Do not follow HTTP redirects with Gaxios

Earlier today I was enhancing some relatively old piece of NodeJS code, so I decided to convert it from axios to gaxios along the way. Most of the work was pretty transparent, since much smaller and better maintained gaxios is pretty much a drop-in replacement for axios, which seems to be stuck in it’s v0.x days forever.

However, there was one request that stood apart.

This request actually resulted in HTTP redirect, and my goal was to intercept the target URL and extract some query string parameters. In axios, that was straight-forward – it was enough to allow at most 0 redirects, and also make sure that HTTP 302 is not treated as an error:

const axios = require('axios');

const response = await axios.request({
  ...
  maxRedirects: 0,
  validateStatus: (status) => (status === 302)
});
const url = response.headers.location;

Yet, the same configuration produced different outcome in gaxios – the request was failing with "max redirects reached" error. My attempt to adjust maxRedirects value to 1 along with keeping status validation in place was not successful either – gaxios just issued another request to the new target.

To my surprise, quick googling did not reveal any relevant results. I took that as a hint that this is not only doable, but likely so trivial that nobody had to ask – so I continued with the digging. One of the merged PRs caused me to realize that gaxios is basically a wrapper on top of the Fetch API, and the configuration object is passed down to fetch implementation with minimal modifications. From here, the solution was simple –

const { request } = require('gaxios');

const response = await request({
  ...
  redirect: 'manual',
  validateStatus: (status) => (status === 302)
});
const url = response.headers.location;

Enjoy!

Repeatable build for Lambda Layers with Yarn Workspaces

I have pretty big monorepo, managed solely by Yarn Workspaces (no Lerna). One of the packages (“workspaces”) contains a set of 3rd party NodeJS libraries that we use as a shared layer for our Lambda functions, collected as dependencies in package.json of this package. Build script for this package is supposed to collect all dependencies in a zip file that will be later published by Terraform. Unfortunately, Yarn cannot build single workspace from the monorepo, so initially we opted to use NPM directly – copy package.json to a build folder, then run “npm install --production” there and zip the resulting node_modules tree.

My main problem with this approach (besides mixing the build tools) was that the build is not repeatable – each time we run npm install we could get newer compatible version of any dependent package, since the version is “locked” by Yarn in the top-level yarn.lock file and NPM (obviously) is not aware about it. So I decided to dive deeper and see how it can be solved in a better way.

It appears that while Yarn hoists all the dependencies to the node_modules of the top-level workspace, you can explicitly opt-out from this behavior for some dependencies – or, in my case, for all dependencies of the given workspace.

Yarn Workspaces configuration before:

"workspaces": [
  "packages/*"
]

Yarn Workspaces configuration after the change, assuming Lambda Layer dependencies are collected under common-lambda workspace:

"workspaces": {
  "packages": [
    "packages/*"
  ],
  "nohoist": [
    "common-lambda/**"
  ]
}

Note that nohoist array should contain the workspace name (including namespace when applicable) and not the workspace folder.

After this change packages/common-lambda/node_modules will contain proper versions of all the dependencies to be packaged as Lambda Layer. Those dependencies will be updated automatically on yarn install and the node_modules folder can be packaged directly.

How to Find a VM by MAC address with VMware PowerCLI

logo-powershellFinding a VM by its MAC address can be a non-trivial task, especially if you have a vCenter with tons of complicated virtual appliances and several network cards on each one of them. Fortunately, VMware has PowerShell CLI that can help here.

First, you will need to install PowerCLI on a Windows box. Check for PowerCLI version you want to use on its PowerShell Gallery page. PowerCLI versions are backward-compatible (PowerCLI v6.5 worked for me on vCenter 5.5), so just pick the latest stable version. On the way, PowerShell may ask you to update NuGet and/or trust installations from PowerShell Gallery.

PS C:\> Install-Module -Name VMware.PowerCLI -RequiredVersion 6.5.4.7155375

NuGet provider is required to continue
PowerShellGet requires NuGet provider version '2.8.5.201' or newer to interact with NuGet-based repositories. The NuGet
 provider must be available in 'C:\Program Files\PackageManagement\ProviderAssemblies' or
'C:\Users\ankhitre\AppData\Local\PackageManagement\ProviderAssemblies'. You can also install the NuGet provider by
running 'Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force'. Do you want PowerShellGet to install
and import the NuGet provider now?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): Y

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from
'PSGallery'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "N"): Y
PS C:\> 

Next step is to import core CLI module. You may need to adjust execution policy to do that – do it on your own risk, and make sure you know what you are doing first! Also, PowerCLI will prompt you to join their data collection program, but you can safely ignore that “warning” for now.

PS C:\> Import-Module VMware.VimAutomation.Core
WARNING: .....
PS C:\> 

Now you can connect to your vCenter:

PS C:\> Connect-VIServer -Server <IP/FQDN> -Protocol https -User <username> -Password <password>

Name                           Port  User
----                           ----  ----
<IP/FQDN>                      443   <user>

PS C:\> 

Note that if you don’t have valid SSL certificate on your vCenter, you will have to configure PowerCLI to deal with that – either prompt you to accept certificate on connection ("Set-PowerCLIConfiguration -InvalidCertificateAction Prompt") or just ignore those warnings altogether ("Set-PowerCLIConfiguration -InvalidCertificateAction Ignore").

Once CLI is connected to vCenter, finding for your VM by MAC address is one step away:

PS C:\> Get-VM | Get-NetworkAdapter | Where {$_.MacAddress -eq “00:50:56:12:34:56”} | Select-Object Parent,Name,MacAddress

Parent              Name                MacAddress
------              ----                  ----------
ubuntu-dev-012      Network adapter 1     00:50:56:12:34:56


PS C:\>

Alternatively, you can “pipe” the result through Format-List instead of Select-Object and get all available properties of this network adapter.

Good luck!

SQL: Find a Row That You Don’t Have

LEFT OUTER JOINSeveral years ago I had to create Oracle DB structure for one of my side projects. One of the data types to be stored contained results of certain observation. Each object had some meta-information (like timestamp, name of the observer, location and so on) plus dynamic collection of various key-value pairs – zero or more per observation. I went with the classic “one to many” pattern – main table with observation metadata and additional table for key-value pairs, connected via observation ID. And this worked pretty well for some time…

OBSERVATIONS:               OBS_VALUES:
+----+-----------+-----+	+--------+-----+-------+
| ID | TIMESTAMP | ... |	| OBS_ID | KEY | VALUE |
+----+-----------+-----+	+--------+-----+-------+
|  1 |           |     |	|      1 | A   |   123 |
|  2 |           |     |	|      1 | B   |   456 |
|  . |           |     |	|      2 | A   |   ... |
|  . |           |     |	|      . |     |       |
+----+-----------+-----+	+--------+-----+-------+

Recently I worked on the next version of that tool. One of the requests that I got was less trivial than others – “we need to find all observations in given time period that do not have key-value records with some given key”. It took me time to recall about outer joins (I rarely work with databases directly nowadays), but the initial version of the query was ready almost instantly:

SELECT obs.id FROM observations obs
LEFT JOIN obs_values v ON obs.id=v.obs_id
WHERE key is null

Unfortunately, this worked only for observations that did not have any key-value pairs at all – the rest of the records had real data rows on the right side of the joined table and did not match the NULL key condition. So, the final version was bit more complex and took much more time to produce. I had to “enrich” main table with additional column containing the key in question (see line 2 below) – and only then I was able to use proper left outer join on both record ID and the key, which resulted in “null” row parts for records with no key in the values table.

SELECT obs.id FROM
(SELECT *,? as key from observations) obs
LEFT JOIN obs_values v ON obs.id=v.obs_id and obs.key=v.key
WHERE v.key is null

It was fun to recall a bit of Oracle world, but I’m glad I don’t have to deal with that in my everyday work 🙂

Eclipse: Configure HTTP Proxy Settings for All Run Configurations

There are many cases nowadays when you have to go through HTTP proxy in order to access the internet. Eclipse provides an option to configure HTTP proxy settings – either by manual configuration or loading native system settings. (It does not support auto-configuration scripts, though, so if this is the way you configure your OS, you’ll have to provide proxy address to Eclipse manually.)

However. the settings configured via Eclipse preferences are not propagated to the Java programs that you develop. Obviously, you can provide proxy settings via VM arguments, but doing that for each and every run configuration may be painful. Fortunately, this can be solved with the help of “Default VM Arguments” setting under JRE/JDK definition. System properties configured there will be in effect for every Java program you are going to run or debug. Continue reading “Eclipse: Configure HTTP Proxy Settings for All Run Configurations”