Repeatable build for Lambda Layers with Yarn Workspaces

I have pretty big monorepo, managed solely by Yarn Workspaces (no Lerna). One of the packages (“workspaces”) contains a set of 3rd party NodeJS libraries that we use as a shared layer for our Lambda functions, collected as dependencies in package.json of this package. Build script for this package is supposed to collect all dependencies in a zip file that will be later published by Terraform. Unfortunately, Yarn cannot build single workspace from the monorepo, so initially we opted to use NPM directly – copy package.json to a build folder, then run “npm install --production” there and zip the resulting node_modules tree.

My main problem with this approach (besides mixing the build tools) was that the build is not repeatable – each time we run npm install we could get newer compatible version of any dependent package, since the version is “locked” by Yarn in the top-level yarn.lock file and NPM (obviously) is not aware about it. So I decided to dive deeper and see how it can be solved in a better way.

It appears that while Yarn hoists all the dependencies to the node_modules of the top-level workspace, you can explicitly opt-out from this behavior for some dependencies – or, in my case, for all dependencies of the given workspace.

Yarn Workspaces configuration before:

"workspaces": [
  "packages/*"
]

Yarn Workspaces configuration after the change, assuming Lambda Layer dependencies are collected under common-lambda workspace:

"workspaces": {
  "packages": [
    "packages/*"
  ],
  "nohoist": [
    "common-lambda/**"
  ]
}

Note that nohoist array should contain the workspace name (including namespace when applicable) and not the workspace folder.

After this change packages/common-lambda/node_modules will contain proper versions of all the dependencies to be packaged as Lambda Layer. Those dependencies will be updated automatically on yarn install and the node_modules folder can be packaged directly.

How to Find a VM by MAC address with VMware PowerCLI

logo-powershellFinding a VM by its MAC address can be a non-trivial task, especially if you have a vCenter with tons of complicated virtual appliances and several network cards on each one of them. Fortunately, VMware has PowerShell CLI that can help here.

First, you will need to install PowerCLI on a Windows box. Check for PowerCLI version you want to use on its PowerShell Gallery page. PowerCLI versions are backward-compatible (PowerCLI v6.5 worked for me on vCenter 5.5), so just pick the latest stable version. On the way, PowerShell may ask you to update NuGet and/or trust installations from PowerShell Gallery.

PS C:\> Install-Module -Name VMware.PowerCLI -RequiredVersion 6.5.4.7155375

NuGet provider is required to continue
PowerShellGet requires NuGet provider version '2.8.5.201' or newer to interact with NuGet-based repositories. The NuGet
 provider must be available in 'C:\Program Files\PackageManagement\ProviderAssemblies' or
'C:\Users\ankhitre\AppData\Local\PackageManagement\ProviderAssemblies'. You can also install the NuGet provider by
running 'Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force'. Do you want PowerShellGet to install
and import the NuGet provider now?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): Y

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from
'PSGallery'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "N"): Y
PS C:\> 

Next step is to import core CLI module. You may need to adjust execution policy to do that – do it on your own risk, and make sure you know what you are doing first! Also, PowerCLI will prompt you to join their data collection program, but you can safely ignore that “warning” for now.

PS C:\> Import-Module VMware.VimAutomation.Core
WARNING: .....
PS C:\> 

Now you can connect to your vCenter:

PS C:\> Connect-VIServer -Server <IP/FQDN> -Protocol https -User <username> -Password <password>

Name                           Port  User
----                           ----  ----
<IP/FQDN>                      443   <user>

PS C:\> 

Note that if you don’t have valid SSL certificate on your vCenter, you will have to configure PowerCLI to deal with that – either prompt you to accept certificate on connection ("Set-PowerCLIConfiguration -InvalidCertificateAction Prompt") or just ignore those warnings altogether ("Set-PowerCLIConfiguration -InvalidCertificateAction Ignore").

Once CLI is connected to vCenter, finding for your VM by MAC address is one step away:

PS C:\> Get-VM | Get-NetworkAdapter | Where {$_.MacAddress -eq “00:50:56:12:34:56”} | Select-Object Parent,Name,MacAddress

Parent              Name                MacAddress
------              ----                  ----------
ubuntu-dev-012      Network adapter 1     00:50:56:12:34:56


PS C:\>

Alternatively, you can “pipe” the result through Format-List instead of Select-Object and get all available properties of this network adapter.

Good luck!

SQL: Find a Row That You Don’t Have

LEFT OUTER JOINSeveral years ago I had to create Oracle DB structure for one of my side projects. One of the data types to be stored contained results of certain observation. Each object had some meta-information (like timestamp, name of the observer, location and so on) plus dynamic collection of various key-value pairs – zero or more per observation. I went with the classic “one to many” pattern – main table with observation metadata and additional table for key-value pairs, connected via observation ID. And this worked pretty well for some time…

OBSERVATIONS:               OBS_VALUES:
+----+-----------+-----+	+--------+-----+-------+
| ID | TIMESTAMP | ... |	| OBS_ID | KEY | VALUE |
+----+-----------+-----+	+--------+-----+-------+
|  1 |           |     |	|      1 | A   |   123 |
|  2 |           |     |	|      1 | B   |   456 |
|  . |           |     |	|      2 | A   |   ... |
|  . |           |     |	|      . |     |       |
+----+-----------+-----+	+--------+-----+-------+

Recently I worked on the next version of that tool. One of the requests that I got was less trivial than others – “we need to find all observations in given time period that do not have key-value records with some given key”. It took me time to recall about outer joins (I rarely work with databases directly nowadays), but the initial version of the query was ready almost instantly:

SELECT obs.id FROM observations obs
LEFT JOIN obs_values v ON obs.id=v.obs_id
WHERE key is null

Unfortunately, this worked only for observations that did not have any key-value pairs at all – the rest of the records had real data rows on the right side of the joined table and did not match the NULL key condition. So, the final version was bit more complex and took much more time to produce. I had to “enrich” main table with additional column containing the key in question (see line 2 below) – and only then I was able to use proper left outer join on both record ID and the key, which resulted in “null” row parts for records with no key in the values table.

SELECT obs.id FROM
(SELECT *,? as key from observations) obs
LEFT JOIN obs_values v ON obs.id=v.obs_id and obs.key=v.key
WHERE v.key is null

It was fun to recall a bit of Oracle world, but I’m glad I don’t have to deal with that in my everyday work 🙂

Eclipse: Configure HTTP Proxy Settings for All Run Configurations

There are many cases nowadays when you have to go through HTTP proxy in order to access the internet. Eclipse provides an option to configure HTTP proxy settings – either by manual configuration or loading native system settings. (It does not support auto-configuration scripts, though, so if this is the way you configure your OS, you’ll have to provide proxy address to Eclipse manually.)

However. the settings configured via Eclipse preferences are not propagated to the Java programs that you develop. Obviously, you can provide proxy settings via VM arguments, but doing that for each and every run configuration may be painful. Fortunately, this can be solved with the help of “Default VM Arguments” setting under JRE/JDK definition. System properties configured there will be in effect for every Java program you are going to run or debug. Continue reading “Eclipse: Configure HTTP Proxy Settings for All Run Configurations”

Summary of Cache-Related HTTP Headers

Long ago (long before the first post in this blog!) I’ve composed a list of cache-related HTTP headers, so I would not need to go through the trial-and-error process of guessing the right combination more than once. Recently I got another question about caching and it took me a lot of time to recall where I saw this list last time. So now I’m placing it here.

Please treat the explanations below as quick and incomplete summary. For full specification of “Pragma”, “Cache-Control” and “Expires” headers refer to HTTP/1.1 specification.

Caching in HTTP 1.1

Following directive does not prevent caching despite its name. It allows caching of the page, but specifies that the cache must ask the originating web server if the page is up-to-date before serving the cached version. So the cached page can still be served up if the originating web server says so. Applies to all caches.

Cache-Control: no-cache

Following directive tells the browser that the page has expired and must be treated as stale. Should be good news as long as the caches obey.

Expires: Thu, 01 Jan 1970 00:00:00 GMT

Following directive specifies that the page contains information intended for a single user only and must not be cached by a shared cache (e.g. a proxy server).

Cache-Control: private

Following directive specifies that a cache must not store any part of the response or the request that elicited it.

Cache-Control: no-store

Following directive tells the cache that the maximum acceptable staleness of a page is 0 seconds.

Cache-Control: max-stale=0

Caching in HTTP 1.0

Following directive is the only cache control directive for HTTP 1.0, so use it in addition to any HTTP 1.1 cache control headers you include.

Pragma: no-cache

Continue reading “Summary of Cache-Related HTTP Headers”