Repeatable build for Lambda Layers with Yarn Workspaces

I have pretty big monorepo, managed solely by Yarn Workspaces (no Lerna). One of the packages (“workspaces”) contains a set of 3rd party NodeJS libraries that we use as a shared layer for our Lambda functions, collected as dependencies in package.json of this package. Build script for this package is supposed to collect all dependencies in a zip file that will be later published by Terraform. Unfortunately, Yarn cannot build single workspace from the monorepo, so initially we opted to use NPM directly – copy package.json to a build folder, then run “npm install --production” there and zip the resulting node_modules tree.

My main problem with this approach (besides mixing the build tools) was that the build is not repeatable – each time we run npm install we could get newer compatible version of any dependent package, since the version is “locked” by Yarn in the top-level yarn.lock file and NPM (obviously) is not aware about it. So I decided to dive deeper and see how it can be solved in a better way.

It appears that while Yarn hoists all the dependencies to the node_modules of the top-level workspace, you can explicitly opt-out from this behavior for some dependencies – or, in my case, for all dependencies of the given workspace.

Yarn Workspaces configuration before:

"workspaces": [
  "packages/*"
]

Yarn Workspaces configuration after the change, assuming Lambda Layer dependencies are collected under common-lambda workspace:

"workspaces": {
  "packages": [
    "packages/*"
  ],
  "nohoist": [
    "common-lambda/**"
  ]
}

Note that nohoist array should contain the workspace name (including namespace when applicable) and not the workspace folder.

After this change packages/common-lambda/node_modules will contain proper versions of all the dependencies to be packaged as Lambda Layer. Those dependencies will be updated automatically on yarn install and the node_modules folder can be packaged directly.

Optimization of Lodash styling and bundle size

Lodash is one of the most widely used libraries in modern web development. Whether your webapp is based on Angular, React, Vue or something more exotic – chances that you use Lodash are pretty high. Lodash provides tons of utility methods, making your code more fast and elegant.

However, those tons of useful methods come with a cost – Lodash “weights” more than 70 KB of minified code. This is a significant addition even for the heavy apps. Moreover, chances are you are using just a handful of methods from the whole library. Tree shaking provided by your favorite packing tool could help, but here it comes the next issue – there are several possible ways you can import Lodash in your ES6-based code, potentially making it “unshakable”:

// Method 1:
// import the whole library
import _ from 'lodash';

// Method 2:
// import only relevant methods
import { map, find, filter } from 'lodash';

// Method 3:
// import individual modules
import map from 'lodash/map';
import find from 'lodash/find';
import filter from 'lodash/filter';

Each one of those methods has own pros and cons (and they are described in depth in this excellent article from BlazeMeter), but in many cases personal choice of the developer plays more significant role than formal recommendations. In big projects dealing with the styling of Lodash imports alone may quickly become a mess, and a single “whole library” import will eliminate all your bundle size optimization efforts.

Fortunately, there is a way to handle both size and styling issues together – with the help of Lodash plugins for Eslint and Babel.

Eslint plugin for Lodash enforces consistent and recommended styling for Lodash usage. I’ve found rules coming from plugin:lodash/canonical configuration to be a perfect fit for our React application. This set enforces whole package imports with _ as the name for Lodash variable, enforces usage of Lodash methods over native counterparts when available, and much more. The only caveat I found was in the testing code, where it confused enzyme.find() with native method. Fortunately, we have separate eslint configuration for testing code, so it was pretty easy to tweak the offending rule without sacrificing the style of the application code.

Babel plugin for Lodash implicitly transforms recommended _ imports to per-module imports, reducing “tree-shaked” bundle size. For the app that I was experimenting with, applying both plugins effectively reduced the size of the app bundle by more than 42 KB – but your mileage may vary.

In addition, I spent some time looking at webpack plugin for Lodash, but ultimately decided to skip it. The plugin strips out significant parts of internal Lodash implementation, claiming that those parts are rarely used. While the plugin is configurable, maintaining this configuration will be a pain. Moreover, the code that is unit-tested may be significantly different from the code that is being packaged, especially around edge cases. In my opinion, potential gain in bundle size does not justify the risks of the implementation.

There is one more thing that is worth mentioning in this context. Don’t use Lodash chaining – neither explicit (with _.chain(arr)...) nor implicit (with _(arr)...), since this again will import the whole Lodash library. There are more details about it in this Medium post. In some cases, where chaining seems to be the most simple and elegant way of doing things, see if you can get away with flow and lodash/fp combination:

import { flow, map, compact } from 'lodash/fp';

const result = 
  flow(
    map(...),
    compact
  )(arr);

You can read more about functional programming with Lodash here.

Happy Lodashing!

How to Find a VM by MAC address with VMware PowerCLI

logo-powershellFinding a VM by its MAC address can be a non-trivial task, especially if you have a vCenter with tons of complicated virtual appliances and several network cards on each one of them. Fortunately, VMware has PowerShell CLI that can help here.

First, you will need to install PowerCLI on a Windows box. Check for PowerCLI version you want to use on its PowerShell Gallery page. PowerCLI versions are backward-compatible (PowerCLI v6.5 worked for me on vCenter 5.5), so just pick the latest stable version. On the way, PowerShell may ask you to update NuGet and/or trust installations from PowerShell Gallery.

PS C:\> Install-Module -Name VMware.PowerCLI -RequiredVersion 6.5.4.7155375

NuGet provider is required to continue
PowerShellGet requires NuGet provider version '2.8.5.201' or newer to interact with NuGet-based repositories. The NuGet
 provider must be available in 'C:\Program Files\PackageManagement\ProviderAssemblies' or
'C:\Users\ankhitre\AppData\Local\PackageManagement\ProviderAssemblies'. You can also install the NuGet provider by
running 'Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force'. Do you want PowerShellGet to install
and import the NuGet provider now?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): Y

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from
'PSGallery'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "N"): Y
PS C:\> 

Next step is to import core CLI module. You may need to adjust execution policy to do that – do it on your own risk, and make sure you know what you are doing first! Also, PowerCLI will prompt you to join their data collection program, but you can safely ignore that “warning” for now.

PS C:\> Import-Module VMware.VimAutomation.Core
WARNING: .....
PS C:\> 

Now you can connect to your vCenter:

PS C:\> Connect-VIServer -Server <IP/FQDN> -Protocol https -User <username> -Password <password>

Name                           Port  User
----                           ----  ----
<IP/FQDN>                      443   <user>

PS C:\> 

Note that if you don’t have valid SSL certificate on your vCenter, you will have to configure PowerCLI to deal with that – either prompt you to accept certificate on connection ("Set-PowerCLIConfiguration -InvalidCertificateAction Prompt") or just ignore those warnings altogether ("Set-PowerCLIConfiguration -InvalidCertificateAction Ignore").

Once CLI is connected to vCenter, finding for your VM by MAC address is one step away:

PS C:\> Get-VM | Get-NetworkAdapter | Where {$_.MacAddress -eq “00:50:56:12:34:56”} | Select-Object Parent,Name,MacAddress

Parent              Name                MacAddress
------              ----                  ----------
ubuntu-dev-012      Network adapter 1     00:50:56:12:34:56


PS C:\>

Alternatively, you can “pipe” the result through Format-List instead of Select-Object and get all available properties of this network adapter.

Good luck!

SQL: Find a Row That You Don’t Have

LEFT OUTER JOINSeveral years ago I had to create Oracle DB structure for one of my side projects. One of the data types to be stored contained results of certain observation. Each object had some meta-information (like timestamp, name of the observer, location and so on) plus dynamic collection of various key-value pairs – zero or more per observation. I went with the classic “one to many” pattern – main table with observation metadata and additional table for key-value pairs, connected via observation ID. And this worked pretty well for some time…

OBSERVATIONS:               OBS_VALUES:
+----+-----------+-----+	+--------+-----+-------+
| ID | TIMESTAMP | ... |	| OBS_ID | KEY | VALUE |
+----+-----------+-----+	+--------+-----+-------+
|  1 |           |     |	|      1 | A   |   123 |
|  2 |           |     |	|      1 | B   |   456 |
|  . |           |     |	|      2 | A   |   ... |
|  . |           |     |	|      . |     |       |
+----+-----------+-----+	+--------+-----+-------+

Recently I worked on the next version of that tool. One of the requests that I got was less trivial than others – “we need to find all observations in given time period that do not have key-value records with some given key”. It took me time to recall about outer joins (I rarely work with databases directly nowadays), but the initial version of the query was ready almost instantly:

SELECT obs.id FROM observations obs
LEFT JOIN obs_values v ON obs.id=v.obs_id
WHERE key is null

Unfortunately, this worked only for observations that did not have any key-value pairs at all – the rest of the records had real data rows on the right side of the joined table and did not match the NULL key condition. So, the final version was bit more complex and took much more time to produce. I had to “enrich” main table with additional column containing the key in question (see line 2 below) – and only then I was able to use proper left outer join on both record ID and the key, which resulted in “null” row parts for records with no key in the values table.

SELECT obs.id FROM
(SELECT *,? as key from observations) obs
LEFT JOIN obs_values v ON obs.id=v.obs_id and obs.key=v.key
WHERE v.key is null

It was fun to recall a bit of Oracle world, but I’m glad I don’t have to deal with that in my everyday work 🙂

Eclipse: Configure HTTP Proxy Settings for All Run Configurations

There are many cases nowadays when you have to go through HTTP proxy in order to access the internet. Eclipse provides an option to configure HTTP proxy settings – either by manual configuration or loading native system settings. (It does not support auto-configuration scripts, though, so if this is the way you configure your OS, you’ll have to provide proxy address to Eclipse manually.)

However. the settings configured via Eclipse preferences are not propagated to the Java programs that you develop. Obviously, you can provide proxy settings via VM arguments, but doing that for each and every run configuration may be painful. Fortunately, this can be solved with the help of “Default VM Arguments” setting under JRE/JDK definition. System properties configured there will be in effect for every Java program you are going to run or debug. Continue reading “Eclipse: Configure HTTP Proxy Settings for All Run Configurations”