This post is inspired by the recent supply chain attack on coa, a popular command-line parser package for NodeJS,. and the discussions that followed that – both on LinkedIn and in the local InfoSec Slack community. This attack was not the first one, and definitely won’t be the last one. It’s true that Node modules are small and bring a lot of sub-dependencies – but they are usually super-focused on what they are doing. The speed of development in Node is largely based on this modularity, which comes with a lot of inherent risks. So, how do you mitigate those risks?
It’s about time to introduce Dependency Hygiene.
TL;DR:
Use lock files.
Pin your dependencies.
Run audits.
Patch frequently – but don’t rush fresh upgrades in.
Now, let’s dive deeper into those items. I’ll explain what do I mean by each one of them and how they are relevant in the mitigation of the modern supply chain attacks.
My primary working environment nowadays is NodeJS-based, so the examples will come from this world, but most of what I’m going to tell applies to other ecosystems as well.
1. Lock Files
I don’t think this is really an open question nowadays. The need to preserve that exact state of your dependency tree (repeatable builds anyone?) was apparent since before I heard about NodeJS for the first time.. The half-baked npm shrinkwrap was there to help, but creating the lock files was not a mandatory part of the process and those often fell out of sync. Yarn was out in 2016 to solve that and other needs via introduction of yarn.lock – and Yarn v1 release post explains all the whys and hows really well. NPM had no choice but to catch up, and NPM v5 included their own package-lock.json, which was really baked in this time.
While repeatable builds are important, it’s even more important to make sure your are not pulling in a fresh malware that was just pushed to the NPM registry by a malicious actor. A lot of small, but widely used packages are so simple and mature that there was no need to update them in years (literally), and their NPM publisher accounts are often still missing the essential 2FA protection, making account takeovers easier than they are supposed to be.
Pay attention that in NodeJS it’s not enough to create the lock file and commit it to the source control system. Your build server should be instructed to verify that what appears in the lock file actually matches the content of package.json and fail if it doesn’t. You can do that with "npm ci" or "yarn install --frozen-lockfile” commands accordingly. (Note that at the moment of writing both NPM v7 and v8 are buggy and do not fail on mismatch, so it’s better to stick with NPM v6).
I’m talking mostly about NodeJS, but any mature ecosystem – from Python to Terraform – now employs own flavor of lock files with the exact version of all the dependencies that were used, whether those are Python libraries of Terraform providers.
2. Dependency Pinning
Dependency “pinning” means that all your dependencies appear in the list with the exact version you plan to use – no ranges, fuzzy logic or other wildcards, please. Renovate has a great article about why you should pin your dependencies, and there is no reason to copy it all here. It is worth mentioning, though, that this primarily applies to a product that you are developing and is less relevant if you are developing a library for other products to use.
One may claim – we have all the versions written down in the lock file, so why bother? Well, sometimes upgrade tasks fail behind (spoiler: I’ll talk about how to avoid this a bit later) and it might be tempting to “upgrade ’em all” in a single commit. If such a “mass upgrade” commit fails in CI, it will be a pain to find out what out which one of the bumps was responsible for the failure – but from security perspective you’ve just lost visibility over what was pulled in, mostly because no human is able to meaningfully review a massive change in the lock file!
3. Security Audits
Both NPM and Yarn provide access to NPM security advisories database (which was recently migrated over to GitHub) via the built-in “npm audit” and "yarn audit" commands respectively. Both commands return non-zero status when a known vulnerability is detected in your repository, making it really trivial to integrate them in the regular build process.
There is one caveat, though. Sometimes it may take days for the fix to become available. Typically you don’t want to block the whole CI pipeline while the audit is red, otherwise developers won’t be able to carry their daily duties. On another hand, the very fact of the failing audit should be prominent enough, so it won’t be easily ignored. Finding the right balance between those needs might be tough.
There is much more than can be said about how to make most of your dependency security audit process, but this deserves its own post that I hope to publish some time later.
4. Dependency Upgrades
The days of manual dependency upgrades are largely over. There are excellent tools available for free to the users of GitHub, GitLab and other platforms that can automate a big chunk of the process. Yes, I’m talking about Dependabot and Renovate. Dependabot, which is now owned by GitHub, lays in the core of GitHub security features and is great for small open-source projects with minimal maintenance required. Renovate, acquired by Whitesource pretty much at the same time, is more powerful and flexible, which comes with some setup cost, and is more suited for private organizations. Both are available as GitHub Apps, Renovate can also run as a command-line tool.
Confession: I’m with Renovate since last March, and I never looked back – so I’m going to talk about Renovate. Some of the configuration choices I’m going to mention were not available to Dependabot users last time I checked.
Patch Frequently
First, don’t delay the dependency upgrades. They tend to accumulate, and at the moment of truth you won’t be able to patch a vulnerability quickly because of the gazillion of other upgrades, sometimes with breaking changes, that it will carry. You want to create pull requests (aka PRs) for the new versions of your dependencies automatically and run them through the same set of checks you run your regular code commits through. Green checks should be a sufficient indicator that the upgrade is good to be merged. (Not sure about it? Then you have more serious problems with your CI system to solve first!..) Typically anyone with the approval permissions can approve and merge the upgrade since the PR was created by “somebody else” – in this case the upgrade bot.
How does one deal with the volume of upgrades in a big project? A lot of options here, actually. You can limit the number of PRs that Renovate may keep open at the same time and the number of PRs that Renovate may open in a time period. You can configure “noisy” dependencies (like AWS SDK, which is bumped almost daily) to be upgraded once a week or once a month. You can group dependencies that must be upgraded together in a single PR (and a lot of those Renovate will group for you automatically). You can define priorities to specific dependencies. Renovate is really flexible here, and the Dependency Dashboard really helps to keep track of things.
Don’t Rush Upgrades
I’ve just convinced you to upgrade frequently, and now I’m telling that you shouldn’t? Well, not exactly – “frequently” does not mean “immediately”. There is a couple of good reasons to wait several days before taking the fresh version of a dependency into your production build. First, NPM packages may be unpublished within 72 hours. Second, sometimes new versions contain new bugs, which are being fixed in the patch versions published shortly after – so save yourself from the hassle of running into fresh bugs or double upgrades. Last and the most important, fresh packages might contain fresh vulnerabilities – remember the coa upgrade I’ve started this post from?
Fortunately, Renovate can help here also. Setting “stability days” to 3 (or any other amount of days you feel comfortable with) will create an additional check on the PR, which was “pass” only after the stability period has passed. Moreover, configuring PR creation to “non-pending” will prevent creation of this PR altogether – what’s the point to create a PR that you cannot merge anyway?
I hope it helped. Please share your thoughts on dependency hygiene in comments.
P.S. While I was working on this post, GitHub shared their commitment to npm ecosystem security post, where they announced upcoming 2FA requirements for publishers of popular packages, and at the same time disclosed a vulnerability that would allow an attacker to publish new versions of any package using an account without proper authorization. What a coincidence…
