The human factor in an economy of scale

Back in the days when hosting was exclusively on-prem, two factors had a major impact on time-to-market: software development iterations – how fast can our developers implement these new features, and scalability – how fast can our engineers rack those new servers. For a long time, these were the key factors defining how fast software platforms and software businesses could grow.

This game has changed with the cloud enabling infrastructural scalability within the snap of a finger. Nowadays, management focus often centers on the speed at which developers race their fingers across keyboards, enabling new functionality. Agile and DevOps have certainly increased the pace at which software iterations are produced, tested and deployed. But it is important to keep in mind that – unlike servers scaling up within minutes and fully automatically – software is still developed by humans: experienced developers, with finite time at their hands.

In this article, we will look at a security vulnerability in Canonicals’ Ubuntu Server installer and use this example to gain insight into how Shift Left and a Security Champions program can help us develop software not just faster, but also more securely.

Subiquity, the new Ubuntu Server installer

In late 2017, Canonical Ltd., the company behind Ubuntu (Linux), remodeled its Advantage subscription model. While previously primarily targeting large customers, they have since introduced a collection of add-on services for both on-premise and the cloud, now affordable to smaller players as well. These services now provide even longer commercial support life-cycles, and Live Kernel Patching, a way for mitigating security flaws in the operating system kernel at runtime. Along came a fully redesigned website to further promote the company’s all-cloud strategy, which Canonical had already adopted back in 2012. Back then, they started offering installing and managing Openstack, the scalable run-your-own-cloud-infrastructure software stack, on their customers premises. An attractive offer, since there were good reasons to be worried about the softwares’ complexity which had already driven companies trying to run it themselves into bankruptcy.

In 2017, new OpenStack point releases seemed to offer much better stability and manageability, so that Canonical spent sizable amounts of developer time on making it yet easier to maintain, thus having a chance to extend the reach and profit margin on their offering. So a lot of developer time was needed to realize these improvements and changes, which also affected the plans of the ‘server’ team – the team primarily working on making Ubuntu Server a success. Now, they did not only need to develop a new server installer, a task they had had on their list for a good while, but also spent a majority of their time on enabling Canonicals’ new and remodeled service offerings. As a result, the spring 2018 long term support (LTS) release 18.04 shipped with two server installers: Subiquity, the new, fast and modern installer, though functionally limited and not free from bugs, and the older, less handsome, but reliable and versatile installer Ubuntu had forked from its origins, Debian GNU/Linux.

A tight schedule…

Ubuntu releases twice a year, and long-term support releases take place in April of even years. Major changes are usually expected to be introduced at least one release before an LTS release, which are favored by corporate customers for server deployments. But 19.10, the release before the long-awaited 20.04 LTS release, did not fill all the gaps, yet. And even in early 2020, Canonical customers were wondering whether history would repeat or whether the April 2020 LTS release would now provide a fully featured new server installer. After all, there were only four months left, which would have to include QA, acceptance testing and brushing up documentation.

Work on extending Subiquity had, however, started. For an early feedback loop, the developers had drafted how unattended installations would work and shared these designs with customers and the greater community in autumn 2019 already. Also, thanks to Subiquity being developed and packaged in Canonicals’ new agile-friendly Snap software packaging format, it was exempt from the early arrival rule.

… which was matched …

And while most of the work on extending Subiquity started only in 2020, its developers were actually able to make it land in time – and with feature parity to the old installer for most, if not all, corporate use cases. Subiquity is now a proper OS installer which covers all classic server installation scenarios – which certainly is an impressive work. Most of all, what seems to have enabled the server team transforming Ubuntu Server 20.04 into a mature product is that they could work on it full time for some weeks, enabling the project to progress quickly. As a result, most of the more serious bugs already present in previous releases have since been examined and fixed. Subiquity is now a proper OS installer which covers most server installation scenarios – which certainly is an impressive work.

… if not without loss

To make this happen, the development team working on Subiquity must have been working under intense stress during those few weeks, and may not have found just enough time to fully focus on all the details. As a result, a few bugs have still slipped through. One of them, recently identified by Alice&Bob solution engineer Moritz Naumann, may leak the disk encryption key entered during installation onto unencrypted storage of the installed system. This vulnerability, also known by its ID CVE-2020-11932, has since – and promptly – been fixed in the Subiquity installer software. To work around this security bug, when booting the Ubuntu Server 20.04.0 installer, Subiquity itself can be live upgraded from the Internet. The Ubuntu Server 20.04.1 release (expected in early August 2020) should contain these patches out of the box.

Technical Details

Vulnerability ID: CVE-2020-11932
Product: Subiquity (Ubuntu Server installer software)
Versions affected: ? < 20.05.2
· Ubuntu Security Tracker
· Launchpad: bug report #1878115
· Subiquity commit fixing vulnerability
· Subiquity release fixing vulnerability

It was discovered that the Subiquity installer for Ubuntu Server logged the LUKS full disk encryption password if one was entered.

Let us take a closer look at the details of this vulnerability, because it may bear a potential to learn from it. At some point, when Subiquity developers implemented disk encryption, they were confronted with the need to pass the passphrase entered by the user to Curtin, the software actually carrying out disk partitioning and instrumenting disk encryption. There is a limited number of possibilities to pass such secrets from one software, or process, to another. The method applied here was to have Subiquity write the plain text passphrase into the Curtin configuration file, so that Curtin could read and apply it when starting up. Additionally, all Subiquity actions, including all parameters, such as the encryption passphrase, were logged to the system log as well as its own log file. The latter is easy to fix by just not writing the passphrase to these logs. But passing the passphrase to Curtin did require keeping it somewhere Subiquity could write to and Curtin could read from. Curtins’ configuration file was stored on an overlay file system – effectively a RAM disk, a type of storage which does not survive reboots. This may have seemed like a good enough approach at first. But we can suspect that the developer already noticed this could become problematic later on. After all, it is Subiquity itself which, by the end of the installation, gathers the installation log files and copies them to the installed system, with the very purpose of retaining these logs.

Finite time, and good solutions

So which other options were there, how else could Subiquity have passed the passphrase to Curtin? It could have done so via the /run file system, which it does now. This, too, is a RAM-backed file system, but one that is separate from the main file system the installer runs from. More importantly, no log files are stored there, or backed up from it. So this approach is a little better, but not perfect: it still means writing a secret to a plain text file. Another option would be to have Subiquity store the passphrase in the Kernels’ secret storage, which also protects against direct memory access (thus preventing some forms of physical attacks). And to have the Kernel only hand it out to what it considers the legitimate ‘curtin’ process, and forget about it after the fact.

While this may seem like a technically sound approach, it is also a much more complex one to implement. A middle ground would be the Subiquity process passing the passphrase to Curtin when calling it, or, in case it could be guaranteed that both processes run at the same time, using inter-process communication (IPC), a common approach enabling processes to talk to each other. These two may, however, be more difficult to work out because both Subiquity and Curtin are restricted by Apparmor, a Linux hardening mechanism Ubuntu uses, which could prevent such communication as a generic security measure.

So passing the secret could have been done better – as there is always a better way – but time is not infinite, and it certainly was not for the Subiquity developers before the Ubuntu 20.04 release. A working condition which can make it difficult to identify the very solution which both provides a sufficient level of security and does not introduce a level of complexity that delays the time to market.

Fast clocks make slow developers

We do, however, need to make these decisions about security related software features and properties at speed, and we need make them right. How can we achieve this?

Taking another step back to look at the greater picture may provide an answer: With infrastructure now scaling up at the snip of a finger in the cloud, a process that would often seem to take way too long in the past, managements’ attention has since shifted to what now seems to take the most time: the software development process. While the velocity of software development has remarkably increased in the agile age, it still takes the majority of a product release cycle. And now that hardware is no longer a concern, it receives the primary managerial focus.

It is entirely logical and expected that software development still takes its time, though. Computers can now write their own code, and there are areas where they can match with and win against humans in doing so. However, software which does not just replicate existing use cases or requires long term neural network training is still exclusively composed by human brains. Especially those human brains with long-term software architecting, problem solving, and development experience. These are very special skills, which computers are not good at reproducing and which thus do not scale the way you can scale up and down compute resources in the cloud.

How to improve?

So, all in all, maybe all we need to do is to make ourselves aware that the strong focus on the seemingly slow software development process is strong because many of the surrounding tasks we needed to optimize have been optimized since. And to remind ourselves that the process which takes most of the time now is one which cannot be reasonably automated. We have grown an understanding that the speed of software development is a very relevant but also not the only relevant factor in shipping good software. We need to ask how we can improve upon software quality, and not only by building exquisite CI/CD pipelines applying code coverage reporting, static and dynamic analysis, and by carrying out QA and acceptance testing?

The Shift Left approach, and the DevSecOps methodology can certainly help here. The earlier security is involved in the development process, the more secure software can become. But, as those of us who spent years around software developers and security teams, know very well, these two do not originally go together well. Developers may be security minded, but their primary objective is to deliver functionality, as requested, and before deadlines. Security staff is usually determined to comply with policies, ensure best practices are met, but in doing so, they can seem like those parents you always wanted to break free from. So with the classic approach, bringing the two together is not an easy task.

This is why Alice&Bob.Company recommends introducing a Security Champions program, where encouraging and incentivizing select developers to regularly discuss with and receiving training from security departments leads to broadening both sides’ views on the challenges the other faces in their daily work. This helps the company form a new security culture within development, and lets security engineers grow a higher appreciation for policy-as-code approaches. With this broadened view, developers will identify security flaws more easily, and will appreciate the possibility to further spread their new perspectives within their teams. At the same time, security departments may gain insight into software and tooling they can make use of to accelerate their work without a need to compromise on security standards.

Ensuring this new culture is seeded well, is commonly accepted and appreciated, is a key factor for companies transforming from a DevOps to a DevSecOps culture. Introducing the Security Champions program properly is a delicate task, and it can fail. In the worst case, all a company achieves is to shift to a formal Sec/Dev/Ops environment, where all parties work side by side, but actually remain separate, and do not share a common security focus. What sets companies ahead of their competition, however, is a security focus and understanding that is appreciated and thrived for by everyone. A well-understood and common goal: to create high quality software (and software based services) which enables customer success – without compromising on security, quality, development cost, or time-to-market.

Image credit: One Click Group UK (Source)