Migrating Cloud-Init to Ignition

Cloud-init is the most common software used to configure cloud instances during runtime. In the devops ‘bake’ or ‘fry’ image paradigm it’s the frypan. Anyone who has ever launched a cloud instance on Amazon, Google, or OpenStack should be well acquainted with it. In short it’s an extremely basic configuration management system that’s available to bootstrap a cloud instance. It’s the perfect example of a technology that’s ‘just good enough to get the job done, but barely’ & one of those things people just use because it’s there. It’s quite common for people to use cloud-init to initialize more robust CM software such as Chef or Ansible.

I was setting about to bump the version on a Container Linux-based Kubernetes install when I noticed a note that CoreOS was in the process of deprecating user_data configurations. The exact definition of the word ‘deprecation’ is fairly vague but ultimately it’s a promise that your currently working system will randomly break in the future unless you act. Having a bit of time I decided to dig further into the situation.

Anyone using cloud systems will undoubtedly be well acquainted with utilizing cloud-init in its’ various names and forms. As it’s been quite popular for some time its usage has been extended to virtual machines as well as bare metal, as such It’s been one of the primary methods for installing Container Linux. Unfortunately as with any vetted technology whose usage has gone past the original intent, limitations are bound to appear. My primary complaints have been file size and formatting. Once you go past the single application setup the files can get quite lengthy. Additionally, there no way to validate your formatting without actual testing. I’ve witnessed a coworker lose several days of their life debugging a single yaml file which ultimately has a couple spaces at the end of a line. So it goes without saying that there improvements that can be made.

CoreOS has decided to address these shortcomings with a couple of projects – ignition and ct (container transpiler). Ignition is their replacement for cloud-init. At first glance it seems like just a formatting shift for cloud-init, but once you dig in significant differences appear. Ignition leverages JSON configurations which are not the easiest thing for humans to create. To make things easier an intermediary yaml specification was created to minimize the bracket matching and effort involved in creating a configuration. As part of this tool you get checks to ensure that your yaml renders into a valid ignition files and useful errors as to where your problem lies, all before you boot anything. Also worthy of mention is matchbox for managing the hardware provisioning, but it’s a bit out of scope for this discussion.

So what improvements does Ignition bring with it? The most obvious is that it runs much earlier in the startup process. Its executed during the initrd prior to the root filesystem mount instead of just running as normal system service. This lets you ensure all your disks, raid volumes, and filesystems mounts are ready to go before init kicks in. And since all your systemd files are in place from the start all dependencies will be correctly honored from the very start. Ultimately this means fewer reboots (any?) during your setup & configuration.

The CT configuration file format also provides a number of improvements. Since its sole focus is Container Linux special directives has have been added for the most commonly utilized components such as etcd, docker, and locksmith. Extra commands such as hostname are no longer needed due to the early exec time and can replaced with just simple file commands. File contests can still be included inline but there’s also an option for loading file contents via http which does a remarkable job of keeping file sizes to a manageable size. Hashes can be used to ensure your content isn’t mangled on the way to the disk. Overall the process isn’t too different that what most people are used to, but the number of small changes make everything much easier to use and manage.

Migrating to the new toolset is fairly painless once you understand each component’s role. First you need to download the CT tool from github. Then comes the task of converting your old userdata file to the new format. A migration page is provided to assist with the most common use cases, more often than not it’s a one to one mapping. The only thing that wasn’t obvious at first glance was the root file definition. Its automatically defined as ‘root’ for file references, if you redefine it CT will simply output a warning. Converting a kubernetes master userdata yaml file took about 30 minutes, most of which was spent referencing documentation. Once the ignition file was rendered I copied it to my server as /usr/share/oem/igntion.json and deleted the old ‘/var/lib/coreos-install/user_data. Additionally I added a /usr/share/oem/grub.cfg with the line

set linux_append=”coreos.config.url=oem://ignition.json”

to tell the initrd where to find the configuration file. A couple reboots later (yes, I managed to mangle a number of urls and such during the conversion process) and all the kinks were worked out. If I was managing more than a couple machines I’d inevitably want leverage PXE boot for all of this and that’s where matchbox would provide benefits.

My conclusion is fairly simple: Adopting Ignition is a no brainer for anyone who’s using Container Linux. Its straightforward to learn plus you get features, flexibility, and simpler configuration files to manage. The drawbacks? You’ve officially abandoned any hope of cross OS compatibility. That’s likely minor considering your applications are bound to be container targeted and any distro differences require provisioning be fairly specific anyway.


Cloud init:

CT github:
Ignition github:
Matchbox github:

CT migration:
CT spec:

Contact Solinea for More Information

Author: Wil Reichert, Director of Delivery, Solinea