Posted on Wed 18 June 2014

Where are the sample config files?

This article is written from an OpenStack operator's perspective.

During the OpenStack Icehouse cycle, Clint Byrum suggested removing all auto-generated configuration file samples from all repositories.

The reason given is that outdated sample config files broke the gate several times in the past due to configuration changes made to shared libraries (such as the keystone authtoken middleware to only name one). This is because 3rd party libraries (using oslo.config) are able to inject their own configuration options into the generated sample config file.

To prevent sample config files from going stalled, each project has a test comparing the current sample config file to a newly generated one. If both are not identical, an error is raised ; forcing the developers to generate a new sample config file before being able to commit anything again. This is bad because each new release of those libraries could modify the content of the generated configuration file, outdating the previous one.

Robert Collins explains why generating sample config files ain't easy:

Note that because libraries now export config options (which is the root of this problem!) you cannot ever know from the source all the options for a service - you must know the library versions you are running, to interrogate them for their options.

However, Tom Fifield who has experience as an OpenStack operator, got concerned about the removal of the sample config files which he used to refer to as an operator:

When I was an operator, I regularly referred to the sample config files in the git repository.

If there weren't generated versions of the sample config in the repo, I would probably grep the code (not an ideal user experience!). Running some random script that I don't know about the existence and might depend on having something else installed of is probably not something that would happen.

And so is Michael Chapman:

I fairly consistently refer to the github copies of the samples. They also allow me to refer to specific lines of the config when explaining concepts over text. I am not against their removal, but if we were to remove them I'd disappointed if I had to search very far on docs.openstack.org to get to them, and I would want the raw files instead of something formatted.

Tom Fifield took the initiative to inform people on the openstack-operators mailinglist that Cinder was about to also remove its sample config file. He asked operators to express their opinions and concerns about this change.

Still, Matt Riedemann explains again the reasoning being the removal in the review:

Nova removed it's sample conf in Icehouse because it was constantly being broken in the gate and blocking out all patches from getting through Jenkins until fixed, we can't operate like that every time python-keystoneclient or oslo.messaging puts out a release.

He also adds:

my team does our own packaging internally and generates the config files during the rpmbuild process so the latest configs are always available to us every day from our CD builds.

Unfortunately, not all OpenStack operators have the resources or needs to have their own internal packaging system. In fact, they more than often depend on others (such as Linux distributions) to maintain and build them. It is unreasonable to expect operators to build an entire (or even partial) packaging system to have access to the latest sample config files.

Note that prior to this discussion, nova.conf.sample was removed from the Nova repository in favor of a note explaining how to (re)generate it:

To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox -egenconfig

Although it sounds like a great idea from the point of view of a developer, asking an operator to use tox to generate a sample config file is unreasonable. It assumes prior Python knowledge which operators often don't have.

To quote my own words from the Cinder review:

You know Python and its tools. I happen to know Python due to a strong programming background, Cinder developers know Python. My coworkers are OpenStack operators, not developers. They don't know Python, tox and its tools.

[...]

Having to install tox and all Cinder dependencies (which can be a challenge by itself on some platforms) to generate a sample configuration file is not an option for them. They used to have a single URL they could consult to have the latest sample, now it's gone.

This raises the barrier of entry for them to get the latest and greatest information and documentation about Cinder. They don't want to go through hoops to get back what they used to have in a single click. That's their realities.

Sean Dague shares the same point of view on the mailinglist:

Because we've already got a working tox environment. Which includes knowing, in advance that you can't just pip install tox (as 1.7.x is broken), and that you need to have postgresql and mysql and libffi dev packages installed, and a C compiler.

Start with a pristine Linux it is a lot manual steps you have to go through to get a working config out of tox -e genconfig.

So I think it's a fair concern that we did just move a burden back onto users because we dug a hole by letting libraries declare arbitrary required variables in our config files.

Where do we go from here?

TL;DR: There is no clear solution.

As Robert Collins stated, one of the root cause of the problem is that 3rd party libraries are able to export configs through oslo.config. This means the resulting content of a sample config file highly depends on the installed versions of those libraries.

A sample config file generated today might be different tomorrow. It might also contain configs not yet available in your production environment due to outdated libraries.

A more perverted side-effect is that libraries could also export configs still not used by the application itself. This could mislead the user into thinking those configs are supported and could be used. One clear example of such configuration option is slave_connection from oslo.db:

Handle slave database connection in EngineFacade

Make it possible to use an additional (aka, 'slave') database connection. This might be useful for offloading of read operations to reduce the load on the RDBMS. Currently, this is only used in Nova, but can be ported to other projects easily.

Make the application responsible of exporting supported configs

A solution to this problem would be to move the responsibility of exporting such configs to the application itself. The application knows better about what is supported and what is not. This paradigm however increases the risk of inconsistency across OpenStack projects which people are still trying to fix as of today. One project could very well name its configuration options differently from the others.

Those kinds of inconsistency and discrepancy between OpenStack projects often result in headaches for operators and other projects trying to automate the deployment of OpenStack. So keeping configuration options consistency between OpenStack projects could be a challenge in itself.

Ignore 3rd party library configs from sample config files

An alternative approach would be to ignore configs exported by 3rd party libraries so we don't end up with outdated sample config files over time.

Ignoring 3rd party library configs has serious downsides, one being that operators won't have a access to a clear and rapid overview of all the available configuration options for a particular project. Those "missing" options are often essential for the application to work properly: keystone authtoken middleware, database and messaging being the main ones.

OpenStack Configuration Reference manual

The code is always the canonical source of documentation and information. Experience shows that documentation is often outdated or missing information compared to the latest version. Any OpenStack documentation made available is created after developers wrote the code by people doing their best to understand or reverse engineer the code.

In the cinder.conf.sample removal review, Matt Riedemann suggests consulting 2 pages in the OpenStack Configuration Reference - Icehouse manual but couldn't tell the difference between those 2 pages:

The first link is missing a lot of configurations which all look related to driver configurations. In fact, you will find them split across multiple pages in the driver sections. As an operator, you might not know it.

If you happen to run OpenStack Havana, there is no Cinder configuration options overview available in the Havana version of the manual. Someone already pointed it out in the comment section and the bug is still pending resolution or never got opened.

An other example, the Configuration options page from the Compute section in the OpenStack Configuration Reference - Havana manual is missing the novncproxy_host option as pointed out by someone in the comment section, and its still missing as of today.

Periodic jobs keeping sample config files in sync

This solution is a middleground. Instead of relying on a human to (re)generate the latest sample configuration files, a periodic job would be responsible of doing it once in a while to keep them in sync with new options introduced by 3rd party libraries.

The sample config files don't have to live in the repository if this is found to be a problem by some people. As long as it's published somewhere anyone can refer to, with ability to link to specific section, it's fine.

Conclusion

As you can see, the operators experience is not great when it comes to find a place where ALL configurations are listed. There is unfortunately no clear solution to address this problem in its entirety, they all have downsides or side-effects. Thanks to Tom Fifield for adding OpenStack operators in the loop, it triggered a lot of interesting discussions and things are moving to improve the situation.

Whatever solutions we end up coming up with, I strongly believe we should involve OpenStack operators in the discussion so we can find a solution that satisfies everyone's needs.

© Mathieu Gagné. Built using Pelican. Theme by Giulio Fidente on github.