When working with an OpenStack installer, it is quite common to see changes happening in upstream projects break it. Options get deprecated every other day, new options show up, projects add dependencies… After all, this is one of the main reasons for RDO to have a continuous packaging pipeline coupled with CI jobs testing the latest commits.
However, in some cases we see some cross-project changes landing, and that takes quite a lot of coordination to happen, specially once we get passed the “it worked with DevStack!” point. As an example, I will discuss the switch from the MySQL-Python library to PyMySQL that happened during the Liberty cycle. We will follow the process from the initial decision until it got implemented in all projects (and DevStack), and then what it took us to get that change propagated into RDO and in the hands of our users.
The change
During the Liberty Summit, there was a cross-project session to discuss the Python 3 effort porting for OpenStack. During this discussion, the MySQL-Python library was found to be a blocker, as it is Python2-only. PyMySQL was found to be a better alternative, so a decision was made: let’s all switch to PyMySQL!
The upstream implementation
So a plan was prepared to do the switch, and it was quickly implemented:
- Early June, DevStack was updated to include PyMySQL as the default library, and it settled after fixing some issues with projects that did not work well with it.
- Around the same time, oslo.db was updated to use PyMySQL by default.
- …and between June and July 2015, all projects transitioned to the new library.
So, in July 2015, all projects had successfully transitioned to PyMySQL, were using it in their CI systems, and everyone was happy.
So that’s is, right?
Well, not really. And this reflects an issue we in the OpenStack distro space find quite often. Changing the project code and updating DevStack only takes you to a point where you can develop using the new code. If you install from source you may use it, provided you have updated your configuration files. But what about users?
To illustrate the situation, let’s have a look at the following diagram, where I describe (at a high level) what is the usual workflow for such a change, from the OpenStack distro point of view.
(Click image to enlarge)
So, once the change was implemented, we had two tasks to do:
- Get PyMySQL packaged and included in the RDO repos
This turned out to be a challenge for us, since a wrong assumption made us ignore the situation for a few weeks, assuming the package already included the required bits. Once we found out, the package was quickly ready in the RDO repository.
- Make the RDO installers support PyMySQL
Since we rely on the upstream Puppet OpenStack modules, the first step to do was to ensure they had proper support for PyMySQL, and this was implemented between November 2015 and January 2016. The Ansible community was a little faster than us, and had it working in October 2015.
Then, the last step was to include that support inside the installers used by RDO: Packstack (December 2015) and TripleO (January 2016).
The whole process didn’t take too long if you think about it, but there was a big gap between having the code ready (July) and us starting to work to get it inside the distribution (October). Why did this happen?
Lessons learned
So yes, the change was implemented and eventually propagated into RDO, although a bit later than the Liberty release date. IMHO, this leaves two lessons for the community to learn:
- From the overall OpenStack community side, it would be great if we started keeping in mind the bigger picture when it comes to disruptive changes. Yes, we have release notes and such, but we insist in considering our work is done once it works in DevStack, when there is a very significant number of users who install OpenStack using vendor-distributed packages and some form of Puppet or Ansible-based automation. I see a need for a tighter feedback loop between those three communities to make everyone’s life easier.
- From the distribution side, we also need to be closer to the community, so we can react faster to these changes. I don’t see this issue as “just a distro thing”, but it’s clear that the more informed we are about upstream changes, the better.
As a starter, I’m doing my homework and I am now subscribed to receive alerts for every new change to DevStack in Gerrit.
Javier Peña's technology blog 