Getting cross-project changes in OpenStack to our user’s hands

When working with an OpenStack installer, it is quite common to see changes happening in upstream projects break it. Options get deprecated every other day, new options show up, projects add dependencies… After all, this is one of the main reasons for RDO to have a continuous packaging pipeline coupled with CI jobs testing the latest commits.

However, in some cases we see some cross-project changes landing, and that takes quite a lot of coordination to happen, specially once we get passed the “it worked with DevStack!” point. As an example, I will discuss the switch from the MySQL-Python library to PyMySQL that happened during the Liberty cycle. We will follow the process from the initial decision until it got implemented in all projects (and DevStack), and then what it took us to get that change propagated into RDO and in the hands of our users.

The change

During the Liberty Summit, there was a cross-project session to discuss the Python 3 effort porting for OpenStack. During this discussion, the MySQL-Python library was found to be a blocker, as it is Python2-only. PyMySQL was found to be a better alternative, so a decision was made: let’s all switch to PyMySQL!

The upstream implementation

So a plan was prepared to do the switch, and it was quickly implemented:

So, in July 2015, all projects had successfully transitioned to PyMySQL, were using it in their CI systems, and everyone was happy.

So that’s is, right?

Well, not really. And this reflects an issue we in the OpenStack distro space find quite often. Changing the project code and updating DevStack only takes you to a point where you can develop using the new code.  If you install from source you may use it, provided you have updated your configuration files. But what about users?

To illustrate the situation, let’s have a look at the following diagram, where I describe (at a high level) what is the usual workflow for such a change, from the OpenStack distro point of view.change_process(Click image to enlarge)

So, once the change was implemented, we had two tasks to do:

  • Get PyMySQL packaged and included in the RDO repos

This turned out to be a challenge for us, since a wrong assumption made us ignore the situation for a few weeks, assuming the package already included the required bits. Once we found out, the package was quickly ready in the RDO repository.

  •  Make the RDO installers support PyMySQL

Since we rely on the upstream Puppet OpenStack modules, the first step to do was to ensure they had proper support for PyMySQL, and this was implemented between November 2015 and January 2016. The Ansible community was a little faster than us, and had it working in October 2015.

Then, the last step was to include that support inside the installers used by RDO: Packstack (December 2015) and TripleO (January 2016).

The whole process didn’t take too long if you think about it, but there was a big gap between having the code ready (July) and us starting to work to get it inside the distribution (October). Why did this happen?

Lessons learned

So yes, the change was implemented and eventually propagated into RDO, although a bit later than the Liberty release date. IMHO, this leaves two lessons for the community to learn:

  1. From the overall OpenStack community side, it would be great if we started keeping in mind the bigger picture when it comes to disruptive changes. Yes, we have release notes and such, but we insist in considering our work is done once it works in DevStack, when there is a very significant number of users who install OpenStack using vendor-distributed packages and some form of Puppet or Ansible-based automation. I see a need for a tighter feedback loop between those three communities to make everyone’s life easier.
  2. From the distribution side, we also need to be closer to the community, so we can react faster to these changes. I don’t see this issue as “just a distro thing”, but it’s clear that the more informed we are about upstream changes, the better.

As a starter, I’m doing my homework and I am now subscribed to receive alerts for every new change to DevStack in Gerrit.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s