OpenStack - Skip level upgrades - Introduction

Comments

I’ve been fortunate enough to be part of a team looking into skip level upgrades recently ahead of the start of the Queens development cycle for OpenStack. What follows is an introduction to the concept of skip level upgrades and an overview of our initial PoC work in this area. Future posts will also cover our plans for enabling skip level upgrades within TripleO and possible work with the wider community to enable this within other deployment tools.

Introduction

Skip level upgrades are as the name suggests, upgrades that move an environment from release N to N+X in a single step, where X is greater than 1 and for skip level upgrades is typically 3. For example in the context of OpenStack N to N+3 can refer to an upgrade from the Newton release of Openstack to the Queens release, skipping Ocata and Pike:

Newton    Ocata     Pike       Queens
+-----+   +-----+   +-----+    +-----+
|     |   | N+1 |   | N+2 |    |     |
|  N  | ---------------------> | N+3 |
|     |   |     |   |     |    |     |
+-----+   +-----+   +-----+    +-----+

There are existing alternative methods available for skipping a number of releases during an upgrade. For example, parallel cloud migration is a commonly cited alternative. This is where an additional environment is stood up alongside the original, with workloads migrated to the new environment:

        Newton    
        +-----+
        |     |
env#1   |  N  |
        |     |
        +-----+
------------------------------------   
           \       Queens
            \      +-----+
             \     |     |
env#2         `->  | N+3 |
                   |     |
                   +-----+

The requirement for this type of upgrade is driven by users looking to standardise on a given release (typically LTS), whilst retaining the ability to skip forward when the release hits EOL. This negates the need to keep up with the major release cycle that in the case of OpenStack continues to be every 6 months.

It is worth highlighting that the topic of skip level upgrades is not new to the OpenStack community, with attempts to provide skip level upgrade functionality within the community before now, typically within the various deployment projects. For example openstack-ansible’s leap-upgrades project that attempted to move environments between Juno/Kilo and Newton.

More recently the topic of skip level upgrades was discussed at the OpenStack Forum in Boston in May. A RFC thread was also posted to the development mailing list, however no formal actions came of either discussion. I’m looking to restart this discussion at the next PTG in Denver, more on that later.

Requirements

Now that we understand what skip level upgrades actually are, it’s time to set out some basic requirements for the state of the environment during the upgrade. At the start of this process our team sat down and drafted the following:

  • The control plane is inaccessible for the duration of the upgrade
  • The upgrade must complete successfully or rollback within 4 hours
  • The data plane and workloads must remain available for the duration of the upgrade.

Proof of concept

With the requirements set out, our first real task was to prove that this was even possible with an OpenStack environment. Given the releases available at the time, we began by manually upgrading an existing Mitaka based RHOSP 9 environment running on RHEL 7.3 to our recently released Ocata based RHOSP 11 release running on RHEL 7.4.

Mitaka    Newton    Ocata
+-----+   +-----+   +-----+
|     |   | N+1 |   |     |
|  N  | ----------> | N+2 |
|     |   |     |   |     |
+-----+   +-----+   +-----+
RHOSP9    RHOSP10   RHOSP11
RHEL73              RHEL74

We were aware that whilst the goal of skip level upgrades is to give the impression of a single jump between releases, in practice this isn’t possible with OpenStack. Upgrades of OpenStack components are verified by the community across N to N+1 jumps, so whilst we wanted to skip ahead to Ocata we knew we would also have to upgrade through Newton to get there.

The following outlines, at a very high level, the steps we followed during the PoC to upgrade the environment from Mitaka to Ocata:

  • Rolling minor update of the underlying OS
  • Disable control plane and compute services
  • Upgrade a single controller to N+1 and then N+2
    • Update packages
    • Introduce new services as required (nova-placement for example)
    • Update service configuration files
    • Run DB syncs, migrations etc
    • Repeat for N+2
  • Upgrade remaining controllers directly to N+2
    • Update packages
    • Introduce new services as required
    • Update service configuration files
  • Upgrade remaining hosts to N+2
    • Update packages
    • Update service configuration files
  • Enable control plane and compute services
  • Verify workload availability during upgrade
  • Validate the post upgrade environment.

Let’s take a look at each of these steps below in more detail.

Rolling minor update of the underlying OS

This initial rolling minor update moved hosts from RHEL 7.3 to RHEL 7.4 whilst also pulling in OVS from our RHOSP11 repos in a bid to limit the number of reboots required in the environment. In practise operators could perform this minor update well ahead of any skip level upgrade, reducing any impact on the overall time required for the upgrade itself.

Disable control plane and compute services

As listed above under requirements, a full control plane outage is accounted for during the skip level upgrade. Note that this does not include the infrastructure services providing the database, messaging queues etc. Compute services are also stopped at this time but should not have any impact on the running workloads and data plane.

Upgrade a single controller to N+1 and then N+2

The main work of upgrading between releases is carried out on a single controller. Packages are updated, new services such as nova-placement are deployed as required, configuration files updated and DB migrations completed. This process is repeated on this host until we reach the target release.

Upgrade remaining controllers directly to N+2

Once the single controller has been upgraded to our target release we then skip any remaining controllers ahead to this target release. Updating packages, introducing new services and updating configuration files on these controllers as required.

Upgrade remaining hosts to N+2

This is then repeated for any remaining hosts, such as computes, object storage hosts etc. Again this should not interrupt running workloads or the data plane.

Enable control plane and compute services

Once all hosts are updated to the target release the control and compute services are restarted.

Verify workload availability during upgrade

During our PoC we ran multiple instances across various L2 and L3 networks, using Ansible to first launch and then later collect the results of asynchronous jobs (ping, ssh etc) that had been running between these instances during the upgrade.

Validate the post upgrade environment

Finally Tempest was used to validate the end state of the environment post upgrade.

After many iterations, the eventual introduction of Ansible to automate all the things and much cursing at the amount of time to reconfigure the environment after each run it was agreed that the team would move on to look into the possible implementation of the above in TripleO.

PTG

Thanks for making it this far! Before I end this post I wanted to highlight that I’ve recently agreed to lead the skip level upgrades room at the upcoming Denver PTG. I’ll be posting more details on this shortly to the OpenStack development mailing list but wanted to take this opportunity to encourage anyone interested in this topic to attend and discuss possible ways we can make skip level upgrades a possibility across the various deployment tools within the community.

Contents

comments powered by Disqus