Update 04/12/17 : The initial deployment documented in this demo no longer works due to the removal of a number of plan migration steps that have now been promoted into the Queens repos. We are currently looking into ways to reintroduce these for use in master UC Newton OC FFU development deployments, until then anyone attempting to run through this demo should start with a Newton OC and UC before upgrading the UC to master.
This is another TripleO fast-forward upgrade demo post, this time focusing on a basic stack of Keystone, Glance, Cinder, Neutron and Nova. At present there are several workarounds still required to allow the upgrade to complete, please see the workaround sections for more details.
Environment
As with the original demo I’m still using tripleo-quickstart to deploy my initial environment, this time with 1 controller and 1 compute, with a Queens undercloud and Newton overcloud. In addition I’m also using a new general config to deploy a minimal control stack able to host Nova.
$ bash quickstart.sh -w $WD -t all -R master-undercloud-newton-overcloud \
-c config/general_config/minimal-nova.yml $VIRTHOST
UC - docker_registry.yaml
Again with this demo we are not caching containers locally, the following
command will create a docker_registry.yaml file referencing the RDO registry
for use during the final deployment of the overcloud to Queens:
$ ssh -F $WD/ssh.config.ansible undercloud
$ openstack overcloud container image prepare \
--namespace trunk.registry.rdoproject.org/master \
--tag tripleo-ci-testing \
--output-env-file ~/docker_registry.yaml
UC - tripleo-heat-templates
We then need to update the version of tripleo-heat-templates deployed on the
undercloud host:
$ ssh -F $WD/ssh.config.ansible undercloud
$ cd /home/stack/tripleo-heat-templates
$ git fetch git://git.openstack.org/openstack/tripleo-heat-templates refs/changes/19/518719/9 && git checkout FETCH_HEAD
Finally, as we are using a customised controller role the following services
need to be added to the overcloud_services.yml file on the undercloud node
under ControllerServices:
parameter_defaults:
ControllerServices:
[..]
- OS::TripleO::Services::Docker
- OS::TripleO::Services::Iscsid
- OS::TripleO::Services::NovaPlacement
UC - tripleo-common
At present we are waiting for a promotion of tripleo-common that includes various bugfixes when updating the overcloud stack, generating outputs etc. For the time being we can simply install directly from master to workaround these issues.
$ ssh -F $WD/ssh.config.ansible undercloud
$ git clone https://github.com/openstack/tripleo-common.git ; cd tripleo-common
$ sudo python setup.py install ; cd ~
OC - Update heat-agents
As documented in my previous demo
post
we need to remove any legacy heiradata from all overcloud hosts prior to
updating the heat stack:
$ sudo rm -f /usr/libexec/os-apply-config/templates/etc/puppet/hiera.yaml \
/usr/libexec/os-refresh-config/configure.d/40-hiera-datafiles \
/etc/puppet/hieradata/*.yaml
We also need to update the heat-agents on all nodes to their Ocata versions:
$ git clone https://github.com/openstack/tripleo-repos.git ; cd tripleo-repos
$ sudo python setup.py install
$ sudo tripleo-repos -b ocata current
$ sudo yum update -y python-heat-agent \
python-heat-agent-puppt
$ sudo yum install -y openstack-heat-agents \
python-heat-agent-ansible \
python-heat-agent-apply-config \
python-heat-agent-docker-cmd \
python-heat-agent-hiera \
python-heat-agent-json-file
OC - Workarounds #1
$ sudo yum remove openstack-ceilometer* -y
UC - Update stack outputs
With the workarounds in place we can now update the stack using the updated version of tripleo-heat-templates on the undercloud. Once again we need to use the original deploy command with a number of additional environment files included:
$ . stackrc
$ openstack overcloud deploy \
--templates /home/stack/tripleo-heat-templates \
[..]
-e /home/stack/docker_registry.yaml \
-e /home/stack/tripleo-heat-templates/environments/docker.yaml \
-e /home/stack/tripleo-heat-templates/environments/fast-forward-upgrade.yaml \
-e /home/stack/tripleo-heat-templates/environments/noop-deploy-steps.yaml
UC - Download config
Once the stack has been updated we can download the config with the following command:
$ . stackrc
$ openstack overcloud config download
The TripleO configuration has been successfully generated into: /home/stack/tripleo-Oalkee-config
UC - FFU and Upgrade plays
Before running through any of the generated playbooks I personally like to add
the profile_tasks callback to the callback_whitelist for Ansible within
/etc/ansible/ansible.cfg. This provides timestamps during the playbook run
and a summary of the slowest tasks at the end.
# enable callback plugins, they can output to stdout but cannot be 'stdout' type.
callback_whitelist = profile_tasks
We first run the fast_forward_upgrade_playbook to complete the upgrade to Pike:
$ . stackrc
$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory \
/home/stack/config/tripleo-jUY9FB-config/fast_forward_upgrade_playbook.yaml
[..]
PLAY RECAP *****************************************************************************************************************************
192.168.24.11 : ok=62 changed=8 unreachable=0 failed=0
192.168.24.16 : ok=123 changed=55 unreachable=0 failed=0
Friday 01 December 2017 20:39:58 +0000 (0:00:03.967) 0:06:16.615 *******
===============================================================================
Stop neutron_server ------------------------------------------------------------------------------------------------------------ 32.53s
stop openstack-cinder-volume --------------------------------------------------------------------------------------------------- 16.03s
Stop neutron_l3_agent ---------------------------------------------------------------------------------------------------------- 14.36s
Stop and disable nova-compute service ------------------------------------------------------------------------------------------ 13.16s
Cinder package update ---------------------------------------------------------------------------------------------------------- 12.73s
stop openstack-cinder-scheduler ------------------------------------------------------------------------------------------------ 12.68s
Setup cell_v2 (sync nova/cell DB) ---------------------------------------------------------------------------------------------- 11.79s
Cinder package update ---------------------------------------------------------------------------------------------------------- 11.30s
Neutron package update --------------------------------------------------------------------------------------------------------- 10.99s
Keystone package update -------------------------------------------------------------------------------------------------------- 10.77s
glance package update ---------------------------------------------------------------------------------------------------------- 10.28s
Keystone package update --------------------------------------------------------------------------------------------------------- 9.80s
glance package update ----------------------------------------------------------------------------------------------------------- 8.72s
Neutron package update ---------------------------------------------------------------------------------------------------------- 8.62s
Stop and disable nova-consoleauth service --------------------------------------------------------------------------------------- 7.94s
Update nova packages ------------------------------------------------------------------------------------------------------------ 7.62s
Update nova packages ------------------------------------------------------------------------------------------------------------ 7.24s
Stop and disable nova-scheduler service ----------------------------------------------------------------------------------------- 6.36s
Run puppet apply to set tranport_url in nova.conf ------------------------------------------------------------------------------- 5.78s
install tripleo-repos ----------------------------------------------------------------------------------------------------------- 4.70s
We then run the upgrade_steps_playbook to start the upgrade to Queens:
$ . stackrc
$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory \
/home/stack/tripleo-Oalkee-config/upgrade_steps_playbook.yaml
[..]
PLAY RECAP *****************************************************************************************************************************
192.168.24.11 : ok=57 changed=45 unreachable=0 failed=0
192.168.24.16 : ok=165 changed=146 unreachable=0 failed=0
Friday 01 December 2017 20:51:55 +0000 (0:00:00.038) 0:10:47.865 *******
===============================================================================
Update all packages ----------------------------------------------------------------------------------------------------------- 263.71s
Update all packages ----------------------------------------------------------------------------------------------------------- 256.79s
Install docker packages on upgrade if missing ---------------------------------------------------------------------------------- 13.77s
Upgrade os-net-config ----------------------------------------------------------------------------------------------------------- 5.71s
Upgrade os-net-config ----------------------------------------------------------------------------------------------------------- 5.12s
Gathering Facts ----------------------------------------------------------------------------------------------------------------- 3.36s
Install docker packages on upgrade if missing ----------------------------------------------------------------------------------- 3.14s
Stop and disable mysql service -------------------------------------------------------------------------------------------------- 1.97s
Check for os-net-config upgrade ------------------------------------------------------------------------------------------------- 1.66s
Check for os-net-config upgrade ------------------------------------------------------------------------------------------------- 1.57s
Stop keepalived service --------------------------------------------------------------------------------------------------------- 1.48s
Stop and disable rabbitmq service ----------------------------------------------------------------------------------------------- 1.47s
take new os-net-config parameters into account now ------------------------------------------------------------------------------ 1.31s
take new os-net-config parameters into account now ------------------------------------------------------------------------------ 1.08s
Check if openstack-ceilometer-compute is deployed ------------------------------------------------------------------------------- 0.70s
Check if iscsid service is deployed --------------------------------------------------------------------------------------------- 0.67s
Start keepalived service -------------------------------------------------------------------------------------------------------- 0.48s
Check for nova placement running under apache ----------------------------------------------------------------------------------- 0.46s
Stop and disable mongodb service on upgrade ------------------------------------------------------------------------------------- 0.45s
remove old cinder cron jobs ----------------------------------------------------------------------------------------------------- 0.45s
OC - Workarounds #2
On overcloud-novacompute-0 the following file needs to be removed to
workaround a known issue:
$ ssh -F $WD/ssh.config.ansible overcloud-novacompute-0
$ sudo rm /etc/iscsi/.initiator_reset
UC - Deploy play
Finally we run through the deploy_steps_playbook:
$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory \
/home/stack/tripleo-Oalkee-config/deploy_steps_playbook.yaml
[..]
PLAY RECAP *****************************************************************************************************************************
192.168.24.11 : ok=48 changed=11 unreachable=0 failed=0
192.168.24.16 : ok=76 changed=10 unreachable=0 failed=0
localhost : ok=1 changed=0 unreachable=0 failed=0
Friday 01 December 2017 21:04:58 +0000 (0:00:00.041) 0:10:24.723 *******
===============================================================================
Run docker-puppet tasks (generate config) ------------------------------------------------------------------------------------- 186.65s
Run docker-puppet tasks (bootstrap tasks) ------------------------------------------------------------------------------------- 101.10s
Start containers for step 3 ---------------------------------------------------------------------------------------------------- 98.61s
Start containers for step 4 ---------------------------------------------------------------------------------------------------- 41.37s
Run puppet host configuration for step 1 --------------------------------------------------------------------------------------- 32.53s
Start containers for step 1 ---------------------------------------------------------------------------------------------------- 25.76s
Run puppet host configuration for step 5 --------------------------------------------------------------------------------------- 17.91s
Run puppet host configuration for step 4 --------------------------------------------------------------------------------------- 14.47s
Run puppet host configuration for step 3 --------------------------------------------------------------------------------------- 13.41s
Run docker-puppet tasks (bootstrap tasks) -------------------------------------------------------------------------------------- 10.39s
Run puppet host configuration for step 2 --------------------------------------------------------------------------------------- 10.37s
Start containers for step 5 ---------------------------------------------------------------------------------------------------- 10.12s
Run docker-puppet tasks (bootstrap tasks) --------------------------------------------------------------------------------------- 9.78s
Start containers for step 2 ----------------------------------------------------------------------------------------------------- 6.32s
Gathering Facts ----------------------------------------------------------------------------------------------------------------- 4.37s
Gathering Facts ----------------------------------------------------------------------------------------------------------------- 3.46s
Write the config_step hieradata ------------------------------------------------------------------------------------------------- 1.80s
create libvirt persistent data directories -------------------------------------------------------------------------------------- 1.21s
Write the config_step hieradata ------------------------------------------------------------------------------------------------- 1.03s
Check if /var/lib/docker-puppet/docker-puppet-tasks4.json exists ---------------------------------------------------------------- 1.00s
Verification
I’ll revisit this in the coming days and add a more complete set of tasks to verify the end environment but for now we can run a simple boot from volume instance (as Swift, the default store for Glance was not installed):
$ cinder create 1
$ cinder set-bootable 46d278f7-31fc-4e45-b5df-eb8220800b1a true
$ nova flavor-create 1 1 512 1 1
$ nova boot --boot-volume 46d278f7-31fc-4e45-b5df-eb8220800b1a --flavor 1 test
[..]
$ nova list
+--------------------------------------+------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+-------------------+
| 05821616-1239-4ca9-8baa-6b0ca4ea3a6b | test | ACTIVE | - | Running | priv=192.168.0.16 |
+--------------------------------------+------+--------+------------+-------------+-------------------+
We can also see the various containerised services running on the overcloud:
$ ssh -F $WD/ssh.config.ansible overcloud-controller-0
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d80d6f072604 trunk.registry.rdoproject.org/master/centos-binary-glance-api:tripleo-ci-testing "kolla_start" 13 minutes ago Up 12 minutes (healthy) glance_api
61fbf47241ce trunk.registry.rdoproject.org/master/centos-binary-nova-api:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes nova_metadata
9defdb5efe0f trunk.registry.rdoproject.org/master/centos-binary-nova-api:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) nova_api
874716d99a44 trunk.registry.rdoproject.org/master/centos-binary-nova-novncproxy:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) nova_vnc_proxy
21ca0fd8d8ec trunk.registry.rdoproject.org/master/centos-binary-neutron-server:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes neutron_api
e0eed85b860a trunk.registry.rdoproject.org/master/centos-binary-cinder-volume:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) cinder_volume
0882e08ac198 trunk.registry.rdoproject.org/master/centos-binary-nova-consoleauth:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) nova_consoleauth
e3ebc4b066c9 trunk.registry.rdoproject.org/master/centos-binary-nova-api:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes nova_api_cron
c7d05a04a8a3 trunk.registry.rdoproject.org/master/centos-binary-cinder-api:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes cinder_api_cron
2f3c1e244997 trunk.registry.rdoproject.org/master/centos-binary-neutron-openvswitch-agent:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) neutron_ovs_agent
bfeb120bf77a trunk.registry.rdoproject.org/master/centos-binary-neutron-metadata-agent:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) neutron_metadata_agent
43b2c09aecf8 trunk.registry.rdoproject.org/master/centos-binary-nova-scheduler:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) nova_scheduler
a7a3024b63f6 trunk.registry.rdoproject.org/master/centos-binary-neutron-dhcp-agent:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) neutron_dhcp
3df990a68046 trunk.registry.rdoproject.org/master/centos-binary-cinder-scheduler:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) cinder_scheduler
94461ba833aa trunk.registry.rdoproject.org/master/centos-binary-neutron-l3-agent:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) neutron_l3_agent
4bee34f9fce2 trunk.registry.rdoproject.org/master/centos-binary-cinder-api:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes cinder_api
e8bec9348fe3 trunk.registry.rdoproject.org/master/centos-binary-nova-conductor:tripleo-ci-testing "kolla_start" 13 minutes ago Up 13 minutes (healthy) nova_conductor
22db40c25881 trunk.registry.rdoproject.org/master/centos-binary-keystone:tripleo-ci-testing "/bin/bash -c '/usr/l" 15 minutes ago Up 15 minutes keystone_cron
26769acaaf5e trunk.registry.rdoproject.org/master/centos-binary-keystone:tripleo-ci-testing "kolla_start" 16 minutes ago Up 16 minutes (healthy) keystone
99037a5e5c36 trunk.registry.rdoproject.org/master/centos-binary-iscsid:tripleo-ci-testing "kolla_start" 16 minutes ago Up 16 minutes iscsid
9f4aae72c201 trunk.registry.rdoproject.org/master/centos-binary-nova-placement-api:tripleo-ci-testing "kolla_start" 16 minutes ago Up 16 minutes nova_placement
311302abc297 trunk.registry.rdoproject.org/master/centos-binary-horizon:tripleo-ci-testing "kolla_start" 16 minutes ago Up 16 minutes horizon
d465e4f5b7e6 trunk.registry.rdoproject.org/master/centos-binary-mariadb:tripleo-ci-testing "kolla_start" 17 minutes ago Up 17 minutes (unhealthy) mysql
b9e062f1d857 trunk.registry.rdoproject.org/master/centos-binary-rabbitmq:tripleo-ci-testing "kolla_start" 18 minutes ago Up 18 minutes (healthy) rabbitmq
a57f053afc03 trunk.registry.rdoproject.org/master/centos-binary-memcached:tripleo-ci-testing "/bin/bash -c 'source" 18 minutes ago Up 18 minutes memcached
baeb6d1087e6 trunk.registry.rdoproject.org/master/centos-binary-redis:tripleo-ci-testing "kolla_start" 18 minutes ago Up 18 minutes redis
faafa1bf2d2e trunk.registry.rdoproject.org/master/centos-binary-haproxy:tripleo-ci-testing "kolla_start" 18 minutes ago Up 18 minutes haproxy
$ exit
$ ssh -F $WD/ssh.config.ansible overcloud-novacompute-0
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0363d7008e87 trunk.registry.rdoproject.org/master/centos-binary-neutron-openvswitch-agent:tripleo-ci-testing "kolla_start" 12 minutes ago Up 12 minutes (healthy) neutron_ovs_agent
c1ff23ee9f16 trunk.registry.rdoproject.org/master/centos-binary-cron:tripleo-ci-testing "kolla_start" 12 minutes ago Up 12 minutes logrotate_crond
d81d8207ec9a trunk.registry.rdoproject.org/master/centos-binary-nova-compute:tripleo-ci-testing "kolla_start" 12 minutes ago Up 12 minutes nova_migration_target
abd9b79e2af8 trunk.registry.rdoproject.org/master/centos-binary-ceilometer-compute:tripleo-ci-testing "kolla_start" 12 minutes ago Up 12 minutes ceilometer_agent_compute
aa581489ac9a trunk.registry.rdoproject.org/master/centos-binary-nova-compute:tripleo-ci-testing "kolla_start" 12 minutes ago Up 12 minutes (healthy) nova_compute
d4ade28175f0 trunk.registry.rdoproject.org/master/centos-binary-iscsid:tripleo-ci-testing "kolla_start" 14 minutes ago Up 14 minutes iscsid
ae4652853098 trunk.registry.rdoproject.org/master/centos-binary-nova-libvirt:tripleo-ci-testing "kolla_start" 14 minutes ago Up 14 minutes nova_libvirt
aac8fea2d496 trunk.registry.rdoproject.org/master/centos-binary-nova-libvirt:tripleo-ci-testing "kolla_start" 14 minutes ago Up 14 minutes nova_virtlogd
Conclusion
So in conclusion this demo takes a simple multi-host OpenStack deployment of Keystone, Glance, Cinder, Neutron and Nova from baremetal Newton to containerised Queens in ~26 minutes. There are many things still to resolve and validate with FFU but for now, ahead of M2 this is a pretty good start.