Ceph integration topics at OpenStack PTG

2018-03-19T03:32:00+01:00

I wanted to share a short summary of the discussions happened around the Ceph integration (in TripleO) at the OpenStack PTG.

ceph-{container,ansible} branching

Together with John Fulton and Guillaume Abrioux (and after PTG, Sebastien Han) we put some thought into how to make the Ceph container images and ceph-ansible releases fit better the OpenStack model; the container images and ceph-ansible are in fact loosely coupled (not all versions of the container images work with all versions of ceph-ansible) and we wanted to move from a "rolling release" into a "point release" approach, mainly to permit regular maintenance of the previous versions known to work with the previous OpenStack versions. The plan goes more or less as follows:

ceph-{container,ansible} should be released together with the regular ceph updates

ceph-container will start using tags and stable branches like ceph-ansible does

The changes for the ceph/daemon docker images are visible already: https://hub.docker.com/r/ceph/daemon/tags/

Multiple Ceph clusters

In the attempt to support better the "edge computing" use case, we discussed adding support for the deployment of multiple Ceph clusters in the overcloud.

Together with John Fulton and Steven Hardy (and after PTG, Gregory Charot) we realized this could be done using multiple stacks and by doing so, hopefully simplify managament of the "cells" and avoid potential issues due to orchestration of large clusters.

Much of this will build on Shardy's blueprint to split the control plane, see spec at: https://review.openstack.org/#/c/523459/

The multiple Ceph clusters specifics will be tracked via another blueprint: https://blueprints.launchpad.net/tripleo/+spec/deploy-multiple-ceph-clusters

ceph-ansible testing with TripleO

We had a very good chat with John Fulton, Guillaume Abrioux, Wesley Hayutin and Javier Pena on how to get tested new pull requests for ceph-ansible with TripleO; basically trigger an existing TripleO scenario on changes proposed to ceph-ansible.

Given ceph-ansible is hosted on github, Wesley's and Javier suggested this should be possible with Zuul v3 and volunteered to help; some of the complications are about building an RPM from uncommitted changes for testing.

Move ceph-ansible triggering from workflow_tasks to external_deploy_tasks

This is a requirement for the Rocky release; we want to migrate away from using workflow_tasks and use external_deploy_tasks instead, to integrate into the "config-download" mechanism.

This work is tracked via a blueprint and we have a WIP submission on review: https://blueprints.launchpad.net/tripleo/+spec/ceph-ansible-external-deploy-tasks

We're also working with Sofer Athlan-Guyot on the enablement of Ceph in the upgrade CI jobs and with Tom Barron on scenario004 to deploy Manila with Ganesha (and CephFS) instead of the CephFS native backend.

Hopefully I didn't forget much; to stay updated on the progress join #tripleo on freenode or check our integration squad status at: https://etherpad.openstack.org/p/tripleo-integration-squad-status

Understanding ceph-ansible in TripleO

2017-07-19T11:00:00+02:00

One of the goals for the TripleO Pike release was to introduce ceph-ansible as an alternative to puppet-ceph for the deployment of Ceph.

More specifically, to put operators in control of the playbook execution as if they were launching ceph-ansible from the commandline, except it would be Heat starting ceph-ansible at the right time during the overcloud deployment.

This demanded for some changes in different tools used by TripleO and went through a pretty long review process, eventually putting in place some useful bits for the future integration of Kubernetes and migration to an ansible driven deployment of the overcloud configuration steps in TripleO.

The idea was to add a generic functionality allowing triggering of a given Mistral workflow during the deployment of a service. Mistral could have then executed any action, including for example an ansible playbook, provided it was given all the necessay input data for the playbook to run and the roles list to build the hosts inventory.

This is how we did it.

Run ansible-playbook from Mistral (1): An initial submission added support for the execution of ansible playbooks as workflow tasks in Mistral https://github.com/openstack/tripleo-common/commit/e6c8a46f00436edfa5de92e97c3a390d90c3ce54

A generic action for Mistral which workflows can use to run an ansible playbook. +2 to Dougal and Ryan.

Deploy external resources from Heat (2): We also needed a new resource in Heat to be able to drive Mistral workflow executions https://github.com/openstack/heat/commit/725b404468bdd2c1bdbaf16e594515475da7bace so that we could orchestrate the executions like any other Heat resource. This is described much in detail in a Heat spec.

With these two, we could run an ansible playbook from a Heat resource, via Mistral. +2 to Zane and Thomas for the help! Enough to start messing in TripleO and glue things together.

Describe what/when to run in TripleO (3): We added a mechanim in the TripleO templates to make it possible to describe, from within a service, a list of tasks or workflows to be executed at any given deployment step https://github.com/openstack/tripleo-heat-templates/commit/71f13388161cbab12fe284f7b251ca8d36f7635c

There aren't restrictions on what the tasks or workflows in the new section should do. These might deploy the service or prepare the environment for it or execute code (eg. build Swift rings). The commit message explains how to use it:

service_workflow_tasks:
  step2:
    - name: my_action_name
      action: std.echo
      input:
        output: 'hello world'

The above snippet would make TripleO to run the Mistral std.echo action during the overcloud deployment, precisely at step 2, assuming you create a new service with the code above and enable it on a role.

For Ceph we wanted to run the new Mistral action (1) and needed to provide it with the config settings for the service, normally described within the config_settings structure of the service template.

Provide config_settings to the workflows (4): The decision was to make available all config settings into the Mistral execution environment so that ansible actions could, for example, use them as extra_vars https://github.com/openstack/tripleo-heat-templates/commit/8b81b363fd48b0080b963fd2b1ab6bfe97b0c204

Now all config settings normally consumed by puppet were available to the Mistral action and playbook settings could be added too, +2 Steven.

Build the data for the hosts inventory (5): Together with the above, another small change provided into the execution environment a dictionary mapping every enabled service to the list of IP address of the nodes where the service is deployed https://github.com/openstack/tripleo-heat-templates/commit/9c1940e461867f2ce986a81fa313d7995592f0c5

This was necessary to be able to build the ansible hosts inventory.

Create a workflow for ceph-ansible (6): Having all pieces available to trigger the workflow and pass to it the service config settings, we needed the workflow which would run ceph-ansible https://github.com/openstack/tripleo-common/commit/fa0b9f52080580b7408dc6f5f2da6fc1dc07d500 plus some new, generic Mistral actions, to run smoothly multiple times (eg. stack updates) https://github.com/openstack/tripleo-common/commit/f81372d85a0a92de455eeaa93162faf09be670cf

This is the glue which runs a ceph-ansible playbook with the given set of parameters. +2 John.

Deploy Ceph via ceph-ansible (7): Finally, the new services definition for Tripleo https://review.openstack.org/#/c/465066/ to deploy Ceph in containers via ceph-ansible, including a couple of params operators can use to push into the Mistral environment arbitrary extra_vars for ceph-ansible.

The deployment with ceph-ansible is activated with the ceph-ansible.yaml environment file.

Interestingly the templates to deploy Ceph using puppet-ceph are unchanged and continue to work as they used to so that for new deployments it is possible to use alternatively the new implementation with ceph-ansible or the pre-existing implementation using puppet-ceph. Only ceph-ansible allows for the deployment of Ceph in containers.

Big +2 also to Jiri (who doesn't even need a blog or twitter) and all the people who helped during the development process with feedback, commits and reviews.