Giulio Fidente - tripleo

Pushing Openstack and Ceph to the Edge at OpenStack summit Berlin 2018

2019-01-02T00:00:00+01:00

We presented at the latest OpenStack (Open Infrastructure ?) summit, held in Berlin after te Rocky release, together with Sebastien Han and Sean Cohen, a session discussing how TripleO will support "Edge" deployments with storage at the edge; colocating Ceph with the OpenStack services in the edge zone yet keeping a small hardware footprint.

Thanks to the OpenStack Foundation for giving us the chance.

Ceph integration topics at OpenStack PTG

2018-03-19T03:32:00+01:00

I wanted to share a short summary of the discussions happened around the Ceph integration (in TripleO) at the OpenStack PTG.

ceph-{container,ansible} branching

Together with John Fulton and Guillaume Abrioux (and after PTG, Sebastien Han) we put some thought into how to make the Ceph container images and ceph-ansible releases fit better the OpenStack model; the container images and ceph-ansible are in fact loosely coupled (not all versions of the container images work with all versions of ceph-ansible) and we wanted to move from a "rolling release" into a "point release" approach, mainly to permit regular maintenance of the previous versions known to work with the previous OpenStack versions. The plan goes more or less as follows:

ceph-{container,ansible} should be released together with the regular ceph updates

ceph-container will start using tags and stable branches like ceph-ansible does

The changes for the ceph/daemon docker images are visible already: https://hub.docker.com/r/ceph/daemon/tags/

Multiple Ceph clusters

In the attempt to support better the "edge computing" use case, we discussed adding support for the deployment of multiple Ceph clusters in the overcloud.

Together with John Fulton and Steven Hardy (and after PTG, Gregory Charot) we realized this could be done using multiple stacks and by doing so, hopefully simplify managament of the "cells" and avoid potential issues due to orchestration of large clusters.

Much of this will build on Shardy's blueprint to split the control plane, see spec at: https://review.openstack.org/#/c/523459/

The multiple Ceph clusters specifics will be tracked via another blueprint: https://blueprints.launchpad.net/tripleo/+spec/deploy-multiple-ceph-clusters

ceph-ansible testing with TripleO

We had a very good chat with John Fulton, Guillaume Abrioux, Wesley Hayutin and Javier Pena on how to get tested new pull requests for ceph-ansible with TripleO; basically trigger an existing TripleO scenario on changes proposed to ceph-ansible.

Given ceph-ansible is hosted on github, Wesley's and Javier suggested this should be possible with Zuul v3 and volunteered to help; some of the complications are about building an RPM from uncommitted changes for testing.

Move ceph-ansible triggering from workflow_tasks to external_deploy_tasks

This is a requirement for the Rocky release; we want to migrate away from using workflow_tasks and use external_deploy_tasks instead, to integrate into the "config-download" mechanism.

This work is tracked via a blueprint and we have a WIP submission on review: https://blueprints.launchpad.net/tripleo/+spec/ceph-ansible-external-deploy-tasks

We're also working with Sofer Athlan-Guyot on the enablement of Ceph in the upgrade CI jobs and with Tom Barron on scenario004 to deploy Manila with Ganesha (and CephFS) instead of the CephFS native backend.

Hopefully I didn't forget much; to stay updated on the progress join #tripleo on freenode or check our integration squad status at: https://etherpad.openstack.org/p/tripleo-integration-squad-status

Understanding ceph-ansible in TripleO

2017-07-19T11:00:00+02:00

One of the goals for the TripleO Pike release was to introduce ceph-ansible as an alternative to puppet-ceph for the deployment of Ceph.

More specifically, to put operators in control of the playbook execution as if they were launching ceph-ansible from the commandline, except it would be Heat starting ceph-ansible at the right time during the overcloud deployment.

This demanded for some changes in different tools used by TripleO and went through a pretty long review process, eventually putting in place some useful bits for the future integration of Kubernetes and migration to an ansible driven deployment of the overcloud configuration steps in TripleO.

The idea was to add a generic functionality allowing triggering of a given Mistral workflow during the deployment of a service. Mistral could have then executed any action, including for example an ansible playbook, provided it was given all the necessay input data for the playbook to run and the roles list to build the hosts inventory.

This is how we did it.

Run ansible-playbook from Mistral (1): An initial submission added support for the execution of ansible playbooks as workflow tasks in Mistral https://github.com/openstack/tripleo-common/commit/e6c8a46f00436edfa5de92e97c3a390d90c3ce54

A generic action for Mistral which workflows can use to run an ansible playbook. +2 to Dougal and Ryan.

Deploy external resources from Heat (2): We also needed a new resource in Heat to be able to drive Mistral workflow executions https://github.com/openstack/heat/commit/725b404468bdd2c1bdbaf16e594515475da7bace so that we could orchestrate the executions like any other Heat resource. This is described much in detail in a Heat spec.

With these two, we could run an ansible playbook from a Heat resource, via Mistral. +2 to Zane and Thomas for the help! Enough to start messing in TripleO and glue things together.

Describe what/when to run in TripleO (3): We added a mechanim in the TripleO templates to make it possible to describe, from within a service, a list of tasks or workflows to be executed at any given deployment step https://github.com/openstack/tripleo-heat-templates/commit/71f13388161cbab12fe284f7b251ca8d36f7635c

There aren't restrictions on what the tasks or workflows in the new section should do. These might deploy the service or prepare the environment for it or execute code (eg. build Swift rings). The commit message explains how to use it:

service_workflow_tasks:
  step2:
    - name: my_action_name
      action: std.echo
      input:
        output: 'hello world'

The above snippet would make TripleO to run the Mistral std.echo action during the overcloud deployment, precisely at step 2, assuming you create a new service with the code above and enable it on a role.

For Ceph we wanted to run the new Mistral action (1) and needed to provide it with the config settings for the service, normally described within the config_settings structure of the service template.

Provide config_settings to the workflows (4): The decision was to make available all config settings into the Mistral execution environment so that ansible actions could, for example, use them as extra_vars https://github.com/openstack/tripleo-heat-templates/commit/8b81b363fd48b0080b963fd2b1ab6bfe97b0c204

Now all config settings normally consumed by puppet were available to the Mistral action and playbook settings could be added too, +2 Steven.

Build the data for the hosts inventory (5): Together with the above, another small change provided into the execution environment a dictionary mapping every enabled service to the list of IP address of the nodes where the service is deployed https://github.com/openstack/tripleo-heat-templates/commit/9c1940e461867f2ce986a81fa313d7995592f0c5

This was necessary to be able to build the ansible hosts inventory.

Create a workflow for ceph-ansible (6): Having all pieces available to trigger the workflow and pass to it the service config settings, we needed the workflow which would run ceph-ansible https://github.com/openstack/tripleo-common/commit/fa0b9f52080580b7408dc6f5f2da6fc1dc07d500 plus some new, generic Mistral actions, to run smoothly multiple times (eg. stack updates) https://github.com/openstack/tripleo-common/commit/f81372d85a0a92de455eeaa93162faf09be670cf

This is the glue which runs a ceph-ansible playbook with the given set of parameters. +2 John.

Deploy Ceph via ceph-ansible (7): Finally, the new services definition for Tripleo https://review.openstack.org/#/c/465066/ to deploy Ceph in containers via ceph-ansible, including a couple of params operators can use to push into the Mistral environment arbitrary extra_vars for ceph-ansible.

The deployment with ceph-ansible is activated with the ceph-ansible.yaml environment file.

Interestingly the templates to deploy Ceph using puppet-ceph are unchanged and continue to work as they used to so that for new deployments it is possible to use alternatively the new implementation with ceph-ansible or the pre-existing implementation using puppet-ceph. Only ceph-ansible allows for the deployment of Ceph in containers.

Big +2 also to Jiri (who doesn't even need a blog or twitter) and all the people who helped during the development process with feedback, commits and reviews.

Soon another article with some usage examples and debugging instructions!

TripleO to deploy Ceph standlone

2016-12-16T23:00:00+01:00

Here is a nice Christmas present: you can use TripleO for a standalone Ceph deployment, with just a few lines of YAML. Assuming you have an undercloud ready for a new overcloud, create an environment file like the following:

resource_registry:
  OS::TripleO::Services::CephMon: /usr/share/openstack-tripleo-heat-templates/puppet/services/ceph-mon.yaml
  OS::TripleO::Services::CephOSD: /usr/share/openstack-tripleo-heat-templates/puppet/services/ceph-osd.yaml

parameters:
  ControllerServices:
    - OS::TripleO::Services::CephMon
  CephStorageServices:
    - OS::TripleO::Services::CephOSD

and launch a deployment with:

openstack overcloud deploy --compute-scale 0 --ceph-storage-scale 1 -e the_above_env_file.yaml

The two lines from the environment file in resource_registry are mapping (and enabling) the CephMon and CephOSD services in TripleO while the lines in parameters are defining which services should be deployed on the controller and cephstorage roles.

This will bring up a two nodes overcloud with one node running ceph-mon and the other ceph-osd but the actual Christmas gift is that it implicitly provides and allows usage of all the features we already know about TripleO, like:

baremetal provisioning

network isolation

a web GUI

lifecycle management

... containers

... upgrades

For example, you can scale up the Ceph cluster with:

openstack overcloud deploy --compute-scale 0 --ceph-storage-scale 2 -e the_above_env_file.yaml

and this will provision a new Ironic node with the cephstorage role, configuring the required networks on it and updating the cluster config for the new OSDs. (Note the --ceph-storage-scale parameter going from 1 to 2 in the second example).

Even more interestingly is that the above will work for any service, not just Ceph, and new services can be added to TripleO with just some YAML and puppet, letting TripleO take care of a number of common issues in any deployment tool, for example:

supports multinode deployments

synchronizes and orders the deployment steps across different nodes

supports propagation of config data across different services

Time to try it and join the fun in #tripleo :)

Ceph, TripleO and the Newton release

2016-08-26T05:00:00+02:00

Time to roll up some notes on the status of Ceph in TripleO. The majority of these functionalities were available in the Mitaka release too but the examples work with code from the Newton release so they might not apply identical to Mitaka.

The TripleO default configuration

No default is going to fit everybody, but we want to know what the default is to improve from there. So let's try and see:

uc$ openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/storage-environment.yaml --ceph-storage-scale 1
Deploying templates in the directory /home/stack/example/tripleo-heat-templates
...
Overcloud Deployed

Monitors go on the controller nodes, one per node, the above command is deploying a single controller though. First interesting thing to point out is:

oc$ ceph --version
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)

Jewel! Kudos to Emilien for bringing support for it in puppet-ceph. Continuing our investigation, we notice the OSDs go on the cephstorage nodes and are backed by the local filesystem, as we didn't tell it to do differently:

oc$ ceph osd tree
ID WEIGHT  TYPE NAME                        UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.03999 root default
-2 0.03999     host overcloud-cephstorage-0
 0 0.03999         osd.0                         up  1.00000          1.00000

Notice we got SELinux covered:

oc$ ls -laZ /srv/data
drwxr-xr-x. ceph ceph system_u:object_r:ceph_var_lib_t:s0 .
...

And use CephX with autogenerated keys:

oc$ ceph auth list
installed auth entries:

client.admin
        key: AQC2Pr9XAAAAABAAOpviw6DqOMG0syeEYmX2EQ==
        caps: [mds] allow *
        caps: [mon] allow *
        caps: [osd] allow *
client.openstack
        key: AQC2Pr9XAAAAABAAA78Svmmt+LVIcRrZRQLacw==
        caps: [mon] allow r
        caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics

But which OpenStack service is using Ceph? The storage-environment.yaml file has some informations:

uc$ grep -v '#' tripleo-heat-templates/environments/storage-environment.yaml | uniq

 resource_registry:
   OS::TripleO::Services::CephMon: ../puppet/services/ceph-mon.yaml
   OS::TripleO::Services::CephOSD: ../puppet/services/ceph-osd.yaml
   OS::TripleO::Services::CephClient: ../puppet/services/ceph-client.yaml

 parameter_defaults:
   CinderEnableIscsiBackend: false
   CinderEnableRbdBackend: true
   CinderBackupBackend: ceph
   NovaEnableRbdBackend: true
   GlanceBackend: rbd
   GnocchiBackend: rbd

The registry lines enable the Ceph services, the parameters instead are setting Ceph as backend for Cinder, Nova, Glance and Gnocchi. They can be configured to use other backends, see the comments in the environment file. Regarding the pools:

oc$ ceph osd lspools
0 rbd,1 metrics,2 images,3 backups,4 volumes,5 vms,

Despite the replica size set by default to 3, we only have a single OSD so with a single OSD the cluster will never get into HEALTH_OK:

oc$ ceph osd pool get vms size
size: 3

Good to know, now a new deployment with more interesting stuff.

A more realistic scenario

What makes it "more realistic"? We'll have enough OSDs to cover the replica size. We'll use physical disks for our OSDs (and journals) and not the local filesystem. We'll cope with a node with a different disks topology and we'll decrease the replica size for one of the pools.

Set a default disks map for the OSD nodes

Define a default configuration for the storage nodes, telling TripleO to use sdb for the OSD data and sdc for the journal:

ceph_default_disks.yaml
  parameter_defaults:
    CephStorageExtraConfig:
      ceph::profile::params::osds:
        /dev/sdb:
          journal: /dev/sdc

Customize the disks map for a specific node

For the node which has two (instead of a single) rotatory disks, we'll need a specific map. First get its system-uuid from the Ironic introspection data:

uc$ openstack baremetal introspection data save | jq .extra.system.product.uuid
"66C033FA-BAC0-4364-9E8A-3184B5952370"

then create the node specific map:

ceph_mynode_disks.yaml
  resource_registry:
    OS::TripleO::CephStorageExtraConfigPre: tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml

  parameter_defaults:
    NodeDataLookup: >
     {"66C033FA-BAC0-4364-9E8A-3184B5952370":
       {"ceph::profile::params::osds":
         {"/dev/sdb": {"journal": "/dev/sdd"},
          "/dev/sdc": {"journal": "/dev/sdd"}
         }
       }
     }

Fine tune pg_num, pgp_num and replica size for a pool

Finally, to override the replica size (and why not, PGs number) of the "vms" pool (where by default the Nova ephemeral disks go):

ceph_pools_config.yaml
  parameter_defaults:
    CephPools:
      vms:
        size: 2
        pg_num: 128
        pgp_num: 128

Zap all disks for the new deployment

We also want to clear and prepare all the non-root disks with a GPT label, which will allow us, for example, to repeat the deployment multiple times reusing the same nodes. The implementation of the disks cleanup script can vary, but we can use a sample script and wire it to the overcloud nodes via NodeUserData:

uc$ curl -O https://gist.githubusercontent.com/gfidente/42d3cdfe0c67f7c95f0c/raw/1f467c6018ada194b54f22113522db61ef944e20/ceph_wipe_disk.yaml

ceph_wipe_env.yaml:
  resource_registry:
    OS::TripleO::NodeUserData: ceph_wipe_disk.yaml

  parameter_defaults:
    ceph_disks: "/dev/sdb /dev/sdc /dev/sdd"

All the above environment files could have been merged in a single one but we split them out in multiple ones for clarity. Now the new deploy command:

uc$ openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/storage-environment.yaml --ceph-storage-scale 3 -e ceph_pools_config.yaml -e ceph_mynode_disks.yaml -e ceph_default_disks.yaml -e ceph_wipe_env.yaml
Deploying templates in the directory /home/stack/example/tripleo-heat-templates
...
Overcloud Deployed

Here is our OSDs tree, with two instances running on the node with two rotatory disks (sharing the same journal disk):

oc$ ceph os tree
ID WEIGHT  TYPE NAME                        UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.03119 root default
-2 0.00780     host overcloud-cephstorage-1
 0 0.00780         osd.0                         up  1.00000          1.00000
-3 0.01559     host overcloud-cephstorage-2
 1 0.00780         osd.1                         up  1.00000          1.00000
 2 0.00780         osd.2                         up  1.00000          1.00000
-4 0.00780     host overcloud-cephstorage-0
 3 0.00780         osd.3                         up  1.00000          1.00000

and the custom PG/size values for for "vms" pool:

oc$ ceph osd pool get vms size
size: 2
oc$ ceph osd pool get vms pg_num
pg_num: 128

Another simple customization could have been to set the journals size. For example:

ceph_journal_size.yaml
  parameter_defaults:
    ExtraConfig:
      ceph::profile::params::osd_journal_size: 1024

Also we did not provide any customization for the crushmap but a recent addition from Erno makes it possible to disable global/osd_crush_update_on_start so that any customization becomes possible after the deployment is finished.

Also we did not deploy the RadosGW service as it is still a work in progress, expected for the Newton release. Submissions for its inclusion are on review.

We're also working on automating the upgrade from the Ceph/Hammer release deployed with TripleO/Mitaka to Ceph/Jewel, installed with TripleO/Newton. The process will be integrated with the OpenStack upgrade and again the submissions are on review in a series.

For more scenarios

The mechanism recently introduced in TripleO to make composable roles, discussed in a Steven's blog post, makes it possible to test a complete Ceph deployment using a single controller node too (hosting the OSD service as well), just by adding OS::TripleO::Services::CephOSD to the list of services deployed on the controller role.

And if the above still wasn't enough, TripleO continues to support configuration of OpenStack with a pre-existing, unmanaged Ceph cluster. To do so we'll want to customize the parameters in puppet-ceph-external.yaml and deploy passing that as argument instead. For example:

puppet-ceph-external.yaml
  resource_registry:
    OS::TripleO::Services::CephExternal: tripleo-heat-templates/puppet/services/ceph-external.yaml

  parameter_defaults:
    # NOTE: These example parameters are required when using Ceph External and must be obtained from the running cluster
    #CephClusterFSID: '4b5c8c0a-ff60-454b-a1b4-9747aa737d19'
    #CephClientKey: 'AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ=='
    #CephExternalMonHost: '172.16.1.7, 172.16.1.8'

    # the following parameters enable Ceph backends for Cinder, Glance, Gnocchi and Nova
    NovaEnableRbdBackend: true
    CinderEnableRbdBackend: true
    CinderBackupBackend: ceph
    GlanceBackend: rbd
    GnocchiBackend: rbd
    # If the Ceph pools which host VMs, Volumes and Images do not match these
    # names OR the client keyring to use is not named 'openstack',  edit the
    # following as needed.
    NovaRbdPoolName: vms
    CinderRbdPoolName: volumes
    GlanceRbdPoolName: images
    GnocchiRbdPoolName: metrics
    CephClientUserName: openstack
    # finally we disable the Cinder LVM backend
    CinderEnableIscsiBackend: false

Come help in #tripleo @ freenode and don't forget to check the docs at tripleo.org! Some related topics are described there, for example, how to set the root device via Ironic for the nodes with multiple disks or how to push in ceph.conf additional arbitraty settings.

OpenStack summit in Tokyo and TripleO

2015-11-03T17:30:00+01:00

OpenStack summit Tokyo anyone? I've been there and thought it was a very well organized event, in a nice location. Every minute together with peers seemed worth it to me. This said, let's talk about the actual sessions. I spent most of my time at the TripleO and Heat sessions, with a little detour on Magnum. Plus some booth crawling.

TripleO

We had three sessions and a full day meetup. The first discussed the recent work done to allow for the deployment of OpenStack in containers, using TripleO; the changes to deploy the compute nodes and a majority of the controller roles in containers are up for review, which is great; some notes can be found in the tripleo-mitaka-containers pad. It was very nice for me to see the TripleO bits delivering on the promise of being flexibile; deploying in containers requires minimal changes to the core tools and yet inherits important features like network isolation and support for Ceph.

Another session was on upgrades, which isn't a trivial goal as it pushes on both Heat and Puppet. The proposed plan is to start with a CI job attempting to upgrade an overcloud from the stable branch to the master branch; more in the tripleo-kilo-to-liberty-upgrades pad. I will hopefully write more about this topic in future posts.

There has also been a session on the CLI and UI clients. Again, not a simple problem because the Heat templates are constantly ongoing refactor to implement new features and because putting too much business logic into the client limited our flexibility in the past. We'll probably see a shared library for the newer CLI/UI, with less business logic and probably a REST API, more in the tripleo-mitaka-restapi pad.

Then the community meetup on Friday covered a few additional topics. We reviewed the status of the CI with ideas for more jobs, discussed some ideas to provision the puppet modules via Heat instead of shipping them with the images, collected lots of good feedback from actual OpenStack operators and, last but not least, discussed the recent work from Dan Prince to have composable roles. Kudos to Dan on the composable roles which also seems to work pretty well with the effort to containerize the deployment. Notes about the community meetup are in the tripleo-mitaka-meetup pad.

Heat

I've joined only those sessions were I could at least understand the topics and yet there is quite a lot to write about. First an interesting session focused on a problem with dependencies across nested stacks, which we make large us of in TripleO. Both the issue and the consequences are much easier to understand with a picture than with words. Zane Bitter had indeed prepared some demo material for the session, including pictures, which should be linked from the mitaka-heat-break-stack-barrier pad.

Then a session to discuss issues affecting large stack deployments. Major points for discussion were of three types: token timeouts, batching of operations and resources status polling, with resources polling being probably the hardest to cope with. More in the mitaka-heat-large-stacks pad.

Then a session to collect input from users and ops. My proposal here was for the introduction of 'immutable resources' where resources with such a proerty wouldn't get deleted or updated during a stack update. One proposed solution for this is a warning message, outputted by the stack validation process, in case of resources deletion. More on this and on the session in general in the mitaka-heat-user-ops pad.

There has also been an interesting conversation about the changes from a spec meant to introduce support for external resources, which is something TripleO could use soon.

Magnum

The Magnum project seems to be facing some of the problems which TripleO faced in past as well, in relation to the Heat templates. The goal seems to be to customize the templates based on the input parameters; I joined the session to learn more about the Magnum approach and the proposed solutions.

It seems to me that while TripleO relies on actual Heat features to allow for different implementations of the same resource, via resource registry, Magnum is working instead on a sort of pre-processor (powered by Jinja) which generates Heat templates using conditionals and includes from the Jinja language. More on this is in the launchpad bug 1501045.

Who said installers?

It was nice to see TripleO mentioned in at least two more sessions, one where Mirantis directly compared Fuel to the Red Hat OpenStack Platform Director and another where four different automated installers were reviewed. Each had its moments.

I think one good thing about TripleO is that it is pretty flexible and offers some capabilities to maintain the environment even after the initial installation; hopefully both will make it interesting to many.

RDO meetup

We had a meetup for the RDO project and it cleared up a lot of questions on how the packages are built and tested for both the trunk and the stable branches. Good news is that it is now possible to try TripleO, on CentOS, with the latest RDO bits. The docs at tripleo.org are documenting exactly that.

Now, what about the event? Well, as an ATC, I have two things to say about the event more in general. First, I'd really like the design summit to start on Tuesday. Simply put, there are so many sessions going on that it is impossible to join anything out of the most immediate interests. My thinking is that one more day could at least lower parallelism!

Another comment is, please make the keynotes a little less marketish. I know the circus runs on actual money but my feeling is that the keynotes, on both Tuesday and Wednesday, left some sessions go a little too far into the product marketing area when they were probably supposed to present an actual OpenStack use case.

Last but not least, it was great to have some time to meet face to face peers and coworkers from all over the world. Including some who are not exactly peers, like Matt, to whom I ended up asking if he was a newcome to the TripleO project during dinner. I'm sure next time I will remember and do better.

On TripleO and the EnablePacemaker bool

2015-05-01T15:02:00+02:00

Yes, EnablePacemaker! A little old fashioned and not very well suited for OpenStack you say heh? Let's try to talk a bit about this then. News is that we're adding into TripleO/Puppet an option to deploy Pacemaker on the Overcloud nodes. Puppet you said? Sure TripleO can use Puppet now for the configuration of the Overcloud nodes, ask Dan Prince :) Actually usage of Puppet is nowdays the preferred approach and many features are available only in the Puppet scenario ... amongst which the EnablePacemaker bool in subject.

To go back to Pacemaker, no this won't be the Active/Passive implementation described in the OpenStack HA docs. Not at all, there are no special resource agents for the OpenStack services and more importantly, there is no Active/Passive, when not needed. Instead, the architecture looks a lot more like the Active/Active scenario described in tha doc, except there is Pacemaker coping with a few uncovered bits.

Whoever tried to setup a RabbitMQ cluster from scratch

dan prince
ha docs

####### OLD STUFF #######

One of the topics discussed during the TripleO mid-cycle meetup in RDU was our status in relation to deploying OpenStack in a highly available manner. This had been worked on for some time and recently reached a usable state.

Majority of complications seem to come from two factors: 1) we need to guarantee availability of external services too, like the database and the message broker, which aren't exactly designed for a scale-out scenario, 2) despite the OpenStack services being designed around a scale-out concept, while attempting to achieve that in TripleO we spotted a number of weak angles, some of which could be worked around, others instead still need some changes in the core service. You're encouraged to try what we have available today and help with the rest.

So to try out OpenStack HA with TripleO you just set a number >= 3 for OVERCLOUD_CONTROLSCALE and continue with devtest as usual. Nodes will be configured appropriately:

export OVERCLOUD_CONTROLSCALE=3
source scripts/devtest_variables.sh
...

Don't forget this is only tested on a few distros for now, I'd pick some Fedora 20.

On the controller nodes, MariaDB with Galera (for Fedora) is going to provide for a reliable SQL. There is still some work in progress to make sure the Galera cluster can be restarted correctly should all the controllers go down at the same time but, for single node failures, this should be safe to use.

RabbitMQ nodes are clustered and balanced (via HAProxy), queues replicated.

And with regards to the OpenStack services, these are configured in a balancing manner (again, using HAProxy) except for those cases where this wouldn't have worked, notably the Neutron L3 agent and the Ceilometer Central agent, yet these are under control via Pacemaker and a single instance is expected to be running at all times. Cinder instead remains uncovered as volumes would require a shared storage for proper HA. A spec has been proposed for this though.

Also, behind the scenes, the Heat template language addon shipped as merge.py and included in tripleo-heat-templates, which allows for example for scaling of the resources definition, is currently going to be removed and replaced with code living entirely in Heat.

And there is more so once you tried, join us on #tripleo @ freenode for the real fun!

Ceph for Cinder in TripleO

2015-01-06T17:15:00+01:00

A wrap up on the status of TripleO's Cinder HA spec. First, a link to the cinder-ha blueprint, where you can find even more links, to the actual spec (under review) and the code changes (again, still under review). Intent of the blueprint is for the TripleO deployments to keep Cinder volumes available and Cinder operational in case of failures of any node.

This said, should $subject sound interesting to you, beware the code still needs reviews, probably polishing and surely more features ... so take this post as a call for help as well! Now some details.

For Cinder to remain operational, in TripleO we deploy the cinder-{api,schduler,volume} services on all controller nodes (as it was even before the proposed spec) but we also customize the host setting so that all instances of cinder-volume will share an identical identifier. This is so that all instances can perform operations on all volumes, otherwise, the particular host which created a volume would be the only one capable of performing further operations on it. This exposed some potential issues with Cinder due to concurrent mutation of same resource from multiple nodes and they already are taken care of, in Cinder, with the state-enforcer blueprint.

Then, for the volumes to remain available, TripleO will deploy a Ceph cluster together with the OpenStack components, to be used as a backend for Cinder. Each and every controller will host a Ceph monitor while an arbitrary number of additional nodes can be configured as Ceph OSDs. The basic idea is that by doing so people will be able to scale out the number of controllers and/or data nodes independently, in an attempt to suit the needs for more space or more CPU resources depending on the use case. In addition to that, network partitioning should also be easy to achieve as traffic from Nova nodes will be directed to the OSDs nodes only for volumes I/O, not to the controllers. Note though that the existing implementation does not yet implement support for the network partitioning but, on a good note, it does use cephx already to secure access to data.

As of today, assuming you checkout by yourself, as linked from the cinder-ha blueprint, the needed TripleO changes, everything should just work by setting some value > 0 for the CEPHSTORAGESCALE env variable, which represents the number of OSD nodes you want to deploy. Should you need it, in the Heat templates it is possible to customize a number of useful Ceph settings, like the number of replicas for the data (needs to be lower than the number of OSD nodes). Now go try it out and come back with some feedback! :)

TripleO vs OpenStack HA

2014-08-21T15:02:00+02:00

So to try out OpenStack HA with TripleO you just set a number >= 3 for OVERCLOUD_CONTROLSCALE and continue with devtest as usual. Nodes will be configured appropriately:

export OVERCLOUD_CONTROLSCALE=3
source scripts/devtest_variables.sh
...

Don't forget this is only tested on a few distros for now, I'd pick some Fedora 20.

RabbitMQ nodes are clustered and balanced (via HAProxy), queues replicated.

And there is more so once you tried, join us on #tripleo @ freenode for the real fun!