Clusters are not stateless

Clustered systems, such as GlusterFS and Ceph for example, cannot be deployed deterministically to groups of computers without coordination. Three ceph monitors, for instance, cannot each be the first one. Gluster peers and volumes cannot be defined until the other peers are up and waiting. This either requires a type of semaphore and wait system, or a stateful callback system.

Solutions

Semaphores using consul k/v

My first attempt at solving this problem uses the consul key/value store as a semaphore to allow ceph monitor nodes to be configured based on the condition of the other nodes. I will demonstrate how we’ve implemented this at IO, and discuss the benefits and pitfalls of this design.

Salt reactor + data

Reactor is a callback system that allows a state to communicate an event back to the master. Combined with the data functions, this allows stateful determination of an order of actions among multiple minions allowing intelligent deployment of clustered systems. I will demonstrate how we are implementing this at IO, and discuss the benefits of this design over the semaphore design.