We currently found our self in a tight spot, a somewhat careless refactoring broke our protobuf format. As we are in multiple datacenters we need to handle 2 versions at a time during transitions between versions. This was not totally unexpected, however the problem occurred a little sooner than we anticipated.
We use Topics* on Azure servicebus to distribute messages to multiple data centers having a subscription for each data center. We then have worker roles reading of the subscriptions, updating DocumentDb and Redis.
A Topic acts as a Queue to the sender. Inside a Topic there can be one or more Queues, called subscriptions. When a message is put on sent to the Topic, it is "copied" to all the subscriptions, which can then be processed individually. Read more here https://azure.microsoft.com/en-us/documentation/articles/service-bus-dotnet-how-to-use-topics-subscriptions/
Having Topics inplace instead of a regular queue proved to be a nice spot when dealing with an existing application. Besides having multiple subscriptions, it is also possible to filter messages on each subscription. Filtering is done using a syntax aligned with SQL-92, and can be easily applied when creating the subscription.
With filtering in hand we could easily get out of our little corner
- Add version as a property to each message
- Add filter to the subscriptions
- Deploy new worker role reading from a subscription with naming convention after the message version.
- Swap staging and production slot for the worker role, so the new version is running in production and the old one in staging.
- When all systems are updated we delete the staging slot with the old version and remove the old subscription.