TL;DR;
Hash indexing is dead, long live Range indexing

I recently did a DocumentBD talk for NNUG in Oslo, and made a discovery during my preparation of a demo on indexing.

The default index configuration when creating a collection in the portal differs from creating one with the API and also the documentation. Both the API and Documentation follows the "old" indexing settings[1] but this will show up as Custom in the portal.

Creating a collection using the portal, the new default is range for both string and numbers. It will also display as default settings in the portal.

Another observation is the new default indexing for the point datatype, points will now by default be indexed for spatial searchs.

I've reached out to the DocumentDB team, and apperantly they are changing the indexing a bit. As far as I understand they have optimized the storage for range, so that the difference between hash and range are very small. This may lead to the deprecation of hash indexing. This is a good thing, a simple product means less developer surprises and frustration.

In my dream world, this is a step on the road to skip and take functionality. The DocumentDB closely monitors our wishlist, so please take your time to upvote skip/take here.

[1] Code for old/API indexing settings,

{
  "indexingMode": "consistent",
  "automatic": true,
  "includedPaths": [
    {
      "path": "/*",
      "indexes": [
        {
          "kind": "Range",
          "dataType": "Number",
          "precision": -1
        },
        {
          "kind": "Hash",
          "dataType": "String",
          "precision": 3
        }
      ]
    }
  ],
  "excludedPaths": []
}

[2]New indexing settings

{
  "indexingMode": "consistent",
  "automatic": true,
  "includedPaths": [
    {
      "path": "/*",
      "indexes": [
        {
          "kind": "Range",
          "dataType": "Number",
          "precision": -1
        },
        {
          "kind": "Range",
          "dataType": "String",
          "precision": -1
        },
        {
          "kind": "Spatial",
          "dataType": "Point"
        }
      ]
    }
  ],
  "excludedPaths": []
}

During Desember last year I created an open source project called DoubleCache. It's a cleaner version of a cache pattern we use at work, written from scratch.

TL;DR

• Great source for a conference talk
• Helped me learn FAKE and Appveyor
• Made me reason about handling pull requests, figuring out a how I prefer to do this (blogpost coming).
• Showcase skills and thoughts for potential customers and employers.

In details

The first benefit my open source project gave me was backing up a talk I had planned for the Norwegian .Net user group. The talk was about Redis and our experiences of moving to a centralized cache. Backing the talk with my own implementation provided very clear samples.

Truth be told, DoubleCache wasn't yet public when I did my talk, but it became public a couple of days later. Pushing code out in the open forced me to setup a proper build and test setup, something I might not have done had it been a private project. With an actual need to learn FAKE it was easy to get the Instant Gratification Monkey out of the way[1]. I prefer xUnit and shoudly when writing C#, and this meant I had to dig into FAKE to a degree where I now feel comfortable with it.

It did not take long time before I had the first pull request, this one from a colleague but never the less I learned how to use AppVeyor. Well, not much to learn really when using FAKE. Having all your build configurations under version control is just great.

After yet another talk, I got the first pull request from someone I did not know, Clayton Sayer. He's done some great things, fixed a few configuration issues, added delete functionality and synchronous support. After looking at our repository at work and reading a few blogposts, I was sure that I wanted single commits when merging pull requests. I needed a way to do this, and at the same time be able to make modifications such as bumping the version number in the build script. More on this in the next blog post.

Finally, putting code out in the open forced me to think things through more than once. Not just because it's public, but it will serve as part of my resume. It will without doubt be a discussion point when talking to potential customers or future employers.

It has been, and continue to be, a great experience.

With great power comes great responsibility. The power of Lua scripting with Redis and what you can achieve using data structures beyond a key/value store is amazing. Starting out is quick and easy, and with some VIM tricks you can easily create an efficient feedback loop.

But then what, after a few iterations, how do you know that your scripts are still working?

Well, there's no reason not to include your Lua scripts in a test regime. To do this we can "simply" fire up a Redis server as a part of our tests. Now, you could argue that these are integration tests as we're using a service running out of process and perhaps that's right. But these tests run so fast, and are so specific, that I prefer to run them as a part of my development cycle. They're also executed alongside our regular unit tests, not our integration tests.

TL;DR

How does it work? We reference a redis server, start the server, execute our tests, and shutdown.

Step by step - creating our test project

0. Here's our system under test, a small and rather useless script for adding a key value to the cache. ``` public class SomeServiceUsingLuaScript { private IConnectionMultiplexer _redisConnection; public SomeCacheService(IConnectionMultiplexer redisConnection) { _redisConnection = redisConnection; } public void RunsALuaScript(string value) { _redisConnection.GetDatabase().ScriptEvaluate(string.Format("redis.call('set','test','{0}')", value)); } } ```
1. Create a new class library for your tests


2. Use nuget and add references to Redis-64, StackExchange.Redis, xunit.core, xunit.runner.visualstudio and Shouldly ``` Install-Package Redis-64 Install-Package StackExchange.Redis Install-Package xunit.core Install-Package xunit.runner.visualstudio Install-Package Shouldly ```
3. Create a folder called Redis4. As linked files, add redis-server.exe and redis.windows.conf from .\packages\redis-64.3.0.501\tools\ and set the Copy to output property to "Copy if newer"


4. To make sure we start the server once for all our tests we'll use the class and collection Fixture from xunit. The class fixture will be responsible for starting the redis server, and the collection fixture will allow us to run multiple test classes with the same instance of the class fixture. For small and closely related classes I prefer to keep them in the same file. Create a new file called RedisFixture.cs with the following content ``` [CollectionDefinition("RedisCollection")] public class RedisCollectionFixture : ICollectionFixture { //// Marker class for Collection Fixture }

   public class RedisFixture : IDisposable
   {
   private static Process redisProcess;

   public RedisFixture()
   {
   if (redisProcess == null || redisProcess.HasExited)
   redisProcess = StartRedis();
   }

   public void Dispose()
   {
   try
   {
   if (redisProcess != null)
   {
   if (redisProcess.HasExited)
   return;

       redisProcess.StandardInput.WriteLine("SHUTDOWN");
       Thread.Sleep(500);
   
       if (!redisProcess.HasExited)
         redisProcess.Kill();
       redisProcess.Dispose();
     }
   }
   catch
   {
   
   }
   

   }

   </li>
   <br/>
   <li>We can now write our first test using Redis
   

   [Collection("RedisCollection")]
   public class MyLuaScriptTests
   {
   [Fact]
   public void RunALuaScriptTest()
   {
   var connection = ConnectionMultiplexer.Connect("localhost");
   var service = new SomeCacheService(connection);

     service.RunsALuaScript("a");
   
     var result = (string)connection.GetDatabase().StringGet("test");
     result.ShouldBe("a");
   }
   

   }

   </li>
   <br/>
   <li>The test is now available in the Visual studio test explorer (it may fail running if you're using the 2.1.0 version of the xunit.runner.visualstudio. In that case, try to downgrade it to 2.0.1).  During the first test exection you may be prompted to adjust firewall settings, as Redis binds to a tcp port. 
   </ol>
   You're now ready to write Lua scripts for Redis in a safe and sound manner. 

Beginning of January I was contacted by the Bitshift meetup organizers, asking if I could come over and do a talk or two. Having two talks more or less prepared I decided that it could be a fun experience and also great practice. I gave a talk on caching (slides), and one on the architecture behind personalization of http://tv.nrk.no, (slides).

Before and during my previous talks I have noticed my heart rate goes somewhat haywire, so I decided it could be fun to track it during a talk. As I got on stage setting up and having a chat with one of the organizers I turned on my garmin and noticed straight away that I was having a small workout, that's a 140bpm. Apparently something happened about 5 minutes in, my pulse peaked out at 165 bmp. Here's the whole story
We had a break about an hour in before we started again, easy to spot.

I have a tendency to talk a bit fast, and also run demos waaay too fast. Heart rate may be a part of it, and I will keep tracking it to see if I can managed to calm down a bit. The first chance seems to be April 26th, when I will give a talk on DocumentDb for NNUG Oslo

Bitshift was a great experience, and it was really fun talking about something I care about and engage me. I was also surprised by the strong turnout, about the same as we see in Oslo.

Thanks for having me.

Picture by http://twitter.com/olavamjelde

If some of your functionality fails, should it take your website down?

Let's say you have a frontpage of a webshop displaying promoted items, a campaign and a list of the most popular items. If you fail fetching the popular items, or have some invalid data in it which throws an exception, having the whole frontpage fail would be really bad. You want the rest of the page to render.

This happened to us recently, a part of the frontpage failed. But instead of killing the whole page, only the failing feature failed. Here's how it looks when you enter the page

and below you have the failing functionality - in this case it was not even visible to the user until they clicked the "sist sent" (recently aired) tab. Crashing our whole page because of this would be plain out stupid.

This is just as much about isolating features as isolating failures. By isolating features we can start applying several patterns which will help in creating a responsive and well behaved page:

1. Circuit breaker for each feature/system. If the system is down, deliver what you got.

2. Timeouts per feature - if the most popular items feature will exceed your response time requirements, treat it as a fail and deliver what you got. That's better than building the request queue with waiting clients.

3. Cache - we can cache each part by itself. If we know when feature A and B changes but not C - we can pre-compute A and B and update the cache, or at least invalidate it. C can then have a less agressive cache setting.

So, by isolating your features you should be able to deliver some value to the user even if a part of your system is in error.

DoubleCache, https://github.com/AurumAS/DoubleCache, is my own open source implementation of a layered distributed cache, it builds upon solid projects like Redis, StackExchange.Redis and MsgPack and combines these with a local cache implementation on the .Net stack.

Some history

We're already running a similar solution with my current customer, NRK. Our motivation for creating this, came after migrating from Azure Managed Cache to Azure Redis Cache. Azure Managed cache has the nice feature of a local cache on the client, in front of the Managed cache. This is not available when using Redis, as Microsoft has not created their own client, instead they recommend using the (excellent) StackExchange.Redis client. When we moved to Redis, our CPU went haywire as we did a LOT of cache requests (often way too many, it's hard to notice when you hit local memory) and used BinaryFormatter for serialization. Besides cleaning up our data access, we needed our own layered cache. Our layered implementation over StackExchange.Redis is quite OK, however it is a bit tangled with old interfaces and not something which can be easily reused by others. DoubleCache aims to fix this.

My goals when creating DoubleCache

I created DoubleCache for a meetup where I talked about the penalties of moving to a remote cache, and I needed some code to highlight the problems with remote caches. As I wanted a clean implementation and other open source projects were too messy, I decided to implement my own with the following criteria

1. It should be possible to use as a local, remote or local and remote cache - with or without sync, through a single interface. Changing from a local to a remote cache should not require any changes in the client using it besides swapping interface implementation.
2. Each implementation should in it self be trivial
3. Follow the cache aside pattern
4. Extending the functionality should not require any modifications to existing imeplementations. Using standard patterns such as decorator ought to do it.
5. Serialization must be pluggable.

Usage

Add a reference to DoubleCache using nuget Install-Package DoubleCache and initialize the DoubleCache with a remote and a local cache.

var connection = ConnectionMultiplexer.Connect("localhost");
var serializer = new MsgPackItemSerializer();

_pubSubCache = CacheFactory.CreatePubSubDoubleCache(connection, serializer);

To use the cache call the GetAsync<T> method. This method takes a Func called dataRetriever. This method should call your repository or other service. The dataRetriever method executes if the requested key does not exist in the cache, adding the result to the cache.

var cacheKey = Request.RequestUri.PathAndQuery;

pubSubCache.GetAsync(cacheKey, () => _repo.GetSingleDummyUser()));

Implementation

The ICacheAside interface is the main part of DoubleCache, all variants relies on implementations of this single interface.

    public interface ICacheAside
    {
        void Add<T>(string key, T item);
        void Add<T>(string key, T item, TimeSpan? timeToLive);

        Task<T> GetAsync<T>(string key, Func<Task<T>> dataRetriever) where T : class;
        Task<T> GetAsync<T>(string key, Func<Task<T>> dataRetriever, TimeSpan? timeToLive) where T : class;

        Task<object> GetAsync(string key, Type type, Func<Task<object>> dataRetriever);
        Task<object> GetAsync(string key, Type type, Func<Task<object>> dataRetriever, TimeSpan? timeToLive);
    }

The Add method is implemented with fire and forget, hence it does not need to be Async as this is handled by the Stackexchange.Redis client.

DoubleCache comes with the following implementations of this interface

• LocalCache.MemCache - using System.Runtime.Memory
• Redis.RedisCache - using StackExchange.Redis client
• DoubleCache - a decorator wrapping a local and a remote cache
• PublishingCache - a decorator publishing cache changes
• SubscribingCache - a decorator supporting push notifications of cache updates

As seen in the usage example, combining the decorators and implementations provide a high level of flexibility. It comes at a little mental cost when wiring up the DoubleCache constructor. This can be relieved to hide the constructor behind a factory method.

LocalCache.MemCache

This uses the System.Runtime.Caching.MemoryCache.Default instance throughout the implementation.

A word of warning: These are mutable objects stored in memory. Be careful if you modify items retrieved from the cache. There can be only one.

Redis.RedisCache

A wrapper around StackExchange.Redis. Since the IConnectionMultiplexer should be a single instance, you will need to pass the Redis database in the constructor. The StringSet call to Redis use the FireAndForget option.

DoubleCache

This is a decorator used to wire the local and remote cache together.

To achieve this, it simply calles the local cache first and wraps the remote cache in the dataRetriever function.

return _localCache.GetAsync(key, type, () => _remoteCache.GetAsync(key, type, dataRetriever), timeToLive);

While DoubleCache will keep the remote cache in sync with any add calls made on the local cache it owns, syncing the local cache with remote cache changes triggered by other clients is not the responsibility of the DoubleCache wrapper. This is covered by the publishing and subscribing cache wrappers.

PublishingCache

A decorator for publishing changes when adding/updating items in the cache. The key and object type is published using the ICachePublisher interface, which then passes the message on to Redis Pub/Sub. As each cache implementation is responsible for adding an item to itself when the dataRetriever function is executed, ICachePublisher wraps this method causing a cache publish after it is executed.

public Task<T> GetAsync<T>(string key, Func<Task<T>> dataRetriever) where T : class
{
    return  _cache.GetAsync(key, async() => {
        var result = await dataRetriever.Invoke();
        _cachePublisher.NotifyUpdate(key, result.GetType().AssemblyQualifiedName);
        return result;
    });
}

The publishing cache is intended to wrap the remote/central cache implementation.

Subscribing cache

This decorator wraps a cache and through the implementation of ICacheSubscriber it will call Add on the wrapped cache when the ICacheSubscriber.CacheUpdate event is fired. The RedisSubscriber implementation of ICacheSubscriber will also need a reference to the remote cache, as the Pub/Sub message does not contain the cache item itself, only the key and the type.

The Subscribing cache decorator is intended to wrap the local cache.

Serialization

As mentioned in in my post on cache speed, there are a lot of options when it comes to serializers. By default, DoubleCache comes with BinaryFormatter and MsgPack.

If your data-types support it, I highly recommend something other than BinaryFormatter.

To implement another serializer you will need to implement the IItemSerializer interface

    public interface IItemSerializer
    {
        byte[] Serialize<T>(T item);

        T Deserialize<T>(byte[] bytes);
        T Deserialize<T>(Stream stream);
        object Deserialize(byte[] bytes, Type type);
    }

Whats next

I have a few open issues on GitHub which I intend to close:

1. Passing TimeToLive with pub/sub, or retrieving it from the cache
2. Creating a factory to make it simpler to get started
3. Adding delete operations

You'll find DoubleCache over at GitHub https://github.com/AurumAS/DoubleCache

Single digit response times are fully possible. And I think we should try to get there, it may not always be possible but that's no reason not to explore our options. A fast site will feel more responsive, and using less resources means that you can handle massive load on less resources.

Caching content is perhaps the single most effective measure. Living in a web world we have three main ways to go

1. Http caching (think RFC7234 and headers such as max-age, etags and If-Modified-Since)
2. Output cache, either built-in or products like Varnish
3. Application cache.

In this post we're going to look at the last of these, the one closest to your stack. This is where you end up when you need to alter the response before you emit data from the server.

Starting out with a small solution, often hosted on a single server, the application cache normally lives in-memory and the objects are kept as is. It's fast and relatively easy, the memory management is often taken care of for you.

After a while your application grows, you have multiple servers and having local caches may no longer be optimal. Problems such as cache sync arise - serving the same user different data just because the user ended up on a different server makes your application feel britle and awkward. Enter the remote cache. These days often implemented using Redis (Redis is really on a roll with both AWS and Microsoft offering Redis as a service).

So, what happens when you switch from a local cache to a remote cache? You get two penalties

1. Network IO
2. Serialization

Redis is fast, we're talking about 6-8ms response times with network IO and serialization, but when your goal is single digit response times, that's too much.

So, how can we have both? A layered cache, using both a remote cache and a local cache. This also plays well with this cloud thing where instances pop up when needed and instance memory is not really expensive.

How much memory do you really need to keep your entire site in memory?

To get this to play well we also need to

1. Keep our local caches in sync
2. A fast serialization when we hit the remote cache

To keep our local caches in sync we need a pub/sub mechanism, Redis has this built in and makes the implementation quite trivial. It's not a reliable sync, i.e. the messages are published, if a client is disconnected for some reason, it will not get the updated. I find that this is something we normally can live with.

The serialization part of it is also solved for us, there are several fast serialization procotols available to us, such as MsgPack, Protobuf, Bond or Avro. You will find that these protocols have their pros and cons, In .Net land MsgPack does not require class markup, but cannot cope with inheritance. Protobuf can, but requires a serialization definition. Yan Cui has a nice benchmark over at http://theburningmonk.com/benchmarks/.

What ever you do, BinaryFormatter is slow.

So, by creating a layered distributed cache we can achieve fast response times and a synchronized cache. As an added benefit we can use the pub/sub mechanism to push data changes to the cache from other services. This comes in real handy in the microservices world.

DoubleCache is my own implementation of a layered distributed cache for .Net. Using Redis as the remote storage, and System.Runtime.MemoryCache (can easily be switched to HttpCache). I've tried to stick to a single interface, making it possible to switch between a local cache, remote cache, local and remote, or a synced local and remote cache. The implementation follows the cache-aside and decorator patterns.

One of the first problems to solve when moving to the cloud is availability, or just plain uptime. Although it seems strange that the cloud should not be stable, but this is one of the constraints that is an enabler for solid solutions. By accepting unexpected service interuptions as a design requirement, the whole solution benefits. You have to design for failure and outages. Think Netflix and their simian army.

Outages in the cloud can be placed in two categories, failures or maintenance, and within maintenance we have planned and unplanned maintenance. To deal with this, and also to meet the SLA requirements, Microsoft has divided Azure into update domains and fault domains. Update domains are related to restarts due to maintenance and upgrades, while fault domains are closely related to HW, such as a rack or power unit https://azure.microsoft.com/nb-no/documentation/articles/virtual-machines-manage-availability/. It is therefore important to have multiple instances, at least one in each fault domain, preferably also one in each update domain. Once you have enough instances you will survive downtime.

Next step on the availability list is to support datacenter outages. To deal with this, one can start using multiple datacenters, potentially from different cloud providers, i.e. use one Azure DS and one Amazon DS. Your ability to move between cloud vendors will normally be limited by the choice of technology and implementation. Architecting for multi-vendor support will require a lot more work when it comes to operations and management. Any SAAS platform must have an equivalent offering with both vendors, not to mention how you will work with the different performance characteristics.

Service overview

Here's a short list of some of the Azure services and how replication works.

Service	Comments
Azure storage	Azure storage comes in four different flavors: From https://azure.microsoft.com/nb-no/documentation/articles/storage-redundancy/. Note that Zone Redundant storage is only available for block blobs.
Azure SQL
DocumentDb	Each collection in DcumentDb resides on multiple servers, however there's no built in geo-replication.
Redis	The basic Redis offering is not replicated. At the standard level there's a master and a slave. And in the recently announced Premium tier there's support for Redis clustering, sharding data over multiple nodes. https://azure.microsoft.com/en-us/documentation/articles/cache-how-to-premium-clustering/
Search	Search is divided in replicas and partitions. For read-write and indexing availability, at least 3 replicas are needed. Partitions relates to the storage volume.
Cloud services	The number of instances determines how good you can handle outages. You need at least one in each fault domain, preferably at least instances.
Cloud services/Virtual machines	The number of instances determines how good you can handle outages. You need at least two or more in an availability set/one in each fault domain..

Also, while a few of these services provide geo replication, supporting writes is a completely different story. Especially if we go down the SAAS path, handling writes during outages are not trivial. It may often seem easier to use a PAAS and then run software designed for multiple datacenter usage on top of that instead. I've started writing on a post where we'll go through a sample setup on how we can achieve a solution built with Azure SAAS and still survive a datacenter outage.