paulmach/consistent.go

Output on the current master branch of github.com/strava/go.serversets results in

2015/08/17 22:50:01 cluster endpoints: [10.0.16.69:11211 10.0.22.188:11211 10.0.26.214:11211]
2015/08/17 22:50:02 mcset results
2015/08/17 22:50:02     58856   10.0.16.69:11211
2015/08/17 22:50:02     18389   10.0.22.188:11211
2015/08/17 22:50:02     22755   10.0.26.214:11211

This is due to the non-uniformity of the crc32 hash function. :(

I ran into this exact non-uniformity when I implemented our assignment randomizer for the experiment reporting framework.

See this article:
http://research.neustar.biz/2012/02/02/choosing-a-good-hash-function-part-3/

I believe we ended up going with Murmur3.

Code sample from wikipedia:

uint32_t murmur3_32(const char *key, uint32_t len, uint32_t seed) {
    static const uint32_t c1 = 0xcc9e2d51;
    static const uint32_t c2 = 0x1b873593;
    static const uint32_t r1 = 15;
    static const uint32_t r2 = 13;
    static const uint32_t m = 5;
    static const uint32_t n = 0xe6546b64;

    uint32_t hash = seed;

    const int nblocks = len / 4;
    const uint32_t *blocks = (const uint32_t *) key;
    int i;
    for (i = 0; i < nblocks; i++) {
        uint32_t k = blocks[i];
        k *= c1;
        k = (k << r1) | (k >> (32 - r1));
        k *= c2;

        hash ^= k;
        hash = ((hash << r2) | (hash >> (32 - r2))) * m + n;
    }

    const uint8_t *tail = (const uint8_t *) (key + nblocks * 4);
    uint32_t k1 = 0;

    switch (len & 3) {
    case 3:
        k1 ^= tail[2] << 16;
    case 2:
        k1 ^= tail[1] << 8;
    case 1:
        k1 ^= tail[0];

        k1 *= c1;
        k1 = (k1 << r1) | (k1 >> (32 - r1));
        k1 *= c2;
        hash ^= k1;
    }

    hash ^= len;
    hash ^= (hash >> 16);
    hash *= 0x85ebca6b;
    hash ^= (hash >> 13);
    hash *= 0xc2b2ae35;
    hash ^= (hash >> 16);

    return hash;
}