M3DB Allow snapshots to be merged for lower active block memory footprint

[robskillington]

Let me know if you feel this warrants a design doc, but this proposal is to allow using the arbitrary out of order writes code path to merge snapshots together if configured correctly so time series that have not been recently written to can be unloaded from memory.

The config will be either set to a specific memory goal (and will unload as best possible to meet that goal) or be set to unload from memory anything that has been snapshotted and hasn't been written to in X minutes/whatever (time duration).

cc @prateek @richardartoul @justinjc

[richardartoul]

@robskillington I think this 100% needs a design doc. There's many different ways this could be implemented so can't really have a reasonable discussion about it until there is a doc to look at.

That said the main thing I find confusing about this is that snapshot files are never converted to real data files, so how does this help us? It seems like the more straightforward path to accomplish what you want is to simply delete the snapshotting code path and trigger more regular flushes that merge with whats already on disk. Once the out of order write stuff lands I don't think we'll need snapshotting at all.