Add Normalized DN Cache with sharded S3-FIFO design

[droideck]

No description provided.

[Firstyear]

Very interesting proposal, I'm curious about the locking design however. :)

[Firstyear]

Interestingly the transactional nature is separate from the algorithm used - if S3-FIFO is very 'good' then there is no reason we couldn't use it in concread instead of ARC.

Also within the way this works in 389-ds, we actually only use readers and we async-submit changes for the main cache to include.

[droideck]

This isn't really a concread-vs-S3-FIFO question. It's whether the NDN cache shape we have today needs the transactional power concread provides.

Concread's ARCache gives readers a transactionally consistent snapshot, a haunted set so evicted-then-revived keys can't clobber newer writes, and an mpsc channel from readers to the writer so reader hits influence eviction (as you describe in CACHE.md). IIUC, these properties are correct for caches whose values can change under readers and whose readers need a stable view across a transaction. The entry cache and the index cache fit that description, yep.

IMO, the NDN cache doesn't. The value is a pure function of the key. Two readers asking for the same DN at different transaction ids get the same answer or a cache miss, never a stale-but-different answer. There's no rollback to support and no temporal ordering between readers and writers to protect. The haunted set and the reader-to-writer channel pay for guarantees the NDN cache doesn't need.

So for this proposal the relevant comparison is concread's transactional shell vs no shell, for a cache whose values are immutable functions of the key. For this specific cache, no shell wins on the hot path, I think. Which is why the new design is sharded S3-FIFO directly rather than ARC-inside-concread.

The reader-doesnt-block-writer property you get from async-submit carries over to the design through a different mechanism: shared read lock with a relaxed atomic counter on hit, write lock only on miss-insert. No reader-to-writer mpsc, but readers stay off the write path on the hot case. Plus the cache is hash sharded across per-shard locks, so reads on different shards never contend at the lock level.

[Firstyear]

This isn't really a concread-vs-S3-FIFO question. It's whether the NDN cache shape we have today needs the transactional power concread provides.

Part of the goal historically of using concread in 389 was to show the cache was effective transactionally, and could then be used in other places where it matters - especially with LMDB for example since it shares an identical transaction model.

However, no one else on the team really ever showed interest in that, and the changes to make 389-ds strongly transactional don't seem to have eventuated even with the changes to support LMDB.

So with that in mind, in this case yes, maybe the s3-fifo is a better option. That's fine. The number of cases for the NDN to rely on transactions would be low.

One area the concread cache has a possibility to outshine the s3-fifo is a majority read-only workset. This is because we end up with most cache lines in the shared state (where s3-fifo will be invalidating them frequently due to atomics). But this is very subjective.

Something to try in your benchmarking is different cache sizes - extremely small, moderate, and large ndn cache sizes. This also affects the performance.

[droideck]

Good point. I added a cache-size + multithreaded sweep.

For the multithreaded memberOf cascade run, changing the configured cache size did not change the direction of the result. This run is mostly lookup-path and contention, not eviction pressure. S3-FIFO stayed around +12% ops/s vs concread, with lower p95 in the same runs.

So yeah, on the read-mostly point, I agree this is where concread has the strongest argument. The useful thing from the new runs is that we see that the hot-DN shape stays close to concread, while the multi-key memberOf shape stays ahead across the cache sizes. Hits take the shared side of a per-shard parking_lot::RwLock, not an exclusive mutex. It is still not a zero-write read path though: the hit/miss stats are per-shard AtomicU64 counters with relaxed updates (we don't need great precision there), and the per-entry AtomicU8 freq counter only CASes until it saturates (which is great for the hot DNs). I think that is a reasonable trade for NDN
WDYT?

[Firstyear]

This run is mostly lookup-path and contention, not eviction pressure. S3-FIFO stayed around +12% ops/s vs concread, with lower p95 in the same runs.

There are some other things I want to suggest you can try, just for comparison.

1. set_reader_quiesce to false and spawn a dedicated quiesce thread.
2. set_look_back_limit to 8 (decreased from 32)

My guess is these two are the biggest factors youre seeing that causes the p95 changes, as well as some of the operational performance difference, especially the with quiesce.

So yeah, on the read-mostly point, I agree this is where concread has the strongest argument. The useful thing from the new runs is that we see that the hot-DN shape stays close to concread, while the multi-key memberOf shape stays ahead across the cache sizes.

A good question is why is memberOf hitting the NDN cache so hard in the firstplace? I assume we're still using the "old" memberOf algorithm (compared to the one I proposed around 2017 that wasn't adopted - note; yes, it's still the slow algo).

Hits take the shared side of a per-shard parking_lot::RwLock, not an exclusive mutex.

It will make the write side of the equation much slower though as you need all readers to drains before a write can proceed. So be mindful of that consequence.

It is still not a zero-write read path though: the hit/miss stats are per-shard AtomicU64 counters with relaxed updates (we don't need great precision there), and the per-entry AtomicU8 freq counter only CASes until it saturates (which is great for the hot DNs). I think that is a reasonable trade for NDN WDYT?

Yeah, that's fine. Realistically if you plan to only do "sampling" based stats and are willing to forego some precision for performance, you can do a trick. You actually only need to record some results, not all to get a hit ratio. For 100,000 events, if you sample 1/10th of them (10,000) as a hit or miss, then you actually end up with a result that is statistically likely to be within 1% of the true value. This means you can probably forgo a lot of atomic calls by sampling instead of recording everything.

(check https://www.abs.gov.au/websitedbs/D3310114.nsf/home/Sample+Size+Calculator for your own homework)

[droideck]

Okay, I tried -- set_reader_quiesce(false) and a dedicated quiesce thread, look_back_limit = 8.
Plus sampling, yeah, I also thought of this before but wasn't sure for the first iteration. I did a 1-in-10 stats version and it's better, thanks!
And beside that, I additionally tuned my S3-FIFO implementation as I found a couple of issues after more benchmarks (but we'll discuss the Rust details later, I think)

Regarding your concread tuning suggestion, yep! It's better with a dedicated quiesce thread. It's clearly visible in Criterion. The tuned concread is a lot faster than the wrapper path we use today, and it wins the isolated single hotkey bench outright (including against S3-FIFO). But only in the isolated Rust microbench.

But with the integration tests... I still can't see it at the LDAP level though. On the memberOf cascade the tuned concread run stays inside the production concread spread. p99 is a bit worse even, and the p95 gap to S3-FIFO does not really move. So my read is that the per-read quiesce cost is measurable, but it is not the thing dominating the server-level p95 here. S3-FIFO still lands around +12% to +21% ops/s on the cascade against either concread mode. One catch from setting this up: exported env never reaches ns-slapd (systemctl drops it, lib389's direct exec passes an empty env), so the benchmark now checks for the quiesce worker thread inside the running process before accepting a run.

Side note on the doc tables: the numbers changed because this is a fresh run with all four variants on one VM and one build, and the fixture is smaller than the earlier draft's (10,220 entries vs ~100k). Hit ratio stays good either way, so the cascade comparison is lookup-path bound; the eviction side is covered by the scan test and the Criterion capacity sweep, which runs a ~100k-DN working set through caps from 1k to 500k entries.

So. On memberOf, IIUC the recompute path goes through memberof_get_groups_r and internal searches; a live modify first walks the nested member graph and then triggers that recompute for each affected entry. The DN work is spread around: the core server normalizes every incoming DN-syntax mod value before the plugin even runs, and the walk builds fresh Slapi_DNs and (member=...) filter values along, each one NDN lookup. So yeah, the cascade is quite representative for the NDN-heavy ops. Plain searches do not hit the same path in the same way.

On the write-side drain, yup.. That's the part I got worried about after your past comment, so I'm glad the measurements look ok so far.

The scan test forces a miss-insert per scanned DN through an undersized cache. That is the worst cache-write/read mix I could come up with at server level. In that run, S3-FIFO scan wall time stays within about 7% of running with no cache at all, so the miss-inserts are nearly free. Concread is 16% to 32% slower than no-cache on the same scan. So I do not see a write-drain cliff there.

Mechanically the drain is per shard, not global. A miss-insert only waits for readers on the shard for that DN. In the warm NDN cases we are mostly looking at hit ratios near 1.0, so cache insert writes are not very common after warmup. That could still be wrong for some workloads, I think...

On the sampled stats: it showed the benefit in Criterion nicely. At the integration bench/LDAP level I don't see the difference... We're still not at the NDN bottle neck in the many places where NDN's involved.

[Firstyear]

Plus sampling, yeah, I also thought of this before but wasn't sure for the first iteration. I did a 1-in-10 stats version and it's better, thanks!

Yep - as I was taught long ago, atomics are fast but they are not free. So that's probably really not helping the comparisons here.

On memberOf, IIUC the recompute path goes through memberof_get_groups_r and internal searches; a live modify first walks the nested member graph and then triggers that recompute for each affected entry.

Again, because it's using the slower MO algo.

Still as we previously mentioned, concread does a lot to create a transactional cache - something you don't (*) need here in the NDN.

(*) technically you should need it, but 389-ds transactions are .... weird.

[Firstyear]

Oh something else to consider is the S3 shards are fixed size where as in ARC they're variable size. S3 uses a ghost set to do promotions, but it seems to be like a fixed set recent ghost list. If anything S3 really is kind of ARC but without the adaptive weights, where you have a small shard/recent list, and then a longer frequent list.

So there is a risk that if you have a lot of high churn on the recent list, your frequent list will never populate since it will be continually pushing out the ghost/shard lists and never able to actually cause the needed 2 hits (either in S or G) that will trigger a move to M. Where as ARC will adapt in this case and extend the S set if there are too many hits into G increasing the likelihood of a promotion. Very subtle edge case but it exists nonetheless.

[droideck]

Oh something else to consider is the S3 shards are fixed size where as in ARC they're variable size. S3 uses a ghost set to do promotions, but it seems to be like a fixed set recent ghost list. If anything S3 really is kind of ARC but without the adaptive weights, where you have a small shard/recent list, and then a longer frequent list.

So there is a risk that if you have a lot of high churn on the recent list, your frequent list will never populate since it will be continually pushing out the ghost/shard lists and never able to actually cause the needed 2 hits (either in S or G) that will trigger a move to M. Where as ARC will adapt in this case and extend the S set if there are too many hits into G increasing the likelihood of a promotion. Very subtle edge case but it exists nonetheless.

Yeah, that's a concern. The paper authors made an explicit static-over-adaptive tradeoff here. The failure mode is bounded and observable, but it's still a tradeoff.

So, to my understanding, that's what we have:

S is sized at around 10% of cache. The window for a second hit before eviction from S is bounded by the number of distinct DNs that land in the same shard. To miss the freq >= 1 threshold, the workload would need to push that many one-hit DNs through the shard faster than the working set cycles.

The ghost queue G is sized to M, not to S. A key evicted from S to G and re-requested while still in G goes straight into M, not back through S (see Algorithm 1 in the paper: if x in G then insert x to head of M). G capacity is much larger than S, so the second-chance window through G is much wider than through S alone.

But the access-bits-cleared rule in the paper's 4.1 section means a G-to-M promotion lands at effective freq=0, and M's eviction will drop it on the first pass if it doesn't earn a hit in M. So a one-hit-via-G entry is still vulnerable. This is the part the static design trades for predictability...

On adaptivity: the paper tested an adaptive S3-FIFO variant (S3-FIFO-d, section 6.2) that resizes S and M based on ghost-queue hits. They report S3-FIFO-d beats static S3-FIFO only on the 2% adversarial tail of their dataset, and underperforms on the rest because the adaptive sizing keeps oscillating. I read the authors as choosing predictability over adaptivity.

For the NDN cache specifically, steady-state hit rate is over 99% with a working set that fits inside M with substantial headroom. I'm not waving this away... IIUC, the scan resistance is one of the reasons why you picked ARC in the entry cache, and that feels right. For the NDN cache, IMO, the static-size choice fits the workload, but it's still a choice. If we see this in production, both S's relative size and G's size are levers we can expose. I'd rather not commit to a cn=config knob until we have a workload that needs them.

[Firstyear]

To miss the freq >= 1 threshold, the workload would need to push that many one-hit DNs through the shard faster than the working set cycles.

Yeah, this could happen with a full table scan though, but I guess at that point you don't mind as this provides scanning resistance.

G capacity is much larger than S, so the second-chance window through G is much wider than through S alone.

Yep, exactly how the ghost sets in arc work (they are inversely sized to their related set, so a small recent set has a large recent ghost set).

I'd rather not commit to a cn=config knob until we have a workload that needs them.

Agreed, it's best not to add too many knobs.

[droideck]

Okay, I’m back from vacation! I did more testing around the hot-DN case and different cache sizes, refined the benchmark section, and updated the doc with the current conclusions. Please review...

[droideck]

@Firstyear @tbordaz gentle ping:)

[Firstyear]

Idea - shards == threads * 4? That way it reduces the likelihood of thread-shard contention? Probably a niche/micro-optimisation so not hugely important, could be a future improvement.