Conversation
c95f733 to
16e155b
Compare
16e155b to
42597c6
Compare
GCdePaula
left a comment
There was a problem hiding this comment.
Nice work!
I’ve left two inline comments (range split + atomic cursor/input persistence).
There are two follow-ups I suggest we handle on top of my open branch:
-
Input reader lifecycle
Right now the reader thread is detached, so we don’t have proper shutdown/join/error propagation tomain.
In my open PR we already have a cleaner lifecycle pattern for background workers; I suggest we wire the reader into that same pattern. -
Config surface
This PR introduces new env knobs.
My branch has moved config parsing toclap, so we should add the new reader options there to keep configuration consistent.
Given that, my recommendation is:
- merge my branch first,
- then rebase this input-reader work on top,
- and apply the two inline fixes plus the two follow-ups above.
| let blocks = 1 + end_block - start_block; | ||
| let half = blocks / 2; | ||
| let middle = start_block + half - 1; |
There was a problem hiding this comment.
We're not handling the base case. Moreover, the middle calculation can underflow.
I propose these changes:
if start_block >= end_block {
// Cannot partition further; bubble original error.
return Err(vec![e]);
}
let middle = start_block + (end_block - start_block) / 2;
| self.storage.append_safe_direct_inputs(&batch)?; | ||
| self.storage | ||
| .input_reader_set_last_processed_block(current)?; |
There was a problem hiding this comment.
These two operations have to be atomic; we need to perform them inside a single db transaction.
I'd create a single function in storage called something like append_safe_inputs_and_advance_cursor(inputs, new_last_processed_block) that performs both operations atomically.
Sounds good, let's merge your PR first, then adjust mine afterwards. |
Add safe-input-reader and related storage logic