feat(block): async epoch fetching #2952
Conversation
|
The latest Buf updates on your PR. Results from workflow CI / buf-check (pull_request).
|
|
Claude finished @julienrbrt's task in 2m 29s —— View job Code Review CompleteI've completed a comprehensive review of the async epoch fetching feature. This is a well-architected addition that introduces background prefetching of DA epoch data to improve performance at epoch boundaries. Below are my findings organized by category. 🟢 StrengthsArchitecture & Design
Concurrency
Testing
🟡 Issues to Address1. Critical: Goroutine Leak Risk in Results IteratorLocation: results, err := f.cache.Query(f.ctx, query)
if err != nil {
f.logger.Warn().Err(err).Msg("failed to query cache for cleanup")
return
}
defer results.Close()
for result := range results.Next() { // ⚠️ Potential leak if context cancelledIssue: If Recommendation: Add explicit context check or use errgroup with cancellation: for {
select {
case <-f.ctx.Done():
return
case result, ok := <-results.Next():
if !ok {
break
}
// process result
}
}2. Race Condition: Context Created in ConstructorLocation: ctx, cancel := context.WithCancel(context.Background())Issue: The context is created from Recommendation: Accept a parent func Start(ctx context.Context) {
f.ctx, f.cancel = context.WithCancel(ctx)
// ...
}3. Missing AsyncFetcher Cleanup on Sequencer ErrorsLocation: Issue: Recommendation: Add cleanup on error paths: s.asyncFetcher = block.NewAsyncEpochFetcher(...)
s.asyncFetcher.Start()
s.fiRetriever = block.NewForcedInclusionRetriever(...)
if err := someOperation(); err != nil {
s.asyncFetcher.Stop() // ← Add this
return nil, err
}Better yet, add a cleanup method or defer pattern. 4. Buffer Overflow Risk in SerializationLocation: Issue: The serialization calculates size upfront but doesn't validate buffer bounds during writes. If size calculation is wrong (e.g., integer overflow), buffer overflow could occur. Recommendation: Use if offset+8 > len(buf) {
return nil, errors.New("buffer overflow during serialization")
}Alternatively, use a 5. Deserialization: Missing Input ValidationLocation: Issue: While there's a minimum length check (
Recommendation: Add safety limits: const (
maxTxCount = 100_000 // Reasonable upper bound
maxTxSize = 2 * 1024 * 1024 // 2MB per tx
)
if txCount > maxTxCount {
return nil, fmt.Errorf("tx count %d exceeds maximum %d", txCount, maxTxCount)
}6. Potential Memory Growth: No Cache Size LimitLocation: Issue: While old epochs are cleaned up, there's no maximum cache size limit. If epochs are very large or the prefetch window is increased, memory could grow unbounded during the grace period. Recommendation: Add a maximum cache size check and evict oldest entries if exceeded: type AsyncEpochFetcher struct {
// ...
maxCacheSize int64 // bytes
currentCacheSize atomic.Int64
}7. Error Handling: Silent Failure in PrefetchLocation: Issue: When prefetch fails (e.g., Recommendation: Consider implementing:
8. Code Duplication:
|
Summary of ChangesHello @julienrbrt, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements an asynchronous epoch fetching mechanism to optimize the retrieval of forced inclusion transactions from the Data Availability (DA) layer. By introducing an AsyncEpochFetcher that prefetches and caches upcoming epoch data in the background, the system can significantly reduce the latency associated with fetching this data, particularly at epoch transitions. This enhancement improves the overall performance and responsiveness of sequencers and the block syncing process by providing immediate access to frequently needed DA epoch information. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an AsyncEpochFetcher component designed to prefetch Data Availability (DA) epoch data in the background, thereby improving performance for sequencers and the block syncer. The new fetcher includes an in-memory cache for storing prefetched epochs, manages background fetching, tracks the current DA height, and cleans up old cached data. The ForcedInclusionRetriever has been updated to leverage this AsyncEpochFetcher, prioritizing cached data before making direct DA client calls. The AsyncEpochFetcher is integrated into the evm, grpc, and testapp applications, the block syncer, and the single sequencer, with corresponding unit tests added and existing tests updated. Review comments indicate that the mu mutex in AsyncEpochFetcher is redundant due to dsync.MutexWrap already providing thread-safety, leading to double-locking, and suggest replacing the fragile fmt.Sscanf for parsing datastore keys with key.Name() and strconv.ParseUint for robustness.
|
|
||
| // In-memory cache for prefetched epoch data | ||
| cache ds.Batching | ||
| mu sync.RWMutex |
There was a problem hiding this comment.
The cache field is already wrapped with dsync.MutexWrap, which makes it thread-safe. The additional mu field is redundant and leads to double-locking. Please remove this mutex and all its usages (f.mu.Lock/Unlock/RLock/RUnlock) throughout the file.
| var epochEnd uint64 | ||
| _, err := fmt.Sscanf(key.String(), "/epoch/%d", &epochEnd) | ||
| if err != nil { | ||
| continue | ||
| } |
There was a problem hiding this comment.
Using fmt.Sscanf to parse the key string is fragile. A more robust and idiomatic way to handle datastore keys is to use key.Name() to get the last component of the key and then parse it using strconv.ParseUint. You will need to import the strconv package.
| var epochEnd uint64 | |
| _, err := fmt.Sscanf(key.String(), "/epoch/%d", &epochEnd) | |
| if err != nil { | |
| continue | |
| } | |
| var epochEnd uint64 | |
| epochEndStr := key.Name() | |
| epochEnd, err := strconv.ParseUint(epochEndStr, 10, 64) | |
| if err != nil { | |
| continue | |
| } |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2952 +/- ##
==========================================
+ Coverage 58.73% 59.28% +0.54%
==========================================
Files 90 91 +1
Lines 8720 9055 +335
==========================================
+ Hits 5122 5368 +246
- Misses 3011 3082 +71
- Partials 587 605 +18
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ref: #2906