input_chunk: configuration option for maximum chunk size#11373
Open
castorsky wants to merge 3 commits intofluent:masterfrom
Open
input_chunk: configuration option for maximum chunk size#11373castorsky wants to merge 3 commits intofluent:masterfrom
castorsky wants to merge 3 commits intofluent:masterfrom
Conversation
📝 WalkthroughWalkthroughA new configurable storage.max_chunk_size option was added to flb_config and wired into input chunk handling and the in_winevtlog plugin. Hard-coded chunk-size constants were replaced with a computed max size obtained via flb_input_chunk_get_max_size(), and config cleanup was added. Changes
Sequence Diagram(s)sequenceDiagram
participant Config
participant InputChunk
participant WinEvtPlugin
Config->>InputChunk: provide storage.max_chunk_size
InputChunk->>InputChunk: flb_input_chunk_get_max_size(config) -> parse value or fallback
WinEvtPlugin->>InputChunk: request max size (flb_input_chunk_get_max_size)
InputChunk-->>WinEvtPlugin: return computed maximum_threshold_size
WinEvtPlugin->>WinEvtPlugin: compute thresholds and apply caps/warnings
InputChunk->>InputChunk: apply max size and 1% threshold during append
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Tip You can disable the changed files summary in the walkthrough.Disable the |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 70b94ff0ad
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
castorsky
force-pushed
the
input_chunk_max_size_configurable
branch
from
70b94ff to
5c70876
Compare
Added configuration parameter for 'service' block to specify maximum chunk size for the 'input_chunk' module. Signed-off-by: Castor Sky <csky57@gmail.com>
Added new function 'flb_input_chunk_get_max_size' that retrieves value of the 'storage.max_chunk_size' parameter from the fluent-bit configuration or sets default value of FLB_INPUT_CHUNK_FS_MAX_SIZE (when user have not set parameter or there is any problem in parsing). Function is exposed to other modules and can be used anywhere to get 'storage.max_chunk_size' parameter. Light optimization: validation of available space in buffer now uses integer division instead of floating point multiplication (should be faster). Signed-off-by: Castor Sky <csky57@gmail.com>
…threshold Used configurable parameter 'storage.max_chunk_size' to calculate reading size threshold instead of fixed FLB_INPUT_CHUNK_FS_MAX_SIZE. Unnecessary type conversions were removed (all related variables are `size_t`). Introduced MAXIMUM_THRESHOLD_PERCENT to replace MAXIMUM_THRESHOLD_SIZE for calculation of threshold as percentage of user configured parameter. Signed-off-by: Castor Sky <csky57@gmail.com>
castorsky
force-pushed
the
input_chunk_max_size_configurable
branch
from
5c70876 to
827dc18
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@plugins/in_winevtlog/in_winevtlog.c`:
- Around line 278-305: The clamp logic can produce contradictory behavior when
maximum_threshold_size is lower than MINIMUM_THRESHOLD_SIZE; change the flow to
first compute and enforce an effective_max = (maximum_threshold_size <
MINIMUM_THRESHOLD_SIZE) ? MINIMUM_THRESHOLD_SIZE : maximum_threshold_size
(logging a single warning via flb_plg_warn if you had to raise the max), then
perform a single clamp of ctx->total_size_threshold into the inclusive range
[MINIMUM_THRESHOLD_SIZE, effective_max], using
flb_utils_bytes_to_human_readable_size and flb_plg_warn/flb_plg_debug for
consistent messages and updating ctx->total_size_threshold accordingly;
reference variables/functions: maximum_threshold_size, MINIMUM_THRESHOLD_SIZE,
ctx->total_size_threshold, flb_utils_bytes_to_human_readable_size, flb_plg_warn,
flb_plg_debug.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b1e15ea6-e2c5-4762-9181-0718278d958f
📒 Files selected for processing (3)
include/fluent-bit/flb_config.hinclude/fluent-bit/flb_input_chunk.hplugins/in_winevtlog/in_winevtlog.c
🚧 Files skipped from review as they are similar to previous changes (2)
- include/fluent-bit/flb_config.h
- include/fluent-bit/flb_input_chunk.h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduced new key
storage.max_chunk_sizefor theserviceblock in configuration. This key regulates maximum size of buffer chunk for input plugins that use the filesystem buffer.Default value of
2048000was preserved for compatibility with older configurations.Getter function for the value of
storage.max_chunk_sizeis exposed for usage by other plugins. Thein_winevtlogwas patched to use this function for calculation of read threshold size.This PR addresses #10327.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
Variant 1 (default value):
Variant 2 (max_chunk_size=16 KB):
Log output for variant 1
Log output for variant 2
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
PR for documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Chores