Read performance - #2645
Merged
Merged
Read performance#2645
Conversation
- skip type narrowing once a column saturates to string - return rows that already match the schema from conform - drop the per-field coercion closure in CSV decode - hoist the stream handle out of the readLines/iterate loops
- Rust CSV tokenizer and schema fold behind the optional extension - adaptive CSVOpenSource: NativeCSVOpenSource / PhpCSVOpenSource - SniffsColumnTypes lets a sample fold itself in SchemaInferrer - escaped or bare enclosure no longer glues two records together - PCRE-limit fallback for record boundaries on huge records
…ively - validate JSON with serde_json behind depth and surrogate pre-filters - assemble Rows through Rows::trusted when no column is absent - cast ISO datetime strings without calling DateTimeType - share the source string with Json and Uuid instead of copying it - fix reference values leaking into natively hydrated rows
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2645 +/- ##
============================================
- Coverage 84.82% 84.77% -0.05%
- Complexity 30939 31000 +61
============================================
Files 2094 2098 +4
Lines 88711 88854 +143
============================================
+ Hits 75245 75330 +85
- Misses 13466 13524 +58 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #xxx
Change Log
Added
flow-php/flow-php-ext- native CSV tokenizer, reader and schema inferenceflow-php/flow-php-ext- native JSON validation and ISO datetime casting in the hydratorflow-php/etl-adapter-csv-NativeCSVOpenSourceused when the extension is loadedflow-php/etl-SniffsColumnTypeslets a sample infer its own column typesFixed
flow-php/etl-adapter-csv- escaped or bare enclosure no longer merges a record with the next lineflow-php/etl-adapter-csv- record boundaries on huge records survive the PCRE backtrack limitflow-php/flow-php-ext- reference values no longer leak into natively hydrated rowsflow-php/filesystem- closing a stream while it is being read no longer throws a TypeErrorChanged
flow-php/etl-adapter-csv-CSVOpenSourceis now an interface implemented byPhpCSVOpenSourceflow-php/etl-adapter-csv-CSVLineReadertakes the separator and escapeflow-php/etl-adapter-csv-CSVFileReader::samples()yieldsCSVFileSampleflow-php/etl- schema inference and row conforming skip redundant workflow-php/flow-php-ext- hydrated rows skip the conform pass when no column is absentRemoved
Deprecated
Security