Extractor statistics - #2647
Merged
Merged
Extractor statistics#2647
Conversation
- Extractor::statistics() with Cardinality and Statistics: rows and bytes known before reading - file sources declare listed bytes; CSV and JSON Lines estimate rows from the schema sample - Parquet and Floe read footer totals once, shared with schema(); Floe footer gains a statistics block - PostgreSQL and Doctrine (PostgreSQL, MySQL) estimate rows with EXPLAIN, never ANALYZE - exact counts for sequences, file listings, cache index and whole-source samples - native CSV reader reports consumed bytes
- physical explain prints every source's declared statistics - analyze reports declared vs. read rows per source (withSourceStatistics(), --stats-sources) - count() is planned as Trigger::count and answered from exact statistics by the CountFromStatistics rule
- rowsError() is null for a read narrowed by a pushed limit/partition filter or stopped early - Cardinality::exactly() and isUnknown() replace re-derived checks in explain and count() - document source statistics: Cardinality, what each source declares, custom extractors
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2647 +/- ##
============================================
+ Coverage 84.77% 84.87% +0.09%
- Complexity 23126 31275 +8149
============================================
Files 2098 2123 +25
Lines 88854 89617 +763
============================================
+ Hits 75329 76064 +735
- Misses 13525 13553 +28
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Log
Added
flow-php/etl-Extractor::statistics(): every source declares its rows and bytes as aCardinalityflow-php/etl- physicalexplain()prints the statistics of every sourceflow-php/etl-analyze()->withSourceStatistics()reports declared vs. read rows per sourceflow-php/etl-count()answers from exact source statistics without reading rowsflow-php/etl- Floe footer carries a statistics blockflow-php/etl-adapter-parquet- exact rows and bytes from file footersflow-php/etl-adapter-csv- row estimate from the sampled row size, listed byte sizeflow-php/etl-adapter-json- JSON Lines row estimate, JSON exact rows on a whole-file sample, listed byte sizeflow-php/etl-adapter-excel- exact rows on a whole-workbook sample, listed byte sizeflow-php/etl-adapter-xml- listed byte sizeflow-php/etl-adapter-text- listed byte sizeflow-php/etl-adapter-doctrine- row estimate fromEXPLAINon PostgreSQL and MySQLflow-php/etl-adapter-postgresql- row estimate fromEXPLAINflow-php/etl-adapter-google-sheet- row bound from the sheet grid sizeflow-php/cli---stats-sourcesoption forpipeline:runandfile:analyzeflow-php/flow-php-ext- native CSV reader reports the bytes it consumedFixed
Changed
flow-php/etl- existing.floefiles must be rewritten for the new footerflow-php/etl-Reportconstructor takes the source statistics of the runflow-php/etl- cache index stores the row count of every entryRemoved
Deprecated
Security