Skip to content

fix(linux): load exported CSV and SQL INSERT files back with the same rows - #3714

Open
datlechin wants to merge 2 commits into
mainfrom
test/3665-linux-export-import-roundtrip
Open

datlechin wants to merge 2 commits into
mainfrom
test/3665-linux-export-import-roundtrip

Conversation

@datlechin

Copy link
Copy Markdown
Member

Summary

  • round_trip_suite! in tablepro-test-support, run by every driver (MySQL and MariaDB both, SQLite in the unit suite):
    • value corpus out as CSV, imported into an empty copy, under four writer/reader pairings (every delimiter, quoting always or where needed, LF, CRLF and CR, NULL as NULL). Row count and every row's text compared.
    • corpus out as SQL INSERT, run statement by statement in an editor session against an empty copy. Every value's text and kind compared.
    • a table whose CSV lands within 1 MiB under the 64 MiB limit, imported back and compared row by row. Off on ClickHouse (below).
  • MAX_CSV_FILE_BYTES moves from the app to tablepro_core::import so the test reads the same limit.
  • Fixed, all found by the new cases:
    • SQLite: a DATE, TIME or DATETIME value with an offset or Z, or in another form SQLite reads, was refused by the import and by cell edits.
    • PostgreSQL: dates and timestamps before year 1 (4713-01-01 BC), from year 10000, or past chrono's last year were refused by the import and by cell edits. One past chrono binds as text cast to the column type.
    • SQL Server: a decimal(38,38) value failed to save or import (tiberius sends precision 39); it now goes as text. Decimal parameters are cast to decimal(38, s), so a multi-row import no longer narrows a full-precision value beside a shorter one.
    • ClickHouse: SQL INSERT export wrote Map and Tuple values as strings the server refuses, and a Variant changed type. They are now JSONExtract('<json>', '<type>').
  • Corpus additions: SQLite DATE, TIME and DATETIME forms, PostgreSQL 10000-01-01.
  • Docs: linux/docs/testing.md (Export round trip), the PostgreSQL, SQL Server and ClickHouse driver pages, ROADMAP.md.
  • No changelog fragment: the Linux app has not shipped.

Not in this PR

  • ClickHouse near the import limit: parameters ride in the request URL (64 KiB client cap), so a statement carries 8 KiB of values. 64 MiB is about 8,000 statements at ~67 ms each (async insert wait), about 9 minutes. 128 KiB batches failed with uri too long and Too many form fields. Sending parameters in the body is its own driver change.
  • Export defaults: NULL in a text column comes back as '', and formula-like text gets an apostrophe. Both by design; a lossless round trip needs Convert NULL to empty off and the NULL marker.
  • Export's Decimal Comma has no import counterpart; a decimal column refuses 1,5 with a clear error.
  • A SQLite DATETIME holding a number (Unix time, Julian day) exports as the number, and the import refuses it.
  • ClickHouse binds a Tuple from a sorting key as its JSON text in a save's probe, which the server would not parse. Not reproduced.

Test plan

  • round_trip_suite! on PostgreSQL, MySQL, MariaDB, SQL Server, ClickHouse (--run-ignored only) and SQLite. Before the fixes the CSV case failed on SQLite, PostgreSQL and SQL Server, and the SQL case on ClickHouse; all pass now.
  • SQL Server: csv_import::a_decimal_at_full_precision_keeps_every_digit_beside_shorter_ones, filter::a_decimal_filter_keeps_the_digits_it_was_given.
  • Unit tests for eras, far years and SQLite forms in parse_literal_text, PostgreSQL far-date binding, SQL Server decimal encoding and cast, ClickHouse JSON literals.
  • Value corpus, CSV import, write batch and filter suites; meson test under xvfb; rustfmt and both clippy runs.

Fixes #3665

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test that a Linux CSV and SQL export imports back with the same rows

1 participant