From 24415bf5ae45c2a0211f88f8ac41a5cdf8fc737f Mon Sep 17 00:00:00 2001 From: Ngo Quoc Dat Date: Wed, 21 Oct 2026 01:10:18 +0700 Subject: [PATCH] fix(linux): load exported CSV and SQL INSERT files back with the same rows --- linux/ROADMAP.md | 4 +- linux/crates/app/src/ui/import_dialog.rs | 8 - linux/crates/app/src/ui/workspace/import.rs | 4 +- .../core/src/edit/parse_literal_text.rs | 217 +++++- linux/crates/core/src/import/csv_import.rs | 8 + linux/crates/core/src/import/mod.rs | 4 +- .../crates/drivers/clickhouse/src/dialect.rs | 30 + .../clickhouse/tests/integration/main.rs | 1 + .../tests/integration/round_trip.rs | 15 + linux/crates/drivers/mssql/src/dialect.rs | 39 ++ linux/crates/drivers/mssql/src/encode.rs | 30 +- .../mssql/tests/integration/csv_import.rs | 69 ++ .../drivers/mssql/tests/integration/filter.rs | 51 +- .../drivers/mssql/tests/integration/main.rs | 1 + .../mssql/tests/integration/round_trip.rs | 13 + .../drivers/mysql/tests/integration/main.rs | 1 + .../mysql/tests/integration/round_trip.rs | 23 + linux/crates/drivers/postgres/src/dialect.rs | 28 +- .../postgres/tests/integration/main.rs | 1 + .../postgres/tests/integration/round_trip.rs | 13 + .../tests/integration/value_corpus.rs | 1 + .../drivers/sqlite/tests/integration/main.rs | 2 + .../sqlite/tests/integration/round_trip.rs | 11 + .../sqlite/tests/integration/value_corpus.rs | 11 +- .../test-support/src/conformance/cases/mod.rs | 1 + .../src/conformance/cases/round_trip.rs | 658 ++++++++++++++++++ .../src/conformance/cases/value_corpus.rs | 62 +- .../test-support/src/conformance/mod.rs | 5 +- .../src/conformance/round_trip_suite_macro.rs | 54 ++ linux/docs/drivers/clickhouse.md | 2 + linux/docs/drivers/postgresql.md | 2 +- linux/docs/drivers/sql-server.md | 2 + linux/docs/testing.md | 18 + 33 files changed, 1338 insertions(+), 51 deletions(-) create mode 100644 linux/crates/drivers/clickhouse/tests/integration/round_trip.rs create mode 100644 linux/crates/drivers/mssql/tests/integration/round_trip.rs create mode 100644 linux/crates/drivers/mysql/tests/integration/round_trip.rs create mode 100644 linux/crates/drivers/postgres/tests/integration/round_trip.rs create mode 100644 linux/crates/drivers/sqlite/tests/integration/round_trip.rs create mode 100644 linux/crates/test-support/src/conformance/cases/round_trip.rs create mode 100644 linux/crates/test-support/src/conformance/round_trip_suite_macro.rs diff --git a/linux/ROADMAP.md b/linux/ROADMAP.md index b4c034280c..df9c7ec5f2 100644 --- a/linux/ROADMAP.md +++ b/linux/ROADMAP.md @@ -114,7 +114,7 @@ The app runs on GNOME 50 ([ADR 0015](docs/decisions/0015-gnome-50-platform-floor | Desktop integration | Desktop file with Open With for SQLite files, D-Bus activation, a man page, metainfo with screenshots, and SVG icons | | Translations | The app's own strings go through gettext, and CI fails when `POTFILES.in` is stale. The template is generated on demand with gettext 1.0. Some error details from the core crates still reach the user in English (see Release criteria) | | Logs | stderr, which the session journals, with JSON lines on request. A panic also writes a crash report under `$XDG_STATE_HOME`, offered with Show Report on the next launch | -| Tests | Unit, gtk (with an accessibility audit of the picker, the connect pages, the workspace, the connect dialog and every preferences page), data and packaging suites; connection, DDL and TLS conformance on every engine; Docker jobs for PostgreSQL with PgBouncer, MySQL and MariaDB, SQL Server, ClickHouse, SSH, the session and the grid save ([testing.md](docs/testing.md)); every critical audit finding has named guards ([regression-tests.md](docs/regression-tests.md)) | +| Tests | Unit, gtk (with an accessibility audit of the picker, the connect pages, the workspace, the connect dialog and every preferences page), data and packaging suites; connection, DDL and TLS conformance, and a CSV and SQL INSERT export round trip, on every engine; Docker jobs for PostgreSQL with PgBouncer, MySQL and MariaDB, SQL Server, ClickHouse, SSH, the session and the grid save ([testing.md](docs/testing.md)); every critical audit finding has named guards ([regression-tests.md](docs/regression-tests.md)) | ## Backlog @@ -180,7 +180,7 @@ What has to hold before the first release, for a user on GNOME 50 with Ubuntu 26 |---|---|---|---| | 1 | Installs `tablepro` from the PPA or the COPR project, and gets updates from the system's updater | CI builds both packages offline the way Launchpad and COPR do, and starts them in fresh containers | The release setup, the first tag, and an install then an upgrade from the PPA and COPR | | 2 | Connects to their everyday PostgreSQL, MySQL, MariaDB, SQL Server, ClickHouse or SQLite database | The supported releases and why others are out, in the README's [supported servers](README.md#supported-servers). CI runs every driver's Docker suite on each tested release, MySQL 8.0 and SQL Server 2017 and 2019 among them | Nothing | -| 3 | Browses tables of real-world data (dates, decimals, JSON, UUIDs, NULLs) and sees correct values | Per-engine decoding with the type fallback and unsupported-type conformance cases ([ADR 0014](docs/decisions/0014-lossless-values-and-dialects.md)). A value corpus on every engine, MariaDB too: dates and times at their extremes and in other zones, decimals, JSON, UUIDs, Unicode and binary values, read back through a table tab, Export All and an editor run | Nothing | +| 3 | Browses tables of real-world data (dates, decimals, JSON, UUIDs, NULLs) and sees correct values | Per-engine decoding with the type fallback and unsupported-type conformance cases ([ADR 0014](docs/decisions/0014-lossless-values-and-dialects.md)). A value corpus on every engine, MariaDB too: dates and times at their extremes and in other zones, decimals, JSON, UUIDs, Unicode and binary values, read back through a table tab, Export All and an editor run, and exported as CSV and SQL INSERT and loaded back into an empty copy | Nothing | | 4 | Runs typical queries against tables of a million rows or more without running out of memory or freezing | The 10,000-row editor limit with Export All, and driver tests over 2,000,000 rows. Pages seek past the edge row's key, or walk PostgreSQL heap block windows for a table with no key; OFFSET only under a user sort, a restored deep offset or a partitioned keyless table. Docker tests page 60k and 25k-row tables both ways and check the plan. A 1,000,000-row table is paged to its end and exported on PostgreSQL, MySQL, MariaDB, SQL Server, ClickHouse and SQLite, with bounds on time and on resident peak growth. SQLite's case runs in the unit suite in a test binary of its own, so the peak is its alone | Nothing | | 5 | Edits data in place, and the changes commit correctly even when the connection or the app misbehaves | Write, row guard and commit resolution conformance on every engine, with the fault fakes and the grid save Docker test. `kill_during_save` kills TablePro halfway through a SQLite save: the file keeps none of it, and the next launch restores the session with its history, settings and saved tabs. A save into a SQLite file another program holds locked fails as `Busy(FileLocked)`, which says another program or an editor tab holds the file's lock, and leaves the file as it was whether the lock shows at `BEGIN IMMEDIATE` or at the `COMMIT` | Nothing | | 6 | Trusts the app with credentials: TLS, SSH tunnels, no password in plain text | Verify full by default, the TLS conformance suite, the Secret Service, and statement logging silenced | An outside security review | diff --git a/linux/crates/app/src/ui/import_dialog.rs b/linux/crates/app/src/ui/import_dialog.rs index 171a383557..71993792cf 100644 --- a/linux/crates/app/src/ui/import_dialog.rs +++ b/linux/crates/app/src/ui/import_dialog.rs @@ -12,14 +12,6 @@ use tablepro_core::import::{ use crate::runtime::DeliverInput; -/// The most a CSV may be before the import refuses it. -/// -/// Every row becomes an INSERT in one transaction, so the file's size -/// decides how much the app holds at once. Refusing is better than a -/// window that stops painting halfway through a file nobody meant to -/// open. -pub const MAX_CSV_FILE_BYTES: usize = 64 * 1024 * 1024; - /// How many values from the file each mapping row shows, so the user /// can see they picked the field they meant. const SAMPLE_VALUES: usize = 3; diff --git a/linux/crates/app/src/ui/workspace/import.rs b/linux/crates/app/src/ui/workspace/import.rs index b63c5d4190..bf895d2fc7 100644 --- a/linux/crates/app/src/ui/workspace/import.rs +++ b/linux/crates/app/src/ui/workspace/import.rs @@ -10,7 +10,7 @@ use std::time::Duration; use tablepro_core::dialect::SqlDialect; use tablepro_core::edit::CellInput; -use tablepro_core::import::{ImportColumn, ImportPlan, plan_import}; +use tablepro_core::import::{ImportColumn, ImportPlan, MAX_CSV_FILE_BYTES, plan_import}; use tablepro_core::meta::TableRef; use tablepro_core::{CallOptions, DriverError, WriteKind}; use tablepro_session::runtime::Tasks; @@ -20,7 +20,7 @@ use tokio_util::sync::CancellationToken; use crate::services::history_service::HistoryService; use crate::ui::error_text::{driver_message, operation_error_message}; use crate::ui::file_read::{FileReadError, read_capped_file}; -use crate::ui::import_dialog::{CsvImport, MAX_CSV_FILE_BYTES, import_error_text}; +use crate::ui::import_dialog::{CsvImport, import_error_text}; use super::{WorkspaceInput, WorkspaceModel}; use crate::runtime::DeliverInput; diff --git a/linux/crates/core/src/edit/parse_literal_text.rs b/linux/crates/core/src/edit/parse_literal_text.rs index 750a2674b6..68c8b49803 100644 --- a/linux/crates/core/src/edit/parse_literal_text.rs +++ b/linux/crates/core/src/edit/parse_literal_text.rs @@ -1,4 +1,6 @@ -use chrono::{DateTime, NaiveDate, NaiveDateTime, NaiveTime, Timelike}; +use std::borrow::Cow; + +use chrono::{DateTime, Datelike, NaiveDate, NaiveDateTime, NaiveTime, Timelike}; use crate::column::{BoolStorage, ColumnKind, ColumnType, FloatWidth, IntegerKind, TemporalZone, TimeRange}; use crate::value::{ @@ -38,6 +40,11 @@ fn parse(input: &str, column: &ColumnType, purpose: Purpose) -> Result Ok(value), Err(_) if column.dynamic_storage && keeps_text_on_failure(&column.kind) => Ok(Value::Text(input.to_owned())), + Err(EditParseError::InvalidDate | EditParseError::InvalidTime | EditParseError::InvalidTimestamp) + if column.dynamic_storage && keeps_text_when_valid(&column.kind) && is_time_value_text(input.trim()) => + { + Ok(Value::Text(input.trim().to_owned())) + } Err(error) => Err(error), } } @@ -55,6 +62,19 @@ fn keeps_text_when_valid(kind: &ColumnKind) -> bool { ) } +/// Whether `input` is a date or time in a form SQLite's date and time +/// functions read, whichever of DATE, TIME or DATETIME the column +/// declares: SQLite gives the three names the same affinity, so any of +/// them holds any form, and a time or timestamp may end in an offset or +/// `Z`. +fn is_time_value_text(input: &str) -> bool { + parse_date(input).is_ok() + || parse_time(input).is_ok() + || parse_timestamp(input).is_ok() + || parse_time_with_offset(input).is_ok() + || parse_timestamp_with_offset(input).is_ok() +} + /// Whether a failed parse should fall back to the text as typed. /// /// Only for kinds whose text form is what the server stores anyway. A @@ -100,7 +120,8 @@ fn strict_value(input: &str, column: &ColumnType, purpose: Purpose) -> Result parse_date(trimmed), + ColumnKind::Date => parse_date(trimmed) + .or_else(|error| far_temporal(trimmed, column, purpose, parse_date).unwrap_or(Err(error))), ColumnKind::Time { with_offset: true, .. } => parse_time_with_offset(trimmed), ColumnKind::Time { range: range @ (TimeRange::Duration | TimeRange::LongDuration), @@ -110,8 +131,10 @@ fn strict_value(input: &str, column: &ColumnType, purpose: Purpose) -> Result parse_timestamp_with_offset(trimmed), - ColumnKind::Timestamp { .. } => parse_timestamp(trimmed), + } => parse_timestamp_with_offset(trimmed) + .or_else(|error| far_temporal(trimmed, column, purpose, parse_timestamp_with_offset).unwrap_or(Err(error))), + ColumnKind::Timestamp { .. } => parse_timestamp(trimmed) + .or_else(|error| far_temporal(trimmed, column, purpose, parse_timestamp).unwrap_or(Err(error))), ColumnKind::Interval => trimmed .parse::() .map(Value::Interval) @@ -265,7 +288,8 @@ fn parse_date(input: &str) -> Result { if let Some(temporal) = infinite_temporal(input) { return Ok(Value::Date(temporal)); } - NaiveDate::parse_from_str(input, "%Y-%m-%d") + let input = chrono_year(input).ok_or(EditParseError::InvalidDate)?; + NaiveDate::parse_from_str(&input, "%Y-%m-%d") .map(|date| Value::Date(Temporal::Finite(date))) .map_err(|_| EditParseError::InvalidDate) } @@ -304,14 +328,72 @@ fn parse_timestamp(input: &str) -> Result { if let Some(temporal) = infinite_temporal(input) { return Ok(Value::Timestamp(temporal)); } + let input = chrono_year(input).ok_or(EditParseError::InvalidTimestamp)?; for format in ["%Y-%m-%d %H:%M:%S%.f", "%Y-%m-%dT%H:%M:%S%.f", "%Y-%m-%d %H:%M"] { - if let Ok(stamp) = NaiveDateTime::parse_from_str(input, format) { + if let Ok(stamp) = NaiveDateTime::parse_from_str(&input, format) { return Ok(Value::Timestamp(Temporal::Finite(stamp))); } } Err(EditParseError::InvalidTimestamp) } +/// The text with its year spelled the way chrono's `%Y` reads one. The +/// grid and PostgreSQL write a year past 9999 without a sign and a year +/// before AD 1 as ` BC` after the whole value, where chrono wants +/// `+10000` and `-4712`: there is no year 0 between 1 BC and AD 1, so +/// 1 BC is chrono's 0. `None` for a ` BC` value with no year to count +/// back from. +fn chrono_year(input: &str) -> Option> { + let (dated, before_christ) = match input.strip_suffix(" BC").or_else(|| input.strip_suffix(" bc")) { + Some(dated) => (dated, true), + None => (input, false), + }; + let year_digits = dated.bytes().take_while(u8::is_ascii_digit).count(); + let (year, rest) = dated.split_at(year_digits); + if !before_christ { + return Some(match year_digits > 4 && rest.starts_with('-') { + true => Cow::Owned(format!("+{dated}")), + false => Cow::Borrowed(input), + }); + } + if !rest.starts_with('-') { + return None; + } + let year = year.parse::().ok().filter(|year| *year >= 1)?; + Some(Cow::Owned(match 1 - year { + 0 => format!("0000{rest}"), + signed => format!("{signed}{rest}"), + })) +} + +/// A date or timestamp past the last year chrono holds, kept as the text +/// it was written in: PostgreSQL keeps dates to 5874897 AD and its driver +/// reads one that far as that text. The Gregorian +/// calendar repeats every 400 years, so the text is checked by `parse` +/// with its year moved into chrono's range, fraction and all. `None` +/// for text that is not such a value. +fn far_temporal( + input: &str, + column: &ColumnType, + purpose: Purpose, + parse: fn(&str) -> Result, +) -> Option> { + let year_digits = input.bytes().take_while(u8::is_ascii_digit).count(); + let (year, rest) = input.split_at(year_digits); + let last_year = u64::try_from(NaiveDate::MAX.year()).ok()?; + let year = year.parse::().ok().filter(|year| *year > last_year)?; + let stand_in = parse(&format!("{}{rest}", 2000 + year % 400)).ok()?; + if purpose == Purpose::Store + && let Err(error) = check_declared_fraction(&stand_in, column.kind.fractional_digits()) + { + return Some(Err(error)); + } + Some(Ok(Value::Other(Box::new(OtherValue { + type_name: column.name.clone(), + text: input.to_owned(), + })))) +} + fn infinite_temporal(input: &str) -> Option> { match input.to_ascii_lowercase().as_str() { "infinity" | "inf" => Some(Temporal::Infinity), @@ -382,18 +464,19 @@ pub fn parse_timestamp_with_offset(input: &str) -> Result if let Some(temporal) = infinite_temporal(trimmed) { return Ok(Value::TimestampTz(temporal)); } + let dated = chrono_year(trimmed).ok_or(EditParseError::InvalidTimestamp)?; for format in [ "%Y-%m-%d %H:%M:%S%.f%#z", "%Y-%m-%dT%H:%M:%S%.f%#z", "%Y-%m-%d %H:%M%#z", ] { - if let Ok(stamp) = DateTime::parse_from_str(trimmed, format) { + if let Ok(stamp) = DateTime::parse_from_str(&dated, format) { return Ok(Value::TimestampTz(Temporal::Finite(OffsetTimestamp::from_datetime( stamp, )))); } } - if NaiveDateTime::parse_from_str(trimmed, "%Y-%m-%d %H:%M:%S%.f").is_ok() { + if NaiveDateTime::parse_from_str(&dated, "%Y-%m-%d %H:%M:%S%.f").is_ok() { return Err(EditParseError::OffsetRequired); } Err(EditParseError::InvalidTimestamp) @@ -880,4 +963,122 @@ mod tests { "a declared column lost its typed value" ); } + + #[test] + fn a_dynamic_column_keeps_any_date_or_time_form_sqlite_reads() { + let stamp = ColumnKind::Timestamp { + fractional_digits: None, + zone: TemporalZone::Naive, + }; + let time_of_day = time(false, TimeRange::TimeOfDay); + for (input, kind) in [ + ("2024-03-10T02:30:00-08:00", &stamp), + ("2024-03-10 10:30:00.5Z", &stamp), + ("2024-03-10 02:30+05:45", &stamp), + ("12:30:00+05:30", &time_of_day), + ("12:30Z", &time_of_day), + ("2024-03-10 02:30:00", &ColumnKind::Date), + ("2024-03-10", &stamp), + ("12:30:00", &ColumnKind::Date), + ] { + assert_eq!( + parse_literal_text(input, &column(kind.clone(), true)), + Ok(Value::Text(input.to_owned())), + "{input} in {kind:?}" + ); + } + } + + #[test] + fn a_date_before_year_one_reads_with_its_era() { + let date = column(ColumnKind::Date, false); + let day = |year, month, day| Value::Date(Temporal::Finite(NaiveDate::from_ymd_opt(year, month, day).unwrap())); + + assert_eq!(parse_literal_text("4713-01-01 BC", &date), Ok(day(-4712, 1, 1))); + assert_eq!(parse_literal_text("0001-12-31 BC", &date), Ok(day(0, 12, 31))); + assert_eq!( + parse_literal_text("0000-01-01 BC", &date), + Err(EditParseError::InvalidDate) + ); + let instant = column( + ColumnKind::Timestamp { + fractional_digits: None, + zone: TemporalZone::Offset, + }, + false, + ); + assert!( + matches!( + parse_literal_text("4713-01-01 00:00:00+00:00 BC", &instant), + Ok(Value::TimestampTz(Temporal::Finite(stamp))) if stamp.to_datetime().year() == -4712 + ), + "an instant before year one lost its era" + ); + } + + #[test] + fn a_year_past_chronos_last_keeps_its_text_once_checked() { + let date = column(ColumnKind::Date, false); + let kept = |text: &str| { + Value::Other(Box::new(OtherValue { + type_name: "t".to_owned(), + text: text.to_owned(), + })) + }; + + assert_eq!( + parse_literal_text("10000-01-01", &date), + Ok(Value::Date(Temporal::Finite( + NaiveDate::from_ymd_opt(10_000, 1, 1).unwrap() + ))) + ); + assert_eq!(parse_literal_text("5874897-12-31", &date), Ok(kept("5874897-12-31"))); + let last = NaiveDate::MAX.year(); + assert_eq!( + parse_literal_text(&format!("{last}-12-31"), &date), + Ok(Value::Date(Temporal::Finite(NaiveDate::MAX))) + ); + let first_far = format!("{}-01-01", last + 1); + assert_eq!(parse_literal_text(&first_far, &date), Ok(kept(&first_far))); + assert_eq!(parse_literal_text("262144-02-29", &date), Ok(kept("262144-02-29"))); + assert_eq!( + parse_literal_text("5874897-02-30", &date), + Err(EditParseError::InvalidDate) + ); + let whole_seconds = column( + ColumnKind::Timestamp { + fractional_digits: Some(0), + zone: TemporalZone::Naive, + }, + false, + ); + assert_eq!( + parse_literal_text("294276-12-31 23:59:59", &whole_seconds), + Ok(kept("294276-12-31 23:59:59")) + ); + assert_eq!( + parse_literal_text("294276-12-31 23:59:59.5", &whole_seconds), + Err(EditParseError::ScaleExceeded { scale: 0 }) + ); + } + + #[test] + fn a_dynamic_column_still_refuses_text_that_is_no_date_or_time() { + let stamp = ColumnKind::Timestamp { + fractional_digits: None, + zone: TemporalZone::Naive, + }; + for input in ["2024-03-10T02:30:00+25:00", "2024-02-30 12:00", "1710037800", "soon"] { + assert_eq!( + parse_literal_text(input, &column(stamp.clone(), true)), + Err(EditParseError::InvalidTimestamp), + "{input}" + ); + } + assert_eq!( + parse_literal_text("2024-03-10T02:30:00-08:00", &column(stamp, false)), + Err(EditParseError::InvalidTimestamp), + "a declared naive timestamp took an offset" + ); + } } diff --git a/linux/crates/core/src/import/csv_import.rs b/linux/crates/core/src/import/csv_import.rs index cb20dff450..38113ec003 100644 --- a/linux/crates/core/src/import/csv_import.rs +++ b/linux/crates/core/src/import/csv_import.rs @@ -6,6 +6,14 @@ use crate::export::CsvDelimiter; use super::ImportColumn; +/// The most a CSV may be before the import refuses it. +/// +/// Every row becomes an INSERT in one transaction, so the file's size +/// decides how much the app holds at once. Refusing is better than a +/// window that stops painting halfway through a file nobody meant to +/// open. +pub const MAX_CSV_FILE_BYTES: usize = 64 * 1024 * 1024; + /// How many rows the mapping dialog shows. Enough to tell whether the /// delimiter and the header switch are right, few enough that a /// gigabyte file still opens the dialog at once. diff --git a/linux/crates/core/src/import/mod.rs b/linux/crates/core/src/import/mod.rs index 145663e5c1..80bca8b77f 100644 --- a/linux/crates/core/src/import/mod.rs +++ b/linux/crates/core/src/import/mod.rs @@ -6,8 +6,8 @@ mod import_column; mod import_plan; pub use csv_import::{ - CsvImportOptions, CsvRowError, CsvSheet, ImportError, MAX_FIELDS, MAX_PREVIEW_ROWS, read_csv, row_to_cells, - suggest_mapping, + CsvImportOptions, CsvRowError, CsvSheet, ImportError, MAX_CSV_FILE_BYTES, MAX_FIELDS, MAX_PREVIEW_ROWS, read_csv, + row_to_cells, suggest_mapping, }; pub use import_column::ImportColumn; pub use import_plan::{ImportPlan, plan_import}; diff --git a/linux/crates/drivers/clickhouse/src/dialect.rs b/linux/crates/drivers/clickhouse/src/dialect.rs index f0c2bca214..b35e1b5379 100644 --- a/linux/crates/drivers/clickhouse/src/dialect.rs +++ b/linux/crates/drivers/clickhouse/src/dialect.rs @@ -16,6 +16,7 @@ use tablepro_core::sql_syntax::{ use tablepro_core::statement::{BoundParam, ParamType}; use tablepro_core::value::{Temporal, Value}; +use crate::cell_seed::CellPlan; use crate::clickhouse_type::ClickHouseType; use crate::param_value::param_name; @@ -319,6 +320,11 @@ fn literal(value: &Value) -> Result { reason: format!("{:?}", undecodable.reason), }); } + // A Map, Tuple, Variant or JSON value is the JSON the server + // wrote, which a string of it does not cast back from. + Value::Other(other) if matches!(ClickHouseType::parse(&other.type_name).plan, CellPlan::Raw { .. }) => { + format!("JSONExtract({}, {})", quoted(&other.text), quoted(&other.type_name)) + } Value::Other(other) => quoted(&other.text), other => match value_to_text(other) { Some(text) => quoted(&text), @@ -559,6 +565,30 @@ mod tests { ); } + #[test] + fn a_value_read_as_json_is_parsed_back_into_its_type() { + let other = |type_name: &str, text: &str| { + Value::Other(Box::new(tablepro_core::value::OtherValue { + type_name: type_name.to_owned(), + text: text.to_owned(), + })) + }; + + assert_eq!( + literal(&other("Map(String, UInt8)", r#"{"a":1}"#)), + Ok(r#"JSONExtract('{"a":1}', 'Map(String, UInt8)')"#.to_owned()) + ); + assert_eq!( + literal(&Value::Array(vec![other("Tuple(String, Int8)", r#"["it's",-1]"#)])), + Ok(r#"[JSONExtract('["it\'s",-1]', 'Tuple(String, Int8)')]"#.to_owned()) + ); + assert_eq!( + literal(&other("IPv4", "192.168.0.1")), + Ok("'192.168.0.1'".to_owned()), + "an address is the server's text, not JSON" + ); + } + #[test] fn like_escapes_with_backslash_and_writes_no_escape_clause() { let dialect = dialect(); diff --git a/linux/crates/drivers/clickhouse/tests/integration/main.rs b/linux/crates/drivers/clickhouse/tests/integration/main.rs index 35de2061c5..3ec640703f 100644 --- a/linux/crates/drivers/clickhouse/tests/integration/main.rs +++ b/linux/crates/drivers/clickhouse/tests/integration/main.rs @@ -11,6 +11,7 @@ mod csv_import; mod ddl; mod keys_and_decimals; mod metadata; +mod round_trip; mod sessions; mod support; mod tls; diff --git a/linux/crates/drivers/clickhouse/tests/integration/round_trip.rs b/linux/crates/drivers/clickhouse/tests/integration/round_trip.rs new file mode 100644 index 0000000000..8d180e969f --- /dev/null +++ b/linux/crates/drivers/clickhouse/tests/integration/round_trip.rs @@ -0,0 +1,15 @@ +//! The export round trip on ClickHouse: the value corpus out as CSV and +//! as SQL INSERT and back into an empty copy. + +mod suite { + use crate::conformance_engine::ClickHouseEngine; + + tablepro_test_support::round_trip_suite! { + engine: ClickHouseEngine, + docker: true, + // An import's values travel in the request URL, 8 KiB a + // statement, so a file near the size limit takes some 8,000 + // statements and minutes longer than the case may run. + near_limit: false, + } +} diff --git a/linux/crates/drivers/mssql/src/dialect.rs b/linux/crates/drivers/mssql/src/dialect.rs index 0cf8b23c52..e6bcee9ec5 100644 --- a/linux/crates/drivers/mssql/src/dialect.rs +++ b/linux/crates/drivers/mssql/src/dialect.rs @@ -16,6 +16,7 @@ use tablepro_core::statement::{BoundParam, ParamType}; use tablepro_core::value::{Temporal, Value}; use crate::catalog::SqlServerType; +use crate::encode::MAX_NUMERIC_DIGITS; /// A string literal in the national character set. Without the `N`, the /// server reads the text in the database's code page and loses every @@ -78,6 +79,19 @@ impl SqlDialect for MssqlDialect { (ParamType::Text | ParamType::FixedText, BindTarget::Column(column)) if non_unicode(column) => { format!("CAST({marker} AS {NON_UNICODE_TEXT})") } + // SQL Server types the rows of a multi-row VALUES as UNION ALL + // does, so a value at full precision beside a shorter one would + // lose digits to a type both fit. Every value that fits the + // column takes the same type, and a filter value keeps the + // digits it has on either side of the point. + (ParamType::Decimal, BindTarget::Column(column)) => match (&column.column_type.kind, &value) { + (ColumnKind::Decimal { scale: Some(scale), .. }, Value::Decimal(decimal)) => { + let room = MAX_NUMERIC_DIGITS.saturating_sub(decimal.integer_digits().unwrap_or(0)); + let scale = decimal.scale().unwrap_or(0).max(*scale).min(room); + format!("CAST({marker} AS decimal({MAX_NUMERIC_DIGITS}, {scale}))") + } + _ => marker, + }, _ => marker, }; Ok(Placeholder::bound(sql, BoundParam::new(value, param_type)?)) @@ -464,6 +478,31 @@ mod tests { .unwrap() } + #[test] + fn a_decimal_binds_at_the_widest_precision_and_the_columns_scale() { + let decimal = |text: &str| Value::Decimal(text.parse().unwrap()); + + assert_eq!( + bound("decimal(38,38)", decimal("0")).sql, + "CAST(@P1 AS decimal(38, 38))" + ); + assert_eq!( + bound("decimal(10,2)", decimal("1.5")).sql, + "CAST(@P1 AS decimal(38, 2))" + ); + assert_eq!( + bound("decimal(10,2)", decimal("1.005")).sql, + "CAST(@P1 AS decimal(38, 3))", + "a filter value finer than the column was rounded" + ); + assert_eq!( + bound("decimal(38,38)", decimal("1")).sql, + "CAST(@P1 AS decimal(38, 37))", + "a filter value past the column's integer digits overflowed" + ); + assert_eq!(bound("decimal(10,2)", Value::Null).sql, "@P1"); + } + #[test] fn null_varbinary_binds_as_bytes() { let placeholder = bound("varbinary(max)", Value::Null); diff --git a/linux/crates/drivers/mssql/src/encode.rs b/linux/crates/drivers/mssql/src/encode.rs index 901fccdc9a..12948ecc04 100644 --- a/linux/crates/drivers/mssql/src/encode.rs +++ b/linux/crates/drivers/mssql/src/encode.rs @@ -1,6 +1,7 @@ use std::borrow::Cow; use chrono::{Datelike, NaiveDate}; +use tablepro_core::export::value_to_text; use tablepro_core::statement::{BoundParam, ParamType}; use tablepro_core::value::{Temporal, Value}; use tablepro_core::{DriverError, UnsupportedFeature}; @@ -8,7 +9,7 @@ use tiberius::numeric::Numeric; use tiberius::{ColumnData, IntoSql, ToSql}; /// The widest decimal SQL Server stores: 38 digits. -const MAX_NUMERIC_DIGITS: u32 = 38; +pub(crate) const MAX_NUMERIC_DIGITS: u32 = 38; /// The parameters of a statement as TDS values, typed by what the /// server expects, so a NULL keeps its type: an untyped NULL arrives as @@ -50,9 +51,15 @@ pub(crate) fn encode(param: &BoundParam) -> Result, DriverEr } // SQL Server has no unsigned 64-bit type; decimal(20,0) holds // every value of one. - ParamType::UInt64 | ParamType::Decimal => { - numeric(param_type, value).map(|numeric| ColumnData::Numeric(Some(numeric))) - } + ParamType::UInt64 | ParamType::Decimal => numeric(param_type, value).map(|numeric| { + // tiberius counts a digit before the point when the integer + // part is 0, so a value at scale 38 goes out as precision 39, + // which the server refuses. Its text converts exactly. + match u32::from(numeric.precision()) > MAX_NUMERIC_DIGITS { + true => text_data(value_to_text(value).unwrap_or_default()), + false => ColumnData::Numeric(Some(numeric)), + } + }), ParamType::Float32 => match value { Value::Float32(real) => Ok(ColumnData::F32(Some(*real))), other => Err(refused(param_type, other)), @@ -258,6 +265,21 @@ mod tests { ); } + #[test] + fn a_decimal_at_scale_38_travels_as_its_text() { + let text = "0.99999999999999999999999999999999999999"; + let decimal: SqlDecimal = text.parse().unwrap(); + + let encoded = encode(&bound(Value::Decimal(decimal), ParamType::Decimal)).unwrap(); + + assert_eq!(encoded, ColumnData::String(Some(Cow::Borrowed(text)))); + let narrower: SqlDecimal = "0.9999999999999999999999999999999999999".parse().unwrap(); + assert!(matches!( + encode(&bound(Value::Decimal(narrower), ParamType::Decimal)).unwrap(), + ColumnData::Numeric(Some(_)) + )); + } + #[test] fn integers_are_checked_against_their_width() { assert_eq!( diff --git a/linux/crates/drivers/mssql/tests/integration/csv_import.rs b/linux/crates/drivers/mssql/tests/integration/csv_import.rs index 3a6bd76966..b5d8168fe4 100644 --- a/linux/crates/drivers/mssql/tests/integration/csv_import.rs +++ b/linux/crates/drivers/mssql/tests/integration/csv_import.rs @@ -115,3 +115,72 @@ async fn a_row_that_fails_in_a_later_statement_rolls_the_whole_file_back() { vec![vec![Value::Int(0)]] ); } + +/// SQL Server types the rows of a multi-row VALUES as UNION ALL types +/// them, so a value at full precision beside a shorter one must not be +/// narrowed to a type both fit. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires docker"] +async fn a_decimal_at_full_precision_keeps_every_digit_beside_shorter_ones() { + let fixture = start().await; + let connection = connect(&fixture, false).await; + setup( + connection.as_ref(), + &[ + "CREATE TABLE dbo.fractions (id int PRIMARY KEY, v decimal(38, 38))", + "CREATE TABLE dbo.amounts (id int PRIMARY KEY, v decimal(38, 2))", + ], + ) + .await; + let files: [(&str, &[u8]); 2] = [ + ( + "fractions", + b"id,v\n1,0.12345678901234567890123456789012345678\n2,0\n3,0.5\n", + ), + ("amounts", b"id,v\n1,123456789012345678901234567890123456.78\n2,1\n"), + ]; + for (table, csv) in files { + import_csv( + connection.as_ref(), + &dbo(table), + csv, + &CsvImportOptions::default(), + within(CALL), + ) + .await + .unwrap_or_else(|failure| panic!("the import into {table}: {failure:?}")); + } + + let text = |rows: Vec>| -> Vec { + rows.into_iter() + .map(|row| match row.into_iter().next() { + Some(Value::Text(text)) => text, + other => panic!("{other:?}"), + }) + .collect() + }; + assert_eq!( + text( + rows( + connection.as_ref(), + "SELECT CAST(v AS nvarchar(60)) FROM dbo.fractions ORDER BY id" + ) + .await + ), + [ + "0.12345678901234567890123456789012345678", + "0.00000000000000000000000000000000000000", + "0.50000000000000000000000000000000000000" + ] + ); + assert_eq!( + text( + rows( + connection.as_ref(), + "SELECT CAST(v AS nvarchar(60)) FROM dbo.amounts ORDER BY id" + ) + .await + ), + ["123456789012345678901234567890123456.78", "1.00"] + ); +} diff --git a/linux/crates/drivers/mssql/tests/integration/filter.rs b/linux/crates/drivers/mssql/tests/integration/filter.rs index 398af57b45..0394fc0259 100644 --- a/linux/crates/drivers/mssql/tests/integration/filter.rs +++ b/linux/crates/drivers/mssql/tests/integration/filter.rs @@ -1,4 +1,5 @@ -//! Filters on an xml column, which takes neither LIKE nor `=`. +//! Filters on an xml column, which takes neither LIKE nor `=`, and on +//! decimals, whose parameters are cast to one type. use tablepro_core::filter::{FilterOp, FilterRule, FilterSet, FilterValue, build_filter}; use tablepro_core::statement::StatementBuilder; @@ -45,3 +46,51 @@ async fn contains_and_equals_on_xml() { assert_eq!(rows, vec![vec![Value::Int(1)]], "{op:?}"); } } + +/// A decimal parameter is cast so the rows of an import share a type. +/// A filter value with more integer digits or a finer scale than the +/// column still compares as written. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires docker"] +async fn a_decimal_filter_keeps_the_digits_it_was_given() { + let fixture = start().await; + let connection = connect(&fixture, false).await; + setup( + connection.as_ref(), + &[ + "CREATE TABLE dbo.fractions (id int PRIMARY KEY, v decimal(38, 38), price decimal(10, 2))", + "INSERT INTO dbo.fractions VALUES (1, 0.5, 1.00), (2, 0.25, 1.01)", + ], + ) + .await; + let table = dbo("fractions"); + let columns = columns(connection.as_ref(), &table).await; + let dialect = connection.dialect(); + + for (column, op, text, expected) in [ + ("v", FilterOp::Lt, "1", vec![1, 2]), + ("price", FilterOp::Gt, "1.005", vec![2]), + ] { + let mut builder = StatementBuilder::new(); + builder.push_sql(&format!("SELECT id FROM {} WHERE ", dialect.qualify(&table))); + let filter = FilterSet { + rules: vec![FilterRule { + column: column.to_owned(), + op, + value: Some(FilterValue::Single(text.to_owned())), + }], + ..FilterSet::default() + }; + build_filter(&mut builder, dialect.as_ref(), &columns, &filter).expect("a filter"); + builder.push_sql(" ORDER BY id"); + + let rows = connection + .query_bound(&builder.finish(), RowLimit::EDITOR_DEFAULT, within(CALL)) + .await + .unwrap_or_else(|error| panic!("{column} {op:?} {text}: {error:?}")) + .rows; + + let expected: Vec> = expected.into_iter().map(|id| vec![Value::Int(id)]).collect(); + assert_eq!(rows, expected, "{column} {op:?} {text}"); + } +} diff --git a/linux/crates/drivers/mssql/tests/integration/main.rs b/linux/crates/drivers/mssql/tests/integration/main.rs index 0d924af9a7..fe2cf6a579 100644 --- a/linux/crates/drivers/mssql/tests/integration/main.rs +++ b/linux/crates/drivers/mssql/tests/integration/main.rs @@ -15,6 +15,7 @@ mod decode; mod filter; mod metadata; mod read_only; +mod round_trip; mod sessions; mod support; mod tls; diff --git a/linux/crates/drivers/mssql/tests/integration/round_trip.rs b/linux/crates/drivers/mssql/tests/integration/round_trip.rs new file mode 100644 index 0000000000..2ff7859c62 --- /dev/null +++ b/linux/crates/drivers/mssql/tests/integration/round_trip.rs @@ -0,0 +1,13 @@ +//! The export round trip on SQL Server: the value corpus out as CSV and +//! as SQL INSERT and back into an empty copy, and a CSV near the import's +//! size limit. + +mod suite { + use crate::conformance_engine::MssqlEngine; + + tablepro_test_support::round_trip_suite! { + engine: MssqlEngine, + docker: true, + near_limit: true, + } +} diff --git a/linux/crates/drivers/mysql/tests/integration/main.rs b/linux/crates/drivers/mysql/tests/integration/main.rs index e94b3b6a40..02b6234b7b 100644 --- a/linux/crates/drivers/mysql/tests/integration/main.rs +++ b/linux/crates/drivers/mysql/tests/integration/main.rs @@ -15,6 +15,7 @@ mod flavour_suite; mod lenient_server; mod metadata; mod non_transactional; +mod round_trip; mod row_limit; mod server_time_zone; mod support; diff --git a/linux/crates/drivers/mysql/tests/integration/round_trip.rs b/linux/crates/drivers/mysql/tests/integration/round_trip.rs new file mode 100644 index 0000000000..4dcf13ef7f --- /dev/null +++ b/linux/crates/drivers/mysql/tests/integration/round_trip.rs @@ -0,0 +1,23 @@ +//! The export round trip on MySQL and MariaDB: the value corpus out as +//! CSV and as SQL INSERT and back into an empty copy, and a CSV near the +//! import's size limit. + +mod mysql_suite { + use crate::conformance_engine::{MySqlEngine, MysqlServer}; + + tablepro_test_support::round_trip_suite! { + engine: MySqlEngine, + docker: true, + near_limit: true, + } +} + +mod mariadb_suite { + use crate::conformance_engine::{MariadbServer, MySqlEngine}; + + tablepro_test_support::round_trip_suite! { + engine: MySqlEngine, + docker: true, + near_limit: true, + } +} diff --git a/linux/crates/drivers/postgres/src/dialect.rs b/linux/crates/drivers/postgres/src/dialect.rs index 66ee5f3123..672e1f2d57 100644 --- a/linux/crates/drivers/postgres/src/dialect.rs +++ b/linux/crates/drivers/postgres/src/dialect.rs @@ -58,7 +58,7 @@ impl SqlDialect for PgDialect { let binding = match target { BindTarget::Pattern => Binding::plain(ParamType::Text), BindTarget::EngineRowId(part) => row_id_binding(part, &value)?, - BindTarget::Column(column) => column_binding(column)?, + BindTarget::Column(column) => column_binding(column, &value)?, }; let param = BoundParam::new(value, binding.param_type)?; let sql = match binding.cast { @@ -231,7 +231,7 @@ fn row_id_binding(part: EngineRowIdPart, value: &Value) -> Result Result { +fn column_binding(column: &ColumnInfo, value: &Value) -> Result { let column_type = &column.column_type; let catalog_cast = || { column_type @@ -257,6 +257,11 @@ fn column_binding(column: &ColumnInfo) -> Result { } => Binding::plain(ParamType::FixedText), ColumnKind::Text { .. } => Binding::plain(ParamType::Text), ColumnKind::Bytes { .. } => Binding::plain(ParamType::Bytes), + // A date or timestamp past chrono's last year is the server's own + // text, which the server reads under a cast to the column's type. + ColumnKind::Date | ColumnKind::Timestamp { .. } if matches!(value, Value::Other(_)) => { + Binding::server_text(catalog_cast()?) + } ColumnKind::Date => Binding::plain(ParamType::Date), ColumnKind::Time { with_offset: false, .. } => Binding::plain(ParamType::Time), ColumnKind::Time { with_offset: true, .. } => Binding::plain(ParamType::TimeTz), @@ -586,6 +591,25 @@ mod tests { ); } + #[test] + fn a_date_past_chronos_last_year_binds_as_the_servers_text() { + let date = column(ColumnKind::Date, Some("date")); + let far = Value::Other(Box::new(tablepro_core::value::OtherValue { + type_name: "date".to_owned(), + text: "5874897-12-31".to_owned(), + })); + + let placeholder = bind(0, far, &date).expect("a far date"); + + assert_eq!(placeholder.sql, "$1::date"); + assert_eq!( + placeholder.param.as_ref().map(BoundParam::param_type), + Some(&ParamType::ServerText(SqlTypeExpr::from_catalog_text("date"))) + ); + let near = bind(0, Value::Date(Temporal::Infinity), &date).expect("an infinite date"); + assert_eq!(near.sql, "$1", "a date chrono holds lost its binary form"); + } + #[test] fn network_and_bit_values_bind_as_their_text() { let inet = column( diff --git a/linux/crates/drivers/postgres/tests/integration/main.rs b/linux/crates/drivers/postgres/tests/integration/main.rs index a878fb62f2..71901286a5 100644 --- a/linux/crates/drivers/postgres/tests/integration/main.rs +++ b/linux/crates/drivers/postgres/tests/integration/main.rs @@ -19,6 +19,7 @@ mod metadata; mod paging; mod pgbouncer; mod read_only; +mod round_trip; mod slow_route; mod support; mod tls; diff --git a/linux/crates/drivers/postgres/tests/integration/round_trip.rs b/linux/crates/drivers/postgres/tests/integration/round_trip.rs new file mode 100644 index 0000000000..1bb2248f02 --- /dev/null +++ b/linux/crates/drivers/postgres/tests/integration/round_trip.rs @@ -0,0 +1,13 @@ +//! The export round trip on PostgreSQL: the value corpus out as CSV and +//! as SQL INSERT and back into an empty copy, and a CSV near the import's +//! size limit. + +mod suite { + use crate::conformance_engine::PgEngine; + + tablepro_test_support::round_trip_suite! { + engine: PgEngine, + docker: true, + near_limit: true, + } +} diff --git a/linux/crates/drivers/postgres/tests/integration/value_corpus.rs b/linux/crates/drivers/postgres/tests/integration/value_corpus.rs index 41db732fd5..a275dbbbee 100644 --- a/linux/crates/drivers/postgres/tests/integration/value_corpus.rs +++ b/linux/crates/drivers/postgres/tests/integration/value_corpus.rs @@ -64,6 +64,7 @@ fn temporal() -> Vec { .shows("'2024-02-29'", "2024-02-29") .shows("'0001-01-01'", "0001-01-01") .shows("'4713-01-01 BC'", "4713-01-01 BC") + .shows("'10000-01-01'", "10000-01-01") .shows("'5874897-12-31'", "5874897-12-31") .shows("'infinity'", "infinity") .null(), diff --git a/linux/crates/drivers/sqlite/tests/integration/main.rs b/linux/crates/drivers/sqlite/tests/integration/main.rs index f0bb977342..3c26ff96ff 100644 --- a/linux/crates/drivers/sqlite/tests/integration/main.rs +++ b/linux/crates/drivers/sqlite/tests/integration/main.rs @@ -23,6 +23,8 @@ mod metadata; #[cfg(test)] mod open_semantics; #[cfg(test)] +mod round_trip; +#[cfg(test)] mod sessions; #[cfg(test)] mod sqlite_engine; diff --git a/linux/crates/drivers/sqlite/tests/integration/round_trip.rs b/linux/crates/drivers/sqlite/tests/integration/round_trip.rs new file mode 100644 index 0000000000..c20c389b3d --- /dev/null +++ b/linux/crates/drivers/sqlite/tests/integration/round_trip.rs @@ -0,0 +1,11 @@ +//! The export round trip on SQLite: the value corpus out as CSV and as +//! SQL INSERT and back into an empty copy, and a CSV near the import's +//! size limit. No server is needed, so it runs in the unit suite. + +use crate::sqlite_engine::SqliteEngine; + +tablepro_test_support::round_trip_suite! { + engine: SqliteEngine, + docker: false, + near_limit: true, +} diff --git a/linux/crates/drivers/sqlite/tests/integration/value_corpus.rs b/linux/crates/drivers/sqlite/tests/integration/value_corpus.rs index 642736c93a..48490f893d 100644 --- a/linux/crates/drivers/sqlite/tests/integration/value_corpus.rs +++ b/linux/crates/drivers/sqlite/tests/integration/value_corpus.rs @@ -34,7 +34,16 @@ impl ValueCorpusEngine for SqliteEngine { CorpusColumn::new("v DECIMAL(10, 2)").shows("-12.5", "-12.5"), CorpusColumn::new("v DATETIME") .shows("'2024-02-29 12:00:00.123'", "2024-02-29 12:00:00.123") - .shows("'2024-03-10T02:30:00-08:00'", "2024-03-10T02:30:00-08:00"), + .shows("'2024-03-10T02:30:00-08:00'", "2024-03-10T02:30:00-08:00") + .shows("'2024-03-10 10:30:00Z'", "2024-03-10 10:30:00Z"), + // SQLite gives DATE, TIME and DATETIME one affinity, so each + // holds any form its date and time functions read. + CorpusColumn::new("v DATE") + .shows("'2024-02-29'", "2024-02-29") + .shows("'2024-03-10 02:30:00'", "2024-03-10 02:30:00"), + CorpusColumn::new("v TIME") + .shows("'12:30:00'", "12:30:00") + .shows("'12:30:00+05:30'", "12:30:00+05:30"), CorpusColumn::new("v BOOLEAN").shows("1", "1").shows("0", "0"), CorpusColumn::new("v JSON").shows(r#"'{"b": 1, "a": [1, 2.50]}'"#, r#"{"b": 1, "a": [1, 2.50]}"#), CorpusColumn::new("v") diff --git a/linux/crates/test-support/src/conformance/cases/mod.rs b/linux/crates/test-support/src/conformance/cases/mod.rs index a277de2a49..52e649e4a3 100644 --- a/linux/crates/test-support/src/conformance/cases/mod.rs +++ b/linux/crates/test-support/src/conformance/cases/mod.rs @@ -11,6 +11,7 @@ pub mod liveness; pub mod metadata; pub mod pool; pub mod read_only; +pub mod round_trip; pub mod row_guards; pub mod row_limit; pub mod statement_timeout; diff --git a/linux/crates/test-support/src/conformance/cases/round_trip.rs b/linux/crates/test-support/src/conformance/cases/round_trip.rs new file mode 100644 index 0000000000..9f22c387b3 --- /dev/null +++ b/linux/crates/test-support/src/conformance/cases/round_trip.rs @@ -0,0 +1,658 @@ +use std::collections::{BTreeMap, BTreeSet}; +use std::hash::{DefaultHasher, Hash, Hasher}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use async_trait::async_trait; +use tablepro_core::browse::{DeclaredPrecisionSink, ExportRequest, build_export}; +use tablepro_core::column::{ColumnInfo, ColumnType, ResultColumn}; +use tablepro_core::dialect::SqlDialect; +use tablepro_core::edit::{EditText, edit_text, read_only_reason}; +use tablepro_core::export::{ + CsvDelimiter, CsvEncoder, CsvLineBreak, CsvOptions, CsvQuote, RowEncoder, SqlInsertEncoder, +}; +use tablepro_core::import::{CsvImportOptions, MAX_CSV_FILE_BYTES}; +use tablepro_core::sql_syntax::script::{LexicalSettings, ScriptPlan}; +use tablepro_core::{ + Connection, EditorSession, EditorSessionOptions, FilterSet, ReadStatement, ResultEvent, ResultSink, SinkFlow, + TableRef, Value, WriteKind, +}; +use tokio_util::sync::CancellationToken; + +use crate::conformance::cases::value_corpus::{corpus_table, load, quoted, shown}; +use crate::conformance::support::{ + CALL_DEADLINE, apply_within, connect, create_scratch_table, expect, run_editor, run_options, unexpected, within, + writable, +}; +use crate::conformance::{ConformanceFailure, LargeTableEngine, ValueCorpusEngine}; +use crate::{import_csv, lock}; + +const CSV_CASE: &str = "csv_export_imports_back_as_the_same_rows"; +const SQL_CASE: &str = "sql_insert_export_runs_back_as_the_same_rows"; +const NEAR_LIMIT_CASE: &str = "csv_export_near_the_import_limit_imports_back"; + +/// What the import reads as NULL in every pairing, and what the export +/// writes for one. +const NULL_MARKER: &str = "NULL"; + +/// The app gives an import the row save's 120 seconds, and five more for +/// each statement. +const IMPORT_DEADLINE: Duration = Duration::from_secs(120); + +/// An export has no deadline in the app. This one only stops a read +/// that hangs. +const EXPORT_DEADLINE: Duration = Duration::from_secs(120); + +/// Under the export defaults these rows of 200-character labels make a +/// file just under `MAX_CSV_FILE_BYTES`. 200 characters is the widest +/// label every engine's scratch table takes. +const NEAR_LIMIT_ROWS: u64 = 315_000; +const NEAR_LIMIT_LABEL_WIDTH: usize = 200; + +/// How far under the limit the file may land and still test it. +const NEAR_LIMIT_MARGIN: usize = 1024 * 1024; + +const FILL_DEADLINE: Duration = Duration::from_secs(120); + +/// The app adds five seconds a statement to the row save's 120, which +/// for a file this size runs past the case's own limit. +const NEAR_LIMIT_IMPORT_DEADLINE: Duration = Duration::from_secs(180); + +/// A case makes a table and its copies for every corpus column, or +/// fills, exports and imports a 64 MiB table, so it takes longer than +/// `CASE_LIMIT`, and still ends before the `ci` nextest profile kills the +/// test at five minutes. +pub const ROUND_TRIP_CASE_LIMIT: Duration = Duration::from_secs(240); + +/// Rows of `id` whose differences are listed when a large table does +/// not come back whole. +const LISTED_IDS: usize = 10; + +/// Every corpus value with a text form exports as CSV and imports into +/// an empty copy of its table with the same rows, under each pairing of +/// writer and reader options that is meant to be lossless. A column of +/// a kind the import refuses, such as a blob or an array, and a value +/// with no text form, such as a blob in a SQLite column typed by its +/// values, are the SQL case's: CSV has no text that reads back as them. +pub async fn csv_export_imports_back_as_the_same_rows( + engine: E, +) -> Result<(), ConformanceFailure> { + let corpus = engine.value_corpus(); + expect(CSV_CASE, !corpus.is_empty(), || "the engine lists no corpus".to_owned())?; + let connection = connect(CSV_CASE, &engine, writable()).await?; + let mut wrong = Vec::new(); + let mut round_trips = 0; + for (index, column) in corpus.iter().enumerate() { + let source = match load(&engine, connection.as_ref(), &format!("csv_{index}"), column).await { + Ok(table) => table, + Err(failure) => { + wrong.push(failure.to_string()); + continue; + } + }; + let columns = connection.fetch_columns(&source, within(CALL_DEADLINE)).await?; + if columns + .iter() + .any(|info| read_only_reason(&info.column_type.kind).is_some()) + { + continue; + } + let expected = read_rows(connection.as_ref(), &source, texts, KeepValues::Yes).await?; + for (writer, reader) in pairings() { + let what = format!( + "`{}` as {:?}, {:?}, {:?}", + column.declaration, writer.delimiter, writer.quote, writer.line_break + ); + let copy = corpus_table(&engine, connection.as_ref(), &format!("csv_{index}_copy"), column, &[]).await?; + let file = encode(&expected.values, &csv_encoder(writer))?; + if let Err(failure) = import_csv(connection.as_ref(), ©, &file, &reader, within(IMPORT_DEADLINE)).await + { + wrong.push(format!("{what}: the import failed: {failure:?}")); + continue; + } + let actual = read_rows(connection.as_ref(), ©, texts, KeepValues::No).await?; + wrong.extend(differences(&what, &expected, &actual, &expected.without_text)); + round_trips += 1; + } + } + connection.close().await?; + expect(CSV_CASE, round_trips > 0, || { + "no corpus column could travel as CSV".to_owned() + })?; + expect(CSV_CASE, wrong.is_empty(), || { + format!( + "{} differences after a CSV round trip:\n{}", + wrong.len(), + wrong.join("\n") + ) + }) +} + +/// Every corpus value exports as SQL INSERT, and the file, run in an +/// editor against an empty copy of its table, gives back the same rows, +/// each value of the same kind. +pub async fn sql_insert_export_runs_back_as_the_same_rows( + engine: E, +) -> Result<(), ConformanceFailure> { + let corpus = engine.value_corpus(); + expect(SQL_CASE, !corpus.is_empty(), || "the engine lists no corpus".to_owned())?; + let connection = connect(SQL_CASE, &engine, writable()).await?; + let session = connection + .open_editor_session(EditorSessionOptions::default(), CancellationToken::new()) + .await?; + let mut wrong = Vec::new(); + for (index, column) in corpus.iter().enumerate() { + let source = match load(&engine, connection.as_ref(), &format!("sql_{index}"), column).await { + Ok(table) => table, + Err(failure) => { + wrong.push(failure.to_string()); + continue; + } + }; + let what = format!("`{}` as SQL INSERT", column.declaration); + let expected = read_rows(connection.as_ref(), &source, typed_texts, KeepValues::No).await?; + let copy = corpus_table(&engine, connection.as_ref(), &format!("sql_{index}_copy"), column, &[]).await?; + let generated = connection + .fetch_columns(&source, within(CALL_DEADLINE)) + .await? + .into_iter() + .filter(|info| info.is_generated) + .map(|info| info.name) + .collect(); + let encoder = insert_encoder(connection.dialect(), copy.clone(), generated); + let script = export_file(connection.as_ref(), &source, encoder).await?; + let script = String::from_utf8(script).map_err(ConformanceFailure::setup)?; + if let Some(failure) = run_script(session.as_ref(), &*connection.dialect(), &script).await? { + wrong.push(format!("{what}: {failure}")); + continue; + } + let actual = read_rows(connection.as_ref(), ©, typed_texts, KeepValues::No).await?; + wrong.extend(differences(&what, &expected, &actual, &BTreeSet::new())); + } + session.close().await; + connection.close().await?; + expect(SQL_CASE, wrong.is_empty(), || { + format!( + "{} differences after a SQL INSERT round trip:\n{}", + wrong.len(), + wrong.join("\n") + ) + }) +} + +/// Run `script` as the editor runs a file: split by the engine's +/// grammar and sent a batch at a time. Returns why it stopped short. +async fn run_script( + session: &dyn EditorSession, + dialect: &dyn SqlDialect, + script: &str, +) -> Result, ConformanceFailure> { + let grammar = dialect.grammar(); + let plan = ScriptPlan::build(script, grammar, LexicalSettings::default_for(grammar)); + if let Some(diagnostic) = plan.diagnostics().first() { + return Ok(Some(format!("the file does not split cleanly: {diagnostic:?}"))); + } + for batch in plan.batches() { + let text = script.get(batch.range.clone()).unwrap_or_default().trim(); + if text.is_empty() { + continue; + } + for _ in 0..batch.repeat.get() { + let (summary, _) = run_editor(session, text, run_options()).await?; + if !summary.succeeded() { + return Ok(Some(format!("`{text}` ended {:?}", summary.end))); + } + } + } + Ok(None) +} + +/// A table whose CSV, under the export defaults, lands just under the +/// import's size limit imports back into an empty copy with every row +/// the same. +pub async fn csv_export_near_the_import_limit_imports_back( + engine: E, +) -> Result<(), ConformanceFailure> { + let connection = connect(NEAR_LIMIT_CASE, &engine, writable()).await?; + let source = create_scratch_table(&engine, connection.as_ref(), "near_limit").await?; + let fill = engine.fill_scratch_table_sql(&source, NEAR_LIMIT_ROWS, NEAR_LIMIT_LABEL_WIDTH); + apply_within( + connection.as_ref(), + &source, + WriteKind::SchemaChanges, + fill, + FILL_DEADLINE, + ) + .await?; + let file = export_file(connection.as_ref(), &source, csv_encoder(CsvOptions::default())).await?; + let size = file.len(); + expect( + NEAR_LIMIT_CASE, + size <= MAX_CSV_FILE_BYTES && size + NEAR_LIMIT_MARGIN > MAX_CSV_FILE_BYTES, + || { + format!( + "the file is {size} bytes, not within {NEAR_LIMIT_MARGIN} under the {MAX_CSV_FILE_BYTES}-byte limit" + ) + }, + )?; + let copy = create_scratch_table(&engine, connection.as_ref(), "near_limit_copy").await?; + import_csv( + connection.as_ref(), + ©, + &file, + &CsvImportOptions::default(), + within(NEAR_LIMIT_IMPORT_DEADLINE), + ) + .await + .map_err(|failure| unexpected(NEAR_LIMIT_CASE, format!("the import failed: {failure:?}")))?; + drop(file); + let expected = read_rows(connection.as_ref(), &source, digest, KeepValues::No).await?; + let actual = read_rows(connection.as_ref(), ©, digest, KeepValues::No).await?; + connection.close().await?; + expect(NEAR_LIMIT_CASE, expected.by_id.len() as u64 == NEAR_LIMIT_ROWS, || { + format!("the table read {} rows, not {NEAR_LIMIT_ROWS}", expected.by_id.len()) + })?; + let wrong = digest_differences(&expected, &actual); + expect(NEAR_LIMIT_CASE, wrong.is_empty(), || wrong.join("\n")) +} + +/// A writer's options and the reader's that match them. Each pairing +/// is meant to be lossless: NULL is written as a marker the reader takes +/// back, so an empty string stays one, and formulas are left as they +/// are. Between them they cover every delimiter, both quoting styles +/// that keep a delimiter inside a value, and every line break. +fn pairings() -> Vec<(CsvOptions, CsvImportOptions)> { + [ + (CsvDelimiter::Comma, CsvQuote::IfNeeded, CsvLineBreak::Lf), + (CsvDelimiter::Semicolon, CsvQuote::Always, CsvLineBreak::CrLf), + (CsvDelimiter::Tab, CsvQuote::IfNeeded, CsvLineBreak::Cr), + (CsvDelimiter::Pipe, CsvQuote::Always, CsvLineBreak::Lf), + ] + .into_iter() + .map(|(delimiter, quote, line_break)| { + let writer = CsvOptions { + null_to_empty: false, + sanitize_formulas: false, + delimiter, + quote, + line_break, + ..CsvOptions::default() + }; + let reader = CsvImportOptions { + delimiter, + has_header: true, + null_marker: NULL_MARKER.to_owned(), + }; + (writer, reader) + }) + .collect() +} + +type EncoderFor = Box) -> Box + Send>; + +fn csv_encoder(options: CsvOptions) -> EncoderFor { + Box::new(move |columns| Box::new(CsvEncoder::new(columns, options.clone()))) +} + +/// INSERT statements into `table`, leaving out the columns the server +/// computes, as a table tab's export does. +fn insert_encoder(dialect: Arc, table: TableRef, generated: Vec) -> EncoderFor { + Box::new(move |columns| Box::new(SqlInsertEncoder::new(dialect.clone(), &table, columns, &generated))) +} + +/// Rows an export read, encoded one at a time from the columns the read +/// announced, as the export sink encodes them as they arrive. The app +/// writes a page the grid holds this way too. +fn encode(read: &ReadValues, encoder_for: &EncoderFor) -> Result, ConformanceFailure> { + let columns = read + .columns + .clone() + .ok_or_else(|| ConformanceFailure::setup("the export read started no result set"))?; + let mut encoder = encoder_for(columns); + let mut bytes = Vec::new(); + encoder.begin(&mut bytes); + for row in &read.rows { + encoder + .row(&mut bytes, &mut row.iter()) + .map_err(|error| ConformanceFailure::setup(format!("the export failed: {error}")))?; + } + encoder.finish(&mut bytes); + Ok(bytes) +} + +/// Every row of `table` encoded by the encoder `encoder_for` builds, as +/// the export dialog's Entire Table writes it. +async fn export_file( + connection: &dyn Connection, + table: &TableRef, + encoder_for: EncoderFor, +) -> Result, ConformanceFailure> { + let file = Arc::new(Mutex::new(Encoded::default())); + let sink = EncodingSink { + file: Arc::clone(&file), + encoder_for, + encoder: None, + }; + stream_table(connection, table, Box::new(sink)).await?; + let file = std::mem::take(&mut *lock(&file)); + match file.failure { + Some(failure) => Err(ConformanceFailure::setup(format!("the export failed: {failure}"))), + None => Ok(file.bytes), + } +} + +/// Stream every row of `table` into `sink`: the statement `build_export` +/// writes, read at the precision the catalogue declares, as the export +/// dialog's Entire Table reads a table tab. +async fn stream_table( + connection: &dyn Connection, + table: &TableRef, + sink: Box, +) -> Result<(), ConformanceFailure> { + let columns: Arc<[ColumnInfo]> = connection.fetch_columns(table, within(CALL_DEADLINE)).await?.into(); + let identity = connection.row_identity(table, within(CALL_DEADLINE)).await?; + let statement = build_export( + &*connection.dialect(), + &ExportRequest { + table, + columns: &columns, + filter: &FilterSet::default(), + sort: &[], + identity: &identity, + }, + ) + .map_err(ConformanceFailure::setup)?; + connection + .stream_read( + &ReadStatement::Generated(statement), + Box::new(DeclaredPrecisionSink::new(sink, columns)), + within(EXPORT_DEADLINE), + ) + .await?; + Ok(()) +} + +#[derive(Default)] +struct Encoded { + bytes: Vec, + failure: Option, +} + +/// Encodes rows as they arrive, as the app's export sink does: the +/// encoder is built from the columns the stream announces. +struct EncodingSink { + file: Arc>, + encoder_for: EncoderFor, + encoder: Option>, +} + +#[async_trait] +impl ResultSink for EncodingSink { + async fn accept(&mut self, event: ResultEvent) -> SinkFlow { + let mut file = lock(&self.file); + match event { + ResultEvent::ResultSetStarted { columns } => { + if self.encoder.is_some() { + file.failure = Some("a second result set started".to_owned()); + return SinkFlow::ConsumerGone; + } + let mut encoder = (self.encoder_for)(columns); + encoder.begin(&mut file.bytes); + self.encoder = Some(encoder); + } + ResultEvent::Rows(rows) => { + let Some(encoder) = self.encoder.as_mut() else { + file.failure = Some("rows arrived before their result set".to_owned()); + return SinkFlow::ConsumerGone; + }; + for row in &rows { + if let Err(error) = encoder.row(&mut file.bytes, &mut row.iter()) { + file.failure = Some(error.to_string()); + return SinkFlow::ConsumerGone; + } + } + } + ResultEvent::ResultSetFinished { .. } => { + if let Some(encoder) = self.encoder.as_mut() { + encoder.finish(&mut file.bytes); + } + } + _ => {} + } + SinkFlow::Continue + } +} + +/// A cell as the grid shows it, with the kind of value behind it. +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +struct Cell { + kind: &'static str, + text: Option, +} + +/// What a comparison keeps of a row. +#[derive(Debug, Clone, PartialEq, Eq)] +enum Kept { + /// What the grid shows. CSV carries no kinds: a column SQLite types + /// by its values takes an imported `42` as text. + Texts(Vec>), + /// The text and the kind of every value, which a SQL literal keeps. + Typed(Vec), +} + +fn texts(cells: Vec) -> Kept { + Kept::Texts(cells.into_iter().map(|cell| cell.text).collect()) +} + +fn typed_texts(cells: Vec) -> Kept { + Kept::Typed(cells) +} + +/// The texts of a row, hashed, so a large table compares without +/// holding every row twice. +fn digest(cells: Vec) -> u64 { + let mut hasher = DefaultHasher::new(); + for cell in cells { + cell.text.hash(&mut hasher); + } + hasher.finish() +} + +/// Every row of a table by the text of its `id`. +struct Rows { + columns: Vec, + by_id: BTreeMap, + /// Rows holding a value with no text form the import reads back. + without_text: BTreeSet, + duplicate: Option, + failure: Option, + values: ReadValues, +} + +impl Default for Rows { + fn default() -> Self { + Self { + columns: Vec::new(), + by_id: BTreeMap::new(), + without_text: BTreeSet::new(), + duplicate: None, + failure: None, + values: ReadValues::default(), + } + } +} + +/// The values a read sent, under the columns its result set started +/// with, which are the ones the export sink builds its encoder from. +#[derive(Default)] +struct ReadValues { + columns: Option>, + rows: Vec>, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum KeepValues { + Yes, + No, +} + +/// Every row of `table`, read as an export reads it and kept by `keep`, +/// with the values themselves when `values` says so. +async fn read_rows( + connection: &dyn Connection, + table: &TableRef, + keep: fn(Vec) -> T, + values: KeepValues, +) -> Result, ConformanceFailure> { + let read = Arc::new(Mutex::new(Rows::default())); + let sink = RowSink { + read: Arc::clone(&read), + keep, + values, + columns: None, + id: None, + }; + stream_table(connection, table, Box::new(sink)).await?; + let rows = std::mem::take(&mut *lock(&read)); + match rows.failure { + Some(failure) => Err(ConformanceFailure::setup(failure)), + None => Ok(rows), + } +} + +struct RowSink { + read: Arc>>, + keep: fn(Vec) -> T, + values: KeepValues, + columns: Option>, + id: Option, +} + +#[async_trait] +impl ResultSink for RowSink { + async fn accept(&mut self, event: ResultEvent) -> SinkFlow { + let mut read = lock(&self.read); + match event { + ResultEvent::ResultSetStarted { columns } | ResultEvent::ColumnsResolved { columns, .. } => { + if read.values.columns.is_none() { + read.values.columns = Some(Arc::clone(&columns)); + } + self.id = columns.iter().position(|column| column.name == "id"); + read.columns = columns.iter().map(|column| column.name.clone()).collect(); + self.columns = Some(columns); + } + ResultEvent::Rows(rows) => { + let (Some(columns), Some(id)) = (&self.columns, self.id) else { + read.failure = Some(format!("no `id` column in {:?}", read.columns)); + return SinkFlow::ConsumerGone; + }; + let untyped = ColumnType::unknown(); + for row in rows { + let column_type = |index: usize| columns.get(index).map_or(&untyped, |column| &column.column_type); + let has_text = row + .iter() + .enumerate() + .all(|(index, value)| matches!(edit_text(value, column_type(index)), EditText::Editable(_))); + let cells: Vec = row + .iter() + .enumerate() + .map(|(index, value)| Cell { + kind: value.variant_name(), + text: shown(value, column_type(index)), + }) + .collect(); + let key = cells.get(id).and_then(|cell| cell.text.clone()).unwrap_or_default(); + if self.values == KeepValues::Yes { + read.values.rows.push(row); + } + if !has_text { + read.without_text.insert(key.clone()); + } + if read.by_id.insert(key.clone(), (self.keep)(cells)).is_some() { + read.duplicate.get_or_insert(key); + } + } + } + _ => {} + } + SinkFlow::Continue + } +} + +/// Every way `copy` differs from `source`, one line each, leaving out +/// the rows `skipped` names. +fn differences(what: &str, source: &Rows, copy: &Rows, skipped: &BTreeSet) -> Vec { + let mut wrong = Vec::new(); + if copy.columns != source.columns { + wrong.push(format!( + "{what}: the copy has the columns {:?}, the table {:?}", + copy.columns, source.columns + )); + } + if let Some(id) = ©.duplicate { + wrong.push(format!("{what}: row {id} came back twice")); + } + let kept = |rows: &Rows| rows.by_id.keys().filter(|id| !skipped.contains(*id)).count(); + let (exported, imported) = (kept(source), kept(copy)); + if exported != imported { + wrong.push(format!("{what}: {imported} rows came back, {exported} were exported")); + } + for (id, expected) in source.by_id.iter().filter(|(id, _)| !skipped.contains(*id)) { + let Some(actual) = copy.by_id.get(id) else { + wrong.push(format!("{what}: row {id} did not come back")); + continue; + }; + if actual != expected { + wrong.push(format!( + "{what}: row {id} came back as {}, exported as {}", + described(actual), + described(expected) + )); + } + } + for id in copy.by_id.keys().filter(|id| !skipped.contains(*id)) { + if !source.by_id.contains_key(id) { + wrong.push(format!("{what}: row {id} came back but was never exported")); + } + } + wrong +} + +fn described(kept: &Kept) -> String { + let cells: Vec = match kept { + Kept::Texts(texts) => texts.iter().map(|text| quoted(text.as_deref())).collect(), + Kept::Typed(cells) => cells + .iter() + .map(|cell| format!("{} {}", cell.kind, quoted(cell.text.as_deref()))) + .collect(), + }; + format!("({})", cells.join(", ")) +} + +/// The rows of a large table that did not come back the same, the +/// first few of each kind by `id`. +fn digest_differences(source: &Rows, copy: &Rows) -> Vec { + let missing: Vec<&String> = source.by_id.keys().filter(|id| !copy.by_id.contains_key(*id)).collect(); + let extra: Vec<&String> = copy.by_id.keys().filter(|id| !source.by_id.contains_key(*id)).collect(); + let changed: Vec<&String> = source + .by_id + .iter() + .filter(|(id, digest)| copy.by_id.get(*id).is_some_and(|other| other != *digest)) + .map(|(id, _)| id) + .collect(); + let mut wrong = Vec::new(); + if let Some(id) = ©.duplicate { + wrong.push(format!("row {id} came back twice")); + } + for (count, ids, says) in [ + (missing.len(), &missing, "did not come back"), + (extra.len(), &extra, "came back but were never exported"), + (changed.len(), &changed, "came back changed"), + ] { + if count > 0 { + let listed: Vec<&str> = ids.iter().take(LISTED_IDS).map(|id| id.as_str()).collect(); + wrong.push(format!("{count} rows {says}, among them {}", listed.join(", "))); + } + } + wrong +} diff --git a/linux/crates/test-support/src/conformance/cases/value_corpus.rs b/linux/crates/test-support/src/conformance/cases/value_corpus.rs index 28e11208b5..20f9d43659 100644 --- a/linux/crates/test-support/src/conformance/cases/value_corpus.rs +++ b/linux/crates/test-support/src/conformance/cases/value_corpus.rs @@ -13,8 +13,7 @@ use tablepro_core::{Connection, DriverError, EditorSession, EditorSessionOptions use tokio_util::sync::CancellationToken; use crate::conformance::support::{ - CALL_DEADLINE, Collected, apply, connect, create_scratch_table, expect, run_completed, run_options, within, - writable, + CALL_DEADLINE, Collected, apply, connect, expect, run_completed, run_options, unique, within, writable, }; use crate::conformance::{ConformanceEngine, ConformanceFailure, CorpusColumn, ValueCorpusEngine}; @@ -47,7 +46,7 @@ pub async fn value_corpus_reads_as_the_grid_shows(engine: for (index, column) in corpus.iter().enumerate() { // A column the server refuses is listed with the wrong values, // so one run reports the whole corpus. - let table = match load(&engine, connection.as_ref(), index, column).await { + let table = match load(&engine, connection.as_ref(), &format!("corpus_{index}"), column).await { Ok(table) => table, Err(failure) => { wrong.push(failure.to_string()); @@ -72,20 +71,17 @@ pub async fn value_corpus_reads_as_the_grid_shows(engine: }) } -async fn load( +/// A scratch table holding `column` as `v`, with a row for each of its +/// values. +pub(crate) async fn load( engine: &E, connection: &dyn Connection, - index: usize, + prefix: &str, column: &CorpusColumn, ) -> Result { - let table = create_scratch_table(engine, connection, &format!("corpus_{index}")).await?; + let table = corpus_table(engine, connection, prefix, column, &column.setup_sql).await?; let dialect = connection.dialect(); let table_sql = dialect.qualify(&table); - let mut schema = column.setup_sql.clone(); - schema.push(format!("ALTER TABLE {table_sql} ADD {}", column.declaration)); - apply(connection, &table, WriteKind::SchemaChanges, schema) - .await - .map_err(|error| ConformanceFailure::setup(format!("`{}`: {error}", column.declaration)))?; let names = [ dialect.quote_identifier("id"), dialect.quote_identifier("label"), @@ -109,6 +105,30 @@ async fn load( Ok(table) } +/// An empty scratch table with `column` added as `v`, after `setup`. A +/// second table of the same column passes no setup: a type it creates +/// already exists. +pub(crate) async fn corpus_table( + engine: &E, + connection: &dyn Connection, + prefix: &str, + column: &CorpusColumn, + setup: &[String], +) -> Result { + let table = engine.scratch_table(&unique(prefix)); + let mut schema = vec![engine.create_scratch_table_sql(&table)]; + schema.extend_from_slice(setup); + schema.push(format!( + "ALTER TABLE {} ADD {}", + connection.dialect().qualify(&table), + column.declaration + )); + apply(connection, &table, WriteKind::SchemaChanges, schema) + .await + .map_err(|error| ConformanceFailure::setup(format!("`{}`: {error}", column.declaration)))?; + Ok(table) +} + /// The first page of the table, read by `read_page` as the browse tab /// reads one. async fn browse(connection: &dyn Connection, table: &TableRef) -> Result { @@ -256,12 +276,7 @@ fn compare(path: Path, column: &CorpusColumn, read: &Read, wrong: &mut Vec None, - // What the grid shows for a value the driver could not read. - Value::Undecodable(undecoded) => Some(format!("", undecoded.type_name)), - other => Some(value_text(other, &read.column_type).unwrap_or_else(|| format!("<{other:?}>"))), - }; + let shown = shown(read_value, &read.column_type); if shown.as_deref() != expected { wrong.push(format!( "{path_name}: `{declaration}` holding {sql} shows {} as {}, expected {}", @@ -273,7 +288,18 @@ fn compare(path: Path, column: &CorpusColumn, read: &Read, wrong: &mut Vec) -> String { +/// The text the grid shows for `value` in a column of `column_type`, +/// `None` for NULL. +pub(crate) fn shown(value: &Value, column_type: &ColumnType) -> Option { + match value { + Value::Null => None, + // What the grid shows for a value the driver could not read. + Value::Undecodable(undecoded) => Some(format!("", undecoded.type_name)), + other => Some(value_text(other, column_type).unwrap_or_else(|| format!("<{other:?}>"))), + } +} + +pub(crate) fn quoted(text: Option<&str>) -> String { match text { Some(text) => format!("{text:?}"), None => "NULL".to_owned(), diff --git a/linux/crates/test-support/src/conformance/mod.rs b/linux/crates/test-support/src/conformance/mod.rs index aafe2a54ab..b11dbfbe62 100644 --- a/linux/crates/test-support/src/conformance/mod.rs +++ b/linux/crates/test-support/src/conformance/mod.rs @@ -2,8 +2,8 @@ //! //! A driver's integration target implements an engine adapter and //! invokes `connection_conformance_suite!`, `ddl_conformance_suite!`, -//! `value_corpus_suite!` and `tls_conformance_suite!` with flags for -//! what the engine supports. A +//! `value_corpus_suite!`, `round_trip_suite!` and `tls_conformance_suite!` +//! with flags for what the engine supports. A //! flag that is off emits no test. The flags with a capability behind them (`endpoint`, //! `editor_transactions` and the TLS `client_identity`) are checked //! against the driver's capability table, so those cases cannot be @@ -30,6 +30,7 @@ mod large_table_suite_macro; mod network_conformance_engine; mod pooled_conformance_engine; mod resident_peak; +mod round_trip_suite_macro; mod run_case; mod suite_endpoint; mod support; diff --git a/linux/crates/test-support/src/conformance/round_trip_suite_macro.rs b/linux/crates/test-support/src/conformance/round_trip_suite_macro.rs new file mode 100644 index 0000000000..b9c281a980 --- /dev/null +++ b/linux/crates/test-support/src/conformance/round_trip_suite_macro.rs @@ -0,0 +1,54 @@ +/// Emit the export round-trip cases for one engine adapter, which +/// implements `ValueCorpusEngine` and `LargeTableEngine`. `docker: true` +/// marks them `#[ignore = "requires docker"]`. +/// +/// The value corpus goes out as CSV and comes back through the import, +/// and goes out as SQL INSERT and comes back through an editor run, each +/// into an empty copy of its table. `near_limit: true` adds a case that +/// exports a table whose CSV lands just under the import's size limit +/// and imports it back. It describes how fast the engine takes an +/// import, so no capability stands behind it. +#[macro_export] +macro_rules! round_trip_suite { + (engine: $engine:ty, docker: $docker:tt, near_limit: $near_limit:tt $(,)?) => { + $crate::__round_trip_case! { $docker, $engine, csv_export_imports_back_as_the_same_rows } + $crate::__round_trip_case! { $docker, $engine, sql_insert_export_runs_back_as_the_same_rows } + $crate::__near_limit_case! { $near_limit, $docker, $engine } + }; +} + +#[doc(hidden)] +#[macro_export] +macro_rules! __near_limit_case { + (true, $docker:tt, $engine:ty) => { + $crate::__round_trip_case! { $docker, $engine, csv_export_near_the_import_limit_imports_back } + }; + (false, $docker:tt, $engine:ty) => {}; +} + +#[doc(hidden)] +#[macro_export] +macro_rules! __round_trip_case { + (true, $engine:ty, $name:ident) => { + #[::tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[ignore = "requires docker"] + async fn $name() -> ::core::result::Result<(), $crate::conformance::ConformanceFailure> { + $crate::__round_trip_case!(run $engine, $name) + } + }; + (false, $engine:ty, $name:ident) => { + #[::tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn $name() -> ::core::result::Result<(), $crate::conformance::ConformanceFailure> { + $crate::__round_trip_case!(run $engine, $name) + } + }; + (run $engine:ty, $name:ident) => { + $crate::conformance::run_case_within( + ::core::stringify!($name), + $crate::conformance::cases::round_trip::ROUND_TRIP_CASE_LIMIT, + <$engine as $crate::conformance::ConformanceEngine>::start(), + $crate::conformance::cases::round_trip::$name::<$engine>, + ) + .await + }; +} diff --git a/linux/docs/drivers/clickhouse.md b/linux/docs/drivers/clickhouse.md index e8d00ce3a8..9160e81c08 100644 --- a/linux/docs/drivers/clickhouse.md +++ b/linux/docs/drivers/clickhouse.md @@ -75,6 +75,8 @@ The grid names a ClickHouse row by its sorting key (`RowIdentity::SortingKey`), A failed statement is sorted by whether it can have landed. A server error that refused it (a syntax error, an unknown column) did not apply. A cancel, a timeout, an unfinished mutation wait (341), a network error, or a body with no error code may have applied, so the outcome comes from the `WriteLedger`, with a `CommitToken::ClickHouseQuery` of every statement's query id. +A Map, Tuple, Variant, Dynamic, JSON or geo value reads as the JSON the server writes for it, which a string of it does not cast back from. A SQL INSERT export writes it as `JSONExtract('', '')`, which parses it into the column's type. + A column made non-nullable in the structure tab is written as `MODIFY COLUMN name T DEFAULT `: from 25.12 the server refuses the change without a DEFAULT to put in the column's NULLs. The draft's default fills them; with none, `defaultValueOfTypeName` does, as older servers did on their own, and a `REMOVE DEFAULT` after it leaves the column with no default. ### Resolving an unknown outcome diff --git a/linux/docs/drivers/postgresql.md b/linux/docs/drivers/postgresql.md index 68ae4c947c..a8bca11a92 100644 --- a/linux/docs/drivers/postgresql.md +++ b/linux/docs/drivers/postgresql.md @@ -38,7 +38,7 @@ COMMIT - A connection whose flight did not end with an acknowledged COMMIT or ROLLBACK is taken out of the pool, because deadpool's recycle check (`RecyclingMethod::Verified`) passes inside a half-finished transaction. - A read that returns no rows still has its columns: the statement runs once more as a subquery right-joined to one row on a false condition, in a read-only transaction of its own. - A column of a type tokio-postgres does not know (an enum, `citext`, a domain) would make it prepare a type lookup under a name. Pages project such columns as text, so a read never does. -- chrono ends at 262143 AD, and the server keeps dates to 5874897 AD and timestamps to 294276 AD. A value past chrono's end reads as the text the server would print for it, so the cell shows it rather than saying it cannot. +- chrono ends at 262143 AD, and the server keeps dates to 5874897 AD and timestamps to 294276 AD. A value past chrono's end reads as the text the server would print for it, so the cell shows it rather than saying it cannot. A cell edit or a CSV import takes such text back, and it binds as text cast to the column's type. `probe` takes a pool slot without waiting; when every slot is busy it opens a connection of its own and runs `SELECT 1`. `stream_read` opens a connection of its own, runs `START TRANSACTION READ ONLY` with no statement timeout, and refuses user SQL that would end that transaction (transaction control, `CALL`, `DO`) before sending anything. diff --git a/linux/docs/drivers/sql-server.md b/linux/docs/drivers/sql-server.md index 1906a2d872..4ae068ac84 100644 --- a/linux/docs/drivers/sql-server.md +++ b/linux/docs/drivers/sql-server.md @@ -94,6 +94,8 @@ SQL Server has no session statement timeout. A workspace call's deadline is the `WriteAtomicity::Transactional` for row and schema changes. A save is `BEGIN TRANSACTION`, each step, then `COMMIT TRANSACTION`: - A statement with parameters runs through tiberius's `execute`; one without runs through `sp_executesql` with the statement alone, because SQL Server refuses `CREATE VIEW` or `CREATE TRIGGER` next to an empty parameter list. +- A decimal parameter is cast to `decimal(38, s)`: `s` is the column's scale, or the value's when finer, as far as the value's integer digits leave room. SQL Server types the rows of a multi-row VALUES as UNION ALL does, so without one type for every row an import would narrow a full-precision value beside a shorter one. +- tiberius gives a decimal parameter whose integer part is 0 one digit more than its scale, so a value at scale 38 would claim precision 39, which the server refuses. Such a value goes as its text, which the cast converts exactly. - The affected-row count includes rows a trigger touched, so a save proves it hit the right row with a probe before the statement rather than with that count. - A `COMMIT` the server answers with an error rolled back; one whose connection dropped is `CommitOutcomeUnknown` with no token. There is nothing to ask the server afterwards (`CommitResolutionSupport::Unavailable`). - SQL Server cannot suspend foreign keys for one batch, so a batch that asks for it is refused before anything is sent. diff --git a/linux/docs/testing.md b/linux/docs/testing.md index 2855c9be25..99aeb728fd 100644 --- a/linux/docs/testing.md +++ b/linux/docs/testing.md @@ -195,6 +195,24 @@ The corpus is a list of `CorpusColumn`s: a column clause, values of it written a Every target runs it, the MySQL target on MariaDB as well, and SQLite's runs in the unit suite. +### Export round trip + +`round_trip_suite!` checks that a file the export writes loads back with the same rows. Each driver's `crates/drivers//tests/integration/round_trip.rs` invokes it with an engine that implements `ValueCorpusEngine` and `LargeTableEngine`: + +```rust +tablepro_test_support::round_trip_suite! { + engine: PgEngine, + docker: true, + near_limit: true, +} +``` + +- `csv_export_imports_back_as_the_same_rows` loads each corpus column into a table, exports it as CSV and imports the file into an empty copy, through `import_csv`, which runs the plan the import dialog runs. It does this under four pairings of export and import options that are meant to lose nothing: every delimiter, quoting always or where needed, LF, CRLF and CR line breaks, NULL written and read as `NULL`, and formulas left as they are. Row count and the text of every row have to match. A column of a kind `read_only_reason` names (bytes, arrays, spatial, variant and composite values) and a value `edit_text` gives no text for, such as a blob in a SQLite column typed by its values, have no CSV text the import reads back, so the SQL case covers them. +- `sql_insert_export_runs_back_as_the_same_rows` exports each corpus column as SQL INSERT into an empty copy and runs the file as the editor runs one: split by the engine's grammar and sent a batch at a time. Every value has to come back with the same text and of the same kind. +- `csv_export_near_the_import_limit_imports_back`, with `near_limit: true`, fills a table through `LargeTableEngine` so its CSV under the export defaults lands within 1 MiB under `MAX_CSV_FILE_BYTES`, imports it into an empty copy, and compares a hash of every row. ClickHouse passes `false`: its import sends values in the request URL, 8 KiB a statement, so such a file takes some 8,000 statements and longer than a case may run. + +Each case has four minutes (`ROUND_TRIP_CASE_LIMIT`). Every target runs the suite, the MySQL target on MariaDB as well, and SQLite's runs in the unit suite. + ### TLS conformance Every network driver's `crates/drivers//tests/integration/tls.rs` also runs `tls_conformance_suite!` against a TLS-only server built from `TestPki`: