Operate in exactly one schema, and filter every catalog query to it

CHANGELOG.md

@@ -339,6 +339,91 @@ hardening standalone use; the highlights:
   Connect — no long-lived token is stored anywhere. (#113)
 - `CITATION.cff`, so GitHub renders a citation for the package. (#124)
 
+### Fixed after the first release candidates
+
+- **`max_id` and `min_id` compose the table name they are given.** Both took a
+  `table=` argument and formatted it into the statement as text, so a name
+  needing quotes was a syntax error and a name carrying its own statement ran
+  it. Both now use `Identifier`. `max_id` returns -1 for an empty table, which
+  is the only empty sentinel: 0 is a real id, and `random()` treated a table
+  whose single row had id 0 as empty.
+- **Approximate statistics are scaled by the table they describe.**
+  `_approx_most_common` took `reltuples` from a hard-coded `public.nf_fields`
+  while taking frequencies from the real table, so on every other table the
+  estimate was that table's frequencies multiplied by an unrelated row count.
+  The column type it interpolates now goes through the validated
+  `column_type_sql`.
+- **`update_from_file` no longer shares log state between calls.** Its
+  `logging` default was a dictionary literal that the method wrote `logid` and
+  `aborted` into, so consecutive default calls saw each other's values and a
+  caller-supplied dictionary came back modified. The default is now `None` and
+  the mapping is copied per call.
+- **Random selection edge cases.** `random(query, pick_first=...)` returned
+  `None` rather than raising `IndexError` when nothing satisfies the query; a
+  projected value of `0`, `False`, `""` or `[]` counts as a result instead of
+  being skipped until `maxtries` ran out; `random_sample` raises `ValueError`
+  naming the accepted modes instead of silently returning `None` for an
+  unknown one; and a repeatable `choice` sample uses a local
+  `random.Random(seed)` rather than reseeding the process-wide generator.
+
+- **`stats_valid` is enforced, not just recorded.** Write paths cleared the
+  flag but read paths ignored it, so a count cached before a `restat=False`
+  write kept being served afterwards -- verified: a query counted at 67, then
+  every matching row changed, still answered 67. Every lookup that would serve
+  a cached answer now goes through one predicate and reports a miss while the
+  flag is false: `quick_count`, `quick_count_distinct` and `_quick_statistic`,
+  which is what makes `count`, `max`, `min` and `sum` compute the answer
+  instead of returning a recorded one. The line is whether a miss costs one
+  bounded query or a rebuild, so these are deliberately not gated: the
+  empty-query `total`, maintained on every write and so exact; the `_status` /
+  `status` / `extra_counts` inventory, which is how `refresh_stats` discovers
+  what to recompute; `_has_stats` / `_has_numstats`, which decide whether a
+  whole statistics family needs computing; and `null_counts`, whose fallback
+  is one full count *per search column*. Gating that last group made
+  `column_counts`, `numstats` and `null_counts` rebuild on every call with
+  nothing to converge on, since only `refresh_stats` restores the flag --
+  measured on the LMFDB, four minutes of downstream suite became over
+  forty-five. **The gap that leaves:** `column_counts`, `numstats` and
+  `null_counts` can still report a value recorded before an unrefreshed
+  write.
+  Closing it needs freshness per statistic rather than one flag per table,
+  which is a metadata format change; `refresh_stats()` is the remedy
+  meanwhile. A suffixed (`_tmp`, `_oldN`) table is not gated by the live
+  table's flag, since it carries its own caches. The flag is restored only by
+  `refresh_stats()`, inside the transaction that rebuilt the caches, so a
+  failed refresh leaves the table invalid; refreshing a `_tmp` copy does not
+  validate the live table.
+- **Bulk paths run `ANALYZE`.** A relation that has just been bulk loaded has
+  no planner statistics until autovacuum reaches it, so queries against it are
+  costed as though it were tiny. Replacement tables are analyzed while still
+  named `_tmp` -- before the swap, and outside its transaction, since the
+  catalog entry follows the relation through the rename -- which covers
+  `reload`, `rewrite`, non-inplace `update_from_file` and staged commits
+  through the one helper they share; `copy_from` analyzes the live table it
+  loaded into.
+
+- **A database operates in exactly one schema.** `PostgresDatabase` takes a
+  `schema=` argument (default `"public"`, so nothing changes for existing
+  deployments), validates it as an identifier, and pins `search_path` to it on
+  the first connection and on every replacement. Catalog inspection was
+  previously a mixture of hard-coded `'public'` and no filter at all -- 30-odd
+  queries across `pg_tables`, `pg_indexes`, `pg_class`, `pg_constraint` and
+  `information_schema` -- so with two schemas holding a relation of the same
+  name, column discovery could mix their columns, an index or constraint in
+  the other schema counted as present, `_all_tablenames` listed the name
+  twice, and a lock on the other schema's copy refused a write to this one.
+  Every one is now filtered to the selected schema, which is bound as a value
+  rather than interpolated. The schema goes into `search_path` as one quoted
+  identifier rather than as text (a `search_path` is a comma-separated list,
+  in which a name needing quotes is parsed rather than used), `pg_temp` is
+  named after it so a temporary relation cannot shadow a real one, and
+  `current_schema()` is checked afterwards, since PostgreSQL silently ignores
+  an entry naming a schema that is missing or not accessible. *Migration:*
+  none unless you were relying on psycodict seeing relations outside `public`,
+  which it did only by accident. The schema must exist and the role needs
+  `USAGE` on it; the name `$user` is refused, PostgreSQL substituting the
+  role's name for it even when quoted.
+
 ### Release candidates
 
 1.0.0 is published as a sequence of release candidates first. `pip` ignores

DataManagement.md

@@ -64,6 +64,29 @@ Copies the column layout, `label_col`, `sort`, `id_ordered`, `id` type and descr
 
 psycodict keeps its own bookkeeping in a handful of tables that live alongside your search tables:
 
+psycodict operates in one PostgreSQL schema, `public` unless the constructor is
+given another (`PostgresDatabase(schema="myschema")`).  Every relation it
+creates goes there, every relation it looks for is looked for there, and every
+catalog query is filtered to it, so a relation of the same name in another
+schema is neither mistaken for one of these nor merged with it.  The schema is
+pinned in each connection's `search_path`, including replacement connections
+after a reconnect, as one quoted identifier (a `search_path` is a
+comma-separated *list*, so a name that needs quoting must not be written to it
+as text).  psycodict names `pg_temp` explicitly *after* it, so that a temporary
+relation cannot shadow `meta_tables` or a search table, and leaves `pg_catalog`
+unnamed, which is what keeps it ahead of both so the built-in types and
+functions stay reachable.  The effective order is therefore `pg_catalog`, the
+selected schema, `pg_temp`.
+
+The schema must already exist: psycodict never creates one, and `create=True`
+bootstraps the meta tables *within* it.  The connecting role needs `USAGE` on
+it, and `CREATE` to make any relation there.  PostgreSQL silently ignores a
+`search_path` entry naming a schema that is missing or not accessible, so
+psycodict checks `current_schema()` after setting it and refuses to connect
+rather than operate somewhere nobody chose.  One name cannot be selected at all:
+`$user`, which PostgreSQL substitutes with the connecting role's name even when
+quoted, is rejected by the constructor.
+
  * **`meta_tables`** — one row per search table, holding `name`, `sort`, `count_cutoff`, `id_ordered`, `out_of_order`, `stats_valid`, `label_col`, `total`, `important` and `include_nones`.  This is the source of truth psycodict reads on connection to reconstruct each table object.
  * **`meta_indexes`** and **`meta_constraints`** — one row per index / constraint, recording how to rebuild it.  `reload` and `restore_indexes` rebuild from these rows, **not** from whatever is physically on the table (see [reload](#reload)).
  * **`meta_tables_hist`**, **`meta_indexes_hist`**, **`meta_constraints_hist`** — versioned history of the three tables above, so that `reload_meta`/`revert_meta` can roll a table's metadata forward and back.
@@ -108,7 +131,32 @@ These mutate the live table directly.  They are convenient for small edits; for
  * **`update(query, changes, resort=False, restat=True)`** — a plain SQL `UPDATE` of every row matching `query`; `changes` maps column names to constants.
  * **`delete(query, restat=True)`** — deletes every row matching `query` and decrements `total`.
 
-**Statistics invalidation.**  Any write that can change the data calls `_break_stats`, which sets `meta_tables.stats_valid = false` so that cached statistics are known to be stale.  If the table has `saving` on and you left `restat=True`, statistics are refreshed at the end of the call; otherwise they are simply marked invalid.  Inserting rows (and updating a sort-key column) also calls `_break_order`, setting `out_of_order = true` to record that the `id` order no longer matches `sort`; `delete` leaves the order flag alone.
+**Statistics invalidation.**  Any write that can change the data calls `_break_stats`, which sets `meta_tables.stats_valid = false` so that cached statistics are known to be stale.  If the table has `saving` on and you left `restat=True`, statistics are refreshed at the end of the call; otherwise they are simply marked invalid.
+
+`stats_valid` is enforced rather than merely recorded: while it is false, a
+cached nonempty-query count, distinct count, minimum, maximum or sum reports a
+miss, and the method computes the answer instead of returning the stored one.
+The empty-query `total` is the exception, since it is maintained on every write
+and so stays exact.
+
+`column_counts`, `numstats` and `null_counts` are the other exception, and a
+caveat worth knowing.  The line is what a cache miss costs: the counts above
+fall back to a single statement about the rows in question, while these fall
+back to rebuilding a whole statistics family, or to one full count per search
+column.  Making them miss while the table is invalid would rebuild on every
+call and never converge, since only `refresh_stats()` restores the flag, so
+they read what is recorded.  A value recorded before an unrefreshed write is
+therefore still reported by them; run `refresh_stats()` after a write you did
+not `restat`.  The flag goes back to true only in `refresh_stats()`, inside
+the transaction that rebuilt the caches, so a refresh that fails part-way leaves
+the table marked invalid rather than claiming a cache it does not have.
+Refreshing a `_tmp` copy does not validate the live table.
+
+Bulk paths also run PostgreSQL's own `ANALYZE`, which is a different thing from
+psycodict's statistics: a freshly loaded relation has no planner statistics
+until autovacuum reaches it.  Replacement tables are analyzed while still named
+`_tmp`, before the swap, since the catalog entry follows the relation through
+the rename; `copy_from` analyzes the live table it loaded into.  Inserting rows (and updating a sort-key column) also calls `_break_order`, setting `out_of_order = true` to record that the `id` order no longer matches `sort`; `delete` leaves the order flag alone.
 
 ### Resorting is disabled
 
@@ -212,7 +260,7 @@ Every search table has two companion tables:
 
 ## Operational tips
 
- * **`db.show_locks()`** prints every lock currently held on any table — name, mode, pid and age.  This is how you find the process id to kill when a write raised `LockError`.
+ * **`db.show_locks()`** prints every lock currently held on a table in this database's schema — name, mode, pid and age.  This is how you find the process id to kill when a write raised `LockError`.  A lock on a same-named table in another schema is not shown, and does not block anything done here.
  * **`table.analyze(query, projection=1, limit=1000, sort=None, explain_only=False)`** prints the SQL it would run and its `EXPLAIN ANALYZE` plan (or just `EXPLAIN`, with `explain_only=True`).  Use it when tuning indexes for a slow search.
  * **Slow-query logging** is configured in `config.ini`: `[logging] slowcutoff` (seconds, default `0.1`) and `slowlogfile` (default `slow_queries.log`).  Any statement slower than `slowcutoff` is logged with its interpolated SQL; slow searches additionally log a `Replicate with db.table.analyze(...)` hint so you can reproduce them.
  * On **PostgreSQL 18+** a `reload` or `update_from_file` prints a harmless warning like `Constraint of ... with name ..._id_not_null does not end with the suffix _tmp` during the swap — Postgres now catalogs `NOT NULL` as a named constraint that the swap logic does not recognize.  The constraint is preserved; the warning can be ignored.

psycodict/base.py

@@ -423,22 +423,47 @@ def _table_exists(self, tablename):
 
         - ``tablename`` -- a string, the name of the table
         """
-        cur = self._execute(SQL("SELECT 1 FROM pg_tables where tablename=%s"), [tablename], silent=True)
+        cur = self._execute(
+            SQL("SELECT 1 FROM pg_tables WHERE schemaname = %s AND tablename = %s"),
+            [self._db.schema, tablename],
+            silent=True,
+        )
         return cur.fetchone() is not None
 
     def _all_tablenames(self):
         """
         Return all (postgres) table names in the database
         """
-        return [rec[0] for rec in self._execute(SQL("SELECT tablename FROM pg_tables ORDER BY tablename"), silent=True)]
+        return [
+            rec[0]
+            for rec in self._execute(
+                SQL("SELECT tablename FROM pg_tables WHERE schemaname = %s ORDER BY tablename"),
+                [self._db.schema],
+                silent=True,
+            )
+        ]
 
     def _get_locks(self):
-        return self._execute(SQL(
-            "SELECT t.relname, l.mode, l.pid, age(clock_timestamp(), a.backend_start) "
-            "FROM pg_locks l "
-            "JOIN pg_stat_all_tables t ON l.relation = t.relid JOIN pg_stat_activity a ON l.pid = a.pid "
-            "WHERE l.granted AND t.schemaname <> 'pg_toast'::name AND t.schemaname <> 'pg_catalog'::name"
-        ))
+        """
+        The granted locks on relations in this database's schema.
+
+        Filtered to that schema in SQL, because the caller has only the
+        relation name to go on: a lock on ``other_schema.same_name`` would
+        otherwise be reported as a lock on this schema's ``same_name``, and
+        ``_check_locks`` would refuse a write that nothing is blocking.  Another
+        session's temporary table of the same name does the same thing.
+        Equality with the schema also subsumes the old exclusions of
+        ``pg_toast`` and ``pg_catalog``.
+        """
+        return self._execute(
+            SQL(
+                "SELECT t.relname, l.mode, l.pid, age(clock_timestamp(), a.backend_start) "
+                "FROM pg_locks l "
+                "JOIN pg_stat_all_tables t ON l.relation = t.relid JOIN pg_stat_activity a ON l.pid = a.pid "
+                "WHERE l.granted AND t.schemaname = %s"
+            ),
+            [self._db.schema],
+        )
 
     def _table_locked(self, tablename, types="all"):
         """
@@ -535,15 +560,18 @@ def _index_exists(self, indexname, tablename=None):
         """
         if tablename:
             cur = self._execute(
-                SQL("SELECT 1 FROM pg_indexes WHERE indexname = %s AND tablename = %s"),
-                [indexname, tablename],
+                SQL(
+                    "SELECT 1 FROM pg_indexes "
+                    "WHERE schemaname = %s AND indexname = %s AND tablename = %s"
+                ),
+                [self._db.schema, indexname, tablename],
                 silent=True,
             )
             return cur.fetchone() is not None
         else:
             cur = self._execute(
-                SQL("SELECT tablename FROM pg_indexes WHERE indexname=%s"),
-                [indexname],
+                SQL("SELECT tablename FROM pg_indexes WHERE schemaname = %s AND indexname = %s"),
+                [self._db.schema, indexname],
                 silent=True,
             )
             table = cur.fetchone()
@@ -560,7 +588,13 @@ def _relation_exists(self, name):
 
         - ``name`` -- a string, the name of the relation
         """
-        cur = self._execute(SQL("SELECT 1 FROM pg_class where relname = %s"), [name])
+        cur = self._execute(
+            SQL(
+                "SELECT 1 FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace "
+                "WHERE n.nspname = %s AND c.relname = %s"
+            ),
+            [self._db.schema, name],
+        )
         return cur.fetchone() is not None
 
     def _constraint_exists(self, constraintname, tablename=None):
@@ -582,19 +616,19 @@ def _constraint_exists(self, constraintname, tablename=None):
             cur = self._execute(
                 SQL(
                     "SELECT 1 from information_schema.table_constraints "
-                    "WHERE table_name=%s and constraint_name=%s"
+                    "WHERE table_schema = %s AND table_name = %s AND constraint_name = %s"
                 ),
-                [tablename, constraintname],
+                [self._db.schema, tablename, constraintname],
                 silent=True,
             )
             return cur.fetchone() is not None
         else:
             cur = self._execute(
                 SQL(
                     "SELECT table_name from information_schema.table_constraints "
-                    "WHERE constraint_name=%s"
+                    "WHERE table_schema = %s AND constraint_name = %s"
                 ),
-                [constraintname],
+                [self._db.schema, constraintname],
                 silent=True,
             )
             table = cur.fetchone()
@@ -608,8 +642,8 @@ def _list_indexes(self, tablename):
         Lists built index names on the search table ``tablename``
         """
         cur = self._execute(
-            SQL("SELECT indexname FROM pg_indexes WHERE tablename = %s"),
-            [tablename],
+            SQL("SELECT indexname FROM pg_indexes WHERE schemaname = %s AND tablename = %s"),
+            [self._db.schema, tablename],
             silent=True,
         )
         return [elt[0] for elt in cur]
@@ -629,9 +663,9 @@ def _list_constraints(self, tablename):
                 "           ON rel.oid = con.conrelid "
                 "INNER JOIN pg_catalog.pg_namespace nsp "
                 "           ON nsp.oid = connamespace "
-                "WHERE rel.relname = %s"
+                "WHERE nsp.nspname = %s AND rel.relname = %s"
             ),
-            [tablename],
+            [self._db.schema, tablename],
             silent=True,
         )
         return [elt[0] for elt in cur]
@@ -809,9 +843,9 @@ def _column_types(self, table_name, data_types=None):
                 cur = self._execute(
                     SQL(
                         "SELECT column_name, udt_name::regtype FROM information_schema.columns "
-                        "WHERE table_name = %s ORDER BY ordinal_position"
+                        "WHERE table_schema = %s AND table_name = %s ORDER BY ordinal_position"
                     ),
-                    [tname],
+                    [self._db.schema, tname],
                 )
             else:
                 cur = data_types[tname]
@@ -838,8 +872,11 @@ def _relation_columns(self, table):
         one about columns.
         """
         cur = self._execute(
-            SQL("SELECT column_name FROM information_schema.columns WHERE table_name = %s"),
-            [table],
+            SQL(
+                "SELECT column_name FROM information_schema.columns "
+                "WHERE table_schema = %s AND table_name = %s"
+            ),
+            [self._db.schema, table],
             silent=True,
             commit=False,
         )