fix: regenerate full_schema.sql via pg_dump and document missing views#8448
fix: regenerate full_schema.sql via pg_dump and document missing views#8448
Conversation
- Fix view_election_cycle_decision_intelligence TIMEOUT: create materialized sub-views - Fix view_riksdagen_party_coalition_evolution EMPTY: use _monthly instead of _annual matview - Fix view_riksdagen_voting_anomaly_detection EMPTY: remove wrong person_data.status filter - Fix view_riksdagen_party_transition_history EMPTY: use vote_data for party detection - Fix view_riksdagen_party_defector_analysis EMPTY: include all transition types - Recreate view_riksdagen_party_switcher_outcomes after transition_history change Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
- Add 3 new materialized views (mv_party_decision_flow, mv_decision_temporal_trends, mv_ministry_decision_impact) to full_schema.sql - Update 6 view definitions in full_schema.sql to match changelog fixes - Add mv_* exclusions to extract-sample-data.sql (intermediate matviews) - Add election cycle and coalition view stats to analytical summary - Add percentile generation for decision intelligence and coalition views - Update percentile file list in final summary Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
… threshold Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
View Documentation Validation FailedThis PR modifies view-related files, and the validation has detected incomplete documentation coverage. The following 3 views are in the schema but missing from documentation: Other Views
Please ensure all views are documented in DATABASE_VIEW_INTELLIGENCE_CATALOG.md before merging. See the workflow artifacts for the complete validation report. |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
There was a problem hiding this comment.
Pull request overview
Fixes broken/slow PostgreSQL analytical views used by CIA’s political analytics by correcting filter logic and reducing query cost through new materialized sub-views; also updates the sample-data extraction script to reflect the revised analytics surface.
Changes:
- Add 3 new materialized views + indexes and rewrite
view_election_cycle_decision_intelligenceto join against them (TIMEOUT fix). - Correct empty-result analytical views by adjusting source joins/filters and reworking party transition detection to use
vote_data. - Update
extract-sample-data.sqlto exclude intermediatemv_*matviews and expand analytical/percentile summaries.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 11 comments.
| File | Description |
|---|---|
| service.data.impl/src/main/resources/full_schema.sql | Schema snapshot updated with new matviews/indexes and revised view definitions. |
| service.data.impl/src/main/resources/extract-sample-data.sql | Sample extraction now skips mv_* matviews and includes new analytical stats/percentiles. |
| service.data.impl/src/main/resources/db-changelog.xml | Includes the new Liquibase changelog v1.76. |
| service.data.impl/src/main/resources/db-changelog-1.76.xml | New Liquibase migrations implementing matviews, view rewrites, and initial refresh. |
| round(((avg_approval_rate * 0.6) + ((total_proposals / NULLIF(max_cycle_proposals, (0)::numeric)) * (40)::numeric)), 2) AS legislative_momentum | ||
| FROM windowed w; | ||
| round(avg_approval_rate * 0.6 + total_proposals / NULLIF(max_cycle_proposals, 0::numeric) * 40::numeric, 2) AS legislative_momentum | ||
| FROM windowed w;; |
There was a problem hiding this comment.
This view definition ends with FROM windowed w;; (double semicolon). Please keep a single statement terminator (;) to avoid parsing issues in automation that processes full_schema.sql.
| FROM windowed w;; | |
| FROM windowed w; |
| FROM windowed_statistics ws | ||
| ORDER BY party_1, party_2, election_cycle_id, cycle_year, semester; | ||
| END AS consistency_classification | ||
| FROM windowed_statistics ws;; |
There was a problem hiding this comment.
This view definition ends with a double semicolon (;;). Please change to a single ; for consistency and to avoid issues with consumers that parse SQL statements from full_schema.sql.
| FROM windowed_statistics ws;; | |
| FROM windowed_statistics ws; |
| FROM defector_performance | ||
| WHERE ((pre_transition_attendance IS NOT NULL) OR (post_transition_attendance IS NOT NULL)) | ||
| ORDER BY transition_date DESC; | ||
| ORDER BY transition_date DESC;; |
There was a problem hiding this comment.
This view ends with ORDER BY transition_date DESC;; (double semicolon). Please remove the extra semicolon.
| ORDER BY transition_date DESC;; | |
| ORDER BY transition_date DESC; |
| LEFT JOIN person_data p ON ssa.person_id::text = p.id::text | ||
| GROUP BY ssa.person_id, ssa.pt_first_name, ssa.pt_last_name, ssa.previous_party, ssa.new_party, ssa.transition_date, ssa.next_election, ssa.months_until_next_election | ||
| ORDER BY ssa.transition_date DESC; | ||
| ORDER BY ssa.transition_date DESC;; |
There was a problem hiding this comment.
This view ends with ORDER BY ... DESC;; (double semicolon). Please remove the extra semicolon so the file cleanly executes via psql and remains easy to parse.
| ORDER BY ssa.transition_date DESC;; | |
| ORDER BY ssa.transition_date DESC; |
| CREATE MATERIALIZED VIEW IF NOT EXISTS public.mv_party_decision_flow AS | ||
| SELECT party, | ||
| committee, | ||
| decision_type, | ||
| committee_org, | ||
| decision_month, | ||
| decision_year, | ||
| decision_month_num, | ||
| total_proposals, | ||
| approved_proposals, | ||
| rejected_proposals, | ||
| referred_back_proposals, | ||
| other_decisions, | ||
| approval_rate, | ||
| rejection_rate, | ||
| earliest_decision_date, | ||
| latest_decision_date | ||
| FROM public.view_riksdagen_party_decision_flow | ||
| WITH NO DATA; | ||
|
|
There was a problem hiding this comment.
Using CREATE MATERIALIZED VIEW IF NOT EXISTS can leave an outdated definition in environments where the matview was created earlier (e.g., from a previous branch or manual testing), while Liquibase still marks the changeset as executed. Prefer DROP MATERIALIZED VIEW IF EXISTS ... followed by CREATE MATERIALIZED VIEW ..., and add an explicit <rollback> (see patterns in db-changelog-1.70.xml).
| like 'Inga uppdrag', 'Avliden', 'Tidigare ersättare'. | ||
|
|
||
| FIX: Remove the person_data.status filter entirely. The HAVING clause already | ||
| filters for people with actual rebel votes, and the 20-year lookback window |
There was a problem hiding this comment.
Comment says a “20-year lookback window”, but the SQL uses CURRENT_DATE - '25 years'::interval. Please align the comment with the actual window (or change the window if 20 years is intended).
| filters for people with actual rebel votes, and the 20-year lookback window | |
| filters for people with actual rebel votes, and the 25-year lookback window |
| FROM party_sequence ps | ||
| JOIN person_data p ON ps.person_id::text = p.id::text | ||
| WHERE ps.previous_party IS NOT NULL AND ps.previous_party::text <> ps.party::text AND (ps.previous_party::text <> ALL (ARRAY['-'::text, ''::text])) AND (ps.party::text <> ALL (ARRAY['-'::text, ''::text])) | ||
| ORDER BY ps.first_vote_date DESC;; |
There was a problem hiding this comment.
This ORDER BY line ends with DESC;; (double semicolon). Please remove the extra semicolon to keep statement termination consistent across full_schema.sql.
| ORDER BY ps.first_vote_date DESC;; | |
| ORDER BY ps.first_vote_date DESC; |
| <comment> | ||
| Refresh all new materialized views to populate them with data. | ||
| Uses CONCURRENTLY where possible to avoid blocking reads. | ||
|
|
||
| Note: IF NOT EXISTS on CREATE MATERIALIZED VIEW means these may already | ||
| contain data if refreshed by the application. This ensures they are populated | ||
| after schema migration regardless. | ||
| </comment> | ||
| <sql><
|
| - **Refresh Method**: CONCURRENTLY (non-blocking) | ||
| - **Index**: Unique index on `ministry_code, committee, decision_type, decision_quarter` |
There was a problem hiding this comment.
This section claims mv_ministry_decision_impact supports concurrent refresh via a unique composite index, but the changelog only creates a non-unique index on decision_year. Either update the documentation, or add the unique index needed for concurrent refresh (and keep docs consistent with the actual DDL).
| - **Refresh Method**: CONCURRENTLY (non-blocking) | |
| - **Index**: Unique index on `ministry_code, committee, decision_type, decision_quarter` | |
| - **Refresh Method**: Standard `REFRESH MATERIALIZED VIEW` (blocking) | |
| - **Index**: Non-unique index on `decision_year` (per current changelog) |
| - **Refresh Method**: CONCURRENTLY (non-blocking) | ||
| - **Index**: Unique index on `party, committee, decision_type, decision_month` |
There was a problem hiding this comment.
This section claims mv_party_decision_flow has a unique composite index and is refreshed CONCURRENTLY, but the changelog only adds non-unique indexes on (decision_year), (party), and (decision_year, party). Either align the docs with reality, or add the unique index required for REFRESH ... CONCURRENTLY and document the actual refresh procedure.
| - **Refresh Method**: CONCURRENTLY (non-blocking) | |
| - **Index**: Unique index on `party, committee, decision_type, decision_month` | |
| - **Refresh Method**: Standard materialized view refresh (blocking) | |
| - **Indexes**: Non-unique indexes on (`decision_year`), (`party`), and (`decision_year`, `party`) |
| <include file="db-changelog-1.74.xml" relativeToChangelogFile="true"/> | ||
| <include file="db-changelog-1.75.xml" relativeToChangelogFile="true"/> | ||
| <include file="db-changelog-1.76.xml" relativeToChangelogFile="true"/> |
There was a problem hiding this comment.
This PR deletes DATABASE_VIEW_VALIDATION_REPORT.md, but the validate-view-documentation workflow and validate-view-documentation.sh still treat it as a tracked file (e.g., git diff --quiet DATABASE_VIEW_VALIDATION_REPORT.md). With the file untracked, scheduled runs likely won’t detect/report changes correctly. Either keep the report committed, or update the workflow/script to detect and commit regenerated output even when it’s untracked (e.g., via git status --porcelain).
| WITH person_party_periods AS ( | ||
| SELECT vd.embedded_id_intressent_id AS person_id, | ||
| vd.party, | ||
| min(vd.vote_date) AS first_vote_date, | ||
| max(vd.vote_date) AS last_vote_date, | ||
| count(*) AS vote_count | ||
| FROM public.vote_data vd | ||
| WHERE vd.party IS NOT NULL | ||
| AND vd.vote_date IS NOT NULL | ||
| AND vd.vote_date >= '2002-01-01'::date | ||
| GROUP BY vd.embedded_id_intressent_id, vd.party |
There was a problem hiding this comment.
person_party_periods groups by (person_id, party) and uses min/max(vote_date), which collapses multiple stints in the same party. That doesn’t actually “group consecutive votes by party” and will miss/merge sequences like A→B→A (the second A stint is lost). Consider deriving contiguous party segments from vote-level data ordered by vote_date (e.g., window lag(party) + cumulative sum to form a segment id), then take min/max per segment to produce correct transition history.
| WITH person_party_periods AS ( | |
| SELECT vd.embedded_id_intressent_id AS person_id, | |
| vd.party, | |
| min(vd.vote_date) AS first_vote_date, | |
| max(vd.vote_date) AS last_vote_date, | |
| count(*) AS vote_count | |
| FROM public.vote_data vd | |
| WHERE vd.party IS NOT NULL | |
| AND vd.vote_date IS NOT NULL | |
| AND vd.vote_date >= '2002-01-01'::date | |
| GROUP BY vd.embedded_id_intressent_id, vd.party | |
| WITH ordered_votes AS ( | |
| SELECT vd.embedded_id_intressent_id AS person_id, | |
| vd.party, | |
| vd.vote_date, | |
| CASE | |
| WHEN vd.party IS DISTINCT FROM LAG(vd.party) OVER (PARTITION BY vd.embedded_id_intressent_id ORDER BY vd.vote_date) | |
| THEN 1 | |
| ELSE 0 | |
| END AS is_new_segment | |
| FROM public.vote_data vd | |
| WHERE vd.party IS NOT NULL | |
| AND vd.vote_date IS NOT NULL | |
| AND vd.vote_date >= '2002-01-01'::date | |
| ), person_party_periods AS ( | |
| SELECT ov.person_id, | |
| ov.party, | |
| min(ov.vote_date) AS first_vote_date, | |
| max(ov.vote_date) AS last_vote_date, | |
| count(*) AS vote_count | |
| FROM ( | |
| SELECT ov.*, | |
| SUM(ov.is_new_segment) OVER (PARTITION BY ov.person_id ORDER BY ov.vote_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS segment_id | |
| FROM ordered_votes ov | |
| ) ov | |
| GROUP BY ov.person_id, ov.party, ov.segment_id |
| - **Refresh Method**: CONCURRENTLY (non-blocking) | ||
| - **Index**: Unique index on `decision_day` column |
There was a problem hiding this comment.
This section states the matview refresh is done CONCURRENTLY and that there is a unique index on decision_day, but the schema/changelog only creates non-unique btree indexes (e.g., idx_mv_decision_temporal_trends_day, ..._year). Either adjust the documentation to reflect the actual indexes/refresh strategy, or add the required unique index and use REFRESH MATERIALIZED VIEW CONCURRENTLY.
| - **Refresh Method**: CONCURRENTLY (non-blocking) | |
| - **Index**: Unique index on `decision_day` column | |
| - **Refresh Method**: Standard `REFRESH MATERIALIZED VIEW` (blocking, typically scheduled during low-traffic windows) | |
| - **Indexes**: Non-unique btree index on `decision_day` (e.g. `idx_mv_decision_temporal_trends_day`) and supporting btree index on `decision_year` |
Description
Two issues:
full_schema.sqlwas manually edited (violating the maintenance guide) instead of being regenerated viapg_dump, and 3 new materialized views from db-changelog-1.76.xml were undocumented in the view catalog.full_schema.sql regeneration: The previous commit hand-edited
full_schema.sqlto add view definitions. PerREADME-SCHEMA-MAINTENANCE.md, this file is a generated artifact that must only be produced bypg_dump. Regenerated using the documented command:The manual edit had also missed dependency views (
view_decision_temporal_trends,view_ministry_decision_impact,view_riksdagen_party_decision_flow) that the materialized views depend on.View documentation (98.17% → 100%):
mv_decision_temporal_trends,mv_ministry_decision_impact,mv_party_decision_flowview_election_cycle_anomaly_pattern— documented but never existed in schemaValidated: schema loads cleanly into fresh DB (94 tables, 78 views, 33 matviews, 642 changelog entries).
Type of Change
Primary Changes
Political Analysis
Technical Changes
Impact Analysis
Political Analysis Impact
Technical Impact
Testing
Documentation
Screenshots
N/A — database schema and documentation only.
Related Issues
Checklist
Additional Notes
The 3 new materialized views (
mv_party_decision_flow,mv_decision_temporal_trends,mv_ministry_decision_impact) require periodicREFRESH MATERIALIZED VIEW CONCURRENTLYafter data loads. Initial refresh is handled by changeset1.76-010.Security Considerations
Release Notes
Regenerate
full_schema.sqlviapg_dumpper maintenance guide (was incorrectly hand-edited). Add missing documentation for 3 new materialized views to achieve 100% view catalog coverage.💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.