fix: production Liquibase migration failure — window function in GROUP BY, vote casing, rebel vote calculation

[Copilot]

Description

Production deploy failed at changeset 1.76-005 (ERROR: cannot drop columns from view) and 1.76-007 (ERROR: window functions are not allowed in GROUP BY). Multiple view SQL correctness issues identified during review.

Fixes applied to db-changelog-1.76.xml:

• 1.76-004/005/007/008: CREATE OR REPLACE VIEW → DROP VIEW IF EXISTS ... CASCADE + CREATE VIEW for views with column changes
• 1.76-005: Restored 10 missing coalition columns (forecast_trend, stability_score, etc.) required by JPA entity
• 1.76-006: UPPER() normalization for vote comparisons in view_riksdagen_voting_anomaly_detection
• 1.76-007: Fixed window functions are not allowed in GROUP BY — split distinct_transitions CTE into three-step islands-and-gaps pattern:

-- Before (illegal): window function directly in GROUP BY
GROUP BY nc.person_id, nc.previous_party, nc.new_party,
         nc.change_seq - ROW_NUMBER() OVER (PARTITION BY ...)

-- After: pre-compute in intermediate CTE, then GROUP BY the result
), grouped_changes AS (
    SELECT ...,
        fc.change_seq - ROW_NUMBER() OVER (...) AS grp
    FROM filtered_changes fc
), distinct_transitions AS (
    SELECT ..., min(gc.transition_date) AS transition_date
    FROM grouped_changes gc
    GROUP BY gc.person_id, gc.previous_party, gc.new_party, gc.grp

Also replaced GROUP BY (person_id, party) with LAG()-based change-point detection to preserve non-contiguous stints (S→M→S).

• 1.76-008: UPPER() normalization for 'Frånvarande' in view_riksdagen_party_defector_analysis
• 1.76-011 (new): Fixed view_politician_risk_summary — rebel_votes compared vote values (JA/NEJ) against party codes (S/M), inflating rebel rates to ~100%. Now computes party consensus via mode() WITHIN GROUP (ORDER BY UPPER(vd.vote)), normalizing case before aggregation.

full_schema.sql: Restored to pre-1.76 state so CI tests the actual migration path.

DATABASE_VIEW_INTELLIGENCE_CATALOG.md: Added missing view_election_cycle_anomaly_pattern.

Validated: mvn liquibase:validate + mvn liquibase:update — all 11 changesets apply cleanly against PostgreSQL 16.

Type of Change

Primary Changes

• 🐛 Bug Fix

Political Analysis

• 📊 Political Data Analysis
  • Party Analysis
  • Riksdagen Integration
• 📈 Analytics & Metrics
  • Risk Assessment

Technical Changes

• 🏗️ Infrastructure
  • Database Changes
  • Performance Optimization
• 📝 Documentation
  • Technical Documentation

Impact Analysis

Political Analysis Impact

• Impact on data quality: Restores 10 coalition columns. Fixes rebel rates from ~100% to actual deviation rates. Normalizes vote casing across all modified views.
• Impact on analysis accuracy: Party transitions correctly track non-contiguous stints (S→M→S). mode() consensus computed on normalized values prevents mixed-case splitting.
• Impact on transparency features: None

Technical Impact

• Performance impact: O(n²) correlated subqueries → window functions + LATERAL JOIN in transition view
• Security implications: None
• Dependency changes: None

Testing

• Integration tests added/updated
• Security compliance verified

Validated: pre-1.76 schema → Liquibase applies all 11 changesets → all views created with correct columns. mvn liquibase:validate and mvn liquibase:update pass.

Documentation

• Political analysis documentation updated

Screenshots

N/A

Related Issues

Checklist

• Code follows project coding standards
• Comments are clear and helpful
• Documentation is updated
• Tests are passing
• Security compliance is maintained
• Performance impact is acceptable
• Breaking changes are documented
• Changes are backward compatible

Additional Notes

Changeset	View	Issue	Fix
1.76-004	view_election_cycle_decision_intelligence	Column structure change	DROP CASCADE + CREATE
1.76-005	view_riksdagen_party_coalition_evolution	10 columns removed	DROP CASCADE + CREATE + restore columns
1.76-006	view_riksdagen_voting_anomaly_detection	Title-case vote literals	UPPER() normalization
1.76-007	view_riksdagen_party_transition_history	Window function in GROUP BY; merged non-contiguous stints	Split into 3 CTEs; LAG()-based change-point detection
1.76-008	view_riksdagen_party_defector_analysis	Case-sensitive 'Frånvarande'	UPPER() normalization
1.76-011	view_politician_risk_summary	rebel_votes compared vote to party code; mode() on mixed-case	Party consensus via mode() over UPPER(vd.vote)

Pre-existing test failures (ViewDecisionTemporalTrendsDAOITest, ViewRiksdagenPartyDecisionFlowDAOITest) confirmed on main — unrelated JPA column mismatches.

Security Considerations

• No sensitive data exposed
• Security best practices followed
• Compliance requirements met

Release Notes

Fix production Liquibase migration failures (cannot drop columns from view, window functions are not allowed in GROUP BY). Uses DROP VIEW CASCADE + CREATE VIEW for column changes, restores 10 missing coalition columns, fixes rebel_votes calculation (was comparing votes against party codes instead of party consensus via mode()), normalizes vote casing with UPPER() pre-aggregation, and corrects party transition history to handle non-contiguous stints via LAG()-based change-point detection.

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[github-actions]

View Documentation Validation Failed

This PR modifies view-related files, and the validation has detected incomplete documentation coverage.

The following 1 views are in the schema but missing from documentation:

Other Views

• view_election_cycle_anomaly_pattern

Please ensure all views are documented in DATABASE_VIEW_INTELLIGENCE_CATALOG.md before merging.

See the workflow artifacts for the complete validation report.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 97e04e9.

Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Files

None

[Copilot]

Pull request overview

This PR updates the PostgreSQL schema artifacts and documentation around the “intelligence” view layer, including a regenerated full_schema.sql dump and adjustments in Liquibase changelog 1.76 for view recreation semantics.

Changes:

• Regenerates full_schema.sql, reordering/relocating multiple view definitions (including intelligence/risk-related views).
• Updates db-changelog-1.76.xml to drop/recreate specific views (using DROP VIEW ... CASCADE) rather than relying on CREATE OR REPLACE.
• Updates DATABASE_VIEW_INTELLIGENCE_CATALOG.md to reflect expanded framework/view coverage (incl. election cycle / pattern recognition content).

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 5 comments.

File	Description
service.data.impl/src/main/resources/full_schema.sql	Regenerated pg_dump baseline; view definitions moved/reordered, including politician risk/anomaly views.
service.data.impl/src/main/resources/db-changelog-1.76.xml	Adjusts Liquibase DDL strategy for several views (drop + recreate with CASCADE).
DATABASE_VIEW_INTELLIGENCE_CATALOG.md	Documentation updates for intelligence framework/view inventory and operational guidance.

[Copilot]

In politician_vote_metrics, rebel_votes is computed as vd.vote <> vd.party, but vd.party is the politician’s party code (e.g., S/M) while vd.vote is a vote choice (Ja/Nej/Avstår/Frånvarande). This condition will be true for almost every non-absent vote, inflating rebel rates and the overall risk score. Compute rebel votes by comparing the member’s vote to the party’s majority/consensus vote for the same ballot (similar to view_riksdagen_voting_anomaly_detection), rather than comparing to the party code.

Suggested change

	WITH politician_vote_metrics AS (
	SELECT p.id AS person_id,
	count(DISTINCT vd.embedded_id_ballot_id) AS total_votes,
	count(DISTINCT vd.embedded_id_ballot_id) FILTER (WHERE ((vd.vote)::text = 'Frånvarande'::text)) AS absent_votes,
	count(DISTINCT vd.embedded_id_ballot_id) FILTER (WHERE (((vd.vote)::text <> (vd.party)::text) AND ((vd.vote)::text <> 'Frånvarande'::text))) AS rebel_votes
	FROM (public.person_data p
	LEFT JOIN public.vote_data vd ON ((((vd.embedded_id_intressent_id)::text = (p.id)::text) AND (vd.vote_date >= (CURRENT_DATE - '2 years'::interval)))))
	WITH party_ballot_party_vote AS (
	SELECT vd.embedded_id_ballot_id,
	vd.party,
	mode() WITHIN GROUP (ORDER BY vd.vote) AS party_vote
	FROM public.vote_data vd
	WHERE ((vd.vote)::text <> 'Frånvarande'::text)
	GROUP BY vd.embedded_id_ballot_id, vd.party
	), politician_vote_metrics AS (
	SELECT p.id AS person_id,
	count(DISTINCT vd.embedded_id_ballot_id) AS total_votes,
	count(DISTINCT vd.embedded_id_ballot_id) FILTER (WHERE ((vd.vote)::text = 'Frånvarande'::text)) AS absent_votes,
	count(DISTINCT vd.embedded_id_ballot_id) FILTER (
	WHERE (
	(pbpv.party_vote IS NOT NULL)
	AND ((vd.vote)::text <> (pbpv.party_vote)::text)
	AND ((vd.vote)::text <> 'Frånvarande'::text)
	)
	) AS rebel_votes
	FROM (public.person_data p
	LEFT JOIN public.vote_data vd ON (
	((vd.embedded_id_intressent_id)::text = (p.id)::text)
	AND (vd.vote_date >= (CURRENT_DATE - '2 years'::interval))
	)
	LEFT JOIN party_ballot_party_vote pbpv ON (
	(pbpv.embedded_id_ballot_id = vd.embedded_id_ballot_id)
	AND ((pbpv.party)::text = (vd.party)::text)
	)
	)

[Copilot]

absent_votes is filtered with vd.vote = 'Frånvarande', but many existing views/changelogs treat vote_data.vote values as uppercase (e.g., FRÅNVARANDE) and sometimes normalize with UPPER(vote). If the stored values are uppercase, this will undercount absences and skew risk scores. Prefer a case-normalized comparison (e.g., UPPER(vd.vote) = 'FRÅNVARANDE').

Suggested change

	count(DISTINCT vd.embedded_id_ballot_id) FILTER (WHERE ((vd.vote)::text = 'Frånvarande'::text)) AS absent_votes,
	count(DISTINCT vd.embedded_id_ballot_id) FILTER (WHERE (((vd.vote)::text <> (vd.party)::text) AND ((vd.vote)::text <> 'Frånvarande'::text))) AS rebel_votes
	count(DISTINCT vd.embedded_id_ballot_id) FILTER (WHERE (upper((vd.vote)::text) = 'FRÅNVARANDE'::text)) AS absent_votes,
	count(DISTINCT vd.embedded_id_ballot_id) FILTER (WHERE (((vd.vote)::text <> (vd.party)::text) AND (upper((vd.vote)::text) <> 'FRÅNVARANDE'::text))) AS rebel_votes

[Copilot]

This view filters vote_data.vote using title-case literals ('Ja','Nej','Avstår'), while other parts of the schema use uppercase vote values (JA/NEJ/AVSTÅR/FRÅNVARANDE) and/or UPPER(vote) normalization. If the underlying data is uppercase, these predicates will silently exclude all rows. Normalize the comparison (e.g., UPPER(vote_data.vote) IN ('JA','NEJ','AVSTÅR')).

[Copilot]

The new view_riksdagen_party_transition_history logic collapses party history by grouping on (person_id, party) (see person_party_periods), which will merge multiple non-contiguous stints in the same party (e.g., S→M→S) into one row and lose the real transition sequence. This can cause missed or misdated transitions. Consider segmenting consecutive party periods by ordering votes per person and grouping by change-points (e.g., via LAG(party) + running group id), then deriving periods from those segments.

[Copilot]

view_riksdagen_party_defector_analysis (re)created here uses case-sensitive literals for vote values later in the view (e.g., comparing to 'Frånvarande'). Elsewhere in the schema/changelogs, vote_data.vote is handled as uppercase (JA/NEJ/AVSTÅR/FRÅNVARANDE) and/or normalized with UPPER(vote). Without normalization this view can silently compute wrong attendance metrics or return empty results depending on stored casing. Normalize vote comparisons (e.g., UPPER(vd.vote)) consistently.

[pethers]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied all review comments in commits e37c68d and 6b5f28f:

1. 1.76-006 (view_riksdagen_voting_anomaly_detection): Normalized all vote comparisons with UPPER() — 'Ja'/'Nej'/'Avstår' → 'JA'/'NEJ'/'AVSTÅR'
2. 1.76-007 (view_riksdagen_party_transition_history): Replaced GROUP BY (person_id, party) with LAG-based change-point detection on individual votes, preserving non-contiguous stints (e.g., S→M→S). Used window functions + LATERAL JOIN instead of correlated subqueries.
3. 1.76-008 (view_riksdagen_party_defector_analysis): Normalized 'Frånvarande' → UPPER() comparison
4. New 1.76-011 (view_politician_risk_summary): Fixed rebel_votes to compare against party consensus vote (via mode() aggregate) instead of party code. Uses CREATE OR REPLACE VIEW to preserve dependent views.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

[Copilot]

In party_ballot_consensus, the consensus vote is computed with mode() WITHIN GROUP (ORDER BY vd.vote) and only uppercased after the mode is chosen. If vote values can be mixed-cased (e.g., 'Ja' vs 'JA'), the mode calculation will treat them as different values and can pick the wrong consensus. Normalize before computing the mode (e.g., compute the mode over UPPER(vd.vote) or otherwise aggregate on an uppercased value).

[pethers]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Fixed in 34378db — mode() now operates over UPPER((vd.vote)::text) instead of uppercasing after mode selection, so mixed-case values like 'Ja' and 'JA' are correctly treated as the same value during consensus calculation.

[pethers]

Fixed the window functions are not allowed in GROUP BY error in changeset 1.76-007 (commit 97e04e9).

Root cause: The distinct_transitions CTE used ROW_NUMBER() OVER (...) directly inside GROUP BY, which PostgreSQL prohibits.

Fix: Split into three CTEs using the standard islands-and-gaps pattern:

1. filtered_changes — applies WHERE filter on LAG values
2. grouped_changes — computes group identifier via ROW_NUMBER() as a SELECT column
3. distinct_transitions — GROUP BY the pre-computed grp column

Validated with Maven Liquibase plugin per the schema maintenance guide:

• mvn liquibase:validate -pl service.data.impl → no validation errors
• mvn liquibase:update -pl service.data.impl → all 11 changesets applied successfully against PostgreSQL 16

[sonarqubecloud]

Quality Gate passed

Issues
15 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
7.0% Duplication on New Code

See analysis details on SonarQube Cloud