Skip to content

Comments

feat(bigquery): add retry for intermittent GCS 401 errors#3903

Draft
jkaflik wants to merge 1 commit intomainfrom
kk/bq-gcs-token-retry
Draft

feat(bigquery): add retry for intermittent GCS 401 errors#3903
jkaflik wants to merge 1 commit intomainfrom
kk/bq-gcs-token-retry

Conversation

@jkaflik
Copy link
Contributor

@jkaflik jkaflik commented Feb 9, 2026

Add exponential backoff retry logic to handle intermittent HTTP 401
Unauthorized errors from GCS when fetching files via ClickHouse's
URL table function during BigQuery source ingestion.

The retry logic:

  • Detects GCS 401 errors (ClickHouse error code 86)
  • Invalidates cached token before retry
  • Uses exponential backoff (2s, 4s delays)
  • Retries up to 3 times

@jkaflik
feat(bigquery): add retry for intermittent GCS 401 errors
c91d389 
Add exponential backoff retry logic to handle intermittent HTTP 401
Unauthorized errors from GCS when fetching files via ClickHouse's
URL table function during BigQuery source ingestion.

The retry logic:
- Detects GCS 401 errors (ClickHouse error code 86)
- Invalidates cached token before retry
- Uses exponential backoff (2s, 4s delays)
- Retries up to 3 times
@codecov
Copy link

codecov bot commented Feb 9, 2026

❌ 4 Tests Failed:

Tests completed Failed Passed Skipped
1678 4 1674 163
View the top 3 failed test(s) by shortest run time 
github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg
Stack Traces | 0.01s run time 
=== RUN   TestApiPg
=== PAUSE TestApiPg
=== CONT  TestApiPg
--- FAIL: TestApiPg (0.01s)
2026/02/09 14:02:40 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/02/09 14:02:40 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/02/09 14:02:40 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/02/09 14:02:40 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}

github.com/PeerDB-io/peerdb/flow/e2e::TestBigQueryClickhouseSuite
Stack Traces | 0.01s run time 
=== RUN   TestBigQueryClickhouseSuite
=== PAUSE TestBigQueryClickhouseSuite
=== CONT  TestBigQueryClickhouseSuite
--- FAIL: TestBigQueryClickhouseSuite (0.01s)
2026/02/09 14:00:24 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 14:00:24 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 14:00:24 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_11894967313103561732 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" args=[]
2026/02/09 14:00:24 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11894967313103561732
2026/02/09 14:00:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11894967313103561732 records=8 bytes=168 channelLen=7
2026/02/09 14:00:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11894967313103561732
2026/02/09 14:00:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11894967313103561732 records=0 bytes=0 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 14:00:24 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=0
2026/02/09 14:00:24 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 14:00:24 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 14:00:24 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_7731802835375335223 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" args=[]
2026/02/09 14:00:24 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7731802835375335223
2026/02/09 14:00:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7731802835375335223 records=3 bytes=63 channelLen=2
2026/02/09 14:00:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7731802835375335223
2026/02/09 14:00:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7731802835375335223 records=0 bytes=0 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 14:00:24 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=0

github.com/PeerDB-io/peerdb/flow/e2e::TestBigQueryClickhouseSuite/Test_GCS_Cleanup_After_Initial_Load
Stack Traces | 43.4s run time 
=== RUN   TestBigQueryClickhouseSuite/Test_GCS_Cleanup_After_Initial_Load
=== PAUSE TestBigQueryClickhouseSuite/Test_GCS_Cleanup_After_Initial_Load
=== CONT  TestBigQueryClickhouseSuite/Test_GCS_Cleanup_After_Initial_Load
    bigquery_source_test.go:846: ClickHouse database: e2e_test_bqch_4flkmka4
    bigquery_source_test.go:852: Creating test table test_gcs_cleanup_p6h25aqr
2026/02/09 13:59:41 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychg_k783tqsr.test_lost_column_bug
    bigquery_source_test.go:900: Inserted 5 rows into source table
    bigquery_source_test.go:922: WaitFor data replicated 2026-02-09 13:59:44.945488377 +0000 UTC m=+205.102777640
2026/02/09 13:59:45 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 13:59:45 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 13:59:45 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_13674446969521249178 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" args=[]
2026/02/09 13:59:45 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" channelLen=0
2026/02/09 13:59:45 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13674446969521249178
2026/02/09 13:59:45 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13674446969521249178 records=8 bytes=168 channelLen=7
2026/02/09 13:59:45 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=7
2026/02/09 13:59:45 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13674446969521249178
2026/02/09 13:59:45 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13674446969521249178 records=0 bytes=0 channelLen=0
2026/02/09 13:59:45 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:45 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:45 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=0
2026/02/09 13:59:45 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 13:59:45 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 13:59:45 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_8628273795720186718 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" args=[]
2026/02/09 13:59:45 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" channelLen=0
2026/02/09 13:59:45 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8628273795720186718
2026/02/09 13:59:45 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8628273795720186718 records=3 bytes=63 channelLen=2
2026/02/09 13:59:45 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=2
2026/02/09 13:59:45 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8628273795720186718
2026/02/09 13:59:45 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8628273795720186718 records=0 bytes=0 channelLen=0
2026/02/09 13:59:45 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:45 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:45 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=0
    bigquery_source_test.go:922: code: 60, message: Unknown table expression identifier 'test_gcs_cleanup_p6h25aqr_dst' in scope SELECT id FROM test_gcs_cleanup_p6h25aqr_dst FINAL WHERE _peerdb_is_deleted = 0 ORDER BY 1 ASC SETTINGS use_query_cache = false
    bigquery_source_test.go:922: q.NumRecords: 5
    bigquery_source_test.go:922: other.NumRecords: 0
2026/02/09 13:59:47 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 13:59:47 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 13:59:47 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_13219882432546262939 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" args=[]
2026/02/09 13:59:47 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" channelLen=0
2026/02/09 13:59:47 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13219882432546262939
2026/02/09 13:59:47 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13219882432546262939 records=8 bytes=168 channelLen=7
2026/02/09 13:59:47 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=7
2026/02/09 13:59:47 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13219882432546262939
2026/02/09 13:59:47 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13219882432546262939 records=0 bytes=0 channelLen=0
2026/02/09 13:59:47 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:47 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:47 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=0
2026/02/09 13:59:47 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 13:59:47 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 13:59:47 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_2651570689587927444 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" args=[]
2026/02/09 13:59:47 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" channelLen=0
2026/02/09 13:59:47 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_2651570689587927444
2026/02/09 13:59:47 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_2651570689587927444 records=3 bytes=63 channelLen=2
2026/02/09 13:59:47 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=0
2026/02/09 13:59:47 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_2651570689587927444
2026/02/09 13:59:47 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_2651570689587927444 records=0 bytes=0 channelLen=0
2026/02/09 13:59:47 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:47 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:47 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=0
    bigquery_source_test.go:922: q.NumRecords: 5
    bigquery_source_test.go:922: other.NumRecords: 0
2026/02/09 13:59:48 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column\" ORDER BY id"
2026/02/09 13:59:48 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column\" ORDER BY id"
2026/02/09 13:59:48 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_12617237290210064649 CURSOR FOR SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column\" ORDER BY id" args=[]
2026/02/09 13:59:48 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column\" ORDER BY id" channelLen=0
2026/02/09 13:59:48 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_12617237290210064649
2026/02/09 13:59:48 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_12617237290210064649 records=2 bytes=12 channelLen=1
2026/02/09 13:59:48 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column\" ORDER BY id" rows=2 bytes=12 channelLen=0
2026/02/09 13:59:48 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_12617237290210064649
2026/02/09 13:59:48 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_12617237290210064649 records=0 bytes=0 channelLen=0
2026/02/09 13:59:48 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:48 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:48 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column\" ORDER BY id" rows=2 bytes=12 channelLen=0
2026/02/09 13:59:48 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column_dst\" ORDER BY id"
2026/02/09 13:59:48 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column_dst\" ORDER BY id"
2026/02/09 13:59:48 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_8023686543779232122 CURSOR FOR SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column_dst\" ORDER BY id" args=[]
2026/02/09 13:59:48 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column_dst\" ORDER BY id" channelLen=0
2026/02/09 13:59:48 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8023686543779232122
2026/02/09 13:59:48 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8023686543779232122 records=2 bytes=12 channelLen=1
2026/02/09 13:59:48 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column_dst\" ORDER BY id" rows=2 bytes=12 channelLen=1
2026/02/09 13:59:48 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8023686543779232122
2026/02/09 13:59:48 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8023686543779232122 records=0 bytes=0 channelLen=0
2026/02/09 13:59:48 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column_dst\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:48 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:48 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id, col_to_add FROM e2e_test_pg_jz97ujdr.\"test_ddl_drop_column_dst\" ORDER BY id" rows=2 bytes=12 channelLen=0
    bigquery_source_test.go:922: q.NumRecords: 5
    bigquery_source_test.go:922: other.NumRecords: 0
2026/02/09 13:59:50 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 13:59:50 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 13:59:50 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_942064873058753223 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" args=[]
2026/02/09 13:59:50 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" channelLen=0
2026/02/09 13:59:50 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_942064873058753223
2026/02/09 13:59:50 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_942064873058753223 records=8 bytes=168 channelLen=7
2026/02/09 13:59:50 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=7
2026/02/09 13:59:50 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_942064873058753223
2026/02/09 13:59:50 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_942064873058753223 records=0 bytes=0 channelLen=0
2026/02/09 13:59:50 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:50 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:50 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=0
2026/02/09 13:59:50 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 13:59:50 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 13:59:50 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_5157330822793114325 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" args=[]
2026/02/09 13:59:50 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" channelLen=0
2026/02/09 13:59:50 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5157330822793114325
2026/02/09 13:59:50 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5157330822793114325 records=3 bytes=63 channelLen=2
2026/02/09 13:59:50 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=0
2026/02/09 13:59:50 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5157330822793114325
2026/02/09 13:59:50 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5157330822793114325 records=0 bytes=0 channelLen=0
2026/02/09 13:59:50 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:50 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:50 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=0
    bigquery_source_test.go:922: q.NumRecords: 5
    bigquery_source_test.go:922: other.NumRecords: 0
    bigquery_source_test.go:924: Verifying data in ClickHouse
    bigquery_source_test.go:931: Checking GCS staging path for exported objects
    bigquery_source_test.go:933: WaitFor GCS path gs:.../peerdb_bigquery_source_test_do_not_remove/test_cleanup/test_gcs_cleanup_p6h25aqr to be cleaned up 2026-02-09 13:59:52.89435491 +0000 UTC m=+213.051644183
2026/02/09 13:59:53 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 13:59:53 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 13:59:53 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_7120307888005418938 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" args=[]
2026/02/09 13:59:53 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" channelLen=0
2026/02/09 13:59:53 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7120307888005418938
2026/02/09 13:59:53 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7120307888005418938 records=8 bytes=168 channelLen=7
2026/02/09 13:59:53 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=7
2026/02/09 13:59:53 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7120307888005418938
2026/02/09 13:59:53 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7120307888005418938 records=0 bytes=0 channelLen=0
2026/02/09 13:59:53 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:53 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:53 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=0
2026/02/09 13:59:53 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 13:59:53 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 13:59:53 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_6325072532569991686 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" args=[]
2026/02/09 13:59:53 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" channelLen=0
2026/02/09 13:59:53 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_6325072532569991686
2026/02/09 13:59:53 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_6325072532569991686 records=3 bytes=63 channelLen=2
2026/02/09 13:59:53 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=2
2026/02/09 13:59:53 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_6325072532569991686
2026/02/09 13:59:53 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_6325072532569991686 records=0 bytes=0 channelLen=0
2026/02/09 13:59:53 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 13:59:53 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 13:59:53 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=0
    bigquery_source_test.go:933: UNEXPECTED TIMEOUT GCS path gs:.../peerdb_bigquery_source_test_do_not_remove/test_cleanup/test_gcs_cleanup_p6h25aqr to be cleaned up 2026-02-09 14:00:23.925701221 +0000 UTC m=+244.082990484
2026/02/09 14:00:24 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'"
2026/02/09 14:00:24 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'"
2026/02/09 14:00:24 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_5178943941061771626 CURSOR FOR \n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" args=[]
2026/02/09 14:00:24 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5178943941061771626
2026/02/09 14:00:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5178943941061771626 records=1 bytes=8 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" rows=1 bytes=8 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5178943941061771626
2026/02/09 14:00:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5178943941061771626 records=0 bytes=0 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" rows=0 bytes=0 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 14:00:24 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" rows=1 bytes=8 channelLen=0
2026/02/09 14:00:24 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'"
2026/02/09 14:00:24 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'"
2026/02/09 14:00:24 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_5696179322022755988 CURSOR FOR \n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" args=[]
2026/02/09 14:00:24 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5696179322022755988
2026/02/09 14:00:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5696179322022755988 records=1 bytes=8 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" rows=1 bytes=8 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5696179322022755988
2026/02/09 14:00:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5696179322022755988 records=0 bytes=0 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" rows=0 bytes=0 channelLen=0
2026/02/09 14:00:24 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 14:00:24 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" rows=1 bytes=8 channelLen=0
--- FAIL: TestBigQueryClickhouseSuite/Test_GCS_Cleanup_After_Initial_Load (43.41s)

github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg/TestResyncFailed
Stack Traces | 138s run time 
=== RUN   TestApiPg/TestResyncFailed
=== PAUSE TestApiPg/TestResyncFailed
=== CONT  TestApiPg/TestResyncFailed
2026/02/09 14:00:23 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/02/09 14:00:23 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
    api_test.go:850: WaitFor wait for MV error messages 2026-02-09 14:00:23.088115197 +0000 UTC m=+243.245404470
2026/02/09 14:00:23 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'"
2026/02/09 14:00:23 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'"
2026/02/09 14:00:23 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_6667225532003833932 CURSOR FOR \n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" args=[]
2026/02/09 14:00:23 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" channelLen=0
2026/02/09 14:00:23 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_6667225532003833932
2026/02/09 14:00:23 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_6667225532003833932 records=1 bytes=8 channelLen=0
2026/02/09 14:00:23 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" rows=1 bytes=8 channelLen=0
2026/02/09 14:00:23 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_6667225532003833932
2026/02/09 14:00:23 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_6667225532003833932 records=0 bytes=0 channelLen=0
2026/02/09 14:00:23 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" rows=0 bytes=0 channelLen=0
2026/02/09 14:00:23 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 14:00:23 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('resync_failed_api_gvw8joq5' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_gvw8joq5.resync_failed_api_gvw8joq5_mv_api_gvw8joq5%'" rows=1 bytes=8 channelLen=0
    api_test.go:863: WaitFor wait for failed 2026-02-09 14:00:25.108717929 +0000 UTC m=+245.266007202
2026/02/09 14:00:25 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'"
2026/02/09 14:00:25 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'"
2026/02/09 14:00:25 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_12769460666182752800 CURSOR FOR \n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" args=[]
2026/02/09 14:00:25 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_12769460666182752800
2026/02/09 14:00:25 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_12769460666182752800 records=1 bytes=8 channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" rows=1 bytes=8 channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_12769460666182752800
2026/02/09 14:00:25 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_12769460666182752800 records=0 bytes=0 channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" rows=0 bytes=0 channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 14:00:25 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="\n\t\tSELECT COUNT(*) FROM peerdb_stats.flow_errors\n\t\tWHERE error_type='error' AND position('cancel_table_addition_test_flow_normal_api_ae4z8cok' in flow_name) > 0\n\t\tAND error_message ILIKE '%while pushing to view e2e_test_api_ae4z8cok.t5_mv_api_ae4z8cok%'" rows=1 bytes=8 channelLen=0
2026/02/09 14:00:25 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 14:00:25 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id"
2026/02/09 14:00:25 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_219458189314954928 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" args=[]
2026/02/09 14:00:25 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_219458189314954928
2026/02/09 14:00:25 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_219458189314954928 records=8 bytes=168 channelLen=7
2026/02/09 14:00:25 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_219458189314954928
2026/02/09 14:00:25 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_219458189314954928 records=0 bytes=0 channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 14:00:25 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf\" ORDER BY id" rows=8 bytes=168 channelLen=0
2026/02/09 14:00:25 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 14:00:25 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id"
2026/02/09 14:00:25 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_8814295221185482748 CURSOR FOR SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" args=[]
2026/02/09 14:00:25 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8814295221185482748
2026/02/09 14:00:25 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8814295221185482748 records=3 bytes=63 channelLen=2
2026/02/09 14:00:25 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=2
2026/02/09 14:00:25 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8814295221185482748
2026/02/09 14:00:25 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_8814295221185482748 records=0 bytes=0 channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/02/09 14:00:25 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/02/09 14:00:25 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,name,created_at FROM e2e_test_pg_gje3fj3e.\"test_inheritance_dynconf_dst\" ORDER BY id" rows=3 bytes=63 channelLen=0
    api_test.go:881: WaitFor resync should have 2 rows 2026-02-09 14:00:36.586266248 +0000 UTC m=+256.743555522
    api_test.go:44: begin tearing down postgres schema api_gvw8joq5
2026/02/09 14:00:40 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychclg_9jjqysy9.test_simple
    api_test.go:44: 
        	Error Trace:	.../flow/e2e/pg.go:149
        	            				.../flow/e2e/api_test.go:44
        	            				.../flow/e2eshared/e2eshared.go:38
        	            				.../hostedtoolcache/go/1.25.6.../src/testing/testing.go:1308
        	            				.../hostedtoolcache/go/1.25.6.../src/testing/testing.go:1572
        	            				.../hostedtoolcache/go/1.25.6.../src/testing/testing.go:1928
        	            				.../hostedtoolcache/go/1.25.6.../src/runtime/asm_amd64.s:1693
        	Error:      	failed to teardown postgres schema
        	Test:       	TestApiPg/TestResyncFailed
        	Messages:   	api_gvw8joq5: failed to drop replication slots: ERROR: replication slot "peerflow_slot_resync_failed_api_gvw8joq5" is active for PID 8219 (SQLSTATE 55006)
--- FAIL: TestApiPg/TestResyncFailed (137.84s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jkaflik