Bug: Query operations don't handle pagination (200 record limit)

[leogdion]

Problem

Query operations are limited to 200 records and don't handle continuationMarker for pagination:

1. Bushel: BushelCloudKitService.queryRecords() returns max 200 records (line 71)
2. Celestra: Delete operations won't remove all records if count exceeds 200

The OpenAPI-generated types include continuationMarker fields, but they're never used.

Proposed Solution

1. Implement continuation marker handling in query operations
2. Add recursive/iterative query pattern:

func queryAllRecords(recordType: String) async throws -> [RecordInfo] {
    var allRecords: [RecordInfo] = []
    var continuationMarker: String? = nil
    
    repeat {
        let (records, marker) = try await queryRecords(
            recordType: recordType,
            continuationMarker: continuationMarker
        )
        allRecords.append(contentsOf: records)
        continuationMarker = marker
    } while continuationMarker != nil
    
    return allRecords
}

3. Update batch delete operations to handle pagination

Impact

• Data Loss: Queries silently truncate large result sets
• Incomplete Operations: Delete operations leave orphaned records
• User Confusion: No indication that results are incomplete

Files Affected

• Examples/Bushel/Sources/BushelImages/CloudKit/BushelCloudKitService.swift:71
• Examples/Celestra/Sources/Celestra/Services/CloudKitService+Celestra.swift
• Sources/MistKit/Service/CloudKitService+RecordManaging.swift:52
• Sources/MistKit/Generated/Types.swift (continuationMarker fields exist but unused)

References

• PR update from writing blog post and examples #129 CodeRabbit Review: update from writing blog post and examples #129 (BushelCloudKitService.swift)
• Related: Feature: Add pagination support with continuation markers for query results #145 (pagination support), Add comprehensive test suite for Bushel demo #136

[leogdion]

Status: TYPES EXIST, LOGIC NOT IMPLEMENTED

The OpenAPI specification and generated types include continuationMarker fields, but pagination logic is not implemented. Queries are currently limited to 200 records per request with no automatic pagination.

Current State

What Exists:

• OpenAPI spec defines continuationMarker in openapi.yaml:1020 (QueryResponse) and openapi.yaml:77 (query request)
• Generated types include QueryResponse.continuationMarker: String? in Sources/MistKit/Generated/Types.swift:1080-1092
• Request types support optional continuationMarker parameter

What's Missing:

• queryRecords() in CloudKitService+Operations.swift:203-324 executes single query only
• No logic to check for continuation marker in response
• No automatic pagination loop
• No AsyncSequence implementation for streaming paginated results
• RecordManaging protocol doesn't support pagination
• Examples (BushelCloud, CelestraCloud) silently truncate results at 200 records

Impact

• Data Loss Risk: Queries silently truncate large result sets
• Incomplete Operations: Delete/sync operations may leave orphaned records
• User Confusion: No indication that results are incomplete

What Remains

1. Modify queryRecords() to Return Continuation Marker

File: Sources/MistKit/Service/CloudKitService+Operations.swift:203-324

Create a QueryResult struct:

public struct QueryResult {
    public let records: [RecordInfo]
    public let continuationMarker: String?
}

Update queryRecords signature:

public func queryRecords(
    recordType: String,
    filters: [QueryFilter]? = nil,
    sortBy: [QuerySort]? = nil,
    limit: Int? = nil,
    desiredKeys: [String]? = nil,
    continuationMarker: String? = nil
) async throws -> QueryResult

2. Implement Pagination Helper

Add automatic pagination method:

public func queryAllRecords(
    recordType: String,
    filters: [QueryFilter]? = nil,
    sortBy: [QuerySort]? = nil,
    desiredKeys: [String]? = nil
) async throws -> [RecordInfo] {
    var allRecords: [RecordInfo] = []
    var continuationMarker: String? = nil
    
    repeat {
        let result = try await queryRecords(
            recordType: recordType,
            filters: filters,
            sortBy: sortBy,
            limit: 200,
            desiredKeys: desiredKeys,
            continuationMarker: continuationMarker
        )
        allRecords.append(contentsOf: result.records)
        continuationMarker = result.continuationMarker
    } while continuationMarker != nil
    
    return allRecords
}

3. Implement AsyncSequence for Streaming

public func queryRecordsSequence(
    recordType: String,
    filters: [QueryFilter]? = nil,
    sortBy: [QuerySort]? = nil,
    desiredKeys: [String]? = nil
) -> AsyncThrowingStream<RecordInfo, Error>

4. Update RecordManaging Protocol

File: Sources/MistKit/Service/CloudKitService+RecordManaging.swift:52

Add pagination support to protocol methods.

5. Update Examples

• BushelCloudKitService (Examples/BushelCloud/Sources/BushelCloudKit/CloudKit/BushelCloudKitService.swift:49)
• CelestraCloud CloudKitService (Examples/CelestraCloud/Sources/CelestraCloudKit/Services/CloudKitService+Celestra.swift:74-97)

6. Add Comprehensive Tests

• Test multi-page queries
• Test continuation marker handling
• Test 200+ record scenarios
• Test AsyncSequence streaming

Files to Create/Modify

• Sources/MistKit/Service/CloudKitService+Operations.swift (modify queryRecords)
• Sources/MistKit/Service/CloudKitService+RecordManaging.swift (add pagination)
• Sources/MistKit/Service/CloudKitService+Pagination.swift (new, AsyncSequence)
• Examples/BushelCloud/Sources/BushelCloudKit/CloudKit/BushelCloudKitService.swift
• Examples/CelestraCloud/Sources/CelestraCloudKit/Services/CloudKitService+Celestra.swift
• Tests/MistKitTests/CloudKitServicePaginationTests.swift (new)

References

• Related: GitHub issue Feature: Add pagination support with continuation markers for query results #145 (pagination support)