Handle permanent image failures in download state tracking#67
Merged
Conversation
Bulk download and retry-on-fail were declaring chapters "Downloaded" even when some images had failed permanently (404/403). The DB isDownloaded flag stuck on true, so opening such chapters surfaced "Image unavailable" for the missing pages, and on next app launch the chapter could flicker to "In library" before the inspect refresh ran. Root cause: inspectCacheInternal counted .failed-sidecar URLs toward effectiveCached and treated isComplete=true as "fully downloaded". syncDownloadedFlag then promoted the DB flag for chapters that were not actually on disk. Fix: - PrefetchResult gains hasPermanentFailures; cachedImages now reflects the actual on-disk count only. - isComplete keeps its "no further work to attempt" meaning so the download loop and auto-resume don't spin forever, but the DB flag and the "Downloaded" UI label both now require !hasPermanentFailures. - syncDownloadedFlag also demotes a previously-downloaded chapter once a finished USER_REQUESTED inspect confirms permanent failures, so chapters wrongly promoted by earlier sessions self-heal on next start. - chapterCacheStatusKind falls back to Downloaded when the DB remembers the chapter and the cache state hasn't been populated yet, so the badge no longer flickers to "In library" right after launch. - Permanent failures now keep isRetryable=true so the retry control stays visible — tapping it clears the sidecar and re-attempts.
Prevent WorkManagerInitializer from auto-initializing so the app's Configuration.Provider controls WorkManager initialization on demand
Replace lastReadOffset with lastReadElementKey and introduce FRACTION_UNKNOWN sentinel for unmeasured intra-item offsets. Bump Room schema to v8 and backup schema to v2, add MIGRATION_7_8, and update DAO, backup, viewmodel, UI, and tests accordingly
Add ImageIntegrity to detect HTML, zero-byte, and truncated files. Use it in ImageCache and WebContentLoader to avoid accepting corrupt fallbacks. Make ImageDownloader verify Content-Length against bytes read and fail on short reads so partial responses don't enter the cache and trigger retry paths. Update tests and fixtures accordingly.
Also relax JPEG EOF integrity check and add reader drag/restore flags; update tests and bump DB schema to v9
Merge image URLs extracted from inline scripts into parsed image lists for JS-driven chapter pages (Mangabat/Asura/Astro patterns) and filter decorative URLs. Add extractInlineScriptImageUrls and helper heuristics. Add hasMangaReaderHints/detectMangaReaderHints so pages that are manga readers but only carry a JS shell aren't marked Downloaded with zero images. Track in-flight chapter prefetches with mode + deferred and ensure a USER_REQUESTED prefetch does not reuse a SPECULATIVE one (cancel and replace speculative reservations as needed)
Invalidate cached media files and re-download visible HTTP images when a displayed image errors. Add ContentRepository/WebContentLoader invalidation, ReaderViewModel repair methods that demote downloaded chapters and re-fetch, UI retry logic in ReaderImageView, and unit tests.
Request only app-renderable image MIME types and treat AVIF/HEIF/SVG as invalid so cached downloads open reliably; add tests
Use "likely" cache state checks and findUsableCachedMediaFile to avoid relying on brittle file-exists logic and to invalidate before re-download. Batch and delay image-dimension persistence to reduce write churn. Narrow prefetch window (ahead 6, behind 2) and skip the current index. Add on-demand library reconciliation and enrich parsed HTML cache with resolved image dimensions; update related APIs and tests.
Add downloadChapter(...) to ContentRepository and WebContentLoader to handle user-requested durable downloads and short-circuit the speculative prefetch path. Introduce ChapterDownloadLimiter (max 2 concurrent) and use it in the worker to limit durable downloads. Reduce WebContentLoader's MAX_CONCURRENT_DOWNLOADS to 4 and adjust in-flight/deduplication logic so cache-tier in-flight images can be promoted into the downloads tier. Add PrefetchResult.isStrictOfflineReady helper and use it in UI, reconciler, and worker. Update ChapterDownloadQueue to prefer a single observed WorkInfo per URL and refresh tests to match the new flow.
Replace ad-hoc per-host throttling with HostThrottle and wire it into image and HTML fetch paths, including outcome classification for rate-limits/retries. Move HTTP timeout constants into HttpTimeouts and apply them to all OkHttp clients. Misc: cap HTML body size, drop diskCacheKey in ReaderImageView, use CacheKeyUtils for worker URL keys, and update tests/comments
Implement a WebOfflineChapterStore for downloading and persisting chapter HTML and images, add a @WebOfflineDownloadsDir DI qualifier, wire the store into WebContentLoader, add resetWebOfflinePipelineData() and a webOfflinePipelineVersion preference, and update tests/benchmarks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR improves handling of permanent image failures (4xx errors) in chapter downloads by distinguishing between "complete" (no more work to attempt) and "fully downloaded" (all images actually on disk). Chapters with permanent failures now correctly show as "Download incomplete" rather than "Downloaded", preventing misleading UI when images are unavailable offline.
Key Changes
PrefetchResult model: Added
hasPermanentFailuresflag to track when images were accounted as permanently failed (in the .failed sidecar) rather than actually cached on diskWebContentLoader inspection logic:
cachedImagesto reflect only on-disk count, excluding permanent failuresisRetryableto allow retry when permanent failures exist (clearing the sidecar can recover misclassified transient errors)LibraryViewModel sync logic:
isDownloadedflag when permanent failures are detectedUI status display:
chapterCacheStatusKind()to show "Download incomplete" whenhasPermanentFailuresis true, even ifisCompleteis trueisDownloadedflag when cache state is not yet available (prevents flickering on app launch)Test coverage: Added tests validating that:
addChaptersdoesn't mark download when permanent failures are presentrefreshChapterCacheStatesdemotes the downloaded flag when permanent failures appearImplementation Details
The key insight is that
isComplete(no more work to attempt) differs from "fully downloaded" (all images on disk). Permanent failures count toward completion but not toward the downloaded state, allowing the download loop to stop while preventing the UI from showing a misleading "Downloaded" badge that would surface "Image unavailable" errors when opened.https://claude.ai/code/session_01UiAnirw1fUqW3itdo6WS2S