Convert P010/P016 hardware-decoder frames to 8-bit before analysis - #7
Merged
Merged
Conversation
Hardware decoders emit P010LE (not YUV420P10LE) for 10-bit HEVC, so the 8-bit conversion from erikkaashoek#143 was skipped and frames were processed as a double-width 8-bit buffer, corrupting brightness, uniformity and logo metrics on Main10 sources.
Decide the 8-bit conversion by the pixel format's luma depth instead of listing formats, so 12-bit and 4:4:4 hardware outputs are covered too. Free the decoder frame it replaces, which leaked one AVFrame per converted frame, and skip the frame when allocation fails. SubmitFrame refuses a frame that is still not 8-bit instead of reading it as a wider buffer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #4 (commit kept, credit to @v8eta) with review fixes on top: conversion keyed on luma depth rather than a format list, the replaced AVFrame freed, allocation failures skipped, and SubmitFrame refusing non-8-bit frames.
Verified on a 10-bit HEVC recording: cut list and CSV identical to the current fancybits build, peak RSS 8 MB lower.