Skip to content

Latest commit

 

History

History
638 lines (469 loc) · 26.9 KB

File metadata and controls

638 lines (469 loc) · 26.9 KB

Developer Guide

This document provides an overview of the most important classes and structures, and how to use them.

Table of Contents


Structures

FFmpeg.Utils.AVResult32-63

AVResult32-63 represents the result of most FFmpeg functions that return an integer. The return value indicates success or failure:

  • Success: >= 0IsError returns false
  • Failure: < 0IsError returns true

Conversion

AVResult32-63 can be implicitly converted from and to both int and long.

Error Codes

The structure provides static properties for the most important FFmpeg error codes.

GNU Error Codes
Code Name Description
-11 TryAgain Used during encoding/decoding or filtering when more data needs to be written or read.
-22 InvalidArgument Indicates missing or incorrect parameters. Check standard output for additional FFmpeg error information.
-1 to -32 Other standard GNU error codes
FFmpeg Error Codes [Negative FourCC]
Code Name Description
"EOF " EndOfFile End of the file has been reached.
...

Exception Handling

Call ThrowIfError() to automatically throw an exception if the value is negative.

Example

AVResult32 res = context.ReceiveFrame(frame);
do while(res == AVResult32.TryAgain)
{
    context.SendPacket(packet).ThrowIfError();
    res = context.ReceiveFrame(frame);
} 

FFmpeg.Utils.Rational

The Rational struct represents a rational number, typically used to express frame rates, timebases, or other ratio-based values.

Conversions

Rational supports implicit conversion from and to double, and TimeSpan.

Operators

The Rational struct supports several operators, including:

  • Arithmetic operations (+, -, *, /)
  • Comparisons (==, !=, <, >, <=, >=)
  • Conversions to numeric types (double, TimeSpan)

Rescale

The method Rescale can be used to rescale a timestamp (usually stored as a long in TimeBase units) from one TimeBase into another.

Example

// TimeSpan = long * Rational
// GetPresentationTimestamp returns the pts or the best effort pts if pts is not available
TimeSpan pts = frame.GetPresentationTimestamp() * frame.TimeBase;
// Conversion from one timebase into another
dstFrame.PresentationTimestamp = dstFrame.TimeBase.Rescale(srcFrame.GetPresentationTimestamp(), srcFrame.TimeBase);

FFmpeg.Codec

Represents a codec in FFmpeg. Provides access to core codec information, supported formats, and allows managed interaction with codec details.

Properties

Property Type Description
Name string Short, symbolic name of the codec (e.g., "h264").
LongName string Human-readable descriptive name of the codec.
MediaType MediaType Type of media handled by the codec (video, audio, subtitle).
CodecID CodecID FFmpeg codec identifier.
SupportedFramerates ReadOnlySpan<Rational> Array of supported frame rates (video codecs).
SupportedPixelFormats ReadOnlySpan<PixelFormat> Array of supported pixel formats (video codecs).
SupportedSampleRates ReadOnlySpan<int> Array of supported audio sample rates.
SupportedSampleFormats ReadOnlySpan<SampleFormat> Array of supported audio sample formats.

Methods

Method Description
GetBestPixelFormat(PixelFormat src, bool alphaUsed, out FFLoss loss) Finds the best pixel format for a source format, considering alpha usage.
GetBestPixelFormat(PixelFormat src) Overload ignoring loss and assuming alpha used.
static ReadOnlyCollection<Codec> GetAllCodecs() Retrieves all registered codecs (encoders and decoders).
static Codec? FindDecoder(CodecID codecID) Finds a decoder by codec ID.
static Codec? FindDecoder(string name) Finds a decoder by name.
static Codec? FindEncoder(CodecID codecID) Finds an encoder by codec ID.
static Codec? FindEncoder(string name) Finds an encoder by name.

Usage Example

// Find a decoder by codec ID
Codec? codec = Codec.FindDecoder(CodecID.H264);

if (codec is not null)
{
    Console.WriteLine($"Decoder: {codec?.Name} ({codec?.LongName})");
    Console.WriteLine($"MediaType: {codec?.MediaType}, CodecID: {codec?.CodecID}");

    // Best pixel format for a given source
    FFLoss loss = default;
    PixelFormat best = codec?.GetBestPixelFormat(PixelFormat.YUV420P, alphaUsed: false, out loss) ?? PixelFormat.None;
    if (best != PixelFormat.None)
        Console.WriteLine($"Best PixelFormat: {best}, Loss: {loss}");
}

// Enumerate all codecs
var allCodecs = Codec.GetAllCodecs();

Enums

FFmpeg.Images.PixelFormat

The PixelFormat enum defines all supported pixel formats within FFmpeg.
It describes how pixel data is stored and interpreted (e.g., RGB, YUV, planar, packed, endianness, alpha channels, etc.).

Extensions

The FFmpeg.Images.PixelFormatExtensions static class provides helper methods to simplify working with pixel formats.

Method Description Returns
SwapEndianness() Swaps the byte order (endianness) of the pixel format (e.g., RGB24BERGB24LE). PixelFormat
PlaneCount() Returns the number of image planes for the given pixel format (e.g., 1 for RGB, 3 for YUV). int
FindBestPixelFormat(ReadOnlySpan<PixelFormat>) Finds the best matching pixel format from a list of candidates. PixelFormat
FindBestPixelFormat(bool useAlpha, ReadOnlySpan<PixelFormat>) Finds the best matching format considering alpha transparency. PixelFormat
FindBestPixelFormat(out FFLoss loss, ReadOnlySpan<PixelFormat>) Finds the best matching format and reports conversion loss. PixelFormat
FindBestPixelFormat(bool useAlpha, out FFLoss loss, ReadOnlySpan<PixelFormat>) Finds the best matching format considering alpha transparency and reports conversion loss. PixelFormat

FFmpeg.Audio.SampleFormat

The SampleFormat enum defines all supported audio sample formats within FFmpeg.
It describes how audio samples are stored — integer vs. floating-point, planar vs. packed, bit depth, etc.

Extensions

The FFmpeg.Audio.SampleExtensions static class provides helper methods for inspecting, converting, and validating audio sample formats.

Method Description Returns
IsPlanar() Determines whether the sample format stores audio data in planar layout (one buffer per channel). bool
IsPacked() Determines whether the sample format stores audio data in packed/interleaved layout. bool
AsPlanar() Converts the current sample format to its planar equivalent (e.g., Int16Int16Planar). SampleFormat
AsPacked() Converts the current sample format to its packed equivalent (e.g., Float32PlanarFloat32). SampleFormat
GetBytesPerSample() Returns the number of bytes per individual sample. int
GetBitsPerSample() Returns the number of bits per individual sample. int
GetBitsPerSample(AutoGen._AVCodecID) Returns the number of bits per sample for a given FFmpeg codec ID. int
GetName() Returns the FFmpeg string name for the sample format (e.g., "s16p", "fltp"). string
ValidateType<T>() Ensures that a given unmanaged .NET type matches the sample format (throws if not). void
GetSampleFormatType() Returns the .NET type (byte, short, float, etc.) corresponding to the sample format. Type

FFmpeg.Formats.DiscardFlags

The DiscardFlags enum specifies which packets of an AVStream should be discarded during decoding or processing.

Example

var mediaSource = MediaSource.Open(file);
int videoIndex = mediaSource.FindBestStream(MediaType.Video);

// Discard all packets from every stream
foreach (var stream in mediaSource.Streams)
    stream.Discard = DiscardFlags.All;

// Keep only the main video stream active
mediaSource.Streams[videoIndex].Discard = DiscardFlags.Default;

FFmpeg.Devices.DeviceType

The DeviceType enum specifies the hardware device types available for hardware‑accelerated decoding in FFmpeg.

DeviceType Description
None No hardware acceleration device.
VDPAU Video Decode and Presentation API for Unix — Linux hardware decoding.
CUDA NVIDIA GPU hardware-accelerated decoding.
VAAPI Video Acceleration API — Linux/Unix.
DXVA2 Microsoft DirectX Video Acceleration 2 — Windows.
QSV Intel Quick Sync Video — Intel platforms.
VideoToolbox Apple hardware-accelerated video API on macOS/iOS.
D3D11VA Direct3D 11 Video Acceleration — Windows.
DRM Direct Rendering Manager — Linux GPU acceleration.
OpenCL Open Computing Language — cross-platform GPU/CPU.
MediaCodec Android hardware-accelerated video API.
Vulkan Cross-platform graphics & compute API.
D3D12VA Direct3D 12 Video Acceleration — Windows.
AMF AMD Advanced Media Framework — AMD hardware acceleration.
OHCODEC OpenHarmony hardware acceleration.

Notes

  • Intended for decoding: choose a DeviceType when creating a decoder context for hardware acceleration.
  • Availability depends on OS, GPU, drivers, and FFmpeg build.
  • Unsupported devices may fall back to software decoding or throw an error.
  • See FFmpeg Wiki: HWAccelIntro

Example

MediaSource video = MediaSource.Open(file, deviceType: DeviceType.Vulkan);

Classes

Most classes contain unmanaged data and implement IDisposable.

FFmpeg.AVFrame

Represents a decoded audio or video frame in memory.

Allocation

Method Description
static AVFrame.Allocate() Allocates a new empty frame.
static AVFrame.Allocate(PixelFormat format, int width, int height) Allocates a video frame with initialized image buffer.
static AVFrame.Allocate(SampleFormat format, ChannelLayout layout, int samples, int? sampleRate = null) Allocates an audio frame with initialized sample buffers.
static AVFrame.Allocate(SampleFormat format, int channels, int samples, int? sampleRate = null) Allocates an audio frame using a default channel layout.

Properties

Property Type Description
Width int Coded width of the video frame.
Height int Coded height of the video frame.
PixelFormat PixelFormat Pixel format for video frames.
SampleFormat SampleFormat Sample format for audio frames.
SampleRate int Audio sample rate in Hz.
SampleCount int Number of audio samples per channel.
ChannelLayout ChannelLayout_ref Channel layout of the audio frame.
Data ReadOnlySpan<IntPtr> Pointers to the frame’s data buffers.
LineSize ref int_array8 Stride/line sizes for each buffer.
PresentationTimestamp long Presentation timestamp (PTS) in stream time base units.
TimeBase Rational Time base for frame timestamps.
Duration long Duration of the frame in the same units as PTS.

Methods

Method Description
GetPresentationTimestamp() Returns the PTS if available, otherwise best-effort timestamp.
CreateBuffer(int align = 1) Allocates new buffers for the frame. Must set Pixel/Sample format, dimensions or channels first.
GetData(int index) Returns a span of bytes for the specified plane/channel.
GetBufferSpan(int bufferIndex) Returns a span of bytes for the specified buffer.
Reference(AVFrame src) Makes the current frame reference another frame (shallow copy).
Unreference() Releases all buffers and resets the frame for reuse.
Dispose() Disposes the AVFrame.

Usage Example

// Allocate a video frame
using var frame = AVFrame.Allocate(PixelFormat.RGB24, 1920, 1080);

// Access pixel data
using Span<byte> plane0 = frame.GetData(0);

// Allocate an audio frame
var audioFrame = AVFrame.Allocate(SampleFormat.Float32, 2, 1024, 44100);

// Reference another frame
var clone = AVFrame.Allocate();
clone.Reference(frame);

// Reset frame for reuse
frame.Unreference();

FFmpeg.AVPacket

Represents a compressed audio or video packet in memory.

Allocation

Method Description
static AVPacket.Allocate() Allocates a new empty packet.
static AVPacket.Allocate(int size) Allocates a packet with a pre-allocated payload.

Properties

Property Type Description
PresentationTimestamp long Packet PTS in stream time base units.
PresentationTime TimeSpan PTS expressed as a TimeSpan using the packet's TimeBase.
DecompressionTimestamp long Packet DTS in stream time base units.
DecompressionTime TimeSpan DTS expressed as a TimeSpan using the packet's TimeBase.
Data Span<byte> Raw packet data.
Size int Size of the packet data in bytes.
StreamIndex int Index of the stream this packet belongs to.
Duration long Duration in stream time base units.
Position long Byte position of the packet in the stream.
TimeBase Rational Time base for the packet's timestamps.

Methods

Method Description
Unreference() Releases the packet and any associated data.
Clone() Creates a new packet referencing the same data (shallow copy).
Dispose() Disposes the AVPacket.

Usage Example

// Allocate a new packet
using var packet = AVPacket.Allocate();

// Receive a packet from the demuxer
formatContext.ReadFrame(packet);

// Send the packet to the decoder
codecContext.SendPacket(packet);

// Access raw packet data
Span<byte> buffer = packet.Data;

// Clone the packet (shallow copy)
using var clone = packet.Clone();

// Release packet resources
packet.Unreference();

FFmpeg.Formats.FormatContext

Represents a managed wrapper around FFmpeg’s native _AVFormatContext, providing container-level functionality shared by both demuxing and muxing operations.
This class is typically used indirectly through DemuxerContext (input) and MuxerContext (output).
It implements IDisposable because it owns unmanaged FFmpeg resources.

Responsibilities

  • Wraps the native _AVFormatContext*
  • Manages container metadata, stream descriptors, chapters, and format options
  • Synchronizes managed stream objects with the native stream array
  • Supports binding to a custom IOContext
  • Provides format-level option querying via OptionQueryBase
  • Handles cleanup of the underlying native context and any associated I/O resources

Properties

Property Type Description
Flags FormatContextFlags Flags applied to the format context.
StreamCount int Number of streams present in the container.
Streams IReadOnlyList<AVStream> Managed list of stream descriptors synchronized with the native context.
Url string? Media URL or file path associated with the context, if available.
Metadata AVDictionary_ref Container-level metadata.
StartTimeRealTime DateTime? Real-time clock value of the first packet, when provided by the container.
Chapters ChapterList List of chapters defined in the container.

Methods

Method Description
FindBestStream(MediaType type) Returns the index of the best-matching stream for the specified media type.
SetContext(IOContext context, IOOptions options, int bufferSize) Associates a custom I/O context and configures it for reading or writing.
GetOutputTimestamp(int streamIndex, out TimeSpan timestamp, out TimeSpan wallClock) Retrieves FFmpeg’s computed output timestamps for muxing operations.
Dispose() Releases the underlying _AVFormatContext* and any associated resources.

Notes

  • This class is not intended for direct allocation or opening of media files.
    Use DemuxerContext for reading and MuxerContext for writing.
  • It functions as the core container representation used internally by both workflows.

FFmpeg.Formats.MuxerContext

Represents an output-oriented wrapper around FFmpeg’s _AVFormatContext, used for creating and writing container formats such as MP4, MKV, MP3, TS, and others.
MuxerContext is responsible for allocating output contexts, configuring streams, writing headers, writing packets, and finalizing the output file.

This class inherits from FormatContext and extends it with muxing-specific functionality.

Responsibilities

  • Allocates and configures output format contexts
  • Manages output streams and associated codec parameters
  • Handles file or custom I/O binding for writing
  • Writes headers, packets, trailers, and flushes buffers
  • Ensures proper disposal of unmanaged FFmpeg structures

Properties

Property Type Description
OutputFormat OutputFormat? The format used for muxing, derived from the underlying _AVFormatContext.oformat.

Context Allocation & Opening

Method Description
Open(string? filename, OutputFormat? format) Creates a muxer context using a filesystem output target. May also allocate an FFmpeg-managed file handle.
Open(Stream stream, OutputFormat format, bool closeOnDispose = true) Opens a muxer context using a managed .NET stream.
Open(IOContext context, OutputFormat format) Opens a muxer context using a custom I/O implementation.

Stream Creation

Method Description
AddStream(Codec codec) Creates a new stream using the given codec. Initializes codec parameters (ID, type).
AddStream(Codec codec, ICodecParameters parameters) Creates a new stream and copies codec parameters into the stream.
AddStream(CodecContext codec) Creates a stream from an existing initialized codec context. Copies codec parameters and uses its time base.

Header Writing

Method Description
WriteHeader() Writes the container header using default options.
WriteHeader(AVDictionary dict) Writes the header using a single dictionary of muxing options.
WriteHeader(AVMultiDictionary dict) Writes the header using multiple option dictionaries.
WriteHeader(IDictionary<string,string> dict) Writes the header and returns updated options after FFmpeg processing.

Writing Packets

Method Description
WriteFrame(IPacket? packet) Writes a packet directly without interleaving.
WriteFrameInterleaved(IPacket? packet) Writes a packet with automatic interleaving.
WritePacket(AVPacket? packet) Writes a packet directly without interleaving.
WritePacketInterleaved(AVPacket? packet) Writes a packet with automatic interleaving.

Finalization

Method Description
WriteTrailer() Writes the container trailer and final metadata.
Flush() Flushes the underlying I/O context if present.

Other Members

Member Description
ToString() Returns the long name of the output format or "Unknown" when unavailable.

Usage Example

    using MuxerContext context = MuxerContext.Open("output.mp4",null!)!;
        
    using CodecContext ctx = CodecContext.Open(Codec.FindEncoder(CodecID.H264)!.Value);
    // ... Set codec parameters (ctx);
    // adds a stream using the codec context and copies the codec parameters
    // its advised to set the codec parameters manually instead of copieng during remuxing,
    // since they might not be valid between Decoder and Encoder
    var videoStream = context.AddStream(ctx);

    videoStream.TimeBase = ctx.TimeBase = new Rational(1, 30); // 30 fps

    var audioStream = context.AddStream(Codec.FindEncoder(CodecID.AAC)!.Value);
    audioStream.TimeBase = new Rational(1, 48000); // 48 kHz
    var codecParameters = audioStream.CodecParameters; // its a struct, VS, does not like changing Properties that are structs.
    codecParameters.SampleRate = 48000;
    codecParameters.ChannelLayout = ChannelLayout.CreateStereo();
    codecParameters.SampleFormat = SampleFormat.Float32Planar;
    audioStream.SetOption("key", "value"); // example to set stream option

    context.WriteHeader();
    // ... Encode and write frames
    while (...)
    {
        AVPacket packet = GetSomePacketFormSomeEncoder();
        packet.StreamIndex = videoStream.Index; // or audioStream.Index
        context.WriteFrameInterleaved(packet);
    }
    context.WriteTrailer();
    context.Dispose(); // or use 'using' statement

FFmpeg.Formats.DemuxerContext

Represents an input-oriented wrapper around FFmpeg’s _AVFormatContext used for reading container formats such as MP4, MKV, MP3, TS, and others.
This class inherits from FormatContext and extends it with demuxing-specific functionality.

Constructions

Constructor Description
protected DemuxerContext(AutoGen._AVFormatContext* context) Initializes a new instance of DemuxerContext with an existing native _AVFormatContext. Ownership of the context depends on internal management.

Properties

Property Type Description
InputFormat InputFormat? Gets the input format associated with this context, if available.
StartTime long Start time of the media file in AV_TIME_BASE units.
Duration long Duration of the media file in AV_TIME_BASE units.
BitRate long Overall bit rate of the media file in bits per second.
Chapters ChapterList List of chapters in the media file. Constructed from the current context.

Methods

Open(...)

Opens an input media source and initializes a DemuxerContext. Overloads exist depending on the type of source:

Source Type Input Format (InputFormat?) Options (IDictionary<string,string> or derived) Find Stream Info (bool) Notes
File path (string) Optional Optional Optional Opens input from a file or URL.
Stream (Stream) Optional Optional Optional Stream must support reading and seeking.
IOContext (IOContext) Optional Optional Optional IOContext handles low-level read/write operations; may support seeking.

Parameter notes:

  • Input Format: Required for non-file-based sources or to override automatic detection.
  • Options: General key/value dictionary controlling input behavior; single or multi-dictionary supported internally.
  • Find Stream Info: If true, calls avformat_find_stream_info after opening to populate stream metadata. FFmpeg might read a few frames to set the AVStream Properties to the correct values.

All overloads internally handle dictionary conversions and FFmpeg context initialization.

Method Description
AVResult32 FindStreamInfo(...) Finds stream information from the media file.
Rational GuessFrameRate(...) Guesses the frame rate of a stream using an optional frame.
AVResult32 Seek(Rational time, int streamIndex) Seeks to a specific time in a given stream. Throws if index out of range.
AVResult32 Seek(long timestamp, int streamIndex) Seeks to a specific timestamp in a given stream.
AVResult32 ReadFrame(AVPacket packet) Reads a frame from the input media. Sets packet time base if not already set.
AVResult32 ReadPacket(AVPacket packet) Reads a frame from the input media. Sets packet time base if not already set.
Dispose() Disposes the context and underlying resources.

Usage Example

    TimeSpan seek = TimeSpan.FromSeconds(10);

    using var demuxerContext = DemuxerContext.Open(videoFile, true); // searchStreamInfo = true
    int videoIndex = context.FindBestStream(MediaType.Video); // find Best Video Stream
    
    // Discard all other streams
    foreach (var stream in context.Streams)
        stream.Discard = DiscardFlags.All;
    context.Streams[index].Discard = DiscardFlags.Default;
    
    // Open Decoder, with the correct parameters
    Codec c = Codec.FindDecoder(demuxerContext.Streams[videoIndex].CodecParameters.CodecId)!.Value;
    using CodecContext decoder = CodecContext.Open(c, demuxerContext.Streams[videoIndex].CodecParameters);
            decoder.TimeBase = input.Streams[videoIndex].TimeBase; // set codec TimeBase just to be sure


    using AVFrame frame = AVFrame.Allocate();
    using var packet = AVPacket.Allocate();
    var result = input.Seek(seek, videoIndex);
    if (result.IsError)
        return result;
    // decoder.FlushBuffers(); // we seeked so flush the codecs internal buffers if something was written to it
    do
    {
        do
        {
            result = input.ReadFrame(packet);
            if (result.IsError)
                return result;
            if(packet.Flags.HasFlag(PacketFlags.Discard) || packet.StreamIndex != videoIndex)
            {
                result = AVResult32.TryAgain;
                continue;
            }
            result = decoder.SendPacket(packet);
            if (result.IsError)
                return result;
            result = decoder.ReceiveFrame(frame);
        } while (result.IsTryAgain);
    } while (!result.IsError && (frame.GetPresentationTimestamp()+frame.Duration) * frame.TimeBase < seek);
    
    return frame;

CodecContext