Skip to content

Fix AudioFrameV3Wrapper::get_data() to return correct float32 audio data - #45

Open
ichux wants to merge 1 commit into
buresu:masterfrom
ichux:fix-audio-frame-v3-get-data
Open

Fix AudioFrameV3Wrapper::get_data() to return correct float32 audio data#45
ichux wants to merge 1 commit into
buresu:masterfrom
ichux:fix-audio-frame-v3-get-data

Conversation

@ichux

@ichux ichux commented Sep 1, 2026

Copy link
Copy Markdown

Problem

AudioFrameV3Wrapper::get_data() wraps NDI's float32 planar audio data as a uint8 numpy array with incorrect strides, making all audio received via framesync_capture_audio_v2() unusable in Python.

Specifically:

  • The returned numpy array has dtype=uint8 and shape (channels, samples)
  • tobytes() returns only 1/4 of the actual audio data (1024 bytes instead of 4096 for a 1024-sample mono frame)
  • The array's internal buffer is a COPY that is too small; reading past it via ctypes returns garbage (uninitialized memory)
  • All samples appear as NaN or silent corruption

Root Cause

AudioFrameV3Wrapper::get_data() uses sizeof(uint8_t) as the element size and format_descriptor<uint8_t> as the numpy format, but the underlying NDI audio data is always FLTP (float planar, 32-bit).

Meanwhile, AudioFrameV2Wrapper::get_data() correctly uses sizeof(float) and format_descriptor: V3 is a copy-paste error.

The Fix

Change AudioFrameV3Wrapper::get_data() to match AudioFrameV2Wrapper:

Before (buggy):
size = sizeof(uint8_t)
format_descriptor<uint8_t>
stride: {col * size * 4, size} // col * 4 bytes, 1 byte

After (fixed):
size = sizeof(float)
format_descriptor
stride: {col * size, size} // col * 4 bytes, 4 bytes

This changes the numpy array dtype from uint8 to float32 and fixes the stride calculation so the full audio data is accessible.

Why This Is Correct

  1. NDI SDK guarantees FLTP format: the FourCC field on AudioFrameV3 is always FOURCC_AUDIO_TYPE_FLTP (float planar). There are no other audio FourCC types in the current SDK.

  2. Matches AudioFrameV2Wrapper: the V2 wrapper already handles this correctly. V3 should match.

  3. Matches DistroAV (OBS NDI plugin): DistroAV's C++ code accesses p_data directly as float* with channel_stride_in_bytes offsets: confirming the data is float32.

  4. Verified on real hardware: tested with NDI Camera Android app sending mono 48kHz audio. After the fix, framesync_capture_audio_v2() returns clean float32 data with shape (channels, samples), dtype=float32, no NaN values, and correct amplitude range.

Reproduction

Before fix:
import NDIlib as ndi
import numpy as np
ndi.initialize()
recv = ndi.recv_create_v3()

... connect to source ...

fs = ndi.framesync_create(recv)
af = ndi.framesync_capture_audio_v2(fs, 0, 0, 1024)
print(af.data.dtype) # uint8 (WRONG)
print(af.data.shape) # (1, 1024) — misleading, only 256 float32 samples fit
print(af.data.nbytes) # 1024 (WRONG, should be 4096)
print(np.any(np.isnan(af.data))) # True (WRONG)

After fix:
print(af.data.dtype) # float32 (CORRECT)
print(af.data.shape) # (1, 1024)
print(af.data.nbytes) # 4096 (CORRECT)
print(np.any(np.isnan(af.data))) # False (CORRECT)
print(af.data.max()) # ~0.002 (real audio from phone mic)

Impact

  • Breaking change: No. The array shape stays (channels, samples). Only the dtype changes from uint8 to float32, which is the correct representation of the data.
  • Backward compatibility: Code that treated the array as raw bytes via tobytes() will get different output, but that was already broken (only returned 1/4 of the data).
  • Performance: No change. Same memory allocation, same data copy.

## Problem

AudioFrameV3Wrapper::get_data() wraps NDI's float32 planar audio data
as a uint8 numpy array with incorrect strides, making all audio
received via framesync_capture_audio_v2() unusable in Python.

Specifically:
- The returned numpy array has dtype=uint8 and shape (channels, samples)
- tobytes() returns only 1/4 of the actual audio data (1024 bytes
  instead of 4096 for a 1024-sample mono frame)
- The array's internal buffer is a COPY that is too small; reading
  past it via ctypes returns garbage (uninitialized memory)
- All samples appear as NaN or silent corruption

## Root Cause

AudioFrameV3Wrapper::get_data() uses sizeof(uint8_t) as the element
size and format_descriptor<uint8_t> as the numpy format, but the
underlying NDI audio data is always FLTP (float planar, 32-bit).

Meanwhile, AudioFrameV2Wrapper::get_data() correctly uses
sizeof(float) and format_descriptor<float> — V3 is a copy-paste error.

## The Fix

Change AudioFrameV3Wrapper::get_data() to match AudioFrameV2Wrapper:

  Before (buggy):
    size = sizeof(uint8_t)
    format_descriptor<uint8_t>
    stride: {col * size * 4, size}   // col * 4 bytes, 1 byte

  After (fixed):
    size = sizeof(float)
    format_descriptor<float>
    stride: {col * size, size}       // col * 4 bytes, 4 bytes

This changes the numpy array dtype from uint8 to float32 and fixes
the stride calculation so the full audio data is accessible.

## Why This Is Correct

1. NDI SDK guarantees FLTP format: the FourCC field on AudioFrameV3
   is always FOURCC_AUDIO_TYPE_FLTP (float planar). There are no
   other audio FourCC types in the current SDK.

2. Matches AudioFrameV2Wrapper: the V2 wrapper already handles this
   correctly. V3 should match.

3. Matches DistroAV (OBS NDI plugin): DistroAV's C++ code accesses
   p_data directly as float* with channel_stride_in_bytes offsets —
   confirming the data is float32.

4. Verified on real hardware: tested with NDI Camera Android app
   sending mono 48kHz audio. After the fix, framesync_capture_audio_v2()
   returns clean float32 data with shape (channels, samples),
   dtype=float32, no NaN values, and correct amplitude range.

## Reproduction

Before fix:
  import NDIlib as ndi
  import numpy as np
  ndi.initialize()
  recv = ndi.recv_create_v3()
  # ... connect to source ...
  fs = ndi.framesync_create(recv)
  af = ndi.framesync_capture_audio_v2(fs, 0, 0, 1024)
  print(af.data.dtype)     # uint8 (WRONG)
  print(af.data.shape)     # (1, 1024) — misleading, only 256 float32 samples fit
  print(af.data.nbytes)    # 1024 (WRONG, should be 4096)
  print(np.any(np.isnan(af.data)))  # True (WRONG)

After fix:
  print(af.data.dtype)     # float32 (CORRECT)
  print(af.data.shape)     # (1, 1024)
  print(af.data.nbytes)    # 4096 (CORRECT)
  print(np.any(np.isnan(af.data)))  # False (CORRECT)
  print(af.data.max())     # ~0.002 (real audio from phone mic)

## Impact

- Breaking change: No. The array shape stays (channels, samples).
  Only the dtype changes from uint8 to float32, which is the
  correct representation of the data.
- Backward compatibility: Code that treated the array as raw bytes
  via tobytes() will get different output, but that was already
  broken (only returned 1/4 of the data).
- Performance: No change. Same memory allocation, same data copy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant