Fix AudioFrameV3Wrapper::get_data() to return correct float32 audio data - #45
Open
ichux wants to merge 1 commit into
Open
Fix AudioFrameV3Wrapper::get_data() to return correct float32 audio data#45ichux wants to merge 1 commit into
ichux wants to merge 1 commit into
Conversation
## Problem
AudioFrameV3Wrapper::get_data() wraps NDI's float32 planar audio data
as a uint8 numpy array with incorrect strides, making all audio
received via framesync_capture_audio_v2() unusable in Python.
Specifically:
- The returned numpy array has dtype=uint8 and shape (channels, samples)
- tobytes() returns only 1/4 of the actual audio data (1024 bytes
instead of 4096 for a 1024-sample mono frame)
- The array's internal buffer is a COPY that is too small; reading
past it via ctypes returns garbage (uninitialized memory)
- All samples appear as NaN or silent corruption
## Root Cause
AudioFrameV3Wrapper::get_data() uses sizeof(uint8_t) as the element
size and format_descriptor<uint8_t> as the numpy format, but the
underlying NDI audio data is always FLTP (float planar, 32-bit).
Meanwhile, AudioFrameV2Wrapper::get_data() correctly uses
sizeof(float) and format_descriptor<float> — V3 is a copy-paste error.
## The Fix
Change AudioFrameV3Wrapper::get_data() to match AudioFrameV2Wrapper:
Before (buggy):
size = sizeof(uint8_t)
format_descriptor<uint8_t>
stride: {col * size * 4, size} // col * 4 bytes, 1 byte
After (fixed):
size = sizeof(float)
format_descriptor<float>
stride: {col * size, size} // col * 4 bytes, 4 bytes
This changes the numpy array dtype from uint8 to float32 and fixes
the stride calculation so the full audio data is accessible.
## Why This Is Correct
1. NDI SDK guarantees FLTP format: the FourCC field on AudioFrameV3
is always FOURCC_AUDIO_TYPE_FLTP (float planar). There are no
other audio FourCC types in the current SDK.
2. Matches AudioFrameV2Wrapper: the V2 wrapper already handles this
correctly. V3 should match.
3. Matches DistroAV (OBS NDI plugin): DistroAV's C++ code accesses
p_data directly as float* with channel_stride_in_bytes offsets —
confirming the data is float32.
4. Verified on real hardware: tested with NDI Camera Android app
sending mono 48kHz audio. After the fix, framesync_capture_audio_v2()
returns clean float32 data with shape (channels, samples),
dtype=float32, no NaN values, and correct amplitude range.
## Reproduction
Before fix:
import NDIlib as ndi
import numpy as np
ndi.initialize()
recv = ndi.recv_create_v3()
# ... connect to source ...
fs = ndi.framesync_create(recv)
af = ndi.framesync_capture_audio_v2(fs, 0, 0, 1024)
print(af.data.dtype) # uint8 (WRONG)
print(af.data.shape) # (1, 1024) — misleading, only 256 float32 samples fit
print(af.data.nbytes) # 1024 (WRONG, should be 4096)
print(np.any(np.isnan(af.data))) # True (WRONG)
After fix:
print(af.data.dtype) # float32 (CORRECT)
print(af.data.shape) # (1, 1024)
print(af.data.nbytes) # 4096 (CORRECT)
print(np.any(np.isnan(af.data))) # False (CORRECT)
print(af.data.max()) # ~0.002 (real audio from phone mic)
## Impact
- Breaking change: No. The array shape stays (channels, samples).
Only the dtype changes from uint8 to float32, which is the
correct representation of the data.
- Backward compatibility: Code that treated the array as raw bytes
via tobytes() will get different output, but that was already
broken (only returned 1/4 of the data).
- Performance: No change. Same memory allocation, same data copy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
AudioFrameV3Wrapper::get_data() wraps NDI's float32 planar audio data as a uint8 numpy array with incorrect strides, making all audio received via framesync_capture_audio_v2() unusable in Python.
Specifically:
Root Cause
AudioFrameV3Wrapper::get_data() uses sizeof(uint8_t) as the element size and format_descriptor<uint8_t> as the numpy format, but the underlying NDI audio data is always FLTP (float planar, 32-bit).
Meanwhile, AudioFrameV2Wrapper::get_data() correctly uses sizeof(float) and format_descriptor: V3 is a copy-paste error.
The Fix
Change AudioFrameV3Wrapper::get_data() to match AudioFrameV2Wrapper:
Before (buggy):
size = sizeof(uint8_t)
format_descriptor<uint8_t>
stride: {col * size * 4, size} // col * 4 bytes, 1 byte
After (fixed):
size = sizeof(float)
format_descriptor
stride: {col * size, size} // col * 4 bytes, 4 bytes
This changes the numpy array dtype from uint8 to float32 and fixes the stride calculation so the full audio data is accessible.
Why This Is Correct
NDI SDK guarantees FLTP format: the FourCC field on AudioFrameV3 is always FOURCC_AUDIO_TYPE_FLTP (float planar). There are no other audio FourCC types in the current SDK.
Matches AudioFrameV2Wrapper: the V2 wrapper already handles this correctly. V3 should match.
Matches DistroAV (OBS NDI plugin): DistroAV's C++ code accesses p_data directly as float* with channel_stride_in_bytes offsets: confirming the data is float32.
Verified on real hardware: tested with NDI Camera Android app sending mono 48kHz audio. After the fix, framesync_capture_audio_v2() returns clean float32 data with shape (channels, samples), dtype=float32, no NaN values, and correct amplitude range.
Reproduction
Before fix:
import NDIlib as ndi
import numpy as np
ndi.initialize()
recv = ndi.recv_create_v3()
... connect to source ...
fs = ndi.framesync_create(recv)
af = ndi.framesync_capture_audio_v2(fs, 0, 0, 1024)
print(af.data.dtype) # uint8 (WRONG)
print(af.data.shape) # (1, 1024) — misleading, only 256 float32 samples fit
print(af.data.nbytes) # 1024 (WRONG, should be 4096)
print(np.any(np.isnan(af.data))) # True (WRONG)
After fix:
print(af.data.dtype) # float32 (CORRECT)
print(af.data.shape) # (1, 1024)
print(af.data.nbytes) # 4096 (CORRECT)
print(np.any(np.isnan(af.data))) # False (CORRECT)
print(af.data.max()) # ~0.002 (real audio from phone mic)
Impact