AudioSource GetOutputData and GetSpectrumData Issues

[Ziugy]

Describe the bug

Output and spectrum data do not exist while running on HoloLens 2.

Sample of how we're using the two methods:

audioSource.GetOutputData(audioSamples, 0);
audioSource.GetSpectrumData(audioSpectrum, 0, FFTWindow.BlackmanHarris);

To reproduce

Steps to reproduce the behavior:

1. Have an AudioSource that's playing looping audio with all spatializer settings enabled.
2. Write a simple script that computes DB, like so: https://answers.unity.com/questions/157940/getoutputdata-and-getspectrumdata-they-represent-t.html
3. Verify this works with MS HRTF Spatializer on HoloLens 2.
4. Swap to Microsoft Spatializer and redeploy.
5. See that the results now on HoloLens 2 are not consistent.

Expected behavior

Behavior should be consistent with MS HRTF Spatializer and what's being seen in the Unity editor.

Your setup

• Unity Version 2019.2.21f1
• Microsoft Spatializer v1.0.0

Target platform

• HoloLens 2
• WMR immersive

[MikeChemi]

Hi @Ziugy , I just tried this out on my PC. I was able to repro the issue you reported. I belive that if you check the "Spatialize Post Effects" checkbox on the Audio Source component, you should see that the GetOutputData and GetSpectrumData calls are returning proper values again. Does this work for you as well?

[ashtat]

Hi @Ziugy, this behavior stems from the architectural difference between the two plugins.

In MS-HRTF plugin spatial audio signal processing occurs within the Unity audio graph itself using the CPU. This new plugin, on the other hand, relies on Windows platform APIs to enable hardware offload where the spatial audio processing is handled by the OS and that signal cannot make it's way back into the Unity audio graph. It's a trade-off with hardware offload where you get a higher source count at a lower CPU cost but you don't have access to the spatialized audio signal.

Hope this answers your question!

[ashtat]

Hi @Ziugy , I just tried this out on my PC. I was able to repro the issue you reported. I belive that if you check the "Spatialize Post Effects" checkbox on the Audio Source component, you should see that the GetOutputData and GetSpectrumData calls are returning proper values again. Does this work for you as well?

@MikeChemi @Ziugy Note that even with "Spatialize Post Effects" the signal available through GetOutputData and GetSepctrumData will be the non-directional wet path reverb signal only. Dry-path HRTF audio data does not make its way back into the Unity audio engine so there will still be a difference in behavior between MS HRTF and this plugin in this scenario.

[Ziugy]

I haven't had time to double back to this yet. When you say:

non-directional wet path reverb signal only

How will that effect the output we're using?

This caveat, gotcha, or additional steps (check Spatialize Post Effects when using GetOutputData/GetSpecturmData) should at least be added in this documentation: https://docs.microsoft.com/en-us/windows/mixed-reality/spatial-sound-in-unity