Skip to content

Native heap corruption (Scudo double free) in libjingle_peerconnection_so during setRemoteDescription while AudioRecord is capturing (2.27.0 and 2.28.2) #1030

Description

@sharibeloelo

Describe the bug

Native crash in production (Crashlytics) during audio-only calls: a heap corruption, meaning a use-after-free or double free, inside libjingle_peerconnection_so.so. Scudo aborts in free() on a WebRTC internal thread. At the same moment, the AudioRecord thread is inside WebRtcAudioRecord.nativeDataIsRecorded, and another thread is inside PeerConnection.nativeSetRemoteDescription.

It reproduces on both LiveKit 2.27.0 and 2.28.2, so upgrading did not fix it. Scale: 745 events from 725 users in 90 days.

Tombstone (aborting thread), from a 2.28.2 event

Scudo ERROR: corrupted chunk header at address 0x2000072bbbe5c90: chunk header is zero and might indicate memory corruption or a double free
#00 pc 0xb90d0  abort (libc.so)
#01 pc 0x6c8cc  scudo::die()
#02 pc 0x6d31c  scudo::reportRawError(char const*)
#03 pc 0x6d290  scudo::ScopedErrorReport::~ScopedErrorReport()
#04 pc 0x6d478  scudo::reportHeaderCorruption(void*, void*)
#05 pc 0x6edd8  scudo::Allocator<scudo::AndroidNormalConfig, &scudo_malloc_postinit>::deallocate(...)
#06 pc 0x918954 libjingle_peerconnection_so.so
#07 pc 0x915bb4 libjingle_peerconnection_so.so
#08 pc 0x91541c libjingle_peerconnection_so.so
#09 pc 0x914b54 libjingle_peerconnection_so.so
#10 pc 0x90b08c libjingle_peerconnection_so.so
#11 pc 0x8eacb4 libjingle_peerconnection_so.so
#12 pc 0x3b3228 libjingle_peerconnection_so.so   (rtc::Thread run loop)
#13 pc 0x3b226c libjingle_peerconnection_so.so
#14 pc 0x3b3698 libjingle_peerconnection_so.so
#15 pc 0xc9df0  __pthread_start
#16 pc 0xbc4bc  __start_thread

BuildId of libjingle_peerconnection_so.so: 8fb495ec1c3cc305 (webrtc-sdk android-prefixed 144.7559.14)

Crashlytics "crashed" thread (the audio record thread)

SIGSEGV 0xb4000072bbc15000
#00 pc 0x42a348 libjingle_peerconnection_so.so
#01 pc 0x42a344 libjingle_peerconnection_so.so
#02 pc 0x4285e4 libjingle_peerconnection_so.so
#03 pc 0x42854c libjingle_peerconnection_so.so
#04 pc 0x427e24 libjingle_peerconnection_so.so
#05 pc 0xa0c0f0 libjingle_peerconnection_so.so
#06 pc 0xa025c0 libjingle_peerconnection_so.so
#07 pc 0x5edeb8 libjingle_peerconnection_so.so
#08 pc 0x86bc48 libjingle_peerconnection_so.so   (near Java_livekit_org_webrtc_audio_WebRtcAudioRecord_nativeDataIsRecorded)
#09 pc 0x72ee77a0  (Java frame: WebRtcAudioRecord$AudioRecordThread)

Other relevant thread at crash time

#03 pc 0x3b682c libjingle_peerconnection_so.so   (waiting on rtc::Event)
#04 pc 0x3b3a60 libjingle_peerconnection_so.so
#05 pc 0x8dd7b8 libjingle_peerconnection_so.so
#06 pc 0x892344 libjingle_peerconnection_so.so
#07 pc 0x87cb4c libjingle_peerconnection_so.so   (Java_livekit_org_webrtc_PeerConnection_nativeSetRemoteDescription)
#08 pc 0x72eea36c

Note: the .so is stripped. Crashlytics labels frames with the nearest exported JNI symbol, such as BuiltinAudioEncoderFactoryFactory_nativeCreateBuiltinAudioEncoderFactory or TurnCustomizer_nativeFreeTurnCustomizer. Those labels are misleading. The raw PCs and the BuildId above should symbolize against your unstripped build of 144.7559.14.

Suspected cause

A race between the capture path and renegotiation. The AudioRecord thread delivers 10 ms frames through nativeDataIsRecorded into the audio transport, then the send stream, APM/encoder, and the Krisp filter. At the same time, setRemoteDescription runs on the signaling thread. That call reconfigures or recreates the audio send stream or encoder on the worker thread and frees objects that the capture thread is still using. The worker thread then frees an already-freed or overwritten chunk, and Scudo aborts.

To Reproduce

We can't reproduce it on demand. It happens in production during active foreground calls, most likely while a renegotiation (a remote SDP update) arrives mid-call. Our app also handles transient audio focus loss during calls (it mutes and unmutes the mic track, and the AudioRecord restarts), which may make the window larger.

Expected behavior

No native heap corruption during renegotiation while the mic is capturing.

Screenshots

N/A. Full tombstone and thread dump are above.

Device Info:

  • Device: mostly budget/mid-range OEM devices: Vivo 46%, Xiaomi 20%, Oppo 18%, Realme 5%, other 10%
  • OS: Android 16 is 56% of events; also seen on Android 12
  • LiveKit SDK version: reproduces on both of these:
    • 2.27.0 (webrtc-sdk android-prefixed 144.7559.09): 739 events
    • 2.28.2 (webrtc-sdk android-prefixed 144.7559.14): 6 events so far (newer release). The tombstone above is from this version, BuildId 8fb495ec1c3cc305
  • krisp-noise-filter 0.0.15 on both. ABI arm64-v8a. Audio only (no video tracks).
  • 0% of events happened while the app was in the background, so these calls were in the foreground.

Additional context

  1. Can you symbolize the PCs above against the 144.7559.14 build (BuildId 8fb495ec1c3cc305)?
  2. Is this a known issue in these WebRTC revisions, or in the Krisp audio processing integration?
  3. Is there a recommended workaround in the meantime, for example serializing mic mute/unmute with renegotiation, or disabling Krisp?

Crashlytics issue: 03c9dc9f88fc6ba514302d1571ebed6b

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions