fishjam-cloud · MiloszFilimowski · Jul 15, 2026
diff --git a/.gitignore b/.gitignore
@@ -138,5 +138,7 @@ docs/api/mobile/**
 docs/api/web/**
 docs/api/server/**
 docs/api/server-python/**
+docs/api/custom-video-source/**
+docs/api/vision-camera-source/**
 
 .cursor/
diff --git a/docs/explanation/custom-sources.mdx b/docs/explanation/custom-sources.mdx
@@ -0,0 +1,82 @@
+---
+title: "How custom sources work"
+sidebar_position: 10
+description: Concepts behind Fishjam custom sources, including track metadata routing, the publish lifecycle, and the zero-copy React Native frame pipeline.
+---
+
+# How custom sources work
+
+[Custom sources](../how-to/client/custom-sources/index.mdx) let an app publish media that Fishjam did not capture itself. This page explains how custom tracks are modeled and routed, when they are published, and how the React Native frame pipeline moves frames to the encoder without copying pixels.
+
+## Built-in source, middleware, or custom source?
+
+Fishjam clients offer three ways to influence what a peer publishes:
+
+| Approach                                                          | What it does                                                         | When to use it                                                            |
+| ----------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------- |
+| Built-in sources (`useCamera`, `useMicrophone`, `useScreenShare`) | Capture and publish a device the SDK manages end-to-end              | Plain camera, microphone or screen sharing                                |
+| [Track middleware](../how-to/client/stream-middleware)            | Transforms a built-in track before it is sent (e.g. background blur) | You want the SDK's device handling, but with a processing step in between |
+| [Custom sources](../how-to/client/custom-sources/index.mdx)       | Publishes a `MediaStream` your app produced entirely on its own      | Content that isn't a managed device: renderers, compositors, ML pipelines |
+
+Capabilities differ slightly per platform:
+
+| Capability                             | Web                                | React Native                                        |
+| -------------------------------------- | ---------------------------------- | --------------------------------------------------- |
+| Publish any `MediaStream`              | ✅                                 | ✅                                                  |
+| Frame-level video injection            | via `canvas.captureStream()`       | ✅ native buffer forwarding / pooled render targets |
+| Custom audio                           | ✅ any audio track                 | ❌ no way to produce a custom audio track yet       |
+| GPU rendering into the published video | ✅ WebGPU/WebGL via canvas capture | ✅ WebGPU toolkit with zero-copy camera import      |
+
+## Streams and source IDs
+
+A custom source is a `MediaStream` registered under a stable **source ID** with `useCustomSource(sourceId)`. The ID makes the source addressable:
+
+- Every `useCustomSource` call with the same ID, in any component, returns the same state, so one component can produce the stream while another controls or renders it.
+- A peer can publish any number of custom sources, each under its own ID.
+
+When the stream is published, Fishjam takes its first video track and first audio track (whichever are present) and publishes them with the track metadata types `customVideo` and `customAudio`. On React Native only video is available today, because the platform offers no way to create a custom audio track. The metadata is how the receiving side tells custom tracks apart from camera, microphone and screen-share tracks: `usePeers` groups each peer's tracks by type and exposes custom tracks in the `customVideoTracks` and `customAudioTracks` arrays.
+
+## The publish lifecycle
+
+`setStream` is asynchronous and connection-aware:
+
+- Tracks are added to the session only while the peer is connected. A stream set before joining is stored as pending and published on connect.
+- On disconnect, sources return to pending; after a reconnect they are re-published automatically.
+- Calls to `setStream` are serialized: replacing one stream with another in quick succession cannot interleave, and a failed call does not affect subsequent ones.
+- `setStream(null)` unpublishes and forgets the source. The tracks themselves are not stopped; the producer of the stream owns them.
+
+## The React Native frame pipeline
+
+On the web, the browser can produce a `MediaStream` from a canvas, a media element or Web Audio, so the base hook is all you need. React Native has no such general facility, so the SDK provides one for video, built into `@fishjam-cloud/react-native-webrtc`:
+
+```mermaid
+flowchart LR
+  frames["Your frames<br/>(native buffers or<br/>rendered surfaces)"] --> track["Custom video track<br/>(react-native-webrtc)"]
+  track --> stream["MediaStream"]
+  stream --> hook["useCustomSource"]
+  hook --> room["Fishjam room"]
+  room --> peers["Other peers<br/>customVideoTracks"]
+```
+
+A **custom video track** looks like any other WebRTC video track to the rest of the stack, but its frames are supplied by your code through one of two modes:
+
+- **Forwarding**: you already have finished frames as native buffers (`CVPixelBufferRef` on iOS, `AHardwareBuffer*` on Android). `forwardFrame` passes the buffer pointer to the track; the SDK retains the buffer for as long as encoding needs it, so you can release your reference immediately.
+- **Render target (pooled)**: you draw the frames yourself. The SDK allocates a small pool of GPU-shareable surfaces (IOSurfaces on iOS, AHardwareBuffers on Android); you import a surface into your GPU once, render into it, and return it with `pushFrame`. A pool of about 3 surfaces keeps drawing and encoding overlapped: one surface can be encoding while the next is being drawn.
+
+Both paths are zero-copy: pixels stay in the same native memory from your producer to the video encoder, and only a pointer moves.
+
+The pipeline also handles:
+
+- **GPU synchronization.** With asynchronous renderers, a pushed surface may not be fully drawn when the encoder reads it. `pushFrame` accepts a _fence_ (an `MTLSharedEvent` on iOS, a sync file descriptor on Android) so the encoder waits for your GPU work. The WebGPU hooks manage this automatically.
+- **One timestamp domain.** Encoders require monotonically increasing timestamps. Forwarded frames without timestamps are stamped from the native monotonic clock, and the VisionCamera adapter normalizes VisionCamera's per-platform timestamp units onto one zero-based nanosecond timeline.
+- **Worklet-friendly handles.** Track handles are plain, worklet-serializable objects, so frames can be pushed straight from a frame-processor worklet without hopping through the JS thread. This synchronous JSI handoff is also why custom video tracks require the New Architecture.
+
+## Platform foundations
+
+- **iOS**: surfaces are IOSurface-backed, BGRA8 (`bgra8unorm` as a render target). The WebGPU camera import relies on Metal external-texture features that are only guaranteed on iOS 17+; publishing frames without WebGPU has no such requirement.
+- **Android**: surfaces are `AHardwareBuffer`s, RGBA8 (`rgba8unorm`). The camera arrives as an opaque YCbCr buffer, so the WebGPU toolkit's `sampleCamera` performs the BT.709 limited-range YUV→RGB conversion in-shader.
+
+## Where to go next
+
+- [Custom sources how-to guides](../how-to/client/custom-sources/index.mdx): publish from the web, React Native, Vision Camera, WebGPU, or your own pipeline
+- API reference: [`useCustomSource` (Web)](../api/web/functions/useCustomSource), [`useCustomSource` (React Native)](../api/mobile/functions/useCustomSource), [Vision Camera Source package](../api/vision-camera-source/index), [Custom Video Source package](../api/custom-video-source/index)
diff --git a/docs/how-to/client/custom-sources/_category_.json b/docs/how-to/client/custom-sources/_category_.json
@@ -0,0 +1,8 @@
+{
+  "label": "Custom sources",
+  "position": 8,
+  "link": {
+    "type": "doc",
+    "id": "how-to/client/custom-sources/index"
+  }
+}
diff --git a/docs/how-to/client/custom-sources/index.mdx b/docs/how-to/client/custom-sources/index.mdx
@@ -0,0 +1,115 @@
+---
+title: "Custom sources"
+description: Publish any video or audio content to a Fishjam room, from canvas and WebGPU on the web to Vision Camera and native frame pipelines in React Native.
+---
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+import DocCardList from "@theme/DocCardList";
+
+# Custom sources
+
+Custom sources allow you to publish media that doesn't come straight from the camera, microphone or screen share, for example a game render loop or an ML-processed camera feed. Any content you can produce as video frames or audio samples can be published.
+
+:::important
+
+If you only wish to send plain camera, microphone or screen share output through Fishjam, then you most likely should refer to [Streaming media](../start-streaming.mdx), [Managing devices](../managing-devices.mdx) and [Screen sharing](../screensharing.mdx) instead of this section.
+
+:::
+
+Every custom source follows the same contract: you register a `MediaStream` under a stable source ID with the [`useCustomSource`](../../../api/mobile/functions/useCustomSource.md) hook, and Fishjam publishes its video track as a `customVideo` track and its audio track as a `customAudio` track. Other peers receive them like any other track.
+
+## Choose a guide
+
+| Guide                                            | Platform           | Use it when                                                                                   |
+| ------------------------------------------------ | ------------------ | --------------------------------------------------------------------------------------------- |
+| [Web](./web.mdx)                                 | 🌐 Web             | You have a `MediaStream` from a canvas, a media element, Web Audio, or a library like Smelter |
+| [React Native](./react-native.mdx)               | 📱 Mobile          | You already have a React Native `MediaStream` to publish                                      |
+| [Vision Camera](./vision-camera.mdx)             | 📱 Mobile          | You want to publish a VisionCamera feed, optionally running frame processors on it            |
+| [WebGPU effects](./webgpu-effects.mdx)           | 🌐 Web · 📱 Mobile | You want to draw your own shaders, overlays or effects into the published video               |
+| [Low-level frame API](./low-level-frame-api.mdx) | 📱 Mobile          | You produce video frames yourself, e.g. from a native pipeline or your own renderer           |
+
+:::info[Platform support]
+
+On the web you can publish both video and audio from any `MediaStream`. React Native custom sources are video-only today. The platform has no way to produce a custom audio track, so custom audio remains a web-only capability. See [What about audio?](./react-native.mdx#what-about-audio).
+
+:::
+
+## Receiving custom tracks
+
+Custom tracks published by other peers show up in the [`usePeers`](../../../api/mobile/functions/usePeers.md) hook, bucketed into the `customVideoTracks` and `customAudioTracks` arrays of each peer.
+
+<Tabs groupId="platform">
+  <TabItem value="web" label="React (Web)">
+
+```tsx
+import React, { FC } from "react";
+
+const VideoRenderer: FC<{ stream?: MediaStream | null }> = (_) => <video />;
+
+// ---cut---
+import { usePeers } from "@fishjam-cloud/react-client";
+
+export function RemoteCustomVideos() {
+  const { remotePeers } = usePeers();
+
+  return (
+    <ul>
+      {remotePeers.map((peer) => (
+        <li key={peer.id}>
+          {peer.customVideoTracks.map((track) => (
+            <VideoRenderer key={track.trackId} stream={track.stream} />
+          ))}
+        </li>
+      ))}
+    </ul>
+  );
+}
+```
+
+Remote `customAudioTracks` are played by attaching each track's `stream` to an `<audio>` element, the same way as microphone tracks.
+
+  </TabItem>
+  <TabItem value="mobile" label="React Native (Mobile)">
+
+```tsx
+import React from "react";
+import { View } from "react-native";
+import { usePeers, RTCView } from "@fishjam-cloud/react-native-client";
+
+export function RemoteCustomVideos() {
+  const { remotePeers } = usePeers();
+
+  return (
+    <View>
+      {remotePeers.map((peer) =>
+        peer.customVideoTracks.map(
+          (track) =>
+            track.stream && (
+              <RTCView
+                key={track.trackId}
+                mediaStream={track.stream}
+                style={{ height: 200, width: "100%" }}
+                objectFit="cover"
+              />
+            ),
+        ),
+      )}
+    </View>
+  );
+}
+```
+
+Incoming `customAudioTracks` are played back automatically; you don't render anything for them.
+
+  </TabItem>
+</Tabs>
+
+## Guides in this section
+
+<DocCardList />
+
+## Further reading
+
+- [How custom sources work](../../../explanation/custom-sources.mdx): concepts behind the pipeline
+- API reference: [`useCustomSource` (Web)](../../../api/web/functions/useCustomSource.md), [`useCustomSource` (React Native)](../../../api/mobile/functions/useCustomSource.md), [Vision Camera Source package](../../../api/vision-camera-source/index.md), [Custom Video Source package](../../../api/custom-video-source/index.md)