diff --git a/.gitignore b/.gitignore index e867858d..c6065300 100644 --- a/.gitignore +++ b/.gitignore @@ -138,5 +138,7 @@ docs/api/mobile/** docs/api/web/** docs/api/server/** docs/api/server-python/** +docs/api/custom-video-source/** +docs/api/vision-camera-source/** .cursor/ diff --git a/docs/explanation/custom-sources.mdx b/docs/explanation/custom-sources.mdx new file mode 100644 index 00000000..6f1919ee --- /dev/null +++ b/docs/explanation/custom-sources.mdx @@ -0,0 +1,82 @@ +--- +title: "How custom sources work" +sidebar_position: 10 +description: Concepts behind Fishjam custom sources, including track metadata routing, the publish lifecycle, and the zero-copy React Native frame pipeline. +--- + +# How custom sources work + +[Custom sources](../how-to/client/custom-sources/index.mdx) let an app publish media that Fishjam did not capture itself. This page explains how custom tracks are modeled and routed, when they are published, and how the React Native frame pipeline moves frames to the encoder without copying pixels. + +## Built-in source, middleware, or custom source? + +Fishjam clients offer three ways to influence what a peer publishes: + +| Approach | What it does | When to use it | +| ----------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------- | +| Built-in sources (`useCamera`, `useMicrophone`, `useScreenShare`) | Capture and publish a device the SDK manages end-to-end | Plain camera, microphone or screen sharing | +| [Track middleware](../how-to/client/stream-middleware) | Transforms a built-in track before it is sent (e.g. background blur) | You want the SDK's device handling, but with a processing step in between | +| [Custom sources](../how-to/client/custom-sources/index.mdx) | Publishes a `MediaStream` your app produced entirely on its own | Content that isn't a managed device: renderers, compositors, ML pipelines | + +Capabilities differ slightly per platform: + +| Capability | Web | React Native | +| -------------------------------------- | ---------------------------------- | --------------------------------------------------- | +| Publish any `MediaStream` | ✅ | ✅ | +| Frame-level video injection | via `canvas.captureStream()` | ✅ native buffer forwarding / pooled render targets | +| Custom audio | ✅ any audio track | ❌ no way to produce a custom audio track yet | +| GPU rendering into the published video | ✅ WebGPU/WebGL via canvas capture | ✅ WebGPU toolkit with zero-copy camera import | + +## Streams and source IDs + +A custom source is a `MediaStream` registered under a stable **source ID** with `useCustomSource(sourceId)`. The ID makes the source addressable: + +- Every `useCustomSource` call with the same ID, in any component, returns the same state, so one component can produce the stream while another controls or renders it. +- A peer can publish any number of custom sources, each under its own ID. + +When the stream is published, Fishjam takes its first video track and first audio track (whichever are present) and publishes them with the track metadata types `customVideo` and `customAudio`. On React Native only video is available today, because the platform offers no way to create a custom audio track. The metadata is how the receiving side tells custom tracks apart from camera, microphone and screen-share tracks: `usePeers` groups each peer's tracks by type and exposes custom tracks in the `customVideoTracks` and `customAudioTracks` arrays. + +## The publish lifecycle + +`setStream` is asynchronous and connection-aware: + +- Tracks are added to the session only while the peer is connected. A stream set before joining is stored as pending and published on connect. +- On disconnect, sources return to pending; after a reconnect they are re-published automatically. +- Calls to `setStream` are serialized: replacing one stream with another in quick succession cannot interleave, and a failed call does not affect subsequent ones. +- `setStream(null)` unpublishes and forgets the source. The tracks themselves are not stopped; the producer of the stream owns them. + +## The React Native frame pipeline + +On the web, the browser can produce a `MediaStream` from a canvas, a media element or Web Audio, so the base hook is all you need. React Native has no such general facility, so the SDK provides one for video, built into `@fishjam-cloud/react-native-webrtc`: + +```mermaid +flowchart LR + frames["Your frames
(native buffers or
rendered surfaces)"] --> track["Custom video track
(react-native-webrtc)"] + track --> stream["MediaStream"] + stream --> hook["useCustomSource"] + hook --> room["Fishjam room"] + room --> peers["Other peers
customVideoTracks"] +``` + +A **custom video track** looks like any other WebRTC video track to the rest of the stack, but its frames are supplied by your code through one of two modes: + +- **Forwarding**: you already have finished frames as native buffers (`CVPixelBufferRef` on iOS, `AHardwareBuffer*` on Android). `forwardFrame` passes the buffer pointer to the track; the SDK retains the buffer for as long as encoding needs it, so you can release your reference immediately. +- **Render target (pooled)**: you draw the frames yourself. The SDK allocates a small pool of GPU-shareable surfaces (IOSurfaces on iOS, AHardwareBuffers on Android); you import a surface into your GPU once, render into it, and return it with `pushFrame`. A pool of about 3 surfaces keeps drawing and encoding overlapped: one surface can be encoding while the next is being drawn. + +Both paths are zero-copy: pixels stay in the same native memory from your producer to the video encoder, and only a pointer moves. + +The pipeline also handles: + +- **GPU synchronization.** With asynchronous renderers, a pushed surface may not be fully drawn when the encoder reads it. `pushFrame` accepts a _fence_ (an `MTLSharedEvent` on iOS, a sync file descriptor on Android) so the encoder waits for your GPU work. The WebGPU hooks manage this automatically. +- **One timestamp domain.** Encoders require monotonically increasing timestamps. Forwarded frames without timestamps are stamped from the native monotonic clock, and the VisionCamera adapter normalizes VisionCamera's per-platform timestamp units onto one zero-based nanosecond timeline. +- **Worklet-friendly handles.** Track handles are plain, worklet-serializable objects, so frames can be pushed straight from a frame-processor worklet without hopping through the JS thread. This synchronous JSI handoff is also why custom video tracks require the New Architecture. + +## Platform foundations + +- **iOS**: surfaces are IOSurface-backed, BGRA8 (`bgra8unorm` as a render target). The WebGPU camera import relies on Metal external-texture features that are only guaranteed on iOS 17+; publishing frames without WebGPU has no such requirement. +- **Android**: surfaces are `AHardwareBuffer`s, RGBA8 (`rgba8unorm`). The camera arrives as an opaque YCbCr buffer, so the WebGPU toolkit's `sampleCamera` performs the BT.709 limited-range YUV→RGB conversion in-shader. + +## Where to go next + +- [Custom sources how-to guides](../how-to/client/custom-sources/index.mdx): publish from the web, React Native, Vision Camera, WebGPU, or your own pipeline +- API reference: [`useCustomSource` (Web)](../api/web/functions/useCustomSource), [`useCustomSource` (React Native)](../api/mobile/functions/useCustomSource), [Vision Camera Source package](../api/vision-camera-source/index), [Custom Video Source package](../api/custom-video-source/index) diff --git a/docs/how-to/client/custom-sources/_category_.json b/docs/how-to/client/custom-sources/_category_.json new file mode 100644 index 00000000..2eaff9da --- /dev/null +++ b/docs/how-to/client/custom-sources/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Custom sources", + "position": 8, + "link": { + "type": "doc", + "id": "how-to/client/custom-sources/index" + } +} diff --git a/docs/how-to/client/custom-sources/index.mdx b/docs/how-to/client/custom-sources/index.mdx new file mode 100644 index 00000000..1c0add58 --- /dev/null +++ b/docs/how-to/client/custom-sources/index.mdx @@ -0,0 +1,115 @@ +--- +title: "Custom sources" +description: Publish any video or audio content to a Fishjam room, from canvas and WebGPU on the web to Vision Camera and native frame pipelines in React Native. +--- + +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; +import DocCardList from "@theme/DocCardList"; + +# Custom sources + +Custom sources allow you to publish media that doesn't come straight from the camera, microphone or screen share, for example a game render loop or an ML-processed camera feed. Any content you can produce as video frames or audio samples can be published. + +:::important + +If you only wish to send plain camera, microphone or screen share output through Fishjam, then you most likely should refer to [Streaming media](../start-streaming.mdx), [Managing devices](../managing-devices.mdx) and [Screen sharing](../screensharing.mdx) instead of this section. + +::: + +Every custom source follows the same contract: you register a `MediaStream` under a stable source ID with the [`useCustomSource`](../../../api/mobile/functions/useCustomSource.md) hook, and Fishjam publishes its video track as a `customVideo` track and its audio track as a `customAudio` track. Other peers receive them like any other track. + +## Choose a guide + +| Guide | Platform | Use it when | +| ------------------------------------------------ | ------------------ | --------------------------------------------------------------------------------------------- | +| [Web](./web.mdx) | 🌐 Web | You have a `MediaStream` from a canvas, a media element, Web Audio, or a library like Smelter | +| [React Native](./react-native.mdx) | 📱 Mobile | You already have a React Native `MediaStream` to publish | +| [Vision Camera](./vision-camera.mdx) | 📱 Mobile | You want to publish a VisionCamera feed, optionally running frame processors on it | +| [WebGPU effects](./webgpu-effects.mdx) | 🌐 Web · 📱 Mobile | You want to draw your own shaders, overlays or effects into the published video | +| [Low-level frame API](./low-level-frame-api.mdx) | 📱 Mobile | You produce video frames yourself, e.g. from a native pipeline or your own renderer | + +:::info[Platform support] + +On the web you can publish both video and audio from any `MediaStream`. React Native custom sources are video-only today. The platform has no way to produce a custom audio track, so custom audio remains a web-only capability. See [What about audio?](./react-native.mdx#what-about-audio). + +::: + +## Receiving custom tracks + +Custom tracks published by other peers show up in the [`usePeers`](../../../api/mobile/functions/usePeers.md) hook, bucketed into the `customVideoTracks` and `customAudioTracks` arrays of each peer. + + + + +```tsx +import React, { FC } from "react"; + +const VideoRenderer: FC<{ stream?: MediaStream | null }> = (_) => + + +```tsx +import React from "react"; +import { View } from "react-native"; +import { usePeers, RTCView } from "@fishjam-cloud/react-native-client"; + +export function RemoteCustomVideos() { + const { remotePeers } = usePeers(); + + return ( + + {remotePeers.map((peer) => + peer.customVideoTracks.map( + (track) => + track.stream && ( + + ), + ), + )} + + ); +} +``` + +Incoming `customAudioTracks` are played back automatically; you don't render anything for them. + + + + +## Guides in this section + + + +## Further reading + +- [How custom sources work](../../../explanation/custom-sources.mdx): concepts behind the pipeline +- API reference: [`useCustomSource` (Web)](../../../api/web/functions/useCustomSource.md), [`useCustomSource` (React Native)](../../../api/mobile/functions/useCustomSource.md), [Vision Camera Source package](../../../api/vision-camera-source/index.md), [Custom Video Source package](../../../api/custom-video-source/index.md) diff --git a/docs/how-to/client/custom-sources/low-level-frame-api.mdx b/docs/how-to/client/custom-sources/low-level-frame-api.mdx new file mode 100644 index 00000000..ecc0195d --- /dev/null +++ b/docs/how-to/client/custom-sources/low-level-frame-api.mdx @@ -0,0 +1,199 @@ +--- +title: "Publish frames with the low-level API" +sidebar_position: 5 +sidebar_label: "Low-level frame API 📱" +description: Publish video frames from any source, such as a native ML pipeline or your own renderer, with the generic custom video source layer. +--- + +# Publish frames with the low-level API Mobile + +:::note +This guide is exclusively for **Mobile** (React Native) applications. +::: + +`@fishjam-cloud/react-native-custom-video-source` is the generic layer under the [Vision Camera integration](./vision-camera). Use it directly when your frames come from another source, such as a native ML pipeline or your own renderer. It is built on the raw track primitives of `@fishjam-cloud/react-native-webrtc`, covered [at the end of this guide](#going-lower-raw-track-primitives). + +## Pick a mode + +The two modes differ in how you produce frames: + +| Mode | Hook | Use it when | +| -------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------- | +| **Forwarding** | `useManagedForwardTrack` | You already have finished native buffers (`CVPixelBufferRef` on iOS, `AHardwareBuffer*` on Android) | +| **Render-target (pooled)** | `useManagedPooledTrack` | You render the frames yourself; the hook allocates GPU-shareable surfaces for you to draw into | + +Each hook manages the track's asynchronous lifecycle: creation, the published `stream`, and teardown. Your code only supplies frames. + +## Prerequisites + +- `@fishjam-cloud/react-native-webrtc` and `@fishjam-cloud/react-native-client`, with your app wrapped in `FishjamProvider` +- The New Architecture; custom video tracks require it and reject with a clear error on the old architecture + +```bash npm2yarn +npm install @fishjam-cloud/react-native-custom-video-source +``` + +## Publish the managed stream + +Both hooks return a `stream`. Publish it with [`useCustomSource`](./react-native), like any other custom source: + +```tsx +import { useEffect } from "react"; +import { + useCustomSource, + type MediaStream, +} from "@fishjam-cloud/react-native-client"; + +// ---cut--- +export function usePublishStream(sourceId: string, stream: MediaStream | null) { + const { setStream } = useCustomSource(sourceId); + + useEffect(() => { + if (!stream) return; + setStream(stream); + return () => { + setStream(null); + }; + }, [stream, setStream]); +} +``` + +## Forward finished buffers + +`useManagedForwardTrack` creates and owns the track; hand each buffer pointer to `forwardFrame`: + +```tsx +import { useManagedForwardTrack } from "@fishjam-cloud/react-native-custom-video-source"; +import { forwardFrame } from "@fishjam-cloud/react-native-webrtc"; + +declare function getNextFrameBuffer(): bigint; // your native pipeline + +// ---cut--- +const { track, stream, error } = useManagedForwardTrack(true /* enabled */); + +// publish `stream` via useCustomSource, then per frame: +if (track) { + forwardFrame(track, { + nativeBuffer: getNextFrameBuffer(), + }); +} +``` + +Buffer requirements: + +- `nativeBuffer` is a retainable, IOSurface-backed `CVPixelBufferRef` (iOS) or an `AHardwareBuffer*` (Android), passed as a `bigint` pointer. +- The SDK retains the buffer for encoding; you may release your reference immediately after `forwardFrame` returns. +- `timestampNs` is optional: when omitted, frames are stamped with the native monotonic clock. Pass your own capture timestamps only if you need to align the video with an audio track you also produce. +- `rotation` (`0 | 90 | 180 | 270`) rotates the frame for the receiver. Don't set it if your buffers are already upright, or the frame gets rotated twice. + +:::warning[One worklet runtime per track] + +`forwardFrame` and `pushFrame` are worklet-safe, but a track's sink binds to the first worklet runtime that touches it. Feed each track from a single runtime (or from the JS thread) consistently. + +::: + +## Render into pooled surfaces + +`useManagedPooledTrack` allocates a pool of native GPU-shareable surfaces; draw into a slot and hand it back with `pushFrame`: + +```tsx +import { useManagedPooledTrack } from "@fishjam-cloud/react-native-custom-video-source"; +import { pushFrame } from "@fishjam-cloud/react-native-webrtc"; + +declare function renderInto(surfaceHandle: bigint): void; // your renderer +declare function nowNanoseconds(): number; + +// ---cut--- +const { track, stream, bufferDescriptors, error } = useManagedPooledTrack( + true, // enabled + 720, // width + 1280, // height + 3, // poolSize +); + +let nextSlot = 0; + +function renderFrame() { + if (!track || !bufferDescriptors) return; + + const descriptor = bufferDescriptors[nextSlot]; + nextSlot = (nextSlot + 1) % bufferDescriptors.length; + + renderInto(descriptor.surfaceHandle); + pushFrame(track, { + bufferIndex: descriptor.index, + timestampNs: nowNanoseconds(), + }); +} +``` + +Each `WorkletBufferDescriptor` carries `{ index, surfaceHandle, width, height }`. Import each `surfaceHandle` into your GPU once and cache the result per index; don't re-import it every frame. + +The surfaces are: + +| Platform | Surface type | Pixel format | Import as | +| -------- | ----------------- | ------------ | ------------ | +| iOS | `IOSurface` | BGRA8 | `bgra8unorm` | +| Android | `AHardwareBuffer` | RGBA8 | `rgba8unorm` | + +Rendering with the wrong format typically shows up as swapped red/blue channels, or the import is rejected outright. + +Unlike forwarding, `timestampNs` is required here and must be strictly increasing. + +### Synchronize with your GPU + +If your renderer works asynchronously, pass a `fence` so the encoder waits for your rendering to finish instead of reading a half-drawn surface: + +- **iOS**: `handle` is an `MTLSharedEvent` pointer and `signaledValue` the value your GPU work signals. +- **Android**: `handle` is a sync file descriptor; pass `0n` as `signaledValue`. + +If you render with WebGPU, prefer [`useVisionCameraWebGpuSource`](./webgpu-effects) or the `/webgpu` toolkit of this package; they handle fencing, surface import, and frame lifetimes for you. + +## Lifecycle and errors + +- `track`, `stream` and `bufferDescriptors` are `null` until asynchronous creation finishes; failures surface in the hook's `error` field rather than throwing. +- Set `enabled: false` to tear everything down; the hooks stop the tracks (and dispose the pool) in the correct order on unmount too. +- Changing the pooled dimensions reallocates the pool, so release any GPU objects you imported when `bufferDescriptors` changes. + +## Going lower: raw track primitives + +The managed hooks wrap four primitives from `@fishjam-cloud/react-native-webrtc`: `createCustomVideoBufferPool`, `createCustomVideoTrack`, `pushFrame` and `forwardFrame`. Use them directly only when a React hook cannot own the lifecycle, for example inside a headless native-driven runtime: + +```tsx +import { + createCustomVideoBufferPool, + createCustomVideoTrack, +} from "@fishjam-cloud/react-native-webrtc"; + +// ---cut--- +const pool = await createCustomVideoBufferPool({ + width: 720, + height: 1280, + poolSize: 3, +}); +const { stream, track } = await createCustomVideoTrack({ pool }); + +// render into pool.buffers[i].surfaceHandle, then pushFrame(track, ...) + +// Teardown; order matters: +stream.getTracks().forEach((mediaStreamTrack) => mediaStreamTrack.stop()); +await pool.dispose(); +``` + +When using the primitives directly, you must follow the rules the hooks otherwise enforce: + +- A pool binds to exactly one track. +- Stop the track's stream before disposing the pool; `dispose()` rejects while the track is live. +- Stopping a track never frees the pool; you own it and must dispose it yourself. +- `createCustomVideoTrack()` without a pool creates a forwarding track for `forwardFrame`. + +## WebGPU render targets + +The `/webgpu` entry point of this package is a camera-rendering toolkit for the pooled mode. It provides a shared camera-capable `GPUDevice`, camera sampling from your shaders, a passthrough pipeline, and crop helpers. It is covered in [WebGPU effects](./webgpu-effects). + +## Related guides + +- [Custom sources in React Native](./react-native): publishing the stream +- [Vision Camera](./vision-camera): the ready-made camera integration built on this layer +- [How custom sources work](../../../explanation/custom-sources) +- API reference: [Custom Video Source package](../../../api/custom-video-source/index) diff --git a/docs/how-to/client/custom-sources/react-native.mdx b/docs/how-to/client/custom-sources/react-native.mdx new file mode 100644 index 00000000..bbcd6f36 --- /dev/null +++ b/docs/how-to/client/custom-sources/react-native.mdx @@ -0,0 +1,108 @@ +--- +title: "Custom sources in React Native" +sidebar_position: 2 +sidebar_label: "React Native 📱" +description: Publish any React Native MediaStream to a Fishjam room with the useCustomSource hook. +--- + +# Custom sources in React Native Mobile + +:::note +This guide is exclusively for **Mobile** (React Native) applications. +::: + +The [`useCustomSource`](../../../api/mobile/functions/useCustomSource) hook from `@fishjam-cloud/react-native-client` is the same hook that is available [on the web](./web), typed for the React Native [`MediaStream`](../../../api/mobile/classes/MediaStream). Your app must be wrapped in [`FishjamProvider`](../../../api/mobile/functions/FishjamProvider). + +On React Native, a `MediaStream` for a custom source typically comes from: + +- the [Vision Camera integration](./vision-camera) or the [low-level frame API](./low-level-frame-api), whose hooks create the stream for you, +- any other `MediaStream` you already hold. + +## Publish a stream + +Set the stream on a stable source ID; set `null` to unpublish. The cleanup function keeps the source in sync with the component lifecycle. + +```tsx +import { useEffect } from "react"; +import { + useCustomSource, + type MediaStream, +} from "@fishjam-cloud/react-native-client"; + +// ---cut--- +export function usePublishStream(stream: MediaStream | null) { + const { setStream } = useCustomSource("my-custom-source"); + + useEffect(() => { + if (!stream) return; + setStream(stream); + return () => { + setStream(null); + }; + }, [stream, setStream]); +} +``` + +:::info[When tracks are actually published] + +Tracks are sent to Fishjam only while the peer is connected to a room. A stream set before joining is queued and published as soon as the connection is established, and pending sources are re-published automatically after a reconnect. Calls to `setStream` are serialized internally, so replacing one stream with another in quick succession is safe. + +::: + +:::important + +Custom sources are shared by ID: every `useCustomSource` call with the same source ID returns the same state, so multiple components can control and read one source. + +::: + +## What about audio? + +Custom sources on React Native are video-only today. The publishing contract is shared with the web (a stream's audio track would go out as a `customAudio` track), but React Native currently has no way to produce a custom audio track: there is no audio-frame API, and no equivalent of the web's `captureStream()` or Web Audio. The microphone is already covered by [`useMicrophone`](../../../api/mobile/functions/useMicrophone). + +If you need to publish generated or processed audio, that is currently a [web-only capability](./web.mdx). + +## Show a self-view + +The `stream` returned by the hook renders like any other stream: + +```tsx +import React from "react"; +import { useCustomSource, RTCView } from "@fishjam-cloud/react-native-client"; + +// ---cut--- +export function SelfView() { + const { stream } = useCustomSource("my-custom-source"); + + if (!stream) return null; + return ( + + ); +} +``` + +## Remove the source + +```tsx +import { useCustomSource } from "@fishjam-cloud/react-native-client"; +// ---cut--- +const { setStream } = useCustomSource("my-custom-source"); +// ... +await setStream(null); +``` + +Removing a source unpublishes its tracks but does not stop them; the producer of the stream owns the tracks and should stop them itself. + +## Receiving on other peers + +Custom tracks arrive in the `customVideoTracks` and `customAudioTracks` arrays of each peer returned by `usePeers`. See [Receiving custom tracks](./index.mdx#receiving-custom-tracks). + +## Related guides + +- [Vision Camera](./vision-camera): publish a camera feed with frame processors +- [WebGPU effects](./webgpu-effects): draw your own content into the published video +- [Low-level frame API](./low-level-frame-api): publish frames from your own pipeline +- [How custom sources work](../../../explanation/custom-sources) diff --git a/docs/how-to/client/custom-sources/vision-camera.mdx b/docs/how-to/client/custom-sources/vision-camera.mdx new file mode 100644 index 00000000..9d106948 --- /dev/null +++ b/docs/how-to/client/custom-sources/vision-camera.mdx @@ -0,0 +1,137 @@ +--- +title: "Publish a Vision Camera feed" +sidebar_position: 3 +sidebar_label: "Vision Camera 📱" +description: Publish a react-native-vision-camera feed to Fishjam, optionally running frame-processor plugins on the published frames. +--- + +# Publish a Vision Camera feed Mobile + +:::note +This guide is exclusively for **Mobile** (React Native) applications. +::: + +`@fishjam-cloud/react-native-vision-camera-source` publishes a [VisionCamera](https://react-native-vision-camera.com) feed to Fishjam. Its hooks work like the other Fishjam source hooks ([`useCamera`](../../../api/mobile/functions/useCamera), [`useScreenShare`](../../../api/mobile/functions/useScreenShare), [`useCustomSource`](../../../api/mobile/functions/useCustomSource)): they create the underlying track, publish it, and clean up on unmount. Each camera frame is handed to Fishjam without copying its pixels. + +Use this package when you need VisionCamera capabilities alongside a Fishjam call, such as frame-processor plugins for on-device ML, precise device control, or [your own WebGPU rendering](./webgpu-effects) drawn into the published video. If you only want to stream the camera, use [`useCamera`](../start-streaming) instead. + +## Prerequisites + +- `react-native-vision-camera` **v5** and `react-native-vision-camera-worklets` +- [`react-native-worklets`](https://docs.swmansion.com/react-native-worklets/) with its Babel plugin configured (required by VisionCamera's frame outputs) +- `@fishjam-cloud/react-native-client` with your app wrapped in `FishjamProvider` (see [Installation](../installation)) +- The New Architecture; custom video tracks require it +- A development build ([Expo Go](https://docs.expo.dev/develop/development-builds/introduction/) cannot load these native modules) + +## Install + +```bash npm2yarn +npm install @fishjam-cloud/react-native-vision-camera-source react-native-vision-camera react-native-vision-camera-worklets react-native-worklets +``` + +Add the worklets Babel plugin as the last entry in your Babel plugins, then restart Metro with a cleared cache: + +```js title='babel.config.js' +module.exports = { + presets: ["babel-preset-expo"], + plugins: ["react-native-worklets/plugin"], +}; +``` + +Camera permission setup (usage strings, requesting at runtime) follows the standard VisionCamera flow; see the [VisionCamera getting started guide](https://react-native-vision-camera.com/docs/guides). + +## Publish the camera + +Pass the hook's `frameOutput` to VisionCamera's `useCamera` outputs. The returned `stream` is your self-view; the same feed is published to the room under the given source ID. + +```tsx +import React from "react"; +import { + useCamera as useVisionCamera, + useCameraPermission, +} from "react-native-vision-camera"; +import { RTCView } from "@fishjam-cloud/react-native-client"; +import { useVisionCameraSource } from "@fishjam-cloud/react-native-vision-camera-source"; + +export function CameraPublisher() { + const { hasPermission } = useCameraPermission(); + + const { frameOutput, stream } = useVisionCameraSource("my-camera"); + + useVisionCamera({ + device: "front", + isActive: hasPermission, + outputs: [frameOutput], + }); + + if (!stream) return null; + return ; +} +``` + +Frame rotation and timestamps are handled for you: the hook normalizes VisionCamera's per-platform timestamp units onto one monotonic timeline and applies the frame's orientation automatically. + +Other peers receive the feed among their [`customVideoTracks`](./index.mdx#receiving-custom-tracks). + +:::warning[On-screen drawing is not published] + +The published track carries the camera frames exactly as captured. Anything you draw on top of the preview (React Native views, a [Skia](https://shopify.github.io/react-native-skia/) canvas, or any other overlay) appears only in your local UI and is **not** sent to Fishjam. To render content into the published video itself, use [WebGPU effects](./webgpu-effects.mdx) for effects and overlays on the camera feed, or the [low-level frame API](./low-level-frame-api.mdx) when you produce the frames yourself. + +::: + +## Run frame processors on published frames + +Pass an `onFrame` worklet to process every published frame, for example to run a VisionCamera frame-processor plugin for pose detection or other on-device inference. `onFrame` is for reading frames; it cannot draw into the published feed: + +```tsx +import { useCallback } from "react"; +import type { Frame } from "react-native-vision-camera"; +import { useVisionCameraSource } from "@fishjam-cloud/react-native-vision-camera-source"; + +declare function detectPose(frame: Frame): unknown; + +// ---cut--- +const onFrame = useCallback((frame: Frame) => { + "worklet"; + const pose = detectPose(frame); // any VisionCamera frame-processor plugin + console.log(pose); +}, []); + +const { frameOutput, stream } = useVisionCameraSource("my-camera", { + onFrame, +}); +``` + +:::danger[Frame lifetime] + +`onFrame` runs **after** the frame has been sent to Fishjam, and the frame is valid only for the duration of your synchronous callback; the hook releases it afterwards. Don't store the frame or its buffers for later use. + +::: + +:::warning + +Keep the identity of `onFrame` stable (`useCallback` or module scope). A new function identity re-registers the frame callback on every render. + +::: + +## Options + +| Option | Default | Description | +| -------------------------- | ------------ | --------------------------------------------------------------------------------------------------- | +| `enabled` | `true` | Creates and publishes the track while `true`; tears everything down when `false`. | +| `onFrame` | — | Worklet called with every camera frame after it has been sent. | +| `onFrameDropped` | — | Worklet called with the reason whenever a frame is skipped (for example when the pipeline is busy). | +| `frameIntervalNanoseconds` | `33_333_333` | Fallback frame spacing used when a frame arrives without a usable native timestamp. | + +The options also accept VisionCamera's `FrameOutputOptions`. The hook forces `pixelFormat: 'native'` (the copy-free path) and defaults `dropFramesWhileBusy` to `true`. + +## Troubleshooting + +- **Track creation fails immediately**: custom video tracks require the New Architecture; the returned `error` carries a clear message when the app runs on the old architecture. +- **No frames flow**: check that `isActive` is `true`, the device permission is granted, and `frameOutput` is included in the `outputs` array of `useCamera`. + +## Next steps + +- Follow the [Vision Camera tutorial](../../../tutorials/vision-camera) for a step-by-step walkthrough +- Draw your own content into the feed with [WebGPU effects](./webgpu-effects) +- API reference: [Vision Camera Source package](../../../api/vision-camera-source/index) diff --git a/docs/how-to/client/custom-sources.mdx b/docs/how-to/client/custom-sources/web.mdx similarity index 78% rename from docs/how-to/client/custom-sources.mdx rename to docs/how-to/client/custom-sources/web.mdx index def2c69b..98f3025b 100644 --- a/docs/how-to/client/custom-sources.mdx +++ b/docs/how-to/client/custom-sources/web.mdx @@ -1,35 +1,30 @@ --- -title: "Custom sources" -sidebar_position: 8 -sidebar_label: "Custom sources 🌐" +title: "Custom sources on the web" +sidebar_position: 1 +sidebar_label: "Web 🌐" description: Stream non-standard video or audio sources (e.g. WebGL, WebGPU, Three.js) through Fishjam in web apps. --- import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; -# Custom sources Web +# Custom sources on the web Web :::note -This guide is exclusively for **Web** (React) applications. +This guide is exclusively for **Web** (React) applications. For React Native, see [Custom sources in React Native](./react-native). ::: -:::important - -If you only wish to send camera, microphone or screen share output through Fishjam, then you most likely should refer to the documentation in [Streaming media](./start-streaming) and [Managing devices](./managing-devices) instead of this page. - -::: - -This section demonstrates how to stream non-standard video or audio to other peers in your web app. +This guide demonstrates how to stream non-standard video or audio to other peers in your web app. The utilities in this section allow you to integrate Fishjam with powerful browser APIs such as [WebGL](https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API) and [WebGPU](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API), -or higher level libraries, which leverage these APIs, such as [Three.js](https://threejs.org/), [Smelter](https://smelter.dev) or [PixiJS](https://pixijs.com/) +or higher level libraries, which leverage these APIs, such as [Three.js](https://threejs.org/), [Smelter](https://smelter.dev) or [PixiJS](https://pixijs.com/). +For a complete WebGPU render-and-publish pipeline, see [WebGPU effects](./webgpu-effects.mdx). -## Creating a custom source - [`useCustomSource()`](../../api/web/functions/useCustomSource) +## Creating a custom source - [`useCustomSource()`](../../../api/web/functions/useCustomSource) To create a custom source, you only need to do two things: 1. Call the `useCustomSource` hook. -2. Call the [`setStream`](../../api/web/functions/useCustomSource#setstream) callback provided by the `useCustomSource` hook with a [MediaStream](#how-to-get-a-mediastream-object) object. +2. Call the [`setStream`](../../../api/web/functions/useCustomSource#setstream) callback provided by the `useCustomSource` hook with a [MediaStream](#how-to-get-a-mediastream-object) object. #### Usage Example @@ -47,12 +42,12 @@ useEffect(() => { ### Using a created custom source -Once you have called [`setStream`](../../api/web/functions/useCustomSource#setstream) for a given source ID (in the above example, `"my-custom-source"`), any subsequent calls to `useCustomSource` with the same ID will return the same state. +Once you have called [`setStream`](../../../api/web/functions/useCustomSource#setstream) for a given source ID (in the above example, `"my-custom-source"`), any subsequent calls to `useCustomSource` with the same ID will return the same state. This enables multiple components to control and read a shared custom source. ### Deleting a custom source -If you wish to remove a custom source, then you should call the [`setStream`](../../api/web/functions/useCustomSource#setstream) callback with `null` as its argument. +If you wish to remove a custom source, then you should call the [`setStream`](../../../api/web/functions/useCustomSource#setstream) callback with `null` as its argument. #### Usage Example @@ -180,3 +175,9 @@ If you have a `