Camera frames

Preview.tsx

1import { CameraFrame, Canvas, Fill, ImageShader, Shader, useCameraFrame } from 'react-native-tgfx'

3export function Preview() {

4 const { image, posture, onFrame } = useCameraFrame()

5 const output = useFrameOutput({ ...OPTIONS, onFrame })

6 useCamera({ isActive: true, device: 'back',

7 outputs: [output] })

9 return (

10 <Canvas style={StyleSheet.absoluteFill} opaque>

11 <CameraFrame image={image} {...posture}>

12 <Fill>

13 <Shader source={NEON}>

14 <ImageShader image={image} />

15 </Shader>

16 </Fill>

17 </CameraFrame>

9:41

npm install react-native-vision-camera react-native-vision-camera-worklets

useCameraFrame turns a VisionCamera frame into a TgfxImage and a CameraFramePosture, together a CameraFrameSource.

Preview.tsx

import { useCamera, useFrameOutput } from 'react-native-vision-camera'
import { Canvas, CameraFrame, useCameraFrame } from 'react-native-tgfx'

export function Preview() {
  const { image, posture, onFrame } = useCameraFrame()

  const frameOutput = useFrameOutput({
    pixelFormat: 'yuv',
    dropFramesWhileBusy: true,
    enablePhysicalBufferRotation: true,
    // Headless: this output is the only one, so deferring it behind a preview starts it late.
    allowDeferredStart: false,
    onFrame,
  })

  useCamera({
    isActive: true,
    device: 'back',
    outputs: [frameOutput],
  })

  return (
    <Canvas style={StyleSheet.absoluteFill} opaque={true}>
      <CameraFrame image={image} {...posture} />
    </Canvas>
  )
}

Get camera permission first. No camera view is mounted. Keep allowDeferredStart: false, or frames start late.

Keep useFrameOutput options stable (a module constant or memoised).

CameraFrameProps · CameraFrameOptions

Zero copy

imageFromNativeBuffer imports the frame's buffer (NativeBufferLike) as a texture with no copy. Later frames reuse the handle:

handle.value == null
  ? imageFromNativeBuffer(buffer.pointer, format)   // first frame
  : handle.value.setNativeBuffer(buffer.pointer, format)  // every frame after

setNativeBuffer releases the previous buffer before returning.

Don't write into a buffer a handle points at.

useCameraFrame manages this for you.

The frame in your callback dies when it returns. Don't retain it, call getNativeBuffer() again, or store its pointer.

No JS per frame

npm install react-native-tgfx-vision-camera

useTgfxCameraSink does the import natively, so frames skip JS. Attach it once the handle exists:

const sink = useTgfxCameraSink()

useEffect(() => {
  if (image != null) sink.attach(frameOutput, image.handle)
}, [sink, frameOutput, image])

After attach, orientation stops reaching JS, so keep enablePhysicalBufferRotation: true. Not for frame processors that run JS, like a detector.

Zooming

The viewport doesn't move a <CameraFrame>. Wrap it in <Group transform> to zoom it.

Shading a frame

<CameraFrame image={image} {...posture}>
  <Fill>
    <Shader source={NEON}>
      <ImageShader image={image} />
    </Shader>
  </Fill>
</CameraFrame>

Keep tile modes at 'clamp', or YUV frames turn transparent.

Face blur

Allocate fixed slots. Zero-size slots draw nothing.

const SLOTS = [0, 1, 2, 3]
const [ovals] = useState(() => SLOTS.map(() => createHandle()))

const detect = useCallback((frame: Frame) => {
  'worklet'
  const faces = detector.detectFaces(frame)
  for (const i of SLOTS) {
    const b = faces[i]?.bounds
    ovals[i]!.set({
      x: b?.x ?? 0,
      y: b?.y ?? 0,
      width: b?.width ?? 0,
      height: b?.height ?? 0,
    })
  }
}, [detector, ovals])

return { ...useCameraFrame(detect), ovals }

<CameraFrame image={image} {...posture}>
  {SLOTS.map((i) => (
    <Oval key={i} ref={ovals[i]} x={0} y={0} width={0} height={0}>
      <Paint>
        <ImageShader image={image} />
        <Blur blur={25} />
      </Paint>
    </Oval>
  ))}
</CameraFrame>