RUStart

ARScene

Augmented reality: a Scene drawn over the device camera's picture, its camera moved as the phone moves, so 3D models stay on a real floor, table or printed picture. A camera preview in the UI, a photo or a QR code is CameraView instead.

  • place a 3D model on a real table or floor in AR
  • tap to place an object on the floor or a table
  • let the user drag, pinch and twist a placed model
  • show a video or a model on a printed picture (image targets)
  • find the floors, tables and walls the camera sees
  • light the model like the room around it
  • take a photo of the AR view

Platforms: iOS, Android, the le.codes viewer app; on the web (Web Lite, lecodes web) plain AR only — the camera picture under the scene, turned by the device's sensors, no surfaces, no image targets, https needed; the web player opens the session but shows no camera picture; not on desktop or in headless tests — check ARScene.isSupported.

At a glance

TypeScript
const main = async () => {
  const scene = new ARScene()
  const chair = await Model.load(asset('./chair.glb'))
  scene.root.add(chair)                         // on the floor, once the session finds it
  scene.root.addEventListener('track', () => toast('Drag to move · pinch to scale'))
  scene.addControls(chair)                      // drag / pinch / twist to place it

  try {
    await scene.open()                          // camera permission → warm render → session → on screen, drawn
  } catch (e) {
    toast('AR is not available here')           // denied, no ARKit / ARCore, or superseded
  }
}
main()

Add AR to an app: the ar plugin, ARScene.isSupported

ARScene is a plugin: list it in app.json — "plugins": { "ar": "*" } (one list for every platform). iOS runs ARKit, Android ARCore (the app installs on a phone without it; the first open() may ask to install Google Play Services for AR). ARScene.isSupported is false on a host without the plugin — check it before showing an AR button. It says the plugin is there, not that the device can track: open() still rejects unavailable on a phone without ARKit / ARCore.

Everything from Scene (options, add/remove, events, overlays, capture()) applies. A session of the plugin drives the scene every frame: it moves scene.camera as the device moves, gives it the camera's lens and draws the camera image behind everything.

Open the camera and start AR: new ARScene, open()

TypeScript
new ARScene(options?: SceneOptions & {
  mode?: 'world' | 'images'    // 'world' (default): planes, hit tests, anchors, images
})                             // 'images': image targets only — lighter; no planes, no hit test, no root

ARScene.isSupported            // static: this host has the "ar" plugin
scene.mode                     // the mode it was created with
scene.useWarmRender = true     // open() warm-renders the scene first (default true)

scene.open(options?: {
  transition?: Transition
  planes?: 'horizontal' | 'vertical' | 'both' | 'none'   // what to look for; default 'horizontal'
  lightEstimation?: boolean                              // 'light' events; default false
  followLight?: boolean                                  // the scene's ambient follows the light
  worldAlignment?: 'gravity' | 'gravityAndHeading'       // -Z north (iOS, with location access)
  autoFocus?: boolean                                    // default true
}): Promise<void>
scene.close(options?): void    // stops the session (the camera is let go) and leaves the screen

open() asks for the camera, warm-renders, starts the session and waits for its first camera frame before the scene becomes current. The current destination (typically a loading screen) stays visible the whole time: the swap — and the transition — happens only once everything is ready. It resolves at the first frame drawn of the presented scene: after await scene.open() the scene is on screen, scene.camera.displaySize is its view's size, and hitTest() and captureFrame() work. It rejects — and leaves the screen as it was — when the camera is denied (denied), the device has no ARKit / ARCore or the user declined installing it (unavailable), the session failed, or another navigation happened meanwhile; a close() or another navigation before that first frame — while it prepares, or after the swap — rejects it as superseded. Router.push(arScene) / Router.replace(arScene) wait the same way — to the first frame — with the options of the last open().

Tracking quality and session errors: trackingState

TypeScript
scene.trackingState            // 'notAvailable' | 'limited' | 'normal'
scene.trackingReason           // with 'limited': 'initializing' | 'excessiveMotion' |
                               //   'insufficientFeatures' | 'relocalizing' — else null
scene.addEventListener('tracking', (state, reason) => {})   // the session's first event, then on change
scene.addEventListener('error', ({ code, message }) => {})  // the session failed and is over

The camera is the session's: read it (scene.camera.worldPosition, getRay), never move it — the session overwrites it every frame.

Keep a model on the floor or at a place in the room: root, anchor()

TypeScript
scene.root: Anchor                                    // the floor (not in 'images')
scene.anchor(pose: { position, rotation }): Anchor    // a place in the room (a hit test's pose)

anchor.tracked                                        // the session holds it now
anchor.addEventListener('track', () => {})
anchor.addEventListener('untrack', () => {})
anchor.remove()                                       // the session lets it go; it leaves the scene

An Anchor is a Node the session moves: put content under it (it joins the scene with it). root is pinned on the first horizontal plane the session finds — it fires track then; until that it sits at the world's origin, where the session began. scene.anchor(pose) keeps a place where it is in the room as the session's map of it improves (iOS reports untrack / track around a relocalization). Every tracked anchor fires untrack when the session ends (close() or an error).

Show a video or a model on a printed picture: createAnchor

TypeScript
scene.createAnchor(source: FetchResponse | File, physicalWidth = 0.2): ImageAnchor  // an image target
imageAnchor.extent                                    // [width, height] of the image, metres

createAnchor follows a picture — a fetched image, physicalWidth its printed width in metres — and returns an ImageAnchor: a rigid pose in metres, like every anchor (no scale): its centre, Y out of the image, the image lying in X (its width) and Z (towards its bottom). How big the image is, is extent — [width, height] in metres, the session's estimate, refined while it tracks the image (ARKit's scale estimate, ARCore's measured extent). Before the image is first found it is [physicalWidth, 0]: the height comes with the image's proportions, which the session reads, at the first track. An image anchor is hidden while not tracked. Image targets can be added (and removed, anchor.remove()) before open() or at any time: on a running scene the image set is updated in place — tracking, planes and anchors stay. On Android an image being tracked is looked for again when the set changes (ARCore starts the new set afresh): an untrack, then a track once the camera sees it again.

A video on a poster — a plane in the image, sized to it:

TypeScript
const main = async () => {
  const scene = new ARScene({ mode: 'images' })
  const poster = scene.createAnchor(await fetch(asset('./poster.jpg')), 0.42)   // printed 42 cm wide
  const player = new VideoPlayer(asset('./clip.mp4'))
  const screen = Mesh.plane({ material: Material.video(player.texture), eulerAngles: [-90, 0, 0] })   // XY → the image's XZ
  poster.add(screen)
  poster.addEventListener('track', () => {
    screen.scale = [poster.extent[0], poster.extent[1], 1]
    player.play()
  })
  await scene.open()
}
main()

Find floors, tables and walls: scene.planes

TypeScript
scene.planes: ARPlane[]                               // what the session holds now
scene.addEventListener('plane', ({ plane, change, mergedInto }) => {})
// change: 'added' | 'updated' | 'removed'; mergedInto: the plane that absorbed this one (Android)
scene.showPlanes = true                               // a grid drawn on every plane

plane.type                     // 'horizontal' | 'vertical'
plane.extent                   // [width, depth], metres, in the plane's X and Z
plane.polygon                  // its outline: x, z pairs in the plane's space
plane.center                   // its centre in the world
plane.planeId                  // what a hit test names

An ARPlane is an anchor at the plane's centre with Y along its normal; it moves and grows as the session learns the surface (updates come at most ten times a second, for a change over a centimetre). Walls come only with open({ planes: 'vertical' }) or 'both':

TypeScript
const scene = new ARScene()
scene.showPlanes = true                         // a grid on every surface found
scene.addEventListener('plane', ({ plane, change }) => {
  if (change === 'added' && plane.type === 'vertical') toast(`A wall, ${plane.extent[0].toFixed(1)} m wide`)
})
scene.open({ planes: 'both' })

Tap to place an object on the floor or a table in AR: hitTest

A tap's point on the screen becomes a pose on the real surface under it; anchor the object there.

TypeScript
scene.hitTest(x, y, target?: 'planes' | 'estimated' | 'any'): Promise<Array<{
  pose: { position: Vec3, rotation: Quat }
  type: 'plane' | 'estimated'    // on a found plane, or on a point / surface the session estimates
  plane: ARPlane | null
  distance: number               // metres from the camera
}>>

What is under a point of the scene's view, nearest first — the point as the scene's touches give it (clientX / clientY of a click or a touchstart on the scene, relative to its view; the session fits the camera to that view, not to the window); target defaults to 'any'. Empty when nothing is. Place things with it:

TypeScript
const scene = new ARScene()
scene.addEventListener('click', async (ev) => {
  const [hit] = await scene.hitTest(ev.clientX, ev.clientY, 'planes')
  if (hit) scene.anchor(hit.pose).add(await Model.load(asset('./lamp.glb')))
})
scene.open()

Light the model like the room: lightEstimate, followLight

TypeScript
scene.lightEstimate            // { intensity, temperature } | null — 1000 = neutral, kelvin
scene.addEventListener('light', ({ intensity, temperature }) => {})   // with lightEstimation, ≤ 10/s

open({ followLight: true }) turns the estimate on and scales the scene's environmentIntensity by it (back to what it was when the session ends).

TypeScript
const main = async () => {
  const scene = new ARScene()
  scene.root.add(await Model.load(asset('./lamp.glb')))
  scene.addEventListener('light', ({ intensity, temperature }) => console.log(intensity, temperature))
  await scene.open({ followLight: true })       // the scene's light dims and brightens with the room
}
main()

Take a photo of the AR view: capture, captureFrame

TypeScript
scene.capture(): Promise<File>         // what the scene shows: the camera image and the 3D, a PNG
scene.captureFrame(): Promise<File>    // the camera's image alone, a JPEG

captureFrame() waits for a readable camera image when the session has none at that instant (the first frames after a start or a resume) — up to about half a second, then it rejects failed; a session stopped meanwhile rejects notRunning.

TypeScript
const scene = new ARScene()
scene.open()
scene.addEventListener('click', async () => {
  const photo = await scene.capture()           // the camera picture with the 3D on it, a PNG
  share(photo)
})

Let the user move, scale and rotate a model: addControls

TypeScript
scene.addControls(target: Node, options?: {
  minScale?: number   // smallest uniform scale the pinch allows; default 0.08
  maxScale?: number   // largest; default 2
  pan?: boolean       // one-finger drag; default true
  pinch?: boolean     // two-finger scale; default true
  twist?: boolean     // two-finger rotate; default true
}): { remove(): void }
  • One finger drags the target over what the session finds under the finger (a plane, through hitTest); where it finds nothing — the web, or before a surface is known — across the ground, relative to where the camera looks, at the target's height.
  • Two fingers pinch to scale (uniform, clamped by minScale/maxScale — the effective range is [minScale × 0.4, maxScale × 1.5]) and twist to rotate the target around its Y axis.

AR on the web: the camera picture turned by the sensors

The plugin's web half (Web Lite) is the plain kind: the camera's picture under the scene, the scene's camera turned as the device turns (the gyroscope and the accelerometer, or the browser's orientation where there is no gyroscope; Safari asks for the sensors inside a tap — open the scene from one). The camera stays at the world's origin and the session starts looking at −Z. It finds nothing in the picture: tracking is 'normal' from the first frame, no plane / light event comes, image anchors never track, hitTest resolves [], root stays at the origin (it never fires track), anchor() places an anchor that stays where it is. The origin is the camera's own position, so content under root sits at the viewer's eye — move it away from the origin. captureFrame() and capture() work. A page needs https (or localhost) for the camera.

The web player runs the same half, but its runtime module has neither the camera picture nor the device's turn: the session opens and reports 'normal', and nothing of the camera shows.

See also

  • Scene — everything inherited: options, add/remove, overlays, capture, events
  • Node — hierarchy, transforms, events
  • Model — loading the GLB you're placing
  • Pointer events & gestures — the touch surface addControls is built on
  • Permissions — the camera, asked by open()
← BackPost-processingNext →Node