Media

AudioPlayer plays a sound file and VideoPlayer a video, through the platform's own player. An app's sounds — a tap sound, music, a voice-over — play through AudioPlayer, on every host; audio (audio.md) is the game engine's mixer and plays on desktop and iOS only — a silent no-op on Android, the web and the web player (audio.supported).

• play a short sound when a button is tapped
• play background music on a loop
• pause, resume or restart a track
• do something when a sound or a video finishes (completed)
• play a file the user picked or recorded
• play a video on a screen (UIVideo)
• put a video on a 3D surface (Material.video)
• stop the music when the screen closes
• pause the music when the app goes to the background

Platforms: iOS, Android, desktop, the web and the web player, with no guard to check; desktop reports no position and no end of playback, Linux plays no video, and headless tests (lecodes render / lecodes test) play nothing (Platform notes).

At a glance

Background music on a loop, and a one-shot sound:

const music = new AudioPlayer(asset('./theme.mp3'))
music.loop = true
music.volume = 0.4
music.play()

new AudioPlayer(asset('./win.wav')).play()   // fire and forget: it plays to the end, then goes

Play a sound when a button is tapped

A tap sound — a short sound effect on every tap — is one AudioPlayer, created once with the screen and played again on each tap: rewind it with time = 0, then play().

const click = new AudioPlayer(asset('./click.wav'))   // created once: it starts loading now, not at the first tap

const screen = UIScreen(
  UIButton(UIText("Tap me")).onClick(() => {
    click.time = 0   // back to the start: replays a finished sound, restarts one still playing
    click.play()
  }),
)
screen.open()

The rewind matters: on iOS a player that reached its end stays there, and play() alone plays nothing. One player plays one sound at a time — for taps that overlap, fire a new player per tap (new AudioPlayer(asset('./click.wav')).play(): it loads the file again, plays to the end and is freed). Dozens of effects at once, with pitch and variants, are the game mixer's job (audio.md), on desktop and iOS only.

Create a player: an asset, a URL or a file

new AudioPlayer(src?: string | File | FetchResponse)   // asset('./sound.mp3'), a remote URL, or bytes
new VideoPlayer(src?: string | File | FetchResponse)

A player starts loading its source when it is created, so create it before it has to play. A URL plays as it downloads; a File or a FetchResponse (a picked, recorded or fetched file) plays from memory, the whole bytes. One player plays one file, and both kinds have the same controls and events. For sound, MP3 and WAV play on every host — the desktop player decodes only these two.

Play a file the user picked or recorded

A file the person picks with openFilePicker is a File, and a player takes it as its source:

const pick = UIButton(UIText("Play a file")).onClick(async () => {
  const file = await openFilePicker({ accept: "audio/*" })   // null when the dialog is cancelled
  if (file) new AudioPlayer(file).play()
})
UIScreen(pick).open()

A recording of the AudioRecorder plugin plays the same way: new AudioPlayer(recording.file).play().

Pause, resume, seek and volume

player.play(): void          // start, or resume where it paused
player.pause(): void         // pause, keeping the position
player.stop(): void          // pause and rewind to the start
player.playing: boolean      // settable — play() / pause() set it
player.volume: number        // 0..1, settable; default 1 — this player's level, not the system volume
player.loop: boolean         // settable: start over at the end instead of stopping; default false
player.time: number          // the position in SECONDS, settable to seek
player.duration: number      // the length in seconds, read-only; 0 until the file's metadata loads

A play / pause button and a restart:

const music = new AudioPlayer(asset('./theme.mp3'))
music.loop = true
const label = UIText("Play")

UIScreen(
  UIButton(label).onClick(() => {
    music.playing = !music.playing   // the same as play() / pause()
    label.text = music.playing ? "Pause" : "Play"
  }),
  UIButton(UIText("From the start")).onClick(() => { music.time = 0 }),
).open()

When a sound finishes playing: completed / loopReached

To do something when a sound or a video finishes playing — start the next track, enable a button again — listen for completed; with loop on, loopReached fires at each lap instead.

player.addEventListener('completed', callback: () => void)     // the end, while loop is false: playing turns false
player.addEventListener('loopReached', callback: () => void)   // each lap, while loop is true: it keeps playing
player.removeEventListener('completed', callback)

Exactly one of the two fires per play-through, decided by loop at the moment the end is reached. A playlist that plays the next track when one ends:

const tracks = [asset('./one.mp3'), asset('./two.mp3'), asset('./three.mp3')]
let index = 0

const playTrack = () => {
  const player = new AudioPlayer(tracks[index])
  player.addEventListener('completed', () => {
    index = (index + 1) % tracks.length
    playTrack()
  })
  player.play()
}
playTrack()

Play a video on a screen

A VideoPlayer plays the video and UIVideo shows it in a screen; the controls are the player's, the same as an AudioPlayer's.

const movie = new VideoPlayer(asset('./intro.mp4'))
const video = UIVideo(movie).style({ width: "100%", height: 220, objectFit: "contain" })

UIScreen(
  video,
  UIButton(UIText("Play / pause")).onClick(() => { movie.playing = !movie.playing }),
)
  .onClose(() => movie.stop())
  .open()

video.open() shows the same video full screen (UI content).

Video on a 3D surface: texture

movie.texture is a 3D Texture that samples the live video; Material.video(texture) puts it on a mesh.

const movie = new VideoPlayer(asset('./clip.mp4'))
movie.loop = true
movie.play()
const tex = movie.texture   // read it once: a 3D Texture sampling the video

const scene = new Scene()
scene.add(Mesh.plane({ material: Material.video(tex), scale: [1.6, 0.9, 1], position: [0, 1, -1] }))
scene.camera.position = [0, 1, 2.5]
scene.open()

Background music for a screen: onOpen / onClose

Music that loops while a screen is shown starts in the screen's onOpen and stops in its onClose — a playing player keeps playing after its screen has closed.

const music = new AudioPlayer(asset('./level.mp3'))
music.loop = true

const level = UIScreen(UIText("Level 1"))
  .onOpen(() => music.play())
  .onClose(() => music.stop())
level.open()

onOpen runs again when the screen comes back, so the music starts over. Players free themselves: a player lives while you hold a reference to it and while it is playing — a one-shot you fire and forget plays to its end and is released then; a paused player you stop referencing is released at once. dispose() survives only as a deprecated alias of stop() — it frees nothing.

Sound in the background and on the lock screen

When the app goes to the background (the app pause event, app.md), what happens to the sound differs per host — none has a background-audio mode or lock-screen controls:

• iOS stops the sound once the app is in the background or the screen locks.
• Android keeps the players playing, but the app's code is paused: completed — and the next track a playlist starts from it — waits until the app is back, and Android may end the app at any time.
• Desktop keeps playing while the window is minimized.
• The web leaves it to the browser: the host pauses no player in a hidden tab, and the browser's own rules apply.

For the same behaviour everywhere, pause on pause and play again on resume:

const music = new AudioPlayer(asset('./theme.mp3'))
music.loop = true
music.play()

let wasPlaying = false
app.addEventListener('pause', () => {     // the app left the foreground
  wasPlaying = music.playing
  music.pause()
})
app.addEventListener('resume', () => {    // it is back
  if (wasPlaying) music.play()
})

Platform notes

There is no support flag: where a host has no media players, every call does nothing and nothing throws.

• iOS — the Ring/Silent switch mutes AudioPlayer, VideoPlayer and game audio: the app keeps iOS's default audio session. Once the app has recorded (AudioRecorder), listened (SpeechRecognizer) or made a call (webrtc), the session is a recording one and the switch no longer mutes it.
• Desktop (Windows, macOS, Linux) plays both kinds but reports no position and no end: time and duration read 0, setting time does nothing (so stop() only pauses, and a tap during a sound does not restart it), completed / loopReached never fire — a finished player still reads playing === true — and its players stay alive until the app exits. An AudioPlayer there plays through game audio's music bus, so audio.bus('music').volume and .muted apply to it — on desktop only.
• Linux desktop has no video backend: a VideoPlayer plays nothing.
• The web and the web player play through the browser's <audio> / <video>. A browser may refuse sound until the person has tapped or clicked the page: music started at launch then stays paused (playing still reads true) — start it from a tap.
• Headless tests (lecodes render / lecodes test) and the windowless server (--sim) have no media players: nothing plays, time and duration read 0 and no end event fires; playing follows play() / pause().

See also

• Game audio — Sound, audio.play, buses: many effects at once, pitch, variants (desktop and iOS)
• UI content — UIVideo shows a VideoPlayer in a screen
• App lifecycle — the pause / resume events
• Audio recorder — recording from the microphone
• Files & share — openFilePicker, sharing captured or downloaded media files