ffmpeg.wasm + Svelte 5

Svelte 5 seems nice. Here's a component that eventually does ffmpeg:

Arrows indicate equivalence, the dotted one also being function call.

Left file being defined and acted upon: set to a File object got from a drop zone.

It then seems to be a container object...

If we devtools / Sources, the nice sourcemapped code shows up first, and has a link down there...

over to the compiled code:

Where the new magic, instead of simply having file, accesses the current value of file from... Itself, which has become some kind of container|proxy object.

I guess this to determine the dependencies of their expressions when they are evaluated as they say in https://svelte.dev/blog/runes.

You will have read about them. https://sveltekit.io/blog/runes#the-derived-rune

What is $derived?

The $derived value will also always be current, so you don’t need to worry about value mismatches when using the derived value in a function right after the dependency changes - your changes will be reflected immediately, and thus before the next executable runs.

Interesting.

Over in https://github.com/stylehouse/bitzliten, I want to make most of its memory hang off each file value that may come along.

Here are two jabs at that:

// now these reset per file:

let muck = () => {

file && 1

debugger

}

let memory = $derived(file && muck())

// the selection

let sel = $state({})

// how-encoded chunks

let dublets = $state([])

$effect(() => {

if (file) {

sel = {in:10,out:20}

dublets = []

debugger

}

// memory && 1

})

It won't do muck() unless you uncomment memory && 1!

I'm not sure why... The $effect fires because it has a file dependency, so are we adding a dependency on memory? But the $derived occurs after the $effect so that conflicts with 

The $derived value will also always be current

I'm just looking for what works at this point, so lets just ~file -> $effect -> reset a bunch of $state!

So letsgo() and $inspect

And so, the focal point is now:

function letsgo() {

if (1) {

if (sel.in == null) return []

let needs = undublets()

$inspect(needs)

// needs.map(joblet => go_ffmpeg(joblet))

}

go_ffmpeg()

}

I have renamed what it used to be to go_ffmpeg() and taken its spot (sublation).

The $inspect errors something about occurring out of component initialisation, so I suppose:

let needs

function letsgo() {

if (1) {

if (sel.in == null) return []

needs = undublets()

// needs.map(joblet => go_ffmpeg(joblet))

}

go_ffmpeg()

}

$inspect(needs)

But I see no output.

What about:

let needs = $state()

function letsgo() {

if (1) {

if (sel.in == null) return []

needs = undublets()

// needs.map(joblet => go_ffmpeg(joblet))

}

go_ffmpeg()

}

$inspect(needs)

This gives me a bunch of console action:

I go and fix something, and:

We are generating chunks fine now!

nublets -> dublets

We now switch up the API to process these items

function letsgo() {

if (sel.in == null) return []

let needs = undublets()

console.log("letsgo() nublets:",needs)

needs.map(joblet => go_ffmpeg(joblet))

}

Making needs a lexical in this letsgo() call, rather than a $state the component keeps around and maybe tells you about, is essential to avoid an infinite loop...

Then we:

async function go_ffmpeg(joblet:adublet) {

...

let result = await FF.transcode(file,joblet.modes)

joblet.objectURL = URL.createObjectURL(result)

...

dublets.push(joblet)

...

We get it done:

<Player {playlets} />

A new thing to pass data to... This data:

function letsgo() {

if (sel.in == null) return []

// make view of what to play

update_playlets()

// find gapos

let needs = playlets.filter(playlet => !playlet.ideal_dub)

// let needs = undublets()

console.log("letsgo() nublets:",needs)

needs.map(joblet => go_ffmpeg(joblet))

}

// generate a bunch of tiles for your ears to walk on

function update_playlets() {

let {n_chunks} = get_timespace(sel)

// a set of dublets stretching across it

let nublets = create_unfulfilled_dublets(sel,n_chunks,chunk_length)

// ready it for being an ffmpeg job

// ie include modes+modes_json, which includes its in|out points

nublets.map(nublet => compile_dublet_modes(nublet))

// link to candidate dublets

nublets.map(nublet => find_dub(nublet))

// and we now call that

playlets = nublets

}

This infinite loops:

Because I was going to get Player to hang some of its state on the  playlets/dublet (that "/" is a path expression, "|" means or), eg getting html ready to play the next chunk:

    {#if dublet.near}

<audio ...

So lets instead:

    {#if near.includes(cuelet)}

Having got a layer of cuelets following the given playlets:

$effect(() => {

// < buffering

console.log("SYNC CUELETS")

init_sel()

sync_cuelets(playlets)

})

You cause an infinite loop if you access near directly for that console.log

Strange.

But because it involves Svelte reactivity, it also handles the infinite loop for us!

My stereo looks strong:

Getting sound to come out

Is iterative: it loops badly, the callback to hit play aiming a little early gives better results,with <audio> tags.

TODO: film them coming and going, at least, for comedic effect.

Having to hit play at the right time is messy and usually stutters noticably.

<Schaud {playlets} />

I go for buffered audio (see 'our loop') and get seamless playthrough!

These are the sounds that play:

  type asource = AudioBufferSourceNode & {fadein:Function,fadeout:Function}

They are born from source_cuelet(cuenow)

Here:

let source:asource = audioContext.createBufferSource() as asource;

    Some type noise: the as casts it to a type, to solve a "... is not assignable to ..." warning.

We dump the wax in:

source.buffer = cuelet.buffer;

For convenience we extend with:

// fades

let fade = (suddenly,thence,fadetime:number) => {

gain.gain.setValueAtTime(suddenly, audioContext.currentTime);

gain.gain.linearRampToValueAtTime(thence, audioContext.currentTime + fadetime);

}

source.fadein = (fadetime) => fade(0,1,fadetime)

source.fadeout = (fadetime) => fade(1,0,fadetime)

And then back in the callee of source_cuelet(cuenow),

    the scheduleNextSound() loop:

// one of these while playing

if (!cuenow.source) {

let source = cuenow.source = source_cuelet(cuenow)

// crossfade from last

if (c.fadein) {

source.fadein(c.fadein)

}

// play

source.start(audioContext.currentTime);

// crossfade to next

let cuenext = next_cuelet(cuenow)

if (cuenext.in != cuenow.out) {

// < how to get to the time to do it more reliably?

let left = source.buffer.duration - fadetime

setTimeout(() => {

cuelet.source.fadeout(fadetime)

cuenow = cuenext

scheduleNextSound(null,{fadein:fadetime})

},left * 1000)

}

source.onended = () => {

// on the cuelet that was cuenow when it started playing

delete cuelet.source

// cuenext will already be cuenow and playing (has .source) if crossfading

// this may trigger if we get >1 cuelet ahead of now somehow

if (cuenow != cuelet && cuenow != cuenext) debugger

if (cuelet == cuenow) {

// nobody else (crossfading) has altered cuenow since we started

cuenow = cuenext

scheduleNextSound(the);

}

};

Note any non-sequential precessing pair will crossfade: cuenext.in != cuenow.out

The the is an attempt to deduplicate waves of going at scheduleNextSound()

Though we only set up callbacks if !cuenow.source, so...

It behaves!

So the art department (Gemini) came up with this:

And gliding it is:

function up_displaytime() {

setTimeout(() => up_displaytime(), 200)

cuelets && 1

if (!(cuenow && cuenow.el)) return

let length = sel.out - sel.in

// a repeating time measure

displaytime = dec(audioContext.currentTime % length)

// move needle

let cueswidth = cuenow.el.offsetWidth

let loop_width = cueswidth * playlets.length

let loop_progress = displaytime / length

needlepos = dec(loop_progress * loop_width,0)

}

I use a spring:

let needlepos = spring(0,{

stiffness: 0.1,

damping: 0.5,

});

Which needs tuning:

Suavis.

I found you can't get a repeating time measure from:

    dec(audioContext.currentTime % length)

Because they basically start whenever, so over in scheduleNextSound()

// play

cuenow.startTime = audioContext.currentTime

source.start(audioContext.currentTime);

And then

// a repeating time measure

let loop_startTime = cuenow.startTime - cuenow.intime

let loop_time = audioContext.currentTime - loop_startTime

displaytime = dec(loop_time)

let cue_time = audioContext.currentTime - cuenow.startTime

is_cue_time_rolling(cue_time)

It also checks time is rolling,

  the browser waits for user interaction before playing audio,

  so the user may need goaded.

Etc etc!

Svelte 5 holds up well.

We render the upcoming tiles of sound:

// talk to cuelet playing now

let needle_uplink = {}

// rearrange joblist so the first is soonest played

function prioritise_needs(needs) {

if (!needs[0] || !needle_uplink.stat) return

let {cuenow,remains} = needle_uplink.stat()

if (!cuenow) return

// wind needs around to where we are

let many = needs.length

while (needs[0].out < cuenow.out) {

if (many-- < 0) break

needs.push(needs.shift())

}

// and maybe just hop to the next one if soon

if (remains < last_job_time+0.2) {

needs.push(needs.shift())

}

}

Having an interface wired in:

<Schaud {playlets} {needle_uplink}/>

Every time Schaud advances, it points to where you are:

Via:

{#each needles as ne (ne.id)}

<Pointer {ne} />

{/each}

Stores (tweened()) must be declared in the top level of the component, without being in a structure.

If you need structure, as above, make its nodes a component, and put there the stores:

<script lang="ts">

import { tweened } from 'svelte/motion';

import { dec } from './FFgemp';

// a needle

let {ne} = $props();

// they needle.left.set(value) etc

let left = tweened(0,{duration:0})

let top = tweened(0,{duration:0})

let opacity = tweened(0,{duration:0})

ne.left = {set:(v,o)=>left.set(v,o)}

ne.top = {set:(v,o)=>top.set(v,o)}

ne.opacity = {set:(v,o)=>opacity.set(v,o)}

let spanclass = ne.mirror ? 'mirror' : ''

</script>

<soundneedle style="

left:{$left}px;

top:{$top}px;

opacity:{dec($opacity,3)};

">

<span class={spanclass}>

<img src="pointer.webp" />

</span>

</soundneedle>

<style>

soundneedle {

position:absolute;

mix-blend-mode: color-dodge;

pointer-events:none;

margin-left:-1em;

}

soundneedle span.mirror {

position:absolute;

transform: scaleX(-1);

}

soundneedle span.mirror img {

margin-right: -20em;

}

img {

margin-top: -7em;

margin-left: -7em;

}

</style>

Looks pretty great.

Next, making it act good as a looper...

Crossfading

To go one further than https://jackyef.com/posts/building-an-audio-loop-player-on-the-web, which is one further than <audio loop>, we must blend one track into another.

// we shall run this whenever a new cuelet needs to .source and .play()

// also runs at random other times

not sure it does run at random other times.

it seems to run twice, the first doing "new"

function scheduleNextSound(c?) {

c ||= {}

let remarks = []

// some weird time to avoid.

if (!cuelets[0]) return

you might be called to look at weird data sometimes!

// on init

cuenow ||= cuelets[0]

if (!cuenow) throw "!cuenow"

// this shall remain the same for source.onended()'s callback

let cuelet = cuenow

// one of these while playing

if (!cuenow.source) {

remarks.push("new")

let source = cuenow.source = source_cuelet(cuenow)

An AudioBuffer is already made - it is async, involves decoding the audio, so is part of the cuelets arriving in Schaud. In there we simply audioContext.createBufferSource()

// this will find the same one unless crossfading

let needle = cuenow.needle = find_unused_needle()

// crossfade from last

if (c.fadein) {

source.fadein(c.fadein)

}

// play

cuenow.startTime = audioContext.currentTime

source.start(audioContext.currentTime);

Simply this creates seamless transitions. As if time is paused while computing to here from the moment the last sound ended (via source.onended which is coming later)

up_displaytime()

if (c.fadein) {

// clobber the default opacity tween to match the fadetime

needle.opacity.set(0)

needle.opacity.set(1,{duration:fadetime*1000})

}

// crossfade to next

let cuenext = next_cuelet(cuenow)

if (cuenext.in != cuenow.out) {

// discontinuity, probably from looping

let left = source.buffer.duration - fadetime

remarks.push("crossfade in "+left)

setTimeout(() => {

cuelet.source.fadeout(fadetime)

needle.opacity.set(0,{duration:fadetime*1000})

cuenow = cuenext

scheduleNextSound({fadein:fadetime})

},left * 1000)

}

So basically we come back and switch things up before the usual "end" of the sound, thus:

source.onended = () => {

console.log(`Ends ${cuelet.in} ne:${needle.id}`,cuelets.map(cu => cu.needle))

// on the cuelet that was cuenow when it started playing

delete cuelet.source

delete cuelet.needle

// cuenext will already be cuenow and playing (has .source) if crossfading

if (cuelet == cuenow) {

// nobody else (crossfading, ~cuelets) has altered cuenow since we started

cuenow = cuenext

scheduleNextSound();

}

};

}

upto = cuelet.in

if (!cuelets.includes(cuenow)) remarks.push("!in")

console.log(`NextSound: ${cuelet.in} ${dec(audioContext.currentTime)} ${remarks.join(" ")}`)

}

Next, other nice features...