streaming-audio-analysis

In-memory, sliding-window PCM analysis engine for music streaming applications. Feed decoded PCM from a rolling history buffer and the crate extracts user-interface-facing features faster than real time — tempo (BPM), key, energy, brightness, spectral contrast, roughness, onset density, valence/arousal, a rhythmic-feel classification, and an approximate mood label — laid out on an append-only Timeline of Markers.

Consumption interface

Two independent surfaces over a shared feed/STFT core: LevelsAnalyzer (the display surface — levels only, no timeline, no machines) and Analyzer (the analysis surface — machines and the marker timeline). Each is fed from the same stream on its own thread; the duplicated STFT is deliberate.

LevelsAnalyzer — display levels

LevelsAnalyzer::new(LevelsAnalyzerConfig) returns the feed side plus a LevelsReaderHandle — the only levels read surface, cheaply cloneable, served from an immutable snapshot refreshed inside each feed call, so feed and read never block each other:

• Feed: feed / feed_f32 (native-endian f32 interleaved), optionally *_with_gain
• Hop-maxima levels: levels(bin_count) / levels_log(bin_count) — drain-free &self reads
• Frame peaks (drain-on-read, per-reader state on the handle): levels_frame(bin_count, gain_db) / levels_frame_log(...) (gain_db: Option<f32>)
• Positional levels: levels_at(at, window, gain_db) / levels_at_log(...) — pure &self queries over the retained per-frame levels history (stream-local sample coords, same denomination as latest_at). Strict containment — the window must lie fully inside the retained history or the query errors. Retention span and storage resolution are consumer-configurable via LevelsAnalyzerConfig::levels_retention_seconds (1–600 s) / levels_retention (Linear(n) / Log(n), folded shapes only — required)
• Lifecycle: advance(bytes_dropped) keeps levels history valid across player overlap gaps. There is no rebase/reset — for a seek, drop and re-construct the object (and its handles); levels history is display-grade and blanking the bars is the desired behavior.

There is no timeline, no feature machinery, and no markers_since on this type — the misuse is impossible at compile time. At 2 s retention with Log(128) the levels store is ~88 KB plus STFT buffers.

Analyzer — analysis timeline

A sync, &mut self-based, runtime-agnostic (Send, !Sync) struct:

• Feed: feed / feed_f32 (native-endian f32 interleaved), optionally *_with_gain
• Timeline: timeline() for the marker stream (plus markers, len, and is_empty direct reads); query with timeline().latest_at(FeatureKind, sample_index) for positional reads, and timeline().markers_since(start_watermark) for incremental publication — the new markers since the last read (start-ordered watermark; cheap partition_point + slice). After a backward rebase, re-stamped markers carry starts below any held watermark, so re-derive the watermark after a rebase
• Features: settled_through(FeatureKind) returns the settled watermark, so consumers know how far a feature is stable; features() returns the active FeatureSet
• Lifecycle: advance(bytes_dropped) as history is trimmed, reset() to clear markers, pending state, and history while keeping the configured feature selection, rebase(sample_index) to re-anchor the sample stamp origin at a discontinuity
• Feature selection: set_features(FeatureSet) enables a subset; backfill fills the gap for newly enabled features

This type has no levels surface — levels live only on LevelsAnalyzer.

Optional feature flags: serde (serialize/deserialize derives on Marker, FeatureValue, FeatureKind, Key, TempoClass, MoodState, and the debug types — note FeatureSet is not serde-derivable) and debug (the synthesize_debug mood reference implementation and DebugHop/KeyElection debug output).

Feed-ahead growth

Analyzer memory grows with fed audio up to the configured caps — a freshly constructed analyzer holds near-zero memory, so priming one per stream ahead of playback is cheap. A consumer priming an analyzer ahead of playback feeds until fed_through() — the received frontier: bytes are in the ring, but may not be STFT-framed yet — reaches the target. settled_through(FeatureKind) is the analyzed watermark: the newest closed hop record's start, lagging by the hop cadence (~1 s) plus the 4096-sample STFT lookback and None until a hop closes.

fed_through() and settled_through(FeatureKind) both count mono/per-channel frames, not interleaved samples — stereo audio counts frames, not samples.

Signal levels

• The analyzer reads absolute, un-normalized RMS — there is no AGC anywhere in the crate.
• The mood lane's energy→arousal mapping assumes input at a known integrated loudness: normalize audio before feeding. deezerd normalizes to −15 dBFS/LUFS integrated; feed_with_gain/feed_f32_with_gain is the correction point.
• The stereo mid-downmix (L + R) / 2 costs ~6 dB for correlated content.

Quickstart

use streaming_audio_analysis::{Analyzer, AnalyzerConfig, FeatureKind};
use streaming_audio_analysis::{LevelsAnalyzer, LevelsAnalyzerConfig, LevelsRetention};

let mut analyzer = Analyzer::new(AnalyzerConfig::default())?;
analyzer.feed_f32(&pcm_samples); // f32 interleaved, channels per config

if let Some(marker) = analyzer.timeline().latest_at(FeatureKind::Tempo, 5_000_000) {
    // marker.value: FeatureValue, marker.confidence: f32
}

let cfg = LevelsAnalyzerConfig {
    sample_rate: 44100,
    channels: 2,
    levels_retention_seconds: 2.0,
    levels_retention: LevelsRetention::Log(128),
};
let (mut levels, mut handle) = LevelsAnalyzer::new(cfg)?;
levels.feed_f32(&pcm_samples);

let bars = handle.levels_log(32)?; // log-spaced amplitude bins, matching Log(128) retention
let frame = handle.levels_frame(128, None)?; // per-reader drain-on-read peaks

Errors are ConfigError (invalid configs — rates below 8 kHz, bad channel counts, AnalyzerConfig history windows outside 30–600 s, LevelsAnalyzerConfig retention seconds outside 1–600, bands below 1) and LevelsError (zero bins, insufficient history before the first STFT frame, or a query grid that doesn't match the configured retention).

See the code and rustdoc for additional detail.