Audio Pipeline Implementation

Caution

This is advanced stuff that you will probably have no idea about if you're a beginner. But! If you're coming here to just read about it, you'll be learning about it if you do proper research.

Note

Nearly this whole page was generated with claude.ai, and has had manual adjustments by the person who proofreads this article.

A comprehensive C++ audio playback system featuring multi-track mixing, lock-free multithreading, sound effects, background music, and time-stretching capabilities. Includes Haxe bindings for easy integration with Haxe/OpenFL projects.

Overview

This audio system provides three main components:

1. AudioSystem - Multi-track music player with time-stretching support
2. AudioMixerManager - Real-time background music and sound effects mixer
3. Platform-specific audio device detection (Windows)

The system uses miniaudio for cross-platform audio I/O, signalsmith-stretch for time-stretching, and stb_vorbis for Vorbis decoding.

Features

• Multi-track playback with individual volume control
• Time-stretching for variable playback rates without pitch change
• Background music with seamless looping
• Sound effect pools supporting overlapping instances
• Automatic latency detection based on audio content
• Device detection (headphones, USB/Bluetooth devices on Windows)
• Lock-free triple-buffered streaming for smooth playback
• RAII design for automatic resource management

Architecture

AudioSystem (Multi-track Player)

The core music playback system supporting multiple synchronized audio tracks.

Key Components

• DecoderStream: Triple-buffered decoder with padding for smooth seeking
• Buffer Management: Triple-buffered sliding window (padding + main buffer + padding)
• Time Stretching: Optional pitch-preserving speed changes via signalsmith-stretch

Buffer Strategy

[PADDING_MS] [BUFFER_MS] [PADDING_MS]
    100ms        750ms        100ms

• Refills when playback reaches nearly halfway through buffer
• Padding enables smooth seeking and continuity
• Total buffer: 950ms with automatic refilling

API Reference

Multi-track Music System

C++ API

// Load and initialize tracks
void loadFiles(std::vector<const char*> filePaths);

// Playback control
void start();                          // Start/resume playback
void stop();                           // Pause playback
bool stopped();                        // Check if stopped
void destroy();                        // Clean up all resources

// Seeking
void seekToPCMFrame(int64_t frame);   // Seek to specific frame

// Playback info
int getMixerState();                   // 1=playing, 2=paused, 3=stopped
double getPlaybackPosition();          // Current position in ms
double getDuration();                  // Total duration in ms

// Track control
void deactivate_decoder(int index);    // Mute specific track
void amplify_decoder(int index, double volume);  // Set track volume

// Time stretching
void setPlaybackRate(float rate);      // Speed without pitch change (0.5-2.0)

// Volume
double getGlobalVolume();
double setGlobalVolume(double volume);

Haxe API

// Load and initialize tracks
MiniAudio.loadFiles(["vocals.ogg", "drums.ogg", "bass.ogg"]);

// Playback control
MiniAudio.start();
MiniAudio.stop();
var isStopped:Bool = MiniAudio.stopped();
MiniAudio.destroy();

// Seeking
MiniAudio.seekToPCMFrame(frameNumber);

// Playback info
var state:Int = MiniAudio.getMixerState();  // 1=playing, 2=paused, 3=stopped
var pos:Float = MiniAudio.getPlaybackPosition();  // ms
var duration:Float = MiniAudio.getDuration();     // ms

// Track control
MiniAudio.deactivate_decoder(trackIndex);
MiniAudio.amplify_decoder(trackIndex, 0.8);

// Time stretching
MiniAudio.setPlaybackRate(1.5);

// Volume
var volume:Float = MiniAudio.getGlobalVolume();
MiniAudio.setGlobalVolume(0.9);

AudioMixerManager (Background & SFX)

Real-time mixer for background music and sound effects running on a separate audio device.

Components

• BackgroundTrack: Streaming decoder with looping support
• SoundEffectPool: Pre-loaded sound with up to 16 simultaneous instances
• Thread-safe mixing: Mutex-protected buffer mixing

Background Music

C++ API

// Load background track
int loadBackgroundTrack(const char* path);

// Playback control
void playBackgroundTrack(int index);
void stopBackgroundTrack(int index);
bool isBackgroundTrackPlaying(int index);

// Configuration
void setBackgroundTrackVolume(int index, float volume);
void setBackgroundTrackLooping(int index, bool looping);

Haxe API

// Load and play background track
var bgIndex:Int = MiniAudio.loadBackgroundTrack("assets/music/ambient.ogg");

// Playback control
MiniAudio.playBackgroundTrack(bgIndex);
MiniAudio.stopBackgroundTrack(bgIndex);
var isPlaying:Bool = MiniAudio.isBackgroundTrackPlaying(bgIndex);

// Configuration
MiniAudio.setBackgroundTrackVolume(bgIndex, 0.7);
MiniAudio.setBackgroundTrackLooping(bgIndex, true);

Sound Effects

C++ API

// Load sound effect (pre-loads entire file)
int loadSoundEffect(const char* path);

// Play (can overlap with itself up to 16 times)
void playSoundEffect(int index, float volume = 1.0f);

// Control
void stopSoundEffect(int index);       // Stop all instances
bool isSoundEffectPlaying(int index);  // Check if any playing

// Cleanup
void destroyMixer();

Haxe API

// Load sound effects
var clickSfx:Int = MiniAudio.loadSoundEffect("assets/sounds/click.ogg");
var explosionSfx:Int = MiniAudio.loadSoundEffect("assets/sounds/explosion.ogg");

// Play (can overlap with itself)
MiniAudio.playSoundEffect(clickSfx, 1.0);
MiniAudio.playSoundEffect(clickSfx, 1.0);  // Second instance overlaps

// Stop all instances
MiniAudio.stopSoundEffect(explosionSfx);

Mixer Volume Control

C++ API

void setMixerMasterVolume(float volume);
float getMixerMasterVolume();

Haxe API

MiniAudio.setMixerMasterVolume(0.8);

Usage Examples

Basic Multi-track Playback (C++)

// Load multiple tracks (e.g., stems)
std::vector<const char*> tracks = {
    "vocals.ogg",
    "drums.ogg",
    "bass.ogg",
    "guitar.ogg"
};
loadFiles(tracks);

// Start playback
start();

// Mute drums
deactivate_decoder(1);

// Lower bass volume
amplify_decoder(2, 0.5);

// Slow down to 80% speed
setPlaybackRate(0.8f);

// Seek to 30 seconds
seekToPCMFrame(30 * 44100);

// Clean up
destroy();

Basic Multi-track Playback (Haxe)

// Load multiple tracks
MiniAudio.loadFiles([
    "assets/music/vocals.ogg",
    "assets/music/drums.ogg",
    "assets/music/bass.ogg",
    "assets/music/guitar.ogg"
]);

// Start playback
MiniAudio.start();

// Mute drums (track index 1)
MiniAudio.deactivate_decoder(1);

// Lower bass volume
MiniAudio.amplify_decoder(2, 0.5);

// Speed up to 120%
MiniAudio.setPlaybackRate(1.2);

// Seek to 30 seconds (30000ms * 44.1 samples/ms)
MiniAudio.seekToPCMFrame(1323000);

// Clean up
MiniAudio.destroy();

Device Detection (Haxe)

// Detect audio device configuration
var hasHeadphones:Bool = MiniAudio.wearingHeadphones();
var isPnPDevice:Bool = MiniAudio.wearingPlugNPlay();

// Get calculated latency for A/V sync
var latencyMs:Int = MiniAudio.detectLatency();
trace('Audio latency: ${latencyMs}ms');

// Adjust video timing accordingly
videoPlayer.setAudioOffset(latencyMs);

Background Music with Sound Effects (C++)

// Load and play background music
int bgMusic = loadBackgroundTrack("ambient.ogg");
setBackgroundTrackVolume(bgMusic, 0.7f);
setBackgroundTrackLooping(bgMusic, true);
playBackgroundTrack(bgMusic);

// Load sound effects
int sfxClick = loadSoundEffect("click.ogg");
int sfxExplosion = loadSoundEffect("explosion.ogg");

// Play sound effects (can overlap)
playSoundEffect(sfxClick, 1.0f);
playSoundEffect(sfxClick, 1.0f);  // Overlaps with previous
playSoundEffect(sfxExplosion, 0.8f);

// Master volume control
setMixerMasterVolume(0.9f);

// Clean up
destroyMixer();

Background Music with Sound Effects (Haxe)

// Load and configure background music
var bgMusic = MiniAudio.loadBackgroundTrack("assets/music/ambient.ogg");
MiniAudio.setBackgroundTrackVolume(bgMusic, 0.7);
MiniAudio.setBackgroundTrackLooping(bgMusic, true);
MiniAudio.playBackgroundTrack(bgMusic);

// Load sound effects
var clickSfx = MiniAudio.loadSoundEffect("assets/sounds/click.ogg");
var explosionSfx = MiniAudio.loadSoundEffect("assets/sounds/explosion.ogg");

// Play overlapping sound effects
MiniAudio.playSoundEffect(clickSfx, 1.0);
MiniAudio.playSoundEffect(clickSfx, 1.0);  // Can play simultaneously
MiniAudio.playSoundEffect(explosionSfx, 0.8);

// Control master volume
MiniAudio.setMixerMasterVolume(0.9);

Adaptive Playback Position Tracking (Haxe)

class MusicPlayer {
    var updateTimer:Timer;
    
    public function new() {
        // Update UI every 100ms
        updateTimer = new Timer(100);
        updateTimer.run = updatePlaybackDisplay;
    }
    
    function updatePlaybackDisplay() {
        var position = MiniAudio.getPlaybackPosition();
        var duration = MiniAudio.getDuration();
        var state = MiniAudio.getMixerState();
        
        // Update progress bar
        var progress = position / duration;
        progressBar.value = progress;
        
        // Update time display
        timeLabel.text = formatTime(position) + " / " + formatTime(duration);
        
        // Check if finished
        if (state == 3) {  // Stopped
            onTrackFinished();
        }
    }
    
    function formatTime(ms:Float):String {
        var seconds = Math.floor(ms / 1000);
        var minutes = Math.floor(seconds / 60);
        seconds = seconds % 60;
        return minutes + ":" + StringTools.lpad(Std.string(seconds), "0", 2);
    }
}

Adaptive Latency Detection

// System automatically detects latency based on:
// - Audio content (silence before first sample)
// - Device type (built-in vs USB/Bluetooth)
// - Headphone connection
// - OS baseline latency

int latencyMs = detectLatency();
printf("Detected latency: %d ms\n", latencyMs);

// Use for audio/visual synchronization
int syncOffset = latencyMs;

Audio Configuration

Constants

#define SAMPLE_FORMAT ma_format_f32
#define CHANNEL_COUNT 2
#define SAMPLE_RATE 44100

// Buffer sizes
#define PADDING_MS 25         // Seek padding
#define BUFFER_MS 500         // Main buffer
#define HALF_BUFFER_MS 225    // Refill threshold

Supported Formats

• WAV, FLAC, MP3, Vorbis/OGG
• Any format supported by miniaudio decoders
• Automatic resampling to 44.1kHz stereo

Haxe Integration

Target Support

The Haxe bindings support multiple compilation targets:

• C++ target (#if cpp): Full native performance
• HashLink target (#if hl): Native extensions via @:hlNative
• Other targets: Stub implementation (no audio)

Build Configuration

For C++ target, include the build XML:

<!-- miniaudioBuild.xml -->
<xml>
  <files id="haxe">
    <compilerflag value="-I${haxelib:your-lib}/cpp/include"/>
    <file name="${haxelib:your-lib}/cpp/ma_thing.cpp"/>
  </files>
</xml>

String Handling

The bindings handle string conversion automatically:

// Haxe strings are automatically converted to native format
MiniAudio.loadBackgroundTrack("path/to/file.ogg");

// Array of strings converted to native vector (C++)
MiniAudio.loadFiles(["track1.ogg", "track2.ogg"]);

Memory Management

The system uses RAII (Resource Acquisition Is Initialization) principles:

• Automatic cleanup on destruction
• Move semantics for efficient resource transfer
• Deleted copy constructors to prevent issues
• Manual destroy() methods for explicit cleanup

Haxe Memory Considerations

When using from Haxe:

• C++ target: RAII handles cleanup automatically
• HashLink target: Native memory managed by HL runtime
• Always call destroy() when done to free resources immediately

Thread Safety

• AudioSystem: Single audio callback thread, no locks needed
• AudioMixerManager: Mutex-protected for thread-safe API calls
• Sound effect pools use pre-allocated instances to avoid runtime allocation

Latency Detection

The system automatically detects audio latency by:

1. Scanning first 100ms of each track for signal above threshold (0.1)
2. Recording maximum latency across all tracks
3. Adding OS-specific baseline latency
4. Adjusting for device type (headphones, USB/Bluetooth)

// Latency calculation formula
int totalLatency = osBaseline + deviceLatency - contentLatency;

Windows baseline: 60ms

• Add 50ms if not PnP device (built-in audio)
• Add 40ms if headphones detected
• Subtract detected content latency

Device Detection (Windows)

bool wearingHeadphones();              // Detect headphone connection
bool wearingPlugNPlay();               // Detect USB/Bluetooth devices
int detectLatency();                   // Calculate system latency in ms

The system checks device names and IDs for keywords:

• Headphones: "headphone", "headset", "earphone", "earbud", "airpod"
• PnP devices: USB, Bluetooth, HID devices
• Internal devices: HD Audio, Realtek, Intel integrated audio

Platform Support

• Cross-platform: Windows, macOS, Linux (via miniaudio)
• Windows-specific features: Headphone/device detection via WASAPI
• Haxe targets: C++, HashLink (with native extensions)

Performance Considerations

• Buffer size: 500ms main buffer provides smooth playback
• Refill threshold: 225ms prevents buffer underruns
• Sound effects: Pre-loaded for zero-latency playback
• Instance pooling: Up to 16 overlapping sound instances per effect
• Time stretching: "cheaper" preset for real-time performance

Dependencies

• miniaudio - Audio I/O
• [signalsmith-stretch](https://github.com/Estrol/signalsmith-stretch) - Time stretching
• [stb_vorbis](https://github.com/nothings/stb) - Vorbis decoding
• Windows: ole32.lib for device detection

Building

C++ Implementation

Include the following in your project:

#include "ma_thing.h"

The implementation uses single-header libraries:

• Define MINIAUDIO_IMPLEMENTATION once
• Define SIGNALSMITH_STRETCH_IMPLEMENTATION once
• Define STB_VORBIS_IMPLEMENTATION once

Haxe Project

This audio system is integrated directly into Funkin' View as an internal system working under the hood. Refer to Extern API for more details.

Troubleshooting

Common Issues

No audio output

• Check if initialize() or loadFiles() was called successfully
• Verify file paths are correct and accessible
• Check system audio settings and default device

Crackling or stuttering

• Check CPU usage

Sound effects not playing

• Verify sound effect was loaded successfully (check return value >= 0)
• Check if MAX_INSTANCES (16) limit reached
• Ensure mixer is initialized

Sync issues with video

• Use detectLatency() for automatic compensation
• Manually adjust with seekToPCMFrame() if needed
• Test on target hardware (latency varies by device)

Haxe-specific Issues

Linking errors (C++ target)

• Verify build XML is correctly included
• Check that ma_thing.cpp is being compiled
• Ensure all dependencies are linked

Runtime errors (HashLink)

• Verify native library (.hdll) is in correct location
• Check that @:hlNative bindings match C function signatures
• Rebuild native extensions after changes

Limitations

• Maximum 16 simultaneous instances per sound effect
• Time stretching range: 0.5x - 2.0x (practical limits)

Future Enhancements

• Nothing, really. Unless I have to just in case.

---

License: Check individual library licenses (miniaudio, signalsmith-stretch, stb_vorbis)

Contributing: Submit issues and PRs to improve the system

Support: For Haxe-specific questions, check the Haxe community forums and OpenFL documentation