Guide: Local Audiobook Pipeline Architecture & Setup

Converting lengthy books and documents into high-quality audio is traditionally dependent on expensive cloud APIs, heavy GPU cluster infrastructures, or tedious manual audio engineering workflows.

My Local Audiobook Pipeline project was created to solve these exact challenges. It establishes a completely local, crash-resilient, manifest-driven workflow utilizing open-source tools running on consumer-grade hardware.

💻 Source Code & Repository: Access full source code, configuration templates, and setup guidelines on GitHub (vyscnktn/local-audiobook-pipeline). Stars and contributions are welcome!

Why Build a Local Pipeline?

Synthesizing a short text snippet is easy. However, generating an entire audiobook from raw EPUB files presents real systems engineering challenges:

• Data Noise: EPUB files contain unstructured cover pages, boilerplate front matter, malformed XHTML tags, and footnotes that disrupt speech synthesis.
• Hardware Limitations: Submitting long book chapters directly to a TTS model causes Video RAM (VRAM) spikes and Out-Of-Memory (OOM) crashes on consumer GPUs.
• Fault Tolerance: If a 5-hour render process encounters an error 4 hours in, the pipeline must seamlessly resume from the exact checkpoint without re-synthesizing completed audio chunks.

Rather than operating as a fragile “black box” script, this project divides the workload into an inspectable, manifest-driven pipeline.

System Architecture & Workflow

The architecture deliberately avoids brittle single-script execution. The job pipeline is decoupled into 7 independent, resumable stages:

1. Ingestion: Parses EPUB structures, metadata, and raw XHTML content.
2. Cleanup: Transforms raw XHTML into clean, structure-preserving plain text (JSONL format) while stripping non-narrative artifacts.
3. Manifest Generation: Constructs chapter-level manifests tracking narration progress.
4. Job Generation: Chunks chapter text into GPU-safe, fixed-size inference jobs.
5. Synthesis: Synthesizes .wav audio segments using Chatterbox Multilingual TTS.
6. Assembly: Stitches individual audio chunks into coherent chapter audio files with configurable crossfading and silence padding.
7. QA (Quality Control): Generates machine-readable reports detailing success rates and missing jobs.

Hardware Profile & VRAM Optimization

The pipeline is engineered to run efficiently on modest consumer hardware. Development and stress tests were conducted on:

• GPU: NVIDIA GeForce RTX 4050 Laptop GPU
• VRAM: 6 GB
• Runtime: Python + PyTorch + CUDA
• TTS Engine: Chatterbox Multilingual TTS

By enforcing conservative text chunking strategies, VRAM utilization remains strictly capped below 5.5 GB, avoiding OOM failures even during extended multi-hour rendering sessions.

Step-by-Step Installation & Usage

Follow these steps to run the pipeline locally on your machine:

1. Environment Setup

python -m venv .venv-chatterbox
source .venv-chatterbox/bin/activate
pip install -r requirements.txt

2. Configuration

Copy the example configuration file and specify your speaker reference audio target:

cp config.example.yaml config.yaml

3. Ingestion & Text Cleanup

Ingest an EPUB book file and clean up HTML tags:

# Ingest raw book
python ingest_epub.py --book my-book --epub /path/to/my-book.epub

# Clean XHTML artifacts
python clean_epub.py --book my-book

4. Manifest & Job Building

Generate synthesis manifests without triggering GPU inference yet:

python build_book_manifests.py
python run_chatterbox_chapter.py --chapter-id <chapter-id> --rebuild-jobs --jobs-only

5. Render & Synthesis

Execute GPU inference to generate the final audio files:

python run_chatterbox_chapter.py --chapter-id <chapter-id>

For advanced configuration options, NVIDIA NIM container support, and contributing guidelines, visit the GitHub Repository.