This is the build log for kokoro-ru: the first Russian version of Kokoro, an 82-million-parameter open text-to-speech model. The aim is a model small enough to run offline on a phone-class CPU, with explicit stress control, fixed voicepacks, and an Apache 2.0 license that lets anyone use it commercially. Nothing like that currently exists for Russian; the free options are either robotic (Piper) or large cloning models that need a GPU and a reference clip.
Contents
- Why fine-tune instead of train from scratch
- The model cannot read: phonemes and stress
- Phase 1: the G2P pipeline
- Phase 2: assembling the corpus
- Phases 3 and 4: training in two stages
- Phase 5: voicepacks and local verification
- Evaluation and release
- The one-sentence version
Why fine-tune instead of train from scratch
Training a speech synthesizer from nothing needs hundreds to thousands of hours of audio and serious GPU money, because the model has to learn everything at once: what human voices sound like, how to produce clean audio, how sentences flow. Instead, this project takes the released Kokoro-82M checkpoint, which already speaks English, Chinese, and a few other languages, and continues its training on Russian.
Kokoro is derived from StyleTTS2 and has four parts worth knowing. A phoneme encoder turns phonetic symbols into internal representations. A prosody predictor, helped by a small distilled language model (PL-BERT), decides duration, pitch, and rhythm. A decoder plus vocoder (iSTFTNet) turns all of that into a 24 kHz waveform. And instead of generating speaking style on the fly like full StyleTTS2, Kokoro uses fixed style vectors, one per voice, called voicepacks. That last design choice is why Kokoro has named voices rather than voice cloning, and it is deliberate safety here too: the model cannot imitate an arbitrary person.
Fine-tuning reuses all of it. Every weight loads from the released checkpoint, then training continues on Russian pairs so the weights adapt. The checkpoint already knows how to be a speech synthesizer; it only needs to learn Russian pronunciation patterns. The kikiri-tts project proved this recipe by taking Kokoro to German on 51 hours of audio, so this run copies their recipe move for move, including the data scale.
The model cannot read: phonemes and stress
The single most important fact about the whole build: the model never sees text. It understands only a fixed alphabet of 178 phonetic symbols (IPA), so every sentence, in training and at inference, must first pass through a grapheme-to-phoneme (G2P) converter.
For English that is a solved problem. For Russian it is the hard part of the entire project, because Russian pronunciation hinges on stress and Russian spelling does not mark it. Stress placement is unpredictable, unstressed vowels reduce (an unstressed о sounds closer to а), and some words are spelled identically and differ only by stress: за́мок is “castle”, замо́к is “lock”. Get the stress wrong and every vowel in the word comes out wrong.
The trained model also cannot rescue a bad conversion. It faithfully pronounces whatever phonemes it is handed, so its stress quality has a hard ceiling: it can never be better than the G2P feeding it. And during training a wrong stress mark is worse than a passthrough, because the model is taught mismatched pairs of symbols and sound, which it learns as blurry, inconsistent vowels. That is why G2P came first and was gated before any money was spent.
Phase 1: the G2P pipeline
The converter is a chain of three tools.
First, RUAccent, a small neural network, reads each sentence, decides where the stress falls in every word, restores ё, and resolves homographs from context, writing accent marks into the text. Second, espeak-ng, a rule-based pronunciation engine, converts the accented text into IPA, deriving vowel reduction from the stress marks. This surfaced the project’s nastiest discovery: stock espeak-ng data ignores accent marks entirely, so the Russian dictionary had to be recompiled from the master dictsource for the marks to take effect. With every word acute-marked, espeak’s dictionary is bypassed and its rules derive both stress and reduction from our marks, making RUAccent the single stress authority. Third, a small cleanup layer applies orthoepic respellings the rules miss (the чн→шн set, stressed -ого/-его endings, clusters like солнце and здравствуйте) and remaps a few symbols so everything lands inside the model’s 178-token vocabulary.
The phase gate: 580 test sentences (Wikipedia plus hand-built edge cases) convert with zero out-of-vocabulary symbols, and nine golden tests covering homographs, ё, and consonant clusters all pass.
Phase 2: assembling the corpus
Fine-tuning needs pairs: a recording of a Russian speaker plus the phoneme transcript of what they said. Sources were restricted to license-clean material only, mainly Russian LibriSpeech (public-domain audiobooks, about 98 hours) plus the Dialogs corpus (about 20 hours of studio speech from three actors who explicitly consented). Anything non-commercial or with murky provenance stays out, mechanically, no exceptions.
Everything was resampled to 24 kHz mono, then filtered hard. An automatic quality scorer (DNSMOS) dropped noisy clips below a citable threshold; clips outside 1-15 seconds were dropped; transcripts containing digits were dropped because the phonemizer expands numbers as uninflected cardinals; every transcript ran through the Phase 1 G2P and anything that failed was dropped.
Of 88 usable hours, the best 50 were kept, capped at 12 hours per speaker so no single narrator dominates the model’s idea of Russian.
Final corpus: 27,246 clips, 49.9 hours, 16 speakers, deliberately matching the proven 51-hour German scale.
Output is plain text file lists in the trainer’s format, one line per clip: audio_path|phonemes|speaker_id.
Phases 3 and 4: training in two stages
A Mac cannot do this part, so training runs on a rented 48 GB CUDA GPU (RunPod, L40S class, roughly 50-100 dollars total). First comes a smoke test: one epoch on a tiny 240-line subset, purely to confirm the plumbing works, losses are finite, and checkpoints save and load, before spending real money.
The real run has two stages, inherited from the StyleTTS2 recipe.
Stage 1 is imitation. The model is shown phonemes plus the real recording and learns to reconstruct that recording; the difference between its output and the real audio (the mel loss) drives the weight updates. This teaches Russian sounds but tends to produce flat, averaged speech.
Stage 2 is polish. Adversarial training joins in: judge networks, including a large pretrained speech model, score whether the output sounds real or synthetic, and the model is trained to fool them. This is what pushes output from intelligible to natural. Stage 2 also trains the predictors that guess timing and pitch from phonemes alone, which is what the model must do at inference when there is no real recording to lean on.
Expected wall time is around 20 GPU-hours. Checkpoints get copied off the rented box regularly, because rented storage dies without warning. If quality disappoints at convergence, the playbook says add data hours before touching hyperparameters.
Phase 5: voicepacks and local verification
The trainer’s checkpoint format is not what the kokoro pip package loads, so the checkpoint is converted back to Kokoro’s format and verified to run unmodified on an M4 Pro.
Then voicepacks are extracted: for each speaker, the model has learned a 256-number style vector capturing how that person sounds, and the best few speakers ship as named voices.
Users pick a voice; they cannot clone one.
Evaluation and release
Evaluation is three-pronged. Native speakers blind-compare the output against Piper’s Russian voices, which it must clearly beat. A homograph test set checks stress correctness end to end, both in the frontend and in the audio. And the synthetic audio runs through strong Russian speech recognizers; if the ASR transcribes it accurately, it was intelligible.
Then everything ships on HuggingFace: the checkpoint, voicepacks, the G2P recipe, and a model card with the full dataset lineage and licenses, all Apache 2.0. The corpus file lists and filtering scripts release separately, and the G2P recipe gets offered upstream so official Kokoro can absorb Russian.
The one-sentence version
Convert clean Russian recordings into phoneme-audio pairs with stress handled carefully, resume training an existing small TTS model on them for about a hundred dollars of rented GPU, convert back, extract fixed voices, prove it beats Piper, ship it open.