Speech-to-Creative Pipeline: speech recognition + language swap + lip-sync
One video creative is scaled to 5+ languages and countries automatically. Whisper transcribes speech → GPT-4o translates and rewrites for the market and persona → ElevenLabs clones the voice with the correct accent → wav2lip syncs lip movement to the new audio. What used to require a recording studio and a week of work — now happens in an evening.
Competitor research exists, but it doesn't scale by hand
In performance marketing, ad intelligence services (AdHeart, AdSpy, AdsLibrary, Anstrex) show thousands of competitors' working ads — but you can only borrow what does not trigger a copyright strike. Narrator voice, background, actor's face, lip-sync — everything covered by DMCA requires recreation.
The workflow used to look like this: watch a competitor's video → write the transcript by hand → hand it to a copywriter to rewrite for your own offer → hire voice actor via Fiverr → wait 1-3 days → edit in Final Cut → iterate. 4-6 hours of work per variation, and that's without lip-sync.
With budgets of $2-5k/day on A/B tests, how fast you can produce is the competitive advantage. When you have 8-10 funnels running at once and each needs 20-30 fresh creatives per week — the manual workflow stops working.
Four stages: speech-to-text → language model → speech synthesis → lip-sync
Whisper transcription of competitor videos
whisper-large-v3 or OpenAI whisper-1 via API. Supports Russian, English, Balkan, Turkic, Spanish. Outputs SRT with timecodes — we know exactly which phrase is on which second.
GPT — translation, localization, persona rewrite
GPT-4o does three tasks in one pass: (a) breaks the transcript into compositional blocks — the hook (first 3 seconds), the pain, the proof, the call to action; (b) translates into the target language (Russian / English / Serbian / Polish / Turkish) with cultural nuance — the way "a native speaker would say it"; (c) rewrites it for the target offer, audience and country, preserving the rhythm and emotional triggers of the original.
Few-shot prompts with ready "before → after" examples per language and niche. The main focus is preserving timecodes: each phrase's duration must match the original or lip-sync will break. Output — 10-30 variants × 5+ languages, ranked by predicted engagement.
ElevenLabs — voice cloning with the right accent
ElevenLabs Multilingual v2 supports 29 languages in a single model — the cloned voice sounds in each with the correct accent. This matters: if the original speaker is an American with a Southern accent, her clone in Serbian won't sound robotic but like a natural native Serbian speaker.
Two strategies: (a) voice cloning — a 30-second sample is enough for a high-quality clone; (b) stock voices for fast A/B by timbre (male/female, age, emotional tone). Stability / similarity / style settings are tuned per niche. Output — audio.wav matching the original's duration (important for the next step).
wav2lip — lip movement synchronization
At this stage the "magic" appears — the actress in the video starts speaking in the new language as if it was re-shot. wav2lip analyzes the source video + new audio.wav and redraws the mouth region frame by frame so the lip movement matches the new speech. A GPU is needed, but it's hours of compute.
Simple case (voice-over / off-screen): ffmpeg just replaces the audio track. Complex case (talking head): wav2lip or SadTalker for face sync. Output — finished mp4 for ad platforms (FB Ads / TikTok / VK Ads / Yandex Direct).
Technologies and infrastructure
- OpenAI Whisper API (whisper-1)
- whisper-large-v3 (self-hosted, at large volumes)
- SRT parsing for timecodes
- GPT-4o / Claude Sonnet 4.5
- few-shot prompts with hook → rewrite examples
- structured outputs (JSON schema)
- ElevenLabs Multilingual v2
- Voice cloning (30-second sample is enough)
- Voice style settings: stability / similarity / style
- ffmpeg for audio-replace (voice-over)
- wav2lip / SadTalker for talking head
- Python orchestration + task queue
What changes in the funnel
per creative variation (audio-replace, no lip-sync)
variants × languages (1 actress → 5+ countries without a re-shoot)
Whisper + GPT + ElevenLabs (at API rates, no lip-sync)
The main effect is iteration speed. An A/B test of 20 opening hooks instead of 2 launches in an evening. The winning combinations show up in the first 24-48 hours; the losers are switched off before they burn budget. Cost per install (CPI) drops 15-30% because the opening matches the audience better.
The same speech synthesis beyond ad creatives
The voice step stands on its own, away from video. Two things run on it today: a spoken notice for telephony, and voice messages sent in a messenger on the manager's behalf through a business account. In both, the voice is fixed and comes out of the same pipeline, so the delivery sounds identical across channels.
The outbound-call line follows a fixed order
The caller is named first, then the line says the conversation is being recorded. The purpose of the recording stays out of the wording. The order is baked into the script, so every call opens the same way.
Audio is rebuilt from the script
The repo holds the synthesis script and the files are regenerated in seconds: two files, 4.0 and 3.2 seconds long. What the version history keeps is the wording of the line.
The voice bubble renders for one container only
A messenger draws the voice-note bubble for a single container format. Any other container arrives as a plain file, so the format is pinned right at the output of the synthesis step.
Number-heavy breakdowns stay in text
Voice carries short notices. Summaries with numbers go out as text, where a reader can glance back at a figure and reread it as many times as it takes.
Open blocker: outbound calls
With the telephony provider, audio playback lives in the inbound-call flow. There is nowhere to drop a prepared file into an outbound call today; a support request is open.
A free local stand-in for paid synthesis
The stack on this page runs a paid synthesis service, and every other client asks what happens without a subscription. The candidate picked for that role is an MIT-licensed model: it handles Russian and clones a voice from a 10-second sample. The two best-sounding models were ruled out by their non-commercial licences. For now the choice sits on paper: the working pipeline runs the paid service.
Key hygiene
The paid service key was reissued to replace one that had surfaced in a log. The replacement carries speech-synthesis rights only, every other API endpoint is closed to it, and a spending cap is set explicitly.
Where it works, where it doesn't
- · Performance agencies with an in-house creative team of 5+
- · Performance teams and online retailers producing a lot of video
- · In-house e-commerce marketing that publishes user-made video regularly
- · Launches in 5+ countries simultaneously (multi-language from one master script)
- · Podcasters and online-course businesses cutting short-form clips
- · Direct copying of other people's creatives (DMCA / Copyright Strike)
- · Regulated niches (medicine, pharma, finance) — need media lawyers for compliance
- · Voice cloning without consent of the voice owner (banned by EU AI Act)
- · Long-form (10+ minute videos) — TTS cost and editing time comparable to an expensive human actor
Ethical note: the pipeline is intended for scaling your own ideas and original scripts. Using others' videos and voices without permission violates Copyright and the AI Act. I transcribe other people's ads as research to understand the market, then write my own script, my own audio, and my own video.
Related cases
GMBSPYLAB — collecting the public FB Ad Library into my own Telegram feed
Async Python and Postgres 16: 30–75k raw cards a day, perceptual dedup leaves 5–12k unique ones, arq queues…
End-to-end attribution: from ad click to order
A stdlib middleware takes Shopify webhooks, verifies HMAC, fires an S2S postback to the tracker and a…
Paid Shopify orders → warehouse spreadsheets
Orders, tracking numbers, statuses and cancellations land by themselves every 30 minutes. An Advanced Sheets…
The 9,900 ₽ audit — with a concrete report and quote
I'll tell you what to deploy in your business first, what the payback looks like, and whether you need AI for the task at all (sometimes you don't).
Or just send your question — I reply within 2 hours