I made voice-scraper as more of an experiment to see how good diarization models (ai models that split audio recordings on the basis of speakers) and character voice embedding models are. And i would say they are pretty good as they are really small and can run on just a cpu. Sample: violet_evergarden (on github) How it works: Queries DuckDuckGo or yt-dlp for videos matching the character name and search terms and downloads audio from the search results using yt-dlp and then ffmpeg for converting mp3 to wav. Runs speaker diarization on each clip to detect how many speakers are present and which segments belong to whom, Splits each clip into per-speaker segments. Merges all matched segments into a single _joined.wav file, and may or may not run one final check. The main challenge is figuring out which of the segmented voices belong to the actual character, on supported way is to just use a sample voice (much more reliable). But another way is to find the common voices across all the embeddings and basically guessing it to be the required voice. It is no where near perfect and makes a lot of mistakes but it is a pretty good test for these diarization models. checkout the repo: link I also wrote a blog on its exact working in more detail link submitted by /u/Kartik_2203
Originally posted by u/Kartik_2203 on r/ArtificialInteligence