Disclosure: I’m the developer of this project. Over the last few months I kept running into the same workflow problem. LLMs have become very good at reasoning over text. We have RAG pipelines, long-term memory, vector databases, and increasingly capable agents that can search documentation, code, and previous conversations. Videos felt completely different. A typical workflow looked like this:
- Upload a bug recording or product demo.
- Ask a few questions.
- End the session.
- Upload the same video again the next day. The model wasn’t the problem. The workflow was. So I built an open-source project called Watch Skill to experiment with a different approach. Instead of treating a video as temporary input, the first analysis creates a persistent local index containing transcripts, OCR, visual observations, timestamps, and embeddings. Future questions become retrieval instead of video processing. The project exposes the same functionality through MCP, a CLI, and a REST API, so it isn’t tied to one specific agent. It also supports running fully offline with local models if that’s important for privacy or cost. Some implementation decisions that ended up mattering more than I expected:
- Scene detection instead of uniform frame extraction.
- Hybrid retrieval (FTS + embeddings) instead of vector search alone.
- Timestamped evidence for every answer.
- Persistent indexing so follow-up questions don’t require another full video analysis.
- A local correction system where mistakes can become reusable lessons for future queries I’m posting this mainly because I’m interested in feedback on the architecture rather than promotion. If you’re building agent systems that work with video, would you solve this differently? Repository: https://github.com/oxbshw/watch-skill submitted by /u/Fearless-Role-2707
Originally posted by u/Fearless-Role-2707 on r/ArtificialInteligence
You must log in or # to comment.