What I built: Buddy, an Android app with a floating chat bubble. You ask for something (“reply to Sam that I’m running late”) and an agent reads the screen, taps and types until it’s done. How it works: - The agents (Claude Code, Codex, OpenCode, Antigravity CLI) run inside the app in a small Alpine Linux (proot). No PC, no root, no adb. - Buddy gives them phone tools through a local MCP server: read the screen, tap, type, scroll, open apps, screenshots, plus a headless Chromium for web pages. - The screen is read through Android’s accessibility tree as text. A button with no text falls back to its view id, then to a screenshot. - A guard sits between the agent and every tap: Send, Pay and Delete get an approval card, each app gets a mode (Trust, Hard, or a second model reviewing), and the agent never types into password fields. What I learned: - Reading the accessibility tree as text is much faster and cheaper than screenshots. Screenshots are only worth it for canvas-drawn apps. - Safety has to live in the app, not the prompt. A prompt can be ignored; a hard stop before Send can’t. - Agents need to re-read the screen after every action. Dialogs and loading screens change things under them. Repo (Apache 2.0): https://github.com/ghorbelhamdi/buddy-android submitted by /u/3amae404
Originally posted by u/3amae404 on r/ArtificialInteligence