disclosure: i built Tolop which is what this post is about this started as a personal frustration with getting surprised by paywalls. turned into a spreadsheet, then a proper dataset, then a site. here is the honest technical breakdown of how i approached it and where the methodology falls short. the approach: every tool gets scraped directly from its pricing page rather than relying on marketing copy. i built a scraper in Node.js that hits each tool’s pricing and feature pages, strips the HTML to readable text, and passes it to an LLM via OpenRouter with a strict system prompt that says “use only what is on this page, do not use your training data.” the output gets structured into a consistent JSON schema across all 135 tools. the schema forces every tool into the same fields regardless of how different they are - free tier summary, feature limits with notes, exhaustion estimates for three usage profiles (light, moderate, heavy), a verdict, and a verdict color. forcing a CLI agent and a browser based app builder into identical fields loses nuance but makes cross category comparison possible which is the whole point. the exhaustion estimates specifically: these are the most useful and most contested part of the dataset. for each tool i estimate how long the free tier lasts under light use (occasional completions, minimal chat), moderate use (active daily development, regular chat), and heavy use (agentic sessions, multi file editing). the estimates come from a combination of the published limits, community reports on Reddit and Discord, and in some cases direct testing. they are estimates not guarantees and i flag them with a data as of date on every entry because pricing changes constantly. what i found that surprised me: the gap between most and least generous free tiers is 90x in the same category. Gemini Code Assist gives 180,000 completions per month. GitHub Copilot Free gives 2,000. identical marketing language, completely different reality. a significant portion of tools with millions of installs are not free in any meaningful sense. Cline, Aider, Continue, Roo Code all require your own API key. the tool costs nothing. Claude Sonnet costs $15 per million output tokens. an active agentic session can cost $5-20 per day. the schema flags these separately as requiresOwnApiKey: true. daily reset limits are structurally better for developer workflows than monthly caps. this seems obvious in hindsight but most tools have not figured it out. the limitations i have not fully solved: javascript rendered pricing pages are the biggest problem. a significant number of tools render their pricing entirely in the browser which means the scraper gets an empty shell. for those i fall back to manual research or screenshots which introduces inconsistency. the schema forces a single verdict color per tool which oversimplifies. a tool can have a genuinely unlimited free completion tier and a terrible free chat tier simultaneously. the current schema averages this into one color which loses information. data freshness is a constant problem. pricing changes without announcement. i have a data as of date on every entry but there is no automated alert when a tool changes its limits. the scraper needs to run on a schedule and flag changes for manual review which i have not fully built yet. the exhaustion estimates for agentic tools are the least reliable part of the dataset because token consumption varies enormously by task complexity. a simple autocomplete session and a complex multi file refactor can differ by 100x in token usage. lessons learned: forcing a consistent schema early was the right decision even though it felt constraining. the alternative - letting each tool define its own structure - would have made comparison impossible. building the scraper before building the UI was backwards. i spent weeks on the bookshelf UI before the data pipeline was solid. the data quality should have come first. the requiresOwnApiKey distinction is the most practically useful field in the schema and i almost did not include it because it seemed obvious. it is not obvious to most developers adopting these tools. submitted by /u/DAK12_YT
Originally posted by u/DAK12_YT on r/ArtificialInteligence