🌐 Web Tools

Three scripts for web search and content extraction. All pre-installed in image v11.0+ — no per-generation pip needed.

web_search.py

Query the web via Brave Search; get titles, URLs, snippets.

• Positional: query string
• --count default 5, capped at 10
• Needs BRAVE_SEARCH_API_KEY in executor environment
• Output: pretty-printed JSON array (not the ok/error envelope)

web_fetch.py

HTTP GET a URL and extract main text (strips boilerplate via trafilatura). Use this first for normal articles and docs — fast and lightweight.

• Positional: url
• Large pages truncated (tens of thousands of chars)
• JS-only sites may return no extractable text — stderr explains

web_render_fetch.py

Open a URL in headless Chromium and extract rendered page text. Use only when web_fetch.py cannot read a page because content is JS-rendered.

• Positional: url
• Optional: --timeout, --wait-until (domcontentloaded|load|networkidle), --max-chars, --lean
• Much slower than web_fetch.py — for one-off high-value pages, not batches
• No login, cookies, forms, file downloads, or screenshots
• Blocks local and private network targets
• If heavy page fails with ERR_INSUFFICIENT_RESOURCES, script retries once with nonessentials blocked