# Screenshot to Steps

> Turn a screenshot of an app or website into numbered steps for what you want to do. Upload a screenshot (or paste the visible labels) and say what you want to achieve. The text on screen is read with OCR, each label is classified as button, input field, navigation, link, checkbox or heading with its position, and the elements relevant to your goal are turned into numbered instructions.

Part of usefulweb.site (https://usefulweb.site/). Results are deterministic calculations from the inputs and the listed assumptions; the API returns exactly what the web page shows.

## Use it
- Human interface: https://screenshottosteps.usefulweb.site/ (interactive tool at https://screenshottosteps.usefulweb.site/tool)
- JSON API: POST https://screenshottosteps.usefulweb.site/api/v1/calculate with a JSON object of inputs (schema: https://screenshottosteps.usefulweb.site/api/v1/tool)
- OpenAPI: https://screenshottosteps.usefulweb.site/openapi.json
- MCP (Streamable HTTP): https://screenshottosteps.usefulweb.site/mcp - tool `screenshot_to_steps`

## Inputs
- goal (text, required): What do you want to do?
- file (file): Screenshot - PNG or JPG, max 5 MiB. Read with OCR on our server; not stored.
- uiText (textarea): Or: visible labels, one per line (top to bottom)

## Method
- OCR: tesseract (English) with word positions; words with confidence below 30 are dropped; words on one line are split into separate elements at gaps wider than 1.8× the text height.
- Roles are assigned by rules: known action words → button; trailing colon or known field names → input field; short labels in the top 12% or known section names → navigation; URLs/"forgot…" → link; ☐/[ ] → checkbox; text 1.5× taller than median → heading.
- Your goal is split into keywords and expanded with synonyms (e.g. password → security, login). Steps: open the best-matching navigation item (if no matching field is visible), fill visible input fields, tick relevant checkboxes, then click the primary button (Save/Continue/OK…).
- Position is reported as a 3×3 grid region (top/middle/bottom × left/center/right).

## Limits
- Steps are derived only from text visible in the screenshot; icons without labels and hidden menus cannot be seen. Never upload screenshots that show passwords or card numbers.