Docs & integrations
Make your first call from cURL, Python, JavaScript, n8n or an AI assistant.
Start here
- Create an account (email and a password of 8+ characters; no credit card).
- Create an API key and copy it straight away: the full key is shown only once.
- Send it on every request as a header:
Authorization: Bearer sk_live_…, with a JSON body saying what to do.
You can try the page endpoints without a key: anonymous calls are capped per IP address (a few requests at a time, a small number of unfinished crawl or batch jobs, at most 200 pages per crawl, 50 URLs per batch). A key lifts those caps to your organisation's limits and unlocks request history, monitors, workflows, webhooks and the growth products.
The snippets below use YOUR_API_KEY. Sign in to see your key name filled in.
Your first two calls
curl -X POST https://api.hardstuck.ai/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "output_format": "markdown"}'curl -X POST https://api.hardstuck.ai/crawl \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'Which endpoint do I use?
All of them start with https://api.hardstuck.ai. The full interactive reference for every field is at api.hardstuck.ai/docs.
| I want to… | Use this endpoint | What you send |
|---|---|---|
| Get the clean text/content of one web page | POST /scrape | {"url": "https://example.com"} |
| Get one page's content plus a simple list of its links and title | POST /crawl | {"url": "https://example.com"} |
| Pull every page from a whole site (a blog, docs site, etc.) | POST /crawl/site | {"url": "https://example.com", "max_pages": 20} |
| Get a list of every URL on a site, without fetching any content | POST /map | {"site": "https://example.com"} |
| Pull specific fields (a price, a title, an author) instead of full text | POST /extract | {"url": "...", "schema": {"price": "string"}} |
| Take a picture of a page | POST /screenshot | {"url": "https://example.com", "full_page": true} |
| Click buttons, fill in forms, or navigate multiple steps on a site | POST /workflows, then POST /workflows/{id}/run | a list of steps (see Advanced → Workflows) |
| Get told automatically when a page changes | POST /monitors | {"url": "...", "schedule": "daily"} |
| Get events pushed to your own server the moment something happens | POST /webhooks | {"target_url": "...", "events": ["monitor.changed"]} |
| Search the web for a topic | POST /search | {"query": "...", "max_results": 10} |
| Search the web AND read the top results (deeper, slower) | POST /research | {"query": "...", "max_results": 5} |
| Scrape a long list of URLs in the background | POST /batch, then GET /jobs/{id} | {"urls": ["...", "..."]} |
Troubleshooting
| Code | What you see | What it means | What to do |
|---|---|---|---|
| 401 | “not authenticated” or “a valid API key is required” | Your API key is missing, mistyped, revoked, or the word Bearer is missing before it. | Check the header reads exactly Authorization: Bearer sk_live_… (with a space after Bearer). |
| 402 | Payment Required from a growth product | Valid key, but your workspace is not subscribed to that specific product. | Plans are listed on the ScorchCrawl page (hardstuck.ai/scorchcrawl/#plans). Checkout is not connected yet, so email support (the address at the bottom of the console navigation) to have the product enabled for your workspace; the key itself is fine. |
| 403 | FORBIDDEN with a scope name | The key was created without that scope (e.g. a search-only key used on /scrape). | Create a key that includes the scope named in the message; the default key has scrape, crawl, extract, search. |
| 409 | “already_decided” | A decide call on a suggestion or flag that was already decided. | Repeat the call with ?force=1 to override the earlier decision. |
| 409 | from a product's /v1/generate | A generation run is already active for that project. | Poll GET /v1/generate/{run_id} until it is done or failed, then start another. |
| 422 | validation error | The JSON you sent does not match what the endpoint expects: a missing url field, or a typo in a field name. | Compare your body against the examples here, or check api.hardstuck.ai/docs for exact field names. |
| 429 | RATE_LIMITED or TOO_MANY_JOBS | You hit the anonymous per-IP cap, or your organisation's limit: too many requests at once, or too many unfinished jobs. | Send your API key if you were not; otherwise wait for running requests to finish (the response carries retry_after) or start fewer at once. |
| 503 | “status”: “not_configured” | You called Citation Tracker or AI Visibility's /check: no AI-answer provider is wired up yet, and the API says so instead of inventing a result. | Not a bug: the query or target was recorded. See that product's page. |
| 503 | from a product's /ready | One of the product's dependencies (its database, SCERM, an upstream product) is not answering. | Read checks in the body for the failing one; retry once it is back. |
The response has "ok": true but the content looks empty? Check completeness_status: it is reported honestly and may say minimal, blocked or failed even when ok is true. Some pages take 10–30+ seconds while several strategies are tried; raise your platform's timeout if it has one.