← Catalog · SEO/AEO

Open web & SEO data

Search, extract and crawl the open web, plus backlinks, referring domains and authority metrics. 202 endpoints from AnyAPI, Apify, Context.dev, Bright Data, DataForSEO, Diffbot, Exa, Firecrawl, Just One API, Keenable, LimaData, Linkup, Majestic, Moz, Olostep, People Data Labs, ScrapeCreators, ScrapeGraphAI, Semrush, SE Ranking, SerpApi, Serper, Serpstat, Tavily, TikHub, TinyFish, Tomba, treg, from <$0.0001 per call. A job several providers do has its own page comparing them by price, measured success and agents' verdicts; where treg offers a routed tool for it, treg can pick the provider for you.

Run an asynchronous browser agent toward a goal

• Run a browser agentStart an asynchronous browser agent on a URL and have it work toward a natural-language goal.TinyFish · live-verified · $0.016-$8/success · tinyfish.web.agent.run

Cancel an asynchronous browser-agent run

• Cancel a browser-agent runStop a pending or running Agent run; completed steps remain billable upstream.TinyFish · live-verified · free · tinyfish.web.agent.run.cancel

Check an asynchronous browser-agent run

• Check a browser-agent runRead an Agent run's state, step count, result, and captured step metadata.TinyFish · live-verified · free · tinyfish.web.agent.run.get

Retrieve browser-agent step HTML

• Retrieve Agent step HTMLRetrieve the HTML captured for one step of a run that requested HTML capture.TinyFish · free · tinyfish.web.agent.step.html

Retrieve a browser-agent step screenshot

• Retrieve an Agent step screenshotRetrieve the JPEG screenshot captured for one step of a run that requested screenshots.TinyFish · free · tinyfish.web.agent.step.screenshot

List the anchor texts used

• web.anchors.list — routed: best of 3 providers, own keys firstAnchor texts of backlinks pointing at a domain — provider-native rows in `anchors`; `limit` is the price dialtreg · $0.000179/success · treg.web.anchors.list
• Anchor text distribution of backlinks for a domainAnchor-text distribution — backlink and referring-domain counts per anchorDataForSEO · live-verified · $0.024/call · dataforseo.web.anchors.list
• Anchor texts in backlinks by domain or URLAnchor-text distribution of inbound links, with domain and page counts per phraseMoz · live-verified · $0.00667/call · moz.web.anchors.list
• Anchor texts in backlinks by domain or URLAnchor texts across a target's backlink profile, with link and domain countsSE Ranking · live-verified · $0.000179/result · seranking.web.anchors.list
• Anchor texts in backlinks by domain or URLAnchor texts pointing at a target, with referring-domain and link counts per phraseMajestic · $0.004/call · majestic.web.anchors.list
• Anchor texts in backlinks by domain or URLAnchor texts across a target's backlink profile, with domain and link countsSemrush · own account · semrush.web.anchors.list
• Anchor texts in backlinks by domain or URLAnchor texts pointing at a target, with referring-domain and link counts per phraseSerpstat · $0.0005/result · serpstat.web.anchors.list

Answer a question from live web research, with cited sources

• Answer a question with citationsGet an LLM answer to a question informed by Exa search results. Fully compatible with OpenAI's chat completions endpoint.Exa · live-verified · $0.005/call · exa.web.answer
• Research a web questionStart asynchronous multi-source research with citations.Linkup · live-verified · $0.25-$2.5/success · linkup.web.answer
• Answer a question with live web researchResearch a task on the live web and return a structured answer with sources.Olostep · live-verified · $0.04/call · olostep.web.answer

Check a web research task

• Check a research taskRead the status and result of a research task started by this team.Linkup · live-verified · free · linkup.web.answer.status

Find domains with overlapping backlinks

• Domains sharing backlink sources with a domainDomains sharing referring domains with the target, ranked by overlapDataForSEO · live-verified · $0.024/call · dataforseo.web.backlinks.competitors
• Backlink competitors by domain — shared referring domainsDomains whose backlink profile overlaps a target's, by shared referring domainsSemrush · own account · semrush.web.backlinks.competitors

Find link gaps across several targets

• Sources linking to competitors but not you — link gapSources that link to the given competitors but not to you — link-gap prospectingMoz · live-verified · $0.0333/call · moz.web.backlinks.intersect

List individual backlinks to a target

• web.backlinks.list — routed: best of 3 providers, own keys firstBacklinks pointing at a domain or page — provider-native backlink rows in `backlinks`; `limit` is the price dialtreg · $0.000179/success · treg.web.backlinks.list
• Backlinks pointing to a domain or URL — anchor, rankIndividual backlinks — source page, anchor, follow status, first seen, page/domain rankDataForSEO · live-verified · $0.024/call · dataforseo.web.backlinks.list
• Backlinks by domain or URL — source and target metricsIndividual inbound links, with full URL metrics for both source and targetMoz · live-verified · $0.00667/call · moz.web.backlinks.list
• Backlinks by domain or URL — source page, anchor textsIndividual backlinks to a domain or URL, with source page, anchor and follow statusSE Ranking · live-verified · $0.000179/result · seranking.web.backlinks.list
• Backlinks by domain or URL — anchor texts, Trust FlowIndividual backlinks to a URL or domain, with anchor text and both ends' Trust FlowMajestic · $0.02/call · majestic.web.backlinks.list
• Backlinks by domain or URL — source page, anchor textsIndividual backlinks to a domain or URL, with source page, anchor and follow statusSemrush · own account · semrush.web.backlinks.list
• Active backlinks by domain or URL — anchor textsCurrently active backlinks to a domain or URL, with source page, anchor and link typeSerpstat · $0.0005/result · serpstat.web.backlinks.list

Check if specific sites link to a target

• Check whether specific sources link to a target domainCheck whether specific sources link to a target — returns one boolean per sourceMoz · live-verified · $0.0133/call · moz.web.backlinks.status

Backlink profile of a domain or URL

• web.backlinks.summary — routed: best of 3 providers, own keys firstBacklink profile summary for a domain — counts in `backlinks` / `referring_domains`, the provider's full row in `summary`treg · $0.0133/success · treg.web.backlinks.summary
• Backlink profile of a domain — counts, rank, spam scoreAggregate backlink profile of a domain or URL — counts, rank, spam score, breakdownsDataForSEO · live-verified · $0.024/call · dataforseo.web.backlinks.summary
• Backlinks summary by domain — authority, spam histogramsAggregate backlink profile plus histograms by DA, spam score and root domainsMoz · live-verified · $0.0133/call · moz.web.backlinks.summary
• Backlinks summary by domain or URL — referring domainsAggregate backlink profile — backlinks, referring domains, IPs, subnets, follow splitSE Ranking · live-verified · $0.0179/result · seranking.web.backlinks.summary
• Backlinks summary by domain — referring domains, .edu/.govAggregate backlink profile of a domain — external links, referring domains, .edu/.govMajestic · $0.0008/result · majestic.web.backlinks.summary
• Backlinks summary by domain or URL — Authority ScoreAggregate backlink profile — Authority Score, links, referring domains, follow splitSemrush · own account · semrush.web.backlinks.summary
• Backlinks summary by domain — referring domains, authorityAggregate backlink profile — external links, referring domains, IPs, domain authoritySerpstat · $0.0025/call · serpstat.web.backlinks.summary

Crawl a website and extract its pages

• Crawl a websiteReturn Markdown from a bounded crawl of linked pages.Context.dev · live-verified · $0.00253-$0.0253/success · branddev.web.crawl
• Crawl a websiteStart a bounded crawl and pay for pages processed.Firecrawl · live-verified · $0.005-$0.05/success · firecrawl.web.crawl
• Crawl a websiteStart a bounded asynchronous website crawl and pay only for completed pages.Olostep · live-verified · $0.002-$0.2/success · olostep.web.crawl
• Crawl one web page asynchronouslyStart a bounded one-page Markdown crawl and return its job id.ScrapeGraphAI · live-verified · $0.012/success · scrapegraphai.web.crawl
• Crawl a websiteMap a website and extract clean content from the discovered pages in one request.Tavily · live-verified · $0.0024-$0.0048/success · tavily.web.crawl
• Website CrawlCrawl a website and get clean text content from up to 10 pages in one normalized response, ideal for feeding sites into LLMs and search indexes.AnyAPI · live-verified · $0.00495/success · anyapi.web.crawl

List the pages produced by a website crawl

• Retrieve website crawl pagesList an owned crawl's pages, optionally including Markdown or HTML content.Olostep · live-verified · free · olostep.web.crawl.results
• Retrieve ScrapeGraphAI crawl pagesFetch owned crawl pages with each page's resolved scrape result.ScrapeGraphAI · live-verified · free · scrapegraphai.web.crawl.pages

Check a website crawl's progress and final state

• Check a website crawlRead an owned crawl's status, result pages and credit usage.Firecrawl · live-verified · free · firecrawl.web.crawl.status
• Check a website crawlRead an owned crawl's progress, completed-page count and terminal credit usage.Olostep · live-verified · free · olostep.web.crawl.status
• Check a ScrapeGraphAI crawlRead an owned crawl's state and lightweight page metadata.ScrapeGraphAI · live-verified · free · scrapegraphai.web.crawl.status

Is a domain disposable or a free webmail host

• Is a domain disposable or free webmailDomain → two booleans: whether it is a disposable/throwaway mail host and whether it is a free webmail provider — the cheap pre-filter before signup or lead acceptanceTomba · live-verified · free · tomba.web.domain.disposable

Extract clean content from web pages

• web.extract — routed: best of 10 providers, own keys firstExtract clean content from one web page — provider-native rows in `pages`treg · free · treg.web.extract
• Scrape a web pageReturn one page as Markdown, HTML, images or a screenshot.Context.dev · live-verified · $0.00253/success · branddev.web.scrape
• Get page contents by URLGet the full page contents, summaries, and metadata for a list of URLs.Exa · live-verified · $0.001/result · exa.web.contents.get
• Scrape a web pageFetch one page as Markdown, HTML or links.Firecrawl · live-verified · $0.005/success · firecrawl.web.scrape
• Fetch page contentFetch an indexed or live webpage as Markdown, optionally reduced by an extraction instruction.Keenable · live-verified · $0.004/call · keenable.web.fetch
• Extract web pagesExtract clean Markdown content, metadata, links, and images from up to ten URLs.LimaData · live-verified · $0.004/result · limadata.web.extract
• Fetch a web pageRetrieve clean Markdown from a public HTML or PDF URL.Linkup · live-verified · $0.001-$0.01/success · linkup.web.fetch
• Scrape a web pageFetch one page as clean Markdown, HTML, text or a hosted screenshot.Olostep · live-verified · $0.002/call · olostep.web.scrape
• Extract structured data from one pageFetch one public HTML page and use a natural-language prompt to return structured JSON.ScrapeGraphAI · live-verified · $0.02/success · scrapegraphai.web.extract
• Scrape one web page as MarkdownFetch one public HTML page and return clean Markdown.ScrapeGraphAI · live-verified · $0.004/success · scrapegraphai.web.scrape
• Extract a web pageFetch one public web page and return clean text plus optional Markdown, links, images, videos, and metadata.Serper · live-verified · $0.01/call · serper.web.extract
• Extract web pagesExtract clean content from one to twenty URLs, with optional query-focused chunks and images.Tavily · live-verified · $0.0016-$0.0032/success · tavily.web.extract
• Fetch clean page contentFetch up to ten URLs as Markdown, HTML, or structured JSON, with optional links, metadata, and selectors.TinyFish · live-verified · free · tinyfish.web.fetch

Scrape a bounded list of web pages

• Scrape a batch of web pagesStart a bounded batch scrape of up to ten URLs.Firecrawl · live-verified · $0.05/success · firecrawl.web.batch-scrape

Read a batch scrape's progress and results

• Check a batch scrapeRead an owned batch's status, result pages and credit usage.Firecrawl · live-verified · free · firecrawl.web.batch-scrape.status

Extract typed JSON from one web page

• Extract structured data from a web pageFetch a page and extract typed JSON with a supplied schema.Linkup · live-verified · $0.002-$0.011/success · linkup.web.fetch.structured

Most-linked domains on the web (index-wide)

• Most-linked domains on the web — Domain Authority, spamMost-linked root domains on the web, with DA and spam score (index-wide, no target)Moz · live-verified · $0.00667/call · moz.web.global.top_domains

Most-linked pages on the web (index-wide)

• Most-linked pages on the web — index-wide, no targetMost-linked pages on the web, with full URL metrics (index-wide, takes no target)Moz · live-verified · $0.00667/call · moz.web.global.top_pages

Link index id & freshness

• Current Moz link index versionWhich Moz link index is serving results — tells you if metrics moved on a refreshMoz · live-verified · $0.00667/call · moz.web.index.metadata

List the referring domains

• Referring domains for a domain or URL — counts, rankReferring domains linking to a target, with backlink counts, rank and first/last seenDataForSEO · live-verified · $0.024/call · dataforseo.web.linking_domains.list
• Referring domains by domain or URL — authority, spamRoot domains linking to a target, with each source's DA, spam score and link countsMoz · live-verified · $0.00667/call · moz.web.linking_domains.list
• Referring domains by domain or URL — authority, link countsReferring domains linking to a target, with each source's link count and authoritySE Ranking · live-verified · $0.000179/result · seranking.web.linking_domains.list
• Referring domains by domain — Trust Flow, country, IPReferring domains linking to a target, with Trust Flow, country, IP and link countMajestic · $0.004/call · majestic.web.linking_domains.list
• Referring domains by domain or URL — Authority ScoreReferring domains linking to a target, with Authority Score, country and link countSemrush · own account · semrush.web.linking_domains.list
• Referring domains by domain — authority, link countsReferring domains linking to a target, with each source's link count and authoritySerpstat · $0.0005/result · serpstat.web.linking_domains.list

Discover the URLs on a website

• web.map — routed: best of 5 providers, own keys firstDiscover the URLs on a website — provider-native map rows in `results`treg · $0.001/success · treg.web.map
• Map a websiteDiscover bounded lists of URLs on a domain without fetching page bodies.Context.dev · live-verified · $0.00253/success · branddev.web.map
• Map a websiteDiscover URLs on a website without scraping their content.Firecrawl · live-verified · $0.005/success · firecrawl.web.map
• Find relevant URLs on a websiteMap a site and return at most 1,000 URLs ranked for a search query.Olostep · live-verified · $0.002/call · olostep.web.map.search
• Map a websiteDiscover URLs from a website with depth, breadth, path and domain controls.Tavily · live-verified · $0.0008-$0.0016/success · tavily.web.map
• Web MapMap an entire website into a clean list of its URLs (with titles and descriptions) in a single call.AnyAPI · live-verified · $0.001/success · anyapi.web.map

Read a scheduled monitor's tick activity

• Read page-monitor activityRead recent ticks, completion states and per-format change diffs.ScrapeGraphAI · live-verified · free · scrapegraphai.web.monitor.activity

Create a scheduled web-page change monitor

• Create a scheduled page monitorSchedule recurring page captures and change detection on the connected account.ScrapeGraphAI · live-verified · $0.004-$0.024/success · scrapegraphai.web.monitor.create

Delete a scheduled web-page monitor

• Delete a page monitorPermanently remove a monitor and its stored tick history.ScrapeGraphAI · live-verified · free · scrapegraphai.web.monitor.delete

Read one scheduled web-page monitor

• Get one page monitorRead one monitor's status, schedule and capture configuration.ScrapeGraphAI · live-verified · free · scrapegraphai.web.monitor.get

List scheduled web-page monitors

• List page monitorsList every scheduled monitor on the connected account.ScrapeGraphAI · live-verified · free · scrapegraphai.web.monitor.list

Pause a scheduled web-page monitor

• Pause a page monitorStop future ticks while preserving the monitor and its activity.ScrapeGraphAI · live-verified · free · scrapegraphai.web.monitor.pause

Resume a scheduled web-page monitor

• Resume a page monitorPut a paused monitor back on its saved schedule.ScrapeGraphAI · live-verified · free · scrapegraphai.web.monitor.resume

Update a scheduled web-page monitor

• Update a page monitorChange a monitor's URL, name, schedule, formats, webhook or fetch configuration.ScrapeGraphAI · live-verified · free · scrapegraphai.web.monitor.update

Run an on-page audit of a URL

• Instant on-page audit of a URL — timings, meta, scoreCrawl one URL on demand — status code, load timings, on-page score, meta and contentDataForSEO · live-verified · $0.00015/result · dataforseo.web.page.audit

Search and synthesize web research for a question

• Research the webRetrieve and synthesize web context for a natural-language question.LimaData · live-verified · $0.006/success · limadata.web.research

Download a finished job's records

• Download an Apify run's results by dataset idDownload the results of a finished scraper runApify · live-verified · free · apify.web.scrape.job.results
• Download scraped records by snapshot idDownload the records produced by a finished scraping jobBright Data · live-verified · $0.0015/result · brightdata.web.scrape.job.results
• Download a bulk extract job's results as JSON or CSVDownloads the extracted results of a bulk job as JSON or CSV, or a diagnostic URL reportDiffbot · diffbot.x.bulk-data
• Download a crawl job's results as JSON or CSVDownloads the extracted results of a crawl job as JSON or CSV, or a diagnostic URL reportDiffbot · diffbot.x.crawl-data

Start a scraping job over many URLs

• Start an Apify actor run in the backgroundStart a scraper run in the background and get back its run idApify · live-verified · free · apify.web.scrape.job.start
• Start an async scrape of a URL batch — snapshot idStart an asynchronous scraping job over a batch of URLs and get a snapshot id backBright Data · live-verified · $0.0015/result · brightdata.web.scrape.job.start
• Start a bulk extract job from a list of URLsSends a large, explicit list of URLs through an Extract API asynchronously and compiles the results into one collection — unlike Crawl, it does not discover linksDiffbot · own account · diffbot.x.bulk-create
• Start a site crawl job from seed URLsSpiders a site for links and pages, running each discovered page through an Extract API and compiling results into a searchable collectionDiffbot · own account · diffbot.x.crawl-create

Check if a scraping job has finished

• Check an Apify actor run by run idCheck whether a background scraper run has finishedApify · live-verified · free · apify.web.scrape.job.status
• Check an async scrape job by snapshot idCheck whether an asynchronous scraping job has finishedBright Data · live-verified · free · brightdata.web.scrape.job.status
• Manage a crawl jobOne endpoint for status, pause/resume, restart and delete of one or more crawl jobsDiffbot · live-verified · diffbot.x.crawl-manage

Scrape a URL into structured records

• Scrape URLs into structured records in one callScrape one or more URLs into structured records and get them back in the same callBright Data · live-verified · $0.0015/result · brightdata.web.scrape.structured
• Auto-detect a page type and extract it by URLAutomatically identifies the type of page being processed and routes it through the relevant Diffbot APIDiffbot · live-verified · $0.0012/call · diffbot.x.extract-analyze

Search the open web by meaning or keyword

• web.search — routed: best of 10 providers, own keys firstSearch the open web — provider-native result rows in `results`treg · free · treg.web.search
• Search the open webFind ranked web pages by query, with optional domain filters.Context.dev · live-verified · $0.00253/success · branddev.web.search
• Search the webThe search endpoint lets you intelligently search the web and extract contents from the results.Exa · live-verified · $0.007/call · exa.web.search
• Search the open webFind up to ten web results without scraping their full pages.Firecrawl · live-verified · $0.01/success · firecrawl.web.search
• Search the open webSearch Keenable's web index with site, publication, acquisition and point-in-time filters.Keenable · live-verified · $0.004/call · keenable.web.search
• Search the open webReturn ranked sources, a sourced answer, or structured JSON from a web query.Linkup · live-verified · $0.005-$0.055/success · linkup.web.search
• Search the open webSearch the web with a natural-language query and domain filters.Olostep · live-verified · $0.01/call · olostep.web.search
• Search the web with fetched page contentSearch the web and return the top HTML pages with Markdown content inline.ScrapeGraphAI · live-verified · $0.008-$0.16/success · scrapegraphai.web.search
• Search the web and extract a structured answerSearch HTML pages, return their Markdown, and run one AI extraction across the results.ScrapeGraphAI · live-verified · $0.02-$0.4/success · scrapegraphai.web.search.extract
• Search the open web with GoogleSearch Google and return organic results plus answer boxes, knowledge panels, related questions, and related searches when present.Serper · live-verified · $0.001/call · serper.web.search
• Search the open webSearch the web with semantic ranking, optional answers, domain and date filters, images, and page content.Tavily · live-verified · $0.008-$0.016/call · tavily.web.search
• Search the open webSearch the web by keyword or meaning, with domain, language, location, and date filters.TinyFish · live-verified · free · tinyfish.web.search

Deep multi-query web search with reasoning

• Deep web searchThe search endpoint lets you intelligently search the web and extract contents from the results.Exa · live-verified · $0.012/call · exa.web.search.deep

Search recent news articles by topic

• Search news by topicThe search endpoint lets you intelligently search the web and extract contents from the results.Exa · live-verified · $0.007/call · exa.web.search.news
• Search recent newsSearch TinyFish's news index with optional recency, date, domain, language, and location filters.TinyFish · live-verified · free · tinyfish.web.search.news

Search research papers and publications

• Search research publicationsThe search endpoint lets you intelligently search the web and extract contents from the results.Exa · live-verified · $0.007/call · exa.web.search.publications
• Search research publicationsSearch research papers by topic, publication year, domain, language, and date.TinyFish · live-verified · free · tinyfish.web.search.publications

Find pages similar to a URL

• Find pages similar to a URLFind similar links to the link provided and optionally return the contents of the pages.Exa · live-verified · $0.007/call · exa.web.similar

A site's highest-authority pages

• Highest-authority pages by domain, with URL metricsHighest-authority pages of a domain, with full URL metrics per pageMoz · live-verified · $0.00667/call · moz.web.site.top_pages
• Most-linked pages by domain — backlinks, Trust FlowA site's most-linked pages, with Trust Flow, Citation Flow and backlink countsMajestic · $0.02/call · majestic.web.site.top_pages
• Most-linked pages by domain — backlinks, referring domainsList a site's most-linked-to pages, ranked by backlinks and referring domainsSerpstat · $0.0005/result · serpstat.web.site.top_pages

Authority metrics for a URL or domain

• web.url.metrics — routed: best of 3 providers, own keys firstAuthority / link metrics for one URL or domain — the provider's own metrics row in `metrics`treg · $0.00268/success · treg.web.url.metrics
• Backlink authority rank for up to 1000 domains or URLsBacklink-derived authority rank (0-1000) for up to 1000 domains or pages per callDataForSEO · live-verified · $0.024/call · dataforseo.web.url.metrics
• Domain Authority, Page Authority and Spam Score by URLDomain Authority, Page Authority, Spam Score and link counts for up to 50 URLsMoz · live-verified · $0.00667/call · moz.web.url.metrics
• Page and domain authority scores by domain or URLGet SE Ranking's page and domain authority scores for a URL or domainSE Ranking · live-verified · $0.00268/result · seranking.web.url.metrics
• Trust Flow and Citation Flow by domain or URLTrust Flow, Citation Flow and link counts for up to 100 URLs or domains per callMajestic · $0.0008/result · majestic.web.url.metrics