About the WebToAPI crawler

Who we are

WebToAPI turns user-selected elements of a web page into a JSON endpoint. Our crawler only visits pages that a user has explicitly configured — we do not spider links or discover pages on our own.

How to recognise us

Our requests carry the user agent:

• WebToAPI/1.0 (+https://webtoapi.net/en/bot)

Requests come from Google Cloud Platform in region us-central1. Because we run on serverless infrastructure the source IP is not stable, so please identify us by user agent rather than by address.

How we behave

• We fetch a page at most once per second per domain.
• We cache each result for 60 seconds, so repeated API calls do not translate into repeated visits.
• We read robots.txt before creating an API and refuse to create one for a disallowed path.
• We do not attempt to bypass logins, paywalls, CAPTCHAs or bot protection.
• We do not submit forms and do not follow links beyond the exact page configured.

How to block us

Add the following to your robots.txt:

• User-agent: WebToAPI
• Disallow: /

We re-read robots.txt at least once a day. If you need something removed sooner, or want to report abuse, email support@webtoapi.net with the URL and we will disable the affected APIs.