VoltcheckBot

VoltcheckBot is the crawler behind Voltcheck, a site that helps shoppers find electric vehicles for sale. It visits public dealer inventory pages and reads the structured vehicle data (schema.org JSON-LD) those pages publish, and links every listing back to its source page on the dealer's own site.

How it behaves

• Identifies itself in the X-Crawler header of every request. It sends a standard browser User-Agent, because many CDNs block non-browser agents by default, including, on a number of sites we checked, blocking access to robots.txt itself, which prevented us from reading the site's own crawling policy.
• Obeys robots.txt, including any rule naming VoltcheckBot.
• Waits at least 1.1 seconds between requests to the same host.
• Backs off and does not retry when a site returns an error or a challenge.
• Does not use proxy rotation, and does not attempt to solve or bypass bot challenges.
• Links every listing back to the original page on your site.

Excluding your site

Add this to your robots.txt and VoltcheckBot will not fetch your pages:

User-agent: VoltcheckBot
Disallow: /