For website owners
About our crawler: AIRecruiterResearch
AIRecruiterResearch reads how many job postings large companies list on their own job boards, once a day, for our hiring tracker. It asks each board the same question the board's own careers page asks when someone searches with no keywords, keeps the total it answers, and publishes the count. It reads nothing else.
The crawler at a glance
- Name
AIRecruiterResearch- User-Agent
AIRecruiterResearch/1.0 (+https://airecruiter.co/crawler)- Operator
- AIRecruiter.co, published by HeroHunt.ai
- What it reads
- The public job search of 431 companies' own job boards (the S&P 500 members we can read, and a few others people ask about), and each of those sites' robots.txt
- How often
- Once a day at 12:10 UTC, one search request per board (a few for a board that caps its total), with two later reads at 16:40 and 21:40 UTC only for the boards that did not answer
- Signature-Agent
"https://airecruiter.co"- Key id
ITMljWh-ywgvAbHRCMW8d8JE0dyF-l05QRGJtuhcQAg- Sent from
- The site's host functions (Vercel, in Amazon's us-east-1 region), with no fixed address
What it does and never does
- It reads robots.txt first. From November 18, 2026, before each board it reads the site's robots.txt and follows the rules for AIRecruiterResearch, or for every crawler (
*) when none names it. A board whose robots.txt asks crawlers not to read it is not read, and a site whose robots.txt cannot be reached is not read that day. - It signs every request. Each request carries a Web Bot Auth signature (HTTP Message Signatures with an Ed25519 key) over the host it is sent to and its Signature-Agent, so a site can check that a request naming AIRecruiterResearch really comes from us. The public key is in the key directory above.
- It never gets around a refusal. A board that refuses the request, or answers with a check for people (a Cloudflare challenge), is recorded as not answering and is not worked around: no other client, no other address, no solving the check. When the first boards of one job system answer with a check, the crawler asks that system no more that day. Workday's boards have answered this way since November 16, 2026.
- It reads only public totals. No sign-ins, no applications, no candidate or employee data, and no posting text is kept: only the number of postings, and where the board's own country filter gives one, how many are in the US.
How to stop it reading your site
Add these lines to your robots.txt, and the next daily read leaves your site alone:
User-agent: AIRecruiterResearch Disallow: /
Or write to us at airecruiter3@founden.cc, and we will stop reading it and say so on the tracker.