Who we are
PreschoolRosterBot is the crawler Preschool Roster, an independent directory of licensed preschools and child care programs, uses to read program websites for facts. It identifies itself in every request with this user agent:
PreschoolRosterBot/1.0 (+https://preschoolroster.com/bot/)
What it reads
It reads a program's own public website to find a few plain facts about the program: opening hours, the ages served, published tuition, the type of program, and the schedule options. It keeps a private copy of each page it reads, so it can find and check those facts and skip pages that have not changed, and it does not publish that copy. A listing uses only the facts above. For each one the crawler records the page address, the short line of text that states it, and the date it read it, so every fact it adds can be traced back to the program's own words. It does not publish descriptions or photos, and it does not extract people's names or other personal details as listing facts.
It does not read the websites of home child care providers who have not claimed their listing.
How it behaves
- It follows your robots.txt, including any crawl-delay. A page your robots.txt disallows is not fetched.
- It asks for about one page every five seconds from a site (longer if your crawl-delay asks). It reads at most eight pages for each program we list, plus your robots.txt and, for a site with many locations, its sitemap.
- We aim to read a site again about once a month, not nightly. A few sites are read more often: to check our own connection before and after a run, to double-check a fact by hand, or to retry after a block.
- When a site provides an ETag or Last-Modified value for a page, the next crawl that reads that page sends it back (If-None-Match or If-Modified-Since), so an unchanged page can answer with a short 304 Not Modified instead of the full page.
- It reads only the HTML a page sends; it does not run the page's scripts or load it in a browser.
- If your site answers with a block, a rate limit, or a challenge page, it stops reading that site and does not start another crawl of it for at least three days. It never uses proxies, never solves CAPTCHAs, and never pretends to be a person's browser.
How to ask it to stop
Add these two lines to your robots.txt and PreschoolRosterBot will stop reading your site:
User-agent: PreschoolRosterBot
Disallow: /
The rule above stops PreschoolRosterBot. Our other checks (whether a site is online, published tuition and openings, and the contact address on a program's site) identify as VentureKitBot/1.0 (+https://preschoolroster.com/bot/), follow robots.txt for VentureKitBot or *, honour the same opt-out list, and do not read unclaimed home providers' sites. To stop those too, add these two lines:
User-agent: VentureKitBot
Disallow: /
You can also write to hello@preschoolroster.com with your website address to ask us to stop crawling it or to review facts taken from it. The robots.txt rule above is the sure way: the crawler reads your robots.txt at the start of every visit.
If a fact about your program is wrong
Write to hello@preschoolroster.com, or claim your listing for free. See corrections.