TrueNorthBot

TrueNorthBot is the crawler behind lookout, the lookup service for True North, a family-safe AI assistant. It reads public pages so that answers can cite current, checked sources.

How it identifies itself

What it does

• It follows robots.txt, including Crawl-delay. When robots.txt can't be read, it doesn't crawl that site.
• It waits between requests to the same site (at least one second by default) and keeps few requests open at once.
• It reads feeds and the pages they list, and pages people ask about. It doesn't follow links found in page text except within a reviewed list of sources.
• It doesn't submit forms, log in, or fetch from private network addresses.
• It doesn't pretend to be a browser.

How to control it

To keep it off your whole site, add this to /robots.txt:

To slow it down:

Changes take effect within a day, when it next reads your robots.txt.