SynapseBot

What it is

Every request it makes carries this User-Agent:

Mozilla/5.0 (compatible; SynapseBot/1.0; +https://synapseorch.com/bot/)

SynapseBot is not a search engine and does not wander the web. It fetches a page only when someone using SynapseOrch has added it to a knowledge base: a single page, a sitemap, or a site to crawl from a starting address. What it fetches is kept in that workspace’s knowledge base, so the workspace’s agents can answer from it and cite it.

Who runs it

SynapseOrch Cloud runs it for our hosted customers. SynapseOrch is also open source, and a self-hosted installation sends the same User-Agent from its operator’s own network. We do not control those installations, and we publish no IP ranges for that reason.

• From SynapseOrch Cloud: one request at a time per site, at most one a second; up to 3 links deep and 500 pages per crawl. A customer cannot raise these.
• Self-hosted: the same defaults. An operator can raise them, to at most 10 requests a second and 5 links deep.

How to limit or stop it

SynapseBot obeys robots.txt. It follows a group for SynapseBot if you have one, and otherwise the * group. To keep it out of your whole site:

User-agent: SynapseBot
Disallow: /

Or out of one part of it:

User-agent: SynapseBot
Disallow: /private/

• Rules are matched as RFC 9309 says. The longest matching rule wins, Allow wins a tie, and * and $ work in patterns. The User-Agent is matched by its product token, ignoring case.
• Crawl-delay is honoured, up to 30 seconds between requests.
• robots.txt is read once per site per crawl, so a change applies from the next crawl.
• A robots.txt that answers 404 (or any 4xx) allows everything. One that cannot be read (a 5xx or no answer) stops the crawl: nothing is fetched.
• Per page, noindex (in a robots <meta> tag or an X-Robots-Tag header) keeps the page out, and nofollow stops its links being followed.

What it reads

• HTML pages up to 5 MB, and documents they link to within the crawl, such as PDFs. It does not run JavaScript, so a page that only renders in a browser reads as empty.
• A site crawl stays on the starting host, under the starting address’s path. A sitemap’s pages must be on the sitemap’s own host.
• Each request gives up after 60 seconds.

How often it comes back

By default a knowledge base re-reads its pages every 7 days. It sends conditional requests (If-None-Match, or If-Modified-Since when your server sends no ETag), so an unchanged page is not downloaded again. On a re-read, a page that answers 404 or 410, now carries noindex, or is newly disallowed is removed from the knowledge base.

A problem with a crawl

If SynapseBot from SynapseOrch Cloud misbehaves on your site, email us at the address below with the URLs and times from your logs. A crawl from a self-hosted installation is its operator’s, so robots.txt is the way to limit it.