About our crawler

Siteoseo is a technical-SEO tool. When one of our customers starts a crawl of a site, our crawler fetches its pages and reports broken links, redirects and metadata problems to that customer. It identifies itself as:

Mozilla/5.0 (compatible; SiteoseoBot/1.0; +https://siteoseo.com/bot)

It follows robots.txt by default and makes up to 5 requests per second unless the customer lowers that. Customers can switch robots.txt off, raise the speed (up to 50 requests per second) or send a different user agent, typically for their own sites.

Blocking or slowing it

Add this to your robots.txt to keep it out of your whole site:

User-agent: SiteoseoBot
Disallow: /

If it causes you a problem, write to [email protected] with your site's address and the time of the requests, and we'll look into it.