Crawl settings
All 72 settings a crawl accepts, with type, limits and default. Only the start URL is required. Field names are the ones saved crawl settings use: dots for nested sections, [] for list entries.
Crawl
| Setting | What it does | Default |
|---|---|---|
Start URLstartUrlURL | Where the crawl begins. Its host defines what counts as internal. Must be a public http or https address. | Default: required |
URL capmaxUrlsinteger, 1 to 100,000 | The most URLs this crawl will fetch, counting internal pages, external links and each redirect hop. Also sets the credit held when the crawl starts. | Default: 500 |
Maximum depthmaxDepthinteger, 0 to 100 | How many clicks away from the start URL the crawler will follow links. 0 crawls only the start URL. | Default: 20 |
Parallel requestsconcurrencyinteger, 1 to 20 | How many requests can be in flight at once. Higher is faster but puts more load on the server. | Default: 5 |
Requests per secondmaxRequestsPerSecondnumber, 0 to 50 | Upper limit on request rate across the whole crawl. 0 removes the limit, leaving only parallel requests to pace it. | Default: 5 |
Request timeoutrequestTimeoutMsinteger, 1,000 to 60,000 | How long to wait for a response, in milliseconds, before recording the URL as no response. | Default: 20,000 |
Respect robots.txtrespectRobotsTxtboolean | Skip URLs that robots.txt disallows for the crawler's user agent. Skipped URLs are listed as blocked and not charged. | Default: true |
Custom robots.txtcustomRobotsTxtstring, max 100,000 chars | Paste a draft robots.txt to use instead of the start host's live file, to see what it would block before you publish it. Subdomains keep their live files. | Default: not set |
User agentuserAgentPresetenum, crawlwise | googlebotSmartphone | googlebotDesktop | chrome | custom | The browser or bot identity sent with each request. Choose custom to supply your own string. | Default: crawlwise |
Custom user agentcustomUserAgentstring, max 500 chars | Used when the user agent is set to custom. Leave empty to fall back to the default crawler identity. | Default: not set |
Crawl subdomainscrawlSubdomainsboolean | Treat subdomains of the start host (for example blog. or shop.) as internal and crawl them too. | Default: false |
Check external linkscheckExternalLinksboolean | Request each external link once to record its status. Turn off to save URLs on link-heavy sites. | Default: true |
Follow nofollow linksfollowNofollowboolean | Also follow links marked rel=nofollow, and links on pages with a nofollow directive. | Default: false |
Include patternsincludePatternsstring[] | Regular expressions matched against the full URL. When any are set, only matching URLs are crawled. Applied before excludes. | Default: none |
Exclude patternsexcludePatternsstring[] | Regular expressions matched against the full URL. Matching URLs are never crawled. | Default: none |
Ignore query stringsstripQueryStringsboolean | Drop everything after ? from discovered URLs, so parameter variations are crawled once. | Default: false |
Render JavaScriptrenderJsboolean | Load pages in headless Chromium and analyse the rendered DOM. Charged as an add-on per page actually rendered. Custom JavaScript and the accessibility audit need it. | Default: false |
Render waitrenderWaitMsinteger, 0 to 30,000 | Extra time, in milliseconds, to wait after the network goes idle before reading the rendered page. | Default: 1,000 |
Custom extractionextractors[]
Pull data out of every internal HTML page. Each extractor adds a column to the Extraction report. Included in the crawl rate. A list: each entry has the fields below.
| Setting | What it does | Default |
|---|---|---|
Nameextractors[].namestring, max 64 chars | Column name in the report and exports. | Default: required |
Selector typeextractors[].typeenum, css | xpath | regex | How the expression is read: a CSS selector, an XPath expression or a regular expression. | Default: required |
Expressionextractors[].expressionstring, max 2,000 chars | The CSS selector, XPath or regex to match in the page HTML. | Default: required |
Outputextractors[].outputenum, text | html | attribute | count | text = trimmed text content; html = the matched element's outer HTML; attribute = the value of the attribute below; count = number of matches. For regex, text is the first capture group, or the whole match. | Default: text |
Attributeextractors[].attributestring, max 100 chars | Attribute to read when output is attribute, for example href or content. | Default: not set |
Keep every matchextractors[].multipleboolean | Store all matches instead of only the first. | Default: false |
Custom searchsearches[]
Find pages whose source contains, or lacks, a piece of text. Each search is a column in the Search report. Included in the crawl rate. A list: each entry has the fields below.
| Setting | What it does | Default |
|---|---|---|
Namesearches[].namestring, max 64 chars | Report name. | Default: required |
Modesearches[].modeenum, contains | not-contains | contains reports pages that match; not-contains reports pages that do not, for example pages missing an analytics tag. | Default: required |
Pattern typesearches[].typeenum, text | regex | Plain text or a regular expression. | Default: required |
Patternsearches[].patternstring, max 1,000 chars | What to look for. | Default: required |
Case sensitivesearches[].caseSensitiveboolean | Match upper and lower case exactly. | Default: false |
Search insearches[].scopeenum, html | visible-text | html searches the raw source; visible-text searches only the text a visitor sees. | Default: html |
Custom JavaScriptcustomScripts[]
Your own snippets, run in each rendered page; whatever they return (as JSON) is stored per URL. Requires JavaScript rendering and is billed per page as the Custom JavaScript add-on. A list: each entry has the fields below.
| Setting | What it does | Default |
|---|---|---|
NamecustomScripts[].namestring, max 64 chars | Column name in the report. | Default: required |
CodecustomScripts[].codestring, max 50,000 chars | Body of an async function. Use return to send back a value, for example return document.querySelectorAll('a').length. | Default: required |
TimeoutcustomScripts[].timeoutMsinteger, 100 to 30,000 | How long the snippet may run on a page, in milliseconds, before it is stopped. | Default: 5,000 |
Segmentssegments[]
Group internal URLs by pattern and see issues per segment. Each URL takes the first segment that matches. Included in the crawl rate. A list: each entry has the fields below.
| Setting | What it does | Default |
|---|---|---|
Namesegments[].namestring, max 64 chars | Segment name shown in reports and exports. | Default: required |
Patternsegments[].patternstring | Regular expression matched against the full URL, for example /blog/. | Default: required |
Near duplicatesnearDuplicates
Finds pages whose text is mostly the same, using MinHash similarity. Included in the crawl rate.
| Setting | What it does | Default |
|---|---|---|
Find near duplicatesnearDuplicates.enabledboolean | Compare the text of every indexable HTML page with at least 50 words. | Default: false |
Similarity thresholdnearDuplicates.thresholdnumber, 0.5 to 0.99 | Estimated share of shared text at or above which two pages count as near duplicates. Higher finds fewer, closer matches. | Default: 0.9 |
Auditsaudits
Per-page checks, each billed as its own add-on per page actually checked.
| Setting | What it does | Default |
|---|---|---|
Accessibility auditaudits.accessibilityboolean | WCAG 2.2 checks with axe-core on each rendered page. Requires JavaScript rendering. Billed per page as the Accessibility audit add-on. | Default: false |
Structured data validationaudits.structuredDataboolean | Read JSON-LD, Microdata and RDFa and check them against schema.org and Google rich-result rules. Billed per page as the Structured data add-on. | Default: false |
AMP validationaudits.ampboolean | Validate AMP pages found in the crawl. Billed per AMP page validated. | Default: false |
Lighthouseaudits.lighthouse
A full Lighthouse run per page: performance, mobile usability and best practices. Billed per page as the Lighthouse add-on.
| Setting | What it does | Default |
|---|---|---|
Run Lighthouseaudits.lighthouse.enabledboolean | Audit pages with Lighthouse. | Default: false |
Deviceaudits.lighthouse.deviceenum, mobile | desktop | Emulate a mobile phone (throttled) or a desktop. | Default: mobile |
Page limitaudits.lighthouse.maxPagesinteger, 1 to 100,000 | The most pages Lighthouse runs on in one crawl, whatever the URL cap. Also caps what the add-on can cost. | Default: 25 |
Spelling and grammaraudits.spelling
Check page copy for spelling and grammar mistakes. Billed per page as the Spelling and grammar add-on.
| Setting | What it does | Default |
|---|---|---|
Check spellingaudits.spelling.enabledboolean | Check the visible text of each page. | Default: false |
Languageaudits.spelling.languagestring, max 20 chars | A BCP 47 code such as en-GB, or auto to use each page's <html lang>. | Default: auto |
Page limitaudits.spelling.maxPagesinteger, 1 to 100,000 | The most pages the spelling check runs on in one crawl, whatever the URL cap. Also caps what the add-on can cost. | Default: 500 |
PageSpeed Insightsintegrations.pagespeed
Core Web Vitals field data and lab scores from Google. Uses your PageSpeed key if you saved one. Billed per page as the PageSpeed Insights add-on.
| Setting | What it does | Default |
|---|---|---|
Fetch PageSpeed dataintegrations.pagespeed.enabledboolean | Query PageSpeed Insights for internal HTML pages. | Default: false |
Strategyintegrations.pagespeed.strategyenum, mobile | desktop | Mobile or desktop results. | Default: mobile |
Page limitintegrations.pagespeed.maxPagesinteger, 1 to 100,000 | The most pages PageSpeed Insights runs on in one crawl, whatever the URL cap. Also caps what the add-on can cost. | Default: 50 |
AI promptsintegrations.ai[]
Prompts run on each page with your own OpenAI, Gemini or Anthropic key. Each prompt on each page is one billed run; the provider also bills you for the tokens. A list: each entry has the fields below.
| Setting | What it does | Default |
|---|---|---|
Nameintegrations.ai[].namestring, max 64 chars | Column name for the answers. | Default: required |
Providerintegrations.ai[].providerenum, openai | gemini | anthropic | Which API to call. Save that provider's key in Settings first. | Default: required |
Modelintegrations.ai[].modelstring, max 100 chars | The provider's model id. | Default: required |
Promptintegrations.ai[].promptstring, max 4,000 chars | Your instruction. The page input below is sent with it. | Default: required |
Page inputintegrations.ai[].inputenum, page-text | html | title-and-meta | What the model sees: the page's visible text, its HTML, or only its title and meta description. | Default: page-text |
Page limitintegrations.ai[].maxPagesinteger, 1 to 100,000 | The most pages this prompt runs on in one crawl, whatever the URL cap. Also caps what the add-on can cost. | Default: 100 |
Google Analytics 4integrations.googleAnalytics
Sessions, users, views, engagement and key events per URL from your connected Google account. Included in the crawl rate.
| Setting | What it does | Default |
|---|---|---|
Fetch GA4 dataintegrations.googleAnalytics.enabledboolean | Join GA4 page data to crawled URLs. | Default: false |
Property idintegrations.googleAnalytics.propertyIdstring, digits only | The numeric GA4 property id, from Admin → Property details. | Default: not set |
Daysintegrations.googleAnalytics.daysinteger, 1 to 365 | Length of the reporting period, ending yesterday. | Default: 28 |
Search Consoleintegrations.searchConsole
Clicks, impressions, CTR and position per URL from your connected Google account (included in the crawl rate), plus optional URL Inspection (billed per URL inspected).
| Setting | What it does | Default |
|---|---|---|
Fetch Search Console dataintegrations.searchConsole.enabledboolean | Join Search Console page data to crawled URLs. | Default: false |
Propertyintegrations.searchConsole.siteUrlstring, max 500 chars | As written in Search Console: sc-domain:example.com for a domain property, or the full URL prefix. | Default: not set |
Daysintegrations.searchConsole.daysinteger, 1 to 480 | Length of the reporting period. | Default: 28 |
URL Inspectionintegrations.searchConsole.urlInspectionboolean | Ask Google for each URL's index status and chosen canonical. | Default: false |
Inspection limitintegrations.searchConsole.maxInspectionsinteger, 1 to 2,000 | The most URLs inspected in one crawl. Google allows 2,000 inspections per property per day. | Default: 100 |
Link metricsintegrations.linkMetrics
Backlink metrics per URL from your own Ahrefs or Majestic key. Billed per URL looked up; the provider also bills you.
| Setting | What it does | Default |
|---|---|---|
Fetch link metricsintegrations.linkMetrics.enabledboolean | Look up internal HTML pages. | Default: false |
Providerintegrations.linkMetrics.providerenum, ahrefs | majestic | Ahrefs or Majestic. Save that provider's key in Settings first. | Default: ahrefs |
Page limitintegrations.linkMetrics.maxPagesinteger, 1 to 100,000 | The most pages the lookup runs on in one crawl, whatever the URL cap. Also caps what the add-on can cost. | Default: 1,000 |
Forms loginformsAuth
Log in through the site's own login form before crawling, then crawl with that session. Included in the crawl rate.
| Setting | What it does | Default |
|---|---|---|
Log in firstformsAuth.enabledboolean | Sign in before the crawl starts. | Default: false |
Login pageformsAuth.loginUrlURL | URL of the page with the login form. | Default: not set |
Username fieldformsAuth.usernameSelectorstring, max 500 chars | CSS selector for the username or email input. | Default: input[type="email"], input[name="username"], input[name="email"] |
Password fieldformsAuth.passwordSelectorstring, max 500 chars | CSS selector for the password input. | Default: input[type="password"] |
Submit buttonformsAuth.submitSelectorstring, max 500 chars | CSS selector for the button that sends the form. | Default: button[type="submit"], input[type="submit"] |
Saved loginformsAuth.secretIdid | Which saved login (Settings → Saved logins) to use. The password is never stored in crawl settings. | Default: not set |
User agents
The presets for userAgentPreset. Robots.txt rules are matched against the agent you pick, so a Googlebot preset shows you what Google is allowed to crawl.
- Siteoseo crawler (default)
crawlwise - Mozilla/5.0 (compatible; SiteoseoBot/1.0; +https://siteoseo.com/bot)
- Googlebot smartphone
googlebotSmartphone - Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- Googlebot desktop
googlebotDesktop - Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/131.0.0.0 Safari/537.36
- Chrome desktop
chrome - Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36