Auditing a 5,000-page site and auditing a 500,000-page site are not the same job. The vocabulary is identical, title tags, canonicals, internal links, status codes, but the methodology, tooling, and order of operations have to change once you cross into enterprise territory. Crawl an enterprise site the way you would crawl a small business site and you will either run out of memory, run out of patience, or, worse, produce a tidy report that confidently misses every issue that actually matters.
This is the audit framework we use on enterprise crawls, refined across e-commerce catalogs, large publishers, marketplaces, and SaaS knowledge bases.
Step 1: Define the crawl perimeter before you press start
The first mistake on any large site is treating "the website" as a single object. It almost never is. A typical enterprise property is a federation of subdomains, regional variants, legacy CMS instances, headless front-ends, and microservice-rendered pages. Before you launch a single crawler, you need an inventory:
- Every subdomain that resolves (often discovered through certificate transparency logs, not the sitemap)
- Every hreflang cluster and the canonical region for each
- Every CDN, edge worker, and middleware layer that can mutate a response
- Every sitemap URL, including the ones in
robots.txtthat nobody updated for three years
Without this perimeter, your crawler will either over-fetch, wasting budget on staging environments and tracking parameters, or under-fetch, missing entire sections of the site that are reachable only through search forms or JavaScript-rendered widgets.
Step 2: Sample before you commit
A full crawl of a 100k-URL site can take days. A full crawl of a 5M-URL site can take weeks and cost real money in compute and proxy bandwidth. Before you commit, run a sample crawl: 1 to 2% of the URL inventory, weighted across templates. The goal is not to find issues yet. The goal is to characterize the site:
- What is the average response size and time?
- What percentage of URLs return non-200 codes?
- How many parameters are present, and how many produce duplicate content?
- Where does JavaScript rendering change the rendered DOM versus the raw HTML?
Sample crawls catch the configuration mistakes that would otherwise burn 40 hours of crawl time. If 30% of your sample returns 302 redirects to a login page, you have a credentialing problem to solve before the full crawl, not after.
Step 3: Segment the crawl by template, not by directory
Directory-based segmentation is intuitive but misleading on large sites. /products/ might contain seven distinct page templates with completely different SEO profiles. Template-based segmentation, grouping URLs by their structural fingerprint (DOM signature, schema type, header navigation pattern), tells you the real story.
Most enterprise crawlers can fingerprint pages either by an XPath signature, a content hash of the chrome-stripped DOM, or schema.org type. Pick one and standardize. Once URLs are grouped by template, every issue you find can be expressed in terms of a fix that applies to thousands of pages at once. That is the only way enterprise SEO scales.
Step 4: Distinguish technical issues from indexation outcomes
A core mistake in enterprise audits is conflating "broken" with "deindexed." Many large sites have technical issues that have no measurable impact on search performance, and many have clean technical signals on pages Google has quietly stopped indexing.
Cross-reference every audit finding with three indexation signals:
site:queries for spot-checks on specific templates- Search Console URL Inspection API for definitive index status on sampled URLs
- Server logs for actual Googlebot fetches in the last 30 days
If a page has a duplicate-content issue but Google is indexing it, ranking it, and refreshing it weekly, that issue moves down the priority list. If a page is technically perfect but hasn't been crawled in six months, that is the headline finding, regardless of how clean the HTML looks.
Step 5: Prioritize by traffic-weighted impact
The default audit report, sorted by issue severity, is wrong for enterprise sites. A "high severity" issue affecting 200,000 zero-traffic URLs matters less than a "medium severity" issue affecting the 50 URLs that drive 30% of organic revenue.
Every issue in the final report should carry three numbers:
- URLs affected (raw count)
- Sessions affected (last 90 days, organic only)
- Revenue or conversion exposure (if available)
Stakeholders fund fixes based on the second and third numbers, not the first. An audit that doesn't carry impact data into the recommendations gets shelved.
Step 6: Separate "fix now" from "fix in the next migration"
Enterprise sites move slowly. A finding that requires changing the URL structure is a 9-month project minimum, gated by stakeholders in product, engineering, and sometimes legal. A finding that requires updating a template's title tag formula is a sprint ticket.
Split the audit deliverable into two tracks from the start:
- Tactical fixes, template-level, no URL changes, deployable in 1 to 2 sprints
- Strategic fixes, anything that touches URL structure, canonicalization logic, or platform architecture
Mixing them produces a single 80-item backlog that no one starts.
What a good enterprise audit deliverable looks like
A small-business audit deliverable can be a PDF. An enterprise audit deliverable is a living dataset: the crawl data, indexation signals, and traffic data joined at the URL level, with a dashboard or notebook view that engineering and content teams can filter on their own. The dashboard outlasts the audit, and as the team ships fixes, the same data feeds a regression view that shows whether the fixes are landing.
The crawl is the beginning, not the deliverable. The deliverable is the system that turns ongoing crawl data into ongoing decisions.