Optimizing Crawl Budget for Large Sites

What is Crawl Budget?

Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. It's determined by two factors:

• Crawl capacity limit: How fast Google can crawl without overloading your server
• Crawl demand: How much Google wants to crawl based on popularity and freshness

When Crawl Budget Matters

Crawl budget is mainly a concern for:

• Large sites: 10,000+ pages
• Sites with many URL parameters: E-commerce filters, sorting options
• Sites with auto-generated content: Search results, infinite calendars
• Sites with duplicate content issues: Multiple URLs for same content
• Sites with slow server response: Google crawls less when servers are slow

For smaller sites (under 10,000 pages), crawl budget usually isn't an issue.

Common Crawl Budget Wasters

1. Faceted Navigation

E-commerce sites are notorious for this:

/products?color=red
/products?color=red&size=large
/products?color=red&size=large&sort=price
/products?color=red&size=large&sort=price&page=2
...

A few filters can create millions of URL combinations.

2. Session IDs and Tracking Parameters

/page?sessionid=abc123
/page?utm_source=google&utm_medium=cpc
/page?ref=homepage

Each parameter creates a "new" URL for Google.

3. Infinite Spaces

• Calendars: /events/2030/12/15/
• Internal search: /search?q=anything
• Pagination: /blog/page/500/

4. Duplicate Content

/page
/page/
/page/index.html
/PAGE
http://example.com/page
https://example.com/page
https://www.example.com/page

Finding Wasted Crawls in Logs

Identify Googlebot Requests

# All Googlebot requests
grep "Googlebot" access.log | awk '{print $7}' > googlebot_urls.txt

# Count requests by URL pattern
cat googlebot_urls.txt | sort | uniq -c | sort -rn | head -50

Find Parameter URLs

# URLs with query strings
grep "Googlebot" access.log | grep "?" | awk '{print $7}' | cut -d? -f1 | sort | uniq -c | sort -rn

# Most crawled parameter combinations
grep "Googlebot" access.log | grep "?" | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

Find Pagination Crawls

# Pagination patterns
grep "Googlebot" access.log | grep -E "/page/[0-9]+" | wc -l

# Deep pagination (page 10+)
grep "Googlebot" access.log | grep -E "/page/(1[0-9]|[2-9][0-9])" | wc -l

Optimization Strategies

1. Use robots.txt

# Block faceted navigation
User-agent: *
Disallow: /products?*color=
Disallow: /products?*size=
Disallow: /products?*sort=

# Block internal search
Disallow: /search

# Block deep pagination
Disallow: /blog/page/

2. Implement Canonical Tags

<link rel="canonical" href="https://example.com/products" />

Tells Google which URL is the "master" version.

3. Use Meta Robots Noindex

<meta name="robots" content="noindex, follow">

For pages that should be crawled but not indexed (like filtered views).

4. Improve Internal Linking

• Link to important pages from navigation
• Reduce links to low-value pages
• Use descriptive anchor text
• Create clear site hierarchy

5. Speed Up Your Server

Faster response = more crawls. Google is more willing to crawl fast sites.

# Check Googlebot response times
grep "Googlebot" access.log | awk '{print $NF}' | sort -n | tail -20

6. Submit XML Sitemaps

Include only indexable, canonical URLs. Update frequently.

Ongoing Monitoring

Key Metrics to Track

• Googlebot requests per day (trend over time)
• Response codes returned to Googlebot
• Most crawled URL patterns
• Crawl frequency of important pages
• Server response time for Googlebot

Weekly Crawl Report

#!/bin/bash
echo "=== Weekly Crawl Budget Report ==="
echo ""
echo "Total Googlebot requests:"
grep "Googlebot" access.log | wc -l
echo ""
echo "Top crawled URL patterns:"
grep "Googlebot" access.log | awk '{print $7}' | cut -d? -f1 | sort | uniq -c | sort -rn | head -20
echo ""
echo "Parameter URLs (potential waste):"
grep "Googlebot" access.log | grep "?" | wc -l