How Does Google Crawl the Web?

How does Google
crawl the web?
A study based on the analysis of 413 million
pages crawled by Botify and 6 billion
Googlebot requests.

Annabelle Bouard
Search Data Strategist @botify.com
Today’s Presenter
@botify - #BotifyWebinar
Dimitri Brunel
Search Data Strategist @botify.com

Today’s Agenda
Goal: Better understand Google's behavior with a scientific approach.
● Methodology & Definitions
● The 1st study of its kind based on real customer data globally
● We put REAL figures on what SEOs have known but could not prove
● Insights from our study
● How can you go further?

Methodology &
Definitions

Scale-up the dataset
from a single website
to a full set of
websites from
different industries.
Add a scientific
approach to the
empiric one for
detailed and
data-backed insights.
Improved SEO
Methodology
More Precise
Analysis
Confirm or Invalidate
Google’s
behavior and
Discover new ones.
Real Insights
Be more efficient in
our SEO in order to
continually improve
Googlebot’s
efficiency and user
experience.
Share a Belief

Definitions for today’s session
URLs that meet the following
criteria:
● HTTP 200 Status Code
● Self Referential Canonical
● No noindex Tag
● HTML Content Type
Percentage of compliant pages
(indexable pages) crawled by
Google in 30 days.
Average number of times a
website’s URL was crawled by
Google in 30 days.
Compliant URL Crawl Ratio Crawl Frequency

The 1st study of its
kind based on real
customer data
globally
* Anonymized data@botify - #BotifyWebinar

We Looked at a Massive Amount of Data
270
413M
6.2B
Websites that fall in one of the following industries:
Retail Publisher Classifieds
Pages analyzed. Data from Botify Analytics and from
log files for these websites.
Googlebot requests analyzed. 30 days of
web server log files for each website.

We also looked at websites of all shapes and sizes
Industries Analyzed Dataset by Website Size (in pages)

And we looked at a lot of different metrics
● Industry
● Size
● Non-Compliant Pages
● Orphan Pages
● PageRank
● Page Depth
● Load Times
● No. of Outlinks
● Template weight
● Content Size
Type of Website Types of Pages Structure & Content

We put REAL
figures on what
SEOs have
known but could
not prove

How Does Google Crawl the Web?

A website’s size is
one of the most
important factors
impacting Google’s
crawl.
Website size matters

KPIs are impacted by website size
Some KPIs like the number of orphan pages, the load time, or the
percentage of words vs. template, have almost no impact on small
websites but have a huge impact on big websites.
Orphan Pages Load Time
% of Words vs. Template
The larger the website, the
greater the impact:
PageRank Depth
Content Size
Huge impact regardless of size:
Other KPIs like the PageRank dilution, depth, or surprisingly content
size, have a big impact on Google’s crawl, regardless of website size.

Insights from our study

What elements really impact Google’s crawl?
WEBSITE
1. Industry ✘
2. Size ✓
STRUCTURAL KPIS
5. PageRank ✓
6. Depth ✓
7. Load Times ✓
8. No. of Outlinks ✘
9. Template Weight ✓
10. Content Size ✓
TYPE OF PAGES
3. Non-Compliant Pages ✓
4. Orphan Pages ✓

Website Size
& Industry

#1 - Industry
Expected Results
Similar Crawl Rate
Different Crawl Frequency
depending on Industries
CLASSIFIEDS PUBLISHERRETAILER

CRAWL RATIO AND ACTIVE PAGES RATIO BY INDUSTRY
● Googlebot impartially crawls on the web.
● Googlebot crawls impartially regardless of
industry.
● Publishers tend to have more active pages (in
%).
Google’s crawl ratio is not influenced by industry
From our Experience
Confirmation
From the analysis of the Dataset

What's your crawl ratio and active pages ratio?

CRAWL FREQUENCY BY INDUSTRY
Publishers are crawled 45% more frequently than other industries
Publishers might be crawled more frequently
because of fresher and higher quality content.
New learnings!

What's your crawl frequency?

2# - Website Size
Expected Results
Decreasing Crawl Ratio Adaptative Crawl Frequency
> 10K
PAGES
> 1 MILLIONS
PAGES
> 100K
PAGES
< 10K
PAGES

Confirmation
CRAWL RATIO AND ACTIVE PAGES RATIO BY WEBSITE SIZE
From our Experience
Websites with >1M pages: Crawl Ratio is significantly
lower than average
● More pages means more difficulties for
Googlebot.
● More pages means fewer active pages in the
SERPs (in %).
● Small websites are better crawled by Google
but still not entirely.
● Large websites have a harder time using
Crawl Budget efficiently.

Confirmation
CRAWL FREQUENCY BY WEBSITE SIZE
Pages on large websites with >1M pages are less frequently
crawled than the average
Large websites tend to have more long tail
pages that will be crawled by Google less
frequently.
Good news: this can be influenced with crawl
budget optimization.

Type of Pages

3# - Non-Compliant Pages
and
or
Canonical tag set not to
self
Not Text/HTML Content
● Poor indexability from a
technical point of vue.
● Negative signals for web spiders.
Risk
Expected Results
Lowest Crawl Ratio Low Indexation Lower Crawl Frequency
At last a composite indicator
NoIndex Status
HTTP codes other
than 200 Status Code

Confirmation
COMPLIANT PAGES CRAWLED BY BOTIFY vs. NON-COMPLIANT PAGES CRAWLED BY BOTIFY
From our Experience
37% of pages crawled by Botify were non-compliant pages
● The proportion of Non-Compliant (37%) pages
is still too important vs. aiming for total
indexability (100% of compliant pages).
● The overall average shows that SEO managers
still have room for improvement.
● From our experience we see that many websites still
face this problem, usually because of:
○ Extensive use of Noindex,
○ Server Errors,
○ Canonical annotations.
413M pages crawled

Confirmation
CRAWLED COMPLIANT PAGES vs. CRAWLED NON-COMPLIANT PAGES
From our Experience
Google is wasting (at least) 16% of its time and resources
crawling non-compliant pages
● As most websites have a huge proportion of
Non-Compliant Pages, Google is on average
wasting 16% of its time crawling these useless
pages, when it could focus on valuable pages
for SEO traffic.
● Google is wasting time crawling
Non-Compliant Pages.

CRAWL RATIO vs. % OF NON-COMPLIANT PAGES CRAWLED BY GOOGLE
From our Experience
Confirmation
The higher the share of Non-Compliant pages crawled, the
lower the Crawl Ratio
● When the proportion of Non-Compliant Pages
crawled by Google increases, the Crawl Ratio
decreases.
● We expect that having more Non-Compliant
Pages crawled by Google will have a negative
impact on the Compliant Page’s Crawl Ratios.

LESS THAN 100K PAGES MORE THAN 100K PAGES
For large websites over 100K pages, Crawl Ratio is strongly
impacted by the number of Non-Compliant pages
Confirmation
● Low impact on small websites but huge on medium ones.

What's your compliant pages ratio?
Globally and among pages crawled by Google?

#4 - Orphan Pages
● that are outside of the website structure,
● that we did not discover,
● that Google crawled,
● that receive crawl budget.
Expected Results
Cannibalization of crawl
budget
Lowering the crawl ratio
of the site structure
PAGES
Crawled by
BOTIFY
Crawled by
GOOGLE
Crawled by
Google AND Botify

Confirmation
CRAWL VOLUME ON PAGES IN STRUCTURE vs. CRAWL VOLUME ON ORPHAN PAGES From our Experience
● On avg. the Orphans Pages steal ¼ of the
crawl.
Orphan pages represent 26% of Google’s crawl
● From our experience, we always see many
Orphans URLs.
● Common reasons:
○ Old implementations, technical regressions,
○ No DNS cleaning, etc.

Confirmation
CRAWL RATIO vs. % OF ORPHAN PAGES CRAWLED BY GOOGLE From our Experience
Too many orphan pages negatively impact the way Google
crawls your site
● These pages tend to cannibalize precious crawl
budget, and impact the Crawl Ratio of pages in
the structure that do not benefit of 100% of the
crawl budget.
● As the percentage of Orphan Pages
increases, the Crawl Ratios should be
negatively impacted.
Let’s dig deeper into the data, once again!
Few orphans
= Higher crawl ratio
More orphans
= Lower crawl ratio

New learnings!
LESS THAN 100K PAGES MORE THAN 100K PAGES
From the analysis of the DatasetFrom our Experience
● This is true on Large and Very Large websites
only.
● Crawl budget cannibalization whatever the
size of the website.
Especially for large websites where Crawl Ratio is badly impacted

What proportion of orphan pages in Google 's crawl,
on your site?
54,57M / (2,86M + 54,57M) = 95%
% among number of URLs
crawled by Botify, here 10M

Website Structure

#5 - Internal PageRank
Expected Results
Diluting the Internal PageRank on Non-Compliant Pages should
negatively impact Google’s Crawl Ratio on Compliant Pages
How popularity is
distributed within
the website's
internal structure
A strong crawl
signal supposed
to guide
Googlebot(s)

Confirmation
CRAWL RATIO vs. % OF INTERNAL PAGERANK SPREAD ACROSS COMPLIANT PAGES
From our Experience
The more the internal PageRank is focusing on compliant
pages, the better will the Crawl ratio be
● If Compliant Pages get more Internal
PageRank, their Crawl Ratio should improve.
Pro Tips:
● Don’t waste PR with Nofollow and Noindex tags.
● Crawl Ratio ⇔ opportunity to improve your links.
Higher crawl ratio
→ Optimize your
linking!

What's your internal PR on compliant and non-compliant pages?
(bottom of page)

Minimum number of clicks to reach the page from the Home
#6 - Depth
Expected Results
Slow down the Crawl Lower Crawl Ratio
# Folders Depth # Clicks from the Home Page

Confirmation
CRAWL RATIO vs. AVG. DEPTH IN ANY WEBSITE STRUCTURE
From our Experience
The depth of a page greatly impacts its chances of being
crawled by Google
● We've known for ages that depth is an SEO /
UX problem:
○ Catalog size,
○ Efficient faceted navigation,
○ Structure pruning to remove useless pages.Avg. Depth
● Websites with a higher average Depth
should be less crawled by Google.

What's your average page depth for compliant pages?

#7 - Load Times
Expected Results
Idle Crawl
Huge Impact on Crawl
Ratio
We consider from a web crawler's “point of view” :
- The Time to first byte (web server responsiveness) +
- The Time to download the page HTML source (last byte).

CRAWL RATIO vs. LOAD TIMES IN MILLISECONDES From our Experience
Can we really believe that Load Times don’t impact Google’s crawl?
● When looking at websites of all sizes, Load
Times don’t seem to have any significant
impact on Google’s crawl.
● With higher average Load Times, the Crawl
Ratios should decrease.
Can we dig deeper into the data to clarify?
Disturbing
fact here
Your
target

New learnings!
LESS THAN 10K PAGES
For large websites, Load Times are definitely impacting
Google’s crawl
● Small websites ⇔ Low impact of Load Times
● Big websites ⇔ Huge impact of Load Times
● With higher average Load Times, the Crawl
Ratios should decrease.
Limited
impact
MORE THAN 10K PAGES
From our Experience
Dramatic
impact

What's your average load time for compliant pages?

#8 - Number of Internal Outlinks
Expected Results
Quantity is not Quality
Impact on Crawl Ratio
when too many
Either
Follow NoFollow
To a
Compliant page
To a Not
Compliant page

No Confirmation
CRAWL RATIO vs. NO. OF OUTLINKS PER PAGE CRAWL RATIO vs. NO. OF OUTLINKS TO NON-COMPLIANT PAGES
From the analysis of the DatasetFrom our Experience
Too many Outlinks don't really impact Google’s crawl
● Google’s crawl doesn’t seem to be very
impacted by the number of Outlinks.
● Less outlinks ⇔ Better Crawl Ratio
● Bad outlinks ⇔ Slightly decrease the Crawl
Ratio

Page Content

#9 - Percentage of Content
Expected Results
Low percentage of “real” content
often mean heavier page
More difficult to crawl for
Google
Percentage
of
Content
REAL Content
TEMPLATE
Content

New learnings!
Confirmation
LESS THAN 10K PAGES MORE THAN 1M PAGES
Heavy templates have a huge impact on large websites’ Crawl
Ratio
● Small websites => Limited impact of the % of
Content VS. Template
Potential reason:
Pages with a low % of Content VS. Template tend to be heavier and therefore have slow Load Times.
We have already seen that large websites are highly impacted by Load Times.
● Big websites => Huge impact of the % of
Content VS. Template
Limited
impact
Significant
impact

What's your average template weight?

#10 - Content Size
The number of words
in page, excluding
template.
Expected Results
The more content in average,
the more crawled
Yet a limited impact on
Google crawl

New learnings!
Confirmation
CRAWL RATIO vs. CONTENT SIZE (in words)
From our Experience
Content Size has a very big impact on Google’s crawl
● Content Size impacts Google’s Crawl.
● Websites with more content should be more
crawled by Google but we do not expect a
very high impact on Google’s crawl.
● Content Size is more impactful on Google’s
crawl than we were expecting.

New learnings!
● Content Size positively impacts Google’s crawl for website of any size.
Some
impact
…whatever the size of the website
LESS THAN 100K PAGES MORE THAN 1M PAGES
Positive
impact
Great
impact
BETWEEN 100K AND 1M PAGES

What's your average content size?

How can you go
further?
Use custom charts

After looking at overall, site-wide figures:
Look at pages by type of SEO role (expected traffic)
○ Product pages (ecommerce), article pages (publishing), ads (classifieds)
○ Category pages (lists of products / articles / ads)
→ Breakdown by segment in the report
Use the “advanced selector” feature to explore your data
○ Breakdown by any relevant dimension (content size, depth, linking…)
○ Combine 2 dimensions (segment + content size…)
How can you go further?

What's the impact of load time on Google's crawl?

What's the impact of content size on Google's crawl?

How to combine with a secondary dimension ?

Avg. load time for compliant pages by segments ?

Key
Takeaways

Website Size
dramatically impacts
Google’s Crawl Ratio.
Even small websites are not
crawled at 100% by Google.
Crawl Budget matters.
#1 #2

Content size
matters:
Build high-quality
unique content
Orphans Volume
matters:
Like water, don’t
waste crawl budget!
Structure Depth
matters:
Don’t be afraid, prune
useless branches!
#3 #4 #5

Thank you
for your attention
Get in touch!
hello@botify.com
Questions?

How Does Google Crawl the Web?

More Related Content

What's hot (20)

Similar to How Does Google Crawl the Web? (20)

More from Botify (20)

Recently uploaded (20)

How Does Google Crawl the Web?