Hey Googlebot, did you cache that ?

hey Googlebot,
did you cache that?
Petra Kis-Herczegh
@chameleon_jrnl | @botify | #brightonSEO
how to help bots discover valuable content

Hello Googlebot, welcome to my website

HTML,
Stylesheets, DOM,
JSON, JS, API
...here are 241 resources per page for you to render

very important
internal
linking
reviews
super awesome
content
more awesome
content
pagination
third party
JavaScript
more
pagination
awesome
content
blocked by
robots.txt
duplicate
content
lazy loading
uncrawlable
links
HTML,
Stylesheets, DOM,
JSON, JS, API

5.2 Billion Webpages

Google’s resources are finite

source: botify.com/blog/do-slow-page-load-times-negatively-impact-how-google-crawls-your-site
For large websites, load times are definitely impacting Google’s crawl

“There’s so many different factors [that cause pages to load slowly]. Sometimes it’s
your servers, or sometimes your servers respond really quickly but there’s a ton
of JavaScript that has to be processed first. JavaScript is a very expensive
resource because it has to be fully downloaded, parsed, and then executed.”
source: https://www.youtube.com/watch?v=XUOD6pcvnso

How long
does Google
take to fetch,
render and
cache
your pages...
source: https://developers.google.com/search/docs/guides/javascript-seo-basics

...and how
long until
your pages
make it to
Google’s
index?

“Googlebot caches aggressively in order to reduce
network requests and resource usage “

“...if we have something that is already cached, that
doesn’t count towards your crawl budget.”
source: https://www.youtube.com/watch?v=EZtCgrpa6ss

GSC Crawl Stats
Average response time
doesn’t include executing
scripts, images, etc.
Google Search Console >> Settings >> Crawl Stats

GSC Crawl Stats
Doesn’t include information
about resources loaded from
different domains, such as
images from your CDN or any
third party scripts.

GSC Crawl Stats
Total download bytes include
HTML, associated images,
scripts, CSS for the associated
domain only. Cached
resources aren’t counted

Understand
JavaScript
JavaScript elements
● Quick JavaScript Switcher
● Chrome DevTools (right click + inspect)
Talk to your developers!

Understand
JavaScript
JavaScript load time on different pages
and against competitors (for users)
● pagespeed.compare
● Lighthouse
● Page Speed Insights

Understand
JavaScript
Rendered Source (for bots)
● URL inspection tool (GSC)
● Mobile-friendly test
● View Rendered Source

Understand
JavaScript
Use JavaScript conﬁgured crawls!
● include resources important to
load content and links
● exclude tracking, analytics, ads

Large & JavaScript heavy sites Every website
Different challenge

Crawl, Render, Index SERP & Website Engagement
Different Journey

Get pages Indexed Buy Services & Product
Different Goal

Full SEO funnel
Crawl & Render Index Rank Click Convert

Server Response
Total Rendering Time
Core Web Vitals
Site Speed

Crawl Web Vitals

Server Response time Download, parse, execute time Page Load
+ =

+ =
Client Side Rendering (CSR)
User
&
Bot
request
Server
response
Page fully
loaded
Initial HTML + JavaScript
No additional
implementation required
+
Users CPU will impact
load time
Potential rendering issues
for bots
-

+ =
Server Side Rendering (SSR)
User
&
Bot
request
Page fully
loaded
Dynamic HTML (+ JS)
Server
response
Initial HTML + JS
Build time
Improved load time for
users and bots
Can provide a solution to
rendering challenges
+
Expensive
Time to First Byte can be
delayed
Complex
Third-party JS might still
not render
-

+ =
Prerendering or Static Site Generation
User
&
Bot
request
Page fully
loaded
Static HTML (+ JS rehydration)
Server
response
Build time
Prerendered
Cache
Static Site
Generation
Build time doesn’t impact
Time to First Byte
No rendering challenges
for bots
Users and bots are
served fast Static HTML
+
Expensive to scale
Interactive content
requires rehydration in
browser for users
-

“cloaking!”
“black hat!”
“penalties”
IT’S
NOT

Dynamic Rendering
source:https://developers.google.com/search/docs/guides/dynamic-rendering

“Dynamic Rendering is not cloaking”
source:https://developers.google.com/search/docs/guides/dynamic-rendering#cloaking
see also: https://blogs.bing.com/webmaster/october-2018/bingbot-Series-JavaScript,-Dynamic-Rendering,-and-Cloaking-Oh-My

Dynamic Rendering
Deliver the same content seen by the user
Avoid rendering issues caused by JavaScript
(empty pages, missing or uncrawlable links,
missing third party content)
Make crawling more efﬁcient by
delivering the content faster
Change the content
Deceive the crawler

+ =
Dynamic Rendering
Page fully
loaded
Static HTML
Server
response
Build time
Prerendering
Cache
Page fully
loaded
Server
response
User
&
Bot
request
User
Bot

+ =
Hybrid Dynamic Rendering
Page fully
loaded
Static HTML
Server
response
Prerendering
Cache
Page fully
loaded
Server
response
User
&
Bot
request
User
Bot
API fetch
Build time

...all resources have been prerendered for you
Static HTML

more
pagination
reviews
super awesome
content
more
awesome
content
pagination
awesome
content
blocked by
robots.txt
duplicate
content
crawlable links
and navigation
Static HTML
very important
internal
linking

It’s time to make a decision

Site needs to be
interactive for Users
Site has over
500,000 pages
SSG
Site has often changing
content and/or elements
CSR with Prerendering or
SSR or SSG with rehydration
No
Yes
No
No
Yes
Yes
Bots: Dynamic Rendering
Bots: Hybrid Dynamic Rendering
Users: combination of SSR, CSR and Prerendering

Is it worth it?

March 2020

Implemented Dynamic Rendering

March 2021 vs March 2020

54

38%
Number of Pages
with Impressions
24%
Number of Pages
with Trafﬁc
19%
Overall
Organic Trafﬁc

Conclusion: Bot and User experience go hand in hand...

Hey Googlebot, did you cache that ?

More Related Content

What's hot (19)

Similar to Hey Googlebot, did you cache that ? (20)

Recently uploaded (20)

Hey Googlebot, did you cache that ?