Bartosz Goralewicz — The Real Problems Behind Indexing | 5 Hours of Technical SEO

Bartosz Góralewicz Nik Ranger Cindy Krum Will Critchlow
!
!
!

Helping Fortune 500's rank
better and get more traffic
Bartosz Góralewicz
@bart_goralewicz
www.onely.com
We're deeply specialized:
Technical SEO
JavaScript SEO
Rendering SEO (!)
Indexing Issues (!)
Web Performance
Link to this deck -> on the
last slide
@bart_goralewiczwww.onely.com

#TechnicalSEOconfessions
Let me tell you
a personal story

We are researching indexing for a while…

… truth be told…
it gets boring sometimes…
More
proxies
Proxies
Scraping
sitemaps
Google banning
our proxies
Websites banning
our proxies because
we scrape sitemaps
More bans, more
captchas

... Locked down in the house, my mind
started to play tricks on me…
?
?
?
?? ?

I felt like Google started to challenge me
?
?
?

Am I a robot?
?
?
?

Then I finally understood why
SEOs do that..
… I felt a strong urge to do something that
we all hate so much.
Sorry:(

A CORRELATION STUDY!

If it correlates = it’s true!

I called Tomek, our head of R&D
we have 1 million URLs
in our database.
AMAZING - Let’s see what
correlates – I’ll talk about this
at 5 hours of technical SEO.
#AmazingContent

Result: 160 columns filled with data
Which one correlates the most
with is_indexed?

Results?
No correlation.
No correlation.
No correlation.
No correlation.
No correlation.

Long story short…
there is no 1 problem
or 1 solution for indexing problems.

than just crawl budget.
is way more complex

on Google=Indexed

Percentage of URLs NOT indexed
30%76%
15%
81%
14%14%
38%71%
98%

Bartosz Goralewicz — The Real Problems Behind Indexing | 5 Hours of Technical SEO

*sorted by the level of complexity, ascending
Every kind of indexing problems comes from different
origins and requires different solutions.
URL indexing
problems
Mobile-first
related indexing
problems
JavaScript related
indexing problems
Layout based
indexing problems
4 kinds of indexing problems*

#SEJSummit
@bart_goralewicz
Discovery Queue Crawl Rendering
Index selectionIndexingRanking
How indexing works

#SEJSummit
@bart_goralewicz
Discovery Queue Crawl Rendering
Index selectionIndexingRanking
*please don’t start a Twitter war after this slide 
Partial indexing issue = URL not
indexed AFTER it was crawled *
How indexing works

Why is indexing going to be
more and more of a problem?

Google’s
challenge
2010 2010
2012 2012
2014
2020

source: https://twitter.com/methode/status/1261259179983081473
@bart_goralewiczwww.onely.com @bart_goralewiczwww.onely.com

1-minute crash course
Index
selection
@bart_goralewiczwww.onely.com @bart_goralewiczwww.onely.com

Index selection for dummies
SOURCE: Patent Method and apparatus for managing a backlog of pending URL crawls (patent US8676783B1)
Limit: 100 people
Rendering
Links
Efficient
crawling
Content
Indexing
strategy

= new challenges
index selection+Limited resources

Let’s start easy with a little
warm up

URL indexing - example
one.ly/alba-shoes

Problem with the site: command
False negatives

Site: command
new challenges
Site:URL – watch out
for false negatives*
*fortunately, there are a few ways to
avoid those and get 100% accuracy

URL indexing - causes
• Thin content
• Duplicate content
• Cannibalization
• Etc.
Content quality Crawler budget
issues
Index bloat

PARTIAL
INDEXING
AHEAD
URL indexing

Mobile-first related partial indexing
not visible on mobile

Mobile-first related partial indexing
- example
one.ly/yoox-pants
mobile
desktop

desktop
not visible on mobile
one.ly/yoox-pants

Diagnosing mobile-first related
indexing problems
Simple way -
Side by side visual
comparison1

Diagnosing mobile-first related
indexing problems
Diffchecker
2

Make sure that all the content on mobile
is on desktop as well.

Thinking about that new,
shiny JS framework?

JavaScript indexing
≈ 25%
trends over time

We need to talk.Before we move forward...

Remember those good old times, when
only SOME websites were JS-powered?

In 2020, Wordpress, Magento, Wix,
Shopify are usually JS-powered too!
DUH!

Google Hangouts
(August 23rd 2019)

JavaScript SEO is not dying.
It's getting even more complex
Is JavaScript SEO dying?

or… in simpler terms.

JavaScript SEO
is not dying

It is getting even
more f..cked up!

JavaScript SEO leveled up over
the last years.
JS

JavaScript-related indexing problems -
example
with JavaScript without JavaScript

Diagnosing JS-related partial indexing
problems

INDEXED
JavaScript indexing problems
= partial indexing
The URL is JavaScript dependent content –
NOT INDEXED.
How to spot JavaScript indexing problems?

WRS*
To understand JS-related indexing problems, we
need to look under Google’s hood a bit.

To understand JS-related indexing problems, we
need to look under Google’s hood a bit.
WRS*
*Web Rendering Service

Google limits CPU consumption
source: Google Webmaster Conference Product Summit, Mountain View, CA
http://services.google.com/fh/files/events/wmconf_product_summit_slides_publish.pdf

Rendering
- a search
engine's
perspective

Confession
time
Father, 81% of my
content is not indexed

Browser BOR
Browser BORvs
source: Patent Batch-optimized render and fetch architecture (patent US20180276220A1)

How Batch-
Optimized
Rendering works
step by step

Step 1.
BOR skips all resources which
are not essential to generate
a preview of your page
Examples:
Tracking scripts
(Google Analytics,
Hotjar etc.)
Ads
Images*
How Batch-optimized rendering works

vs
Browser BOR
Load: 4.24s Load: 1.91s

Set the value of a Virtual
Clock
Step 2.

1. Virtual Clock’s time runs out*
2. Website’s layout is generated
*simplification
Step 3.

Using this data to rank better
Virtual
Clock
Layout

Virtual Clock
=
Rendering Budget*
*simplification

Rendering pauses
while waiting for
scripts, CSS files etc.
Cost of our
website’s rendering
A script/CSS heavy
website needs
more “virtual time”
on the virtual clock
Source: Patent Batch-optimized render and fetch architecture (patent US20180276220A1)
Virtual Clock

BOR – a place
where real time
doesn’t matter.

Where is
the limit?

How resource-hungry is your website?
Superfast CPUSlower CPU

Measuring the
Virtual Clock load*
of your website.
*Ubersimplification
2 options

Use TLDR
one.ly/tldr
Simulate BOR
in your Chrome
Dev Tools
one.ly/bor
Detailed walkthrough

Virtual clock’s
time runs out
the LAYOUT is
generated

Pre-layout times
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta http-equiv="X-UA-
Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-
scale=1"/> <script>ss_u = false;
ss_g_var = 'no';
if (document.cookie.indexOf("scroll0=") > -1 && window.location.href.indexOf('no_scroll') === -1 )
{
ss_u = true;
ss_g_var = 'yes';
}</script> <link rel="preconnect" href="https://c.amazon-adsystem.com" /><link rel="preconnect"
href="https://aax.amazon-adsystem.com" /><link rel="preconnect" href="https://adserver-
us.adtech.advertising.com" /><link rel="preconnect" href="https://as-sec.casalemedia.com" /><link
rel="preconnect" href="https://ib.adnxs.com" /><link rel="preconnect" href="https://ap.lijit.com"
/><link rel="preconnect" href="https://vap2sfo1.lijit.com" /><link rel="preconnect"
href="https://g2.gumgum.com" /><link rel="preconnect" href="https://tag.1rx.io" /><link
rel="preconnect" href="https://btlr.sharethrough.com" /><link rel="preconnect"
href="https://dmx.districtm.io" /><link rel="preconnect" href="https://apex.go.sonobi.com" /><link
rel="preconnect" href="https://hb.emxdgt.com" /><link rel="preconnect"
href="https://biddr.brealtime.com" /><link rel="preconnect"
href="https://web.hb.ad.cpe.dotomi.com" /><link rel="preconnect"
href="https://s3.amazonaws.com" /><link rel="preconnect" href="https://a.teads.tv" /><link
rel="preconnect" href="https://c.aaxads.com" /><link rel="preconnect" href="https://s.aaxads.com"
/><link rel="preconnect" href="https://related.searchenginejournal.com" /><link rel="preconnect"
href="https://cpm.webtradingspot.com" /><link rel="preconnect" href="https://cdn.jsdelivr.net"
/><link rel="preconnect" href="https://adservice.google.com" /><link rel="preconnect"
href="https://tpc.googlesyndication.com" /><link rel="preconnect"
href="https://www.googletagservices.com" /><link rel="preconnect"
href="https://pagead2.googlesyndication.com" /><link rel="preconnect"
href="https://pubads.g.doubleclick.net" /><link rel="preconnect" href="https://www.google.com"
/><link rel="preconnect" href="https://googleads4.g.doubleclick.net" /><link rel="preconnect"
href="https://cdn.adnxs.com" /><link rel="preconnect" href="https://www.google-analytics.com"
/><link rel="preconnect" href="https://www.googletagservices.com" /><link rel="preconnect"
href="https://connect.facebook.net" /><link rel="dns-prefetch" href="https://platform.twitter.com"
/><link rel="dns-prefetch" href="https://www.youtube.com" /><link rel="preconnect"
href="https://cdn.searchenginejournal.com" /> <script>width_param = 'large';
if( window.innerWidth < 1024 ){
width_param = 'small';
Before 2011
rendering
After 2011
Google Panda
Content quality
updates

Layout vs.
Rendering
new findings

A lot of focus on… layout.
Source: BOR patents (2012 -2018)

text appearing above-the-fold (e.g.,
visible without scrolling) may be
considered more important than text
below-the-line.”
Content location
matters
„

Patent on Scheduling
resource crawls (filed
in 2011)
The importance of the
section is based on (...)
prominence of the section
within the rendered layout.
Source: Patent Scheduling resource crawls (US20130144858A1)
ads ads
„

(…) link positioned under the
“More Top Stories” heading on the
cnn.com has a high probability
of being selected.
„
Some sections may get more
“Link Juice”* from Google
*Wink, Wink John Mu ;)
source: Google patent Ranking documents based on user behavior and/or feature data (US10152520B1)

Google seems to struggle with
indexing “related items”, “you may
also be interested in”.

..more
findings

Going even more
beyond JavaScript…
Ekhm…

…which brings us to
the 4th kind of
partial indexing
problems

http://one.ly/target

URL is indexed

Not indexed

…all those partial indexing
problems are not THAT serious.

*sorted by the level of complexity, ascending
Every kind of indexing problems comes from different
origins and requires different solutions.
Mobile-first
related indexing
problems
JavaScript related
indexing problems
Layout based
indexing problems
Every kind of indexing problem*
URL indexing
problems

How are indexing problems killing
your traffic?

Let’s investigate Target.com
again.

Js + mobile
Quality
Quality
Indexed

Shipping info

Main content

87%
62,24%
https://www.target.com/p/nhl-chicago-blackhawks-checkers-game/-/A-54589615
87%

43,04%

87%
Js + mobile
Indexed
87%
62,24%

43,04%
Quality

Summary

Summary
?
?

Summary

Let’s talk about the results…

Indexed content = rankings.

Bartosz Goralewicz — The Real Problems Behind Indexing | 5 Hours of Technical SEO

More Related Content

What's hot (20)

Similar to Bartosz Goralewicz — The Real Problems Behind Indexing | 5 Hours of Technical SEO (20)

More from Semrush (20)

Recently uploaded (20)

Bartosz Goralewicz — The Real Problems Behind Indexing | 5 Hours of Technical SEO