Optimizing with Server Logs | Jamie Alberico @ #TechSEO Boost 2018

Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Watching
Googlebot
Watching You
–
Optimizing with Server Logs

All of us share a common goal
To be crawled, indexed, and ranked.

We spend a lot of time and energy
figuring out how to do it better.

Hey! It’s
me talking
about
how to
pwn JS
SEO.
(That was
rad.)

Aaaaaand
here’s my
twitter feed
two weeks
later…
#whompwhomp

Happy ending,
but more fable
than fairy tale.

Actions > Words.

What exactly is Googlebot
crawling?

Things this report is not
● solely traditional web pages
● details about which
Googlebot is crawling
● just pages 200 response
codes
● reflective of how many
unique pages are crawled
● bigger ≠ better

Does reality match our
expectations?

https://twitter.com/JohnMu/status/856449976351825921

Was the crawl healthy?

Googlebot is designed to be a
good citizen of the web...
For Googlebot a speedy site is
a sign of healthy servers...
If the site slows down or
responds with server errors,
the [crawl rate] limit goes down
and Googlebot crawls less.
Politeness is
job 0
https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-
googlebot.html

Is it already too late?

https://twitter.com/JohnMu/status/1032553570468552704

Server logs
are a record
of every
request a
server
receives.

How do I get logs?

credit: https://flic.kr/p/cnorAf
Make new allies.

Ask: Is there already a log
management platform in place?
Be Clear: We do not want Personal
Identification Information (PII) and
request it be removed
BE SPECIFIC.

Web
Server 1
Web
Server 2
Web
Server 3
CDN
DDOS Mitigation/Bot Manager
Logs can
come from
multiple
places in
your stack.
Load Balancer

Accessing Log Files
Apache (Linux Server)
NGINX (Linux Server)
IIS log files (Windows Server)
AWS Load Balancer (Load Balancer)
Google Cloud Load Balancer (Load Balancer)
AWS Cloudfront (CDN)
Accessing CloudFare log files (CDN)
Incapsula (CDN/DDoS Mitigation)
Akamai logs (CDN/DDoS Mitigation)

Aggregate Log
Data
Validate
Googlebot
Read Log Data
Parse logs for
meaningful
search and
analysis
Log
Source
1

Some
assembly
required.

Ways to read logs
Paid: Botify, Logz.io, Sumo Logic, Splunk
Free(mium): Screaming Frog Log Analyzer,
Big Query
Masochistic: Excel, Command Line

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
/twiki/bin/view/TWiki/WikiSyntax?q=ntoon
HTTP/1.1 www.arrow.com 200 7352 616 -
Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+5X+Bu
ild/MMB29P)+AppleWebKit/537.36+(KHTML,+like+Ge
cko)+Chrome/41.0.2272.96+Mobile+Safari/537.36+
(compatible;+Googlebot/2.1;++http://www.google
.com/bot.html) https://www.arrow.com/en/
indiegogo
Server IP

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
indiegogo
Server Name*

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
HTTP/1.1 www.arrow.com 200 7352 616"-"
indiegogo
Date & Time

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
HTTP/1.1 www.arrow.com 200 7352 616"-"
indiegogo
Requester’s IP

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
indiegogo
Request Method

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
indiegogo
URI

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
indiegogo
Hostname*

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
indiegogo
Response Code

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
indiegogo
Response Size*

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
indiegogo
Response Time

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
indiegogo
Requester’s User Agent

216.150.168.131 emeasrvr003
[07/Mar/2018:16:11:58 -0800] 66.249.66.1 GET
indiegogo
Referring URL*

https://support.google.com/webmasters/answer/80553?hl=en
Validate Googlebot IPs: Manual

Validate Googlebot IPs: Bulk with Script
https://dzone.com/articles/shell-script-to-detect-if-the-ip-address-is-google-1

Validate Googlebot IPs: Log
Analyser Functionality

Unlock logs ≤ 6 lines
• Data Source
• Condition
• Parse
• Aggregate
• Sort
• Limit

!!Every site will be different.
Make a new engineering ally.

Use Case
Site section with low
index coverage

Parsing URL Structure
/en/products/blam-o/log-12345
}
}
Language
App
}
Manufacturer
}
SKU

LPT: Limit is how you keep
your access to server logs.

If 40% of my site is articles,
should those URLS
represent 80% of crawl?

Use Case
Google chose a
different canonical than
user.

Query
Duplicate domains by
looking for ‘hostname’
values

Use Case
Sudden crawl flux

Query
Count by response code

No clear answers?
Dig deeper.

Query
Broken JS, CSS, or
AJAX Endpoints

Use Case
Intermittent crawl
errors

Query
Server parity

Analysing Log Files
Screaming Frog Log Analysis
BigQuery + Google Cloud Storage Services
Excel + .csv
Big Query + .csv
Command Line

Making the most of logs means adapting to
your environment and making new friends.

Iterate.
Test.
Share what you learn.

Thank you for your time, energy, and
being part of this wonderful
community
–
@Jammer_Volts
totally@not-a-robot.com

Optimizing with Server Logs | Jamie Alberico @ #TechSEO Boost 2018

More Related Content

What's hot (20)

Similar to Optimizing with Server Logs | Jamie Alberico @ #TechSEO Boost 2018 (20)

Recently uploaded (20)

Optimizing with Server Logs | Jamie Alberico @ #TechSEO Boost 2018