Logstash for SEO: come monitorare i Log del Web Server in realtime

SEOLog
Alla scoperta
del "dato" perduto
@CardinaleAndrea
#AdvSEOtool
Advanced SEO Tool - 20 Maggio 2014 - Bologna

About me
WPO
Web developer
SysAdmin
SEO
Andrea Cardinale
@CardinaleAndrea
+AndreaCardinale
www.andrea-cardinale.it
www.webandtech.it

Di cosa parlo
● Log: chi, cosa e perché
● Log & Crawler
● Installazione sistema log management
● Panormica interfaccia web

I’m not a
SysAdmin
I’m a
SEO !!!

L o g & S E O
# andiamo al sodo

CrawlBudget
www.sito1.it www.sito2.it www.sito3.it

CRAWLER & LOG
$ la strana coppia

HTTP status codes
200301
302
404
500 503304
410

Importanzadei contenuti
la pensiamo allo stesso modo?

ANALISI FAI DA TE
$ SEOroy Merlin

Logstash for SEO: come monitorare i Log del Web Server in realtime

Complesso?
Noioso?
Lungo?
eppure c’è chi oggi lo fa

IL SEO “DELEGANTE”
SysAdmin, non ti crucciare:
Vuolsi così colà dove si puote
ciò che si vuole, io sono SEO
ed a te devo demandare

IL SEO MITOLOGICO
metà Uomo
metà Excel

Problematiche
$ ‘na parola !!!

Formatononleggibile
Unamareadi dati
Piùserver
Piùclienti

Nessun accesso a server
Real Time
Centralizzato
Accettare formati differenti
Performante
Scalabile
Ricerche semplici
Alerts
ideale

… e gli “amici degli amici”
…isuoi“amici”

● raccoglie, trasmette, interpreta, memorizza
● libero e open source
● scritta principalmente da Jordan Sissel
If a newbie has a bad time,
it's a bug in Lotstash

SHIPPER
STORAGE
+
SEARCH
WEB
INTERFACE
ANALYZER
+
INDEXER
SHIPPER
SHIPPER
BROKER

SHIPPER
STORAGE
+
SEARCH
WEB
INTERFACE
ANALYZER
+
INDEXER
SHIPPER
SHIPPER
BROKER
AGENT SERVER CENTRAL SERVER
logstash.miosito.itsitoesempio.it

$ il gioco si fa duro
Installazioni

# java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (rhel-2.4.7.1.el6_5-x86_64
u55-b13)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)
GIÀ INSTALLATO ?

# sudo yum install java-1.7.0-openjdk
INSTALLIAMO

Logstash
$ Install
Agent & Central

# sudo vi /etc/yum.repos.d/logstash.repo
[logstash-1.4]
name=logstash repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Logstash
$ Install
Central only

# sudo rpm -Uvh http://download.fedoraproject.
org/pub/epel/6/x86_64/epel-release-6-8.
noarch.rpm
# sudo rpm -Uvh http://rpms.famillecollet.
com/enterprise/remi-release-6.rpm

# sudo yum --enablerepo=remi install redis

# sudo vi /etc/redis.conf
# commentiamo la seguente riga
# bind: 127.0.0.1

# sudo vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-1.1]
name=Elasticsearch repository for 1.1.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.1/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

# sudo yum install elasticsearch

# sudo vi /etc/elasticsearch/elasticsearch.yml
# aggiungamo le due seguenti righe
cluster.name: logstash_test
node.name: "elasticsearch_server1"

TRE SEZIONI
input {
}
filter {
}
output {
}

TRE SEZIONI
input {
}
filter {
}
output {
}
Da dove arrivano i log?

TRE SEZIONI
input {
}
filter {
}
output {
}
Qui li manipoliamo

TRE SEZIONI
input {
}
filter {
}
output {
}
Qui li manipoliamo
Che cosa ce ne facciamo?

Logstash
$ Shipper Config
Input

collectd drupal_dblog elasticsearch eventlog
exec file ganglia gelf gemfire generator
graphite heroku imap invalid_input irc jmx
log4j lumberjack pipe puppet_facter rabbitmq
redis relp s3 snmptrap sqlite sqs stdin stomp
syslog tcp twitter udp unix varnishlog
websocket wmi xmpp zenoss zeromq

input {
file {
path => "/var/log/httpd/access-sitoesempio.log"
type => "sitoesempio-apache-access"
# e se si vuol partire da inizio file
start_position => "beginning"
}
}

Logstash
$ Shipper Config
Output

boundary circonus cloudwatch csv datadog datadog_metrics
elasticsearch elasticsearch_http elasticsearch_river
email exec file ganglia gelf gemfire google_bigquery
google_cloud_storage graphite graphtastic hipchat http irc
jira juggernaut librato loggly lumberjack metriccatcher
mongodb nagios nagios_nsca null opentsdb pagerduty pipe
rabbitmq redisriak riemann s3 sns solr_http sqs statsd
stdout stomp syslog tcp udp websocket xmpp zabbix zeromq

output {
redis {
host => "logstash.miosito.it"
data_type => "list"
key => "logstash"
}
}

Logstash
$ Shipper Config
Filter

advisor alter anonymize checksum cidr cipher clone
collate csv datedns drop elapsed elasticsearch
environment extractnumbers fingerprint gelfify geoip
grep grokgrokdiscovery i18n json json_encode kv
metaevent metrics multiline mutate noop prune punct
railsparallelrequest range ruby sleep split sumnumbers
syslog_pri throttle translate unique urldecode
useragent uuid wms wmts xml zeromq

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
locale => "en"
}
}

Logstash
$ Shipper Config
Grok Filter

+ di 120 patterns
https://github.com/elasticsearch/logstash/tree/master/patterns
in particolare
https://github.com/elasticsearch/logstash/blob/master/patterns/grok-patterns
%{pattern:nome_campo}

http://grokdebug.herokuapp.com/

Grok - %{COMBINEDAPACHELOG}
Campo Valore Campo Valore
clientip 99.222.33.44 ident -
auth - timestamp 04/Apr/2014:00:16:14 +0200
verb “GET” request "/robots.txt"
httpversion "1.1" response "200"
"bytes" "66" referrer ""-""
agent ""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)""
99.222.33.44 - - [04/Apr/2014:00:16:14 +0200] "GET /robots.txt HTTP/1.1" 200 66 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Logstash
$ Shipper Config
Final result

input {
file {
path => "/var/log/httpd/access-sitoesempio.log"
type => "sitoesempio-apache-access"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
locale => "en"
}
}
output {
redis {
host => "logstash.miosito.it"
data_type => "list"
key => "logstash"
}
}
# sudo vi /etc/logstash/conf.d/shipper.conf

Logstash
$ Central Config
Input

collectd drupal_dblog elasticsearch eventlog
exec file ganglia gelf gemfire generator
graphite heroku imap invalid_input irc jmx
log4j lumberjack pipe puppet_facter rabbitmq
redis relp s3 snmptrap sqlite sqs stdin
stomp syslog tcp twitter udp unix varnishlog
websocket wmi xmpp zenoss zeromq

input {
redis {
host => "localhost"
type => "redis"
data_type => "list"
key => "logstash"
}
}

Logstash
$ Central Config
Output

boundary circonus cloudwatch csv datadog datadog_metrics
elasticsearchelasticsearch_http
elasticsearch_river email exec file ganglia gelf gemfire
google_bigquery google_cloud_storage graphite graphtastic
hipchat http irc jira juggernaut librato loggly lumberjack
metriccatcher mongodb nagios nagios_nsca null opentsdb
pagerduty pipe rabbitmq redis riak riemann s3 sns
solr_http sqs statsd stdout stomp syslog tcp udp websocket
xmpp zabbix zeromq

output {
elasticsearch {
cluster => "logstash_test"
}
}

output {
elasticsearch {
}
if [response] in ["500", "404"] and [ua_name] = “Googlebot” {
email {
from => "a.cardinale@logstash.webandtech.it"
subject => "Error %{response}"
to => "a.cardinale80@gmail.com"
via => "sendmail"
body => "Error: %{message}"
htmlbody => "<h2>Error %{response}</h2><div align='center'>%{message}</div>"
}
}
}

Logstash
$ Central Config
Filter

advisor alter anonymize checksum cidr cipher clone
collate csv date dns drop elapsed elasticsearch
environment extractnumbers fingerprint gelfify geoip
grep grok grokdiscovery i18n json json_encode kv
metaevent metrics multiline mutate noop prune punct
railsparallelrequest range ruby sleep split sumnumbers
syslog_pri throttle translate unique urldecode
useragent uuid wms wmts xml zeromq

filter {
… tutto ciò che vi occorre ...
}

if [agent] != "-" and [agent] != "" {
useragent {
source => "agent"
prefix => "ua_"
}
}
Mi occorrono info su User Agent?

Mi occorrono info su User Agent?
Campo Valore Valore
ua_device Other Spider
ua_major 26 2
ua_minor 0 1
ua_name Firefox Googlebot
ua_os Ubuntu Other
ua_os_name Ubuntu Other

if [clientip] {
geoip {
source => "clientip"
target => "geoip"
}
}
Mi occorrono info su Geolocalizzazione?

Campo Valore Campo Valore
geoip.continent_code EU geoip.ip 52.11.66.144
geoip.coordinates 9,51 geoip.latitude 51
geoip.country_code2 DE geoip.location 9,51
geoip.country_code3 DEU geoip.longitude 9
geoip.country_name Germany geoip.timezone Europe/Berlin
Mi occorrono info su Geolocalizzazione?

mutate {
convert => [ "bytes", "integer" ]
convert => [ "[geoip][longitude]", "float" ]
convert => [ "[geoip][latitude]", "float" ]
}
Devo convertire un campo?

if [request] =~ //.*?.(jpg|gif|png)/ {
mutate {
add_field => [ "file_type", "image" ]
}
}
Devo aggiungere un campo?

mutate {
remove_field => [ "nomecampoinutile1", "nomecampoinutile2" ]
}
Devo rimuovere un campo?

if [referrer] =~ /^http(s?)://[a-zA-Z0-9-.]+.[a-zA-Z]{2,3}(/S*)?/ {
grok {
match => [
"referrer",
"%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?
(?:%{URIHOST:referrer_host})?(?:%{URIPATHPARAM})?"
]
}
}
Devo estrapolare un campo?

kv {
source => "request"
field_split => "&?"
include_keys => [ "page", "q" ]
prefix => “qs_”
}
Mi occorrono alcuni parametri QueryString?

Logstash
$ Server Config
Final result

input {
redis {
host => "localhost"
type => "redis"
data_type => "list"
key => "logstash"
}
}
filter {
………………………………………….
………………………………………….
………………………………………….
}
output {
elasticsearch {
}
}
# sudo vi /etc/logstash/conf.d/central.conf

# sudo chkconfig redis on
# sudo service redis start
# sudo chkconfig elasticsearch on
# sudo service elasticsearch start
# sudo chkconfig logstash on
# sudo service logstash start

$ guardiamolo in faccia
Interfaccia web

# sudo /opt/logstash/bin/logstash web
Integrato con Logstash
http://logstash.miosito.it:9292

imposto le queries
in questo esempio:
gli http status code che mi
interessano
imposto i filtri
in questo esempio:
solo le richieste
effettuate dagli Spiders

Per ogni query posso
assegnare
un alias ed un colore

Histogram Panel
Posso impostare il tipo di grafico e l’intervallo temporale

Grafico a Torta Grafico a barre
Trends
posso visualizzare i trends su
base personalizzabile
(1 giorno, 7 giorni, mensile, etc…)

Table Panel
Visualizzo gli eventi in tabella
Posso selezionare i
campi da visualizzare

Posso aggiungere nuove righe
ed ordinarle

Posso aggiungere nuovi panels
e successivamente spostarli
con drag and drop

Ogni panel ha diverse
opzioni di configurazione
che variano in base al tipo di panel

Per ogni panel posso scegliere su quali
queries deve “lavorare”

https://support.google.com/webmasters/answer/80553?hl=en

Il log di
Apache è
fatto cosi:

Possiamo“costruirlo” a nostropiacimento

Ti occorrelavelocitàdierogazionediunarichiesta?
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i" %D" supercombined

http://httpd.apache.org/docs/2.2/mod/mod_log_config.html

Links
Logstash Site
http://www.logstash.net/
The Logstash cookbook
http://cookbook.logstash.net/
Logstash source code on on GitHub
https://github.com/elasticsearch/logstash/
Grok debugger
http://grokdebug.herokuapp.com/
Regular expressions
http://regexpal.com/ - http://www.regexr.com/ - http://regexone.com/
Apache - Custom Log Formats
http://httpd.apache.org/docs/2.2/mod/mod_log_config.html
Matt Cutts Interviewed by Eric Enge
http://www.stonetemple.com/articles/interview-matt-cutts-012510.
shtml
Our new search index: Caffeine
http://googleblog.blogspot.it/2010/06/our-new-search-index-
caffeine.html
Large-scale Incremental Processing Using Distributed Transactions and
Notifications
http://research.google.com/pubs/pub36726.html

Images
https://www.flickr.com/photos/carbonnyc/6415460111 https://www.flickr.com/photos/carbonnyc/5757504378
https://www.flickr.com/photos/giuseppesavo/6153578409 https://www.flickr.com/photos/sybrenstuvel/2468506922
https://www.flickr.com/photos/bortescristian/8442828274 https://www.flickr.com/photos/ajari/3921502672
https://www.flickr.com/photos/76657755 https://www.flickr.com/photos/marcobellucci/3534516458
http://it.wikipedia.org/wiki/Nana_bianca https://www.flickr.com/photos/walkingsf/5912385701
http://belladavendere.blogspot.it/2012/12/maria-cristina-franco-next-big-thing.html https://www.flickr.com/photos/liquene/3949414617
https://www.flickr.com/photos/dandechiaro/4151566643 https://www.flickr.com/photos/schroedinger/105285724
https://www.flickr.com/photos/peddhapati/11671457605 https://www.flickr.com/photos/fdecomite/2552413544
https://www.flickr.com/photos/luchilu/495547910 http://en.wikipedia.org/wiki/File:Perkulator2.jpg
https://www.flickr.com/photos/86979666@N00/7623744452

Domande
Andrea Cardinale
@CardinaleAndrea
+AndreaCardinale
www.andrea-cardinale.it
www.webandtech.it
Advanced SEO Tool
http://www.gtmasterclub.it
20 Maggio 2014 - Bologna

Logstash for SEO: come monitorare i Log del Web Server in realtime

More Related Content

What's hot (20)

Similar to Logstash for SEO: come monitorare i Log del Web Server in realtime (20)

Recently uploaded (20)

Logstash for SEO: come monitorare i Log del Web Server in realtime