Why you should be using structured logs

Why you should be
using structured logs
Stefan Krawczyk
@stefkrawczyk
linkedin.com/in/skrawczyk
Try out Stitch Fix → goo.gl/Q3tCQ3

Outline:
What is Stitch Fix/Who am I?
What logs am I talking about?
What are structured logs?
Why are structured logs more fun?
Caveats with structured logs
How to implement a structured logger
Conclusion

Outline:
> What is Stitch Fix/Who am I?
Conclusion

What
•
Personal Styling Service

Stitch Fix: Personal Styling Service
•
•
•
•
•

Stitch Fix: Algorithms Org.
•
•
•
•
•
•
•
•
•
•

Stitch Fix: Algorithms Org.
•
•
•
•
• < - - - - - - - - - - - - - - - - Manager of Algo. Dev. Platform Team.
•
•
•
•
•

Outline:
> What logs am I talking about?
Conclusion

Aug 13 14:08:58 ip-10-8-0-122 com.apple.xpc.launchd[1] (com.google.keystone.daemon[20144]): Endpoint has been activated through legac
launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.google.Keystone.Daemon.Administration
Aug 13 14:08:58 ip-10-8-0-122 com.apple.xpc.launchd[1] (com.google.keystone.daemon[20144]): Endpoint has been activated through legac
launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.google.Keystone.Daemon.UpdateEngine
Aug 13 14:11:30 ip-10-8-0-122 com.apple.xpc.launchd[1] (com.jamfsoftware.task.Every 15 Minutes[20014]): Service exited with abnormal
code: 1
Aug 13 14:19:47 ip-10-8-0-122 syslogd[64]: ASL Sender Statistics
Aug 13 14:20:45 ip-10-8-0-122 AOUDownloadCount[20978]: ERROR|AOUDownloadCount.m|376L|Error:AOUDownloadCount::sendDownloadCountInfo:ge
DownloadCountInfo failed.
code: 1
Aug 13 14:30:58 ip-10-8-0-122 com.apple.xpc.launchd[1] (com.apple.quicklook[21738]): Endpoint has been activated through legacy
launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.quicklook
Aug 13 14:35:23 ip-10-8-0-122 com.apple.xpc.launchd[1] (com.apple.imfoundation.IMRemoteURLConnectionAgent): Unknown key for integer:
_DirtyJetsamMemoryLimit
code: 1
Aug 13 14:47:20 ip-10-8-0-122 com.apple.xpc.launchd[1] (com.apple.xpc.launchd.domain.user.502): Service
"com.apple.xpc.launchd.unmanaged.loginwindow.123" tried to register for endpoint "com.apple.tsm.uiserver" already registered by owner
com.apple.SystemUIServer.agent

●
○
●
●
○
import logging
logger = logging.getLogger(__name__)
...
logger.info('Computing %s for client_id: %s, shipment_id: %s', item_id, client_id, shipment_id)

Example use cases for logs
●
●
●
●
●
●

Examples on how you likely consume logs
●
○ grep “client”
●
○ egrep “client[_id: ]+(d+)”
●
○

Question:
Who here likes using logs?

Outline:
> What are structured logs?
Conclusion

2019-07-10 18:46:34,501 INFO logging_talk_code __main__ 18 Computing 10001 for client_id 1 shipment_id
2
{
"message": "Computing 10001 for client_id 1 shipment_id 2",
"payload": {
"client_id": 1,
"shipment_id": 2,
"item_id": 10001
},
"metadata": {
"code": {
"file_url": "/a/url/code.py",
"line_number": 186,
"file_name": "code.py",

logger.info(f'Computing {item_id} for client_id: {client_id}, shipment_id: {shipment_id}')
logger.info('Computing %(item_id)s for client_id: %(client_id)s shipment_id: %(shipment_id)s',
{'client_id': client_id, 'item_id': item_id, 'shipment_id': shipment_id})

{
"message": "Computing 10001 for client_id 1
shipment_id 2",
"payload": {
"client_id": 1,
"shipment_id": 2,
"item_id": 10001
},
"metadata": {
"code": {
"file_url": "/a/url/path/code.py",
"line_number": 186,
Computing 10001 for client_id 1 shipment_id

-
-
-
-
-
-

●
●
●
●
What are structured logs @ Stitch Fix?
logger.info('Some message optional %(interpolation)s', log_payload)

●
●
●
●
●
●
●
●

●
●
●
●
●
●
●
●
https://commons.wikimedia.org/wiki/File:Python_batteries_included.jpg

Structured logs @ Stitch Fix: Example Usage
Launched an experiment

Outline:
> Why are structured logs more fun?
Conclusion

●
●
●
{
"message": "Computing 10001 for client_id 1
shipment_id 2",
"payload": {
"client_id": 1,
"shipment_id": 2,
"item_id": 10001
},
"metadata": {
"code": {
"line_number": 186,
"module": "code",
"function_name": "up_next"
},
"instance": {
"ami_id": "XXXX",
"id": "XXXX",
"type": "c4.2xlarge",
"tags": {
"aws:autoscaling:groupName":

Slow, Tedious; Not Fun
Streamlined, fast; Fun!

○
-

● Developer knowledge transfer
○ much easier to pick up, use, and evangelize.

Leverage: Developer Knowledge Transfer
● one incentivized way to write a logline!
● only one format that logs come out in!
●
logger.info('some text %(var1)s, %(var2)s', {'var1': value1, 'var2': value2})

●
○
● Oncall debugging/monitoring:
○ inspecting & using logs is easier since it's in a structured format (e.g. stacktraces)

Leverage: Oncall & Debugging/Monitoring

●

payload.warehouse payload.row payload.shipment_idpayload.client_id

secrets

●
○
●
○
● Easy to ingest/use in multiple places/pieces of infrastructure
○ e.g. elasticsearch, presto, ETL, custom reports

Leverage: Easy to reuse in multiple places
●
●
with m_app_traces AS (
SELECT
json_extract_scalar(m.payload, '$.metadata.http_headers["stitchfix-request-id"]') as request_id,
json_extract_scalar(m.payload, '$.payload.data.functionality') as m_functionality,
json_extract_scalar(m.payload, '$.payload.data.context.client_id') as client_id,
json_extract_scalar(m.payload, '$.payload.data.context.shipment_id') as shipment_id,
json_extract_scalar(m.payload, '$.payload.data.context.stylist_id') as stylist_id
FROM
kafkatopic.secret_app_logs m
WHERE
m.date = '20190817'
),
...

Leverage: Easy to reuse in multiple places
def create_df_for_analysis(lines: List[str]) -> int:
"""Counts number of errors in log lines"""
df_base = {'level': [], 'message': [], 'client_id': [], 'timestamp': []}
for line in lines:
json_dict = json.loads(line) # We know the structure so easy to analyze
df_base['timestamp'] = json_dict['timestamp']
df_base['level'] = json_dict['level']
df_base['message'] = json_dict['message']
df_base['client_id'] = json_dict['payload'].get('client_id')
df = pd.DataFrame(df_base)
df.index = df['timestamp']
return df

●
○
●
○
●
○
● Can tie other systems together more easily:
○ e.g. nginx, web apps, ETLs, etc.; can use/join on the same “key” names.

Leverage: tie systems together
●
with app1 AS (
SELECT
m.event_timestamp,
json_extract_scalar(m.payload, '$.query_params.client_id') as client_id,
json_extract_scalar(m.payload, '$.response_time') as response_time
FROM
kafkatopic.nginx_logs m
WHERE
m.date = '20190817'
),
app2 AS (
SELECT
m.event_timestamp,
json_extract_scalar(m.payload, '$.payload.client_id') as client_id,
json_extract_scalar(m.payload, '$.payload.score') as score
FROM
kafkatopic.app1_logs m
WHERE
m.date = '20190817'
)
SELECT
a1.client_id, a1.event_timestamp,
a1.response_time,
a2.score
FROM
app1 a1
JOIN
app2 a2
ON
a1.client_id = a2.client_id

Outline:
> Caveats with structured logs
Conclusion

Untyped -> Typed
○
logger.info('Computing scores for client_id: %(client_id)s shipment_id: %(shipment_id)s',
{'client_id': client_id, 'shipment_id': shipment_id})

Untyped -> Typed
○
●
○
●
○

Naming
●

Naming
●
○
○
■

Serialization
●
○
●
●
●
●

Human Readable Logs
https://docs.python.org/3/howto/logging-cookbook.html#logging-to-multiple-destinations
Computing scores for client_id 1 shipment_id 2
"metadata": {
"code": {
"line_number": 186,
"module": "code",
},
"instance": {
"ami_id": "XXXX",
"id": "XXXX",
"tags": {
"aws:autoscaling:groupName": "k-prod",
"stack": "flotilla",
"stitchfix:clusterName": "k-prod",
"clusterName": "k-prod",
"stitchfix:applicationName": "our_app",
"name": "k-prod"
}
},

Human Readable Logs
https://docs.python.org/3/howto/logging-cookbook.html#logging-to-multiple-destinations
Computing scores for client_id 1 shipment_id 2
"metadata": {
"code": {
"line_number": 186,
"module": "code",
},
"instance": {
"ami_id": "XXXX",
"id": "XXXX",
"tags": {
"aws:autoscaling:groupName": "k-prod",
"stack": "flotilla",
"stitchfix:clusterName": "k-prod",
"clusterName": "k-prod",
"stitchfix:applicationName": "our_app",
"name": "k-prod"
}
},
●
●
○
○

Outline:
> How to implement a structured logger
Conclusion

Python
Logging
Module 3.6+
●
●
●
https://docs.python.org/3/howto/logging.html
https://docs.python.org/3/howto/logging-cookbook.html

Python
Logging
Module 3.6+
●
●
●

Python
Logging
Module 3.6+
●

TL;DR: Logging module (3.6+)
• LogRecord
• Formatter
•
• Handler
•
•
• Logger
•
•

Second: Options for implementation

Very happy we're in Python
•
def info(self, msg, *args, **kwargs):
•
•
•
•

JSON structured logger
●
○
■ format()
●
○
■ format()/emit()
○
■ makeRecord()
■ format()

Code
class PyBayFormatter(logging.Formatter):
"""Implementation of JSON structured logging that works for most handlers."""
def format(self, record: logging.LogRecord) -> str:
"""Overrides parent format function.
:param record: logging.LogRecord object
:return: JSON string
"""
payload = self.make_structured_dict(record)
if hasattr(record, 'exc_info') and record.exc_info: # adds stack traces!
payload['stack_trace'] = self.formatException(record.exc_info)
return json.dumps(payload)
@staticmethod
def make_structured_dict(record: logging.LogRecord) -> dict:
"""Creates dictionary that we want to JSON-ify.
:param record: the LogRecord object.
:return: dict
"""
# next slide

Code
@staticmethod
def make_structured_dict(record: logging.LogRecord) -> dict:
"""Creates dictionary that we want to JSON-ify.
:param record: the LogRecord object.
:return: dict
"""
message_str = record.getMessage() # this interpolates '%' markers in the message.
if isinstance(record.args, dict): # handles case for backwards compatibility of people passing multiple args.
user_payload = record.args
else:
user_payload = {'value': repr(record.args)}
return {
'payload': user_payload,
'message': message_str,
'version': '0.0.1', # always good to have a schema version field
'meta': {
'level': record.levelname,
'name': record.name,
'pid': record.process,
'code': {
'file_name': record.filename,
'file_url': record.pathname,
# etc ...
},
}
}

Hooking this into logger conﬁg
[formatters]
keys=PyBayFormatter
[formatter_PyBayFormatter]
class=package.PyBayFormatter
[handlers]
keys=console_handler
[handler_console_handler]
class=StreamHandler
formatter=PyBayFormatter
args=(sys.stdout,)
[loggers]
keys=root
[logger_root]
level=INFO
handlers=console_handler
import logging
import logging.config
import sys
config_path = 'some_dir/config.ini'
logging.config.fileConfig(config_path)
logger.info('Winning', {'key1': 1, 'key2': 'a'})
### ------- OR manually ----------------
sh = logging.StreamHandler(sys.stdout)
sh.setFormatter(PyBayFormatter())
logger.addHandler(sh)
logger.setLevel(logging.INFO)
logger.info('Hi this is a message', {'and': 'some', 'structured': 'logs'})

Hooking this into logger conﬁg
[formatters]
keys=PyBayFormatter
[formatter_PyBayFormatter]
class=package.PyBayFormatter
[handlers]
keys=console_handler
[handler_console_handler]
class=StreamHandler
formatter =PyBayFormatter
args=(sys.stdout,)
[loggers]
keys=root
[logger_root]
level=INFO
handlers =console_handler
import logging
import logging.config
import sys
config_path = 'some_dir/config.ini'
logging.config.fileConfig(config_path)
logger.info('Winning', {'key1': 1, 'key2': 'a'})
### ------- OR manually ----------------
sh = logging.StreamHandler(sys.stdout)
sh.setFormatter(PyBayFormatter())
logger.addHandler(sh)
logger.setLevel(logging.INFO)
logger.info('Hi this is a message', {'and': 'some', 'structured': 'logs'})

You now just need to push the logs...
→ ✅
→
-
-
-
-
-
-

Outline:
> Conclusion

Conclusion
•
logger.info('Computing %(item_id)s for client_id: %(client_id)s shipment_id: %(shipment_id)s',
{'client_id': client_id, 'item_id': item_id, 'shipment_id': shipment_id})
{
"message": "Computing 10001 for client_id 1 shipment_id 2",
"payload": {
"client_id": 1,
"shipment_id": 2,
"item_id": 10001
},
"metadata": {
"code": {
"line_number": 186,
"module": "code",
},
"instance": {

Conclusion
•
•
●
●
●

Conclusion
•
•
•
●
●
●
●

Conclusion
•
•
•
•
•

Thank you! We’re hiring! Questions?
Try out Stitch Fix → goo.gl/Q3tCQ3@stefkrawczyk
linkedin.com/in/skrawczyk

Why you should be using structured logs

More Related Content

What's hot (20)

Similar to Why you should be using structured logs (20)

Recently uploaded (20)

Why you should be using structured logs