The journey of an (un)orthodox optimization

(UN)ORTHODOX
THE JOURNEY OF AN OPTIMIZATION

SIAN LERK LAU
linkedin.com/in/sianlerk
@kiawin

A little bit about
Jewel…We’re a Singapore-based company with offices in KL and HK too.
We’re hiring!
We’re always looking for talented developers, data scientists, data engineers, devops engineers and more!
Check out our Careers page at
https://www.jewelpaymentech.com/careers.html
Image
Jewel Paymentech
© Copyright 2018 Jewel Paymentech Pte .Ltd. – Reproduction and distribution of this presentation without written permission is
prohibited.
VISION

JOIN THE TEAM
Confidenti
al
Python Developers
Data Scientists
Data Engineers
And many more…
Scan the QR code to
visit our Careers
page:

Mohamad Yusri Shaharin Woon Wai Keen

The journey of an (un)orthodox optimization

SIZE DOES MATTER
result:
+----------------------+
| count(DISTINCT r.id) |
+----------------------+
| 294493 |
+----------------------+
1 row in set (4.58 sec)

JOIN PARTY!
MariaDB> SELECT count(DISTINCT r.id) FROM resource r LEFT JOIN resource_location rl
ON (r.id = rl.resource AND ...) LEFT JOIN `ssl` s ON (r.ssl = s.id) LEFT JOIN
location ON (rl.location = location.id AND ...) LEFT JOIN user pub ON (r.publisher =
pub.id AND ...) LEFT JOIN user resOp ON (pub.principal = resOp.id AND ...) LEFT JOIN
user edgeOp ON (location.operator = edgeOp.id AND ...) LEFT JOIN peer ON (location.id
= peer.location AND peer.buyer = resOp.id) LEFT JOIN edge ON (location.id =
edge.location AND ... AND (edge.operator = resOp.id OR (edge.operator = peer.seller
AND ...)) ) LEFT JOIN location edgeLocation ON (edgeLocation.id = edge.location);
(resource_location_sql)

MORE JOIN PARTIES!
MariaDB> SELECT count(DISTINCT r.id) FROM resource r JOIN resource_edgeGroup reg ON
(r.id = reg.resource AND ...) LEFT JOIN `ssl` s ON (r.ssl = s.id) LEFT JOIN edgeGroup
eg ON (eg.id = reg.edgeGroup AND ...) LEFT JOIN edgeGroup_location egl ON (eg.id =
egl.edgeGroup) LEFT JOIN location ON (egl.location = location.id AND ...) LEFT JOIN
user pub ON (r.publisher = pub.id AND ...) LEFT JOIN user resOp ON (pub.principal =
resOp.id AND ...) LEFT JOIN user edgeOp ON (location.operator = edgeOp.id AND ...)
LEFT JOIN peer ON (location.id = peer.location AND peer.buyer = resOp.id ) LEFT JOIN
edge ON (location.id = edge.location AND ... AND (edge.operator = resOp.id OR
(edge.operator = peer.seller AND ...)) ) LEFT JOIN location edgeLocation ON
(edgeLocation.id = edge.location);
(resource_edgegroup_sql)

THE COST OF THE PARTIES
result:
1272632 rows in set (6 min 34.43 sec)

IT’S ALL ABOUT INDEXING
MariaDB> show create table resourceG
*************************** 1. row ***************************
Table: resource
Create Table: CREATE TABLE `resource` (
...
PRIMARY KEY (`id`),
UNIQUE KEY `publishedName_extant` (`publishedName`,`extant`),
KEY `publisher` (`publisher`),
KEY `pushOrigin` (`pushOrigin`),
KEY `ssl` (`ssl`),
...
)
1 row in set (0.00 sec)

Temporary Table is Faster *
CREATE OR REPLACE TEMPORARY TABLE active_resource (
PRIMARY KEY(`id`),
KEY `publisher` (`publisher`),
KEY `pushOrigin` (`pushOrigin`),
KEY `ssl` (`ssl`)
) AS (
SELECT ...
FROM resource
WHERE ... AND extant = 1
);
CREATE OR REPLACE TEMPORARY TABLE active_peer (
...
) AS (
...
);
CREATE OR REPLACE TEMPORARY TABLE active_edge (
...
) AS (
...
);

CREATE OR REPLACE TEMPORARY TABLE temp_result AS (
SELECT DISTINCT
r.id AS resource,
...
FROM active_resource r
JOIN resource_edgeGroup reg ON (r.id = reg.resource)
LEFT JOIN `ssl` s ON (r.ssl = s.id)
LEFT JOIN edgeGroup eg ON (eg.id = reg.edgeGroup)
LEFT JOIN edgeGroup_location egl ON (eg.id = egl.edgeGroup)
LEFT JOIN location ON (egl.location = location.id AND ...)
LEFT JOIN user pub ON (r.publisher = pub.id AND ...)
LEFT JOIN user resOp ON (pub.principal = resOp.id AND ...)
LEFT JOIN user edgeOp ON (location.operator = edgeOp.id AND ...)
LEFT JOIN active_peer peer ON (location.id = peer.location AND
LEFT JOIN active_edge edge ON (location.id = edge.location AND
(edge.operator = resOp.id OR (edge.operator = peer.seller AND ...)
)
LEFT JOIN location edgeLocation ON (edgeLocation.id = edge.location
WHERE eg.status = "ACTIVE"
AND location.status = "ACTIVE" AND location.extant = 1
AND pub.status IN ("RESTRICTED", "ACTIVE") AND pub.extant = 1
AND resOp.status = "ACTIVE" AND resOp.extant = 1
AND edgeOp.status IN ("RESTRICTED", "ACTIVE") AND edgeOp.extant = 1
);
Temporary Table is Faster *

FOR ALL WE KNOW ABOUT MYSQL
result:
Query OK, 22127 rows affected (1.09 sec)
Query OK, 966172 rows affected (3 min 50.94 sec)
(correction: MariaDB)

PROFILING MYSQL
mysql> SET PROFILING=1;
mysql> SELECT * FROM resource;
mysql> SHOW PROFILES;
+----------+----------+--------------------------+
| Query_ID | Duration | Query |
+----------+----------+--------------------------+
| 0 | 0.000088 | SET PROFILING = 1 |
| 1 | 0.000136 | SELECT * FROM resource |
+----------+----------+--------------------------+
2 rows in set (0.00 sec)
Disclaimer: Adapted from https://dev.mysql.com/doc/refman/8.0/en/show-profile.html

PROFILING MYSQL
mysql> SHOW PROFILE FOR QUERY 1;
+--------------------+----------+
| Status | Duration |
+--------------------+----------+
| query end | 0.000107 |
| freeing items | 0.000008 |
| logging slow query | 0.000015 |
| cleaning up | 0.000006 |
+--------------------+----------+

PROFILING MYSQL
mysql> SHOW PROFILE CPU FOR QUERY 1;
+----------------------+----------+----------+------------+
| Status | Duration | CPU_user | CPU_system |
+----------------------+----------+----------+------------+
| checking permissions | 0.000040 | 0.000038 | 0.000002 |
| creating table | 0.000056 | 0.000028 | 0.000028 |
| After create | 0.011363 | 0.000217 | 0.001571 |
| query end | 0.000375 | 0.000013 | 0.000028 |
| freeing items | 0.000089 | 0.000010 | 0.000014 |
| logging slow query | 0.000019 | 0.000009 | 0.000010 |
| cleaning up | 0.000005 | 0.000003 | 0.000002 |
+----------------------+----------+----------+------------+

“Self-Documenting” Code .
4

REALITY
HURTS
(if not your eyes)

SETTING OUR BASELINE
sianlerk@host:~$ time python oldbdbtest.py
Resource Locations: 23695
Resource Locations (non-empty): 18718
Resource Type: 23695
name=oldbdbtest.py level=INFO {"msg": "Store resource completed",
"completed_ms": 347310}
real 5m48.521s
user 0m43.167s
sys 0m1.264s

if … not in …
resource_id = x['resource']
if resource_id not in existing_resource_locations:
existing_resource_locations[resource_id] = []

1. if … not in …
O(n)

1. USE setdefault FOR NEW value
existing_resource_locations.setdefault(resource_id, [])
O(1)

2. NO set for uniqueness
resource_locations = set()
O(n)

2. USE DISTINCT for uniqueness
resource_locations = set()
SELECT DISTINCT id ...

3. USE MORE dict FOR COMMON USED value
while True:
x = cursor.fetchone()
if not x:
break
resource_operator[x['resource']] = x['resourceOperator']
resource_type[resource_id] = x["type"]
O(1)

4. USE get
if resource_is_http:
edges = mart_location_http_edges.get(location_id, {}) if
is_marketplace else location_http_edges.get(location_id, {})
elif resource_is_stream:
edges = mart_location_stream_edges.get(location_id, {}) if
is_marketplace else location_stream_edges.get(location_id, {})
else:
continue
O(1)

5. MORE dict REPLACING JOIN
# The problematic resource_edgegroup_sql
- resource
- ssl
- edgeGroup
- location
- user, user, user (!!)
- peer
- edge
- location (again)

PROCESS OF ELIMINATION
(by isolation)

(5th Attempt)
sianlerk@host:~$ time python bdb-v5.py
Resource Operator: 19071
real 0m47.964s
user 0m34.398s
sys 0m0.804s

6. FULLY DECONSTRUCT JOIN
# The problematic resource_edgegroup_sql
- resource
- ssl
- edgeGroup
- location
- user, user, user (!!)
- peer
- edge
- location (again)

7. USE try … except … *
try:
edgegroups = resource_edgegroups[resource_id]
except KeyError:
continue

(9th Attempt)
real 0m17.424s
user 0m15.069s
sys 0m0.596s

OVER-ENGINEERING
(we forgot, let the master does its job)

8. KEEP CALM, dict IS NOT THE SOLUTION
resource_edgegroup_sql = """
SELECT
reg.resource, reg.edgeGroup
FROM resource_edgeGroup reg
LEFT JOIN edgeGroup eg ON (eg.id = reg.edgeGroup)
WHERE eg.status = "ACTIVE"
"""
(for everything)

8. KEEP CALM, USE GROUP_CONCAT
SELECT
r.id AS resource,
...
GROUP_CONCAT(reg.edgeGroup) AS edgeGroups,
...

(10th Attempt)
real 0m16.764s
user 0m13.549s
sys 0m0.512s

(11th Attempt)
real 0m15.207s
user 0m11.945s
sys 0m0.384s

(12th Attempt)
real 0m14.721s
user 0m11.313s
sys 0m0.380s

DISCLAIMER
(we have to credit one more thing)

Python Profiling
# Part 1 - Execute the code with cProfile
python -m cProfile -o something.prof something.py
# Part 2 - Visualize the profiling result with SnakeViz
sianlerk@host~$ pip install snakeviz
sianlerk@host~$ snakeviz result.prof

THANK YOU
linkedin.com/in/sianlerk
@kiawin

The journey of an (un)orthodox optimization

More Related Content

What's hot (20)

Similar to The journey of an (un)orthodox optimization (20)

More from Sian Lerk Lau (7)

Recently uploaded (20)

The journey of an (un)orthodox optimization