SlideShare a Scribd company logo

How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project - Pycon Belarus 2019

O
Oleksandr Tarasenko

The document outlines the transition of a high-load Python project, prom.ua, from a monolithic architecture with SQLAlchemy and REST API to a GraphQL-based system. Key steps include separating the frontend and backend, implementing mobile app APIs with GraphQL, and utilizing mutations for data handling. This transformation aimed to enhance request handling efficiency, streamline feature development, and improve overall system performance.

Engineering
Related topics:
Node.js Development • Search Engine Optimization
1 of 69
Downloaded 10 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project Oleksandr Tarasenko EVO.company / prom.ua
Some of prom.ua numbers ● RPS 2500-3500 ● total sites over 300 000 ● products ~ 150 million ● pages ~ 400 million 2
What we had ● monolithic WSGI app (11 years) ● mako templates ● monolithic database ● monolithic Nginx ● poor REST API for all sub-services ● slow Delivery and Deployment ● new features were hard to develop 3
4 GraphQL
http://graphql.org/users/ 5
http://graphql.org/users/ 6
Key History Notes ● 2008 - Start. SQLAlchemy ● 2014 - SQLConstruct ● 2014 - ElasticSearch ecosystem ● 2016 - GraphQL ● 2017 - Microservices way ● 2018 - Python 3.7 + async 7
GraphQL concept in prom.ua ● new client-proposal paradigm ● good for read-only requests and data ● two-level (x-level) graph: ○ low-level graph for database mapping (data loading) ○ high-level graph for business logic ● auto documentation and easy testing with graphiql tool ● data validation 8
9 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
Why we choose GraphQL ● good for read-only requests and data ● no mutations needed ● hard to update REST API versions (v1, v2,...vn) ● auto generated documentation ● new client-proposal paradigm ● difficult routes 10
2016 ● hiku, https://hiku.readthedocs.io/ ● two-level graph ● mobile API with DSL-QL (EDN) 11
from hiku.graph import Graph, Node, Link, Field, ... from hiku.sources import sqlalchemy as sa from hiku.types import Boolean, Integer, String, TypeRef, ... product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 12
from hiku.graph import Graph, Node, Link, Field from hiku.sources import sqlalchemy as sa from hiku.types import Boolean, Integer, String, TypeRef product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 13
from hiku.graph import Graph, Node, Link, Field from hiku.sources import sqlalchemy as sa from hiku.types import Boolean, Integer, String, TypeRef product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 14
15 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
2017 ● hiku upgrade to native GraphQL ● new site 2nd level graph ● reimplementing mobile sites ● SSR Node.js Apollo Client ● success story with metrics 16
17 Frontend
18 Node.js
Node.js implementation ● Only for the first request (SSR) ● Good for React prerender ● Routing ● Two-step implementation 19
20
21
22
Success metrics 23 from 20.5% to 14.5%
Success metrics 24 from 57.5% to 49%
??? metrics 25 from 46 to 36
Some of the numbers of the new scheme ● node.js workers 3-6 per front ● React render 5 ms ● prom.ua workers over 750 ● number of requests split to node/python 26
27 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
Early 2018 ● new order and shopping cart graphs ● new user cabinet ● Node.js and apollo for stitching two graphs ● REST API under GraphQL 28
GraphQL schema stitching 29https://labs.getninjas.com.br/sharing-data-in-a-microservices-architecture-using-graphql-97db59357602
How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project - Pycon Belarus 2019
31 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
Middle 2018 ● Hiku upgrade to aliases and new data types ● Use SSR + GraphQL for portal mobile version ● Graph for replace SQLAlchemy models logic and queries ● Rewriting sites SQL queries to GraphQL ● Remove Models from BL ● https://hiku.readthedocs.io/ 32
product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 33
product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 34
product(ids: [1234, 1235]) { id name price } """ SELECT id, name, price FROM product WHERE id IN (:id1, :id2, ..., :idN) """ 35
product(ids: [1234, 1235]) { id name price } """ SELECT id, name, price FROM product WHERE id IN (:id1, :id2, ..., :idN) """ 36
low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), ... Link( 'discount', Optional[TypeRef['Discount']], product_to_discount_query, requires='id', ), ]), Node('Discount', [ Field('id', Integer, discount_query), Field('amount', Integer, discount_query), ]) ]) 37
low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), ... Link( 'discount', Optional[TypeRef['Discount']], product_to_discount_query, requires='id', ), ]), Node('Discount', [ Field('id', Integer, discount_query), Field('amount', Integer, discount_query), ]) ]) 38
low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), ... Link( 'discount', Optional[TypeRef['Discount']], product_to_discount_query, requires='id', ), ]), Node('Discount', [ Field('id', Integer, discount_query), Field('amount', Integer, discount_query), ]) ]) 39
40 discount_query = sa.FieldsQuery('db.session', Discount.__table__) """ SELECT id, name, price, price_currency_id FROM product WHERE id IN (:id1, :id2, ..., :idN) """ product_to_discount_query = sa.LinkQuery( 'db.session', from_column=Discount.product_id, to_column=Discount.product_id, ) """ SELECT product_id FROM discount WHERE product_id IN (:id1, :id2, ..., :idN) """
41 discount_query = sa.FieldsQuery('db.session', Discount.__table__) """ SELECT id, name, price, price_currency_id FROM product WHERE id IN (:id1, :id2, ..., :idN) """ product_to_discount_query = sa.LinkQuery( 'db.session', from_column=Discount.product_id, to_column=Discount.product_id, ) """ SELECT product_id FROM discount WHERE product_id IN (:id1, :id2, ..., :idN) """
42 product_sg = SubGraph(low_level_graph, 'Product') high_level_graph = Graph([ Node('Product', [ Field('id', String, product_sg), Field('name', String, product_sg.compile(S.this.name)), Field('priceText', String, product_sg.compile( get_price_text(S.this) )), Field('hasDiscount', Boolean, product_sg.compile( is_discount_available(S.this, S.this.discount) )), Field('discountedPriceText', String, product_sg.compile( get_discounted_price_text(S.this, S.this.discount) )), ]), ])
43 product_sg = SubGraph(low_level_graph, 'Product') high_level_graph = Graph([ Node('Product', [ Field('id', String, product_sg), Field('name', String, product_sg.compile(S.this.name)), Field('priceText', String, product_sg.compile( get_price_text(S.this) )), Field('hasDiscount', Boolean, product_sg.compile( is_discount_available(S.this, S.this.discount) )), Field('discountedPriceText', String, product_sg.compile( get_discounted_price_text(S.this, S.this.discount) )), ]), ])
44 product_sg = SubGraph(low_level_graph, 'Product') high_level_graph = Graph([ Node('Product', [ Field('id', String, product_sg), Field('name', String, product_sg.compile(S.this.name)), Field('priceText', String, product_sg.compile( get_price_text(S.this) )), Field('hasDiscount', Boolean, product_sg.compile( is_discount_available(S.this, S.this.discount) )), Field('discountedPriceText', String, product_sg.compile( get_discounted_price_text(S.this, S.this.discount) )), ]), ])
45 @define(Record[{ 'price': Float, 'price_currency_id': Integer, 'currency_settings': Any }]) def get_price_text(product): product_price_text = format_currency_data( product_price, product['price_currency_id'], ) ... //.. return product_price_text.formatted_number
Key History Notes ● 2008 - Start. SqlAlchemy ● 2014 - SQL Construct ● 2014 - ElasticSearch ecosystem ● 2016 - GraphQL ● 2017 - Microservices way ● 2018 - Python 3.7 + async 46
47 Python 3.7 + Async + GraphQL
48 def link_to_some_product(opts): product = db.session.query( Product.id ).filter( Product.id == opts['id'], Product.status_on_display(), ).first() if product is not None: return product.id else: return Nothing
49 async def link_to_some_product(opts): expr = select([Product.id]).where( and_( Product.id == opts['id'], Product.status_on_display() ) ) async with async_engine() as query_ctx: product_id = await query_ctx.scalar(expr) return product_id or Nothing
50 async def link_to_some_product(opts): expr = select([Product.id]).where( and_( Product.id == opts['id'], Product.status_on_display() ) ) async with async_engine() as query_ctx: product_id = await query_ctx.scalar(expr) return product_id or Nothing
51 product_query = sa.FieldsQuery( 'db.session', Product.__table__) product_query = asyncsa.FieldsQuery( 'db.session_async', Product.table)
52 product_query = sa.FieldsQuery( 'db.session', Product.__table__) product_query = asyncsa.FieldsQuery( 'db.session_async', Product.__table__)
Mobile API average across all queries: 383 ms -> 323 ms 15% Catalog Graph API average across all queries: 82 ms -> 62 ms 25% Site Graph Api average across all queries 121 ms -> 108 ms 11% Async + GraphQL results
54 Example with aliases def get_price_lists_data(): query = build([ Q.priceLists[ Q.id, Q.name, Q.file_id << Q.priceFileId, Q.date_posted << Q.datePosted, Q.url << Q.fileUrl, ] ]) graph_data = execute(query) return graph_data.priceLists
55 Example with aliases def get_price_lists_data(): query = build([ Q.priceLists[ Q.id, Q.name, Q.file_id << Q.priceFileId, Q.date_posted << Q.datePosted, Q.url << Q.fileUrl, ] ]) graph_data = execute(query) return graph_data.priceLists
A few facts of prom.ua graphs ● number of graphs: ~ 20 ● number of fields: ~ 2000 ● number of links: ~ 300 ● number of nodes: ~ 250 ● single entry point /graphql ● refactoring and new vision for code ● better monitoring ● easy to test API with graphiql 56 Hiku
57 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
End of 2018 - now ● Hiku upgrade to mutations ● New 2nd level graph for opinions and mobile cabinet API ● Read and write with graph 58
mutation_graph = Graph(repr_graph.nodes + [ Root([ Link( 'addNewOpinion', TypeRef['NewCommentResult'], add_new_comment_for_opinion, options=[ Option('opinion_id', Integer), Option('comment', String), ], requires=None, ), ]), ])
mutation_graph = Graph(repr_graph.nodes + [ Root([ Link( 'addNewOpinion', TypeRef['NewCommentResult'], add_new_comment_for_opinion, options=[ Option('opinion_id', Integer), Option('comment', String), ], requires=None, ), ]), ])
def add_new_comment_for_opinion(ctx, options): opinion_id = options['opinion_id'] comment = options['comment'] user = ctx['user'] opinion = Opinion.get(opinion_id) ... form = OpinionCommentForm() if not form.validate(): return NewCommentResult(id=None, errors=form.validate_resp()) comment = opinion.new_comment(...) comment.message = clear_text(form.data['comment']) comment.author_user_id = user.id db.session.commit() return NewCommentResult(id=comment.id, errors=[])
def add_new_comment_for_opinion(ctx, options): opinion_id = options['opinion_id'] comment = options['comment'] user = ctx['user'] opinion = Opinion.get(opinion_id) ... form = OpinionCommentForm() if not form.validate(): return NewCommentResult(id=None, errors=form.validate_resp()) comment = opinion.new_comment(...) comment.message = clear_text(form.data['comment']) comment.author_user_id = user.id db.session.commit() return NewCommentResult(id=comment.id, errors=[])
def add_new_comment_for_opinion(ctx, options): opinion_id = options['opinion_id'] comment = options['comment'] user = ctx['user'] opinion = Opinion.get(opinion_id) ... form = OpinionCommentForm() if not form.validate(): return NewCommentResult(id=None, errors=form.validate_resp()) comment = opinion.new_comment(...) comment.message = clear_text(form.data['comment']) comment.author_user_id = user.id db.session.commit() return NewCommentResult(id=comment.id, errors=[])
def add_new_comment_for_opinion(ctx, options): opinion_id = options['opinion_id'] comment = options['comment'] user = ctx['user'] opinion = Opinion.get(opinion_id) ... form = OpinionCommentForm() if not form.validate(): return NewCommentResult(id=None, errors=form.validate_resp()) comment = opinion.new_comment(...) comment.message = clear_text(form.data['comment']) comment.author_user_id = user.id db.session.commit() return NewCommentResult(id=comment.id, errors=[])
65 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
New client-proposal paradigm via GraphQL
GraphQL everywhere
Testing with hypothesis
Q/A Hiku

More Related Content

ODP
java ee 6 Petcatalog
Carol McDonald
 
PDF
Introduction to interactive data visualisation using R Shiny
anamarisaguedes
 
PDF
ASP.Net 3.5 SP1 Dynamic Data
micham
 
PDF
iOS Reactive Cocoa Pipeline
Ivan Trifonov
 
PDF
Salesforce meetup | Custom document generation
Accenture Hungary
 
PDF
Getting Started with GraphQL && PHP
Andrew Rota
 
PDF
Salesforce meetup | Lightning Web Component
Accenture Hungary
 
PDF
Tutorial: Building a GraphQL API in PHP
Andrew Rota
 
java ee 6 Petcatalog
Carol McDonald
 
Introduction to interactive data visualisation using R Shiny
anamarisaguedes
 
ASP.Net 3.5 SP1 Dynamic Data
micham
 
iOS Reactive Cocoa Pipeline
Ivan Trifonov
 
Salesforce meetup | Custom document generation
Accenture Hungary
 
Getting Started with GraphQL && PHP
Andrew Rota
 
Salesforce meetup | Lightning Web Component
Accenture Hungary
 
Tutorial: Building a GraphQL API in PHP
Andrew Rota
 

What's hot (8)

PDF
Building interactive web app with shiny
Vivian S. Zhang
 
PDF
Recoil at Codete Webinars #3
Mateusz Bryła
 
PPTX
Building a GraphQL API in PHP
Andrew Rota
 
PPTX
Smarter data analysis with JavaScript and Azure ML functions in Excel
Microsoft Tech Community
 
PDF
Performant APIs with GraphQL and PHP (Dutch PHP 2019)
Andrew Rota
 
PDF
Charting with Google
Russell Heimlich
 
PPTX
Google Chart Tools Kanika Garg (10BM60035) Lavanya R. (10BM60042)
Kanika Garg
 
PDF
Integrating React.js Into a PHP Application: Dutch PHP 2019
Andrew Rota
 
Building interactive web app with shiny
Vivian S. Zhang
 
Recoil at Codete Webinars #3
Mateusz Bryła
 
Building a GraphQL API in PHP
Andrew Rota
 
Smarter data analysis with JavaScript and Azure ML functions in Excel
Microsoft Tech Community
 
Performant APIs with GraphQL and PHP (Dutch PHP 2019)
Andrew Rota
 
Charting with Google
Russell Heimlich
 
Google Chart Tools Kanika Garg (10BM60035) Lavanya R. (10BM60042)
Kanika Garg
 
Integrating React.js Into a PHP Application: Dutch PHP 2019
Andrew Rota
 
Ad

Similar to How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project - Pycon Belarus 2019 (20)

PDF
How to separate frontend from a highload python project with no problems - Py...
Oleksandr Tarasenko
 
PDF
apidays LIVE Paris - Augmenting a Legacy REST API with GraphQL by Clément Vil...
apidays
 
DOCX
GraphQL Advanced Concepts A Comprehensive Guide.docx
ssuser5583681
 
PPTX
Introduction to GraphQL
Rodrigo Prates
 
PDF
Graphql
Niv Ben David
 
PDF
Intro to GraphQL
Charles Burgess
 
PDF
apidays LIVE Paris - GraphQL meshes by Jens Neuse
apidays
 
PDF
Discover GraphQL with Python, Graphene and Odoo
Odoo
 
DOCX
How to Deploy a GraphQL API A Comprehensive Guide.docx
ssuser5583681
 
PDF
GraphQL the holy contract between client and server
Pavel Chertorogov
 
PDF
Adding GraphQL to your existing architecture
Sashko Stubailo
 
PDF
GraphQL across the stack: How everything fits together
Sashko Stubailo
 
PDF
GraphQL Isn't An Excuse To Stop Writing Docs
Pronovix
 
PDF
GraphQL Munich Meetup #1 - How We Use GraphQL At Commercetools
Nicola Molinari
 
PDF
apidays LIVE Hong Kong 2021 - GraphQL : Beyond APIs, graph your enterprise by...
apidays
 
PPTX
Green Custard Friday Talk 8: GraphQL
Green Custard
 
DOCX
Graphql for Frontend Developers Simplifying Data Fetching.docx
ssuser5583681
 
PDF
Scaling Your Team With GraphQL: Why Relationships Matter
Joel Bowen
 
PDF
GraphQL Bangkok meetup 5.0
Tobias Meixner
 
PPTX
GraphQL - an elegant weapon... for more civilized age
Bartosz Sypytkowski
 
How to separate frontend from a highload python project with no problems - Py...
Oleksandr Tarasenko
 
apidays LIVE Paris - Augmenting a Legacy REST API with GraphQL by Clément Vil...
apidays
 
GraphQL Advanced Concepts A Comprehensive Guide.docx
ssuser5583681
 
Introduction to GraphQL
Rodrigo Prates
 
Graphql
Niv Ben David
 
Intro to GraphQL
Charles Burgess
 
apidays LIVE Paris - GraphQL meshes by Jens Neuse
apidays
 
Discover GraphQL with Python, Graphene and Odoo
Odoo
 
How to Deploy a GraphQL API A Comprehensive Guide.docx
ssuser5583681
 
GraphQL the holy contract between client and server
Pavel Chertorogov
 
Adding GraphQL to your existing architecture
Sashko Stubailo
 
GraphQL across the stack: How everything fits together
Sashko Stubailo
 
GraphQL Isn't An Excuse To Stop Writing Docs
Pronovix
 
GraphQL Munich Meetup #1 - How We Use GraphQL At Commercetools
Nicola Molinari
 
apidays LIVE Hong Kong 2021 - GraphQL : Beyond APIs, graph your enterprise by...
apidays
 
Green Custard Friday Talk 8: GraphQL
Green Custard
 
Graphql for Frontend Developers Simplifying Data Fetching.docx
ssuser5583681
 
Scaling Your Team With GraphQL: Why Relationships Matter
Joel Bowen
 
GraphQL Bangkok meetup 5.0
Tobias Meixner
 
GraphQL - an elegant weapon... for more civilized age
Bartosz Sypytkowski
 
Ad

Recently uploaded (20)

PPTX
Lecture Notes Electrical Wiring System Components
eedgecbhojpur
 
PDF
Mohammad Mahdi Farshadian CV - Prospective PhD Student 2026
Educational Group Mohammad Farshadian
 
PPTX
UNIT-1 - COAL BASED THERMAL POWER PLANTS
Chandra Kumar S
 
PPTX
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
PPTX
CYBER-CRIMES AND SECURITY A guide to understanding
Davies Chacko
 
PPTX
MCN 401 KTU-2019-PPE KITS-MODULE 2.pptx
VinayB68
 
PPTX
MET 305 2019 SCHEME MODULE 2 COMPLETE.pptx
VinayB68
 
PPT
CRASH COURSE IN ALTERNATIVE PLUMBING CLASS
rene militante
 
PDF
July 2025 - Top 10 Read Articles in International Journal of Software Enginee...
sebastianku31
 
PDF
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 
PPTX
Engineering Ethics, Safety and Environment [Autosaved] (1).pptx
MehrabTaseenTalukder
 
PPTX
CH1 Production IntroductoryConcepts.pptx
RacemMellouli1
 
PDF
Mitigating Risks through Effective Management for Enhancing Organizational Pe...
IJAEMSJORNAL
 
PDF
SM_6th-Sem__Cse_Internet-of-Things.pdf IOT
smceramu
 
PDF
PRIZ Academy - 9 Windows Thinking Where to Invest Today to Win Tomorrow.pdf
PRIZ Guru
 
PPT
Mechanical Engineering MATERIALS Selection
arsalanahmad705384
 
PPTX
UNIT 4 Total Quality Management .pptx
gokuld13012005
 
PPTX
CARTOGRAPHY AND GEOINFORMATION VISUALIZATION chapter1 NPTE (2).pptx
abhi1361yadav
 
PDF
Evaluating the Democratization of the Turkish Armed Forces from a Normative P...
jpsjournal1
 
DOCX
573137875-Attendance-Management-System-original
AYUSHMANAV
 
Lecture Notes Electrical Wiring System Components
eedgecbhojpur
 
Mohammad Mahdi Farshadian CV - Prospective PhD Student 2026
Educational Group Mohammad Farshadian
 
UNIT-1 - COAL BASED THERMAL POWER PLANTS
Chandra Kumar S
 
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
CYBER-CRIMES AND SECURITY A guide to understanding
Davies Chacko
 
MCN 401 KTU-2019-PPE KITS-MODULE 2.pptx
VinayB68
 
MET 305 2019 SCHEME MODULE 2 COMPLETE.pptx
VinayB68
 
CRASH COURSE IN ALTERNATIVE PLUMBING CLASS
rene militante
 
July 2025 - Top 10 Read Articles in International Journal of Software Enginee...
sebastianku31
 
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 
Engineering Ethics, Safety and Environment [Autosaved] (1).pptx
MehrabTaseenTalukder
 
CH1 Production IntroductoryConcepts.pptx
RacemMellouli1
 
Mitigating Risks through Effective Management for Enhancing Organizational Pe...
IJAEMSJORNAL
 
SM_6th-Sem__Cse_Internet-of-Things.pdf IOT
smceramu
 
PRIZ Academy - 9 Windows Thinking Where to Invest Today to Win Tomorrow.pdf
PRIZ Guru
 
Mechanical Engineering MATERIALS Selection
arsalanahmad705384
 
UNIT 4 Total Quality Management .pptx
gokuld13012005
 
CARTOGRAPHY AND GEOINFORMATION VISUALIZATION chapter1 NPTE (2).pptx
abhi1361yadav
 
Evaluating the Democratization of the Turkish Armed Forces from a Normative P...
jpsjournal1
 
573137875-Attendance-Management-System-original
AYUSHMANAV
 

How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project - Pycon Belarus 2019

  • 1. How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project Oleksandr Tarasenko EVO.company / prom.ua
  • 2. Some of prom.ua numbers ● RPS 2500-3500 ● total sites over 300 000 ● products ~ 150 million ● pages ~ 400 million 2
  • 3. What we had ● monolithic WSGI app (11 years) ● mako templates ● monolithic database ● monolithic Nginx ● poor REST API for all sub-services ● slow Delivery and Deployment ● new features were hard to develop 3
  • 7. Key History Notes ● 2008 - Start. SQLAlchemy ● 2014 - SQLConstruct ● 2014 - ElasticSearch ecosystem ● 2016 - GraphQL ● 2017 - Microservices way ● 2018 - Python 3.7 + async 7
  • 8. GraphQL concept in prom.ua ● new client-proposal paradigm ● good for read-only requests and data ● two-level (x-level) graph: ○ low-level graph for database mapping (data loading) ○ high-level graph for business logic ● auto documentation and easy testing with graphiql tool ● data validation 8
  • 9. 9 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
  • 10. Why we choose GraphQL ● good for read-only requests and data ● no mutations needed ● hard to update REST API versions (v1, v2,...vn) ● auto generated documentation ● new client-proposal paradigm ● difficult routes 10
  • 11. 2016 ● hiku, https://hiku.readthedocs.io/ ● two-level graph ● mobile API with DSL-QL (EDN) 11
  • 12. from hiku.graph import Graph, Node, Link, Field, ... from hiku.sources import sqlalchemy as sa from hiku.types import Boolean, Integer, String, TypeRef, ... product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 12
  • 13. from hiku.graph import Graph, Node, Link, Field from hiku.sources import sqlalchemy as sa from hiku.types import Boolean, Integer, String, TypeRef product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 13
  • 14. from hiku.graph import Graph, Node, Link, Field from hiku.sources import sqlalchemy as sa from hiku.types import Boolean, Integer, String, TypeRef product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 14
  • 15. 15 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
  • 16. 2017 ● hiku upgrade to native GraphQL ● new site 2nd level graph ● reimplementing mobile sites ● SSR Node.js Apollo Client ● success story with metrics 16
  • 19. Node.js implementation ● Only for the first request (SSR) ● Good for React prerender ● Routing ● Two-step implementation 19
  • 20. 20
  • 21. 21
  • 22. 22
  • 26. Some of the numbers of the new scheme ● node.js workers 3-6 per front ● React render 5 ms ● prom.ua workers over 750 ● number of requests split to node/python 26
  • 27. 27 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
  • 28. Early 2018 ● new order and shopping cart graphs ● new user cabinet ● Node.js and apollo for stitching two graphs ● REST API under GraphQL 28
  • 31. 31 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
  • 32. Middle 2018 ● Hiku upgrade to aliases and new data types ● Use SSR + GraphQL for portal mobile version ● Graph for replace SQLAlchemy models logic and queries ● Rewriting sites SQL queries to GraphQL ● Remove Models from BL ● https://hiku.readthedocs.io/ 32
  • 33. product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 33
  • 34. product_query = sa.FieldsQuery('db.session', Product.__table__) low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), Field('name', String, product_query), Field('price', None, product_query), ... ]) 34
  • 35. product(ids: [1234, 1235]) { id name price } """ SELECT id, name, price FROM product WHERE id IN (:id1, :id2, ..., :idN) """ 35
  • 36. product(ids: [1234, 1235]) { id name price } """ SELECT id, name, price FROM product WHERE id IN (:id1, :id2, ..., :idN) """ 36
  • 37. low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), ... Link( 'discount', Optional[TypeRef['Discount']], product_to_discount_query, requires='id', ), ]), Node('Discount', [ Field('id', Integer, discount_query), Field('amount', Integer, discount_query), ]) ]) 37
  • 38. low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), ... Link( 'discount', Optional[TypeRef['Discount']], product_to_discount_query, requires='id', ), ]), Node('Discount', [ Field('id', Integer, discount_query), Field('amount', Integer, discount_query), ]) ]) 38
  • 39. low_level_graph = Graph([ Node('Product', [ Field('id', Integer, product_query), ... Link( 'discount', Optional[TypeRef['Discount']], product_to_discount_query, requires='id', ), ]), Node('Discount', [ Field('id', Integer, discount_query), Field('amount', Integer, discount_query), ]) ]) 39
  • 40. 40 discount_query = sa.FieldsQuery('db.session', Discount.__table__) """ SELECT id, name, price, price_currency_id FROM product WHERE id IN (:id1, :id2, ..., :idN) """ product_to_discount_query = sa.LinkQuery( 'db.session', from_column=Discount.product_id, to_column=Discount.product_id, ) """ SELECT product_id FROM discount WHERE product_id IN (:id1, :id2, ..., :idN) """
  • 41. 41 discount_query = sa.FieldsQuery('db.session', Discount.__table__) """ SELECT id, name, price, price_currency_id FROM product WHERE id IN (:id1, :id2, ..., :idN) """ product_to_discount_query = sa.LinkQuery( 'db.session', from_column=Discount.product_id, to_column=Discount.product_id, ) """ SELECT product_id FROM discount WHERE product_id IN (:id1, :id2, ..., :idN) """
  • 42. 42 product_sg = SubGraph(low_level_graph, 'Product') high_level_graph = Graph([ Node('Product', [ Field('id', String, product_sg), Field('name', String, product_sg.compile(S.this.name)), Field('priceText', String, product_sg.compile( get_price_text(S.this) )), Field('hasDiscount', Boolean, product_sg.compile( is_discount_available(S.this, S.this.discount) )), Field('discountedPriceText', String, product_sg.compile( get_discounted_price_text(S.this, S.this.discount) )), ]), ])
  • 43. 43 product_sg = SubGraph(low_level_graph, 'Product') high_level_graph = Graph([ Node('Product', [ Field('id', String, product_sg), Field('name', String, product_sg.compile(S.this.name)), Field('priceText', String, product_sg.compile( get_price_text(S.this) )), Field('hasDiscount', Boolean, product_sg.compile( is_discount_available(S.this, S.this.discount) )), Field('discountedPriceText', String, product_sg.compile( get_discounted_price_text(S.this, S.this.discount) )), ]), ])
  • 44. 44 product_sg = SubGraph(low_level_graph, 'Product') high_level_graph = Graph([ Node('Product', [ Field('id', String, product_sg), Field('name', String, product_sg.compile(S.this.name)), Field('priceText', String, product_sg.compile( get_price_text(S.this) )), Field('hasDiscount', Boolean, product_sg.compile( is_discount_available(S.this, S.this.discount) )), Field('discountedPriceText', String, product_sg.compile( get_discounted_price_text(S.this, S.this.discount) )), ]), ])
  • 45. 45 @define(Record[{ 'price': Float, 'price_currency_id': Integer, 'currency_settings': Any }]) def get_price_text(product): product_price_text = format_currency_data( product_price, product['price_currency_id'], ) ... //.. return product_price_text.formatted_number
  • 46. Key History Notes ● 2008 - Start. SqlAlchemy ● 2014 - SQL Construct ● 2014 - ElasticSearch ecosystem ● 2016 - GraphQL ● 2017 - Microservices way ● 2018 - Python 3.7 + async 46
  • 47. 47 Python 3.7 + Async + GraphQL
  • 48. 48 def link_to_some_product(opts): product = db.session.query( Product.id ).filter( Product.id == opts['id'], Product.status_on_display(), ).first() if product is not None: return product.id else: return Nothing
  • 49. 49 async def link_to_some_product(opts): expr = select([Product.id]).where( and_( Product.id == opts['id'], Product.status_on_display() ) ) async with async_engine() as query_ctx: product_id = await query_ctx.scalar(expr) return product_id or Nothing
  • 50. 50 async def link_to_some_product(opts): expr = select([Product.id]).where( and_( Product.id == opts['id'], Product.status_on_display() ) ) async with async_engine() as query_ctx: product_id = await query_ctx.scalar(expr) return product_id or Nothing
  • 51. 51 product_query = sa.FieldsQuery( 'db.session', Product.__table__) product_query = asyncsa.FieldsQuery( 'db.session_async', Product.table)
  • 52. 52 product_query = sa.FieldsQuery( 'db.session', Product.__table__) product_query = asyncsa.FieldsQuery( 'db.session_async', Product.__table__)
  • 53. Mobile API average across all queries: 383 ms -> 323 ms 15% Catalog Graph API average across all queries: 82 ms -> 62 ms 25% Site Graph Api average across all queries 121 ms -> 108 ms 11% Async + GraphQL results
  • 54. 54 Example with aliases def get_price_lists_data(): query = build([ Q.priceLists[ Q.id, Q.name, Q.file_id << Q.priceFileId, Q.date_posted << Q.datePosted, Q.url << Q.fileUrl, ] ]) graph_data = execute(query) return graph_data.priceLists
  • 55. 55 Example with aliases def get_price_lists_data(): query = build([ Q.priceLists[ Q.id, Q.name, Q.file_id << Q.priceFileId, Q.date_posted << Q.datePosted, Q.url << Q.fileUrl, ] ]) graph_data = execute(query) return graph_data.priceLists
  • 56. A few facts of prom.ua graphs ● number of graphs: ~ 20 ● number of fields: ~ 2000 ● number of links: ~ 300 ● number of nodes: ~ 250 ● single entry point /graphql ● refactoring and new vision for code ● better monitoring ● easy to test API with graphiql 56 Hiku
  • 57. 57 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL
  • 58. End of 2018 - now ● Hiku upgrade to mutations ● New 2nd level graph for opinions and mobile cabinet API ● Read and write with graph 58
  • 59. mutation_graph = Graph(repr_graph.nodes + [ Root([ Link( 'addNewOpinion', TypeRef['NewCommentResult'], add_new_comment_for_opinion, options=[ Option('opinion_id', Integer), Option('comment', String), ], requires=None, ), ]), ])
  • 60. mutation_graph = Graph(repr_graph.nodes + [ Root([ Link( 'addNewOpinion', TypeRef['NewCommentResult'], add_new_comment_for_opinion, options=[ Option('opinion_id', Integer), Option('comment', String), ], requires=None, ), ]), ])
  • 61. def add_new_comment_for_opinion(ctx, options): opinion_id = options['opinion_id'] comment = options['comment'] user = ctx['user'] opinion = Opinion.get(opinion_id) ... form = OpinionCommentForm() if not form.validate(): return NewCommentResult(id=None, errors=form.validate_resp()) comment = opinion.new_comment(...) comment.message = clear_text(form.data['comment']) comment.author_user_id = user.id db.session.commit() return NewCommentResult(id=comment.id, errors=[])
  • 62. def add_new_comment_for_opinion(ctx, options): opinion_id = options['opinion_id'] comment = options['comment'] user = ctx['user'] opinion = Opinion.get(opinion_id) ... form = OpinionCommentForm() if not form.validate(): return NewCommentResult(id=None, errors=form.validate_resp()) comment = opinion.new_comment(...) comment.message = clear_text(form.data['comment']) comment.author_user_id = user.id db.session.commit() return NewCommentResult(id=comment.id, errors=[])
  • 63. def add_new_comment_for_opinion(ctx, options): opinion_id = options['opinion_id'] comment = options['comment'] user = ctx['user'] opinion = Opinion.get(opinion_id) ... form = OpinionCommentForm() if not form.validate(): return NewCommentResult(id=None, errors=form.validate_resp()) comment = opinion.new_comment(...) comment.message = clear_text(form.data['comment']) comment.author_user_id = user.id db.session.commit() return NewCommentResult(id=comment.id, errors=[])
  • 64. def add_new_comment_for_opinion(ctx, options): opinion_id = options['opinion_id'] comment = options['comment'] user = ctx['user'] opinion = Opinion.get(opinion_id) ... form = OpinionCommentForm() if not form.validate(): return NewCommentResult(id=None, errors=form.validate_resp()) comment = opinion.new_comment(...) comment.message = clear_text(form.data['comment']) comment.author_user_id = user.id db.session.commit() return NewCommentResult(id=comment.id, errors=[])
  • 65. 65 Step 1. Mobile App API with GraphQL Step 2. Separating Frontend from Backend Step 3. Graph services as Proxy via different Graph APIs Step 4. Replace SQLAlchemy models logic via Graph Step 5. Mutations in Graph API Step 6. A brave new world with GraphQL