Serverless flow programming a new perspective (py web meetup, sept 2nd, 2019) (1)

BLACKSWAN TECHNOLOGIES 2019.ALL RIGHTS
WWW.BLACKSWANTECHNOLOGIES.AI
1
Serverless Flow
Programming
A new perspective
BLACKSWAN TECHNOLOGIES 2019.ALL RIGHTS WWW.BLACKSWANTECHNOLOGIES.AI
Asher Sterkin, VP Engineering
Core Technology Group

2
● (Strategic) Domain-Driven Design
● Serverless Architecture
● Cyneﬁn
● Wardley Maps
● Promise Theory
Software Technologist/Architect
Who I am
VP Engineering at Black Swan Technologies

3
Big Data, Cognitive Computing &
Artiﬁcial Intelligence Software Company
BlackSwan Technologies
United Kingdom
Israel
Who we are
Tier 1 enterprise customers (banking, ensurance)
United States
Sri Lanka
Pakistan
Ukraine
Ofﬁce zero, distributed work
We are after top talents, rather than cheap labor

4
Conventional programming languages are growing
ever more enormous, but not stronger. Inherent defects
at the most basic level cause them to be both fat and
weak: their primitive word-at-a-time style of
programming inherited from their common
ancestor--the von Neumann computer, their close
coupling of semantics to state transitions, their division
of programming into a world of expressions and a
world of statements, their inability to effectively use
powerful combining forms for building new programs
from existing ones, and their lack of useful
mathematical properties for reasoning about programs.
An alternative functional style of programming is founded on the use of
combining forms for creating programs.

5
Quiz
● Modular?
● Functional?
● Object-Oriented?
● None from above?

6
The Answer
Python is a Modular Programming language built on the top of Abstract Data Type system
(called objects).
It does support, at certain degree, Functional and Object-Oriented Programming styles.

7
Among the mental aids available to
understand a program (or a proof of its
correctness) there are three that I should
like to mention explicitly:
(1) Enumeration
(2) Mathematical induction
(3) Abstraction.

8
● Data Types (representation and manipulation)
● Unstructured
● Structured
○ Cartesian Product (tuple)
○ Discriminated Union (dict - it’s a tabulated function - me)
○ Array (it’s a tabulated function - Dijkstra)
○ Powerset
○ Sequence
○ Recursive Data Structures
There are certain close analogies between the methods used for structuring data and
the methods for structuring a program which processes that data. Thus, a Cartesian
product corresponds to a compound statement, which assigns values to its
components. Similarly, a discriminated union corresponds to a conditional or case
construction, selecting an appropriate processing method for each alternative. Arrays
and powersets correspond to for statements sequencing through their elements, with
an essentially bounded number of iterations. The sequence structure is the first that
permits construction of types of infinite cardinality, with values of unbounded length;
and it corresponds to the unbounded form of looping, with a while condition to control
termination. The reason why the sequence is unbounded is that one of its
components (i.e. the initial segment) from which it is built up belongs to the same type
as itself, in the same way as the statement which remains to be obeyed after any
iteration of a while loop is the same statement as before.

9
Modular Structured Programming in Python
module.py
def abstraction_A(param1, param2):
abstraction_B(...)
abstraction_C(...)
if condition_D:
abstraction_E(...)
else:
abstraction_F(...)
while condition_G:
abstraction_H(...)

10
Data Stream (Functional Programming)
head
tail
filter(predicate)
head
map(transformer)
head
reduce(accumulator)
flatMap(...)
foldl(...)
foldr(...)

11
Control Flow (Structured Programming)
Step1
Step2
cond
Step3
State1
State2
State3
State0
Functional Programming
restores control flow via Monads

12
Future users of large data banks must be protected from
knowing how the data is organized in the machine (internal
representation)
SELECT * from table WHERE <condition>

13
● Communicating sequential processes (Hoare)
● Statecharts (Harel)
● Actor computing (Hewitt)
● Object-oriented programming (Kay)
● Tuple spaces (Glenerter)
● Reactive Extensions (Meijer)
Not covered here, yet important for the big picture

14
Fast Forward

15
15 years to proceed from Distributed Functional Programming back to SQL
https://towardsdatascience.com/i-have-a-lot-of-data-i-just-dont-know-where-421d98c3cc3a
I would say GFS and BigTable
papers

16

17

18
Amazon Simple Storage
Service (S3)
AWS Glue
Amazon Athena
Amazon EMR
AWS Tools and SDKs

19
PartiQL: one query language for all your data

20
Serverless Map/Reduce

21
● Distributed SQL covers 80% of data
transformation needs, but not all of
them
● What if a sequence of data
transformation steps is required?
● What about rest of 20% which does not
fit naturally into SQL paradigm (e.g.
data download, manual approvals,
etc.)?
Apache Airflow
Data Stream processing ≠
Control Flow Execution

22
Flow Direct Acyclic Graph is composed as if
principles of Structured Programming have never
existed.
Any structured program is a DAG, so why are we
making the same mistake of re-inventing the wheel
over and over again and hoping for a different
result?
It’s workish, but Ugly!!!
dag = DAG('tutorial', default_args=default_args,
schedule_interval=timedelta(days=1))
# t1, t2 and t3 are examples of tasks created by
instantiating operators
t1 = BashOperator(
task_id='print_date',
bash_command='date',
dag=dag)
t2 = BashOperator(
task_id='sleep',
bash_command='sleep 5',
retries=3,
dag=dag)
templated_command = """
{% for i in range(5) %}
echo "{{ ds }}"
echo "{{ macros.ds_add(ds, 7)}}"
echo "{{ params.my_param }}"
{% endfor %}
"""
t3 = BashOperator(
task_id='templated',
bash_command=templated_command,
params={'my_param': 'Parameter I passed in'},
dag=dag)
t2.set_upstream(t1)
t3.set_upstream(t1)
My Godness! This is how you define the flow???

23
● Mature distributed Data Stream and Control Flow solutions are based on late 90-ties innovations:
○ Built around (unreliable) boxes and disks
○ Trying to compensate network latency and stability
○ Leak out lot of heavy IT lifting abstractions (try to configure a Hadoop cluster yourself)
● As such, modern Serverless cloud offerings make all these assumptions irrelevant:
○ No boxes and disks to worry about
○ Cloud infrastructure is inherently unpredictable - study CAP theorem and learn to live with it
● Emerging Serveless Big Data and Workflow solutions are still immature and suffer from the
Innovator's’ Dilemma
It’s just a symptom of deeper problem

24
Overshoot regular user needs at expense
of inconvenience. Most of the time Big
Data is not that big.

25
Announcing Cloud AI Operating System (CAIOS)
● Treat one Region of one Cloud Vendor Account as an AI-power Supercomputer
● Build a portable Operating System to provide:
○ Hardware abstraction layer to insulate different cloud platform specifics
■ Treat native automation solutions (e.g. Cloud Formation) as machine-level code
○ State-of-the art productivity and quality
○ Secure by design application solutions
○ Efficient utilization of the Cloud Supercomputer (still limited) resources

26
System Layers
CAIOS (Cloud AI Operating System)
● Standard CAIOS <Language> Library (example scpl):
○ Common AI and IT primitives
○ Lambda Layer, Lambda Function, Fargate Container, Sage Maker Endpoint
● Staged Flow Programming Environment
○ Glues scXl primitives into coherent flow
○ Triggered by external events
○ Concurrency structure optimization
○ Multi-paradigm Programming (Activity Flow, Data Flow, Map/Reduce, State Chart, …)
Domain Data Research Platform Development Platform
Application Services
Custom Solutions
EngineeringUI
Core Services
BYOS (Bring Your Own Scheme) Middleware

27
In general case, there is no right choice between
multithreading sequential and event-driven asynchronous
programming models. The both need to be combined in
order to achieve a robust and scalable distributed
architecture.

28
Staged Flow Oriented Architecture (SFOA)
External Triggers
Stage
External Data Internal Data
Stage
Stage
InternalTriggers
Internal Data
External Triggers
External Data

29
Universal Pipeline and Workflow Platform
Activity
Choice
Start End
State State
event[guard]/action event
State
event[guard]
event[guard]/action
Source Transform
Split
Transform
Transform
Transform
Merge
Sync
Start
ChoiceStep
Step
Step
Step
Join End
Fork

30
Upload World Check PEPs List
xxx-exchange
/watchlists/wcp/wcp-<version>.zip
build-wcp-index-core
zip2gzip build-db build-tables query-person-names build-person-index
wcp-<version>-person-<matcher-version>
get-next-page
Today:
● Structured Python pseudo-code
● Simulated async/await semantics
● AWS
● Automatically translation to AWS Step
Functions State Machine calling Lambda
functions
Tomorrow:
● Structured Python code with full support of
async i/o
● Multi-cloud
● Automatic selection of the most optimal
concurrency model
be_build_wcp_index = Flow(
'Build World Check Index',
assign(data.db, zip2gzip, data.db),
assign(data.db, create_athena_db, data),
Do(
assign(data.db.query_status, get_athena_query_status, data.db.executions[0])
).While(Or(data.db.query_status.State.eq('QUEUED'), data.db.query_status.State.eq('RUNNING'))).Do(wait(10)),
assign(data.db, create_athena_tables, data),
Do(
).While(Or(data.db.query_status.State.eq('QUEUED'), data.db.query_status.State.eq('RUNNING'))).Do(wait(10)),
assign(data.db.index_name, make_index_name, data.db),
call(delete_person_index, data.db.index_name),
call(create_person_index, data.db.index_name),
assign(data.db.executions, query_person_names, data),
Do(
).While(Or(data.db.query_status.State.eq('QUEUED'), data.db.query_status.State.eq('RUNNING'))).Do( wait(10)),
Repeat(
assign(data.db.next_token, index_person_names, data.db)
).Until(data.db.next_token.eq('None'))
)

31
The hierarchical structure proved to be vital for the
verification of the logical soundness of the design and the
correctness of its implementation.

32
Multi-Dimensional Layering
<<layer>>
Application
<<layer>>
Framework
<<layer>>
Middleware
<<layer>>
OS
<<layer>>
Drivers
<<layer>>
HW
<<layer>>
Presentation
<<layer>>
Application Logic
<<layer>>
Domian Logic
<<layer>>
Data Access
<<layer>>
Data
Individual ComputerOptimal Resource
Utilization of one
Computer
(Process, File System)
Optimal Resource
Utilization of
Distributed System
(Interprocess
Communication,
Distributed File
System)
Programming
Language Specific
Interface
Nework

33
Multi-Dimensional Layering
<<layer>>
Application
<<layer>>
Framework
<<layer>>
Middleware
<<layer>>
OS
<<layer>>
Drivers
<<layer>>
HW
<<layer>>
Presentation
<<layer>>
Application Logic
<<layer>>
Domian Logic
<<layer>>
Data Access
<<layer>>
Data
Individual ComputerOptimal Resource
Utilization of one
Computer
(Process, File System)
Optimal Resource
Utilization of
Distributed System
(Interprocess
Communication,
Distributed File
System)
Programming
Language Specific
Interface
One Region of One
Account of One
Vendor (native SDK)
CAIOS
(Flow Stage, Entity
Store)
BYOS
(Federated Distributed
Entity Graph)
Standard CAIOS X
Library
SCPL
...

34
Optimal Concurrency and Storage Structure Selection
Coroutine (Fiber, Green Thread)
POSIX Thread
OS Process
Container (Lambda, Fargate, Sage Maker)
State Machine (Step Functions)
Flow Stage Specification in
a Mainstream
Programming Language
(eg caios_py_flow DSL)
Basic Primitives (eg scpl)
Intermediate
Representation
Platform-Specific
Representation
(eg Cloud Formation
Stack Template)
Translation
Code Generation
and Optimization
Optimization Models
Operational Statistics: Volume,
Velocity, Variety
Machine Learning
RAM
KV Store, RDBMS
Object Store

35
Why Python is not a perfect cloud programming language, yet?
● Lack of dict vs class interoperability (as in JavaScript):
○ why should I write event[‘Records’][0][‘s3’][‘bucket’][‘name’] rather than event.Records[0].s3.bucket.name?
● Lack of monadic protection of nested dict access:
○ why should I write this ugly piece of code has_problems = 'indexing' not in response or 'errors' not in response.indexing or
response.indexing.errors rather than if response.indexing.errors in {NullObject, True}: ?
● Lack of list/dict vs functions interoperability:
○ cannot pass tabulated functions to algorithms
● Lack of list/dict vs cloud resources interoperability:
○ why I cannot treat Bucket or DynamoDB as a dict thus simplifying local unit testing?
● SQL vs comprehensions vs algorithms confusion
○ don’t push me your sqlalchemy, I do not have any Object-Relational gap
● Lack of uniform way of asynchronous functions invocation
Some of these complains are more about cloud SDK than language, but who we are
talking about ideal cloud programming language.

36
Contact/Follow
asher@blackswan-technologies.comhttps://www.slideshare.net/AsherSterkinhttps://medium.com/@asher.sterkin

Serverless flow programming a new perspective (py web meetup, sept 2nd, 2019) (1)

More Related Content

What's hot (20)

Similar to Serverless flow programming a new perspective (py web meetup, sept 2nd, 2019) (1) (20)

More from Asher Sterkin (14)

Recently uploaded (20)

Serverless flow programming a new perspective (py web meetup, sept 2nd, 2019) (1)