pyjamas22_ generic composite in python.pdf

© BST LABS 2022 ALL RIGHTS RESERVED
Generic Composite
in Python
Rationale behind the pycomposite
Open Source Library
Asher Sterkin
SVP Engineering, GM
Pyjamas Conf 2022

● A 40-year industry veteran specializing in software architecture and technology
● SVP Engineering @ BST LABS, BlackSwan Technologies
● Former VP Technology @ NDS and Distinguished Engineer @ Cisco
● Initiator and chief technology lead of the Cloud AI Operating System project
● Areas of interest:
○ Serverless Cloud Architecture
○ (Strategic) Domain-Driven Design
○ Promise Theory and Semantic SpaceTimes
○ Complex Adaptive Systems
● Contacts:
○ Linkedin: https://www.linkedin.com/in/asher-sterkin-10a1063
○ Twitter: https://www.twitter.com/asterkin
○ Email: asher.sterkin@gmail.com
○ Slideshare: https://www.slideshare.net/AsherSterkin
○ Medium: https://asher-sterkin.medium.com/
Who am I?

● Design Patterns
● The “Composite” Design Pattern
● Need for a Generic One
● Design Patterns Density
● Implementation
● More Advanced Case Study
● Things to Remember
Table of Contents

The Wikipedia definition:
“In software engineering, a software design pattern
is a general, reusable solution to a commonly
occurring problem within a given context in
software design.”
Quality Without a Name (QWAN):
“To seek the timeless way we must first know the
quality without a name. There is a central quality
which is the root criterion of life and spirit in a man,
a town, a building, or a wilderness. This quality is
objective and precise, but it cannot be named.”
Design Patterns
A good collection of Design Patterns in Python:
https://github.com/faif/python-patterns
Fundamental, though a bit
outdated, with examples in C++
and Java
Origin of the idea of Design
Patterns

Whenever we have a tree-like data structure,
be it a file system, cloud stack resources,
code, or anything composed from smaller
parts, the Composite Design Pattern is the
first candidate to consider.
“Composite” Design Pattern

Composite Design Pattern (from Wikipedia)
Traditional OO Implementation

● The Composite class has to overload all the abstract interface methods.
● The implementation of Composite methods would most likely be trivial:
○ Go over all child components and invoke the corresponding methods
○ Could be Depth-First or Breadth-First traversal.
○ In most of cases nobody cares which one.
● Outcome of the Composite operation is an aggregation of the child operations.
● Generally, not a big deal if the number of methods is small
Limitations of the Traditional OO Implementation

I
F
C
services vendors APIs location
acquire
configure
consume
operate
My Context: Cloud Infrastructure From Python Code
service.py
service_conﬁg.py

build_service_parameters()
build_service_template_variables()
build_service_conditions()
build_service_outputs()
build_service_resources()
build_function_permissions()
build_function_resources()
build_function_resource_properties()
build_function_environment()
Core Mechanism: Service Template Builder Composite
ResourceTemplateBuilder
CompositeResourceTemplateBuilder
*
● Every high-level feature (e.g. MySQL Database) has multiple sub-features:
○ Data on rest encryption: yes/no
○ Private network protection: yes/no
○ Interface: (e.g. db_connection vs SQLAlchemy)
○ Allocation: internal/external
○ Access: read-only/read-write
○ User authentication: yes/no
● Each sub-feature might have multiple flavours (e.g. type of encryption)
● Every feature or sub-feature requires one or more cloud resources
● Every service function has to be properly configured to get an access to eachy
feature resources
● Every service function implements a particular API, which might bring its own
resource feature(s)
Service Template
Builder Composite
API Template Builder
Composite
API Resource
Template Builder
Feature Template
Builder Composite
Feature Resource
Template Builder

● The measurement of the amount of design that can
be represented as instances of design patterns
● When applied correctly, higher design pattern
density implies a higher maturity of the design.
● When applied incorrectly, leads to disastrous
over-engineering.
Design Pattern Density

build_service_parameters()
build_service_template_variables()
build_service_conditions()
build_service_outputs()
build_service_resources()
build_function_permissions()
build_function_resources()
build_function_resource_properties()
build_function_environment()
<<null object>>
<<builder>>
ResourceTemplateBuilder
Putting All Patterns Together
*
<<composite>>
<<iterator>>
CompositeResourceTemplateBuilder
<<decorator>>
composite(cls)
<<factory method>>
get_builder(config)
XyzResourceTemplateBuilder
<<specification>>
XyzResourceSpec
*
For each method, define
a default implementation
that returns an empty
structure
<<decorator>>
ResourceTemplateBuilderDecorator

Let’s Go to the Code
@composite
class CompositeResourceTemplateBuilder(ResourceTemplateBuilder):
"""
Implements a Composite Design Pattern to support combining multiple ResourceTemplateBuilders.
By default, works as a PassThroughGate for underlying builder(s).
"""
pass
class ResourceTemplateBuilderDecorator(ResourceTemplateBuilder):
"""
Base class for generic (e.g. Storage) ResourceTemplateBuilders selectively delegating specific tasks to
platform-dependent builders. By default, works as a StopGate for the underlying builder.
:param builder: reference to external platform-specific builder
"""
def __init__(self, builder: ResourceTemplateBuilder):
self._builder = builder

composite class decorator
def composite(cls: type) -> type:
"""
Generic class decorator to create a Composite from the original class.
Notes:
1. the constructor does not make copy, so do not pass generators,
if you plan to invoke more than one operation.
2. it will return always flattened results of any operation.
:param cls: original class
:return: Composite version of original class
"""
attrs = {
n: _make_method(n, f)
for n, f in getmembers(cls, predicate=isfunction)
if not n.startswith("_")
}
attrs["__init__"] = _constructor
composite_cls = type(cls.__name__, cls.__bases__, attrs)
composite_cls.__iter__ = _make_iterator(composite_cls)
return composite_cls

_constructor is simple
def _constructor(self, *parts) -> None:
self._parts = parts

_make_method is where the real stuff is done
def _make_method(name: str, func: callable) -> callable:
> def _make_reduce(m: str, rt: type) -> callable: <-- Python Closure Gotcha
> def _make_foreach(m) -> callable: <-- Python Closure Gotcha
rt_ = signature(func).return_annotation
rt = get_origin(rt_) or rt_ # strip type annotation parameters like tuple[int, ...] if present
return _make_foreach(name) if rt is None else _make_reduce(name, rt)
# Top-down decomposition in action

_make_foreach is boring
def _make_foreach(m) -> callable:
def _foreach_parts(self, *args, **kwargs) -> callable: <-- Python Closure Gotcha
# this is a member function, hence self
# self is iterable, concrete functions invoked depth first
for obj in self:
getattr(obj, m)(*args, **kwargs)
return _foreach_parts

_make_reduce is more interesting
def _make_method(name: str, func: callable) -> callable:
def _make_reduce(m: str, rt: type) -> callable: <-- Python Closure Gotcha
init_value = rt()
combine = add if rt in (int, str, tuple) else always_merger.merge
def _reduce_parts(self, *args, **kwargs):
# this is a member function, hence self
# self is iterable, results come out flattened
return reduce(
lambda acc, obj: combine(acc, getattr(obj, m)(*args, **kwargs)),
self,
init_value
)
return _reduce_parts

_make_iterator is the most tricky
def _make_iterator(cls):
def _iterator(self):
# Simple depth-first composite Iterator
# Recursive version did not work for some mysterious reason
# This one proved to be more reliable
# Credit: https://stackoverflow.com/questions/26145678/implementing-a-depth-first-tree-iterator-in-python
stack = deque(self._parts)
while stack:
# Pop out the first element in the stack
part = stack.popleft()
if cls == type(part): # The same composite exactly
stack.extendleft(reversed(part._parts))
elif isinstance(part, cls) or not isinstance(part, Iterable):
yield part # derived classes presumably have overloads
else: # Iterable
stack.extendleft(reversed(part))
return _iterator

<<builder>>
K8SManifestBuilder
__call__(...)
<<builder>>
<<template method>>
K8SManifestPartBuilder
build_manifest_part(...)
_build_manifest_part_data(...)
__subclasses__()
More Advanced Use Case: K8S Manifest Builder
*
<<composite>>
<<iterator>>
K8SManifestPartCompositeBuilder
<<decorator>>
composite(cls)
<<factory method>>
get_builders()
K8SXYZManifestBuilder
Even with one function, this approach proved to lead to better modularity and
testability

I need ‘em all
@composite
class K8SManifestPartCompositeBuilder(K8SManifestPartBuilder):
pass
def get_builders():
for m in iter_modules(__path__): # ensure all builders are registered
import_module(f'{__package__}.{m.name}')
return K8SManifestPartCompositeBuilder(
*(sb() for sb in K8SManifestPartBuilder.__subclasses__() if 'K8SManifestPartCompositeBuilder' not in sb.__name__)
)

● In this solution I did not implement the Visitor Design Pattern. While it is not hard to
implement, I just did not have any real use case for it. If you do, drop me a line
● The current set of aggregation functions is a bit limited reflecting what I needed at the
moment. Support for a larger set might come in the future.
● This presentation used a more advanced version of code, currently at the separate
branch
Concluding Remarks

● Design Patterns are extremely powerful tools.
● Design Patterns work best in concert (high density).
● Composite Design Pattern is suitable for implementing the on/off logic a complex
feature
● Also, for implementing a variable list of components you do not want to enumerate
explicitly
● Python metaprogramming could make miracles.
● With more power comes more responsibility.
● An unjustified chase after advanced techniques is a sure road to hell.
Things to Remember

Questions?

pyjamas22_ generic composite in python.pdf

More Related Content

Similar to pyjamas22_ generic composite in python.pdf (20)

More from Asher Sterkin (19)

Recently uploaded (20)

pyjamas22_ generic composite in python.pdf