Refactoring toward deeper insight java forum

Disclaimer!!!
.NET not Java
Work In Progress…

Refactoring Toward
Deeper Insight
DDD Findings In Batch Processing
A Case Study
Andreas Brink, factor10

Danica Pension
• Core financial business system.
• ASP.NET / RDBMS / Integrations
• Generations of technologies / architectures.
• Lots of financial business rules.
• Lots of Batch Processing.
• Mission: Taking Control of the Code Base
– DRY, Understandability, Testability, Automation
•  DDD/DM + ORM fits well!

My View of the DDD Toolbox
Model Design & Impl.
Philosophy & Principles • Entity
• Ubiquitous Language • Aggregate
• Model Driven • Value Object
• Declarativity • Respository
• Distilling The Model • Service
• Specifiction
• Breaktrhough
• Side-Effect Free Functions
• … • …

Strategic Design
• Bounded Contexts
• Responsibilty Layers
• …

Domain Model Pattern (DM)
Basic Building Blocks Supple Design
• Entity, • Assertions
• Aggregate • Side-effect free
• Value Object Functions
• Specification • Standalone Classes
• Factory • Closure of Operations

Design Sweet-Spot
• Understanding & Communication
• Testability, Executable Specifications
But, not like the Office Object Model…
• Must scale
• Not one single file on disk

Implementability

Some
Layer…
Repository ORM
Service

• Object Navigation does not scale
 Repositories
• DM does not scale well with composition & coupling
Services
• Problem Solved !?!?

Implementation Mess
Repository

Some ORM
Scenario…

Service

• Less focus on Domain Model
• Services – The Ultimate Loophole
– Touches big parts of the system – horizontally, vertically
– Side Effects
Understandability & Testing Problems
• Decentralized Flow Control & Data Access…
 Global Optimization & Generic Processing hard or impossible
 Performance Problems

Why DDD is Hard
Repository

Some ORM
Scenario…

Service

• Model Design is hard to begin with
– OO Expertise is still quite rare 
• Have to be a Design Patterns / ORM / Architecture Expert
• Fail to get an Easily Consumable & Testable Model
• Same old Procedural Service Soup (+ some entities…)

My DDD Challenge

• Reclaiming the Domain Model
– Easy Reasoning, Consumption & Testing
• REAL Separation of Concerns
– Not just a complex web of objects and method
calls behind superficially simple interfaces
• And with Batch Processing in the mix…

IS THIS POSSIBLE??

Batch in the mix…
• ”ORM is not for Batch, use the right tool…”
• DDD/ORM vs Stored Procedures
• Service Chattiness  Performance Problem
• Batch becomes a domain service in itself
– Business Workflow as a mini program
– Hard to decompose/compose without Service
– I want the business rules in the Domain Model…

Billing Batch – Pseudo Code
foreach (PlanAgreement planAgreement in GetPlanAgreements())
{
Agreement agreement = GetAgreement(planAgreement.Id);

foreach (PlanCategory planCategory in GetPlanCategories(planAgreement.Id))
{
PremiumTable premiumTable = GetPremiumTable(planCategory.Id);

foreach (PlanInsurance planInsurance in GetPlanInsurances(planCategory.Id))
{
Insurance insurance = GetInsurance(planInsurance.InsuranceNumber);
InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber);
AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber);

double premium = CalculatePremium(planAgreement, agreement, planCategory,
premiumTable, planInsurance, insurance);

List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account);

...
...
...
}
}
}

Billing Batch – Levels

Agreement

• Agreement
• PlanAgreement Category
• PlanCategory
• PremiumTable Insurance

• PlanInsuranceHistory
Misc…
• LatestPlanInsurance
• InsuranceHistory • …
• LatestInsurance • …
• InsuranceAccount
• AdviceHistory
• …

Batch Observations
• High input/output entity ratio
– 11 Entity Types as Input – Often Complex
– 2 as Output (1 Create, 1 Update) – Simple
– Simple State Semantics
– Opportunities for Caching
– (Responsibility Layers Analysis…)
• Data is centered around a few central business
keys.
Potential for generalizing / streamlining the batch
processing pipeline??

Billing Batch – Loops Flattened
PlanAgreement planAgreement = null;
Agreement agreement = null;
PlanCategory planCategory = null;
PremiumTable premiumTable = null;

foreach (PlanInsurance planInsurance in GetPlanInsurances()) {
if (planInsurance.PlanAgreement != planAgreement) {
planAgreement = planInsurance.PlanAgreement;
agreement = GetAgreement(planAgreement.Id);
}
if (planInsurance.PlanCategory != planCategory) {
planCategory = planInsurance.PlanCategory;
premiumTable = GetPremiumTable(planCategory.Id);
}
Insurance insurance = GetInsurance(planInsurance.InsuranceNumber);
InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber);
AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber);

double premium = CalculatePremium(planAgreement, agreement, planCategory,
premiumTable, planInsurance, insurance);
List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account);
...
}

Billing Batch – Levels Flattened

Agreement Category Insurance Misc…



• Agreement • PlanCategory • PlanInsuranceHistory • …
• PlanAgreement • PremiumTable • LatestPlanInsurance • …
• InsuranceHistory
• LatestInsurance
• AdviceHistory

Billing Batch – Entity Level Keys
Agreement Category Insurance
Level Level Level
Key Key Key




• Agreement • PlanCategory • PlanInsuranceHistory • …
• PlanAgreement • PremiumTable • LatestPlanInsurance • …
• InsuranceHistory
• LatestInsurance
• AdviceHistory

Billing Batch – Generic Cursor Style

Master Keys Entities
Agreement Level Plan Agreement Agreement Insurance History

Category Level Advice History Premium Table Plan Category

Insurance Level Plan Insurance History … …

• Cursor Semantics
• A Set of Master Keys Drives the Cursor
• Entities Associated with Keys in Master
• Each Row Contains Entities for a Unit-Of-Work

Entity Level Keys

Level Keys Plan Insurance

Agreement Level = 4501 Agreement ID = 4501

Category Level = 78  Category ID = 78

Insurance Level = ”56076” ID = ”56076”

• Map of EntityLevel & Values
– Dictionary<EntityLevel, object>
• Or derived from Entity Properties

The Entity Level Abstraction

class AgreementLevel : EntityLevel {}

class CategoryLevel : EntityLevel {}

class InsuranceLevel : EntityLevel {}

public class PlanAgreement
{
[Level(typeof(AgreementLevel), IdentityType.Full)]
public int Id;
}

Entity Cursor: Master + Entities
void Initialize()
{
var cursor = EntityCursor.For(SessionFactory, MetaData);

// MASTER: IEnumerable<object[]> OR IEnumerable<TEntity>
cursor.Master = GetMyMaster();
cursor.MasterLevels(new AgreementLevel(), new InsuranceLevel());

cursor.Add(Query.For<PlanAgreement>());

// ADD MORE ENTITIES TO THE CURSOR...

while (cursor.MoveNext()) {
var currentPlanAgreement = cursor.Get<PlanAgreement>();

// PROCESS EACH ROW IN THE CURSOR...
}
}

IoC Style + Syntactic Sugar
class MyBatch : BaseBatch
{
PlanAgreement planAgreement;

EntityLevel[] Levels() { return ... }

object[] Master() { return ... }

void Initialize() {
// Query Defintions that are not simple
// Query.For<MyEntity>()
Add<PlanAgreement>()
.Where(pa => pa.Foo != null);

}

void ProcessRow() {
var foo = this.planAgreement.Foo ...
// PROCESS THE ROW...
}
}

Row Processing

Master Keys Agreement Insurance
Key 1: Agreement Id ChunkSize: 2 ChunkSize: 2
Key 2: Insurance No

(1, “InNo-1")
(1, “InNo-2")
(1, “InNo-3")
(2, “InNo-4")
(2, “InNo-5")
(3, “InNo-6")
...
(n, “InNo-n")

Row Processing  Chunked Data Fetch

Key 1: Agreement Id ChunkSize: 2 ChunkSize: 2
Key 2: Insurance No

(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")
(1, “InNo-2") -||- Insurance(1, “InNo-2")
(1, “InNo-3") -||-
(2, “InNo-4") Agreement(2)
Query<Insurance>()
(2, “InNo-5") -||- .Where ...
(3, “InNo-6")
... Query<Agreement>()
(n, “InNo-n") .Where(a => a.Id)
.IsIn(1, 2)

• Entities are fetched in Chunks
• Multiple chunk queries executed in one DB round-trip.
• NHibernate MultiCriteria (or Futures).

Row Processing  Indexing

Key 1: Agreement ChunkSize: 2 ChunkSize: 2
Key 2: Insurance

(1, “InNo-3") Insurance(1, “InNo-3")
... ...
(n, “InNo-n")

• Each entity is indexed with the identifying level key(s).
• Entities in chunks synced with key for current row as the
cursor proceeds forward.

Entity Grouping
class InsuranceHistory : GroupingEntity<Insurance>
{
static readonly Grouping Grouping = Grouping.For<Insurance>()
.By(i => i.AgreementId)
.By(i => i.InsuranceNumber);

public InsuranceHistory(IList<Insurance> values) { ... }
}

Cursor.Add<PlanInsuranceHistory>();

Cursor.Add<PlanInsuranceHistory, PlanInsurance>()
.Where(...); // OK to override filter??

• Groups as a 1st class modeling concept
• Enriching the Domain Model
• “Virtual Aggregate Root” – Model Integrity
• Declarative expression (By, Where, Load)

Complex Grouping – PremiumTable
Row
10-22 23-30 31-45 Interval

0-20 100 120 135
Value
20-40 110 130 150
40-65 130 160 190
Column
Interval

• Rich Model Abstraction
• Complex data structure with lookup semantics
• No natural aggregate root
• Not cacheable in NHibernate session
• Fits well as a GroupingEntity

Querying
Conceptual API:
Cursor.Add(Query entityProvider)

Query.For<PlanInsurance>()
.Where(insurance => insurance.IsActive)
.Load(insurance => insurance.Category)

Query.For<AdviceHistory>()

Query.For(PremiumTable.ByAgreement)
.IndexBy(table => table.TableId)

• Filter per Entity – Cursor “Joins” using Shared Level Keys
• ORM-semantics: Where, Load
• Grouping Entity has query like qualities
• Level Queries are statically defined query using Entity Levels
Keys to construct underlying ORM query (yes, coupling)

Versioning
public class PlanInsurance
{
[Level(typeof(AgreementLevel), IdentityType.Partial)]
public int AgreementId;

[Level(typeof(InsuranceLevel), IdentityType.Partial)]
public string InsuranceNumber;

[VersionLevel(typeof(PlanInsurance), IdentityType.Partial)]
public int Version;
}

• Core to many business domains
• Has its own set of semantics
• Common in Groups – Latest<Insurance> vs InsuranceHistory
• Implemented in different ways in the DB
• Expressed declaratively
• Uniform Query Semantics

What About The Services?
void ProcessRow()
{
...
var premiumService = new PremiumService
{
PlanAgreement = Cursor.Get<PlanAgreement>(),
PlanInsurance = Cursor.Get<PlanInsurance>(),
Insurance = Cursor.Get<Insurance>(),
Insured = Cursor.Get<Person>(),
PriceBaseAmountTable = Cursor.Get<PriceBaseAmountTable>(),
PremiumTable = Cursor.Get<PremiumTable>(),
RiskTable = Cursor.Get<RiskTable>()
};

var premium = premiumService.CalculatePremium(advicePeriod);
...
}
• Service has pure calculation responsibility
• Dependencies are injected by client
• Coupling…? Boilerplate Smell…?

Conclusions
• Data Access Abstraction with Power & Ease of Use
• Declarative & Composable Entity Pipeline
• Minimizes DB Round-trips; Favors Eager Loading
• Repositories Become Redundant
• No More Unconstrained Services – “Calculators” / …???
• Richer Domain Model – Less Supporting Objects, More Domain
Related Objects
• DDD/ORM + Legacy DB == True
• Composite DB Key Thinking Essential to the Solution
• Patching the DB Model with Entity Level Abstraction…
• What’s Next? – Lots of Low Hanging Fruit…
 TOWARDS AN EXECUTABLE ARCHITECTURE…???

What’s Next? – Entity Injection
Cursor.Add<PremiumCalculator>();

void ProcessRow()
{
...
var calculator = Get<PremiumCalculator>();
var premium = calculator.Calculate(advicePeriod);
...
}

• Cursor can inject entity dependencies automatically
• Calculators dependencies can be inferred and added
to cursor automatically
•  ”Calculator” define Cursor Entities Implicitly

What’s Next? – Stateful Calculators?
class PremiumCalculator class PremiumCalculation
{ {
... ...
double CalculatePremium(...) {} double Premium;
... ...
} }

• What if we treated a calculation as a stateful object?
• Calculations become data flows through the system
• Stateful Objects as the Uniform Expression – Simplifies
declarative programming
• Captures Multiple / Intermediate Calculation Results
• Can be bound to a UI
• Additional state in the cursor – UI could add presentation
model/wrapper to the cursor

What’s Next? – Entity Pipeline
class BillingCalculation : EntityPipeline
{
void Initialize() {
Add<PlanAgreement>();
...
}
}

var monthlyBatch = new BillingCalculation();
monthlyBatch.Master = GetMasterForMonthlyBatch();
monthlyBatch.Persist<AdviceCalculation>(ac => ac.Advice).BatchSize(20);
monthlyBatch.Execute();

var singleInstance = new BillingCalculation();
singleInstance.Master = new object[]{ 24, "InNo-1"};
singleInstance.Persist<AdviceCalculation>(ac => ac.Advice);
singleInstance.Execute();

var nextUIPage = new BillingCalculation();
nextUIPage.Add<MyUIModel>();
nextUIPage.Master = GetMasterForNextPage();
myGrid.DataSource = nextUIPage.Select(cursor => cursor.Get<MyUIModel>())

What’s Next? – New Data Providers
• File Processing for Data Imports
– Prototyped batch framework
• Document Based Persistence
– Premium Table for example
• Hybrid Persistence
– Serialized object graphs in SQLServer
• SOA Integrations
– Loosely Coupled Bounded Contexts
• Parallel data fetch
– Multiple DBs / Data Services

What’s Next? – Business Events
• Entity Processing Pipeline seems to be a good
environment for triggering and/or handling
business events based on persistence events.
• Poor man’s Business Events!?!?

What’s Next? – Greenfield
• Search the Core Domain/Application Semantics
– Built-in Versioning from the start e.g.
– Semantic Storage…
• Streamline
– Uniform Expression
– Semantics
– Patterns
• Be Opinionted
– Constraints are Liberating
•  Executable Architecture

Thanks For Listening!!!

Questions?

Refactoring toward deeper insight java forum

More Related Content

Similar to Refactoring toward deeper insight java forum (20)

More from Squeed (7)

Recently uploaded (20)

Refactoring toward deeper insight java forum