SlideShare a Scribd company logo
Data Mesh in Action (MEAP V04) Jacek Majchrzak
install download
https://ebookmeta.com/product/data-mesh-in-action-meap-v04-jacek-
majchrzak/
Download more ebook from https://ebookmeta.com
We believe these products will be a great fit for you. Click
the link to download now, or visit ebookmeta.com
to discover even more!
Data Mesh in Action 1st Edition Jacek Majchrzak Sven
Balnojan Marian Siwiak
https://ebookmeta.com/product/data-mesh-in-action-1st-edition-
jacek-majchrzak-sven-balnojan-marian-siwiak/
Everyday Data Visualization (MEAP V04) Desiree Abbott
https://ebookmeta.com/product/everyday-data-visualization-
meap-v04-desiree-abbott/
Data Mesh Delivering Data Driven Value at Scale 1st
Edition Zhamak Dehghani
https://ebookmeta.com/product/data-mesh-delivering-data-driven-
value-at-scale-1st-edition-zhamak-dehghani/
Unfree Migrant Domestic Work in Arab States 1st Edition
Rhacel Salazar Parreñas
https://ebookmeta.com/product/unfree-migrant-domestic-work-in-
arab-states-1st-edition-rhacel-salazar-parrenas/
Lock Mercy 3 1st Edition Debra Anastasia Anastasia
Debra
https://ebookmeta.com/product/lock-mercy-3-1st-edition-debra-
anastasia-anastasia-debra/
Multiphysics Modeling Using COMSOL 5 and MATLAB 2nd
Edition Pryor Phd
https://ebookmeta.com/product/multiphysics-modeling-using-
comsol-5-and-matlab-2nd-edition-pryor-phd/
Stress Management Through Mind Engineering 1st Edition
R.P. Banerjee
https://ebookmeta.com/product/stress-management-through-mind-
engineering-1st-edition-r-p-banerjee/
Democratizing the Economics Debate Pluralism and
Research Evaluation 1st Edition Carlo D Ippoliti
https://ebookmeta.com/product/democratizing-the-economics-debate-
pluralism-and-research-evaluation-1st-edition-carlo-d-ippoliti/
Off The Path Beyond the Merillian 3 1st Edition
Courtney Leigh
https://ebookmeta.com/product/off-the-path-beyond-the-
merillian-3-1st-edition-courtney-leigh/
Fundamentals of sustainable business : a guide for the
next 100 years Second Edition Matthew Tueth
https://ebookmeta.com/product/fundamentals-of-sustainable-
business-a-guide-for-the-next-100-years-second-edition-matthew-
tueth/
Data Mesh in Action (MEAP V04) Jacek Majchrzak
©Manning Publications Co. To comment go to liveBook
MEAP Edition
Manning Early Access Program
Data Mesh in Action
Version 4
Copyright 2022 Manning Publications
For more information on this and other Manning titles go to
manning.com
©Manning Publications Co. To comment go to liveBook
welcome
Thank you for purchasing the MEAP edition of Data Mesh in Action. We’re excited to already
be able to share some parts of this book with you as it still is being actively written! Since
we’re still writing on it, this is also your chance to leave us some comments to make the final
book even better.
Data Mesh is one of the hottest topics in the data world right now. However, the growing
number of blog articles and discussions in the communities show growing dispersion in Data
Mesh understanding, mainly based on theoretical debates and considerations.
This book is trying to bring the Data Mesh world together by:
1. focusing on applied examples all the way throughout the book, including a long case
study as well as multiple real-world examples,
2. bringing structure to the discussion based on ideas that have been around for a long
time like Socio-Technical Architecture or Domain-Driven Design,
3. and finally by providing an open-minded and flexible approach, with a focus on making
data valuable for every company, without fixed roles, fixed definitions, or dogmas.
We know from (at least) three different perspectives, that the development of Data Mesh
touches a lot of different aspects of the organization and usually starts at very different
places on the data maturity scale.
It involves organizational structure, technology stack, and different levels of architecture. It
can start in any size of a company, be it a company of 20, or a company of 1,000. The data
maturity levels can differ inside the areas of operation of the company.
So, we decided to write a book about how the Data Mesh implementation may look from
different perspectives. We’re trying to achieve this by always highlighting that almost any
part of the Data Mesh, be it the governance policies or the data platform itself are not fixed
concepts but rather points on a spectrum where you can choose the right one for you and
your situation.
To make it even more applied, we created a fictional company, in which we walk from zero to
hero in their data maturity, offering you insight into a set of cohesive, step-by-step
examples.
To take full advantage of this book, you would be directly or indirectly involved in creating,
managing, and utilizing data within your organization. Be that as an architect, a leader in the
data or software space, a developer inside a data-focused team, or on the data-consuming
side of things, like an analyst, a decision-maker, or a machine learner.
We hope you enjoy Data Mesh in Action and that it will occupy an important place on your
bookshelf.
©Manning Publications Co. To comment go to liveBook
We also encourage you to post any questions or comments you have about the content in
the liveBook Discussion forum. We appreciate knowing where we can make improvements
and increase your understanding of the material.
- Marian Siwiak, Jacek Majchrzak, and Sven Balnojan
©Manning Publications Co. To comment go to liveBook
brief contents
Front matter
PART 1: FOUNDATIONS
1 The What and Why of Data Mesh
2 Is Data Mesh Right for You?
3 Kickstart your Data Mesh MVP in a month
PART 2: THE FOUR PRINCIPLES IN PRACTICE
4 Domain ownership
5 Data as a Product
6 Federated Computational Governance
7 The self-serve data platform
PART 3: INFRASTRUCTURE AND TECHNICAL ARCHITECTURE
8 Self-Serve Data Platforms Applications in Comparison
9 Solution architecture design
©Manning Publications Co. To comment go to liveBook
Front matter
preface
All three of us authors experienced at length, at different companies, the old way of “doing
data”, usually through centralized data lakes and data warehouses in combination with a set
of central teams organized inside an “analytics function” to handle this.
The old way basically looked like this:
1. Multiple Decentralized development teams have data that is accessible through its
storage systems like a shared drive, a decentralized database, a REST API, or any
other interface.
2. One or more centralized data teams are tasked with collecting this data into one
monolithic pot. This is either a data lake or a data warehouse.
3. The same set of teams is tasked with transforming this data into something useful.
4. Multiple decentralized analysts, development teams or machine learning teams pick up
that transformed data and transform this data finally into value in the form of reports,
recommendation systems, or anything else they can think of.
All three of us learned the hard way how this concept has its limits, producing a
bottleneck both in terms of technology as well as in terms of the team capacities. We all saw
how companies all around us were struggling to get the flow from data to value to be as
productive as the companies needed it to be. Then the Data Mesh and the ideas behind it
appeared on the horizon.
The Data Mesh is a decentralization paradigm. It decentralizes the ownership of data, its
transformation into information as well as its serving. It aims to increase the value extraction
from data by removing bottlenecks in the data value stream by these means.
The concept of the Data Mesh appeared on the stage in 2019 and since has lit not just
the data world, but the whole technology world on fire. The Data Mesh concept breaks with
the current world of data where data is usually treated as a “by-product” of software
components. It turns the spotlight on the producers of the data and gives them the
responsibility to handle this data just as they would handle their software.
With this, the data mesh takes the same journey software components have taken, with
microservices architectures and with the DevOps movement. It takes the same journey
frontends are currently taking with what is called “microfrontends”. And just as in these
previous examples, we believe that the Data Mesh approach is the right approach to finally
1
©Manning Publications Co. To comment go to liveBook
gain the flexibility to extract value from our data at scale, be that in business intelligence,
machine learning, or any other use case you can think of.
The Data Mesh concept is often referred to as a socio-technical paradigm shift, meaning
the core of it is not about technology but about the alignment of people, processes, and
organization. This significant complexity is why we wrote this book. However, we didn’t just
try to bring together the available theoretical knowledge that is out there about the Data
Mesh; we focused on parts that are, in our experience, critical for successful implementation.
We organized those parts into a digestible resource to help you, the reader, put Data Mesh in
Action!
To guide you through the process, we prepared hands-on examples with a lot of
architecture sketches, describing different technologies, workshop techniques, team
organization forms, and the like. After reading this book you should be able:
1. evaluate if Data Mesh will suit your organization’s business needs,
2. lay the groundwork for Data Mesh development,
3. develop a minimal Data Mesh to start their journey,
4. keep iteratively developing and expanding your Data Mesh.
Don’t expect to find a lot of code in this book though, other than a few JSONs here and
there. That’s because we truly believe the magic is not in the technology, but in the people,
processes, and the organization. But of course, you can expect to find a lot of technology
inside this book in the form of very deep architecture sketches with reference to various
technologies and cloud providers, explanations, and blueprints inspired by multiple real-world
examples.
That said, we don’t believe in a black-and-white implementation of the data mesh idea.
For that reason, we wrote a book that will help you adjust the data mesh idea to your
company by offering a lot of degrees of freedom, shortcuts, and a healthy level of
pragmatism.
To tie together our experience we will use an imaginary company called Messflix LLC,
which resembles a lot of what we’ve seen out there in the data world. This case study will
guide you through all chapters of the book. Toward the end of this front matter, we provide a
brief introduction to Messflix LLC by taking a look at the data mess the company has gotten
itself into.
acknowledgments
First, we would like to express our gratitude to the community engaged with Data Mesh
development. Their discussions, openness about problems and issues helped us broaden our
perspectives and put our particular experiences into the generalized framework you’ll find in
this book.
We owe our thanks to the wonderful people at Manning who made this book possible:
Publisher Marjan Bace, Development Editor Ian Hough, and last, but not least Acquisitions
Editor Andrew Waldron. Without their patience with our ever-evolving view on Data Mesh,
and ability to make us synthetize it into a coherent view, we wouldn’t be able to finish Data
Mesh in Action in a form we could so proudly present to you. We would like also to thank
2
©Manning Publications Co. To comment go to liveBook
marketing, editorial and production teams without whom this book would gather dust in
Manning’s drawer.
A heartfelt thanks also to Michael Jensen and Al Kinker for technical reviews, which
allowed us to further condense and clarify Data Mesh concepts.
We would also like to thank all our reviewers, who trusted us and invested their time in
reading this book, even when no one was sure if it’ll become read-worthy.
about this book
This book is intended to serve two purposes. First, it is intended to collect and organize the
knowledge about the new socio-technological paradigm of Data Mesh. Second, it’s supposed
to help the reader in Data Mesh implementation. From consideration as to whether Data
Mesh is a suitable solution for their organization, to laying the groundwork, to development
of MVP, to implementation of Data Mesh principles, we hope that this book will provide you
with the tools needed to get well on your way on your data mesh journey.
Who should read this book
The most general description of our reader is someone who is involved in extracting value
from data. However, as in modern economy it describes almost anyone, let’s try to see what
benefits will it bring to different audiences.
The first group, is people involved in creating, managing, and utilizing data within a
companies of:
• high socio-technological complexity (e.g., big corporations),
• complex data use cases,
• many and diverse data sources.
This would encompass, but not be limited to, roles like: data architects, data engineers,
software architects, tech leads, senior developers.
The more you feel like these quantifiers apply to your business, the more likely it is that
Data Mesh could be a good solution. This book will help you understand Data Mesh concepts,
whose cooperation you need to secure, and what steps to take in both your organization and
technical environment are needed to move from data mess to Data Mesh.
Beyond that, as the data mesh is a company-wide transformation process, a the book's
content will be directly useful to executive level personnel, like technical C-suite, engineering
directors/ managers, enterprise architects, chief and lead architects, solution/program
owners. This book will help you decide in what extent and priorities should you shift your
company’s data environment into Data Mesh direction, and help you plan the change
management.
How to Navigate this Book
While the book is meant to be read linearly, it is broken into four main parts and allows you
to skip different sections. The first part is a quick and hands-on introduction, the second
explains the four principles of the data mesh in detail, the third tackles the technical side of
things in detail, and the fourth focuses on the complete enterprise journey.
3
©Manning Publications Co. To comment go to liveBook
PART 1: FOUNDATIONS
The goal of the first part of the book is to get you familiarized with the data mesh paradigm
as quickly as possible. To do so we first go through the basics of the data mesh and then get
our hands dirty by building our first data mesh within a month.
Chapter 1: The What and Why of Data Mesh
This chapter gives the overview needed to put the rest of the book into the proper context,
including why you might want to consider following the data mesh mindset shift as well as a
short explanation of the four key principles we dive deeper into in the next part.
Chapter 2: Is Data Mesh Right for You?
This chapter provides you with the context of the Data Mesh implementation and different
drivers to consider when deciding on the transformation. It helps you decide whether you
want to start the journey now, and at which point you are on the data maturity scale. This
helps you to match your Data Mesh journey to your particular situation.
Chapter 3: Kickstart your Data Mesh MVP in a Month
This chapter is a hands-on example of how to go about building an MVP. The Messflix MVP
focuses a lot on the organizational challenges and stays very light on the technology side of
things, which an MVP should. These will be picked up later. The chapter provides you with
tools like stakeholder mappings and FAIR principles to get you started.
PART 2: THE FOUR PRINCIPLES IN PRACTICE
The goal of the second part of the book is to provide you with the tools to tackle the four
principles of the data mesh for advancing your data mesh beyond the first month.
Chapter 4: Domain Ownership
This chapter is all about domains and business capabilities and how data can find its owner
inside a company. It provides you with a lot of workshop techniques like domain storytelling
applied to Messflix example and the Hitchcock Movie Maker and the likes.
Chapter 5: Domain Data as a Product
Data is often treated as a “by-product”. The topic of this chapter is about changing to a
product perspective called “data as a product”. It provides examples of data products from
Messflix LLC and explains concepts like the data product canvas and data ports in detail.
Chapter 6: Federated Computational Governance
This chapter tackles the concept of data governance in the data mesh context. Inside data
meshes, it’s called federated computational governance because both a federated nature of
governance and an automated execution are needed to unfold the data mesh character. This
chapter contains a discussion of centralized vs. decentralized aspects, hands-on examples
from Messflix LLC, and a guide for setting up a governance team.
Chapter 7: Self-Serve Data Platform
The last chapter on data mesh principles covers the platform, the enabling technology that
makes the data mesh work. It works through three iterations on our data platform for
4
©Manning Publications Co. To comment go to liveBook
Messflix LLC and explains important concepts like “Platform Thinking” along with these
examples.
PART 3: INFRASTRUCTURE AND TECHNICAL ARCHITECTURE
The third part focuses on all things technical. We break out of the Messflix example to
highlight different architectures and discuss multiple different options for moving from your
existing structure to a data mesh.
Chapter 8: Self-Serve Data Platforms Applications in Comparison
This chapter explains different blueprints for data mesh platforms that fit different cloud
providers as well as different sizes of companies.
Chapter 9: Technical architecture design
In this chapter, we focus on the migration from your existing system to various different
kinds of architectures step by step and component by component. We talk about data lakes,
data warehouses, REST APIs, and more.
How to Use this Book
We don’t just want to present yet another theory of Data Mesh. This book is more of a
structured, collective diary of actions leading to Data Mesh development in different
environments. The emphasis is on “actions leading to”. We’ve arrived at Data Mesh after the
long and often painful journey through multiple other solutions. Over the years, we’d been
testing, researching, discussing, and, last but not least, failing a lot in the process. In this
book, we share with you the summary of “I wish someone had told me earlier” insights. We
hope you will be able to immediately put the information you’ll get out of it, well, in Action.
Depending on your goal, there are a few focal points you could set while reading this
book and dive deeper into.
If your interest is purely informational, and you might want to explain the concepts to
your team, your management, your company, we recommend you put a lot of focus onto
chapters 1 and 2, for the quick overview as well as the MVP presented in chapter 4. If, in
addition, you will read through chapter 9 for a deeper dive into the reasons for this paradigm
shift and a lighter look into Part 2, you will be well equipped to explain the data mesh
paradigm to someone else.
If you want to launch a larger initiative inside your company, you’ll need to be
convincing. In that case, we recommend you take a deep dive into the complete chapter 9
and follow with caution chapter 3, offering the insight into the question of whether you
should start this journey at all. Chapter 4, presenting the full-scale Data Mesh MVP
development and chapter 2 offering a quick glance into a lightweight application of Data
Mesh principles will allow you to balance the big picture view with notes on requirements of
quick implementation and getting results fast. All together this material should equip you
with enough convincing material to get top-level buy-in.
If you’re interested in the technical side of things like automated governance and the
self-serve platform, then chapters 5 to 8 will provide you with a lot of interesting content to
dig through.
5
©Manning Publications Co. To comment go to liveBook
If you work inside a development team, we’d particularly recommend you to turn your
attention to chapter 4, as it explains exactly what is broken in the current mode of thinking,
and it should also help you advance your ways of working without ever touching the data
mesh concept. Additionally we recommend Chapter 8 as it explains possible architecture
alternatives for serving data from a development teams’ point of view.
If you want to advance the way you work inside your data team, then you could put a
focus on chapters 3 & 4 to deeply understand where your current troubles may stem from.
You could also focus on chapter 6 to understand what “platform thinking” in a data context
means. Both could help you advance your ways of working without actually going full data
mesh inside the company.
We’re sure there are many more reasons for you to open up this book, these are simply a
few possible ways you could go about putting this book into use.
The Messflix Case Study
To help you conceptualize the practical aspects of putting Data Mesh in action, we combined
our experiences and merged them into a single Data Mesh journey of Messflix LLC.
Messflix LLC, a movie & TV-show streaming platform, just hit a wall. A data wall. They
have all the data in the world but complain about not even being able to build a proper
recommendation system for their movies and shows. The competition seems to be able to
get it done, in fact, the competition is famous for being the first-movers in a lot of
technology sectors.
Other companies in equally complex industries seem to be able to put their data to work.
Messflix LLC does work with data, analysts are able to get some insights from it, but the
organization doesn’t feel like they can call themselves “data-driven”.
The data science trial runs seem to all end in “pretty prototypes” with no clear business
value. The data scientists tell their managers that it’s because the “product team just doesn’t
want to put these great prototypes on the roadmap” or in another instance “because the
data from the source is way too messy and inconsistent”.
In short, Messflix LLC hopefully sounds like your average business, but for some reason
doesn’t feel like it’s able to let the right data flow to the right use cases.
The data landscape, just as the technology landscape grew organically over time and
became quite complex.
The two key technology components of Messflix LLC are the “Messflix Streaming
Platform” and the so-called “Hitchcock Movie Maker”. Where the streaming platform does
just what it says, enabling subscribers to watch shows and movies, and the latter is a set of
tools helping the movie production teams to choose good movie topics, themes, and content.
Additionally, Messflix LLC has a data lake with an analytics platform on top of it taking
data from everywhere.
A few teams manage these components, The teams “Orange” and “White” together
operate a few of the Hitchcock Movie Maker tools. Team “Green” is all about the
subscriptions, the log-in processes, etc., and team Yellow is responsible for getting things on
the screen inside the streaming platform.
Figure I depicts a very rough architecture sketch of a few of these components before we
discuss quickly, how data is currently handled at Messflix LLC.
6
©Manning Publications Co. To comment go to liveBook
Figure I Selective architecture sketch of the Messflix main software components. Clearly visible is a plenitude
of data sources Data Team under Data Team responsibility. Depicted are Messflix internal systems (blue),
teams (red squares with little team icon above), and users (black silhouettes).
7
©Manning Publications Co. To comment go to liveBook
The data team gets data into the data warehouse from a few different places like cost
statements from the hitchcock movie makers and subscriptions from the subscriptions
service.
It also gets streaming data and subscription profiles into the data lake.
Then the data team does some number crunching to transform this data and transforms
this data into information for fraud analysis and business decisions.
Finally, this information is then used by decentralized units to make business decisions
and other use cases. This currently is a very centralized workflow as described above. The
data team “sits in the middle”.
No matter where you’re coming from and where you want to go, you will find yourself
somewhere along the Messflix journey. So let us take one final look at the complete journey
our company is going through.
No data journey is a simple straight line. Likewise, we don’t like to pretend the Messflix
journey is a simple linear progression of a series of steps. So while we will go back and forth
a bit between the chapters to show different approaches and ways to make the data mesh
match any company, there is a green line.
You can follow the main thread for the Messflix LLC along with Chapters 2-6 and 9. Table
I. below gives you an overview of the stages of the company, we highlight two dimensions
alongside the journey. The first is the number of organizational units and teams affected. The
second is the levels of responsibilities that are decentralized.
The core of the data mesh paradigm shift is the decentralization of the responsibility for
data. But “responsibility for data” today is practically split into multiple parts, all of which
need to be decentralized. Thus we highlight all four kinds below. They each correspond to
one of the principles we will discuss in Part 2.
8
©Manning Publications Co. To comment go to liveBook
Table I. Messflix LLC Journey
Chapter 4 Chapter 5 Chapter 6 Chapter 7 Chapter 8
Affected Teams 2
(two data
producing
teams)
Few
collaborators
across the
company
2+ 2+ 2+ 3-10
(a dedicated
platform team +
more producing &
consuming
teams)
Levels of
Responsibility
A small bit of
everything.
true
ownership
through
knowing the
actual (data)
domains.
responsibility
for serving
data
customers.
responsibility for
security,
governance,
etc., in short, the
needs of the
whole company.
responsibility for
infrastructure,
technical
components.
About the authors
Jacek Majchrzak is a lead architect in the area of drug discovery where he implements the
Data Mesh idea. Jacek is a workshop facilitator with an experience in moderating many
workshop techniques like, Event Storming, Domain Storytelling, Business Capability
Modelling, Bounded Context Canvas and many other. He designed his own technique that
helps discover and define Data Products in a domain oriented way, he called this exercise a
Data Bazaar. He is using an aforementioned techniques, to help people make best possible
decisions about their technology and strategy. His areas of expertise are Domain-Driven
Design, Event-Driven Architecture, solution architecture, socio-technical system design,
architecture governance and strategy. He thinks that understanding business domain and
needs are the key to a succesful architecture. He blogs on jacekmajchrzak.com. Privately,
he's a husband and a father of two, motorcyclist, addicted to sports, especially aquatic.
Dr. Sven Balnojan, Sven Balnojan is a data technologist and product person focused on
helping the world extract more value from the exponentially growing amount of data. He’s
passionate about all things data, machine learning, AI, business intelligence and many
related fields. His endeavors include managing internal data teams and managing the
transition from service-oriented team to platform-team as well as getting his hands dirty as
data developer be that in the field of machine learning, data engineering or Data DevOps.
Sven holds a phD in mathematics with a thesis in the field of singularity theory. He is the
author of an opinionated newsletter “Three Data Point Thursday”. Additionally he blogs on
datacisions.com and appears in talks here and there over the internet.
9
©Manning Publications Co. To comment go to liveBook
Dr. Marian Siwiak is a jack of all data related trades and master of their integration with
business operations. As a Chief Data Science Officer of Cognition Shared Solutions LLC, he’s
a strategic-level data use consultant and the designer of Trilayer Business Proces Analysis™
framework connecting process execution, data environment and risk management. He has a
track record of deliveing multimillion IT, scientific and technical projects covering various
areas, from life sciences to robotics. He lectured data management at various business
universities. His team first published a global spread model of COVID-19. Privately he’s a
husband, father of three, skier, sailor, and sci-fi writer.
10
©Manning Publications Co. To comment go to liveBook
1
The What and Why of Data Mesh
This chapter covers
• What is a “Data Mesh”? Our definition of a Data Mesh
• What are the key concepts of the Data Mesh paradigm?
• Why is it a “socio-technical” paradigm shift?
• What are the advantages of the Data Mesh?
• What are the possible challenges of a Data Mesh implementation?
The Data Mesh is a decentralization paradigm. It decentralizes the ownership for data, its
transformation into information as well as its serving. It aims to increase the value extraction
from data by removing bottlenecks in the data value stream by these means.
The Data Mesh paradigm is disrupting the data space. Large and small companies are
racing to showcase “their Data Mesh-like journey” all over the internet. It’s becoming the new
“thing” to try out for any company that wants to extract more value from their data. This book
describes the Data Mesh paradigm as a socio-technical architecture with an emphasis on the
socio. The main focus is on people, processes, and organization, not technology. Data Meshes
can, but don’t have to, be implemented using the same technologies most current data systems
run on.
But as a topic of ongoing debate and only slowly emerging best practices and standards,
we found the need for an in-depth book that covers both the key principles that make Data
Meshes work and examples and variations needed to adapt this to any company.
This book is designed to do just that: help you begin your own Data Mesh journey. We will
provide you with conceptual tools, including but not limited to the key Data Mesh principles
and boots-on-the-ground insights and examples. And as it’s designed to be of help, it’s not
meant to be a comprehensive theoretical study, but offers readers the practical guidance
necessary to get set up.
11
©Manning Publications Co. To comment go to liveBook
To start off, we will look at the core ideas of the Data Mesh as well as the benefits and the
challenges associated with it. We’ll also give you our definition of a Data Mesh focused on
outcomes and practicality.
1.1 Data Mesh 101
The Data Mesh paradigm in this book is all about decentralizing responsibility.
For instance, the development team for the “Customer Registration Component” of a
company also creates a dataset for analytical purposes of “registered customers”. They ensure
it is in an easy-to-digest format by transforming the data, e.g., to a CSV file, and serving it
the way the consumers would like it, e.g., on a central file-sharing system.
But this deceptively simple definition has a lot of implications because, in most companies,
data is handled as a “by-product”. It is usually turned into value only after being put as a by-
product into some storage, then pulled into a central technology by a centralized data team,
and then decentralized actors again pick up the data. Be that as an analyst in the marketing
department, inside a recommendation system used in a marketing campaign, or displayed
inside the frontend.
Figure 1.1 depicts a common form of data architecture, both organizational and technical.
It hopefully also shows its pitfalls.
Figure 1.1 Decentralized data emission and central transformation causes problems for the users due to e.g.
unclear ownership and responsibility for data and its quality."
We can see here two levels of centralization:
12
©Manning Publications Co. To comment go to liveBook
• the centralized technology in the form of storage and the usual data engineering, data
science machinery,
• the organizational centralization of the data team.
Since the development team considers the data a “by-product”, the ownership is implicitly
assigned to the data team. But such central teams can usually not keep up with the business
domain knowledge inside multiple data source domains. The developer responsible for
Customer Registrations would only need to know the language and the updates inside that
component and the associated business. But the central data team will have to have the same
understanding of a domain multiplied by the number of domains. Such overload makes it
unlikely that the central team will understand even a single domain to the degree the
responsible development team does. As a result, the data team cannot tell whether the data
is correct, what it actually means, or what specific metrics might mean.
The Data Mesh paradigm shift calls for decentralization of the responsibility for data - to
consider it an actual product. The situation depicted in figure 1.1 can turn into a data mesh if,
e.g., the development team provides the data product straight to the analysts through some
standardized data port. It could be something as simple as a plain CSV file hosted in the
appropriate cloud storage spot, easy to access for the analyst. Take a look at figure 1.2 to see
this shift in action.
Figure 1.2 Decentralized data transformation makes data consumers happy by offering simple access to well
described data.
A platform team could help provide a simple technology as a service to be used by
development teams to deploy such data products, including the data ports, quickly.
Data producers focus on developing data products, which, together with data consumers,
start to form connections and compose a network. We call such a network the mesh, where
the individual nodes are data products and consumers.
13
©Manning Publications Co. To comment go to liveBook
Even in our small example, we observe a significant operational paradigm change. It
encompasses both a shift in the ownership responsibility (from a central data team to the
development teams) and the technical challenge of making the new setup work.
Introducing changes in the operational paradigm will result in ripples affecting many areas
of your business. To stop it from becoming chaos, we need guiding principles. We will quickly
introduce them later in this chapter and examine them in detail in chapters 4 to 7.
Before that, you must understand our definition of the “Data Mesh” and its non-technical
aspects.
DEFINITION OF A DATA MESH
Zhamak Dehghani made an incredible effort in curating the idea of the Data Mesh starting in
20191. She provided us with all its critical elements and introduced a structured approach to
the previously discussed paradigm shift.
Since the first introduction of the “Data Mesh” approach by Zhamak, many “Data Mesh”-
inspired business-derived and theoretical examples have appeared. A lot of this content might
not perfectly fit into the initial description of the Data Mesh framework. A lot of businesses
seemed somewhat unsure about what exactly conforms to the definition of a Data Mesh and
what doesn’t.
In our work, as well as in this book, we opt for solutions that are first and foremost practical
(hence the title “Data Mesh in Action”). Therefore, the Data Mesh definition we coined aims to
be broad, functional, and emphasize decentralized efforts to maximize the value derived from
data:
Definition: Data Mesh
The Data Mesh is a decentralization paradigm. It decentralizes the ownership for data, the transformation of data
into information, and data serving.
It aims to increase the value extraction from data by removing bottlenecks in the data value stream.
The Data Mesh paradigm is guided by four principles2, helping to make the data operations efficient at scale:
domain ownership, domain data as a product, federated computational governance, and self-serve data platform.
Data Mesh implementations may differ in scope and degree of utilization of different principles.
The goal of implementing a Data Mesh is to extract more value from the company's data
assets. That is also the reason we keep this definition so lightweight and inclusive in relation
to the level at which each of the principles are followed. The following non-technical use case
of a Data Mesh will hopefully explain what we mean by that.
1.2 Why the Data Mesh?
We see three main reasons why the data world is in need of decentralization in the form of the
Data Mesh:
1 https://martinfowler.com/articles/data-monolith-to-mesh.html
2 The four Data Mesh guiding principles are described in section 1.4 of this chapter and in chapters 4 to 7.
14
©Manning Publications Co. To comment go to liveBook
• With the proliferation of data sources and data consumers, a central team in between
creates an organizational bottleneck.
• With multiple data emitting and consuming technologies, a central monolithic data
storage in between creates a technological bottleneck, which loses a lot of the
information in this step in the middle.
• Both data quality and data ownership are only implicitly assigned, which causes
confusion and a lack of control over both.
Over the past thirty years, most data architectures were designed to integrate multiple
data sources, i.e., central data teams merged data from all kinds of source systems and
provided harmonized sets to users, who in turn tried to use it to drive business value.
Yet, for over a decade now, the problem of big-data hangovers has plagued companies of
all sizes. The data environments struggle with the scalability of the solutions, completeness of
the data, accessibility issues, and the likes. We bet you can tell that in your company you’re
also fighting a struggle to extract value from data, right? Some things simply seem to not work
out. Dozens of reports & dashboards seem to be of no use compared to the costs of creating
& maintaining them. A bunch of data science projects seems to stay stuck in the “prototype”
phase, and the running data-intensive applications probably are facing a bunch of data-related
problems. At least it should seem that way compared to the effort it takes to get a software
component to run. Just not yet right.
One of the reasons for the scalability problem is the proliferation of data sources and data
consumers. An obvious bottleneck emerges when one central team manages and owns data
along its whole journey: from ingestion through transformation and harmonization to serving
it to all potential users. Splitting the team along the data pipe does not help much either. When
engineers working on data ingestion change anything, they need to inform the group
responsible for transformation. Otherwise, the upstream systems may fail, or worse will
process the data incorrectly. Required close collaboration between the engineers leads to the
tight coupling of all data-related systems.
The other problem arises from the monolithic nature of data platforms, such as warehouses
and lakes. As a result, they often lack the diversity to reflect the reality encoded in data derived
from sources and domain-specific structures. Moreover, enforced flattening of data structures
reduces the ability to generate valuable insights from the collected data, as crucial domain-
specific knowledge gets lost in these centralized platforms. We could observe it in one of the
projects we worked on. The car parts manufacturing company was buying data related to
failures of different parts. Even though the provider had information on the part provenance,
i.e., the model the part was installed in, the buyer had no data models allowing it to store this
information. As a result, components were analyzed separately, hampering R&D’s attempts to
understand the big picture better.
Two more interwoven factors exacerbate the problems described above. One is unclear
data ownership structure; the other is the responsibility for data quality. Data traveling through
different specialized teams loses its connection to its business meaning, developers of
centralized data processing systems and applications can’t and won’t fully understand the data
content. In contrast, data quality cannot be assessed in disconnection from its meaning.
Similar problems were recognized in other areas of software engineering and resulted in
the emergence (and success!) of domain-driven development and microservices. Application
15
©Manning Publications Co. To comment go to liveBook
of similar thinking (i.e., focus on data ownership and shared tooling) to data engineering led
to the development of the idea of the Data Mesh.
1.2.1 Alternatives
There are two main alternative models to Data Mesh’s decentralization of responsibility for
data. We discuss their setup in more detail in chapter 6, here providing a quick overview.
The first option is the centralization of both people and technology. This is the default setup
for any start-up. And it’s a very decent default option, just like the monolith is a decent default
option for any software component. In the beginning, the costs of decentralization outweigh
its benefits. The benefits brought in by working closely together inside one data team, having
just one technology to use, makes things a lot easier.
INSIGHT: CENTRALIZATION IS A SENSIBLE DEFAULT OPTION Centralized data work both
organizational and technical can make sense as a default option. Decentralization does carry costs, and
centralization can mitigate those. That does imply though, that the value derived from centralized and
decentralized data is roughly equal.
The second option is the idea of splitting up the work not by business domains as the Data
Mesh suggests, but by technology. This usually results in one core data engineering team
responsible mostly for ingesting data and provisioning a data storage infrastructure and
multiple other teams, analytics teams, data science teams, analysts you name it. These pick
up the raw data and turn it into something meaningful down the road. You might first centralize
your data system and then layer up with this option to increase the flow.
There is nothing wrong with these two options. They might be reasonable default options,
but both options fail to align with the value creation, which is deeply tied to business domains.
None of the options are able to address sudden changes in just one business domain. As with
microservices, where the strength is the ability to quickly extract value from one specific
service by scaling it up all by itself, the data mesh is able to scale up value extraction in just
one domain. All other options need to scale up everything to scale up value extraction in just
one domain.
So in one way or another, both of these options will hit a wall at some point in time, in
which it will feel like adding the next data source, or adding the next data science project will
feel extremely complex & costly compared to the first ones. That’s the point where you want
to switch to a Data Mesh.
1.2.2 Data Warehouses & Data Lakes Inside the Data Mesh
There is a misconception about the Data Mesh. It is sometimes perceived as an exclusive
alternative to the central data lake or the central data warehouse.
But that does not take into account what the Data Mesh is, a combination of two things,
technology & organization. The Data Mesh is an alternative to having one centralized data unit
taking care of the data inside a central data storage.
16
That still leaves the option to have central data storage and decentralized units working &
owning the data. Indeed that is a common implementation in companies that don’t need
complete flexibility on the data producers’ side.
It also is a common approach to keep data lakes & data warehouses inside a business
intelligence or data science team. The data lakes & data warehouses then become a node
inside the Data Mesh.
Figure 1.3 Data Mesh can still use Data Lakes, e.g. a Data Science team building data products may use Data
Lakes as nodes within the Data Mesh.
Data Meshes make heavy use of both data lakes and data warehouses in various formats,
Data Meshes in general do not try to focus on any specific technology. We discuss this
dichotomy a bit deeper in section 1.6. For now, let’s try to keep spirits high and focus on the
benefits of Data Mesh.
1.2.3 Data Mesh benefits
Let’s analyze the potential lying in Data Mesh implementation from two different perspectives:
through the eyes of the business decision-makers and the technologist.
THE BUSINESS PERSPECTIVE
From the business perspective, data itself is of little value. Worse, it means incurred costs!
Sounds like heresy? To understand that statement, and if needed, convey it to your business
partners, you need to understand the different levels at which people can cognize reality. A
©Manning Publications Co. To comment go to liveBook
17
©Manning Publications Co. To comment go to liveBook
good approximation of this phenomenon is the so-called DIKW pyramid3, derived from 1934’s
“The Rock” play by T.S. Eliot. It represents data, information, knowledge, and wisdom as a
hierarchical structure, where each next element can be derived from the former.
The data in this context is just a set of values (storing of which costs money!). To derive
the value from it, one needs to build up the context allowing for informed decision-making.
The Data Mesh improves the robustness of the whole pyramid.
As we mentioned, having the raw data is of no use for decision-makers. One can argue
that they can download it to their laptops and analyze it themselves. It is true! It has, however,
two underlying assumptions:
1. To download the data, it needs to be accessible.
2. To ensure the value of any performed analysis, data needs to be as complete as
possible.
To address the first assumption: we mentioned already and will say repeatedly, Data Mesh
is very much focused on making data accessible. Not only accessible but findable,
interoperable, and reusable as well! It’s embedded in one of the four Data Mesh principles:
Data as a Product is all about making sure data is there for the taking.
Completeness of the data is another issue where Data Mesh shines. Unlike most Data
Warehouse or Data Lake architectures, Data Products and their data models are not developed
by IT specialists in disconnection to business. Instead, it’s a joint effort, ensuring the data
presented outside the domain is sufficient to derive meaningful conclusions.
Data Mesh also helps add value to elements higher in the hierarchy. The teams
transforming data into information, knowledge, and wisdom, which the business environment
likes to call “insight,” gain instant access to multiple interoperable data sources.
Of course, in theory, it’s possible to make it happen in Data Lake as well. However, the
reality of as a single team managing technical aspects of the environment, as well as data
access and transfer rights, in our experience always makes it infeasible. And if required bits of
data are stored in two different Data Lakes (or four, which is not that unusual), getting them
all to work together is next to impossible.
In short, having access to read-optimized Data Products enables quick prototyping of new
analytical methods and opens a path for the rapid development of new business capabilities.
THE TECHNOLOGY PERSPECTIVE
The main benefit from the technological perspective is maintaining the speed of development
with the organization’s growth, as we mentioned previously, required to keep bringing business
benefits from the data. Data Mesh is meant to address the shortcomings of other data
architectures, like Data Warehouse or Data Lake, by decentralizing data production and
governance. Those architectures introduce a bottleneck — a central team responsible for
harmonizing all the data for the whole of your company and making it ready for consumption.
A single team cannot scale to accommodate the varied data needs of a growing organization.
Both the technology as well as the team knowledge quickly becomes a scale problem.
Eventually, more time is spent on maintenance, and new projects become more and more
delayed.
3 https://en.wikipedia.org/wiki/DIKW_pyramid
18
©Manning Publications Co. To comment go to liveBook
The other benefit of Data Mesh is the clarity of data ownership right from the point of its
creation. It flattens the data management structure leaving just a thin layer of a Federated
Governance Team. And even that team’s activities are limited to agreeing on standards within
autonomous domains.
The speedup of development also comes from empowering the implementation teams.
Since producing and maintaining the Data Products lies on their shoulders, the speed of change
is not limited by a single central integrations team’s backlog of tasks. It means that both the
evolution and fixes to the Data Product happen quicker. It is especially prominent in case of
any bug fixing and downtimes. Furthermore, the Data Product owning team is better equipped
to react faster because there is no context switching, as in the one central team’s case.
The other factor worth mentioning is data environment stability. With Data Products
offering access to contracted versions of its datasets, pipelines built on them are much more
robust and require much less maintenance.
1.3 Use Case: The Snow Shoveling Business
Candace operates a snow shoveling business. She is a proud business woman who started the
business shoveling snow every winter herself. After a couple of years, she scaled up her
operations. She focused on logistics, billing, setting the prices across the business, etc., and
hired three employees.
Adam, who shovels the houses at Pine Road; Eve, who does the houses on Oak Street; and
Bob who is responsible for ordering new shovels, as they seem to break all the time.
Adam and Eve are responsible both for shoveling as well as bringing in new clients on their
respective streets. After all, they spend quite some time around people there, shoveling snow
each winter!
But, Candace ain’t happy.
19
©Manning Publications Co. To comment go to liveBook
Figure 1.4 Candace’s (C) Snow Shoveling Business. Adam (A) and Eve (E) do their work, while Bob (B) creates
a stack of inventory, freezing the capital.
The previous year, Candace asked Adam & Eve to write down their working times and the
number of houses cleared. She put that data in a fancy Excel file to do some calculations and
set the prices. Figure 1.4 displays that data flow.
20
Figure 1.5 Centralized Data Flow from Adam & Eve to Candace for Decisions
This way, one could say, she turned the data received from Adam & Eve into information
in her Excel, and then further into decisions yielding the business value.
Additionally, she asked Adam & Eve to provide her with a rough guesstimation on how
many shovels they will need. Figure 1.5 shows the data flow to Bob.
Figure 1.6 Centralized Data Flow from Adam & Eve through Candace to Bob
We could say,that Adam and Eve provided raw data, Candace aggregated it into information
and handed it over to Bob to turn the data into decisions & business value.
Notice how you again can see a “pipeline”, a sequence of steps that turns data into value.
The pipeline has two steps for setting prices, and three for the shovel procurement with Bob.
©Manning Publications Co. To comment go to liveBook
21
But Candace is not happy with the situation.
Profits are not too good and the shovel procurement seems to always be off, sometimes
inventory seems to stack up, sometimes Adam and Eve need to delay their work, as they run
out of shovels.
This year, after reading a book called “Data Mesh in Action”, she decided to experiment.
She decided to “build the foundation for a Data Mesh” described in chapter 3 there. She used
knowledge form chapter 4 to define domain boundaries, and give the ownership of data to her
main shoveling domain owners, namely Adam and Eve. That means:
1. She stops collecting the times, and instead she asks Adam & Eve to record their times
themselves in any way they see fit
2. Adam and Eve will set the pricing for their streets
Then something interesting happened. Eve decided to to increase prices on Oak Street,
whereas Adam decreased the prices on Pine Road. Turns out, they simply knew more about
their neighborhoods, than Candace did. On Oak Street, the houses had long driveways, so
Adam needed to charge more per driveway. On Pine Road, a local kid was shoveling cheaper,
so to be competitive Eve needed to get her price down. Figure 1.6 shows the decentralized
data flow now taking place.
Figure 1.7 Decentralized Data Flow inside Adams and Eve domain
If we look at the flow of data now, we could say it stays completely with Adam and Eve
respectively, from data to information and finally into the decision - the pricing model.
As profits increased, Candace decided to take a step further. In the second year of the
experiment, Candace decided to tackle the procurement problem. To do that, she decided to
ask Adam & Eve to directly inform Bob about the shovels they will likely need.
©Manning Publications Co. To comment go to liveBook
22
To avoid confusion and to-and-fro email, they decided to create a spreadsheet, Adam and
Eve updated regularly, if e.g. current conditions or weather forecast changed their estimations.
Figure 1.7 explains the data flow serving data to Bob.
Figure 1.8 Decentralized Data Provisioning in Adams & Eves Domains served to Bob for decision-making.
In our data flow, Adam & Eve now keep both Data and Information in their domain and try
to supply Bob with a proper set of information to make the procurement decisions. Not just
the raw data they were asked to collect.
Bob seems much happier. When asked by Candace why is so, he tells her that Eve realized
Bob also needs to know how often shovels break. Eve seems to simply break them more often
because of the longer driveways.
An unexpected business benefit of shortening the data flow emerged. Adam and Eve usually
presented Candace with pessimistic approximations, not wanting to be left without shovels.
Sometimes, however, being afraid of her reaction to too many broken shovels they hoped for
the best and presented optimistic ones. Candace usually added an extra “safety margin” before
sending the order to Bob, but sometimes she got worried about spending and took a risk of
lowering the received estimation.
Bob, knowing that the numbers he receives are unreliable, often tried to adjust the order
based on his gut feeling (and telling Candace, that e.g. required numbers of shovels was not
available on the market. To keep the company running, he had to create a buffer of shovels,
decreasing the company’s financial liquidity by freezing the capital and increasing warehousing
costs. After he got access to Adam and Eve’s direct forecasts, he was able to procure just
enough shovels, ending the year with barely any inventory left and without running out of
them in a meantime.
©Manning Publications Co. To comment go to liveBook
23
©Manning Publications Co. To comment go to liveBook
For Candace, this resulted in a nice profit thanks to reduced costs in year two, as well as
three happy employees.
If you compare that to the data pipe above, you should notice that again, this “pipeline” ,
the flow of data from one unit, in this case, Adam & Eve, to Candace and further to Bob, is the
source of all problems. All that Candace did was break up that pipe and “package it into one”
or at most two pieces. That is decentralization at work.
And that is really all there is to it, by breaking up this pipe in certain situations, you will
end up with better outcomes, because the decentralized parts, like Adam & Eve in our case,
simply can turn that data into value better, whereas the pipe could not.
Probably next year she’ll want to introduce some form of governance, e.g. introducing rules
on the frequency of updates, or ensuring the security of her spreadsheets, so the kid from Pine
Street doesn’t mess up with their data. Maybe she’ll hire someone to create a website, where
people will be able to book their services online - to do that she’d have to develop data products
with Adam and Eve’s working times and connect them with the new system using some sort
of the platform. The future is full of potential when you have operational profit, isn’t it?
Next, we will show you the four principles of Data Mesh. We consider them the cornerstones
of its implementation. But as we mentioned previously, our Data Mesh definition is very
inclusive and business value-oriented. It means you need to prioritize them for yourself and
decide to what degree you should follow them to achieve your business goals.
Just as Candace did, you will need to keep the following principles as “guides” to
successfully implement a Data Mesh.
1.4 Data Mesh principles
Zhamak Dehghani first described the current incarnation of the Data Mesh concept as a set of
four principles. Throughout the book, we will focus on their implementation details. Below is a
summary of these four Data Mesh cornerstones.
1.4.1 Domain-oriented decentralized data ownership and architecture
The first principle is that data and its relevant components should be owned, maintained, and
developed by the people closest to it, the people inside the data’s domain. This calls for the
application of the concepts of domains and bounded contexts to the data world. This also
means the application of decentralization to ownership and architecture.
The idea is, that just like in the example above, domain-internal engineers are responsible
for developing data interfaces, allowing other users, be it data scientists, self-service BI users,
data engineers, and system developers, to use the domain data. Data engineers of a data
product are expected to be experts within remits of a single domain, which minimizes
communication problems and misinterpretations of data.
Data should have its clear ownership, and it should not be on the centralized level of
organization; we should put that responsibility into the hands of the people closest to it. That
might be the domain team in the case of a source-oriented dataset, or it might be a data
engineering team, an analytics engineering team, or a data science team, for new datasets
created out of multiple others. It could also be the organizational unit using the data if our
dataset is a very consumer-oriented dataset.
24
The domain is an area or part of our business. It is a way of slicing and decomposing the
company. Quite often, our organizational structure resembles business domains.
At Messflix LLC, we can, for instance, find domains like Content Distribution, Content
Production & Market, and Brand Development.
Figure 1.9 Simplified Messflix business domain sketch
The Domain Ownership principle says that each of the teams or units owning those domains
should also own data that has been created inside this domain. So the team developing
software to support the content production is also responsible for Content Production Data.
But what does it mean to be responsible for data?
Being responsible for data means hosting and serving datasets to other parts of the
organization, to other domains. Ownership not only spans data; the team should also be
responsible for pipelines, software, cleansing, deduplicating, or enriching of that data.
©Manning Publications Co. To comment go to liveBook
25
©Manning Publications Co. To comment go to liveBook
The same as with agile principles and agile teams, ownership on its own would not make
much sense without having autonomy at the same time. This is why teams should be able to
release and deploy their operational and analytical data systems autonomously.
As you can see in the example of Messflix LLC, every business consists of many domains.
Each of those domains can usually be further split into subdomains. By applying this principle,
we will end up with a mesh of interconnected domain data nodes. Nodes of the mesh can and
even should be connected. Data can be moved, duplicated, and change shape between the
domains when it is needed. For example, data will usually move from source-oriented data
domains (output of the operational systems) to more consumer-oriented data domains, where
raw data will be customarily aggregated and transformed into a more consumer-friendly shape
and format.
In such a mesh of interconnected domains and subdomains, only the people close to the
data know it well enough to actually work with it. Take an example from above: Does the word
“content” mean the same in all three domains? Hopefully not, because in the “Produce content”
domain, we will have both “draft versions and ideas” of not yet produced content, as well as
content that will then become truly productionized content. However, in the “Distribute
content” domain, people will always mean productionized final content when they say content.
Just imagine a developer from the “Distribute content” domain who is supposed to compile
a list of “content pieces” from the “Produce content” domain. He will likely produce a list of
produced content pieces. However, that will miss the point. The requirement likely asks for a
list of all content pieces, including ideas, things that are still in production, and canceled pieces.
Additionally, the status of these content pieces should also be included.
However, people outside this domain will not even know that these are important pieces of
data. Hence making the people inside the domain the only people truly able to work with this
kind of data.
Source Data Domains usually serve data and information that represent facts about the
business. Data should be exposed to other domains to serve operational purposes (like REST
API) and analytical purposes. Source Data Domain should expose domain events and historical
snapshots aggregated over time. With the latter, we should make sure that underlying storage
is optimized for Big Data analytical consumption (like Data Lake). In the above example, the
“Produce content” domain becomes a source data domain, when it exposes lists of produced
content pieces. This is an original piece of data created by the business process of creating
content.
Consumer Data Domains are aligned closely with consumption. An excellent example of
such a domain could be Management Reporting and Predictions. In the case of Messflix LLC, it
could be the Content Recommendations domain which might be a subdomain of Content
Distribution. If now the marketing team takes the list of produced content pieces and enriches
it with relevant marketing materials, the tweets it uses to promote it, etc., then the new list
slips into a consumer data domain.
Parts of the data between Data Domains can be duplicated. Still, because it is serving a
different purpose, it will also be modeled differently, so usually, domain boundaries will, at the
same time, be data model boundaries. Because we want to give teams as much autonomy as
possible, we are not trying to achieve a single canonical model for the whole organization. We
are giving them the freedom to model the data in the way they need it. Besides the Source
26
©Manning Publications Co. To comment go to liveBook
and Consumer aligned domain, we could also encounter core data domains used across the
organization and usually represent key entities or objects.
When we share the responsibility for data across the organization, we gain tremendous
scalability and maintainability. For example, when we want to add new datasets to the Mesh,
we will be adding a new autonomous node. At the same time, teams that own datasets will be
in a very comfortable situation. They will only own data that they truly understand.
1.4.2 Data as a product
The second principle is that data must be viewed as a product. This calls for an introduction of
product thinking, integrated into data management.
Usually, when we talk about data, the first thing that comes to our mind is either a file or
a table in a database. We often see a spreadsheet or a series of rows in a file with named
columns when we think of data. Taking this perspective, it is easy to reduce data to technical
details. But, instead, the more important question is - what gives value to the data from the
organization's point of view? Or reversing this question - what is stopping our data from being
turned into valuable decisions?
Without a proper set of descriptions, even the best-prepared set of data will not be found,
and thus no value extracted from it. Missing information on the recency or completeness of
data might render it completely useless in cases that require these attributes. That's why it's
worthwhile to think of data as a product - a larger whole that ultimately makes up the
experience of the users who use it.
Data offered by a data team shall follow typical product features like:
• Viable quality - ensured by specialized domain experts
• Anticipation of user needs - team offering the data to the outside world, shall
understand the enterprise's business environment, e.g., present the data in a format
easily digested by existing data pipelines to ensure availability and effortless usefulness
of the provided data
• Secured availability - ensuring availability of product to fulfill user needs whenever he
needs it
• Focus on user goals - the focus on own domain shall not mean lack of communication
of data team with other users; on the contrary, search for synergies and shared toolset
shall create an opportunity to understand each other’s needs better
• Findable - any data product should be discoverable by a simple means, something a
random table in a database lacks
• Interoperable - different data products should be combinable in a way that increases
their value
WHEN CAN WE CALL DATA A PRODUCT?
In everyday life, we deal with many products. A product might be defined as, in general, "the
result of a conscious action.” If we take a pair of jeans as an example, we are convinced that
it meets certain predetermined conditions. For a product to be called a pair of jeans, it must
have a suitable form and shape. In addition, it should have its unique name (especially if the
manufacturer provides many products of a given type) because we buy a specific model and
not a pair of jeans in general.
27
©Manning Publications Co. To comment go to liveBook
It should also meet some standards to ensure that a pair of jeans will not break down after
a few hours of use and is made of a safe material. In addition to this, there is someone
responsible for this product, its quality, and the consequences of its use within the accepted
terms of use. We can use this kind of thinking to think about data.
Treating data as a product will mean that someone has consciously designed the product
and then created and released it, is responsible for it, and is mainly responsible for its quality.
In the context of Data Mesh, this will be the responsibility of the Data Product Owner - who
designs the Data Product - and the Data Product Developers who implement it. Just like a
product on a shelf in a store - a Data Product has its unique name, and it has established
characteristics, including:
• Level of quality
• Level of availability
• Security rules
• Frequency of updates
• Specific content
When we think about products - it's not enough to just expose data. We also need to make
sure that we maximize its usability for the end-users. In this context, the role of the Data
Product Owner is critical, as he or she is responsible for the final User Experience of the data.
Referring to the Messflix example mentioned earlier, we can imagine a few exemplary Data
Products in the Produce Content domain:
• Cost statement
• Scripts
• Cast
• Movie popularity
• Movie trends
Treating data as a product brings us straight to the Product Thinking approach - start with
what problems your customers want to be solved, which should drive the Data Product design
process. Then, as a Data Product Owner - you should deliver a Data Product addressing these
problems based on predefined success criteria.
Data as a product also implies a degree of standardization that allows a single element to
be incorporated into a larger Data Mesh ecosystem. To call a given set of data a Data Product,
it should be:
• Self-described and discoverable - data should be described, and this description
should be an integral part of the Data Product. Data Product should be able to register
itself in the Data Mesh ecosystem and, as a result, should be discoverable,
• Addressable - it should have its unique address (e.g., in the form of URI address) so
that it can be referred to and relationships between Data Products can be created,
• Interoperable - a Data Product should be made according to predefined standards,
concerning the form of data sharing, standardized formats, vocabularies, terminology
or identifiers, secure and trust-worthy. Data Product should meet the established and
declared SLA, and enable controlled access, ensuring data security from both
perspectives: intellectual property and GDPR-type regulations.
28
©Manning Publications Co. To comment go to liveBook
DATA PRODUCT AS AN AUTONOMOUS COMPONENT
While fulfilling the previously established conditions, a Data Product should at the same time
constitute an autonomous component so that it can be independently developed by the team
responsible for a given Data Product. From the point of view of a technical solution, we can
see a Data Product as an analog of a microservice in the data world. In addition to making
data available, the Data Product also embeds code related to data transformation, cleaning, or
harmonization. It also exposes interfaces for automatic integration with the Data Mesh
environment and platform by providing, among other things:
• Input logic and output ports - form and protocols used to ingest from source and
expose data to consumers;
• Operational metrics - number of users, throughput, amount of data fetched, etc.;
• Data quality reports - quantity of incomplete data, format incompatibilities, statistical
measures of outliers, etc.;
• Metadata - specification and description of the data schema, domain description,
business ownership, etc.;
• Configuration endpoint - means to configure the Data Product in runtime, e.g.,
setting the security rules.
These are a few examples of what can constitute a Data Product as a technical component.
1.4.3 Federated computational governance
The third principle is to federate and automate the data governance across all of the
participants of the Data Mesh. It aims to provide a unified framework and interoperability to
the ecosystem of largely independent data products. Its purpose is to make the autonomous
data products work in an actual data mesh, not just as stand-alone products.
Federated computational governance requires two inseparable elements: the governing
body and means of rules enforcement.
Overarching rules and regulations shall be agreed upon by a body composed of Data
Product Owners, self-serve data infrastructure platform team, security specialists, as well as
CDO/CIO/CTO representatives. The body would also serve as a place for discussion regarding,
e.g., development of new data products vs. adding new datasets to existing data products,
methods of ingesting new external data sources, priorities for central platform development,
etc.
Effective data governance is crucial, with data security being one of the main concerns of
CDOs/CIOs in companies across all sectors and sizes. Also, most large enterprises need to
introduce controls on data security and governance enforced by governmental or business
regulations, e.g., GDPR, HIPAA, PCI DSS, or SoX.
FEDERALIZATION OF DATA GOVERNANCE
At first glance, Data Mesh adds a new layer of complexity to the already vast scope of Data
Governance, as it now also needs to address a shift of responsibilities to Data Products. Each
of the Data Products, in turn, needs to be equipped with processes allowing safe and efficient
ways of handling owned infrastructure, code, and data (and metadata). Moreover, data
governance processes need to balance company-wide cohesion of data solutions, usually
29
©Manning Publications Co. To comment go to liveBook
achieved through standardization vs. autonomy-driven flexibility and creativity offered to Data
Product teams.
It is imperative to understand that there is no silver bullet solution to balance central
governance and local autonomy! It will always depend on the specifics of your organization.
For example, what are the needs and the maturity of your Data Product Owners? What is the
data-related risk appetite of the organization? What is the level of expertise of your central
data governance team? How sensitive is the data you’re working with? You need to answer
these and a lot more questions before you’ll be able to set the remits of central and local
teams.
Another set of policies is required to ensure interoperability of Data Products and the ability
of data consumers to join them together readily. Finally, the procedures need to provide the
compatibility of different Data Products without explicitly enforcing an overarching data model,
as such an approach has been proved to create a bottleneck in data operations.
Data Mesh tries to answer that need with the Federalization of Data Governance. The
federalization in this context means governing structure operating at parity of distinct levels -
central and local. The central level of governance, executed by the data governance council
(of course, the central level governance body may use another name, e.g., in MVP example
we’ll present in chapter 3, it will be called simply a data governance team), will decide on a
minimal set of global rules required to ensure safe and secure discoverability and
interoperability of data products.
Data Product teams, led by Data Product Owners, are responsible for developing their
products with a high degree of autonomy, deciding on all technical and procedural issues within
its remits (within boundaries of the enterprise technology stack).
There is no silver bullet solution as to the precise division of responsibilities, the structure
of the data governance council, and exact rules governing the data world. Instead, each
business will have its own set of global rules, leaving domain teams with different levels of
control over their Data Products.
Chapter 4 will discuss the effects of different levels of domain team autonomy on their
agility and overall system complexity.
COMPUTATIONAL ELEMENTS OF DATA GOVERNANCE
The name Federated Computational Governance was coined by Zhamak Daghani in her de
facto “Data Mesh manifesto”4 blog article. However, as Computational Governance is not
defined anywhere else, throughout this book, we will assume that this term relates to elements
of Data Governance automation, enabled through the existence of the Central Platform (as we
call self-serve infrastructure-as-a-platform presented in a bit more detail in section 1.7, and in
a whole lot more detail in chapters 7 and 8), serving as a medium connecting different Data
Products.
Data Governance elements, which can be automated and embedded into Data Platform
include, but are not limited to:
4 https://martinfowler.com/articles/data-mesh-principles.html
30
©Manning Publications Co. To comment go to liveBook
• Metadata
• Catalog, reference, and master data
• Lineage & provenance
• Validation & verification
• Storage and operations
• Security & privacy
Once again, there is no silver bullet solution for deciding which of these should be
automated. It will require your teams’ hard work to identify which data governance tasks create
bottlenecks in Data Mesh development in your organization and automate them. It will pay off,
however. First, it will offer Data Product Owners the solution allowing them a frictionless
connection of their Data Products. Second, it will enable users to make efficient use of the
exposed data.
You will learn more about automating elements of data governance when we will discuss
the details of the self-serve data infrastructure-as-a-platform.
1.4.4 Self-serve data infrastructure as a platform
The fourth principle is to extract the duplicated efforts of the Data Mesh into a platform. It calls
for the application of “platform thinking” to the data context. Platform thinking means that
efforts that are duplicated throughout the company to a larger extent can also be packaged
into a “platform” and thus only done once, but offered as a “service” to others.
Just like anyone can rent cloud resources on one of the major cloud providers and
customize them to fit his needs, taking away to duplicate the effort of maintaining one’s own
cloud, his idea can be shrunk to efforts inside a company.
Building and maintaining data products is resource-intensive and requires a set of very
specialized skills (ranging from computational environment management to security).
Multiplication of the required effort by the number of Data Products would endanger the
feasibility of the whole Data Mesh idea. The idea behind the Central Platform is to centralize
repeatable and generalizable actions to the degree necessary (yet again depending on the
context of the company!) and to offer a set of tools abstracting away specialized skills. It would
reduce entry and access barriers for both Data Product developers and Data Product
consumers.
You may start with the requirements of your first Data Product Owners and iteratively build
the setup up to meet your organization’s needs.
The infrastructural support can be offered at on-premise computing power or in the cloud,
depending on the enterprise policies. Its elements can be available in IaaS, PaaS, SaaS, or
their hybrid models. Data Mesh platform could, e.g., support:
31
©Manning Publications Co. To comment go to liveBook
• Governability - all data computation related to Data Governance processes needs to
be incorporated and automatically enforced on every Data Product connected to the
Data Infrastructure.
• Security - the infrastructure solution shall ensure that all Data Products offer freedom
of operation for users whose access allows them to meet their information requirements
and ensure safety from unauthorized access. To do that set of ready-to-use processes,
tools and procedures shall be accessible to Data Product creators and users.
• Flexibility, adaptability, and elasticity - the infrastructure needs to support multiple
types of business domain data. It means enabling different data storage solutions, ETL
and query operations, deployments, data processing engines, and pipelines. All that
needs to be scalable, to serve business needs as they arise.
• Resilience - smooth operations of data-driven businesses require high availability of
data. Therefore, they are ensuring redundancies and disaster recovery protocols at the
design level of each infrastructural element.
• Process automation - from data metadata injection at Data Product registry to
ensuring access control, data flow through central infrastructure needs to be fully
automated, possibly using machine learning and artificial intelligence to ensure efficient
data processing, quality, and monitoring.
After you’ve got a short summary of the four key Data Mesh principles, let us visit Candace
again and see how she scores on these four principles.
1.5 Back to Snow Shoveling
In Candace’s business, you might identify a few “expert domains”. Seems like shovel
procurement is one, but also the two streets seem to be separate domains. They definitely are
different in some key elements and local knowledge is needed to work effectively. Let’s look
at how the business evolved over time:
• In Candace’s first year, her team does not own the data, even though they own their
“domains” in the sense that they are responsible for operations there as well as creating
domain data. There is no data governance mentioned, and not much of a platform to
help them do anything with data. Data is a by-product of their work.
• In Candace’s second year, she makes a key change. She gives the data ownership into
Eve’s & Adam’s domain. She also extends their responsibility in general, something that
in many companies happens before transferring the data ownership. However, there is
still no data governance, platform, and not much data is being treated as a product.
• In Candace’s third year, she essentially tells Adam & Eve to finally treat the data not
just for themselves as valuable, but also as a product for others, in this case, Bob.
Finally supplied with a proper “data product”, Bob is able to also optimize the costs of
the business.
In summary, Candace focuses mostly on the principle of decentralized data ownership and
data product thinking. Since her operation is very small, she doesn’t need to do much
governance at all besides talking with her employees about using data in decision-making. She
also does not have to use a large platform, or any central platform at all besides maybe email
simply because there isn’t yet a large mesh of data products to connect.
32
©Manning Publications Co. To comment go to liveBook
INSIGHT: THE VALUE LIES IN DECENTRALIZATION, NOT IN ALL FOUR PRINCIPLES We think
that Data Meshes at the core are just about the decentralization of data ownership into any kind of autonomous
unit. That can be an individual in our case, it can be a whole department, it can be anything depending on the
situation. But, the key principles are guidelines for helping us execute this decentralization process well.
We believe that the Data Mesh journey might begin at any size of an enterprise if you “feel
the pull of decentralization”. We will discuss that in more depth in the next chapter.
INSIGHT: DATA MESHES ARE FOR COMPANIES OF ALL SIZES The Data Mesh is a possible tool
for companies of all sizes. But that does not mean it is the right tool for every company in their current stage.
It can also be the wrong tool for a company, again independent of its size. We discuss the decision process in
the next chapter.
So why did Candace feel this “pull of decentralization”? And why will you feel it too at some
point?
1.6 Socio-technical architecture
Data Mesh is not a technical solution for data problems. Data Mesh solves these problems
using socio-technical architecture or socio-technical system design. We believe this is the
biggest strength of Data Mesh as a paradigm, and we think you should treat socio-technical
architecture as a foundation of data mesh implementation. You need to apply socio-technical
architecture consciously to make it successful.
But before doing it, you need first to understand what it means, how it originated, and
most importantly, what are the underpinning laws behind it. For that, we look into Conway’s
law to understand why we won’t be able to change technology just on its own and then into
the Team Topologies framework which is in essence about socio-technical architecture. Finally,
we look into Cognitive Load as one of the main ideas leveraged by the Team Topologies
framework.
1.6.1 Conway’s law
"Any organization that designs a system (defined broadly) will produce a design whose
structure is a copy of the organization's communication structure." (Conway’s Law)
Melvin Conway made this observation in 1967, so just ten years after FORTRAN, the first
widely used high-level language, was released. It didn’t take us long to see this strong, almost
gravitational force in practice. This ‘force’ will sooner or later transform our architecture so it
will look like our organizational structure. As architects, we need to be fully aware of the implied
consequences.
We should take as an example the martial art called Aikido. ‘Aiki’ refers to the martial arts
principle or tactic of blending with an attacker's movements to control their actions with
minimal effort. The masters of Aikido teach you that you should not put your force directly
against the force of your opponent. Instead, you should take advantage of the force he is
exerting. In the same way, we should be aware and take advantage of Conway's Law.
33
©Manning Publications Co. To comment go to liveBook
EXAMPLE: CENTRAL DATA WAREHOUSES Conway’s law is a good explanation on why people say
“central data warehouse” but mean “central data warehouse operated by a central data team”. It’s because
usually when you find a central data warehouse, it's been built by a central data team.
Conway’s law simply tells us to take care of both the organizational side of things and the
technical. Just changing technical things isn’t going to make an impact. The organization will
cramp itself into the new technology.
1.6.2 Team Topologies
In the Socio-technical architecture approach, as the name suggests, we are trying to co-design
both the social and the technical architecture elements simultaneously. This way, we are
thinking upfront about all constraints and concerns.
IMPERFECT EXAMPLE: CONSTRUCTING TEAM TOPOLOGIES Try to think of the process of building
a skyscraper. It’s structurally different from villagers collectively building a barn for one of the members of
their community. A Skyscraper is built by highly specialized teams, where some have “front line”, and some
have “support” roles. You don’t send electricians to install windows (pun intended). The requirements of the
technology used in a given location at a given time define the team carrying out the task.
This section is about the one main architecture method that is used today and within the
Data Mesh community to do socio-technical architecture well.
“Team Topologies is a clear, easy-to-follow approach to modern software delivery with an
emphasis on optimizing team interactions for flow.” (source: teamtopologies.com) Matthew
Skelton and Manuel Pais created team topologies, and they are starting to get some traction
in the community because of their simplicity and straightforward application. Team Topologies
is an approach that helps you not fall into the trap of Conway's Law. The Team Topologies
framework is designed to optimize the teams inside a company for optimal flow. It relies
heavily on the idea of cognitive load to properly split workloads across this flow and separates
teams into just a handful of teams as well as interaction modes into ones that are designed to
minimize cognitive load.
Since Team Topologies is focused on the optimal flow of the company, it will help us to
optimize the data to value flow inside the company.
1.6.3 Cognitive Load
Cognitive load is a term from cognitive load theory, and it means the amount of mental
resources (working memory) used by a person. This theory was at the beginning applied to
the teaching field, and it influenced how we write instruction so readers can easily digest them.
But after that, it was used in more and more areas, like IT.
There are three types of Cognitive Load:
34
©Manning Publications Co. To comment go to liveBook
• Intrinsic: the skills and knowledge you need to build your product (like programming
languages, framework, patterns).
• Extraneous activities are not part of product development but are needed to release the
product (like infrastructure, deployment, monitoring).
• Germane: knowledge of business and the problem space (like, types of movies if you
are a Messflix LLC developer).
As socio-technical architects, we should shape our teams so that the total cognitive load is
limited. It is very similar to road traffic, and you will not maximize the flow of cars if you fill
up every square meter of the road with cars; traffic moves in the fastest way when there is
enough space between vehicles to enable them to drive near the speed limit. It is the same
with the teams.
So, what can we do to make Data Mesh teams performant? So firstly, we should plan our
technology stack to be simple to limit intrinsic load. Secondly, we should embrace platform
thinking as not to force teams to reinvent the wheel every time with yet another monitoring
and logging solution to limit extraneous load. Doing both will allow the team to entirely focus
on what is essential: the germane load, a.k.a Business Domain. But germane load should also
be limited, and we will do it by decomposing solutions into smaller domain-oriented data
products.
All of the principles are influenced by socio-technical ways of thinking.
Domain-oriented decentralized data ownership and architecture reduce the germane load
by decentralizing responsibility for domains and corresponding data.
Data as a product reduces collaboration between teams to access data by making data
findable, accessible, interoperable, and reusable; it limits extraneous load by exposing data in
the expected and known format and ports by consuming teams.
Self-serve data infrastructure as a platform - it reduces extraneous load by providing a
self-service platform.
Federated computational governance makes collaboration between teams easier by
enforcing common standards and patterns; it enables team members’ transfers by reducing
possible technology stack within the company.
As you can see, at the heart of every principle, there is a socio-technical way of thinking.
Due to the socio-technical nature of this paradigm shift, it's important to be clear about the
benefits that this shift can possibly result in. So next, we look at the key benefits.
While Data Mesh is a socio-technical solution to many problems, there are some major
challenges associated with it. Let’s explore them in detail.
1.7 Data Mesh challenges
Data Mesh is neither a silver bullet nor a plug-and-play solution. Instead, its successful
implementation requires overcoming an array of challenges.
We will cover the technological, data management, and then the organizational challenge
in turn.
35
©Manning Publications Co. To comment go to liveBook
1.7.1 Technological challenge
The most crucial technological challenge of Data Mesh implementation is ensuring proper
tooling, available to both domain-focused and central platform data teams. Only providing data
discoverability and links between domains may prevent the whole ecosystem from becoming
a siloed world.
The other challenge is ensuring the tooling developed locally is shareable across domains.
Flexibility and autonomy of teams shall not lead to inefficiencies at the company level and
reduced synergy leverage.
To effectively develop and maintain their data solutions, domain data teams need safe and
secure access to data storage services. Centralizing, e.g., maintenance of cloud computing
environment is much more efficient than the multiplication of this effort by each team. In
addition, centralized control over cloud storage and computing can also help introduce a proper
spending regime. We’ve seen way too often situations where seemingly rational spending by
multiple separated teams summed up to amounts far exceeding levels justifiable at the
company level.
With a plethora of distributed, semi-independent Data Products, the development of
consistent monitoring, alerting, and logging procedures becomes a challenge. As a result, both
central platform and domain teams need to collaborate closely to ensure continuous insight
and control over data quality and security and maintenance efficiency.
1.7.2 Data Management challenge
Like the first technological challenge, the critical data-management-related challenge is
ensuring data is easy to locate and comprehensively documented. The importance of a data
catalog cannot be overestimated. It is not a challenge unique to Data Mesh, but as it’s based
on the decentralization paradigm, the consequences of overlooking it would be much direr.
There will be no central data team capable of ad hoc identification of available data assets (not
that such ad hoc querying would be an advisable form of interacting with, e.g., Data Lake
contents!).
Another challenge is ensuring control over the duplication of data across different domains.
With each new business domain needing to be served with repurposed data, redundancy may
emerge. It is not only impacting the costs of data storage and management but introduces
lineage and versioning problems. Data consumers may analyze outdated, incomplete, or
altered data and, as a result, get to misleading conclusions.
The third challenge is not specific to Data Mesh, however, one needs to remember about it
when embarrassing on Data Mesh journey. It lies in cross-domain analytics. As business
analysts and data scientists will not have access to a harmonized enterprise-wide data model,
they may need additional platforms to aggregate and consolidate the various data products.
Work on distributed data sources may also lead to spaghetti data pipelines, reducing their
efficiency and reusability. Cross-domain access also challenges the data safeguarding and
access controls, as many-to-many relationships require simultaneous access to multiple data
products.
36
©Manning Publications Co. To comment go to liveBook
1.7.3 Organizational challenge
Organizational challenges in the Data Mesh context break down into three different aspects,
scope, data practices and culture.
The first organizational challenge is the scope of such a paradigm shift. It requires
coordinated efforts from multiple engaged parties from different business areas: IT, the
business units, and senior management.
Second, it does require a level of maturity in the handling of data by multiple parties.
Centralized parties, data creators, managers, and users, will need to learn to handle data with
a certain level of proficiency.
Third, in most enterprises, going from data “as-a-byproduct” to “data-as-a-product” will be
a substantial cultural shift. A shift that will not only involve data engineers and the IT
departments, but the product, management, marketing, sales, and most other departments
working with data.
Decentralizing the decision-making process, shaping Data Product Owner positions,
developing multiple cross-functional teams, and shifting responsibilities require a significant
restructuring of existing organizational charts. Moreover, such changes come with costs in
time, brainpower, and finances.
Federated governance requires the development of novel procedures, starting from
decision making to communication methods. These responsibilities and principles have to be
appropriately identified and federated.
As we come to the end of this chapter, you should now have a better understanding of
Data Mesh as a joint technological and organizational paradigm, as well as its underlying
principles. In the next chapter, we will show you how to build a start Data Mesh in your
organization within a month.
1.8 Summary
• The cornerstone of Data Mesh is decentralization: moving ownership and responsibility
for data closer to its source. It aims to keep data in sync with the subject matter, to
remove bottlenecks of long data pipes and central team inefficiencies.
• Data Mesh is a socio-technological architecture paradigm based on principles of:
domain-oriented data ownership, treating data as a product, federated computational
governance, and self-serve data infrastructure as a platform.
• Domain-oriented data ownership means the data and its relevant components should
be owned, maintained, and developed by the people closest to it, the people inside the
data’s domain.
• Treating data as a product means a conscious design of the outcome presented ot the
outside environment, clearly assigned role for ensuring said outcome availability and
quality, and autonomy of development required to produce said outcome.
• Federated computational governance means two things. First, the responsibility for data
governance is split between central body responsible for company-wide policies and
standards, and owners of highly autonomous units, usually domain-oriented,
responsible for development of actual systems. Second, means of rules enforcement
should be automatically executed wherever it’s feasible.
37
©Manning Publications Co. To comment go to liveBook
• Self-serve data infrastructure means extracting the duplicated efforts of the Data Mesh
into a platform, thus only done once, but offered as a “service” to others.
• Data Mesh, is not a precisely defined solution. The extent of application of each principle
differ widely from application to application, and it will vary depending on your business
case.
• When considering the Data Mesh implementation, you must take into account major
technological, data management-related and organizational challenges.
38
©Manning Publications Co. To comment go to liveBook
2
Is Data Mesh Right for You?
This chapter covers
• How to decide whether you should introduce Data Mesh to your organization
• Decision drivers to consider before choosing a data architecture
• Data Mesh compared to other popular data architectures
• The steps for transforming your organization architecture into the Data Mesh
In the first chapter we explained to you what the Data Mesh is and why you should consider
implementing it in your company. In this chapter, we will answer two other essential questions:
1. Should I implement Data Mesh in my business, i.e., what are Data Mesh drivers?
2. How much effort does it take to implement Data Mesh, i.e., would benefits outweigh
the effort of its implementation?
The presence of the first question may surprise you in the context of this book. Data Mesh
has become one of the hottest buzzwords in the industry. Many started to implement it without
considering if it fits their organizations, or without considering all requirements and
ramifications. We observed many similar rushes on patterns or practices in the past, like
microservices or agile. Many of these ended up pretty badly, and people started to blame these
patterns or practices afterward. The truth is, there is no silver bullet, and every pattern or
practice has its area of applicability, and they come with trade-offs. Treat Data Mesh as yet
another tool in your toolbox.
In the second part of this chapter, we will focus on the example process of implementing
Data Mesh. We will show you possible steps towards this goal and how all skills and knowledge
you will learn in upcoming chapters fit together. But as we said before, the most important
thing is to know if you need a data mesh. So let’s start there.
39
©Manning Publications Co. To comment go to liveBook
2.1 Analyzing Data Mesh drivers
To answer whether Data Mesh would be the right choice for you, we need to analyze the
decision drivers behind architecture selection. We split these drivers into three categories:
• Business drivers
• Organizational drivers
• Domain data drivers
Let’s start from the most crucial aspect, i.e., business reasons.
2.1.1 Business drivers
We think you should start your analysis from the business side. Stop there if you can’t find a
suitable business reason to start your Data Mesh journey. You’ll already have an answer: you
don’t need a data mesh.
BUSINESS STRATEGY
Look at your business strategy at the company level or in the case of big corporations on the
unit level. Search for the answers to the following questions:
• Are we defining our company as data-driven, or are we planning to shift our strategy
into more data-driven?
• Do we have Objectives and Key Results (OKRs), or any goal-setting methodology, that
require data to be met?
NOTE: OKR OKRs are a methodology used to define goals for the company and on the individual level. This
methodology is quite often used as a tool to implement business strategy.
If the answer to any of these questions is yes, you can dig deeper and look for the specific
business cases requiring data (which we cover in the next step). If the answer is no, your work
is going to be more challenging because business cases for data needs are often hidden in
business unit strategies or in specific business processes. It is going to take some laborious
searching to find what you need.
BUSINESS CASE AND ITS DATA NEEDS COMPLEXITY
To build a viable business case for data mesh, start with answering the following questions:
• Do you have specific business processes starting or ongoing with complex data needs?
• Do you need to kick-off a product/project with complex data needs to accomplish OKR?
By complex data need, we mean multidimensional analysis (ad hoc, AI/ML, reporting) run
on top of the data coming from multiple sources. If answers to these questions are positive,
then there is a chance you have a compelling business case.
KEY POINT Data Mesh requires a business case and is useful when there are related complex data needs.
Let’s look at the examples of business cases and their data needs.
40
Other documents randomly have
different content
Data Mesh in Action (MEAP V04) Jacek Majchrzak
Data Mesh in Action (MEAP V04) Jacek Majchrzak
Data Mesh in Action (MEAP V04) Jacek Majchrzak
The Project Gutenberg eBook of Bothwell; or,
The Days of Mary Queen of Scots, Volume 1 (of
3)
This ebook is for the use of anyone anywhere in the United
States and most other parts of the world at no cost and with
almost no restrictions whatsoever. You may copy it, give it away
or re-use it under the terms of the Project Gutenberg License
included with this ebook or online at www.gutenberg.org. If you
are not located in the United States, you will have to check the
laws of the country where you are located before using this
eBook.
Title: Bothwell; or, The Days of Mary Queen of Scots, Volume 1
(of 3)
Creator: James Grant
Release date: September 11, 2017 [eBook #55527]
Language: English
Credits: Produced by Al Haines
*** START OF THE PROJECT GUTENBERG EBOOK BOTHWELL; OR,
THE DAYS OF MARY QUEEN OF SCOTS, VOLUME 1 (OF 3) ***
BOTHWELL:
OR,
THE DAYS OF MARY QUEEN OF SCOTS.
BY JAMES GRANT, ESQ.,
AUTHOR OF
"THE ROMANCE OF WAR," "MEMORIALS OF EDINBURGH CASTLE,"
"THE SCOTTISH CAVALIER," &c., &c.
IN THREE VOLUMES.
VOL. I.
LONDON:
PARRY & CO., LEADENHALL STREET.
MDCCCLI.
M'CORQUODALE AND CO., PRINTERS, LONDON.
WORKS, NEWTON.
PREFACE.
The leading event upon which the following story hinges, will be
found in the illustrative notes at the end of the third volume, which
will show that the Magister Absalom (so frequently referred to) was a
real personage, who, in the days of Earl Bothwell, was a Protestant
clergyman at Bergen, and author of a Diary named The Chapter Book.
There is no style of reading more conducive to a good or evil
result, than the historical romance, according to the manner it is
treated, by a judicious or injudicious writer. I have been studious in
avoiding any distortion of history, the tenor of which is so often
misconstructed wilfully by writers of romance; for there are bounds
beyond which not even they are entitled to go. The Scottish reader
will find how closely I have woven up the stirring events of 1567 with
my own story, which, in reality, contains much more that is veracious
than fictitious.
Thus, Bothwell's journey to Denmark—his conflict with John of
Park—the Queen's visit to Hermitage—the assault on the house of
Alison Craig—the brawl and assistance given the Earl (in mistake for
Arran) by the Abbot of Kilwinning—and many other incidents, all
occurred actually as related.
With one or two exceptions, every character in the following
pages was a bona fide personage "of flesh and blood," who existed at
the time, and was an actor in the scenes narrated.
In the general grouping, costume, and other dramatic
accessaries, I have endeavoured (as closely as I could) to draw a
picture of the Scottish court and metropolis in the year 1567, at a
time when the splendour of both was dimmed by the poverty which
followed the wars and tumults of the Reformation; and with what
success, I may say with the old knights of Cumbernauld—"Let the
deed shaw!"
EDINBURGH, September, 1851.
CONTENTS OF VOL. I.
CHAPTER
I. The Castle of Bergen
II. Erick Rosenkrantz
III. The Strangers
IV. A Norse Supper
V. The Earl and Hob Discourse
VI. Anna
VII. Konrad
VIII. The Cock-of-the-Woods
IX. Lord Huntly's Letter
X. The Hermit of Bergen
XI. The Fleur-de-Lys
XII. The Isle of Westeray
XIII. Noltland
XIV. The Separation
XV. Doubt and Despair
XVI. Blantyre Priory
XVII. The Countess of Bothwell
VIII. The Rescue
XIX. The Rejected and the Rival
XX. Konrad and the Countess
XXI. Disappointment
XXII. The Countess Jane
XXIII. The Pursuit
XXIV. Mary, Queen of Scots
BOTHWELL;
OR,
THE DAYS OF MARY QUEEN OF SCOTS.
CHAPTER I.
THE CASTLE Of BERGEN.
The stern old shepherd of the air,
The spirit of the whistling hair,
The wind has risen drearily
In the northern evening sea,
And is piping long and loud
From many a heavy up-coming cloud.
Leigh Hunt.
It was the autumn of a bleak day in the September of 1566.
Enveloped in murky clouds, through which, at times, its red rays
shot along the crested waves, the Norwegian sun was verging to the
westward. From the frozen Baltic a cold wind swept down the Skager
Rack, and, urged by the whole force of the Atlantic ocean, the sullen
waves poured their foam upon the rocky bluffs and fissured crags that
overhang the fiord of Christiana.
In those days, a vessel in the fiord proved an object of the
greatest interest to the inhabitants of the hamlet; and it was with
growing fears that the anxious housewives and weatherwise
fishermen of Bergen, a little wooden town situated on the bay of
Christiana, watched the exertions made by the crew of a small crayer
or brigantine, of some eighty tons or so, that under bare poles, or
having at least only her great square spritsail and jib set,
endeavoured to weather the rocky headland to the east, and gain
their little harbour, within which the water lay smooth as a millpond,
forming by its placidity a strong contrast to the boiling and heaving
ocean without.
The last rays of the September sun had died away on the pine-
clad hills of Christiana and the cathedral spire of Bergen. Night came
on sooner than usual, and the sky was rendered opaque by sable
clouds, through which the red streaks of lightning shot red and
forklike; while the hollow thunder reverberated afar off among the
splintered summits of the Silverbergen.
Then through the flying vapour, where, parted by the levin brand,
the misty rain poured down in torrents on the pathless sea, and the
goodwives of Bergen told their beads, and muttered a Hail Mary! or a
prayer to Saint Erick the Martyr for the souls of the poor mariners,
who, they were assured, would find their graves at the bottom of the
deep Skager Rack ere morning brightened on the waters of the
Sound.
The royal castle of Bergen, a great square tower of vast strength
and unknown antiquity, reared on a point of rock, still overlooks the
town that in the year of our story was little more than a fisher hamlet.
Swung in an iron grating on its battlement, a huge beacon fire had
been lighted by order of the governor to direct the struggling ship;
and now the flames from the blazing mass of tarred fagots and well-
oiled flax streamed like a torn banner on the stormy wind, and lit up
the weatherbeaten visages of a few Danish soldiers who were
grouped on the keep, glinting on their steel caps and mail shirts, and
on the little brass minions and iron drakes that peeped between the
timeworn embrasures.
Another group, which since sunset had been watching the
strange ship, was crowded under the sheltering arch of the castle
gate, watching for the dispersion of the clouds or the rising of the
moon to reveal her whereabouts.
"Hans Knuber," said a young man who appeared at the wicket,
and whose half military attire showed that he was captain of the
king's crossbowmen at Bergen, "dost thou think she will weather the
Devil's Nose on the next tack?"
"I doubt it much, Captain Konrad," replied the fisherman,
removing his right hand from the pocket of his voluminous red
breeches to the front of his fur cap, "unless they steer with the keep
of Bergen and the spire of the bishop's church in a line; which I saw
they did not do. Ugh! yonder she looms! and what a sea she shipped!
How heavily her fore and after castles and all her top-hamper make
her heel to leeward!"
"They who man her seem to have but small skill in pilot-craft,"
said one.
"By Saint Olaus!" cried another, "unless some one boards and
pilots her, another quarter of an hour will see her run full plump on
the reef; and then God assoilzie both master and mariner!"
"Luff—luff—timoneer!" exclaimed the first seaman. "Now keep
her full! Would I had my hands on thy tiller!"
"Every moment the night groweth darker," said the young man
whom they called Konrad, and whom they treated with marked
respect: "as the clouds darken the lightning brightens. A foul shame it
were to old Norway, to have it said that so many of us—stout fellows
all—stood idly and saw yonder struggling ship lost for lack of a little
pilot-craft: for as thou sayest, Hans, if she runs so far again eastward
on the next tack she must strike on the sunken reefs."
"No boat could live in such a sea," muttered the fishermen as
they drew back, none appearing solicitous of the selection which they
expected the young man would make.
"The mists are coming down from the Arctic ocean—the west
wind always brings them," said Jans Thorson; "and we all know 'tis in
these mists that the spirits of the mountain and storm travel."
"Come hither, Hans Knuber," said the captain, whose plumed cap
and rich dress of scarlet velvet, trimmed with white fur, and braided
with silver like a hussar pelisse, were rapidly changing their hues
under the drenching rain that lashed the castle wall, and hissed
through the deep-mouthed archway. "Come hither, thou great
seahorse! Dost mean to tell me thou art afraid?"
"Sir captain, I fear neither the storm nor the spirit of the mist;
but Zernebok the lord of evil may be abroad to-night, and he and the
Hermit of the Rock may chance to remember how once in my cups,
like an ass as I was, I reviled and mocked them both."
"Bah!" retorted Konrad, whose superstition did not go so far as
that of the seaman; "Jans Thorson, I will give thee this silver chain to
launch and put forth to yonder ship. Come, man—away, for the
honour of old Norway!"
"Not for all the silver in yonder hills, sir captain, nor the copper in
the mines of Fahlun to boot, would I trust myself beyond the Devil's
Nose to-night," said the old fisherman bluntly. "I have just refused
Master Sueno, the chamberlain."
"Why, 'twas just in such a storm old Christian Alborg, and his
stout ship the Biornen, were blown away into the wide ocean," said
another; "and I marvel much, noble Konrad, that you would urge poor
fellows like us"——
"On a venture which I would not attempt myself!" exclaimed the
young man, whose dark blue eyes flashed at his own suggestion.
"Now, Saint Olaus forefend thou shouldst say so!"
"Nay, noble Konrad"——
"But thou dost think so?"
The fisherman was silent.
A flush crossed the handsome face of Konrad of Saltzberg. He
looked seaward a moment. The wind was roaring fearfully among the
bare summits of the cliffs that towered abruptly from the shore to the
very clouds—absolute mountains of rock rising peak above peak; and
when the blue lightning flashed among them, their granite tops were
seen stretching away in the distance, while the giant pines that
flourished in their clefts and gorges, were tossing like black ostrich
feathers in the storm.
At the harbour mouth the waves of snow-white foam were visible
through the gloom, as they lashed, and hissed, and burst in
successive mountains on the rocks of worn granite that fringed the
entrance of the haven.
Konrad cast a rapid glance around him, and the appalling fury of
the northern storm made even his gallant heart waver for a moment
in its generous purpose; but a fair female face, that with all its waving
ringlets appeared at a little casement overlooking the portal, and a
kiss wafted to him from "a quick small hand," decided him. His eyes
sparkled, and turning briskly round to the fishermen, he said,—
"By my honour, Sirs, though knowing less of pilot-craft than of
handling the boll of an arblast, I will prove to you that I require
nothing of any man that I dare not myself attempt—so thus will I put
forth alone—and even if I perish shame you all."
And, throwing aside his sword and short mantle, the young man
rushed down the steep pathway that led to the little pier, and leaped
on board one of the long light whaleboats that lay there; but ere his
ready hand had quite cast off the rope that bound it to a ringbolt on
the mole, both Hans Knuber and Jans Thorson, fired by his example,
sprang on board, and with more of the action of elephants, in their
wide fur boots and mighty breeches, than the agility of seamen, they
seized each an oar, and pushed off.
In Denmark and Norway, there were and are few titles of honour;
but there has always existed in the latter an untitled nobility, like our
Scottish lairds and English squires, consisting of very old families, who
are more highly revered than those ennobled by Norway's Danish
rulers; and many of these can trace their blood back to those terrible
vikingr or ocean kings, who were so long the conquerors of the
English Saxons, and the scourge of the Scottish shores.
Konrad of the Saltzberg (for he had no other name than that
which he took from a solitary and half-ruined tower overlooking the
fiord) was the representative of one of those time-honoured races.
The fame his brave ancestors had won under the enchanted
banner of Regner Lodbrog, Erick with the bloody axe, and Sigwardis
Ring, yet lived in the songs and stories of the northern harpers; and
Konrad was revered for these old memories of Norway's ancient days,
while his own bravery, affability, and handsome exterior, gained him
the love of the Norse burghers of Bergen, the Danish bowmen he
commanded, the fishermen of the fiord, and the huntsmen of the
woods of Aggerhuis.
By the glare of the beacon on the castle wall, his boat was briefly
seen amid the deepening gloom as it rose on the heaving swell, and
the broad-bladed oars of his lusty companions flashed as they were
dipped in the sparkling water. A moment, and a moment only, they
were visible; Konrad was seen to move his plumed cap, and his
cheerful hallo was heard; the next, they had vanished into obscurity.
The fishers gazed on the gloom with intensity, but could discover
nothing; and there was no other sound came on the bellowing wind,
save the roar of the resounding breakers as they broke on the
impending bluffs.
CHAPTER II.
ERICK ROSENKRANTZ.
Turn round, turn round, thou Scottish youth,
Or loud thy sire shall mourn;
For if thou touchest Norway's strand,
Thou never shalt return.
Vedder.
The hall of the castle of Bergen was a spacious but rude apartment,
spanned by a stone arch, ribbed with massive groins, that sprung
from the ponderous walls.
Its floor was composed of oak planks, and two clumsy stone
columns, surmounted by grotesque capitals, supported the round
archway of the fireplace, above which was a rudely carved, and still
more rudely painted, shield, bearing the golden lion of ancient
Norway in a field gules. Piled within the arch lay a heap of roots and
billets, blazing and rumbling in the recesses of the great stone
chimney. Eight tall candles, each like a small flambeau, flared in an
iron candelabrum, and sputtered in the currents of air that swept
through the hall.
Various weapons hung on the rough walls of red sandstone;
there were heavy Danish ghisannas or battle-axes of steel, iron mauls,
ponderous maces, and deadly morglays, two-handed swords of
enormous length, iron bucklers, chain hauberks, and leathern
surcoats, all of uncouth fashion, and fully two hundred years behind
the arms then used by the more southern nations of Europe.
The long table occupying the centre of the hall was of wood that
had grown in the forests of Memel; it was black as ebony with age,
and the clumsy chairs and stools that were ranged against the walls
were all of the same homely material. Several deerskins were spread
before the hearth, and thereon reposed a couple of shaggy wolf-
hounds, that ever and anon cocked their ears when a louder gust
than usual shook the hall windows, or when the rain swept the
feathery soot down the wide chimney to hiss in the sparkling fire.
Near the hearth stood a chair covered with gilded leather, and
studded with brass nails; and so different was its aspect from the rest
of the unornamented furniture, that there was no difficulty in
recognising it as the seat of state. A long sword, the silver hilt of
which was covered with a curious network of steel, hung by an
embroidered baldrick on one knob thereof, balanced by a little velvet
cap adorned with a long scarlet feather, on the other.
The proprietor of these articles, a stout old man, somewhere
about sixty-five, whose rotundity had been considerably increased by
good living, was standing in the arched recess of a well-grated
window, peering earnestly out upon the blackness of the night, in
hope to discern some trace of that strange vessel, concerning which
all Bergen was agog. His complexion was fair and florid, and though
his head was bald and polished, the long hair that hung from his
temples and mingled with his bushy beard and heavy mustaches,
was, like them, of a decided yellow; but his round visage was of the
ruddiest and most weatherbeaten brown. There was a bold and frank
expression in his keen blue eye, that with his air and aspect forcibly
realized the idea of those Scandinavian vikingr who were once the
tyrants of Saxons, and the terror of the Scots.
His flowing robe of scarlet cloth, trimmed with black fur and laced
with gold, his Norwayn anlace or dagger, sheathed in crimson leather
sown with pearls, and the large rowelled spurs that glittered on the
heels of his Muscovite leather boots, announced him one of Norway's
untitled noblesse. He was Erick Rosenkrantz of Welsöö, governor of
the province of Aggerhuis, castellan of Bergen, and knight of the
Danish orders of the Elephant and Dannebrog.
"Sueno Throndson," said he to a little old man who entered the
hall, muffled in a mantle of red deerskin, which was drenched with
rain, "dost thou think there is any chance of yonder strange bark
weathering the storm, and getting under the lee of our ramparts?"
"I know not, noble sir," replied Sueno, casting his drenched cloak
on the floor, and displaying his under attire, which (saith the Magister
Absalom Beyer, whose minute narrative we follow) consisted of a
green cloth gaberdine, trimmed with the fur of the black fox, and girt
at the waist by a broad belt, sustaining a black bugle-horn and short
hunting sword. "I have serious doubts; for the waves of the fiord are
combating with the currents from the Skager Rack, and whirling like a
maelstrom. I have been through the whole town of Bergen; but
neither offer nor bribe—no, not even the bishop's blessing, a hundred
pieces of silver, and thrice as many deer hides—will induce one of the
knavish fishermen or white-livered pilots to put forth a boat to pick up
any of these strangers, who must all drown the moment their ship
strikes; and strike she must, if the wind holds."
"The curse of Saint Olaus be on them!" grumbled the governor,
glancing at a rude image of Norway's tutelary saint.
"Amen!" added Sueno, as he wrung the wet tails of his
gaberdine.
"Didst thou try threats, then?"
"By my soul, I did so; and with equal success."
"Dost thou gibe me, Throndson? This to me, the governor of
Aggerhuis, and captain of the king's castle of Bergen!" muttered the
portly official, walking to and fro, and swelling with importance as he
spoke.
"The oldest of our fishermen are ready to swear on the blessed
gospels that there has not been seen such a storm since Christian
Alborg, in the Biornen, was blown from his moorings."
"Under the ramparts of this, the king's castle, by foul sorcery;
and on the vigil of Saint Erick the king, and martyr too! I remember it
well, Sueno. But what! is the old Norse spirit fallen so far, that these
villains have become so economical of their persons that they shrink
from a little salt water? and that none will launch a shallop in such to
save these poor strangers, who, unless they know the coast, will
assuredly run full tilt on the Devil's Nose at the haven mouth? By
Saint Olaus! I can see the white surf curling over its terrible ridge,
through the gloom, even at this moment."
"I said all this, noble sir," replied Sueno, brushing the rain from
his fur bonnet; "but none attended to me save young Konrad of the
Saltzberg, the captain of our Danish crossbowmen, who cursed them
for white-livered coistrils, and launching a boat, with Hans Knuber and
Jans Thorson the pilot, pushed off from the mole, like brave hearts as
they are, in the direction of the labouring ship, which Konrad vowed
to pilot round the Devil's Nose or perish."
"Fool! and thou only tellest me of this now! Konrad—the boldest
youth and the best in all old Norway!" exclaimed the burly governor.
"Hah! and hath the last of an ancient and gallant race to peril his life
on such a night as this, when these baseborn drawers of nets and
fishers of seals hang back?"
"His boat vanished into the gloom in a moment, and we heard
but one gallant blast from his bugle ring above the roar of the waves
that boil round that terrible promontory."
"The mother of God pray for him—brave lad! What the devil!
Sueno, I would not for all the ships on the northern seas, a hair of
Konrad's head were injured; for though he is no kin to me, I love the
lad as if he were mine own and only son. See that my niece Anna
knoweth not of this wild adventure till he returns safe. She has
seemed somewhat cold to him of late; some lover's pique"——
"I pray he may return, Sir Erick."
"He must—he shall return!" rejoined the impetuous old knight,
stamping his foot. "Yea, and in safety too, or I will sack Bergen, and
scourge every fisher in it. From whence thought these knaves the
stranger came?"
"From Denmark."
"Malediction on Denmark!" said Rosenkrantz, feeling his old
Norse prejudices rising in his breast. "Assure me that she is Danish,
and I will extinguish the beacon and let them all drown and be——!"
"Nay, nay, Sir Governor, they know her to be a good ship of
Scotland, commanded by a certain great lord of that country, who is
on an embassy to Frederick of Denmark, and hath been cruising in
these seas."
"Then my double malediction on the Scots, too!" said the
governor, as he turned away from the hall window.
"And so say I, noble Sir," chimed in the obsequious chamberlain,
as he raised the skirts of his gaberdine, and warmed his voluminous
trunk hosen before the great fire.
"Right, Throndson! though eight of our monarchs are buried in
Iona, under the Ridge of the Kings, the death of Coelus of Norway,
who is graved in the Scottish Kyles, still lives in our songs; and the
fatal field of Largs, when aided by such a storm as this, the Scots laid
Haco's enchanted banner in the waves."
"And the wars of Erick with the bloody axe."
"And of Harold Graafeldt, his son."
"And Magnus with the Barefeet," continued the old man, whose
eyes gleamed at the names of these savage kings of early
Scandinavia.
"Enough, Sueno," said the governor, who was again peering from
the window into the darkness; "enough, or thou wilt fire my old Norse
heart in such wise by these fierce memories, that no remnant of
Christian feeling will remain in it. After all, it matters not, Scots or
Danes, we ought to pray for the souls that are now perhaps, from
yonder dark abyss, ascending to the throne of God unblessed and
unconfessed," added the old knight, with a sudden burst of religious
feeling.
"God assoil them!" added Sueno crossing himself, and becoming
pious too.
From the windows of the hall little else was seen but the dark
masses of cloud that flew hither and thither on the stormy wind; at
times a red star shot a tremulous ray through the openings, and was
again hidden. Far down, beneath the castle windows, boiled the fierce
ocean, and its white foam was visible when the lofty waves reared up
their crested heads to lash the impending cliffs; but we have said that
the bosom of the harbour was smooth as a summer lake when
compared with the tumult of the fiord of Christiana. Overhead,
showers of red sparks were swept away through the gloom, from the
beacon that blazed on the keep to direct the waveworn ship.
"What led Hans Knuber and his brother knave of the net, to
deem the stranger was a Scot? By her lumbering leeboard I would
have sworn she was a Lubecker."
"Nay, Sir, her high fore and after castles marked her Scottish
build; and both Hans Knuber and Jans Thorson, who have eyes for
these matters, and have traded to Kirkwall—yea, and even to that
Scottish sea the fiord of Forth—averred she bore Saint Andrew's
saltire flying at her mizen-peak——I see nothing of her now,"
continued Sueno.
"See! why, 'tis so dark, one cannot see the length of one's own
nose. They must have perished!"
At that moment the flash of a culverin glared amid the obscurity
far down below; but its report was borne away on the wind that
roared down the narrow fiord to bury its fury in the Skager Rack.
"God and St. Olaus be praised!" muttered the old knight, rubbing
his hands: "they are almost within the haven mouth; another
moment, and they will be safe."
"Thou forgettest, noble sir," said the chamberlain, "that the
stranger's pilot may be unacquainted with the nooks and crooks of
our harbour, the rocks and reefs that fringe it, and that the water in
some parts is two hundred fathoms deep."
"Saidst thou not that Konrad and Hans Knuber had put off in a
boat?"
"True, true! A ray of light is shining on the water now."
"Whence comes it?"
"'Tis the hermit in the cavern under the rocks, who hath lit a
beacon on the beach to direct the benighted ship."
"Saint Olaf bless him! Hoh! there goeth the culverin again. We
heard the report this time. They are saved! 'Tis Konrad of Saltzberg
hath done this gallant deed, and heaven reward him! for many a poor
fellow had perished else. Now that they are in safe anchorage, away
Sueno Throndson, take thy chamberlain's staff and chain, man a boat,
board this seaworn ship, and invite this Scottish lord to Bergen; for a
foul shame it were in a knight of the Elephant, to permit the
ambassador of a queen, to remain on shipboard after such a storm,
and within a bowshot of his Danish majesty's castle: we would be
worse than Finns or Muscovites. Away, Sueno! for now the storm is
lulling, and under the lee of its high hills the harbour is smooth as a
mirror."
Thus commanded, Sueno unwillingly enveloped himself once
more in the before-mentioned fur mantle, and retired.
A blast of his horn was heard to ring in the yard as he summoned
certain followers, who grumbled and swore in guttural Norse as they
scrambled after him down the steep and winding pathway, that led
from the castle gate to the mole of Bergen.
CHAPTER III.
THE STRANGERS.
To tell the terrors of deep untried,
What toils we suffer'd, and what storms defied;
What mountain surges, mountain surges lash'd,
What sudden hurricanes the canvass dash'd;
To tell each horror in the deep reveal'd,
Would ask an iron throat with tenfold vigour steel'd.
Lusiad of Camoens.
"How now, Anna! thou lookest as pale as if all the gnomes of the
Silverbergen, or Nippen and Zernebok to boot, had been about thee.
Art thou affrighted by the storm, child?" asked Erick, pinching the soft
cheek of his niece, who at that moment had entered the hall, and
glided to his side in one of the great windows.
Her only reply was to clasp her hands upon his arm, and look up
in his face with a fond smile.
Anna Rosenkrantz was the only daughter of Svend of Aggerhuis,
the governor's younger brother, who had fallen in battle with the
Holsteiners. In stature she was rather under the middle height; and
so full and round was her outline, that many might have considered it
too much so, but for the exquisite fairness of her skin, the beauty of
her features, and the grace pervading every motion. Norway is famed
for its fair beauties, but the lustre of Anna's complexion was dazzling;
her neck and forehead were white as the unmelting snows of the
Dovrefeldt. From under the lappets of a little velvet cap, which was
edged by a row of Onslo pearls, her dark-brown ringlets flowed in
heavy profusion, and seemed almost black when contrasted with the
neck on which they waved. Her eyes were of a decided grey, dark, but
clear and sparkling. The curve of her mouth and chin were very
piquant and arch in expression; her smile was ever one of surpassing
sweetness, and at times of coquetry.
A jacket of black velvet, fashioned like a Bohemian vest, trimmed
with narrow edgings of white fur, and studded with seed pearls,
displayed the full contour of her beautiful bust; but unhappily her skirt
was one of those enormous fardingales which were then becoming
the rage over all Europe.
"Have the roaring of the wind and the screaming of the water-
sprite scared thee, Anna?" continued the old man, who, like a true
Nordlander, believed every element to be peopled by unseen spirits
and imps. "By the bones of Lodbrog!" he added, patting her soft
cheek with his huge bony hand, "my mind misgave me much that this
last year's sojourn at the palace of Kiobenhafen would fairly undo
thee."
"How, good uncle?" said Anna, blushing slightly.
"By tainting thine inbred hardiment of soul, my little damsel, and
making thee, instead of a fearless Norse maiden, and a dweller in the
land of hills and cataracts, like one of those sickly moppets whom I
have seen clustered round the tabouret of Frederick's queen, when,
for my sins, I spent a summer at his court during the war with
Christian II., that tyrant and tool of the Dutch harlot, Sigiberta."
"Indeed, uncle mine, you mistake me," replied Anna, "though I
will own myself somewhat terrified by this unwonted storm."
"There now! said I not so? Three years ago, would the screaming
of the eagles, the yelling of the wood-demon, the howl of the wind, or
the tumult of the ocean, when all the spirits of the Skager Rack are
rolling its billows on the rocks, have affrighted thee? Bah! what is
there so terrible in all that? Do not forget, my girl, that thou comest of
a race of sea-kings who trace their blood from O'Ivarre—he who with
Andd and Olaff ravaged all the Scottish shores from Thurso to the
Clyde, and once even placed the red lion of Norway on the double
dun of Alcluyd.[*] But I warrant thou art only terrified for young
Konrad, who, like a gallant Norseman, hath run his life into such
deadly peril."
[*] A.D. 870 (Note by Mag. Absalom Beyer.)
"Konrad—tush!" said Anna pettishly.
"Ay, Konrad!" reiterated Erick testily; "which way doth the wind
blow now? By my soul, damosel, thou takest very quietly the danger
in which the finest young fellow in all Norway has thrust himself—
when even the boldest of our fishers drew back. He departed in a
poor shallop to guide yonder devilish ship round the dangerous
promontory, and if the blessed saints have not prevailed over the
spirits of evil, who make their bourne in the caverns of that dark
ocean—then I say, God help thee, Konrad of Saltzberg! But fear not,
Anna," continued the old man kindly, perceiving that she turned away
as if to conceal tears; "for thy lover is stout of heart and strong of
hand—and—there now!—the devil's in my old gossiping tongue—pest
upon it!—I have made thee weep."
Anna's breast heaved very perceptibly, and she covered her face,
not to conceal her tears, but the smile that spread over her features.
"Come, damosel—away to thy toilet; for know there is in yonder
ship which we have watched the livelong day, and which has escaped
destruction so narrowly, a certain great lord, who this night shall sup
with us; for I have sent Sueno with a courteous message, inviting him
to abide, so long as it pleases him, in the king's castle of Bergen. Be
gay, Anna; for I doubt not thou wilt be dying to hear tidings of what is
astir in the great world around Aggerhuis; for, during the last month
since thy return here, thou hast moped like some melancholy oyster
on the frozen cape yonder."
"A great lord, saidst thou, uncle?" asked Anna with sudden
animation.
"Of Scotland—so said Sueno."
Anna blushed scarlet; but the momentary expression of confusion
was replaced by one of pride and triumph.
"Did thou hear of any such at Frederick's court, little one?"
"Yes—oh yes! there were two on an embassy concerning the isles
of Shetland."
"Ah! which that fool, Christian of Oldenburg, gave to the Scottish
king with his daughter Margaret? Their names?"
"I marked them not," replied Anna with hesitation; "for thou
knowest, uncle mine, I bear no good-will unto these rough-footed
Scots."
"Keep all thy good-will for the lad who loves thee so well," said
the old man smiling, as he pressed his wiry mustaches against her
white forehead. "I see thou hast still the old Norse spirit, Anna.
Though three centuries have come and gone since the field of Largs
was lost by Haco and his host, we have not forgotten it; and
vengeance for that day's slaughter and defeat still forms no small item
in our oaths of fealty and of knighthood. But hark! the horn of Sueno!
There are torches flashing on the windows, and strange voices
echoing, in the court. Away, girl! and bring me my sword and collars
of knighthood from yonder cabinet; for I must receive these guests as
becomes the king's representative at Aggerhuis, and captain of his
castle of Bergen."
Anna glided from his side, and in a minute returned with a casket
from the cabinet, and the long heavy sword that lay on the chair at
the fireplace. She clasped the rich waistbelt round the old man's burly
figure, and drawing from the casket the gold chain with the diamond
Elephant, having under its feet the enamelled motto,
"Trew is Wildbrat,"—
and the woven collar bearing the red cross of the Dannebrog, she
placed them round Sir Brick's neck, and the jewels sparkled brightly
among the red hair of his bushy beard.
She then glanced hurriedly at her own figure in an opposite
mirror; adjusted the jaunty little cap before mentioned; ran her
slender fingers through her long dark ringlets; smiled with satisfaction
at her own beauty; and took her seat on a low tabouret near the
great stuffed chair, between the gilded arms of which the pompous
old governor wedged his rotund figure, with an energy that made his
visage flush scarlet to the temples; and he had barely time to assume
his most imposing aspect of official dignity, when the light of several
flambeaux flashed through the dark doorway at the lower end of the
hall, and the handsome commander of his crossbowmen, Konrad of
Saltzberg, with his features pale from fatigue, and his long locks, like
his furred pelisse, damp with salt water, and Sueno wearing his gold
chain and key, having his white wand uplifted, and attended by
several torch-bearers in the king's livery, preceded the strangers.
The first who approached was a tall and handsome man, in
whose strong figure there was a certain jaunty air, that suited well the
peculiar daredevil expression of his deep dark eye, which bespoke the
confirmed man of pleasure. He seemed to be about thirty years of
age, and was clad in a shining doublet of cloth of gold, over which he
wore a cuirass of the finest steel, attached to the backplate by braces
of burnished silver. His mantle was of purple velvet lined with white
satin; his trunk breeches were of the latter material slashed with
scarlet silk, and were of that enormous fashion then so much in
vogue, being so preposterously stuffed with tow, hair, or bombast, as
to render even greaves useless in battle. He wore a long sword and
Scottish dagger. His blue velvet bonnet was adorned by a diamond
aigrette, from which sprung three tall white ostrich feathers. His eyes
were keen, dark, and proud, and their brows nearly met over his
nose, which was straight; he wore little beard, but his mustaches
were thick and pointed upward. His page, a saucy-looking lad of
sixteen, whom he jocularly called Nick (for his name was Nicholas

More Related Content

PDF
Data Management at Scale, Second Edition Piethein Strengholt
PDF
The analytics-stack-guidebook
PDF
The Analytics Stack Guidebook (Holistics)
PDF
Data Management at Scale Piethein Strengholt
PDF
Agnostic Tool Chain Key to Fixing the Broken State of Data and Information Ma...
Data Management at Scale, Second Edition Piethein Strengholt
The analytics-stack-guidebook
The Analytics Stack Guidebook (Holistics)
Data Management at Scale Piethein Strengholt
Agnostic Tool Chain Key to Fixing the Broken State of Data and Information Ma...

Similar to Data Mesh in Action (MEAP V04) Jacek Majchrzak (20)

PDF
Mighty Guides- Data Disruption
DOCX
BIAM 410 Final Paper - Beyond the Buzzwords: Big Data, Machine Learning, What...
DOCX
Data governance for now
PDF
How the Journey to Modern Data Management is Paved with an Inclusive Edge-to-...
PDF
Document Based Data Modeling Technique
PDF
The IT Intelligence Foundation For Digital Business Transformation Builds fro...
PDF
Gerenral insurance Accounts IT and Investment
PDF
Complex Carrier Network Performance Data on Vertica Yields Performance and Cu...
PDF
A Primer for a layman about Big Data, Business Analytics and Cloud
DOCX
Discussion post· The proper implementation of a database is es.docx
PDF
How 3 trends are shaping analytics and data management
PDF
Streamlining Your Path to Metadata Charlotte Robidoux Stacey Swart
PDF
Learning Azure Synapse Analytics (Third Early Release) Paul Andrew
PPTX
SegmentOfOne
PDF
Enterprise Search White Paper: Beyond the Enterprise Data Warehouse - The Eme...
PDF
Harness the power of data
PDF
Big Data
PPTX
Data lake ppt
DOCX
A Practical Guide to Data Mesh Architecture Blog.docx
Mighty Guides- Data Disruption
BIAM 410 Final Paper - Beyond the Buzzwords: Big Data, Machine Learning, What...
Data governance for now
How the Journey to Modern Data Management is Paved with an Inclusive Edge-to-...
Document Based Data Modeling Technique
The IT Intelligence Foundation For Digital Business Transformation Builds fro...
Gerenral insurance Accounts IT and Investment
Complex Carrier Network Performance Data on Vertica Yields Performance and Cu...
A Primer for a layman about Big Data, Business Analytics and Cloud
Discussion post· The proper implementation of a database is es.docx
How 3 trends are shaping analytics and data management
Streamlining Your Path to Metadata Charlotte Robidoux Stacey Swart
Learning Azure Synapse Analytics (Third Early Release) Paul Andrew
SegmentOfOne
Enterprise Search White Paper: Beyond the Enterprise Data Warehouse - The Eme...
Harness the power of data
Big Data
Data lake ppt
A Practical Guide to Data Mesh Architecture Blog.docx
Ad

Recently uploaded (20)

PPTX
Pharma ospi slides which help in ospi learning
PDF
Module 4: Burden of Disease Tutorial Slides S2 2025
PDF
Chapter 2 Heredity, Prenatal Development, and Birth.pdf
PDF
O7-L3 Supply Chain Operations - ICLT Program
PDF
Basic Mud Logging Guide for educational purpose
PDF
Supply Chain Operations Speaking Notes -ICLT Program
PPTX
BOWEL ELIMINATION FACTORS AFFECTING AND TYPES
PPTX
Cell Structure & Organelles in detailed.
PDF
102 student loan defaulters named and shamed – Is someone you know on the list?
PDF
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH 9 GLOBAL SUCCESS - CẢ NĂM - BÁM SÁT FORM Đ...
PDF
Pre independence Education in Inndia.pdf
PDF
ANTIBIOTICS.pptx.pdf………………… xxxxxxxxxxxxx
PDF
01-Introduction-to-Information-Management.pdf
PDF
Business Ethics Teaching Materials for college
PDF
Anesthesia in Laparoscopic Surgery in India
PDF
Physiotherapy_for_Respiratory_and_Cardiac_Problems WEBBER.pdf
PPTX
Introduction to Child Health Nursing – Unit I | Child Health Nursing I | B.Sc...
PDF
3rd Neelam Sanjeevareddy Memorial Lecture.pdf
PPTX
Institutional Correction lecture only . . .
PDF
Classroom Observation Tools for Teachers
Pharma ospi slides which help in ospi learning
Module 4: Burden of Disease Tutorial Slides S2 2025
Chapter 2 Heredity, Prenatal Development, and Birth.pdf
O7-L3 Supply Chain Operations - ICLT Program
Basic Mud Logging Guide for educational purpose
Supply Chain Operations Speaking Notes -ICLT Program
BOWEL ELIMINATION FACTORS AFFECTING AND TYPES
Cell Structure & Organelles in detailed.
102 student loan defaulters named and shamed – Is someone you know on the list?
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH 9 GLOBAL SUCCESS - CẢ NĂM - BÁM SÁT FORM Đ...
Pre independence Education in Inndia.pdf
ANTIBIOTICS.pptx.pdf………………… xxxxxxxxxxxxx
01-Introduction-to-Information-Management.pdf
Business Ethics Teaching Materials for college
Anesthesia in Laparoscopic Surgery in India
Physiotherapy_for_Respiratory_and_Cardiac_Problems WEBBER.pdf
Introduction to Child Health Nursing – Unit I | Child Health Nursing I | B.Sc...
3rd Neelam Sanjeevareddy Memorial Lecture.pdf
Institutional Correction lecture only . . .
Classroom Observation Tools for Teachers
Ad

Data Mesh in Action (MEAP V04) Jacek Majchrzak

  • 1. Data Mesh in Action (MEAP V04) Jacek Majchrzak install download https://ebookmeta.com/product/data-mesh-in-action-meap-v04-jacek- majchrzak/ Download more ebook from https://ebookmeta.com
  • 2. We believe these products will be a great fit for you. Click the link to download now, or visit ebookmeta.com to discover even more! Data Mesh in Action 1st Edition Jacek Majchrzak Sven Balnojan Marian Siwiak https://ebookmeta.com/product/data-mesh-in-action-1st-edition- jacek-majchrzak-sven-balnojan-marian-siwiak/ Everyday Data Visualization (MEAP V04) Desiree Abbott https://ebookmeta.com/product/everyday-data-visualization- meap-v04-desiree-abbott/ Data Mesh Delivering Data Driven Value at Scale 1st Edition Zhamak Dehghani https://ebookmeta.com/product/data-mesh-delivering-data-driven- value-at-scale-1st-edition-zhamak-dehghani/ Unfree Migrant Domestic Work in Arab States 1st Edition Rhacel Salazar Parreñas https://ebookmeta.com/product/unfree-migrant-domestic-work-in- arab-states-1st-edition-rhacel-salazar-parrenas/
  • 3. Lock Mercy 3 1st Edition Debra Anastasia Anastasia Debra https://ebookmeta.com/product/lock-mercy-3-1st-edition-debra- anastasia-anastasia-debra/ Multiphysics Modeling Using COMSOL 5 and MATLAB 2nd Edition Pryor Phd https://ebookmeta.com/product/multiphysics-modeling-using- comsol-5-and-matlab-2nd-edition-pryor-phd/ Stress Management Through Mind Engineering 1st Edition R.P. Banerjee https://ebookmeta.com/product/stress-management-through-mind- engineering-1st-edition-r-p-banerjee/ Democratizing the Economics Debate Pluralism and Research Evaluation 1st Edition Carlo D Ippoliti https://ebookmeta.com/product/democratizing-the-economics-debate- pluralism-and-research-evaluation-1st-edition-carlo-d-ippoliti/ Off The Path Beyond the Merillian 3 1st Edition Courtney Leigh https://ebookmeta.com/product/off-the-path-beyond-the- merillian-3-1st-edition-courtney-leigh/
  • 4. Fundamentals of sustainable business : a guide for the next 100 years Second Edition Matthew Tueth https://ebookmeta.com/product/fundamentals-of-sustainable- business-a-guide-for-the-next-100-years-second-edition-matthew- tueth/
  • 6. ©Manning Publications Co. To comment go to liveBook MEAP Edition Manning Early Access Program Data Mesh in Action Version 4 Copyright 2022 Manning Publications For more information on this and other Manning titles go to manning.com
  • 7. ©Manning Publications Co. To comment go to liveBook welcome Thank you for purchasing the MEAP edition of Data Mesh in Action. We’re excited to already be able to share some parts of this book with you as it still is being actively written! Since we’re still writing on it, this is also your chance to leave us some comments to make the final book even better. Data Mesh is one of the hottest topics in the data world right now. However, the growing number of blog articles and discussions in the communities show growing dispersion in Data Mesh understanding, mainly based on theoretical debates and considerations. This book is trying to bring the Data Mesh world together by: 1. focusing on applied examples all the way throughout the book, including a long case study as well as multiple real-world examples, 2. bringing structure to the discussion based on ideas that have been around for a long time like Socio-Technical Architecture or Domain-Driven Design, 3. and finally by providing an open-minded and flexible approach, with a focus on making data valuable for every company, without fixed roles, fixed definitions, or dogmas. We know from (at least) three different perspectives, that the development of Data Mesh touches a lot of different aspects of the organization and usually starts at very different places on the data maturity scale. It involves organizational structure, technology stack, and different levels of architecture. It can start in any size of a company, be it a company of 20, or a company of 1,000. The data maturity levels can differ inside the areas of operation of the company. So, we decided to write a book about how the Data Mesh implementation may look from different perspectives. We’re trying to achieve this by always highlighting that almost any part of the Data Mesh, be it the governance policies or the data platform itself are not fixed concepts but rather points on a spectrum where you can choose the right one for you and your situation. To make it even more applied, we created a fictional company, in which we walk from zero to hero in their data maturity, offering you insight into a set of cohesive, step-by-step examples. To take full advantage of this book, you would be directly or indirectly involved in creating, managing, and utilizing data within your organization. Be that as an architect, a leader in the data or software space, a developer inside a data-focused team, or on the data-consuming side of things, like an analyst, a decision-maker, or a machine learner. We hope you enjoy Data Mesh in Action and that it will occupy an important place on your bookshelf.
  • 8. ©Manning Publications Co. To comment go to liveBook We also encourage you to post any questions or comments you have about the content in the liveBook Discussion forum. We appreciate knowing where we can make improvements and increase your understanding of the material. - Marian Siwiak, Jacek Majchrzak, and Sven Balnojan
  • 9. ©Manning Publications Co. To comment go to liveBook brief contents Front matter PART 1: FOUNDATIONS 1 The What and Why of Data Mesh 2 Is Data Mesh Right for You? 3 Kickstart your Data Mesh MVP in a month PART 2: THE FOUR PRINCIPLES IN PRACTICE 4 Domain ownership 5 Data as a Product 6 Federated Computational Governance 7 The self-serve data platform PART 3: INFRASTRUCTURE AND TECHNICAL ARCHITECTURE 8 Self-Serve Data Platforms Applications in Comparison 9 Solution architecture design
  • 10. ©Manning Publications Co. To comment go to liveBook Front matter preface All three of us authors experienced at length, at different companies, the old way of “doing data”, usually through centralized data lakes and data warehouses in combination with a set of central teams organized inside an “analytics function” to handle this. The old way basically looked like this: 1. Multiple Decentralized development teams have data that is accessible through its storage systems like a shared drive, a decentralized database, a REST API, or any other interface. 2. One or more centralized data teams are tasked with collecting this data into one monolithic pot. This is either a data lake or a data warehouse. 3. The same set of teams is tasked with transforming this data into something useful. 4. Multiple decentralized analysts, development teams or machine learning teams pick up that transformed data and transform this data finally into value in the form of reports, recommendation systems, or anything else they can think of. All three of us learned the hard way how this concept has its limits, producing a bottleneck both in terms of technology as well as in terms of the team capacities. We all saw how companies all around us were struggling to get the flow from data to value to be as productive as the companies needed it to be. Then the Data Mesh and the ideas behind it appeared on the horizon. The Data Mesh is a decentralization paradigm. It decentralizes the ownership of data, its transformation into information as well as its serving. It aims to increase the value extraction from data by removing bottlenecks in the data value stream by these means. The concept of the Data Mesh appeared on the stage in 2019 and since has lit not just the data world, but the whole technology world on fire. The Data Mesh concept breaks with the current world of data where data is usually treated as a “by-product” of software components. It turns the spotlight on the producers of the data and gives them the responsibility to handle this data just as they would handle their software. With this, the data mesh takes the same journey software components have taken, with microservices architectures and with the DevOps movement. It takes the same journey frontends are currently taking with what is called “microfrontends”. And just as in these previous examples, we believe that the Data Mesh approach is the right approach to finally 1
  • 11. ©Manning Publications Co. To comment go to liveBook gain the flexibility to extract value from our data at scale, be that in business intelligence, machine learning, or any other use case you can think of. The Data Mesh concept is often referred to as a socio-technical paradigm shift, meaning the core of it is not about technology but about the alignment of people, processes, and organization. This significant complexity is why we wrote this book. However, we didn’t just try to bring together the available theoretical knowledge that is out there about the Data Mesh; we focused on parts that are, in our experience, critical for successful implementation. We organized those parts into a digestible resource to help you, the reader, put Data Mesh in Action! To guide you through the process, we prepared hands-on examples with a lot of architecture sketches, describing different technologies, workshop techniques, team organization forms, and the like. After reading this book you should be able: 1. evaluate if Data Mesh will suit your organization’s business needs, 2. lay the groundwork for Data Mesh development, 3. develop a minimal Data Mesh to start their journey, 4. keep iteratively developing and expanding your Data Mesh. Don’t expect to find a lot of code in this book though, other than a few JSONs here and there. That’s because we truly believe the magic is not in the technology, but in the people, processes, and the organization. But of course, you can expect to find a lot of technology inside this book in the form of very deep architecture sketches with reference to various technologies and cloud providers, explanations, and blueprints inspired by multiple real-world examples. That said, we don’t believe in a black-and-white implementation of the data mesh idea. For that reason, we wrote a book that will help you adjust the data mesh idea to your company by offering a lot of degrees of freedom, shortcuts, and a healthy level of pragmatism. To tie together our experience we will use an imaginary company called Messflix LLC, which resembles a lot of what we’ve seen out there in the data world. This case study will guide you through all chapters of the book. Toward the end of this front matter, we provide a brief introduction to Messflix LLC by taking a look at the data mess the company has gotten itself into. acknowledgments First, we would like to express our gratitude to the community engaged with Data Mesh development. Their discussions, openness about problems and issues helped us broaden our perspectives and put our particular experiences into the generalized framework you’ll find in this book. We owe our thanks to the wonderful people at Manning who made this book possible: Publisher Marjan Bace, Development Editor Ian Hough, and last, but not least Acquisitions Editor Andrew Waldron. Without their patience with our ever-evolving view on Data Mesh, and ability to make us synthetize it into a coherent view, we wouldn’t be able to finish Data Mesh in Action in a form we could so proudly present to you. We would like also to thank 2
  • 12. ©Manning Publications Co. To comment go to liveBook marketing, editorial and production teams without whom this book would gather dust in Manning’s drawer. A heartfelt thanks also to Michael Jensen and Al Kinker for technical reviews, which allowed us to further condense and clarify Data Mesh concepts. We would also like to thank all our reviewers, who trusted us and invested their time in reading this book, even when no one was sure if it’ll become read-worthy. about this book This book is intended to serve two purposes. First, it is intended to collect and organize the knowledge about the new socio-technological paradigm of Data Mesh. Second, it’s supposed to help the reader in Data Mesh implementation. From consideration as to whether Data Mesh is a suitable solution for their organization, to laying the groundwork, to development of MVP, to implementation of Data Mesh principles, we hope that this book will provide you with the tools needed to get well on your way on your data mesh journey. Who should read this book The most general description of our reader is someone who is involved in extracting value from data. However, as in modern economy it describes almost anyone, let’s try to see what benefits will it bring to different audiences. The first group, is people involved in creating, managing, and utilizing data within a companies of: • high socio-technological complexity (e.g., big corporations), • complex data use cases, • many and diverse data sources. This would encompass, but not be limited to, roles like: data architects, data engineers, software architects, tech leads, senior developers. The more you feel like these quantifiers apply to your business, the more likely it is that Data Mesh could be a good solution. This book will help you understand Data Mesh concepts, whose cooperation you need to secure, and what steps to take in both your organization and technical environment are needed to move from data mess to Data Mesh. Beyond that, as the data mesh is a company-wide transformation process, a the book's content will be directly useful to executive level personnel, like technical C-suite, engineering directors/ managers, enterprise architects, chief and lead architects, solution/program owners. This book will help you decide in what extent and priorities should you shift your company’s data environment into Data Mesh direction, and help you plan the change management. How to Navigate this Book While the book is meant to be read linearly, it is broken into four main parts and allows you to skip different sections. The first part is a quick and hands-on introduction, the second explains the four principles of the data mesh in detail, the third tackles the technical side of things in detail, and the fourth focuses on the complete enterprise journey. 3
  • 13. ©Manning Publications Co. To comment go to liveBook PART 1: FOUNDATIONS The goal of the first part of the book is to get you familiarized with the data mesh paradigm as quickly as possible. To do so we first go through the basics of the data mesh and then get our hands dirty by building our first data mesh within a month. Chapter 1: The What and Why of Data Mesh This chapter gives the overview needed to put the rest of the book into the proper context, including why you might want to consider following the data mesh mindset shift as well as a short explanation of the four key principles we dive deeper into in the next part. Chapter 2: Is Data Mesh Right for You? This chapter provides you with the context of the Data Mesh implementation and different drivers to consider when deciding on the transformation. It helps you decide whether you want to start the journey now, and at which point you are on the data maturity scale. This helps you to match your Data Mesh journey to your particular situation. Chapter 3: Kickstart your Data Mesh MVP in a Month This chapter is a hands-on example of how to go about building an MVP. The Messflix MVP focuses a lot on the organizational challenges and stays very light on the technology side of things, which an MVP should. These will be picked up later. The chapter provides you with tools like stakeholder mappings and FAIR principles to get you started. PART 2: THE FOUR PRINCIPLES IN PRACTICE The goal of the second part of the book is to provide you with the tools to tackle the four principles of the data mesh for advancing your data mesh beyond the first month. Chapter 4: Domain Ownership This chapter is all about domains and business capabilities and how data can find its owner inside a company. It provides you with a lot of workshop techniques like domain storytelling applied to Messflix example and the Hitchcock Movie Maker and the likes. Chapter 5: Domain Data as a Product Data is often treated as a “by-product”. The topic of this chapter is about changing to a product perspective called “data as a product”. It provides examples of data products from Messflix LLC and explains concepts like the data product canvas and data ports in detail. Chapter 6: Federated Computational Governance This chapter tackles the concept of data governance in the data mesh context. Inside data meshes, it’s called federated computational governance because both a federated nature of governance and an automated execution are needed to unfold the data mesh character. This chapter contains a discussion of centralized vs. decentralized aspects, hands-on examples from Messflix LLC, and a guide for setting up a governance team. Chapter 7: Self-Serve Data Platform The last chapter on data mesh principles covers the platform, the enabling technology that makes the data mesh work. It works through three iterations on our data platform for 4
  • 14. ©Manning Publications Co. To comment go to liveBook Messflix LLC and explains important concepts like “Platform Thinking” along with these examples. PART 3: INFRASTRUCTURE AND TECHNICAL ARCHITECTURE The third part focuses on all things technical. We break out of the Messflix example to highlight different architectures and discuss multiple different options for moving from your existing structure to a data mesh. Chapter 8: Self-Serve Data Platforms Applications in Comparison This chapter explains different blueprints for data mesh platforms that fit different cloud providers as well as different sizes of companies. Chapter 9: Technical architecture design In this chapter, we focus on the migration from your existing system to various different kinds of architectures step by step and component by component. We talk about data lakes, data warehouses, REST APIs, and more. How to Use this Book We don’t just want to present yet another theory of Data Mesh. This book is more of a structured, collective diary of actions leading to Data Mesh development in different environments. The emphasis is on “actions leading to”. We’ve arrived at Data Mesh after the long and often painful journey through multiple other solutions. Over the years, we’d been testing, researching, discussing, and, last but not least, failing a lot in the process. In this book, we share with you the summary of “I wish someone had told me earlier” insights. We hope you will be able to immediately put the information you’ll get out of it, well, in Action. Depending on your goal, there are a few focal points you could set while reading this book and dive deeper into. If your interest is purely informational, and you might want to explain the concepts to your team, your management, your company, we recommend you put a lot of focus onto chapters 1 and 2, for the quick overview as well as the MVP presented in chapter 4. If, in addition, you will read through chapter 9 for a deeper dive into the reasons for this paradigm shift and a lighter look into Part 2, you will be well equipped to explain the data mesh paradigm to someone else. If you want to launch a larger initiative inside your company, you’ll need to be convincing. In that case, we recommend you take a deep dive into the complete chapter 9 and follow with caution chapter 3, offering the insight into the question of whether you should start this journey at all. Chapter 4, presenting the full-scale Data Mesh MVP development and chapter 2 offering a quick glance into a lightweight application of Data Mesh principles will allow you to balance the big picture view with notes on requirements of quick implementation and getting results fast. All together this material should equip you with enough convincing material to get top-level buy-in. If you’re interested in the technical side of things like automated governance and the self-serve platform, then chapters 5 to 8 will provide you with a lot of interesting content to dig through. 5
  • 15. ©Manning Publications Co. To comment go to liveBook If you work inside a development team, we’d particularly recommend you to turn your attention to chapter 4, as it explains exactly what is broken in the current mode of thinking, and it should also help you advance your ways of working without ever touching the data mesh concept. Additionally we recommend Chapter 8 as it explains possible architecture alternatives for serving data from a development teams’ point of view. If you want to advance the way you work inside your data team, then you could put a focus on chapters 3 & 4 to deeply understand where your current troubles may stem from. You could also focus on chapter 6 to understand what “platform thinking” in a data context means. Both could help you advance your ways of working without actually going full data mesh inside the company. We’re sure there are many more reasons for you to open up this book, these are simply a few possible ways you could go about putting this book into use. The Messflix Case Study To help you conceptualize the practical aspects of putting Data Mesh in action, we combined our experiences and merged them into a single Data Mesh journey of Messflix LLC. Messflix LLC, a movie & TV-show streaming platform, just hit a wall. A data wall. They have all the data in the world but complain about not even being able to build a proper recommendation system for their movies and shows. The competition seems to be able to get it done, in fact, the competition is famous for being the first-movers in a lot of technology sectors. Other companies in equally complex industries seem to be able to put their data to work. Messflix LLC does work with data, analysts are able to get some insights from it, but the organization doesn’t feel like they can call themselves “data-driven”. The data science trial runs seem to all end in “pretty prototypes” with no clear business value. The data scientists tell their managers that it’s because the “product team just doesn’t want to put these great prototypes on the roadmap” or in another instance “because the data from the source is way too messy and inconsistent”. In short, Messflix LLC hopefully sounds like your average business, but for some reason doesn’t feel like it’s able to let the right data flow to the right use cases. The data landscape, just as the technology landscape grew organically over time and became quite complex. The two key technology components of Messflix LLC are the “Messflix Streaming Platform” and the so-called “Hitchcock Movie Maker”. Where the streaming platform does just what it says, enabling subscribers to watch shows and movies, and the latter is a set of tools helping the movie production teams to choose good movie topics, themes, and content. Additionally, Messflix LLC has a data lake with an analytics platform on top of it taking data from everywhere. A few teams manage these components, The teams “Orange” and “White” together operate a few of the Hitchcock Movie Maker tools. Team “Green” is all about the subscriptions, the log-in processes, etc., and team Yellow is responsible for getting things on the screen inside the streaming platform. Figure I depicts a very rough architecture sketch of a few of these components before we discuss quickly, how data is currently handled at Messflix LLC. 6
  • 16. ©Manning Publications Co. To comment go to liveBook Figure I Selective architecture sketch of the Messflix main software components. Clearly visible is a plenitude of data sources Data Team under Data Team responsibility. Depicted are Messflix internal systems (blue), teams (red squares with little team icon above), and users (black silhouettes). 7
  • 17. ©Manning Publications Co. To comment go to liveBook The data team gets data into the data warehouse from a few different places like cost statements from the hitchcock movie makers and subscriptions from the subscriptions service. It also gets streaming data and subscription profiles into the data lake. Then the data team does some number crunching to transform this data and transforms this data into information for fraud analysis and business decisions. Finally, this information is then used by decentralized units to make business decisions and other use cases. This currently is a very centralized workflow as described above. The data team “sits in the middle”. No matter where you’re coming from and where you want to go, you will find yourself somewhere along the Messflix journey. So let us take one final look at the complete journey our company is going through. No data journey is a simple straight line. Likewise, we don’t like to pretend the Messflix journey is a simple linear progression of a series of steps. So while we will go back and forth a bit between the chapters to show different approaches and ways to make the data mesh match any company, there is a green line. You can follow the main thread for the Messflix LLC along with Chapters 2-6 and 9. Table I. below gives you an overview of the stages of the company, we highlight two dimensions alongside the journey. The first is the number of organizational units and teams affected. The second is the levels of responsibilities that are decentralized. The core of the data mesh paradigm shift is the decentralization of the responsibility for data. But “responsibility for data” today is practically split into multiple parts, all of which need to be decentralized. Thus we highlight all four kinds below. They each correspond to one of the principles we will discuss in Part 2. 8
  • 18. ©Manning Publications Co. To comment go to liveBook Table I. Messflix LLC Journey Chapter 4 Chapter 5 Chapter 6 Chapter 7 Chapter 8 Affected Teams 2 (two data producing teams) Few collaborators across the company 2+ 2+ 2+ 3-10 (a dedicated platform team + more producing & consuming teams) Levels of Responsibility A small bit of everything. true ownership through knowing the actual (data) domains. responsibility for serving data customers. responsibility for security, governance, etc., in short, the needs of the whole company. responsibility for infrastructure, technical components. About the authors Jacek Majchrzak is a lead architect in the area of drug discovery where he implements the Data Mesh idea. Jacek is a workshop facilitator with an experience in moderating many workshop techniques like, Event Storming, Domain Storytelling, Business Capability Modelling, Bounded Context Canvas and many other. He designed his own technique that helps discover and define Data Products in a domain oriented way, he called this exercise a Data Bazaar. He is using an aforementioned techniques, to help people make best possible decisions about their technology and strategy. His areas of expertise are Domain-Driven Design, Event-Driven Architecture, solution architecture, socio-technical system design, architecture governance and strategy. He thinks that understanding business domain and needs are the key to a succesful architecture. He blogs on jacekmajchrzak.com. Privately, he's a husband and a father of two, motorcyclist, addicted to sports, especially aquatic. Dr. Sven Balnojan, Sven Balnojan is a data technologist and product person focused on helping the world extract more value from the exponentially growing amount of data. He’s passionate about all things data, machine learning, AI, business intelligence and many related fields. His endeavors include managing internal data teams and managing the transition from service-oriented team to platform-team as well as getting his hands dirty as data developer be that in the field of machine learning, data engineering or Data DevOps. Sven holds a phD in mathematics with a thesis in the field of singularity theory. He is the author of an opinionated newsletter “Three Data Point Thursday”. Additionally he blogs on datacisions.com and appears in talks here and there over the internet. 9
  • 19. ©Manning Publications Co. To comment go to liveBook Dr. Marian Siwiak is a jack of all data related trades and master of their integration with business operations. As a Chief Data Science Officer of Cognition Shared Solutions LLC, he’s a strategic-level data use consultant and the designer of Trilayer Business Proces Analysis™ framework connecting process execution, data environment and risk management. He has a track record of deliveing multimillion IT, scientific and technical projects covering various areas, from life sciences to robotics. He lectured data management at various business universities. His team first published a global spread model of COVID-19. Privately he’s a husband, father of three, skier, sailor, and sci-fi writer. 10
  • 20. ©Manning Publications Co. To comment go to liveBook 1 The What and Why of Data Mesh This chapter covers • What is a “Data Mesh”? Our definition of a Data Mesh • What are the key concepts of the Data Mesh paradigm? • Why is it a “socio-technical” paradigm shift? • What are the advantages of the Data Mesh? • What are the possible challenges of a Data Mesh implementation? The Data Mesh is a decentralization paradigm. It decentralizes the ownership for data, its transformation into information as well as its serving. It aims to increase the value extraction from data by removing bottlenecks in the data value stream by these means. The Data Mesh paradigm is disrupting the data space. Large and small companies are racing to showcase “their Data Mesh-like journey” all over the internet. It’s becoming the new “thing” to try out for any company that wants to extract more value from their data. This book describes the Data Mesh paradigm as a socio-technical architecture with an emphasis on the socio. The main focus is on people, processes, and organization, not technology. Data Meshes can, but don’t have to, be implemented using the same technologies most current data systems run on. But as a topic of ongoing debate and only slowly emerging best practices and standards, we found the need for an in-depth book that covers both the key principles that make Data Meshes work and examples and variations needed to adapt this to any company. This book is designed to do just that: help you begin your own Data Mesh journey. We will provide you with conceptual tools, including but not limited to the key Data Mesh principles and boots-on-the-ground insights and examples. And as it’s designed to be of help, it’s not meant to be a comprehensive theoretical study, but offers readers the practical guidance necessary to get set up. 11
  • 21. ©Manning Publications Co. To comment go to liveBook To start off, we will look at the core ideas of the Data Mesh as well as the benefits and the challenges associated with it. We’ll also give you our definition of a Data Mesh focused on outcomes and practicality. 1.1 Data Mesh 101 The Data Mesh paradigm in this book is all about decentralizing responsibility. For instance, the development team for the “Customer Registration Component” of a company also creates a dataset for analytical purposes of “registered customers”. They ensure it is in an easy-to-digest format by transforming the data, e.g., to a CSV file, and serving it the way the consumers would like it, e.g., on a central file-sharing system. But this deceptively simple definition has a lot of implications because, in most companies, data is handled as a “by-product”. It is usually turned into value only after being put as a by- product into some storage, then pulled into a central technology by a centralized data team, and then decentralized actors again pick up the data. Be that as an analyst in the marketing department, inside a recommendation system used in a marketing campaign, or displayed inside the frontend. Figure 1.1 depicts a common form of data architecture, both organizational and technical. It hopefully also shows its pitfalls. Figure 1.1 Decentralized data emission and central transformation causes problems for the users due to e.g. unclear ownership and responsibility for data and its quality." We can see here two levels of centralization: 12
  • 22. ©Manning Publications Co. To comment go to liveBook • the centralized technology in the form of storage and the usual data engineering, data science machinery, • the organizational centralization of the data team. Since the development team considers the data a “by-product”, the ownership is implicitly assigned to the data team. But such central teams can usually not keep up with the business domain knowledge inside multiple data source domains. The developer responsible for Customer Registrations would only need to know the language and the updates inside that component and the associated business. But the central data team will have to have the same understanding of a domain multiplied by the number of domains. Such overload makes it unlikely that the central team will understand even a single domain to the degree the responsible development team does. As a result, the data team cannot tell whether the data is correct, what it actually means, or what specific metrics might mean. The Data Mesh paradigm shift calls for decentralization of the responsibility for data - to consider it an actual product. The situation depicted in figure 1.1 can turn into a data mesh if, e.g., the development team provides the data product straight to the analysts through some standardized data port. It could be something as simple as a plain CSV file hosted in the appropriate cloud storage spot, easy to access for the analyst. Take a look at figure 1.2 to see this shift in action. Figure 1.2 Decentralized data transformation makes data consumers happy by offering simple access to well described data. A platform team could help provide a simple technology as a service to be used by development teams to deploy such data products, including the data ports, quickly. Data producers focus on developing data products, which, together with data consumers, start to form connections and compose a network. We call such a network the mesh, where the individual nodes are data products and consumers. 13
  • 23. ©Manning Publications Co. To comment go to liveBook Even in our small example, we observe a significant operational paradigm change. It encompasses both a shift in the ownership responsibility (from a central data team to the development teams) and the technical challenge of making the new setup work. Introducing changes in the operational paradigm will result in ripples affecting many areas of your business. To stop it from becoming chaos, we need guiding principles. We will quickly introduce them later in this chapter and examine them in detail in chapters 4 to 7. Before that, you must understand our definition of the “Data Mesh” and its non-technical aspects. DEFINITION OF A DATA MESH Zhamak Dehghani made an incredible effort in curating the idea of the Data Mesh starting in 20191. She provided us with all its critical elements and introduced a structured approach to the previously discussed paradigm shift. Since the first introduction of the “Data Mesh” approach by Zhamak, many “Data Mesh”- inspired business-derived and theoretical examples have appeared. A lot of this content might not perfectly fit into the initial description of the Data Mesh framework. A lot of businesses seemed somewhat unsure about what exactly conforms to the definition of a Data Mesh and what doesn’t. In our work, as well as in this book, we opt for solutions that are first and foremost practical (hence the title “Data Mesh in Action”). Therefore, the Data Mesh definition we coined aims to be broad, functional, and emphasize decentralized efforts to maximize the value derived from data: Definition: Data Mesh The Data Mesh is a decentralization paradigm. It decentralizes the ownership for data, the transformation of data into information, and data serving. It aims to increase the value extraction from data by removing bottlenecks in the data value stream. The Data Mesh paradigm is guided by four principles2, helping to make the data operations efficient at scale: domain ownership, domain data as a product, federated computational governance, and self-serve data platform. Data Mesh implementations may differ in scope and degree of utilization of different principles. The goal of implementing a Data Mesh is to extract more value from the company's data assets. That is also the reason we keep this definition so lightweight and inclusive in relation to the level at which each of the principles are followed. The following non-technical use case of a Data Mesh will hopefully explain what we mean by that. 1.2 Why the Data Mesh? We see three main reasons why the data world is in need of decentralization in the form of the Data Mesh: 1 https://martinfowler.com/articles/data-monolith-to-mesh.html 2 The four Data Mesh guiding principles are described in section 1.4 of this chapter and in chapters 4 to 7. 14
  • 24. ©Manning Publications Co. To comment go to liveBook • With the proliferation of data sources and data consumers, a central team in between creates an organizational bottleneck. • With multiple data emitting and consuming technologies, a central monolithic data storage in between creates a technological bottleneck, which loses a lot of the information in this step in the middle. • Both data quality and data ownership are only implicitly assigned, which causes confusion and a lack of control over both. Over the past thirty years, most data architectures were designed to integrate multiple data sources, i.e., central data teams merged data from all kinds of source systems and provided harmonized sets to users, who in turn tried to use it to drive business value. Yet, for over a decade now, the problem of big-data hangovers has plagued companies of all sizes. The data environments struggle with the scalability of the solutions, completeness of the data, accessibility issues, and the likes. We bet you can tell that in your company you’re also fighting a struggle to extract value from data, right? Some things simply seem to not work out. Dozens of reports & dashboards seem to be of no use compared to the costs of creating & maintaining them. A bunch of data science projects seems to stay stuck in the “prototype” phase, and the running data-intensive applications probably are facing a bunch of data-related problems. At least it should seem that way compared to the effort it takes to get a software component to run. Just not yet right. One of the reasons for the scalability problem is the proliferation of data sources and data consumers. An obvious bottleneck emerges when one central team manages and owns data along its whole journey: from ingestion through transformation and harmonization to serving it to all potential users. Splitting the team along the data pipe does not help much either. When engineers working on data ingestion change anything, they need to inform the group responsible for transformation. Otherwise, the upstream systems may fail, or worse will process the data incorrectly. Required close collaboration between the engineers leads to the tight coupling of all data-related systems. The other problem arises from the monolithic nature of data platforms, such as warehouses and lakes. As a result, they often lack the diversity to reflect the reality encoded in data derived from sources and domain-specific structures. Moreover, enforced flattening of data structures reduces the ability to generate valuable insights from the collected data, as crucial domain- specific knowledge gets lost in these centralized platforms. We could observe it in one of the projects we worked on. The car parts manufacturing company was buying data related to failures of different parts. Even though the provider had information on the part provenance, i.e., the model the part was installed in, the buyer had no data models allowing it to store this information. As a result, components were analyzed separately, hampering R&D’s attempts to understand the big picture better. Two more interwoven factors exacerbate the problems described above. One is unclear data ownership structure; the other is the responsibility for data quality. Data traveling through different specialized teams loses its connection to its business meaning, developers of centralized data processing systems and applications can’t and won’t fully understand the data content. In contrast, data quality cannot be assessed in disconnection from its meaning. Similar problems were recognized in other areas of software engineering and resulted in the emergence (and success!) of domain-driven development and microservices. Application 15
  • 25. ©Manning Publications Co. To comment go to liveBook of similar thinking (i.e., focus on data ownership and shared tooling) to data engineering led to the development of the idea of the Data Mesh. 1.2.1 Alternatives There are two main alternative models to Data Mesh’s decentralization of responsibility for data. We discuss their setup in more detail in chapter 6, here providing a quick overview. The first option is the centralization of both people and technology. This is the default setup for any start-up. And it’s a very decent default option, just like the monolith is a decent default option for any software component. In the beginning, the costs of decentralization outweigh its benefits. The benefits brought in by working closely together inside one data team, having just one technology to use, makes things a lot easier. INSIGHT: CENTRALIZATION IS A SENSIBLE DEFAULT OPTION Centralized data work both organizational and technical can make sense as a default option. Decentralization does carry costs, and centralization can mitigate those. That does imply though, that the value derived from centralized and decentralized data is roughly equal. The second option is the idea of splitting up the work not by business domains as the Data Mesh suggests, but by technology. This usually results in one core data engineering team responsible mostly for ingesting data and provisioning a data storage infrastructure and multiple other teams, analytics teams, data science teams, analysts you name it. These pick up the raw data and turn it into something meaningful down the road. You might first centralize your data system and then layer up with this option to increase the flow. There is nothing wrong with these two options. They might be reasonable default options, but both options fail to align with the value creation, which is deeply tied to business domains. None of the options are able to address sudden changes in just one business domain. As with microservices, where the strength is the ability to quickly extract value from one specific service by scaling it up all by itself, the data mesh is able to scale up value extraction in just one domain. All other options need to scale up everything to scale up value extraction in just one domain. So in one way or another, both of these options will hit a wall at some point in time, in which it will feel like adding the next data source, or adding the next data science project will feel extremely complex & costly compared to the first ones. That’s the point where you want to switch to a Data Mesh. 1.2.2 Data Warehouses & Data Lakes Inside the Data Mesh There is a misconception about the Data Mesh. It is sometimes perceived as an exclusive alternative to the central data lake or the central data warehouse. But that does not take into account what the Data Mesh is, a combination of two things, technology & organization. The Data Mesh is an alternative to having one centralized data unit taking care of the data inside a central data storage. 16
  • 26. That still leaves the option to have central data storage and decentralized units working & owning the data. Indeed that is a common implementation in companies that don’t need complete flexibility on the data producers’ side. It also is a common approach to keep data lakes & data warehouses inside a business intelligence or data science team. The data lakes & data warehouses then become a node inside the Data Mesh. Figure 1.3 Data Mesh can still use Data Lakes, e.g. a Data Science team building data products may use Data Lakes as nodes within the Data Mesh. Data Meshes make heavy use of both data lakes and data warehouses in various formats, Data Meshes in general do not try to focus on any specific technology. We discuss this dichotomy a bit deeper in section 1.6. For now, let’s try to keep spirits high and focus on the benefits of Data Mesh. 1.2.3 Data Mesh benefits Let’s analyze the potential lying in Data Mesh implementation from two different perspectives: through the eyes of the business decision-makers and the technologist. THE BUSINESS PERSPECTIVE From the business perspective, data itself is of little value. Worse, it means incurred costs! Sounds like heresy? To understand that statement, and if needed, convey it to your business partners, you need to understand the different levels at which people can cognize reality. A ©Manning Publications Co. To comment go to liveBook 17
  • 27. ©Manning Publications Co. To comment go to liveBook good approximation of this phenomenon is the so-called DIKW pyramid3, derived from 1934’s “The Rock” play by T.S. Eliot. It represents data, information, knowledge, and wisdom as a hierarchical structure, where each next element can be derived from the former. The data in this context is just a set of values (storing of which costs money!). To derive the value from it, one needs to build up the context allowing for informed decision-making. The Data Mesh improves the robustness of the whole pyramid. As we mentioned, having the raw data is of no use for decision-makers. One can argue that they can download it to their laptops and analyze it themselves. It is true! It has, however, two underlying assumptions: 1. To download the data, it needs to be accessible. 2. To ensure the value of any performed analysis, data needs to be as complete as possible. To address the first assumption: we mentioned already and will say repeatedly, Data Mesh is very much focused on making data accessible. Not only accessible but findable, interoperable, and reusable as well! It’s embedded in one of the four Data Mesh principles: Data as a Product is all about making sure data is there for the taking. Completeness of the data is another issue where Data Mesh shines. Unlike most Data Warehouse or Data Lake architectures, Data Products and their data models are not developed by IT specialists in disconnection to business. Instead, it’s a joint effort, ensuring the data presented outside the domain is sufficient to derive meaningful conclusions. Data Mesh also helps add value to elements higher in the hierarchy. The teams transforming data into information, knowledge, and wisdom, which the business environment likes to call “insight,” gain instant access to multiple interoperable data sources. Of course, in theory, it’s possible to make it happen in Data Lake as well. However, the reality of as a single team managing technical aspects of the environment, as well as data access and transfer rights, in our experience always makes it infeasible. And if required bits of data are stored in two different Data Lakes (or four, which is not that unusual), getting them all to work together is next to impossible. In short, having access to read-optimized Data Products enables quick prototyping of new analytical methods and opens a path for the rapid development of new business capabilities. THE TECHNOLOGY PERSPECTIVE The main benefit from the technological perspective is maintaining the speed of development with the organization’s growth, as we mentioned previously, required to keep bringing business benefits from the data. Data Mesh is meant to address the shortcomings of other data architectures, like Data Warehouse or Data Lake, by decentralizing data production and governance. Those architectures introduce a bottleneck — a central team responsible for harmonizing all the data for the whole of your company and making it ready for consumption. A single team cannot scale to accommodate the varied data needs of a growing organization. Both the technology as well as the team knowledge quickly becomes a scale problem. Eventually, more time is spent on maintenance, and new projects become more and more delayed. 3 https://en.wikipedia.org/wiki/DIKW_pyramid 18
  • 28. ©Manning Publications Co. To comment go to liveBook The other benefit of Data Mesh is the clarity of data ownership right from the point of its creation. It flattens the data management structure leaving just a thin layer of a Federated Governance Team. And even that team’s activities are limited to agreeing on standards within autonomous domains. The speedup of development also comes from empowering the implementation teams. Since producing and maintaining the Data Products lies on their shoulders, the speed of change is not limited by a single central integrations team’s backlog of tasks. It means that both the evolution and fixes to the Data Product happen quicker. It is especially prominent in case of any bug fixing and downtimes. Furthermore, the Data Product owning team is better equipped to react faster because there is no context switching, as in the one central team’s case. The other factor worth mentioning is data environment stability. With Data Products offering access to contracted versions of its datasets, pipelines built on them are much more robust and require much less maintenance. 1.3 Use Case: The Snow Shoveling Business Candace operates a snow shoveling business. She is a proud business woman who started the business shoveling snow every winter herself. After a couple of years, she scaled up her operations. She focused on logistics, billing, setting the prices across the business, etc., and hired three employees. Adam, who shovels the houses at Pine Road; Eve, who does the houses on Oak Street; and Bob who is responsible for ordering new shovels, as they seem to break all the time. Adam and Eve are responsible both for shoveling as well as bringing in new clients on their respective streets. After all, they spend quite some time around people there, shoveling snow each winter! But, Candace ain’t happy. 19
  • 29. ©Manning Publications Co. To comment go to liveBook Figure 1.4 Candace’s (C) Snow Shoveling Business. Adam (A) and Eve (E) do their work, while Bob (B) creates a stack of inventory, freezing the capital. The previous year, Candace asked Adam & Eve to write down their working times and the number of houses cleared. She put that data in a fancy Excel file to do some calculations and set the prices. Figure 1.4 displays that data flow. 20
  • 30. Figure 1.5 Centralized Data Flow from Adam & Eve to Candace for Decisions This way, one could say, she turned the data received from Adam & Eve into information in her Excel, and then further into decisions yielding the business value. Additionally, she asked Adam & Eve to provide her with a rough guesstimation on how many shovels they will need. Figure 1.5 shows the data flow to Bob. Figure 1.6 Centralized Data Flow from Adam & Eve through Candace to Bob We could say,that Adam and Eve provided raw data, Candace aggregated it into information and handed it over to Bob to turn the data into decisions & business value. Notice how you again can see a “pipeline”, a sequence of steps that turns data into value. The pipeline has two steps for setting prices, and three for the shovel procurement with Bob. ©Manning Publications Co. To comment go to liveBook 21
  • 31. But Candace is not happy with the situation. Profits are not too good and the shovel procurement seems to always be off, sometimes inventory seems to stack up, sometimes Adam and Eve need to delay their work, as they run out of shovels. This year, after reading a book called “Data Mesh in Action”, she decided to experiment. She decided to “build the foundation for a Data Mesh” described in chapter 3 there. She used knowledge form chapter 4 to define domain boundaries, and give the ownership of data to her main shoveling domain owners, namely Adam and Eve. That means: 1. She stops collecting the times, and instead she asks Adam & Eve to record their times themselves in any way they see fit 2. Adam and Eve will set the pricing for their streets Then something interesting happened. Eve decided to to increase prices on Oak Street, whereas Adam decreased the prices on Pine Road. Turns out, they simply knew more about their neighborhoods, than Candace did. On Oak Street, the houses had long driveways, so Adam needed to charge more per driveway. On Pine Road, a local kid was shoveling cheaper, so to be competitive Eve needed to get her price down. Figure 1.6 shows the decentralized data flow now taking place. Figure 1.7 Decentralized Data Flow inside Adams and Eve domain If we look at the flow of data now, we could say it stays completely with Adam and Eve respectively, from data to information and finally into the decision - the pricing model. As profits increased, Candace decided to take a step further. In the second year of the experiment, Candace decided to tackle the procurement problem. To do that, she decided to ask Adam & Eve to directly inform Bob about the shovels they will likely need. ©Manning Publications Co. To comment go to liveBook 22
  • 32. To avoid confusion and to-and-fro email, they decided to create a spreadsheet, Adam and Eve updated regularly, if e.g. current conditions or weather forecast changed their estimations. Figure 1.7 explains the data flow serving data to Bob. Figure 1.8 Decentralized Data Provisioning in Adams & Eves Domains served to Bob for decision-making. In our data flow, Adam & Eve now keep both Data and Information in their domain and try to supply Bob with a proper set of information to make the procurement decisions. Not just the raw data they were asked to collect. Bob seems much happier. When asked by Candace why is so, he tells her that Eve realized Bob also needs to know how often shovels break. Eve seems to simply break them more often because of the longer driveways. An unexpected business benefit of shortening the data flow emerged. Adam and Eve usually presented Candace with pessimistic approximations, not wanting to be left without shovels. Sometimes, however, being afraid of her reaction to too many broken shovels they hoped for the best and presented optimistic ones. Candace usually added an extra “safety margin” before sending the order to Bob, but sometimes she got worried about spending and took a risk of lowering the received estimation. Bob, knowing that the numbers he receives are unreliable, often tried to adjust the order based on his gut feeling (and telling Candace, that e.g. required numbers of shovels was not available on the market. To keep the company running, he had to create a buffer of shovels, decreasing the company’s financial liquidity by freezing the capital and increasing warehousing costs. After he got access to Adam and Eve’s direct forecasts, he was able to procure just enough shovels, ending the year with barely any inventory left and without running out of them in a meantime. ©Manning Publications Co. To comment go to liveBook 23
  • 33. ©Manning Publications Co. To comment go to liveBook For Candace, this resulted in a nice profit thanks to reduced costs in year two, as well as three happy employees. If you compare that to the data pipe above, you should notice that again, this “pipeline” , the flow of data from one unit, in this case, Adam & Eve, to Candace and further to Bob, is the source of all problems. All that Candace did was break up that pipe and “package it into one” or at most two pieces. That is decentralization at work. And that is really all there is to it, by breaking up this pipe in certain situations, you will end up with better outcomes, because the decentralized parts, like Adam & Eve in our case, simply can turn that data into value better, whereas the pipe could not. Probably next year she’ll want to introduce some form of governance, e.g. introducing rules on the frequency of updates, or ensuring the security of her spreadsheets, so the kid from Pine Street doesn’t mess up with their data. Maybe she’ll hire someone to create a website, where people will be able to book their services online - to do that she’d have to develop data products with Adam and Eve’s working times and connect them with the new system using some sort of the platform. The future is full of potential when you have operational profit, isn’t it? Next, we will show you the four principles of Data Mesh. We consider them the cornerstones of its implementation. But as we mentioned previously, our Data Mesh definition is very inclusive and business value-oriented. It means you need to prioritize them for yourself and decide to what degree you should follow them to achieve your business goals. Just as Candace did, you will need to keep the following principles as “guides” to successfully implement a Data Mesh. 1.4 Data Mesh principles Zhamak Dehghani first described the current incarnation of the Data Mesh concept as a set of four principles. Throughout the book, we will focus on their implementation details. Below is a summary of these four Data Mesh cornerstones. 1.4.1 Domain-oriented decentralized data ownership and architecture The first principle is that data and its relevant components should be owned, maintained, and developed by the people closest to it, the people inside the data’s domain. This calls for the application of the concepts of domains and bounded contexts to the data world. This also means the application of decentralization to ownership and architecture. The idea is, that just like in the example above, domain-internal engineers are responsible for developing data interfaces, allowing other users, be it data scientists, self-service BI users, data engineers, and system developers, to use the domain data. Data engineers of a data product are expected to be experts within remits of a single domain, which minimizes communication problems and misinterpretations of data. Data should have its clear ownership, and it should not be on the centralized level of organization; we should put that responsibility into the hands of the people closest to it. That might be the domain team in the case of a source-oriented dataset, or it might be a data engineering team, an analytics engineering team, or a data science team, for new datasets created out of multiple others. It could also be the organizational unit using the data if our dataset is a very consumer-oriented dataset. 24
  • 34. The domain is an area or part of our business. It is a way of slicing and decomposing the company. Quite often, our organizational structure resembles business domains. At Messflix LLC, we can, for instance, find domains like Content Distribution, Content Production & Market, and Brand Development. Figure 1.9 Simplified Messflix business domain sketch The Domain Ownership principle says that each of the teams or units owning those domains should also own data that has been created inside this domain. So the team developing software to support the content production is also responsible for Content Production Data. But what does it mean to be responsible for data? Being responsible for data means hosting and serving datasets to other parts of the organization, to other domains. Ownership not only spans data; the team should also be responsible for pipelines, software, cleansing, deduplicating, or enriching of that data. ©Manning Publications Co. To comment go to liveBook 25
  • 35. ©Manning Publications Co. To comment go to liveBook The same as with agile principles and agile teams, ownership on its own would not make much sense without having autonomy at the same time. This is why teams should be able to release and deploy their operational and analytical data systems autonomously. As you can see in the example of Messflix LLC, every business consists of many domains. Each of those domains can usually be further split into subdomains. By applying this principle, we will end up with a mesh of interconnected domain data nodes. Nodes of the mesh can and even should be connected. Data can be moved, duplicated, and change shape between the domains when it is needed. For example, data will usually move from source-oriented data domains (output of the operational systems) to more consumer-oriented data domains, where raw data will be customarily aggregated and transformed into a more consumer-friendly shape and format. In such a mesh of interconnected domains and subdomains, only the people close to the data know it well enough to actually work with it. Take an example from above: Does the word “content” mean the same in all three domains? Hopefully not, because in the “Produce content” domain, we will have both “draft versions and ideas” of not yet produced content, as well as content that will then become truly productionized content. However, in the “Distribute content” domain, people will always mean productionized final content when they say content. Just imagine a developer from the “Distribute content” domain who is supposed to compile a list of “content pieces” from the “Produce content” domain. He will likely produce a list of produced content pieces. However, that will miss the point. The requirement likely asks for a list of all content pieces, including ideas, things that are still in production, and canceled pieces. Additionally, the status of these content pieces should also be included. However, people outside this domain will not even know that these are important pieces of data. Hence making the people inside the domain the only people truly able to work with this kind of data. Source Data Domains usually serve data and information that represent facts about the business. Data should be exposed to other domains to serve operational purposes (like REST API) and analytical purposes. Source Data Domain should expose domain events and historical snapshots aggregated over time. With the latter, we should make sure that underlying storage is optimized for Big Data analytical consumption (like Data Lake). In the above example, the “Produce content” domain becomes a source data domain, when it exposes lists of produced content pieces. This is an original piece of data created by the business process of creating content. Consumer Data Domains are aligned closely with consumption. An excellent example of such a domain could be Management Reporting and Predictions. In the case of Messflix LLC, it could be the Content Recommendations domain which might be a subdomain of Content Distribution. If now the marketing team takes the list of produced content pieces and enriches it with relevant marketing materials, the tweets it uses to promote it, etc., then the new list slips into a consumer data domain. Parts of the data between Data Domains can be duplicated. Still, because it is serving a different purpose, it will also be modeled differently, so usually, domain boundaries will, at the same time, be data model boundaries. Because we want to give teams as much autonomy as possible, we are not trying to achieve a single canonical model for the whole organization. We are giving them the freedom to model the data in the way they need it. Besides the Source 26
  • 36. ©Manning Publications Co. To comment go to liveBook and Consumer aligned domain, we could also encounter core data domains used across the organization and usually represent key entities or objects. When we share the responsibility for data across the organization, we gain tremendous scalability and maintainability. For example, when we want to add new datasets to the Mesh, we will be adding a new autonomous node. At the same time, teams that own datasets will be in a very comfortable situation. They will only own data that they truly understand. 1.4.2 Data as a product The second principle is that data must be viewed as a product. This calls for an introduction of product thinking, integrated into data management. Usually, when we talk about data, the first thing that comes to our mind is either a file or a table in a database. We often see a spreadsheet or a series of rows in a file with named columns when we think of data. Taking this perspective, it is easy to reduce data to technical details. But, instead, the more important question is - what gives value to the data from the organization's point of view? Or reversing this question - what is stopping our data from being turned into valuable decisions? Without a proper set of descriptions, even the best-prepared set of data will not be found, and thus no value extracted from it. Missing information on the recency or completeness of data might render it completely useless in cases that require these attributes. That's why it's worthwhile to think of data as a product - a larger whole that ultimately makes up the experience of the users who use it. Data offered by a data team shall follow typical product features like: • Viable quality - ensured by specialized domain experts • Anticipation of user needs - team offering the data to the outside world, shall understand the enterprise's business environment, e.g., present the data in a format easily digested by existing data pipelines to ensure availability and effortless usefulness of the provided data • Secured availability - ensuring availability of product to fulfill user needs whenever he needs it • Focus on user goals - the focus on own domain shall not mean lack of communication of data team with other users; on the contrary, search for synergies and shared toolset shall create an opportunity to understand each other’s needs better • Findable - any data product should be discoverable by a simple means, something a random table in a database lacks • Interoperable - different data products should be combinable in a way that increases their value WHEN CAN WE CALL DATA A PRODUCT? In everyday life, we deal with many products. A product might be defined as, in general, "the result of a conscious action.” If we take a pair of jeans as an example, we are convinced that it meets certain predetermined conditions. For a product to be called a pair of jeans, it must have a suitable form and shape. In addition, it should have its unique name (especially if the manufacturer provides many products of a given type) because we buy a specific model and not a pair of jeans in general. 27
  • 37. ©Manning Publications Co. To comment go to liveBook It should also meet some standards to ensure that a pair of jeans will not break down after a few hours of use and is made of a safe material. In addition to this, there is someone responsible for this product, its quality, and the consequences of its use within the accepted terms of use. We can use this kind of thinking to think about data. Treating data as a product will mean that someone has consciously designed the product and then created and released it, is responsible for it, and is mainly responsible for its quality. In the context of Data Mesh, this will be the responsibility of the Data Product Owner - who designs the Data Product - and the Data Product Developers who implement it. Just like a product on a shelf in a store - a Data Product has its unique name, and it has established characteristics, including: • Level of quality • Level of availability • Security rules • Frequency of updates • Specific content When we think about products - it's not enough to just expose data. We also need to make sure that we maximize its usability for the end-users. In this context, the role of the Data Product Owner is critical, as he or she is responsible for the final User Experience of the data. Referring to the Messflix example mentioned earlier, we can imagine a few exemplary Data Products in the Produce Content domain: • Cost statement • Scripts • Cast • Movie popularity • Movie trends Treating data as a product brings us straight to the Product Thinking approach - start with what problems your customers want to be solved, which should drive the Data Product design process. Then, as a Data Product Owner - you should deliver a Data Product addressing these problems based on predefined success criteria. Data as a product also implies a degree of standardization that allows a single element to be incorporated into a larger Data Mesh ecosystem. To call a given set of data a Data Product, it should be: • Self-described and discoverable - data should be described, and this description should be an integral part of the Data Product. Data Product should be able to register itself in the Data Mesh ecosystem and, as a result, should be discoverable, • Addressable - it should have its unique address (e.g., in the form of URI address) so that it can be referred to and relationships between Data Products can be created, • Interoperable - a Data Product should be made according to predefined standards, concerning the form of data sharing, standardized formats, vocabularies, terminology or identifiers, secure and trust-worthy. Data Product should meet the established and declared SLA, and enable controlled access, ensuring data security from both perspectives: intellectual property and GDPR-type regulations. 28
  • 38. ©Manning Publications Co. To comment go to liveBook DATA PRODUCT AS AN AUTONOMOUS COMPONENT While fulfilling the previously established conditions, a Data Product should at the same time constitute an autonomous component so that it can be independently developed by the team responsible for a given Data Product. From the point of view of a technical solution, we can see a Data Product as an analog of a microservice in the data world. In addition to making data available, the Data Product also embeds code related to data transformation, cleaning, or harmonization. It also exposes interfaces for automatic integration with the Data Mesh environment and platform by providing, among other things: • Input logic and output ports - form and protocols used to ingest from source and expose data to consumers; • Operational metrics - number of users, throughput, amount of data fetched, etc.; • Data quality reports - quantity of incomplete data, format incompatibilities, statistical measures of outliers, etc.; • Metadata - specification and description of the data schema, domain description, business ownership, etc.; • Configuration endpoint - means to configure the Data Product in runtime, e.g., setting the security rules. These are a few examples of what can constitute a Data Product as a technical component. 1.4.3 Federated computational governance The third principle is to federate and automate the data governance across all of the participants of the Data Mesh. It aims to provide a unified framework and interoperability to the ecosystem of largely independent data products. Its purpose is to make the autonomous data products work in an actual data mesh, not just as stand-alone products. Federated computational governance requires two inseparable elements: the governing body and means of rules enforcement. Overarching rules and regulations shall be agreed upon by a body composed of Data Product Owners, self-serve data infrastructure platform team, security specialists, as well as CDO/CIO/CTO representatives. The body would also serve as a place for discussion regarding, e.g., development of new data products vs. adding new datasets to existing data products, methods of ingesting new external data sources, priorities for central platform development, etc. Effective data governance is crucial, with data security being one of the main concerns of CDOs/CIOs in companies across all sectors and sizes. Also, most large enterprises need to introduce controls on data security and governance enforced by governmental or business regulations, e.g., GDPR, HIPAA, PCI DSS, or SoX. FEDERALIZATION OF DATA GOVERNANCE At first glance, Data Mesh adds a new layer of complexity to the already vast scope of Data Governance, as it now also needs to address a shift of responsibilities to Data Products. Each of the Data Products, in turn, needs to be equipped with processes allowing safe and efficient ways of handling owned infrastructure, code, and data (and metadata). Moreover, data governance processes need to balance company-wide cohesion of data solutions, usually 29
  • 39. ©Manning Publications Co. To comment go to liveBook achieved through standardization vs. autonomy-driven flexibility and creativity offered to Data Product teams. It is imperative to understand that there is no silver bullet solution to balance central governance and local autonomy! It will always depend on the specifics of your organization. For example, what are the needs and the maturity of your Data Product Owners? What is the data-related risk appetite of the organization? What is the level of expertise of your central data governance team? How sensitive is the data you’re working with? You need to answer these and a lot more questions before you’ll be able to set the remits of central and local teams. Another set of policies is required to ensure interoperability of Data Products and the ability of data consumers to join them together readily. Finally, the procedures need to provide the compatibility of different Data Products without explicitly enforcing an overarching data model, as such an approach has been proved to create a bottleneck in data operations. Data Mesh tries to answer that need with the Federalization of Data Governance. The federalization in this context means governing structure operating at parity of distinct levels - central and local. The central level of governance, executed by the data governance council (of course, the central level governance body may use another name, e.g., in MVP example we’ll present in chapter 3, it will be called simply a data governance team), will decide on a minimal set of global rules required to ensure safe and secure discoverability and interoperability of data products. Data Product teams, led by Data Product Owners, are responsible for developing their products with a high degree of autonomy, deciding on all technical and procedural issues within its remits (within boundaries of the enterprise technology stack). There is no silver bullet solution as to the precise division of responsibilities, the structure of the data governance council, and exact rules governing the data world. Instead, each business will have its own set of global rules, leaving domain teams with different levels of control over their Data Products. Chapter 4 will discuss the effects of different levels of domain team autonomy on their agility and overall system complexity. COMPUTATIONAL ELEMENTS OF DATA GOVERNANCE The name Federated Computational Governance was coined by Zhamak Daghani in her de facto “Data Mesh manifesto”4 blog article. However, as Computational Governance is not defined anywhere else, throughout this book, we will assume that this term relates to elements of Data Governance automation, enabled through the existence of the Central Platform (as we call self-serve infrastructure-as-a-platform presented in a bit more detail in section 1.7, and in a whole lot more detail in chapters 7 and 8), serving as a medium connecting different Data Products. Data Governance elements, which can be automated and embedded into Data Platform include, but are not limited to: 4 https://martinfowler.com/articles/data-mesh-principles.html 30
  • 40. ©Manning Publications Co. To comment go to liveBook • Metadata • Catalog, reference, and master data • Lineage & provenance • Validation & verification • Storage and operations • Security & privacy Once again, there is no silver bullet solution for deciding which of these should be automated. It will require your teams’ hard work to identify which data governance tasks create bottlenecks in Data Mesh development in your organization and automate them. It will pay off, however. First, it will offer Data Product Owners the solution allowing them a frictionless connection of their Data Products. Second, it will enable users to make efficient use of the exposed data. You will learn more about automating elements of data governance when we will discuss the details of the self-serve data infrastructure-as-a-platform. 1.4.4 Self-serve data infrastructure as a platform The fourth principle is to extract the duplicated efforts of the Data Mesh into a platform. It calls for the application of “platform thinking” to the data context. Platform thinking means that efforts that are duplicated throughout the company to a larger extent can also be packaged into a “platform” and thus only done once, but offered as a “service” to others. Just like anyone can rent cloud resources on one of the major cloud providers and customize them to fit his needs, taking away to duplicate the effort of maintaining one’s own cloud, his idea can be shrunk to efforts inside a company. Building and maintaining data products is resource-intensive and requires a set of very specialized skills (ranging from computational environment management to security). Multiplication of the required effort by the number of Data Products would endanger the feasibility of the whole Data Mesh idea. The idea behind the Central Platform is to centralize repeatable and generalizable actions to the degree necessary (yet again depending on the context of the company!) and to offer a set of tools abstracting away specialized skills. It would reduce entry and access barriers for both Data Product developers and Data Product consumers. You may start with the requirements of your first Data Product Owners and iteratively build the setup up to meet your organization’s needs. The infrastructural support can be offered at on-premise computing power or in the cloud, depending on the enterprise policies. Its elements can be available in IaaS, PaaS, SaaS, or their hybrid models. Data Mesh platform could, e.g., support: 31
  • 41. ©Manning Publications Co. To comment go to liveBook • Governability - all data computation related to Data Governance processes needs to be incorporated and automatically enforced on every Data Product connected to the Data Infrastructure. • Security - the infrastructure solution shall ensure that all Data Products offer freedom of operation for users whose access allows them to meet their information requirements and ensure safety from unauthorized access. To do that set of ready-to-use processes, tools and procedures shall be accessible to Data Product creators and users. • Flexibility, adaptability, and elasticity - the infrastructure needs to support multiple types of business domain data. It means enabling different data storage solutions, ETL and query operations, deployments, data processing engines, and pipelines. All that needs to be scalable, to serve business needs as they arise. • Resilience - smooth operations of data-driven businesses require high availability of data. Therefore, they are ensuring redundancies and disaster recovery protocols at the design level of each infrastructural element. • Process automation - from data metadata injection at Data Product registry to ensuring access control, data flow through central infrastructure needs to be fully automated, possibly using machine learning and artificial intelligence to ensure efficient data processing, quality, and monitoring. After you’ve got a short summary of the four key Data Mesh principles, let us visit Candace again and see how she scores on these four principles. 1.5 Back to Snow Shoveling In Candace’s business, you might identify a few “expert domains”. Seems like shovel procurement is one, but also the two streets seem to be separate domains. They definitely are different in some key elements and local knowledge is needed to work effectively. Let’s look at how the business evolved over time: • In Candace’s first year, her team does not own the data, even though they own their “domains” in the sense that they are responsible for operations there as well as creating domain data. There is no data governance mentioned, and not much of a platform to help them do anything with data. Data is a by-product of their work. • In Candace’s second year, she makes a key change. She gives the data ownership into Eve’s & Adam’s domain. She also extends their responsibility in general, something that in many companies happens before transferring the data ownership. However, there is still no data governance, platform, and not much data is being treated as a product. • In Candace’s third year, she essentially tells Adam & Eve to finally treat the data not just for themselves as valuable, but also as a product for others, in this case, Bob. Finally supplied with a proper “data product”, Bob is able to also optimize the costs of the business. In summary, Candace focuses mostly on the principle of decentralized data ownership and data product thinking. Since her operation is very small, she doesn’t need to do much governance at all besides talking with her employees about using data in decision-making. She also does not have to use a large platform, or any central platform at all besides maybe email simply because there isn’t yet a large mesh of data products to connect. 32
  • 42. ©Manning Publications Co. To comment go to liveBook INSIGHT: THE VALUE LIES IN DECENTRALIZATION, NOT IN ALL FOUR PRINCIPLES We think that Data Meshes at the core are just about the decentralization of data ownership into any kind of autonomous unit. That can be an individual in our case, it can be a whole department, it can be anything depending on the situation. But, the key principles are guidelines for helping us execute this decentralization process well. We believe that the Data Mesh journey might begin at any size of an enterprise if you “feel the pull of decentralization”. We will discuss that in more depth in the next chapter. INSIGHT: DATA MESHES ARE FOR COMPANIES OF ALL SIZES The Data Mesh is a possible tool for companies of all sizes. But that does not mean it is the right tool for every company in their current stage. It can also be the wrong tool for a company, again independent of its size. We discuss the decision process in the next chapter. So why did Candace feel this “pull of decentralization”? And why will you feel it too at some point? 1.6 Socio-technical architecture Data Mesh is not a technical solution for data problems. Data Mesh solves these problems using socio-technical architecture or socio-technical system design. We believe this is the biggest strength of Data Mesh as a paradigm, and we think you should treat socio-technical architecture as a foundation of data mesh implementation. You need to apply socio-technical architecture consciously to make it successful. But before doing it, you need first to understand what it means, how it originated, and most importantly, what are the underpinning laws behind it. For that, we look into Conway’s law to understand why we won’t be able to change technology just on its own and then into the Team Topologies framework which is in essence about socio-technical architecture. Finally, we look into Cognitive Load as one of the main ideas leveraged by the Team Topologies framework. 1.6.1 Conway’s law "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." (Conway’s Law) Melvin Conway made this observation in 1967, so just ten years after FORTRAN, the first widely used high-level language, was released. It didn’t take us long to see this strong, almost gravitational force in practice. This ‘force’ will sooner or later transform our architecture so it will look like our organizational structure. As architects, we need to be fully aware of the implied consequences. We should take as an example the martial art called Aikido. ‘Aiki’ refers to the martial arts principle or tactic of blending with an attacker's movements to control their actions with minimal effort. The masters of Aikido teach you that you should not put your force directly against the force of your opponent. Instead, you should take advantage of the force he is exerting. In the same way, we should be aware and take advantage of Conway's Law. 33
  • 43. ©Manning Publications Co. To comment go to liveBook EXAMPLE: CENTRAL DATA WAREHOUSES Conway’s law is a good explanation on why people say “central data warehouse” but mean “central data warehouse operated by a central data team”. It’s because usually when you find a central data warehouse, it's been built by a central data team. Conway’s law simply tells us to take care of both the organizational side of things and the technical. Just changing technical things isn’t going to make an impact. The organization will cramp itself into the new technology. 1.6.2 Team Topologies In the Socio-technical architecture approach, as the name suggests, we are trying to co-design both the social and the technical architecture elements simultaneously. This way, we are thinking upfront about all constraints and concerns. IMPERFECT EXAMPLE: CONSTRUCTING TEAM TOPOLOGIES Try to think of the process of building a skyscraper. It’s structurally different from villagers collectively building a barn for one of the members of their community. A Skyscraper is built by highly specialized teams, where some have “front line”, and some have “support” roles. You don’t send electricians to install windows (pun intended). The requirements of the technology used in a given location at a given time define the team carrying out the task. This section is about the one main architecture method that is used today and within the Data Mesh community to do socio-technical architecture well. “Team Topologies is a clear, easy-to-follow approach to modern software delivery with an emphasis on optimizing team interactions for flow.” (source: teamtopologies.com) Matthew Skelton and Manuel Pais created team topologies, and they are starting to get some traction in the community because of their simplicity and straightforward application. Team Topologies is an approach that helps you not fall into the trap of Conway's Law. The Team Topologies framework is designed to optimize the teams inside a company for optimal flow. It relies heavily on the idea of cognitive load to properly split workloads across this flow and separates teams into just a handful of teams as well as interaction modes into ones that are designed to minimize cognitive load. Since Team Topologies is focused on the optimal flow of the company, it will help us to optimize the data to value flow inside the company. 1.6.3 Cognitive Load Cognitive load is a term from cognitive load theory, and it means the amount of mental resources (working memory) used by a person. This theory was at the beginning applied to the teaching field, and it influenced how we write instruction so readers can easily digest them. But after that, it was used in more and more areas, like IT. There are three types of Cognitive Load: 34
  • 44. ©Manning Publications Co. To comment go to liveBook • Intrinsic: the skills and knowledge you need to build your product (like programming languages, framework, patterns). • Extraneous activities are not part of product development but are needed to release the product (like infrastructure, deployment, monitoring). • Germane: knowledge of business and the problem space (like, types of movies if you are a Messflix LLC developer). As socio-technical architects, we should shape our teams so that the total cognitive load is limited. It is very similar to road traffic, and you will not maximize the flow of cars if you fill up every square meter of the road with cars; traffic moves in the fastest way when there is enough space between vehicles to enable them to drive near the speed limit. It is the same with the teams. So, what can we do to make Data Mesh teams performant? So firstly, we should plan our technology stack to be simple to limit intrinsic load. Secondly, we should embrace platform thinking as not to force teams to reinvent the wheel every time with yet another monitoring and logging solution to limit extraneous load. Doing both will allow the team to entirely focus on what is essential: the germane load, a.k.a Business Domain. But germane load should also be limited, and we will do it by decomposing solutions into smaller domain-oriented data products. All of the principles are influenced by socio-technical ways of thinking. Domain-oriented decentralized data ownership and architecture reduce the germane load by decentralizing responsibility for domains and corresponding data. Data as a product reduces collaboration between teams to access data by making data findable, accessible, interoperable, and reusable; it limits extraneous load by exposing data in the expected and known format and ports by consuming teams. Self-serve data infrastructure as a platform - it reduces extraneous load by providing a self-service platform. Federated computational governance makes collaboration between teams easier by enforcing common standards and patterns; it enables team members’ transfers by reducing possible technology stack within the company. As you can see, at the heart of every principle, there is a socio-technical way of thinking. Due to the socio-technical nature of this paradigm shift, it's important to be clear about the benefits that this shift can possibly result in. So next, we look at the key benefits. While Data Mesh is a socio-technical solution to many problems, there are some major challenges associated with it. Let’s explore them in detail. 1.7 Data Mesh challenges Data Mesh is neither a silver bullet nor a plug-and-play solution. Instead, its successful implementation requires overcoming an array of challenges. We will cover the technological, data management, and then the organizational challenge in turn. 35
  • 45. ©Manning Publications Co. To comment go to liveBook 1.7.1 Technological challenge The most crucial technological challenge of Data Mesh implementation is ensuring proper tooling, available to both domain-focused and central platform data teams. Only providing data discoverability and links between domains may prevent the whole ecosystem from becoming a siloed world. The other challenge is ensuring the tooling developed locally is shareable across domains. Flexibility and autonomy of teams shall not lead to inefficiencies at the company level and reduced synergy leverage. To effectively develop and maintain their data solutions, domain data teams need safe and secure access to data storage services. Centralizing, e.g., maintenance of cloud computing environment is much more efficient than the multiplication of this effort by each team. In addition, centralized control over cloud storage and computing can also help introduce a proper spending regime. We’ve seen way too often situations where seemingly rational spending by multiple separated teams summed up to amounts far exceeding levels justifiable at the company level. With a plethora of distributed, semi-independent Data Products, the development of consistent monitoring, alerting, and logging procedures becomes a challenge. As a result, both central platform and domain teams need to collaborate closely to ensure continuous insight and control over data quality and security and maintenance efficiency. 1.7.2 Data Management challenge Like the first technological challenge, the critical data-management-related challenge is ensuring data is easy to locate and comprehensively documented. The importance of a data catalog cannot be overestimated. It is not a challenge unique to Data Mesh, but as it’s based on the decentralization paradigm, the consequences of overlooking it would be much direr. There will be no central data team capable of ad hoc identification of available data assets (not that such ad hoc querying would be an advisable form of interacting with, e.g., Data Lake contents!). Another challenge is ensuring control over the duplication of data across different domains. With each new business domain needing to be served with repurposed data, redundancy may emerge. It is not only impacting the costs of data storage and management but introduces lineage and versioning problems. Data consumers may analyze outdated, incomplete, or altered data and, as a result, get to misleading conclusions. The third challenge is not specific to Data Mesh, however, one needs to remember about it when embarrassing on Data Mesh journey. It lies in cross-domain analytics. As business analysts and data scientists will not have access to a harmonized enterprise-wide data model, they may need additional platforms to aggregate and consolidate the various data products. Work on distributed data sources may also lead to spaghetti data pipelines, reducing their efficiency and reusability. Cross-domain access also challenges the data safeguarding and access controls, as many-to-many relationships require simultaneous access to multiple data products. 36
  • 46. ©Manning Publications Co. To comment go to liveBook 1.7.3 Organizational challenge Organizational challenges in the Data Mesh context break down into three different aspects, scope, data practices and culture. The first organizational challenge is the scope of such a paradigm shift. It requires coordinated efforts from multiple engaged parties from different business areas: IT, the business units, and senior management. Second, it does require a level of maturity in the handling of data by multiple parties. Centralized parties, data creators, managers, and users, will need to learn to handle data with a certain level of proficiency. Third, in most enterprises, going from data “as-a-byproduct” to “data-as-a-product” will be a substantial cultural shift. A shift that will not only involve data engineers and the IT departments, but the product, management, marketing, sales, and most other departments working with data. Decentralizing the decision-making process, shaping Data Product Owner positions, developing multiple cross-functional teams, and shifting responsibilities require a significant restructuring of existing organizational charts. Moreover, such changes come with costs in time, brainpower, and finances. Federated governance requires the development of novel procedures, starting from decision making to communication methods. These responsibilities and principles have to be appropriately identified and federated. As we come to the end of this chapter, you should now have a better understanding of Data Mesh as a joint technological and organizational paradigm, as well as its underlying principles. In the next chapter, we will show you how to build a start Data Mesh in your organization within a month. 1.8 Summary • The cornerstone of Data Mesh is decentralization: moving ownership and responsibility for data closer to its source. It aims to keep data in sync with the subject matter, to remove bottlenecks of long data pipes and central team inefficiencies. • Data Mesh is a socio-technological architecture paradigm based on principles of: domain-oriented data ownership, treating data as a product, federated computational governance, and self-serve data infrastructure as a platform. • Domain-oriented data ownership means the data and its relevant components should be owned, maintained, and developed by the people closest to it, the people inside the data’s domain. • Treating data as a product means a conscious design of the outcome presented ot the outside environment, clearly assigned role for ensuring said outcome availability and quality, and autonomy of development required to produce said outcome. • Federated computational governance means two things. First, the responsibility for data governance is split between central body responsible for company-wide policies and standards, and owners of highly autonomous units, usually domain-oriented, responsible for development of actual systems. Second, means of rules enforcement should be automatically executed wherever it’s feasible. 37
  • 47. ©Manning Publications Co. To comment go to liveBook • Self-serve data infrastructure means extracting the duplicated efforts of the Data Mesh into a platform, thus only done once, but offered as a “service” to others. • Data Mesh, is not a precisely defined solution. The extent of application of each principle differ widely from application to application, and it will vary depending on your business case. • When considering the Data Mesh implementation, you must take into account major technological, data management-related and organizational challenges. 38
  • 48. ©Manning Publications Co. To comment go to liveBook 2 Is Data Mesh Right for You? This chapter covers • How to decide whether you should introduce Data Mesh to your organization • Decision drivers to consider before choosing a data architecture • Data Mesh compared to other popular data architectures • The steps for transforming your organization architecture into the Data Mesh In the first chapter we explained to you what the Data Mesh is and why you should consider implementing it in your company. In this chapter, we will answer two other essential questions: 1. Should I implement Data Mesh in my business, i.e., what are Data Mesh drivers? 2. How much effort does it take to implement Data Mesh, i.e., would benefits outweigh the effort of its implementation? The presence of the first question may surprise you in the context of this book. Data Mesh has become one of the hottest buzzwords in the industry. Many started to implement it without considering if it fits their organizations, or without considering all requirements and ramifications. We observed many similar rushes on patterns or practices in the past, like microservices or agile. Many of these ended up pretty badly, and people started to blame these patterns or practices afterward. The truth is, there is no silver bullet, and every pattern or practice has its area of applicability, and they come with trade-offs. Treat Data Mesh as yet another tool in your toolbox. In the second part of this chapter, we will focus on the example process of implementing Data Mesh. We will show you possible steps towards this goal and how all skills and knowledge you will learn in upcoming chapters fit together. But as we said before, the most important thing is to know if you need a data mesh. So let’s start there. 39
  • 49. ©Manning Publications Co. To comment go to liveBook 2.1 Analyzing Data Mesh drivers To answer whether Data Mesh would be the right choice for you, we need to analyze the decision drivers behind architecture selection. We split these drivers into three categories: • Business drivers • Organizational drivers • Domain data drivers Let’s start from the most crucial aspect, i.e., business reasons. 2.1.1 Business drivers We think you should start your analysis from the business side. Stop there if you can’t find a suitable business reason to start your Data Mesh journey. You’ll already have an answer: you don’t need a data mesh. BUSINESS STRATEGY Look at your business strategy at the company level or in the case of big corporations on the unit level. Search for the answers to the following questions: • Are we defining our company as data-driven, or are we planning to shift our strategy into more data-driven? • Do we have Objectives and Key Results (OKRs), or any goal-setting methodology, that require data to be met? NOTE: OKR OKRs are a methodology used to define goals for the company and on the individual level. This methodology is quite often used as a tool to implement business strategy. If the answer to any of these questions is yes, you can dig deeper and look for the specific business cases requiring data (which we cover in the next step). If the answer is no, your work is going to be more challenging because business cases for data needs are often hidden in business unit strategies or in specific business processes. It is going to take some laborious searching to find what you need. BUSINESS CASE AND ITS DATA NEEDS COMPLEXITY To build a viable business case for data mesh, start with answering the following questions: • Do you have specific business processes starting or ongoing with complex data needs? • Do you need to kick-off a product/project with complex data needs to accomplish OKR? By complex data need, we mean multidimensional analysis (ad hoc, AI/ML, reporting) run on top of the data coming from multiple sources. If answers to these questions are positive, then there is a chance you have a compelling business case. KEY POINT Data Mesh requires a business case and is useful when there are related complex data needs. Let’s look at the examples of business cases and their data needs. 40
  • 50. Other documents randomly have different content
  • 54. The Project Gutenberg eBook of Bothwell; or, The Days of Mary Queen of Scots, Volume 1 (of 3)
  • 55. This ebook is for the use of anyone anywhere in the United States and most other parts of the world at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this ebook or online at www.gutenberg.org. If you are not located in the United States, you will have to check the laws of the country where you are located before using this eBook. Title: Bothwell; or, The Days of Mary Queen of Scots, Volume 1 (of 3) Creator: James Grant Release date: September 11, 2017 [eBook #55527] Language: English Credits: Produced by Al Haines *** START OF THE PROJECT GUTENBERG EBOOK BOTHWELL; OR, THE DAYS OF MARY QUEEN OF SCOTS, VOLUME 1 (OF 3) ***
  • 56. BOTHWELL: OR, THE DAYS OF MARY QUEEN OF SCOTS. BY JAMES GRANT, ESQ., AUTHOR OF "THE ROMANCE OF WAR," "MEMORIALS OF EDINBURGH CASTLE," "THE SCOTTISH CAVALIER," &c., &c. IN THREE VOLUMES. VOL. I. LONDON: PARRY & CO., LEADENHALL STREET. MDCCCLI. M'CORQUODALE AND CO., PRINTERS, LONDON. WORKS, NEWTON. PREFACE.
  • 57. The leading event upon which the following story hinges, will be found in the illustrative notes at the end of the third volume, which will show that the Magister Absalom (so frequently referred to) was a real personage, who, in the days of Earl Bothwell, was a Protestant clergyman at Bergen, and author of a Diary named The Chapter Book. There is no style of reading more conducive to a good or evil result, than the historical romance, according to the manner it is treated, by a judicious or injudicious writer. I have been studious in avoiding any distortion of history, the tenor of which is so often misconstructed wilfully by writers of romance; for there are bounds beyond which not even they are entitled to go. The Scottish reader will find how closely I have woven up the stirring events of 1567 with my own story, which, in reality, contains much more that is veracious than fictitious. Thus, Bothwell's journey to Denmark—his conflict with John of Park—the Queen's visit to Hermitage—the assault on the house of Alison Craig—the brawl and assistance given the Earl (in mistake for Arran) by the Abbot of Kilwinning—and many other incidents, all occurred actually as related. With one or two exceptions, every character in the following pages was a bona fide personage "of flesh and blood," who existed at the time, and was an actor in the scenes narrated. In the general grouping, costume, and other dramatic accessaries, I have endeavoured (as closely as I could) to draw a picture of the Scottish court and metropolis in the year 1567, at a time when the splendour of both was dimmed by the poverty which followed the wars and tumults of the Reformation; and with what
  • 58. success, I may say with the old knights of Cumbernauld—"Let the deed shaw!" EDINBURGH, September, 1851. CONTENTS OF VOL. I. CHAPTER I. The Castle of Bergen II. Erick Rosenkrantz III. The Strangers IV. A Norse Supper V. The Earl and Hob Discourse VI. Anna VII. Konrad VIII. The Cock-of-the-Woods IX. Lord Huntly's Letter X. The Hermit of Bergen XI. The Fleur-de-Lys XII. The Isle of Westeray XIII. Noltland XIV. The Separation XV. Doubt and Despair XVI. Blantyre Priory XVII. The Countess of Bothwell
  • 59. VIII. The Rescue XIX. The Rejected and the Rival XX. Konrad and the Countess XXI. Disappointment XXII. The Countess Jane XXIII. The Pursuit XXIV. Mary, Queen of Scots BOTHWELL; OR, THE DAYS OF MARY QUEEN OF SCOTS. CHAPTER I. THE CASTLE Of BERGEN. The stern old shepherd of the air, The spirit of the whistling hair, The wind has risen drearily In the northern evening sea, And is piping long and loud From many a heavy up-coming cloud. Leigh Hunt.
  • 60. It was the autumn of a bleak day in the September of 1566. Enveloped in murky clouds, through which, at times, its red rays shot along the crested waves, the Norwegian sun was verging to the westward. From the frozen Baltic a cold wind swept down the Skager Rack, and, urged by the whole force of the Atlantic ocean, the sullen waves poured their foam upon the rocky bluffs and fissured crags that overhang the fiord of Christiana. In those days, a vessel in the fiord proved an object of the greatest interest to the inhabitants of the hamlet; and it was with growing fears that the anxious housewives and weatherwise fishermen of Bergen, a little wooden town situated on the bay of Christiana, watched the exertions made by the crew of a small crayer or brigantine, of some eighty tons or so, that under bare poles, or having at least only her great square spritsail and jib set, endeavoured to weather the rocky headland to the east, and gain their little harbour, within which the water lay smooth as a millpond, forming by its placidity a strong contrast to the boiling and heaving ocean without. The last rays of the September sun had died away on the pine- clad hills of Christiana and the cathedral spire of Bergen. Night came on sooner than usual, and the sky was rendered opaque by sable clouds, through which the red streaks of lightning shot red and forklike; while the hollow thunder reverberated afar off among the splintered summits of the Silverbergen. Then through the flying vapour, where, parted by the levin brand, the misty rain poured down in torrents on the pathless sea, and the goodwives of Bergen told their beads, and muttered a Hail Mary! or a prayer to Saint Erick the Martyr for the souls of the poor mariners, who, they were assured, would find their graves at the bottom of the
  • 61. deep Skager Rack ere morning brightened on the waters of the Sound. The royal castle of Bergen, a great square tower of vast strength and unknown antiquity, reared on a point of rock, still overlooks the town that in the year of our story was little more than a fisher hamlet. Swung in an iron grating on its battlement, a huge beacon fire had been lighted by order of the governor to direct the struggling ship; and now the flames from the blazing mass of tarred fagots and well- oiled flax streamed like a torn banner on the stormy wind, and lit up the weatherbeaten visages of a few Danish soldiers who were grouped on the keep, glinting on their steel caps and mail shirts, and on the little brass minions and iron drakes that peeped between the timeworn embrasures. Another group, which since sunset had been watching the strange ship, was crowded under the sheltering arch of the castle gate, watching for the dispersion of the clouds or the rising of the moon to reveal her whereabouts. "Hans Knuber," said a young man who appeared at the wicket, and whose half military attire showed that he was captain of the king's crossbowmen at Bergen, "dost thou think she will weather the Devil's Nose on the next tack?" "I doubt it much, Captain Konrad," replied the fisherman, removing his right hand from the pocket of his voluminous red breeches to the front of his fur cap, "unless they steer with the keep of Bergen and the spire of the bishop's church in a line; which I saw they did not do. Ugh! yonder she looms! and what a sea she shipped! How heavily her fore and after castles and all her top-hamper make her heel to leeward!"
  • 62. "They who man her seem to have but small skill in pilot-craft," said one. "By Saint Olaus!" cried another, "unless some one boards and pilots her, another quarter of an hour will see her run full plump on the reef; and then God assoilzie both master and mariner!" "Luff—luff—timoneer!" exclaimed the first seaman. "Now keep her full! Would I had my hands on thy tiller!" "Every moment the night groweth darker," said the young man whom they called Konrad, and whom they treated with marked respect: "as the clouds darken the lightning brightens. A foul shame it were to old Norway, to have it said that so many of us—stout fellows all—stood idly and saw yonder struggling ship lost for lack of a little pilot-craft: for as thou sayest, Hans, if she runs so far again eastward on the next tack she must strike on the sunken reefs." "No boat could live in such a sea," muttered the fishermen as they drew back, none appearing solicitous of the selection which they expected the young man would make. "The mists are coming down from the Arctic ocean—the west wind always brings them," said Jans Thorson; "and we all know 'tis in these mists that the spirits of the mountain and storm travel." "Come hither, Hans Knuber," said the captain, whose plumed cap and rich dress of scarlet velvet, trimmed with white fur, and braided with silver like a hussar pelisse, were rapidly changing their hues under the drenching rain that lashed the castle wall, and hissed through the deep-mouthed archway. "Come hither, thou great seahorse! Dost mean to tell me thou art afraid?" "Sir captain, I fear neither the storm nor the spirit of the mist; but Zernebok the lord of evil may be abroad to-night, and he and the
  • 63. Hermit of the Rock may chance to remember how once in my cups, like an ass as I was, I reviled and mocked them both." "Bah!" retorted Konrad, whose superstition did not go so far as that of the seaman; "Jans Thorson, I will give thee this silver chain to launch and put forth to yonder ship. Come, man—away, for the honour of old Norway!" "Not for all the silver in yonder hills, sir captain, nor the copper in the mines of Fahlun to boot, would I trust myself beyond the Devil's Nose to-night," said the old fisherman bluntly. "I have just refused Master Sueno, the chamberlain." "Why, 'twas just in such a storm old Christian Alborg, and his stout ship the Biornen, were blown away into the wide ocean," said another; "and I marvel much, noble Konrad, that you would urge poor fellows like us"—— "On a venture which I would not attempt myself!" exclaimed the young man, whose dark blue eyes flashed at his own suggestion. "Now, Saint Olaus forefend thou shouldst say so!" "Nay, noble Konrad"—— "But thou dost think so?" The fisherman was silent. A flush crossed the handsome face of Konrad of Saltzberg. He looked seaward a moment. The wind was roaring fearfully among the bare summits of the cliffs that towered abruptly from the shore to the very clouds—absolute mountains of rock rising peak above peak; and when the blue lightning flashed among them, their granite tops were seen stretching away in the distance, while the giant pines that flourished in their clefts and gorges, were tossing like black ostrich feathers in the storm.
  • 64. At the harbour mouth the waves of snow-white foam were visible through the gloom, as they lashed, and hissed, and burst in successive mountains on the rocks of worn granite that fringed the entrance of the haven. Konrad cast a rapid glance around him, and the appalling fury of the northern storm made even his gallant heart waver for a moment in its generous purpose; but a fair female face, that with all its waving ringlets appeared at a little casement overlooking the portal, and a kiss wafted to him from "a quick small hand," decided him. His eyes sparkled, and turning briskly round to the fishermen, he said,— "By my honour, Sirs, though knowing less of pilot-craft than of handling the boll of an arblast, I will prove to you that I require nothing of any man that I dare not myself attempt—so thus will I put forth alone—and even if I perish shame you all." And, throwing aside his sword and short mantle, the young man rushed down the steep pathway that led to the little pier, and leaped on board one of the long light whaleboats that lay there; but ere his ready hand had quite cast off the rope that bound it to a ringbolt on the mole, both Hans Knuber and Jans Thorson, fired by his example, sprang on board, and with more of the action of elephants, in their wide fur boots and mighty breeches, than the agility of seamen, they seized each an oar, and pushed off. In Denmark and Norway, there were and are few titles of honour; but there has always existed in the latter an untitled nobility, like our Scottish lairds and English squires, consisting of very old families, who are more highly revered than those ennobled by Norway's Danish rulers; and many of these can trace their blood back to those terrible vikingr or ocean kings, who were so long the conquerors of the English Saxons, and the scourge of the Scottish shores.
  • 65. Konrad of the Saltzberg (for he had no other name than that which he took from a solitary and half-ruined tower overlooking the fiord) was the representative of one of those time-honoured races. The fame his brave ancestors had won under the enchanted banner of Regner Lodbrog, Erick with the bloody axe, and Sigwardis Ring, yet lived in the songs and stories of the northern harpers; and Konrad was revered for these old memories of Norway's ancient days, while his own bravery, affability, and handsome exterior, gained him the love of the Norse burghers of Bergen, the Danish bowmen he commanded, the fishermen of the fiord, and the huntsmen of the woods of Aggerhuis. By the glare of the beacon on the castle wall, his boat was briefly seen amid the deepening gloom as it rose on the heaving swell, and the broad-bladed oars of his lusty companions flashed as they were dipped in the sparkling water. A moment, and a moment only, they were visible; Konrad was seen to move his plumed cap, and his cheerful hallo was heard; the next, they had vanished into obscurity. The fishers gazed on the gloom with intensity, but could discover nothing; and there was no other sound came on the bellowing wind, save the roar of the resounding breakers as they broke on the impending bluffs. CHAPTER II. ERICK ROSENKRANTZ. Turn round, turn round, thou Scottish youth,
  • 66. Or loud thy sire shall mourn; For if thou touchest Norway's strand, Thou never shalt return. Vedder. The hall of the castle of Bergen was a spacious but rude apartment, spanned by a stone arch, ribbed with massive groins, that sprung from the ponderous walls. Its floor was composed of oak planks, and two clumsy stone columns, surmounted by grotesque capitals, supported the round archway of the fireplace, above which was a rudely carved, and still more rudely painted, shield, bearing the golden lion of ancient Norway in a field gules. Piled within the arch lay a heap of roots and billets, blazing and rumbling in the recesses of the great stone chimney. Eight tall candles, each like a small flambeau, flared in an iron candelabrum, and sputtered in the currents of air that swept through the hall. Various weapons hung on the rough walls of red sandstone; there were heavy Danish ghisannas or battle-axes of steel, iron mauls, ponderous maces, and deadly morglays, two-handed swords of enormous length, iron bucklers, chain hauberks, and leathern surcoats, all of uncouth fashion, and fully two hundred years behind the arms then used by the more southern nations of Europe. The long table occupying the centre of the hall was of wood that had grown in the forests of Memel; it was black as ebony with age, and the clumsy chairs and stools that were ranged against the walls were all of the same homely material. Several deerskins were spread before the hearth, and thereon reposed a couple of shaggy wolf-
  • 67. hounds, that ever and anon cocked their ears when a louder gust than usual shook the hall windows, or when the rain swept the feathery soot down the wide chimney to hiss in the sparkling fire. Near the hearth stood a chair covered with gilded leather, and studded with brass nails; and so different was its aspect from the rest of the unornamented furniture, that there was no difficulty in recognising it as the seat of state. A long sword, the silver hilt of which was covered with a curious network of steel, hung by an embroidered baldrick on one knob thereof, balanced by a little velvet cap adorned with a long scarlet feather, on the other. The proprietor of these articles, a stout old man, somewhere about sixty-five, whose rotundity had been considerably increased by good living, was standing in the arched recess of a well-grated window, peering earnestly out upon the blackness of the night, in hope to discern some trace of that strange vessel, concerning which all Bergen was agog. His complexion was fair and florid, and though his head was bald and polished, the long hair that hung from his temples and mingled with his bushy beard and heavy mustaches, was, like them, of a decided yellow; but his round visage was of the ruddiest and most weatherbeaten brown. There was a bold and frank expression in his keen blue eye, that with his air and aspect forcibly realized the idea of those Scandinavian vikingr who were once the tyrants of Saxons, and the terror of the Scots. His flowing robe of scarlet cloth, trimmed with black fur and laced with gold, his Norwayn anlace or dagger, sheathed in crimson leather sown with pearls, and the large rowelled spurs that glittered on the heels of his Muscovite leather boots, announced him one of Norway's untitled noblesse. He was Erick Rosenkrantz of Welsöö, governor of
  • 68. the province of Aggerhuis, castellan of Bergen, and knight of the Danish orders of the Elephant and Dannebrog. "Sueno Throndson," said he to a little old man who entered the hall, muffled in a mantle of red deerskin, which was drenched with rain, "dost thou think there is any chance of yonder strange bark weathering the storm, and getting under the lee of our ramparts?" "I know not, noble sir," replied Sueno, casting his drenched cloak on the floor, and displaying his under attire, which (saith the Magister Absalom Beyer, whose minute narrative we follow) consisted of a green cloth gaberdine, trimmed with the fur of the black fox, and girt at the waist by a broad belt, sustaining a black bugle-horn and short hunting sword. "I have serious doubts; for the waves of the fiord are combating with the currents from the Skager Rack, and whirling like a maelstrom. I have been through the whole town of Bergen; but neither offer nor bribe—no, not even the bishop's blessing, a hundred pieces of silver, and thrice as many deer hides—will induce one of the knavish fishermen or white-livered pilots to put forth a boat to pick up any of these strangers, who must all drown the moment their ship strikes; and strike she must, if the wind holds." "The curse of Saint Olaus be on them!" grumbled the governor, glancing at a rude image of Norway's tutelary saint. "Amen!" added Sueno, as he wrung the wet tails of his gaberdine. "Didst thou try threats, then?" "By my soul, I did so; and with equal success." "Dost thou gibe me, Throndson? This to me, the governor of Aggerhuis, and captain of the king's castle of Bergen!" muttered the portly official, walking to and fro, and swelling with importance as he spoke.
  • 69. "The oldest of our fishermen are ready to swear on the blessed gospels that there has not been seen such a storm since Christian Alborg, in the Biornen, was blown from his moorings." "Under the ramparts of this, the king's castle, by foul sorcery; and on the vigil of Saint Erick the king, and martyr too! I remember it well, Sueno. But what! is the old Norse spirit fallen so far, that these villains have become so economical of their persons that they shrink from a little salt water? and that none will launch a shallop in such to save these poor strangers, who, unless they know the coast, will assuredly run full tilt on the Devil's Nose at the haven mouth? By Saint Olaus! I can see the white surf curling over its terrible ridge, through the gloom, even at this moment." "I said all this, noble sir," replied Sueno, brushing the rain from his fur bonnet; "but none attended to me save young Konrad of the Saltzberg, the captain of our Danish crossbowmen, who cursed them for white-livered coistrils, and launching a boat, with Hans Knuber and Jans Thorson the pilot, pushed off from the mole, like brave hearts as they are, in the direction of the labouring ship, which Konrad vowed to pilot round the Devil's Nose or perish." "Fool! and thou only tellest me of this now! Konrad—the boldest youth and the best in all old Norway!" exclaimed the burly governor. "Hah! and hath the last of an ancient and gallant race to peril his life on such a night as this, when these baseborn drawers of nets and fishers of seals hang back?" "His boat vanished into the gloom in a moment, and we heard but one gallant blast from his bugle ring above the roar of the waves that boil round that terrible promontory." "The mother of God pray for him—brave lad! What the devil! Sueno, I would not for all the ships on the northern seas, a hair of
  • 70. Konrad's head were injured; for though he is no kin to me, I love the lad as if he were mine own and only son. See that my niece Anna knoweth not of this wild adventure till he returns safe. She has seemed somewhat cold to him of late; some lover's pique"—— "I pray he may return, Sir Erick." "He must—he shall return!" rejoined the impetuous old knight, stamping his foot. "Yea, and in safety too, or I will sack Bergen, and scourge every fisher in it. From whence thought these knaves the stranger came?" "From Denmark." "Malediction on Denmark!" said Rosenkrantz, feeling his old Norse prejudices rising in his breast. "Assure me that she is Danish, and I will extinguish the beacon and let them all drown and be——!" "Nay, nay, Sir Governor, they know her to be a good ship of Scotland, commanded by a certain great lord of that country, who is on an embassy to Frederick of Denmark, and hath been cruising in these seas." "Then my double malediction on the Scots, too!" said the governor, as he turned away from the hall window. "And so say I, noble Sir," chimed in the obsequious chamberlain, as he raised the skirts of his gaberdine, and warmed his voluminous trunk hosen before the great fire. "Right, Throndson! though eight of our monarchs are buried in Iona, under the Ridge of the Kings, the death of Coelus of Norway, who is graved in the Scottish Kyles, still lives in our songs; and the fatal field of Largs, when aided by such a storm as this, the Scots laid Haco's enchanted banner in the waves." "And the wars of Erick with the bloody axe." "And of Harold Graafeldt, his son."
  • 71. "And Magnus with the Barefeet," continued the old man, whose eyes gleamed at the names of these savage kings of early Scandinavia. "Enough, Sueno," said the governor, who was again peering from the window into the darkness; "enough, or thou wilt fire my old Norse heart in such wise by these fierce memories, that no remnant of Christian feeling will remain in it. After all, it matters not, Scots or Danes, we ought to pray for the souls that are now perhaps, from yonder dark abyss, ascending to the throne of God unblessed and unconfessed," added the old knight, with a sudden burst of religious feeling. "God assoil them!" added Sueno crossing himself, and becoming pious too. From the windows of the hall little else was seen but the dark masses of cloud that flew hither and thither on the stormy wind; at times a red star shot a tremulous ray through the openings, and was again hidden. Far down, beneath the castle windows, boiled the fierce ocean, and its white foam was visible when the lofty waves reared up their crested heads to lash the impending cliffs; but we have said that the bosom of the harbour was smooth as a summer lake when compared with the tumult of the fiord of Christiana. Overhead, showers of red sparks were swept away through the gloom, from the beacon that blazed on the keep to direct the waveworn ship. "What led Hans Knuber and his brother knave of the net, to deem the stranger was a Scot? By her lumbering leeboard I would have sworn she was a Lubecker." "Nay, Sir, her high fore and after castles marked her Scottish build; and both Hans Knuber and Jans Thorson, who have eyes for these matters, and have traded to Kirkwall—yea, and even to that
  • 72. Scottish sea the fiord of Forth—averred she bore Saint Andrew's saltire flying at her mizen-peak——I see nothing of her now," continued Sueno. "See! why, 'tis so dark, one cannot see the length of one's own nose. They must have perished!" At that moment the flash of a culverin glared amid the obscurity far down below; but its report was borne away on the wind that roared down the narrow fiord to bury its fury in the Skager Rack. "God and St. Olaus be praised!" muttered the old knight, rubbing his hands: "they are almost within the haven mouth; another moment, and they will be safe." "Thou forgettest, noble sir," said the chamberlain, "that the stranger's pilot may be unacquainted with the nooks and crooks of our harbour, the rocks and reefs that fringe it, and that the water in some parts is two hundred fathoms deep." "Saidst thou not that Konrad and Hans Knuber had put off in a boat?" "True, true! A ray of light is shining on the water now." "Whence comes it?" "'Tis the hermit in the cavern under the rocks, who hath lit a beacon on the beach to direct the benighted ship." "Saint Olaf bless him! Hoh! there goeth the culverin again. We heard the report this time. They are saved! 'Tis Konrad of Saltzberg hath done this gallant deed, and heaven reward him! for many a poor fellow had perished else. Now that they are in safe anchorage, away Sueno Throndson, take thy chamberlain's staff and chain, man a boat, board this seaworn ship, and invite this Scottish lord to Bergen; for a foul shame it were in a knight of the Elephant, to permit the ambassador of a queen, to remain on shipboard after such a storm,
  • 73. and within a bowshot of his Danish majesty's castle: we would be worse than Finns or Muscovites. Away, Sueno! for now the storm is lulling, and under the lee of its high hills the harbour is smooth as a mirror." Thus commanded, Sueno unwillingly enveloped himself once more in the before-mentioned fur mantle, and retired. A blast of his horn was heard to ring in the yard as he summoned certain followers, who grumbled and swore in guttural Norse as they scrambled after him down the steep and winding pathway, that led from the castle gate to the mole of Bergen. CHAPTER III. THE STRANGERS. To tell the terrors of deep untried, What toils we suffer'd, and what storms defied; What mountain surges, mountain surges lash'd, What sudden hurricanes the canvass dash'd; To tell each horror in the deep reveal'd, Would ask an iron throat with tenfold vigour steel'd. Lusiad of Camoens. "How now, Anna! thou lookest as pale as if all the gnomes of the Silverbergen, or Nippen and Zernebok to boot, had been about thee. Art thou affrighted by the storm, child?" asked Erick, pinching the soft
  • 74. cheek of his niece, who at that moment had entered the hall, and glided to his side in one of the great windows. Her only reply was to clasp her hands upon his arm, and look up in his face with a fond smile. Anna Rosenkrantz was the only daughter of Svend of Aggerhuis, the governor's younger brother, who had fallen in battle with the Holsteiners. In stature she was rather under the middle height; and so full and round was her outline, that many might have considered it too much so, but for the exquisite fairness of her skin, the beauty of her features, and the grace pervading every motion. Norway is famed for its fair beauties, but the lustre of Anna's complexion was dazzling; her neck and forehead were white as the unmelting snows of the Dovrefeldt. From under the lappets of a little velvet cap, which was edged by a row of Onslo pearls, her dark-brown ringlets flowed in heavy profusion, and seemed almost black when contrasted with the neck on which they waved. Her eyes were of a decided grey, dark, but clear and sparkling. The curve of her mouth and chin were very piquant and arch in expression; her smile was ever one of surpassing sweetness, and at times of coquetry. A jacket of black velvet, fashioned like a Bohemian vest, trimmed with narrow edgings of white fur, and studded with seed pearls, displayed the full contour of her beautiful bust; but unhappily her skirt was one of those enormous fardingales which were then becoming the rage over all Europe. "Have the roaring of the wind and the screaming of the water- sprite scared thee, Anna?" continued the old man, who, like a true Nordlander, believed every element to be peopled by unseen spirits and imps. "By the bones of Lodbrog!" he added, patting her soft cheek with his huge bony hand, "my mind misgave me much that this
  • 75. last year's sojourn at the palace of Kiobenhafen would fairly undo thee." "How, good uncle?" said Anna, blushing slightly. "By tainting thine inbred hardiment of soul, my little damsel, and making thee, instead of a fearless Norse maiden, and a dweller in the land of hills and cataracts, like one of those sickly moppets whom I have seen clustered round the tabouret of Frederick's queen, when, for my sins, I spent a summer at his court during the war with Christian II., that tyrant and tool of the Dutch harlot, Sigiberta." "Indeed, uncle mine, you mistake me," replied Anna, "though I will own myself somewhat terrified by this unwonted storm." "There now! said I not so? Three years ago, would the screaming of the eagles, the yelling of the wood-demon, the howl of the wind, or the tumult of the ocean, when all the spirits of the Skager Rack are rolling its billows on the rocks, have affrighted thee? Bah! what is there so terrible in all that? Do not forget, my girl, that thou comest of a race of sea-kings who trace their blood from O'Ivarre—he who with Andd and Olaff ravaged all the Scottish shores from Thurso to the Clyde, and once even placed the red lion of Norway on the double dun of Alcluyd.[*] But I warrant thou art only terrified for young Konrad, who, like a gallant Norseman, hath run his life into such deadly peril." [*] A.D. 870 (Note by Mag. Absalom Beyer.) "Konrad—tush!" said Anna pettishly. "Ay, Konrad!" reiterated Erick testily; "which way doth the wind blow now? By my soul, damosel, thou takest very quietly the danger
  • 76. in which the finest young fellow in all Norway has thrust himself— when even the boldest of our fishers drew back. He departed in a poor shallop to guide yonder devilish ship round the dangerous promontory, and if the blessed saints have not prevailed over the spirits of evil, who make their bourne in the caverns of that dark ocean—then I say, God help thee, Konrad of Saltzberg! But fear not, Anna," continued the old man kindly, perceiving that she turned away as if to conceal tears; "for thy lover is stout of heart and strong of hand—and—there now!—the devil's in my old gossiping tongue—pest upon it!—I have made thee weep." Anna's breast heaved very perceptibly, and she covered her face, not to conceal her tears, but the smile that spread over her features. "Come, damosel—away to thy toilet; for know there is in yonder ship which we have watched the livelong day, and which has escaped destruction so narrowly, a certain great lord, who this night shall sup with us; for I have sent Sueno with a courteous message, inviting him to abide, so long as it pleases him, in the king's castle of Bergen. Be gay, Anna; for I doubt not thou wilt be dying to hear tidings of what is astir in the great world around Aggerhuis; for, during the last month since thy return here, thou hast moped like some melancholy oyster on the frozen cape yonder." "A great lord, saidst thou, uncle?" asked Anna with sudden animation. "Of Scotland—so said Sueno." Anna blushed scarlet; but the momentary expression of confusion was replaced by one of pride and triumph. "Did thou hear of any such at Frederick's court, little one?" "Yes—oh yes! there were two on an embassy concerning the isles of Shetland."
  • 77. "Ah! which that fool, Christian of Oldenburg, gave to the Scottish king with his daughter Margaret? Their names?" "I marked them not," replied Anna with hesitation; "for thou knowest, uncle mine, I bear no good-will unto these rough-footed Scots." "Keep all thy good-will for the lad who loves thee so well," said the old man smiling, as he pressed his wiry mustaches against her white forehead. "I see thou hast still the old Norse spirit, Anna. Though three centuries have come and gone since the field of Largs was lost by Haco and his host, we have not forgotten it; and vengeance for that day's slaughter and defeat still forms no small item in our oaths of fealty and of knighthood. But hark! the horn of Sueno! There are torches flashing on the windows, and strange voices echoing, in the court. Away, girl! and bring me my sword and collars of knighthood from yonder cabinet; for I must receive these guests as becomes the king's representative at Aggerhuis, and captain of his castle of Bergen." Anna glided from his side, and in a minute returned with a casket from the cabinet, and the long heavy sword that lay on the chair at the fireplace. She clasped the rich waistbelt round the old man's burly figure, and drawing from the casket the gold chain with the diamond Elephant, having under its feet the enamelled motto, "Trew is Wildbrat,"— and the woven collar bearing the red cross of the Dannebrog, she placed them round Sir Brick's neck, and the jewels sparkled brightly among the red hair of his bushy beard. She then glanced hurriedly at her own figure in an opposite mirror; adjusted the jaunty little cap before mentioned; ran her
  • 78. slender fingers through her long dark ringlets; smiled with satisfaction at her own beauty; and took her seat on a low tabouret near the great stuffed chair, between the gilded arms of which the pompous old governor wedged his rotund figure, with an energy that made his visage flush scarlet to the temples; and he had barely time to assume his most imposing aspect of official dignity, when the light of several flambeaux flashed through the dark doorway at the lower end of the hall, and the handsome commander of his crossbowmen, Konrad of Saltzberg, with his features pale from fatigue, and his long locks, like his furred pelisse, damp with salt water, and Sueno wearing his gold chain and key, having his white wand uplifted, and attended by several torch-bearers in the king's livery, preceded the strangers. The first who approached was a tall and handsome man, in whose strong figure there was a certain jaunty air, that suited well the peculiar daredevil expression of his deep dark eye, which bespoke the confirmed man of pleasure. He seemed to be about thirty years of age, and was clad in a shining doublet of cloth of gold, over which he wore a cuirass of the finest steel, attached to the backplate by braces of burnished silver. His mantle was of purple velvet lined with white satin; his trunk breeches were of the latter material slashed with scarlet silk, and were of that enormous fashion then so much in vogue, being so preposterously stuffed with tow, hair, or bombast, as to render even greaves useless in battle. He wore a long sword and Scottish dagger. His blue velvet bonnet was adorned by a diamond aigrette, from which sprung three tall white ostrich feathers. His eyes were keen, dark, and proud, and their brows nearly met over his nose, which was straight; he wore little beard, but his mustaches were thick and pointed upward. His page, a saucy-looking lad of sixteen, whom he jocularly called Nick (for his name was Nicholas