Lunch and Learn: You have the data, now what?

25th July 2023
2
Lunch & learn
You have the data,
now what?

Who is DiUS?
● Ideation, innovation and labs
● Discovery services
● Experience and services design
● Digital product development
● Application modernisation
● Data platforms, analytics and AI/machine learning
● Internet of Things
Our services.
19 Years. 150 People. Australia + New Zealand
Specialists in using emerging technology to solve difficult
problems, get new ideas to market or disrupt traditional
business models.

Got a question?
slidio.com | #2040429

Data and analytics at DiUS
What we’re doing/
what we’re seeing

6
Meet your speakers
Nabi Rezvani
Lead Data Engineer
E: nrezvani@dius.com.au
Gaurav Thadani
Lead Software Engineer
E: gthadani@dius.com.au

● Introduction
● Problem space
- What are we dealing with
- Explosion of tools
- The process forward
● Product led thinking
- Applied lens
- Themes
- Approach
● Panel discussion
Agenda

Lunch and Learn: You have the data, now what?

● Data lake
● Data warehouse
● Data lakehouse
● Delta lake
● Data mesh
● Data fabric
Some terms to demystify?

Aspects - Data engineering vision
14

Data as a product
Product thinking
towards data

Tech vs product lens
Different lenses towards data initiatives
Data architecture and
infrastructure
Data processing
Data visualisation
Data streaming
Data pipelines
Data governance
ML engineering
Technology lens
Data led decision
making
Add business value
using data products
Measure success using
data
Predict patterns using
data
Product lens

Just like software products, we start from the business objective, then deﬁne the product speciﬁcations.
We need to build an API that shows us the best
selling products per category on a daily basis.
“ ”
Actor
Functional requirements
Non-functional requirements
Build a popular
products API
Marketing manager
Show best selling
products per category
Update daily
We do not have visibility on the best selling products
to run marketing campaigns on them.
“ ”
Increase revenue by
upselling popular
products via marketing
campaigns
How?
What?
Why?
Product lens towards data

Building a little bit in each
layer, not the entire layer
The “thin slice” approach

DiUS did some data engineering work with “Organisation X”.
The history:
● Organisation X, had invested in building their shiny data platform for three years.
● The objective was to build a self served, scalable and data platform which makes the data
discoverable and allows performant data processing.
Case study 1: Organisation X

We have a lot of
data across our
verticals, let’s get
everyone to ingest
their data into the
platform!
22

The “Data warehouse integration” project:
● Building the platform did not start with asking “what business problem we do we want to
address?” in the first place.
● The integration of the new data warehouse did not seem to add much value.
A business led project:
● Eventually a product leader came onboard and started bringing business problems to the
table to solve using the platform and the data(calculate the annual recurring revenue).
Recap of the Organisation X situation.

We gathered around the table with questions like:
● “Can we have more visibility on the installation
performance of the devices to measure the sales metrics
of the partners faster?”
● “Can we have the fleet view data of the battery and
bluetooth performance to be able to make informed
product roadmap decisions?”
● “Can we ingest the energy data and make it queribale so
that we can build new features around energy
consumption patterns?”
Case study 2: Powersensor
Powersensor is a self-installed energy measurement product that
enables households to monitor their home energy consumption and
solar generation easily.

Recap of the Powersensor use case:
● We initially started with the problem that we wanted to solve and focused on the business
outcome.
● We built thin vertical slices rather than making a huge investment in making the best “data
platform”.
● We focused on the functional requirements and did a trade-off between non-functional
requirements along the way to keep the solution cost low.
Case study 2: Powersensor

You have the data, now what?
Panel discussion

29
Meet your speakers
Jonathan Gomez
Head of Data Platforms,
Wesfarmers OneDigital - OnePass
John Sullivan
CEO,
ChargeFox

Know what you need/want
Avoid buying technology or using products just like that. Wire up to product requirements.
● Interviews and discussions with the product to understand the current status of the platform and the
business
● Anticipated requirements - short-mid-long term goals
● Conversation on why analytics?
○ Who needs to view it?
● Product Demo and walkthrough
● Review of the current product vision and roadmap
● Review of current backlog and proposed MVP
Presentation title
31

Data & tech lens
What are the data engineering tools and best practices in the market?

PRESALES FRAMEWORK
Data product
Collecting and making available data as it is
(perhaps with some small processing or
cleansing steps). User does all the processing.
Doing some of the processing on our side.
Example: with customer data, add additional
attributes like assigning a customer segment to
each customer.
Receive some data, run it through the algorithm
(machine learning or otherwise) then return
information or insights. Example: Google Image
(user uploads a picture, and receives a set of
images that are the same or similar)
Provide information to the user to help them with
decision-making but we are not taking the decision
ourselves. Examples: sales dashboard, Google
analytics
Outsource intelligence within a specific
domain. Example: recommending
products to users
From a blog post by Simon O’Regan https://towardsdatascience.com/designing-data-products-b6b93edf3d23

PRESALES FRAMEWORK
Data product
Voice, robotics, augmented
reality

Chatbots
segmentation dashboards
ML models
Recommendation
systems
Data product matrix
36

Chatbots
segmentation dashboards
ML models
Recommendation
systems
Data product matrix
37

Technology lens
○ Design, build and revamp
data platforms
○ Build data processing
pipelines
○ Improve the performance
of data processing
pipelines
Product lens
○ Address a business
problem using data
○ Design data products
○ Build, ship and surface data
products
Data services
38

Technology lens Product lens
Data services
39
Data architecture and
infrastructure
Data processing
Data visualisation
Data streaming
Data pipelines
Data governance
ML engineering
Data led decision
making
Add business value using
data products
Measure success
using data
Predict patterns
using data

Just like software world, we could approach each data opportunity with the product lens as well.
With software products, we start from the business objective, then define use cases and actors and
specify functional and non-functional requirements.
Product lens towards data
We need to build a sales dashboard that shows us
the best selling products per category
in a real time manner.
“ ”
Use case
Actor
Functional requirements
Non-functional requirements
Build a sales dashboard
Marketing manager
Show best selling
products per category
Real-time (10 seconds
latency)
We do not have visibility on the best selling products
to recommend them to more users.
“ ”
40
Business objective Upselling the best selling
products

They ended up with a complicated ecosystem of components, a lot of complexity and cost to deal with.
Motivation of the project:
● We have 1600 datasets in the platform and zero users, let’s change that by making people’s
lives easier. How about buying “Data warehouse Y”?
Outcome of the project:
● “Data warehouse Y” was integrated, but there was still no interest in the platform!
41
Use case 1

The Data warehouse integration project:
● Building the platform did not start with asking “what business problem we do we want to
address?” in the first place.
● No functional requirements and business outcomes were discussed in the first place.
● The integration of the new data warehouse did not seem to add much value
A business led project:
● Eventually a product leader came onboard and started bringing business problems to the
table to solve using the data and the platform (calculate the annual recurring revenue).
Recap of the Organisation X situation.
Use case 1

Powersensor is a self-installed energy measurement product that enables
households to monitor their home energy consumption and solar generation
easily.
We gathered around the table with questions like:
● “Can we have more visibility on the onboarding of the
devices to improve customer retention?”
● “Can we have the fleet view data of the battery and
bluetooth performance to be able to make informed
product roadmap decisions?”
● “Can we ingest the energy data and make it queribale for
further exploration?”
Case study 2:Powersensor

44
Research
Build
Rollout
Plan
Backlog of tasks
Prioritise into
phases 1,2,..
Define non-functional
requirements
Cost analysis
Data readiness
assessment
Build & test
Ship to prod
Data
onboarding
Iterate for
each phase...

cost latency
The devices are sending data
constantly, can we build a real
time dashboard showing battery
and bluetooth signal status?
Data engineer Product owner
The solution looks really good
but I can see that there is a
spike in the AWS cost
Yes we can, let’s design and
test a streaming solution to
achieve that
Alright, let’s go with a batch
design then
We can live with a few hours
latency, keeping the cost down
is very important though.
Compromising the latency
for lower cost
Yeah, streaming solutions are
expensive in general, unless
we don’t need to build a real
time dashboard.

Lunch and Learn: You have the data, now what?

More Related Content

Similar to Lunch and Learn: You have the data, now what? (20)

More from DiUS (20)

Recently uploaded (20)

Lunch and Learn: You have the data, now what?