Big Data London did not disappoint!

This year's #bigdataldn was a blast and it was fantastic to see so many familiar faces and many, many more startling innovations!

Thought I would drop my big takeaways from the conference this year below!

1. #dataproducts & #datamesh

Whilst being two separate (but related) topics, we saw a lot of insights in how people are formulating best practices for data rooted in these ideas & principles- and it was refreshing to see people embracing & adapting these notions to their context, rather than hesitating.

Big takeaway for me was "make good ideas your own, don't be wedded to dogmas".

2. #dataculture & #datatalent

We all know just how severe the global digital and data skills gap is, and there is lots of innovation in the recruitment industry & in management philosophies.

Big takeaway is while we continue to solve the global skills gap, we have to think about how our new ways of working will solve atrophy, disillusionment & attrition.

Life remains tough for #dataleaders trying to drive change.

3. #dataprivacy & #dataregulation

As the world becomes more globalised, we need to understand the implications of data crossing borders and being used in new contexts outside of their own in jurisdiction better.

There's lots of an innovation from data management & privacy vendors, but the big takeaway for me is that #dataprivacy & #datasecurity has to become more contextual to use- cases (i.e. the #dataproduct).

We need to exploit our #metadata, connect our tooling better and embed #datacultures.

4. #dataobservability & #dataops

Data management paradigms have tended to focus on achieving a target state, rather than focusing on gradual, micro changes and their potential (i.e. unknown) implications. E.g. focusing on periodic DQ metrics, rather than whether a change in a schema might impact downstream applications & users.

There was a lot of talk this year about data observability, and the big takeaway for me is that this represents a shift in the focus of data management away from "is it right?" towards "is this interesting?".

As we build increasingly complex, integrated and real-time solutions, we need to approaches rooted in gradual, incremental improvements over time- approaches rooted in #dataops.

5. #knowledgegraphs & #dataontologies

Historically bound to specific use cases, we are now seeing #knowledgegraphs becoming core to everything we do in Data. From Data Governance to Data Quality to Master Data Management, we saw how starting with the data and focusing on curating useful knowledge is the way forward.

The big takeaway for me is that #knowledgegraphs can both accelerate #datatransformation and ensure we focus on what matters- useful knowledge.

Everything we do in #datamanagement should focus on that.

6. #datafabric & #datavirtualization

We continue to see modern data architecture's plan for change, employing #datavirtualisation and #datafabric techniques to enable modularised #dataarchitectures.

How this fits together at scale is still a little unclear, but the big takeaway is the continued trend towards:

1. Leaving data where & as it is,
2. Making it accessible with deep context,
3. Enable multiple modes of security & control,
4. Processing in ways built for change, and
5. Avoiding legacy bottlenecks & #techdebt.

Lot's to solve in this space, but a direction of travel remains clear.

7. #technologypartners & #opensource

The technology landscape for data has exploded in the last few decades, with competing solutions geared for increasingly niche use cases. For example, building understanding of data and keeping it secure both rely on common capabilities for #metadatamanagement, etc. But there was a notable shift away from walled gardens and towards integration & sharing, as we realise that we need portability & integration to achieve #bestofbreed.

So a big takwaway is avoiding dependencies and designing architectures to leverage multiple, niche technologies seamlessly. For example, having a #datacatalogue behind the scenes but using a purpose- built tool for #dataprivacy with baked in regulatory insights.

A complex challenge for technology offenders, and a daunting one for #dataleaders. But no doubt a direction of travel that is suited to the need for #dynamic #dataarchitectures.

Juan Sequeda Jon Cook Chris Probert Amit G. James Miller James Baldry Stephen Perry Zhamak Dehghani Scott (not coming back) Hirleman Jubair Patel Omar Khawaja