The map is NOT the territory.

I was bemoaning that ‘Nobody gets it!’ when discussing the recent feedback from my last article with my business partner. Why has there been such resistance to what I believe is a simple and more pragmatic approach to data warehousing?

My business partner, who happens to be a reservist in the Australian Defence Force (ADF), gets it. When we first started working together, he told me that as a data engineer, he hated data modellers - is that a general sentiment, please let me know? But over time, he began to appreciate the minimal modelling paradigm of Hook and is now a firm supporter; that’s why we’re business partners. During this particular conversation, he used a phrase which resonated with me. He said that in the ADF, they often say, “The map is NOT the territory”.

So I googled it, and discovered that “the map is not the territory” was a remark made by the Polish-American scientist/philosopher Alfred Korzybski. The meaning is that a map is only an abstract representation of reality. We need to unpack that a bit more.  

Maps are abstractions that describe the territory using a set of predefined concepts, such as roads, contours, buildings, train lines, rivers, and lakes. If we understand the fundamental concepts of a map, then we know how to read it and figure out how to navigate the territory.

But the map is not the territory; it doesn’t tell us what is happening at any given time. Are there road closures or floods? What is the weather like? Are the schools open? It is impossible to visualise the landscape by looking at a map without going out and seeing for yourself.

But are maps useful..?

It’s a rhetorical question; the answer is obvious. Of course maps are useful, which is why we have them. They enable us to focus on elements that help us figure out how to get from A to B. Drop a stranger into unknown territory and ask them to find their way home; they would be lost. Given them a map, they would have a fighting chance.

And so it is with a Hook data warehouse. Hook provides the map to help you navigate the data assets you have. We don’t need to model/shape the data into a logical structure, because that isn’t the structure of the data. Hook’s approach to integration allows you to navigate these assets regardless.

Is this map enough? In most cases, yes; armed with the raw source data organised around business concepts using frames and hooks, I would be confident that we can meet 80% of analytical use cases. Unfortunately, I can’t back that number up because you can’t measure what you don’t need to build!!!

I want to turn this argument around to those telling me that it is imperative to reshape the data to match some predefined view of what the business thinks it should be. Let’s apply that thinking to our map. If we build our map according to what we want to see in the landscape, we’d have to create/alter the landscape accordingly. That sounds like a lot of effort and time. It sounds expensive.

If you absolutely do want that kind of modelling, it doesn’t belong in the data warehouse. It should either be “shifted left” into the design of the operational source systems themselves, or “shifted right” into consumable assets for specific use cases. Dare I say it, we need to “shift out” from the data warehouse.

There is no substitute for shifting the problem left, but, as is often the case, especially in the data warehouse world, we have no control or influence over the design of operational source systems. We cannot ensure they are designed properly, and we must eat whatever is served to us. However, by following the Hook approach, we can at least minimise the modelling we need to perform on the right.