Get a "Farm to Table" View of Your Data: Tracking Data Quality and Lineage, on Premise and in the Cloud, On and Off the Cluster

Editor's Notes

• #3: Tendü Yoğurtçu, Ph.D., is Syncsort’s Chief Technology Officer (CTO). She has 20+ years of software industry experience, including extensive Big Data and Hadoop industry knowledge. As CTO, Tendu directs the company’s technology strategy and innovation, leading all product research and development programs. Prior to her CTO role, Tendü has served as Syncsort’s General Manager of Big Data, leading the global software business for Data Integration, Hadoop and Cloud, including sales, marketing, engineering and support. Tendu has held several engineering management roles where she directed the development of ETL, Sort, and Application Modernization products for Syncsort’s Data Integration business. She also was an Adjunct Faculty Member at the Computer Science Department at Stevens Institute of Technology.
• #4: So, think about this scenario… You’re a chef at a high-end restaurant that has made a name for itself by providing meals of the highest quality made with the freshest organic food. Not only do you need to run a tight ship inside your kitchen – selecting the best ingredients, ensuring high standards for how your food is stored, prepared and cooked… But you also need to know everything about that food BEFORE it was delivered to your kitchen door. Where and how was your food produced? Were your chicken’s free range? Are all the organic certifications valid and up to date? How was the food handled, transported and stored along the way? Was it kept at the proper temperatures to maintain freshness and safety? Did it come into contact with any other foods it shouldn’t have? What about ingredients – like your rice, pasta and spices -- that are imported from other countries? How do you trace them all the way back to where they were harvested or produced? out .. And how long will it take? An end to end view of where your ingredients came from … and what happened to them along the way … is critically important to maintain the restaurant’s reputation for high-quality food. However, this is obviously a complicated and difficult task. While different industries, there are many similarities between the chef’s mission to track her food supply from the farm to the table – and our need to have an end-to-end view of our data. We need to know all the various sources of the data we are working with – how it was moved, transformed, combined along the way – and then what’s happening to it once we are working with it in our Enterprise Data Hub or Data Lake. And, like the restaurant scenario, the task of doing this is critical to complying with regulations and ensuring high quality for our end consumers. And, the task of accomplishing this end to end view is also very challenging.
• #5: Some of the top technology trends are advancing what we do with data but also make it harder to get this view. Cloud: Rapid growth in cloud data volumes IT is a consumer, provider and broker of cloud services Hybrid is becoming the standard And this all drives the need for Data Governance – but also makes it harder to achieve
• #6: Data Science & AI: Investment growing Early adopters: Financial Services, Banking, Retail, Telco
• #7: IoT and Streaming Data: IoT adoption doubled between 2013-2017 Links to analytics, artificial intelligence (AI) and other critical digital initiatives
• #8: And this all drives the need for Data Governance – but also makes it harder to achieve
• #11: A better way is needed – so that, just like the chef, we can have a complete view of our data, from the origin to the data hub – and know what has happened to it at every step of the way
• #13: A better way is needed – so that, just like the chef, we can have a complete view of our data, from the origin to the data hub – and know what has happened to it at every step of the way
• #14: As mentioned earlier – lineage is very important, but it’s not the only factor to consider for governance. Data Quality plays a critical role in enterprise data governance as well. To meet the needs of our customers with Hadoop and Spark, we recently released Trillium Quality for Big Data, which runs natively in the cluster to ensure that the data lake doesn’t turn into a data swamp, while harnessing the processing power of the cluster to scale for massive data volumes.
• #15: Let’s take a look at how this works in the real world…
• #16: Let’s take a look at how this works in the real world…
• #19: #1 BIBD solutions provider # countries ?? # partners ??