Key Data Management Requirements for the IoT

Editor's Notes

• #12: At the heart of the change is data generated by a myriad of new sensors and devices, and the role in plays in modern apps • 90% of the world’s data was created in the last two years • 80% of enterprise data is unstructured • Unstructured data growing 2x faster than structured In the IoT economy, data is the raw currency. How you stores, manages, analyzes and uses data has a direct impact on the your success.
• #13: RDBMS was only real database option up until relatively recently – great for structured data, but no good for multi-structured, polymorphic data generated by todays IoT applications (polymorphic = similar but different, e.g. address fields, or user profiles). Even historically, the RDBMS only held 15-20% of an organisation’s information assets. We now have the tools and technologies that can harness the other 80%
• #14: Based on both ours and Bosch’s own research and experience, we have identified the following 5 key capabilities for data management in IoT: Creating rich, functional applications: Data management must support the development of functionally rich applications with complex data and algorithms, with fast time to market and at low cost. Unlocking Business Agility: The ability to support many new and frequently changing business requirements, causing fast and continuous evolution of the underlying data model. Enabling a Single Point of Truth & Business Convergence: Aggregate multiple views of related data from multiple systems into one consistent version of the data. Real-Time Operational Insight: Support both transactional as well as analytical applications from the same data source Enterprise-Grade Platform: Provide highly scalable, cloud-based, robust and secure applications. We have all seen the impact of security breaches on a organisations. Reputational risk is a key challenge for a modern global brand. Explore each of these in turn
• #15: Work through requirements, tie back to 3 industry use cases we used in the first webinar – Retail, Manu and Telematics Today’s applications now incorporate a wide variety of data, bringing structured, semi-structured and unstructured data together to yield deeper operational insight into all areas of the business: • Retail: “farm-to-fork” initiatives and increasing regulatory requirements to prove food lineage require the addi-tion of sensors to generate audit trails tracking food production and transportation through the supply chain. • Manufacturing: Capturing time-series, event based sensor data directly from the production line enables manu-facturers to detect when processes are exceeding predefined tolerances and quickly take corrective action to avoid product wastage. • Telematics and Mobility: The engine data bus is already established as a standard in luxury cars and trucks, con-solidating individual events from sensors for engine diagnostics. The addition of new sensors enables richer appli-cations and an increasing amount of data is being pushed back to central servers, enabling a fleet management company to start building new asset management applications that compare drivers and vehicles across their fleet, identifying best -- and worst -- practices.
• #16: Semi-structured and unstructured data does not lend itself to be stored and processed in the rigid row and column format imposed by relational databases, and cannot be fully harnessed for analytics if stored in BLOBS or flat files. With sub-documents and arrays, JSON documents also align with the data structure of objects at the application level. This makes it easy for developers to map the information model of the device or asset to its associated document in the database. In contrast, trying to map the same object representation of the data to the tabular representation of an RDBMS slows down development. Adding Object Relational Mappers (ORMs) can create additional complexity by reducing the flexibility to evolve schemas and to optimize queries to meet new application requirements. Instead of spending a lot of time dealing with the impedance mismatch between the programming language and the database, the developers must be enabled to focus on creating rich, functional applications.
• #17: IoT is in its infancy. Changes in customer requirements, emerging standards and new use-cases demand flexible and dynamic development methodologies and data storage architecture. Retail: Technologies such as NFC and Apple’s iBeacon enable retailers to derive as much insight from customer movement around their physical stores as they are used to getting from tracking customer movement around their eCommerce stores. By capturing and visualizing data from these location-based sensors, retailers can build heat maps to optimize the placement of high-margin products. Manufacturing: Smart Factory concepts are proposing more flexible assembly lines and support for smaller batch sizes by moving away from centrally controlled systems towards chains of intelligent and more autonomous production modules which interact with each other directly via the products, e.g. using RFID to create a “product memory”. Telematics and Mobility: Rapid developments in automation and fleet management see each new generation of vehicle bristling with more sensors! Established telematics applications such as geo-location and engine diagnostics are being complemented by new services extending to areas such as fuel efficiency, driver safety, theft prevention and more.
• #18: Key here is the dynamic database schema The rapid evolution of IoT applications can be constrained by traditional software development methodologies -- for example, the waterfall approach places enormous dependency on the requirements defined upfront. In the IoT age organizations need flexible, iterative development practices to make it easy for teams to respond to new business and market demands, without being held back by rigid data models. We can’t predict what we will want to do with the data, how we will enrich it in the future and how it will be linked to other data. MongoDB’s dynamic schema means that application development and ongoing evolution are straightforward, enabling continuous integration as developers add new features. MongoDB enables developers to evolve the database schema through iterative and agile methodologies. Developers can start writing code and persist the objects without first pre-defining their structure. Each document (analogous to a row in a relational database) can have its own set of fields. Users can adapt the structure of a document’s schema just by adding new fields or deleting existing ones, making it very simple to handle to the rapidly changing data generated fast moving IoT applications. Contrast this with a traditional relational database -- the developer and DBA working on a new project must first start by specifying the database schema, before any code is written. At minimum this will take days; it often takes weeks or months. When in production need to schedule the necessary ALTER TABLE operations – can cause downtime, take weeks on large databases. As MongoDB allows schemas to evolve dynamically, such an upgrade requires modifying just the application, with typically no action required for MongoDB.
• #19: Building a single view of a business entity -- whether a physical asset or a customer -- can deliver a range of benefits, from improved cross-sell and upsell to enhanced operational insight and reduced costs: Retail: Building on the NFC and Beacon example earlier, retailers can instantiate a single view of their customers in real time, converging actual location in the store with their profile, purchase history and loyalty card details in order to deliver timely and targeted promotions. Manufacturing: Production line machines contain many discrete components, each with their own sensors. Bringing these together, along with the relevant service history can ensure optimum asset utilization and production line efficiency. Telematics / Mobility: Fleet managers can blend views of a vehicle’s real time operational performance and diagnostics against asset registers that track service history to optimize preventative maintenance schedules.
• #20: Creating this “single point of truth” requires aggregating multiple views of related data distributed across different source systems into one consistent view. Using a relational database, the development and DBA team would first have to undertake lengthy design reviews in order to pre-define a common schema. Subsequent changes to the any of the source schemas would then necessitate associated changes to the single view schema. MongoDB’s dynamic schema and flexible document model do not impose the same constraints, enabling source systems to continuously evolve without impacting the single view needed by the business.
• #21: IoT applications enable new levels of operational insight and business discovery, but their value can only be fully realized when analysis is delivered in real time -- providing the ability to react and respond as processes are in-flight. Retail: Inventory is tracked as it moves from shelf to basket while the retailer concurrently performs analytics that attempt to match available supply to predicted demand, adjusting for any deviation automatically through warehouse operations and the supply chain. Manufacturing: Sensor data from robotic systems is persisted to the database while analytics work in the background to identify optimizations to the production line Telematics / Mobility: Engine diagnostics is enhanced by writing a continuous stream of sensor data to the database while simultaneously performing analytics comparing current status to historical baseline readings in order to proactively identify deviations and potential faults. For example, changes in oil or engine temperature may indicate the need to perform preventative maintenance.
• #22: Many traditional databases support operational applications by capturing structured data as it is generated. They then rely on slow moving batch ETL (Extract Transform Load) processes to replicate the data to the Enterprise Data Warehouse (EDW) where it is blended with semi-and-unstructured data for OLAP (OnLine Analytical Processing). To eliminate the analytics latency that inhibits real time business insight, it is necessary for the database to support both operational and analytical processes across the same data source handling structured, semi-structured and unstructured data. You can do this on MongoDB – powerful query framework over multi-structured data. Could be run against secondaries so separate operational from analytical workloads
• #23: As IoT applications become embedded within the operational fabric of the business, they must deliver the scalability, availability and security demanded by any enterprise application. Business continuity and security are typically governed by strict mandates in every industry vertical. Specific examples include: ● Retail: The recent security breach of 70 million customer accounts at Target contributed to a 46% drop in net profits. The 2011 Playstation network breach at Sony Corporation is estimated to have cost the business over $4.5bn. As IoT applications are integrated into retail operations, they must provide security against attack if other businesses are not to suffer the same costs. Standards such as PCI-DSS and HIPAA (for those retailers selling pharmaceutical products) are also top of mind. ● Manufacturing: If a production line were to stop due to an unplanned failure -- even for a short period of time -- the costs can be significant, including lost production capacity, idle workers and scrapped product. As IoT is at the very heart of many production line systems, continuous availability should be a prime concern. ● Telematics / Mobility: more sensors in each vehicle. Smart vehicle technology filters down product lines into economy models
• #24: While databases such as MongoDB offer new capabilities for flexible data management and agile development methodologies, they cannot compromise on the enterprise-grade capabilities of traditional relational databases. Using MongoDB organizations can build fault tolerant and secure applications that scale-out on commodity hardware as data volumes generated by sensors continues to explode. Scale out on commodity hardware using application transparent automatic sharding Integrated replication enables us to create self healing clusters replicated within and across data centers Most robust security mechanisms of any leading NoSQL database, with authentication via LDAP, Kerberos or PKI certificates, authorisation via RBAC, field level redaction to restrict access to specific fields, in built auditing and encryption
• #28: We can help you get started – Bosch and MongoDB collaboratively deliver consulting and training