![© 2016 IBM Corporation13
Streaming Analytics Service in Bluemix
Provides the capabilities of Streams in the Bluemix Cloud
• Deploy any Streams application to run in the cloud
• The service has been generally available since October 2015
• Available in the US South, United Kingdom, and Sydney* Bluemix Regions
Capabilities at the initial GA included:
• Integration with key Bluemix services such as Watson IoT, Message Hub, and Cloudant.
• Sample, demo and reference applications.
• Support for many Streams toolkits
Users can create their Streams instance in seconds from the bluemix.net
web portal.
• Streams instances can be used interactively through the Streams web console or
programmatically via the Streaming Analytics REST API
• Instances are created initially with two dedicated application nodes, but users can
dynamically resize their instance
• Streams instances in Bluemix are highly available
Streams apps must be developed locally before deploying to the cloud
[ * syndicated in Sydney catalog with deployment in London]](https://image.slidesharecdn.com/ibmstreamsv4-161018144557/85/What-s-New-in-IBM-Streams-V4-2-13-320.jpg)
What's New in IBM Streams V4.2
- 1. © 2016 IBM Corporation What’s New in IBM Streams V4.2 Mike Spicer Distinguished Engineer - Lead Architect, IBM Streams
- 2. © 2016 IBM Corporation2 Important Disclaimer 2 The information contained in this presentation is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided “as is”, without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other documentation. Nothing contained in this presentation is intended to, or shall have the effect of: Creating any warranty or representation from IBM (or its affiliates or its or their suppliers and/or licensors); or Altering the terms and conditions of the applicable license agreement governing the use of IBM software. IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
- 3. © 2016 IBM Corporation3 IBM Streams V4.2 Overview Edge to Center analytics for IoT use cases • Integration with Apache Edgent and Watson IoT Platform Development with Python & Jupyter Notebooks • Python application API to develop Streams apps entirely in Python • Support for Python operators Business Rules integration • Business rules editor and associated SPL Type generation • Business Rules compiler for optimized execution Performance & application deployment improvements Automatic fusion & dynamic threading Security, serviceability & administration improvements New and improved toolkits • Speech to text
- 4. © 2016 IBM Corporation4 Edge to Center analytics for IoT use cases Apache Edgent for analytics at the edge • Programming model and lightweight embeddable runtime • Functional Flow API • Connectors and Runtime (executable graph of oplets) • Dynamic control (Change queue size, change out of band data limits) • Metrics (Optional, rich set of metrics using open source package) • Open Source Apache Incubation Project • Developed in the open at Apache using github and JIRA Streams Integration for Center to Edge Analytics • Streams pluggable “Device Hub” interface with IoT Platform implementation • Manage Edgent apps by device type from Streams • Control Edgent apps from Streams apps • Streams console shows summary health of devices and Edgent apps • Edgent app connects to Streams via IoTP (automatically configured) IBM Confidential
- 5. © 2016 IBM Corporation5 Streams, Edgent & Watson IoT Platform Integration Edgent for edge analytics – on device or gateway • Lightweight embedded streaming analytics runtime • Analyze events locally on the edge • Reduce communication costs by only sending relevant events IBM Watson IoT Platform for optional device hub • Device management • Message broker (MQTT) • Public device hub API supports custom device hub • Includes message brokers like Kafka IBM Streams for streaming analytics • High performance, full featured streaming analytics • Build windows of state and correlate across devices • Have access to data-of-record systems, e.g. medical history • Control edge device based upon analytics • Central job management/health summary • Automatic application connectivity IBM Watson IoT Platform Device Hub IBM Streams Streaming Analytics Edgent on Edge Devices and Gateways Cluster Gateway Edge DeviceEdge Device Messaging (MQTT, Kafka etc.)
- 6. © 2016 IBM Corporation6 Business Rules Integration Deep Integration of ODM Business Rules Tooling Integration • Embedded Business Rules project in Streams Studio • Business rules editor (ODM Rules Designer) in Streams Studio • Generates SPL types from business rules Rules Compiler • Business rules compiled to C++ operator(s) • Can generate an Operator per rules package Dynamic update of rules for running applications • Compiler generates new library for rules • Ruleset Operator loads new library dynamically
- 7. © 2016 IBM Corporation7 Python Topology API Implement Streams applications entirely in Python • Functional Flow API similar to Java Topology API • Define topology with source, sink and operations • Submit to standalone and distributed contexts • Use existing Python modules • Can use and invoke popular modules like SciKits • Support for Python V3.5 with Anaconda • Support for Python 2.7 planned Support for Jupyter notebooks • Build and submit Python Streams application directly from notebook • View stream data directly in python notebook • Generate dynamically updating visualisations Support for Python functions as Streams operators
- 8. © 2016 IBM Corporation8 Python Topology API
- 9. © 2016 IBM Corporation9 Performance and Application Deployment Improvements Parallel Region Improvements • Nested parallel regions • Parallel split to send tuples to all channels Consistent Region Improvements • Asynchronous non-blocking checkpoints • Increase concurrency of establishing and restoring consistent states • Streams Hyperstate Accelerator support for checkpoints Scheduling Improvements • Better utilization across different sized resources • The ability to reserve application resources for a specific purpose Job Configuration Overlays • Submission time Configuration & parameters in separate config file • Same application can be run with different configurations with no code changes Secure Application Configuration • A set of properties accessible to applications by name • Update notifications through JMX • Scoped at Streams Domain or Instance level
- 10. © 2016 IBM Corporation10 Automatic Fusion and Dynamic Threading Pre V4.2 Behaviour • Fusion determined at compile time • Changes to fusion require a recompile • Default 1 PE process per Operator • Threads defined manually and statically • Threaded ports New V4.2 Behaviour • Fusion determined at submission time • Default automatic fusion • ~ 1 PE process per 4 Cores • Based on resources available • Optional directed fusion • Specify number of PEs • 1 to 1 Legacy behaviour • Manual placement constraints take precedence • Automatic threading determined at runtime • Dynamically adjusts based on throughput All fused No fusing
- 11. © 2016 IBM Corporation11 Security, Serviceability and Administration Improvements Security Improvements • Inter host communication encryption • Management services and Inter PE connection using TLS/SSL • Kerberos support • Single Sign On (SSO) authentication • Microsoft Active Directory Authentication through Kerberos Tokens Serviceability Improvements • Improved defensive coding and error case handling • Improved logging and troubleshooting • Improved metrics and zookeeper monitoring Rolling Upgrades • Domain can be upgraded to new version with instances and jobs running • Each instance can be at a different version and upgraded independently • Rolling upgrade of instance with jobs running planned for future release Console custom templates for hyperlink targets Streamtool Improvements • Persistent command history in console mode • Manage jobs by jobname
- 12. © 2016 IBM Corporation12 New and Improved Toolkits Speech to Text toolkit • Converts audio input stream to text output • Uses Watson Speech to text technology Open Source toolkits added to Product in V4.2 • Datetime Toolkit • IoT Toolkit • Connectivity with IoT platform • JDBC toolkit • JSON toolkit • Network Toolkit • Ingest, parse and process network packets • Map ip addresses to locations, reassemble files from packets Geospatial toolkit enhancements • Support for shared maps for matching points to a map (PointMapMatcher) Cybersecurity toolkit enhancements • QRadar integration • Improved algorithms Telecommunications Event Data Analytics (TEDA) toolkit enhancements DPS toolkit support for Streams Hyperstate accelerator
- 13. © 2016 IBM Corporation13 Streaming Analytics Service in Bluemix Provides the capabilities of Streams in the Bluemix Cloud • Deploy any Streams application to run in the cloud • The service has been generally available since October 2015 • Available in the US South, United Kingdom, and Sydney* Bluemix Regions Capabilities at the initial GA included: • Integration with key Bluemix services such as Watson IoT, Message Hub, and Cloudant. • Sample, demo and reference applications. • Support for many Streams toolkits Users can create their Streams instance in seconds from the bluemix.net web portal. • Streams instances can be used interactively through the Streams web console or programmatically via the Streaming Analytics REST API • Instances are created initially with two dedicated application nodes, but users can dynamically resize their instance • Streams instances in Bluemix are highly available Streams apps must be developed locally before deploying to the cloud [ * syndicated in Sydney catalog with deployment in London]
- 14. © 2016 IBM Corporation14 Bluemix Streams Analytics – What’s new since Streams 4.1? New features of the service • Streaming Analytics Development Guide • Enabled use of Streams product REST API in the cloud • Subscription Price Plan • Auto-stop of idle PayGo instances • Real-time Hangout Detection (Geospatial Analytics Service) Integration with other Bluemix services • HDFS and HBase (BigInsights on Cloud) • Object Storage • RabbitMQ • DashDB and SQL Database • Secure Gateway • Message Connect (experimental) Additional demo and sample applications • Twitter Smackdown Sample for Development Guide • Water Conservation Starter Kit (IoT use case with Apache Edgent and IoT platform) • Airport Sentiment Demo
- 15. © 2016 IBM Corporation15 Bluemix Streaming Analytics – Exploiting new Streams 4.2 features Most new Streams 4.2 features are relevant to and can be used with the Streaming Analytics service in Bluemix: • Edge to Center analytics for IoT use cases • Development with Python & Jupyter Notebooks • Compiled business rules • Performance & application deployment improvements • Automatic fusion & dynamic threading • Inter host communication encryption • Toolkit improvements Target for Streams 4.2 availability in Bluemix is October 19th.
- 16. © 2016 IBM Corporation16 Find out more and download V4.2 QSE at STREAMSDEV https://developer.ibm.com/streamsdev/ Streaming Analytics Service in Bluemix https://console.ng.bluemix.net/catalog/services/streaming-analytics Get involved in the Apache Edgent Community http://edgent.incubator.apache.org/ THANK YOU