Stream Analytics

Editor's Notes

• #6: *visualize real time *detect urgent situations *automate immediate actions Normal Programming >> code execution controls data Streaming Applications >> incoming data controls the code
• #7: «loosely real time»
• #8: pipeline
• #10: Atomic:  (aka one-key-value-at-a-time) Processes each inbound data event as a separate element.  Seems logical but this is the most computationally expensive design.  For example, it’s used to guarantee fastest processing of individual events with least delay in transmitting the event to the subscriber.  Seen often for customer transactional inputs so that if some element of the event block fails the entire block is not deleted but moved to a bad record file that can later be processed further.  Apache Storm uses this paradigm. Micro batching:  Yes these are batches of events but typically those that occur within only a few milliseconds.  You can adjust the time window.  This makes the process somewhat more efficient.  Spark Streaming uses this paradigm. Windowing:  Similar to batching, Windowing allows the design of micro batches that may be simple time-based batches, but also allows for many more sophisticated interpretations such as sliding windows (e.g. everything that occurred in the last X period of time).  This can be very useful for aggregating events or determining outliers when compared to averages or standard deviation.
• #11: At most once: data loss possible At least once: no data loss- redelivery attemps made, duplication possible Exactly once: typically require use of idempotent updates
• #12: Compute Intensity, the number of arithmetic operations per I/O or global memory reference. In many signal processing applications today it is well over 50:1 and increasing with algorithmic complexity. Data Parallelism exists in a kernel if the same function is applied to all records of an input stream and a number of records can be processed simultaneously without waiting for results from previous records. Data Locality is a specific type of temporal locality common in signal and media processing applications where data is produced once, read once or twice later in the application, and never read again. Intermediate streams passed between kernels as well as intermediate data within kernel functions can capture this locality directly using the stream processing programming model.
• #13: Most of these can be hidden within modern analytics products so that the user does not have to be aware of exactly how they are being used. Analytic logic must be predefined – daha önceki datalara dayanmamalı, independent belirlenmiş kurallar pipelineından geçiyor Stream processing is essentially a compromise, driven by a data-centric model that works very well for traditional DSP or GPU-type applications (such as image, video and digital signal processing) but less so for general purpose processing with more randomized data access (such as databases). By sacrificing some flexibility in the model, the implications allow easier, faster and more efficient execution. Depending on the context, processor design may be tuned for maximum efficiency or a trade-off for flexibility.
• #14: Zero-copy Kafka neden hızlı? **Arada context switch yok**   Zero Copy - basically it calls the OS kernel direct rather than at the application layer to move data fast. Batch Data in Chunks - Kafka is all about batching the data into chunks. This minimises cross machine latency with all the buffering/copying that accompanies this. Avoids Random Disk Access - as Kafka is an immutable commit log it does not need to rewind the disk and do many random I/O operations and can just access the disk in a sequential manner. This enables it to get similar speeds from a physical disk compared with memory. Can Scale Horizontally - The ability to have thousands of partitions for a single topic spread among thousands of machines means Kafka can handle huge loads
• #17: Storm is designed for massive scalability, supports fault-tolerance with a “fail fast, auto restart” approach to processes, and offers a strong guarantee that every tuple will be processed. Storm defaults to an “at least once” guarantee for messages, but offers the ability to implement “exactly once” processing as well. Storm is written primarily in Clojure and is designed to support wiring “spouts” (think input streams) and “bolts” (processing and output modules) together as a directed acyclic graph (DAG) called a topology. Storm topologies run on clusters and the Storm scheduler distributes work to nodes around the cluster, based on the topology configuration. You can think of topologies as roughly analogous to a MapReduce job in Hadoop, except that given Storm’s focus on real-time, stream-based processing, topologies default to running forever or until manually terminated. Once a topology is started, the spouts bring data into the system and hand the data off to bolts (which may in turn hand data to subsequent bolts) where the main computational work is done. As processing progresses, one or more bolts may write data out to a database or file system, send a message to another external system, or otherwise make the results of the computation available to the users. Hadoop: Store & Query. Store the data first, and indefinitely. Analyse when you like. Storm: Analyze while data is being produced. No need to store anything at all.
• #21: Spouttan her tuple unique bir messageid ile çıkıyor ve bu messageidnin ack veya fail edilmesi gerekiyor. Ack edilmeyenler failed sayılıp spouttan tekrar tetikleniyor. Her boltun işlediği tupleı acklemesi lazım.
• #27: Storm  cbdevapp02 Redis  nsidevapp01 redis/redis-4.0.2/src/redis-cli psubscribe WordCountTopology redis/redis-4.0.2/src/redis-server /WebSphere85/AppServer/java_1.8_64/bin/java -jar storm/Storm.jar 1>/dev/null 2>&1