Kafka Summit SF 2017 - Accelerating Particles to Explore the Mysteries of the Universe and How Kafka Can Help on That
1 like625 views
The document discusses the CERN particle accelerator, highlighting its complexity and unique challenges in control and operation, including systems for monitoring and data handling. It emphasizes the use of Kafka for logging, computing security, and data extraction processes, showcasing the significant data throughput and user engagement. Additionally, it outlines the Kafka as a Service infrastructure, offering self-service cluster management and configurations tailored to various use cases.
![18
0
200
400
600
800
2014
Data ingestion Per day GB
requests per day. Direct SQL access is not permitted.
• A generic Java GUI called TIMBER is also provided
as a means to visualize and extract logged data. The
tool is heavily used, with more than 800 active users.
Figure 2: Logging ervice architecture overview.
The Java APIs for both logging and extracting data are
procedures have been he
an understanding of h
performing, in terms of
how it is being done, and
Optimal Use of Softw
The database mod
business logic (written i
Java infrastructure intera
engineered to use the
Oracle, to maximize per
the LS systems are be
aforementioned instrume
which features and tech
performance.
Data Quality Contro
The MDB (introduce
filtering capabilities wh
for long-term storage.
effort [4] to ensure
configurations, the MD
Filters for
data
Reduction
~ 250’000Signals
~ 50 data loading processes
~ 5.5 billion records per day
~ 275 GB / day
à 100 TB / year throughput
~ 1 million signals
~ 300 data loading processes
~ 4 billion records per day
~ 160 GB / day
à 52 TB / year stored
Ø +800 extraction clients
Ø +5 million extraction requests per day
Ø 130 custom applications
Credit: BE-CO-DS
CERN ACCELERATORLOGGING](https://image.slidesharecdn.com/martinmarquezcern8-170907182638/85/Kafka-Summit-SF-2017-Accelerating-Particles-to-Explore-the-Mysteries-of-the-Universe-and-How-Kafka-Can-Help-on-That-18-320.jpg)
Kafka Summit SF 2017 - Accelerating Particles to Explore the Mysteries of the Universe and How Kafka Can Help on That
- 2. ACCELERATINGPARTICLESTO EXPLORE THEMISTERIESOF THE UNIVERSE ANDHOWKAFKACANONTHAT Manuel Martín Márquez CERN – Scalable Analytics Services Kafka Summit 2017 – San Francisco
- 4. 4Manuel Martin Marquez Intel IoT Ignition Lab – Cloud and Big Data Munich, September 17thCERN - European Laboratory for Particle Physics
- 5. 5Manuel Martin Marquez Intel IoT Ignition Lab – Cloud and Big Data Munich, September 17th A World-Wide Collaboration
- 6. 6 WHAT ISTHE UNIVERSEMADE OF? HOWDITIT START? FUNDAMENTALRESEARCH
- 7. 7Manuel Martin Marquez Intel IoT Ignition Lab – Cloud and Big Data Munich, September 17th
- 8. 8
- 9. 98/9/17
- 11. 8/9/17 Documentreference 11 CERN Aerial View World’s largest scientific instrument 27km, 6000+ superconducting magnets Emptiest place in the solar system High vacuum inside the magnets Hottest spot in the galaxy During Lead ion collisions create temperatures 100 000x hotter than the heart of the sun; Fastest racetrack on Earth Protons circulate 11245 times/s (99.9999991% the speed of light)
- 12. 8/9/17 Documentreference 12 LHC Installation CERN Accelerator Complex is unique installation Therefore, we have to face unique challenges Control and Operations Million of sensors, large number of control devices, front-end equipment, etc. Many critical systems: Cryogenics, Vacuums, Machine Protection, etc.
- 13. 8/9/17 Documentreference 13 CMS Detector 150 Million of sensor Control and detection sensors Massive 3D camera Capturing 40+ million collisions per second
- 15. 15
- 16. Post-LHC accelerator projects (80-100 km)
- 17. 17 HOWKAFKACAN SUPPORTUS? CERN ACCELERATORLOGGING MONITORING COMPUTINGSECURITY PHYSICSJOBSMONITORINGAND ANALISIS PHYSICSDATA?
- 18. 18 0 200 400 600 800 2014 Data ingestion Per day GB requests per day. Direct SQL access is not permitted. • A generic Java GUI called TIMBER is also provided as a means to visualize and extract logged data. The tool is heavily used, with more than 800 active users. Figure 2: Logging ervice architecture overview. The Java APIs for both logging and extracting data are procedures have been he an understanding of h performing, in terms of how it is being done, and Optimal Use of Softw The database mod business logic (written i Java infrastructure intera engineered to use the Oracle, to maximize per the LS systems are be aforementioned instrume which features and tech performance. Data Quality Contro The MDB (introduce filtering capabilities wh for long-term storage. effort [4] to ensure configurations, the MD Filters for data Reduction ~ 250’000Signals ~ 50 data loading processes ~ 5.5 billion records per day ~ 275 GB / day à 100 TB / year throughput ~ 1 million signals ~ 300 data loading processes ~ 4 billion records per day ~ 160 GB / day à 52 TB / year stored Ø +800 extraction clients Ø +5 million extraction requests per day Ø 130 custom applications Credit: BE-CO-DS CERN ACCELERATORLOGGING
- 20. 20 Database Futures Workshop (May 2017) it-db-nile-admins@cern.ch Credit: IT-CM-MM KAFKABASED MONITORING
- 22. 22 HETEROGENEOUS ENVIRONMENTS KAFKA ON DEMAND SERVICE IaaSCLOUD
- 23. 23 On-demand service approach Ø Self-Service Cluster Management Ø Creation, Deletion, Extension Ø Parameters configuration Ø Self-Service Topics Management Ø Configuration, ACLs Ø Dedicated and shared clusters depending on the use case Ø Cluster and Topic Owners have administrative rights Ø But no access to the underlying hardware Ø Best effort support KAFKAAS ASERVICE
- 24. 24 KAFKAAS ASERVICE Development and Supported Functionalities Ø Transparent integration with rest of CERN services Ø Storage: different storage systems depending on the requirements Ø Cloud based (Openstack) but also bare metal as an option Ø Monitoring capabilities Ø Cluster management RestAPI and Web interface Ø Kafka Mirroring for high availability Ø Security: Ø Kerberos + ACLs Ø SSL Ø Support for service continuity in case of hardware failure