Resource Aware Scheduling in Storm (Hadoop Summit 2016)

Editor's Notes

• #2: Good afternoon, My name is Boyang Jerry Peng and I am here to present Resource Aware Scheduling in Apache.
• #3: A little about me, apache storm committer and pmc member I am currently apart of the low latency team at Yahoo. Our team primarily works on projects that provide data processing solutions with low latency to yahoo and Apache storm is one of the platforms we work on. Prior to me joining Yahoo, I was a graduate student at the University of Iilinois, urbana champaign with a research emphasis in distributed systems.
• #4: First, going to provide a brief overview of Apache Storm Then, I will discuss the problems and challenges of running apache storm at yahoo. Next, I will get to the core of this presentation and talk about resource aware scheduling in Storm. Define what it is and how to use it and how it helps us overcome the problems and challenges I have mentioned Lastly, I will present some results.
• #5: Apache Storm is a popular open source distributed data stream processing platform used by many companies in industry There are many use cases for Apache Storm such as: Real-time analytics , Online machine learning , Continuous computation , Distributed RPC , and ETL operations
• #6: In apache storm, an application or workload is called a Storm topology. A storm topology, like applications in other stream processing systems, can be represented as a directed graph In which each edge represents a flow of data and each vertex a location where processing data occurs. In Storm, there are two types of operators or component. First type is called a spout. Spouts are sources of information and are responsible for injecting data into the storm topology Second type is called a bolt. Bolts consume streams of data, conduct any user defined processing, and potentially emit new streams of data downstream to be processed by other bolts
• #7: Briefly go over some definitions in Storm
• #8: Two types of nodes in a Storm cluster A master node that runs a daemon called Nimbus. The master node and the Nimbus daemon is responsible (with the help of Apache Zookeeper) for maintaining the active membership of the storm cluster. The nimbus Node is also responsible for computing schedulings of topologies in the Storm cluster. A worker node in Storm is a node that runs a daemon called supervisor that is responsible for retrieving schedulings from nimbus via zookeeper and launching the necessary processes according to the scheduling to realize the computation of the topology
• #9: Let me also talk about the difference between logical and physical connections in Storm. The diagram on the left is an example of a storm topology where executors are organized by component. And each line connecting two executors represents a logical connection. In The diagram on your right, executors are organized by the physical machines they are scheduled on and each line represents a physical connection. As you can see logical connections can vary quite a bit from the physical connections that need to be made in a topology This is where the scheduler can play an important part. How the topology is scheduled can have major impacts on performance of the topology.
• #10: Let me talk about how scheduling is done in storm Default scheduler schedules executors in a round robin fashion Uses the concept of worker slots to limit the computation load on a single machine. Can only Launch as many worker processes as worker slots. Each worker can run any number of executors that requires any amount of resources to run. Because not resource aware customers want isolated nodes Not very effective Not resource aware. Executors use any arbitrary amount of resources. See some loads overloaded and some nodes empty
• #11: Let me talk about some challenges of running storm at yahoo Our clusters have become increasingly heterogeneous. Made up of older nodes and new nodes that have different hardware specs Handing out dedicated nodes heterogeneous cluster, some times nodes on size some time another Not utilizing resources well. Customers used more nodes then they need. Because they don’t think about resource requirements as well. Nothing else can run on those isolated nodes
• #13: Fine grain resource control Deprecates the notion of using worker slots to limit load and removes the need to use isolated nodes. Resource isolation via cgroups
• #14: Let me go over the some of the core API for scheduling with resource aware scheduler Allows users to specify the resource requirements for each component…
• #15: Cluster admins can specify how much of each resource is available for user on each worker machine
• #16: Let me talk about some features Resource Aware Scheduler provides One of them is have pluggable per topology scheduling strategies. We have identified that different topologies might have different scheduling needs Constraint based scheduling strategy: An internal user has some scheduling requirements in which Users can can describe these constraints and the strategy will attempt to find a scheduling that satisfies these constraints
• #17: Only neat features we developed to support RAS is resource isolation via cgroups Get rid of delagating isolated nodes that was killing out utilization Rhel 7 cgroup and java memory do play well. Bugs in kernel
• #18: Taken into account in the scheduling priority and eviction strategies I will mention latter
• #19: Taken into account in scheduling priority and eviction strategies
• #20: pluggable In what order should the topologies be scheduled
• #21: Pluggable Different clusters should have different eviction policies (Production vs Staging) How much over his or her resource guarantee a user is Not enough resources or sudden failure
• #24: Still in the process of migration. The average amount of assigned memory has decreased. Which implies that topologies are becoming more resource efficient to run Using less memory to run Run more topologies
• #26: Working out the kinks. Cgroup and memory. Complete migration, beta quality