In Flux Limiting for a multi-tenant logging service

Editor's Notes

• #2: welcome to talk In-Flux Limiting for a Multi-Tenant Logging Service introduce yourself we are from CPE at symantec.
• #3: today we are here to talk about how we do event throttling, rate limiting for real time streaming, go over architecture internal details of streaming pipeline influx issue different approaches of solving prob show you results also want to cover an efficient pattern of computing if there are any pressing questions pls feel free to stop us. but we would prefer to take questions to the end
• #4: we are part of symantec internal cloud team that hosts app’s generating 1B revenue specifically our team builds, owns and runs 3 primary services. we call them Logging as a Service, Metering as a Service and Alerting as a Service Side note we are hiring if anyone is interested in joining the effort to build the biggest security data lake in the world, please stop by after the presentation. Side note we have open sourced project called hendrix , which is our alerting as a service . pls feel free to go and check it at out at github.com slash Symantec slash hendrix
• #5: before we jump on into the actual design and architecture of our system, lets talk about the data that we get and what is the problem that we are solving basically we offer logging and APM (app performance and monitoring) as a service. APM stands for application performance monitoring. our customers are Symantec product teams and they send us app and system logs generated on VM’s and containers. The teams use these for troubleshooting their applications. This is basically our own version of splunk. On The metrics side of the story we get app and system telemtetries which the teams use for application performance monitoring. Here is our sample events. we accept data in Json format and this is what it looks like on the left is the log event and the right is the metric event. If you look at log events you notice that there are 2 special fields one is called tenant id and the other is the api key. so whats the tenant and an api key ? Every customer which is a P&S teams at Symantec have something called tenants. the concept of tenant comes from our Openstack cloud. A given P&S teams can have more than one tenant. For example, their production App A can have a tenant and prod app B can have another tenant. Basically every tenant is a unit of isolation for us. An api key is a token used to allow and revoke flow of logs for a given tenant. lets say you wanted to stop a given tenant from sending data, you can revoke the api key. this means we start discarding the events. we call this process event authentication.
• #6: Now lets get into our architecture. basically customers run agents like flume, logstash, collectd and statsd which send data to our kafka cluster which is exposed over loadbalancers. we then run a set of storm topologies which write data to destination data stores. incase of logs its ES and incase of metrics its Influx db. We use Kibana as a front end for ES and Grafana as a front end for the influx db so that customers can graph and query the data. Redis is where we store tenant id’s and api keys.
• #7: Here’s what happens inside our streaming pipeline that is that storm topology. First like we showed earlier, events arrive in Kafka; we use the Storm Kafka Spout to read them and then we validate these events against the format and schema specifications that we publish to our customers, example if it’s malformed JSON we will drop the event. Next like we check whether the Tenant Id and API Key are valid. And lastly we Index the data to Elasticsearch or insert it into InfluxDB Each of the above stages are in separate Storm Bolts.
• #8: So now that you that you have a fair idea of our pipeline let’s understand the Influx issue. Influx means arrival of a large quantity of something in a short time, in this case that is events. When you are writing data to a data store like Hbase, Cassandra or Elasticsearch you provision a capacity in the cluster as in your cluster will have X number of nodes and they can support let’s say 10000 inserts per second. You can gauge this number / capacity running benchmarks. For us these inserts per second can also be referred to as Event per Second or EPS. EPS sent by our tenants is never linear, it fluctuates quite a lot as you can see in the graph on the right, each line in this graph represents the EPS from a tenant. At times we get spikes which is bound to happen in any real-time event processing system, when the load increases on the applications, they generate a lot more logs and we get a spike. So when a spike happens we don’t have provisioned capacity to handle indexing of the additional influx of data that came in because of the spike. So wouldn’t it be great if you could normalize these spikes? What we mean is have an almost flat EPS curve for every tenant.
• #9: Let’s understand how we can limit the influx and normalize these spikes. Think about event streams as a river and if there’s a cloud burst (no pun intended) the river will get temporarily flooded so much so that the banks will overflow right? So to fix this problem we can build a Dam and this Dam will buffer the additional influx of water we just got and you can control the rate at which this dam is being drained. Since we are using Kafka we already have a buffer however we have no control over the buffer as it will be controlled by the back pressure our elasticsearch cluster creates because it can take just sot many writes. So the purpose of this work was to have controlled back pressure into Kafka for our streaming pipelines to let us quantitatively determine how many events we would like to let flow through our pipeline into Elasticsearch. But we would like to do this on a tenant by tenant basis as you can see in the diagrams before you have different tenants sending different quantities of data shown in different colors. If we were to have a controlled system we can normalize and evenly divide the capacity among all of them or knowingly make it uneven that is if 1 customer has more need we allocate more capacity to them rather than others.
• #10: How can we do this? Well there are 2 approaches we thought of, one is you can write a substream where if a tenant exceeds their allocated throughput capacity we divert the extra event traffic that is over the capacity to a separate queue which we will then drain at a slower pace. In technical that means a separate kafka topic and a separate storm topology with a lower parallelism configuration. The other way of solving this problem is to pause the processing of events in the existing streaming pipeline for the tenant that is sending more data. Both approaches have some pros and cons. If you go with the first one you will see events out of order that means some data you will see right way which is flowing through our main pipeline and the some data will be delayed because it’s flowing though the slower pipeline. For the second approach you will always see events in order but if the queue back pressures too much you may loose data but that is true for either approaches if you share the kafka cluster because for a given cluster you disk space is limited.
• #13: What do we do inside the bolt and how do we track the Event rate for tenants? To track counts of events we keep a hashtable of tenant id and an integer counter which every time we see an event from a tenant we increment. But our customers wanted policies that define event rates differently for every tenant that is someone wants to be allowed to send 300 events in 2 minutes while the other one wanted 5000 in 10 minutes which don’t mean the same EPS so we had to come up with a way to track this for every tenant and we came up with an interesting way of solving this problem without using multiple threads. What we built logically was a sort of mary go round where every tenant is allowed to go round once. How it’s setup is every tenant influx limiting policy has 2 parts, 1. the number of events they would like to send 2. the time duration for those events so what we do is we take the time duration we we place tenants on this virtual mary go round based on their policy time duration.