IT Days - Parse huge JSON files in a streaming way.pptx

Editor's Notes

• #2: Hello & Welcome A production issue fixed with the help of an open source library we built
• #3: Let’s start from the top Why do we even have to process huge JSON files? By process I mean actually manipulating these files not only storing them for import/export flows or analytics
• #4: We, in RWS, do translation’s a lot of them for a lot of clients In our system whatever the customer uploads turns into a JSON that we call BCM (Billingual Content Model)
• #5: How does a simple translation flow Upload files Convert to BCM’s (which are JSON’s that hold both the original file details and the resulting translated document details) Apply Machine Translation Maybe someone will then rewrite parts of the machined translated text Finally we convert this JSON back to the original file type with the translated text
• #6: To give you a sense of how big these files can get I will quickly show some books and tell you the BCM’s size for each of them Keep in mind this is before applying any translation which will of course make files even bigger
• #7: About 50 thousand words Gets to 3Mb
• #8: About 105 thousand words Gets you to 5 Mb
• #9: Couldn’t skip Uncle Bob’s book About 68 thousand words Gets you to 8 and a half Mb JSON
• #10: Let’s jump to one of the longest novels About 570 thousand words Gives us 33 Mb Json
• #11: The entire Lord of The Rings Trilogy Slightly less words than the previous book 53 Mb Json
• #12: Finally, Introduction to Algorithms About 450 thousand words Almost 150 Mb JSON
• #13: Finally just to introduce an Inception moment This presentation has about 2.5k words including notes And the JSON we will process to translate is a little over 1Mb
• #14: - Now that you have a sense of how big some of our BCM’s can be let’s zoom into the place where we first noticed problems
• #15: - First we receive a message that we have to apply translation to a file
• #16: - Then we download the file in memory
• #17: - Remove the already translated paragraphs
• #18: - Send the file to translation
• #19: The send to translation might give us some problems But when we were merging the translation back into the original file we had more problems This is triggered by receiving another message that the translation for a given file is done
• #20: First we download the original file again
• #21: Then we download the translation
• #22: When we have both files in memory we merge them together
• #23: Then we upload the BCM to be used by other services Where’s the problem?
• #24: Go back to the merging Its here where we have 2 JSON’s in memory to be able to merge them together Let me show you how much memory this might use
• #25: What happens if we try to translate Introduction to Algorithms The file without any translation is 150Mb Assume the translation doubles the file just to make math easier In memory those same files can be a lot bigger with one bigger JSON that I’ve tested it was 3 times bigger Assuming the same multiplier results in almost 1.3 Gb of memory being used This is on top of everything else running on the service
• #26: What’s the immediate fix you can do? Pay more for more memory Reduce your throughput This is not a long term solution
• #27: Here is where we started working on a different approach Working in the Java ecosystem we looked around for a solution that fits our needs Here is where I found about Jackson’s streaming capabilities
• #28: Say we have this JSON We receive a request from specifying an action to perform on a list of numbers from a user The JSON is too big to deserialize in memory How do we sum up all numbers with limited memory use using Jackson?
• #29: First we add the Jackson dependency to our project
• #30: Then we will be creating a JsonParser To do this we will need an InputSource from where the JSON will be read, let’s assume a file
• #31: - Then we will be using the parser.nextToken() the most to process this JSON
• #32: If we keep calling the .nextToken() method If you want to sum the numbers you will have a while block Find the numbers field After that iterate over all numbers and add them to a total Doing this will ensure that you have at most a few Kb of memory being used for the bytes that Jackson will buffer in memory for better performance
• #33: This is how a version of the code might look like We managed to sum the numbers The only issue is that even for our simple example this is the logic, imagine how your own logic mapped to tokens would look like Performance wise this is the best but our use cases are too complex to implement them using with tokens
• #34: We built a new library Backed by Jackson but without the token logic Let’s try again to sum all the numbers with our new library
• #35: As of last night you can import the library in your own project
• #36: We initialize a processor to read from our file We define the path where we can find the list of numbers After that using the processor and the path defined we can say that at the given path we expect to find Integer’s and to read them all Using the iterator we can quickly sum up all the numbers
• #37: Most of our operations on the BCM’s require us to rewrite it and apply different logic while we do that We made sure that the library we wrote permits us to do that as well Let’s look over another exercise and see how we can do it in a streamed way
• #38: won’t bother you with how you can do the rewrite and +1 using Jackson because it’s a lot I’m just showing you a snippet of how one might be able to do it in over 20 lines of codes
• #39: Using our library it’s a lot easier to do this We create a processor builder and initialize with the input source pointing to the numbers file and and outputSource where we write our JSON Define numbers path again Create what we call a transformer that we tell to look at the numbers array path, expect to find Integer’s and apply the following function Then we create a visitor using this transformer Build our Visiting processor And we call the visit method of the processor We will iterate over JSON and apply all the transformers specified while rewriting the entire JSON in the OutputStream
• #40: - If we want to complicate the problem even more and say we want to extract the username from the JSON
• #41: Remember we had a requester field that has an object including the username
• #42: Using our library we can quickly achieve this Define our path matchers Define an object where we will be storing out username value Define our JSON visitor specifying the 2 transformers Then we visit the JSON At the end we will have the username value stored and we can do whatever we want with it
• #43: How does our library compare with Jackson and processing files in memory?
• #44: Did some benchmark tests just to give you a glimpse of how Jackson our Library and processing files in memory compare As you can see from 10Mb files to 100Mb JSON’s, if we want to sum the numbers Jackson will be the fastest, followed closely by our library Starting from about 60Mb files processing in memory becomes way slower I don’t have a graph to show you how much memory is being used but is Kb vs Mb’s For my numbers JSON I’ve only used 3 digit numbers. 3 digits on disk it’s 3 bits, same 3 digits as an int in Java will use 32 bits. Each number is more than 10 times bigger in memory.
• #45: Comparing performance for rewriting JSON’s shows us a similar story Doing anything in memory is always slower than doing it in a streamed way
• #46: If you know your JSON’s are too big for you to hold in memory, when is it a good idea to use our library over Jackson?
• #47: - This is a good tell-tale sign that you should be using our library
• #48: I’ve only shown you deserialization to Integer’s and String’s but using our library you can deserialize to an entire object Split your logic into smaller processable units
• #49: Like I mentioned the slide before you can deserialize to any object and process into smaller units Deserialization has it’s performance penalty, you have to check how many deserialization’s you end up doing
• #50: Even for the easiest of logics building that logic using our library will be faster
• #51: For now we use the library in 4 distinct flows What are our own personal results from processing files using the library we built For our BCM’s our processable units are Paragraphs and we were able to do everything we needed by only deserializing one paragraph at a time
• #53: - Doesn’t matter the size of a BCM a paragraph can’t be so big that we can’t hold in memory
• #54: - Back to processing multiple files in parallel since we use less memory
• #55: - We simply don’t need as much memory added to these services because we don’t use as much now
• #56: On GitHub you can find the source code, all the other transformers you can use and tests that show you how they all work I’ve got a list of refactorings, new api’s, tests to add so if you like the library please do follow the progress there Would love to receive any feedback on it If you have a use case where you think this library might work for you try using the library or come contact me If you use a different programming language than Java and you need the same functionality also contact me because we can make something work