Kallio bosc2010 chipster-cloud
Download as PPT, PDF1 like369 views
This document discusses connecting Chipster, an interactive genome browser, to the cloud. It describes how Chipster's loosely coupled, message-oriented architecture makes it well-suited for the cloud. The document then outlines a proposal to preprocess genomic data using Hadoop in the cloud to generate summaries, which would be queried by the genome browser for faster, more scalable visualizations while reducing disk seeks. The goal is to leverage cheap cloud resources to improve the user experience of data analysis and visualization.
Kallio bosc2010 chipster-cloud
- 1. Connecting Chipster genome browser to the cloud Aleksi Kallio CSC – IT Center for Science, Finland
- 2. Architecture of Chipster platform Authentication Management service service Message broker File broker Clients Brokers Computing services Loosely coupled, independent components Message oriented communications Flexible, scalable, robust In other words, very cloud like
- 4. Chipster in the cloud 1) Deploying compute nodes in the cloud • Easy, because architecture already loosely coupled and based on message passing 2) Running large parallel jobs in the cloud • Architecture allows this easily • Cloud compatible tools can be integrated quickly 3) Using cloud as a back end for interactive visualisations • Not maybe so obvious • So let's dig into this further...
- 5. Background: Chipster Genome Browser Interactive Swing-based GUI Shows reads and analysis results in genomic context Interactive zooming from chromosome down to nucleotide level Ensembl annotations for genes and transcripts Integrated with the rest of the Chipster Parallel, distributed to some extent
- 9. Basic idea Preprocess data with Hadoop / MapReduce Generate powers of two summaries for the data, like in Google Earth • Doubles the data size Current genome browser samples data to produce summaries Now summaries can be read directly – Accurate results, significantly less disk seeks Distribute data to scale into massive datasets • Use messaging to query independent data providers Aggregate results as/if they appear to the visualiser
- 10. Work in progress... Genome browser up and running Hadoop based data processing at very early stages Currently trying to get it scale well
- 11. What's the point? Besides items (e.g., reads), visualiser can receive “superitems” (e.g., summaries of reads) • Summarises coverage, quality, SNP's etc. of the original reads All kinds of advanced information can be generated in the preprocessing step – Such as features that combine large number of genomes – Generators should be pluggable We spend resources on the server side to improve user experience on the client side • At server side CPU, memory and disk space required • But only for a short time (like in large batch jobs) • Cheap commodity servers can be used • And the experiment has already been expensive
- 12. Summary Use cheap server resources to enable better user experience Goal: to make data analysis quicker (and more fun) Tackle server side unreliability on the client side Future development – If this works out, it could be used in other Chipster visualisers also – Integrating Hbase queries to interactive visualisations – Optimising data summarising for visual truthfulness For more info: aleksi.kallio@csc.fi,