Job Talk Iowa State University Ag Bio Engineering

Editor's Notes

• #2: Hi, thanks for inviting me to talk to you today and taking the time to come learn a little bit of my research. I’ll admit that this is one of the longer talks I’ve ever given. I’ve had only one other 90 minute talk and it was to a group of Korean government officials who were interested in what a framework for big data analysis for a community might look like. But since they did not speak English, half of that talk was given by a translator. 
• #3: Today, I’m going to give you an overview of my research which is very much interdisciplinary, living on the edge of both computational biology and microbial ecology (the study of natural communities in the environment). I’m going to tell you a bit about my background and what shaped the research I’ve become involved in. Then I’m going to highlight a couple research projects – the first will be more of a computational focus where I tell you about research that tackled the “data deluge” that emerged from fast changing sequencing technologies. Then I’ll tell you a story of how we used these tools on data to investigate how our bodies (which our in themsleves a natural ecosystem) respond to dietary changes? Then finally, I’ll conclude with a discussion on where I view these efforts going in the future.
• #4: Folks often ask me how I went from a Mechanical Engineering degree to microbial ecology, as its not the most conventional track. And I actually think if you talk to most people in the field now, many of them have arrived here in unconventional paths. As a ME at Purdue, I had the opportunity to do two internships where I looked at the environmental impacts of industrial machinery. In particular, my junior year, I worked at Exxon Mobil evaluating sustainable replacments for outdated compressors on an oil platform which pump the unrefined oil from the platform to the refinery. This got me very interested in understanding the environmental impacts of economic decisions and how we should evaluate them, and this brought me into a program that was in its first year at Purdue in Environmental Engineering focusing on sustainability research. It was here where I first learned about microbiology and how it impacts our lives in so many different ways – visiting a wastewater treatment plant was a bit of a life changing moment for me. I really fell in love with the natural ability of these invisbile lifeforms in creating the world around us. So I went to grad school at the University of Iowa where I worked on how to monitor the activity of microbes which degrade pollutants in both groundwater and soil and one of the main challenges I had was that we were always workign with “model organisms” which didn’t necessarily match what was in the natural environment. After my PhD, the field of metagenomes was in its infancy but was being touted as a huge opportunity for studying natural environments. Jim Tiedje at MSU needed someone to start working with this sort of data, and thought I had no experience in it, I was willing to give it a try, and with the support of the NSF and Titus Brown at MSU, that’s what we did. I must’ve done a decent job of it because I was recruited by Argonne National Lab a couple years ago to provide support for some projects ongoing locally there.
• #5: Folks often ask me how I went from a Mechanical Engineering degree to microbial ecology, as its not the most conventional track. And I actually think if you talk to most people in the field now, many of them have arrived here in unconventional paths. But something that I think is shared is that you pursue research that is enabling and that can make a real difference. As a ME at Purdue, I had the opportunity to do two internships where I looked at the environmental impacts of industrial machinery. In particular, my junior year, I worked at Exxon Mobil evaluating sustainable replacments for outdated compressors on an oil platform which pump the unrefined oil from the platform to the refinery. This got me very interested in understanding the environmental impacts of economic decisions and how we should evaluate them, and this brought me into a program that was in its first year at Purdue in Environmental Engineering focusing on sustainability research. It was here where I first learned about microbiology and how it impacts our lives in so many different ways – visiting a wastewater treatment plant was a bit of a life changing moment for me. I really fell in love with the natural ability of these invisbile lifeforms in creating the world around us. So I went to grad school at the University of Iowa where I worked on how to monitor the activity of microbes which degrade pollutants in both groundwater and soil and one of the main challenges I had was that we were always workign with “model organisms” which didn’t necessarily match what was in the natural environment. After my PhD, the field of metagenomes was in its infancy but was being touted as a huge opportunity for studying natural environments. Jim Tiedje at MSU needed someone to start working with this sort of data, and thought I had no experience in it, I was willing to give it a try, and with the support of the NSF and Titus Brown at MSU, that’s what we did. I must’ve done a decent job of it because I was recruited by Argonne National Lab a couple years ago to provide support for some projects ongoing locally there.
• #6: Folks often ask me how I went from a Mechanical Engineering degree to microbial ecology, as its not the most conventional track. And I actually think if you talk to most people in the field now, many of them have arrived here in unconventional paths. But something that I think is shared is that you pursue research that is enabling and that can make a real difference. As a ME at Purdue, I had the opportunity to do two internships where I looked at the environmental impacts of industrial machinery. In particular, my junior year, I worked at Exxon Mobil evaluating sustainable replacments for outdated compressors on an oil platform which pump the unrefined oil from the platform to the refinery. This got me very interested in understanding the environmental impacts of economic decisions and how we should evaluate them, and this brought me into a program that was in its first year at Purdue in Environmental Engineering focusing on sustainability research. It was here where I first learned about microbiology and how it impacts our lives in so many different ways – visiting a wastewater treatment plant was a bit of a life changing moment for me. I really fell in love with the natural ability of these invisbile lifeforms in creating the world around us. So I went to grad school at the University of Iowa where I worked on how to monitor the activity of microbes which degrade pollutants in both groundwater and soil and one of the main challenges I had was that we were always workign with “model organisms” which didn’t necessarily match what was in the natural environment. After my PhD, the field of metagenomes was in its infancy but was being touted as a huge opportunity for studying natural environments. Jim Tiedje at MSU needed someone to start working with this sort of data, and thought I had no experience in it, I was willing to give it a try, and with the support of the NSF and Titus Brown at MSU, that’s what we did. I must’ve done a decent job of it because I was recruited by Argonne National Lab a couple years ago to provide support for some projects ongoing locally there.
• #7: Folks often ask me how I went from a Mechanical Engineering degree to microbial ecology, as its not the most conventional track. And I actually think if you talk to most people in the field now, many of them have arrived here in unconventional paths. But something that I think is shared is that you pursue research that is enabling and that can make a real difference. As a ME at Purdue, I had the opportunity to do two internships where I looked at the environmental impacts of industrial machinery. In particular, my junior year, I worked at Exxon Mobil evaluating sustainable replacments for outdated compressors on an oil platform which pump the unrefined oil from the platform to the refinery. This got me very interested in understanding the environmental impacts of economic decisions and how we should evaluate them, and this brought me into a program that was in its first year at Purdue in Environmental Engineering focusing on sustainability research. It was here where I first learned about microbiology and how it impacts our lives in so many different ways – visiting a wastewater treatment plant was a bit of a life changing moment for me. I really fell in love with the natural ability of these invisbile lifeforms in creating the world around us. So I went to grad school at the University of Iowa where I worked on how to monitor the activity of microbes which degrade pollutants in both groundwater and soil and one of the main challenges I had was that we were always workign with “model organisms” which didn’t necessarily match what was in the natural environment. After my PhD, the field of metagenomes was in its infancy but was being touted as a huge opportunity for studying natural environments. Jim Tiedje at MSU needed someone to start working with this sort of data, and thought I had no experience in it, I was willing to give it a try, and with the support of the NSF and Titus Brown at MSU, that’s what we did. I must’ve done a decent job of it because I was recruited by Argonne National Lab a couple years ago to provide support for some projects ongoing locally there.
• #8: There are several grand challenges that our society is currently facing which I think are of paramount importance. These are predicting and managing the impacts of climate change, finding sustainable sources of liquid fuels, and understanding the emerging pandemics facing human health in recent years. From carbon emissions from land use (which is magnitudes more than that of car emissions), degrading cellulosic biomass, to pathogens in our bodies, microbes are involved in complex communities that drive the health and productivity of either our natural resources or our own bodies. And its buidling up the expertise to ask
• #9: My research explores these complex communities. These microbial communities are all connected, the food we eat contains microbes which then we “introduce into the environment” (mainly through wastewater treatment”), and then these and other microbes then impact biogeochemical cycling which affects the global climate cycle and the flow of nutrients in natural systems. As I talk about my research today, then I want to be sure to emphasize that they are broadly applicable.
• #10: We’ve known about the importance of these environmental microbes for a long time, and much research has been spent answering three seemingly simple questions. One of the reasons this has been historically so challenging is something known as the great plate count anomoly. We know that there is a diverse world of microbes out there, but when we go into the laboratory and try to study their characteristics, we cannot grow them.
• #11: When first automated DNA sequencing machines came online in the late 80s, microbiologists had a new way to ask questions like who is there and what are they doing? If we could access the microbe that we were interestd in, we could extract and sequence its DNA. We could then compare this DNA to previously seen DNA, and we could then identify the “Who” and “What” assuming and add to the encyclopedia of genes we had information about.
• #12: Sequencing opened up the door to start building a catalog of some observed key microbial players. Iit was expensive but effective. Many of the choices of who got sequenced was driven here mostly by health and biotech. And this is the same set of reference genes that are still in use today. This graph here shows how expensive sequencng was in the early 90s, and how with early sequencing technologies this cost has changed over the years.
• #13: What changed the field was the invention of next generation technlogies, bascially allowing the throughput of these automated sequencers to be much higher and the cost of sequencing much cheaper. So cheap in fact that instead of sequencing only one bacteria you could start sequencing multiple, even bacteria from complex environments.
• #14: You may have seen this in the news and recently highlighted in NPR under the subject of personalized medicine and how it getting to the point where we can all have our genomes sequenced as a baseline for our future health.
• #15: This sequencing also opens up the door to start studyig not only single isolates, but all the organims in a natural system. So then the question is not only who is there and what they are doing? But what are they doing together and how?
• #16: With this growth and opportunity, however, has come other challenges. we are now dealing with though is that the growht of sequencing technology is growing more rapidly than the computers used to even the store the data on, let alone the types of analysis that we need to make this data informative.
• #17: 25x million times….And this is when I started my postdoc. To give some very concrete examples. Within the first year of my postdoc, the data I had to analyze grew from the largest known soil metagenome (a collection of environmental DNA sequences) at 50 million reads to about 40x that within literally 9 months. At that time, we were already overwhelmed with this much data. And to put this in the perspective of other datasets that were avilable at the time.
• #18: So where as I had spent a lot of time learning about how to grow bacteria and design an experiment during my PhD, now I was faced with an experiment that was designed to collect a lot of essential data but no way to start analyzing the data simply because available tools wouldn’t work.
• #19: One of the most effective ways to reduce genomic sequencing data is do something which is referred to as genomic or in our case metagenomic assembly.
• #20: Assembly is the process of rebuilding the original genome from the fragments of sequences we get from a sequencing machine. Essentially, its solving a puzzle where you look for overlaps of sequences among shredded information to predict a consensus sequence. If you do this process without any previous information (without a guided reference), you would call this process de novo assembly. Assembly has several advantages.
• #21: I want to emphasize that the difference between a single genome assembly (like that of a pure culture) vs metagenomic assembly (like that of DNA from a complex environment of soil) is a huge difference of scale. And with this difference, come multiple challenges.
• #22: Again, assembly is the process of trying to come up with a consensus sequence based on finding overlaps in small fragments. Here is an example of how “an assembly” of a sampling of the novel “A Tale of two cities”. You’ll notice here that because you have enough sampling of this sentence, you can get a good guess of what the original information would look like. You’ll also notice that there are some obstacles in getting the right solution, there are mistakes in the sampling which is analagous to sequencing errors that you would have to decide some criteria to estimate. In this example, we are coming up with a solution of one sentence using 8 fragments. In metagenomic assembly, you are trying to come up with hundreds to thousands to even millions of genomes using billions of fragments. And to do this, you have to compare each fragment to every other one in the dataset, making it very computationally intensive.
• #23: To give you an idea of what computational intensive means, even the smallest dataset that I had at the beginning of my postdoc required several months on a supercomputer, something having over 100 GB of RAM. These were resources I simply didn’t have at this time. And for my larger datasets, there was simply nothing I could do with them, they would essentially crash any available assembly program that existed. So I had to come up with a way to deal with all of this data or essentially, there were a handful of Pis that had just invested tens of thousands of dollars in a project where we couldn’t tractably handle the datasets.
• #24: To give you an idea of what computational intensive means, even the smallest dataset that I had at the beginning of my postdoc required several months on a supercomputer, something having over 100 GB of RAM. These were resources I simply didn’t have at this time. And for my larger datasets, there was simply nothing I could do with them, they would essentially crash any available assembly program that existed. So I had to come up with a way to deal with all of this data or essentially, there were a handful of Pis that had just invested tens of thousands of dollars in a project where we couldn’t tractably handle the datasets. I’m going to tell you now about how we wee able to do this and there actually two different strategies we had to combine.
• #25: So one of the first things we thought about is what makes natural communities different than single organisms and there are two main factors. One is that natural communities are diverse. There are multiple genomes, and even potentially millions of species, in a sample. And this is represented here by the presence of red, blue, and green organisms. Another main difference is that these organisms are present at a variable abundance in nature, some are highly abundant some are not.
• #26: Firstly, let me acknowledge that assembly for single organisms (especially bacterial) is relatively mature. So one of the first things we thought about is what makes natural communities different than single organisms and there are two main factors. One is that natural communities are diverse. There are multiple genomes, and even potentially millions of species, in a sample. And this is represented here by the presence of red, blue, and green organisms. Another main difference is that these organisms are present at a variable abundance in nature, some are highly abundant some are not.
• #27: Firstly, let me acknowledge that assembly for single organisms (especially bacterial) is relatively mature. So one of the first things we thought about is what makes natural communities different than single organisms and there are two main factors. One is that natural communities are diverse. There are multiple genomes, and even potentially millions of species, in a sample. And this is represented here by the presence of red, blue, and green organisms. Another main difference is that these organisms are present at a variable abundance in nature, some are highly abundant some are not.
• #28: Firstly, let me acknowledge that assembly for single organisms (especially bacterial) is relatively mature. So one of the first things we thought about is what makes natural communities different than single organisms and there are two main factors. One is that natural communities are diverse. There are multiple genomes, and even potentially millions of species, in a sample. And this is represented here by the presence of red, blue, and green organisms. Another main difference is that these organisms are present at a variable abundance in nature, some are highly abundant some are not. A strategy we came up with was can we come up with a way to come up with the minimal dataset that you need for assembly, discarding these reads from this overkill section?
• #29: From a sequencing standpoint then, what we see is that for a given genome (represented here as a dotted line), we start sampling fragments from it.
• #30: As we sample more, we will have some sequences which will have errors in it.
• #31: And we’ll keep sequencing this genome, randomly sampling different parts of it. We’ll get to a point, where we’ll have enough sequences where we can make a good guess at what the original sequence may have looked like.
• #32: For example, here we have a total of 6 sequences for which this particular part highlighted by the black arrow where we can be confident in saying we know what that is. From experience, I know this number here should be about 6 sequences to get an accurate assembly. So anything beyond this 6 is excessive or redundant information.
• #33: So we can discard or set aside this read and not use it for our assembly. And that actually turns out to be a good thing because in discarding this information, we’re actually removing data with errors in it.
• #34: In the end, we end up with a minimal dataset needed for an assembly of the dataset here in pink and a redundant set of information which we have set aside. In setting aside these reads here in the red, we actually get to discard sequencing errors which actually ends up in improved results for assembly. So eessentially, what we’ve shown is that the assembly of all this data and just the pink data ends up with at least the same assembly if not improved ones. In assembling just the pink dataset though, we’re able to reduce the amount of data we’re working with up to 95% in some environmental datasets I’m working with.
• #35: Another tool we’ve developed to deal with biolgoical big data is a lightweight data structure that can break apart these datasets by connectivity.
• #36: The system that these methods were developed for was sequencing data that was investigating soil biodiversity in both managed and natural soil systems. Soil biodiversity is amazing. Great Prairie – world’s most fertile. Important reference site for the biological baseis and ecosystems of soil microbial communities. It sequesters most carbon, produces large amount of biomass anually, key for biofuels and security. We know surprisingly little about the identities and functions of the microbes inhabiting soil,” With applications of DNA sequencing, the field was really excited about how we could now gauge this specific ecological niche and its responsiveness to change. Once we came up with how do deal with the data, and sift through the gleaned information, it was a sobering reality check on just how hard a challenge these environments will be.
• #37: Overall, many funcitons are shared between corn and prairies soils. Interestingly, prairie soils have much many more unique functions (indicated here as blue bars) compared to unique functions in the corn (here green). This result may reflect the varying management history of these two soils. Unlike the prairie soils, which have never been tilled, the corn soils have been cultivated for more than 100 y and have had annual additions of animal manure that potentially could enrich specific metabolic pathways with decreased diversity.
• #38: I’m fortunate to have worked on many projects in which I feel that this is true.
• #39: Ok, let’s talk about the gut microbiome and how it responds to different diets. This study is a collaboration with the UC.
• #40: In In recent years, there has been a growing appreciation for the fact that, as humans, we are in fact supraorganisms composed of both human and microbial cells, and as such we carry two sets of genes, those encoded in our own genome and those encoded in our microbiota. We genetically inherit only ~1% of our genes from our parents, and the remaining ~99% is mainly acquired from the immedi- ate environment when we are born. Importantly, all the genes in our body, whether human or microbiome encoded, have the potential to have an impact on our health. The gut is the most densely colonized microbial community in the human body and is also one of the most diverse. The gut functions as a chemostat, a continuous culture system for microorganisms (mostly bacteria) in which fresh nutrients enter the system and cultured microorganisms leave at a relatively constant rate. Approximately 1.5 kg of bacteria are resident in our gut, and 50% of our faecal matter biomass is bacterial cells.
• #41: These gut microbiota interacts with our genetics and our environment (mainly diet) to influence our health. The gut microbiota releases toxins, such as lipopolysaccharides, and beneficial metabolites, such as vitamins and short-chain fatty acids, to damage or nourish humans. We know that diet has a greater potential to shape the structure and function of the gut microbiota than host genetics, thus influencing our health state directly.
• #42: There are two key efforts that have looked into the response of gut communities to diet changes – one was Zhang et al which worked in mice.
• #43: After reading these studies, a key question I had was has anyone looked into the viral components of the human gut. Much of the gut microbiome literature focuses only on the bacterial component of the gut, but we know that viruses are abundant in the gut environment, present at a ratio of 1:1. So beyond bacterial cells, there are…viruses as prophage… As far as I know, there are only 3 studies that have looked at the gut virome. From these preliminary studies, …
• #44: The majority of these viruses are phages, or viruses that only infect bacteria. These viruses are pivotal driver of gut health and disease as these phages are able to redirect the structure and function of the entire gut microbial community.
• #45: Specifically, phages can alter the fitness and function of bacterial populations through their transfer of genetic material,
• #46: skew the abundance of bacterial population by infection, and drive the evolution of the community with their diversity and modifications of bacterial hosts. The broad range of subtle and robust effects that phages can exert on the gut microbial community makes them key targets in understanding health and the pathogenesis of gut diseases. Overlooked
• #47: We know that we’ve previously seen in bacterial communities…
• #48: Acess throught faecal matter.
• #49: We targeted each of these communities in mice that had been fed a baseline diet, then switched diets, and then returned to their original diet. This would allow us to see what was being altered by diet and how much and if functions returned to normal if returned to the baseline diet (or washout). We did this on two diets to see if they had distinct reponses. Additionally, we took 16S rRNA DNA samples from fecal samples every week of the experiment so we could get a better resolution of the changes on community structure. The diets that we are studying here are basically diffefrent in their fat content. There is a low fat and milk fat diet where there is about a four fold difference in diet. But another thing to note is that these diets also differ in the amount of corn starch and sugar…and the LF diet has more corn starch – so complex carbs and the MF diet has more simple sugars, like sucrose. Finally also note that I designed this experiment to really access the virome part of the microbiome, much more so than has previously been looked at.
• #51: A key result I would like to discuss is How different communities (both bacterial and viral) change over the course of this experiment. To talk about this result, I’m going to show you the change of abundances over of over 200,000 genes in a way that you can visually interpret. These analyses products are actually another whole talk about challenges of presenting biologial big data. Basically, to see how communities change over time, I’m going to estimate how different the are as a “distance from their original baseline communities). You can imagine three possible ideal sitautions.
• #57: The data allows me to look at not only “if communities change” but “how communities change” One of the strongest signals in the viruses in the MF baseline over time is the decrease of phage related functions which is accompanied by a decreaes in the richness and diveristy of the viral communities. (BLUE) The strongest signal is that phage functions significantly in free living virus communities, and that there is loss of both the abundance of the free living viral community membership and diversity during MF diet. This corresponds to siginficant decreases in functions encoded within genes of this commuity. Reduced availability for the total community, through viral infection.
• #58: When we looked at significant changes of sequences associated with different organisms in the MF diet, the phyla Bacetrioidets and Firmicutes showed significant decreases, especially in viruses related to these hosts. This is consistent with previous reports.
• #59: We know that diet can be a cause of obesity and that we all respond to diferent diets. Yet, we find that in general, bacteria in our guts our resliient to change. Then, my thought is that there must be something else that is directing our bodies response to diet change…and I would suggest that it could be viral populations…
• #60: These viruses are likely to be a pivotal driver of gut health and disease as these phages are able to redirect the structure and function of the entire gut microbial community.
• #63: Potential consequences of a temperate phage life cycle in the human gut. Metagenomic studies of viruses Nature Reviews | Microbiology suggest that a temperate lifestyle is dominant in the distal human gut, in contrast to the the lytic lifestyle observed in open oceans. This temperate lifestyle can have benefits for the phage and the bacterial host, and can alter phage–host dynamics. Integration as a prophage (part a) protects the host from superinfection, effectively ‘immunizing’ the bacterial host against infection from the same or a closely related phage. Furthermore, the genes encoded by the phage genome may expand the niche of the bacterial host by enabling metabolism of new nutrient sources (for example, carbohydrates), providing antibiotic resistance, conveying virulence factors or altering host gene expression. This temperate (lysogenic) life cycle allows phage expansion in a 1:1 ratio with the bacterial host. If the prophage conveys increased fitness to its bacterial host, there will be an increased prevalence of the host and phage in the microbiota. Induction of a lytic cycle (part b) can follow a lysogenic state and can be triggered by environmental stress. As a consequence, bacterial turnover is accelerated and energy utilization is optimized through a ‘phage shunt’, in which the debris remaining after lysis is used as a nutrient source by the surviving bacterial population. Furthermore, a bacterial subpopulation that undergoes lytic induction sweeps away other sensitive species and increases the niche for survivors (that is, bacteria that already have the specific phage integrated into their genome). Periodic induction of prophages can also lead to a constant-diversity dynamic139, which helps maintain community structure and functional efficiency. Novel infections or infections of novel bacterial hosts by phages (part c) bring the benefit of horizontally transferred genes and create selective pressure on the hosts for diversification of their phage receptors, which are often involved in carbohydrate utilization. HGT, horizontal gene transfer.
• #64: For the final part of my talk today, I wanted to present some of my ideas for future work, especially work that I feel would be successful here at Iowa State with your expertise and resources.
• #65: I hope I’ve presented to you how I’ve been able to leverage next generation sequencing data and big data in biology to start enabling dat driven discovery in this field. Iowa State has done what I think is a really smart thing in identifying it as a great opportunity for research in the future. This parallels an heavy investment at both government and private funding agencies. Philip E. Bourne, Ph.D., as the first permanent Associate Director for Data Science (ADDS). Dr. Bourne is expected to join the NIH in early 2014. “Phil will lead an NIH-wide priority initiative to take better advantage of the exponential growth of biomedical research datasets, which is an area of critical importance to biomedical research. The era of ‘Big Data’ has arrived, and it is vital that the NIH play a major role in coordinating access to and analysis of many different data types that make up this revolution in biological information,” said Collins.
• #66: As my research is broadly applicable in multiple enviornments within a continuum. I’m also very multidisciplinary and my research occurs on a continuum connecting High Thoughput Data discovery frameworks, big data sacling approaches, and trying to identify the best model systems or biomarkers to investigate complex natural communities.
• #67: In the next few slides, I’m going to go through some ideas I’ve had for grants that I will be writing in the next 5 years. But before I do that, I wanted to emphasize what I think are my core research values.
• #68: Buliding on viruses and their impacts on human health, a natural extension of the research I presented on the gut microbiome and diet change is to extend to the human system. The NIH is heaveily invested into questions like this.
• #69: The grant here is actually unique among the next slides in that it is already funded for the next few years. This is a grant with Kirsten Hofmockel here at ISU, where we’ve been funded to look at microbial drivers of carbon metabolism and warming. Our key goal here is to understand…
• #70: Unknowns in soil….A large opportunity for data driven discovery…
• #71: Finally,another area of interest I have is the origin and fat of our microbiomes.
• #74: So we’ve talked about digital normalization as an effective way of compressing the data or reducing it for assembly. Another strategy we used to deal with very large and complex metagenomes was to come up with a way to break it into pieces that could be analyzed separately. If you’re workign on a jigsaw puzzle, this would be akin to thinking about separating out different colored pieces to work on separatetly. To do this, we first needed a more efficient way to both store and query our data. And next I’ll tell you about how we did this.
• #75: The main challenge for dealing with our datasets is that we could not figure out if a Sequence A was connected to sequence B within practical resources – both time and computation. The first trick we could use is that we could break each sequencing read into words of length k. This way rather than comparing long sequences and aligning them and checking for overlaps, we could just look for the presence of the same “word” in two sequences to say that they should be connected together. This trick is a common one used in assembly but isn’t enough for our dataset volume. We simply had too many words to store efficiently.
• #76: So we came up with the application of a data structure called a bloom filter. And its this data structure that allowed us to start storing our huge datasets. A good way to think about bloom filters is to think about what I call the mailbox analogy.
• #77: Say you have to store the mail of everyone in this room in 3 slots. A solution would be to divide these three slots let people know if they had mail. For example, if there is a marker in a slot that contained your last name, you would go check for mail.
• #78: So if an asterix here represents that there is mail in that slot, and I wanted to know if I had mail, there are a couple of outcomes. One is that a slot that doesn’t represent my last name has mail. In that case, I know for sure that I do not have mail. Another possibility is that my representative slot would have mail. I could then possibly have mail or not have mail.
• #79: So you can see how we can use these smaller mailboxes to query whether or not we have mail for a large number of people hopefully.
• #80: If I wanted to make this even more efficeint, I could add a couple more mailbox setups with varying divisions of last names. So if I came into this mailroom, and looked for mail, I could see that mailbox 1 and 2 say I could have mail but since mailbox 3 is empty, I know I don’t have mail. And its likely someone else who has mail in the two other mailboxes.
• #81: False negatives = structure sequences that should not be connected will never be identified as being connected False positives = sequences may be identified as connected when its not And you can see how this sort of storage system allows us to have no false negatives. So we apply this same strategy to check for the presence or absense of a sequence, or more specifically a word of length k. And this allows us then to ask is Sequence A connected to Sequence B?