Molecular QC: Using Reference Standards in NGS Pipelines

Editor's Notes

• #7: Since it’s inception, next-generation sequencing has found utility in a diverse set of industries, from biomarker discovery in pharma to ancestral identification in archeology. Across the board, NGS has the advantage of allowing us to answer questions that require a lot of data. Next-generation sequencing provides orders of magnitude more data than traditional Sanger sequencing shown on the left as hundreds of “lanes” analyzed in parallel vs. hundreds of millions of “clusters” of molecules as shown on the right, which allows for many samples to be multiplexed on a single-run.
• #8: By starting with different genetic material and following specific experimental workflows, NGS can be applied to many applications.
• #9: For today’s webinar, we will focus on DNA resequencing applications, which implies the data generated will be compared to an existing reference sequence (such as the human genome). Specifically, we’ll focus on how we can analyze patient-derived material to identify onco-relevant mutations including single-nucleotide variants, insertions-deletions, copy number variants and translocations. We’ll also focus on how known reference standards have been shown to be vital in ensuring data generated from NGS assays is accurate and reproducible.
• #10: Patient-derived samples, whether they are used for diagnostic purposes, clinical-trial stratification, or translational research, have their own unique challenges. These materials are often heterogenous, consisting of multiple cell populations and even multiple sub-clones of cancer. Tissue and fluids are often at preciously low quantities, which results in very low yields of DNA. And, depending on the type of material and how it was preserved and stored, the nucleic acids may be highly degraded, further complicating analysis. Finally, the clinical need for earlier detection of diseases and non-invasive monitoring drives improvement to the technical limitations of detecting very low allelic frequency mutations.
• #11: As with other scientific assays, from simple pH testing to complex laser spectroscopy, the resonating solution for many of these challenges is making use of well-characterized reference standards to better understand, optimize, and control variability in each step within a protocol and ultimately, provide a predictive result for routine validation. In the case of high-throughput DNA sequencing, requirements for this reference standard include well-defined, orthogonally-validated single-nucleotide polymorphisms (SNPs) and small insertions/deletions (INDELs) multiplexed in a single sample, within the range of allelic frequencies you wish to detect with your assay. Furthermore, providing this material in a format which also allows for the evaluation of the pre-analytical portion of the NGS workflow is highly desirable. In this webinar, we’ll walk through each of the stages of the workflow, and address the challenges associated within each.
• #12: Specifically, laboratories that wish to analyze patient-derived materials must optimize and validate their NGS workflow following recommendations provided by the College of American Pathologists (CAP) and the American College of Medical Genetics and Genomics (ACMG). As shown in the figure, some of these requirements include defining performance specifications for any instrumentation or reagents used throughout the assay, validating the performance of a specific protocol, implementing appropriate quality control and quality assurance protocols, and performing proficiency testing of personnel involved in these assays. In order to accomplish many of these requirements, a set of well-characterized reference materials are required. Ideally, this reference must be orthogonally-validated to accommodate platform-specific errors, as well as renewable, so that deviations to the protocol are not introduced after it is established. Even outside the realm of clinical testing, these requirements also constitute “good practices” which will help make your NGS assay more robust and reliable.  
• #13: Now we’ll briefly walk through each stage in a typical NGS workflow, starting from extraction of DNA from your sample.
• #14: There is significant variability in the quantity of DNA recovered, depending on the extraction kit used, as shown in this comparative analysis. Even for diagnostic-grade kits, certain parameters may be optimized before establishing an approved protocol. Incubation periods, temperatures and elution volumes all can play a role in whether or not DNA of sufficient quantity, quality, and concentration is extracted. During protocol optimization, a well-characterized standard of known quantity and quality presented in the same format as the patient samples, such as FFPE in this example data, is ideal to evaluate how protocol modifications affect resulting DNA yield.  Importantly, making use of renewable reference standards ensures that this optimization may be performed not only at the beginning of assay design, but even after selecting a kit and protocol. Routine monitoring ensures that we do not introduce inconsistencies due to lot-to-lot variability of reagent kits.
• #15: Following extraction, genomic material must be qualified and quantified. Accurate DNA quantification is imperative to ensure the appropriate amount of material is input for downstream sequencing library preparation, as well as to understand theoretical allele frequency thresholds, as we’ll come back to later.
• #16: Importantly, the method of quantification will greatly affect your results. As shown in the figure, across measurements collected by multiple laboratories, there is notable variability between DNA quantified spectrophotometrically by Nanodrop vs. fluorometrically by Qubit. While spectroscopic methods provide adequate estimation at concentrations higher than 10 ng/µl, fluorometric quantification will arguably be more specific, measuring only DNA via intercalating dyes, and more accurate for quantification. That being said, spectroscopic analysis has other utility in evaluating DNA quality. Spectroscopic analysis to determine the 260/280 and 260/230 ratios can provide insight into potential contaminates in the sample, which may affect downstream library preparation. And finally, the level of fragmentation, often defined by the DNA integrity number or DIN may also affect downstream sample processing. Horizon offers reference standards in multiple formats, designed to mimic samples fragmented by formalin treatment or isolated from fluids, such as cell-free DNA. Having standards which reflect the actual state of your samples for analysis can help you select the appropriate equipment, and set standardized quality and quantity thresholds to ensure your protocol yields actionable data and minimizes assay failure with irreplaceable patient samples.
• #17: Once you’ve determined some general thresholds and expectations for the DNA quality and quantity your samples will yield, this will influence the library kit or protocol you choose to implement.
• #18: And while our knowledge of the human genome has been significantly enhanced since the completion of the Human Genome Project, there are still many regions that are either not well understood, or are repetitive. As such, it is often preferred to interrogate a reduced representation of the genome, focusing specifically on the exonic regions or smaller targeted regions of interest with short, 100-300 bp sequencing reads. Given the high-throughput nature of NGS, multiple samples may be multiplexed on a single sequencing run, making each assay more affordable and efficient. There are two primary ways of approaching targeted sequencing, solution-based enrichment using probes to capture specific regions, and targeted PCR to generate short amplicons.   For both capture and amplicon-based libraries, sample input must be tuned to meet the needs of the protocol. Capture-based experiments generally require higher input; although both Agilent SureSelect™ and Illumina Nextera™ kits are pushing this down to 200 ng and 50 ng, respectively, moving these into the same range as amplicon library protocols. And, as we’ll discuss later, this is important because input amount and unique molecules generated during library preparation affect the limits of allelic frequency detection. (CLICK) Reference standards can play an integral role during protocol optimization of sequencing library preparation. It’s ideal to have a standard that mimics the concentration and genomic diversity of your actual clinical sample. Ensuring there are no competing regions for probe or primer binding, and that the input quantity, quality, concentration yields the optimal library output can be achieved with the appropriate reference standard. Horizon’s reference standards are all derived from genomic DNA from human cell lines, making them more realistic than plasmid or synthetic standards.
• #19: Sequencing output, read length, run time, and quality of sequencing data continue to improve across all commercially available platforms. The two most common benchtop sequencers include the MiSeq™ from Illumina and the Ion Torrent PGM™ from Life Technologies. While each have certain advantages and disadvantages, both have found utility for the analysis of patient-derived samples.
• #20: One of the most important areas of optimization that directly impacts the detection limit of the assay is sequencing depth or coverage. If sufficient coverage of the positions of interest is not achieved during sequencing, it may be difficult or impossible to interpret the results. However, each individual laboratory may find their instrumentation performs differently, and results in a different limit of detection. As you can see from these results from 3 different laboratories, sufficient average coverage does not guarantee detection. This demonstrates the need for optimization in your own laboratory, with the technicians and instrumentation used for your own assays. Furthermore, on a more routine basis, having a positive reference control to run alongside patient samples is useful. This allows for confirmation that unexpected results or variability are biological in nature, and not inherent to the protocol. For example, if a patient sample performs poorly or does not yield the expected results, use of a positive reference standard will confirm whether the protocol was performing as expected and the result is real, or if there are protocol errors which must be addressed.   While Sanger sequencing has often been considered the “gold standard” for validation, it is becoming more commonplace to validate detected mutations and their allelic frequencies using a second NGS platform and/or entire workflow. This is especially true when allelic frequencies are outsides of the limit of detection of Sanger sequencing. Additionally, this parallel confirmation also helps account for inherent biases associated with each platform (i.e. homopolymer errors in Ion Torrent and 454 sequencing).
• #21: To wrap up the NGS workflow, I’ll briefly mention the role bioinformatics analysis can play in introducing variability. We’ll discuss this in more detail during our June webinar, where we dig further into the role reference standards can play in optimizing your informatics workflow. For both of the benchtop sequencers, the researcher may use the on machine analysis software. However, depending on the goals or flexibility of the assay, a researcher may choose to use an external, off-the-shelf software package or custom-developed pipeline. These may vary in parameters such as alignment stringency, which is how closely the sequence must match the reference, or the threshold of coverage required to make a variant call, such as requiring that the position must be sequenced at coverage of 4x or greater. For parameters such as these, it's important to evaluate how changes to the pipeline affect the final data output.
• #22: An example of the immense influence the informatics software/pipeline plays on resulting data is demonstrated with the results compiled by the Genome in a Bottle Consortium.3 By analyzing the same data using 3 different informatics pipelines, the authors observed significant variability in the resulting variant calls, with only 60% of variant calls being shared by all pipelines. As discussed for the molecular side of the assay, having a well-characterized reference standard with known allelic frequencies allows a researcher or clinician to ensure that the informatics are not introducing unwanted bias and changes made to the pipeline over time such as when the software is upgraded, do not affect downstream results.
• #23: And finally, annotation of the variants generated using databases such as COSMIC, dbSNP and SnpEFF can help impart additional information about the clinical relevance of the mutations identified by the assay. After generating a list of annotated variants, it is valuable to combine replicate data, positive and negative control data, and any relevant patient metadata for interpretation. Having experimental replicates from a single patient (especially from different tissue/fluid types) can confirm observed variants, and impart additional statistical confidence. We’ll pause briefly for a poll…and then come back to dive deeper into the theoretical limits of detection of NGS…
• #24: Welcome back! Now we’ll dive into how we can determine the theoretical and actual limit of detection of our NGS assay. As we improve library prep protocols, allowing us to generate sequencing libraries from lower and lower input amounts, we must also consider the impact this has on the theoretical limits of detection of minor allele frequencies. If we only have a small amount of material, the sample may not be a good representation of the patient. The theoretical limit of detection is dependent on the following: How many unique molecules are present in the sample, the amount of material used as sample input, and sequencing coverage. Deduplication/Molecular Uniqueness   To first discuss depduplication or unique molecules in the sample, we’ll consider the case of a traditional amplicon panel. Here, short amplicons are generated with the same forward and reverse primer sequences, targeting a specific region of interest. These are the grey “reads” represented in figure. The identical ends of these sequences makes it difficult to disambiguate which molecules are unique, and which have been “duplicated” during PCR amplification. In an ideal world, only unique molecules would be used to calculate allelic frequencies of mutations in order to avoid amplified PCR errors and bias. Capture-based libraries represented by the blue “reads” overcome this issue because DNA is first randomly sheared, thereby generating random start/stop sites that can be disambiguated, or “deduplicated”. However, the input amount required for capture-based libraries is generally higher, so this may not be ideal for all sample types.
• #25:  Input   The amount of input material in an NGS experiment greatly influences the lowest possible allelic frequencies that may be detected. Essentially, it is not possible to cover the position of interest more times than you have copies of DNA in your biological sample. It is common for an informatics pipeline to remove PCR duplicates and require a minimum of 4 reads to show the mutation of interest in order to make a confident variant call. So, if the number of copies of DNA present in a sample is estimated using 3.3 pg for a single, haploid copy, then for DNA samples of 10 ng, the theoretical limit of allelic frequency detection is 0.12% as shown in the equation. As the input amount increases, the theoretical detection limit decreases. It’s important to note that this is a theoretical calculation, and when experimental noise is factored in, the actual limit of detection may be much higher.  
• #26: Coverage   Finally, the sequencing coverage also plays a role in the limit of detection of the assay. If we are not limited by input material, then the denominator of the above equation is not limited by molecules, but rather becomes reads exclusively. Again, assuming we require that 4 reads contain the mutation of interest, then in order to achieve detection of allelic frequencies of 0.01% or lower, we must sequence this positon at least 40,000x. Importantly, coverage distribution is not even across a sample, therefore, it may be necessary to sequence significantly deeper to achieve coverages of 40,000x at each position of interest. Using reference standards to evaluate the actual limit of detection of your specific assay can be very useful.
• #27: By making use of our single engineered mutant standards, and the corresponding wild type, you can generate a set of samples where the concentration of mutant is serially diluted to the lower limit of detection you wish to achieve. An example dilution series ranging from 20% to 1% with the negative control of 0% mutant, or 100% wild type is shown.
• #28: After taking these samples through your NGS workflow, you will be able to determine if your assay is able to detect down to the lower allelic frequency. In this example, the 1% dilution was not detected by the NGS pipeline, and therefore, we know we cannot confidently detect any mutants below the 5% allelic frequency threshold. Alternatively, making use of a multiplex reference standard such as the Tru-Q Multiplex Series allows you to evaluate multiple genes and mutants in a single sample. These standards cover common mutants covered by oncology panels including EGFR, KRAS, NRAS, BRAF, etc. at defined frequencies of 5%, 2.5% and 1.25%.
• #29: Finally, reference standards are perhaps even more useful for addressing the up-and-coming challenges facing those adopting NGS for clinically-relevant analyses. An example of this is the analysis of cell-free DNA or cfDNA, for onco-relevant mutations. In these patient samples, copies of the disease-relevant mutant DNA may be very low relative to the healthy/wild-type DNA. Other examples include the early detection of organ rejection following transplantation, prenatal non-invasive analysis, and disease monitoring during treatment. Fragmented materials, such as cell-free DNA or formalin-compromised DNA may perform differently in an NGS assay due to variability in primer binding, amplification, or exclusion during size selection as a result of the high fragmentation. Horizon offers genetically-equivalent multiplex reference standards in varying formats, from high molecular weight or high DNA Integrity Number to formalin-compromised and fragmented DNA with low DINs, to allow researchers to evaluate how fragmentation affects detection.
• #30: This analysis, performed by Dr. Matthew Smith at the Molecular Pathology Diagnostic Service of University Hospital Birmingham, compares some of the different available formats of the Quantitative Multiplex reference standards side-by-side. This NGS assay made use of the Qiagen Actionable Tumor Panel, the MiSeq platform, and CLC Cancer Workbench for analysis. The table highlights the mutations and frequencies detected for HD701 which is the high molecular-weight genomic DNA extracted directly from unfixed cells, HD751: the formalin-compromised genomic DNA, and HD200: the mildly fixed and paraffin-embedded cells ready for extraction. You’ll see that the detected frequencies across the reference types are measured accurately for nearly all of the mutations. What’s even more impressive is that the frequencies for the mutations are also similar across formats, even for the lowest allelic frequencies. And, there is not significant variability between the DNA extracted at Horizon and shipped as gDNA (red bar) and the DNA extracted from FFPE by the Smith laboratory (purple bar). Dr. Smith also mentioned he has observed impressive consistency between replicates, which provides not only a solid foundation for validating the NGS panels but also as a QC benchmark for each of the NGS runs. Big thanks to Dr. Smith for allowing us to share this data. Having a single reference standard in multiple formats is imperative for evaluating how sample fragmentation will affect the results of the assay, and as discussed previously, to pinpoint where protocol problems occur.
• #31: So, in summary, as we continue to push NGS to its theoretical limits, it’s imperative that the performance of protocols are evaluated with known standards before implementation into clinical workflows. While as scientists we continually work to optimize assay parameters for the sake of efficiency and accuracy, it is perhaps most necessary when the outcome of an assay can affect the downstream stratification, diagnosis, or treatment of patients. Reference standards have the potential to play a key role in engineering checkpoints for validation into any NGS workflow, from pre-analytical through informatics analysis. Routine validation ensures consistency between reagent lots and personnel. And, when a reference standard is used across multiple laboratories and workflows, this data may be shared and used to improve platforms, kits, protocols and assay development overall.