BLAST_CSS2.ppt

BLAST
Getting the most from your
cycles.
Tom Madden
NCBI/NLM/NIH
madden@ncbi.nlm.nih.gov

WHAT IS BLAST?
 Basic Local Alignment Search Tool
 Calculates similarity for biological
sequences.
 Produces local alignments: only a portion of
each sequence must be aligned.
 Uses statistical theory to determine if a
match might have occurred by chance.

The BLAST family of programs allows all combinations of
DNA or protein query sequences with searches against DNA
or protein databases:
Protein-protein (blastp): compares an amino acid sequence against a
protein sequence database.
Nucl.-nucl (blastn): compares a nucleotide query sequence against a
nucleotide sequence database (in general optimized for speed, not
sensitivity).
Translated nucl.-protein (blastx): compares the six-frame conceptual
translation products of a nucleotide query against a protein sequence
database.
Protein-translated nucl (tblastn): compares a protein query sequence
against a sequence database dynamically translated in all six reading
frames (useful for searching proteins against EST’s).
Translated nucl-translated nucl. (tblastx): compares the six frame
translation of a nucleotide query sequence against the six-frame
translations of a nucleotide sequence database.

BLAST IS A HEURISTIC.
 A lookup table is made of all the “words”
(short subsequences) in the query sequence.
In many types of searches “neighboring”
words are included.
 The database is scanned for matching words
(“hot spots”).
 Gapped and un-gapped extensions are
initiated from these matches.

THERE ARE MANY DIFFERENT BLAST OUTPUT
FORMATS.
 Pair-wise report
 Query-anchored report
 Hit-table
 Tax BLAST
 Abstract Syntax Notation 1
 XML

BLAST REPORT DESIGNED FOR
HUMAN READABILITY.
 One-line descriptions provide overview
designed for human “browsing”.
 Redundant information is presented in the
report (e.g., one-line descriptions and
alignments both contain expect values, scores,
descriptions) so a user does not need to move
back and forth between sections.
 HTML version has lots of links for a user to
explore.
 It can change as new features/information
becomes available.

HIT-TABLE
 Contains no sequence or definition lines, but does contain
sequence identifiers, starts/stops (one-offset), percent
identity of match as well as expect value etc.
 Simple format is ideal for automated tasks such as
screening of sequence for contamination or sequence
assembly.

THERE ARE DRAWBACKS TO PARSING THE
BLAST REPORT AND HIT-TABLE.
 No way to automatically check for
truncated output.
 No way to rigorously check for syntax
changes in the output.

STRUCTURED OUTPUT ALLOWS AUTOMATIC
AND RIGOROUS CHECKS FOR SYNTAX
ERRORS AND CHANGES.

ABSTRACT SYNTAX NOTATION 1 (ASN.1)
 Is an International Standards Organization
(ISO) standard for describing structured
data and reliably encoding it.
 Used extensively in the
telecommunications industry.
 Both a binary and a text format.
 NCBI data model is written in ASN.1.
 Asntool can produce C object loaders from
an ASN.1 specification.
 Datatool can produce C++ classes from an
ASN.1 specification.

ASN.1 IS USED FOR THE NCBI BLAST WEB PAGE.
server
ASN.1
Return formatted
results
BLAST DB
Fetch ASN.1 Fetch sequence
Request results

DIFFERENT REPORTS CAN BE PRODUCED FROM
THE ASN.1 OF ONE SEARCH.

ASN.1
TaxBlast
report
Query-anchored
BLAST report
Pair-wise
BLAST report
HTML
text
Hit-table
XML
HTML
text

THE BLAST ASN.1 (“SEQALIGN”) CONTAINS:
 Start, stop, and gap
information (zero-
offset).
 Score, bit-score,
expect-value.
 Sequence identifiers.
 Strand information.

THREE FLAVORS OF SEQ-ALIGN,
SCORE-BLOCK(S) PLUS ONE OF:
 Dense-diag: series of unconnected diagonals. No
coordinate “stretching” (e.g., cannot be used for
protein-nucl. alignments). Used for ungapped
BLASTN/BLASTP.
 Dense-seg: describes an alignment containing
many segments. No coordinate “stretching”. Used
for gapped BLASTN/BLASTP.
 Std-seg: a collection of locations. No restriction on
stretching of coordinates. Used for
gapped/ungapped translating searches. Generic.

SCORE BLOCK
Score ::= SEQUENCE {
id Object-id OPTIONAL , -- identifies Score type
value CHOICE { -- actual value
real REAL , -- floating point value
int INTEGER } } -- integer
SEQUENCE is an ordered list of elements,
each of which is an ASN.1 type.
Required unless DEFAULT or OPTIONAL

DENSE-SEG DEFINITION
Dense-seg ::= SEQUENCE { -- for (multiway) global or partial alignments
dim INTEGER DEFAULT 2 , -- dimensionality
numseg INTEGER , -- number of segments here
ids SEQUENCE OF Seq-id , -- sequences in order
starts SEQUENCE OF INTEGER , -- start OFFSETS in ids order within segs
lens SEQUENCE OF INTEGER , -- lengths in ids order within segs
strands SEQUENCE OF Na-strand OPTIONAL ,
scores SEQUENCE OF Score OPTIONAL } -- score for each seg
SEQUENCE OF is an ordered list of
the same type of element.

DEMO PROGRAM (“BLREPLAY”) TO REPRODUCE
BLAST RESULTS FROM ASN.1
 Start/stops and identifiers read in from
ASN.1 (SeqAlign).
 Sequences and definition lines fetched
from BLAST databases.

ASNTOOL CAN PRODUCE XML FROM ASN.1
 Really a
transliteration, not a
new specification
 A Document Type
Definition (DTD) can
also be produced.

ASN.1 AND XML VALIDATION DIFFERENCES.
 XML can be “well-formed” (does not
break any XML syntax rules) or
“validated” (checked against a DTD).
 ASN.1 must always be valid (checked
against a specification).

SPECIAL PURPOSE XML
 NCBI specification does not fit the needs of
some users (the sequence is not provided in
the SeqAlign, when fetched the sequence is
packed 2/4 bp’s per byte).
 Possible to produce XML with more/less
information or in a different format.
 First done as an ASN.1 specification, which
is then dumped as XML.

BLAST XML DESIGNED TO BE SELF-
CONTAINED.
 Query sequence,
database sequence,
etc.
 Sequence definition
lines.
 Start, stop, etc. (one-
offset).
 Scores, expect values,
% identity etc.
 Produced by BLAST
binaries and on NCBI
Web page.

OVERVIEW OF THE BLAST
XML
<!ELEMENT BlastOutput (
BlastOutput_program , BLAST program, e.g., blastp, etc
BlastOutput_version , version of BLAST engine (e.g., 2.1.2)
BlastOutput_reference , Reference about algorithm
BlastOutput_db , Database(s) searched
BlastOutput_query-ID , query identifier
BlastOutput_query-def , query definition
BlastOutput_query-len , query length
BlastOutput_query-seq? , query sequence
BlastOutput_param , BLAST search parameters
BlastOutput_iterations BLAST results for each iteration/run
)>

PARSING BLAST XML WITH EXPAT.
 Expat is a popular
free-ware used for
parsing XML.
 Non-validating.
 Simple C (demo)
program to parse
BLAST output.

OUTPUT SIZES FOR A BLASTP SEARCH OF
GI|178628 VS. NR.
 Hit-table: 16 kb
 Binary ASN.1 (SeqAlign): 35 kb
 Text ASN.1 (SeqAlign): 144 kb
 XML (SeqAlign): 392 kb
 XML: 288 kb
 BLAST report (text): 232 kb
 BLAST report (html): 272 kb

SPECIFICATION (I.E., “DATA MODEL”)
ISSUES SHOULD NOT BE CONFUSED WITH
THE QUESTION ABOUT WHETHER TO USE
ASN.1 OR XML.

STRUCTURED OUTPUT IS NOT A PANACEA.
 Design issues must still be addressed.
 Semantic issues still exist, e.g. is a start/stop
value zero-offset or one-offset.
 Data issues still exist, e.g., is the correct
sequence shown, are the offsets correct, was
the DNA translated with the correct genetic
code?

USE MEGABLAST TO ALIGN VERY SIMILAR
SEQUENCES.
 Best if alignments between query and target
will be 97-99% identical.
 Word-size: 28; an exact match of 28-31
bases required to initiate extensions.
 A greedy gapped alignment routine with non-
affine gapping (constant cost per
insertion/deletion) used to perform extensions
 Use for aligning sequences from the same
organism.

EXAMPLE: SEARCH U93237 (HUMAN MEN1 GENE) VS
HUMAN EST’S WITH FILTERING FOR LOW-
COMPLEXITY AND HUMAN REPEATS AND EXPECT
VALUE 1.0E-6
MEGABLAST:
 word size: 28
 run time: 19 seconds
 469 alignments found
 345 alignments more
than 98% identical
BLASTN
 word size: 11
 run time: 148
seconds
 491 alignments
found
 359 alignments more
than 98% identical

BLASTN alignment:
MEGABLAST alignment:

INCREASING THE THRESHOLD FOR
BLAST[PX]/TBLAST[NX] SPEEDS UP
SEARCH.
 these programs use exact and “neighboring” (three-
letter) words as initial hits.
 increasing threshold decreases the number of
neighboring words.
 fewer neighboring words mean fewer extensions.
 if the threshold is a high value (e.g., 100) only exact
matches are used.
 more subtle alignments will be missed.

EXAMPLE: BLASTX SEARCH OF
NT_078011 (41887 BASES HUMAN
CONTIG) AGAINST NR.
Threshold Time (seconds) Alignments with
expect <= 1.0e-6
12 4050 962
13 2452 955
14 1581 954
15 1139 947
17 770 916
100 658 879

This alignment is found with thresholds
12,13,14,15, and 17. It is not found if only exact
matches are used (threshold = 100).
The default mode of blast[px] and tblast[nx]
requires two hits on the same diagonal to initate
an extension.

DO AS LITTLE WORK (WITH THE DATABASE)
AS POSSIBLE.
 Scanning the database and/or reading it from disk
can be a significant portion of some searches.
 Use the “concatenation” feature of megablast to
concatenate multiple queries and scan the
database only once for all queries.
 The OS will cache recently used files. Make use of
this by grouping together queries for one database
before searching another one.
 Buy more memory if you cannot fit a database into
memory.

MEGABLAST CONCATENATES QUERIES
 searching 50 human EST’s (total of 15,700 bases)
against the human EST database took 23.5
seconds (668 bases/second).
 searching one human EST (gi|272208, 211 bases)
took 13 seconds (16 bases/second).
 Concatenation minimizes time spent scanning the
database.
 The more stringent the search, the more the
savings.

THREE DIFFERENT STRATEGIES TO
SEARCH THREE EST’S AGAINST THE NT
AND EST DATABASES.
 26 minutes (wall clock time) if searches
are grouped by query.
 10 minutes if searches are grouped by
database.
 7.5 minutes if megablast concatenates
the queries.

BLAST DATABASES:
 can be produced with stand-alone formatdb
and a FASTA file.
 are always (?) produced with the formatdb
API (e.g., stand-alone formatdb).
 are almost always read with the readdb API
(recommended).
 pack nucleotide sequences 4-to-1.
 are architecture independent.

THE (PHYSICAL) BLAST DATABASES
COMPRISE FILES IN BINARY FORMAT.
pin or nin Index into sequence and header files
psq or nsq Sequence data
phr or nhr Sequence identifier, definition, taxonomic
information, etc. stored as binary ASN.1
pni or nni * ISAM index file for GI identifiers
pnd or nnd * ISAM data file for GI identifiers
psi or nsi * ISAM index file for other identifiers
psd or nsd * ISAM data file for other identifiers
* only created if “-o” option used with formatdb.

ASN.1 SPEC. USED FOR HEADER FILES
-- one Blast-def-line-set for each entry
Blast-def-line-set ::= SEQUENCE OF Blast-def-line
Blast-def-line ::= SEQUENCE {
title VisibleString OPTIONAL, -- simple title
seqid SEQUENCE OF Seq-id, -- Regular NCBI Seq-Id
taxid INTEGER OPTIONAL, -- taxonomy id
memberships SEQUENCE OF INTEGER OPTIONAL, -- bit arrays
links SEQUENCE OF INTEGER OPTIONAL, -- bit arrays
other-info SEQUENCE OF INTEGER OPTIONAL -- future use
}

USE ASNTOOL TO VIEW HEADER FILE CONTENT.
asntool -m fastadl.asn -M asn.all -d nr.phr -t Blast-def-line-set -p stdout
GI number assigned by NCBI
Swissprot locus name and accession
Tax ID for “house mouse”
Is in swissprot subset of nr
Has information in LocusLink
Used for Protein-Identifier-Group

ASNTOOL CAN ALSO PRODUCE HEADER FILES IN XML.
asntool -m fastadl.asn -M asn.all -d nr.phr -t Blast-def-line-set -x stdout

ALIAS FILES
 Virtual databases uses alias files to instruct BLAST
which physical database(s) to search.
 Alias files have extensions “.nal” or “.pal.
 Alias files can specify multiple databases.
 The search can be limited by a list of GI’s or ordinal
id’s (sequences in BLAST database) specified in
the alias file.
 Alias files hide physical databases with the same
(root) name.

ALIAS FILE USING GI LIST
Physical database
to search
The GI list can be either text or binary; formatdb can
produce a binary GI list from a text one.
Formatdb can be used to produce this alias file with
statistics.
Limits search to this list
of GI’s.
Virtual database statistics

ALIAS FILE USING ORDINAL ID LIST.
Physical database
to search
Specifies subset
to search.
Virtual database statistics

VERY LARGE DATA SETS.
 An array of four byte integers is used in the
pin/nin file to record offsets in the sequence and
header files.
 The header and sequence files are limited to
about 4 gig (4 billion residues or 16 billion bases)
for each physical database.
 Using four byte integers for the offsets means the
pin/nin file stays small for all databases.
 Multiple volumes are used for larger data sets.
 Multiple volumes are produced by formatdb as
needed along with the corresponding alias file.

FASTACMD: A COMMAND-LINE PROGRAM
TO QUERY BLAST DATABASES.
FASTA for one
sequence
Taxonomic
information
Database summary
FASTA dump of database

RESOURCES
 BLAST Home page:
http://www.ncbi.nlm.nih.gov/BLAST/
 NCBI Information Engineering Branch home
page: http://www.ncbi.nlm.nih.gov/IEB/
 Demonstration programs (parsing XML with
EXPAT, blreplay.c, doblast.c, db2fasta.c):
ftp://ftp.ncbi.nih.gov/blast/demo
 NCBI Handbook chapter:
http://www.ncbi.nlm.nih.gov/books/bv.fcgi?call=b
v.View..ShowSection&rid=handbook.chapter.61
0

ASN.1 RESOURCES
 The Open Book : A Practical Perspective on OSI
by Marshall T. Rose (Prentice Hall).
 OSS Nokalva Web site:
http://www.oss.com/asn1/overview.html
 NCBI toolkit documentation on ASN.1:
http://www.ncbi.nlm.nih.gov/IEB/ToolBox/SDKDOC
S/ASNLIB.HTML

EMAIL ADDRESSES
 General questions about running BLAST:
blast-help@ncbi.nlm.nih.gov
 Questions about compiling the toolkit and
requests for hard-copy of documentation:
toolbox@ncbi.nlm.nih.gov

SELECTED BLAST REFERENCES
 Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic
local alignment search tool. J Mol Biol. 1990 Oct
5;215(3):403-10.
 Altschul SF, Madden TL, Schaffer AA, Zhang J, Zhang Z,
Miller W, Lipman DJ. Gapped BLAST and PSI-BLAST: a
new generation of protein database search programs.
Nucleic Acids Res. 1997 Sep 1;25(17):3389-402.
 Altschul SF. Amino acid substitution matrices from an
information theoretic perspective. J Mol Biol. 1991 Jun
5;219(3):555-65.
 Altschul SF, Boguski MS, Gish W, Wootton JC. Issues in
searching molecular sequence databases. Nat Genet. 1994
Feb;6(2):119-29.

(SOME OF THE) PEOPLE (CURRENTLY)
WORKING ON BLAST
 Kevin Bealer
 Christiam Camacho
 George Coulouris
 Ilya Dondoshansky
 Tom Madden
 Yuri Merezhuk
 Yan Raytselis
 Jian Ye
 Richa Agarwala
 Stephen Altschul
 Peter Cooper
 Susan Dombrowski
 David Lipman
 Wayne Matten
 Scott McGinnis
 Alexander Morgulis
 Alejandro Schaffer
 Tao Tao
 David Wheeler

BLAST_CSS2.ppt

More Related Content

Similar to BLAST_CSS2.ppt (20)

More from Silpa87 (14)

Recently uploaded (20)

BLAST_CSS2.ppt