Iaetsd effective method for searching substrings in large databases

Effective Method for searching substrings in large Databases
By using Query based approach
J.RamNaresh Yadav
M.Tech 2nd
year, Dept. of CSE, ASCET, Gudur, India
Email: ramnaresh167@gmail.com
Abstract – This paper mainly deal with the direct
substring search in long string and a query point in
the string for searching large database. Traditional
approaches only focuses on effective search using
approximation string search but they don’t
consider the point that how to find exact substring
query for searching in large database. We address
this problem in this paper and we develop an
effective algorithm for query – answering. First,
we develop an algorithm to answer a smallest
substring queries in O(n) time using suffix tree
index. Second we also compute unique substring
in every position of a given string. Once the
smallest unique substrings are pre-computed,
smallest unique substring queries can be answered
online in constant time.
Index terms- substring queries, Query-Answering,
suffix tree.
I. INTRODUCTION
You are searching the Complete Works of
William Shakespeare using query term “king”.
The term “king” occurs 1, 546 times in 1, 392
speeches within 40 works, even without counting
those related words like “king’s” and “kings”.
Using modern information retrieval techniques,
such as an inverted index, one can find all
occurrence positions of a query word easily. How
can a search engine, however, present an
informative list of the search results? Showing all
occurrence positions. On the one hand, if the
length is short, some snippets may be identical and
thus those occurrences still cannot be
distinguished.
A smarter way is to present for each
occurrence a smallest snippet that contains the
query term and is different from all other snippets
of the query term. The above simple yet effective
application in document search introduces an
interesting novel problem to be tackled in this
paper. Given a (long) string S and a query point q
in S, we want to conduct a smallest unique
substring query that finds a smallest unique
substring containing q.
Shortest unique substring queries have
many potential applications. In addition to the
above document search example, shortest unique
substring queries can be used in bioinformatics.
Moreover, finding shortest unique substrings on
DNA sequences can help polymerase chain
reaction (PCR) primer design in molecule biology.
Also, it can help to identify unique DNA
signatures of closely related species or organisms.
The shortest unique substring of the event under
INTERNATIONAL CONFERENCE ON DEVELOPMENTS IN ENGINEERING RESEARCH, ICDER - 2014
INTERNATIONAL ASSOCIATION OF ENGINEERING & TECHNOLOGY FOR SKILL DEVELOPMENT www.iaetsd.in
108

investigation may serve as a concrete working
base of the event context.
Answering shortest unique substring
queries efficiently is far from trivial. A brute-force
(heuristic) search may easily lead to cost in time
quadratic to the length of the string, which is
unacceptable in practice when the string is long
and queries are expected to be answered online.
In this paper, we address the problem of answering
shortest unique substring queries from the
algorithmic point of view and make several
contributions. First, we model shortest unique
substring queries and explore their properties
thoroughly. The properties clearly distinguish
shortest unique substring queries from the existing
related problems, such as computing global
minimal substrings.
Second, we present an algorithm to
answer a shortest unique substring query in O(n)
time using a suffix tree index, which can be
constructed in O(n) time and space, where n is the
length of string S.
Third, we show that, using O(n · h) time
and O(n) space, we can compute a shortest unique
substring for every position in a given string,
where h is variable theoretically in O(n) but on
real data sets often much smaller than n and can be
treated as a constant.
II. SMALLEST UNIQUE SUBSTRING QUERIES
In this section, we formulate the shortest
unique substring queries, and discuss the
properties of several critical concepts.
A. Smallest Unique Substring Queries
Let S be the string of length n. Denoted by
S[i] the value of i-th position of S and S[i,j] =
S[i]….S[j] the substring at position i and ending at
position j. Substring starting at position p. if two
string are identical, denoted by X = Y, and for
every 1 ≤ ≤ | |, [ ] = [ ]. X is called a
substring of Y. we call X a proper substring of Y,
denoted by ⊂ , ⊆ ≠ .
Definition 1 (Minimal unique Substring (MUS)):
A substring S[i, j] is unique in S if there doen not
exist another substring S[ ′
, ′
] such that S[i,j] =
S[ ′
, ′
]. S[i,j]is called a minimal unique substring
if S[i, j] is unique and there is not any proper
substring of S[i, j] that is also unique.
Definition 2 (small unique Substring (SUS)) :
Given a string S and a position p in S, substring
S[i, j] is a small unique substring at position p if
S[i,j] is unique and contains p, and there does not
exists another unique substring S[ ′
, ′
] such that
S[ ′
, ′
] is also contains p and ′
− ′
< − .
Definition 3 (Problem definition): Given a String
S and a position p, the small unique substring
query (SUSQ) is to find a SUS at position p. Any
number in SUS (p) is valid answer.
In our algorithm design, we often consider two
types of unique substring that may be candidates
of SUSs. We give the definitions here and will
pursue further discussion later.
Definition 4 Given a string S and a Position p in S,
a substring S[p, j] is called the left-bound SUS
(LSUS) position p, denoted by LSUS(p), if S[p, j]
is unique and no other substring [ , ′
] is also
109

unique for ≤ ′
< . Symmetrically, [ , ] is
called the right-bound SUS (RSUS) for position p,
denoted by RSUS(p), if [ , ] is unique and no
other substring [ ′
, ] is also unique for < ′
≤
.
Moreover, we define the left SUS be the
SUS whose starting point is smallest, denoted by
leftmost-SUS(p).
Figure 1 shows the relationship among
LSUS (p), RSUS(p) and the MUSs contain p.
Figure 2 illustrates the concept of leftmost SUS.
Figure1. The relationship among three cases
Figure2. The leftmost SUS at a position p.
It is easy to see the following property.
Property 1 (LSUS and RSUS): Given a string S, for
every position p in S, LSUS (p) and RSUS(p), if
exist, are respectively. In some cases, LSUSs or
RSUSs may not exist. Moreover, position p in S,
LSUSs or RSUSs may not be SUSs.
III. QUERY ANSWERING USING SUFFIX TREES
In this section, we first review suffix tree
and the construction. Then we discuss how to use
a suffix tree as index to answer smallest unique
substring queries.
A. Suffix Trees and Construction
A suffix tree is data structure that
concisely records all possible suffixes of a given
string allows fast string search operations. A string
S of length n suffixes S[i,n], i=1, …..,n. in the
suffix tree of S, each edge represents a substring S,
and a path from the root to a leaf node represents
exactly on suffix of S.
Ukkonen’s proposed a well known suffix
tree construction method that requires only linear
time and space when the alphabet size of a string
is a constant. Taking S=11011001 as an example,
we briefly show to construct its suffix tree as
shown in Figure 3, using Ukkonen’s algorithm.
Figure3. The suffix tree of S= 11011001
The construction procedure is illustrated in Figure
4.
Figure 4 The construction of suffix tree
S = 11011001.
110

To extend the suffix tree of S[1, i] to S[1,
i+1] we need to extend S[j,i] for 1 ≤ ≤ , S[i+1].
There are three possible cases.
1) [ , ] ends at a leaf node. Then, we pad
[ + 1] to the corresponding leaf edge.
2) [ , ] does not end at a leaf node but followed
by [ + 1]. Then, we split the edge and creat
a new node.
3) [ , ] does not end at a leaf node but followed
[ + 1]. In this case, we do not need to do
anything.
When we expand the tree from j = 1 to j =
I during phase i+1, the occurrences of this three
phases follow some properties. First, after case 2
or case 3 happen, and then case 1 will never
happens. With these properties, once we meet case
3 at step j of phase i, we can immediately finish
the current phase and start the phase i+1 at step j.
To ensure O(n) construction time,
ukkonne’s algorithm uses the suffix links and the
skip/count technique during the tree construction.
A suffix link is a directed path from an internal
node associated with substring S [i, j] to another
internal node associated with substring S[i+1,j],
which allows fast jump to the next extension point
in the tree. The skip/count technique enables us to
add the new character S[i+1] at phase i+1 quickly.
To save more space, instead of storing copies of
substrings, we label edges using start and end
indexes. To end index of a leaf edge is omitted and
denoted by− . Finally, an end symbol $ is padded
at each path as a leaf node. As a result, the space
used for a suffix tree is reduced to O(n).
B. Query Answering Using Suffix Trees
Given a string S, we first build its suffix
tree in O(n) space and O(n) time using Ukkonen’s
algorithm. We further store all leaf node into an
array so that we can access a specific leaf node
Leaf(i), its edge Edge(Leaf(i)) and its associated
string Sedge(Leaf(i)) in constant time.
We can use the suffix tree to get LSUS (p)
in constant time, as shown in Algorithm 1.
Algorithm 1 the LSUS finding algorithm
Input: string S[1, n], a position p, and the suffix
tree T of S
Output: LSUS (p)
1. Find the leaf node of S[p, n] in T;
//The leaf node can be indexed during the
construction of the suffix tree, so the access to the
leaf node costs O(1) time
2. If the label of leaf edge is $ then return null;
3. end if
4. l←the length of the lablel of the leaf edge;
// the padded terminal charater is not counted into
the length of the leaf edge.
5. Return [ , − + 1]
We first target at the corresponding leaf
node, of p in the suffix tree. Backtracking along
the leaf node, we meet an internal node. Base on
the property of the suffix tree, the represented
string from the root to this internal node is a
common prefix of different suffixes.
With LSUS (p), we can now find a SUS(Smallest
Unique Substring) containing position p as shown
in Algorithm 2.
111

Algorithm 2 the baseline SUS finding algorithm
Input: string S, a position p, and the suffix tree
T of S
Output: the leftmost SUS containing position p
1. Find LSUS (p)
2. if ( ) ≠ them
3. [ , ] be ( );
4. else
5. [ , ] be [1, ];
6. else if
7. for k← p-1; k>0 and − ≤ − ; ←
− 1 do
8. if LSUS(k) is null then continue;
9. end if
10. [ , ] be the LSUS (k);
11. < then ← p;
12. end if
13. if − ≤ − 1 then = ; = ,
14. end if
15. End for
16. Return S[i,j];
IV. A CONSTANT TIME ONLINE QUERY
ANSWERING ALGORITHM
In this section, we develop a method that
pre-computes the leftmost SUS for every position
using linear space. Then, online query answering
can be conducted in constant time.
A. Ideas
We first observe that a smallest unique
substring must fall into three cases: a MUS, a
LSUS, or a RSUS. This can be achieving by using
various theorems. In this paper, we briefly define
theorem which are useful.
Theorem 1 Given a string S and a position p in S,
if [ , ]is a SUS at portion p but not a MUS, then
either i = p or j = p.
Theorem 2 indicates that, by one scan of S and
obtain LSUS at each position, we can find all
MUSs and LSUSs that are not MUSs. To
determine whether a SUS is a RSUSs we have the
following result.
Theorem 3 Given a string S and a position p, a
substring [ , ] ( ) if and only only if
[ , ] contains only one MUS [ , ]( ≤ ).
B. The Framework
Given a string S, our algorithm first
constructs a suffix tree. This takes (| |) time and
(| |) space. For each position p, algorithm
maintains a currently shortest MUS that contains
position p, denoted by p.cand. It also takes (| |)
space to store of the nMUS obtained at the last
position. Therefore, our algorithm needs only
(| |) space overall.
Algorithm 3 shows the pseudo-code of our
method.
Algorithm 3 the pre-computation algorithm.
Input: string s
Output: a SUS for each position 1 ≤ ≤ | |
1. Build a suffix tree for String S;
2. Initialize . ← for 1 ≤ ≤
| |;
3. Output LSUS (1), denoted by [1, ] as
the SUS at Position 1;
4. ← 1, ← ;
// use LSUS (1) to initialize the SUS at position
p-1
5. For p=2 to | | do
112

6. Let [ , ] ( ) obtained from the
suffix tree;
7. Let S[i, j] be the shortest substring among the
following 4 strings: (1) [ , ], (2)
. . ≠ , (3)
, < and (4) [ , ]
if ≥ . If there are more than one
sybstring having the smallest length, pick the
left most one. Output [ , ] as a SUS at
position ;
8. suppose . = [ , ] . ≠ ;
9. if . ≠ > ℎ
10. call PROPAGATE ( [ , ], + 1
11. end if
12. if ( ) = [ , ] is a MUS and is not
[ , ], and > ℎ
call PROPAGTE( [ , ], + 1
13. end if
14. ← , ← ;
15.
At the beginning, we initialize p.cand to
null for all positions p. our algorithm scans string
S from the beginning to the end. At position 1,
LSUS (p) is the only SUS containing position 1.
At each position p (p>1), we compute LSUS (p)
using the suffix tree in constant time.
C. MUS Propagation
Although we reverse space to record the
smallest MUS for each position, we do not need to
explicitly store one MUS at each p.cand. Instead,
we only need to store at those positions p where
the smallest MUS may not be obtained by LSUS
(p) and RSUS (p). Algorithm 4 gives the pseudo-
code of the propagation procedure.
Algorithm 4 the Propagation procedure.
1: procedure PROPAGATE(MUS [ , ]), the
position k to propagate
2: < > then
3: k is not in the range of [i ,j]
4: return
5: else if . I null then
6: . ← [ , ]; return
7: end if
8: supposed . = ′
, ′
;
9: ′
− ′
> − 1 then . is longer than
[ , ]
10: . ← [ , ]
11: ′
> ℎ [ , ] ends before .
12: call PROPAGATE ( ′
, ′
, + 1);
13: end if
14: else if ′
− ′
< − ′
< then
15: . is shorter than [ , ] and ends before
[ , ]
16: call PROPAGATE ( [ , ], ′
+ 1);
17: else [ , ] . have the same length
18: if < ′ then
19: . ← [ , ]
20: call PROPAGATE( ′
, ′
, + 1)
21: else > ′
22: call PROPAGATE ( [ , ], ′
+ 1)
23: end if
24: end if
25: return
26: end procedure
113

V. EXPECTED RESULTS
Suppose we want to conduct extensive
experiments on three real data sets and a group of
synthetic data sets to evaluate our methods.
Mainly we consider three real data sets will be
used in our experiments. the first data set is an
introduction of R language, appear of the FAQa on
the R project website (http://www.r-project.org/).
The second real data set is the genome sequence of
Mycoplasma genitalium, the pathogentic bacter-
ium that has one of the smallest genomes known
for any free-living organism. The third data is the
Bible.
For the R-sequence, we chose the
positions where the language name R appeared as
the query index. For a given sequence, we can
compute one SUS at each position. We want to
observe the distribution of the SUS counts over
different SUS lengths on the R-Sequence. In
addition to the original R-sequence itself, we
further generated three mutations with the same
string length using the alphabet set. For the
original R-sequence, most SUSs are of length of 3.
As the length increases, the corresponding counts
decreases.
VI. CONCLUSION
In this paper, we formulated a novel type
of interesting queries- smallest unique substring
queries, which have many applications. We
developed efficient algorithms. Furthermore, our
study leads to new direction on string queries. As
future work, it is interesting to extend and
generalize smallest unique substring queries.
REFERENCES
[1] B. Haubold, N. Pierstorff, F. M¨oller, and T.
Wiehe, “Genome comparison without alignment
using shortest unique substrings,” BMC
Bioinformat- ics, vol. 6, no. 123, May 2005.
[2] P. Weiner, “Linear pattern matching
algorithms,” in Proc. of the 14th Annual
Symposium on Switching and Automata Theory
(swat 1973), 1973, pp. 1–11.
[3] U. Manber and G. Myers, “Suffix arrays: a
new method for on-line string searches,” in
Proceedings of the first annual ACM-SIAM
symposium on Discrete algorithms, 1990, pp.
319–327.
[4] E. Ukkonen, “On-line construction of suffix
trees,” Algorithmica, vol. 14, pp. 249–260, 1995.
[5] M. Farach, “Optimal suffix tree construction
with large alphabets,” in Proc. of the 38th Annual
Symposium on Foundations of Computer Science
(FOCS’97), 1997.
[6] S. J. Puglisi, W. F. Smyth, and A. H. Turpin,
“A taxonomy of suffix array construction
algorithms,” ACM Computer Survey, vol. 39, no.
2, July 2007.
[7] G. Nong, S. Zhang, and W. H. Chan, “Linear
time suffix array construction using d-critical
substrings,” in Proc. 20th Annual
Symp.Combinatorial Pattern Matching, 2009, pp.
54–67.
[8] D. Cusfield, Algorithms on Strings, Trees, and
Sequences: Computer Science and Computational
Biology. Cambridge University Press, 1997.
114

[9] L. Ilie and W. F. Smyth, “Minimum unique
substrings and maximum repeats,” Fundamenta
Informaticae, vol. 110, no. 1-4, pp. 183–195,
2011.
[10] K. Ye, Z. Jia, Y. Wang, P. Flicek, and R.
Apweiler, “Mining unique-m substrings from
genomes,” Journal of Proteomics and
Bioinformatics, vol. 3, no. 3, pp. 99–100, 2010.
AUTHORS
First Author
Second Author
115

Iaetsd effective method for searching substrings in large databases

More Related Content

What's hot (9)

Viewers also liked (20)

Similar to Iaetsd effective method for searching substrings in large databases (20)

More from Iaetsd Iaetsd (20)

Recently uploaded (20)

Iaetsd effective method for searching substrings in large databases