Why Mark Logic Addressing The Challenges Of Unstructured Information
- 1. Why MarkLogic: Addressing the Challenges of Unstructured Information with Purpose-built Technology B
- 2. Table of Contents 1 | Introduction 2 | Characteristics of Unstructured Information Why MarkLogic: 4 | MarkLogic Addresses Unstructured Information Addressing the Challenges of Unstructured Information 6 | Summary with Purpose-built Technology 7 | About MarkLogic Abstract Rapidly changing conditions are forcing organizations to re-think how they use information to meet their objectives. Whether battling in the market place or on the battlefield, the need for flexibility and agility with information has never been greater. Organizations are looking to integrate and enrich information to create additional value for users. User ex- pectations are changing too, as they demand Web 2.0 and Enterprise 2.0 style applications that provide modern search capabilities, as well as an ability to interact with information through tagging and user generated comments. And various distribution channels present new challenges for information providers in exposing their information through rich user in- terfaces or through syndicated services like RSS and Atom feeds, allowing users to explore and access information in their own context. Choosing the right technology at the core of their application architecture is critical for any organization to provide them with the agility they need to meet these goals and rapidly respond to unforeseen changes. XML servers such as MarkLogic Server provide that agility by providing a single unified platform for storing, manipulating and delivering XML and building innovative information applications. This paper provides a technical overview of MarkLogic Server, the industry’s leading XML server, and also discusses some of the challenges facing organizations today for storing, repurposing, and dynamically delivering information.
- 3. Introduction MarkLogic Server is a purpose-built database for unstructured informa- tion. In this context, “unstructured information” refers to all information that does not fit well in the rows and columns of a relational database management system (RDBMS). In some cases, unstructured information might be semi- or even highly structured, but due to specific characteris- tics discussed in this paper, requires significant efforts to load, store, and query in an RDBMS. Most organizations recognize unstructured information as documents, such as policies, manuals, contracts, reports, articles, cables, journals, and legal briefs. Even media such as user-generated content, RSS feeds, emails, social graphs, metadata, images, videos, and audio files are widely used forms of unstructured information. Most existing tools such as RDBMSs were not built to handle the challeng- es of unstructured information. These tools either require rigid adherence to a specific structure or ignore any existing structure altogether. In other words, they treat unstructured information as second class citizens. This precludes organizations from effectively leveraging information. 1 | MarkLogic whitepaper
- 4. Characteristics of Unstructured • MDDL – Market Data Definition Language Information • DDMS – Department of Defense To understand why today’s most common Discovery Metadata Specification tools are insufficient for leveraging unstructured information, it is useful to Also consider the different document review the specific characteristics of formats such as PDF, HTML, Microsoft unstructured information that require it Office, RTF, etc. These options represent to be treated differently than structured the different ways unstructured infor- information. This section discusses these mation is stored. characteristics while the next section will Contrast this heterogeneity to the homo- discuss how MarkLogic addresses them. geneity of structured information, which Heterogeneous is stored in a consistent, tabular form. The first important characteristic of The data types in structured information unstructured information is it is hetero- primarily consist of numbers, dates, and geneous. In other words, not only does it fixed-length text strings, which limits its look different from structured informa- format variation. Database tables were tion, but the many formats of unstructured invented with this limited variation in mind. information vary significantly from one Since unstructured information varies another. Unstructured information includes greatly, it is not easily stored in tables. non-discrete data types such as words, The challenge is unstructured information sentences, and concepts, in conjunction must be mapped into tables and discrete with discrete data types such as numbers, data types, which entails an unnatural and dates, and identifiers. Many combina- time-consuming effort. As an alterna- tions of these data types are possible, so tive, data types such as character/binary standards are created to maintain manage- large objects (i.e., CLOBs and BLOBs) of ability. However, the gains are not always an RDBMS were created to overcome the clear, since great variance still exists as limitations of the discrete data types, but evidenced by the many domain-specific they facilitate only storage, not querying. standards such as: Therefore, CLOBs/BLOBs are marginally • FpML – Financial products Markup better than storage on a filesystem. The Language problem remains that RDBMSs treat unstructured information as second- • OOXML – Office Open XML for Microsoft Office 2007/2010 class citizens. The monolithic approach of CLOBs/BLOBs ignores the important • ISO 20022 – the ISO Standard for context in unstructured information, and Financial Services Messaging thus precludes analysis, retrieval, and • XBRL – eXtensible Business Reporting updates at a granular level. Language Complex • RixML – Research Information Markup In addition to heterogeneity, unstruc- Language tured information is also very complex. There are several characteristics that • DocBook – a popular markup language for documentation contribute to complexity, any combina- tion of which are found in unstructured information. 2 | MarkLogic whitepaper
- 5. For one, unstructured information is Changing in Unpredictable Ways typically hierarchical, with nested parent/ When unstructured information evolves, child relationships. Often these relation- it changes in unpredictable and unan- ships are not obvious, but examples nounced ways. New standards, new include subsections in a chapter of a book sources, and new applications are created or sub-clauses in a contract. On the other continually. And there are generally no hand, structured information typically restrictions on how it is updated. Take an has flat, tabular relationships that may be example such as a contract. If an attorney expressed as one-to-one, one-to-many, or amends a contract to revise terms, she many-to-many. Since RDBMSs were not updates it in any way she desires without designed for hierarchies, a query to join formatting restrictions. She is not limited rows to recreate the hierarchy is slow and by the number of words or sentences, inefficient. or even by the location of the amended text. She typically uses a word processing Unstructured information is irregular, program like Microsoft Word to make meaning unstructured information does updates, and the user interface does not fit in neat, predefined data elements. not have hard rules on how the contract Information may vary greatly in length, should be changed. There also is no with no pre-definition or bounded data preparation required by IT staff to plan lengths. It might also be sparsely popu- for the changes, as the attorney makes lated, meaning across a collection of the changes ad hoc. information, there might be thousands of known data elements, many of which Contrast this to structured information, are blank. These characteristics are which changes in well-known ways. inconsistent from what RDBMSs expect, For example, each value in a RDBMS in which most columns are expected to changes in an expected way—numbers be filled with values. are increased or decreased, dates are modified with other dates, and text Finally, unstructured information may strings are updated within predefined or may not conform to a predefined lengths. And when the schema changes, schema. If it does conform, the schema the system is first updated to accom- might be poorly defined, not followed modate that change. Schema changes strictly, or not known in advance. Even must be announced before they can in the case of predefined schemas, large be handled by the system. The IT staff variances may be allowed, making each necessarily knows what type of changes item appear very different from the will be made by users to structured next. RDBMSs expect rigid, predefined information before the changes can be schemas with predefined data elements, made. RDBMSs are good for predictable so unstructured information is a poor fit. and announced changes, but are not While some organizations try to map efficient for the changes that unstructured unstructured information into rows and information undergoes. columns, they face huge tradeoffs. Either Text-Centric data accessibility is compromised, or the Unstructured information is heavily text- system takes a significant performance centric. It contains language ambiguities hit due to inefficient storage and indexing. 3 | MarkLogic whitepaper
- 6. typically not clear for processing by comput- MarkLogic Addresses ers. For example, a word such as “foot” can Unstructured Information have several different meanings including a Based on the characteristics of un- body part, the bottom of something, or 12 structured information in the previous inches. The definition is dependent on the section, it is clear today’s most popular context. Without proper context, users may technologies are not able to fully lever- encounter many false positives, in which they age unstructured information. RDBMSs retrieve irrelevant information. They may also lack the flexibility to efficiently handle “MarkLogic’s Universal Index is a key encounter many false negatives, in which unstructured information, and search feature for addressing the heterogeneity they miss relevant information described engines lack the management and update of unstructured information.” using different terminology. capabilities that applications require. Content management systems, which are Also, text within unstructured information largely workflow-oriented applications lacks specific identifiers to help define built on RDBMSs and search engines, various data elements. In comparison, suffer the same challenges because of column names such as “first_name” in an the limitations of the underlying platform. RDBMS table leave no ambiguity about meaning of the data values. While human Despite this, many organizations still readers can easily find names in unstructured try to use their current tools with information such as in a contract, it is limited success. But now organizations far less obvious when processed by a no longer have to compromise. Since computer. Since RDBMSs were designed MarkLogic was designed for leveraging for tabular data, they do not have the unstructured information, it has impor- functionality to properly handle the text- tant features that lead to significant centric nature of unstructured information. benefits. Some of those key features are described below. Exponentially Growing Analysts estimate unstructured information Universal Index grows 10 to 50 times faster than struc- MarkLogic’s Universal Index is a key tured information. Information in gen- feature for addressing the heterogeneity eral continues to grow at a tremendous of unstructured information. It captures rate with one estimate at 800% over all information users need for precise, the next five years. This rapid growth of high-performance queries. Application unstructured information requires new development teams spend less time on data approaches and strategies pertaining modeling, re-modeling, and performance to performance and scalability. Though tuning, thus expediting time-to-market and hardware advancements help with lowering total cost of ownership. Unstruc- scaling, those are only part of the solu- tured information wants to be unrestricted, tion. Software must be optimized with and the Universal Index allows that. modern hardware in mind to maximize efficiency. Organizations that rely on The Universal Index allows users to older technologies must choose between query all information that the system excessive expenditures or insufficient sees, rather than only the information functionality when facing today’s the system is told to see. In other words, unstructured information loads. the Universal Index enables MarkLogic to make no presumptions around what 4 | MarkLogic whitepaper
- 7. information should be expected and can be added ad hoc without having to enables the system to store information redesign a schema. Third, XML has the “To properly handle the complexity of unstructured “as is” without requiring time-consuming flexibility to fully capture and model information, MarkLogic uses a data model based data modeling to standardize dispa- the unpredictable and irregular aspects on XML documents, which is more efficient and rate information formats. This is also of unstructured information, including effective for storing unstructured information referred to as being “schema-agnostic” non-discrete data elements, hierarchical than the relational model.” or “schema-permissive” in which any elements, variable length characters, and schema, or even non-existent schemas, sparseness of data. can be loaded into MarkLogic with no Using XML documents as the data model prior planning. It automatically captures was a natural architectural decision for all elements in information, including MarkLogic Server. XML is ideal for fully words, structure, dates, and numbers. exploiting unstructured information This means no information is lost, and all despite the heterogeneity, complexity, elements can be queried and retrieved. and unpredictable change. MarkLogic’s In addition to effectively handling het- use of XML ensures it can handle current erogeneous information, the Universal and future requirements around unstruc- Index also addresses the complexity of tured information. unstructured information due to hierarchy, Transaction Controller irregularity, and poor schema definition. It Delays in access to information are often also provides the flexibility to accommo- due to limitations in technology. With date the wide variety of changes end users unpredictable changes in unstructured make with their information. information—including those pertaining XML Documents as the Data Model to standards, formats, and content— To properly handle the complexity of the potential for delay is increased. unstructured information, MarkLogic MarkLogic Server was designed to “MarkLogic Server was designed to immediately uses a data model based on XML docu- immediately accommodate those types accommodate unannounced changes, thus eliminating ments, which is more efficient and of changes, thus eliminating the latency the latency found in structured technologies.” effective for storing unstructured found in structured technologies. As information than the relational model. mentioned earlier, MarkLogic’s Universal Support for W3C-standard XSLT and Index and XML data model provide the XQuery, both purpose-built for XML, flexibility to offset the design overhead enables fast and easy querying and for new information types. transformation. MarkLogic customers Those features represent only part have experienced significant improvements of the real-time access capability. in agility and efficiency by eliminating the MarkLogic’s ACID (atomicity, consist- resource drain of trying to model and store ency, isolation, durability) transaction unstructured information in an RDBMS. controller ensures newly inserted An XML data model gives MarkLogic information is indexed in real time several important advantages for and available to users immediately. leveraging unstructured information. Its multi-version concurrency control First, embedded markup in XML creates (MVCC) ensures rapid insertion with context to enable granularity for access, minimal resource contention. Index- updates, reuse, and repurposing. Second, ing can be done simultaneously with XML is extensible so new data elements heavy query loads with no blocking so 5 | MarkLogic whitepaper
- 8. organizations do not have to settle for faster discovery by end users. Geospatial delayed information access. And for searching enables location-based in- “MarkLogic Server provides features to make the most time-sensitive information, formation retrieval. And finally, built-in information clearer, and also provides several MarkLogic’s real-time alerting quickly co-occurrence analysis reveals hidden techniques for finding evidence as the basis and efficiently processes millions or relationships between various entities for relevance.” billions of queries against a fast incoming in a collection of information. feed of new information. Shared Nothing Architecture Search and Analytics Capabilities MarkLogic’s shared nothing architecture Resolving language ambiguities is an allows high performance and massive “important requirement in handling text- scalability to address the unanticipated centric unstructured information. MarkLogic growth of unstructured information. Server helps in two ways to let end users MarkLogic is optimized for commodity find and make sense of the information they hardware, and exhibits linear scaling have. First, it provides features to make to easily and efficiently grow to handle information clearer. Second, it provides future needs. As the user or informa- several techniques for finding evidence as tion load increases, performance and the basis for relevance. response times can be maintained by adding servers to a cluster. To make information more clear, “MarkLogic is optimized for commodity hard- MarkLogic helps with the identification MarkLogic has been deployed in clusters of ware, and exhibits linear scaling to easily and of meaning and context in information. over 100 hardware servers, with expecta- efficiently grow to handle future needs.” For example, integration with entity tions of customers moving well beyond that enrichment tools enables identification in the near future. Not only do customers of entities such as people, places, and gain cost savings by leveraging commodity things. Range indexes provide structure hardware, and fewer of them, but the lower around specific values to enable precise administrative overhead has resulted in and fast retrievals, as well as sorting, the ability to reallocate human resources aggregations, and lookups. Support for to higher value activities. At one customer extensible metadata schemas allows site, only one-half of a full-time equivalent adding any type of identifying data to is required to administer the 100-server existing documents. MarkLogic cluster. To improve relevance in searches, MarkLogic Summary Server provides capabilities found The focus on unstructured information in leading enterprise search engines has increased over the years, but the such as phrase, proximity, and thesaurus ubiquity of RDBMSs has misled many searches. In addition, MarkLogic sup- organizations to make tradeoffs around ports highly tunable relevance ranking functionality, time-to-market, total to more precisely match the end user’s costs, and performance. Since RDBMSs needs. The Universal Index captures all were designed for structured information, components of information to enable a which is greatly different from unstruc- higher level of specificity, granularity, tured information, there is a clear and structure in searches. Range indexes mismatch that leads to costly inefficiencies. enable classification and faceted With its Universal Index, XML data navigation, to help organize information model, transaction controller, search and in meaningful and structured ways for 6 | MarkLogic whitepaper
- 9. analytics capabilities, and shared nothing architecture, MarkLogic is the right choice for tackling the challenges of unstructured information. Customers report significant gains with MarkLogic Server, including 10 to 100 times performance improvements, time- to-market in weeks instead of years, and scaling to hundreds of terabytes today and petabytes tomorrow. About MarkLogic MarkLogic Corporation is revolutionizing the way organizations leverage information. Our flagship product is a purpose-built data- base for unstructured information. Based on patented innovations, MarkLogic Server enables customers in industries including media, government and financial services to develop and deploy information appli- cations at a fraction of the time and cost it takes with conventional approaches. The company is led by pioneers in search engine technologies, database management systems, and business intelligence software. Our founder saw that the traditional ways of managing and delivering information using relational databases and search engines were no longer sufficient. The increasing volume and variety of information necessary for enterprises to leverage required a radically new approach. 7 | MarkLogic whitepaper
- 10. MarkLogic Corporation www.marklogic.com sales@marklogic.com + 1 877 992 8885 Headquarters 999 Skyway Road, Suite 200 San Carlos, CA 94070 Why MarkLogic: Addressing the Challenges of Unstructured Information with Purpose-built Technology © Copyright 2010 MarkLogic Corporation. MarkLogic is a registered trademark and MarkLogic Server is a trademark of MarkLogic Corporation, all rights reserved. All other product names mentioned herein are the property of their respective owners.