Chapter 2.ppt on Types of Digital f Data

Editor's Notes

• #7: Gartner estimates that unstructured data constitutes 80% of the whole enterprise data. Bank of America Merrill Lynch is the investment banking and wealth management division of Bank of America.
• #8: Data can be classified as- Unstructured data- Data which does not conform to a data model or is not in a form which can be used easily by a computer program. 80-90% of the data of an organization is in this format. E.g. memos, chat rooms, PowerPoint presentations, images, videos, letters, researches, white papers, body of an e-mail etc. Semi- Structured data- Data which does not conform to a data model but has some structure. It is not in a form which can be used easily by a computer program. Metadata is available but it is not sufficient. E.g. e-mails, XML, mark- up languages like HTML Structured data- Data which is in an organized form e.g. in rows and columns; which can be easily used by a computer program. Relationships exist between entities such as classes and their objects. E.g. data stored in databases
• #10: Unstructured data is growing at an alarming rate. 80-85% data in any organization is unstructured. Enormous amount of knowledge is buried in this data. Unstructured data is one which cannot be stored in form of rows and columns in a database and does not conform to any data model i.e. it is difficult to determine the meaning of data. It does not follow any rule or semantics. It can be of any type hence is unpredictable.
• #11: Unstructured Data: Broadly speaking anything that is in a non-database form is unstructured data. It can be classified into 2 broad categories: Bitmap Objects: E.g.: image, video or audio files. Textual Objects: E.g.: Microsoft Word documents, e-mails or Microsoft Excel spreadsheets. Let us take the example of e-mail: Even though email messages may be organized in databases such as Microsoft Exchange or Lotus Notes, the body of the email is essentially raw data – freeform text without any structure. A lot of unstructured data is also noisy text. Noisy text such as chat, emails and SMS texts wherein the language differs significantly from the standard form of the language.
• #12: It is difficult to store and manage unstructured data. Huge amounts of space is required to store unstructured data. It is difficult store images, videos, audios etc.. As the data grows scalability becomes an issue and costs of storing such data grows. Even if they are stored it is difficult to retrieve and recover unstructured data. Updating and deleting unstructured data is very difficult as retrieval is difficult due to no clear structure. Indexing unstructured data is difficult and error prone as the structure is not clear and attributes are not pre-defined. As a result the search results are not very accurate. Indexing becomes all the more difficult as the volume of data grows.
• #13: CAS or Content Addressable Storage. It stores data based their meta-data. It assigns a unique name to every object stored in it. The object is retrieved based on its content and not its location. It is used extensively to store e-mails etc. Unstructured data can be stored in BLOBs or Binary Large Objects in relational databases. While unstructured data such as video or image file cannot be stored fairly neatly into a relational column, there is no such problem when it comes to store its metadata such as the date and time of its creation, the owner. author of the data etc. XML or Extensible MarkUp Language will be explained in detail in semi-structured data section.
• #15: Possible solutions: Naming conventions/ standards: Organization wide file naming conventions or standards need to be defined to ensure easier storage and indexing/ searching of data Tags: Storing unstructured data in a virtual repository which automatically tags the data and helps in retrieval and searching of the data is a possible solution. Documentum provides this kind of solution. Text Mining: Various text mining tools are available which either classify unstructured data or analyze and search based on grammar, context, synonyms etc.
• #20: Semi-structured data does not conform to any data model i.e. it is difficult to determine the meaning of data neither can data be stored in rows and columns as in a database but semi-structured data has tags and markers which help to group data and describe how data is stored, giving some metadata but it is not sufficient for management and automation of data. Similar entities in the data are grouped and organized in a hierarchy. The attributes or the properties within a group may or may not be the same. For example two addresses may or may not contain the same number of properties as in Address 1 <house number><street name><area name><city> Address 2 <house number><street name><city> For example an e-mail follows a standard format To: <Name> From: <Name> Subject: <Text> CC: <Name> Body: <Text, Graphics, Images etc. > The tags give us some metadata but the body of the e-mail contains no format neither is such which conveys meaning of the data it contains. There is very fine line between unstructured and semi-structured data.
• #21: E-mail, XML, TCP/IP packets, Zipped files etc. are semi- structured data as all have certain amount of metadata. Integration of data from heterogeneous sources leads to semi-structured data as data from one source may not have the adequate structure while others may have information which is not required or is missing. In short, semi-structured data is: organized into semantic entities similar entities are grouped together entities in same group may not have same attributes order of attributes is not necessarily important not always all attributes are required size of same attributes in a group may differ type of same attributes in a group may differ For example name and e-mail of different people can be stored in more than one way as shown below: name: Patrick Wood email: ptw@dcs.abc.ac.uk, p.wood@ymail.uk name: first name: Mark last name: Taylor email: MarkT@dcs.ymail.ac.uk name: Alex Bourdoo email: AlexBourdoo@dcs.ymail.ac.uk
• #22: Schemas can be used to describe the structure of data. Schemas define the constraint on the structure, content of the document etc. The problem with schemas is that requirements are ever changing and changes required in data also require changes in schema. Graph based data models can be used to describe data. This is ‘schema less’ approach and also known as ‘self describing’ as data is presented in such a way that it explains itself. The relationships and hierarchies are represented in form of a tree like structure where the vertices contain the object or entity and the leaves contain data. XML is widely used to store and exchange semi-structured data. It allows user to define tags to store data in hierarchical or nested form. Schemas in XML are not tightly coupled to data.
• #23: Semi-structured data usually has irregular and partial structure. Data from sources may have partial structure while some may have none at all. Structure in some sources is implicit which makes it very difficult to interpret relationships between data. In case of semi-structured data schema and data is usually tightly coupled. Same queries may update both schema and data, schema is updated very frequently. Sometimes distinction between schema and data is very vague. E.g. in some cases the data from source may contain the ‘status’ i.e. married or single as true or false and consider it as a separate attribute while in some sources it may be a attribute of a larger set or class. These problems complicate the designing of structure for the data.
• #24: OEM or Object Exchange Model is a model for storing and exchanging semi-structured data. It structures data in from of graphs. The entities are taken as objects which are represented as the vertices of the graph. Labels connect objects and sub-objects hence defining the relationship between them
• #25: Data coming from heterogeneous sources contain different structures (in some cases none at all!) and it is difficult to tag and index them.
• #26: Storing data in a graph based form enables indexing in semi-structured data. It makes search quick and efficient as it is easy to search leaf based data.
• #28: XSLT: eXtensible Stylesheet Language Transformation
• #32: Structured data is organized in semantic chunks (entities) Similar entities are grouped together (relations or classes) Entities in the same group have the same descriptions (attributes) Descriptions for all entities in a group (schema) have the same defined format have a predefined length are all present and follow the same order
• #35: Consider the example taken for semi-structured data. name: Patrick Wood email: ptw@dcs.abc.ac.uk, p.wood@ymail.uk name: first name: Mark last name: Taylor email: MarkT@dcs.ymail.ac.uk name: Alex Bourdoo email: AlexBourdoo@dcs.ymail.ac.uk The same when stored in structured form will be in form of rows and columns each having a defined format as shown above.