I say emulate
1 like1,059 views
The document summarizes a project testing the feasibility of emulation and migration as strategies for preserving web archives. It describes the National Library of Australia's web collections and common web object types. It then details tests conducted on a sample of the NLA's PANDORA archive using emulation and migration tools. The tests found that emulation was complex and slow, while available migration tools were imperfect and slow. Both strategies require significant resources and clearly defined preservation policies. The document concludes that no proven digital preservation methods currently exist and more real-world testing is needed.
I say emulate
- 1. I Say Emulate; He Says Migrate Are emulation or migration feasible preservation strategies? National Library of Australia Prepared by: Andrew Stawowczyk Long Presented by: 1 David Pearson
- 2. Archiving the Web • Many institutions actively harvest the web • Collecting scale vary • Preservation practices not well understood and implemented • Collecting intent may differ depending on the institution 2
- 3. Web Archives • Type • Text oriented • Multimedia (video/audio) oriented • Picture oriented • Databases • Combination of all types • Storage • Uncompressed • Compressed (WARC) • Combination 3
- 4. Web Objects and Elements • Challenge: Web archives may contain any type of digital object • Common objects • HTML/XML and related (htm, html, xml, css, etc.) • Images (raster images – JPEG, GIF, PNG) • Media • Audio files (au, wav, aiff, midi, mp3) • Video files (mov, mpg, wmv, rm) • Other objects • File Archives (usually compressed – zip, tar, gz, arc, sit) • Images (raster images – bmp, tiff) • Images (vector images - SVG) • Text files (txt, csv, rtf) • Document files • PDF • Microsoft Word, Excel, Power Point 4
- 5. Comparative statistics of NLA web collections PANDORA (selective) .au Domain Harvests Files: 73 million Files: 2.3 billion Size: 3.26 TB Size: 78.75 TB Domain 2005 2006 2007 2008 Harvest Unique 185 million 596 million 516 million 1 billion files Hosts 811,523 1,046,038 1,247,614 3,038,658 crawled Size 6.69 TB 19.04 18.47TB 34.55 TB 5
- 6. What are we preserving? Preservation Intent • Preservation of: • Physical media? • Bit-stream (logical form of data)? • Action (rendering data into something useful to user)? • User experience? • Important Considerations • Creator’s perceived intent • Institution’s preservation intent 6 Based on Heslop and Davis (2002)
- 7. What are we preserving? Properties • Object Properties (Properties regarded as important would vary depending on the intention of the collecting institution) • • Derived from file format High-level – e.g. layout, formatting or WEB • Measured – identified directly by computer • Intended – Set by the collecting body 7
- 8. Possible Preservation Actions 1 • Emulation The original environment is recreated on a contemporary hardware using specialised software (emulator) and original software. • Renderers • Specialised software, operating in the contemporary environment and used to access (render) original files. It is similar to emulation. 8
- 9. Possible Preservation Actions 2 • Migration Original file formats are migrated (converted) to another format, which is supported by current hardware/software. e.g. MS Word 3.0 to MS Word 2008 9
- 10. Possible Preservation Actions 3 Not long-term sustainable • Technological Museum Collect and maintain the original hardware and software • Take No Action Do nothing 10
- 11. Digital Preservation Preliminaries • Collection objects need to be correctly recognised and identified • Preservation intent(s) need to be defined • High-level preservation actions need to be defined (e.g. shall we use emulation or migration?) • Practical-level preservation actions need to be defined Object Format + Preservation Intent = Appropriate Action Dillema: How to properly migrate data if preservation intent(s) are unknown or not defined 11
- 12. Tools Required for Emulation • Emulators • Fast, stable, flexible, extendable • Licenced Operating Systems • Various drivers • Web browsers • Browser plug-ins • Other programs as required (e.g. Java, Adobe Acrobat Reader) 12
- 13. Tools Required in Migration • Format identifiers • Format converters • Link updaters • QA automatons CAMiLEON project – Migration on Request Tool XENA 13
- 14. Project Tests General Testing Environment • Large slice of uncompressed PANDORA archive (random selection) • Whole Domain Harvest archive have not been included in tests (WARC files) • Multiple hardware combinations • Multiple OS combinations • Multiple Web Browsers 14
- 15. Project Tests Material Sample Testing the industrial scale tools • PANDORA slice • 861Gb • 18,019,172 files • 2,379,326 folders Testing object properties • Smaller slice of PANDORA slice • 20 objects of each selected types •Audio, html, images, pdf, video, zip, MS documents 15
- 16. Project Tests Methodology • Large sample testing (861Gb, 18,019,172 files) • Attempt to identify objects in the sample using DROID • Attempt to migrate jpeg images to png and update links • Small sample testing • Select smaller sub-sample, with objects mostly created before year 2000 • Identify objects in the sample • View and experience selected objects in contemporary environments using various platforms, OS and browsers • View and experience selected objects in old environments using emulations on various platforms, using different OS and browsers • Migrate selected objects and review them in various environments 16
- 17. Project Tests Tools tested • Common • Emulation • DROID • QEMU • JHOVE • Bochs • TRiID • File Identifier • MS Virtual PC (Not exactly an emulator) • Lister (dev. in-house) • OS ● Dioscuri – MS Win XP Pro – MS Win 3.1 • Migration – MS Win 98SE • ImageMagick – Ubuntu 9.04 • MediaCoder • Web Browsers – MS IE 7 • Swf>>avi – Firefox 3 • OpenOffice Tools – Arachne 1.2 • XENA – Mosaic 2 17 – Netscape 4
- 18. Project Tests Control – Current Environment • Properties observed in selected files Object Basic Characteristics (based on Emulation Project by KB) 1. Content : the text, images, etc. from the object 2. Structure : the cohesion between different parts of the object 3. Context : the meaning of the object. 4. Appearance : the way an object is presented to the user. 5. Behaviour : the interaction of the object with the user or system. E.g. for HTML pages: •Rendering of text, images, media files • Font, layout, colours, contrast, brightness, animation smoothness, sound quality, etc. •Objects dependencies •Mouse & keyboard behaviour •Data extraction 18
- 19. Project Tests Emulated Environments • Hardware • Dell Optiplex GX620, P4, 4.4GHz x 3.39GHZ, 3.5Gb RAM • Power Mac G4 EMULATORS: • Bochs • Host: WinXP Pro v2002 SP3 Ubuntu 9.04 • Client: Win 3.1, MS DOS 6.2 WinXP Pro SP2 • Dioscuri 0.4.0 • Host: WinXP Pro v2002 SP3 • Client: Win3.1, MS DOS 6.2 19
- 20. Project Tests Emulated Environments • Qemu • Host: MS WinXP Pro v2002 SP3 • Clients: MS Win98SE MS Win 3.1 MS DOS 6.2 Ubuntu 9.04 • Host: Ubuntu 9.04 • Clients: MS WinXP Pro SP2, P4, 12.92GHz, 256Mb RAM MS Win98SE MS Win 3.1 • Microsoft Virtual PC • Host: MS WinXP Pro v2002 SP3 • Clients: MS Win 3.1 MS Win98SE 20
- 21. Tests - Summary Emulation •Setting up emulators was relatively simple •Additional software (especially to work with disk images) proved to be extremely useful. •Licencing was at times a big obstacle. (E.g. Impossible to emulate Macintosh environment legally). •A lot of dependencies exist. It is a complex task to make programs work correctly. •e.g Windows XP requires internet or over-the-phone activation after 30 days 21
- 22. Tests – Summary Emulation • All Some of the dll libraries in Win 3.1 did not agree with 16-bit Netscape and Mosaic programs • Bochs 2.3.7 for Windows • Extremely slow in GUI environments • No full screen mode. Limited end-user experience. • Dioscuri • Sluggish at times • Didn’t like some of the images created in WinImage • Qemu 0.9.0 for Windows and Linux • Much faster but still sluggish at times • Win98SE couldn't run in hi-res, hi-colour mode • Microsoft Virtual PC Relatively fast (it's a virtualisation software on PC) but still sluggish at times 22
- 23. Tests - Summary Migration Environment •Dell Optiplex GX620 •MS Windows XP Pro v2002 SP3 •Networked drive with PANDORA sample 23
- 24. Tests - Summary Migration •Available tools are imperfect and slow. • e.g. DROID took more than two weeks to examine slightly over 18 million files and many of them were not recognised •It is very difficult to examine contents of the container formats (e.g. avi or rm) •Network connections need to be as fast as possible •It is difficult to make informed decision about migration without preservation intent clearly defined 24
- 25. Tests - General Comments • No proven methods exist Real-world testing is needed • Most documented approaches are ad-hoc - no commodity solutions • Tools are few and inadequate 25
- 26. Tests - General Comments • Preservation policies, especially about preservation intent are needed • Significant resources are needed to practically tackle the problem 26
- 27. Andrew Stawowczyk Long Strategist Digital Preservation Standards NLA anlong@nla.gov.au David Pearson Director (Acting) Web Archiving and Digital Preservation Branch NLA dapearso@nla.gov.au Project Report is due end of October 2009 27