Workshop on Preservation and Access for Audio and Video
Download as PPTX, PDF0 likes475 views
This document discusses the history of audio and video formats and the challenges of preserving large broadcast archives as formats become obsolete. It notes that 70% of broadcast archive holdings from 2001 had problems with decay, obsolescence or fragility. Digitization is presented as the solution but challenges remain around encoding, compression and long-term digital preservation. The importance of formal digital preservation practices like migration and emulation are discussed. Statistics are provided on the BBC archive holdings in 1995 and challenges they faced with video and audio formats. Tools for modeling storage costs and risks over time are presented.
![Reference and Citation
• the core requirement for scholarly discourse
– along with a major change in attitude!
• Needs a permanent place for “things to be”
– Hence the need for stable audiovisual collections
“Hamlet, for example, is comparable to Saxo
Grammaticus' Gesta Danorum.[citation needed]
King Lear is based on King Leir in Historia
Regum Britanniae by Geoffrey of Monmouth,
retold in 1587 by Raphael Holinshed.[citation
needed] “
wikipedia](https://image.slidesharecdn.com/wrightfonotecaworkshop23jun14-150209041608-conversion-gate02/85/Workshop-on-Preservation-and-Access-for-Audio-and-Video-59-320.jpg)
Workshop on Preservation and Access for Audio and Video
- 1. Workshop: Preservation and Access for Audio and Video Richard Wright BBC R&D – PrestoCentre Goportis Digital Preservation Summit Hamburg – 19Oct 2011
- 2. We are on a journey
- 4. And here:
- 5. And here: phonautograph, developed in 1857 by Parisian inventor Édouard-Léon Scott de Martinville
- 6. And here:
- 7. And here: "The cinema is an invention without a future" - Louis Lumière
- 8. And here:
- 9. All in the 19th Century In the 20th C, came broadcasting
- 10. And the format wars: List formats and dates
- 11. Leaving us with this:
- 12. The BBC Archive, 1995 • About 1 million hours BBC radio and television • 1.5 million items of film and videotape • 750,000 radio recordings • 3 million photographs • 1.2 million commercial recordings (“grams”) • 4 million items of sheet music • 22 million newspaper cuttings • 550,000 document files; 20,000 rolls of microfilm • 500,000 phonetic pronunciations
- 13. TV Archive Holdings, 1999 Film 30% D3 6% Digibeta 1% Betacam 11% VHS 14% Umatic 4.5% 1” C Format 12% 2” Quad 1% Ekta, Reversal 12%
- 14. TV Formats and Holdings • 2” Quad, 60s-80s: Dr Who, Dad’s Army, Steptoe & Son, Forsythe Saga, Fawlty Towers, Secret Army • 1” C, 70s-90s: Yes Minister, Eastenders, Angels, Wogan, All Our Working Lives • Film: 50 years, 40s-90s: Man Alive, 1984, Ascent of Man, British Empire, Omnibus, Pride and Prejudice 95 • Ektachrome: 67-82 news: Vietnam War, Yom Kippur war, all major domestic stories • U-Matic: 82-90’s: Lockerbie, General Elections in 1983 & 1987, the Gulf War
- 15. Radio Archive, 1999 • Radio holdings: 750,000 recordings, 300,000 hours • Fewer technical problems with audio recordings 1/4” tapes in regions 37% 1/4” tapes in London 32% CD Sound effects 0.02% LP Sound effects 0.4% DAT NCA sequence programmes , bulletins 1% 1/4” News bulletins 3% Cassette NCA sequence programmes 4% CD Programme extract compilation 0.1% 1/4” complete programmes 15% 1/4” film unit tapes 1% DAT, London & Region 2% LP & 78RPM Programme Extract, 5% of sound holdings
- 16. Radio Formats and Holdings • Radio One sessions: 40k recordings, all BBC copyright; mainly ¼” tape – Rolling Stones, Beatles, Who, Jimi Hendrix, Led Zeppelin, The Fall ... and so many more • News off-air on DAT 1990-2001, then CD to 06 • Files from tapeless production onto CD (6 yrs) • Shellac and vinyl pressings from 20s-60s – Special problems with lacquer, acetate, aluminium master discs: 16”, fragile, deteriorate
- 17. The Problem: Analogue Media Decaying Obsolete Fragile Presto Survey, 2001 5 million hours of holdings (10 European broadcasters)
- 18. Decaying Obsolete Fragile • Obsolescence: at least 2/3 of the material • Deterioration: approximately 1/3 of the material • Fragile media: roughly 1/4 of the material Overall: 70% of holdings have problems (2001) The Solution: digitisation
- 19. Obsolescence • Videotape –2”; 1”; U-Matic: no playback equipment • Film –Disappearing in post production • Audio formats –Grams : no playback equipment (in BBC !!) –¼” no longer accepted in BBC radio production and playout systems
- 20. Deterioration • Videotape – decay of adhesive – 2”; 1”; U-Matic (30% read failures at BBC) • Audio – decay of adhesive: ¼” tape – Lacquer discs • Magnetic sound tracks: vinegar syndrome • Other Acetate • Decay of film splices • General decay of polymer materials – Even the sleeves on vinyl LPs
- 21. Fragile Media • Vinyl (45s, LPs) – and shellac (78s) • Film – 10 plays per print (videotape: 50) • Video or audiotape – physical damage – magnetic fields
- 22. Size of the Problem – in Europe • Presto: found 5 million hours 2001 – Mainly broadcast archives • Prestospace: found 10 million hours 2004 – Broadcast and large national collections • TAPE: found additional 20 million hours – In collections not covered previously • UNESCO estimate: 200 million hours worldwide (100 million in Europe)
- 23. Where is the material? • Broadcast archives 30% (roughly) • National collections 15% • Other major collections 15% • Small and specialist collections 40% NB: all these figures refer to archived material ONLY (TAPE survey)
- 24. What to do about it? Presto preservation factory • Efficient workflow for digitisation –Staff specialisation –Cartography and Triage Adam Smith: “the division of labour in pin manufacturing – and the great increase in the quantity of work that results” (UK £20 note) wiki.prestospace.org preservation guide
- 25. Problems with the solution: digitisation not fully accepted “You’re not preserving anything; you’re only making more proxies and adding to the problem” • Not accepted as a solution for film • Not easy to implement for video (in full quality); problems: encodings, compression, formats, file size, bandwidth ... • But – very much accepted for audio: BWF
- 26. Problems with the solution: needs Digital Preservation The approach in 2006: •Media •Multiple copies • Maintenance •Migration
- 27. Media • Datatape : cheaper that hard drives – Needs an expensive tape drive – And has reliability issues • Optical is cheapest of all – But isn’t really mass storage (DVD=4.7 GB) • New DVD format(s) promise 20 to 100 GB – And has reliability issues • Hard drives prices have dropped sharply – Easiest to automate management – And has reliability issues
- 28. Multiple copies • Two copies – Two technologies • In two places • But fastest recovery is by mirroring – Which means identical technologies • Big arguments about RAID vs simpler options vs more complex options
- 29. Maintenance • Life cycle management • Should be every archive’s built-in process • Begins with blank media – Then the writing – Then the initial checking – Then the periodic checking: aerobics, scrubbing • Ends with migration to the next format
- 30. Migration • A fact of life • Every five years • Can involve a lot of manual handling (of datatapes or optical media) • Or can be nearly transparant (disc upgrades) – but: every three years! • Best practice: uncompressed file formats
- 31. Digital Preservation 2009 formal management model Is the format a problem? START HERE Archive for a few years What cost/quality/risk option can you afford Compress lossy YES NO UncompressCompress lossless END HERE (1) (2) (3) (4) (5a)(5b) (5c)
- 32. ... with emulation Is the format at risk? START HERE Archive for a few years What cost/ quality/risk can you afford? Compress lossy YES NO Uncompress Compress lossless END HEREMultivalent
- 33. Stepping back: the real problem with storage (1) Medium Bits/cm² Life Stone 10 10 000 Paper 104 1000 Film 107 100 Disc 1010 10 Each change 1000 times cheaper, but lasts 1/10th as long
- 34. The problem (2) • Current storage media are unreliable – Discs fail – Data tapes fail – Optical media fail (and are easily damaged) – Companies fail – exceptions? Glass discs; Holographic media; Going back to film; Digital film;
- 35. The problem (3) • Storage isn’t just about media – Encoding and obsolescence – File formats and obsolescence – File management systems and obsolescence – Physical interfaces and obsolescence – Operating systems and obsolescence – System complexity and associated risks – Human errors –Cost: continuous maintenance
- 36. What is the cost of continuous maintenance? • You need a model of storage operating and replacement costs, into the future • What storage? So you need a storage strategy: – Allocation of storage: primary, backup, cloud ... – Operation of storage: cycles for copying, checking – Some idea of relating costs to risks !!! • NOT available from storage vendors
- 38. And now: one PrestoPRIME tool • A model for storage systems, to calculate – Cost – Risk – Loss – And compare what-if scenarios • http://prestoprime.it-innovation.soton.ac.uk/
- 41. Storage Systems HDD in servers Migration required every 4 years. Running Costs Access: €0.1 per GB Storage: €1 per GB per year Corruption Rates Access: avg. 1 in 500 files Latent: avg. 1 in 750 files per year HDD on shelves Migration required every 4 years. Running Costs Access: €1 per GB Storage: €0.25 per GB per year Corruption Rates Access: avg. 1 in 100 files Latent: avg. 1 in 500 files per year
- 42. More Storage Systems Data tapes in a robot Migration required every 6 years. Running Costs Access: €0.2 per GB Storage: €0.4 per GB per year Corruption Rates Access: avg. 1 in 1x104 files Latent: avg. 1 in 1x105 files per year Data tapes on shelves Migration required every 6 years. Running Costs Access: €1 per GB Storage: €0.1 per GB per year Corruption Rates Access: avg. 1 in 1x104 files Latent: avg. 1 in 1x105 files per year
- 44. Storage Configuration Found 3 storage configurations. Add... Disk with Tape System 1: HDD in servers Files accessed avg of 0.25 times per year, staying constant Scrubbing every 1 year(s) System 2: Data tapes in a robot Files accessed avg of 0 times per year, staying constant Scrubbing every 3 year(s)
- 46. File Collections • Found 1 file collection. Add... • read-only • Default File Collection • Length of cost/loss projection is 25 year(s). Files • 100 thousand initially, staying constant. • Average File Size • 25 GB.
- 48. Plans Found 3 plans. Add... Disk and Tape edit Delete Evaluate File Collection: Default File Collection 25 year lifetime. 100 files, avg. 25 GB in size. Storage Configuration: Disk with Tape Uses HDD in servers and Data tapes in a robot systems.
- 51. http://prestoprime.it-innovation.soton.ac.uk/ Now: Three Areas of Digital Preservation, and PrestoPRIME tools 1) Digitisation – going digital 2) Digital Workflows – working digitally 3) Digital Preservation (proper)
- 52. 1. Digitisation – Key Ideas • Cartography and Triage – Make a map of your holdings – Decide on priorities • Make a preservation plan – Digitisation: in-house or a service provider? • Better – Faster – Cheaper – Division of labour: Adam Smith, industrial process – Lower prices by contractors for archive work Joanneum: Quality Analysis Tool
- 53. 2- Working with digital content (lots of files) It’s all about management – DAM/MAM and Trusted Repositories – what do they do, what don’t they do -- White Papers – Storage –ITI online free tools – Metadata – Joanneum mapping and validation ; “tag gardening” Univ Amsterdam; fingerprinting INA – Digital library technology RAI, BBC MXF support – Access – Joanneum Time-based navigation, annotation – Rights – RAI ontology, Eurix implementation
- 54. 3- Preserving the digital content • Preservation Platform: P4=PrestoPRIME Preservation Platfom, Eurix; Rosetta, Ex Libris • Standards: OAIS; formal control; formal preservation actions eg migration; P4 • Emulation – Multivalent, Univ of Liverpool • Formats, carriers, storage: Planning and strategy: PrestoPRIME white papers • Managing and maintaining storage into the future –SLA’s for outsourced service; white papers, software for real-time SLA monitoring; modelling and simulation tools
- 55. Access: Audiovisual Content and Digital Libraries • Digitisation makes audiovisual content available to web access, including digital libraries • Broadcast archive projects: Birth of TV, VIdeoActive, EUScreen (link to Europeana) • BUT – what are digital libraries doing to provide access to audiovisual content?
- 56. Four requirements for sensible access • Granularity • Navigation • Reference and Citation • Annotation
- 57. Granularity - division into meaningful units • Keyframes • Other methods to represent video • and audio:
- 58. Navigation • "Click and play" on visual representation of the meaningful units
- 59. Reference and Citation • the core requirement for scholarly discourse – along with a major change in attitude! • Needs a permanent place for “things to be” – Hence the need for stable audiovisual collections “Hamlet, for example, is comparable to Saxo Grammaticus' Gesta Danorum.[citation needed] King Lear is based on King Leir in Historia Regum Britanniae by Geoffrey of Monmouth, retold in 1587 by Raphael Holinshed.[citation needed] “ wikipedia
- 60. Annotation • the core requirement for social web = interactivity • individual interacts with content • individuals interact with other individuals
- 61. Thank You Preservation Guide preservationguide.co.uk PrestoCentre prestocentre.eu Richard Wright preservation.guide@gmail.com