Building Digital Collections: Managing and Sharing

Editor's Notes

• #5: As part of that, they developed 6 modules regarding different aspects of managing e-records and have trained several groups of people to bring those modules to groups dedicated to working with e-records.
• #6: We are looking to digital preservation for an answer because we realize that being in digital form is not the same as being digitally preserved. Digital preservation is active management of digital content over the long term with access as it ’s ultimate goal. With books or documents – We can read it and put it on the shelf and continue to open it and read it for decades with proper handling. However, once something is digitized, we can ’t expect to set it aside and then open it in 10 years much less 50 without active management. We must find ways to ensure that the digital item is accessible. In order determine how we are going to preserve something, we must first have an understanding of what we have. We must IDENTIFY it
• #13: Ask if anyone currently has an inventory and what software is being used
• #14: Ask if anyone currently has an inventory and what software is being used
• #16: Refer to the handout here ….. Extent – How much is there? Nature and Location – is all the information onsite, or would you need to travel to multiple locations to capture everything? Resources – How many people can you get to help – is it just you, a small staff, volunteers? Timeframe – Give yourself a time frame for this. Keep in mind this is never “done”
• #17: HANDOUT – What? - Work at the collection level, not the item level. What is the familiar title for the collection? Description – Provide a brief description of what is in the collection. (photos, postcards, diaries, maps, etc) You are collecting information about items that are known and may be in your catalog + items that have come in your door that are waiting to be dealt with + items that are being created (digitization projects) + things you may not even know about yet…….
• #18: Creator – so that you can go back to them with any issues
• #19: It ’ s a good idea to note the format of you digital media, or what the digital content is stored on, since some format types last longer than others. Digital content on more fragile media (floppy disc) might be a higher priority.
• #21: You should make sure to specify the location of digital content in your inventory. Some things you will want to consider: How will you specify whether content is located online (meaning on your computer hard drive or a network server), or offline (meaning stored on some removable piece of media, like a CD or flash drive)? Location in storage system Keep in mind that you will need to update the inventory whenever the content moves. If you get too specific you might spend all your time updating file locations. Ask Audience - WHAT OTHER FIELDS ARE NOT INCLUDED THAT WOULD BE HELPFUL
• #23: After you ’ve compiled your inventory, it can be easy to get overwhelmed. You know you’ve got lots of digital content, but how much of it is really your organization’s responsibility to preserve? Meanwhile, you’ve still got more logs—more new digital content—coming in down the river. One of the goals of selecting content to preserve is to help get your logs moving again—start setting priorities and pick a few things to tackle first so everything can start flowing more efficiently.
• #24: Not all of the content you ’re dealing with may in fact be appropriate or necessary for you to preserve, and you don’t want to commit resources to preserving materials you don’t have to. You may hear people argue that storage is cheap so we should keep everything. Unfortunately that perspective is rather short-sighted. Storage may be cheap, but preserving the quality of content over the long-term is not. There are periodic migration costs , moving the digital materials into systems where you will preserve it. Monitoring files for corruption and change . [have you lost bits? Are the files degrading? Not to mention maintaining access to the files, which means updating your discovery and dissemination services every time hardware and software change. [an ongoing, recurring cost] The idea behind long-term preservation is that you will be making this content available in the future . It isn ’t enough just to save the content if you can’t access it any more. [ Quality ] Even if we could keep everything forever, would we want to? Is that manageable given the type of content that you hold? Not all digital content may be preservation quality – if you have high resolution scans of your photos, do you also need to preserve the lower quality versions of these scans? And not all will be significant enough to warrant preservation. [that string of emails about organizing the staff Christmas party…] Does the digital content we take in match our mission and scope of collections ? Quite often materials find their way to us that have little or nothing to do with our mission, yet we give hem a home and expend our resources on maintaining them. Maybe there is a better/more logical home for that content? [Maybe you could partner with another org that is better placed to hold and preserve that content.] The selection process for digital content is very analogous to the selection process for non-digital materials – you don ’t collect materials for your archive that don’t match your mission, and you should keep that same principles in mind when selecting digital content.
• #25: When you ’ re first getting started, it ’ s helpful to treat selection as a managed, structured project in order to plan and coordinate the process [and plan for the future]. The selection criteria you choose will be unique specific to your situation, your organization and its mission. Once you have your selection criteria, it may not be possible to review/select everything at once, so how might you sequence the process? Again, the answer will be different for each organization. Think about what ’s most significant to your organization? Look inside your organization first: are there mission-related documents that might give you clues? existing manuals and policies , such as records retention schedules? Or Collecting policies? On the question of uniqueness , Is this the only source or is it preserved elsewhere? Avoid duplication The value of material can be determined by a variety of factors - historical, evidential, can ’t reproduce? This must be assessed within the context of your own institution. most extensive ? (and therefore a more coherent body of material to manage) most requested/used ? Easiest to tackle (e.g. most familiar, most ready for ingest – a quick win for your digital preservation process; very helpful when you are having to prove the value of your efforts to a reluctant administration ) Oldest ( possible historical importance) Newest (possible immediate interest) Mandated (via local policies, legislation, etc.) At risk ? If it were no longer available, what digital files would be the hardest to replace? Some formats become obsolete a lot faster than other formats. PDFs are viable for a really long time – video files, however, get old very quickly.
• #26: Even if something fits your desired criteria, it still might not be reasonable for you to select it. You can use decision tree or list of questions to help you decide what ’s practical to preserve. You ’ve already considered the content in view of your selection criteria. And you should already have answered ‘yes’ to both of these Qs to continue considering the materials you hold. does the content have long term value? does it fit your scope and mission ? Next you need to consider Technical issues: is it feasible for you to preserve the content? [Is it a “ digital time bomb ” ? S ome formats are a challenge to preserve, such as video/time-based media. Some may be too damaged to preserve. Do you have the skills and resources (either to undertake the preservation yourself or to buy the skills in)? Some types of material may require far more expertise and resources than you have available. And Access. Even if we ’re not making it public, how useful is a server full of digital content that is safe, but that we can’t access? We need to ask is it possible to make the content available over time? Are you the only holder of this content? [ Duplication] If it is not feasible to preserve the content, and not possible to make it available and usable, then it probably shouldn ’t be included in your selection –especially if you know you are not the only holder of this digital content.
• #27: Identify and Select An Introduction to Digital Preservation
• #29: Identify and Select An Introduction to Digital Preservation
• #32: Co-locate – It ’s OK to move things around if it makes sense to do so. Layers – If you have several layers to hunt through, it can be really hard to find anything – Shallow is better Searching is really difficult if you have to search through multiple layers Breadcrumbs – OK to leave “sticky notes” (AKA “READ ME”) files in folders. Can give a brief description of contents, retention schedule, any naming conventions Don’t know – unknown file formats, files on old media (floppies), password protected
• #33: File backups – EX: Speeches had multiple drafts  Final + copies in several different font sizes Supplementary files – folder of images that were used in a power point. Files you can’t open – Corrupted Formats – may receive Word and pdf – May not want to keep both. As you are creating your inventory, you are likely to discover a lot of really simple places you can clean up the files you are reviewing. Co-locate – It ’s OK to move things around if it makes sense to do so. Bury – If you have several layers to hunt through, it can be really hard to find anything – Shallow is better
• #34: The outcome of going through the work of selection is to gain a sense of control over what you have to deal with, what your scope is, and what your policies and priorities are for selection. This is critical to developing a sustainable program for support of long-term preservation and access. By applying your selection criteria to your inventory, you will have more detailed information to work with in your planning. This documentation can also inform your work with creators of digital content. This might include the creation of submission agreements or other policies so that the content coming in to your organization fits your selection criteria for long-term support. The selection process puts you on the path to a sustainable program. Selecting content is ultimately not a one-time project but a long-term, ongoing process, so formalizing it through policies, schedules and other documented criteria will help you avoid more log jams in the future. Identify and Select An Introduction to Digital Preservation
• #37: We ’ve learned that it is essential to remove duplicates first. Once you start using other tools and changing things, the duplicate finder applications are no longer as accurate. These are all FREE We have used all three of these and they all work a little bit differently under the covers so the results vary a bit. Auslogics has a number of products that are for sale, but this one falls under their “freebie” category We found it really helpful with documents and wanted to try it with the images. Similar Images - It creates a database, so consecutive runs go faster but the first run while it is creating the database can be really slow. This application works with lots of file formats Visipics – This application only works with a handful of file formats, but it hits the main ones and does it really well. It will detect two different resolution files of the same picture as a duplicate (we had a number of photos that were corrected with Photoshop and this picks those up), or the same picture saved in different formats, or duplicates where only minor cosmetic changes have taken place.
• #38: This application had some real strengths + You can run it against USB removable devices (but not against a network drive) When it searches, it pulls duplicates by comparing the MD5 checksum of a file – so we were highly confident that these were the exact same image + Can “bulk delete” if you are confident so that is a speed/efficiency plus Negative parts of this tool Unlike the other two programs, there is no “preview mode” so you can’t see the images although you can pull them up if you click on them. You can also right-click and check the properties of the images and get all kinds of metadata – (camera make and model, f stop, exposure time, ISO Speed, date, time, dimensions, dpi, bit depth, This doesn ’t catch the “altered” images (changed the lighting) or “Similar” images
• #40: When run again choosing “Ignore File Names” and “Ignore File Dates” more duplicates are likely to come up It will find: + It found duplicates with a name change + Found some photo glitches that the other programs didn ’t It also found a lot of problems (same picture with different dates) so it may provide you some information to resolve metadata issues.
• #43: Similar Images Will show errors (.avi file, image file) – often you will find these are corrupted images. Default threshold is 35 and should catch “pairs” that are pretty similar This found 483 duplicates, 18 of those are “similar images” with the threshold set at 15. (483 results) (0  100). When run at a threshold of 30 – there were 538 results. Default threshold is 35 and should catch “pairs” that are pretty similar Fast / Duplicates : 0 - 10 Photos : About 12 Collages : 12 – 35 (Images close to each other) Collections of Scans : 50 - 60 Mixed Stuff : 20 - 40 Semi Automated Deletion – Based on rules you set –this feature pre-select the file to delete but you have to confirm Lower/higher resolution, bigger/smaller file size, older/newer file, left/right file – The rules will be checked from top to bottom LOTS of file types to choose from to include in the comparison Database stores – file name, file size, last change, dimensions, color/model and will remember if you indicate a pair is not a duplicate
• #44: Similar Images has a nice working screen – photos are side by side and a lot of the key metadata is there to compare. File name and path are there. File size – the bigger one is in green Things that are == show in red CRC32 – fast hashing algorithm that generates 32 bit checksum on the file – if they match, they are red Type of file – match == red Last modified – the most recent will be green if they are different. + Nice large screen in which to view both photos together + Easy GUI interface + Colorcoding + They have a pretty decent basic “Help” document to explain the icons/colorings/setup. Negatives When the view window is up, you can ’t do anything on the main window which sits behind it Can only view one set at a time and use the Next/Previous to move around
• #45: Everything is TOTALLY identical except for the name – they are even in the same folder Keep the labeled photo
• #46: Adjusted photo – The right photo now gives us a name and cleans it up
• #47: Visipics Like Similar Images – You choose the folders you want to compare and set the Threshhold The pictures appear along the left and you can skip around between the “pages” You can mark the deletions by clicking on the photo you want to delete. Runs faster than Similar Images Auto Select allows you to preselect images for deletion based on rules (but many fewer rules available than Similar Images) You can also move or rename photos from this screen if you need to
• #48: We ’ve learned that it is essential to remove duplicates first. Once you start using other tools and changing things, the duplicate finder applications are no longer as accurate. These are all FREE We have used all three of these and they all work a little bit differently under the covers so the results vary a bit. Auslogics has a number of products that are for sale, but this one falls under their “freebie” category We found it really helpful with documents and wanted to try it with the images. Similar Images - It creates a database, so consecutive runs go faster but the first run while it is creating the database can be really slow. This application works with lots of file formats Visipics – This application only works with a handful of file formats, but it hits the main ones and does it really well. It will detect two different resolution files of the same picture as a duplicate (we had a number of photos that were corrected with Photoshop and this picks those up), or the same picture saved in different formats, or duplicates where only minor cosmetic changes have taken place.
• #49: This screen is very representative of what many of our folders look like when we open them. Irfanview ’s strengths + Can change the size of these photos so that you can see more/less + Can delete from this page without having to go back into the folder + Can double-click on photos to see full screen versions and move through the images that way
• #57: Resulting MD5 file can be opened in any text editor…..
• #64: WHAT are you going to store it on? WHERE you are going to store it? HOW MANY COPIES are you going to make?
• #65: WHERE are you going to store it? What are your Options? Decisions can be determined by a number of things……. Size – The options you consider will vary depending on how much you have to store. Media – CDs – on average 5 years Gold CDs - more If you’ve burned it – as little as 2 depending among other things on the quality of the CD to begin with. Magnetic Tape – could last 30 years but its very sensitive to heat, magnetic fields and dust. Is the company producing the hardware you are using to run the storage media still around? Cloud – what’s it going to cost to rent space. Sometimes it costs more when you pull it out than when you put it in. You also need to determine where you don’t want to store it and migrate it off those devices accordingly USB drives, old media,
• #66: Three copies is a happy medium if you are able
• #67: RAID = Redundant Array of Independent Disks = multiple hard drives in one package
• #72: These are the elements you want to aim for when planning for providing access to your content over time. Easily: this will change over time and will depend a lot on your staff, users, nature of your institution Coherently: document what you ’re doing, provide context Completely, Correctly, Reliably: these all relate to how you manage the technology Consistently, Fairly: write good policies and procedures and stick to them
• #98: Some ideas: Think about providing an online survey instrument at various strategic points within your access environment. Implement some free web analytics – find out where people are linking into your site, what are they requesting, how much time do they spend on what materials. If you have email subscribers– solicit some inquiries from your regular users about their visits to your digital collections and how they might be finding them useful, as well as ways of improving upon your existing levels of service. If you are only providing access on-site, make sure you add some lines on your visitor forms that account for their use of your digital collections. So, you may get very good at serving your digital collections up to your current stakeholders through these different monitoring measures. But what about new users, users who could really benefit from exposure to your materials but may require it in a different form or through different means, like tablets or cell phones. Maybe near-future users will want to run all sorts of sophisticated services over the top of your materials along with materials from all sorts of other institutions. How might you be able to track trends or use cases from other similar institutions to find out where access needs are heading?
• #100: Mention funding with media replacement