Digital preservation and preserving the digital

I've been doing digital preservation stuff in one way or another for over twenty years, managing projects, building servers, installing systems, even writing code.

In the early days it was all a bit finger in the ear - no agreed standards, no agreed preservation formats, and lots of home grown and ultimately unmaintainable solutions.

Nowadays, and I'm possibly out of the loop a little, being retired, it's basically a just add dollars problem.

There is a tacit agreement about what to do, how to do it, and what software to deploy.

For some people a proprietary solution with predictable recurrent costs makes sense, others may find an open source solution with its low entry costs. but implied long term funding for software maintenance ( and by implication maintaining a specialist team) more to their taste.

Either are good, either will work, it's basically a financial decision as how best to proceed.

The days of having a test server sitting under your desk (and I've been there) are gone.

But there's an elephant in the room.

To date most digital preservation efforts have been focused on preserving digitised artefacts.

Books, early modern ballad sheets, insects collected in the nineteenth century, rude french postcards, etc etc.

And this is partly because a lot of digital preservation efforts have been driven by librarians and archivists who wished to digitise items under their care for preservation purposes.

And this model works well, and can be easily extended to born digital records, be they photographs, medical records, or research data - it's all ones and zeros and we know how to store them.

And being linear media with a beginning and an end we can read the file as long as we understand the format.

What it doesn't work well for is for dynamic hypertextual resources that do not have beginnings or ends but are instead database driven query centric artefacts:

From memory, the wagiman dictionary was written mostly in perl and did a whole lot of queries on a database. I know of other similar projects, such as the Malay Concordance Project, that use similar technologies.

Essentially what they have is a web based front end, and a software mechanism for making queries into a database. The precise details of how individual projects work are not really relevant, but what is is that the webserver needs to be maintained and not only has to be upgraded to handle modern browsers, it needs to be made secure, the database software needs to be maintained and of course the query mechanism needs to be upgraded.

Big commercial sites do this all the time of course, but academic projects suffer from the curse of the three year funding cycle - once it's developed there's usually no funding for sustainability which means that even if it becomes very useful, no one's there to upgrade the environment, which means that it starts to suffer from bitrot. Left alone, sitting on a single machine with a fixed operating system version it would probably run forever, but hardware dies, operating systems are upgraded and all sorts of incompatibilities set in.

While it's not quite the same thing, go and take that old tablet or computer you've had sitting on a shelf since 2010 and have never quite got round to throwing out. Try accessing some internet based services. Some will work, some won't.

And the reason is of course that things have change in the intervening ten years. New versions, new protocols and so on.

So, what to do?

One solution is to build a virtual machine using all the old versions of software (assuming of course you can virtualise things successfully). The people who get old computer games running have a lot to teach us here - most games used every tweak possible to get as much performance as possible out of the hardware of the time. The fact that hardware is now immensely more powerful than even five years ago means that the performance cost of running emulations can be ignored, even if they're particularly complex.

This gets rids of the hardware maintenance problem - as long as your virtualisation system allows. and continues to allow you to emulate a machine that old you're home dry - except you're not.

You need to think about secure access, and that probably means a second machine that is outward facing and passes the queries to your virtual machine. This isn't rocket science, there's a surprising number of commercial legacy systems out there - business process automation solutions for example - that work like this.

The other thing that needs to be done is that the system needs to be documented. People forget, people retire to Bali and so on, and sooner or later the software solution will need to be fixed because of an unforeseen problem ...

Diaries

Well it’s almost the end of the  year and that means a new diary.

So, why in 2020 do I still use a diary?

It’s not as if I’m technophobic, I was an early and enthusiastic adopter of the Palm pilot, so much so that around about 2000 I simply stopped using my diary as an appointment book, and then there was a gap of around seven or eight years when I simply didn’t use a diary.

And then I started using them again, simply because I started travelling for work and using a diary as a planning tool gave me visibility of what I was doing and where I would be. It also had the advantage that you could stick postit notes in it on appropriate pages with names of hotels, airport shuttle booking numbers and the rest to save wading through a loosleaf folder full of booking dockets.

I use to favour the Leuchturm planner diaries with a week on the left and a notes page on the right, and what I would do is write down a two or three line summary of any important meeting such as a meeting, and system upgrade or a system failure of some sort.

I was even geeky enough to use different coloured inks for notes relating to different projects.

And then what I would do is, at the end of the week, scan the diary and notes page - the Leuchturm books opened flat to A4 - for that week and save it in Evernote, which made compiling project progress reports and activity reports a hell of a lot easier and less of a work of fiction that they might otherwise have been.

After I retired I carried on with my Leuchturm planners for a couple of years, but this year, 2019 I used a more normal week to a page diary with any notes simply attached to a postit note.

That seemed to work reasonably well - I certainly don’t need the notes page anymore, but I found the individual day boxes a bit small and restricting, so next year (2020) I’m going to try a day-to-a-page diary which, while it’s still the A5 format is quite a bit bulkier than anything I’ve used in the last 20 years.

It’s not an expensive imported one - no Moleskines or Leuchturms this year, but one I bought in a discount store in Queensland last August for ten dollars or thereabouts.

The reason for buying one so early is that we’re planning a reasonably complicated overseas trip in 2020 and already by August we had payments for flights, deposits, and due dates for payment, and even by August this was overflowing into 2020, and I needed a diary for planning purposes.

It’s strangely difficult to buy a decent diary for the following year in August - the calendar year diaries are not yet in stock and the unsold financial year diaries (July to June) are being sold off.

We’ll see how it goes - you never know it might wean me off my taste for expensive import designer diaries ...

Bushfire planning

[This post is about bushfire planning for small museums. I've been getting some hits from people looking for help getting their bushfire plan together. If you've come here looking for general information check the relevant CFA or RFS pages.]

It's hardly the most festive of topics, but with the ongoing bushfire emergency on the east coast the time has come to update our disaster planning.

We've over four thousand artefacts - I'm not entirely sure how many, I havn't finished documenting the collection yet - so any evacuation would mean taking only the highlights.

It's a pity, but in an emergency we would have to abandon most of the collection to its fate.

I tried looking on the web for any useful examples of disaster planning for small museums or historic buildings but did not find any relevant examples, so I wrote my own based on my past experience of writing disaster plans as to what to do with archival computer tapes. (Since the recent bushfires more resources have become available online - see the Blue Shield resources page)

It's pretty minimal at the moment, and in no way an official document, but I've placed a redacted copy online to help anyone putting together a similar plan against a deadline.

As the document evolves and changes I'll put the changes into the redacted version as well.

Please feel free to copy, download or modify the plan - the redacted version is a Google Document and can be downloaded here.

If you do find it useful, please let me know how you've used it and what changes you've made - you can find my contact details online if you don't know them already - in fact if you end up using the plan, send me a postcard from your museum/historic house.

2019 - what worked

For a long time I used to write a 'what worked' review of my personal use of technology in the preceding year.

Since I retired (almost exactly four years ago now) posting has been a little erratic with the last post being in August 2018, so I thought it was time for an update.

The success

I finally bought myself a new computer back in March and it's been an undoubted success, quick, fast, reliable - no comparison with my old Dell Inspiron

The not so much a success

My chromebook had gone end of life and my Macbook battery was seriously showing its age, so back in April I bought myself a second hand Yoga 11e. It's been a success as a second take to meetings laptop (yes I still go to meetings occasionally) but compared to my old Macbook Air it's still a bit on the heavy side. I don't regret the purchase, it's a useful versatile machine, but it's simply just not as light as my old Air, so much so that I've just paid out for a replacement battery for the air while I look for a sensible lightweight replacement

Hanging in there

The old Thinkpad X230 continues on as a useful workhorse, principally used for for family history work and for storing a second local copy of my work on Dow's Pharmacy. My coffee mediated disaster earlier this year taught me the value of having a second machine available as a backup.

While it's older, and is still Windows 7 based, the 500GB disk is big enough to store a local copy of  the family history database and associated documentation, as well as a second copy of the Dow's documentation.

Having a local copy has made it possible to get things done when I didn't have local access.

And of course coupling this with OneNote and Onedrive has of course meant that any new material saved locally is automatically uploaded as soon as the network becomes available.

This has also proved to be one of the problems with the Yoga - at 128GB its SSD is a little too small to comfortably store a copy of everything I've got on OneDrive.

My really old imac continues to be useful by dint of its large screen, which is ideal for viewing and transcribing scans of old documents, so while obviously there will come a time when it starts to suffer from software rot, everything is still at a recent enough version to be highly usable.

Not quite so useful

The iPad mini and keyboard combo I put together last year hasn't turned out to be quite as useful as I thought it would be.  I don't regret its purchase, its been incredibly useful on some occasions, but my work patterns have remained more computer focussed than I expected, hence my search for a light, versatile and cost effective windows machine. Currently it seems that you can have any two of these if you are on a budget, but not all three.

Other Changes

I've gone from using a Samsung Galaxy to an iPhone which has meant some changes in my work on the documentation project, mainly using a camera in place of the Samsung to take pictures of the artefacts.

Software and operating systems

Basically it's been Windows and the Microsoft ecology, with OS X and iOS playing aminor supporting role. While I still use Google docs for those living documents I have in docs, my use of the Google world has dropped right off since I made the change away from Android tablets and phones.

The other thing that's gone is my use of Linux. I always used to argue that software choice to a large extent dictates operating system choice and its true - my need for One Note and Evernote, not to mention OneDrive has forced me to abandon the use of linux in favour of windows.

Perversely, some of the genealogy software I use is designed to run on Linux, with the windows version being very much a follow on.

I've never got around to building a virtual machine to run it on - perhaps I should - to see how much better performance is. I have thought about converting my X230 to linux with the end of Windows 7 support next month, but the advantages of having a second machine to slot in are such I'll probably keep it windows until the end of the pharmacy documentation project.

That said I need to do something about getting rid of my old Inspiron. I havn't powered it up for about 9 months so I'm guessing that there's nothing on it I really need. Perhaps before I wipe it and take it to the eWaste disposal centre I'll stick linux on it and see how useful (or not) it is and how the genealogy software runs natively on an old machine ...

Joined the herd - bought an iPhone

For years I've been a Samsung Galaxy user, but I've finally cracked and got myself an iPhone.

A refurbished one admittedly, but a pretty recent one.

Not that I've never been an iPhone user - work used to give us iPhones as the corporate phone, but I always had my own phone as well, and that was also a Samsung - good battery life, decent screen, reasonable camera, reliable, and since I retired four years ago now my Galaxy's been my one and only phone - and it's been a really good productivity aid.

But after four and half years, it's beginning to show its age. It's still stuck on Android 6, and it needs a new battery - only twenty five bucks for a third party one admittedly -but I took that as a sign that it was time to change.

Now, where we live, we have to have Telstra.

Non negotiable.

Optus's coverage is ok over most of the town, but we live in a blackspot. Vodafone's not great either, but Telstra works, a bit on the edge but we get a usable signal.

This means that you can either buy a phone from Telstra, either straight up or on a plan, or else you buy an unlocked phone from someone else.

Now you can buy a reasonable midrange phone from Telstra for not a bad price, and if you wanted the convenience of a phone on a plan it's probably not too bad a deal, but you can probably do better buying from one of the online retailers.

And that's why I ended up with a 12 month old refurbished iPhone.

Most corporates have standardised on iPhones as a company phone creating a healthy market in good second hand and refurbished phones, but it's not the case with Samsung phones meaning you probably are going to end up buying new.

So on the day I did this I could get a 12 moth old refurbished iPhone 8 for $600 and a new Galaxy S10 for a $1000. As always your mileage will vary, and it could be that next week someone drops a whole lot of decent Galaxy's on the market.

Being a cheapskate, I went for the iPhone and Apple's well known build quality.

I'm not a heavy phone user, and I'll probably keep the phone for a few years. So build quality and long term reliability is important to me, other things less so.

As I'm no longer working, I hardly call anyone these days, and if I do, I use my Skype account half the time, and  I'm not a great user of mobile data.

In fact the principal things I use my phone for is

1. receiving verification codes via SMS
2. calling J to let her know where I've parked when picking her up
3. a little bit of twitter and email checking
4. checking the weather
5. looking at maps if I'm not sure where I am in Melbourne/Sydney/Brisbane

in fact just about any phone would do, but the iphone has a useful ecology round it of extra apps, bump covers, hands free holders for use in the car etc etc I thought I'd go for the iphone and Apple's build quality ...

[update 29 November 2019]

and I get Telstra's wifi calling service in the box - which given the sometimes spotty local cell coverage is a definite plus ...

Documenting artefacts - using a camera instead of a phone

The methodology I wrote up a couple of years ago has proven pretty robust - even coping with with me spilling coffee on my laptop - but the time has come to introduce a change.

Previously I've been using my Samsung Galaxy 5 to take images of artefacts. That phone is now about four and a half years old, and definitely due for replacement. What I havn't decided what I'm going to replace it with, some experiments with my iPad and general frustration at easily getting images out of Apple's walled garden has led me to suspect that if my next phone was an iPhone it wouldn't fit quite so well into the methodology.

However I have a Nikon CoolPix AW100 that was bought second hand on eBay for another project entirely that is as good as the Samsung's camera. Reasonable lens, reasonable zoom, and the added advantage of being designed for use in challenging environments it can be used comfortably while wearing nitrile gloves.

Getting images out of it is as easy as taking the SD card out and plugging the card into an external SD Card reader - external because my work laptop doesn't have an SD card slot, but does have a spare USB port - actually I use a little portable Belkin hub - and copying the files across and reviewing them.

The procedure works fine - the only downside is that the Nikon, like most cameras, uses a sequential naming system for images of the format DSCNnnnn rather than the Samsung's yyyymmddhhmmss format which has the advantage of making the date and order the image was taken immediately obvious.

The Nikon does record the date and time in the EXIF data (as well as the GPS information), but not using the date in the filename does mean recording the image name really has to be done at the time, or more accurately, during the review process, before the individual artefacts are entered into the documentation spreadsheet.

Otherwise, procedurally, the process is exactly the same and works well ...

The ipad as a research tool

It probably seems really strange, given it's 2019 and almost Hallowe'en to be writing about using the ipad as a research tool. but last week was the first time I'd used the second hand iPad I bought at the start of the year as an impromptu research tool.

I'd been down in Chiltern working on the documentation project, when I was chased out of the back room due to the unexpected arrival of a large tour group.

Now I'd been researching Karna Vita desiccated ox liver pills - it may not be your thing, but between the first and second world wars pills containing desiccated liver were very much a thing.

They were held to be really good for perking people up, getting extra iron into your system if you were a bit anaemic, and some doctors advised women to take them during pregnancy if they were feeling tired and worn out.

So, having been chased out I went down the road to the local library, which was (a) open and (b) had reasonable free wifi - in fact zippier wifi than last time to do my research.

But rather than drag my work laptop in, I simply took my iPad mini into the library.

Now I'm not a virgin as far as tablet and keyboard combo's for work are concerned - I've been through two seven inch android tablet and keyboard combos in the last seven years, most recently a cheap Alcatel tablet bought from Telstra's remainder shop.

It's around the same vintage as my iPad, and certainly feels as capable, even if the apple slickness is not there - you have to work at integration, but basically it does everything you would want to do:

• search for stuff of the web
• capture url's and web pages
• write notes about stuff
• export stuff to somewhere useful

absolutely no surprises there and equally so using the iPad - chrome basically is just chrome, and you can save stuff to evernote or one note as easily as anywhere else and apple's notes app certainly frees you from dependence on weird editors, of course it has restrictions - no markdown for example, and it only wants to talk to applications it knows about - and that revealed a little gotcha.

Now you might think, as I did, that if you save something from notes to Onenote, say, via Airdrop, it would save it to the cloud and then sync it back to local copies.

But no, what it actually seems to do is transfer data to the local instance of the application and then leave it to the application to do any synchronisation required.

Now you can see immediately why Apple implemented it this way - make it a universal on device service with a documented interface, and leave it up to the application implementers to work with the service and do any synchronisation required.

Fortunately it integrates with OneNote and with apple's email service meaning that content can be saved easily, or if you don't have the software installed use email to post content, as with Evernote.

Unfortunately, the email integration puts the content in the body of the message, rather than creating an attachment, which means that attachment based services such as sendtodropbox or Epsonconnect, are of little use to you.

And that's the rub. With the iPad, everything is slicker and easier than it is with the Android environment - but that's only providing your'e happy playing inside of Apple's walled garden ...