Preserving spreadsheets

Spreadsheets are used in lots of ways in research, and that means that we need to think about their preservation as part of the long term preservation of data.

And this is actually more complicated than it sounds - as demonstrated by a recent post on preserving Google Spreadsheets.

The best preservation practice really comes down to how the spreadsheet was used.

If we are using it passively, ie as a way of recording data in the way that I’m doing so on the Dow’s pharmacy project, export as comma separated, tab separated etc, is the way to go, and also circumvents the Year 1900 problem in excel. Basically you just get the characters and that’s all you want.

And this is great for survey data, botanical field data, archaeological data and the rest - a true lowest common denominator format.

And that’s a very good thing as if you have any pre-1900 dates in your spreadsheet exporting from Excel to Libre Office calc on the basis that calc’s .ods format is open, and non proprietary can cause problems.

And that’s the problem with spreadsheets, if there’s any calculation you need to ensure that the exported version correctly reproduces both the calculations and the results, which is a complicated problem.

It would probably be simple to start with a product that uses an open format - such as Gnumeric or Libre Office calc and then export the document to Google Drive, Dropbox or OneDrive for sharing rather than start with an online spreadsheet - and if you need to start with an online spreadsheet, Microsoft’s online version of Excel might be a better departure point due to it’s compatibility with the stand alone version of Excel giving a better chance of conversion to an archival format ...

Gnumeric ...

At various times I've said in my hand waving way that you could as easily use Gnumeric as a spreadsheet for recording data as use excel or a more heavy weight alternative  product such as Libre Office Calc.

However I've got to confess I've never actually used it for real work.

That may be about to change as my corporate supplied copy of excel has started wingeing about product activation failures. I'm sure it's just an expired licence key, and not being on the corporate network it can't see the licence server to update.

The only problem is that as a volunteer archivist I can't contact corporate IT support directly, my boss has to log the job, and just by chance she's overseas at the moment.

It's not a drama - for the moment everything seems to work, I can still create and save data, but just in case there's a grace period that's about to expire on me I installed the latest commonly available windows version of Gnumeric this morning so I can keep working if I get locked out.

If I end up using it in anger I'll post on my experiences ...

[update 20/12/2017]

Well, everything kept on running until my boss got back from overseas, and even though my first attempt to contact IT support was stymied by a power outage, I did get on to them, and with a combination of remote access via TeamViewer and my being a meat puppet when we required we got a new product key despite not being on the corporate network.

But having spruiked Gnumeric, I guess I should come up with a little side data collection project to see how well it works out ...

Fun with a legacy wireless bridge ...

A long time ago, more or less ten years ago, I bought myself a Linksys WET54G wireless bridge which let me connect an old mac (running linux) and a couple of home made linux servers cobbled out of scrap machines that I had in the garage to our home network.

The setup was fairly simple, linux boxes, a $20 white box unmanaged switch and the wreless bridge. Performance was fairly impressive given that the garage was built of corrugated iron and did a nice imitation of a faraday cage. Putting the bridge next the sole window gave me a reasonable signal.

Fast forward to 2017.

I no longer have any homemade servers - they died horribly in a flood, and I now live in a wooden house nicely lined with metallized sarking - hello Faraday age.

We also have a studio, which is a separate building, and is in fact a converted garage, and is lined with guess what ? metallized sarking.

The net result is that getting a network signal in the studio is a big ask. I bought one of these no name $15 repeaters, which managed to get a decent signal onto the back deck and a weak but stable one into the studio.

Machines are usable with the current signal  but I wanted to move my old imac into the studio and set up a second desk in there for a book scanning project I have in mind which would involve shoving some large files about.

Now the linksys is quite good with weak signals so I thought I could use it to get a better signal and then use an old wireless router to drive a local network, or indeed a local wired network.

I still had the bridge, but of course no configuration manual, but about twenty minutes with google told me all I needed to know. A little bit of network jiggery pokery and I could both see the home network and the wireless repeater and get a better signal than by relying on my old imac’s hardware alone.

I could connect, but not really. The linksys doesn’t support WPA2 even though you can run WPA with AES encryption, which mean that to authenticate I’d have to lower security on my home network. The linksys lets you apparently reauthenticate but actually fails silently. I had the same problem with my old Asus Internet Radio, which is why it’s now plugged into the wired network at home.

Wireless bridges of course need a wireless connection.

During testing I even managed to fool myself into thinking that I’d got it to work - I hadn’t, after changing the encryption from TKIP to AES I’d forgotten to turn off wireless networking on my laptop after rebooting it for testing, but that wasted an hour while I worked out I’d been an idiot, rather than having broken something.

So, basically the Linksys is useless, or more or less useless. A hunt for firmware updates that support wpa2 drew a blank. Still I had fun playing with hardware for the first time in years, so the time wasn't wasted, even if I did spend almost a day playing with it.

I’ve now admitted defeat and ordered myself a second no name whitebox wifi repeater. The studio has a decently large glass door and the home repeater for the back deck is next the door so hopefully I can daisy chain the two ...

In praise of Linux (again)

A few days ago there was an article in the Irish Times praising linux on the desktop for its utility and ability to extend the life of old and otherwise perfectly usable hardware.

I am in fact writing this on my five nearly six year old Linux netbook.

Why?

Windows updates. Ever since I had the Windows 10 creators update installed I've had a storm of minor fixes and updates, all off which seem to leave my machine in an odd state requiring not only a reboot but a fifteen minute session of placatory messages while Windows plays with itself.

That said I actually quite like Windows 10 as an environment and am quite happy with the fact that when I eventually replace my elderly Dell Inspiron it'll be with a Windows machine.

However, I can't help but contrast the paind I'm going through with Windows at the moment with the ease at which I ran my latest set of Linux of updates it was a fairly painless exercise.

What's more I even installed a suite of optical character recognition software. Think about it - running OCR software on a six year old Intel Atom powered machine.

That said my first attempt, with OCRfeeder, which I'd successfully used with Debian to OCR a collection Vietnam war era newspaper cuttings from North Vietnam didn't quite work - basically OCRfeeder and Xfce seem to have an incompatibility. Changing to Yagf which uses the same underlying recognition engines, tesseract and cuneiform - seemed to work.

Preliminary, and fairly basic tests, seem to show that it works, if a little slowly, but good enough for some of J's family history stuff where we have some good jpegs of documents.

And that of course is the other great virtue of Linux - there's always more than one way of solving a problem or carrying out a particular task.

Now I'm not going to tell you that Linux is a panacea. It's not. Sometimes it's flexibility is a curse more than a blessing - for example I have never ever been able to get bluetooth to work with Xfce despite having it work successfully with other Linux front ends.

I am not going to tell you to throw out your Macs and your windows machines. My MacBook Air for example remains one of the best machines I have ever owned for travelling and note taking in the field - the only machine that ever came close was the Linux EeePc 701SD. But what I will say that if you need a low cost and effective solution try Linux.

Zpad six and a bit years on

Six and a bit years ago I bought myself a zPad, a no name Chinese android 2.2 tablet skinned to look like an iPad.

It was bought as an experiment at a time when iPads seemed to be taking over the world to see if cheap whitebox Android devices could mount a challenge, and provide an alternative tablet based solution.

Ipads are of course still dominant but Samsung, Lenovo and the others have turned Android into a viable alternative platform for tablet computing. What hasn't happened is that cheap whitebox devices have taken over the world - most Android tablet sales are for brand name devices, most of which are both cheap and offer reasonable performance.

Enough history - back to the zPad.

Amazingly I'm still using it (occasionally) six and a bit years on.

The operating system is hopelessly out of date, upgrades just don't happen anymore but gMail and twitter still work, as does a weather app, and for that reason it continues to live on a shelf in my shed so that I can check the weather and my email with I'm covered in dirt after a serious gardening session.

Surprised (a) that I still use it, and that (b) it's still proving useful ...

Technology and travel

Usually when we've had a trip overseas I do a little blog post on how our technology worked.

This time I have almost nothing to report. I took my MacBook Air and J took her trusty samsung tablet and both worked well, and we had no connectivity problems, basically everywhere we went or stayed there was free fast wifi.

Our only problem was that we didn't have data roaming on our phones, but then there was enough free wifi around to not really need it. Next time we'll probably take an old unlocked smartphone and buy a local sim just for the convenience but it's by no means essential.

We did take the old Nokia phone with a travel sim that we'd used in 2015 and that again performed excellently as far as calls and texts went, and having a UK number people were happy to call it given that roaming charges in Europe are now a thing of the past.

As always I took an Australian powerboard and a pair of adaptors - a UK pattern one for the UK and Singapore, and a European one for Portugal.

Hardly anywhere has these so called universal sockets, and while UK and European plugs fit fine I've yet to find one where an Australian plug fits well - either too tight or too loose, never right.

I'd bought myself an SD Card reader for my MacBook for the princely sum of $3, and that worked well, meaning we had no problem backing up our cameras.

As in 2015, I took a GPS with me but this time both hire cars we had came with an inbuilt GPS and there were no silly extra charges to use them - I'd guess they'll be standard on hire cars next time we travel to Europe.

It's actually getting easier ...

4G routers

I've previously written about our adventures with 3g routers, first to provide backup for our (then) flaky home ADSL connection, and also to provide us with a portable network solution when we go bush.

Well, the need for a backup connection for our home network link is long gone, but we still use our little portable 3g network box quite extensively when we go travelling in Australia.

While some country motels offer decent quality wifi some don't, or if they do still charge silly money, and quite a few holiday cottages, unlike in Europe, come without a wifi connection.

Latterly, we've been buying 10GB of data with 365days expiry as the best offer around.

When we're away we average around 300MB a day, but we perhaps use this for 10 or 15 days year, ie while we've 10GB to play with we roughly use half.

As our use of the service is bursty, ie we might use it for three or four days in a row and then not at all for a month or so, having a long 'use it or lose it' period allowed us plenty of headroom without excess data charges.

Well, long story short, I went to renew our data service for the 3G unit, and, well, our provider no longer offered year long data packs, it was all by the month (actually 28 days to be exact) and the cheapest offer was for 1.5GB/28days for $15, ie a 5 day trip away with our average usage would just fit inside our allowance (excess data charges are still a thing), and yet we'd be paying twice as much as before for dead data.

So, first thought was to change providers. Unfortunately no one really does this any more - no one offers a long expiry prepaid broadband service, or if they do, it's not long for this world, and no one offers anything reasonable say 2.5GB/28days at an economic price.

Now just by chance, we'd just changed our phones from Virgin to Telstra, and Telstra had not only given us a silly monthly data allowance (15GB each) they'd put us in a family pool so we were sharing 30GB.

This meant that the cheapest option was to buy the cheapest data service (1GB/month) that Telstra allows you to add to a family pool, and that way we would end up with more than enough headroom.

Which is what we did.

Now as our portable 3G box is unlocked I could have simply replaced the SIM and left it at that, but given that 4G is widely available I decided to get us a 4G router so we could have faster network speeds where possible (Telstra really does have the fastest and most pervasive rural network).

Simplest solution was to go hunting on ebay and buy an unlocked Telstra MF910V 4G router made by ZTE - this being the previous model of the unit Telstra now sell.

I went for the older unit as it was a bit cheaper, and having an unlocked unit means we can always put a different SIM in it, like if we were overseas. Being about the size and weight of an iPhone 4 the unit's highly portable, and with an internal battery it can be used (for a limited time at least) somewhere without power - camping, or on the wifi-less V/line train from Wangaratta to the city.

I did some rough tests yesterday when the unit arrived and performance looks good - how well it performs in remote areas will have to wait until our next trip away ...