The Domesday Book as structured data

Happened across this rather nice searchable version of the Domesday book from the University of Hull.

It's nice on two counts, firstly the presentation and look of the site and secondly through the way that the site developers have taken the text of the Domesday book, marked it up and created a searchable resource.

This of course only works because the Domesday book is an implicitly structured document - when the clerks went out they always asked the same questions and wrote the same things down, so that each entry had more or less the same format and organisation, making it then possible for someone 900 years later to take the information and turn it into a database.

Being able to turn pre Information Technology era records into structured data is not unique to the Domesday book, there are plenty of other potential sources from Mughal landholding ad tax records in India, through the New Statistical Account of Scotland to Poorhouse Famine and Parish records in Ireland, which are equally useful and could be used to tell a story as data.

Telling stories as data can be very powerful. The Domesday book for example records the rentable value of a parish in 1066 and in 1086. If one looks at the entry for Fangfoss you can see that the value has dropped massively - the same being true of just about every else around.

One could guess that this could be a reflection of the continuing impact of the Harrying of the North some sixteen years earlier. Certainly if one has a not very scientific look at parishes around Reading - for example Rotherfield Peppard one sees that the values of rents are the same or perhaps a little more.
With this data it would be quite a simple task to take the 1086 to 1066 rent data and plot it on a map to show how widespread this effect is - for example we would expect there to be less of an impact in areas such as Cheshire and Shropshire where the Harrying was less intense, and consequently a more normal ratio for the rent figures than the extreme case of Fangfoss ...

You've got webmail

As I've mentioned before we've changed over to Office 365 at the day job, and this has shown up an interesting little problem - before people would know they've got mail because they would have a minimized client in the system tray, and the client would provide a visual cue by changing its icon, ringing a bell, or some such event.

With webmail you don't get any of that. You have to have the email tab open in the browser on your desktop.

The same problem applies whether or not you use 365 directly as Microsoft intended, or, like me, funnel your email to another webmail service.

Essentially, what you need is a mail notifier, something like xbiff, that runs in the system tray and alerts the user.

This is especially useful for people who spend a lot to their day inside of an editor, be it a document or a code editor.

Basically there are two approaches - either use an existing notification engine such as growl or else write a small stand alone client - for example something like those listed at http://www.emailclients.net/email-notifiers.htm.

Either way the logic would be quite simple - using a minimal pop (or imap) implementation poll the server separately and get the new message count, and when it increments display a visual cue.

While you could develop it as a browser application it is probably better done as a platform specific system tray application - after all not everyone has a browser open all of teh time ...

Aboriginal contacts with the outside world and Kilwa coins

It was once a commonly held view that the Aboriginal peoples of Australia had little or no contact with the outside world. We now know that not to be the case with the Yolngu having substantial trading contacts with the Macassan from Sulawesi.

Given these, it wouldn't be in the least surprising if some Dutch East India company coins turned up somewhere on the northern coats of Australia, either as a result of shipwreck or indirect contact via a trading vessel from what is now Indonesia.

There is news today of some coins doing exactly that. The interesting thing is that they appear to have been found alongside some coins from the Kilwa sultanate in East Africa.

This is in fact not new news, more rediscovered news. The coins have been known for some time and are in the PowerHouse Museum in Sydney.

The PowerHouse Museum acquisition record suggests that the Kilwa and the VoC coins represent two separate deposits, perhaps as a result of two shipwrecks separated by a couple of centuries.

Given however that the actual details of the find and their discovery are vague, there remains the possibility that they are a single deposit

This might seem surprising, but not very. Somalis have been crewing dhows across the Indian ocean for millenia. And Somali sailors are known to have  worked on British vessels in the nineteenth century, establishing migrant communities in Cardiff and Liverpool.

There's no reason why some Somali sailors did not sign on to work on a Dutch ship - and we might even be able to find records to that effect. The date of 1690 for the VoC coins gives us the earliest possible date for the coins ending up in Australia, but given that coins are fairly indestructable,  they could have ended up in Australia a hundred years later.

The same with the Kilwa coins - they could have been loose change, but given there age that seems unlikley, but they could have been used as counters in a game, or simply kept for sentimental value. The fact that the only other Kilwa coin to have been found outside of East Africa was found in Oman points to them being carried along the spice route.

We can spend all day waving our hands and speculating. We simply don't know. Like Roman style pottery in Bali, they'll probably remain one of these little tantalising enigmas of history.

But, if it is a shipwreck, and other evidence is found to date it more securely, we could probably trawl the VoC records to see if there is evidence of Somalis or other East Africans crewing ships ...

Living in the noughties ...

The internet went out last night at home.

Seriously out, not even syncing, let alone a connect to our providers network. Let's just say it neatly demonstrated one of the inherent problems with a Chromebook - great though they may be, offline is a problem if you havn't synced your work - and yes the documents I wanted were in Dropbox and Google docs.

So, out with the windows netbook and the 3G modem - the reason for choosing the netbook was (a) it had the 3G software already installed and (b) I could sit on the sofa and work,

No our house is on the edge of a slope and we have had problems with cellphone reception. These are mostly gone now, since they improved the infrastructure, but the data is a bit wibbly - like you have to choose a location with decent signal strength for a reasonably fast connect.

Dropbox wasn't too bad. Niether was Evernote. Synchronisation was not stellar but fast enough to download what I needed, do my work and then sync stuff back. Definitely old school but it worked.

Trying to use Google Docs over a slow connection was interesting (it turned out I'd never installed the Google drive synchronisation software on the netbook and now didn't seem the time to start). It did work but it was more than easy to type ahead and suddenly have things catch up, and spreadsheets were a bit like playing wack-a-mole, but I got there.

What this did teach me is that the old style way of working where you use local as opposed to web apps remains the best approach when working on flaky connections. It reminded me of the slightly odd experience of using WordPerfect on Microvax back in the nineties - most times it was ok but sometimes it was unresponsive and more like a game of 'guess how many backspaces' I need - and this was in an operating system where they tweaked the kernel parameters to up the priority of the text input routines.

However, while the interactive stuff was bad, the sync based stuff worked well - Evernote is good, as you do all your stuff and sync, and dropbox is similarly good. Likewise using a local email client, or a local editor to blog and then post is a lot better.

There are some implications to this. In the last quarter in the UK tablets outsold pcs and laptops combined. A lot of tablet apps tend to be quite verbose in their traffic and the heavy weight ones tend to offload work to the cloud - just as something like a chromebook does.

This of course means that they consume a lot of bandwidth. Now while our home ADSL link is less than wonderful, it works most of the time, and when it works it is reasonably fast. Imagine for a moment I was running on a slow connection, like last night's wireless broadband. A lot of the tablet based stuff I do probably wouldn't really have been possible.

A lot of the services would be almost unusable due to lack of responsiveness - not lack of connectivity.
For whatever reason just about everything seems to want to connect these days, and that's why we need  decent bandwidth end to end.

It's telling that the only way I could find out about the outage was from our internet providers website - and one that so slow to load I couldn't confirm it was a known outage until I came into work this morning.

Now, I'm not grizzling (well maybe just a little), but if we expect that people will use online services for most of daily life, they do need to have access to decent connectivity and decent bandwidth ...

The day job ...

has kind of been getting in the way of posting recently.

So much end of project documentation, not to mention the mildly painful exercise of auditing the project accounts. We are now almost done and in anticipaton of this we've started tentatively to get the message out there.


We have a  piece in this month’s ANU IT newsletter – as well as a piece in the previous month’s newsletter  - it's towards the end of the pdf so you will need to scroll down quite a bit. Besides that, we’ve implemented support for Digital Object Identifiers in our publication workflow, meaning it is possible to request a DOI for your data set if it is saved in the ANU Data Commons.

My colleague, Rahul Khanna, and myself recently gave a short web based presentation on our implementation of DOIs.  The presentation, with accompanying sound track, is available on the ANDS channel on YouTube if you are interested in following up on our use of DOIs.

All our source code is available via our project landing page. We're just about done with the documentation for our metadata stores project to tie dataset publication to grant activity and person records - it you're interested in that there should be a link on the project landing page early next week.

Chromebooks and usefulness

Having done a little bit of reading around the various Chromebook specifications it looks as if they might make a decent Eee substitute.

The only major downside is the lack of skype. Microsoft have recently announced a skype plugin, which might provide an escape route.

The use case is a trifle messy here, because quite often I use Skype to call phones and send text messages when I'm away as well as the basic Skype to Skype communications, so the skype plugin will have to have a pretty complex range of capabilities.

Until then the Skype and Chromebook saga is pretty dismal with messaging being the most you can manage, unless you have a suitable alternative device to hand - begging the question as to why you would bother with a chromebook.

However the good thing is that Chromebooks do seem to support Eduroam. This might seem pretty esoteric but, given I work in a university being able to access the network when I'm at meetings at other campuses is a definite advantage over the Eee. This was one of my drivers for going for the seven inch tablet as a note taker as Android also has good Eduroam support.

Also, our main campus network uses an identical authentication mechanism to that used in Eduroam, which means that if a chrome book supports eduroam, it will support our main campus network.
I couldn't find any Australian universities with web pages on how to use a chromebooks with eduroam, but there are plenty UK examples such as this one from Sheffield which look entirely plausible when compared with our Linux connection guide.

And I do know from direct personal experience that you can use Australian eduroam settings overseas without having to worry about version incompatibilities.
This would seem to combine to make the Chromebook a pretty sensible travel computer, especially given its statelessnes, and by extension the ease of getting information off of it.

Maybe I will crack an buy one ...