Further thoughts on eresearch services

I’ve been watching the eresearch services thread from the eresearch 2013 conference. I’m beginning to regret not having gone - it looks to have been more interesting than I expected, but that’s life.
A lot of people seem to be getting interested in eresearch services. I’ve already expressed my opinions about such things, but generally a heightened level of interest is probably a good thing.
However there seems to be a bit of conflation going on, so here’s my $0.02:

• eresearch is not the same as big data - big data refers to the handling of very big data sets and their analysis using inferential analyses, eresearch refers to the application of computational and numerical techniques to research data
• eresearch is not the same as digital humanities - digital humanities really refers to the move to using electronic resources in the humanities - this move may enable the application of eresearch techniques to humanities research
• astronomers, physicists, economists and many more have been using inferential analyses such as cluster analysis for many years - eresearch is not new, but its spread and penetration is
• the rise of cheap (cloud) computing and cheap storage are key drivers in the adoption of eresearch by allowing bigger datasets to be handled more easily and cheaply

In short, you can do perfectly good eresearch with an old laptop and an internet connection, you don’t need all the gee-whizzy stuff, all you need is a problem, the desire to solve it, and a little bit of programming knowledge.
Any eresearch service is going to be there to support such work, by faciliating access and providing advice and support to researchers, in fact it’s taking on the role of research support that belonged to computing services in the days before the commodification of computing and the rise of the internet when in the main computing meant time on a timesharing system to run some analytical software …

Written with StackEdit.

Data Science for business [book review]

Data Science for Business: What you need to know about data mining and data-analytic thinking Foster Provost and Tom Fawcett O'Reilly Media  Data science is the new best thing, but like Aristotle’s elephant people study to define  exactly what data science is and what the skills required are. When we see data science we tend to recognise what it is, that mixture  of analysis, inference and logic  that pulls information out of numbers, be it social  network analysis, plotting interest in a topic over time, or predicting the impact of the  weather on supermarket stock levels. This book serves as an introduction to the topic. It’s designed for use as a  college textbook and perhaps  aimed at business management courses. It starts at a very  low level, assuming little or no knowledge of statistics or of any of the more advanced  techniques such as cluster analysis or topic modelling. If all you ever do is read the first two chapters you’ll come away with enough  high level knowledge to fluff your way through a job interview as long as you’re  not expected to get your hands dirty. Chapter three and things get a bit more rigorous. The book noticably changes  gear and takes you through some fairly advanced mathematics, discussing  regression, cluster analysis and the overfitting  of mathematical models, all of  which are handled fairly well It’s difficult to know where this book sits. The first two chapters are most  definitely ‘fluffy’, the remainder demand some knowledge of probability theory  and statistics of the reader, plus an ability not to be scared by equations embedded  in the text. It’s a good book, it’s a useful book. It probably asks too much to be ideal for the  general reader or even the non numerate graduate, I’d position it more as an  introduction to data analysis for beginning researchers and statisticians more than  anything else, rather than as a backgrounder on data science. [originally written for LibraryThing]

What does an eresearch service look like ?

There has been a lot of discussion about eresearch and eresearch services. However when you try and pin down what constitutes an eresearch service it seems to be all things to all people.

In an effort to try and find some consensu I did a very simple survey. I typed 'eresearch services' into Google and chose pages from Australian universities. I've tabulated the results of this fairly unscientific survey in a google spreadsheet.

Each institution of course described the service on offer differently, so the spreadsheet is purely my interpretation of the information available on the web.

There are some clear trends - all sites offer help with

• storage
• access to compute/virtual machines
• cloud services
• collaboration (which includes data transfer and video conferencing)

Other services tend to be more idiosyncratic, perhaps reflecting the strengths of individual institutions. However it's clear that a lot of the effort revolves around facilitation.

My personal view is that we do not try to second guess researchers. Instead of prescribing we facilitate by helping researchers get on with the business of research.

This is based on my experience. Over the course of the data commons project we fielded a number of out of band questions such as

• Access to storage for work in progress data
• Data management and the use of external services like dropbox and skydrive
• Access to a bare vm to run an application or service
• Starting a blog to chronicle a project
• What is the best tablet for a field trip
• How can I publish my data on the web

which suggests what researchers want is advice and someone to help them do what they want to do - a single point of contact.

Provision of a single point of contact hides any internal organisational complexity from the researcher, it becomes the contact’s problem to manage access to services and not the researcher’s.

There are of course other views - for example this presentation from eresearch 2013 but I think we can agree that what researchers want is easy access to services and a small amount of support and help ...

Recovering data serially

Over the past few weeks I've noticed a number of posts along the lines of

we've an old XYZ machine without a network connection, can anyone help with recovering data from it?

Not having an ethernet connection is a problem, but assuming that the machine still powers ups and the disk spins, it might not be so much of a problem.

The key is to go looking to see if it has terminal application. This isn't as odd a question as a lot of computers were used to access timesharing systems back then in these pre web days, and a terminal application was fairly standard.

The good thing about terminal applications back then is that they usually incorporated a serial file transfer protocol such as xmodem, ymodem, zmodem or kermit. Of these kermit is perhaps the best, not the least because it can be put into server mode and you can push files from your host in batches.

The good news is that both lrzsz, the ?modem client for linux and ckermit are available for install on ubuntu from the repositories via apt-get.

Then all you need is a usb to 9 pin serial adapter cable and a serial nine pin null modem cable - both avaiable from ebay for a few dollars and then you should be ble to transfer data from your old machine to the new.

Yo will of course need to set up things like parity and baudrate, and it might be an idea to practice transfering data first by setting up a second linux machine and transferring data between the two - see [http://www.stlinux.com/install/getting-started/serial](http://www.stlinux.com/install/getting-started/serial for an exaple.

Despite this sounding a bit of black art, it's actually quite easy. The other good thing is that a number of embedded communications devices are still configured over a serial port, so most network technicians still know something about debugging serial connections.

Once you have a managed to establish a working connection you should then be able to get the serial communications software on your problematical machine to talk to your newly enabled serial host.

From there it's simply a matter of transferring the files across one by one and converting them to something usable - if they're wordprocessor files, LibreOffice can read most of the legacy formats and web based services like cloudconvert and zamzar can read many more ...

Written with StackEdit.

Archiving, persistence and robots.txt

Web archiving is a hazardous business. Content gets created, content gets deleted, content gets changed every minute of every day. There's basically so much content you can't hope to archive it all.

Also a lot of web archiving assume that the pages are static, even if they've been generated from a script - pure on the fly pages have no chance of being archived.

However you can usually make an assumption that if something was a static web page and there long enough, that it will be on the wayback machine in some form.

Not necessarily it turns out. I recently wanted to look at some content I'd written years ago. I didn't have the original source, but I did have the url and I did remember searching successfully for the same content on the wayback machine some years ago. (I even had a screenshot as proof that my memory wasn't playing tricks).

So, you would think it would be easy. Nope. Access is denied because the wayback machine honours the sites current robots.txt file, not the one current at the time of the snapshot, meaning that if your favouriet site changes its robots.txt between then and now to deny access you are locked out.

Now there's a lot of reasons why they've enacted the policy they have but it effectively locks away content that was once public, and that doesn't seem quite right ...

Written with StackEdit.

Bufferapp

If you're someone who follows my twitter stream you may have noticed that I seem to post bursts of tweets around the same time every day.

This because I've taken to using Bufferapp to stage some of my tweets. Basically bufferapp is a little application that integrates nicely with Chrome and AddThis which allows you to put tweets into a buffer to be reposted later in the day.

I only use the free version, which means that my buffer is only 10 deep, but that seems to cover most of the tweets I'm likely to make in a day. I'm not obsessive compulsive about twitter, no matter what it seems like.

Why use it?

One could imagine lots of scenarios including making it look as if one was online when one wasn't but my reasons are a little different. Basically I tweet about two topics - geeky stuff to do with computing and data storage, and equally geeky stuff about history and archaeology. There is of course an overlap, for example, big digitisation projects and computational text analysis do provide a degree of overlap but in the main there are two topic groups and two topic audiences. (I had the the same thing with my blogs, which is why I split them - this one is more technically focussed, while the other one is a bit more discursive and random)

When I look at my twitter followers I can say very roughly that the computing and data people are in the same or adjacent timezones to me, but the people interested in the geeky history stuff are clustered in North America and Western Europe - of course that's not quite true, I have followers in South Africa and Chile to name but two, but it's a good enough approximation.

In other words the history followers tend to be between eight and eighteen time zones away from me on the east coast of Australia, and hence unlikely to be awake when I'm tweeting (well except for Chile and the west coast of America where there's a few hours of overlap).

So I've taken to using bufferapp to delay the tweets for that audience, which has the effect of de cluttering the feed for the computing and data people.

I'm still tweaking the schedule and I'm conscious (because some of my followers have said so) that some of both communities like a leavening of the other sort of information so it's not a hard split, and of course there's always the paper.li daily summary of the most popular tweets from both me and the people I follow ...

Written with StackEdit.

Usenet, VM's and Pan

Like most people who were in at the beginnings of the internet as something widespread (I'll say sometime around 1991 when JANET connected to the Internet and abandoned Coloured Books for TCP/IP for me) Usenet News filled the niche taken nowadays by twitter and blog feeds.

Usenet news fell apart and lost popularity in the main due to it being hijacked by trolls and other malefactors with the result that people walked away from it when the signal to noise ratio got too high.

In fact I closed down work's usenet news server a few years ago. It was quite an interesting experience as we had a couple of downstream servers elsewhere that we provided a feed to under an SLA. Finding someone at the downstream sites who could remember what a usenet news server was and why we should agree to terminate the SLA (and the feed) was a task in itself. People really don't use it anymore.

However, despite that there's still a couple of technical newsgroups I still find useful, especially now the trolls have abandoned it for twitter and facebook, making the experience kind of like the old days.

To access them I use pan running on a minimal crunchbang linux vm.

This of course has the problem of getting the information out of pan and into somewhere useful - having that useful post sitting on a vm you run up once a week isn't really that useful.

There's lots of ways of solving that problem, but I didn't want to spend a lot of time installing extra software such as dropbox on the vm. My answer is incredibly simple and incredibly old school - install alpine on the vm, set up a dummy account on outlook.com, manually attach the usenet posts as text file and email them to my evernote account, my work account, or where ever suits.

Remarkably old school, but remarkably efficient ...

Written with StackEdit.