Thursday, 6 December 2007

How yahoo [does | will do] big file systems ...

hadoop in a word. Very involved in the development of the hadopp file system and I guess it's going to be their googlefs equivalent for content hosting.

I'm also guessing that flickr will be in the [does | will do] box as well.

Which is kind of interesting. And leveraging off opensource save some of your development costs as well.

And as a bit of non sequiter (ok I want to keep track of the url), there's an interesting tale from the NYT about hadoop, Amazon's cloud computing environment adn digital content management

1 comment:

Anonymous said...

See also:

Amazon EC2 and S3 (I use these)

In particular, Hapdoop on S3 not hadoop-fs:

Amazon's Dynamo:

It's a key-value system not a normal filesystem and only available internally. Still a good read.

The Ceph filesystem

Interestingly, only the last of these even tries to preserve posix semantics.