Thursday, 6 December 2007

How yahoo [does | will do] big file systems ...

hadoop in a word. Very involved in the development of the hadopp file system and I guess it's going to be their googlefs equivalent for content hosting.

I'm also guessing that flickr will be in the [does | will do] box as well.

Which is kind of interesting. And leveraging off opensource save some of your development costs as well.

And as a bit of non sequiter (ok I want to keep track of the url), there's an interesting tale from the NYT about hadoop, Amazon's cloud computing environment adn digital content management

1 comment:

Arthur said...

See also:

Amazon EC2 and S3 (I use these)
s3.amazonews.com

In particular, Hapdoop on S3 not hadoop-fs:

http://wiki.apache.org/lucene-hadoop/AmazonS3

Amazon's Dynamo:

http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf

It's a key-value system not a normal filesystem and only available internally. Still a good read.

The Ceph filesystem

http://lwn.net/Articles/258516/


Interestingly, only the last of these even tries to preserve posix semantics.