Archive for December, 2009

Hadoop reduce method reuses objects

Monday, December 28th, 2009

Over the holiday break I was playing around with creating a Map/Reduce job that would scan through all of the content items and then create a link graph. It was a fairly straightforward job. I would scan each content item for all of the hrefs and for each one would ...

Hadoop and HBase in production

Monday, December 28th, 2009

At the beginning of December, I ported several portions of the ReadPath infrastructure from MySQL to HBase with Hadoop. So far it's been a complete success. At first I had focused most of my attention on HBase, seeing it as a way to scale systems beyond a single MySQL instance without ...