Friday, September 5, 2008

Great open source scalability tools from Danga

If you are trying to build a scalable LAMP service, it's always best to start with the original and still quite relevant presentation, from Brad Fitzpatrick when he was at LiveJournal. You can find the 2005 version here. You'll learn how they pioneered the use of a lot of open source tools at new levels of scale, and even created quite a few more, that are essential scaling aids. Three of my favorite:

  • memcached - an in-memory object caching system. For tips on how to integrate it into your database and application layers, you can see the tail-end of my JIT Scalability talk.
  • MogileFS - "Distributed (meta) file system. Spray files across cheap disks on your network. Pay less for storage. No proprietary on-disk file formats." Excellent if you happen to be building a site that, for example, receives and stores gigabytes of photo uploads on a regular basis.
  • Perlbal - reverse proxy written in Perl. At IMVU, we used its relatively straightforward plugin system to implement a simple long-poll solution for chat.
If I was building something from scratch today, I'd probably use Gearman for asynchronous task processing, too.

Thanks, Danga (and everyone else who's contributed to these projects).

1 comment:

  1. UnknownOctober 13, 2008 at 9:44 PM

    I'd also recommend Cal Henderson's (flickr) presentations (and book), e.g. http://www.slideshare.net/iamcal/scalable-web-architectures-common-patterns-approaches-presentation/

    And - of course - going through http://highscalability.com/ and reading up on the various different approaches different companies have (successfully) taken towards getting scalability at reasonably low cost.

    (sadly, this blog forces you to use a google account. Yuck. So, for reference, I am Adam, of http://t-machine.org . Death to the demon single-sign-on-single-identity madness!)

    ReplyDelete

Subscribe to: Post Comments (Atom)