craigslist > about > open source

craigslist is a big believer in open source software, and relies heavily on Linux, MySQL, Perl, Apache, Sphinx, Redis, Haraka, and many others.

craigslist (CL) has released the following open source projects:

CL blob service - a simple HTTP-based, multi-master storage service designed for scale-out and multi-datacenter deployments:

  • multi-master architecture with no single point of failure
  • scales well for both read and write intensive workloads  
  • tracks TTLs for automatic expiration and purging of blobs   
  • designed for both local and multiple data center replication
  • pluggable index and storage interface (uses SQLite for index and filesystem for storage by default)

CL image service - large scale image resizing and processing HTTP service:

  • flexible API to choose any quality, size, and center-cropping
  • automatically handles orientation operations from EXIF data  
  • stores resized images in the blob service (could use other stores)

memcache cluster proxy (MCP) - a high-performance modular clustering HTTP reverse proxy:

  • high-performance event-driven daemon
  • flexible configuration, easily extensible framework
  • uses standard memcached memory store for caching   
  • supports PCRE and rule-based traffic routing by URL, headers, and more
  • tracks health of origin servers, routes around failures
  • provides configurable URL redirects, rewrites, and real-time header transformations

In addition, craigslist contributes code to the following open source projects:

Haraka:

  • added the smtp_client code that helps support pooled connections for proxy queues
  • rewrote all of the smtp_proxy and smtp_forward code
  • major contributions to the test suite
  • plugin that does strict DNS checking, DNS access control lists
  • contributions to bannering support
  • logging infrastructure changes for more robust logging
  • plugins for aliases, access lists, etc.

Redis:

  • perl-AnyEvent-Redis-Federated: an event-based redis client that implements client-side sharding in Perl
  • redis_util: s collection of utilities for node-redis
  • provided support for included config files

Sphinx:

  • provided the original persistent connections implementation
  • craigslist sponsors useful sphinx feature developments, such as the TRUNCATE INDEX command for real-time sphinx

The craigslist Charitable Fund supports open source nonprofit organizations including: