Michael G. Noll

Applied Research. Big Data. Distributed Systems. Open Source.

Building a Scalable Collaborative Web Filter With Free and Open Source Software

My paper “Building a Scalable Collaborative Web Filter with Free and Open Source Software” has been accepted for publication and presentation at this year’s IEEE International Conference on Signal-Image Technology & Internet-based Systems (SITIS) which will be held in Bali, Indonesia, from November 30 - December 03, 2008. IEEE SITIS 2008 Conference Logo


In this case study, we describe the design and architecture of a scalable collaborative web filtering service, TaggyBear, which is powered by free and open source software. We will introduce the reader to the ideas and concepts behind TaggyBear, and discuss why we picked the software components that form the basis of the service. We will talk about how we combined or extended their functionality to build the TaggyBear service, and provide some initial benchmarking results and performance figures. Note for fellow developers: The described system components include Pylons, MySQL, Tokyo Cabinet/Tokyo Tyrant, Pylog (custom app based on Twisted), Hadoop, and on the client side a Firefox Add-On.

Full Paper & Presentation

Related Links