recentpopularlog in

ignatz : distributed   4

Heroku finds out that distributed queueing is hard
Stage 3 of the Rap Genius/Heroku blog drama. Summary (as far as I can tell): Heroku gave up on a fully-synchronised load-balancing setup ("intelligent routing"), since it didn't scale, in favour of randomised queue selection; they didn't sufficiently inform their customers, and metrics and docs were not updated to make this change public; the pessimal case became pretty damn pessimal; a customer eventually noticed and complained publicly, creating a public shit-storm.

Comments: 1. this is why you monitor real HTTP request latency (scroll down for crazy graphs!). 2. include 90/99 percentiles to catch the "tail" of poorly-performing requests. 3. Load balancers are hard.

http://aphyr.com/posts/277-timelike-a-network-simulator has more info on the intricacies of distributed load balancing -- worth a read.
heroku  rap-genius  via:hn  networking  distcomp  distributed  load-balancing  ip  queueing  percentiles  monitoring  via:jm 
february 2013 by ignatz
Quake-Catcher Network
distributed network of computers with motion sensors, detecting quakes
earthquakes  distributed  geology  science  measurement 
december 2008 by ignatz
irrepressible.info
Put a badge on your website that publishes random bits of censored material from all over the world. Neato
censorship  internet  widgets  webtoys  politics  oppression  distributed 
august 2007 by ignatz

Copy this bookmark:





to read