recentpopularlog in

ignatz : cloudflare   2

Details of the Cloudflare outage on July 2, 2019
Great writeup from jgc. Worth noting some important lessons:

* config changes should be rolled out carefully and gradually, just like code;

* particularly regexps, which are effectively code anyway;

* emergency-use rollback systems need to work, of course!;

* having emergency-only systems is a risk, too, since infrequently-used code paths are likely to atrophy and break without anyone noticing (as nsheridan said);

* /.*/ in a regexp is pretty much always bad news, and would have been worth a linter to catch before commit.
cloudflare  outages  regex  postmortems  regexps  deployment  rollback  via:jgc  via:jm 
8 weeks ago by ignatz

Copy this bookmark:





to read