recentpopularlog in

ignatz : ops   11

AWS S3 Event Notifications have "probably once" delivery
you get the following

Messages that are delivered once;
Messages that are delivered multiple times;
Messages that are not delivered

This is in fact equivalent to "no guarantees at all" but the phrase "probably once" has a certain appeal to it. In my case I have an application that writes files to S3 at a regular interval. These files are processed by a lambda so they can be loaded into a database. This database is ultimately used in a customer facing application, so any duplicates gets noticed very quickly. Somehow I needed to come up with a way to deal with this pecuilar behavior of S3 Event Notifications.

Christ, what a mess. Sounds like S3 Event Notifications are best ignored for production use. Disappointing :(
aws  s3  event-notifications  consistency  durability  reliability  ops  via:jm 
18 days ago by ignatz
An update on Sunday’s service disruption | Google Cloud Blog
postmortem on google outage. note that once again it's a config error, which is how it happen now
gcp  google  odd  outages  post-mortems  networking  config  sysadmin  ops  via:jm 
10 weeks ago by ignatz
OVH suffer 24-hour outage (The Register)
Choice quotes:

‘At 6:48pm, Thursday, June 29, in Room 3 of the P19 datacenter, due to a crack on a soft plastic pipe in our water-cooling system, a coolant leak causes fluid to enter the system';
‘This process had been tested in principle but not at a 50,000-website scale’
postmortems  ovh  outages  liquid-cooling  datacenters  dr  disaster-recovery  ops  via:jm 
july 2017 by ignatz
Ubuntu on AWS gets serious performance boost with AWS-tuned kernel
interesting -- faster boots, CPU throttling resolved on t2.micros, other nice stuff
aws  ubuntu  ec2  kernel  linux  ops  via:jm 
april 2017 by ignatz
Instapaper Outage Cause & Recovery
Hard to see this as anything other than a pretty awful documentation fail by the AWS RDS service:
Without knowledge of the pre-April 2014 file size limit, it was difficult to foresee and prevent this issue. As far as we can tell, there’s no information in the RDS console in the form of monitoring, alerts or logging that would have let us know we were approaching the 2TB file size limit, or that we were subject to it in the first place. Even now, there’s nothing to indicate that our hosted database has a critical issue.
limits  aws  rds  databases  mysql  filesystems  ops  instapaper  risks  via:jm 
february 2017 by ignatz

Copy this bookmark:

to read