A guide for snapshotting your MySQL data on EBS volumes.
You have data. You don't want to lose your data, do you? You probably take MySQL dumps, but they can take hours to rebuild. What would you do if you lost your main database and somehow your slave...
View ArticlePrune EBS Snapshots with ec2-expire-snapshots
In a previous post, I discussed how to snapshot your EBS volumes for disaster recovery and for starting slaves. If you're using that than great, you're being proactive. Here's the problem though: your...
View ArticleAWS::S3::Errors::AccessDenied when calling bucket.objects.with_prefix
While tightening up our security policies for S3 I encountered an error on deletion which we did not expect. The Problem Our policy looked like this: { "Version": "2012-10-17", "Statement": [ {...
View ArticleCodePen Outage - Mar 20, 2015
CodePen suffered an outage today and the service was unavailable for about 20 minutes from 7:10 am to 7:40am PST. After this time, we were intermittently flaky until 9:30am PST. Here's a quick rundown...
View Articles3-backed docker private registry on AWS
You use Docker, but you're not open-source, so putting your containers on the docker registry is not a bright idea. If you want to host your own private registry like we did, follow these...
View ArticleClik here to view.
apparmor aa-logprof hangs, maxes out cpu
Apparmor has always been finicky to get running for me. This is just the latest in the saga. Running Ubuntu Trusty, and following instructions for creating an apparmor nginx profile, I ran into an...
View Articledocker deployment without downtime
Problem You have a container serving traffic on port 80 via an exposed port. You build a new image and want to launch it as a new container. But, you don't want any downtime while the cutover takes...
View Articleinstalling solr on tomcat7
Why Because you can't index text by hand, now can you? Also, I always reference my private gist when I rebuild boxes, so I figured I'd put it out there for the world. Finally, no mortal remembers how...
View ArticleAWS CLI Script to Assign a Secondary IP
In high stress admin situations, you don't want to have to click around inside the AWS console to move a VIP from one instance to the other. Instead, you should automate! Here's a quick script which...
View ArticleClik here to view.
Assign a floating (secondary) IP address in AWS VPC
What is a Virtual IP? A Virtual IP (VIP) is a networking pattern which allows an administrator to quickly move and IP address from one server to another with virtually no downtime. Also known as a...
View ArticleRemove non-reporting servers from New Relic using the API
During our latest network cutover, we were left with lots of servers orphaned in New Relic because we turned them off in AWS. New Relic calls these types of servers non-reporting. I call these kinds...
View ArticleClik here to view.
Diagnosing disk contention on AWS with iotop and iostat
Finding the bottlenecks in your application can be tricky. Here's a story of how I used iotop and iostat to help build evidence to choose an ebs-optimized disk to solve a saturated disk problem....
View Articleback up gitlab configuration to s3
The gitlab backup instructions are pretty clear on how to how to back up your entire gitlab repo to S3 on a schedule. However, that backup does not include your secrets or configuration. This post...
View ArticleAWS::S3::Errors::AccessDenied when calling bucket.objects.with_prefix
While tightening up our security policies for S3 I encountered an error on deletion which we did not expect. The Problem Our policy looked like this: { "Version": "2012-10-17", "Statement": [ {...
View ArticleCodePen Outage - Mar 20, 2015
CodePen suffered an outage today and the service was unavailable for about 20 minutes from 7:10 am to 7:40am PST. After this time, we were intermittently flaky until 9:30am PST. Here's a quick rundown...
View Articles3-backed docker private registry on AWS
You use Docker, but you're not open-source, so putting your containers on the docker registry is not a bright idea. If you want to host your own private registry like we did, follow these...
View ArticleClik here to view.
apparmor aa-logprof hangs, maxes out cpu
Apparmor has always been finicky to get running for me. This is just the latest in the saga. Running Ubuntu Trusty, and following instructions for creating an apparmor nginx profile, I ran into an...
View Articledocker deployment without downtime
Problem You have a container serving traffic on port 80 via an exposed port. You build a new image and want to launch it as a new container. But, you don't want any downtime while the cutover takes...
View Articleinstalling solr on tomcat7
Why Because you can't index text by hand, now can you? Also, I always reference my private gist when I rebuild boxes, so I figured I'd put it out there for the world. Finally, no mortal remembers how...
View Article
