Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Based on my unscientific poll of friends, one of the least used and most overlooked features of AWS is CloudWatch. Not only can CloudWatch be used to monitor the availability of your AWS services, but it can also be used as anomaly an detection tool. Did I mention that these feature are free?
Since the holy grail of DevOps is to bring the security and the operations teams together to meet the same goals in an automated fashion; Cloud Watch is a perfect DevOps tool. In fact, once it’s implemented for your environment you are one step closer to the exalted state of SecDevOps.
If you aren’t familiar with AWS CloudWatch then head on over to https://aws.amazon.com/cloudwatch/. As a quick introduction, here’s how AWS website describes the service:
“Amazon CloudWatch provides monitoring for AWS cloud resources and the applications customers run on AWS. Developers and system administrators can use it to collect and track metrics, gain insight, and react immediately to keep their applications and businesses running smoothly. Amazon CloudWatch monitors AWS resources such as Amazon EC2 and Amazon RDS DB instances, and can also monitor custom metrics generated by a customer’s applications and services. With Amazon CloudWatch, you gain system-wide visibility into resource utilization, application performance, and operational health.”
Let’s assume that both security and Ops teams need to be aware when weird things are happening with their servers and applications. Events outside the norm (aka weird things) can be either be related to security or operational events; either way they may merit investigation. Here’s a simple CloudWatch monitoring process you can use to give you a heads up on events that could indicate someone is trying to exfiltrate data from your systems.
Opening Scene
Think about what kind of information attackers are after. They want your intellectual property and your customer data because it can be converted to cold, hard cash. Next, consider the architecture of your systems and where that data is stored. Most likely your treasures are stored in databases or Git repositories.
Guess what? Attackers know that so that’s exactly what they are looking for. Take a few minutes and think like an attacker bent on exfiltrating your data. Obviously, the first thing you should do is carefully monitor your egress traffic for suspicious activity. In addition, look for anomalies in traffic flows – they can also help determine if suspicious events are happening. Amazon CloudWatch can help you do both of these things.(Note: This process assumes you haven’t already been breached.)
1. Enable CloudWatch
2. Establish a baseline
Review the historical data on a daily, weekly and monthly basis. Soon you will start to see a trend in your egress bandwidth usage. Start with eyeballing the min, max and average usage over time. Put these in your favorite tracking tool (text file, spreadsheet, abacus, whatever).
3. Configure alerts
When you feel comfortable with your baseline, configure your first alert to send an email if traffic usage goes over the baseline. Consider how “chatty” you want this event to be. For example, alerting over the average baseline will often generate more alerts than alerts over maximum usage. This is a tricky balance and you’ll need to make trade offs between missing problems and the time spent responding to potential false positives.
4. Respond
However you choose to respond is up to you. I suggest you start with automatically opening a ticket in your tracking system so that someone will be sure to review the alert and make notes on their findings.
5. Tune as needed
Make a plan to review the alerts and the data on a regular basis. Architecture and other changes to the system are going to change the baseline and the alerts may need to be reconfigured.
Building and automating security into DevOps is a requirement that everyone should be working toward. It’s a theme I’m going to be returning to in future posts because it’s really important.
AWS Cloudwatch is a free tool that you can help you get started automating security right now, so what are you waiting for?
