Rank of Automating Amazon EBS snapshot managements

Brittany Doncaster, Solutions Architect

Business continuity is important to create critical workloads for AWS. As an AWS client, you can set recovery point objectives (RPOs) and recovery time goals (RTOs) for different level applications in your organization. Once the RPO and RTO requirements are defined, it is up to your architects to determine how to meet these requirements.

You are probably storing persistent data in Amazon's EBS volumes, which live in a single availability area. And by following best practices, you take snapshots of your EBS volumes to back up the data on Amazon S3, which offers a sustainability of 11.9. If you follow these best practices, you have probably recognized the need to manage the number of snapshots that you keep for a particular EBS volume and remove old and unnecessary snapshots. Doing this cleaning saves on storage costs.

Some customers also have policies indicating that backups should be stored at a certain number of miles as part of a disaster recovery (DR) plan. To meet these requirements, customers copy their EBS snapshots into the DR region. Then, the same management and cleaning of the snapshots must also be done in the DR region.

All of this snapshot management logic consists of different components. You must first tag your snapshots so that you can manage them. Then, determine how many snapshots you currently have for a particular EBS volume and evaluate that value against a retention rule. If the number of snapshots was greater than your retention value, you should clean up the old snapshots. And finally, you can copy the last snapshot on your DR region. All of these steps are just one example of a simple workflow for snapshot management. But how to automate something like this in AWS? How do you do it without servers

Videos 1.2 AWS training in chennai

One of the most powerful AWS services released in 2016 was Amazon CloudWatch Events. It allows you to build event automation based on events in your AWS infrastructure. CloudWatch Events integrates with AWS Lambda to allow you to run your custom code when one of these events occurs. However, the actions to be taken as a function of these events are not always composed of a single Lambda function. Instead, your business logic can consist of several steps (as in the example of the snapshot management flow example described earlier). And you may want to perform these steps in sequence or in parallel. You can also retry logic or exception handling for each step.

AWS Step Functions serves only this purpose: to help you coordinate your functions and microservices. The step functions allow you to simplify your efforts and extract the Lambda code from error handling, retry and workflow logic. The step functions integrate with Lambda to provide a mechanism for creating complex serverless applications. Now, you can run a Step Functions state machine based on a CloudWatch event.

In this publication, I discuss how you can target step functions in a CloudWatch event rule. This allows you to have instant snapshot management based on the completion events of the snapshot triggering in the CloudWatch event rules.

As an example of what you could do with Step Functions and CloudWatch Events, we have developed a reference architecture that manages your EBS snapshots.

Automation of EBS snapshot management with step functions

This architecture assumes that you have already configured CloudWatch Events to create snapshots on a schedule or use other ways to create snapshots as needed.

This architecture covers the workflow elements that must occur after creating a snapshot.

It creates a CloudWatch event rule to call a state machine execution of step functions when an EBS snapshot is created.

The state machine then records the snapshot, cleans the oldest snapshots if the number of snapshots is greater than the number defined to hold and copies the snapshot into a DR region.