Based on code from Oren Solomianik’s ec2-delete-old-snapshots ec2-manage-snapshots does things a little differently.
Oren’s script would delete any snapshots older than n days for (a) given volume(s). I improved the code a little bit so as to handle a –region parameter ( –region eu-west-1 is an example) an –noop so as to tell us what it would do but not actually do it. That’s the ec2-delete-old-snapshots you’ll find in the included archive.
[ad#Amazon Cloud inline]
Because I wanted to manage snapshots differently I heavily modified the original script to create ec2-manage-snapshots so that for a given volumes, it would keep snapshots made:
- in the last 7 days
- the past 4 sundays
- every first day of the month
and erase the rest.
Its behavior can be easily modified but I wanted to make sure I kept monthly snapshots, last 4 weekly & last 7 days worth. It assumes you’re doing one daily snapshot of each volume.
Download here: ec2-manage-snapshots
Failsafe
Like its ancestor, the program checks to see that each vol-id entered on the command line has at least one newer snapshot than the deletion date, to prevent deletion of all snapshots of a certain volume. The deletion process will commence for a volume only if such a snapshot was found. More importantly, you can use the --noop command line option or the constant in the code (NOOP) and the script will tell you what it would have deleted but won’t actually delete anything.
Disclaimer: This program is in development. Although it has been tested and worked on production environments, it can’t be guaranteed to perform without unexpected results. Use at your own risk.
This is great. What license is is released under?
I’ll put it under Apache License as soon as I have the time. Consider it so for now. The original had no license info nor did its google code site.
Hi Erik,
Thanks for these improvements! I really neglected the development of this script…
If you would like to commit your changes, please send me an email with your Google code user and I’ll add you to the team there.
Oren
On my system, I was getting a fatal error:
Fatal error: Class ‘XsltProcessor’ not found
This was on Ubuntu 9.10 desktop. It was because I didn’t have the package php5-xsl installed. To get around this:
sudo apt-get install php5-xsl
Then I re-ran the script and all was working.
Great, thanks for the insight on php5-xsl package requirement.
It’s also worth noting that on most systems running PHP prior to 5.3, it is not possible to use long options, so the –noop and –region will not work. It’s actually quite dangerous for –noop to fail unexpectedly, for obvious reasons! I saw this error:
Warning: getopt(): No support for long options in this build
I would recommend updating to PHP 5.3, but we’re using Drupal 6 which is not compatible with it yet.
As a workaround, I suggest adding short versions of each option, so at least people have the choice.
Good point, I’ll soon release a version that supports short options for region and noop
I found Oren’s script a little too cumbersome to install and run (you have to download a bunch of PHP libraries and it works with time period not number of backups to keep!) so I’ve created a simpler alternative here if anyone is interested:
http://www.sambastream.com/blogs/dgildeh/12-03-10/implementing-revolving-backups-aws-ec2
Hope this helps!
Well, Oren’s script main goal is to delete the last n days. Yours deletes n backups. The script I added (based on Oren) sees things more like backups and always keeps the monthlies, a month worth of weeklies and the last seven days of dailies. I guess it depends on what you need.
As for the library requirements, I have to say I didn’t look too much into it since my home system and my servers seemed to have all that it needed. Your mileage (did) may varies.
Hi Cloud Walker,
I noticed one issue with the script… It will fail if the snapshot is part of an EBS AMI snapshot. The issue with 5.3 compatibility could probably be overcome by using Zend Framework’s Zend_Console_Getopt class. I write a lot of PHP and could help really spruce this up!
BTW, thanks for the post.
Nick, feel free to do so. Make changes to http://github.com/edasque/ec2-manage-snapshots ? Does using the Zend getOpt class mean bringing in a slew of dependencies though?
Hey Cloud Walker! Sorry for the long delay. I am going to take a look at this today to see if there’s anything I can help contribute.
Howdy Again,
I spent the last 8 hours really putting some polish on this script. Here’s a link to the project files. (http://www.upsellnetwork.com/ec2-manage-snapshots.export.tar.gz)
Some of the improvements are:
=========================================
* OO Redesign of the logic
* Accounts for more than one backup a day
* Uses Amazon’s newest API
* Handles snapshots that are in use by AMI images
* access key and secret access key can now be included as command line arguments
* configuration has been moved outside of the script, so there should be no editing there.
* added validation for amazon regions with graceful exit when a bad region is supplied
* added quiet mode
Right now the entire Zend Framework is included under the library directory, but I bet I can weed that down.
When unzipping the script, you will have to create a file called config.ini under the /etc directory. There is a config.ini-dist to be used as an example. To see all of the options, just run ec2-manage-snapshots.php –help
Let me know what you guys think!!!!
Best,
Nick
Posted the code on GoogleCode http://code.google.com/p/ec2-prune-snapshots/
Great work but I would have rather you put it on github since we already have a project there. Also, I’d rather we keep the same name otherwise there will be confusion?
I haven’t really played with GitHub before. I’ll take a look in a few hours and try to integrate what I’ve done with your project. If that all works, I’ll just close the project at Google. Were you able to run the script?
Hi Erik,
I just forked your project and uploaded the files. You can see the project here now. http://github.com/NicholasMiller/ec2-manage-snapshots
Awesome, I’ll take a look very soon
Thanks Nick, looks great. I am wondering if you could add a script to create the snapshots. The guys over at http://tiger-fish.com/blog/automated-ebs-backup-solution-using-amazon-ec2 have done this but I just thought it would be nice to have one package that could do it all and yours seems like the most robust one out there.
Pingback: Manage EBS snapshots with a python script
Pingback: Making the move from Centos 5 to Ubuntu Server | Simon Morley | Digital Savage
I am not a PHP developer and want to convert your script to C#. Before I take the time to learn PHP for the conversion, I want to make sure that your script does not prune the original snapshot. I am under the belief that pruning the original snapshot will invalidate the subsequent snapshots. Please let me know.
“I am under the belief that pruning the original snapshot will invalidate the subsequent snapshots”. That’s actually not true. AWS behind the scenes always keeps enough information to be able to restore a snapshot, no matter what other snapshots still exist.
I believe you and I am trying to find the post that made me think I had to keep the original… no luck. Anyway, it is easy to prove with some testing. Thanks for your help and your script.
This one didn’t want to play ball for some reason so I wrote my own based on yours but using the AWS PHP SDK.
https://github.com/jdelaune/AWS-EC2-Snapshot-Management
I’m using Nick Millers fork of this. I’m running an Ubuntu 10.04 server with PHP 5.3
When I run this command, this is all I get. I can’t tell if it’s working or not.
PHP 5.3.2-1ubuntu4.7 with Suhosin-Patch (cli) (built: Jan 12 2011 18:36:08)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
I have only started creating snapshots over the last couple of days, so maybe it doesn’t have anything to delete.
Any tips appreciated.
Yes, we always keep the last 7 days worth of snapshots if I recall correctly
I found that I had a problem before in the script and now I’ve figured out that I wasn’t passing — args in the command. This is what I’m getting now.
$ sudo /usr/bin/php -f /opt/scripts/bin/ec2-manage-snapshots.php — args -v vol-9cc356e5
There was an error running /opt/scripts/bin/ec2-manage-snapshots.php
Error Message: The provided “vol-9cc356e5″ volume does not exist
I’m 100% sure that the volume with that volume id does exist. Any ideas?
if I run ec2-describe-volumes –region eu-west-1, I see vol-9cc356e5 in the list of my volumes.
I couldn’t figure this out, so I went back to the script here -> http://www.sambastream.com/blogs/dgildeh/12-03-10/implementing-revolving-backups-aws-ec2
You have a bug when using a different region than US.
Although you give a choice to select a region in the config file, it is never passed to the Zend EC2 classes.
This meant they were always talking to the US webservice and I never got any results back
When I tried changing the bin/ec2-manage-snapshots.php file to include this:
Zend_Service_Amazon_Ec2_Ebs::setRegion(‘eu-west-1′);
It worked
very helpful and much appreciated tool for cleaning out our 600+ snapshots!
Hi,
Great script and very useful to me. I’ve forked and modified it to handle specifying snapshots using tags. This is because of the way I manage my volumes. I have a single instance which is automatically rebuilt by Chef it if fails. To do this it starts new volumes from existing snapshots. This means that volume ids don’t work but tagging the snapshots does.
The commit is: https://github.com/ejsarge/ec2-purge-snapshots/commit/3d942a92cc4363d7f7a2d0d0bcdf8ed12580e289
I’ve added a pull request to your queue. Feel free to ask me for whatever licence you need.
Note that I had to refactor the main code in order to do this so the commit looks scarier than it is.
Note that I’m a Ruby and Git newbie so my apologies if I’ve screwed something up.
Thanks for the hard work.
Cheers,
Edward
Pingback: Amazon EC2 Backup: Purging old snapshots | TrailHunger.com Technical Blog