Help! I can’t delete my Elastic Beanstalk environment…

In follow up to his last post on using the MATLAB Compiler Runtime with AWS Elastic Beanstalk, Lead Developer Phil Kendall now explains how he worked round some issues he had when deleting an Elastic Beanstalk environment.

So, you’ve had your nice AWS Elastic Beanstalk environment up and running for a while, but for whatever reason you’ve decided its now time to retire that environment. That should be nice and simple: just go to the AWS console and hit the “Terminate Environment” option. It starts chugging away, removing your EC2 instances and the like, but then suddenly it stops with an obtuse error message:

EnvironmentTerminationFailed

(The blacked out squares there are our AWS account number)

With a little bit of thought, you can probably work out that “:DeletionTest” refers to an EC2 security group in your environment, and if we go and look at the security group, we can see that it does in fact contain a reference to the security group we’re trying to delete:

SecurityGroupReference

Removing that reference helps, but we’re still stuck with those “030008803268:rds.ec2sg.765575” and “644125515248:elmo.ec2sg.446077” references. What’s going on there?

The simpler of the two is probably “030008803268:rds.ec2sg.765575″: the clue’s in the name – it’s an RDS security group (note for the potentially confused: RDS security groups are/were a concept which applied only to the older EC2-Classic platform; if your AWS account is relatively recent, you can only use the newer EC2-VPC platform and so won’t ever see an RDS security group – but if you’re reading this post it may well be because you’ve got an RDS security group somewhere!)

However, the console isn’t going to give you any sort of clue as to which RDS security group it is. If you’ve got only a few security groups, you may be able to eyeball them and work out which one contains the reference, but if not you’re probably going to have to pull out the API. The example here is in PowerShell, but you should fairly easily be able to convert it to whatever your weapon of choice is:

Get-RDSDBSecurityGroup | Where-Object { ($_.EC2SecurityGroups | Select-Object -ExpandProperty EC2SecurityGroupName) -contains "awseb-e-mvuaz2etah-stack-AWSEBSecurityGroup-L5YQ4RLDJATB" }

In our case, that finds us the RDS security group quite nicely, and we can again remove the reference to “awseb-e-mvuaz2etah-stack-AWSEBSecurityGroup-L5YQ4RLDJATB”.

Unfortunately, the “644125515248:elmo.ec2sg.446077” reference doesn’t give us nearly such a nice hint as to which service that’s coming from. However, with a hat tip to Craig Watcham, we get the hint that it might be an ElastiCache security group. We can again deploy the API to find out exactly which group it is:

Get-ECCacheSecurityGroup | Where-Object { ($_.EC2SecurityGroups | Select-Object -ExpandProperty EC2SecurityGroupName) -contains "awseb-e-mvuaz2etah-stack-AWSEBSecurityGroup-L5YQ4RLDJATB" }

…and then delete the reference.

After all that, we can hit “Terminate Environment” again and, with a following wind, your environment should go away.

For completeness, if you do want to use the API to find which EC2 security groups reference another one, there’s a couple of commands to run (because the references can be on either the ingress or egress rules):

Get-EC2SecurityGroup | Where-Object { ($_.IpPermissions | Select-Object -ExpandProperty UserIdGroupPairs | Select-Object -ExpandProperty GroupName) -contains "awseb-e-mvuaz2etah-stack-AWSEBSecurityGroup-L5YQ4RLDJATB" }
Get-EC2SecurityGroup | Where-Object { ($_.IpPermissionsEgress | Select-Object -ExpandProperty UserIdGroupPairs | Select-Object -ExpandProperty GroupName) -contains "awseb-e-mvuaz2etah-stack-AWSEBSecurityGroup-L5YQ4RLDJATB" }

As always, any comments, improvements or such like are very welcome.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s