This blog includes joint work from the Cloud Operations team Adam Stegman and Vick Kelkar.
In the Cloud Operations team at Pivotal, we value documentation that is helpful and easy to discover. One method we use at Pivotal to ensure documentation is easy to discover is to make it visible within the normal workflow. What follows is one example of the value of in-context documentation, and how we used that idea to better manage our Elastic IPs (EIPs) in our Amazon Web Services (AWS) accounts.
One day in the fall of 2013, someone “helpfully” cleaned up several unassociated EIPs from our AWS EC2 account. In this particular case we recovered easily, but it highlighted a weak spot in our EIP management—there’s no way to tag, protect or otherwise share context about unassociated EIPs.
When we enabled the Cloud Foundry log drains service in our production environment we had to provide a set of EIPs that could be whitelisted for our log-ingest partners, like Splunk. These EIPs become the definitive list for groups inside and outside of Pivotal to use in their security policies. Out of the 12 EIPs we published, we only used 3. The rest are reserved for future expansion of services.
The extra EIPs are not associated to any instances, so when looking at the AWS console, they seem to be unused. This leaves us exposed to the “helpful” cleanup problem. We knew that if these EIPs were released, we would have a problem that could impact our customers as they grow and expand.
If You Release An EIP By Accident, You Will Not Get It Back
AWS does not allow us to recover an EIP that has been released. If the spare EIPs are released, we would not be able to move to one of the white-listed IPs we gave our our log-ingest parterns. In addition, it would make our partners’ systems vulnerable to third parties who may be able to allocate our published addresses after their release. To avoid this issue, we decided to annotate the IP addresses as a reminder of what these IP addresses are used for, and in this case they are reserved for future use.
Hacking the AWS Console for EIP Management
Amazon EC2 will allow you to “tag” a VM instance with metadata that indicates what it is or what it is used for. However, EIPs have no such feature. So, we came up with a hack to protect the EIPs and create a context around them for the future.
Amazon allows multiple EIPs to be assigned to an instance inside a Virtual Private Cloud (VPC). We use this feature to assure EIPs are not accidently released back to the Amazon pool. To do so, we created an instance with termination protection and assigned it the EIPs we need to keep. Then we named it “nat_pool_x see issue nnnnnnnn DONT remove”. Then, we stopped the instance to avoid paying for it. Anyone running across this oddly named VM can immediately get context on the issue, complete with the technical and business reasons for reserving the EIPs.
The Value of Discoverability in Context
With this “fix” firmly in place we are able to mitigate a messy risk in our production AWS environment. In the future, anyone looking at the EIPs in the AWS console will find clear and obvious bread crumbs back to our issue-tracking system. This in-context documentation is easily discoverable and requires no prior knowledge of the environment. Ensuring that our EIPs are safe-guarded for future expansion.
- Learn More About Pivotal Cloud Foundry
- How to Use Third-Party Log Management Services
- Read More Pivotal Cloud Foundry Blogs
About the AuthorMore Content by Adam Stegman