Diego, Cloud Foundry’s new runtime, is shaping up to be a versatile tool for a variety of applications beyond its original purpose as the next-generation Cloud Foundry runtime. Lattice, for instance, incorporates Diego to enable deployment of containerized applications across a cluster of machines. In its Vagrant virtual machine form, Lattice is very useful for experimenting with Diego’s features on your local machine without the overhead of deploying Cloud Foundry.
One-off tasks have been difficult to incorporate into an application running on Cloud Foundry for some time (see the hoops you need to jump through to get Rails migrations to run, for example). To solve this problem, Diego’s design incorporates Tasks as first-class citizens. Tasks aren’t yet first-class citizens in Lattice, as the support in the ltc command line isn’t ready. However, you may use Diego’s Receptor API for the purpose of scheduling Tasks.
How To Fail Like Me
As part of some research into this new Tasks capability, I attempted to schedule a simple Ruby app as a Task. I soon ran into a ‘Permission denied’ exception when Bundler tried to install gems. I could reproduce it like this:
- Make a Dockerfile that copies an app into a directory and runs ‘bundle install’.
- Build the Docker image and push it to Docker Hub.
- Make a JSON task payload to send to Diego’s REST API. Set the ‘rootfs’ property to the URL of the image pushed in the previous step. Set the ‘action>run>path’ property to a script that also runs ‘bundle install’.
- Start watching the logs for the GUID specified in the JSON:
ltc logs task1 - Send the task to Diego via its REST API using curl:
curl -X POST http://192.168.11.11:8888/v1/tasks -d@task-payload.json - The Permission denied exception appears in the watched logs.
You can see an example of this on GitHub.
I’d made a silly mistake: bundling once in the Dockerfile and again in the script. But bundling twice isn’t a million miles away from a real-world requirement: it might be expedient to run Tasks on a root filesystem with a base set of gems, and refresh those gems when a task is run, just in case bug fixes are available.
Solving The Problem
The Dockerfile in this example creates everything as the root user. It’s not good practice to run Bundler as root, and Bundler will tell you this much if you build the Dockerfile yourself:
bundleinstall$ docker build .
Sending build context to Docker daemon 121.3 kB
Sending build context to Docker daemon
Step 0 : FROM ruby:2.2.1-wheezy
---> 89a773167e29
Step 1 : ADD . /app
---> 54f76e9e60d4
Removing intermediate container 8e9651e8672a
Step 2 : WORKDIR /app
---> Running in 52a9c2a29812
---> 4b00258de138
Removing intermediate container 52a9c2a29812
Step 3 : RUN bundle install
---> Running in 2f9a2fb270e1
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will break this application for all non-root users on this machine.
The Gemfile specifies no dependencies
Resolving dependencies...
Bundle complete! 0 Gemfile dependencies, 1 gem now installed.
Bundled gems are installed into /usr/local/bundle.
(and so on)
Fixing this problem is straightforward in our case. Docker images specified as a rootfs of an LRP or Task are, by default, run as an unprivileged user called ‘vcap’ (an artifact of our VMware heritage: VMware Cloud Application Platform). Therefore, if your Docker image contains files owned by root and your executable expects to be able to modify those files, as running Bundler will do, you’ll run into permission denied errors unless you’ve explicitly set those files’ permissions appropriately.
So, how would we set permissions appropriately in order to prime our gems in the Docker image, allowing a later bundle install on the resulting container to refresh them? The answer is, for the time being, to allow write access to the app directory for all users, and to run Bundler as an unprivileged user inside the Dockerfile when installing gems for the first time.
You can see the modified Dockerfile in the fixed branch.
Note that the USER instruction will not have an effect on the running container, but only on the built image. As explained in the issue encountered by another user, Garden (the container management technology we’re developing) currently ignores USER metadata set in Dockerfiles, and always runs the container as ‘vcap’.
Further Cloud Foundry On Docker Reading
- The Road to Persistent Data Services on Cloud Foundry Diego
- New Cloud Foundry Service Broker Updates
- Deploying Cloud Foundry Microservices
About the Author
