Concourse is a simple-yet-flexible open source CI system which is maintained and developed by a team of engineers at Pivotal. For the Pivotal Cloud Foundry team in Toronto, which focuses on mobile services, Concourse was a useful upgrade from GoCD, our previous CI/CD system. The flexibility of Concourse enables us to continuously test and deploy our Pivotal Cloud Foundry Services, and with minimal setup, add a Mac Mini worker for on-device mobile testing.
Manually Provisioning An OS X Worker
To begin running mobile tests on a Concourse pipeline, you first need to set up a worker on an OS X machine. For our purposes, we’ve set this worker up on a Mac Mini with a USB hub to expand the number of devices that can be used for running tests in parallel. The Concourse Docs describe how to manually provision workers on custom hardware without using BOSH. This process involves installing and setting up Houdini, a no-op Garden backend for OS X and Windows, and a SSH forward between Houdini and your Concourse cluster.
Keeping It All Running Smoothly
Once you’ve installed everything, and tested running Houdini and the SSH tunnel from a user shell, there are a few steps to follow to keep everything running smoothly. Since the houdini and ssh processes aren’t managed by BOSH, we need to daemonize these tasks and restart them if they happen to crash or lose connection to Concourse. We’ve opted to create Bash scripts which run them, and add them as agents using OS X’s launchctl command to keep them running.
Houdini Script
Because OS X launch agents don’t run in a user shell, we need to define some environment variables for Houdini to launch both XCodebuild and Android gradlew tests, while retaining a sane PATH setup. To eliminate a lot of the guesswork, while including all of the environment variables required to run XCode’s build tools from a Houdini container1, you can echo the `env` of a logged in user’s shell to a ‘.houdini_profile’ ( eg. env > .houdini_profile ) and source that file before starting Houdini.
run-houdini.sh
#!/bin/sh
source /Users/pivotal/concourse_worker/.houdini_profile
houdini -depot=/Users/pivotal/concourse_worker/containers
Next we want to write the .plist file to be loaded by launchd, which will keep the script alive and run it at startup. I won’t go into too much detail about OS X Launch Daemons, as there are resources available on the Mac Developer Library which describe what each field does. In this case, we placed all of our plist files in the `/System/Library/LaunchAgents` folder to ensure they run at startup. Be sure to set the ‘Label’ key to something which is uniquely identifiable, and will be clear to anyone else managing the machine.
io.pivotal.concourse.houdini.plist
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>AbandonProcessGroup</key> <true/> <key>KeepAlive</key> <true/> <key>Label</key> <string>io.pivotal.concourse.houdini</string> <key>Nice</key> <integer>0</integer> <key>ProgramArguments</key> <array> <string>/Users/pivotal/concourse_worker/run-houdini.sh</string> </array> <key>RunAtLoad</key> <true/> </dict> </plist>
Advertising Worker To Concourse
Most of the process of forwarding a local Garden server is covered in the Concourse Docs, but writing Bash scripts and managing them as LaunchAgent .plist files makes these commands more robust by restarting them on boot, and whenever they crash. When run from a user shell, the SSH forward command will lose connection and stop running if there is any network disruption, so it is a good idea to run this command as a LaunchAgent as well.
The worker.json file is used to advertise your worker’s details over the SSH tunnel to the Concourse TSA. We’ve set our Mac Mini up as a ‘darwin’ platform worker with tags listing the supported devices.
worker.json
{"platform":"darwin","tags":[‘ios’, ‘android’],"resource_types":[]}
The next step is to create a Bash script which will run the ssh forward-worker command to tunnel the Houdini server to the Concourse TSA.
#!/bin/sh
ssh -vvv -p 2222 your.concourse-uri.com
-i /Users/pivotal/concourse_worker/mac_mini_worker_key
-R 0.0.0.0:0:127.0.0.1:7777
forward-worker < /Users/pivotal/concourse_worker/worker.json
forward-worker.sh
The .plist for running this script is the same deal as it was for Houdini, just be sure to set a unique filename, label, and point to the forward-worker.sh script.
io.pivotal.concourse.forward-worker.plist
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>AbandonProcessGroup</key> <true/> <key>KeepAlive</key> <true/> <key>Label</key> <string>io.pivotal.concourse.forward-worker</string> <key>Nice</key> <integer>0</integer> <key>ProgramArguments</key> <array> <string>/Users/pivotal/concourse_worker/forward-worker.sh</string> </array> <key>RunAtLoad</key> <true/> </dict> </plist>
Pooling iOS Test Devices
For our on-device iOS app testing, we use ios-deploy to install the app, and Calabash-ios to run the tests. We have multiple iOS devices connected to the Mac Mini, and want to ensure that devices can only be used by one job at a time. Concourse’s pool resource allows us to model locks on the devices, and integrate them with our Concourse jobs to avoid messing around with them mid-test.
.
├── ios-devices
│ ├── claimed
│ └── unclaimed
│ ├── a0819
│ └── a1370
├── pool-party.gif
Inside the device pool repo we created plaintext files for each iOS device attached to the Mac Mini, which contain details required to target them for testing. We also have a GIF of a bear jumping into a pool because it’s funny.
Our iOS device lock files contain the following parameters:
DEVICE_UDID | the device’s UDID found in the Devices window of Xcode |
DEVICE_IP | the device’s IP address internal to your network or, preferably, the devicename.local (this is the phone’s name with spaces converted to dashes and apostrophes removed) |
PROFILE_ID | the provisioning profile ID, found by selecting the provisioning profile in xcode and then selecting “other” (must be installed on the worker) |
DEVICE_NAME | the device’s name ( optional ) |
Running Tests on Device
Now we’re ready to bring everything together in a script which will be run by our Concourse job. We include a `run-tests` script which contains the commands to build the app and run tests on device in a repo along with all of our CalaBash tests.
Here’s a brief rundown of our `run-tests` script:
Once we’ve sourced the metadata from the iOS pool and done some test data setup, we run xcodebuild and pass in the PROFILE_ID parameter for the device we’ve acquired a lock on:
xcodebuild [...] clean build PROVISIONING_PROFILE=$PROFILE_ID
Once we’ve built the app for that device’s profile, we can use ios-deploy to install the app onto the device using the –id flag.
ios-deploy --bundle $(dirname $0)/app_name.app --id $DEVICE_UDID
Finally, to run the tests on the device, we call cucumber with a few flags for the target device and its endpoint, along with the bundle id of the app.
BUNDLE_ID=io.pivotal.io.app_name DEVICE_TARGET=$DEVICE_UDID DEVICE_ENDPOINT=http://$DEVICE_IP:37265 cucumber
Here’s a trimmed down example of how our job uses the ios-devices pool resource to run the tests on device:
- name: ios-sample-test plan: - put: ios-devices params: acquire: true - put: env resource: acceptance-env params: acquire: true - get: ios-app-tests trigger: true - get: ios-app-package trigger: true - task: test-on-device config: platform: darwin run: path: /bin/Bash args: ["-c","source ios-devices/metadata && env/metadata && cd ./ios-app-tests/scripts/run-tests"] inputs: - name: ios-app-package - name: ios-devices ensure: aggregate: - put: ios-devices params: release: ios-devices - put: acceptance-env params: release: env
Now that we’ve got a pipeline task setup to run our tests on device, lets kick off a build:
Because we’ve set up all of the iOS code repos as triggers, the job will run any time a new build of the code is created.
When this task runs, Concourse will:
- Acquire a lock on an available ‘device’ metadata file out of the ios-device pool
- Acquire a lock on an available acceptance environment where the backend for the app is deployed
- Download our app package from S3
- Clone the acceptance tests from Github
- Source the metadata about the device and test environment
- Run the `run-test` script on the Mac Mini on the device against our test environment
- Release the locks on the claimed device and environment
Here’s a look at a device picked out of the pool by the Concourse job, which runs through Calabash tests:
Next Steps
Now that we have set up a pipeline to run tests on devices, we can work on our iOS codebase and iterate rapidly, knowing that our tests are being continuously run on devices. From a product standpoint, Concourse’s pipeline definition semantics let us pass along the latest build of our iOS app to other jobs which can perform more tests, or promote a passing build to a release of the product.
About the Author