Continuous Integration for iOS Apps using Jenkins and Fastlane: A thorough guide.
There are two main recommended paths toward continuous integration for iOS. The path supported and recommended by Apple relies on Xcode server, which allows you to configure and run ‘Bots’ to execute tests within your project, however at the time of writing (Xcode 10) these integration bots do not support pushing the builds to the app store and I have had no luck working around this restriction, see my post here ( https://forums.developer.apple.com/thread/110079).
The second path, which is outlined below involves using a Jenkins server to handle task management and then a library called Fastlane that helps to manage all the building, signing and uploading of the builds. Here, Jenkins is listening for new events and then launching our builds and Fastlane is actually performing the heavy lifting and managing all of the command line utilities to perform the actions that we need locally on the server.
The setup outlined below is a fairly advanced setup, in which a single app has 4 different targets, one which is a prototype and 3 pointing at different environments (development, staging and production). Each of these targets has a different app on appstore connect for testing through TestFlight (though of course only prod is live). The dev and prototype builds will be executed when a feature is merged into the dev branch, and a staging and production build will occur when those changes are merged into the master branch.
So, Jenkins will be set up with two tasks.
- One of those will look for changes to the dev branch and then when found, run our tests and then build and test our prototype and development targets.
- The second task will look for changes to the master branch, and when found run tests and build and push our staging and production targets.
Fastlane will be configured with a few simple jobs to make this possible.
- Run our automated UI tests. This will launch the app UI and click through all of our key work flows in the app.
- Run our automated unit and integration tests. Our unit tests will test specific key methods at a function level, allowing us to test stuff like signing a transaction with a given private key. Our integration tests will run against the development api and perform a set of api requests to make sure that our results are as expected
- A set of 4 jobs exist to automate the building and uploading of each of our 4 targets. Prototype, Dev, Staging and Prod. Each of these targets have separate app store profiles allowing each to be available and testable via testflight.
To deploy a new Prototype and Dev build, we need to create a PR from our feature branch into Development and merge it after approval. The Jenkins server will detect this change and launch a new Build-Dev task, which will run tests and push to the store.
To push to staging / prod, we need to PR from Development into Master. This will launch a new Build-Prod task which will run tests and push to the TestFlight.
Note: In addition to pushing to the correct git branch, you must also ensure that for each target,
- the app store listing has a new version pending with the correct new version number in the target. This is done by logging into appstoreconnect.apple.com, going to the given app, and pressing Add new version. The version found in the info.plist for the given target in xcode must be updated to match this version.
To check on the status of a build, you can log in to jenkins from a remote computer by opening the IP address of the remote machine, at port 8080 in any web browser. Once logged in, you may view the status of any previous jobs, view the console from any running jobs and manually launch a new job.
This is a guide for setting up a new CI environment as described above
Access Remote Server (via macstadium.com)
Create a new account at macstadium.com. This will give you access to a fresh Mac Mini that is all set up for remote access.
To access the machine remotely, you may use Apple’s ‘Screen Sharing’ software and enter the IP address for the new machine as found in the Subscriptions tab in macstadium. The user name will be administrator and the default password will be provided through their portal. Once logged in, change this default password.
Next, make sure the following are installed:
- the lastest version of Mac OS (currently Mojave)
- Xcode (from the App Store)
- Xcode command line utilities
- xcode-select –install
- Ruby Version Manager
- \curl -sSL https://get.rvm.io| bash -s stable –ruby
- /usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”
- gem install cocoapods
- sudo gem install fastlane -NV
Install Jenkins Server
While logged in as the administrator, install Jenkins. While there is a dmg installer available to install, I was unable to get Jenkins set up with the installer and instead installed using Homebrew.
brew install jenkins
When installing, you only need a minimal set of plugins, so don’t install with all the defaults, instead look through the plugins and only install ones that look useful, like github.
Once the installer is run, you have to unlock it. Do this by running
jenkins from the command line to launch jenkins. Then from a web browser, navigation to localhost:8080 to open the jenkins application, which will direct you to access a local password file to unlock it.
Next you’ll want to create a new Jenkins User. Open the jenkins console in a web browser on the remote machine and hit Jenkins → Manage Jenkins → Manage Users → Add New. Create a new user ‘jenkinsuser’ and assign them a password.
Close the command line console to close jenkins. Note that by default the workspace will be set at ~/.jenkins. You may wish to rename it,
mv ~/.jenkins ~/jenkins to make the workspace more visible.
Configure Jenkins to Launch on Boot
In the current configuration, Jenkins will only run when the administrator is logged in and must be manually started by the user. We’ll want Jenkins to launch every time the computer boots without requiring a login.
To help make this task a bit easier, I enlisted help from a program called LaunchControl available here for $10: http://www.soma-zone.com/LaunchControl/
Install, purchase and open LaunchControl on the remote computer and lets create a new Global Daemon.
On the top left of the screen, you’ll see a drop down with User Agents selected. Change that to Global Daemons and press File→New.
We’ll configure this job as shown below:
- Rename the File in the left hand pane to Jenkins
- The Program to Run should be set to
- Drag and drop a Standard Output object from the right pane to configure your logging output location
- Drag and drop an environment variable object from the right pane to setup your environment variables. You can press Import from Shell to import your path. We must also configure our jenkins home path and set our language to UTF8 by setting LANG, LANGUAGE and LC_ALL to en_US.UTF8. Note that the default jenkins home path is ~/.jenkins, but i manually moved my jenkins files to ~/jenkins.
- Drag and drop the user name object from the right pane and set it to run as Administrator and the group ‘wheel’.
- Make sure Run at Load is checked.
- Save this job, then right click it and ‘Load it’. This will launch the job and make sure it launches on boot from here on out. After loading, the job should show as ‘running’ and Jenkins should be accessible at localhost:8080 from the remote machine, or from any outside machine at http://[ip address]:8080.
We must now setup fastlane in our local xcode project.
From the project directory, on your mac, run
sudo gem install fastlane -NV fastlane init
(this assumes you already have ruby installed on your local machine, if not see above to install rvm and ruby)
You’ll also want to add these lines into your ~/.bash_profile
export LANGUAGE=en_US.UTF8 export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8
This will generate all the fastlane files and integrate it into your project. You’ll likely want to add fastlane/fastfile into your project so you can edit it in xcode. You likely want to add some of the fastlane files to your .gitignore
fastlane/report.xml fastlane/Preview.html fastlane/screenshots fastlane/test_output
You may also want to follow the instructions here to add a gemfile and lock the versioning of fastlane: https://docs.fastlane.tools/getting-started/ios/setup/
Now in Xcode we can create our fastlane tasks. Open the fastfile generated by fastlane and add some tasks:
desc "Push a new proto dev build to the App Store" lane :release_dev do set_info_plist_value(path: "Project/Info-Proto.plist", key: "commit_hash", value: last_git_commit[:commit_hash] ) build_app(workspace: "Project.xcworkspace", scheme: "Project-Dev") upload_to_app_store(skip_metadata: true, skip_screenshots: true, app_identifier: "com.app.id", username: "firstname.lastname@example.org") end desc "Run automated tests" lane :test do run_tests(workspace: "Project.xcworkspace", devices: ["iPhone 6"], scheme: "Project-Dev", clean: true) end desc "Run automated ui tests" lane :test_ui do run_tests(workspace: "Project.xcworkspace", devices: ["iPhone 6"], scheme: "Project-Proto", clean: true) end
Note the set_info_plist command is going to take the commit hash of the current build that we are compiling and add it into our project at build time so we can access it and display it in code within our project.
Now commit this code and push it up to the server.
git checkout -b 'fastlane' git add . git commit -am 'add fastlane'
Push this new branch and merge it into dev.
Configure Github webhooks
Generate a new secret key for the webhook. Open the jenkins console from a web browser and go to Manage Jenkins → Configure Jenkins → Github. If you do not see the Github section ensure the github plugin is installed.
Press Advanced Settings and you’ll see a drop down for Shared Secret. Press add and follow the steps to generate and store a new shared secret.
We need to configure github to send webhooks for our repository to Jenkins. Goto github.com and navigate to the repo settings → webhooks → Add webhook
- set the URL: http://[server ip]:8080/github-webhook/ NOTE THE TRAILING /
- Set content type to application/json
- Add the shared secret we generated in jenkins
- Specify custom events and select pushes and pull requests
Configure Jenkins to pull from github
From the remote server, we need to add an ssh key to the admin user and add the key to our github account. Follow these instructions to generate a key and add it to github.
Configure Jenkins Tasks
Now we need to get our project compiling and running tests on the server. Log in to the remove server via screen sharing. Open Jenkins via localhost:8080
Press New Item and create a free style project. Call it Build-Dev.
- give it a general description
- under general in github project, link it to the http url of your repo: https://github.com/Org/repo/
- under source code management, select git and add a new repository
- under repository add the ssh url like so: [
- under credentials, select add new and add a new key secured with the Jenkins Credentials Provider.
- Select ssh username and private key and enter the username and private key generated in the previous steps.
- Press Advanced Details and add a name for the repo: Org/repo
- Under branches to build, add a new branch and enter the name of the development branch
- under repository add the ssh url like so: [
- Under Build Triggers, select “GitHub hook trigger for GITScm polling”
- Under Build, add a new Build Step and enter in the commands to execute the build.
- pod install
- fastlane test_ui
- fastlane test
- fastlane release_dev
We’ll want to then create a task for production releases. You can do this by simply copying the settings from the dev build, but change the names of the branches and the fastlane targets to execute.
Configure Email notifications
To have Jenkins send emails after every build, first create a new gmail account just for sending these build summary emails.
Next configure Jenkins with a default email configuration
Goto Configure Jenkins – Manage Configuration
Under Extended E-mail Notification:
- set smtp server: smtp.gmail.com
- Click Advanced
- Click Use SMTP Authentication
- Enter the username and password for your gmail account
- Check Use SSL
- set smtp port: 465
- Sent Default Reply-To email: email@example.com
- For Default recipients add any developers that want to be emailed for EVERY action
- Under Default smtp suffix add your mail server suffix (@yourorg.com)
Scroll up to Jenkins Location
- set Jenkins Location to your external server url.
- Set your System Admin e-mail address to the address you want to display in your emails
Now configure your tasks:
- Go to your build task configuration
- Under Post-build Actions, add a new Editable Build Notification
- Click advanced settings
- Under attach build log, select ‘Attach Build Log’
- Under Triggers, remove the default mail trigger
- Now add a new trigger for Always to send out an email to the default recipient list after the build runs
- Add a second trigger for Before Build to send to the default recipient list before a build is run
- Add a final trigger for Success. Under recipients, press advanced and add the email addresses for any additional mailing lists that would like to be notified after successful builds, like ‘firstname.lastname@example.org’
Install certificates and provision profiles
This should now compile our project, but builds cannot successfully upload because the server does not have the certificates required to sign our code and to push the builds to the app store.
Certificate and provisioning profile management in iOS can be quite a pain, but here are the basic steps to get this working.
- log in to the apple developer portal at developer.apple.com/ios
- created a new app ID to match the project’s bundle in xcode if it hasn’t been created yet
- Create a new private key to store on the server by opening keychain access from the server, then keychain assistant → request new certificate from a certificate authority.
- go to manage certificates and create a new app store distribution certificate for the specific app ID. You’ll have to upload the signing request you generated in the previous step. Download and open that certificate to load it into the keychain on the CI server.
- go to provisioning profiles and add a new distribution profile for app store distribution. Select the certificate we generated previously. Download and install that profile to load it into xcode on the CI server.
- Now on your local machine, download those same files from the developer portal. The private key will not be installed on your local machine since it was generated remotely. This is good since we only want the server to have access to sign builds.
- Open xcode on your local machine and make sure ‘automatically manage code signing’ is unchecked. Then select the certificate and provisioning profiles you downloaded under the release version of the target.
- Do the above for each project target that you want to upload to the app store.
- Commit and push these changes to git.
Now the project is set up to use our new profiles and the profiles are stored on the admin keychain on the server, but we need to move them to the system keychain so that the jenkins server can access them without the admin user being signed in.
On the remote server, open keychain access. From the Login keychain, open the certificates tab and find the certificates you downloaded previously.
Copy those certificates, by clicking them to expand them and highlighting both the certificate and the associated private key. Then Press Edit→ Copy. Then Paste those into the System keychain.
App Store Access
The last thing we need is access to upload the build to the app store. We’ll create a new user in appstore connect and add that user’s credentials to the server’s keychain.
- goto appstoreconnect.apple.com Users → Add New
- Add a new user with developer access assigned to the specific apps that you want the server to be able to push to. You’ll need to confirm the email address and assign a set of security questions.
- Now we need to add these credentials to the keychain on the remote server. I was unable to actually add this to the keychain manually, however fastlane has a tool that will add this for us. We we are going to use this tool running with Jenkins so that we can be sure it gets stored to the system keychain instead of the admin user keychain, otherwise Jenkins won’t be able to access it.
- Go to the Jenkins console
- Add a new Jenkins Task
- Leave everything blank, but add a new build script
- in the build script enter this, replacing the two passwords and user name
- echo ‘[the servers admin password]’ | sudo -S fastlane fastlane-credentials add –username [itunes username] –password [itunes password]
- Now run this task. It will save the password to your keychain
- Delete the script out of the task and save and then delete the entire task to remove traces of the passwords.
Enable Server side build versioning
Let’s allow the CI system to increment our build numbers for us. That way, when we push new code we won’t have to also increment the build numbers manually. Note that if we try to push builds to the app store with the same build number, the upload and hence, the build will fail.
First, we need to make sure our build number versioning is set to use whole numbers. If we are using something like 188.8.131.52, this system won’t work since we will have to do a bunch of parsing to successfully increment the build number.
Now we just add this ‘lane’ to fastlane in our fastfile for prod, and a similar one for dev.
lane :build_number_prod do version_number = get_version_number(:target => "Project-Prod") last_build_dev = latest_testflight_build_number( :app_identifier => "com.org.appid", :version => version_number, :initial_build_number => 0, username: "email@example.com") last_build_prod = latest_testflight_build_number( :app_identifier => "com.org.appid2", :version => version_number, :initial_build_number => 0, username: "firstname.lastname@example.org") last_build = last_build_dev > last_build_prod ? last_build_dev : last_build_prod set_info_plist_value( path: "Project/Info-Staging.plist", key: "CFBundleVersion", value: (last_build + 1).to_s) set_info_plist_value( path: "Project/Info-Prod.plist", key: "CFBundleVersion", value: (last_build + 1).to_s) #tag our new build tag = version_number.to_s + "." + (last_build + 1).to_s if git_tag_exists tag: tag tag = tag + "-" + Time.now.to_i.to_s end git_add_tag tag: tag push_git_tags, remote: "Org/project" end
These tasks will essentially check testflight for the lastest build number that has been pushed for the current version and then increment our build accordingly. It will then add a unique tag for this build in github.
Once these lanes are created, modify your Jenkins tasks and add the commands
fastlane build_number_dev to the dev build before
fastlane build_number_prod to the prod build before
Locking It Down
The server is currently a little too easy to access. We need to turn off VNC / remote desktop access by default and require logins to come into the machine via SSH instead, where the SSH key is a hardware authentication device like a yubikey.
SSH Restrictions (Optional)
Restrict SSH logins and require all logins to use a Hardware Token from our YubiKey.
- ssh into the machine
- add any yubikey public keys for devices that need access to the ~/.ssh/authorized_keys file
- Prevent people from logging in with passwords, and only with public key based authentication by editing the /etc/ssh/sshd_config file and set:
- Restart the machine and and login via ssh to confirm all is working well. The login should not prompt you for a password and instead should require your yubikey.
Screen sharing restrictions
Now login to the machine via VNC and create these scripts to start and stop VNC. Save them in the administrators’ home folder and make sure they have executable permissions.
#!/bin/bash sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -allowAccessFor -specifiedUsers sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -configure -users administrator -access -on -privs -all -setmenuextra -menuextra no -restart -agent
#!/bin/bash sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -deactivate -configure -access -off
Now we’ll want to run the stop_vnc command on boot, so open launch control and add a new Global Daemon job.
- Configure the job to run your new stop vnc script by setting the executable to run to
- Make sure run on load is checked
Now load the job and start it. This should kill your VNC connection. To launch a new one, just login over ssh and run the start_vnc.sh command.
That’s it!! That wasn’t so bad? Was it…???