Slides: A Primer to Unit Testing and Continuous Integration in iOS

Many thanks to Ivan for taking the photo.

Earlier this week, I gave a talk at the April 2013 meetup event of iOS Dev Scout. It was my first public speaking, so obviously I was nervous. I felt that I left out a lot of the details, so I decides to follow up with a blog post and hope to cover the remaining grounds.

Summary

The talk began with my thoughts about software development and how testing fit into the picture. Then, I start describing the various techniques of testing: from unit to user acceptance, specification-based to scenario-based, mocking object to matcher. After which I introduced command line tools of Xcode and how you can manage your tests with Makefile or Rakefile. Finally, with that you can set up a Continuous Integration Server to execute the tests at specific interval or on every commit. All these contents created with a focus on iOS development. As I only had less than an hour of speaking time, my contents only touches the surface. Nevertheless, I hope you enjoy the slides.

Slides

Follow Up

I would like to take this opportunity to expand on some subject I touched on, they are as follows.

Unit Testing

Run loop are NOT running during tests

One of the concept you should be familar with when writing for iOS is the run loop. It is essential for many time-based and asynchronous operations like timers and network operations, which work in conjuction with a run loop.

This introduces a slight inconvience when it comes to testing in OCUnit though, as the run loop are not running during the tests. But if you find yourself needing to test operations that require the run loop to work, you can exercise it with the follow code snippet:

{% gist 2586763 %}

Third Party Solutions

In the past, OCUnit wasn't as good as it is now. Apple definitely have taken their time to improve OCUnit. During that period, many third party solutions are created to address OCUnit's limitations like its inability to set a breakpoint in unit test.

But as Apple improves OCUnit, it didn't force these third party solution into redundancy. Many offers a lot of unique features like capturing UI layout differences, running test in parrallel, etc. Its definitely worth checking these third party testing framework out if you are serious in the field of testing.

Techniques of Testing

Mocking Object provides a way to test scenario where the result is non-deterministic or determined by external factors that are hard to recreate in a test environment, although it is not limited to these use cases.

As of matchers, they allow you to be more descriptive in what you are testing for, instead of just a bunch of assertions which is often hard to read.

Behavior Driven Testing

One of the hardest part of testing is to determine how much to test. It is not just about the breadth of the codes to cover, it also about the level (high level user experience or low level developer experience) to be in control. So it quickly become a width times height problem when one start doing testing. Many methods of testing are developed to address this question. Behavior Driven Testing are one of them, but within itself there are branches based on specifications (Specta, UISpec, Kiwi) or scenarios (Frank, KIF), which are usually more focus on integration and user acceptances.

My advice is to focus on the extreme of the levels (highest, integration or user acceptance, and lowest, logic or algorithmic) and focus on the critical logic or user flow. You have to accept that different regions of your app are of different priorities. In the ideal world, all regions should be covered, but in reality, where time and efforts are of constraints, you will want to cover the important one first.

Using Frank

Frank is a user acceptance testing framework similar to UIAutomation, providing ability to query for UI elements on screen and interacting with them.

I integrated Frank on one of my open source project on called LXPagingView as a seperate branch and used it as an example in my talk. To do the same for your own project.

  1. Make sure you have frank-cucumber gem installed using the following command:
    gem install frank-cucumber
  2. Navigates to your project root directory, the directory where your .xcodeproj file resides, in a terminal.
  3. Frankified your project with the following command:
    frank setup
  4. Build and launch your project using Frank with the following command:
    frank build_and_launch
  5. To run your test, navigates to the Frank subdirectory and run the following command:
    cucumber

A project that is frankified will have its .xcodeproj modified and has a subdirectory called Frank created. The Frank subdirectory contains apps, bundles, libraries and features. Features are probably what you will be interested in most of the time as it contains your test scenarios, steps and step definitions, allowing you to define your test.

In my example, I created a feature file named navigation.feature to test the navigation of my demo app. I uses the built-in steps that come with Frank except for this one: When I drag to the 3rd page in "PagingView". This is a custom step I introduced and I have to define its behavior in a step definition, which I did with the file named drag_steps.rb.

Continuous Integration

Setting Up a CI Server

Although one of the main topic I speak about is Continuous Integration, I did not plan to touch on setting up a CI Server. I believe others have already done a better than me in explaining the steps involved. One of them is written by Manuel Binna titled Continuous Integration of iOS Projects using Jenkins, CocoaPods, and Kiwi.

Examples

I have prepared a few of examples for the talk and although I couldn't cover all of them at the end, I will be listing them all below:

  • LXSupport
    Using (Ruby) Rakefile to manage testing task.

    • .travis.yml contains configurations and commands to instruct Travis CI on how to run the test.
    • Gemfile are use to ensure all the gems are installed with a simple bundle install command.
    • Rakefile are used to manage the test task.

  • My Fork of CargoBay
    Demostrates the availability of homebrew to install ios-sim. Using Makefile to manage testing task.

    • .travis.yml contains configurations and commands to instruct Travis CI on how to run the test.
    • Makefile are used to manage the test task.
    • In CargoBay.xcodeproj, under Targets > CargoBayTests > Build Phases, Run Script, we extend on the default command to support using ios-sim to run your test. The command snippet is as below:
      {% gist 5473043 %}

  • UIAutomation on CI
    Based on Apple's Recipes sample code, I wrote a UIAutomation script that interacts with the UI elements and run it with using instrument on Travis CI.

    • .travis.yml contains configurations and commands to instruct Travis CI on how to run the test.
    • Makefile are used to manage the test task.
    • In Recipes.xcodeproj, under Targets > Recipes > Build Phases, Run Script, we attempt to bring up instrument to run the UIAutomation script. The command snippet is as below:
      {% gist 5473077 %}

Final Thoughts

Testing seats between the development and deployment phase of the software deployment life cycle. It main goal is to make sure of the quality of the app is not being hindered by bugs. As your software start taking shape, its quality often deteriorates, its stability often shaken.

Experiences are able to buy you a stronger foundation and better scaffolding that these softwares are built and dependent on but bugs are still not uncommon, just lesser, especially the repeated ones.

Test driven development reinforces your foundation and scaffolding at the expense of time and efforts. Rat holes are cemented upon first sight, not sealed up with thin woods that the rat can crawl away eventually and ran havoc in your home, leaving you with bigger problems.

Though its really hard to quantify these testing efforts as they are more preventive then cure. And many people would immediate relate to the resources that would be taken away for perceived value creating tasks. Worst, testing are translated to quality and stability that users and stakeholders come to expect from you so they think it should come at your expenses, not theirs. Scaffolding are afterall, a temporary structure used to support the project. Convincing people to spent more time and efforts on foundation and scaffolding can be a daunting tasks.

But the funny things is that as the software develops, we spent more time and resource testing; The more rats running around the house, the more resources are being summoned. These resources continue to be spent on repeatable tasks just to catch rat the developers frantically attempts to squash; Testing the algorithmic tasks, certain user flows, over and over again. Soon enough, you end up spend more resources finding and squashing bugs than working on the next feature.

As Douglas McIlroy once said, "As a programmer, it is your job to put yourself out of business. What you do today can be automated tomorrow." Even with the automation programming has empowered us, many of us still end up with manual solution in the testing phase. Programming don't just covers the development phase, it should cover as much of your software development spectrum.

Whether these are short sightedness, its all dependent on the context. Some companies run on quantity of contracts constrainted by tight schedules, development resources just aren't enough thus quality have to suffer. Strong business development ensures the company can survive with the dip in quality in their delivery. This is just one of the many examples that developing codes to test the software might not be feasible. Ultimately, we can have to put theory in practice.

Regardless, you want to avoid either extremes; bug infested products which you end up spending bulk of your time debugging or too much testing that you end up not writing any meaningful codes.