Sanely Test Your Android UI Libraries With Paparazzi

Create a proper verification process for Android UI libraries using snapshot testing

Photo by Mauro Gigli on Unsplash

Testing should be the most important thing a dev thinks about while writing (or attempting to write) shippable code.

No matter how important it is though, testing sometimes can be inherently difficult.

This is especially true on Android UI libraries where things like Unit testing or UI testing (using Espressο, UiAutomator) start to make less sense.

My use case

The problem I had to sort was trying to test my library StageStepBar.

It is essentially a progress bar that can contain Stages (milestones) and individual steps between them. It is also quite configurable, effectively allowing you to change the look and behavior of all its components. It is really a big Custom View with most of its code inside its onDraw.

All seemed fine and dandy until I realized that the way I could test it after changing stuff was very cumbersome. I did thankfully include a sample application but I essentially had to modify, build and run the application to check that my changes were affecting the library in the correct way. But even that process was not bulletproof enough as:

• Sometimes I would forget to retest some cases.
• Other times I would not test some cases that would seem irrelevant but would be “mysteriously" affected.
• This whole thing still involved one device. What happens with different ones? RTL locales? The list could go on…

Enter snapshot testing

While attending Droidcon 2021 @ London I got introduced to another way of testing things in the Android world - snapshot testing. Here is a nice talk about this topic.

The gist of it is:

• You take a bunch of screenshots and/or videos of your UI.
• If you are happy with them you store them in someplace in your repo as golden values/sources of truth.
• If any changes are made, a verification algorithm (usual pixel by pixel comparison) runs through your code and compares the golden values with the new screenshots.
• If these changes are expected/desired, the screenshots will be replaced. Otherwise, the code is breaking some things that it shouldn’t.

This seemed like just the right approach for my use case. However, a specific library that was showcased at that event called Paparazzi actually convinced me that it is.

What is Paparazzi

This is yet another snapshot testing library but the difference with other currently available solutions (see Shot for example) is that it runs completely on the JVM without the need of a device or emulator.

This talk from the same Droidcon goes into more detail about how it works.

https://github.com/cashapp/paparazzi

This was really convenient for my use case as it meant that I could use Github Actions to create a way for both me and any future contributors to get some fast feedback on whether their changes broke something.

It is worth noting that at the time of writing (12th of Feb 2022) the library is not stable yet. Complete documentation is not fully there and stuff seems like it is still being worked on. So it might be that what you see here may not work the same in a few months. Also at the time of writing again, there was no Compose support.

So with all that out of the way, here is the full process I followed:

Writing tests (Recording process)

First of all, I created a simple library module. This module:

• Depends on the StageStepBar module (which is the one under test).
• Has the Paparazzi plugin applied to it. This gives it the record and verifies Gradle tasks which we will use later.
• Contains the tests and the golden values (screenshots) under src/test and src/test/snapshots accordingly.

An example of the tests we can write is the following:

Essentially we have a layout file with the view that we want to test. Paparazzi inflates it with its special context using paparazzi.inflate().

We can then get access to that view and get it to a state that we want it to be in for each particular test case.

Then calling paparazzi.snapshot() seals the deal as it will render the first frame of everything that we have set up into a screenshot file. If we want more than just a frame (say we are testing animations), then we can use the handy paparazzi.gif() method. The result after running gradle module-name:testDebug on that module is these two screenshots:

All these snapshot files sit inside the build/ folder now. If we are happy with how everything looks we can just run:

gradle module-name:recordPaparazziDebug

This will put these files under src/test/snapshots as well thus making them available to be checked in with git. Now our source of truth has been created!

Note: It is worth noting that using LFS for these files is highly recommended by the Paparazzi authors. As screenshots will change in the future so will the git history become full of these binary blobs. LFS makes it possible to instead save pointers that will point to remote files on another server, make the whole process easier and faster for you. Both Paparazzi and Github docs contain setup guide.

Testing against code changes (Verifying process)

After all the above has been pushed to the repo it’s time to think about how will these things be enforced from now on. So for that I created a GitHub actions workflow that will for every push on an open PR:

• Run the gradle module-name:verifyPaparazziDebug task to compare the screenshots that the PR code would output against the golden values.
• If any of the tests fail, then an image with the expected/actual screenshots together with their delta will be uploaded as an artifact in that run (per test failed). It will also put a comment on the PR that will point the dev to the artifacts section so that they can study the errors. The delta is a really nice extra that we get from the library. Here is how that looks:

Image showing the Actual/Expected/Delta of a single test (created by Paparazzi)

PR message by CI when Snapshot testing has failed

• If all tests pass instead then a happy message is posted on the PR instead.

PR message by CI when Snapshot testing has succeeded

The full CI workflow looks like this:

But what happens if changes are desired? In that case, the reviewer (well me in this case) can approve the PR and ask the author to run the record task on their code and push the resulting snapshots.

Then finally when that PR is merged into the main branch the golden values will be updated.

Final Thoughts

I really believe that snapshot testing with Paparazzi can be a particularly effective way to create a proper verification process for Android UI libraries that had a hard time being tested before. The fact that it runs straight on the JVM means no emulator/device setup, fast execution, and easier integration with CI.

Massive props to the folks that are working on this great library that I am merely using!

I am quite keen to see what you people think about all this!