Making Mocking Libraries Mock Themselves

Is static mocking too powerful for its own good? What happens when the mocker becomes the mocked?

Photo by Nijwam Swargiary on Unsplash

Mocking libraries are powerful tools that can swap real objects for fake ones in unit tests, isolating the code under test from its dependencies. With the help of Java bytecode manipulation tools like ByteBuddy, a library like Mockito or MockK can swap out the functionality of almost any code on the JVM classpath.

So if the mocking framework code can stealth-drop a replacement for any piece of code in the application, what happens when you point it at itself? What could possibly go wrong?

Test 1: Does It Even Work?

I’m testing this in Kotlin with two frameworks: MockK and Mockito. For the first challenge, let’s start simple. Does mocking the mocking library even work?

mockStatic(Mockito::class.java, Answers.CALLS_REAL_METHODS)
println(mockingDetails(Mockito::class.java).isMock)

Here, I’m asking Mockito to mock itself, and then asking it whether it now thinks the Mockito class is a mock. Sure enough, it prints true. Cool! Now for MockK. This is a little harder because instead of having a single class to mock, we have a bunch of top-level functions. I settled on mocking the file that contains those functions.

val mockk = Class.forName("io.mockk.MockKKt")
mockkStatic(mockk.kotlin)
println(isMockKMock(mockk, staticMock = true))

Success! At least it prints true, so we know we’ve mocked something.

Test 2: Can It Verify?

One everyday use of mocks is to record function invocations and their arguments so that a test can verify that a particular function was called at the right time. Can the mock framework spy on calls to its own functions?

First, I’ll create a simple mock Supplier<String> that returns "foo". I’ll call the supplier method and use the mocking library to verify that function call took place. Then comes the real test: verifying that the verifying took place.

val staticMock = mockStatic(Mockito::class.java, CALLS_REAL_METHODS)
val test = mock(Supplier::class.java) { "foo" }
println(test.get())
verify(test).get()
staticMock.verify { verify(test) }

Mockito takes a commanding lead right away, sailing through the test as if this was a perfectly normal and sensible thing to do. I even checked that it really is performing the verification and that it fails if the invocation didn’t happen exactly as expected. How does MockK compare?

mockkStatic("io.mockk.MockK")
val test = mockk<Supplier<String>> { every { get() } returns "foo" }
println(test.get())
verify { test.get() }
verify { verify(verifyBlock = any()) }

MockK starts to struggle here, throwing an AbstractMethodError about how the object passed to verifyBlock doesn’t implement one of its required functions. Actually, it throws the same error even if I don’t include the initial mockkStatic call, so it looks like it simply isn’t possible to nest calls to verify. Passing an actual lambda in place of the any matcher doesn’t help either, producing a different exception. If you can find a way to make this work, leave a comment!

The score is 2–1 to Mockito. Can MockK claw it back in the final round?

Test 3: Self-Sabotage

The other common reason to use a mock is to provide dummy behaviour in place of the real functionality. How will the mocking libraries do when we ask them to stub their own code? Place your bets now.

mockStatic(Mockito::class.java, CALLS_REAL_METHODS)
    .`when`<Any> { mock(any(Class::class.java)) }
    .thenReturn(Supplier { "Imposter!" })
println(mock(Supplier::class.java).get())

To keep things simple, I’m not using mockito-kotlin, hence the backticks required in the call to when. In this test, I’ve told Mockito to return my imposter Supplier object every time anybody tries to mock anything. Maybe I should have expected it by now, but I was still pretty surprised to find that this works with no problem. The code runs with no errors and even prints the expected text "Imposter!". Is Mockito actually just magic?

I’m not confident about MockK’s chances, though. All those top-level inline functions make things messy.

mockkStatic("io.mockk.MockK")
every { mockk<Any>(block = any()) } returns Supplier { "Imposter!"}
println(mockk<Supplier<String>>().get())

Sure enough, it’s that pesky AbstractMethodError again. At this point, I decided to dig into the MockK implementation and see if I could get past that. Since I’m on the JVM, the actual implementation of the mockk function is provided by a class called JvmMockKGateway. Mocking that class would be more similar to what I was doing with Mockito, and might level the playing field a bit.

mockkObject(JvmMockKGateway)
every {
    JvmMockKGateway.defaultImplementation.mockFactory
        .mockk<Any>(any(), any(), any(), any(), any())
} returns Supplier { "Imposter!" }
println(mockk<Supplier<String>>().get())

It was a worthy attempt, but this one fails with a StackOverflowError. To be honest, that’s exactly what I would expect to happen when you ask a mocking framework to attack itself, and I’m amazed it never happened in my experiments with Mockito. I guess MockK is destined to only ever be used for actually sensible things.

The Moral of the Story

This is a dumb thing to do, but I do have a point. Mocking frameworks like MockK and Mockito offer two very distinct types of functionality.

• Functions like mock(…) create a new object that can be passed to a function or constructor in place of the real thing. Creating the mock instance doesn’t change the behaviour of any other objects or existing code.
• On the other hand, functions like mockkObject or mockStatic change the behaviour of something that already exists. They can be used to rip out and replace the implementation of a top-level function or an entire class and its constructor. This is often called static mocking.

Static mocking makes it possible to do dumb things, like having the mocking framework mock itself. I don’t like static mocks. By their nature, constructors and top-level functions can be called from anywhere at any time. Mocking them lets you make sweeping changes to system behaviour that can have unintended consequences well beyond the code you were originally trying to test.

The other day I came across a real example that had me scratching my head for some time. A test failed with a strange stacktrace that seemed entirely unrelated to what the test was doing. Eventually, I tracked the problem down to a static mock. The code under test was generating an object containing a timestamp.

In the test setup, the system Calendar was mocked to use a fixed timestamp which could then be checked in the test’s assertion. The problem was the test framework also initialised a logger, which was trying to write a log message that included a timestamp. The misbehaving Calendar made the logger very confused, and the test framework crashed.

In that example, one solution would have been to pass a Clock object to the constructor of the code under test. That way, it would have been easy to mock the individual Clock object without affecting anything else in the application.

Always look for strategies to inject individual mock objects instead of relying on static mocks. Using a static mock is like trying to pass a math test by hunting down and changing every copy of the textbook. Yes, studying for the test might take some effort, but I bet it’ll create less trouble in the long run.