Understanding fuzzing in Go

Fuzzing in action

Photo by Tim Hüfner on Unsplash

“Fuzz testing is a novel way to discover security vulnerabilities or bugs in software applications.”

The concept of fuzzing was introduced in 1988, when Prof. Barton Miller and his students discovered that an OS “would routinely crash when pinged by random unexpected inputs.”

Fuzz tests generate input for a function, by using supplied seed data. It is a great way to discover unexpected behavior from your code. When I first saw the concept of fuzzing, I thought it will be a great way to automate tests.

To put this idea into context, I’ve found it difficult to think of test input cases in the past, fuzzing seemed like a great way to automate this. fuzzing isn’t an excuse for not implementing Unit and other forms of tests, as fuzzing generates input for a function.

Fuzzing was introduced in Go 1.18, so I’m a bit late to the party. In this post, I’ll attempt to describe a basic use case for fuzzing.

A CSV row parser

For my use case, I’ll write a function that parses a CSV row.

The function will take a comma-delimited string and process some of the data.

The function will attempt to convert the first column to an int, as well as append Senegal to the value of the 3rd column.

The 3rd column represents an address without a specified country.

Here is how the function will look like :

package fuzzing
import (
 "strings"
 "strconv"
)
func ParseString(s string) error {
  parts := strings.Split(s, ",")
  _, err := strconv.Atoi(parts[0])
 if err != nil {
  return err
 }
 parts[2] += ", Senegal"
 return nil
}

Now that I have my function, I’ll proceed by defining the fuzz test. Unlike unit tests, I must supply seed data for the fuzzer.

In the case of my function ParseString, I supplied 2 comma delimited CSV rows, with 3 columns.

While writing a fuzz test it is important to name your test with the following format FuzzXxx. I have the test check for errors originating from function ParseString.

Here is the test file :

package fuzzing
import (
 "testing"
)
func FuzzParseString(f *testing.F){
 f.Add("1, Name, 23rd Fifth Street")
 f.Add("2, Name2, 1 Main street")
 f.Fuzz(func(t *testing.T, s string){
    err := ParseString(s)
    if err != nil {
     t.Errorf("%v", err)
    }
 })
}

My biggest weakness as a programmer is failing to understand edge cases. So when I ran the test, it failed on the first run.

When I checked the supplied data, an empty string was provided.

Here is the output from the test :

cheikh@cheikh-s5-1110:~/go/src/fuzzing$ go test -fuzz FuzzParseString
fuzz: elapsed: 0s, gathering baseline coverage: 0/8 completed
fuzz: minimizing 41-byte failing input file
fuzz: elapsed: 0s, gathering baseline coverage: 2/8 completed
--- FAIL: FuzzParseString (0.01s)
    --- FAIL: FuzzParseString (0.00s)
        testing.go:1349: panic: runtime error: index out of range [2] with length 1
            goroutine 33 [running]:
            runtime/debug.Stack()
             /usr/lib/go-1.18/src/runtime/debug/stack.go:24 +0x90
            testing.tRunner.func1()
             /usr/lib/go-1.18/src/testing/testing.go:1349 +0x1f2
            panic({0x5b3700, 0xc000116210})
             /usr/lib/go-1.18/src/runtime/panic.go:838 +0x207
            fuzzing.ParseString({0x6dcc20?, 0xc00008ea50?})
             /home/cheikh/go/src/fuzzing/lib.go:17 +0x105
            fuzzing.FuzzParseString.func1(0xc000130680, {0x6dcc20, 0x1})
             /home/cheikh/go/src/fuzzing/lib_test.go:12 +0x45
            reflect.Value.call({0x594a60?, 0x5cfb08?, 0x13?}, {0x5c17a6, 0x4}, {0xc00008f050, 0x2, 0x2?})
             /usr/lib/go-1.18/src/reflect/value.go:556 +0x845
            reflect.Value.Call({0x594a60?, 0x5cfb08?, 0x514?}, {0xc00008f050, 0x2, 0x2})
             /usr/lib/go-1.18/src/reflect/value.go:339 +0xbf
            testing.(*F).Fuzz.func1.1(0x0?)
             /usr/lib/go-1.18/src/testing/fuzz.go:337 +0x231
            testing.tRunner(0xc000130680, 0xc0000e8990)
             /usr/lib/go-1.18/src/testing/testing.go:1439 +0x102
            created by testing.(*F).Fuzz.func1
             /usr/lib/go-1.18/src/testing/fuzz.go:324 +0x5b8

The fuzzer detected a runtime issue. With the error in mind, I’ll edit my function again to support empty or incomplete strings. The new function will look like this : (new additions are in bold)

func ParseString(s string) error {
  parts := strings.Split(s, ",")
  if(len(parts) < 3){
    return errors.New("Invalid string")
  }
  _, err := strconv.Atoi(parts[0])
  if err != nil {
    return err
  }
  parts[2] += ", Senegal"
  return nil
}

My program will return an error for malformed strings, but I don’t want my fuzzer to stop there. To do so, I’ll add an early return if the error returned is Invalid string . Here is the updated test function :

func FuzzParseString(f *testing.F){
  f.Add("1, Name, 23rd Fifth Street")
  f.Add("2, Name2, 1 Main street")
  f.Fuzz(func(t *testing.T, s string){
    err := ParseString(s)
    if err != nil {
      if err.Error() == "Invalid string" {
        return
      }
      t.Errorf("%v", err)
    }  
  })
}

After re-running, The fuzzer detected another issue by supplying the following input string(",,") . Here is the output from the terminal :

cheikh@cheikh-s5-1110:~/go/src/fuzzing$ go test -fuzz FuzzParseString
fuzz: elapsed: 0s, gathering baseline coverage: 0/8 completed
fuzz: elapsed: 0s, gathering baseline coverage: 8/8 completed, now fuzzing with 2 workers
fuzz: minimizing 61-byte failing input file
fuzz: elapsed: 0s, minimizing
--- FAIL: FuzzParseString (0.02s)
    --- FAIL: FuzzParseString (0.00s)
        lib_test.go:20: strconv.Atoi: parsing "": invalid syntax
    
    Failing input written to testdata/fuzz/FuzzParseString/74498b2a8282174de6870175ab847ba882cd8356506d60b9a28f23ec8e721c76
    To re-run:
    go test -run=FuzzParseString/74498b2a8282174de6870175ab847ba882cd8356506d60b9a28f23ec8e721c76
FAIL
exit status 1
FAIL fuzzing 0.025s

Let’s say I managed to find a way around the missing ID bug, will the fuzzer find other bugs?

The next issue the fuzzer found was to assign a non-numeric value to the first column. I can keep repeating this process until the code is stable.

This is the benefit of Fuzzing. Here is another output of my test :

cheikh@cheikh-s5-1110:~/go/src/fuzzing$ go test -fuzz FuzzParseString
fuzz: elapsed: 0s, gathering baseline coverage: 0/10 completed
fuzz: elapsed: 0s, gathering baseline coverage: 10/10 completed, now fuzzing with 2 workers
fuzz: minimizing 46-byte failing input file
fuzz: elapsed: 0s, minimizing
--- FAIL: FuzzParseString (0.02s)
    --- FAIL: FuzzParseString (0.00s)
        lib_test.go:20: strconv.Atoi: parsing "A": invalid syntax
    
    Failing input written to testdata/fuzz/FuzzParseString/215375fdfbd654b9fd5b654b45a8676d2b81e538d2a852773673a3713deb50a6
    To re-run:
    go test -run=FuzzParseString/215375fdfbd654b9fd5b654b45a8676d2b81e538d2a852773673a3713deb50a6
FAIL
exit status 1
FAIL fuzzing 0.025s

Conclusion

Testing is a core aspect of Go. Is it possible to write a program without testing? Yes, if you want to sacrifice the quality of your program.

Fuzzing enables developers to test for the unexpected. It does not replace the need for other types of test, but rather complement it. It is a great way to increase test coverage and identify test cases.

After writing our beautiful idiomatic code, we may be a bit exhausted to generate test cases, this is where fuzzing shines.

It is important to note that the testing package does not support complex types, it is something I hope to see in the future.

References

What is fuzz testing?

medium_examples/fuzzing at main · cheikhshift/medium_examples

Go Fuzzing - The Go Programming Language