Go Commands and Env Vars Every Developer Should Know

GOPATH, GOPRIVATE, and more

Photo by Alex Andrews on Pexels

If you are a developer in Go, chances are you have struggled with these cryptic-looking terms before — GOPATH, GOPRIVATE, Go111module, etc.

Most of the time, you don’t need to care what they are. But at times, you might bump into errors that drastically stall your development.

In this article, I will save you the pain and show you the basics you need.

Let’s get started!

Note: The following details only apply to Go1.15 and above.

GOROOT

GOROOT stores the Go SDK

GOROOT is where your Go SDK is stored.

It stores the default Go compilers, Go executable commands, and Go libraries.

You don’t need to amend this variable unless you plan to use a different Go version.

When importing a library, Go first searches the file in GOROOT. If the file doesn’t exist, it then fallbacks to the GOPATH.

GOPATH

GOPATH contains three subdirectories

The GOPATH contains three directories: pkg, src, and bin.

$GOPATH/pkg

• $GOPATH/pkg/mod is the default location for GOMODCACHE.
• Hence, $GOPATH/pkg/mod stores and caches dependencies downloaded via go get or go install.

$GOPATH/bin

• The bin dir stores executable commands installed via go install.
• These commands include 3rd parties commands and your source files’ commands.
• Go default commands such as gofmt are stored in the GOROOT/bin dir instead.

$GOPATH/src

• The src dir stores Go source files before the introduction of Go modules.
• It has become less significant since the advent of Go modules.

Go Modules

Photo by Ekaterina Belinskaya on Pexels

Before the Go modules were introduced, Go projects and dependencies were required to store under the $GOPATH/src/ directory.

There was no version management. Projects under the src directory shared libraries under the same stable version, the master branch.

After the introduction of Go Modules, the following has changed:

• Developers are allowed to create projects outside of the $GOPATH/src/dir.
• Each project, aka module, is a collection of files and packages meant to be released together.
• Each module contains a go.mod file that defines the dependencies and versions required.
• The dependencies are downloaded from the respective repositories during a build.

Go.mod

Go.mod contains the module path and module dependencies

The go.mod file resides in every go module.

The go.mod file does the following:

• Defines the module’s path — The path used when importing packages/files under the same go module.
• Defines the dependencies and versions required for a successful build of a module.

Go.sum

Go.sum contains a list of checksum

When a remote server builds a module, it pulls and downloads dependencies specified in the go.mod file from the respective VCSs.

This inevitably leads to several issues:

• What happens if someone maliciously tampers with the specified version of the library?
• How can we ensure that the remote server pulls a library containing the exact content as our local copy?

This is where the go.sum comes into play.

• It’s the checksum of all the direct and indirect dependencies listed in the go.mod.
• It validates and ensures that the local downloaded copy of the dependencies is similar to the remote copy.

Go mod tidy

Go mod tidy ensures that go.mod is updated

The go.mod tidy ensures that the go.mod is updated.

When you run go mod tidy, it does the following,

• Downloads required dependencies and update go.mod.
• Removes unnecessary dependencies from go.mod. (Note: It doesn’t remove the libraries from local cache, aka, $GOPATH/pkg/mod)

Go clean -modcache

Photo by Yan Krukov on Pexels

When you perform go get or go mod tidy , Go downloads the dependencies and caches them in GOMODCACHE, which is defaulted to be $GOPATH/pkg/mod/.

Sometimes, you may wish to remove all downloaded packages and that’s what go clean -modcache does.

It removes all downloaded packages in the GOMODCACHE directory.

Go mod vendor

Go mod vendor creates a vendor dir in the current module

When building a module in a remote server, the server pulls and downloads the dependencies specified in the go.modfile.

If there’s a network issue or one of the dependencies is removed in the remote VCS, it will lead to a failing build.

To ensure a deterministic build, go mod vendor creates a folder in your Go module and stores your dependencies’ source files in the directory.

The vendor directory will be committed together with your code change.

This allows

• Deterministic and consistent build.
• Changes to dependencies can easily be seen and reviewed.

However, it takes up additional space and increases the time spent cloning a repo leading to a longer CI/CD.

GOPROXY

Before the advent of GOPROXY, dependencies were downloaded directly from remote VCSs.

This resulted in two fundamental issues — security and availability.

• Dependencies in a remote VCS could be removed anytime.
• Dependencies in a remote VCS could be tampered with and compromised.

The GOPROXY is a centralised repository that hosts and caches third parties modules that are publicly available.

Instead of getting from the remote VCSs, Go redirects every download request to a public GOPROXY and validates the downloaded module against a public Go checksum database.

The GOPROXY is defaulted to be

GOPROXY="https://proxy.golang.org,direct"

Adding a direct in the GOPROXY allows Go to fallback to VCSs if the requested module is not found in the GOPROXY.

GOPRIVATE

GOPROXY works well with public modules. However, it leaves us with another issue.

What happens if the requested dependency is from a private repo, for example, your company repo?

This is where the GOPRIVATE comes to the rescue. It controls which modules the Go command considers to be private, for example

GOPRIVATE=*github.com/org_name

Go treats any path that matches this pattern as private and, therefore, will not use the proxy and global checksum database.

Go111module

The GO111module stands for module-aware mode.

When Go was first shipped, there was no package manager until Go module was introduced in Go1.11.

During that time, source files were stored under $GOPATH/src and go get would fetch dependencies and store them in $GOPATH/src.

In short, the way we store, download, and import dependencies differs between the pre- and post-Go module era.

GO111module=on forces Go to behave the Go modules way, while GO111module=off forces Go to behave the GOPATH way.

Since Go modules are now the de facto practice, GO111module=on is the default behavior.

Closing

There is much more to cover regarding the details of Go commands and env variables.

You will bump into the mentioned ones almost every day in your development workflow.

I hope you find this helpful, and I will see you at the next one!