Go Basics: Filesystem IO
A basic guide to working with files & directories in Go
Introduction
Reading and writing to disk and navigating the filesystem is a staple in any language. Let’s learn how to do it in Go with os, a package that lets us interact with operating system functionality.
Agenda
Files
Creating & Opening Files
Reading Files
Writing & Appending to Files
Removing FilesDirectories
Creating Directories
Navigating & Reading Directories
Walking through a Directory
Files
Creating & Opening Files
File creation can be done with os.Create and opening a file is done with os.Open. Both take in a file path and return a File struct and a non-nill error if unsuccessful.
If you call os.Create on an existing file, it will truncate the file: the file’s data is erased. In contrast, calling os.Open on a non-existing file will result in an error.
If successful, we can then use the returned File struct to write and read data to the file (There will be an example of opening a file, reading it chunk by chunk, and closing it in the next section).
Finally, it is good to close the returned file after interacting with it by using File.Close.
Reading Files
One way we can process a file is by reading all its data at one time. We can do this with os.ReadFile. The input is the file path and the output is a byte array of the file’s data and an error if unsuccessful.
If we are processing a text file, we need to convert the output byte array into a string in order to get the file text.
Keep in mind that os.ReadFile will read the entire file and load its data into memory. If the file is large, using os.ReadFile will consume large amounts of memory.
A memory performant approach is to process the file chunk by chunk, which can be done using os.Open.
After opening the file, File.Read is repeatedly called till EOF (end of file).
File.Read takes in a byte array, b and loads up to len(b) bytes from the file into b. It then returns the number of bytes read, bytesRead and an error if something goes wrong. If bytesRead is 0, that means we hit EOF, and are done processing the file.
In the above code, we are loading a max of 10 bytes from the file, processing the bytes, and repeating this process till EOF.
For larger files, this approach consumes a lot less memory than loading the entire file once.
Writing & Appending to Files
Analogous to os.ReadFile , there is os.WriteFile , a function to write bytes to a file.
Things to note about os.WriteFile
- Be sure to convert the data you want to write into
[]bytebefore passing it intoos.WriteFile. - The permission bits are needed to create the file if it doesn’t already exist. Don’t worry too much about them.
- If the file path already exists,
os.WriteFilewill override the initial data in the file with the new data that is being written.
os.WriteFile is a neat way to make a new file or override it, but it doesn’t work when we need to append to a file’s existing contents. In order to append to a file, we will need to leverage os.OpenFile .
According to the docs, os.OpenFile is a more generalized version of os.Open and os.Create. Both os.Create and os.Open internally call it.
In addition to the file path, os.OpenFile takes in an int flags and an int perm, the permission bits, and returns a File struct. In order to perform operations like reading and writing, the right combination of flags must be provided to os.OpenFile.
We can combine O_APPEND and O_WRONLY with a bitwise OR and pass it into os.OpenFile to get a File struct. If we then call File.Write with whatever data we pass in, the data will get appended to the end of the file.
Removing Files
os.Remove takes in a path to a file or an empty directory and deletes the file/directory. If the file doesn’t exist, a non-nil error will be returned.
Now that we learned the basics of working with files, let’s dive into working with directories.
Directories
Creating Directories
In order to create a new directory, we can use os.Mkdir. os.Mkdir takes in the directory name and the permission bits, and makes a new directory. A non-nil error will be returned if the function cannot make the directory.
In some situations, we may need temporary directories which exist only during the execution of the program. os.MkdirTemp can be utilized to make such directories.
os.MkdirTemp ensures that the temp directories created have unique names even when called by multiple goroutines or programs (source). Moreover, once you are done working with the temp directory, ensure to delete it and its contents with os.RemoveAll.
Navigating & Reading Directories
To begin, we can get the current working directory with os.Getwd.
Then, os.Chdir can be used to change the working directory.
In addition to changing the working directory, we can get directory children with os.ReadDir. os.ReadDir takes in a directory path and returns an array of DirEntry structs and a non-nill error if unsuccessful.
Here is its usage:
Walking through a Directory
With os.Chdir & os.ReadDir , we can walk through all the files and sub-directories in a parent directory, but the path/filepath package provides an elegant way of doing this with the filepath.WalkDir function.
filepath.WalkDir takes in root , the directory where will walk from, and a callback function fn of the following type:
type WalkDirFunc func(path string, d DirEntry, err error) errorfn will be called on each file and sub-directory in root. Here is an example of counting all the files in a root directory.
path/filepath provides another function called filepath.Walk which has the same behavior as filepath.WalkDir. However, the docs state that filepath.Walk is less efficient than filepath.WalkDir. As a result, using filepath.WalkDir may be more ideal.
Conclusion
I wrote this article as an exercise for learning Go, and with it, I hoped to introduce you to the basics of working with files & directories. As an exercise, I recommend implementing your own version of filepath.WalkDir.
Thank you for your reading!
