Embedding files natively in Go 1.16

Gophers with a wrench by Renee French

One of the most commonly noted upsides of Go is the compilation to a single binary. This makes deployments and dependency requirements easier to handle compared to other languages that require to install in the target system the individual dependencies which can potentially conflict with other software running or require to install duplicated packages.

However, sometimes Go programs not always can be reduced to a single file. Assets like templates or images are not included as part of the binary and they need to manage and deployed independently. Although this has its benefits, like customising templates without having to recompile the whole program, it loses the benefits of a single unit when you do not want or do not expect to make a change in the assets, like with games.

Traditionally, we have relied on third-party packages like go-bindata or statik to embed static assets in Go binaries. These tools essentially require to run a command against the assets we want to embed to create a Go file which contains a binary representation of them. Later, you import this generated file in your code and you access the files using some key based on the path or a virtual file system.

These solutions have some issues in terms of development experience. They require an extra step to add/update assets every time they change and having the tool available in each environment in some concrete version. It is also an external dependency, which might be complicated to adopt in certain corporate environments. Also, to keep the component “go-getable”, you need to commit the generated files, which is not ideal either.

In version 1.16 (not available yet as of December 2020), the Go team has introduced a new package named embed which solves these trade-offs by embedding the files during the building of the binary. Hence, we don't need to run additional commands before running go build or to track generated files. The go build system will recognise the directives and populate the variables with the matching files.

How it works

General conditions

1. If any patterns are invalid or have invalid matches, the build will fail.
2. The directive must immediately precede a line containing the declaration of a single variable. Only blank lines and comments are permitted between the directive and the declaration.
3. Patterns must not contain . or .. path elements nor begin with a leading slash.
4. To match everything in the current directory, use *.
5. Patterns must not match files outside the package’s module, such as .git/ or symbolic links.
6. Each pattern in a line must match at least one file or non-empty directory.

Embedding a file as a string

• The files are stored as string or []byte.
• You can use a single go:embed directive per variable.
• The directive must refer to a single file.
• You can use a blank import.

package mypackage
import _ "embed"
import "fmt"
//go:embed README.md
var r string
//go:embed image.png
var img []byte 
func main() {
    fmt.Println(r);
}

Embedding several files as a file system.

• The files are stored as embed.FS
• embed.FS is read-only and safe to use from different goroutines concurrently.
• embed.FS implements fs.FS, so it can be used with any package that understands file system interfaces
• You can use a single or multiple go:embed declarations. Each declaration can refer to one or multiple paths separated by spaces.
• If a pattern names a directory, all files in the directory are embedded recursively, except that files with names beginning with . or _ .
• If you use * after the directory, it will add also files prefixed with . and _ .

package mypackage
import (
    "embed"
    "net/http"
    "log"
)
// It will add the specified files.
//go:embed favicon.ico robots.txt index.html
// It will add all non-hidden file in images, css, and js.
//go:embed image/ css/ js/
// It will add all the files in downloads, including hidden files.
//go:embed downloads/*
var static embed.FS
func main() {
    http.Handle("/", http.FileServer(http.FS(static)))
    log.Fatal(http.ListenAndServe(":8080", nil))
}

For more information, check the references below. Do you have any feedback? You can reach me on Twitter.

References

• https://tip.golang.org/pkg/embed/
• https://github.com/go-bindata/go-bindata/
• https://github.com/rakyll/statik/

Changelog

• 2020–02–01: As pointed in the comments, the article had a typo in the code directives, which should not contain a space.