Preface

What is included in this blog:

  • A discussion about how to convert multiple Go libraries in the same repository to Go modules
  • A discussion about how to utilize Go Modules in microservices

prerequisites

Go Modules

Go Modules is an experimental opt-in feature in Go 1.11 with the plan of finalizing feature for Go 1.13. The definition of a Go module from this proposal is “a group of packages that share a common prefix, the module path, and are versioned together as a single unit”. It is designed for resolving dependency hell problems in Go, like conflicting dependencies and diamond dependency.

An Example

Here is an example of Go Modules:

path/to/my-repo:
bar:
go.mod
bar-file1.go
bar-file2.go
foo:
foo-file1.go
foo-file2.go
mixi:
go.mod
mixi-file1.go
mixi-file2.go
module path/to/my-repo/bar

require (
golang.org/x/text v0.3.0
rsc.io/sampler v1.99.99
// Other dependencies
)
import "path/to/my-repo/bar/foo"

func main () {
foo.DoSomething()
}

How to Enable Go Modules

In order to use Go Modules, you need to upgrade your Go to v1.11 or any later version and set the environment variable export GO111MODULE=on.

When to Use Go Modules

The major purpose of Go Modules is to let one or more packages be versioned, released, and retrieved together as a single unit. Therefore, the public packages, for example, Go libraries and SDKs, are major targets of Go Modules as they need to be published properly for public use. You do not need to convert internal packages or any internal-used-only packages within a microservice repository to Go modules. These packages can directly import and use modules once Go Modules feature is enabled, even if they are not converted to modules.

Semantic Import Versioning

Semantic Import Versioning is a method proposed for adopting Semantic Versioning in Go packages and modules. The idea behind it is embedding the major version (say v2) in the package path (for packages) or the module path (for modules) with the following rules:

  • The Major versions which are higher than v1 must be embedded in the package path or the module path so that Semantic Versioning can be applied to Go packages and modules.

Releasing

With Go Modules and Semantic Import Versioning, you can release your modules by creating git tags. A tag corresponds to a version. For example, the following git command releases the bar module v2.3.3:

git tag bar/v2.3.3 && git push -q origin master bar/v2.3.3

Utilizing Go Modules

General Guide of Converting Go Packages to Go Modules

I wrote a dummy package called module for demonstrating how to convert one or more Go packages to a Go module.

Converting

It is very easy to convert one or more Go packages to a Go module. Take the module package as an example, here are the steps of converting it to a Go module:

  1. Convert the package to a module: go mod init github.com/azhuox/blogs/golang/go_modules/example/module
  2. Compile the module and its dependencies: go build
  3. Commit the changes automatically generated by Go: git add ./go.mod ./go.sum && git commit -q -m "Convert the package to a module" && git push origin master -q
  4. (Optional) you can run go mod vendor to reset the module's vendor directory to include all the packages and modules which are required for building and testing all of the module's packages. This is the way to provide dependencies for the older versions of Go that do not fully understand Go modules. Any version of Go >= v1.11 does not need this.
module github.com/azhuox/blogs/golang/go_modules/example/module

go 1.12

require (
golang.org/x/net v0.0.0-20190328230028-74de082e2cca
rsc.io/quote v1.5.2
)
  1. It grabs the latest commit for the packages that have not been converted to modules with the format v0.0.0-{date}-{first_12_characters_of_commit_id}. For example, golang.org/x/net v0.0.0-20190328230028-74de082e2cca.

Releasing

A module can only be used as a module after it is released. A module is released by creating git tags and each tag corresponds to a version. However, there are two problems we need to solve before releasing a module.

  1. The conversion from Go package(s) to a Go module does not add any new feature or fix any bug. So upgrading the Minor or Patch version, in this case, does not make sense either.
  1. Add a note under the v2.0.1 release note in the CHANGELOG.md file to indicate that the package is converted to a module in and after this version.
  2. Release v2.0.1 by creating a git tag: git tag golang/go_modules/example/module/v2.0.1 && git push -q origin master golang/go_modules/example/module/v2.0.1

Consuming A Module

You can still use this package, without Go Modules enabled, by using some Go dependency management tool (e.g. dep) with the following specification. This will grab the whole repository which includes the module module for your build.

[[constraint]]
name = "github.com/azhuox/blogs"
branch = "master"

Converting Go Libraries to Go Modules

The section above already demonstrates how to convert one or more Go packages to a Go module. This section majorly talks about how to convert all the Go packages (libraries) within the same repository to Go modules.

go mod init github.com/azhuox/blogs/golang/go_modules/example/libs/libc
go: creating new go.mod: module github.com/azhuox/blogs/golang/go_modules/example/libs/libc
go build:

can't load package: package github.com/azhuox/blogs/golang/go_modules/example/libs/libc: unknown import path "github.com/azhuox/blogs/golang/go_modules/example/libs/libc": ambiguous import: found github.com/azhuox/blogs/golang/go_modules/example/libs/libc in multiple modules:
github.com/azhuox/blogs/golang/go_modules/example/libs/libc (/Users/achuo/go/src/github.com/azhuox/blogs/golang/go_modules/example/libs/libc)
github.com/azhuox/blogs v0.0.0-20190330175117-09a7dbd4a3ce (/Users/achuo/go/pkg/mod/github.com/azhuox/blogs@v0.0.0-20190330175117-09a7dbd4a3ce/golang/go_modules/example/libs/libc)
cd path/to/libs/liba
go mod init github.com/azhuox/blogs/golang/go_modules/example/libs/liba
go: creating new go.mod: module github.com/azhuox/blogs/golang/go_modules/example/libs/liba
go build
go: finding golang.org/x/net/context latest
go: finding golang.org/x/net latest

# Commit changes
#
git add ./go.mod ./go.sum
git commit ./go.mod ./go.sum -q -m "Convert liba to a module" && git push origin master -q

# Release the latest version (v1.1.0):
#
git tag golang/go_modules/example/libs/liba/v1.1.0 && git push -q origin master golang/go_modules/example/libs/liba/v1.1.0
go mod init github.com/azhuox/blogs/golang/go_modules/example/libs/libb
go: creating new go.mod: module github.com/azhuox/blogs/golang/go_modules/example/libs/libb
go build
go: downloading github.com/azhuox/blogs/golang/go_modules/example/libs/liba v1.1.0
go: extracting github.com/azhuox/blogs/golang/go_modules/example/libs/liba v1.1.0
...

git add ./go.mod ./go.sum
git commit ./go.mod ./go.sum -q -m "Convert libb to a module" && git push origin master -q
git tag golang/go_modules/example/libs/libb/v1.0.0 && git push -q origin master golang/go_modules/example/libs/libb/v1.0.0
go mod init github.com/azhuox/blogs/golang/go_modules/example/libs/libc
go build
go: downloading github.com/azhuox/blogs/golang/go_modules/example/libs/libb v1.0.0
go: extracting github.com/azhuox/blogs/golang/go_modules/example/libs/libb v1.0.0
...

git add ./go.mod ./go.sum
git commit ./go.mod ./go.sum -q -m "Convert libc to a module" && git push origin master -q
git tag golang/go_modules/example/libs/libc/v1.0.0 && git push -q origin master golang/go_modules/example/libs/libc/v1.0.0

Go Modules and Microservices

I wrote a dummy micro-service for demonstrating how to utilize Go Modules in a microservice. Here is its project layout:

github.com/azhuox/blogs/tree/master/golang/go_modules/example/micro-service:
- sdks
- go
- internal
- api
- pkga
- pkgb
- server
- main.go
- vendor
- Gopkg.toml
- Gopkg.lock
- Dockerfile
FROM golang:1.12-alpine3.9

RUN apk add --update \
ca-certificates \
git

COPY . $GOPATH/src/github.com/azhuox/blogs/golang/go_modules/example/micro-service
RUN go build -o /usr/bin/micro-service github.com/azhuox/blogs/golang/go_modules/example/micro-service/server && rm -rf $GOPATH/*

ENTRYPOINT ["/usr/bin/micro-service"]
go mod init github.com/azhuox/blogs/golang/go_modules/example/micro-service/sdks/go
go build
...
git add ./go.mod ./go.sum
git commit ./go.mod ./go.sum -q -m "Convert micro-service/sdks/go to a module" && git push origin master -q
git tag golang/go_modules/example/micro-service/sdks/go/v1.0.2 && git push -q origin master golang/go_modules/example/micro-service/sdks/go/v1.0.2
  1. Cd the root directory of the microservice.
  2. Add a go.mod file to the root directory of the microservice: go mod init github.com/azhuox/blogs/golang/go_modules/example/micro-service.
  3. Run or test the microservice to ensure that everything works fine: go run ./server/main.go. This will generate a file called go.sum if everything goes well.
  4. Remove the files for the old dependency management tool, which is Gopkg.toml and Gopkg.lock in this case.
  5. Commit the changes.

CI Without Vendor

Without vendors means utilizing Go Modules to dynamically grab dependencies when building docker images during the CI process. To do this, we need to do the following steps:

  1. Remove the vendor directory since we don't need it anymore.
  2. Commit the changes.

CI With Vendor

With vendor means we want to dump all the dependencies into the vendor directory and let the CI build the docker image based on the vendor directory. The following steps demonstrate how to do it:

  1. Commit the changes.
  2. If Go Modules is enabled in the CI tool, add the -mod=vendor in the go build step in the Dockerfile: go build -mod=vendor -o /usr/bin/micro-service github.com/azhuox/blogs/golang/go_modules/example/micro-service/server && rm -rf $GOPATH/*.

Update A Dependency in the vendor Directory

Suppose we want to build docker images with the vendor directory and use the latest version of libc (say v1.5.0) in the microservice. The following steps demonstrate the update process:

  1. Update the vendor directory: go mod vendor.
replace (
github.com/azhuox/blogs/golang/go_modules/example/libs/libc v1.0.0 github.com/azhuox/blogs/golang/go_modules/example/libs/libc v1.5.0
)

Summary

  • Go Modules allows you to group one or more packages to a single unit which is released and retrieved together.
  • Semantic Import Versioning is a method for applying Semantic Versioning to Go packages and modules to make them versioned.
  • Only the publicly-used packages, for example, Go libraries and SDKs, need to convert to Go modules (which produces the modules).
  • It is very easy to replace a legacy Go package management tool (e.g. dep) with Go modules (which consumes the modules).

Reference

A software engineer