Image for post
Image for post

Preface

  • A brief introduction of Go modules and Semantic Import Versioning
  • A discussion about how to convert multiple Go libraries in the same repository to Go modules
  • A discussion about how to utilize Go Modules in microservices

prerequisites

Go Modules

An Example

path/to/my-repo:
bar:
go.mod
bar-file1.go
bar-file2.go
foo:
foo-file1.go
foo-file2.go
mixi:
go.mod
mixi-file1.go
mixi-file2.go
Image for post
Image for post

As shown in the picture, the repository my-repo has two modules bar and mixi. Take the module bar as an example, it contains two packages: the package bar and the packagefoo. The file go.mod under the directory path/to/my-repo/bar defines the module's path and its dependencies:

module path/to/my-repo/bar

require (
golang.org/x/text v0.3.0
rsc.io/sampler v1.99.99
// Other dependencies
)

The filego.mod bundles the package bar and the packagefoo together as a unit. For example, the import statement in the following code will import the module path/to/my-repo/bar (which includes the package foo) rather than the package path/to/my-repo/bar/foo when Go Modules is enabled. Even though the code looks the same, the path in the import statement is recognized as the module path, not the package path, once Go Modules is used

import "path/to/my-repo/bar/foo"

func main () {
foo.DoSomething()
}

How to Enable Go Modules

When to Use Go Modules

Semantic Import Versioning

  • v1 must be omitted from the module path. This post explains the reason. You may need to follow this rule in your packages if you are thinking of converting your packages to modules one day.
  • The Major versions which are higher than v1 must be embedded in the package path or the module path so that Semantic Versioning can be applied to Go packages and modules.

The following picture demonstrates the rules above:

Image for post
Image for post

Releasing

git tag bar/v2.3.3 && git push -q origin master bar/v2.3.3

You can read my last blog for more details about how to releases modules with Semantic Import Versioning.

All in all, Go Modules provides a way to group one or more packages as a single retrievable unit, while Semantic Import Versioning is a method for applying Semantic Versioning in Go packages and modules to make them versioned. These two things are designed for breaking a repository into multiple retrievable units (modules) so that Go can grab dependencies at the module granularity rather than the repository granularity.

Utilizing Go Modules

General Guide of Converting Go Packages to Go Modules

Converting

  1. Cd to the root directory of the module package: cd path/to/module
  2. Convert the package to a module: go mod init github.com/azhuox/blogs/golang/go_modules/example/module
  3. Compile the module and its dependencies: go build
  4. Commit the changes automatically generated by Go: git add ./go.mod ./go.sum && git commit -q -m "Convert the package to a module" && git push origin master -q
  5. (Optional) you can run go mod vendor to reset the module's vendor directory to include all the packages and modules which are required for building and testing all of the module's packages. This is the way to provide dependencies for the older versions of Go that do not fully understand Go modules. Any version of Go >= v1.11 does not need this.

Here are the contents of the go.mod file automatically generated by Go. You can see that it defines the module's path, glues anything under the path/to/example/module directory as a single unit and lists all of its dependencies.

module github.com/azhuox/blogs/golang/go_modules/example/module

go 1.12

require (
golang.org/x/net v0.0.0-20190328230028-74de082e2cca
rsc.io/quote v1.5.2
)

Go utilizes the following roles to grab the module’s dependencies:

  1. It grabs the latest version for the packages that have been converted to modules. For example, rsc.io/quote v1.5.2.
  2. It grabs the latest commit for the packages that have not been converted to modules with the format v0.0.0-{date}-{first_12_characters_of_commit_id}. For example, golang.org/x/net v0.0.0-20190328230028-74de082e2cca.

Releasing

The first problem is how to release v2 or higher Major versions. Go utilizes two methods, Major Branch and Major Subdirectory, which are provided by this proposal to solve this problem. This blog demonstrates these two methods and compares their advantages and disadvantages. In this blog, Major Subdirectory is used for all the examples as it does not require to duplicate any code.

The second problem is we need to figure out whether to consider the conversion from Go package(s) to a Go module a breaking change or not. If so, we need to upgrade the Major version using Semantic Versioning. If not, we need to decide what versions we need to release. I prefer to just release the latest version of the package(s) listed in the CHANGELOG.md file for the following reasons:

  1. The conversion from Go package(s) to a Go module is not a breaking change as the package(s) can still work with older versions of Go even if the package(s) are converted to a module. So it does not make sense to upgrade the Major version for this kind of change.
  2. The conversion from Go package(s) to a Go module does not add any new feature or fix any bug. So upgrading the Minor or Patch version, in this case, does not make sense either.

Now let us come back to the module example and release its latest version. Here is what I did:

  1. Appended v2 to the end of the module path (module github.com/azhuox/blogs/golang/go_modules/example/module/v2) as the latest version of the module package is v2.0.1.
  2. Add a note under the v2.0.1 release note in the CHANGELOG.md file to indicate that the package is converted to a module in and after this version.
  3. Release v2.0.1 by creating a git tag: git tag golang/go_modules/example/module/v2.0.1 && git push -q origin master golang/go_modules/example/module/v2.0.1

Consuming A Module

[[constraint]]
name = "github.com/azhuox/blogs"
branch = "master"

With Go Modules, what you need to do is import and use the module in your Go program and run go build. It will automatically grab the golang/go_modules/example/module/v2.0.1 module other than the whole repository for your build.

Converting Go Libraries to Go Modules

I wrote three packages liba libb and libc under the github.com/azhuox/blogs/golang/go_modules/example/libs/ directory for the demo purpose. Among these three packages, the package libb depends on the package libawhile the package libc depends on the packagelibb and libc.

A principle that we need to follow in this case is to convert the packages that have no dependency on other packages within the same repository, and then convert the packages which dependencies have been converted Go modules. This indicates that we need to convert the package liba first, then the package libb and then the package libc in this case.

Let us see what will happen if we convert libc first:

go mod init github.com/azhuox/blogs/golang/go_modules/example/libs/libc
go: creating new go.mod: module github.com/azhuox/blogs/golang/go_modules/example/libs/libc
go build:

can't load package: package github.com/azhuox/blogs/golang/go_modules/example/libs/libc: unknown import path "github.com/azhuox/blogs/golang/go_modules/example/libs/libc": ambiguous import: found github.com/azhuox/blogs/golang/go_modules/example/libs/libc in multiple modules:
github.com/azhuox/blogs/golang/go_modules/example/libs/libc (/Users/achuo/go/src/github.com/azhuox/blogs/golang/go_modules/example/libs/libc)
github.com/azhuox/blogs v0.0.0-20190330175117-09a7dbd4a3ce (/Users/achuo/go/pkg/mod/github.com/azhuox/blogs@v0.0.0-20190330175117-09a7dbd4a3ce/golang/go_modules/example/libs/libc)

The cause of this ambiguous import problem is Go grabs the whole repository github.com/azhuox/blogs v0.0.0-20190330175117-09a7dbd4a3ce to get the liba and libb package for satisfying the dependencies of the libc module. However, github.com/azhuox/blogs v0.0.0-20190330175117-09a7dbd4a3ce also includes a copy of the libc package, which confuses the Go compiler. To fix this, we need to convert the liba and libb package to Go modules and release them, so that they can be retrieved and parsed properly as two individual modules by Go.

Now let us convert these three libs in the correct order.

Convert the packageliba to a module:

cd path/to/libs/liba
go mod init github.com/azhuox/blogs/golang/go_modules/example/libs/liba
go: creating new go.mod: module github.com/azhuox/blogs/golang/go_modules/example/libs/liba
go build
go: finding golang.org/x/net/context latest
go: finding golang.org/x/net latest

# Commit changes
#
git add ./go.mod ./go.sum
git commit ./go.mod ./go.sum -q -m "Convert liba to a module" && git push origin master -q

# Release the latest version (v1.1.0):
#
git tag golang/go_modules/example/libs/liba/v1.1.0 && git push -q origin master golang/go_modules/example/libs/liba/v1.1.0

convert the packagelibb to a module:

go mod init github.com/azhuox/blogs/golang/go_modules/example/libs/libb
go: creating new go.mod: module github.com/azhuox/blogs/golang/go_modules/example/libs/libb
go build
go: downloading github.com/azhuox/blogs/golang/go_modules/example/libs/liba v1.1.0
go: extracting github.com/azhuox/blogs/golang/go_modules/example/libs/liba v1.1.0
...

git add ./go.mod ./go.sum
git commit ./go.mod ./go.sum -q -m "Convert libb to a module" && git push origin master -q
git tag golang/go_modules/example/libs/libb/v1.0.0 && git push -q origin master golang/go_modules/example/libs/libb/v1.0.0

Convert the package libc to a module:

go mod init github.com/azhuox/blogs/golang/go_modules/example/libs/libc
go build
go: downloading github.com/azhuox/blogs/golang/go_modules/example/libs/libb v1.0.0
go: extracting github.com/azhuox/blogs/golang/go_modules/example/libs/libb v1.0.0
...

git add ./go.mod ./go.sum
git commit ./go.mod ./go.sum -q -m "Convert libc to a module" && git push origin master -q
git tag golang/go_modules/example/libs/libc/v1.0.0 && git push -q origin master golang/go_modules/example/libs/libc/v1.0.0

You can see the package libc is converted to a module correctly and it can retrieve the modules liba and libb in its build without any problem.

Go Modules and Microservices

github.com/azhuox/blogs/tree/master/golang/go_modules/example/micro-service:
- sdks
- go
- internal
- api
- pkga
- pkgb
- server
- main.go
- vendor
- Gopkg.toml
- Gopkg.lock
- Dockerfile

I want to mention that the package internal/pkgb is using the package libc that we just converted to a Go module above. In this case, libc is retrieved together with liba and libb from the github.com/azhuox/blogs repository when Go Modules is not enabled. But it is retrieved individually as a single unit when Go Modules is enabled.

From the project layout, you can also see that the microservice is built as a docker image with the following Dockerfile:

FROM golang:1.12-alpine3.9

RUN apk add --update \
ca-certificates \
git

COPY . $GOPATH/src/github.com/azhuox/blogs/golang/go_modules/example/micro-service
RUN go build -o /usr/bin/micro-service github.com/azhuox/blogs/golang/go_modules/example/micro-service/server && rm -rf $GOPATH/*

ENTRYPOINT ["/usr/bin/micro-service"]

As mentioned in the When to Use Go Modules section, only public packages need to be converted to modules. In this case, the package sdks/go is the only package that gets publicly used. Therefore, we only need to convert this package to a module and releases its latest version:

go mod init github.com/azhuox/blogs/golang/go_modules/example/micro-service/sdks/go
go build
...
git add ./go.mod ./go.sum
git commit ./go.mod ./go.sum -q -m "Convert micro-service/sdks/go to a module" && git push origin master -q
git tag golang/go_modules/example/micro-service/sdks/go/v1.0.2 && git push -q origin master golang/go_modules/example/micro-service/sdks/go/v1.0.2

Go Modules, in this case, refers to the new Go package management tool called vgo which is integrated into go tools like go get and go mod. The following steps demonstrate how to use it to manage the dependencies for the microservice:

  1. Launch a terminal and then enable Go Modules in the terminal: export GO111MODULE=on.
  2. Cd the root directory of the microservice.
  3. Add a go.mod file to the root directory of the microservice: go mod init github.com/azhuox/blogs/golang/go_modules/example/micro-service.
  4. Run or test the microservice to ensure that everything works fine: go run ./server/main.go. This will generate a file called go.sum if everything goes well.
  5. Remove the files for the old dependency management tool, which is Gopkg.toml and Gopkg.lock in this case.
  6. Commit the changes.

Now we successfully replace the old dependency management tool with Go Modules. However, there are two cases we need to deal with in the Continuous Integration (CI) process: with a vendor or without a vendor.

CI Without Vendor

  1. Add an environment variable ENV GO111MODULE=on in the Dockerfile to enable Go Modules.
  2. Remove the vendor directory since we don't need it anymore.
  3. Commit the changes.

CI With Vendor

  1. Dump all the dependencies into the vendor directory: go mod vendor.
  2. Commit the changes.
  3. If Go Modules is enabled in the CI tool, add the -mod=vendor in the go build step in the Dockerfile: go build -mod=vendor -o /usr/bin/micro-service github.com/azhuox/blogs/golang/go_modules/example/micro-service/server && rm -rf $GOPATH/*.

Update A Dependency in the vendor Directory

  1. Get the version: go get github.com/azhuox/blogs/golang/go_modules/example/libs/libc@v1.5.0.
  2. Update the vendor directory: go mod vendor.

This may not work when the microservice is not using any new feature released after the current version of libc (v1.0.0 in this case). To force update it, we need to add a replace statement in the go.mod file and then run go mod vendor:

replace (
github.com/azhuox/blogs/golang/go_modules/example/libs/libc v1.0.0 github.com/azhuox/blogs/golang/go_modules/example/libs/libc v1.5.0
)

Summary

  • Semantic Import Versioning is a method for applying Semantic Versioning to Go packages and modules to make them versioned.
  • Only the publicly-used packages, for example, Go libraries and SDKs, need to convert to Go modules (which produces the modules).
  • It is very easy to replace a legacy Go package management tool (e.g. dep) with Go modules (which consumes the modules).

Reference

A software engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store