Dwi Wahyudi
Senior Software Engineer (Ruby, Golang, Java)
We’ll continue our Golang API app from previous article. We’ll apply some separation concerns and writing unit test for it.
Overview
Previously, in this article we’ve set up a web application with some libraries such as Cobra and Viper, together along with a database migration setup as well. In that particular set up, we’ve made the app to utilize Cobra to handle the command line parts, while Viper is responsible for managing configs, environment variables and secrets.
Now let’s deal with another important things: interacting with database and writing unit test for each application part. In some simple web applications, doing all this thing altogether (without separation concerns) in one place is good enough, but when the size of an application grows, it is always a good idea to start thinking about the separation concerns and making it testable. A testable component will be easy to understand, to refactor and to reuse.
Application Structure
The application structure will involve the usage of inversion of control with the help of interface, this will also help with unit testing by using mocks. By utilizing interfaces, we decouple the code between caller and callee. This will make the code unit-testable, because we can just use mocks in place of database/API/IO interaction.
First, we’re going to create a dir and install Gin and Testify (unit test library) framework, don’t forget to include go-sqlmock as well (this is for mocking database call) and mockgen for generating mocks:
mkdir internal/rest
go get -u github.com/gin-gonic/gin
go get -u github.com/stretchr/testify
go get -u github.com/DATA-DOG/go-sqlmock
go install github.com/golang/mock/mockgen@v1.5.0
This internal/rest directory (rest package) will contain everything related to the restful web server handlers. This will also contain our restful server routing code.
Let’s start with the foundation of the rest package:
package rest
type restServer struct {
}
func NewServer() *restServer {
return &restServer{}
}
Nothing fancy, just a restServer struct and NewServer() function, we’ll call this function when creating the web server.
Ping
Ping is just a simple endpoint that will response with a simple string, this is usually made for health-checking.
The Route
Here’s the Route() function in the same package. Let’s define a route for /ping and assign its handler. Do note that when using Gin, the handler needs to have *gin.Engine as the parameter.
package rest
import "github.com/gin-gonic/gin"
func (rs *restServer) Route(r *gin.Engine) {
r.GET("/ping", rs.Ping)
}
The Handler
Still inside the rest package, we’re creating the rs.Ping handler. This handler will do nothing but responding with pong.
package rest
import (
"github.com/gin-gonic/gin"
"net/http"
)
func (rs *restServer) Ping(c *gin.Context) {
c.String(http.StatusOK, "pong")
}
Updating the Server Command
Now, let’s take a look at server cmd in cmd/server/cmd.go, let’s update launchRestServer() function. Because we already place the web handler in rest package before, launchRestServer() will be smaller.
func launchRestServer() {
log.Println("Running the web server")
router := gin.New()
rs := rest.NewServer()
rs.Route(router)
router.Run(":8080")
}
Let’s test by running the server with go run main.go server, we’ll see that now the web server is provided by Gin.
$ go run main.go server
2021/04/24 17:51:35 Running the web server
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET /ping --> updated-golang-api/internal/rest.(*restServer).Ping-fm (3 handlers)
[GIN-debug] Listening and serving HTTP on :8080
We can then double-check with by visiting http://localhost:8080/ping.
The Test
We’re not done with this /ping path, we need to test the handler as well. This could be done with this test function, place this in rest folder but name the package rest_test. We’re going to utilize testify library (for assertion) and net/http/httptest package (for http testing).
We’re going to use testify, it provides assertions API for us to use.
package rest_test
import (
"github.com/gin-gonic/gin"
"github.com/stretchr/testify/assert"
"net/http"
"net/http/httptest"
"testing"
"updated-golang-api/internal/rest"
)
func TestPing(t *testing.T) {
router := gin.Default()
rs := rest.NewServer()
rs.Route(router)
w := httptest.NewRecorder()
req, _ := http.NewRequest("GET", "/ping", nil)
router.ServeHTTP(w, req)
assert.Equal(t, 200, w.Code)
assert.Equal(t, "pong", w.Body.String())
}
Run this test file with: go test ./... and make sure test result is ok.
$ go test ./...
? updated-golang-api [no test files]
? updated-golang-api/cmd [no test files]
? updated-golang-api/cmd/migration [no test files]
? updated-golang-api/cmd/server [no test files]
? updated-golang-api/config [no test files]
? updated-golang-api/db [no test files]
ok updated-golang-api/internal/rest 0.005s
So far there’s no interface involved because testing /ping path is still straight-forward, no database involved, there’s no need to utilize any mock.
Get Users
Now let’s get to the part where database interaction is needed. We’re going to create a GET endpoint that will return all the users from MySQL table we created in previous part.
This time we have to create several interfaces in order to separate between 3 important distinct component types:
- the handler,
- the service (reusable components), and
- the repository (getting data from DB, API, etc).
Handlers will call the service(s), and each service may call the repository. We’re going to decouple these dependencies by using inversion of control.
The Repo and Its Test
We’ll start from the bottom, the component that doesn’t have dependencies, the repository. We’ve made db migration setup in previous article, we’re going to use it. We simply want to get users data from database. We’re going to map the query result to some specific structs. Let’s create a new package model, and create a new file user.go which contains structs related to users data in MySQL.
GetUsersRepoResult will hold the data from GetUsersRepo method in repository code. While GetUsersRepoParam will hold parameters to be passed to that method.
package model
type GetUsersRepoResult struct {
ID int
FirstName string
LastName string
Email string
}
type GetUsersRepoParam struct {
Limit int
Offset int
}
Now let’s create the repository code, this should be done inside user package with the file name mysql.go. This file name is only for convention, we want to get the data from MySQL table. If we want to do something with external API, we should name the file api.go instead.
This repository code will be straight-forward, just do query to user table with certain limit and offset (via GetUsersRepoParam struct). In this code, we also create a function, called NewRepo which will receive a *sql.DB variable and return the userRepo object, this will be mostly called by the services.
package user
import (
"context"
"database/sql"
"updated-golang-api/internal/model"
)
type userRepo struct {
appDB *sql.DB
}
func NewRepo(appDB *sql.DB) *userRepo {
return &userRepo{
appDB: appDB,
}
}
func (ur userRepo) GetUsersRepo(ctx context.Context, param model.GetUsersRepoParam) ([]model.GetUsersRepoResult, error) {
dbConn := ur.appDB
query := `SELECT id, firstname, lastname, email FROM user LIMIT ? OFFSET ?`
rows, err := dbConn.Query(query, param.Limit, param.Offset)
defer rows.Close()
if err != nil {
return []model.GetUsersRepoResult{}, err
}
users := make([]model.GetUsersRepoResult, 0)
for rows.Next() {
var user model.GetUsersRepoResult
err = rows.Scan(&user.ID, &user.FirstName, &user.LastName, &user.Email)
users = append(users, user)
}
return users, nil
}
Now let’s make the test, we will utilize go-sqlmock library. Because our repository code is already isolated, we can easily create the unit test.
Let’s create mysql_test.go in user directory, but with user_test as the package name. We’ll name our test function: TestUserRepo_GetUsersRepo, this means, this test function will test GetUsersRepo under user package.
Instead of opening the real database connection, we will just get *sql.DB from sqlmock.New(), this will be used for user repo that we want to test: userRepo := user.NewRepo(db).
We will try to mock some rows by using NewRows() assign them to a new variable: rows, just pretend that these rows are the real result of the query (because that’s the goal of the mocks) without really hitting the mysql database.
rows := mock.NewRows([]string{"id", "firstname", "lastname", "email"}).
AddRow(1, "john", "doe", "test@email.com").
AddRow(2, "jane", "doe", "test2@email.com")
We then expect query := SELECT id, firstname, lastname, email FROM user LIMIT ? OFFSET ?`` to be called with args of 10 and 10 (which are the limit and the offset) and return rows above.
We then call the method:
results, err := userRepo.GetUsersRepo(context.Background(), model.GetUsersRepoParam{
Limit: 10,
Offset: 10,
})
From these setups, we expect that repository code correctly call the query with limit and offset that we specify in the parameters, plus we expect it to correctly return the results (in the form of GetUsersRepoResult struct).
package user_test
import (
"context"
"github.com/DATA-DOG/go-sqlmock"
"github.com/stretchr/testify/assert"
"regexp"
"testing"
"updated-golang-api/internal/model"
"updated-golang-api/internal/user"
)
func TestUserRepo_GetUsersRepo(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
panic(err)
}
rows := mock.NewRows([]string{"id", "firstname", "lastname", "email"}).
AddRow(1, "john", "doe", "test@email.com").
AddRow(2, "jane", "doe", "test2@email.com")
query := `SELECT id, firstname, lastname, email FROM user LIMIT ? OFFSET ?`
mock.ExpectQuery(regexp.QuoteMeta(query)).
WithArgs(10, 10).WillReturnRows(rows)
userRepo := user.NewRepo(db)
results, err := userRepo.GetUsersRepo(context.Background(), model.GetUsersRepoParam{
Limit: 10,
Offset: 10,
})
if err != nil {
panic(err)
}
expectedResults := []model.GetUsersRepoResult{
{
ID: 1,
FirstName: "john",
LastName: "doe",
Email: "test@email.com",
},
{
ID: 2,
FirstName: "jane",
LastName: "doe",
Email: "test2@email.com",
},
}
assert.Equal(t, expectedResults, results)
}
The Service and Its Test
From the repository code, we’ll continue to service code. Service code is called by the rest handler, it also needs to be decoupled (to repository and to handler) by using inversion of control. First, let’s create the user.go under the user package. Because this service code depends on the repository, we’ll need to declare it here.
type UserRepository interface {
GetUsersRepo(ctx context.Context, param model.GetUsersRepoParam) ([]model.GetUsersRepoResult, error)
}
type UserService struct {
userRepository UserRepository
}
And because this service is needed by the rest handler, we’ll expose a new function for rest handler to use. This also means that at the top stack trace, we’ll need to instantiate the repository first then assign it to this service.
func NewService(userRepository UserRepository) *UserService {
return &UserService{
userRepository: userRepository,
}
}
Before we go to the service method, we’re going to write the struct models needed, let’s add this in model/user.go file.
type GetUsersServiceResult struct {
ID int
FirstName string
LastName string
Email string
}
type GetUsersServiceParam struct {
Limit int
Offset int
}
Wait, isn’t it the same with GetUsersRepoResult and GetUsersRepoParam structs we made before? Yes, that’s true, we create other structs for service code too, because we don’t want to reuse the structs, so that both components (repository and service) are flexible, change in one struct won’t affect both components.
Here’s the method, to get user data, we call the repository by using us.userRepository.GetUsersRepo.
func (us UserService) GetUsers(ctx context.Context, param model.GetUsersServiceParam) ([]model.GetUsersServiceResult, error) {
usersRepo, err := us.userRepository.GetUsersRepo(ctx, model.GetUsersRepoParam{
Limit: param.Limit,
Offset: param.Offset,
})
if err != nil {
return []model.GetUsersServiceResult{}, err
}
result := make([]model.GetUsersServiceResult, 0)
for _, userRepo := range usersRepo {
result = append(result, model.GetUsersServiceResult{
ID: userRepo.ID,
FirstName: userRepo.FirstName,
LastName: userRepo.LastName,
Email: userRepo.Email,
})
}
return result, nil
}
It’s time to write the test for this service code. Let’s create user_test.go in user package. Before we write any service code test, we need to remember that service code above depends on the repository code, but we don’t want to really hit the repository code (we don’t want to hit the database). Mock is needed here in place of repository code.
Add this just above the interface definition, don’t forget to create mock directory inside user directory.
This definition is just a comment, but some text editors can pick it up, and we can run the command from inside the text editors.
//go:generate mockgen -source=./user.go -destination=mock/repository.go -package=usermock
type UserRepository interface {
GetUsersRepo(ctx context.Context, param model.GetUsersRepoParam) ([]model.GetUsersRepoResult, error)
}
// ... rest of the code
We can also run this command in the terminal, the result will be the same.
go generate -run "mockgen -source=./user.go -destination=mock/repository.go -package=usermock"
Mocks will then be generated inside user/mock directory with usermock as its package name. Now let’s create user_test.go inside user directory. This time test function will be named TestUserService_GetUsers.
First, we’re going to create userRepoMock, this is a mock that will replace real user repository.
ctrl := gomock.NewController(t)
defer ctrl.Finish()
userRepoMock := usermock.NewMockUserRepository(ctrl)
us := user.NewService(userRepoMock)
We then expect the service to call user repo’s GetUsersRepo with userRepoMock.EXPECT().GetUsersRepo, we also expect the parameters and the results. For the expectation, the first param we pass is gomock.Any(), this means that we don’t care about the parameter we pass there (it’s a context, btw). What we want to test is whether the limit, and the offset are correctly called on repository code.
After the mock expectation, we then call the method we test, get the results and assert them with our expectation. We assert that data we mock and returned from the repository are correctly returned by the service.
package user_test
import (
"context"
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"testing"
"updated-golang-api/internal/model"
"updated-golang-api/internal/user"
usermock "updated-golang-api/internal/user/mock"
)
func TestUserService_GetUsers(t *testing.T) {
ctrl := gomock.NewController(t)
defer ctrl.Finish()
userRepoMock := usermock.NewMockUserRepository(ctrl)
us := user.NewService(userRepoMock)
userRepoMock.EXPECT().GetUsersRepo(gomock.Any(), model.GetUsersRepoParam{
Limit: 10,
Offset: 100,
}).Return([]model.GetUsersRepoResult{
{
ID: 1,
FirstName: "alpha",
LastName: "beta",
Email: "alpha@email.com",
},
{
ID: 2,
FirstName: "gamma",
LastName: "zona",
Email: "gamma@email.com",
},
}, nil)
results, err := us.GetUsers(context.Background(), model.GetUsersServiceParam{
Limit: 10,
Offset: 100,
})
if err != nil {
panic(err)
}
expected := []model.GetUsersServiceResult{
{
ID: 1,
FirstName: "alpha",
LastName: "beta",
Email: "alpha@email.com",
},
{
ID: 2,
FirstName: "gamma",
LastName: "zona",
Email: "gamma@email.com",
},
}
assert.Equal(t, expected, results)
}
The Handler and Route and Its Test
It’s similar to Ping handler above, which we create a test for, we will also create a test for GetUsers handler. Before that, we need to make sure that we decouple the service from the handler.
Let’s update rest.NewServer function to receive UserService. restServer struct will need to include UserService as well, so that we can call it from our rest handler.
type UserService interface {
GetUsers(ctx context.Context, param model.GetUsersServiceParam) ([]model.GetUsersServiceResult, error)
}
type restServer struct {
userService UserService
}
func NewServer(service UserService) *restServer {
return &restServer{
userService: service,
}
}
Now, let’s find a way to serve above GetUsers service for our rest server. We’ll make a handler for it. Let’s create rest/user.go together along with rest/util.go.
Here’s our rest/util.go, it contains a simple function to calculate the offset needed for fetching data to database. We set the limit to 10.
package rest
import (
"github.com/gin-gonic/gin"
"strconv"
)
func GetLimitOffset(c *gin.Context) (int, int, int) {
pageParam, _ := c.GetQuery("page")
page, _ := strconv.Atoi(pageParam)
if page == 0 {
page = 1
}
limit := 10
offset := limit * (page - 1)
return limit, offset, page
}
Here’s rest/user.go, it calls GetLimitOffset function above, get the limit and offset and send it to service code, which return results data that we’ll marshal as json data.
package rest
import (
"context"
"encoding/json"
"github.com/gin-gonic/gin"
"updated-golang-api/internal/model"
)
func (rs *restServer) GetUsers(c *gin.Context) {
limit, offset, _ := GetLimitOffset(c)
ctx := context.Background()
users, err := rs.userService.GetUsers(ctx, model.GetUsersServiceParam{
Limit: limit,
Offset: offset,
})
if err != nil {
c.String(500, "unexpected error")
}
result, err := json.Marshal(users)
if err != nil {
c.String(500, "unexpected error")
}
c.String(200, string(result))
}
Don’t forget to add the route:
func (rs *restServer) Route(r *gin.Engine) {
r.GET("/ping", rs.Ping)
r.GET("/users", rs.GetUsers) // <== add this
}
Now, let’s create the test for this handler, but before that, let’s generate the mock for UserService interface.
//go:generate mockgen -source=./rest.go -destination=mock/service.go -package=restmock
type UserService interface {
GetUsers(ctx context.Context, param model.GetUsersServiceParam) ([]model.GetUsersServiceResult, error)
}
// ... rest of the code
Or we can just run this in terminal: go generate -run "mockgen -source=./rest.go -destination=mock/service.go -package=restmock", mocks will be generated in rest/mock/ directory. Everytime we add, update or delete a function, mocks need to be regenerated.
Let’s use the mock in the test, create a new test file rest/user_test.go, with the package name rest_test.
ctrl := gomock.NewController(t)
defer ctrl.Finish()
userMock := restmock.NewMockUserService(ctrl)
We also mock the call to service code with userServiceMock.EXPECT().GetUsers.
Here’s how it looks like:
package rest_test
import (
"github.com/gin-gonic/gin"
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"net/http"
"net/http/httptest"
"testing"
"updated-golang-api/internal/model"
"updated-golang-api/internal/rest"
restmock "updated-golang-api/internal/rest/mock"
)
func TestRestServer_GetUsers(t *testing.T) {
ctrl := gomock.NewController(t)
defer ctrl.Finish()
userServiceMock := restmock.NewMockUserService(ctrl)
userServiceMock.EXPECT().GetUsers(gomock.Any(), model.GetUsersServiceParam{
Limit: 10,
Offset: 0,
}).Return([]model.GetUsersServiceResult{
{
ID: 1,
FirstName: "alpha",
LastName: "beta",
Email: "alpha@email.com",
},
{
ID: 2,
FirstName: "gamma",
LastName: "zona",
Email: "gamma@email.com",
},
}, nil)
router := gin.Default()
rs := rest.NewServer(userServiceMock)
rs.Route(router)
w := httptest.NewRecorder()
req, _ := http.NewRequest("GET", "/users", nil)
router.ServeHTTP(w, req)
expectedBody := `[{"ID":1,"FirstName":"alpha","LastName":"beta","Email":"alpha@email.com"},{"ID":2,"FirstName":"gamma","LastName":"zona","Email":"gamma@email.com"}]`
assert.Equal(t, 200, w.Code)
assert.Equal(t, expectedBody, w.Body.String())
}
Updating the Server Command
After this, update the server command to create instances for user repo and user service.
First, create database connection from config:
dbConn, err := database.Conn(config.DBConnectionCfg())
Then, create user repo, send database connection.
userRepo := user.NewRepo(dbConn)
Then, call user repo’s NewService function with the userRepo parameter.
userService := user.NewService(userRepo)
Then, call rest’s NewServer function with the userService parameter.
rs := rest.NewServer(userService)
Here’s how it looks like:
func launchRestServer() {
log.Println("Running the web server")
router := gin.New()
dbConn, err := database.Conn(config.DBConnectionCfg())
if err != nil {
log.Panic("cannot open db connection:", err)
}
userRepo := user.NewRepo(dbConn)
userService := user.NewService(userRepo)
rs := rest.NewServer(userService)
rs.Route(router)
router.Run(":8080")
}
Update Regressed Ping Test
Don’t forget to update the Ping test above because we update the rest NewServer function to receive UserService.
// ... rest of the code
ctrl := gomock.NewController(t)
defer ctrl.Finish()
userServiceMock := restmock.NewMockUserService(ctrl)
router := gin.Default()
rs := rest.NewServer(userServiceMock)
// ... rest of the code
After this, we can run all the test files, to make sure that they’re all correct.
$ go test ./...
? updated-golang-api [no test files]
? updated-golang-api/cmd [no test files]
? updated-golang-api/cmd/migration [no test files]
? updated-golang-api/cmd/server [no test files]
? updated-golang-api/config [no test files]
? updated-golang-api/db [no test files]
? updated-golang-api/internal/model [no test files]
ok updated-golang-api/internal/rest 0.005s
? updated-golang-api/internal/rest/mock [no test files]
ok updated-golang-api/internal/user (cached)
? updated-golang-api/internal/user/mock [no test files]
Makefile
So far, we’ve made an example of web application that is testable and take some attentions to separation concerns. Now we would like to create a makefile, one of its purpose is making every command available more visible to us.
.PHONY: server
export GO111MODULE=on
test:
go test ./...
server:
go run main.go server
migrations-up-all:
go run main.go migrations 9999
migrations-rollback:
go run main.go migrations -- -1
Now we can just run make test in order to run all the tests we’ve written, and make server to start the web server.
Let’s Manually Test It
docker run -itd --rm -p 3306:3306 -p 33060:33060 -e MYSQL_ROOT_HOST='%' -e MYSQL_ROOT_PASSWORD='chocolatecake' -e MYSQL_DATABASE=sampleapp --name mysql-8.0.23 mysql/mysql-server:8.0.23
make migrations-up-all
make server
Then insert some data to user table:
INSERT INTO sampleapp.`user` (firstname,lastname,email) VALUES
('test','test2','test@email.com'),
('test3','test4','test2@email.com');
Open new terminal, and run this curl command, if everything is ok, we will see the json result fetched from the database.
$ curl localhost:8080/users
[{"ID":1,"FirstName":"test","LastName":"test2","Email":"test@email.com"},{"ID":2,"FirstName":"test3","LastName":"test4","Email":"test2@email.com"}]