Dwi Wahyudi

Senior Software Engineer (Ruby, Golang, Java)

In this article, we’re going to build a Golang web application, but this time with the help of Cobra, Viper and DB migration.

Overview

In this article, we’re going to leverage some popular tools for enhancing our web application, notably Cobra, Viper and DB migration tool.

Initializing the Application

Let’s start by initializing a new Golang app, and install some dependencies/libraries. The web app will serve a http request that will return a json response, the json values will be fetched from the database (MySQL 8).

mkdir updated-golang-api
cd updated-golang-api
go mod init updated-golang-api
go get -u github.com/spf13/cobra
go get -u github.com/spf13/viper
go get -u github.com/golang-migrate/migrate

Root And Server Commands

Let’s now create cmd directory (cmd package), where we will place a root.go file, but before we go to that file, let’s create a new go file under cmd/server (server package), we’ll give it a name server.go, this will contain command configurations for our web server:

var ServerCmd = &cobra.Command{
	Use:   "server",
	Short: "A very simple web app server",
	Long:  `A sample restful web app communicating with mysql db.`,
	Run: func(cmd *cobra.Command, args []string) {
		launchRestServer()
	},
}

func Execute() {
	if err := ServerCmd.Execute(); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

func launchRestServer() {
	log.Println("Running the web server")

	http.HandleFunc("/ping", func(w http.ResponseWriter, req *http.Request) {
		fmt.Fprintf(w, "pong\n")
	})

	http.ListenAndServe(":8080", nil)
}

Nothing really special, except for the &cobra.Command struct, it contains some fields, Use, Short, Long and Run.

  • Use is the command we will use in command line later. This means we will run the server with go run main.go server.
  • Short is short description of the command. This will be shown in help output.
  • Long is long description of the command. This will be shown when we run help <command>.
  • Run is the function we want to run for the command. In above code Run calls launchRestServer() which in turn will create a simple web server.

There are other fields like Version, Args, Aliases, etc that we can use on that struct.

Every command will need to have Execute() function, this is the convention used by Cobra library. This is important for registering this command. Registered command will be executed by calling this function, which in turns call the function of Run field. After registering the command, we can use it with the value of Use field. In this case, Use field above is server. When we use server we will call launchRestServer(). In other words, running go run main.go server will indirectly call launchRestServer().

Now let’s deal with the cmd/root.go, we will add the server cmd above. Make sure that ServerCmd on server package above is public, so we can add it here:

import (
    // ... other imports
    "updated-golang-api/cmd/server"
)

var rootCmd = &cobra.Command{
	Short: "A very simple web app",
	Long:  `A very simple web app, it has server and db migration`,
}

func Execute() {
	rootCmd.AddCommand(server.ServerCmd)
	if err := rootCmd.Execute(); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

Above command code has no Run field on the cobra.Command struct, this means go run main.go (root command) will not do anything other than printing some information message about the app.

AddCommand() function above, registers the server command to the root command.

With these setups in place, when we want to start the server, we can just run go run main.go server, or ./updated-golang-api server (when from compiled binary).

Main Function

The main function should be small, it calls the rootCmd above.

package main

import "updated-golang-api/cmd"

func main() {
	cmd.Execute()
}

From here, let’s run the server with either go build && ./updated-golang-api server or go run main.go server. We can then visit localhost:8080/ping to check if our server runs correctly.

As stated above running the app without server, in other words: the root command, will just print out standard CLI information message (this is nicely provided by Cobra by default).

$ go run main.go
A very simple web app, it has server and db migration

Usage:
   [command]

Available Commands:
  help        Help about any command
  server      A very simple web app server

Flags:
  -h, --help   help for this command

Use " [command] --help" for more information about a command.

As we can see above, Cobra gives us nice information regarding the application including the registered commands.

If we want it (the root command) to do something, we’ll need to fill Run field with a function for the rootCmd cobra.Command struct, just like we did for the server.

Configurations and Secrets

Before we go to database migrations, let’s prepare the configurations first using Viper. These configurations are needed ultimately for database connection. Let’s create the config directory, this will contain config.go which will read configurations/secrets from config/.env (make sure to put this in .gitignore as well).

Here’s how the .env file would look like, this setup should match with our MySQL server setup in the next section.

DBHOST=localhost
DBPORT=3306
DBUSERNAME=root
DBPASSWORD=chocolatecake
DBNAME=sampleapp

Now, onto the Golang code to read this .env file. Viper is very easy to use but offers a lots of awesome features for us to use. It offers us reading configs from json, yaml, or even toml file, but in this article, we’re going to use .env file.

Here’s how the config.go file will look like:

package config

import (
	"fmt"
	"github.com/spf13/viper"
)

type DBConnection struct {
	Host         string
	Port         string
	Username     string
	Password     string
	DatabaseName string
}

func DBConnectionCfg() DBConnection {
	viper.SetConfigFile("config/.env")

	err := viper.ReadInConfig()
	if err != nil {
		panic(fmt.Errorf("Fatal error config file: %s \n", err))
	}

	return DBConnection{
		Host:         viper.Get("DBHOST").(string),
		Port:         viper.Get("DBPORT").(string),
		Username:     viper.Get("DBUSERNAME").(string),
		Password:     viper.Get("DBPASSWORD").(string),
		DatabaseName: viper.Get("DBNAME").(string),
	}
}

This code is straight-forward, we just read the .env file, read the configuration one by one and assign it to the DBConnection struct. We should make the config as lightweight as possible, only pass around the necessary config for each function call.

DB Migration Command

Now, let’s take a look at the db migration. We’re going to use MySQL 8.0.23 in this article. Let’s get it from the Docker repo (and set a root password for it):

docker run -itd --rm -p 3306:3306 -p 33060:33060 -e MYSQL_ROOT_HOST='%' -e MYSQL_ROOT_PASSWORD='chocolatecake' -e MYSQL_DATABASE=sampleapp --name mysql-8.0.23 mysql/mysql-server:8.0.23

Make sure that the password match with the password we specified in config/.env above. This docker command will also create a new database sampleapp.

Then create a new directory migrations/, this is where our database migration files will reside. Let’s create some:

For the up migration, 1_create_user.up.sql:

CREATE TABLE user (
    id INT(11) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    firstname VARCHAR(200) NOT NULL,
    lastname VARCHAR(200) NOT NULL,
    email VARCHAR(200)
);

For the down migration, 1_create_user.down.sql:

DROP TABLE user;

Before we write DB migration command, let’s create db connection code db/conn.go. This will use the DB connection configurations above. It’s very simple:

package db

import (
	"database/sql"
	"fmt"
	"updated-golang-api/config"
)

func Conn(connection config.DBConnection) (*sql.DB, error) {
	host := connection.Host
	port := connection.Port
	username := connection.Username
	password := connection.Password
	databaseName := connection.DatabaseName

	dataSourceName := fmt.Sprintf("%v:%v@tcp(%v:%v)/%v", username, password, host, port, databaseName)

	db, err := sql.Open("mysql", dataSourceName)

	if err != nil {
		return nil, err
	}

	return db, nil
}

We’ve written our db connection code above, let’s use this for our db migration command.

Creating the DB migration command will be like this:

package migration

import (
	"fmt"
	"github.com/spf13/cobra"
	"log"
	"os"
	"strconv"
	"updated-golang-api/config"
	database "updated-golang-api/db"

	_ "github.com/go-sql-driver/mysql"
	"github.com/golang-migrate/migrate"
	"github.com/golang-migrate/migrate/database/mysql"
	_ "github.com/golang-migrate/migrate/source/file"
)

var MigrationCmd = &cobra.Command{
	Use:   "migrations",
	Short: "DB Migration for the app",
	Long:  `DB Migration for the app.`,
	Run: func(cmd *cobra.Command, args []string) {
		launchDBMigration(args)
	},
}

func Execute() {
	if err := MigrationCmd.Execute(); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

func launchDBMigration(args []string) {
	db, err := database.Conn(config.DBConnectionCfg())
	if err != nil {
		log.Panic("cannot open db connection:", err)
	}

	driver, err := mysql.WithInstance(db, &mysql.Config{})
	if err != nil {
		log.Panic("cannot setup driver:", err)
	}

	m, err := migrate.NewWithDatabaseInstance(
		"file://migrations",
		"mysql",
		driver,
	)
	if err != nil {
		log.Panic("cannot setup migrations:", err)
	}

	var steps int
	if len(args) > 0 {
		stepArg := args[0]
		steps, err = strconv.Atoi(stepArg)
		if err != nil {
			log.Panic("invalid arg for migrations:", err)
		}
	} else {
		steps = 99999
	}

	m.Steps(steps)
}

Some explanations about this command:

  • This command will use migrations, which means, in order to run db migrations, we need to run go run main.go migrations.
  • This command will call launchDBMigration along with the args from the command line. This is needed for db migrations steps. Steps here means: how many migrations scripts we want to run. We only have 1 so far, so specifying the args with 1 or more is not a problem.
    • If we want to do up migrations we specify the args to be > 0, ex: go run main.go migrations 3 or go run.main.go 99.
    • For down migrations (rolling back), negative number must be specified instead, ex: go run main.go migrations -- -3 or go run main.go migrations -- -99.
    • In above code, if we don’t specify the args, we will just do up migrations for all scripts.
  • Inside the launchDBMigration function, we open the database with Conn() function we already wrote before. We then set up the driver, and then tell the db migration library to look for migration scripts inside the migrations directory with NewWithDatabaseInstance function.
  • We then execute the db migration with the specified steps. This is done with Steps function.

Since we only have 1 migration script so far, we only need 1 step, running go run main.go migrations 1 will be just fine. If somehow we want to roll it back, we can run it with go run main.go migrations -- -1.

If we look into the sampleapp database, there are 2 tables now: user and schema_migrations. schema_migrations table is there to keep track what db migrations is already up, it is used by database migration library to find out which migration script to run. If we roll back a migration, such migration will be deleted from schema_migrations table.

Here’s the code: https://github.com/dwahyudi/updated-golang-api.

In the next part we’re going to implement our web server to serve data from the database. We’re going to write some unit tests for it as well.