In this article, we’re going to demonstrate an application that serves as an API caching service.

Overview

This application will serve as a service that operates between clients and third party API services. Clients can get data from such third party API services directly, but we’re going to provide them with a service layer with caching and storage. Why do we need this? In some cases, the third party API services take long times to process, and sometimes because they’re not in our control, we have no idea if any service is down/unavailable, plus some third party API services impose some requests limits/quota, if their data don’t change too much in some periods of times, caching will be a good idea.

By providing a service inbetween, we can have better reliability and control over data fetched from those third party services. In real-world, this application can range from caching a third party gold price API to weather information API. Here in this article, we’re going to create an application that provides Marvel™ characters data from its official API service. We’re going to provides:

1. All Characters IDs
2. Character data based on provided ID

Marvel™ API impose some quota/limits, so again, this service is even more applicable. We’re going to cache the data with our in-memory storage.

The reason for in-memory cache (instead of redis or memcached):

• It is simpler (no need to run additional programs to store the data), I assume this app will run in a single Golang application.
• No need to serialize / deserialize the data to and from redis/memcached.
• No network cost.
• The cons of using this approach is when we want to run this app in multiple instances of Golang apps (then redis/memcached/RDBMS should be chosen) and when this application is restarted (caches will be gone, because they’re stored in memory).

Getting Started

Let’s create a new Golang application.

mkdir marvel-app
cd marvel-app
go mod init marvel-app

We’re going to use gin as our web server framework to serve the clients: go get -u github.com/gin-gonic/gin

Before we continue, we have to go to Marvel™ developer site and register a new account so that we can get API keys (we’ll get 2 keys, 1 public key and 1 private key) needed to interact with their API. The documentation for the API can be found here and register a new account so that we can get API keys (we’ll get 2 keys, 1 public key and 1 private key) needed to interact with their API. The documentation for the API can be found here, and register a new account so that we can get API keys (we’ll get 2 keys, 1 public key and 1 private key) needed to interact with their API. The documentation for the API can be found here: <https://developer.marvel.com/docs").

First, let’s create a util package, our first utilities would be like this:

API Request

If we look at the Marvel™ developer documentation, our request will need to be like this:

1. For characters lists: https://gateway.marvel.com:443/v1/public/characters?ts=0000&apikey=apikey&hash=hash&limit=99&offset=99
2. For character data based on ID: https://gateway.marvel.com:443/v1/public/characters/1?ts=0000&apikey=apikey&hash=hash

• ts can be a random string.
• apikey is our public key.
• hash is MD5 of ts + private key + public key.

Since Marvel™ only receive requests from url and query params, we will demonstrate this with a simple http request function:

func LogPanic(err error) {
	if err != nil {
		log.Panic(err)
	}
}

func RandomString(length int) string {
	var letters = []rune("abcdefghijklmnopqrstuvwxyz0123456789")

	s := make([]rune, length)
	for i := range s {
		s[i] = letters[rand.Intn(len(letters))]
	}
	return string(s)
}

func MD5Encode(text string) string {
	hash := md5.New()
	hash.Write([]byte(text))
	return hex.EncodeToString(hash.Sum(nil))
}

func SimpleAPIGet(url string) string {
	method := "GET"

	client := &http.Client{}
	req, err := http.NewRequest(method, url, nil)
	LogPanic(err)

	res, err := client.Do(req)
	LogPanic(err)

	body, err := ioutil.ReadAll(res.Body)
	LogPanic(err)

	defer res.Body.Close()

	return string(body)
}

Our SimpleAPIGet function will just simply GET request to a specified url, and return a string response. RandomString function will be used to generate ts for our request. MD5Encode function will MD5-encode a string, this is used to generate our hash.

Now, we’ll need a way to build our request url strings.

Characters List URL Builder

For requesting characters list, Marvel™ developer documentation specifies offset and limit for listing their characters, we’re going to use it as our parameters to our url builder function, we just merge everything we have into a single request url. At the end of this function, we’re going to call SimpleAPIGet function from util package above. SimpleAPIGet will request to Marvel™ API using the built url string, use 1 credit of Marvel™ API, and return a response for us to cache and serve it to the clients.

In real-world application, our envs should be managed by something more robust like viper https://github.com/spf13/viper, but for demonstration purpose, os.Getenv will suffice.

func CharacterListWithOffsetAndLimit(offset int, limit int) string {
	ts := util.RandomString(8)
	publicKey := os.Getenv("MARVEL_PUBLIC_KEY")
	privateKey := os.Getenv("MARVEL_PRIVATE_KEY")
	hash := util.MD5Encode(ts + privateKey + publicKey)
	url := "https://gateway.marvel.com:443/v1/public/characters?ts=" + ts +
		"&apikey=" + publicKey +
		"&hash=" + hash +
		"&limit=" + strconv.Itoa(limit) +
		"&offset=" + strconv.Itoa(offset)

	log.Println("Using 1 credit of Marvel API.")
	return util.SimpleAPIGet(url)
}

Let’s also create some structs to model the response from this function (will be conveniently used by gin):

type CharactersResponse struct {
	Code   int    `json:"code"`
	Status string `json:"status"`
	Data   CharactersResponseData
}

type CharactersResponseData struct {
	Count   int `json:"count"`
	Total   int `json:"total"`
	Results []Character
}

Character Data By ID URL Builder

Now let’s create another url builder for getting a single character data based on the specified ID. It will look identical with CharacterListWithOffsetAndLimit function above, but here we specify a single ID as parameter. This function will return a single character data. This can be cached as well.

func GetCharacterByID(ID int) string {
	ts := util.RandomString(8)
	publicKey := os.Getenv("MARVEL_PUBLIC_KEY")
	privateKey := os.Getenv("MARVEL_PRIVATE_KEY")
	hash := util.MD5Encode(ts + privateKey + publicKey)
	url := "https://gateway.marvel.com:443/v1/public/characters/" + strconv.Itoa(ID) +
		"?ts=" + ts +
		"&apikey=" + publicKey +
		"&hash=" + hash

	log.Println("Using 1 credit of Marvel API.")
	return util.SimpleAPIGet(url)
}

And here’s the struct for modelling the response (will be conveniently used by gin):

type Character struct {
	ID          int    `json:"id"`
	Name        string `json:"name"`
	Description string `json:"description"`
}

Caching

Now we already have 2 functions for interacting with Marvel™ API, one for getting characters list, one for getting a character data based on an ID. It’s time to create a repository package for the characters, this package will be the central piece of code for our characters storage, we will use the caching mechanism here.

But before going into the repository package, we need to setup our cache.

For our in-memory cache, we’ll use: https://github.com/patrickmn/go-cache. This library is very easy to use, just call New function with expiration and cleaning duration. The in-memory cache instance and data will need to live throughout the entire lifetime of our app, thus we’ll need to make it as a singleton, that is instantiated only once and the instantiation will be done when we run the service/application, calling the function multiple times will get us the identical cache instance. Golang provides us with sync.Once type and sync.Do method for us to perform such instantiation.

In later section, we’re going to define cache keys needed.

package util

import (
	"sync"
	"time"

	"github.com/patrickmn/go-cache"
)

var (
	singleton *cache.Cache
	once      sync.Once
)

func Caching() *cache.Cache {
	once.Do(func() {
		singleton = cache.New(6*time.Hour, 12*time.Hour)
	})
	return singleton
}

Characters Repository

After setting up the cache, we’ll create a repository package, call this repo package. We’ll publish 2 public functions: AllCharacterIds and GetCharacterById. These functions will be called by our API server. Our first task: return all marvel characters IDs.

Marvel™ has more than 3800 characters stored at the API at the moment, and Marvel™ API will only return 100 characters data in a single request. And after initial observation, a single request to Marvel™ API takes quite some times to complete, around 2-4 seconds. Caching is a sure strategy here, but getting 38 requests sequentially will require lots of times. We’ll need to do the requests asynchrounously (creating multiple requests at the same time to Marvel™ API), but we will need to wait for all of those requests to complete, because we want to gather all of the IDs. This is a perfect job for goroutines, sync.Mutex and sync.WaitGroup.

Goroutines will do bulk requests to Marvel™ API, while sync.WaitGroup will wait for those concurrent requests to complete, sync.Mutex will synchronize access to a struct containing our character IDs (only one goroutine will be able to access the struct), without this mutex, reading and writing to the character IDs data will be in a race condition.

type SyncCharacterIds struct {
	mutex   sync.Mutex
	charIds []int
}

Before we initialize our requests call to Marvel™ API, we need to know, how many characters there are currently, this is important to know our limit and offset parameters in each iteration and how many requests we should make. Luckily we can find that in a single request to Marvel™ API public/characters endpoint, it tells us the total number of characters. Let’s call this with offset of 0 and limit of 1, and get the Total data.

func obtainCharactersTotalNumber() int {
	jsonResult := apirequest.CharacterListWithOffsetAndLimit(0, 1)
	charactersResponse := apirequest.CharactersResponse{}
	json.Unmarshal([]byte(jsonResult), &charactersResponse)
	return charactersResponse.Data.Total
}

Marvel™ API will only return 100 characters data per single request, so we’ll need to set it as a constant:

const CharacterRequestLimit = 100

With the preparations we have until now, we can create some functions to concurrently call Marvel™ API, and collect the IDs.

• asyncCharactersRequests receives an int number (the total of characters) and charIds, which is a pointer to an empty SyncCharacterIds. Here, without waiting we call processCharacterIdsPerAPICall with a certain offset (that is incremented by CharacterRequestLimit, which is 100). If there are 3800 characters, there will be 38 calls to processCharacterIdsPerAPICall concurrently, per each iteration, we’re incrementing the waitGroups by 1.
• processCharacterIdsPerAPICall mission is to call the Marvel™ API with certain offset and limit, get the response, parse it to CharactersResponse struct. From there, it has json element of Data and then Results. Results contains the 100 characters data (including their IDs). For each character, we’re going to save each ID, collect them to a temporary collectedIds, since we now have characters data, we will aggressively cache them, one by one, call the util.Caching() and set the cache value with our predefined key like this char-999. And then after iterating each character data, we send those IDs, to syncAppend function. Don’t forget to decrement the waitGroup.
• syncAppend function is a simple function that receive a pointer to the struct containing whole character IDs, and the IDs to be appended. It is a simple function that lock such struct containing the character IDs, appending the new character IDs, and then unlocking it for other goroutines to use. syncAppend has some consequences on blocking each processCharacterIdsPerAPICall completion, but it is intended, we don’t want the waitGroup to immediately “done” before the 100 character IDs for each processCharacterIdsPerAPICall to be successfully appended.

const CharactersDetailCacheKey = "char-"

func asyncCharactersRequests(total int, charIds *SyncCharacterIds) {
	var waitGroups sync.WaitGroup

	for offset := 0; offset < total; offset += CharacterRequestLimit {
		waitGroups.Add(1)
		go processCharacterIdsPerAPICall(offset, charIds, &waitGroups)
	}

	waitGroups.Wait()
}

func processCharacterIdsPerAPICall(offset int, charIds *SyncCharacterIds, waitGroup *sync.WaitGroup) {
	defer waitGroup.Done()

	jsonResult := apirequest.CharacterListWithOffsetAndLimit(offset, CharacterRequestLimit)
	charactersResponse := apirequest.CharactersResponse{}
	json.Unmarshal([]byte(jsonResult), &charactersResponse)
	results := charactersResponse.Data.Results

	var collectedIds []int
	for _, character := range results {
		collectedIds = append(collectedIds, character.ID)

		// Aggressive caching.
		cch := util.Caching()
		key := CharactersDetailCacheKey + strconv.Itoa(character.ID)
		cch.Set(key, character, 1*time.Hour)
	}

	syncAppend(charIds, collectedIds)
}

func syncAppend(charIds *SyncCharacterIds, characterID []int) {
	charIds.mutex.Lock()
	charIds.charIds = append(charIds.charIds, characterID...)
	charIds.mutex.Unlock()
}

After this, let’s create the function that call asyncCharactersRequests, this is the public function that our web server will need to call in order to serve the data to our clients.

In this function, we will need to get the whole character IDs from asyncCharactersRequests, this is the layer where we read and write our cache of character IDs. If cache is found, we just use it from the cache, if not, we will do concurrent requests to Marvel™ API and write the cache from it.

const CharactersIdsCacheKey = "charsIds"

func AllCharacterIds() []int {
	var charIds SyncCharacterIds

	cch := util.Caching()

	cachedIds, found := cch.Get(CharactersIdsCacheKey)

	if found {
		charIds.charIds = cachedIds.([]int)
	} else {
		total := obtainCharactersTotalNumber()
		asyncCharactersRequests(total, &charIds)
		cch.Set(CharactersIdsCacheKey, charIds.charIds, cache.DefaultExpiration)
	}

	return charIds.charIds
}

Now, let’s move to another repo, this time for getting character data from a specified ID, this can be simply written like below code. Again we utilize the in-memory cache so if such character ID data present in our cache we will just use it, if not, we will call the API for requesting a single character data, and cache it.

Note that AllCharacterIds above, when called, will iterate all of characters data present in Marvel™ API, and cache the character data one by one using the same key. So calling it will ultimately cache all of character ID data for GetCharacterById to use.

func GetCharacterById(ID int) apirequest.Character {
	cch := util.Caching()
	key := CharactersDetailCacheKey + strconv.Itoa(ID)

	cachedChar, found := cch.Get(key)
	var char apirequest.Character
	if found {
		char = cachedChar.(apirequest.Character)
	} else {
		jsonResult := apirequest.GetCharacterByID(ID)
		charactersResponse := apirequest.CharactersResponse{}
		json.Unmarshal([]byte(jsonResult), &charactersResponse)

		results := charactersResponse.Data.Results

		if len(results) == 1 {
			char = charactersResponse.Data.Results[0]
		}

		cch.Set(key, char, cache.DefaultExpiration)
	}

	return char
}

API Server

With those repository codes in place, we can now create the server endpoint to serve our clients.

func CharacterList(c *gin.Context) {
	ids := repo.AllCharacterIds()
	c.JSON(http.StatusOK, ids)
}

func CharacterDetail(c *gin.Context) {
	idFromParam := c.Param("id")
	id, err := strconv.Atoi(idFromParam)

	if err != nil {
		c.JSON(http.StatusBadRequest, "Invalid ID")
		return
	}

	character := repo.GetCharacterById(id)
	var status int
	var response interface{}

	if character.ID == 0 {
		status = http.StatusNotFound
		response = "Character with that ID doesn't exist"
	} else {
		status = http.StatusOK
		response = character
	}

	c.JSON(status, response)
}

And here’s our main function:

func main() {
	log.Println("Starting Marvel")
	startWeb()
}

func startWeb() {
	web := gin.Default()
	setAPIRoutes(web)
	web.Run()
}

func setAPIRoutes(web *gin.Engine) *gin.Engine {
	web.GET("/ping", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "pong",
		})
	})

	web.GET("/characters", apiserve.CharacterList)
	web.GET("/characters/:id", apiserve.CharacterDetail)

	return web
}

The code for this article can be found here: https://github.com/dwahyudi/marvel-app.