How to Build a URL Shortener in Go with Redis

How to Build a URL Shortener in Go with Redis
Full Code

Let’s admit it. We all have used a shortened URL at least once.

URL Shortener is an application that allows users to shorten unwieldy links into better-looking URLs. This is useful for a variety of purposes, such as sharing links on social media or simply sending links via email.

To build such an application, we want a powerful and efficient programming language. Go was designed by Google engineers with the needs of modern developers in mind. It’s a compiled language that runs quickly and efficiently on a variety of platforms. Plus, its syntax is clean and easy to read, making it a great choice for those just starting with coding.

This article will explore how to write a URL shortener in the Go programming language and Redis as a storing mechanism.

Part I: Initial Setup 🛠

We will need Go 1.14+ installed in our system. You can check the Go version in your terminal by writing go version. If you still don’t have Go on your system, you can easily download one with brew install go using Homebrew. Or follow the official documentation on how to download and install Go.

Let’s initialize the project.

go mod init go-redis-url-shortener

📝 This will create go.mod file in the project folder.

After successfully initializing our project, the next step is creating the main.go file and add the following code.

package main

import "fmt"

func main() {
	fmt.Printf("Welcome to Go URL Shortener with Redis !🚀")
}

If we run go run main.go in the terminal inside the project directory (where our main.go file is located) we should see this output:

Welcome to Go URL Shortener with Redis !🚀

Amazing! The project setup was successful.

Now let’s add Echo, an open-source Go web application framework, so we can easily build our web application. For more details visit their website.

go get github.com/labstack/echo/v4

📝 This will create go.sum file in the project folder.

All right, all right. Now we are ready to start the web server and return some data in JSON format. Update the main.go file to reflect these changes.

package main

import (
	"github.com/labstack/echo/v4"
	"net/http"
)

func main() {
	e := echo.New()
	e.GET("/", func(c echo.Context) error {
		return c.JSON(http.StatusOK, map[string]interface{}{
			"message": "Welcome to Go URL Shortener with Redis !🚀",
		})
	})
	e.Logger.Fatal(e.Start(":1323"))
}

Run again the main.go file (command: go run main.go ) and open http://localhost:1323/ in your browser or other rest client tool. The output should look like this.

{
  "message": "Welcome to Go URL Shortener with Redis !🚀"
}

If you got this JSON response, it’s time to celebrate. This means that our setup was done successfully. 👏

Part II: Algorithm For Generating a Short Link 🧮

In the previous part, we manage to set up the server using Golang Echo. In this part, we are going to work on the algorithm that creates short URL from long URL.

For the implementation, we are going to use two main schemes - a hash function and a binary-to-text encoding algorithm.

First, we will need to create two files - shortener.go and shortener_test.go, inside a folder named shortener

├── go.mod
├── go.sum
├── main.go
└── shortener
   ├── shortener.go
   └── shortener_test.go

Then, we open shortener.go file and add the SHA-256 code below. We will use the Golang built-in implementation of this hash function. SHA-256 is a patented cryptographic hash function that outputs a value that is 256 bits long.

package shortener

import (
	"crypto/sha256"
)

func sha256Of(input string) []byte {
	algorithm := sha256.New()
	algorithm.Write([]byte(input))
	return algorithm.Sum(nil)
}

Next thing is to add encoding. In this tutorial, we will go with BASE58 one. I can see you re-reading the name of the encoding and thinking “58? 🤔 Did the author make a typo?”. Well no, this encoding is a modification of the classic BASE64. it makes it easier for humans to read the result by eliminating 0 (zero), O (capital o), I (capital i), l (lower L), + (plus), and / (slash) characters to avoid confusion. Well, let’s add then the BASE58 encoding.

First things first - let’s load the BASE58 dependency library.

go get github.com/itchyny/base58-go/cmd/base58

Now, we’ll extend the code with another function.

func base58Encoded(bytes []byte) string {
	encoding := base58.BitcoinEncoding
	encoded, err := encoding.Encode(bytes)
	if err != nil {
		fmt.Println(err.Error())
		os.Exit(1)
	}
	return string(encoded)
}

Since we have our two main building blocks (hashing and encoding) already in place, the final algorithm is easy.

  • Hashing initialLink + userId with SHA-256. We use userId for preventing similar shortened URLs to different users in case they want to shorten the same URL.
  • Derive a big integer number from the hash bytes generated during the hashing.
  • Apply BASE58 on the derived big integer value and pick the first 8 characters.
func GenerateShortURL(initialLink string, userId string) string {
	urlHashBytes := sha256Of(initialLink + userId)
	generatedNumber := new(big.Int).SetBytes(urlHashBytes).Uint64()
	finalString := base58Encoded([]byte(fmt.Sprintf("%d", generatedNumber)))
	return finalString[:8]
}

Our algorithm is done. Let’s do some unit testing to be sure that it works as expected. We already created our test file shortener_test.go, so now we only need to add testify toolkit

go get github.com/stretchr/testify/assert

and some pieces of code.

package shortener

import (
	"github.com/stretchr/testify/assert"
	"testing"
)

func TestShortLinkGenerator(t *testing.T) {
	initialLink := "https://go.dev/doc/tutorial/getting-started"
	userId := "UwQPr3aIf9dM5x7r"
	shortLink := GenerateShortURL(initialLink, userId)

	assert.Equal(t, shortLink, "dysg5Fas")
}

With this, we are done with the implementation of the algorithm for generating a short URL. 🥳 We can execute all tests using the command go test ./... or only those inside the shortener folder with go test ./shortener assuming we are at the root of the project (on the same level with main.go)

Part III: Storage Layer 📦

In this part, we will be working on building the storage layer of our short URL application, so let’s start!

📝 If Redis is not yet installed on your computer, you can do it following the instructions for installation with respect to your operating system.

First thing first, we will install Redis client for Golang

go get github.com/go-redis/redis/v9

then we create the store folder in the project, and two empty Go files: store.go and store_test.go

├── go.mod
├── go.sum
├── main.go
└── shortener
   ├── shortener.go
   └── shortener_test.go
└── store
   ├── store.go
   └── store_test.go

Now we can define wrapper structs and initialize the store service, in this case our Redis client.

package store

import (
	"context"
	"fmt"
	"github.com/go-redis/redis/v9"
	"time"
)

// StorageService is struct wrapper around raw Redis client
type StorageService struct {
	redisClient *redis.Client
}

// Top level declarations for the storeService and Redis context
var (
	storeService = &StorageService{}
	ctx          = context.Background()
)

const CacheDuration = 6 * time.Hour

// InitializeStore is initializing the store service and return a store pointer
func InitializeStore() *StorageService {
	rdb := redis.NewClient(&redis.Options{
		Addr:     "localhost:6379",
		Password: "", // no password set
		DB:       0,  // use default DB
	})

	pong, err := rdb.Ping(ctx).Result()
	if err != nil {
		panic(fmt.Sprintf("Error init Redis: %v", err))
	}

	fmt.Printf("\nRedis started successfully: pong message = {%s}", pong)
	storeService.redisClient = rdb
	return storeService
}

Now that our data store service has successfully been initialized, it’s time to consider what storage API to offer for our shortener server.

  1. We want to be able to save the mapping between the original URL and the generated short URL.
  2. We should be able to retrieve the initial long URL once the short is provided.

As the next step let’s update store.go and implement our storage API.

func SaveURLInRedis(shortURL, originalURL string) {
	err := storeService.redisClient.Set(ctx, shortURL, originalURL, CacheDuration).Err()
	if err != nil {
		panic(fmt.Sprintf("Failed SaveURLInRedis | Error: %v - shortURL: %s - originalURL: %s\n", 
			err, shortURL, originalURL))
	}
}

func RetrieveInitialURLFromRedis(shortURL string) string {
	result, err := storeService.redisClient.Get(ctx, shortURL).Result()
	if err != nil {
		panic(fmt.Sprintf("Failed RetrieveInitialURLFromRedis | Error: %v - shortURL: %s\n", 
			err, shortURL))
	}
	return result
}

It was fairly straightforward. Great job!

Since we already created store_test.go file, we will set up the test shell first and then write unit test for the storage APIs.

package store

import (
	"github.com/stretchr/testify/assert"
	"testing"
)

var testStoreService = &StorageService{}

func init() {
	testStoreService = InitializeStore()
}

func TestStoreInit(t *testing.T) {
	assert.True(t, testStoreService.redisClient != nil)
}

func TestInsertionAndRetrieval(t *testing.T) {
	initialLink := "https://go.dev/doc/tutorial/getting-started"
	shortURL := "dysg5Fas"

	// Persist data mapping
	SaveURLInRedis(shortURL, initialLink)

	// Retrieve initial URL
	retrievedUrl := RetrieveInitialURLFromRedis(shortURL)
	assert.Equal(t, initialLink, retrievedUrl)
}

📝 Don’t forget to run go test ./... to check if all your tests work as expected (✅).

As our storage service is set up, we will expose a Rest API endpoint for encoding and decoding the URLs in the next part.

Part IV: Handlers and Endpoints 🧶

Now it’s time to put the components we built previously to good use, so during this part, we will be making sure those previously built components work together as expected.

We are going to build two endpoints to our API service :

  • One endpoint that will be used to generate a short URL and return it, when the initial long URL is provided. /encode
  • The other one will be used to return the original long URL when the short URL is provided. /decode/:short-url

Let’s go ahead and create the handler package and define our handler’s functions there. Create a folder called handler and create a file called handlers.go inside the folder. After that our project directory should look like the tree below :

├── go.mod
├── go.sum
├── main.go
└── shortener
   ├── shortener.go
   └── shortener_test.go
└── store
   ├── store.go
   └── store_test.go
└── handler
   └── handlers.go

Now let’s define and implement our handlers.

We will be starting with implementing the CreateShortURL() handler function, this should be very straightforward :

  1. We will get the creation request body, parse it and extract the initial longURL and userId.
  2. Call our shortener.GenerateShortURL that we implemented in PART II and generate our shortened hash.
  3. Finally store the mapping of our output hash / shortURL with the initial longURL, here, we will be using the store.SaveURLInRedis() we implemented back in PART III
package handler

import (
	"github.com/labstack/echo/v4"
	"go-redis-url-shortener/shortener"
	"go-redis-url-shortener/store"
)

const host = "http://localhost:1323/"

// URLCreationRequest is request model definition
type URLCreationRequest struct {
	LongURL string `json:"long_url" binding:"required"`
	UserId  string `json:"user_id" binding:"required"`
}

func CreateShortURL(c echo.Context) error {
	cr := new(URLCreationRequest)
	if err := c.Bind(cr); err != nil {
		return err
	}

	shortUrl := shortener.GenerateShortURL(cr.LongURL, cr.UserId)
	store.SaveURLInRedis(shortUrl, cr.LongURL)

	return c.JSON(200, map[string]interface{}{
		"short_url": host + shortUrl,
	})
}

The next step will be about returning the original URL, ReturnLongURL(), it will consist of :

  1. Getting the short URL from the path parameter /:shortUrl
  2. Call the store to retrieve the initial URL that corresponds to the short one provided in the path.
  3. And finally, return the long URL
func ReturnLongURL(c echo.Context) error {
	shortUrl := c.Param("short-url")
	initialUrl := store.RetrieveInitialURLFromRedis(shortUrl)
	return c.JSON(200, map[string]interface{}{
		"short_url": host + shortUrl,
		"long_url":  initialUrl,
	})
}

After implementing our handlers, we should go straight to the main.go file to add the needed endpoints and initialize the store.

📝 Also don’t forget to add the middleware library from Echo that we will use to solve our CORS issues. Middleware is a function chained in the HTTP request-response cycle with access to Echo#Context which it uses to perform a specific action, for example, logging every request or limiting the number of requests. Handler is processed in the end after all middleware are finished executing.

go get github.com/labstack/echo/v4/middleware

package main

import (
	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
	"go-redis-url-shortener/handlers"
	"go-redis-url-shortener/store"
	"net/http"
)

func main() {
	e := echo.New()

	e.Use(middleware.CORS())

	e.GET("/", func(c echo.Context) error {
		return c.JSON(http.StatusOK, map[string]interface{}{
			"message": "Welcome to Go URL Shortener with Redis !🚀",
		})
	})

	e.POST("/encode", func(c echo.Context) error {
		return handler.CreateShortURL(c)
	})

	e.GET("/decode/:short-url", func(c echo.Context) error {
		return handler.ReturnLongURL(c)
	})

	// Store initialization happens here
	store.InitializeStore()
	
	e.Logger.Fatal(e.Start(":1323"))
}

📝 In more complex applications, the endpoints should live in a separate file, but for the sake of simplicity and since they are just two endpoints, we will be having them in the main.go file

Part V: Testing The Endpoints 🧪

After completing the implementation of the handlers and the endpoints, now it’s time to test the all thing.

Start the project with go run main.go (main.go file is the entry point) and open http://localhost:1323/ in browser, or run the following command in your terminal

curl -X GET http://localhost:1323

If you got the following message that means that the server is working as expected.

{
  "message":"Welcome to Go URL Shortener with Redis !🚀"
}

Now let’s create short URL for the provided long URL. For this scenario, use the POST endpoint /encode, with the following body as an example:

{
  "long_url": "https://go.dev/doc/tutorial/getting-started",
  "user_id": "UwQPr3aIf9dM5x7r"
}

curl -X POST http://localhost:1323/encode \ 
-H 'Content-Type: application/json' \
-d '{"long_url": "https://go.dev/doc/tutorial/getting-started", "user_id": "UwQPr3aIf9dM5x7r"}'

📝 You can use any rest client you have installed locally.

The response should look like the json below:

{
  "short_url":"http://localhost:1323/dysg5Fas"
}

And finally, to get the initial long URL, use the GET endpoint /decode with the shortURL specified as path parameter:

curl -X GET http://localhost:1323/decode/dysg5Fas

As response we should get JSON with the long URL corresponding to the provided short URL:

{
  "long_url":"https://go.dev/doc/tutorial/getting-started",
  "short_url":"http://localhost:1323/dysg5Fas"
}

If everything was right, we can say that our application works flawlessly! 🎉