#! Dev's Bytes
Random programming stuff
#! Dev's Bytes
Building GoPics - part 1, The Basics

GoPics is a simple image sharing board built with Go, Redis, UIkit, and good intentions.

GoPics When I started to build my first project in Go, what I really needed was an app to be studied that was bigger than the classics gowiki and WebSocket chat examples, but still small enough to be useful as a learning resource. It came out that I was not alone, and that other people too were searching for a similar thing. After more than a year from the original stackoverflow question, the best answers to someone who is learning Go are still to read the code of gddo or go read.These two are great apps, but they are also big, they both have thousands of lines of code in their repository.

So I have started to build GoPics, a simple image sharing board that is smaller than them, but big enough to show the problems that raises in real world web app development and all the good Go capabilities in handling them. GoPics implements only the CR parts of the CRUD operations, so we can save some code but we are still able to do all the interesting stuff.

The idea is to start with something that has some simple, basic functionalities and then progressively enhanche it by adding new features and improving the code, as it happens in the real world. While I'll go on adding and modifing pieces of the app, I'll document and write what I have done here, on my blog. Every post will coincide with a major release on the app repository (so the release that concurs with the next post will be tagged as v2.0).

GoPics is built with Redis and UIkit. I've already talked about Redis and its easy of use. Redis is a gret tool and fit perfectly in the Keep It Simple philosophy of this project. I've discovered UIkit only recently, but it has become quickly my tool of choice for web interfaces (this blog is built with it). It is super simple, it has a great grid system and you can hack it with Sass.

A screenshot of GoPics user timeline

I assume that you have already completed the Tour of Go and the wiki tutorial. Otherwise, have a look at them and then come back here for GoPics.

I've talked enough, it's time to start with some real code!

Installation

Guess what? To play with GoPics, an instance of Redis needs to be up, running, and listening on the default port 6379. So, if you don't have already installed Redis, you should do it now.

To install GoPics, you can simply use go get

$ go get github.com/lucachr/gopics

To install and run the code of the repository relative to this article, cd in the source code directory, then

$ git chechout v1.0
$ go install
$ gopics

Open your browser at localhost:8080 and have fun!

Login and secure cookie with Gorilla

The code used to generate secure cookie is in the auth package. Cookies are secured with the Gorilla securecookie package. The authentication cookie contains the username of the logged user and is validated with HMAC and encrypted with AES-256. The cookie expires after a week.

package auth

import (
        "net/http"

        "github.com/gorilla/securecookie"
)

const (
    defaultAge = 60 * 60 * 24 * 7 // One week
    authCookie = "AUTH"           // The name of the auth cookie
)

// A Keyring is used to encrypt and decrypt auth cookies.
type Keyring struct {
    *securecookie.SecureCookie
}

// NewKeyring creates a new Keyring with the given hashKey and blockKey.
// The hashKey is required, it has a length of 32 or 64 bytes.
// The blockKey is optional, it has a length of 16, 24 or 32 bytes.
func NewKeyring(hashKey, blockKey []byte) Keyring {
    return Keyring{securecookie.New(hashKey, blockKey)}
}

// newCookie creates a new authentication cookie
func newCookie(value string, maxAge int) *http.Cookie {
    return &http.Cookie{
        Name:     authCookie,
        Value:    value,
        Path:     "/",
        MaxAge:   maxAge,
        HttpOnly: true,
    }

}

// SetCookie creates a new authentication cookie.
func SetCookie(w http.ResponseWriter, k Keyring, value string) error {

    encoded, err := k.Encode(authCookie, value)
    if err != nil {
        return err
    }

    http.SetCookie(w, newCookie(encoded, defaultAge))
    return nil
}

// GetCookie reads the authentication cookie from a request and returns the
// authentication credentials in the cookie
func GetCookie(r *http.Request, k Keyring) (value string, err error) {
    c, err := r.Cookie(authCookie)
    if err != nil {
        return
    }

    err = k.Decode(authCookie, c.Value, &value)
    if err != nil {
        return
    }

    return
}

// DelCookie removes the authentication cookie.
func DelCookie(w http.ResponseWriter) {
    http.SetCookie(w, newCookie("", -1))
}

The Gorilla securecookie package is part of the Gorilla web toolkit, a collection of utilities for handling forms, routing URLs, managins sessions and tons of other things. You should give it a glance, it can save you a good ammount of coding.

Validation and flash messages

Currently, if a validation error occurs, it is stored in a flash cookie and it is returned to the user. Only one error at time is displayed and the old form value are flushed away. The code to set and get flash cookies is very simple and is located in the "flash" package.

package flash

import "net/http"

const cookieName = "FLASH"

// SetCookie set a  flash cookie.
func SetCookie(w http.ResponseWriter, value string) error {
    c := &http.Cookie{Name: cookieName, Value: value, HttpOnly: true}
    http.SetCookie(w, c)
    return nil
}

// GetCookie get a flash cookie and delete it.
func GetCookie(w http.ResponseWriter, r *http.Request) (val string,
    err error) {
        c, err := r.Cookie(cookieName)
    if err != nil {
        return
    }
    value = c.Value

    // Remove the flash cookie
    c = &http.Cookie{Name: cookieName, Value: "", MaxAge: -1}
    http.SetCookie(w, c)
    return
}

Hadling user data

Every GoPics user has a name, an email address, a profile picture URL, a password, and, of course, a slice with his published posts. The name that the users chose on registration must be unique among app users (it is used to generate the user's home page URL). The user's profile picture URL is a Gravatar URL generated from his email address. Users have a couple of convenience method too, to store and validate their data.

package main

import (
    "strings"

    "github.com/garyburd/redigo/redis"
    "github.com/lucachr/gopics/reutils"
    "github.com/ungerik/go-gravatar"
)

// An user of the image board
type User struct {
    Name     string `redis:"name"`
    Email    string `redis:"email"`
    Password []byte `redis:"password"`
    PicURL   string `redis:"pic_url"`
    Posts    []Post `redis:"-"`
}

// validate is a convenience method for validating user data.
func (usr *User) validate(conn redis.Conn) error {
    if !reutils.MatchName(usr.Name) {
        return ErrValidation("Your username is invalid!")
    }

    for _, name := range invalidUser {
        if strings.Contains(usr.Name, name) {
            return ErrValidation("You cannot choose that name!")
        }
    }

    reg, err := redisGetUser(conn, usr.Name)
    if err != nil && err != redis.ErrNil {
        return err
    }
    if reg != nil {
        return ErrValidation("A user with the same name already exist!")
    }

    if !reutils.MatchEmail(usr.Email) {
        return ErrValidation("Your email is invalid!")
    }

    if len(usr.Password) < 8 {
        return ErrValidation("Your password is too short!")
    }

    return nil
}

// save is a convenience method for adding a new user.
func (usr *User) save(conn redis.Conn) error {
    usr.PicURL = gravatar.Url(usr.Email)

    _, err := conn.Do("HMSET", redisFlat(userTag+usr.Name, usr)...)
    return err
}

Regular expressions validators are from the package reutils, while redisFlatten is an utility function to flatten the User struct.

// redisFlat flatens a struct for Redis HMSET
func redisFlat(key string, value interface{}) redis.Args {
    return redis.Args{}.Add(key).AddFlat(value)
}

Posts management

When an user publishes a new post, the image is converted to JPEG and stored on the hard drive. Images bigger than 2MB are not allowed and all the images are resized before the conversion. The image name is generated with an UUID.

// Code excerpt from the handlePost function
// ...

// Check the content lenght
switch {
case r.ContentLength == -1:
    return &appError{
        Err:  ErrInvalidLength,
        Code: http.StatusLengthRequired,
    }
case r.ContentLength > maxPicBytes:
    return &appError{
        Err:  ErrInvalidLength,
        Code: http.StatusRequestEntityTooLarge,
    }
}

// Read only the first maxPicBytes of the request's body
r.Body = http.MaxBytesReader(w, r.Body, maxPicBytes)

// Try to get the content of the form picture field
f, _, err := r.FormFile("picture")
if err != nil {
    return &appError{
        Err:  err,
        Code: http.StatusBadRequest,
    }
}

// Try to decode the content of f as an image
src, _, err := image.Decode(f)
if err != nil {
    return &appError{
        Err:  err,
        Code: http.StatusUnsupportedMediaType,
    }
}

// Get the ratio d of the image
bound := src.Bounds()
x, y := bound.Max.X, bound.Max.Y
d := float32(x) / float32(y)

// Check the image sizes
if x > maxWidth || y > maxHeight {
    if x > y {
        img = resize.Resize(uint(maxWidth), uint(1/d*maxWidth),
            src, resize.Lanczos3)

    } else {
        img = resize.Resize(uint(d*maxHeight), uint(maxHeight),
            src, resize.Lanczos3)
    }
} else {
    img = resize.Resize(uint(x), uint(y), src, resize.Lanczos3)
}

// The image name is generated as an uuid
picName := uuid.New() + ".jpeg"
path := append(mediaPath, picName)

// Create a new file
dst, err := os.Create(filepath.Join(path...))
if err != nil {
    return &appError{
        Err:  err,
        Code: http.StatusInternalServerError,
    }
}
defer dst.Close()

// Write the image in the file
err = jpeg.Encode(dst, img, nil)
if err != nil {
    return &appError{
        Err:  err,
        Code: http.StatusInternalServerError,
    }
}

    // ...

A sorted set is used to store the id of the user's posts. The score of the post's id is equal to the UNIX time of the post publishing date. A hash table is used to store the post data and the transaction is made atomic with a "MULTI-EXEC" block.

//Code excerpt from the handlePost function

// Create the post and add it to the user timeline
conn.Send("MULTI")
conn.Send("HMSET", redisFlat(postTag+p.Name, p)...)
conn.Send("ZADD", userTimeline+usr.Name, unixTimeNow(), p.Name)
_, err = conn.Do("EXEC")
if err != nil {
    return &appError{
        Err:  err,
        Code: http.StatusInternalServerError,
    }
}

To get the posts back in order when an user requests a timeline, all we need to do is a ZREVRANGE on the sorted set with the posts' ids, loop over the results and fetch the posts.

// redisGetPosts return a list of the latest one hundred posts in postSet.
// Posts are sorted by publishing date, starting from the latest one.
func redisGetPosts(conn redis.Conn, postSet string) ([]Post, error) {
    postNames, err := redis.Strings(conn.Do("ZREVRANGE", postSet, 0, 100))
    if err != nil {
        return nil, err
    }

    posts := []Post{}

    for _, name := range postNames {
        val, err := redis.Values(conn.Do("HGETALL", postTag+name))
        if err != nil {
            return nil, err
        }

        p := new(Post)
        err = redis.ScanStruct(val, p)
        if err != nil {
            return nil, err
        }
        posts = append(posts, *p)
    }

    return posts, nil
}

Handlers, Redis and errors

When you are at the beginning with Go, you could think that it is very verbose compared to your language of choice. Actually, Go is not so much verbose if you learn to use interfaces. GoPics has a lot of functions and methods that require a connection to Redis and need to handle errors, if we put the same code for a connection request and error handling in every function, we'll end up with a lot of boilerplate. But we know that an http.Handler is an interface that is satisfied with a ServeHTTP method, so we can simply create a custom request handler for our scope.

// redisHandler is a request handler that needs a connection to Redis and
// returns a pointer to an appError.
type redisHandler func(http.ResponseWriter, *http.Request, redis.Conn) *appError

func (fn redisHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
        conn := pool.Get()
        defer conn.Close()

        httpAppError(w, fn(w, r, conn))
}

// Excerpt from utils.go

// httpAppError send an error reponse to the user if
// is not nil.
func httpAppError(w http.ResponseWriter, ae *appError) {
    if ae != nil {
        http.Error(w, ae.Error(), ae.Code)
    }
}

// Excerpt from errors.go

type appError struct {
    Err  error
    Code int
}

func (ae appError) Error() string {
    return ae.Err.Error()
}

And this save us from the boilerplate code. The same is true for connections handlers that uses the flash cookie.

// appHandler is an handler that takes a Page and returns a pointer to an
// appError.
type appHandler func(http.ResponseWriter, *http.Request, *Page) *appError

func (fn appHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // Check for a validation error in the form
    msg, err := flash.GetCookie(w, r)
    switch {
    case err == http.ErrNoCookie: // Do nothing
    case err != nil:
        err := &appError{
            Err:  err,
            Code: http.StatusBadRequest,
        }
        httpAppError(w, err)
    }

    p := new(Page)
    p.ValError = msg
    httpAppError(w, fn(w, r, p))
}

Serving static files

In a production environment you should prefer to put NGINX in front of your Go app and let it serve you static files, but GoPics, for the love of simplicity, serves static files directly.

// Excerpt from the main function

media := http.FileServer(http.Dir(filepath.Join(mediaPath...)))
http.Handle("/media/", http.StripPrefix("/media/", media))

static := http.FileServer(http.Dir(filepath.Join(staticPath...)))
http.Handle("/static/", http.StripPrefix("/static/", static))

The media directory is used to store the users submitted pictures, while the static one is used for all the rest.

Further reading and useful links

What do you think of GoPics? How would you improve it? Please, leave a comment and let me know it.


Full Stack Development: from the backend to the app

Recently, I have worked on the entire stack to develop a SaaS, web client and Android app included. Thoughts, tips, and tricks on the process and the software.

This week, I have had a lot of things to do. I ended the work on mg the Pelican theme that powered this blog, and I have started to learn Erlang, so I didn't have enough time to write a technical tutorial, but I want to share some thoughts on a project on which I have worked from July of the last year, Clochera. Clochera should have been an...

Read More
Simple orchestration with Ansible

Ansible is an IT automation tool, powerful and simple to use. It helps you with software installation, systems configuration, and app deployment.

Let's suppose that you have just finished your new, shiny web app. This is the big day and you are ready to upload the backend on your remote server, when you realize that the end of your coding sessions is not the end of your work. Now, you need to set up your remote machine, configure Supervisor, nginx, the firewall and tons of other stuff. And what if something, one day, goes terribly wrong and...

Read More
Browser automation with Nightmare.js

Nightmare.js allows you to automate browser tasks, as clicking buttons, opening links, navigating pages and much more.

I want to start the first post of this blog writing about something that, for its potential, impressed me when I have seen it for the first time. I'm talking about Nightmare.js, a high level wrapper for already the well-known PhantomJS. Nightmare.js is a rather new module for the Node.js platform, its first stable release (1.0.0), according to its GitHub repository, dates back to May of...

Read More

Receive Updates

ATOM

Contacts