CI

GOLANG: CI using Git Post Receive Hook

admin Leave a comment

Continuous Integration (CI) using automated build and test tools was well covered by Jason McVetta – read his blog post to learn how to get Status and Test-Coverage badges on your GitHub project page. What I’d like to document here is a process of deploying your GOLANG code to a staging server using git push and then physically QA the live project (in this case a website) before deploying it to production.

There are other ways of building out a staging/QA site – you could cross compile the binary on your local development machine and then deploy a self contained binary to staging. However, my goal is to build the binary on an identical copy of production directly from source and then deploy the binary to production. Plus I’d like to have an option of multiple developers pushing to a centralized repo.

The process consists of the following steps:

  1. setup a bare GIT repo on the staging server
  2. setup remote repo called staging_qa on your dev machine and point it to 1)
  3. push to it from your development machine using git push staging_qa master
  4. have the post receive hook on staging server checkout the code to a local directory
  5. compile the GO code using go get and go install
  6. refresh/bounce QA live site with new code (it’s served via NGINX)

Environment:

  • Development: OS X [10.9.5]
  • Staging: Ubuntu 14.04.2 LTS

Lets dive in!

Setup Staging

Setup GOLANG on Staging

Purge default golang installation using “apt-get –purge autoremove”

sudo apt-get --purge autoremove golang

Now install golang directly from the source to get the latest version (in this case it’s 1.4.2):

wget https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.4.2.linux-amd64.tar.gz

Test installation:

vi hello.go
--------- paste --------------
package main

import "fmt"

func main() {
    fmt.Printf("hello, world\n")
}

# go run hello.go

setup GO workspace [maps to GOPATH – see below]

mkdir -p /apps/dev/golang

setup go/bin env in global profile

sudo vi /etc/profile

add the following:

export PATH=$PATH:/usr/local/go/bin

setup GOPATH env in local .profile

vi $HOME/.profile

edit/add as follows:

GOPATH=/apps/dev/golang; export GOPATH
PATH=$PATH:$GOPATH/bin; export PATH

Create Directory Structure

I will use my GitHub project DOP – Day One Parser for this example. The directory structure will live under APP_TOP (/apps) and will have three major components as follows:

/apps                        <— APP_TOP
├── dev
|    └── golang              <— GOPATH
|        ├── bin
|        ├── pkg
|        └── src
|            └── github.com
|                └── vmogilev
|                    └── dop <— [1] Source Code
|
├── stage
|    └── git
|        └── vmogilev
|            └── dop.git     <— [3] BARE Git Repo
|
|
└── vmogilev.com              
     └── dop                 <— [2] http root
         ├── conf
         ├── static
         │   ├── css
         │   ├── fonts
         │   └── js
         └── templates

Here’s how to create this structure:

First set APP_TOP:

APP_TOP=/apps; export APP_TOP

Next create three directories/structure:

  1. ${GOPATH}/src/github.com/vmogilev/dop – project’s source code directory in GOPATH (also under APP_TOP) so we can compile using go install, the code will be copied here by the git post-receive hook right after we push from development: 
    mkdir -p ${GOPATH}/src/github.com/vmogilev/dop
  2. ${APP_TOP}/vmogilev.com/dop – project’s http root directory where the site config files will live. We’ll also stage and serve the website assets from this directory using go’s http server and proxy it using NGINX (this allows running go’s http server on port other than 80 so we can have multiple apps running on the same server all sharing port:80 end-point via NGINX). Some of the assets in this directory (go templates, css and java script) are versioned in the git repo so they will be copied here by the git’s post-receive hook (see further down the writeup): 
    mkdir -p ${APP_TOP}/vmogilev.com/dop
  3. ${APP_TOP}/stage/git/vmogilev/dop.git – bare git repo that we’ll push to from development [must end with .git]: 
    mkdir -p ${APP_TOP}/stage/git/vmogilev/dop.git
cd ${APP_TOP}/stage/git/vmogilev/dop.git
git init --bare

Create Post Receive Hook

cd ${APP_TOP}/stage/git/vmogilev/dop.git
touch hooks/post-receive
chmod +x hooks/post-receive
vi hooks/post-receive

paste the following:

#!/bin/bash

# ----------- EDIT BEGIN ----------- #

APP_TOP=/apps; export APP_TOP
GO=/usr/local/go/bin/go; export GO

## go [get|install] ${SRC_PATH}/${APP_NAME}
SRC_PATH=github.com/vmogilev; export SRC_PATH
APP_NAME=dop; export APP_NAME

## local http root directory served by go http - ${APP_TOP}/${WWW_PATH}
## for / directory use /root:
##      site.com        -> site.com/root
##      blog.site.com   -> blog.site.com/root
##      site.com/mnt    -> site.com/mnt
WWW_PATH=vmogilev.com/dop; export WWW_PATH

## local bare git repo path - ${SRC_NAME}/${APP_NAME}.git
SRC_NAME=vmogilev; export SRC_NAME

# ----------- EDIT END ----------- #


GOPATH=${APP_TOP}/dev/golang; export GOPATH
SOURCE=${GOPATH}/src/${SRC_PATH}/${APP_NAME}; export SOURCE
TARGET=${APP_TOP}/${WWW_PATH}; export TARGET
GIT_DIR=${APP_TOP}/stage/git/${SRC_NAME}/${APP_NAME}.git; export GIT_DIR

## pre-creating SOURCE DIR solves the issue with:
##  "remote: fatal: This operation must be run in a work tree"
mkdir -p ${SOURCE}
mkdir -p ${TARGET}

GIT_WORK_TREE=${SOURCE} git checkout -f


## do not prefix go get with GIT_WORK_TREE - it causes the following errors:
##  remote: # cd .; git clone https://github.com/vmogilev/dlog /apps/dev/golang/src/github.com/vmogilev/dlog
##  remote: fatal: working tree '/apps/dev/golang/src/github.com/vmogilev/dop' already exists.
##
##GIT_WORK_TREE=${SOURCE} $GO get github.com/vmogilev/dop
##GIT_WORK_TREE=${SOURCE} $GO install github.com/vmogilev/dop

unset GOBIN
unset GIT_DIR
$GO get ${SRC_PATH}/${APP_NAME}
$GO install ${SRC_PATH}/${APP_NAME}

if [ $? -gt 0 ]; then
    echo "ERROR: compiling ${APP_NAME} - exiting!"
    exit 1
fi

sudo setcap 'cap_net_bind_service=+ep' $GOPATH/bin/${APP_NAME}


# ----------- DEPLOY BEGIN ----------- #

cp -pr ${SOURCE}/static     ${TARGET}/
cp -pr ${SOURCE}/templates  ${TARGET}/
cp -p ${SOURCE}/*.sh        ${TARGET}/

. ${TARGET}/conf/${APP_NAME}.env
${TARGET}/stop.sh >> ${TARGET}/server.log 2>&1 </dev/null
${TARGET}/start.sh >> ${TARGET}/server.log 2>&1 </dev/null

# ----------- DEPLOY END ----------- #

What’s happening here? Lets break it down:

  1. APP_TOP – top level mount point where everything lives under
  2. GO – complete path to go binary
  3. SRC_PATH and APP_NAME – the combination of the two is what will be passed to go [get|install] ${SRC_PATH}/${APP_NAME}. APP_NAME is the actual binary name – $GOPATH/bin/${APP_NAME} on which we’ll set a special flag sudo setcap that allows to bind on privileged ports <1024
  4. WWW_PATH – since our app has static assets we need an http root directory to serve them from. Depending on your app you can serve these using GO’s http server or NGINX directly. I use GO’s http server and then proxy everything via NGINX to simply configuration. These assets are part of the git repo and will be copied to${APP_TOP}/${WWW_PATH} using post receive hook (see DEPLOY BEGIN|END section). The convention for top level domain is site.com/root
  5. SRC_NAME – this becomes part of the GIT’s bare repo path in the following format ${SRC_NAME}/${APP_NAME}.git – this is what you’ll map to on the development machine using git remote add … (see Setup Git Repo further down)

Now lets talk about what’s going on in the DEPLOY section:

  1. Part of the source code are startup/shutdown scripts named: stop.sh and start.sh and two assets directories named: static and templates – we copy all of this from go’s project directory to target located in WWW_PATH.
  2. We then expect an env file to be present in ${TARGET}/conf/${APP_NAME}.env that sets up our environmental variables for the app’s runtime on this staging box so that when we execute stop.sh and start.sh these envs are passed to our app. Here are the contents of the env file: 
    DOPROOT="/apps/vmogilev.com/dop"; export DOPROOT
HTTPHOST="http://localhost"; export HTTPHOST
HTTPMOUNT="/dop"; export HTTPMOUNT
HTTPPORT="3001"; export HTTPPORT
HTTPHOSTEXT="http://vmogilev.com/dop"; export HTTPHOSTEXT
  3. Here’s an excerpt of the start.sh that passes these to the app: 
    nohup $GOPATH/bin/dop \
    -dopRoot="${DOPROOT}" \
    -httpHost="${HTTPHOST}" \
    -httpMount="${HTTPMOUNT}" \
    -httpPort="${HTTPPORT}" \
    -httpHostExt="${HTTPHOSTEXT}" >> ${DOPROOT}/server.log 2>&1 </dev/null &

Setup NGINX

I am using NGINX on Port 80 and proxy GO’s HTTP server that runs on higher port number – this allows running multiple go apps on different ports yet all accessible via regular http port on the same server:

nginx:80/app1 -> app1:3001
nginx:80/app2 -> app2:3002
nginx:80/app[n] -> app[n]:300[n]

Install nginx:

sudo apt-get install nginx
sudo service nginx start
sudo service nginx stop

Make sure that nginx starts automatically:

sudo update-rc.d nginx defaults

To set up NGINX can be as simple as this:

vi /etc/nginx/sites-available/default

edit as follows which will setup proxy mount point for your app (in this case /dop via port 3001):

server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    # Make site accessible from http://localhost/
    server_name localhost;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
        # Uncomment to enable naxsi on this location
        # include /etc/nginx/naxsi.rules
    }

    location /dop {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header Host $host;
            proxy_pass http://127.0.0.1:3001;
    }
}

next bounce nginx server:

    sudo service nginx restart

if you run into any problems check log file under /var/log/nginx/error.log (this is defined in /etc/nginx/nginx.conf.

Setup Development

Setup Password-less SSH

In order to git push via ssh we’ll need to paste our personal SSH Public KEY into ~/.ssh/authorized_keys on the staging server. Here’s how to do this:

  1. On your development machine copy the contents of your ~/.ssh/id_rsa.pub
  2. Go back to the staging server and paste it to ~/.ssh/authorized_keys
  3. Back on development machine make sure you can ssh user@my-staging-box without supplying the password

As a bonus point setup a bastion host on your network and only allow ssh traffic to pass through it. That’s what I am doing in our infrastructure.

Setup Git Repo

first we need to setup global git prefs (if not already):

git config --global user.name "myusername"
git config --global user.email "myusername@gmail.com"
git config --global core.autocrlf input

next cd into your go project’s directory and setup git repo with a remote origin pointing to the staging’s bare repo we created earlier:

cd $GOPATH/src/github.com/vmogilev/dop
git init
git remote add staging_qa ubuntu@staging-box:/apps/stage/git/vmogilev/dop.git
git add .
git commit -a -m "Initial Commit"
git push staging_qa master

End!