Golang at Runscope

When we started Runscope, we used Python to power all our services.  Python’s flexibility, utility, and maturity has allowed us to move quickly from idea to prototype to production. Consequently, we’ve invested in building a lot of tools and processes for developing and deploying Python services within our infrastructure.

With our recent announcement of Runscope Enterprise, the story of Runscope expands beyond just deploying code to servers that we manage to having code run in a variety of different customer environments.  This was a perfect time to step back and ask ourselves if Python was the best language for the job.

Distribution and Python

Runscope Passageway is a client application  that provides HTTP tunneling to your local machine from the public internet. These HTTP requests are automatically fed into your Runscope stream so you can easily debug a local web service the same way you would a public API. During the initial beta period, we distributed the client via the Python package manager pip.

In theory, this seems like a perfectly reasonable, single line, method for downloading a package.  In practice, it's much more complicated.   For starters, not everyone is a Python developer and not everyone is running a *nix environment.  There are a lot of prerequisites that you need to get to the point where you can even attempt to install from pip.  Once you’ve cleared those hurdles and have pip installed, you’re not done yet.  For instance, one of the most common issues we’ve seen on Windows platforms was gevent/greenlet support.  Not to mention that the most recent version of pip no longer supports dependency_links by default.   Python packaging is great for Python libraries, but for general purpose utilities it requires you to write a lot of supporting scripts to check and fix the environment you’re going to be running in.

Our next foray into Python client distribution was an early alpha of an agent for Runscope Radar that you can run from within your own infrastructure.  For this project we decided to look into various Python binary generation frameworks. We tried py2exe, cx_Freeze, and PyInstaller.  Each of these fell short when trying to build binaries for multiple platforms.  While it was possible to generate binaries, the configuration and build process was time consuming and poorly documented.  Furthermore, it required each build to happen on target platform.  For us, maintaining three separate build servers running three different operating systems was less than ideal.  Even then, many of our early test builds failed because of one or more modules not building correctly on the target platform or some dynamic library not being present.

There are ways around a lot of the issues we saw, but it largely required rewriting, rearchitecting, and simplifying the code and its dependencies.  If we were going to spend a large amount of effort starting from scratch, we had the opportunity to try another approach.

Adventures in Go

One of the requirements for Runscope Enterprise agents is a truly effortless install, no matter what platform the agent is running on.  Go’s ability to easily cross-compile to multiple architectures made it a top candidate.  And based on the experiences and recommendations of those who’ve worked with go in these situations, we decided to try it out.

First Impressions

Go is fun.  Go has an interesting feature set that makes the language feel both familiar and new at the same time.  Using goroutines and channels for the first time makes you go “wow, that was neat!”

Learning by example is by far the easiest way to learn go.  While godoc.org has a nice complete reference for go packages, as a beginner the most useful documentation are the short Example usages in the godocs.  The Go Blog (http://blog.golang.org/) is also filled with a wide variety of detailed examples and solutions.  Seeing how functions behave in context helps cement a lot of the new language idioms.

The concepts of go are simple enough to understand (especially coming from a C/C++/Java/Python background).   Most importantly, we were able to be productive in the first week without any issue.  After getting our feet wet, we were able to learn and use Go-specific features as we went along.

Essential Tools & Resources

Go Fmt - Auto formatting of go code makes a bigger difference than you’d imagine.  Having a language that can self enforce a style guide lets you write code your way but be readable for everyone.  When coupled with the go vim/emacs plugins, you spend less time organizing code and more time getting stuff done.

Go Vet - "Vet examines Go source code and reports suspicious constructs, such as Printf calls whose arguments do not align with the format string. Vet uses heuristics that do not guarantee all reports are genuine problems, but it can find errors not caught by the compilers."

The Go Blog  - Filled with great detailed posts about individual langauge features or patterns.  Usually when we were stuck on something a google search ended up on the this blog.  It’s nice having an authoritative source for information when you’re learning the tricks of a new language.

Go By Example  - A great reference that gets stright to the point.  Go by Example gives you complete working examples to help you understand Go’s finer points.  Each example comes complete with imports, a main function, and the command to run, and example output.  By giving you the full picture, it helps cement your understanding of how thing actually work, rather than giving you snippets of code out of context.

Build & Deployment

Our builds are done by a Jenkins server.  We use the project’s workspace as our GOPATH to isolate each go project from each other.  Each build runs a go get -u and a go build task.  For our cross-compiled libraries we have the build task repeat the go build for each target OS + Architecture combination.   Once a successful build happens, we publish the built binaries into an AWS S3 bucket.  These binaries are downloaded to our cluster when the appropriate service gets deployed.

Shortcomings

With any young language, there will be some rough edges.  We did encounter, from time to time, some small issues as we explored Go.  Here’s a brief overview of those issues:

Repositories Everywhere

Most of the code we write as internal libraries are in private GitHub repositories.  Unfortunately ‘go get’  doesn’t clone a private repo using your ssh keys.  To bootstrap our build system, we had to manually clone those dependencies into our workspace.

hg, bzr, git, …  Public Go libraries are hosted all over the place, and each repository has it’s own SCM dependency.  If your package import is github.com then go get will use git.  If it’s code.google.com you need to install Mercurial.  If you import from launchpad.net, you need Bazaar.   And it’s not just your direct dependencies, if one of your dependencies has it’s own dependency on launchpad, then you might need to install a new source control tool.  There are a lot more moving parts to Go under the hood than it seems at first glance.

Things that do not cross-compile.

There are a handful of functions that do not play well when you cross-compile and that panic at runtime.  For example, user.Current() cannot be called in a cross-compiled binary (aka CGO_ENABLED=0), and it seems like they’re going to stay that way.  These functions that aren’t available when you cross-compile aren’t documented well and end up dying at runtime instead of compile time, it’s also unclear if a library or dependency of a library will have issues until you run it and find out the hard way.

Another shortcoming of cross-compiling is that you cannot use system CA certs for SSL connections.  You now have to accept a path to the cacert files from the user, or bundle your own.

Final Thoughts

Adding Go to our repertoire has enabled to do do things we couldn’t easily do in Python alone. It has allowed us to quickly build stable, high-performance systems which we can deploy to a variety of environments with ease. While still rough around the edges, with each new release we’ve seen some great improvements to the language.  When we started our Go experiment, our focus was on easy to distribute client binaries.  Since then, we’ve found more uses for Go in our internal services, and our use continues to expand as the language matures and our overall go skill level improves.  

If you’d like to work on high performance Go services at Runscope, check out our jobs page and drop us an email!