How the Go programming language leads to fast, (energy) efficient and scaleable enterprise applications.
Go has not only become a popular language for writing small and fast server applications, it is also used by several companies to build huge scaleable web services. One example is the open source file server and groupware Infinite Scale by ownCloud. Such enterprise applications benefit from several Go features.
Time to divide
The well-known file sharing platform ownCloud 10 consists of more than 28,000 source code files. This heap of code is difficult to maintain and hard to extend. Furthermore, a small change could impact several other parts of the application. To get rid of these problems, the ownCloud team rewrote the whole system from scratch in Go. This new implementation has been named Infinite Scale.
In Infinite Scale, each feature is provided by a small single service. These microservices are working together to build the whole file sharing platform. Due to this distribution of work, it’s much easier to scale the system and extend single microservices. Go supports this approach in several aspects.
– Tim Schürmann, senior IT journalist, shares his expert insights on ownCloud Infinite Scale and Go.
ownCloud Infinite Scale has been extensively featured in the press, including several publications by Tim Schürmann.
Usually, microservices exchange information in established data formats like JSON or YAML. The Go standard library offers ready-to-use functions, that read, write and modify those formats. Microservices offer their services through a REST or GraphGL based API. Such interfaces could be created with the build-in HTTP functions or with the help of modules by third-party developers. One example is the Gin framework.
Fast, faster, fastest
ownCloud 10 was written in PHP. Thereby the application will run on every system with a corresponding interpreter. On the other hand, you have to pay for this platform independence with slower code execution. Even with sophisticated methods like a just-in-time compiler, load balancing or a page cache, the application won’t be as fast as a compiled version.
That’s another reason why the developers of ownCloud decided to switch to Go: The compiler builds a native binary that runs much faster – in most cases the performance is similar to an equivalent C program.
Simple dependency management
ownCloud 10 requires a web server, a MariaDB-database, PHP and several PHP modules – each of these in a specific version and with a specific configuration. Administrators have to take care of each of these dependencies. They are responsible for the installation, regular updates and the configuration.
The developers in turn have to test their application with several databases and PHP versions. In addition, most web applications are using PHP code from third-party developers. For example, ownCloud 10 relies on Archive_Tar to open Tar archives and parts of the Symphony framework. In the PHP world, there is some help to maintain these dependencies, like the tool Composer. Nonetheless, maintaining these components remains still a time-consuming, tiresome task.
Go to the rescue
Go compiles the source code into a single, statically-linked program. This binary is everything the users need. Administrators have to maintain the operating system and this binary – and nothing more. In addition, the developers no longer have to test their program in complex environments. As a side effect, Go supports the DevOps principle: The build pipeline generates the program, executes some tests and deploys it automatically on the server or within a Docker container. This is especially useful if you are developing microservices like the developers of Infinite Scale. They build the file sharing platform with the CI/CD system DRONE. The build process is controlled by Make and takes place in a Docker container.
Take care of your dependencies
On the downside, the developers now have to take care of every third-party component they used. Fortunately Go offers a clever module management. The developers just have to collect the URLs and specific versions of the needed third-party components in a configuration file (“go.mod”). The Go compiler will then download and build the corresponding source code autonomously.
A Go-module could use several other modules. This leads to a cascade of a vast number of modules that are linked into the binary – including security holes or bugs. That’s why the Infinite Scale developers recommend to select each module carefully and clean up the dependencies regularly. Go supports this by generating a dependency graph (with the compiler command “go mod graph”).
Another problem emerges if the application uses external services or programs. For example Infinite Scale supports the authentication via LDAP. So the developers still have to test their file sharing platform with several LDAP servers.
Have a look upstream
If a developer team considers to use a module, it should have a deeper look at the community behind that module. If it is implemented and maintained by a single person, then this person could abandon the project suddenly at any time. In such cases, the developers have to migrate to an alternative module, which could be a huge amount of work, especially in an enterprise application like Infinite Scale. That’s why the Infinite Scale developers are choosing only modules that have a stable and agile project team. Ideally the module is maintained by a foundation or a bigger company. But still then the developers should keep in mind that they might have to replace one or more modules within a few days.
To minimize the risk, the developers should support the teams behind the chosen modules and components. ownCloud, for example, participates actively in the Reva project that provides the storage backend in Infinite Scale. This engagement not only ensures that the project is vital, the file sharing platform will also benefit from the improvements made by the ownCloud developers in Reva.
Cross compiling
Just tell the compiler to build a binary for a specific platform and you’ll get the corresponding binary. What C++ programmers are dreaming for is available to all Go programmers, since version 1.5 Go supports this easy-to-use cross compiling. The developer tells the compiler the name of the operating system and the platform – that’s all. The following command generates a binary called “hello” that runs on a 64-Bit Windows on the x86 architecture:
GOOS=windows GOARCH=amd64 go build -o hello hello.go
The Infinite Scale developers are using this neat cross-compiling feature to build their application for the three popular operating systems and for two different platforms. The command “go tool dist list” presents every supported operating systems and architectures plus the keywords you have to use in the environment variables GOOS and GOARCH.
Unfortunately, several Go modules from third-party developers don’t work on every platform. So you have to choose the modules carefully. Another problem is CGO: If you call C code from Go programs, you have to set up a cross-compiling environment for a C compiler. To speed up the building process, initially the Infinite Scale developers used GOX. This helper tool builds the binary for multiple platforms simultaneously. Unfortunately the GOX development has slowed down, so the Infinite Scale team decided to abandon GOX.
In parallel
Sometimes a service has to solve several different tasks simultaneously. For example, a file sharing platform could scan an uploaded photo for malware while generating a thumbnail. Depending on the language, implementing such parallel tasks isn’t trivial. Current PHP versions, for example, offer Fibers and the extension “parallel”, which are both difficult to handle.
Go instead supports so-called Goroutines. These are code parts (or to be more precise functions) that are running concurrently. In a web application a Goroutine could scan a file for malware, while another one could generate thumbnails. Unlike threads in other languages, the Go runtime manages these the Goroutines by itself. The developers of an application just have to start a Goroutine and catch the results if necessary. The build-in scheduler asks the operating system to create an appropriate amount of real threads and starts the Goroutines within these threads.
Up and down
In Infinite Scale, each microservice runs in its own Goroutine and not as an independent process. Since the Go compiler builds one static linked binary, administrators just have to run one single program to start all needed microservices in a controlled manner. On the other hand, if the Go program ends or crashes, all microservices will go down too.
Go provides several features to control and monitor Goroutines. WaitGroups, for example, keep track of the running Goroutines, which can safely exchange data with so-called Channels. The latter work like a pipe: A Goroutine stuffs some data into it while another Goroutine receives this payload at the end of the pipe. Channels are used by Infinite Scale among other things to establish a connection with a LDAP server or during the creation of logs.
Unfortunately, Go can’t restart a single Goroutine and thus in the case of Infinite Scale a microservice automatically. To achieve this, the Infinite Scale developers are using Suture. Based on the corresponding concept in Erlang, this component provides a supervisor that monitors the Goroutines and restarts them as needed. Furthermore Infinite Scale uses the “go-micro” package to simplify the creation and management of the distributed microservice architecture. “go-micro” is even able to move a service to another server, despite the fact that the services are running as Goroutines in one process.
Green energy
Go offers a small syntax, spreads the Goroutines in an efficient manner over the CPU cores and compiles the code into a fast native binary. Thus, running applications written in Go should consume less energy than their PHP counterparts, where every single line has to be analyzed and executed by an interpreter. Web applications written in Go doesn’t need a web server, a database or a PHP environment in most cases. This saves memory and energy. So especially large enterprise Go applications like Infinite Scale should save a significant amount of energy costs. A paper from 2017 confirms this theory: According to a team of Portuguese researchers, a Go program is approximately nine times faster than his PHP counterpart.
But the researchers compared only small programs and algorithms, not complex web applications with several dependencies and a long-running build pipeline. Furthermore, the energy consumption is affected significantly by the architecture of the software and the use of algorithms. In Infinite Scale, the communication between the microservices is the most energy-consuming action. For example, the propagation of a modified access rights could lead to many messages to different microservices that have to been coordinated and evaluated. So each developer has to do its own measurements.
Unfortunately, this isn’t an easy task. Go itself can’t measure the energy consumption of the running code and there isn’t even a third-party framework for that purpose at the moment. As a consequence, developers have to use other tools and hardware. The Infinite Scale developers even wrote their own test scripts on top of K6.
During the measurements, the Go application should run without an error. Otherwise an operation would stop prematurely and thus consume less energy. Every part of the software should be tested under realistic conditions. The Infinite Scale developers used the basic authentication method in their first energy measurements. As a result, the whole system decoded the security tokens several times. The switch to OpenID Connect led to a much faster system.
The Go compiler produces fast and small binaries without any dependencies, which are easy to deploy on several platforms. Goroutines simplify the programming of asynchronous functions and microservices. If developers carefully choose their third-party modules, Go is suitable for scalable enterprise web applications. The best proof is the file sharing platform Infinite Scale.
Learn more about how ownCloud Infinite Scale can help boost productivity and collaboration at a reduced cost in your enterprise. If you have any questions, don’t hesitate to get in touch.
