Optimizing Java Microservices with the smallest container on the market

BellSoft prepares the final changes to upstream the Alpine Linux port to OpenJDK

Alpine is perfect for container deployments

Various sources glorified Alpine Linux as “the new default OS” and “distribution of choice” when it replaced Ubuntu as the base image for Docker in 2016¹. Thanks to its small image size, this distribution is widely adopted in cloud deployments, microservices, and container environments. For instance, Liberica JDK Alpine musl image is at least 200 MB less than the others available (43.5 MB vs. 228 MB for Debian or even 319 MB for CentOS). With a truly minuscule Docker base image of no more than 6 MB and an embedded Docker build process, it is perfect for minimizing valuable cloud resources.

Another positive feature is that the service management subsystem is not even included by default in basic Alpine Linux containers. As a result, it frees up RAM, simplifies diagnostics and system configuration and boosts launch time. At a certain angle, having less contents in a distribution reduces a possible attack surface.

Maximizing business value with containers: less is more

But what lies behind the need for small images in development?

A simple answer is “cost efficiency”. A longer one starts with an understanding that containers are gaining traction each year. Users want to get more value while spending less, that is the gospel truth. And here we can see the absolute superiority of containerization over virtualization.

Docker containers are great for streamlining development with the microservice architecture. If you plan to move from Java monolith or create a new solution based on Java microservices, count on our engineers to pave the most efficient migration path. We are offering an entirely free consultation with a BellSoft expert. Fill out the simple form below and schedule a meeting now!

First, bug fixes and changes are delivered to the whole application package continuously and fast. Deployment may include many new instances, for example, when the application is dynamically scaled in some region. A major problem that usually arises during this process is high traffic expenditures. Also, the less the image you have to upload, the less time you waste before the application begins to perform business tasks.

Second is the age-old problem of storage space. In the case of containers, an isolated environment requires tens of megabytes. In contrast, a virtual volume with full OS will be sized in gigabytes. Just imagine what your company can do with the saved space and money not spent on extra cloud storage, let alone boosted overall performance.

Third, waiting for deployment rollout to finish typically takes up huge chunks of programmer’s work. It is easy to calculate profits if this time is reduced to a minimum with containers deployed almost in an instant.

Let’s turn to a hypothetical enterprise that transfers to containerization. Reducing overhead charges, no longer having to set up a hypervisor and run a virtual machine… All good. New shiny containers deploy on a clean target device. Why wouldn’t the company want to save even more with smaller images? Which brings it to an idea of JDK containers on an intrinsically lightweight Alpine Linux.

Before unraveling the history behind the JDK port to Alpine, we want to say that BellSoft’s binaries (the smallest in the industry!) are supported not only on Linux but on all known platforms.

JEP’s background, demand and idea

Mikael Vidstedt, now Director of JVM Software Engineering at Oracle, was the first to consider bringing support for alternative implementations of standard C libraries to OpenJDK. The problem was self-evident: say, a person trying to set up a Java VM on an OpenWrt router would soon find it impossible. Mr. Vidstedt decided to change the status quo by starting a venture dubbed the Portola Project in February 2017. He could easily be the one to propose the port to the master branch. But his team had neither sufficiently analyzed the changes (to define their purpose and check other possible solutions) nor done enough testing on Alpine Linux and then all available platforms: Linux, macOS X, Windows, arm, and aarch64, as the patch would affect shared code in JDK.

The work on this JEP started a while back when BellSoft understood that containerized JDK 9 is quite favored among users. We supported our Liberica distribution on various platforms and made delivery convenient with a choice of channels: you could download a binary build from the website, install a repository for Linux, download an installer, find publicly available builds on package managers (like SDKMAN), or pick a container image.

BellSoft saw the big future in supporting such builds in commercial terms and first created containers for Alpine Linux with glibc compatibility layer. The earliest Liberica version containing Alpine Docker images was 9.0.1 on armhf (for Raspberry Pi). The Portola Project started at times of JDK 10.

Portola is about building OpenJDK linked directly to musl. In a musl-based system without the compatibility layer, programs linked to glibc (including regular JDK builds) won’t start. Such builds with a different standard library become some sort of a port. We also should take into account all other peculiarities of target systems, some covered by the existing tests and some not.

New containers with OpenJDK took off fast. Liberica OpenJDK statistics on Docker Hub really shows how popular Alpine images are: 100K+ downloads and 10-star rating. The two reasons are that regular JVM images do not work on Linux and that Alpine is inherently lightweight.

Given the success of Liberica containers with glibc, we, engineers at BellSoft, decided to take the “intermediary” out of the equation and enter the Portola Project to help produce a rounded and single-purpose JDK for Alpine. In the team’s view, the procedures inside the kit should address the standard library without emulating glibc behavior.

We have been shipping Liberica-openjdk-alpine-musl container images since JDK 11. By getting rid of this layer, a new container was expected to save approximately 24 MB—an obvious driver for a company that strives to minimize costs for clients.

BellSoft’s contribution to the Portola port

Reasonable questions here would be: Why did the JDK not work with musl libc? What changes has BellSoft had to make to its Alpine Linux containers?

First, the team prepared a merge from the Portola repository to the JDK master branch, which due to the branch’s ever evolving nature needed to be synchronized non-stop before it was approved and pushed. The patch was split into categories, each one examined and code-named by changes it provides. Some are minor improvements, like defining libc in config:

Before: ldd_version=`ldd --version 2>&1 | head -1 | cut -f1 -d' '`

After: libc_vendor=`ldd --version 2>&1 | sed -n '1s/.*\(musl\).*/\1/p’`

Note: with this improvement, it is also possible to cross-compile the Linux-musl OpenJDK images on traditional Linux x64 based systems using GNU/Linux distributions.

Portola cross-compilation:

1. Clone hg.openjdk.java.net/jdk/jdk repository

    hg clone https://hg.openjdk.java.net/jdk/jdk
   
   

2. Apply Portola patch
3. Build jdk for the current platform which will be used as boot and build jdk for cross-compilation

    mkdir -p build/linux-x86_64-boot-release
   
    cd build/linux-x86_64-boot-release
   
    sh ../../configure --with-boot-jdk=<path-to-jdk>/jdk-14.0.1
   
    make jdk-image
   
   

4. Download cross musl-based toolchain from https://musl.cc
5. Copy sysroot from target system (for example from Alpine Lunux docker image)
6. Configure musl build with devkit and sysroot

    export DEVKIT=<path-to-devkit>/x86_64-linux-musl-cross
   
    export SYSROOT=<path-to-sysroot>/alpine3.8-sysroot
   
    export TARGET=x86_64-linux-musl
   
    export BOOT_JDK=build/linux-x86_64-boot-release/images/jdk
   
    sh ./configure --with-jvm-variants=server \
   
    --with-boot-jdk=$BOOT_JDK  \
   
    --with-build-jdk=$BOOT_JDK \
   
    --openjdk-target=x86_64-unknown-linux-musl \
   
    --with-devkit=${DEVKIT} \
   
    --with-sysroot=${SYSROOT}
   
   

7. Build jdk image

   cd build/linux-x86_64-server-release
   
   make jdk-image
   
   

Or, instead of static int sigWakeup = (__SIGRTMAX - 2), using #define INTERRUPT_SIGNAL (SIGRTMAX - 2) where SIGRTMAX is a function in musl libc.

Others encouraged BellSoft to search for different solutions. One example here is that the musl library does not implement dlvsym since it is not a part of POSIX. The initial Portola Project tries to load dlvsym dynamically and provides fall back in case it is absent. The engineers decided to replace it with dlvsym stub and rewrite the code to:

#ifdef MUSL_LIBC

static void *dlvsym(void *handle, const char *symbol,

const char *version) {

   return NULL;

}

#endif

Certain solutions did not make the final cut just yet. Such as execvp in musl libc that improperly executes shell files without a valid shebang line—a known issue, actually².

Finally, the team singled out parts of code to submit them for a standalone review and fix independently of the Portola patch. A major and the most labor-intensive part was jpackage, which could even be considered a subproject of sorts. Under particular circumstances, java launcher executor re-executed the application. This is a corner case for musl libc which, when loading shared libraries, does not check them by short name. It was found that the issue with jpackage application re-execution is not specific to Alpine Linux and can be reproduced on other Linux systems that change LD_LIBRARY_PATH³.

The other huge chunk of work was testing. BellSoft’s first step was to run JCK, jtreg, vmTestbase and jvmTestbase. The team got cornered a little: we tested the master repository constantly under heavy development. Fails occurred even without the Portola patch. The decision was to run all tests with JDK master branch and use the passed ones as a reference to the Portola build configured for fastdebug. As a result, some fixes were made to the patch before the merge, some already in the merge, and some were reported as musl libc bugs (like the execvp issue discussed above).

To recap, by porting the JDK to musl-based Linux distributions, developers will find updating base layers and storing them in a private repository much simpler. Among other benefits are reduced time spent on deployment to a new host, shortened redeploy in general, and saved storage. All these are critical in facilitating business processes and cutting down costs, especially when a business pays for CPU consumed in the cloud.

Applying jlink to reduce the size of the Java runtime, a user will be able to create an even smaller image targeted to run a specific application. But sometimes it looks too complex because of an extra step in the deployment pipeline or because of dynamic dependencies. So our alpine-musl docker images can be used in the ‘docker build’ process with extra options. For instance, a user can easily get a runtime that contains either only java.base (to run simplest apps), or some typical modules for a web application, or all the modules. The preparation process is much simpler than the full-fledged jlink tool. A user has no need to recalculate actual dependencies and just chooses between the three lightweight JDK 14.0.1 presets: Base (43.5 MB) / Light (111 MB) / Full (192 MB).

The JEP will further support Java runtime not only on Alpine but on all Linux systems with musl. We claim (although some additional testing is still to come) that the JDK port runs on OpenWrt, a popular Linux distribution also based on musl. It powers many small smart devices such as routers.

BellSoft guarantees the quality of its Alpine images. The company assures every container image is TCK verified. It is sometimes not so simple to combine things. For example, in some older versions of Alpine, JVM simply shut down on launch because of experimental security patches included. Containers with Liberica JDK, properly tested and verified, show none of these problems. The team is focusing major engineering efforts on this patch, supports the builds, resolves all problems in the timeliest manner, and plans to further follow musl-related part of OpenJDK.

Expectations and prospects

Overall, the Portola Project is moving at its designated pace towards finalization. However, to become part of the upcoming JDK 16 next year, it still should be refactored and polished. Late June saw BellSoft taking a refreshed JEP implementation for review to fix certain non-musl-related issues. There were several new patches (that are concerned with running jpackage; on AIX they are checked by SAP) integrated into the main branch.

After jpackage change was tested by SAP, now, at the time of writing, it is waiting for a public review on core-libs-dev³. Portola polishing changes passed from internal review to public discussion on portola-dev⁴, where developers already responded to some concerns. Next is to introduce the first change into the mainline JDK, then integrate polishing changes into the Portola repository. The final touch is a public review of all Portola changes going into the JDK mainline branch. BellSoft expects the whole JEP to get processed by mid-August.

We see exciting benefits and stability in Portola-powered builds. At the same time, there is a strong demand, as always, for JDK 8. One path for improvement is to build JDK 8 for Linux-musl. The proposed changes apply to the JDK most beloved by the developers (aka “the best release”) since JDK 8u had lots of containerization-related features backported⁵. It would still be labor- and time-intensive as the current Portola status is downstream. After the JEP is fully integrated, we might find resources to provide this option for the latest updates of version 8.

What is next for BellSoft in this area? Our team wants to tackle musl libc for GraalVM within the Graal Project. This library and issues it raises for Java runtime developers are a nice spot for the long-term. Whereas previously the company poured resources into maintaining correctness in new versions of their product, now it is ready to produce novel technical features and support containers. Virtual machines are no longer the talk of the JDK community. Working with containers is seen as mainstream and the primary way to use JavaTM in production. This year and beyond, we promise to dive deeper into the topic and make the developers’ job easier and more enjoyable than it has ever been.

And BellSoft likes to keep promises: We are proud to announce that the latest Liberica JDK’s versions, 8u262, 11.0.8 and 14.0.2, include the long-awaited Alpine Linux musl libc support and lots of other exciting features.

Download Liberica JDK

Thanks to the profound knowledge of this OS, we have created the smallest known containers among OpenJDK contributors. Our customers have used Liberica JDK containers with Alpine musl in production environment for years and admitted savings on cloud traffic, storage and even engineering time. If you want to start working in the cloud and be most efficient, if you are ready to save precious time and money, contact BellSoft right away!

Links:

1. Review: Alpine Linux is made for Docker
2. https://www.openwall.com/lists/musl/2020/02/12/9
3. https://mail.openjdk.java.net/pipermail/core-libs-dev/2020-June/067456.html
4. https://mail.openjdk.java.net/pipermail/portola-dev/2020-June/000441.html
5. GeeCON 2019: Dmitry Chuyko - Do not put all eggs in one container