We at BellSoft take nothing more seriously than the quality, security, and performance of our products. These are the most common concerns for any organization that chooses to develop with OpenJDK. No need to worry—you do not have to “only pick two.”
In this material, we go over the main components that constitute Liberica JDK quality control (or QC for short). The goal is to explain how we work on the Quality–Security–Performance triangle, manage to attain a balance, and produce the best Java runtime we can. Check it out for yourself: click the button below to see all its benefits and Liberica binaries distributed entirely for free.
Release cycle
Before we unravel our QC intricacies, let’s look into why we need it, exactly. OpenJDK Critical Patch Updates (CPUs), containing fixes to vulnerabilities, are date-driven and released quarterly. These are not insignificant; for instance, last year saw 1957 defects and 95 CVE fixed or backported in JDK 8 and 11 combined. In addition to that, major feature releases are developed in a 6-month cadence. Plus, all vendors customarily deliver bug fixes to customers before or after the next feature release.
Now that we have described the existing state of affairs for OpenJDK in general, what about BellSoft specifically? What does our patch flow look like?
As you can see, the process is somewhat confusing and has many actors. To reach eventual consistency with OpenJDK, to ensure all Liberica JDK releases are up to par, QC is simply imperative.
Quality criteria for releases
In 2020, the Java™ programming language celebrated its 25th anniversary and is still thriving. BellSoft can acclaim to meet the high standards to quality the industry has set over this time. Let’s review the main criteria for a compliant release.
A Technology Compatibility Kit (TCK) is a set of tests provided by Oracle. They indicate if a particular Java™ implementation is compatible with Java SE specifications. This suite is the Swiss army knife of QC. It indicates that a Java™ platform is highly performant and will not damage your critical applications. All Liberica binaries have passed the TCK tests and are fully ratified in the Java Community Process. As for the numbers of tests in JDKs (Java Compatibility Kits) for the latest LTS releases and current 15, here they are:
- JCK-8c: over 126,000
- JCK-11: over 139,000
- JCK-15: over 152,000
One of the commonly used approaches to security analysis is proof-of-concept (PoC) exploits. They are non-harmful attacks meant to unveil software weaknesses. In our case, the engineering team develops a piece of PoC code to verify whether a particular OpenJDK vulnerability is exploitable. Once a security flaw is found, we provide a fix. This process is repeated for every release.
Another part of quality control at BellSoft is functional OpenJDK tests executed by the Java Regression Test Harness (jtreg). It is used for functional and regression Java SE testing, which belongs strictly to a TCK. We run somewhere between 11,000 and 34,000 regression tests per binary before pushing a new Liberica JDK version, including hotspot, jdk, langtools, nashorn and, in certain cases, jcstress.
A set of OpenJDK regression tests is also complemented by the largest industry frameworks: Spring and Apache Tomcat. During the most recent check, BellSoft performed more than 21,000 tests that are part of Spring and 40,000 from Tomcat (including APR connector tests running with Tomcat Native), which revealed no regressions.
Benchmark testing is as essential as all the previous steps. There are two sides to this process. First, we evaluate Liberica JDK’s core functionality with major industry-standard benchmarks, such as SPECjbb and SPECjvm. Second, our team runs microbenchmarks embedded in the OpenJDK code. Based on the community-developed Java Microbenchmark Harness (JMH), they measure the performance of individual functions.
How do we make it happen on time?
Work on each release starts well in advance, sometimes more than a year before. It depends on the scope of work and helps us not rush the development.
Key factors for timely quality control:
- push all test patches upstream, to the mainline JDK branch, as soon as they are developed;
- agree within the OpenJDK Vulnerability Group on the scope of each release. Since it is combined of engineers from major distributors such as Oracle, SAP, Amazon, and Red Hat, BellSoft has a clear understanding of timelines;
- test all platforms with and without security fixes;
- verify that final testing right before the release finds no new issues in the build but reconfirms its finished status;
- make sure binaries stored on CDN are not corrupt during network transfer (using checksums).
The role of CI/CD pipelines
An OpenJDK distributor does quite a lot of testing to a single build. And yet, this lot pales before the massive number of binaries to be checked before release.
- 12 platforms (Windows x86 32/64 bit, macOS, Linux x86 32/64 bit, Alpine Linux 64 bit/musl, Linux ARM 32/64 bit, Solaris Sparc/x86)
- 3 JVMs (server, client, minimal)
- JDK and JRE
- 7 types of installers: msi, pkg, dmg, deb, rpm, zip, tar.gz
- 3 presets: lite, standard, full
Multiplied and with thousands of Java and JVM command-line options added, this impressive array is impossible to test manually.
Strict guidelines that help with timing are great. But no amount of discipline beats automation. Continuous integration and continuous delivery (CI/CD) systems enter the scene.
A way to implement a CI/CD system, or pipeline, is to use Jenkins as a core technology. It allows plugging in APIs, software libraries, and build tools. On its own, Jenkins does not have any functionality but gets powerful with other various tools. Groups of events or jobs are aligned together in a sequence.
Automated Liberica JDK building and testing is Jenkins-based and is executed on nodes, computers with various OS and microprocessor architectures. Not all nodes are up simultaneously; they are only enabled when needed. BellSoft has a minimum amount of them on at all times for production, to fix own and customers’ bugs.
At peak load during the release, we engage extra nodes, and the whole CI farm with cloud instances accounts for 400 to 1,200 nodes.
CI/CD pipelines are one of the reasons we manage to finish Liberica JDK quality control on time.
Final stages
Above, we have discussed events happening several weeks before the release. Everything we do is to be sure that day X goes without a hitch. On the penultimate day, X-1, we prepare Liberica JDK bundles, make release notes, and push the builds to content delivery networks (CDNs) for a better user experience.
Extra patches for customers
Finally, open source JDKs would not evolve so rapidly if it was not for the community. BellSoft is among the top 5 OpenJDK contributors because of the number of fixes and proposals the company makes. However, we have ranked this high thanks to the work we do in cahoots with Liberica JDK users.
By providing early patches and binaries for testing to our customers, we get to find solutions to known (and sometimes hidden) issues much faster. Furthermore, we send patches to functional issues before the next update release to expedite their resolution.
Want to get valuable patches before the rest of the world knows about them? Contact our engineering team. We will examine your unique case to offer what is best for your enterprise. Learn how to make the most out of the high-quality Java runtime that is Liberica JDK.
