Profiling Java Docker containers with Async Profiler

In the previous article, we looked at ways of profiling Java applications in containers with Java Flight Recorder (JFR), a profiling a diagnostics tool built into the JVM.

In this article, I will show you how to do the same with Async Profiler, an open-source profiler.

There are two approaches to profiling a containerized Java application with a third-party profiler: from within the container with the application and from the host. I will show you both techniques.

What is Async Profiler?

Async Profiler is an open-source low-overhead profiler for Java. It collects information about various JVM events, including but not limited to CPU usage, heap allocation, method invocation, lock contention.

Async Profiler is easy to use, it is very small and without GUI, so it can easily be embedded into other solutions. For instance, it is used under the hood of IntelliJ Profiler.

You might be familiar with the in-built JDK Flight Recorder, but how does Async Profiler differ from it? Async Profiler is a third-party profiler, so you have to download the binary to use it. If you want to use it in the container, the easiest way is to install it as a package from the Linux repository. Async Profiler can profile all calls, including native ones, and kernel functions.

The data is collected in the form of flame graphs, plain text, or JFR files.

Flame graphs are a hierarchical visualization of stack traces of profiled software. They can help developers quickly identify most frequently executed code paths. Async Profiler supports generation of interactive Flame graphs that accumulate data for one JVM event.

Example of a Flame graph

If you want to profile multiple JVM events at once, the only output format is a JFR file. JFR files can be analyzed in Mission Control.

You can read more about this and other profiling tools in the Overview of top Java profiling tools.

Use Async Profiler in a Docker container upon application start

To profile an application with Async Profiler from a container, you need to include it into the image. It is easy to do with Alpaquita Linux as you can simply install the async-profiler package from the repository.

FROM bellsoft/liberica-runtime-container:jdk-21-stream-musl as builder

WORKDIR /app
ADD spring-petclinic-main /app/spring-petclinic-main
RUN cd spring-petclinic-main && ./mvnw clean package


FROM bellsoft/liberica-runtime-container:jre-21-stream-musl
RUN apk add async-profiler

WORKDIR /app
EXPOSE 8082
ENTRYPOINT ["java", "-agentpath:/opt/async-profiler/lib/libasyncProfiler.so=start,event=cpu,file=/tmp/profile.html", \
"-jar", "-XX:MaxRAMPercentage=80.0", "/app/petclinic.jar"]
COPY --from=builder /app/spring-petclinic-main/target/*.jar /app/petclinic.jar

Note that we are not using a “slim” base image because we need a package manager to add Async Profiler. Also note that when you add the async-profiler package from the Alpaquita repository, the libstdc++ package required by Async Profiler will be added automatically.

In the ENTRYPOINT, we attach the profiler as a Java agent and provide some options such as starting the profiler upon application startup, the event we want to profile, and the name of the file with the profiling data. Other available options are described in the docs.

It is also possible to perform interactive profiling in the container. I’ll show you how to do that below.

After building the image, start the container. As Docker containers restrict the access to perf_event_open syscall, you can use one or several of the following approaches:

• Add the --cap-add SYS_ADMIN option to access privileged perf event information;
• Disable the default Docker seccomp profile that disables some system calls with the --security-opt seccomp=unconfined option;
• Use --fdtransfer together with running the container as privileged user to allow using perf_events;
• Fall back to -e ctimer profiling mode, which is similar to cpu mode, but does not require perf_events support.

In addition, you need to configure the Linux kernel to allow access to the performance events system. For that purpose, set the perf_event_paranoid to 1 and kernel.kptr_restrict to 0:

sudo sysctl kernel.perf_event_paranoid=1
sudo sysctl kernel.kptr_restrict=0

Note that these options are available for Linux only, so if you use macOS, the profiling information will be limited.

The command for running the container may look like this:

docker run -p 8082:8080 --name pet --cap-add SYS_ADMIN --security-opt seccomp=unconfined petclinic-async
Profiling started

Now, how do we get the profiling data? The profile.html file will be created automatically in the /tmp subdirectory of the container when you stop it, but what if we don’t want to stop the container running in production? In this case, you can sh into the running container with:

docker exec -it pet sh

We are inside the container. As we have installed the async-profiler package, the asprof tool is at our command. It can be used to stop the profiling session, resume it, or dump data without stopping the profiling.

Let’s find out the PID of the process, which we need to pass to the asprof tool. Then, use asprof to dump the data into the file:

/app # ps
PID   USER     TIME  COMMAND
    1 root      0:14 java -agentpath:/opt/async-profiler/lib/libasyncProfiler.so=start,event=cpu,file=/tmp/profile.html -jar -XX:MaxRAMPercentage=80.0 /app/petclinic.jar
   47 root      0:00 sh
   64 root      0:00 ps
/app # asprof dump -f /tmp/profile.html 1

After that, copy the file to the current directory by running:

docker cp <container-id>:/tmp/profile.html .

This approach can be used for interactive profiling as well. Simply sh into the running container and use asprof to start the profiling session. It can be as simple as running

/app # asprof start 1

Alternatively, use the options specified in the documentation to adjust the profiling as you see fit.

Profile Java containers with Async Profiler from host

If you have full access to the host server, profiling from the host can be very convenient. First of all, you can tap into a running container at any point in time without having to run a profiler in the background continuously. Secondly, you don’t have to fiddle with additional settings in the image or Docker.

To profile a Docker container from the host, you need to run Async Profiler as a privileged user. Also, the container must be able to access the profiler by the same absolute path as on the host. To do that, start the container with the following command:

docker run -v /absolute/path/to/async-profiler-3.0-linux-x64:/absolute/path/to/async-profiler-3.0-linux-x64 -d -p 8082:8080 --name pet --cap-add SYS_ADMIN petclinic

Now, let’s find out the PID of a running container as we need to provide it to the profiler:

docker top pet

Finally, you can start the profiling session. Instead of PID use the number you got as a result of the previous command:

sudo ~/async-profiler-3.0-linux-x64/bin/asprof -e cpu -d 15 -f /tmp/profile.html PID

The profile.html file will be saved to the /tmp subdirectory of the container. 

Use Async Profiler with buildpacks

If you use buildpacks to build your application container images, adding additional OS packages or placing additional files to the image is a bit more complicated.

Buildpacks create a layered JAR by default. Layered JARs are a cool feature that helps to accelerate the updates of Docker images, but the problem is that files that don’t match the layout of four default layers are automatically discarded.

If you want to profile the application with Async Profiler from the start, the profiler must be in the target container. One way of solving the above issue is to mount the profiler to the ready image.

First of all, we need to add the JVM option specifying the path to the profiler and its options with BPE_APPEND_JAVA_TOOL_OPTIONS:

<BPE_DELIM_JAVA_TOOL_OPTIONS xml:space="preserve"> </BPE_DELIM_JAVA_TOOL_OPTIONS>
<BPE_APPEND_JAVA_TOOL_OPTIONS>-agentpath:/workspace/profiler/lib/libasyncProfiler.so=start,event=cpu,file=/tmp/profile.html</BPE_APPEND_JAVA_TOOL_OPTIONS>

After that, build the container image.

Then, start the container with a volume that contains the profiler. Use -v with two fields, where the first field is the path to the profiler on the host, and the second field is the path where the profiler is mounted in the container (the one we specified in the plugin config):

docker run -v ./profiler/lib/libasyncProfiler.so:/workspace/profiler/lib/libasyncProfiler.so -p 8082:8080 --name pet --cap-add SYS_ADMIN petclinic

The file with the profiling data can be found in the /tmp subdirectory of the container.

Another solution is to create a custom layer for a profiler or any other necessary files such as configs while we are at it. The topic of creating custom Spring Boot layers deserves a separate article, which I am already working on. You can subscribe to our newsletter so as not to miss it.

Conclusion

In this article, we looked into approaches to profiling containerized Java applications with Async Profiler, a low-overhead open source profiler.

You can also profile Java apps with Java Flight Recorder, a profiling and diagnostics tool built into the OpenJDK. Refer to my previous guide for more details.