How to profile Java applications in Docker containers

Profiling the containerized application is essential for monitoring the behavior of the program. It also helps you to promptly pinpoint the root causes of possible performance issues.

In my previous article, I looked at different profiling tools for Java applications. In this article, I will use one of them, Java Flight Recorder to work with containerized workloads. In the following blog post, I will show you how to use Async Profiler to profile Java apps in containers.

JFR, which is an integral part of JVM, is a low-overhead profiler and fairly easy to use. We will look into two approaches to profiling the containerized application: continuously from the application start and an arbitrary point of time. I will also show how to profile applications if you use buildpacks. Feel free to jump to the section most relevant to you.

Profiling in containers: basic concepts

There are three key concepts related to profiling containerized applications we need to understand.

We have a target environment, which is our containerized application, and a host environment, which is a developer machine used for controlling the profiling process and analysing the resulting profiling data. We also have a profiling tool: it can be part of a target container or attached to it externally. JFR is part of JVM and so it is always included into the target environment. 

Profiling Java applications in containers

Target and host environments can be located on one or two machines. 

There are two approaches to profiling the application. You can perform continuous profiling right from the application start. In this case, the profiler will be a part of the target environment, namely the container image with the application. If you use a third-party profiler, you need to add it to the container image.

However, in most cases, we don’t want to profile the application continuously, but rather attach to it, take a few samples, and see how the app is doing at this moment. In this case, the profiler will be in a separate environment, namely the ephemeral container. Note that we can’t detach the JFR from the JVM, so it will remain in the target container. However, we can control it with the help of jcmd, which is part of the JDK. So, the ephemeral container will be based on a full Java Development Kit with jcmd.

If you use a third-party profiler, you can also add it to the ephemeral container.

The ephemeral containers are started at an arbitrary point of time and run temporarily to accomplish some task: for instance, perform the debugging or profiling session.

These containers are useful when you deploy distroless containers that don’t contain a shell, debugging utilities, or profiling tools.

Setting up the environment for profiling

For this tutorial, I will use a reference Spring Boot application, Petclinic.

There are two ways to containerize the application: using a Dockerfile or buildpacks. I will show you the work process with both approaches.

To build a container, I will use Liberica Runtime Container based on Liberica JDK Lite optimized for the cloud and Alpaquita Linux. Liberica JDK is recommended by Spring and is used by default in Spring buildpacks.

Profiling Docker containers with Java Flight Recorder

Use JFR in a Docker container at application start

You can start a JFR recording right at application startup. JFR is part of JVM, so you don’t have to add it manually.

If you use Dockerfiles, you need to specify the -XX:StartFlightRecorder option in the Java command line with a set of necessary arguments separated by comma. You can also specify the -XX:+UnlockDiagnosticVMOptions and -XX:+DebugNonSafepoints to make the profiling more accurate:

FROM bellsoft/liberica-runtime-container:jdk-21-stream-musl as builder

WORKDIR /app
ADD spring-petclinic-main /app/spring-petclinic-main
RUN cd spring-petclinic-main && ./mvnw clean package

FROM bellsoft/liberica-runtime-container:jre-21-stream-musl

WORKDIR /app
EXPOSE 8081
ENTRYPOINT ["java", "-jar", \
"-XX:+UnlockDiagnosticVMOptions" \
"-XX:+DebugNonSafepoints" \
"-XX:StartFlightRecorder=duration=30s,filename=/tmp/recording.jfr", \
"-XX:MaxRAMPercentage=80.0", "/app/petclinic.jar"]

COPY --from=builder /app/spring-petclinic-main/target/*.jar /app/petclinic.jar

Here, we have set the recording time to 30 seconds. But you can let JFR run silently in the background. As JFR is a low-overhead profiler, continuous profiling won’t significantly impact the application performance.

To do that, you need to specify 

-XX:StartFlightRecording=name=background,maxsize=100m

The maxsize is the limit to the events stored in memory.

Then, start the container the usual way:

docker run -p 8082:8080 --name pet petclinic

The recording will be in the /tmp directory of the container.

To copy the file to the current directory, run docker cp:

docker cp <container-id>:/tmp/recording.jfr .

Use JDK Mission Control with Docker containers

You can observe the running Java application and receive profiling data in real time using JDK Mission Control.

First of all, you need to install JDK Mission Control. It is not part of Java runtimes, but you can download it separately from vendors that provide it. For instance, you can get Liberica Mission Control.

Next, to connect to a remote JVM you need to configure the Java Management Extensions (JMX) with the following JVM environment variables:

• -Dcom.sun.management.jmxremote to enable the monitoring from a remote system;
• -Djava.rmi.server.hostname to specify the host. Set 127.0.0.1 for localhost;
• -Dcom.sun.management.jmxremote.rmi.port for the RMI port and -Dcom.sun.management.jmxremote.port for the JMX port. Both must have the same value;
• -Dcom.sun.management.jmxremote.authenticate and -Dcom.sun.management.jmxremote.ssl to set JMX authentication and SSL if required. Set to false if not needed.

Specify these options in the ENTRYPOINT in your Dockerfile:

FROM bellsoft/liberica-runtime-container:jdk-21-stream-musl as builder

WORKDIR /app
ADD spring-petclinic-main /app/spring-petclinic-main
RUN cd spring-petclinic-main && ./mvnw clean package


FROM bellsoft/liberica-runtime-container:jre-21-slim-musl

WORKDIR /app
EXPOSE 8082

ENTRYPOINT ["java", "-jar", "-Dcom.sun.management.jmxremote", \
"-Djava.rmi.server.hostname=127.0.0.1", \
"-Dcom.sun.management.jmxremote.rmi.port=7091", \
"-Dcom.sun.management.jmxremote.port=7091", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
"-XX:MaxRAMPercentage=80.0", "/app/petclinic.jar"]

COPY --from=builder /app/spring-petclinic-main/target/*.jar /app/petclinic.jar

Build the container and start it. Note that you need to specify ports both for the application and the connection ports:

docker run -p 8082:8080 -p 7091:7091 --name pet petclinic-jmx

Note that instead of port mapping you can use the host name that Docker generates for each container.

After starting a container with enabled JMX, you need to connect to the remote JVM. Open Liberica Mission Control, select New Custom JVM Connection, and specify the host name and port:

After that, you can connect to the JVM in your container. Right click on the created connection and select Start JMX Console:

Use ephemeral containers to profile an application with JFR and jcmd

The scenario we discussed above was fairly simple to implement. But what if you want to attach to a running container at an arbitrary point of time?

In this case, you will need jcmd to control the JFR. It is a utility that sends diagnostic commands to the JVM. 

The jcmd tool is part of JDK, but we don’t want to use a JDK base image for running the application and increase the size of the container, do we?

The solution is to use ephemeral containers.

Containerize the application without any additional JFR-related options.

Then, build another container image based on the following Dockerfile:

FROM bellsoft/liberica-runtime-container:jdk-21-stream-musl

This will be our ephemeral container. It is based on Alpaquita Linux and Liberica JDK containing jcmd.

Run the containerized application with a standard command:

docker run -p 8082:8080 --name pet pet-prof-jfr

 

To attach the ephemeral container, we need to place it in the same PID namespace as our application with --pid=container:<container-name>. We also need the -it flag to run it in the interactive mode.

Note that Docker containers restrict the access to perf_event_open syscall. So, to gain access to the JVM performance information, you need to specify the --cap-add SYS_ADMIN capability when starting the container. You may also need to modify the seccomp profile or disable it with --security-opt seccomp=unconfined.

docker run --cap-add SYS_PTRACE --security-opt=apparmor:unconfined --name ephem --pid=container:pet -it prof-jcmd sh

Cool! We are inside the ephemeral container. From here, run the jcmd command to find out the PID of the Java application running in a container we attached to:

/app # jcmd
1 /app/petclinic.jar
62 jdk.jcmd/sun.tools.jcmd.JCmd

Finally, you can start a JFR recording with the jcmd command specifying the PID of the process, 1 in this case. You can also configure the recording as you see fit. For instance, you can set the duration and filename (the full list of options can be found in the docs):

/app # jcmd 1 JFR.start duration=15s filename=/tmp/recording.jfr
1:
Started recording 1. The result will be written to:

/tmp/recording.jfr

After the profiling is complete, you can retrieve the recording from the target container with docker cp:

docker cp <container-id>:/tmp/recording.jfr .

Profiling containers with JFR and buildpacks

First, let’s look at the scenario where you want to start the profiling session right at the application start.

In the case of buildpacks, build a container image without any additional configuration.

Maven:

mvn spring-boot:build-image

Gradle:

gradle bootBuildImage

Then, specify the following environment variables when starting a container:

• BPL_JFR_ENABLED set to true to enable JFR;
• BPL_JFR_ARGS (optional) to specify a list of necessary arguments separated with a comma. If you don’t specify this variable, the default JFR settings will be used: dumponexit=true and filename=<system-temp-dir>/recording.jfr.

An example command for enabling JFR in a container built with buildpacks:

docker run --env BPL_JFR_ENABLED=true --env BPL_JFR_ARGS=filename=/tmp/jfr-recording.jfr,duration=15s -p 8082:8080 --name pet petclinic

The recording will be in the /tmp directory of the container. You can get it with:

docker cp <container-id>:/tmp/recording.jfr .

What if you want to monitor the application in Mission Control?

If you use buildpacks, you should use the BPL_JMX_ENABLED environment variable set to true when starting the container. The following settings will be provided to the JVM by default: 

-Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.rmi.port=5000

You can also use the BPL_JMX_PORT environment variable to change the port.

An example command for enabling JFR in a container built with buildpacks:

docker run -e BPL_JMX_ENABLED=true -e BPL_JMX_PORT=5001 -p 8082:8080 -p 5001:5001 --name pet petclinic

If you need to change the host name, you should provide the JMX arguments as JVM options in the buildpacks configuration INSTEAD of using the BPL_JMX_ENABLED variable. In Maven, you can do that with BPE_APPEND_JAVA_TOOL_OPTIONS:

<BPE_DELIM_JAVA_TOOL_OPTIONS xml:space="preserve"> </BPE_DELIM_JAVA_TOOL_OPTIONS>
<BPE_APPEND_JAVA_TOOL_OPTIONS>-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=192.33.22.11 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=5001 -Dcom.sun.management.jmxremote.rmi.port=5001</BPE_APPEND_JAVA_TOOL_OPTIONS>

And then, start the container:

docker run -p 8082:8080 -p 5001:5001 --name pet petclinic

You can now connect to the remote JVM via Java Mission Control (see the instructions in the Use Java Mission Control Section).

And finally, if you want to start the recording at an arbitrary point of time, you can use jcmd and ephemeral containers as described in the Section above.

Conclusion

In this article, we looked into profiling containerized Java applications with Java Flight Recorder. As JFR is part of JVM, using it is fairly easy. You can profile the application at start up or attach an ephemeral container with jcmd to profile the app at any time.

In the following blog post, we will see how to profile Java apps with Async Profiler, another low-overhead open-source Java profiler. Subscribe to our newsletter so as not to miss it!