Async Profiler: Uncover Hidden Performance Issues in Java!

Transcript:

Some profilers are like elephants in a china shop. They are heavy and disruptive. A sync profiler is a ninja - it is small, silent, and incredibly effective. Join me as we profile a Java application with a sync profiler, both locally and in containers.

What is a sync profiler? A sync profiler is a low-overhead open-source profiler for HotSpot JVM-based applications. It is really small and without a graphical user interface, so it can be embedded into other systems, and its work is almost invisible to the application and has almost no effect on performance. But don't be misled by its miniature size - it's really powerful. A sync profiler can collect data on various JVM events: CPU usage, allocation, methods, and so on.

But how does it differ from other open-source profilers? Well, it can monitor non-Java threads, native calls, and kernel functions. And you know what? You don't need elaborate setup to use this profiler - just one command to attach it to the application and that's it. You will get profiling data as a JFR file or a flame graph. Of course, there are a couple of additional steps to use it with containers, but don't worry - you'll get a grip on it really quickly, especially after I show you how to work with this tool.

Setting up async profiler Async profiler is available for macOS and Linux. You can get the necessary package on GitHub. Locally, I'm using macOS, so I've got this build. After you unpack the archive, you will see that there are two directories: bin contains asprof that attaches and controls the agent to the running process. lib contains the profiler library that can be attached to the application as a Java agent.

Profiling a Java application locally Right, so let's first look at how we can attach to the running Java process. Start any Java application - I'm using reference Spring Boot TP Clinic. We need a PID of the process, and the easiest way is to use jps. So here it is. Instead of PID, you can use the name of the application, so you can simply use asprof to start and stop the application. Right, so you can say asprof start and PID or name of the application, and then after you're done or after some period of time, you can say asprof stop and PID.

But of course, there are additional options to configure the profiling session. So let's do that. Right, so again asprof, and you can set the event with the flag -e, for instance -e cpu. You can set the duration with -d, let's set it to 30 seconds. And you can also specify the name and the extension of the file with -f. So let's call it profile.html.

HTML means that we will get a flame graph. But of course, if you want a JFR file, the extension will be .jfr. There are lots of other options for the profiler, right? You can find them in the documentation. So there's start, stop, and then resume, dump, and so on - options for any format (HTML or JFR). Then there are options only for JFR files, and options only for flame graph files. You can profile several events, but in this case, you can get only the JFR file.

Alright, so the profiling has started. Let's just push some buttons while we wait, so it's not just staring into the screen. Right, so the profiling is done and you can now open the file in the browser. Right, so that's our flame graph. That's the color-coded representation of all the samples that profiler took. And of course, you can inspect each call in more detail. You can also attach async profiler at application start and the profiling will start at once with the application. This way you can get valuable information about application behavior when it starts. So for that purpose, we need to attach async profiler as a Java agent.

We remember that in the lib folder we have this agent libasyncProfiler.dylib in case of macOS and .so in case of Linux. If you use Linux. So let's copy the full path to the agent.Now we need to attach this agent and we are going to use the JAR file. I have already prepared one. So -javaagent:path - that's the option that we need. Then here you specify the full path to the agent. Then you say start and here you can also specify various options like events - so again, let's specify cpu. You can specify the file - right, so it's going to be again profile.html.

You can of course use other options, but for the demonstration, that's enough.

Then next we say -jar and point to our JAR file, which is in the target folder. That's basically it. So when you start the JAR file, the profiling will start. Here it is - you can see that profiling has started. Okay, and then you can stop the application and the file will be in the project directory. You can experiment with profiling modes and settings later on your own.

Profiling Java applications in containers. To profile an application with async profiler from a container, you need to include it into the image. It is really easy to do with Alpine Linux because you can simply install the async profiler package from the Alpine repository. We will need this Dockerfile. During the first stage, we specify the image that we're going to use as a builder. I'm using the Liberica Runtime container - it is based on Liberica JDK and Alpine Linux.

Right, so first stage is super obvious - we simply package our application into a JAR file. Then on the second stage, we take Liberica Runtime container with JRE (because we don't need JDK). Here the important thing is that we run apk add async-profiler, right - we are adding the async profiler package. Then you just specify the entry point.

Alright, so again, we need to specify the absolute path to the agent - async profiler. The profiler will be in the /opt folder - /opt/async-profiler/lib/asyncProfiler.so, right. So as I said, that's the extension for Linux. This package also contains asprof, so we get both the agent and the tool that controls it - this is very useful. I'll show it to you later. Of course, you can specify the necessary options - right, so start the profiler with the application. Let's specify the event and file - it's going to be saved into the /tmp folder of the container. Then you just run the JAR file. Great. But before we run the container image, we need to configure the Linux kernel to allow access to the performance event. For that purpose, we need to set two options: sudo sysctl kernel.perf_event_paranoid=1 sudo sysctl kernel.kptr_restrict=0 I have already enabled these options, so I'm not going to do that. But you have to.

These options are available for Linux only and are necessary in the container - otherwise, the profiling data won't be complete. For macOS, you might not need them, because we are profiling containers and there's Linux in containers usually. Before we go any further, it's important to know that Docker containers restrict access to performance events. So you can use one or several of the following approaches: Add the capability SYS_ADMIN to access privileged performance event info Disable the default seccomp profile with --security-opt seccomp=unconfined Use FD transfer with privileged container Fall back to cstack timer mode which doesn't need perf events Taking that into consideration, the command for running the container may look like this: docker run -p 8080:8080 --name pet \ --cap-add=SYS_ADMIN \ --security-opt seccomp=unconfined \ your-image-name The profiling has started. Success!

How do we get the profiling data? The file profile.html will be created automatically in the /tmp directory when you stop the container. But what if you don't want to stop a running container (e.g., in production)? In this case, you can stop the profiling session manually. Let's enter the container shell: docker exec -it pet /bin/sh We are inside. I already said the async profiler package includes asprof.

We can use it to stop the profiling session, or resume/dump data without stopping the session. Let's find the PID of the process - usually it's 1, but let's be sure: ps Yes, it's 1. Now use asprof: asprof dump -f /tmp/profile.html -p 1 That's it. Now exit the container. To copy the file: docker ps -a docker cp :/tmp/profile.html . Success! That's your file. Buildpacks If you use Buildpacks, there are some considerations for using async profiler. First: async profiler requires the libstdc++ library. Alpine includes it, but Buildpacks (like Spring Boot 3.4's tiny builder) do not. So, you should change the builder to paketobuildpacks/builder-jammy-base. Add this config to spring-boot-maven-plugin: paketobuildpacks/builder-jammy-base

Also, Buildpacks create layered jars. Extra files outside standard layout will be discarded. Solution: put async profiler in the resources directory. Rename the profiler lib to just profiler.so, put it in resources, and remember that it ends up under: /workspace/BOOT-INF/classes/ Add the JVM option: BP_JAVA_OPTS="-javaagent:/workspace/BOOT-INF/classes/profiler.so=start,event=cpu,file=/tmp/profile.html" Then build the image: ./mvnw spring-boot:build-image Now run the container like before. Catch: If you enter the container and try to stop profiling with asprof, it won't work - permission denied. Because the container runs as non-root, and this can't be changed. So you need to stop the container and then extract the file with docker cp. But there's another way - profiling from the host You don't need to include the profiler in the container. You just bind-mount it and use it externally. docker run -d -p 8080:8080 --name pet \ --cap-add=SYS_ADMIN \ -v /absolute/path/to/profiler:/profiler \ your-image-name Then, on the host: docker top pet sudo /profiler/asprof -e cpu -d 15 -f /tmp/profile.html -p Once done: docker ps docker cp :/tmp/profile.html . There it is - your profiling file. Async profiler is small, powerful, and easy to use. Try it with your application. Code snippets are available in the article linked in the description.

Don't forget to like, subscribe, and see you next time!