JCMD everywhere — locally, containerized, and remotely
Published October 14, 2021
jcmd” is the command line executed with JVM diagnostic tools shipped with OpenJDK. Before the introduction of
jcmd, there were multiple tools to run live and post mortem diagnostics for JVM.
jcmd became a to-go utility for live diagnostics of JVM executed from the command line (e.g., dumping threads or inspecting JVM configuration). The post mortem diagnostics are performed with other tools.
- Why do the command line tools matter?
- Common tasks for JCMD
- Example of jcmd usage
- JVM attach protocol under Linux
- jattach — a light alternative for jcmd
- Sending jcmd commands via JMX
- Remaining caveat — accessing dump files
- Complete list of jcmd commands
- My articles
Why do the command line tools matter?
There exists a wide range of tools in the JVM ecosystem, including graphical and web, so why use the command line tools? There are a few reasons:
- Command line tools can be executed with a shell / SSH terminal, and an SSH terminal could be the only secure way to work effectively with the JVM;
- Command line tools from JDK do not need upfront configuration and rely on OS access control for security;
- Command line tools are convenient for scripts and other means of automation you may introduce.
Common tasks for JCMD
jcmd displays a long list of commands (which is included at the end of the article), and a lot of them work on the low level of programming. Here is an overview of operations available via
Capturing thread and heap dumps — this is a pretty common task for Java engineers. Such operations were available long before jcmd was introduced and were utilized by means of the dedicated tools
Control of flight recorder —
jcmd has multiple commands to start/stop/dump JFR (JDK Flight Recorder) sessions. JFR is a powerful profiling and diagnostic subsystem in OpenJDK that we already discussed.
Inspection of various aspects of JVM configuration — with
jcmd you can check the HotSpot JVM runtime setting, system properties, and command line parameters.
Control of JMX socket at runtime — using
jcmd, you can open a socket accepting JMX connection without JVM restart. This is useful if you want to use GUI tools such as Mission Control while avoiding configuring the JMX upfront.
These are the most common operations with
jcmd utilized in the daily life of a Java engineer.
Example of jcmd usage
Next, we will discover the typical jcmd workflow with examples. If you are familiar with
jcmd, you can skip the following section.
jcmd is added to PATH when installing OpenJDK. If for some reason, it does not exist in your path, you can find binary under
bin at your OpenJDK installation directory.
The first thing
jcmd requires is the PID of the JVM process you want to work with. There are multiple ways to get the PID of a JVM, and one of them is to list locally running JVM processes with
jcmd without parameters will list locally running JVMs and their respective PIDs.
> jcmd 23876 sun.tools.jcmd.JCmd 32311 HelloJDK
As you see,
jcmd includes itself in the list, because it is implemented in Java.
Once you’ve got a PID, you can list commands exposed by the JVM.
> jcmd 32311 help VM.unlock_commercial_features JFR.configure JFR.stop JFR.start JFR.dump JFR.check VM.native_memory ManagementAgent.stop ManagementAgent.start_local ManagementAgent.start VM.classloader_stats GC.rotate_log Thread.print GC.class_stats GC.class_histogram GC.heap_dump GC.finalizer_info GC.heap_info GC.run_finalization GC.run VM.uptime VM.dynlibs VM.flags VM.system_properties VM.command_line VM.version help
The list of commands may vary between versions of JDK.
Taking a thread dump is a common task beneficial for quick diagnosis. You can execute the “
Thread.print” command for that.
> jcmd 32311 Thread.print 32311: 2021-10-06 20:25:43 Full thread dump OpenJDK 64-Bit Server VM (25.302-b08 mixed mode): "Attach Listener" #9 daemon prio=9 os_prio=0 tid=0x00007f34c4001000 nid=0x5dbc waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f35040d1000 nid=0x7e44 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f35040c3800 nid=0x7e43 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f35040c2000 nid=0x7e42 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f35040bf000 nid=0x7e41 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f35040bc000 nid=0x7e40 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f3504088000 nid=0x7e3f in Object.wait() [0x00007f34efaf9000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x000000076e208ee0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked <0x000000076e208ee0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216) ...
Another typical action is dumping a heap of JVM.
> jcmd 32311 GC.heap_dump `pwd`/dump.hprof 32311: Heap dump file created
The command above will create a “
dump.hprof” file in the current directory. You can open this file with VisualVm or Eclipse Memory Analyzer Tool for detailed analysis.
Notice that I provided the absolute path (via pwd command). The relative path is interpreted from the working directory of JVM, which is not the best place for heap dump creation.
If you struggle to recall arguments for a specific command, you can use built-in help.
> jcmd 32311 help GC.heap_dump 32311: GC.heap_dump Generate a HPROF format dump of the Java heap. Impact: High: Depends on Java heap size and content. Request a full GC unless the '-all' option is specified. Permission: java.lang.management.ManagementPermission(monitor) Syntax : GC.heap_dump [options] <filename> Arguments: filename : Name of the dump file (STRING, no default value) Options: (options must be specified using the <key> or <key>=<value> syntax) -all : [optional] Dump all objects, including unreachable objects (BOOLEAN, false) -gz : [optional] If specified, the heap dump is written in gzipped format using the given compression level. 1 (recommended) is the fastest, 9 the strongest compression. (INT, 1)
An excellent addition in OpenJDK 11 and above is the
-gz option which allows the making of a compressed heap dump. Heap dumps compression is usually effective, saving both space and disk IO.
This is a typical usage of
jcmd, and I’m utilizing it reasonably often in my daily job.
When developing and running Java applications on your desktop,
jcmd works excellent. The main issue is running
jcmd as the same user as the JVM you are connecting to.
jcmd and other JDK tools work seamlessly on all supported platforms.
But in modern times, you often have to run software in the containers or on some kind of remote host.
jcmd is essential for ad hoc diagnostics of Java applications, and the ability to use it in a variety of circumstances is vital.
Running a JVM in a container is pretty standard these days, so let’s see how jcmd deals with containers.
jcmd and containerized JVM: It just works!
If you will get a fresh Liberica JDK 17 image and use it with docker,
jcmd would very likely work as expected (you will need to run
jcmd as root). Although it was not the case not so long ago, JDK made tremendous progress with the native support of Linux containers over the last few years.
But let’s say something goes wrong, and you need to take a thread dump. What next?
Why does jcmd not work?
To solve the issue, let’s check a few things first.
Are you using Linux? Linux containers run only on Linux OS. If you utilize macOS or Windows, your container is actually running in Linux VM under hypervisor on your host OS.
jcmd uses IPC and cannot access processes running under different OS.
Are you running jcmd as root? Jcmd should be able to connect to target JVM using IPC to function. That means it should either run under the same uid or as root.
What version of Java are you running? When you execute
jcmd from the console, you are using the system default JDK installation. Verify its version with
java --version command. You need to have JDK 11 or higher on your host to work with containers. Your container could use an older version of JDK, although
jcmd will still be able to see it.
What version of Linux kernel do you have? You need at least a 4.1 version of the Linux kernel because it is the release in which some container-related data in procfs was introduced, and it is essential for the functionality of
These are the most common reasons
jcmd could not be working.
What if you still need to use jcmd?
There are several options available in this situation.
Run jcmd inside the container — this is the most straightforward way. If you can open the terminal in a container and run
jcmd there, it would just work. The caveat here is that
jcmd could be missing in the container environment.
Run jcmd on the host Linux VM. If you are on Windows or macOS, try opening the terminal in the Linux VM hosting your container and run
Execute command remotely with JMX. It is possible to execute a jcmd command remotely with JMX. While it requires upfront configuration, it could be a good option once you have finished with local development and start moving things to the target platform.
JVM attach protocol under Linux
Before moving forward, I would like to give some background on the inner workings of
There are two protocols implemented in
jcmd — discovery and commands.
Once started, JVM automatically creates a file in
/tmp/hsperfdata_<user>/<pid> (id does the same for non-Linux OSes, though the path may be different).
Older versions of
jcmd were just scanning
/tmp/hsperfdata_<user> to get the list of PIDs. That doesn’t work well with containers as
/tmp in the container may not be the same as
/tmp of the host. Starting with JDK 11,
jcmd scans the list of processes and checks the presence of the
hsperfdata file for each process (using respective
pid). Thanks to the new approach,
jcmd running under root can now see JVM from any user regardless of container boundaries.
jcmd is using a Unix domain socket to establish a connection to the target JVM. JVM should open a socket for jcmd to connect.
jcmd may trigger creating a socket by sending a QUIT signal to target JVM (don’t worry, your JVM will not shut down). Socket is created under the current directory of target JVM (or under /tmp) and named
That socket uses a simple text protocol,
jcmd sends a command and receives some output and status code back.
As often occurs with Linux containers, things are not what they seem. The same JVM process may have different PIDs (e.g., PID 1 in a container and some 12345 on host OS). This was fixed for
jcmd in JDK 11, and now the container boundary is not a barrier for this tool.
jattach — a light alternative for jcmd
jattach is a compact tool written in C by Andrei Pangin that implements the protocol described above. You can find it at https://github.com/apangin/jattach.
jattach is very handy if
jcmd is not available for some reason (e.g., you have to use stripped-down JRE instead of JDK). Unlike
jcmd, which you cannot just copy as a binary,
jattach is a standalone binary of just a few kilobytes. It is easy to download and run when you need it.
jattach <PID> "jcmd …" you can enjoy the full functionality of
jcmd from JDK. Note that you need to input the whole command starting with jcmd in quotes.
Sending jcmd commands via JMX
I have mentioned the lack of need for upfront configuration as one of the advantages of
jcmd. Sometimes though, JMX access is available while terminal access is not.
In this case, you cannot use
jcmd, but precisely the same list of commands is available via JVM.
I suggest using Mission Control by BellSoft.
Execute Mission Control, connect to the remote process with JMX Console and switch to the “Diagnostics Commands” tab.
Commands are exposed as MBean operations, so they are available for other tools too.
Unfortunately, JConsole cannot deal with argument types, so only the operations without parameters can be invoked.
SJK is another open source tool for JVM diagnostics. It is capable of invoking any MBean operations through a JMX connection. If you need to invoke the
jcmd command from the terminal on a remote JVM, you can use SJK.
> java -jar sjk.jar jcmd -s host:port help The following commands are available: VM.unlock_commercial_features JFR.configure JFR.stop JFR.start JFR.dump JFR.check VM.native_memory VM.classloader_stats GC.rotate_log Thread.print GC.class_stats GC.class_histogram GC.finalizer_info GC.heap_info GC.run_finalization GC.run VM.uptime VM.dynlibs VM.flags VM.system_properties VM.command_line VM.version help For more information about a specific command use 'help <command>'.
Remaining caveat — accessing dump files
Commands that produce files do that relative to the JVM working directory. If a JVM is running in a container, then the container’s FS will be used. If you need to take a heap dump of a JVM running in a container, it is better to use a path mounted from the host as a filename for the dump file.
If no good mount is available, you can fall back to
/proc/PID/root/ mount point to access the container’s file system from the host. Note that it will not help if the container is running in a VM or remote host.
Complete list of jcmd commands
While the list of commands supported by jcmd is long, many are more useful for JVM engineers than application developers. In any case, I would like to point out all of them.
Compiler.CodeHeap_Analytics (available since JDK 11)
Dumps pretty detailed information about code produced by the JIT compiler, including addresses of compiled code blocks.
Compiler.codecache (available since JDK 11)
Prints code cache layout and bounds.
Compiler.codelist (available since JDK 17)
Prints all compiled methods in the code cache that are alive.
Compiler.directives_[add, clear, print, remove] (available since JDK 11)
Allows you to mess with the JIT compiler option on flight.
Compiler.perfmap (available since JDK 17)
Native Linux profilers expect debug symbol information for binary to be available in a specific format under
/tmp/perf-<pid>.map. Without symbol file addresses from the stack, a trace cannot be translated to meaningful method names. Previously I have been using https://github.com/jvm-profiling-tools/perf-map-agent simultaneously with Linux perf for advanced profiling. With JDK 17 release, that functionality is built into JVM and exposed via
Compiler.queue (available since JDK 11)
Prints methods queued for compilation.
This command walks heap content (beware of Stop-the-World pause here) and dumps statistics aggregated by classes. Pretty valuable for spotting obvious memory hogs without firing up a full-featured heap analyzer tool.
GC.class_stats (discontinued after JDK 11)
Similar to the command above, but shows much more statistics.
Prints info about heap objects pending for finalization.
Dumps content of JVM heap into a file for further analysis.
Prints details of heap memory spaces.
GC.rotate_log (discontinued after JDK 8)
Triggers GC log rotation. Requires specific GC log configuration.
jcmd allows you to control JDK Flight Recorder sessions. You can start and stop recording sessions and dump results to file from the terminal with
jcmd. JDK has another tool —
jfr, which can help analyze the content of JFR recording without ever leaving the terminal.
JVMTI.* (available since JDK 11)
This command could be used to enable JVMTI profiling agent without JVM restart.
A handy set of commands, which enables the JMX port without restarting JVM. Typically, JMX port should be configured via JVM start up command. With
jcmd, you can start the JMX port when you need it and start using JMX-based tools such as Mission Control, even if the JVM is not configured upfront.
Produces a JVM thread dump.
VM.cds (available since JDK 17)
CDS (Class Data Sharing) is a pretty old feature that has received many updates in JDK 12. Usually, shared class data archives are generated at JVM exit, if all required XX options were specified at the start.
jcmd offers an alternative way to create such archives.
Prints statistics about all ClassLoaders.
VM.classloaders (available since JDK 11)
Prints a hierarchy of classloaders. Optionally can include a list of classes. This command could be very helpful for troubleshooting the “A cannot be cast to A” problem.
Prints all arguments of JVM start command.
Prints list of loaded native dynamic libraries.
VM.events (available since JDK 17)
Prints latest VM events. Events are produced by various JVM substems and may be useful for troubleshooting JVM issues.
Prints effective values of HotSpot JVM XX options.
VM.info (available since JDK 11)
Prints details similar to
hserror crash dump, though without crashing the JVM :)
VM.log (available since JDK 11)
This command allows changing the configuration for JVM logging.
VM.metaspace (available since JDK 11)
This command can be used to retrieve various information about metaspace configuration and content.
This command works together with JVM native memory tracking. Tracking should be enabled on JVM startup.
VM.print_touched_methods (available since JDK 11)
Prints all methods that have ever been touched during the lifetime of this JVM. Requires
VM.set_flag (available since JDK 11)
The command allows you to modify manageable XX options of Hotspot JVM.
VM.symboltable (available since JDK 11)
Prints statistics for string and symbol tables.
Dumps properties from
VM.systemdictionary (available since JDK 11)
Prints statistics for system dictionaries per class loader.
VM.unlock_commercial_features (discontinued after JDK 8)
Rudimentary option for compatibility with older JDK. Noop in OpenJDK.
Prints uptime of JVM.
Prints JVM version details.
This is a “secret” option, not listed in help, and not available via JVM. This command prints the content of the perf counter accumulated by the JVM. Unlike other
jcmd commands, it does not require connecting to the JVM. Perf counters are exposed by JVM via
Still got questions? Book a consultation with Java runtime experts from BellSoft.
jcmd is a simple tool with a command line interface. You can use it for troubleshooting tasks, such as taking thread and heap dumps, introspecting JVM configuration, controlling flight recorder, and some others.
Starting with JDK 11,
jcmd is built with excellent support for Linux containers. In the case of using the containers, there are few key points to keep in mind:
- If you are on Windows or macOS, containers are running on Linux VM, and you have to run
jcmdon the same VM for it to work;
- You usually need to run
jcmdfrom the host can see JVM processes in the container, but if launched within the container, jcmd will not see processes outside its environment.
If you do not have
jcmd installed on the host or in the container,
jattach is a reliable and lightweight drop-in replacement that can execute the same exact commands.
You also can run the
jcmd command remotely via the JMX interface.
- Mission Control has a dedicated tab for
jcmdin JMX Console;
- SJK supports sending
jcmdcommand remotely via JMX from the command line.
jcmd is a convenient utility that will be a great help in maintaining your VM, if you learn how to run and use it properly.
Check out my other articles on JDK Flight Recorder and JVM in Linux containers.
- JDK Flight Recorder – a gem hidden in OpenJDK (part 1)
- Hunting down code hotspots with JDK Flight Recorder (part 2)
- Hunting down memory issues with JDK Flight Recorder (part 3)
- JVM in Linux containers, surviving the isolation
- JDK Flight Recorder, The Programmatic Way
- JDK Flight Recorder vs. Stop the World Pauses