Fun with Apache Tomcat JMX MBeans

developer

To enable JMX on Tomcat add these flags, we can pass this to Tomcat using the CATALINA_OPS environment variable. Here I am using port 8880, you change to any port you like.

CATALINA_OPTS="-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8880 -Dcom.sun.management.jmxremote.rmi.port=8880 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost" /system/server/tomcat/bin/startup.sh

When it starts, it should indicate that it is using our settings

# CATALINA_OPTS="-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8880 -Dcom.sun.management.jmxremote.rmi.port=8880 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost" /system/server/tomcat/bin/startup.sh
Using CATALINA_BASE:   /system/server/tomcat
Using CATALINA_HOME:   /system/server/tomcat
Using CATALINA_TMPDIR: /system/server/tomcat/temp
Using JRE_HOME:        /system/jdk
Using CLASSPATH:       /system/server/tomcat/bin/bootstrap.jar:/system/server/tomcat/bin/tomcat-juli.jar
Using CATALINA_OPTS:   -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8880 -Dcom.sun.management.jmxremote.rmi.port=8880 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost
Tomcat started.

We can confirm using ps fauxgww also.

# ps fauxgww|grep java
root       39238  0.0  0.2   7004  2296 pts/1    S+   13:59   0:00                          \_ grep --color=auto java
root       39205 14.2  9.4 2796696 90940 pts/1   Sl   13:58   0:04 /system/jdk/bin/java -Djava.util.logging.config.file=/system/server/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8880 -Dcom.sun.management.jmxremote.rmi.port=8880 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost -classpath /system/server/tomcat/bin/bootstrap.jar:/system/server/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/system/server/tomcat -Dcatalina.home=/system/server/tomcat -Djava.io.tmpdir=/system/server/tomcat/temp org.apache.catalina.startup.Bootstrap start

And via netstat if indeed that port is listening.

# netstat -nutap|grep java
tcp        0      0 0.0.0.0:8880            0.0.0.0:*               LISTEN      39205/java
tcp        0      0 127.0.0.1:8005          0.0.0.0:*               LISTEN      39205/java
tcp        0      0 0.0.0.0:45239           0.0.0.0:*               LISTEN      39205/java
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      39205/java

Connecting to the JMX port

To connect we can use 2 applications

  • jconsole.exe, usually in the bin/ folder of the Java SDK
  • jmc.exe (Java Mission Control), download from https://www.oracle.com/java/technologies/jdk-mission-control.html

I found that there are some rules you need to follow to make the connection to work. Let examine each case.

  • Tomcat is running locally, then can run jconsole/jmx and connect by attaching to Local Process or by Remote Process to localhost:8880
  • Tomcat is running on a server and there is no firewall between you and the server. Then change java.rmi.server.hostname to you server’s IP address. For example
CATALINA_OPTS="-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8880 -Dcom.sun.management.jmxremote.rmi.port=8880 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.1.127" /system/server/tomcat/bin/startup.sh

In jconsole/jmx, connect by using Remote Process to 192.168.1.127:8880

  • Tomcat is running on a server and there is a firewall between you and the server (for example an AWS security group only allowing inbound traffic on port 22, 80 and 443 only). This is the most typical scenario. Then you connect using SSH tunnel. Set java.rmi.server.hostname to localhost and if set the tunnel to listen and redirect port to the same number as the JMX port, this is important, if you make them different you won’t be able to connect.
If JMX port is 8880, then source port has to be 8880 also i.e. L8880)

Once you setup the tunnel, connect via SSH, then connect using jconsole/jmx using Remote Process on localhost:8880. There is going to be a warning about insecure connection, that can be ignored since we’re using an SSH tunnel. the tunnel is encrypted.

Jconsole

Once connected you can gather some information

See threads and what they are doing at the moment, can also detec deadlock.
What garbage collectors are running and system information
Information on MBeans, can even do operations, shown here if I click stop, then Tomcat will stop responding on port 80 (since I’m stopping the Connector:80), until I start it again.
Can be used to confirm values that are being used by Tomcat.

JMC

Java Mission Control is a more comprehensive tool (also prettier).

Can also run diagnostic commands, here I’m printing the class histogram (to see what objects are occupying memory the most)

I can also start profiling Tomcat. This is not available in jconsole.

Result of flight recorder. There are a lot of information in the report.

Note

If you get this kind of error, you need to specify -Dcom.sun.management.jmxremote.rmi.port=8880 and/or -Djava.rmi.server.hostname

org.openjdk.jmc.rjmx.ConnectionException caused by java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is: 
	java.net.ConnectException: Connection refused: connect
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:345)
	at org.openjdk.jmc.rjmx.internal.ServerHandle.doConnect(ServerHandle.java:116)
	at org.openjdk.jmc.rjmx.internal.ServerHandle.connect(ServerHandle.java:100)
	at org.openjdk.jmc.browser.wizards.ConnectionWizardPage$ConnectionTester.run(ConnectionWizardPage.java:792)
	at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is: 
	java.net.ConnectException: Connection refused: connect
	at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
	at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:209)
	at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)
	at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2105)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connectJmxConnector(RJMXConnection.java:575)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.establishConnection(RJMXConnection.java:552)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:338)
	... 4 more
Caused by: java.net.ConnectException: Connection refused: connect
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:503)
	at java.base/sun.nio.ch.Net.connect(Net.java:492)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333)
	at java.base/java.net.Socket.connect(Socket.java:648)
	at java.base/java.net.Socket.connect(Socket.java:597)
	at java.base/java.net.Socket.<init>(Socket.java:520)
	at java.base/java.net.Socket.<init>(Socket.java:294)
	at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createSocket(TCPDirectSocketFactory.java:40)
	at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
	... 13 more

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *