Production setup & configuration & settings for Apache Tomcat 8 & 9 & 10 on Ubuntu 20.04 & Ubuntu 22.04

	Ubuntu 20.04 focal fossa	Ubuntu 22.04 jammy jellyfish
Tomcat 8	✅	✅
Tomcat 9	✅	✅
Tomcat 10	✅	✅

Preparation

Let’s create a separate folder structure to house our work. Download a JDK, I use OpenJDK (https://jdk.java.net/archive/), get the latest or LTS (Long Term Support) version.
I also find https://endoflife.date/openjdk-builds-from-oracle quite helpful to figure out end of life. Download the .tar.gz version of Apache Tomcat from https://tomcat.apache.org/.
Run all commands as root (run sudo bash to become root).

mkdir -p /system/storage /system/server
cd /system/storage
wget https://download.java.net/java/GA/jdk20.0.2/6e380f22cbe7469fa75fb448bd903d8e/9/GPL/openjdk-20.0.2_linux-x64_bin.tar.gz
wget https://dlcdn.apache.org/tomcat/tomcat-9/v9.0.80/bin/apache-tomcat-9.0.80.tar.gz

Don’t forget to check checksums and compare.

# sha256sum openjdk-20.0.2_linux-x64_bin.tar.gz
beaf61959c2953310595e1162b0c626aef33d58628771033ff2936609661956c  openjdk-20.0.2_linux-x64_bin.tar.gz
# sha512sum apache-tomcat-9.0.80.tar.gz
24014441b0ccdd2dda238efa56e1a039476488943e6cf04f8a372a340a49dd21ce174ed68e2f5fcc43401e85fae6d00c5eac3d357653e91601737b6fa94476de  apache-tomcat-9.0.80.tar.gz

I usually do a search (Ctrl-F) in the browser and paste the checksum, if it matches it will be highlighted fully.

Extract, make sure root owns the files and create a symlink to the current version we are using. If later we install a newer version, delete the symlink and create a new one pointing to the newer version. This way we don’t need to change our scripts.

mkdir -p /system/server
cd /system/server
tar -zxvf /system/storage/apache-tomcat-9.0.80.tar.gz
chown root:root -R apache-tomcat-9.0.80
ln -s apache-tomcat-9.0.80 tomcat
cd /system
tar -zxvf /system/storage/openjdk-20.0.2_linux-x64_bin.tar.gz
chown root:root -R jdk-20.0.2
ln -s jdk-20.0.2 jdk

It might be worthwhile to create a bash script called env.sh in /system (you can call it any name you want) so Tomcat knows where JAVA_HOME is and to put our environment variables like CATALINA_BASE, CATALINA_HOME and CATALINA_OPTS.

Content of /system/env.sh.

#!/bin/bash
export PATH=/system/jdk/bin:$PATH
export JAVA_HOME=/system/jdk
export CATALINA_OPTS=

To use this script, use this command:

# source /system/env.sh

This will sort of “import” those variables into your current bash session.
To check use the command env.

# env 
or
# env | grep -i java

Let’s clean up Tomcat files, to make it more secure. Remove everything under webapps/. Under conf/ rename some files to “disable” them.

cd /system/server/tomcat
rm -rf webapps/*
cd /system/server/tomcat/conf
mv tomcat-users.xml tomcat-users.xml.original
mv tomcat-users.xsd tomcat-users.xsd.original
cp server.xml server.xml.original
cp catalina.properties catalina.properties.original
cp web.xml web.xml.original

I know of 2 ways to run Tomcat:

1. Run directly from the Tomcat folder (this is what we are using now), which means you will be changing files under the conf/ folder and having log files under log/ folder.
2. Create a new folder structure containing configuration and deployment files and leave the Tomcat folder pristine. This is basically you telling Tomcat, use the executables in /system/server/tomcat but for configuration use this other folder. This way you can switch/upgrade to another Tomcat version a lot easier. This is covered in another guide.

There are also several ways of deploying applications in Tomcat:

• Drop a war in webapps/
• Drop an “exploded” war (i.e. an extracted war file) under webapps/
• Use Catalina/ folder
• Specifying each application’s folder in server.xml (this is what we are using now).
  All these options are explained in another guide.

Configuration file server.xml

Here is minimalist version of server.xml. This should work on Tomcat version 8 or 9 or 10.

<?xml version="1.0" encoding="UTF-8"?>

<Server port="8005" shutdown="TERMINATE">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <Service name="Catalina">

    <Connector port="80" protocol="HTTP/1.1"
               maxThreads="200"
               connectionTimeout="20000"
               redirectPort="443"
               compression="on"
               compressionMinSize="2048"
               compressibleMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript" />

    <Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="200"
               SSLEnabled="true"
               relaxedPathChars="^"
               compression="on"
               compressionMinSize="2048"
               compressibleMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript">
        <SSLHostConfig protocols="TLSv1.2, TLSv1.3">
            <Certificate certificateKeyFile="/system/pki/privkey.pem"
                         certificateFile="/system/pki/cert.pem"
                         certificateChainFile="/system/pki/chain.pem"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

    <Engine name="Catalina" defaultHost="default">
      <Host name="default" autoDeploy="false" appBase="/system/site/default">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs/default" prefix="access_log" suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b %{User-Agent}i" />
      </Host>
      <Host name="one.zestycoder.com" autoDeploy="false" appBase="/system/site/one">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs/one" prefix="access_log" suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b %{User-Agent}i" />
      </Host>
      <Host name="two.zestycoder.com" autoDeploy="false" appBase="/system/site/two">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs/two" prefix="access_log" suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b %{User-Agent}i" />
      </Host>
    </Engine>
  </Service>
</Server>

Some explanations

<Server port="8005" shutdown="TERMINATE"> <- change this keyword to something else, in this case it has been changed to "TERMINATE"

  <Service name="Catalina">

    <Connector port="80" protocol="HTTP/1.1"
               maxThreads="200" <- change maxThreads depending on capacity i.e. number of cores, RAM size, maybe 200 per core?
               connectionTimeout="20000"
               redirectPort="443"
               compression="on"
               compressionMinSize="2048" <- if size of data to be transferred is more than 2048 bytes, Tomcat will compress before sending               compressibleMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript" />

    <Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="200"
               SSLEnabled="true"
               relaxedPathChars="^" <- optional, our application sometimes need to send the character "^" in HTTP GET requests, since not part of specification, we are telling Tomcat, do not panic if you see the "^" character
               compression="on"
               compressionMinSize="2048"
compressibleMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript"> <- remove this connector if you do not need https at the moment
        <SSLHostConfig protocols="TLSv1.2, TLSv1.3">
            <Certificate certificateKeyFile="/system/pki/privkey.pem"
                         certificateFile="/system/pki/cert.pem"
                         certificateChainFile="/system/pki/chain.pem"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

    <Engine name="Catalina" defaultHost="default">
      <Host name="default" autoDeploy="false" appBase="/system/site/default"> <- make sure autoDeploy is set to false, so Tomcat does not keep scanning if there is a newer version of the application, if you only have 1 site you do this, if you have many i.e. virtual hosting, you can add more Host elements
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs/default" prefix="access_log" suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b %{User-Agent}i" /> <- change this according to your needs, for me, I need to know User-Agent
      </Host>
    </Engine>

  </Service>
</Server>

There are other settings for advanced tuning

1. Linux’s /etc/security/limits.conf
2. Tomcat’s Java TCP socket attributes, i.e. the NIO and NIO2 settings

Disable jar scanning to (slightly) speed up startup

Disable jar scanning during startup, this should speed up your boot time
Open /system/server/tomcat/conf/catalina.properties
Under tomcat.util.scan.StandardJarScanFilter.jarsToSkip.
Add *.jar
You may want to modify the tomcat.util.scan.StandardJarScanFilter.jarsToScan as well (i.e. remove all the entries there)

Enable precompression

Enable “send precompressed version of file if available” feature.
What this does is, if Tomcat sees there is compressed version of a file, it will send that instead (instead of either sending it directly to the browser or compressing it first on the fly and send to browser).
This is especially useful if you have large uncompressed static files, e.g. “transpiled” Vue.js or Reactjs dist/ files.
Let’s say you have these 2 large transpiled JavaScript files: app.e16fb0c6.js and chunk-vendors.823f1411.js.
Compress them with this command:
gzip -k app.e16fb0c6.js chunk-vendors.823f1411.js
You’ll then end up with this.
Now, whenever a browser requests for chunk-vendors.823f1411.js, Tomcat sees that there is chunk-vendors.823f1411.js.gz, it will send that instead and indicate to the browser that this is compressed version, once you receive it, uncompress it.
But the amount of data being sent will be smaller since it is compressed, and Tomcat doesn’t need to do any work in compressing it on the fly again and again.

Open /system/server/tomcat/conf/web.xml and add this piece of text

<init-param>
	<param-name>precompressed</param-name>
	<param-value>true</param-value>
</init-param>

Change server version text

Optionally you may want to change the server version information that is returned to clients (it is also possible to disable it entirely by configuring an ErrorReportValve).

If the version number of Apache Tomcat were visible to an intruder, they could use that information to search for known vulnerabilities of the app. Removal of unneeded or non-secure functions, ports, protocols, and services mitigate the risk of unauthorized connection of devices, unauthorized transfer of information, or other exploitation of these resources.

stigviewer.com

Create a file called /system/server/tomcat/lib/org/apache/catalina/util/ServerInfo.properties with this content (you can write any text you want obviously)

server.info=My Ultimate Server v5.10.9

Run server

Let’s start without SSL and only 1 virtual host defined (the default one).
Use this server.xml

<?xml version="1.0" encoding="UTF-8"?>

<Server port="8005" shutdown="TERMINATE">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <Service name="Catalina">

    <Connector port="80" protocol="HTTP/1.1"
               maxThreads="200"
               connectionTimeout="20000"
               redirectPort="443"
               compression="on"
               compressionMinSize="2048"
               compressibleMimeType="text/html,text/xml,text/plain,text/css,application/json,application/javascript" />

    <Engine name="Catalina" defaultHost="default">
      <Host name="default" autoDeploy="false" appBase="/system/site/default">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs/default" prefix="access_log" suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b %{User-Agent}i" />
      </Host>
    </Engine>

  </Service>
</Server>

Create the folder structure that hosts our application. Create /system/site/default/ROOT/index.html

mkdir -p /system/site/default/ROOT
cd /system/site/default/ROOT
echo 'Welcome' > index.html

The “ROOT” there indicates that that is the application that should run if clients goes to “http://[domainName]/” (i.e. the root context path).
If you want an application to be running on a different context path, you can define /system/site/default/anotherContextPath/index.html, then clients will see that application when they go to “http://[domainName]/anotherContextPath/” on their browsers.

If Tomcat sees a WEB-INF folder under a folder, it will read and deploy the web.xml in it, thus deploying your Java web application.

If you need to specify a DataSource or JNDI resource, you can do so by creating a context.xml.default file in /system/server/tomcat/conf/Catalina/default/context.xml.default.

Here is an example, Tomcat by default is only able to recognize certain classes only e.g. javax.sql.DataSource and JavaBeans (you can write your own factory if you need).

<Context>
        <Resource name="myDataSource"
                auth="Container"
                type="javax.sql.DataSource"
                driverClassName="org.mariadb.jdbc.Driver"
                url="jdbc:mariadb://localhost:3306/database"
                username="user"
                password="password"
                maxActive="100"
                maxIdle="20"
                minIdle="5"
                maxWait="10000"
        />
        <Resource name="myVariable"
                type="example.SimpleBean"
                factory="org.apache.naming.factory.BeanFactory"
                fieldOfSimpleBean="Initial value"
        />
</Context>

If you want to have a global JNDI resource that are shared between applications, then you have to specify it server.xml and in each application that needs to use it, use a ResourceLink.
If you use global JNDI resource and the resource has minIdle=”5″ Tomcat will open 5 idle connections to the database.
If you use application level JNDI resource, and 2 applications uses their own JNDI resource, and each resource has minIdle=”5″, then Tomcat will open 2 x 5 = 10 idle connections.
So if the JNDI resource is the same database, it might be beneficial to use a single global JNDI resource. This is explain in this guide.

For testing, let’s start and stop Tomcat manually.
Make sure JAVA_HOME is defined.
Here is one way to find out

# env|grep JAVA_HOME
JAVA_HOME=/system/jdk

If you don’t see JAVA_HOME defined there, you can source /system/env.sh or define it manually with export JAVA_HOME=/system/jdk

Then run /system/server/tomcat/bin/startup.sh (since Tomcat needs to bind to port under 1024, namely 80 and/or 443, you need to run this as root)
You should see something like this

# /system/server/tomcat/bin/startup.sh
Using CATALINA_BASE:   /system/server/tomcat
Using CATALINA_HOME:   /system/server/tomcat
Using CATALINA_TMPDIR: /system/server/tomcat/temp
Using JRE_HOME:        /system/jdk
Using CLASSPATH:       /system/server/tomcat/bin/bootstrap.jar:/system/server/tomcat/bin/tomcat-juli.jar
Using CATALINA_OPTS:
Tomcat started.

To shutdown do

# /system/server/tomcat/bin/shutdown.sh
Using CATALINA_BASE:   /system/server/tomcat
Using CATALINA_HOME:   /system/server/tomcat
Using CATALINA_TMPDIR: /system/server/tomcat/temp
Using JRE_HOME:        /system/jdk
Using CLASSPATH:       /system/server/tomcat/bin/bootstrap.jar:/system/server/tomcat/bin/tomcat-juli.jar
Using CATALINA_OPTS:
NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

Check server log at /system/server/tomcat/logs/catalina.out, it should say something like below, if Tomcat encounters errors/unable to deploy an application, if will say so in catalina.out, you do need to scroll up to find them i.e. Tomcat may still start but not run the troubled application.

29-Sep-2023 05:59:46.159 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/system/site/default/ROOT] has finished in [417] ms
29-Sep-2023 05:59:46.165 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-80"]
29-Sep-2023 05:59:46.200 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [504] milliseconds

Check that Tomcat is running and listening to port 80 or 443.
You can use netstat -nutap
To check if Tomcat process is running use ps -fauxgww, you can see here the full list of command arguments.
You can check the process tree using pstree.
You can check top Tomcat threads that are consuming CPU time using top -H (another guide describes why you may need this info, namely to find bad performing code or deadlocks).

# netstat -nutap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:27272           0.0.0.0:*               LISTEN      718/sshd: /usr/sbin
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      79617/java
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      423/systemd-resolve
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      79617/java

# ps fauxgww
root        1105 16.6  9.2 2794648 89080 pts/1   Sl   03:35   0:03 /system/jdk/bin/java -Djava.util.logging.config.file=/system/server/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /system/server/tomcat/bin/bootstrap.jar:/system/server/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/system/server/tomcat -Dcatalina.home=/system/server/tomcat -Djava.io.tmpdir=/system/server/tomcat/temp org.apache.catalina.startup.Bootstrap start

# pstree
systemd─┬─accounts-daemon───2*[{accounts-daemon}]
        ├─acpid
        ├─2*[agetty]
        ├─amazon-ssm-agen───8*[{amazon-ssm-agen}]
        ├─atd
        ├─cron
        ├─dbus-daemon
        ├─irqbalance───{irqbalance}
        ├─java───72*[{java}]

Configure for systemd

To make Tomcat start automatically during server startup/boot and to make it more resilient i.e. restarts automatically, we need to configure a Tomcat systemd service.
Create the folder /system/service/tomcat, this folder will just contain the file catalina.pid (containing the current process id).
Modify CATALINA_OPTS to suit your needs, especially, change the Xms and Xmx according to your server size.

mkdir -p /system/service/tomcat

Create a file called /etc/systemd/system/tomcat.service

[Unit]
Description=Apache Tomcat server daemon
After=syslog.target network.target

[Service]
Environment="JAVA_HOME=/system/jdk"
Environment="CATALINA_HOME=/system/server/tomcat"
Environment="CATALINA_OPTS=-Xms32m -Xmx512M -Djava.net.preferIPv4Stack=true --add-opens java.base/java.time=ALL-UNNAMED"
Environment="CATALINA_PID=/system/service/tomcat/catalina.pid"
WorkingDirectory=/system/service/tomcat
User=root
Group=root
UMask=0077
Type=forking
Restart=always
RestartSec=5
PIDFile=/system/service/tomcat/catalina.pid
ExecStart=/system/server/tomcat/bin/startup.sh
ExecStop=/system/server/tomcat/bin/shutdown.sh

[Install]
WantedBy=multi-user.target

Another alternative to Restart=always is Restart=on-failure. Read the systemd documentation on what they mean.

This tells systemd that this service depends on syslog and network (the “After” directive), it tells Tomcat via CATALINA_PID, where to put the pid file, which will be monitored by systemd (the “PIDFile” directive).
And it should run this service if Linux is started ini init level multi-user.

Make sure systemd notices this new file.

systemctl daemon-reload

Tell systemd to setup the necessary symlink, so the service starts up during booting. We want Tomcat to run if server starts up in multi-user mode/level.

systemctl enable tomcat

Try running and stopping it.

systemctl start tomcat
systemctl status tomcat
systemctl stop tomcat

Try rebooting the server and check if Tomcat runs during startup.

Log rotation and deleting old logs

Create these folders /system/job and /system/backup. Make these scripts run daily via cron.

mkdir -p /system/job /system/backup

Create file /system/job/logrotate. Run via cron with /usr/sbin/logrotate /system/job/logrotate-tomcat
Content of logrotate-tomcat, change the settings to suit your needs

/system/server/tomcat/logs/catalina.out {
  copytruncate
  daily
  rotate 7
  compress
  missingok
  size 5M
}

Create file /system/job/log-remove.sh. Run via cron with /system/job/log-remove.sh (make sure to make file executable first chmod u+x log-remove.sh). This script deletes access logs that are more than 1 year old (and empty files).

#!/bin/bash -x

id=$(date +"%F_%H%M")

log=/system/backup/log-remove-${id}.log

find /system/server/tomcat/logs -mtime +365 -type f -name '*.txt' -delete 2> ${log}
find /system/server/tomcat/logs -mtime +365 -type f -name '*.log' -delete 2> ${log}
find /system/server/tomcat/logs -empty -delete 2> ${log}