Profiling the Lotus Notes Domino JVM
“When you’re debugging something and then you figure out that not only is the map wrong but the tools are broken—that’s a good time.” ― Peter Seibel, Coders at Work
- Monitoring Tools
- Profiling Tools
- Poor man’s Heap Analysis
This is a mini case study or a pseudo tutorial, if you will, that aims to summarize how to monitor and profile Java applications running on a remote IBM Lotus Notes Domino. Remote implies Domino is not running on your development machine, but on a separate box or in a VM.
Since version 8.5.2 Domino supports OSGi integration for the HTTP task. Developers may deploy OSGi plugins that utilize the HTTP Service to process HTTP requests. Together with the Java Notes API this has paved a way for developers to integrate and run non-Java EE web applications that have direct access to Domino databases.
side note rant, IBM have sadly been doing a poor job updating the JVM since then. To this day (Sep-2016) Domino still does not support Java 8, which starts to seriously affect the development of web applications for the platform.
An important aspect of Java development is for programmers to be able to monitor the performance and memory allocations of their applications. This allows for CPU and memory intensive operations to be pinpointed and optimized where necessary, thus increasing the software’s stability and responsiveness.
Monitoring generally relies on the JMX technology. To quote the docs:
JMX provides a standard way of managing resources such as applications, devices, and services. Because the JMX technology is dynamic, you can use it to monitor and manage resources as they are created, installed and implemented. You can also use the JMX technology to monitor and manage the Java Virtual Machine (Java VM).
The JConsole is probably the oldest Java monitoring tool in the box. It provides general information about the performance and resource consumption of the applications running on a Java platform.
There are many tutorials on how to utilize JMX in order to connect JConsle to an IBM JVM.
The fastest way to do that is to create a simple text file, e.g.,
jvmoptions.txt, and put the following inside:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.host=192.168.1.127 -Dcom.sun.management.jmxremote.port=5151
com.sun.management.jmxremote.port describe the IP and port of the Domino server.
In your Domino
notes.ini add the following line:
You need to then restart the Domino HTTP task to reload the configuration.
jconsole (it comes with a JDK installation) on your develpment machine and use the
Remote Process options to connect to the address and port you specified earlier.
Password should be left empty, because we did not specify any in the configuration above.
There’s a pop up messages about this being an insecure connection, but it may safely be ignored as long as you’re in a local network environment.
And there we have it. You are now monitoring your Domino JVM.
A glimpse of heap allocations and GC activity.
For many years now Sun Microsystems’ (now Oracle’s) VisualVM has been a rigid part of every Java developer’s toolset. Well, technically it’s there since Java 6 Update 7, but it is indeed a large and prettier step forward from JConsole.
Unfortunately, there currently isn’t a VisualVM Java agent that supports the IBM JVM, therefore an important VisualVM feature like
Heap Dump, that provides a way for developers to profile memory allocations, is not going to work! However, tools like the
CPU Sampler are available and they may help you optimize your application a lot.
Add a your remote Domino server via
Add Remote Host. Once the host is added, use
Add JMX Connection to connect to it.
Monitoring CPU usage, heap, threads, etc.
When activated, the
CPU Sampler analyzes what parts of the JVM runtime affect and/or are hogging the CPU. This provides you with information about what parts of your application code to analyze and possibly optimize.
Profilers can basically do everything a monitoring tool does. Using a profiler you can view heap dumps, track memory leaks, study memory consumption patterns, generate reports and more. Profilers usually require you to setup an agent which is a piece of native or Java code that hooks up to the JVM.
The tools I evaluate below are commercial, but you can download and use a trial version for each one. At this time I was unable to find a free and feature complete profiler tool that works with Domino.
YourKit is a commercial tool and comes with a 14 day trial period.
YourKit is much more advanced than VisualVM. It provides native Java agents for various platforms, so heap analysis for IBM JVM is absolutely possible. You can also compare memory snapshots, find deadlocks, do network I/O analysis and much more.
To enable YourKit, you first need to setup a Java agent. Native agent builds are found in the
bin folder within the installation. For example, for a 64-bit Domino server running on Linux you’ll need
jvmoptions.txt file and add the following:
port is one of the many agent startup options you may specify.
You need to then restart the Domino HTTP task to reload the configuration.
Open YourKit and create a new
remote application connection by specifying the server IP or hostname and the
6161 port number configured.
The amount of information YourKit provides is mind-boggling! What is really cool is that you can dynamically sample the JVM by selecting a time window from the graphs.
An example of CPU usage break up for a selected period.
An example of heap allocations break up for a selected period.
You can also use the
Object Allocation Recording option to analyze memory allocations in a specific time window.
JProfiler is another advanced commercial profiling tool that comes with a 10 days evaluation period.
One functionality that I think is very well presented is the
Live memory monitoring. You actually observe how objects are alloacted on the fly. Pretty cool!
Another powerful features is the
Locking Graph. A visual representation of threads and locks.
Just like with YourKit, you first need to setup a Java agent. Download a JProfiler package for the target platform on which Domino is running. For example, for a 64-bit Domino server running on Linux you’ll need
Add the following to your
jvmoptions.txt file and then restart the Domino HTTP task.
Open JProfiler and select the
Profile an application server, locally or remotely option. In the
Choose integration wizard select
Generic application server.
On the next screen, select a
remote computer profiled application and specify your platform. For a 64-bit Linux it would be
Choose a JVM vendor by specifying IBM Java 1.6.0.
Specify your Domino server’s remote IP address and port where the agent is located.
You need to also specify the path to the agent on the Domino server machine, e.g.,
Phew! That should be it. You can now open a remote profiling session.
When a profiling tool is not available, one can still analyze the heap by producing a dump manually. Here are a couple of hints on how to produce heap memory dumps and analyze them in Eclipse.
As written in the Domino documentation you need to issue the following command in the Domino console:
tell http dump java heap
This will produce a message that points to the directory where the dump file is located.
HTTP JVM: JVMDUMP034I User requested Heap dump using ‘/local/notesdata/heapdump.20160930.165511.32223.0001.phd’ through com.ibm.jvm.Dump.HeapDump HTTP JVM: JVMDUMP010I Heap dump written to /local/notesdata/heapdump.20160930.165511.32223.0001.phd
Now, this is all good, but how does one read a
You’ll need the following 2 plugins for Eclipse:
The Memory Analzer can be found in the Eclipse Marketplace.
The IBM Diagnostic Tool must be installed from an update site. Go to Eclipse
Install New Software and add the update site repository.
After restarting Eclipse open the
Memory Analysis perspective and then
Open Heap Dump. Eclipse parses the
.phd file and produces a nice allocated objects diagram.
Notable tools to use are the
Dominator Tree that will show you a break up of the heap consumption and
Leak Suspects, which produces a report about possible memory leaks.
This method is, of course, not as nice as using a profiler, but still a great option for Domino Java programmers to remedy coding issues.
- A a great presentation on Domino & OSGi from Lotusphere 2012 - slideshare.net/fiorep/domino-osgi-development
- JConsole setup for the IBM JVM - www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.lnx.80.doc/diag/tools/JConsole.html
- JConsole vs. VisualVM - www.javaworld.com/article/2072322/from-jconsole-to-visualvm.html
- Domino JVM options - www-10.lotus.com/ldd/dominowiki.nsf/dx/HTTPJVM_Out_of_memory