paper-thin-bits

Tech articles, mini tech blog

Profiling the Lotus Notes Domino JVM

“When you’re debugging something and then you figure out that not only is the map wrong but the tools are broken—that’s a good time.” ― Peter Seibel, Coders at Work

Contents

Preface

This is a mini case study or a pseudo tutorial, if you will, that aims to summarize how to monitor and profile Java applications running on a remote IBM Lotus Notes Domino. Remote implies Domino is not running on your development machine, but on a separate box or in a VM.

Since version 8.5.2 Domino supports OSGi integration for the HTTP task. Developers may deploy OSGi plugins that utilize the HTTP Service to process HTTP requests. Together with the Java Notes API this has paved a way for developers to integrate and run non-Java EE web applications that have direct access to Domino databases.

As a side note rant, IBM have sadly been doing a poor job updating the JVM since then. To this day (Sep-2016) Domino still does not support Java 8, which starts to seriously affect the development of web applications for the platform.

Monitoring Tools

An important aspect of Java development is for programmers to be able to monitor the performance and memory allocations of their applications. This allows for CPU and memory intensive operations to be pinpointed and optimized where necessary, thus increasing the software’s stability and responsiveness.

Monitoring generally relies on the JMX technology. To quote the docs:

JMX provides a standard way of managing resources such as applications, devices, and services. Because the JMX technology is dynamic, you can use it to monitor and manage resources as they are created, installed and implemented. You can also use the JMX technology to monitor and manage the Java Virtual Machine (Java VM).

JConsole

The JConsole is probably the oldest Java monitoring tool in the box. It provides general information about the performance and resource consumption of the applications running on a Java platform.

Configuration

There are many tutorials on how to utilize JMX in order to connect JConsle to an IBM JVM.

The fastest way to do that is to create a simple text file, e.g., jvmoptions.txt, and put the following inside:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.host=192.168.1.127
-Dcom.sun.management.jmxremote.port=5151

com.sun.management.jmxremote.host and com.sun.management.jmxremote.port describe the IP and port of the Domino server.

In your Domino notes.ini add the following line:

JavaOptionsFile=<path/jvmoptions.txt

You need to then restart the Domino HTTP task to reload the configuration.

Next, open jconsole (it comes with a JDK installation) on your develpment machine and use the Remote Process options to connect to the address and port you specified earlier. Username and Password should be left empty, because we did not specify any in the configuration above.

JConsole Connection

There’s a pop up messages about this being an insecure connection, but it may safely be ignored as long as you’re in a local network environment.

JConsole SSL Warning

And there we have it. You are now monitoring your Domino JVM.

JConsole Monitoring

A glimpse of heap allocations and GC activity.

JConsole Heap & GC

VisualVM

For many years now Sun Microsystems’ (now Oracle’s) VisualVM has been a rigid part of every Java developer’s toolset. Well, technically it’s there since Java 6 Update 7, but it is indeed a large and prettier step forward from JConsole.

Unfortunately, there currently isn’t a VisualVM Java agent that supports the IBM JVM, therefore an important VisualVM feature like Heap Dump, that provides a way for developers to profile memory allocations, is not going to work! However, tools like the CPU Sampler are available and they may help you optimize your application a lot.

Configuration

Add a your remote Domino server via File -> Add Remote Host. Once the host is added, use Add JMX Connection to connect to it.

Visual VM JMX Connection

Monitoring CPU usage, heap, threads, etc.

Visual VM Monitoring

When activated, the CPU Sampler analyzes what parts of the JVM runtime affect and/or are hogging the CPU. This provides you with information about what parts of your application code to analyze and possibly optimize.

Visual VM CPU Sampler

Profiling Tools

Profilers can basically do everything a monitoring tool does. Using a profiler you can view heap dumps, track memory leaks, study memory consumption patterns, generate reports and more. Profilers usually require you to setup an agent which is a piece of native or Java code that hooks up to the JVM.

The tools I evaluate below are commercial, but you can download and use a trial version for each one. At this time I was unable to find a free and feature complete profiler tool that works with Domino.

YourKit

YourKit is a commercial tool and comes with a 14 day trial period.

YourKit is much more advanced than VisualVM. It provides native Java agents for various platforms, so heap analysis for IBM JVM is absolutely possible. You can also compare memory snapshots, find deadlocks, do network I/O analysis and much more.

Configuration

To enable YourKit, you first need to setup a Java agent. Native agent builds are found in the bin folder within the installation. For example, for a 64-bit Domino server running on Linux you’ll need linux-x86-64/libyjpagent.so.

YourKit Agents

Open your jvmoptions.txt file and add the following:

-agentpath:<server-path>/libyjpagent.so=port=6161

port is one of the many agent startup options you may specify.

You need to then restart the Domino HTTP task to reload the configuration.

Open YourKit and create a new remote application connection by specifying the server IP or hostname and the 6161 port number configured.

YourKit Connection

The amount of information YourKit provides is mind-boggling! What is really cool is that you can dynamically sample the JVM by selecting a time window from the graphs.

An example of CPU usage break up for a selected period.

YourKit CPU Usage Sample

An example of heap allocations break up for a selected period.

YourKit Memory Analysis

You can also use the Object Allocation Recording option to analyze memory allocations in a specific time window.

YK Allcation Recroding

JProfiler

JProfiler is another advanced commercial profiling tool that comes with a 10 days evaluation period.

One functionality that I think is very well presented is the Live memory monitoring. You actually observe how objects are alloacted on the fly. Pretty cool!

JProfiler Live Memory

Another powerful features is the Locking Graph. A visual representation of threads and locks.

JProfiler Locking Graph

Configuration

Just like with YourKit, you first need to setup a Java agent. Download a JProfiler package for the target platform on which Domino is running. For example, for a 64-bit Domino server running on Linux you’ll need linux-64/libjprofilerti.so.

JProfiler Setup

Add the following to your jvmoptions.txt file and then restart the Domino HTTP task.

-agentpath:/opt/jrebel/agents/libjprofilerti.so=port=6161

Open JProfiler and select the Profile an application server, locally or remotely option. In the Choose integration wizard select Generic application server.

On the next screen, select a remote computer profiled application and specify your platform. For a 64-bit Linux it would be Linux x86/AMD64

Choose a JVM vendor by specifying IBM Java 1.6.0.

JProfiler JVM

Specify your Domino server’s remote IP address and port where the agent is located.

JProfiler IP

JProfiler Port

You need to also specify the path to the agent on the Domino server machine, e.g.,

JProfiler Agent Path

Phew! That should be it. You can now open a remote profiling session.

Poor man’s Heap Analysis

When a profiling tool is not available, one can still analyze the heap by producing a dump manually. Here are a couple of hints on how to produce heap memory dumps and analyze them in Eclipse.

As written in the Domino documentation you need to issue the following command in the Domino console:

tell http dump java heap

This will produce a message that points to the directory where the dump file is located.

HTTP JVM: JVMDUMP034I User requested Heap dump using ‘/local/notesdata/heapdump.20160930.165511.32223.0001.phd’ through com.ibm.jvm.Dump.HeapDump HTTP JVM: JVMDUMP010I Heap dump written to /local/notesdata/heapdump.20160930.165511.32223.0001.phd

Now, this is all good, but how does one read a .phd file?

You’ll need the following 2 plugins for Eclipse:

The Memory Analzer can be found in the Eclipse Marketplace.

Eclipse Memory Analyzer

The IBM Diagnostic Tool must be installed from an update site. Go to Eclipse Help - Install New Software and add the update site repository.

Eclipse IBM DTFG

After restarting Eclipse open the Memory Analysis perspective and then File - Open Heap Dump. Eclipse parses the .phd file and produces a nice allocated objects diagram.

Eclipse IBM DTFG #2

Notable tools to use are the Dominator Tree that will show you a break up of the heap consumption and Leak Suspects, which produces a report about possible memory leaks.

This method is, of course, not as nice as using a profiler, but still a great option for Domino Java programmers to remedy coding issues.

References