All About the Apps
cancel

Basic Custom Instrumentation for the HPE Diagnostics Java Agent

Basic Custom Instrumentation for the HPE Diagnostics Java Agent

APM_Piotr

One of the most essential techniques for acquiring performance data from a running application is through code instrumentation. During the process of instrumentation, the original application code is modified by inserting additional code which makes calls to the monitoring tool. These calls notify the tool that a specific activity, such as a method entry or a method exit, is being performed by the application. In turn, the tool is able to measure the time elapsed between the method start and the method end (latency). For the sake of this blog we will limit ourselves to only this basic code instrumentation. For more advanced instrumentation, such as capturing SQL statements, outbound calls or Web Services, refer to the Diagnostics Java Agent Guide.

Custom instrumentationCustom instrumentation

Load-time Instrumentation

When the application classes are loaded by the Java Virtual Machine (JVM), the probe is given a unique chance to look at the content of the class before it is used by the JVM and modify it. This process is called "load-time instrumentation". Almost all classes instrumented by the probe are instrumented in this way.

During load-time instrumentation, the probe will not instrument classes loaded by the boot classloader (the standard J2SE classes), for example, java.lang.String, even if they are loaded after the probe is initialized.

Instrumentation Points

To perform instrumentation automatically, Diagnostics Java probes receive precise instrumentation instructions organized as a number of individual instrumentation points. An instrumentation point is a small and independent piece of specification controlling the instrumentation process. The probes come with a large number of instrumentation points already installed. The instrumentation points are located in the auto_detect.points file in the etc directory. This is a plain text file, which can be edited by end users, providing the capability of defining custom instrumentation.

In the following sections we will describe how simple instrumentation points can be constructed

Each instrumentation point has a unique name. The point names are just identifying labels. Their presence makes troubleshooting and debugging of instrumentation easier. The point name is provided within [square brackets] as the first line of the point definition. The rest of the point definition takes a form of multiple specification lines. Lines that start with a semicolon (;) are ignored, and can be used for comments or for temporary disablement of certain elements of the point. Individual points need to be separated by one or more blank lines.

All instrumentation points need to provide a layer to which the latency of the method should be contributed. The notion of the layer assumes a hierarchical structure of the application, which can be composed from multiple and inter-dependent frameworks. Layers are specified as a path, for example, MainLayer/FirstSublayer/SecondSublayer. There are no constraints on the number of sublayers or layer names. The performance data is aggregated by layer(s).

During load-time instrumentation, while the probe intercepts the loaded class body, for each method of the class it finds the instrumentation points that match the method. The probe will then attempt to instrument the methods using the matching point(s). Thus, a single point can instrument several code locations. There is a number of different ways the matching can be defined. The following sections will describe them one by one.

Instrumenting by Class and Method Name

In the simplest case, the class name and the method name are specified directly. For example,

[Example point - Direct method specification]
class     = com.mycompany.application.Server
method    = serviceRequest
signature = (Lcom/mycompany/application/Request;)Lcom/mycompany/application/Response;
layer     = My Company Server/Service

This instrumentation point will match only one method, named serviceRequest, defined in the class com.mycompany.application.Server, taking one argument of type com.mycompany.application.Request and returning a value of type com.mycompany.application.Response. The method signature has to be provided in the JVM type signature format (used also by Java JNI). This format is summarized in the following table:

Java VM Type Signatures

Type Signature    Java Type
Z boolean
B byte
C char
S short
I int
J long
F float
D double
V void   (only used as method return type)
Lfully-qualified-class-name-separated-by-slashes; Class identified by the fully-qualified-class-name
[type type[]
(arg-signatures)return-type Method type

For example, the Java method

int[] foo(String[] s, Map<String,Object> m, double d, boolean b)

has the following type signature

([Ljava/lang/String;Ljava/util/Map;DZ)[I

Note that the generics are not reflected in the method signatures at all.

Instrumenting by Interfaces

Another way of matching points with methods is by specifying an interface method, which the method to instrument must implement. Interface methods cannot be instrumented directly, because they have no code. Here is an example:

[Example - Interface specification]
class     = com.mycompany.application.Callback
; Callback can be an interface, or an abstract or concrete class
method    = execute
signature = (Lcom/mycompany/application/Message;)V
deep_mode = soft
layer     = My Company Server/Callbacks

This example differs from the previous one by another specification line deep_mode = soft. This makes the probe look at the class hierarchy, and match all methods that implement (or overload) the method:

void execute(com.mycompany.application.Message);

declared in class or interface com.mycompany.application.Callback. There is no limit on the number of methods this instrumentation point will instrument - any and all methods implementing the specified interface method will get instrumented. However, sometimes such behavior is undesired. The following sections show how to limit the instrumentation.

Excluding Classes from Instrumentation

Use the ignore_cl or ignore_tree class within the specification lines to make an instrumentation point not match any methods from a specific class. Both can take a list of items, separated by commas. They are shown in the following example.

[Example - Interface specification with class exclusion]
class       = com.mycompany.application.Callback
method      = execute
signature   = (Lcom/mycompany/application/Message;)V
deep_mode   = soft
ignore_cl   = org.acme.SpecialClass1,org.acme.SpecialClass2
ignore_tree = org.acme.BackDoorServer
layer       = My Company Server/Callbacks

The new specification lines force the probe to ignore the current instrumentation point if the fully qualified class name is org.acme.SpecialClass1 or org.acme.SpecialClass2, or if the class is a subclass of org.acme.BackDoorServer.

Regular Expressions

Regular expressions can be used to streamline the creation of instrumentation points.The probes accept regular expressions for most of the specification lines that are used for matching the point. For example, all the specification lines described so far can also take regular expressions. If a specification item is a regular expression pattern, rather than a string to match verbatim, it must be preceded by exclamation mark (!). The syntax of the regular expressions accepted by the probe is the same as for the regular expressions handled by the standard J2SE package java.util.regex. Even though most instrumentation points use only a small subset of full capabilities of Java regular expressions, note that any metacharacter used by this API must be escaped by a preceding '\' (backslash), if it needs to match that specific character. The syntax of regular expressions used in instrumentation points is summarized in the following table.

Regular expression constructs used by instrumentation points

Construct

        

Matches

Non-special character x The character x
\( (
\) )
\[ [
\$ $
\. .
. Any single character
X? X, once or not at all
X* X, zero or more times
XY X followed by Y
X|Y Either X or Y
(X) X, as a group (often used in combination with ?, * or |)

The probe always looks at the whole string to match the specified regular expression. It is not sufficient for the string to just contain a matching substring. For example, regular expression A(BC)?(X|Y) matches strings ABCX, and AY, but does not match strings BCX, ABY, or AAY.

Excluding Methods from Instrumentation

In addition to the mechanisms for excluding classes, the probe also offers ways to exclude individual methods. This can be done by using the ignore_method specification line. It can take a list of items (plain strings or regular expressions) separated by commas. Each method is specified by the method name followed by a single space, and by the method signature. For example,

ignore_method = <clinit> V()

instructs the probe not to instrument the static class initializer (the code that has been provided by the static block, if present). It is also possible to exclude methods based on their access level modifiers. For example, the following specification line

method_access_filter = private,protected,package

makes the point match only the public methods.

Instrumenting by Annotations

Method matching for the purpose of instrumentation can be also defined by class and/or method annotations. At least one of the classAnnotation and methodAnnotation elements needs to be provided. If classAnnotation element is missing, all classes will match, and if methodAnnotation is missing, all non-static public methods will match.

Let's look at an example:

[Example - Annotations]
classAnnotation  = javax.jws.WebService
methodAnnotation = javax.jws.WebMethod
ignore_method    = !(get.*)|(set.*)
layer            = Web Services

This instrumentation point matches all methods with @javax.jws.WebMethod annotations, provided they are defined in a class with @javax.jws.WebService annotation. The class name does not matter, but methods with names starting with get or set will not get instrumented even if they happen to be annotated.

Caller-side Instrumentation

All points presented so far instrument the specified method body, inserting code at the beginning and end of its original code. This way, each invocation of the method will be captured by the probe, no matter who made the call. Sometimes it is useful to go with an alternate approach, by instrumenting the selected calls instead. This approach, called caller-side instrumentation, modifies the caller method body and inserts the new code just before the call is made and just after the called method returns. Both approaches capture the same data about the method invocation.

Caller-side instrumentation can be used to fine-tune the invocations you want to capture, or to capture latencies of standard J2SE methods (remember, the probe cannot instrument them directly). To use caller-side instrumentation, the instrumentation point needs to define the scope of instrumentation, which is the set of callers of the specified methods that should get instrumented. The scope is a comma separated list of elements. Each element is a fully qualified class name, followed by dot, method name, space character, and method signature. For example,

scope = com.acme.app.Backbone.service (Ljava/lang/String;)Z

In many cases, a regular expression will be used as the scope, to specify all methods belonging to a given class or package. Here is an example:

[Example - Caller side]
class     = java.util.regex.Matcher
method    = !(find)|(matches)|(lookingAt)
signature = !.*
scope     = !com\.mycompany\.publishing\..*
layer     = Publishing/Regex

This point instruments all calls to methods find, matches, and lookingAt from the JDK standard class Matcher, as long as they are made from any code belonging to any package with name starting with com.mycompany.publishing.

Annotation based instrumentation, method_access_filter or deep_mode cannot be used in caller-side instrumentation points, because no information on the target method (except its name and signature) is available to the probe while instrumenting its caller.

Customizing Performance Data Capture

By default, the probe will measure the latency of each instrumented call, and, as long as the latency is below the declared threshold (see minimum.method.latency in etc/capture.properties), it will present the call as a tree node in a Call Profile. Otherwise it will trim the invocation (not report it). It is possible to override this latency trimming mechanism, and request that all invocations are reported by specifying the following additional instrumentation point line:

detail = method-no-trim

It is also possible to specify the latency trimming directly at the instrumentation point, by specifying the latency trimming threshold in milliseconds. This threshold will be used for the instrumented location regardless of the actual value of minimum.method.latency. For example,

detail = method-trim:25

Sometimes it is good to know the CPU consumption of a specific method (including all methods invoked from it). Specifying

detail = method-cpu-time

tells the probe to measure not only the invocation latency, but the CPU consumption as well. Unless the method is latency trimmed, the CPU consumption for each invocation will be visible in Call Profile.

In some cases, it can be useful to capture the value of one of the method arguments to have a better insight into what the application was doing while it encountered a performance issue. This can be accomplished by specifying

detail = args:<n>

where <n> is the index of the argument (the argument numbering starts from 1). Be careful when capturing arguments: they have to be of a type that allows conversion to String. Also a great number of different argument values will put a lot of stress on the storage keeping the performance data. The captured arguments can be seen in Call Profiles.

Note

Only one detail specification line may appear in any instrumentation point definition. In case you'd like to specify multiple details, combine them into a comma separated list of items.

Application Code Built-in Instrumentation

Developers, who elect to test performance of their application with the probe as one of the steps in their application development lifecycle, can specify instrumentation directly within the application code, using the @InstrumentationPoint annotation. For example,

@InstrumentationPoint(layer = "Processing/Spreadsheet", detail = "method-cpu-time")
private void calculateSummary(DataSheet ds) {

When the application code is loading, the probe will find the @InstrumentationPoint annotation, and instrument the annotated method. There is no need to update auto_detect.points for that.

The mercury.diagnostics.common.api.InstrumentationPoint annotation is defined in the file annotation.jar in the Java Agent lib directory. This file can be copied by the developers to their build environment for convenience.

Instrumentation Point Conflicts

It is recommended that for each Java method used by the application there is at most one instrumentation point which matches. Otherwise, with the exception of a few special cases, the probe will chose just one of the multiple matching points, according to some built-in algorithms. This can be confusing, as the behavior is not transparent to the users, and can even be non-deterministic. There are two ways to avoid instrumentation point conflicts.

One way is to use regular expressions sparingly, and use ignore_cl and ignore_method specification to narrow down the set of matching methods to those which are really important. This approach has also the benefit of increasing the runtime probe overhead only by the minimum amount necessary to provide the required performance data.

Another way is to use point priorities. When more than one point matches a method, the probe will select the one with the highest priority. The point priority can be given by optional specification line as in the following example:

priority   = -3

The default point priority is zero. It is recommended that the instrumentation points added by the user have negative priority. If some of the standard instrumentation points enclosed with the Java Agent are made inactive by point conflicts, the consistency of performance data reported by the probe can get compromised.

Dynamic Instrumentation

Dynamic instrumentation, as opposed to load time instrumentation, takes place after the class of interest has been loaded. During dynamic instrumentation, the body of affected methods is replaced while the application keeps working. The old method body will be used for any method executions that were still in progress when the instrumentation was invoked.

The only way to specify the class and method for dynamic instrumentation is by naming them explicitly. Instrumentation by annotations or by interfaces is not available. The probe will attempt to instrument all matching methods from the already loaded classes with the given name ( there can be more than one), as long as method names and signatures match the point specification. Dynamic instrumentation takes place in-process only. It does not modify the agent configuration. To preserve the instrumentation points used in dynamic instrumentation, the user has to edit the auto_detect.points file.

Dynamic instrumentation can use substantial CPU and memory resources, and can cause JVM instability. It is not recommended for production environment. There's no way to undo dynamic instrumentation, other than restarting the application.

Instrumenting from Call Profile

When viewing a Call Profile in Profiler with some nodes created by stack trace sampling, it is possible to dynamically instrument selected captured method by right clicking on the method bar and selecting the item, as seen below.

Profiler-DynamicInstrumentation0.gif

This action opens a Dynamic Instrumentation dialog with the fields prefilled according to the selected method. The user can modify all of these fields before performing the instrumentation. Regular expressions can be used for method name and method signature.

Profiler-DynamicInstrumentation2.gif

After applying the instrumentation, the probe responds with the definition of the point it actually used, and provides the status of the operation.

cropped.gif

Using Class Map

To help navigate the code of the application, Diagnostics Java Agent offers the ClassMap feature. This is a simple browsing interface to the application static call graph, as shown in the screenshot below.

Diag_class_map3.gif

From the ClassMap, the user can dynamically instrument all methods for the currently listed class. To enable ClassMap, set use.class.map=true in probe.properties.

Dynamic Instrumentation Servlet

Sometimes it is inconvenient to use the ClassMap or the Profiler and look for the method to instrument in stack trace samples shown in Call Profiles. Knowing the exact class and method name makes it possible to dynamically instrument classes directly, using a probe servlet. For example,

http://<probe-host>:35000/inst/instrumentMethod?class_name=com.acme.DB&method_name=foo

instruments all methods named "foo" in the specified class. One can use a regular expression (preceded by the exclamation mark) for the method name, but not for the class name. There is no way to specify the method signature. It is also possible to instrument all methods of a given class by using

http://<probe-host>:35000/inst/instrumentClass?name=com.acme.Persistency

These servlets respond with the status of performed operation, indicating whether any matching classes or methods were found. As with any other probe servlets, the user will be asked for credentials before any action is performed.

Dynamic Capture Control

The probe keeps track of all instrumented code locations and can tailor data capture for these locations while the application runs. In particular, individual locations can be disabled, meaning no performance data will be collected from these locations, and no overhead will be incurred.

Monitoring Profiles

Monitoring profile is a predefined level or amount of data captured by the probe. Monitoring profiles are identified by a numerical value. The larger the value the higher the data volume (and overhead) for the probe. The monitoring profile for the probe can be adjusted dynamically. It is possible to declare a numerical threshold for the profile for which the locations instrumented by a given point remain enabled. For example,

detail = profile:140

indicates that the instrumented locations should be enabled only when the current monitoring profile is 140 or greater. When the current monitoring profile changes, the probe adjusts the enabled/disabled status of each instrumented location accordingly.

Manual Control

It is also possible to control the status of each instrumented location manually, regardless of the current monitoring profile.This applies to both load-time and dynamic instrumentation. The probe provides a web page at

http://<probe-host>:35000/inst/layer

which displays the current state of all instrumented code locations. The locations are organized by layers, as seen in the following screenshot.

InstrumentedLayers1a.gif

The user has an opportunity to disable or enable the whole layer, that is, all instrumented locations belonging to the given layer, by a single click to the Disable or Enable action. It is also possible to reset the manually changed locations back to the default, which means that the status will depend on the current monitoring profile.

The page also displays the total number of location executions (hits). This is very helpful in fine tuning the instrumentation to decrease the probe overhead.

Alternatively, the user can click on the layer name to get to the individual locations page.

InstrumentedLayers3.gif

The instrumented class and method name, as well as the instrumentation point name are provided here. In addition to enable or disable individual code location, the user also has a chance to dynamically set or clear certain detail elements, such as CPU time collection for the method.

If the instrumentation point contains specification line

detail = disabled

all instrumented locations resulting from applying this point will be initially disabled. One can later select and enable them one by one manually. This approach is useful, if the expected impact of having all locations enabled at the same time is too large.

Troubleshooting

Sometimes the newly added instrumentation points do not work as intended. When the expected performance data are missing, there could be a number of different reasons for the failure. In most cases, following the simple steps listed below helps find the cause:

  1. Remember that any changes in auto_detect.points require application restart to take effect.
  2. Check if the instrumentation point is given a unique name.
  3. Check in probe.log if the probe does not complain with warnings about the instrumentation point syntax.
  4. Check in probe.log if the probe does not complain about instrumentation point conflicts involving the new point. You can use conflict.resolution.logging=auto in inst.properties.
  5. Check in detailReport.txt that the new point has been applied to the classes and methods as expected.
  6. Check for typos in package, class, or method names and that all special characters have been properly escaped in regular expressions.
  7. Verify that the application actually uses the classes the point is trying to instrument. To see the names of all loaded classes use the -verbose option for the Java command line.
  8. Check at http://<probe-host>:35000/inst/layer that the location of interest is enabled and that the application actually executed the instrumented code (look at the number of hits).
  9. Check if the latency trimming threshold is sufficiently low to record the instrumented method(s) execution.
  10. Check in probe.log that the probe does not report that it ceased data capture because some of its internal limits are exceeded - increase the limits or cut down on instrumentation as necessary.

Final Remarks

Always be aware that adding instrumentation points increases the probe overhead. In most cases, especially in production environment, the requirements for probe overhead are very strict. Keep in mind that even if method invocations are latency trimmed, the cost of capturing the invocation and measuring the latency can be substantial. In extreme cases, over-instrumentation can cripple application performance, or even cause failures (for example, running out of heap space).

Code instrumentation performed by the probe is a broad topic, and this article intends to show only some aspects of it, without covering them in depth. It serves as a good starting point for those who plan to complement the probe's behavior by their own instrumentation points. For more information on instrumentation, consult comments in auto_detect.points and the Advanced Java Agent Configuration and Instrumentation section in the Diagnostics Java Agent Guide.

 

For more information on Diagnostics please follow this link.

  • Application Performance mgnt
About the Author

APM_Piotr

Technical Lead for Diagnostics Java Agent, Application Performance Management

//Add this to "OnDomLoad" event