Lumigo OpenTelemetry Distribution for Java

Overview

The Lumigo OpenTelemetry Distribution for Java is a package that integrates multiple upstream OpenTelemetry components with additional automated quality assurance and custom optimizations. It is designed for seamless, no-code instrumentation, allowing you to enable OpenTelemetry tracing, and the sending of logs to Lumigo, in your application without modifying a single line of code.

Setup

The setup of Lumigo OpenTelemetry Distribution for Java is made of the three following steps:

1. Download the Lumigo OpenTelemetry Distribution for Java.
2. Configure the Lumigo tracer token.
3. Activate the Lumigo OpenTelemetry Distribution for Java.

1. Download

Download the latest version from the Releases page.

To load the Lumigo OpenTelemetry Distribution for Java on startup, configure your Java Virtual Machine (JVM) using one of the available options.

2. Environment-based configuration

To enable OpenTelemetry tracing with Lumigo, whether by using manual or no-code instrumentation, you must set the required environment variables.

• LUMIGO_TRACER_TOKEN: The authentication token provided by Lumigo. Found under Settings → Tracing → Manual Tracing in the Lumigo platform. see the Lumigo Tokens documentation for more details on how to retrieve your Lumigo token.

LUMIGO_TRACER_TOKEN=<token>

Replace <token> in the command above with the token generated for you by the Lumigo platform.

• OTEL_SERVICE_NAME: The name you assign to your service for OpenTelemetry monitoring. We recommend defining the OTEL_SERVICE_NAME environment variable to specify the service name for your application:

OTEL_SERVICE_NAME=<service name>

Replace <service name> with the desired name of the service.

Note: Note: If you are setting environment variables for configuration, consider also defining the necessary variable for no-code tracer activation.

3. Activation

The Lumigo OpenTelemetry Distribution for Java must be loaded by the Java Virtual Machine before your application is loaded.The two supported ways to achieve this are:

• Recommended: Setting the JAVA_TOOL_OPTIONS environment variable
• Setting the -javaagent command-line parameter

We recommend the JAVA_TOOL_OPTIONS method because setting environment variables is almost always easier than modifying the entrypoint of a container image, which is usually where you would set the -javaagent property.

JAVA_TOOL_OPTIONS

Set the JAVA_TOOL_OPTIONS environment variable on your Java Virtual Machine as follows:

export JAVA_TOOL_OPTIONS="-javaagent:<path-to-lumigo-otel-javaagent>"

This is the preferred method for containerized applications or when running your Java application as a SystemD unit. We recommend this method as in such environments, it is easier to set environment variables than to change the startup arguments of the Java Virtual Machine.

Command-line parameters

Pass the -javaagent property to the startup command as follows:

java -javaagent:<path-to-lumigo-otel-javaagent> -jar app.jar

Advanced Configuration

OpenTelemetry configurations

The Lumigo OpenTelemetry Distribution for Java builds upon multiple upstream OpenTelemetry packages while adding custom enhancements with additional logic. Since it extends standard OpenTelemetry functionality, it fully supports the environment variables used in vanilla OpenTelemetry configurations.

The following specific configurations are supported:

• General configurations

The system properties are lower case versions of the environment variables, and with dots replaced by underscores.For example, to set the environment variable LUMIGO_TRACER_TOKEN use the system property lumigo.tracer.token.

Lumigo-specific configurations

The Lumigo OpenTelemetry Distribution for Java additionally supports the following configuration options as environment variables:

The @lumigo/opentelemetry package additionally supports the following configuration options as environment variables:

For more configuration options, see the Upstream Agent Configuration.

Execution Tags in different spans of an invocation

Execution Tags allow you to dynamically add dimensions to your invocations so that they can be identified, searched for, and filtered in Lumigo. For example, in multi-tenanted systems, execution tags are often used to mark the identifiers of the end-users that trigger them for analysis (Such as Explore view) and for alerting purposes.

By leveraging execution tags, you can gain deeper insights into your application's runtime behavior.

Adding Execution Tags

In the Lumigo OpenTelemetry Distribution for Java, execution tags are represented as span attributes with the lumigo.execution_tags. prefix. You can dynamically add these tags to your spans to enrich them with additional metadata. For example, you could add an execution tag as follows:

import io.opentelemetry.api.trace.Span;

Span.current().setAttribute("lumigo.execution_tags.foo","bar");

When using OpenTelemetry's trace.getActiveSpan() API, you can dynamically retrieve the current span at any point in your program execution without needing to explicitly track it.

In OpenTelemetry, span attributes can be strings, numbers (double precision floating point or signed 64 bit integer), booleans (Also known as "primitive types"), and arrays of one primitive type (Such as an array of string, and array of numbers or an array of booleans). In Lumigo, booleans and numbers are transformed to strings.

When using the Span.setAttribute API multiple times on the same span for the same key, new values may overwrite the previous values instead of adding to them:

import io.opentelemetry.api.trace.Span;

Span.current().setAttribute("lumigo.execution_tags.foo","bar");
Span.current().setAttribute("lumigo.execution_tags.foo","baz");

In the snippet above, the foo execution tag will only have the baz value in Lumigo. The bar value will have been overriden.

If you want to set multiple values for an execution tag:

import io.opentelemetry.api.trace.Span;

Span.current().setAttribute(AttributeKey.stringArrayKey("lumigo.execution_tags.foo"), Arrays.asList("bar", "baz"));

In the examples above, the resulting invocation in Lumigo will contain both bar and baz values for the foo execution tag.

Another way to associate multiple values with an execution tag is by setting execution tags in different spans within the same invocation. See execution Tags in different spans of an invocation for more details.

Execution Tags in different spans of an invocation

In Lumigo, multiple spans can be grouped together into a single invocation, which represents the full context of your operation. This is the same entry you can see in the Explore view.

When spans are grouped into an invocation, all execution tags from each span are included and their values are merged:

import io.opentelemetry.api.trace.Span;

Span.current().setAttribute("lumigo.execution_tags.foo", "bar");
Tracer tracer = GlobalOpenTelemetry.getTracer("my-app");
Span nestedSpan = tracer.spanBuilder("child_span").startSpan();

try (Scope scope = nestedSpan.makeCurrent()) {
    // Do something interesting
    nestedSpan.setAttribute("lumigo.execution_tags.foo", "baz");
} finally {
    nestedSpan.end();
}

In the examples above, the resulting invocation in Lumigo will contain both bar and baz values for the foo execution tag.

The spans that are merged into a single invocation depend on their parent-child relationships. This means that child spans, such as the nestedSpan in the example above, will be grouped under the same invocation as their parent span. For a more in-depth understanding of span relationships, refer to the Traces documentation in OpenTelemetry. In case your execution tags on different spans appear on different invocations than what you would expect, get in touch with Lumigo support.

Execution Tag Limitations

• Up to a max of 50 execution tag keys per invocation in Lumigo, irrespective of how many spans are part of the invocation or how many values each execution tag has.
• The key of an execution tag cannot contain the . character. for example, lumigo.execution_tags.my.tag is not a valid tag. The OpenTelemetry Span.set_attribute() API will not fail or log warnings, but that will be displayed as my in Lumigo.
• Each execution tag key can be at most 50 characters long. The lumigo.execution_tags. prefix does not count against the 50 characters limit.
• Each execution tag value can be at most 70 characters long.

Programmatic Errors

Programmatic Errors allow you to customize errors, on top of monitoring and troubleshooting issues that should not necessarily interfere with the service. For example, an application tries to remove a user who does not exist. These custom errors can be captured by adding just a few lines of additional code to your application.

Programmatic errors indicate that a non-fatal error occurred, such as an application error. You can also log programmatic errors, track custom error issues, and trigger Alerts.

Creating a Programmatic Error

You can create Programmatic errors by adding span events with a custom attribute being set with the key name lumigo.type.

For example, you can add a programmatic error as follows:

Attributes eventAttributes = Attributes.of(
    AttributeKey.stringKey("lumigo.type"), "<error-type>"
);

Span.current().addEvent("<error-message>", eventAttributes);

Supported runtimes

The following runtimes are supported:

• JDK: 11.x, 17.x, 21.x

Supported packages

See the latest list of updated packages supported out of the box and regularly tested by Lumigo here: here

Baseline setup

The Lumigo OpenTelemetry Distribution will automatically create the following OpenTelemetry constructs provided to a NodeTraceProvider.

Resource attributes

SDK resource attributes

• The attributes from the default resource:

  • telemetry.sdk.language: java
  • telemetry.sdk.name: opentelemetry
  • telemetry.sdk.version: depends on the version of the io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom included in the dependencies

• The lumigo.distro.version containing the version of the Lumigo OpenTelemetry Distribution for Java

Process resource attributes

• The following process.runtime.* attributes as specified in the Process Semantic Conventions:

  • process.runtime.description
  • process.runtime.name
  • process.runtime.version

Amazon ECS resource attributes

If the instrumented application is running on the Amazon Elastic Container Service (ECS):

• cloud.provider attribute with value aws
• cloud.platform with value aws_ecs
• container.name with the hostname of the ECS Task container
• container.id with the ID of the Docker container (based on the cgroup id)

If the ECS task uses the ECS agent v1.4.0, and therefore has access to Task metadata endpoint version 4, the following experimental attributes are added, as specified in the AWS ECS Resource Attributes specification:

• aws.ecs.container.arn
• aws.ecs.cluster.arn
• aws.ecs.launchtype
• aws.ecs.task.arn
• aws.ecs.task.family
• aws.ecs.task.revision

Kubernetes resource attributes

• k8s.pod.uid with the Pod identifier is supported for both cgroups v1 and v2.
• k8s.container.name with the name of the container.

Span exporters

• If the LUMIGO_TRACER_TOKEN environment variable is set: an OTLP Exporter is configured to push data to Lumigo.
• If the LUMIGO_DEBUG_SPANDUMP environment variable is set to a path-like value, we define the FileLoggingSpanExporter, to save to file the spans collected.

NOTE: Do not use this flag in production unless you need to troubleshoot trace data!

SDK configuration

The default maximum span attribute length is 1024. This can be overwritten with the OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT and OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT environment variables, but when sending data to Lumigo, the span attributes will be truncated to a maximum length of 2048.

The Lumigo OpenTelemetry Java Distribution automatically configures a BatchSpanProcessor to export tracing data to Lumigo, with the following settings:

• otel.bsp.schedule.delay: 10ms. This is how the minimum delay before the BatchSpanProcessor will flush data to Lumigo.
• otel.bsp.max.export.batch.size: 100. This is the maximum amount of spans queued before flushing. When the limit is passed, a flush will occur.
• otel.bsp.export.timeout: 1s. This is the timeout for flushing data to Lumigo.

The metrics and logs exporters are disabled (otel.logs.exporter and otel.metrics.exporter are set to none) as Lumigo OpenTelemetry endpoint currently does not provide support for the /v1/metrics and /v1/logs endpoints.

Contributing

For guidelines on contributing, please see CONTRIBUTING.md.