Lumigo OpenTelemetry Distribution for Javascript

The Lumigo OpenTelemetry Distribution for JavaScript is a package that integrates multiple upstream OpenTelemetry components with additional automated quality assurance and custom optimizations. It is designed for seamless, no-code instrumentation, allowing you to enable OpenTelemetry tracing in your application without modifying a single line of code, instrumenting your application with distributed traces, and sending both traces and logs to Lumigo.

See the No-code activation section for auto-instrumentation instructions.

📘

Note

If you are looking for the Lumigo Node.js tracer for AWS Lambda functions, use the @lumigo/tracer package.

Setup

There are three steps to adding the Lumigo OpenTelemetry Distribution for Javascript to your application.

1. Add dependencies

To use the Lumigo OpenTelemetry Distribution for Node.js, you need to install the @lumigo/opentelemetry package.

Add @lumigo/opentelemetry as a dependency using your preferred package manager:

npm install @lumigo/opentelemetry

yarn add @lumigo/opentelemetry

2. Environment-based configuration

Configure the Lumigo OpenTelemetry Python distribution package using environment variables.

LUMIGO_TRACER_TOKEN=<token>
OTEL_SERVICE_NAME="your-service-name"
LUMIGO_ENABLE_LOGS='true'

3. Tracer activation

There are two approaches to activating the @lumigo/opentelemetry package. No-code activation imports the package via the environment. Manual activation imports the package in code. We recommend using no-code activation

export NODE_OPTIONS="${NODE_OPTIONS} -r '@lumigo/opentelemetry/sync'"

// javascript
const lumigo = require("@lumigo/opentelemetry");

// typescript
import * as lumigo from "@lumigo/opentelemetry/sync";

For details on initialization behavior, see Waiting for the initialization of the Lumigo OpenTelemetry Distro.

Setup for npm package.json start script

If your application starts via an npm script (as defined in package.json), you can configure no-code activation directly inside the start script. This ensures that OpenTelemetry tracing is automatically enabled when your application runs. Modify your package.json to include the following:

{
    "scripts": {
        "start": "LUMIGO_TRACER_TOKEN=<token> OTEL_SERVICE_NAME=<service name> node -r @lumigo/opentelemetry <main_file>.js"
    }
}

make sure to replace <token> with your Lumigo tracer token, <service name> with your desired service name, and <main_file>.js with your application’s entry file.

Advanced Configuration

The Lumigo OpenTelemetry Distribution for Javascript supports both standard OpenTelemetry settings and Lumigo-specific configurations. Use the following environment variables to configure your tracing setup.

OpenTelemetry configurations

The distribution integrates several upstream OpenTelemetry packages with additional logic. Due to this, any environment variables for vanilla, unmodified OpenTelemetry also apply here. Key configurations we support include:

• General configurations. Standard environment variables for configuring OpenTelemetry behavior.
• Batch span processor configurations: The Lumigo OpenTelemetry Distro for Node.js uses a batch processor to send data to Lumigo.

Lumigo-specific configurations

The @lumigo/opentelemetry package additionally supports the following configuration options as environment variables:

For more information, check out Filtering http endpoints.

Logging instrumentation

The Lumigo OpenTelemetry Distribution for Node.js supports automatic logging instrumentation, allowing log records to be captured and sent to Lumigo with trace context.

• LUMIGO_ENABLE_LOGS: If true, turns on the logging instrumentations to capture log records, and send them to Lumigo. Currently, this function is available for Winston, Pino and Bunyan loggers. By default, this is set to false. Emitted logs will also get injected with the active span context, for example:

  // ...
  "body": "Hello Winston!",
  "attributes": {
    "trace_id": "1fce43bfd3fdde3f1a9ea1adc78b521d",
    "span_id": "13c05292d3b5f5e8",
    "trace_flags": "01"
  }
  "severityText": "info",
  // ...

📘

Note

Logging support is applicable only when using versions of the logging libraries listed here.

• LUMIGO_DEBUG_LOGDUMP: This functions similarly to LUMIGO_DEBUG_SPANDUMP, but for logs instead of spans. This option is only effective when LUMIGO_ENABLE_LOGS is set to true.

Execution Tags

Execution Tags allow you to dynamically add dimensions to your invocations so that they can be identified, searched for, and filtered in Lumigo. For example, in multi-tenanted systems, execution tags are often used to mark the identifiers of the end-users that trigger them for analysis (Such as Explore view) and for alerting purposes.

By leveraging execution tags, you can gain deeper insights into your application's runtime behavior.

Adding Execution Tags

In the Lumigo OpenTelemetry Distro for JS, execution tags are represented as span attributes with the lumigo.execution_tags. prefix. You can dynamically add these tags to your spans to enrich them with additional metadata. For example, you could add an execution tag as follows:

// Typescript
import { trace } from '@opentelemetry/api';

/*
 * In Node.js 14+, the '?' coalescing operator ensures that your code is
 * safe even if the tracing is not active, and `trace.getActiveSpan()` returns
 * `undefined`.
 */
trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo','bar');

// Javascript
const { trace } = require('@opentelemetry/api');

/*
 * In Node.js 14+, the '?' coalescing operator ensures that your code is
 * safe even if the tracing is not active, and `trace.getActiveSpan()` returns
 * `undefined`.
 */
trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo','bar');

When using OpenTelemetry's trace.getActiveSpan() API, you can dynamically retrieve the current span at any point in your program execution without needing to explicitly track it.

In OpenTelemetry, span attributes can be strings, numbers (double precision floating point or signed 64 bit integer), booleans (Also known as "primitive types"), and arrays of one primitive type (Such as an array of string, and array of numbers or an array of booleans). In Lumigo, booleans and numbers are transformed to strings.

When using the Span.setAttribute API multiple times on the same span for the same key, new values may overwrite the previous values instead of adding to them:

// Typescript
import { trace } from '@opentelemetry/api';

trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo', 'bar');
trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo', 'baz');

// Javascript
const { trace } = require('@opentelemetry/api');

trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo', 'bar');
trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo', 'baz');

In the snippet above, the foo execution tag will only have the baz value in Lumigo. The bar value will have been overriden.

If you want to set multiple values for an execution tag:

// Typescript
import { trace } from '@opentelemetry/api';

trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo', ['bar', 'baz']);

// Javascript
const { trace } = require('@opentelemetry/api');

trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo', ['bar', 'baz']);

The snippets above will produce in Lumigo the foo tag having both bar and baz values. Another way to associate multiple values with an execution tag is by setting execution tags in different spans within the same invocation. See execution Tags in different spans of an invocation for more details.

Execution Tags in different spans of an invocation

In Lumigo, multiple spans can be grouped together into a single invocation, which represents the full context of your operation. This is the same entry you can see in the Explore view.

When spans are grouped into an invocation, all execution tags from each span are included and their values are merged:

// Typescript
const tracer = tracerProvider.getTracer(__filename)

trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo','bar');

const tracer = tracerProvider.getTracer(__filename)

const nestedSpan = tracer.startSpan('child_span');

// Do something interesting
nestedSpan.setAttribute('lumigo.execution_tags.foo','baz');

nestedSpan.end();

// Javascript
const { trace } = require('@opentelemetry/api');

trace.getActiveSpan()?.setAttribute('lumigo.execution_tags.foo','bar');

const tracer = tracerProvider.getTracer(__filename)

const nestedSpan = tracer.startSpan('child_span');

// Do something interesting
nestedSpan.setAttribute('lumigo.execution_tags.foo','baz');

nestedSpan.end();

In the examples above, the resulting invocation in Lumigo will contain both bar and baz values for the foo execution tag.

The spans that are merged into a single invocation depend on their parent-child relationships. This means that child spans, such as the nestedSpan in the example above, will be grouped under the same invocation as their parent span. For a more in-depth understanding of span relationships, refer to the Traces documentation in OpenTelemetry. In case your execution tags on different spans appear on different invocations than what you would expect, get in touch with Lumigo support.

Execution Tag Limitations

Execution tags in Lumigo have the following limitations:

• Up to a max of 50 execution tag keys per invocation in Lumigo, irrespective of how many spans are part of the invocation or how many values each execution tag has.
• The key of an execution tag cannot contain the . character. for example, lumigo.execution_tags.my.tag is not a valid tag. The OpenTelemetry Span.set_attribute() API will not fail or log warnings, but that will be displayed as my in Lumigo.
• Each execution tag key can be at most 50 characters long. The lumigo.execution_tags. prefix does not count against the 50 characters limit.
• Each execution tag value can be at most 70 characters long.

Programmatic Errors

Programmatic Errors allow you to customize errors, on top of monitoring and troubleshooting issues that should not necessarily interfere with the service. For example, an application tries to remove a user who does not exist. These custom errors can be captured by adding just a few lines of additional code to your application.

Programmatic errors indicate that a non-fatal error occurred, such as an application error. You can also log programmatic errors, track custom error issues, and trigger Alerts.

Creating a Programmatic Error

You can create Programmatic errors by adding span events with a custom attribute being set with the key name lumigo.type.

For example, you can add a programmatic error as follows:

// Typescript
import { trace } from '@opentelemetry/api';

trace.getActiveSpan()?.addEvent('<error-message>', {'lumigo.type': '<error-type>'});

// Javascript
const { trace } = require('@opentelemetry/api');

trace.getActiveSpan()?.addEvent('<error-message>', {'lumigo.type': '<error-type>'});

Supported runtimes

• Node.js: 14.x, 16.x, 18.x, 20.x

Supported packages

See the latest list of updated packages supported out of the box and regularly tested by Lumigo here: here

Activating your Prisma client instrumentation

If you are using Prisma and you want it instrumented, the only thing you will need to do is ensure that your schema file's generator client has the tracing preview feature enabled before generating the client itself.

generator client {
  provider = "prisma-client-js"
  previewFeatures = ["tracing"]
}

📘

Note

There have been reports of a possible bug that interferes with tracing when multiple Prisma clients have been instantiated. For details, see Prisma issue #20779.

Automated dependency reporting

To enhance support and inform data-driven decisions regarding which packages to support next, the Lumigo OpenTelemetry Distribution for JavaScript reports the packages and their versions used in your application to Lumigo SaaS at startup. This report also includes OpenTelemetry resource data, enabling analytics that reveal which platforms are utilizing which dependencies.

The uploaded data consists of a set of key-value pairs representing package names and their corresponding versions. This information complements the tracing data sent to Lumigo by covering dependencies that may not yet have dedicated instrumentation in the Lumigo OpenTelemetry Distribution for JavaScript. The sole purpose of these analytics is to ensure you receive the necessary instrumentations without having to explicitly request them.

Dependencies data is transmitted only when a LUMIGO_TRACER_TOKEN is present in the environment. You can opt out of this reporting by setting the environment variable LUMIGO_REPORT_DEPENDENCIES= to false.

Baseline setup

The Lumigo OpenTelemetry Distro will automatically create the following OpenTelemetry constructs provided to a NodeTraceProvider.

Resources

A Resource built from the default OpenTelemetry resource with the sdk... attributes, plus The lumigo.distro.version documenting the version of this package.

Additional resource attributes depending on the compute platform.

Amazon Elastic Container Service

• cloud.provider with the value aws.
• cloud.platform with the value aws_ecs.
• container.name with the container name as defined in the task definition as the value.
• container.id with the container id as defined by the underpinning Docker runtime as the value.

You can ensure the Task Metadata endpoint v4 is available, which is done through making sure the ECS_CONTAINER_METADATA_URI_V4 environment variable is set. In this case, the following resource attributes as specified in the AWS ECS Resource Semantic conventions are also set:

• aws.ecs.container.arn
• aws.ecs.cluster.arn
• aws.ecs.launchtype
• aws.ecs.task.arn
• aws.ecs.task.family
• aws.ecs.task.revision

Kubernetes resource attributes

• k8s.pod.uid with the Pod identifier is supported for both the v1 and v2 cgroup.

Exporters

• If the LUMIGO_TRACER_TOKEN environment variable is set: a BatchSpanProcessor, which uses an OTLPTraceExporter to push tracing data to Lumigo is used.
• If the LUMIGO_DEBUG_SPANDUMP environment variable is set: a SimpleSpanProcessor, which uses an FileSpanExporter to save to file the spans collected is used.

📘

Note

Do not use LUMIGO_DEBUG_SPANDUMP in production.

Process resource attributes

• As specified in the Process Semantic Conventions, the following process.runtime.* attributes are provided:

  • process.runtime.description
  • process.runtime.name
  • process.runtime.version

• process.environ: A non-standard resource attribute, which contains a stringified representation of the process environment, with environment variables scrubbed based on the LUMIGO_SECRET_MASKING_REGEX configuration.

SDK configuration

• The following SDK environment variables are supported:

  • OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT
  • OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT
  
  

  📘

  Note

  If the OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT environment variable is not set, the span attribute size limit will be taken from OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT environment variable. The default size limit when both are not set is 2048.

Advanced use cases

Waiting for the initialization of the Lumigo OpenTelemetry Distro

By default, the Lumigo OpenTelemetry Distro initializes asynchronously to avoid blocking your application’s startup process. However, this asynchronous behavior may cause issues in certain scenarios, such as CLI or batch-based applications that need to capture telemetry data immediately during startup. For further details, see the synchronous initialization section for an alternate method of initializing.

In such cases, the first span—which represents the application’s startup—may be missed, since the tracing logic hasn’t fully initialized by the time the application begins its execution.

To ensure that all spans are captured, including the initial startup span, you can wait for the initialization process to complete by using the init promise provided by the Lumigo Distro.

Asynchronous initialization of the Lumigo OpenTelemetry Distro

For use cases where the startup time of the application is a critical concern, you can use the asynchronous initialization method. This method will not block the main thread until the Lumigo OpenTelemetry Distro is fully initialized, which means that the collection of logs might not work properly.

To use this entry point, remove the /sync part from the module import:```

import { tracerProvider, loggerProvider } from '@lumigo/opentelemetry';

const { tracerProvider, loggerProvider } = require('@lumigo/opentelemetry/sync');

This will also possible with preloading, using the -r None option:

node -r '@lumigo/opentelemetry' your-app-main-file.js

Node.js

This differs between node.js before and after v18.

// Node.js prior to v18 (`async` as a top-level construct in your main file is not supported)
import * as lumigo from '@lumigo/opentelemetry';

// Some initialization code for your application.

lumigo.init
.then(()=>{
    // From this point on you are guaranteed that the SDK is initialized.
})
.catch(err => {
   // The sdk initialization failed :-(
   // Please let us know at support@lumigo.io!
});

// Node.js v18+ (`async` as a top-level construct in your main file is supported)
import * as lumigo from '@lumigo/opentelemetry';

// Some initialization code for your application.

try {
  await lumigo.init;
} catch (err) {
   // The sdk initialization failed :-(
   // Please let us know at support@lumigo.io!
}

// From this point on you are guaranteed that the SDK is initialized.

Access to the TracerProvider

The Lumigo OpenTelemetry Distro provides access to the TracerProvider it configures through the resolution of the init promise. See the Baseline setup section for more information.

import * as lumigo from '@lumigo/opentelemetry';
import { Resource } from '@opentelemetry/resources';
import { BasicTracerProvider } from '@opentelemetry/sdk-trace-base';

const tracerProvider: BasicTracerProvider = await lumigo.init.tracerProvider;
// Now you may want to add additional exporters using `tracerProvider.addSpanProcessor(spanProcessor: SpanProcessor)`

// The TracerProvider also provides access to the underpinning resource
const resource: Resource = tracerProvider.resource;

Ensure spans are flushed to Lumigo before shutdown

For short-running processes, the BatchProcessor configured by the Lumigo OpenTelemetry Distro may not ensure that tracing data are sent to Lumigo. See the Baseline setup section for more information.
Through access to the tracerProvider however, it is possible to ensure that all spans are flushed to Lumigo. To force a flush of all spans:

import * as lumigo from '@lumigo/opentelemetry';
import { Resource } from '@opentelemetry/resources';
import { BasicTracerProvider } from '@opentelemetry/sdk-trace-base';

const tracerProvider: BasicTracerProvider = (await lumigo.init).tracerProvider;

// Do some quick logic

try {
  await tracerProvider.forceFlush();
} catch (err) {
  console.error(err);
}

// Now the Node.js process can terminate, with all the spans closed so far sent to Lumigo

Filtering out empty SQS messages

SQS-based applications often continuously poll an SQS queue for messages then process them as they arrive. Empty responses can clutter your tracing data. By default, empty SQS polling messages are filtered out and not sent to Lumigo. To modify this behavior, set the boolean environment variable LUMIGO_AUTO_FILTER_EMPTY_SQS to false.

• LUMIGO_AUTO_FILTER_EMPTY_SQS=true If true, filters out empty SQS polling messages. By default, this is set to true.

Filtering http endpoints

You can selectively filter spans based on HTTP server/client endpoints for various components, not limited to web frameworks.

Global filtering

Set the LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX environment variable to a list of regex strings. Spans with matching server/client endpoints will not be traced.

Specific Filtering

For exclusive server (inbound) or client (outbound) span filtering, use the environment variables:

• LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_SERVER
• LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_CLIENT

The environment variable must be a valid JSON array of strings, so if you want to match endpoint with the hostname google.com the environment variable value should be ["google\\.com"].
If we are filtering out an HTTP call to an opentelemetry traced component, every subsequent invocation made by that component will also not be traced.

Examples:

• Filtering out every incoming HTTP request to the /login endpoint (will also match requests such as /login?user=foo, /login/bar))):
  • LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_SERVER=["\\/login"]
• Filtering out every outgoing HTTP request to the google.com domain (will also match requests such as google.com/foo, bar.google.com):
  • LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_CLIENT=["google\\.com"]'
• Filtering out every outgoing HTTP request to https://www.google.com (will also match requests such as https://www.google.com/, https://www.google.com/foo)
  • LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_CLIENT=["https:\\/\\/www\\.google\\.com"]
• Filtering out every HTTP request (incoming or outgoing) with the word login:
  • LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX=["login"]

Important notes

Using the distro with esbuild

Applications bundled with esbuild using @lumigo/opentelemetry will not be able to instrument any libraries for tracing or logging purposes unless the instrumented module is marked as external.

For instance, instrumenting Postgres calls via the pg library requires the following esbuild setup:

esbuild your-app-file-name.js --bundle --external:pg

or in the config file:

{
    // ...
    external: ["pg"]
}

Contributing

For guidelines on contributing, please see CONTRIBUTING.md.