Secret Masking / Data Scrubbing

Overview

Lumigo's Secret Masking and Data Scrubbing features are designed to automatically detect and conceal confidential data, such as API keys, passwords, authentication tokens, and environment variables, before they are processed or displayed. This approach ensures that sensitive information remains protected throughout your observability workflows, aiding in compliance with data privacy practices.

By leveraging pattern-based matching using regular regex expressions expressions, Lumigo identifies and masks sensitive data across various components, including HTTP headers, request and response bodies, query parameters, and environment variables. This ensures that your logs and traces are free from inadvertent exposure of confidential information.

Secret masking allows you to keep sensitive data private. This sensitive data can include information like API keys, authentication tokens, passwords, or other types of critical information. Secret masking obscures the values stored for some data fields, preventing sensitive data from being processed and displayed by Lumigo.

This article is divided into two sections. The first is concerned with these features for tracing, the latter for logging.

Default Behavior in Tracing

By default, Lumigo will apply secret masking on fields where the key matches common regexes like _".pass.", "._key.*", and so on. The full list of regexes matched by default is below:

[".*pass.*",".*key.*",".*secret.*",".*credential.*",".*passphrase.*","SessionToken","x-amz-security-token","Signature","Credential","Authorization"]

When a data field's key matches any of these patterns, Lumigo obscures its value, preventing it from appearing in logs, traces, or dashboards. This default behavior provides an immediate layer of security without requiring additional configuration.

For example, if your application logs include a field like "db_password": "SuperSecret123", Lumigo will detect the "password" key and mask its value, ensuring that the sensitive information is not exposed in any observability data.

Customization

While Lumigo's default masking covers common scenarios, you can define custom masking rules to tailor the data scrubbing process to your application's specific needs. This is achieved by setting environment variables that specify regex patterns for different data components.

To override the default regular expressions, add the LUMIGO_SECRET_MASKING_REGEX environment variable to your Lambda function’s definition:

We support more granular masking using the following parameters. If not given, the above configuration is the fallback

By configuring these environment variables with appropriate regex patterns, you can ensure that any sensitive data unique to your application is effectively masked. For instance, if your application uses a custom header like "X-API-Token", you can add a regex pattern to LUMIGO_SECRET_MASKING_REGEX_HTTP_REQUEST_HEADERS to mask its value.

We support the masking of exact paths, specifically in order to support masking of specific fields in nested JSON objects:

Whitelisting

To unhide KeyConditionExpression and ExclusiveStartKey, add the LUMIGO_WHITELIST_KEYS_REGEXES environment variable to your Lambda function’s definition.

For example: LUMIGO_WHITELIST_KEYS_REGEXES = '[".TableName.", "KeyConditionExpression"]'

🚧

Notes

• All the regexes are case-insensitive.
• To completely disable secret masking default behavior, use an empty list value: LUMIGO_SECRET_MASKING_REGEX = []
• Lumigo will apply secret masking on http requests, triggered by Lambda, only in case the headers contain 'content-type': 'application/json'
• Please ensure that the values of environment variables are strings enclosed with ".
  Additionally, the list of values inside the [] should be enclose with '. Correct example: '[".*TableName*.", "KeyConditionExpression"]' Incorrect example: "['.*TableName*.', 'KeyConditionExpression']"

Secret masking in Logs

Lumigo's secret masking in Logs feature provides an additional layer of security by allowing developers to define rules that hide sensitive data within logs before they are processed and stored. We provide options for client-side log masking, and lg masking via Lumigo's platform.

For further information on this, consult our Secret masking in Logs documentation.