Pipeline Rules

Cardinal Collector Pipelines support a variety of rule types that can be applied to telemetry data.

What Are Rules?

Rules are flexible, user-defined transformations and filters applied to your telemetry data as it passes through the pipeline. By shaping and refining data before it lands in your storage or monitoring tools, rules help you:

• Control costs by removing unnecessary data.
• Enhance data quality and context by adding tags or extracting metrics.
• Improve privacy by masking or dropping sensitive information.
• Streamline observability efforts, making it easier to investigate and understand system behavior.

Pipeline Processors

Cardinal pipelines give you full flexibility in where you can place rules. Each receiver and exporter you add to your pipeline can individually process your telemetry data. We also provide you a shared node to enable common rules.

In order to configure each processor, click on the node in the graph and then choose a telemetry type to edit rules. Rules are evaluated in a priority order top to bottom.

Below is a brief overview of the key types of rules you can configure.

PII Removal/Masking Rules

Remove or obscure sensitive Personally Identifiable Information (PII) from your telemetry. Use Case: Comply with privacy regulations and prevent sensitive data from reaching logs, metrics, or external services.

Extract Metrics from Logs

Convert log fields into metrics on the fly. Use Case: Turn unstructured logs into actionable metrics for easier aggregation, alerting, and performance analysis.

Custom OTTL (OpenTelemetry Transformation Language) Rules

Write custom conditions and statements using OTTL to programmatically transform data. Use Case: Apply highly flexible, code-like logic to tailor telemetry to your specific business needs.

Filter Rules

Completely remove certain telemetry items based on defined conditions. Use Case: Reduce noise, control costs, and simplify data by dropping irrelevant or unwanted telemetry before it’s stored.

Drop Tags/Dimensions

Remove high-cardinality tags or dimensions that inflate cost and complexity. Use Case: Prevent expensive, noisy dimensions from reaching your monitoring platform, focusing on the most valuable metrics.

Decoration (Add Tags) Rules

Enrich telemetry by adding tags (key-value pairs) based on conditions. Use Case: Provide more context to your data—for example, labeling telemetry by team name, environment, or tier—making it easier to query and correlate.

Sampling Rules

Reduce telemetry volume by sampling logs—either randomly or proportionally—to maintain representative but smaller datasets. Use Case: Control ingestion costs and noise levels while preserving overall insights.

Extract Tags from Log Data

Parse unstructured log messages to extract variable parts (e.g., user IDs) and turn them into structured tags. Use Case: Gain richer analytics and filtering capabilities by converting raw strings into searchable attributes.

With these rules at your disposal, you can intelligently sculpt your telemetry pipeline to be more cost-effective, insightful, and compliant. Each rule can be combined and fine-tuned to meet your unique observability and data management requirements.

Refer to the documentation for each Rule type for more information on how to configure it.