Go App Instrumentation

Follow the steps below to instrument your Go service. When completed, your service will appear in the Cardinal Service Catalog, and Chip will begin monitoring it.

Local Testing

Get a Cardinal API key:

Export the following environment variables:

export OTEL_SERVICE_NAME="your-service-name"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=local" # local/dev/staging/prod
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otelhttp.intake.us-east-2.aws.cardinalhq.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-cardinalhq-api-key=<your-api-key>" # Set your API key

Follow the steps in the Instrumenting Your Application section below to instrument your application, depending on the libraries you are using in your application.
Run your application:

go run <your-app.go>

Validate that Cardinal is receiving data:

Exercise the service by calling some API endpoints, or causing some logs or metrics to be emitted. Wait for a few minutes, then visit the Service Catalog in the Cardinal UI to check that your service appears in the list.

Docker

Set the environment variables outlined above in your Docker container runtime configuration.

Update OTEL_RESOURCE_ATTRIBUTES to set the deployment.environment.name value to your environment.
If you run an OpenTelemetry Collector, set the OTEL_EXPORTER_OTLP_ENDPOINT to your Collector's OTLP Receiver endpoint, and follow the steps in the OTLP Export to Cardinal section to forward data to Cardinal.

Kubernetes

Set the environment variables outlined above in your Deployment or StatefulSet manifest.

Update OTEL_RESOURCE_ATTRIBUTES to set the deployment.environment.name value to your environment.
If you run an OpenTelemetry Collector, set the OTEL_EXPORTER_OTLP_ENDPOINT to your Collector's OTLP Receiver endpoint, and follow the steps in the OTLP Export to Cardinal section to forward data to Cardinal.

Instrumenting Your Application

To instrument your Go application, you will need to set up the OpenTelemetry SDK and configure it to export data to Cardinal.

Setup `OTEL` SDK

This step will setup the OTEL SDK and configure it to export data to Cardinal.

include(
    "context"
    "log/slog"
    "os"
    "os/signal"
  	"syscall"
 
	"github.com/cardinalhq/oteltools/pkg/telemetry"
)
 
func main() {
    // Handle ^C and kubernetes termination gracefully.
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()
 
    // Check to see if the environment variable is set, and only enable it
    // when it is set.
    enableOtel := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") != ""
 
    if enableOtel {
        slog.Info("OpenTelemetry exporting enabled")
		otelShutdown, err := telemetry.SetupOTelSDK(ctx)
        if err != nil {
            return err
        }
        defer func() {
            err = errors.Join(err, otelShutdown(context.Background()))
        }()
    }
 
    // Your application logic here
}

Send Logs

`SLog`

include(
    "log/slog"
    "os"
 
    slogmulti "github.com/samber/slog-multi"
    "go.opentelemetry.io/contrib/bridges/otelslog"
)
 
func main() {
    // SDK setup from the previous section
 
    slog.SetDefault(slog.New(slogmulti.Fanout(
        slog.NewJSONHandler(os.Stdout, nil),
        otelslog.NewHandler(os.Getenv("OTEL_SERVICE_NAME")),
    )))
 
    // Your application logic here
}

Send Traces

HTTP Client Tracing

include(
    "net/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
 
// Create a new HTTP client with OpenTelemetry tracing enabled.
// Now all outgoing HTTP requests made with this client will be automatically traced.
client := &http.Client{
			Transport: otelhttp.NewTransport(http.DefaultTransport),
}

HTTP Server Tracing

// currentHttpHandler is your existing HTTP handler, e.g., http.HandlerFunc(myHandler).
// or it can be your existing middleware chain.
handlerWrappedWithOtel := otelhttp.NewHandler(currentHttpHandler, "")
 
s := &http.Server{
		Addr:    address,
		Handler: handlerWrappedWithOtel,
}

Database Query Tracing

To trace database queries made with the pgx/v5 package, you can use the otelpgx package. This package provides middleware that automatically instruments database queries made by your application.

import (
	"context"
 
	"github.com/jackc/pgx/v5/pgxpool"
	"github.com/pgx-contrib/pgxotel"
)
 
func NewConnectionPool(ctx context.Context, url string, dbName string) (*pgxpool.Pool, error) {
	cfg, err := pgxpool.ParseConfig(url)
	if err != nil {
		return nil, err
	}
 
	cfg.ConnConfig.Tracer = &pgxotel.QueryTracer{
		Name: dbName,
	}
 
	return pgxpool.NewWithConfig(ctx, cfg)
}

Message Queue Tracing

Kafka (`github.com/segmentio/kafka-go`)

Create a wrapper for the Kafka TextMapCarrier, that will be used to propagate tracing context through Kafka messages.

import (
	"github.com/segmentio/kafka-go"
)
 
// Create a wrapper for the Kafka TextMapCarrier, that will be used to propagate tracing context through Kafka messages.
type KafkaHeaderCarrier struct {
	Headers *[]kafka.Header
}
 
func (c KafkaHeaderCarrier) Get(key string) string {
	for _, h := range *c.Headers {
		if h.Key == key {
			return string(h.Value)
		}
	}
	return ""
}
 
func (c KafkaHeaderCarrier) Set(key, value string) {
	for i, h := range *c.Headers {
		if h.Key == key {
			(*c.Headers)[i].Value = []byte(value)
			return
		}
	}
	*c.Headers = append(*c.Headers, kafka.Header{Key: key, Value: []byte(value)})
}
 
func (c KafkaHeaderCarrier) Keys() []string {
	keys := make([]string, 0, len(*c.Headers))
	for _, h := range *c.Headers {
		keys = append(keys, h.Key)
	}
	return keys
}

Producer: Attach trace context to Kafka messages when producing them.

  headers := make([]kafka.Header, 0)
	carrier := KafkaHeaderCarrier{Headers: &headers}
	otel.GetTextMapPropagator().Inject(ctx, &carrier)
 
	messages := []kafka.Message{
		{
			Key:     []byte(key),
			Value:   morePermanentBuffer.Bytes(),
			Topic:   topic,
			Headers: headers, // Attach the trace headers to the message
		},
	}
 
	return kafka.WriteMessages(ctx, messages...)

Consumer: Extract trace context from Kafka messages when consuming them (Rehydrate span context).

import (
	"github.com/segmentio/kafka-go"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	semconv "go.opentelemetry.io/otel/semconv/v1.30.0"
)
 
var tracer = otel.Tracer("kafka-consumer") // Create a tracer for your kafka consumer
var propagator = otel.GetTextMapPropagator() // Get a propagator to extract context from messages
 
 
// Fetch a message from Kafka and start a new span with the extracted context
msg, err := reader.FetchMessage(ctx)
msgCtx := propagator.Extract(context.Background(), KafkaHeaderCarrier{&msg.Headers})
 
// OTEL: Start a new span using pre-created tracer
spanName := fmt.Sprintf("kafka.consume %s", topic)
msgCtx, span := tracer.Start(msgCtx, spanName)
span.SetAttributes(attribute.String(string(semconv.MessagingSystemKey), semconv.MessagingSystemKafka.Value.AsString()),
	attribute.String(string(semconv.MessagingDestinationNameKey), topic),
	attribute.Int(string(semconv.MessagingDestinationPartitionIDKey), partition),
 )
 
// Process the message
 
// After processing the message, end the span
span.End()

Send Metrics

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/metric"
)
	meter  = otel.Meter("github.com/<company>/<project>")
 
m, err := meter.Int64Histogram(
		"request_latency",
		metric.WithUnit("ms"),
		metric.WithDescription("The delay in millis for a request."),
)
m.Record(ctx, latency, attribute.String("method", "GET"), attribute.String("path", "/api/v1/resource"))

Java React.js