Go semi-automatic instrumentation
The Go auto-instrumentation obtained from the OpenTelemetry Operator for Kubernetes enables traces with minimal effort, but other signal types are not currently automatic. This includes application logs, custom metrics, and custom spans. To obtain logs and enable metrics as well as manual span generation, some code needs to be added to your Go application.
This guide describes how to add OpenTelemetry to your Go application.
Differences from other languages
Most other languages handle automatic instrumentation by injecting a library into the application's runtime. Go does not have a way to do this, so the OpenTelemetry Operator for Kubernetes instead runs a "sidecar" container which uses eBPF to intercept system calls and application function calls to generate traces. This is a very different approach from other languages, and it means that the Go application itself must be configured to export logs and custom metrics.
Setting up telemetry
For Go, we need to configure various components. These are propagators for traces, and providers and exporters for all signal types. By default, any instrumentation that uses the OpenTelemetry Go SDK will use a "no-op" provider, meaning all data will be dropped. This feature allows setting up the telemetry exporting path after logging is configured and metrics may have been defined, and those will still work.
On to the code!
File otel_setup.go
Create this file, and put it somewhere you can call it, likely near your main.go or similar entrypoint.
This example is similar to the example in the Getting Started (opens in a new tab) guide for the OpenTelemetry Go SDK.
package cmd
import (
"context"
"errors"
"log/slog"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/exporters/stdout/stdoutlog"
"go.opentelemetry.io/otel/log/global"
"go.opentelemetry.io/otel/propagation"
otellog "go.opentelemetry.io/otel/sdk/log"
"go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/trace"
)
// setupOTelSDK bootstraps the OpenTelemetry pipeline.
// If it does not return an error, make sure to call shutdown for proper cleanup.
func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) {
var shutdownFuncs []func(context.Context) error
// shutdown calls cleanup functions registered via shutdownFuncs.
// The errors from the calls are joined.
// Each registered cleanup will be invoked once.
shutdown = func(ctx context.Context) error {
var err error
for _, fn := range shutdownFuncs {
err = errors.Join(err, fn(ctx))
}
shutdownFuncs = nil
return err
}
// handleErr calls shutdown for cleanup and makes sure that all errors are returned.
handleErr := func(inErr error) {
err = errors.Join(inErr, shutdown(ctx))
}
// Set up propagator.
prop := newPropagator()
otel.SetTextMapPropagator(prop)
// Set up trace provider.
tracerProvider, err := newTracerProvider(ctx)
if err != nil {
handleErr(err)
return
}
shutdownFuncs = append(shutdownFuncs, tracerProvider.Shutdown)
otel.SetTracerProvider(tracerProvider)
// Set up meter provider.
meterProvider, err := newMeterProvider(ctx)
if err != nil {
handleErr(err)
return
}
shutdownFuncs = append(shutdownFuncs, meterProvider.Shutdown)
otel.SetMeterProvider(meterProvider)
// Set up logger provider.
loggerProvider, err := newLoggerProvider(ctx)
if err != nil {
handleErr(err)
return
}
shutdownFuncs = append(shutdownFuncs, loggerProvider.Shutdown)
global.SetLoggerProvider(loggerProvider)
return
}
func newPropagator() propagation.TextMapPropagator {
return propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
)
}
func newTracerProvider(ctx context.Context) (*trace.TracerProvider, error) {
traceExporter, err := otlptracehttp.New(ctx, otlptracehttp.WithInsecure())
if err != nil {
return nil, err
}
tracerProvider := trace.NewTracerProvider(
trace.WithBatcher(traceExporter),
)
return tracerProvider, nil
}
func newMeterProvider(ctx context.Context) (*metric.MeterProvider, error) {
metricExporter, err := otlpmetrichttp.New(ctx, otlpmetrichttp.WithInsecure())
if err != nil {
return nil, err
}
meterProvider := metric.NewMeterProvider(
metric.WithReader(metric.NewPeriodicReader(metricExporter)),
)
return meterProvider, nil
}
func newLoggerProvider(ctx context.Context) (*otellog.LoggerProvider, error) {
logExporter, err := otlploghttp.New(ctx, otlploghttp.WithInsecure())
if err != nil {
return nil, err
}
loggerProvider := otellog.NewLoggerProvider(
otellog.WithProcessor(otellog.NewBatchProcessor(logExporter)),
)
return loggerProvider, nil
}Hooking up logging via slog
Go uses a package called log/slog that implements a structured logging framework. While this guide only covers using slog, other logging systems exist. Each one uses a "bridge" that implements some way to take log messages from a Go logging framework and convert it into something that the OpenTelemetry SDK will consume and export.
We will be using the slog bridge (opens in a new tab). A list of other bridges is available in the OpenTelemetry Registry (opens in a new tab).
Environment Variables
Due the Go using a sidecar container, that container is typically configured with the necessary environment variables. As we are doing semi-automatic instrumentation, we need to set these environment variables ourselves.
The most important ones are OTEL_EXPORTER_OTLP_ENDPOINT and
OTEL_SERVICE_NAME. The endpoint is the address of the OpenTelemetry
Collector, and the service name is the name of the service that is
being instrumented. The service name is used to identify the service
in the traces that are generated.
These are usually set in the Kubernetes manifest for your Go application container.
Example Kubernetes manifest
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-go-app
spec:
template:
spec:
containers:
- name: my-go-app
env:
# Set the OpenTelemetry Collector endpoint to the collector
# service for the agent collector.
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://collector-name-agent.collector.svc.cluster.local:4318"
- name: OTEL_SERVICE_NAME
value: "my-go-app"The OTEL_EXPORTER_OTLP_ENDPOINT should point to the agent collector,
which will add the additional Kubernetes and container resource attributes.
See the the recommended Instrumentation configuration for more information.
Go code additions
Add this code near the top of your main() function:
includes to add:
include(
"context"
"log/slog"
"os"
"os/signal"
slogmulti "github.com/samber/slog-multi"
"go.opentelemetry.io/contrib/bridges/otelslog"
...
func main() {
// enableOtel could be set via environment variables or command line flags.
// Here, we check to see if the environment variable is set, and only enable it
// when it is set.
enableOtel := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") != ""
// Handle SIGINT (CTRL+C) gracefully.
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
if enableOtel {
slog.Info("OpenTelemetry exporting enabled")
otelShutdown, err := setupOTelSDK(ctx)
if err != nil {
return err
}
defer func() {
err = errors.Join(err, otelShutdown(context.Background()))
}()
}
slog.SetDefault(slog.New(slogmulti.Fanout(
slog.NewJSONHandler(os.Stdout, nil),
otelslog.NewHandler(os.Getenv("OTEL_SERVICE_NAME")),
)))
}You can choose a different slog.Handler implementation if you prefer to not have JSON logged to the console.
You can also directly use the OpenTelemetry Go SDK to make additional metrics manually create spans. It will do nothing useful without the enableOtel flag being set to true, but it will not cause any problems. One common reason to disable telemetry is for testing and for local development.