Open Telemetry

Open Telemetry (Otel) is an observability framework and toolkit designed to facilitate the

  • Generation

  • Export

  • Collection

of telemetry data such as traces, metrics, and logs.

The Digital Enterprise Suite (and the Digital Distributed Container) provide support for Open Telemetry out of the box. Its default configuration focuses mainly on generation of the telemetry data.

The Digital Enterprise Suite (and the Digital Distributed Container) can also be configured to export of the telemetry.

There is no explicit collection of telemetry data (other than for the purpose of exporting) in the Digital Enterprise Suite (and the Digital Distributed Container).

Main functionality from automation perspective focuses around two parts of open telemetry:

  • traces

  • logging

Traces provide a distributed view over execution of services. Especially important when multiple services are involved (either hosted on DES/DDC or externally). When all systems involved in the execution support open telemetry then there will be a end to end track of execution which can significantly improve troubleshooting sessions.

Logging provide correlation between request handling and log entries. Enabling simpler look up queries to find relevant log entries for particular execution.

Metrics are produced out of the box in Prometheus format so can also be easily exported to look at them with dashboards.

How it works

Telemetry data in DES/DDC are automatically generated for automation endpoints - endpoints that start with /execution path. Out of the box, tracing information are attached for

  • incoming HTTP requests

  • selected inbound connectors

  • selected outbound connectors

This enables the distributed view of execution as systems calling DES/DDC or systems that DES/DDC calls will participate in the same distributed trace.

In addition, trace information (trace_id and span_id) is logged so can be exported to central logging systems and linked when browsing/searching through the logs.

Traces also have extra information (attributes) attached which differ between type of producer of the trace (e.g. connector) but all of them always have following:

  • service.repository - environment where the service is deployed to

  • service.group - group identifier of the service

  • service.artifact - artifact identifier of the service

  • service.version - version identifier of the service

  • status - status of the execution (ok, error, unset)

  • error.type - in case of error status, type of error is provided

Message broker based connectors have following attributes:

  • messaging.operation - operation performed (send, ack, nack)

  • messaging.system - type of messaging system (rabbitmq, kafka)

  • messaging.destination - name of the destination e.g. queue, topic, etc

  • server.address - address of the server/broker

HTTP/REST based connectors have following attributes:

  • http.request.method - HTTP method name e.g. GET or POST

  • http.response.status_code - HTTP response status code e.g. 200

  • network.protocol.name - URL protocol used e.g. https

  • url.full - complete URL used

  • server.address - address of the server

  • server.port - port of the server

Configuration

Telemetry data are generated by default but can be completely disabled via helm parameter --set telemetry.disabled=true.

Exporters configuration

Exporters can be configured as: - none to not send any data (it defaults to that option) - otlp to configure OpenTelementry protocol for sending data out

To enable logs exporter --set telemetry.exporters.logs=otlp

To enable metrics exporter --set telemetry.exporters.metrics=otlp

To enable traces exporter --set telemetry.exporters.traces=otlp

When enabling exporters you can also set the name that will be used to identify this service --set telemetry.name=my-service. By default it uses the name used as part of helm install/upgrade command.

OTLP configuration

OTLP protocol requires additional configuration such as endpoint to be used etc

Complete and latest configuration options for OTLP can be found in the official documentation

They can be set via telemetry.config key and ideally via additional otlp-values.yaml file with following content:

telemetry:
  config:
    OTEL_EXPORTER_OTLP_ENDPOINT: "http...."
    OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf"
All telemetry related options can be stored in this extra values file 
telemetry:
  # controls if the open telemetry is completely disabled
  disabled: false
  # specifies service name to be used
  name: des
  # configures available exporters
  exporters:
    # supported exporters none or otlp
    logs: otlp
    metrics: otlp
    traces: otlp
  # additional configuration of open telemetry runtime, for available options see https://opentelemetry.io/docs/specs/otel/protocol/exporter/
  config:
    OTEL_EXPORTER_OTLP_ENDPOINT: "http...."
    OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf"