Skip to content

Observability & health

Logs, audit events and metrics: what HelixIAM writes, how a collector of your choice reads it, and what you must know about its limits.

The rule

HelixIAM never pushes audit, log or telemetry data. It writes one-line JSON to standard output and serves a Prometheus scrape on the management port. A collector that you run, such as Fluent Bit or the OpenTelemetry Collector, reads both and routes them to your SIEM, Loki, Elastic, Kafka or any other destination.

Identity features that call other systems are not part of this and are unchanged: back-channel logout, SCIM provisioning, identity-provider calls, e-mail and SMS delivery, and the verification API's result webhook.

Signal Output Collected with
Application logs One-line JSON on stdout (log_type app) Fluent Bit, Vector, Alloy, OpenTelemetry Collector, Filebeat
Audit events One-line JSON on stdout (log_type audit), the only copy: HelixIAM keeps none in its database The same collectors, delivered to your audit tool (SIEM, ELK, Loki, Splunk)
Metrics Prometheus scrape at /actuator/prometheus Prometheus, Mimir, VictoriaMetrics, Alloy, OpenTelemetry Collector
Request correlation trace_id and span_id in log lines Whatever stores the logs

What we have tested

Level Tools
Built in JSON on stdout, the Prometheus scrape
Tested by us in a k3d gate Fluent Bit 4.0.7, OpenTelemetry Collector contrib 0.128.0, Loki 3.5.3, Kafka 3.9.1, Prometheus v3.5.4, Grafana 12.0.2. The OTLP exporter in the Collector configuration below is a template that the gate does not exercise.
Standards-compatible, not claimed as tested Splunk, Elastic, Datadog, Dynatrace, Sentinel, syslog, Debezium

Log schema

There is one stream on stdout per process, with two kinds of line told apart by log_type.

Format: HELIX_LOG_FORMAT

Value Output
json Every application line is one JSON object (below). The Helm chart sets this by default (config.logFormat: json).
text Spring Boot's console pattern, one line per message and the stack trace on the following lines. Running the jar locally stays text.

Audit lines are JSON in both formats. A JSON line is larger than the text line of the same event (about 286 bytes against 111 in one measurement), so log volume and its storage rise when you switch.

Application line (json)

{"timestamp":"2026-11-05T13:50:34.351Z","level":"INFO","logger":"io.helixiam.authorization.security.token.TokenIssuer","thread":"http-nio-8080-exec-7","message":"token issued","log_type":"app","realm":"acme","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"b7ad6b7169203331"}
Field Always there Meaning
timestamp yes UTC, ISO 8601, milliseconds
level yes TRACE, DEBUG, INFO, WARN, ERROR, FATAL
logger, thread, message yes Logger name, thread name, the formatted message
log_type yes Always app
exception no The stack trace as one string, only when an exception was logged
realm no The realm the request or task belongs to, where known
trace_id no 32 hex characters (W3C trace id)
span_id no 16 hex characters; this request's own span

Unknown values are omitted, never null. Every string is JSON-escaped, so a username or header with a line break cannot start a second line or overwrite a field such as realm. A message or stack trace longer than 32768 characters is truncated.

Audit line

{"ts":"2026-11-05T13:51:02Z","kind":"audit","category":"AUTHN","type":"LOGIN_SUCCESS","realm":"acme","actor":"jansen","sourceIp":"192.0.2.10","outcome":"SUCCESS","eventId":"0b8e0a52-3d1e-4a0e-9a57-1f1f6f7d2c11","log_type":"audit"}
Field Meaning
log_type Always audit
eventId A UUID that identifies the event, so you can find it in your audit tool and drop duplicates a collector may deliver twice
ts, kind, category, type Time (UTC, seconds), audit, AUTHN or ADMIN, and the event type
realm The realm, when the event has one
actor, sourceIp Who and from where
resourceType, resourceId, outcome, detail What was acted on, SUCCESS, FAILURE or DENIED, extra context (omitted when empty)

Field names are stable: new fields may appear, existing ones are not renamed or retyped within a major version.

Correlation

Every request gets a trace_id and span_id in its log lines. Nothing is exported, there is no tracing SDK, and neither id is sent in a response or on a call to another system. An inbound traceparent header supplies the trace id only when the TCP peer is listed in helix.security.trusted-proxies (Helm config.trustedProxies); from any other peer it is ignored and a new id is generated.

Where lines go: three routing setups

Where audit and log lines go is the configuration of your collector, which can route on the realm field. HelixIAM has no per-realm files, appenders or destinations.

Setup What you configure When a new realm is added
All realms to one system One rule; realm is a searchable field Nothing to do
The same system, separated per realm (a Kafka topic, a Loki label or an Elastic index per realm) One rule; realm fills in the name Nothing, apart from access rights on that system
A realm with its own separate SIEM One extra output for that realm The operator adds that output

A realm admin cannot connect their own SIEM from the console: the operator does it, and only in the third setup.

Fluent Bit

The configuration below reads the container logs of the node, splits application and audit lines, and buffers on disk. It ran in our k3d gate on Fluent Bit 4.0.7. Run it as a DaemonSet with /var/log mounted read-only and a state directory on the node that survives a restart. Adjust the path to your release name and namespace; the container is called server.

# fluent-bit.conf
[SERVICE]
    Flush                     1
    Parsers_File              parsers.conf
    storage.path              /var/fluent-bit/state/flb-storage/
    storage.sync              normal
    storage.backlog.mem_limit 64M
    storage.max_chunks_up     64

[INPUT]
    Name                      tail
    Tag                       kube.*
    Path                      /var/log/containers/helixiam-*_*_server-*.log
    Exclude_Path              /var/log/containers/*-console-*
    multiline.parser          cri
    Read_from_Head            On
    Refresh_Interval          2
    Skip_Long_Lines           Off
    DB                        /var/fluent-bit/state/tail-helixiam.db
    DB.sync                   normal
    storage.type              filesystem
    Mem_Buf_Limit             16MB

[FILTER]
    Name                      parser
    Match                     kube.*
    Key_Name                  log
    Parser                    helix_json
    Reserve_Data              Off
    Preserve_Key              Off

# A line that is not JSON (text mode) keeps its `log` field and counts as an application line.
[FILTER]
    Name                      modify
    Match                     kube.*
    Condition                 Key_does_not_exist log_type
    Add                       log_type app

[FILTER]
    Name                      rewrite_tag
    Match                     kube.*
    Rule                      $log_type ^audit$ hx.audit false
    Rule                      $log_type ^app$   hx.app   false
    Emitter_Name              hx_split
    Emitter_Storage.type      filesystem
    Emitter_Mem_Buf_Limit     16M
# parsers.conf
[PARSER]
    Name        helix_json
    Format      json

The record keeps the time of the container log line, so a line collected late after an outage still has its original time. The filesystem buffering keeps chunks that an output has not delivered yet on disk, which reduces loss when a destination is unreachable or the collector restarts. It cannot bring back a line that the node has already rotated away.

Setup 1, all realms to one system (Loki). Two labels, log_type and job; the whole record stays in the line, so realm is a field to filter on at query time ({job="helixiam", log_type="audit"} | json | realm="acme").

[OUTPUT]
    Name                      loki
    Match                     hx.*
    Host                      ${LOKI_HOST}
    Port                      ${LOKI_PORT}
    Labels                    job=helixiam
    Label_Keys                $log_type
    Line_Format               json
    Retry_Limit               no_limits
    storage.total_limit_size  512M

Setup 2, separated per realm from one rule (Kafka topic helixiam.audit.<realm>). A copy of each audit record gets a topic field from a small Lua function; one output with Dynamic_Topic On produces to it. A realm created later is routed with no change. The broker creates the topic on first use when auto.create.topics.enable is on (the Kafka default); where it forbids that, the operator creates the topic when the realm is created. Events that belong to no realm go to helixiam.audit-norealm. In the topic name . is written as _2e and _ as _5f, because Kafka refuses a second topic whose name differs from the first only in those two characters: realm acme.nl has topic helixiam.audit.acme_2enl.

[FILTER]
    Name                      rewrite_tag
    Match                     hx.audit
    Rule                      $log_type ^audit$ kafka.audit true
    Emitter_Name              hx_kafka
    Emitter_Storage.type      filesystem
    Emitter_Mem_Buf_Limit     16M

[FILTER]
    Name                      lua
    Match                     kafka.audit
    script                    kafka-topic.lua
    call                      set_topic

[OUTPUT]
    Name                      kafka
    Match                     kafka.audit
    Brokers                   ${KAFKA_BROKERS}
    Topics                    helixiam.audit-norealm
    Topic_Key                 topic
    Dynamic_Topic             On
    Format                    json
    rdkafka.request.required.acks all
    rdkafka.enable.idempotence true
    rdkafka.queue.buffering.max.ms 50
    Retry_Limit               no_limits
    storage.total_limit_size  512M
-- kafka-topic.lua
function set_topic(tag, timestamp, record)
    local realm = record["realm"]
    local topic = "helixiam.audit-norealm"
    if type(realm) == "string" and string.match(realm, "^[A-Za-z0-9._-]+$") and #realm <= 100 then
        local name = "helixiam.audit." .. string.gsub(string.gsub(realm, "_", "_5f"), "%.", "_2e")
        if #name <= 249 then
            topic = name
        end
    end
    record["topic"] = topic
    return 1, timestamp, record
end

Before production. These outputs have no TLS or authentication, and they carry audit data (actors, source addresses): add your system's TLS and authentication settings. Fluent Bit's buffer and tail database hold audit lines, so keep their path readable by the shipper only. Keep Fluent Bit's HTTP port inside the cluster. Replace the OpenTelemetry Collector's debug exporter, which prints every line to the Collector's own stdout.

Setup 3, one realm also to its own SIEM. One rule copies the records of that realm (here acme) to a tag, and one output sends them to that realm's destination. Add one rule and one output per further realm. The output below is a TCP listener that takes JSON lines; swap in the output your SIEM uses.

[FILTER]
    Name                      rewrite_tag
    Match                     hx.audit
    Rule                      $realm ^acme$ siem.acme true
    Emitter_Name              hx_siem
    Emitter_Storage.type      filesystem
    Emitter_Mem_Buf_Limit     16M

[OUTPUT]
    Name                      tcp
    Match                     siem.acme
    Host                      ${SIEM_HOST}
    Port                      ${SIEM_PORT}
    Format                    json_lines
    Retry_Limit               no_limits
    storage.total_limit_size  256M

OpenTelemetry Collector

For platforms that take OTLP, the Collector reads the container log files and scrapes the metrics, and does the OTLP export itself: HelixIAM speaks no OTLP. The filelog receiver and the Prometheus receiver ran in our gate (OpenTelemetry Collector contrib 0.128.0) with the debug exporter. The otlphttp exporter is a template and was not exercised.

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true

receivers:
  filelog:
    include:
      - /var/log/pods/*_helixiam-*/server/*.log
    exclude:
      - /var/log/pods/*_helixiam-console-*/*/*.log
    start_at: beginning
    include_file_path: true
    storage: file_storage
    operators:
      - type: container
        id: container
      - type: json_parser
        id: helix_json
        if: 'body matches "^\\s*\\{"'
        parse_from: body
        parse_to: attributes
      - type: time_parser
        id: app_time
        if: 'attributes.timestamp != nil'
        parse_from: attributes.timestamp
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      - type: time_parser
        id: audit_time
        if: 'attributes.timestamp == nil && attributes.ts != nil'
        parse_from: attributes.ts
        layout: '%Y-%m-%dT%H:%M:%SZ'
      - type: severity_parser
        id: level
        if: 'attributes.level != nil'
        parse_from: attributes.level
        mapping: {debug: DEBUG, info: INFO, warn: WARN, error: ERROR, fatal: FATAL}
      - type: move
        id: message_to_body
        if: 'attributes.message != nil'
        from: attributes.message
        to: body

  prometheus:
    config:
      scrape_configs:
        - job_name: helixiam
          scrape_interval: 15s
          metrics_path: /actuator/prometheus
          static_configs:
            - targets: ["${env:HELIX_MANAGEMENT_TARGET}"]

processors:
  batch:
    timeout: 1s

exporters:
  debug:
    verbosity: detailed
  # Template for an OTLP platform, not exercised: set your endpoint and add `otlphttp` to the pipelines.
  # otlphttp:
  #   endpoint: https://otlp.example.com
  #   headers: {authorization: "Bearer ${env:OTLP_TOKEN}"}

service:
  extensions: [file_storage]
  pipelines:
    logs:    {receivers: [filelog],    processors: [batch], exporters: [debug]}
    metrics: {receivers: [prometheus], processors: [batch], exporters: [debug]}

Per realm: audit categories and temporary DEBUG

There is still one stream per process. In the console under Realm settings → Logging (and over GET/PUT /admin/realms/{realm}/logging) a realm has two settings:

Setting Who may change it Effect
Audit categories (AUTHN, ADMIN) A realm admin may add to the application-wide set (helix.audit.categories) or follow it again. Writing fewer categories needs a master-realm admin (a realm admin gets 403). The realm's audit events are written (on stdout) only for its categories. Events with no realm use the application-wide set. helix.audit.enabled=false still turns everything off.
Temporary DEBUG A master-realm admin only DEBUG lines of HelixIAM's own loggers with that realm, for 1 to 60 minutes (the server refuses more). Never TRACE, never third-party loggers. It stops by itself at its deadline on every replica.

Debug lines can hold more personal data than normal lines. Every change is audited (REALM_LOGGING_UPDATE, with the previous and new categories), and that event is written whatever categories the realm ships, so a change that reduces auditing cannot hide itself. These settings are not part of a realm export.

Checking your pipeline: audit status and the test event

The console's Audit page shows whether audit is enabled, the audit categories and the log format in effect (GET /admin/audit/status), and links to the realm's audit categories. On the master console (master-realm admins only) Send test event writes an AUDIT_TEST event (detail.marker is audit-test) to stdout, whatever the audit categories are, and shows its eventId; search for that id in your audit tool. At most 5 a minute per pod.

curl -s -b cookies.txt "$HELIX_URL/admin/audit/status"
CSRF=$(awk '($6=="__Host-XSRF-TOKEN"||$6=="XSRF-TOKEN")&&$7~/^r1\./{t=$7} END{print t}' cookies.txt)
curl -s -b cookies.txt -X POST -H "X-XSRF-TOKEN: $CSRF" "$HELIX_URL/admin/audit/test-event"

The test event answers 409 when audit is turned off (helix.audit.enabled=false), because no line would be written. HelixIAM knows only that it wrote the line to its standard output; it cannot know whether your collector received it.

Audit data is not stored by HelixIAM

HelixIAM keeps no audit events in its database. Each event is one synchronous JSON line on stdout, written before the request returns, and your audit tool stores and searches it. There is no audit writer, queue or audit metric to watch. With grafanaDashboards.enabled=true the Helm chart ships three Grafana dashboards (Overview, Sign-in and tokens, Key custody) as ConfigMaps for the Grafana sidecar. They are off by default and no Grafana is bundled.

Realm admins see their realm's audit events only through your audit tool, for example a per-realm Kafka topic or a Loki label as in the shipped Fluent Bit examples. Give them access there.

Limits, stated plainly

  • Audit is not atomic with the action. A crash between the action and its audit line can lose the record.
  • There is no second copy. If the collector is down for longer than the node keeps container logs, or a pod is deleted before its lines are read, those lines are gone. Fluent Bit filesystem buffering (as in our shipped example) and adequate container log rotation are required for production. HelixIAM cannot know whether a collector received a line.
  • Kafka must keep its data. In our test, Fluent Bit's Kafka output dropped messages when the broker had lost its topics ("Unknown partition") and did not resend them. A stopped broker that kept its data lost nothing. Use persistent storage for Kafka.
  • Stdout holds all realms. A tenant that must be separated from the operator needs its own deployment.
  • A per-client sign-in rate-limit refusal (429) writes no audit event.
  • /actuator/prometheus is authenticated unless config.prometheusAnonymous=true.

Moving from forwarding and webhooks

Audit HTTP forwarding (helix.audit.http.*) and realm webhooks were removed. Audit events are on stdout in every release; set up a collector as above before you upgrade if you relied on either. The webhook_subscription table is dropped by a migration. The audit_log table is dropped too (migration V205): export it before you upgrade if you need the history, for example psql "$DATABASE_URL" -c "\copy (SELECT * FROM audit_log) TO 'audit_log.csv' CSV HEADER". The audit writer, its metrics (helix_audit_write_*, helix_audit_queue_*), the alerts HelixIAMAuditWriteMissed and HelixIAMAuditQueueBacklog, the helix.audit.write.* settings, the audit retention setting and the admin events search endpoint are removed with it. A realm export from an older release that still has a webhooks section imports normally: the section is skipped, reported as skipped, and one warning is logged.

Metrics and health

Prometheus scrapes the standard endpoint. Alongside JVM and HTTP metrics, HelixIAM publishes purpose-built counters, each tagged by realm:

Counter Measures
helix_login_total Login attempts / outcomes
helix_tokens_issued_total Tokens issued
helix_mfa_challenge_total MFA challenges raised
helix_admin_write_total Administrative write operations

The console Health dashboard summarises component health, realm inventory, and live login/token metrics in one place:

GET /admin/realms/{realm}/health
GET /admin/metrics/summary

Over the monitoring endpoints

The server exposes two actuator endpoints: health (with its liveness / readiness groups) and prometheus. /actuator/info is not exposed. The examples assume $HELIX_URL from Authenticating to the API.

Liveness & readiness — GET /actuator/health

Health is anonymous — point your load balancer and Kubernetes probes here, no session or token:

curl -s "$HELIX_URL/actuator/health"
curl -s "$HELIX_URL/actuator/health/liveness"
curl -s "$HELIX_URL/actuator/health/readiness"
{"status":"UP","groups":["liveness","readiness"]}
{"status":"UP"}
{"status":"UP"}

Probe the group endpoints — /actuator/health/liveness and /actuator/health/readiness — to gate restarts and traffic; the Helm chart's probes do. With both stores on queue, Redis is never contacted and its health indicator is off, so readiness does not depend on it.

Scrape metrics — GET /actuator/prometheus

The Prometheus endpoint is not anonymous by default: without credentials it answers 401.

curl -s -o /dev/null -w '%{http_code}\n' "$HELIX_URL/actuator/prometheus"
401

To let Prometheus scrape it, set HELIX_ACTUATOR_PROMETHEUS_ANONYMOUS=true (Helm: config.prometheusAnonymous) and restrict the path at the network layer — the chart's NetworkPolicy admits scrapes from the namespace you set in networkPolicy.prometheusNamespaceSelector. With an admin session (-b cookies.txt) you can read it for a quick look:

curl -s -b cookies.txt "$HELIX_URL/actuator/prometheus" | grep '^helix_'
helix_db_encryption_legacy_failed_columns 0.0
helix_email_retry_queued 0.0
helix_login_total{outcome="success",realm="master"} 1.0

The purpose-built counters carry a realm tag, so a single Prometheus target gives you per-realm login, token and admin-write rates; a counter appears once it has been incremented. helix_db_encryption_legacy_failed_columns should stay 0 — see Security hardening.

Health dashboard API

The console's Health dashboard reads these admin endpoints (admin session):

curl -s -b cookies.txt "$HELIX_URL/admin/realms/$REALM/health"
curl -s -b cookies.txt "$HELIX_URL/admin/metrics/summary"
{"overall":"healthy","components":[{"name":"Database (PostgreSQL)","status":"up","detail":"Reads succeeding"}, …],"counts":{"applications":3,"oidcClients":3,"samlRelyingParties":0,"identityProviders":0},"checkedAt":"2026-10-13T05:00:37.372594717Z"}
{"realm":null,"loginSuccess":1,"loginFailure":0,"loginTotal":1,"loginSuccessRate":1.0,"mfaSuccess":0,"mfaFailure":0,"tokensIssued":0,"adminWrites":0}

Which release is running

/actuator/info is not exposed (build information is not disclosed). Check the image tag you deployed (kubectl get deploy helixiam -o jsonpath='{.spec.template.spec.containers[0].image}', or docker compose images) and that readiness is UP after a rollout.

Latency histograms

Enable HTTP latency histograms to unlock p99 panels:

HELIX_HTTP_HISTOGRAM=true

Leave this off unless you need percentile latency, since histograms add metric cardinality.

See also