Observability & health¶
Logs, audit events and metrics: what HelixIAM writes, how a collector of your choice reads it, and what you must know about its limits.
The rule¶
HelixIAM never pushes audit, log or telemetry data. It writes one-line JSON to standard output and serves a Prometheus scrape on the management port. A collector that you run, such as Fluent Bit or the OpenTelemetry Collector, reads both and routes them to your SIEM, Loki, Elastic, Kafka or any other destination.
Identity features that call other systems are not part of this and are unchanged: back-channel logout, SCIM provisioning, identity-provider calls, e-mail and SMS delivery, and the verification API's result webhook.
| Signal | Output | Collected with |
|---|---|---|
| Application logs | One-line JSON on stdout (log_type app) |
Fluent Bit, Vector, Alloy, OpenTelemetry Collector, Filebeat |
| Audit events | One-line JSON on stdout (log_type audit), the only copy: HelixIAM keeps none in its database |
The same collectors, delivered to your audit tool (SIEM, ELK, Loki, Splunk) |
| Metrics | Prometheus scrape at /actuator/prometheus |
Prometheus, Mimir, VictoriaMetrics, Alloy, OpenTelemetry Collector |
| Request correlation | trace_id and span_id in log lines |
Whatever stores the logs |
What we have tested¶
| Level | Tools |
|---|---|
| Built in | JSON on stdout, the Prometheus scrape |
| Tested by us in a k3d gate | Fluent Bit 4.0.7, OpenTelemetry Collector contrib 0.128.0, Loki 3.5.3, Kafka 3.9.1, Prometheus v3.5.4, Grafana 12.0.2. The OTLP exporter in the Collector configuration below is a template that the gate does not exercise. |
| Standards-compatible, not claimed as tested | Splunk, Elastic, Datadog, Dynatrace, Sentinel, syslog, Debezium |
Log schema¶
There is one stream on stdout per process, with two kinds of line told apart by log_type.
Format: HELIX_LOG_FORMAT¶
| Value | Output |
|---|---|
json |
Every application line is one JSON object (below). The Helm chart sets this by default (config.logFormat: json). |
text |
Spring Boot's console pattern, one line per message and the stack trace on the following lines. Running the jar locally stays text. |
Audit lines are JSON in both formats. A JSON line is larger than the text line of the same event (about 286 bytes against 111 in one measurement), so log volume and its storage rise when you switch.
Application line (json)¶
{"timestamp":"2026-11-05T13:50:34.351Z","level":"INFO","logger":"io.helixiam.authorization.security.token.TokenIssuer","thread":"http-nio-8080-exec-7","message":"token issued","log_type":"app","realm":"acme","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"b7ad6b7169203331"}
| Field | Always there | Meaning |
|---|---|---|
timestamp |
yes | UTC, ISO 8601, milliseconds |
level |
yes | TRACE, DEBUG, INFO, WARN, ERROR, FATAL |
logger, thread, message |
yes | Logger name, thread name, the formatted message |
log_type |
yes | Always app |
exception |
no | The stack trace as one string, only when an exception was logged |
realm |
no | The realm the request or task belongs to, where known |
trace_id |
no | 32 hex characters (W3C trace id) |
span_id |
no | 16 hex characters; this request's own span |
Unknown values are omitted, never null. Every string is JSON-escaped, so a username or header with a line break cannot start a second line or overwrite a field such as realm. A message or stack trace longer than 32768 characters is truncated.
Audit line¶
{"ts":"2026-11-05T13:51:02Z","kind":"audit","category":"AUTHN","type":"LOGIN_SUCCESS","realm":"acme","actor":"jansen","sourceIp":"192.0.2.10","outcome":"SUCCESS","eventId":"0b8e0a52-3d1e-4a0e-9a57-1f1f6f7d2c11","log_type":"audit"}
| Field | Meaning |
|---|---|
log_type |
Always audit |
eventId |
A UUID that identifies the event, so you can find it in your audit tool and drop duplicates a collector may deliver twice |
ts, kind, category, type |
Time (UTC, seconds), audit, AUTHN or ADMIN, and the event type |
realm |
The realm, when the event has one |
actor, sourceIp |
Who and from where |
resourceType, resourceId, outcome, detail |
What was acted on, SUCCESS, FAILURE or DENIED, extra context (omitted when empty) |
Field names are stable: new fields may appear, existing ones are not renamed or retyped within a major version.
Correlation¶
Every request gets a trace_id and span_id in its log lines. Nothing is exported, there is no tracing SDK, and neither id is sent in a response or on a call to another system. An inbound traceparent header supplies the trace id only when the TCP peer is listed in helix.security.trusted-proxies (Helm config.trustedProxies); from any other peer it is ignored and a new id is generated.
Where lines go: three routing setups¶
Where audit and log lines go is the configuration of your collector, which can route on the realm field. HelixIAM has no per-realm files, appenders or destinations.
| Setup | What you configure | When a new realm is added |
|---|---|---|
| All realms to one system | One rule; realm is a searchable field |
Nothing to do |
| The same system, separated per realm (a Kafka topic, a Loki label or an Elastic index per realm) | One rule; realm fills in the name |
Nothing, apart from access rights on that system |
| A realm with its own separate SIEM | One extra output for that realm | The operator adds that output |
A realm admin cannot connect their own SIEM from the console: the operator does it, and only in the third setup.
Fluent Bit¶
The configuration below reads the container logs of the node, splits application and audit lines, and buffers on disk. It ran in our k3d gate on Fluent Bit 4.0.7. Run it as a DaemonSet with /var/log mounted read-only and a state directory on the node that survives a restart. Adjust the path to your release name and namespace; the container is called server.
# fluent-bit.conf
[SERVICE]
Flush 1
Parsers_File parsers.conf
storage.path /var/fluent-bit/state/flb-storage/
storage.sync normal
storage.backlog.mem_limit 64M
storage.max_chunks_up 64
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/helixiam-*_*_server-*.log
Exclude_Path /var/log/containers/*-console-*
multiline.parser cri
Read_from_Head On
Refresh_Interval 2
Skip_Long_Lines Off
DB /var/fluent-bit/state/tail-helixiam.db
DB.sync normal
storage.type filesystem
Mem_Buf_Limit 16MB
[FILTER]
Name parser
Match kube.*
Key_Name log
Parser helix_json
Reserve_Data Off
Preserve_Key Off
# A line that is not JSON (text mode) keeps its `log` field and counts as an application line.
[FILTER]
Name modify
Match kube.*
Condition Key_does_not_exist log_type
Add log_type app
[FILTER]
Name rewrite_tag
Match kube.*
Rule $log_type ^audit$ hx.audit false
Rule $log_type ^app$ hx.app false
Emitter_Name hx_split
Emitter_Storage.type filesystem
Emitter_Mem_Buf_Limit 16M
The record keeps the time of the container log line, so a line collected late after an outage still has its original time. The filesystem buffering keeps chunks that an output has not delivered yet on disk, which reduces loss when a destination is unreachable or the collector restarts. It cannot bring back a line that the node has already rotated away.
Setup 1, all realms to one system (Loki). Two labels, log_type and job; the whole record stays in the line, so realm is a field to filter on at query time ({job="helixiam", log_type="audit"} | json | realm="acme").
[OUTPUT]
Name loki
Match hx.*
Host ${LOKI_HOST}
Port ${LOKI_PORT}
Labels job=helixiam
Label_Keys $log_type
Line_Format json
Retry_Limit no_limits
storage.total_limit_size 512M
Setup 2, separated per realm from one rule (Kafka topic helixiam.audit.<realm>). A copy of each audit record gets a topic field from a small Lua function; one output with Dynamic_Topic On produces to it. A realm created later is routed with no change. The broker creates the topic on first use when auto.create.topics.enable is on (the Kafka default); where it forbids that, the operator creates the topic when the realm is created. Events that belong to no realm go to helixiam.audit-norealm. In the topic name . is written as _2e and _ as _5f, because Kafka refuses a second topic whose name differs from the first only in those two characters: realm acme.nl has topic helixiam.audit.acme_2enl.
[FILTER]
Name rewrite_tag
Match hx.audit
Rule $log_type ^audit$ kafka.audit true
Emitter_Name hx_kafka
Emitter_Storage.type filesystem
Emitter_Mem_Buf_Limit 16M
[FILTER]
Name lua
Match kafka.audit
script kafka-topic.lua
call set_topic
[OUTPUT]
Name kafka
Match kafka.audit
Brokers ${KAFKA_BROKERS}
Topics helixiam.audit-norealm
Topic_Key topic
Dynamic_Topic On
Format json
rdkafka.request.required.acks all
rdkafka.enable.idempotence true
rdkafka.queue.buffering.max.ms 50
Retry_Limit no_limits
storage.total_limit_size 512M
-- kafka-topic.lua
function set_topic(tag, timestamp, record)
local realm = record["realm"]
local topic = "helixiam.audit-norealm"
if type(realm) == "string" and string.match(realm, "^[A-Za-z0-9._-]+$") and #realm <= 100 then
local name = "helixiam.audit." .. string.gsub(string.gsub(realm, "_", "_5f"), "%.", "_2e")
if #name <= 249 then
topic = name
end
end
record["topic"] = topic
return 1, timestamp, record
end
Before production. These outputs have no TLS or authentication, and they carry audit data (actors, source addresses): add your system's TLS and authentication settings. Fluent Bit's buffer and tail database hold audit lines, so keep their path readable by the shipper only. Keep Fluent Bit's HTTP port inside the cluster. Replace the OpenTelemetry Collector's debug exporter, which prints every line to the Collector's own stdout.
Setup 3, one realm also to its own SIEM. One rule copies the records of that realm (here acme) to a tag, and one output sends them to that realm's destination. Add one rule and one output per further realm. The output below is a TCP listener that takes JSON lines; swap in the output your SIEM uses.
[FILTER]
Name rewrite_tag
Match hx.audit
Rule $realm ^acme$ siem.acme true
Emitter_Name hx_siem
Emitter_Storage.type filesystem
Emitter_Mem_Buf_Limit 16M
[OUTPUT]
Name tcp
Match siem.acme
Host ${SIEM_HOST}
Port ${SIEM_PORT}
Format json_lines
Retry_Limit no_limits
storage.total_limit_size 256M
OpenTelemetry Collector¶
For platforms that take OTLP, the Collector reads the container log files and scrapes the metrics, and does the OTLP export itself: HelixIAM speaks no OTLP. The filelog receiver and the Prometheus receiver ran in our gate (OpenTelemetry Collector contrib 0.128.0) with the debug exporter. The otlphttp exporter is a template and was not exercised.
extensions:
file_storage:
directory: /var/lib/otelcol/storage
create_directory: true
receivers:
filelog:
include:
- /var/log/pods/*_helixiam-*/server/*.log
exclude:
- /var/log/pods/*_helixiam-console-*/*/*.log
start_at: beginning
include_file_path: true
storage: file_storage
operators:
- type: container
id: container
- type: json_parser
id: helix_json
if: 'body matches "^\\s*\\{"'
parse_from: body
parse_to: attributes
- type: time_parser
id: app_time
if: 'attributes.timestamp != nil'
parse_from: attributes.timestamp
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
- type: time_parser
id: audit_time
if: 'attributes.timestamp == nil && attributes.ts != nil'
parse_from: attributes.ts
layout: '%Y-%m-%dT%H:%M:%SZ'
- type: severity_parser
id: level
if: 'attributes.level != nil'
parse_from: attributes.level
mapping: {debug: DEBUG, info: INFO, warn: WARN, error: ERROR, fatal: FATAL}
- type: move
id: message_to_body
if: 'attributes.message != nil'
from: attributes.message
to: body
prometheus:
config:
scrape_configs:
- job_name: helixiam
scrape_interval: 15s
metrics_path: /actuator/prometheus
static_configs:
- targets: ["${env:HELIX_MANAGEMENT_TARGET}"]
processors:
batch:
timeout: 1s
exporters:
debug:
verbosity: detailed
# Template for an OTLP platform, not exercised: set your endpoint and add `otlphttp` to the pipelines.
# otlphttp:
# endpoint: https://otlp.example.com
# headers: {authorization: "Bearer ${env:OTLP_TOKEN}"}
service:
extensions: [file_storage]
pipelines:
logs: {receivers: [filelog], processors: [batch], exporters: [debug]}
metrics: {receivers: [prometheus], processors: [batch], exporters: [debug]}
Per realm: audit categories and temporary DEBUG¶
There is still one stream per process. In the console under Realm settings → Logging (and over GET/PUT /admin/realms/{realm}/logging) a realm has two settings:
| Setting | Who may change it | Effect |
|---|---|---|
Audit categories (AUTHN, ADMIN) |
A realm admin may add to the application-wide set (helix.audit.categories) or follow it again. Writing fewer categories needs a master-realm admin (a realm admin gets 403). |
The realm's audit events are written (on stdout) only for its categories. Events with no realm use the application-wide set. helix.audit.enabled=false still turns everything off. |
| Temporary DEBUG | A master-realm admin only | DEBUG lines of HelixIAM's own loggers with that realm, for 1 to 60 minutes (the server refuses more). Never TRACE, never third-party loggers. It stops by itself at its deadline on every replica. |
Debug lines can hold more personal data than normal lines. Every change is audited (REALM_LOGGING_UPDATE, with the previous and new categories), and that event is written whatever categories the realm ships, so a change that reduces auditing cannot hide itself. These settings are not part of a realm export.
Checking your pipeline: audit status and the test event¶
The console's Audit page shows whether audit is enabled, the audit categories and the log format in effect (GET /admin/audit/status), and links to the realm's audit categories. On the master console (master-realm admins only) Send test event writes an AUDIT_TEST event (detail.marker is audit-test) to stdout, whatever the audit categories are, and shows its eventId; search for that id in your audit tool. At most 5 a minute per pod.
curl -s -b cookies.txt "$HELIX_URL/admin/audit/status"
CSRF=$(awk '($6=="__Host-XSRF-TOKEN"||$6=="XSRF-TOKEN")&&$7~/^r1\./{t=$7} END{print t}' cookies.txt)
curl -s -b cookies.txt -X POST -H "X-XSRF-TOKEN: $CSRF" "$HELIX_URL/admin/audit/test-event"
The test event answers 409 when audit is turned off (helix.audit.enabled=false), because no line would be written. HelixIAM knows only that it wrote the line to its standard output; it cannot know whether your collector received it.
Audit data is not stored by HelixIAM¶
HelixIAM keeps no audit events in its database. Each event is one synchronous JSON line on stdout, written before the request returns, and your audit tool stores and searches it. There is no audit writer, queue or audit metric to watch. With grafanaDashboards.enabled=true the Helm chart ships three Grafana dashboards (Overview, Sign-in and tokens, Key custody) as ConfigMaps for the Grafana sidecar. They are off by default and no Grafana is bundled.
Realm admins see their realm's audit events only through your audit tool, for example a per-realm Kafka topic or a Loki label as in the shipped Fluent Bit examples. Give them access there.
Limits, stated plainly¶
- Audit is not atomic with the action. A crash between the action and its audit line can lose the record.
- There is no second copy. If the collector is down for longer than the node keeps container logs, or a pod is deleted before its lines are read, those lines are gone. Fluent Bit filesystem buffering (as in our shipped example) and adequate container log rotation are required for production. HelixIAM cannot know whether a collector received a line.
- Kafka must keep its data. In our test, Fluent Bit's Kafka output dropped messages when the broker had lost its topics ("Unknown partition") and did not resend them. A stopped broker that kept its data lost nothing. Use persistent storage for Kafka.
- Stdout holds all realms. A tenant that must be separated from the operator needs its own deployment.
- A per-client sign-in rate-limit refusal (
429) writes no audit event. /actuator/prometheusis authenticated unlessconfig.prometheusAnonymous=true.
Moving from forwarding and webhooks¶
Audit HTTP forwarding (helix.audit.http.*) and realm webhooks were removed. Audit events are on stdout in every release; set up a collector as above before you upgrade if you relied on either. The webhook_subscription table is dropped by a migration. The audit_log table is dropped too (migration V205): export it before you upgrade if you need the history, for example psql "$DATABASE_URL" -c "\copy (SELECT * FROM audit_log) TO 'audit_log.csv' CSV HEADER". The audit writer, its metrics (helix_audit_write_*, helix_audit_queue_*), the alerts HelixIAMAuditWriteMissed and HelixIAMAuditQueueBacklog, the helix.audit.write.* settings, the audit retention setting and the admin events search endpoint are removed with it. A realm export from an older release that still has a webhooks section imports normally: the section is skipped, reported as skipped, and one warning is logged.
Metrics and health¶
Prometheus scrapes the standard endpoint. Alongside JVM and HTTP metrics, HelixIAM publishes purpose-built counters, each tagged by realm:
| Counter | Measures |
|---|---|
helix_login_total |
Login attempts / outcomes |
helix_tokens_issued_total |
Tokens issued |
helix_mfa_challenge_total |
MFA challenges raised |
helix_admin_write_total |
Administrative write operations |
The console Health dashboard summarises component health, realm inventory, and live login/token metrics in one place:
Over the monitoring endpoints¶
The server exposes two actuator endpoints: health (with its liveness / readiness groups) and prometheus. /actuator/info is not exposed. The examples assume $HELIX_URL from Authenticating to the API.
Liveness & readiness — GET /actuator/health¶
Health is anonymous — point your load balancer and Kubernetes probes here, no session or token:
curl -s "$HELIX_URL/actuator/health"
curl -s "$HELIX_URL/actuator/health/liveness"
curl -s "$HELIX_URL/actuator/health/readiness"
Probe the group endpoints — /actuator/health/liveness and /actuator/health/readiness — to gate restarts and traffic; the Helm chart's probes do. With both stores on queue, Redis is never contacted and its health indicator is off, so readiness does not depend on it.
Scrape metrics — GET /actuator/prometheus¶
The Prometheus endpoint is not anonymous by default: without credentials it answers 401.
To let Prometheus scrape it, set HELIX_ACTUATOR_PROMETHEUS_ANONYMOUS=true (Helm: config.prometheusAnonymous) and restrict the path at the network layer — the chart's NetworkPolicy admits scrapes from the namespace you set in networkPolicy.prometheusNamespaceSelector. With an admin session (-b cookies.txt) you can read it for a quick look:
helix_db_encryption_legacy_failed_columns 0.0
helix_email_retry_queued 0.0
helix_login_total{outcome="success",realm="master"} 1.0
The purpose-built counters carry a realm tag, so a single Prometheus target gives you per-realm login, token and admin-write rates; a counter appears once it has been incremented. helix_db_encryption_legacy_failed_columns should stay 0 — see Security hardening.
Health dashboard API¶
The console's Health dashboard reads these admin endpoints (admin session):
curl -s -b cookies.txt "$HELIX_URL/admin/realms/$REALM/health"
curl -s -b cookies.txt "$HELIX_URL/admin/metrics/summary"
{"overall":"healthy","components":[{"name":"Database (PostgreSQL)","status":"up","detail":"Reads succeeding"}, …],"counts":{"applications":3,"oidcClients":3,"samlRelyingParties":0,"identityProviders":0},"checkedAt":"2026-10-13T05:00:37.372594717Z"}
{"realm":null,"loginSuccess":1,"loginFailure":0,"loginTotal":1,"loginSuccessRate":1.0,"mfaSuccess":0,"mfaFailure":0,"tokensIssued":0,"adminWrites":0}
Which release is running¶
/actuator/info is not exposed (build information is not disclosed). Check the image tag you deployed (kubectl get deploy helixiam -o jsonpath='{.spec.template.spec.containers[0].image}', or docker compose images) and that readiness is UP after a rollout.
Latency histograms¶
Enable HTTP latency histograms to unlock p99 panels:
Leave this off unless you need percentile latency, since histograms add metric cardinality.