WARNING: THIS SITE IS A MIRROR OF GITHUB.COM / IT CANNOT LOGIN OR REGISTER ACCOUNTS / THE CONTENTS ARE PROVIDED AS-IS / THIS SITE ASSUMES NO RESPONSIBILITY FOR ANY DISPLAYED CONTENT OR LINKS / IF YOU FOUND SOMETHING MAY NOT GOOD FOR EVERYONE, CONTACT ADMIN AT ilovescratch@foxmail.com
Skip to content

[beats receivers] Log messages from the collector are not always identified as being from the collector #11785

@cmacknz

Description

@cmacknz

In 9.3.0-SNAPSHOT using the any output configuration with max_retries: -1 will cause the collector exit at startup.

The collector error logs that result (the final two in the logs below) are not obviously sourced from the collector and contain no "log":{"source" object.

{"log.level":"info","@timestamp":"2025-12-12T19:07:34.149Z","log.logger":"otel_manager","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/otel/manager.(*subprocessExecution).startCollector","file.name":"manager/execution_subprocess.go","file.line":134},"message":"supervised collector started with pid: 33738 and healthcheck port: 61826","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-12T19:07:34.778Z","message":"invalid configuration: exporters::elasticsearch/_agent-component/default: retry::max_retries should be non-negative","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-12T19:07:34.778Z","message":"exporters::elasticsearch/_agent-component/monitoring: retry::max_retries should be non-negative","ecs.version":"1.6.0"}

#11448 improved this a bit and we now get the final logs from the collector identified as being source from the otel_manager but the original logs from the collector appear as if they are from Elastic Agent itself making it impossible to filter out collector logs.

{"log.level":"info","@timestamp":"2025-12-12T19:07:34.149Z","log.logger":"otel_manager","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/otel/manager.(*subprocessExecution).startCollector","file.name":"manager/execution_subprocess.go","file.line":134},"message":"supervised collector started with pid: 33738 and healthcheck port: 61826","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-12T19:07:34.778Z","message":"invalid configuration: exporters::elasticsearch/_agent-component/default: retry::max_retries should be non-negative","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-12T19:07:34.778Z","message":"exporters::elasticsearch/_agent-component/monitoring: retry::max_retries should be non-negative","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-12-12T19:07:34.785Z","log.logger":"otel_manager","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/otel/manager.(*OTelManager).Run","file.name":"manager/manager.go","file.line":295},"message":"collector exited with error (will try to recover in 159ns): exporters::elasticsearch/_agent-component/monitoring: retry::max_retries should be non-negative","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-12T19:07:34.785Z","log.logger":"otel_manager","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/otel/manager.(*OTelManager).Run","file.name":"manager/manager.go","file.line":239},"message":"collector recovery restarting, total retries: 2","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

All logs from the collector should include a "log":{"source":"elastic-otel-collector"} property. This potentially will be addressed by the binary split when the collector binary is separated from that of Elastic Agent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions