WARNING: THIS SITE IS A MIRROR OF GITHUB.COM / IT CANNOT LOGIN OR REGISTER ACCOUNTS / THE CONTENTS ARE PROVIDED AS-IS / THIS SITE ASSUMES NO RESPONSIBILITY FOR ANY DISPLAYED CONTENT OR LINKS / IF YOU FOUND SOMETHING MAY NOT GOOD FOR EVERYONE, CONTACT ADMIN AT ilovescratch@foxmail.com
Skip to content

Conversation

@ahsanfarooq210
Copy link
Contributor

🐞 Add Fallback Mechanism for OpenTelemetry Server
Problem
When the OpenTelemetry (OTEL) server becomes unresponsive, the application could potentially hang or crash due to lack of proper fallback mechanisms. This creates reliability issues in production environments.

Solution
Implemented a robust fallback system for OpenTelemetry tracing that ensures the application continues to function even when the OTEL server is unavailable:

Added a console-based fallback exporter that's always active
Implemented timeout and concurrency limits for OTEL operations
Added graceful degradation to console logging
Improved error handling and logging
Key Changes
Created PrettyConsoleSpanExporter as a reliable fallback
Added configuration for OTEL timeouts (5 seconds) and concurrency limits
Implemented environment-aware processing (Batch vs Simple)
Added dual processing setup with automatic fallback
Improved error handling and logging
Testing
To verify the changes:

Start the application normally
Verify spans are being exported to OTEL
Disable OTEL server
Verify application continues running with console logging
Verify no data loss during OTEL outage
Impact
Improved system reliability
No service disruption during OTEL outages
Better observability through fallback logging
Zero downtime during telemetry issues
Related Issues
Fixes #158 - Application crashes when OTEL server is unresponsive

@m1212e
Copy link
Member

m1212e commented Aug 12, 2025

Thank you very much for your contribution, the whole OTEL stuff is on hold for now, since we'll have to work out some implementation details which might be subject to change. As soon as it comes to fully implementing and using OTEL as it is intended, we will get back to this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Check if otel is able to crash the application at runtime

2 participants