🐞 fix: streamline OTLP exporter initialization and fallback handling #231
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🐞 Add Fallback Mechanism for OpenTelemetry Server
Problem
When the OpenTelemetry (OTEL) server becomes unresponsive, the application could potentially hang or crash due to lack of proper fallback mechanisms. This creates reliability issues in production environments.
Solution
Implemented a robust fallback system for OpenTelemetry tracing that ensures the application continues to function even when the OTEL server is unavailable:
Added a console-based fallback exporter that's always active
Implemented timeout and concurrency limits for OTEL operations
Added graceful degradation to console logging
Improved error handling and logging
Key Changes
Created PrettyConsoleSpanExporter as a reliable fallback
Added configuration for OTEL timeouts (5 seconds) and concurrency limits
Implemented environment-aware processing (Batch vs Simple)
Added dual processing setup with automatic fallback
Improved error handling and logging
Testing
To verify the changes:
Start the application normally
Verify spans are being exported to OTEL
Disable OTEL server
Verify application continues running with console logging
Verify no data loss during OTEL outage
Impact
Improved system reliability
No service disruption during OTEL outages
Better observability through fallback logging
Zero downtime during telemetry issues
Related Issues
Fixes #158 - Application crashes when OTEL server is unresponsive