Stream Interruption on Ant Media Server 2.14 due to sudden memory spike on Vultr Kubernetes Engine

[oleul-bjit]

All RTMP streams (~268 cameras) stopped simultaneously on Ant Media Server 2.14 running in Kubernetes. Streams threw memory spiked, heap dump API failed, pod OOMKilled, and all streams resumed only after restart.

Environment

• Operating system and version: Ubuntu 22.04 (Vultr Kubernetes Engine node image)
• Java version: 17
• Ant Media Server version: Enterprise Edition 2.14
• Browser name and version: N/A (issue occurs server-side with RTMP ingest)

Steps to reproduce

1. Run Ant Media Server 2.14 on Vultr Kubernetes Engine (VKE).
2. Ingest ~268 RTMP camera streams continuously with enabling "addDateTimeToHlsFileName": true
3. Observe server behavior during prolonged load.

Expected behavior

RTMP streams should continue generate .ts files
Memory usage should remain stable under expected load.
Heap dump API should work when triggered under memory pressure to allow analysis.

Actual behavior

On 2025-08-18 between 22:11 JST and 22:21 JST, all RTMP streams stopped simultaneously.

Timeline (JST):

22:10:59 → Last successful stream recording observed.
22:11:06 → Stream stop notifications received.
22:11:08 → Multiple MongoInterruptedException from RTMPConnectionExecutor.
22:11:10 → Errors logged: Error executing call: Service: null Method: publish Num Params: 2 0: {streamId}?subscriberId=ru9JD0Dg&subscriberCode=758169 1: live exception message.
22:13:41 → Memory usage spiked to ~50%; heap dump API triggered but did not work.
22:16:01 → Continuous MongoInterruptedException and publish error logs until this time.
22:20:33 → Pod terminated due to OOMKilled.
22:21:21 → Pod restarted; all streams resumed automatically.

Why did all streams stop simultaneously and begin sending exception messages?

What does the following mean?

Error executing call: Service: null Method: publish Num Params: 2
0: {streamId}?subscriberId=ru9JD0Dg&subscriberCode=758169
1: live exception message

When and why does this exception occur?

Why did memory spike sharply during this exception period?

Why did GC + heap dump API fail under memory pressure? Is there any alternative safe way to collect a reliable heap dump in such cases?

Logs

Logs and heap dump are available here:
https://drive.google.com/drive/folders/1HOIOEYU0-yV9JjufHdmZq1TYwYdddN0x?usp=sharing

[burak-58]

Hi @oleul-bjit,
I have checked the logs you shared. Everything starts with Mongo access issues. Mongo queries took long and this causes everything you listed.
ı am converting this issue a discussion so that you can check the Mongo and your setup with one of my friend from support team.

[oleul-bjit]

Hello @burak-58,
Thank you for your investigation.
We did not encounter this issue with version 2.11.3. Is there anything specific about version 2.14 that affects MongoDB? Do we need to update any configurations?

[yashtandon113]

Hi @oleul-bjit

Which version of MongoDB are you using?

Actually in v2.14, some optimizations are done for MongoDB to make it work faster for queries. You can check the 2.14 release logs.

https://github.com/ant-media/Ant-Media-Server/releases/tag/ams-v2.14.0

[oleul-bjit]

Hello @yashtandon113 ,
We are using MongoDB Community version: "7.0.5" with compatible version is "6.0.5"