Collection creation hangs after redeployment of Milvus v2.6.5 in kubernetes cluster

[Siddeshvr]

Environment:
Milvus version: v2.6.5
Deployment mode(standalone or cluster): cluster (helm)
MQ type: Woodpecker
Storage: S3

I deployed Milvus v2.6.5 fresh on my AWS Kubernetes cluster and ran a collection creation operation — it worked as expected. However, after redeploying the same version (v2.6.5) to the same cluster multiple times, a subsequent collection creation operation hangs indefinitely (never completes).
Then I cleaned up the deployment (deleted PVC, changed S3 storage path, terminated all milvus nodes) and then freshly deployed it, now the collection creation operation worked as expected.
The issue repeats if I do redeployment.

Error logs from Streaming Node :

[WARN] [walmanager/manager_impl.go:88] ["remove wal failed"] [module=streamingnode] [component=wal-manager] [error="when recovering recovery storage: failed to recover from wal: context deadline exceeded"] [errorVerbose="when recovering recovery storage: failed to recover from wal: context deadline exceeded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/adaptor.(*openerAdaptorImpl).openRWWAL\n | \t/workspace/source/internal/streamingnode/server/wal/adaptor/opener.go:102\n | [...repeated from below...]\nWraps: (2) when recovering recovery storage\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/recovery.(*recoveryStorageImpl).recoverFromStream\n | \t/workspace/source/internal/streamingnode/server/wal/recovery/recovery_stream.go:55\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/recovery.RecoverRecoveryStorage\n | \t/workspace/source/internal/streamingnode/server/wal/recovery/recovery_storage_impl.go:45\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/adaptor.(*openerAdaptorImpl).openRWWAL\n | \t/workspace/source/internal/streamingnode/server/wal/adaptor/opener.go:98\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/adaptor.(*openerAdaptorImpl).Open\n | \t/workspace/source/internal/streamingnode/server/wal/adaptor/opener.go:71\n | github.com/milvus-io/milvus/internal/streamingnode/server/walmanager.(*walLifetime).doLifetimeChanged\n | \t/workspace/source/internal/streamingnode/server/walmanager/wal_lifetime.go:159\n | github.com/milvus-io/milvus/internal/streamingnode/server/walmanager.(*walLifetime).backgroundTask\n | \t/workspace/source/internal/streamingnode/server/walmanager/wal_lifetime.go:120\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (4) failed to recover from wal\nWraps: (5) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) context.deadlineExceededError"] [channel=by-dev-rootcoord-dml_10] [term=749]

[INFO] [walmanager/wal_lifetime.go:132] ["current state is not before expected state, do nothing"] [module=streamingnode] [component=wal-manager] [channel=by-dev-rootcoord-dml_10] [expected=(749,false)] [current=(749,false)] [2025/11/06 13:28:57.487 +00:00] [WARN] [walmanager/manager_impl.go:88] ["remove wal failed"] [module=streamingnode] [component=wal-manager] [error="code: STREAMING_CODE_IGNORED_OPERATION, cause: expired term 14260, cannot change expected state for remove"] [channel=by-dev-rootcoord-dml_0] [term=14260]

What might be causing this issue ?

[xiaofan-luan]

what do you mean by redeployment?

you can not create two milvus on same path.

My suggeston:

1. give the operations you did to reproduce this issue
2. offer full logs of your milvus by Could you please refer this doc 

[Siddeshvr]

Hi @xiaofan-luan ,
Thanks for you response.

sure I will export complete logs and share here.

Reproduce steps :
I have deployed Milvus in EKS cluster using helm, now if I update replicas or resources or any thing. I need to redeploy the service. After redeploying the same version (v2.6.5) with updated config to the same cluster multiple times, a subsequent collection creation operation hangs indefinitely (never completes).

This is the error/warn logs from Streaming Node :

[WARN] [walmanager/manager_impl.go:88] ["remove wal failed"] [module=streamingnode] [component=wal-manager] [error="when recovering recovery storage: failed to recover from wal: context deadline exceeded"] [errorVerbose="when recovering recovery storage: failed to recover from wal: context deadline exceeded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/adaptor.(*openerAdaptorImpl).openRWWAL\n | \t/workspace/source/internal/streamingnode/server/wal/adaptor/opener.go:102\n | [...repeated from below...]\nWraps: (2) when recovering recovery storage\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/recovery.(*recoveryStorageImpl).recoverFromStream\n | \t/workspace/source/internal/streamingnode/server/wal/recovery/recovery_stream.go:55\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/recovery.RecoverRecoveryStorage\n | \t/workspace/source/internal/streamingnode/server/wal/recovery/recovery_storage_impl.go:45\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/adaptor.(*openerAdaptorImpl).openRWWAL\n | \t/workspace/source/internal/streamingnode/server/wal/adaptor/opener.go:98\n | github.com/milvus-io/milvus/internal/streamingnode/server/wal/adaptor.(*openerAdaptorImpl).Open\n | \t/workspace/source/internal/streamingnode/server/wal/adaptor/opener.go:71\n | github.com/milvus-io/milvus/internal/streamingnode/server/walmanager.(*walLifetime).doLifetimeChanged\n | \t/workspace/source/internal/streamingnode/server/walmanager/wal_lifetime.go:159\n | github.com/milvus-io/milvus/internal/streamingnode/server/walmanager.(*walLifetime).backgroundTask\n | \t/workspace/source/internal/streamingnode/server/walmanager/wal_lifetime.go:120\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (4) failed to recover from wal\nWraps: (5) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) context.deadlineExceededError"] [channel=by-dev-rootcoord-dml_10] [term=749]

[INFO] [walmanager/wal_lifetime.go:132] ["current state is not before expected state, do nothing"] [module=streamingnode] [component=wal-manager] [channel=by-dev-rootcoord-dml_10] [expected=(749,false)] [current=(749,false)] [2025/11/06 13:28:57.487 +00:00] [WARN] [walmanager/manager_impl.go:88] ["remove wal failed"] [module=streamingnode] [component=wal-manager] [error="code: STREAMING_CODE_IGNORED_OPERATION, cause: expired term 14260, cannot change expected state for remove"] [channel=by-dev-rootcoord-dml_0] [term=14260]

[xiaofan-luan]

could you offer logs of your died service?

[Siddeshvr]

@xiaofan-luan ,
Sorry for the delay.
I was trying to reproduce this from last 2 days, now I got that.
Attached the exported logs here
logs.tar.gz

[xiaofan-luan]

/assign @czs007

please help on invetigating it

[tinswzy]

Thanks for the feedback, I’ll try it and see.

[tinswzy]

I retried more than twenty times by modifying values.yaml, and then running:

helm upgrade my-release zilliztech/milvus \
  --set image.all.tag=v2.6.6 \
  --set pulsarv3.enabled=false \
  --set woodpecker.enabled=true \
  --set streaming.enabled=true \
  --set indexNode.enabled=false \
  -f values.yaml

But I still couldn’t reproduce the issue.

I don’t think the restart itself is what caused the problem. We need to understand why the object disappeared here. Is it possible that your bucket has an S3 TTL (lifecycle expiration) policy configured?

What is the reproduction rate? Does this never occur in development or testing, and only appear in production?
What are the differences between these environments, especially regarding the S3-related configurations?

Also, if you are able to reproduce the issue, could you print the config diff before and after, as well as the exact commands you executed?
It would be very helpful if you could share the full sequence of operations and the configuration changes.
If this is a bug, using the same execution sequence and configuration changes may make it easier to reproduce the issue.

[Siddeshvr]

@tinswzy ,

1. S3 TTL is not configured
2. Reproduction rate is not frequent (once in 10 to 20 times)
3. It is first observed in production env, but after trying for some times it is also reproducible in testing environments also
4. production and testing environments S3 related configs are same
5. command used :
   helm upgrade --install --history-max 3 milvusdb-tst-01 milvus-1-milvus-develop-b293243.tgz --namespace tst-01 -f /home/jenkins/agent/workspace/Services/milvus_db/milvus_db-DEPLOY/configWorkspace/configuration/values/milvus/common-values.yaml -f /home/jenkins/agent/workspace/Services/milvus_db/milvus_db-DEPLOY/configWorkspace/configuration/values/milvus/tst-01/tst-01-milvusdb-values.yaml -f tempValues.yaml --kubeconfig kubeconfigs/kubeconfig-eks-iqa-us-01-sa -v=3 --atomic --timeout 1h

[tinswzy]

@yanliang567 Please help set up an environment to reproduce and confirm this issue.

[yanliang567]

@LoveEachDay could you please take a look first? We need an EKS on AWS to reporduce this issue

[xiaofan-luan]

@Siddeshvr

can you offer full logs for investigation?

[Siddeshvr]

@xiaofan-luan
Sure, below is the complete logs extracted during the issue
logs (1).tar.gz