The Channel has been closed exception from HeartBeatReadCallback

[patriktiain]

Describe the bug

RabbitMQ client 7.2.0
RabbitMQ server 3.11

During one of our long running load tests, our application crashed during the night while we were rebuilding our SQL Server indexes. We have noticed during this rebuild, our application and RabbitMQ are performing worse regarding time to publish. Probably because of disc access on the server is reduced because of the index rebuild.

The application crashed, according to the Event viewer and the stack trace, because of a channel already closed exception from, what I can tell, the heartbeat read callback.

StackTrace:
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Threading.Channels.ChannelClosedException: The channel has been closed.
   at System.Threading.Channels.ChannelWriter`1.Complete(Exception error)
   at RabbitMQ.Client.Impl.Channel.OnSessionShutdownAsync(Object sender, ShutdownEventArgs reason)
   at RabbitMQ.Client.Impl.AsyncEventingWrapper`1.InternalInvoke(Delegate[] handlers, Object sender, TEvent event)
   at RabbitMQ.Client.Impl.AsyncEventingWrapper`1.InternalInvoke(Delegate[] handlers, Object sender, TEvent event)
   at RabbitMQ.Client.Impl.Channel.FinishCloseAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.FinishCloseAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.HeartbeatReadTimerCallback(Object state)
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

To me it looks like an unhandled exception within the client library as there is no application reference in the stack trace.

I have observed this twice and it happened both times during the index rebuild.

Reproduction steps

1.High concurrent load
2.Increase contention on server by rebuilding SQL server index
3. Application crashed
...

Expected behavior

Channel closed exception should no cause application to crash.

Additional context

No response

[patriktiain]

I tracked the reason why this happen is due to a race condition between these two methods.

protected override Task InternalShutdownAsync()
{
    _writer.Complete();
    return _worker;
}

If InternalShutdown has already been called before, this will fail on .Complete() as it has already been closed.

Is there any reason for this not being TryComplete() instead?

[michaelklishin]

"Handling" shutdown time exceptions by ignoring them is very common in every client.

So adopting TryComplete sounds reasonable. Please submit a PR.

That said, the idea of colocating RabbitMQ nodes with a database that will fight for disk, network I/O and CPU resources is not something I would consider a reasonable solution. It's wrong and not something anyone should do to any pair of data services (not just RabbitMQ).