Invoker can panic if invocation was manually paused

[tillrohrmann]

@nikrooz ran into an invoker panic when doing the following:

• Have a handler with a combinator for 3 ctx.runs, 2 will always fail (not terminal) and the other takes very long time
• In one stage that it was backing-off, I tried to pause the invocation
• and then tried to resume it

[Some("rt:pp-8")]  PANIC!!!
panicked at crates/invoker-impl/src/invocation_state_machine.rs:404:9:
assertion failed: matches!(&self.invocation_state, AttemptState::WaitingRetry { .. })


thread 'rt:pp-8' (1478256) panicked at crates/invoker-impl/src/invocation_state_machine.rs:404:9:
assertion failed: matches!(&self.invocation_state, AttemptState::WaitingRetry { .. })
2025-11-25T15:58:43.850858Z ERROR tracing_panic
  A panic occurred
    panic.payload: "assertion failed: matches!(&self.invocation_state, AttemptState::WaitingRetry { .. })"
    panic.location: "crates/invoker-impl/src/invocation_state_machine.rs:404:9"
    panic.backtrace: disabled backtrace
    panic.note: "run with RUST_BACKTRACE=1 environment variable to display a backtrace"
on rt:pp-8
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It seems that we are violating an invariant in the invoker where we expect it to be in the AttemptState::WaitingRetry state.

[tillrohrmann]

The relevant part that is causing the panic (

restate/crates/invoker-impl/src/invocation_state_machine.rs

Lines 417 to 420 in 8359b9a

	debug_assert!(matches!(
	&self.invocation_state,
	AttemptState::WaitingRetry { .. }
	));

) has been added via #3881.

[tillrohrmann]

@slinkydeveloper why did we introduce the debug_assert! in the notify_retry_timer_fired method?

It seems that the following can happen with the manual pause/resume feature: There is a retrying invocation with a retry timer. This invocation gets manually paused. It looks as if we won't delete the retry timer in this case. Now if the invocation is resumed, it will create a new InvocationStateMachine which can be in AttemptState::InFlight. If now the old retry timer fires, it will see the invocation in a wrong state and then panics.

Did you consider fencing off old retry timers or deleting them when pausing an invocation? If we don't do this, then old retry timers can interfere with newly resumed/started invocations.

[slinkydeveloper]

Did you consider fencing off old retry timers or deleting them when pausing an invocation? If we don't do this, then old retry timers can interfere with newly resumed/started invocations.

Indeed, or simpler just ignore the retry timer if the invocation is already running?