The problem was that we failed to initialize node N2. It failed with a terminal error when trying to write to the metadata store running on N1. ``` node-2 | Shutting down: task 5 failed with: Failed initializing the node node-2 | node-2 | Caused by: node-2 | 0: failed writing to metadata store: other error: [terminal] [unix:/home/runner/work/_temp/lcr/raft_metadata_cluster_chaos_test/node-1/fabric.sock] Internal error: h2 protocol error: http2 error node-2 | 1: other error: [terminal] [unix:/home/runner/work/_temp/lcr/raft_metadata_cluster_chaos_test/node-1/fabric.sock] Internal error: h2 protocol error: http2 error node-2 | node-2 | Stack backtrace: node-2 | 0: anyhow::error:: for anyhow::Error>::from node-2 | 1: >::into node-2 | 2: core::ops::function::FnOnce::call_once node-2 | 3: core::result::Result::map_err node-2 | 4: restate_node::init::NodeInit::join_cluster::{{closure}} node-2 | 5: as core::future::future::Future>::poll node-2 | 6: tokio_util::sync::cancellation_token::CancellationToken::run_until_cancelled::{{closure}} node-2 | 7: restate_node::init::NodeInit::init::{{closure}} node-2 | 8: as core::future::future::Future>::poll node-2 | 2025-11-20T12:04:34.725165Z INFO restate_core::network::net_util node-2 | 9: restate_node::Node::start::{{closure}} node-2 | Stopped listening node-2 | 10: as core::future::future::Future>::poll node-2 | on rs:worker-0 node-2 | 11: as core::future::future::Future>::poll::{{closure}} node-2 | in restate_core::network::net_util::server node-2 | 12: as core::ops::function::FnOnce<()>>::call_once node-2 | server_name: message-fabric-server node-2 | 13: std::panicking::catch_unwind::do_call node-2 | uds.path: "/home/runner/work/_temp/lcr/raft_metadata_cluster_chaos_test/node-2/fabric.sock" node-2 | 14: __rust_try node-2 | 15: std::panic::catch_unwind node-2 | 16: as core::future::future::Future>::poll node-2 | 17: as core::future::future::Future>::poll::{{closure}} node-2 | 18: tokio::task::task_local::LocalKey::scope_inner node-2 | 19: as core::future::future::Future>::poll node-2 | 20: as core::future::future::Future>::poll::{{closure}} node-2 | 21: tokio::task::task_local::LocalKey::scope_inner node-2 | 22: as core::future::future::Future>::poll node-2 | 23: as core::future::future::Future>::poll::{{closure}} node-2 | 24: tokio::task::task_local::LocalKey::scope_inner node-2 | 25: as core::future::future::Future>::poll node-2 | 26: restate_core::task_center::wrapper::{{closure}} node-2 | 27: as core::future::future::Future>::poll node-2 | 28: as core::future::future::Future>::poll node-2 | 29: as core::future::future::Future>::poll node-2 | 30: tokio::runtime::task::core::Core::poll::{{closure}} node-2 | 31: tokio::runtime::task::core::Core::poll node-2 | 32: tokio::runtime::task::harness::poll_future::{{closure}} node-2 | 33: as core::ops::function::FnOnce<()>>::call_once node-2 | 34: std::panicking::catch_unwind::do_call node-2 | 35: __rust_try node-2 | 36: std::panic::catch_unwind node-2 | 37: tokio::runtime::task::harness::poll_future node-2 | 38: tokio::runtime::task::harness::Harness::poll_inner node-2 | 39: tokio::runtime::task::harness::Harness::poll node-2 | 40: tokio::runtime::task::raw::poll node-2 | 41: tokio::runtime::task::raw::RawTask::poll node-2 | 42: tokio::runtime::task::LocalNotified::run node-2 | 43: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}} node-2 | 44: tokio::runtime::scheduler::multi_thread::worker::Context::run_task node-2 | 45: tokio::runtime::scheduler::multi_thread::worker::Context::run node-2 | 46: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}} node-2 | 47: tokio::runtime::context::scoped::Scoped::set node-2 | 48: tokio::runtime::context::set_scheduler::{{closure}} node-2 | 49: std::thread::local::LocalKey::try_with node-2 | 50: std::thread::local::LocalKey::with node-2 | 51: tokio::runtime::context::set_scheduler node-2 | 52: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}} node-2 | 53: tokio::runtime::context::runtime::enter_runtime node-2 | 54: tokio::runtime::scheduler::multi_thread::worker::run node-2 | 55: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}} node-2 | 56: as core::future::future::Future>::poll node-2 | 57: as core::future::future::Future>::poll node-2 | 58: tokio::runtime::task::core::Core::poll::{{closure}} node-2 | 59: tokio::runtime::task::core::Core::poll node-2 | 60: tokio::runtime::task::harness::poll_future::{{closure}} node-2 | 61: as core::ops::function::FnOnce<()>>::call_once node-2 | 62: std::panicking::catch_unwind::do_call node-2 | 63: __rust_try node-2 | 64: std::panic::catch_unwind node-2 | 65: tokio::runtime::task::harness::poll_future node-2 | 66: tokio::runtime::task::harness::Harness::poll_inner node-2 | 67: tokio::runtime::task::harness::Harness::poll node-2 | 68: tokio::runtime::task::raw::poll node-2 | 69: tokio::runtime::task::raw::RawTask::poll node-2 | 70: tokio::runtime::task::UnownedTask::run node-2 | 71: tokio::runtime::blocking::pool::Task::run node-2 | 72: tokio::runtime::blocking::pool::Inner::run node-2 | 73: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}} node-2 | 74: std::sys::backtrace::__rust_begin_short_backtrace node-2 | 75: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} node-2 | 76: as core::ops::function::FnOnce<()>>::call_once node-2 | 77: std::panicking::catch_unwind::do_call node-2 | 78: __rust_try node-2 | 79: std::thread::Builder::spawn_unchecked_::{{closure}} node-2 | 80: core::ops::function::FnOnce::call_once{{vtable.shim}} node-2 | 81: std::sys::thread::unix::Thread::new::thread_start node-2 | 82: node-2 | 83: node-2 | kind: SystemBoot node-2 | name: "init" ```

raft_metadata_cluster_chaos_test is unstable #4049

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions