CMS decoding issue breaks task queue

[mheijkoop]

Had an issue with adding a new parent to my CA yesterday (Error: RFC 6492 Issue: Could not decode CMS: missing further values (at position 0)) which we haven't solved yet but the bigger issue it causes stuck tasks in my queue even without the new parent which in turn caused my ROAs not to be updated.

Monitoring caught it before it was a problem but I don't think a broken task should stop processing of all tasks, but krill should just move to the next available task rather than give up on processing tasks at all.

My log was flooded with instances of:
2025-06-14 10:46:00 [ERROR] Could not get pending task from queue: unknown key

I stopped krill, deleted everything in /var/lib/krill/data/tasks/pending and started krill again. Everything seems to have recovered properly.

[partim]

Did you perchance delete the parent again? That would at least explain the stuck task.

Obviously, you are correct that this shouldn’t break Krill.

[mheijkoop]

No, the CMS error causes the parent to not be created in the first place (or it gets rolled back internally - haven't actually dug in that deep). So maybe the task gets created before the new parent is committed to the db?

[partim]

Curiously, it looks to me that the error message suggests that Krill couldn’t actually read the file – unfortunately it discards the underlying error, so we don’t really know why.

[yoitszigi]

Having the same issue here and it spams the logs every second with same error. It blocks the ROAs from being renewed

Jun 26 23:52:52 rpki-01 krill[973419]: 2025-06-26 23:52:52 [WARN] Rescheduling running tasks at startup, note that multi-node Krill servers are not yet supported.
Jun 26 23:52:52 rpki-01 krill[973419]: 2025-06-26 23:52:52 [WARN]   - rescheduling: 1750979945510-refresh_bgp_announcements_info
Jun 26 23:52:52 rpki-01 krill[973419]: 2025-06-26 23:52:52 [WARN]   - rescheduling: 1750715158263-all_cas_republish_if_needed
Jun 26 23:52:52 rpki-01 krill[973419]: 2025-06-26 23:52:52 [ERROR] Could not get pending task from queue: unknown key
Jun 26 23:52:52 rpki-01 krill[973419]: 2025-06-26 23:52:52 [ERROR] Could not get pending task from queue: unknown key
Jun 26 23:52:53 rpki-01 krill[973419]: 2025-06-26 23:52:53 [ERROR] Could not get pending task from queue: unknown key
Jun 26 23:52:53 rpki-01 krill[973419]: 2025-06-26 23:52:53 [ERROR] Could not get pending task from queue: unknown key
Jun 26 23:52:54 rpki-01 krill[973419]: 2025-06-26 23:52:54 [ERROR] Could not get pending task from queue: unknown key
Jun 26 23:52:54 rpki-01 krill[973419]: 2025-06-26 23:52:54 [ERROR] Could not get pending task from queue: unknown key
Jun 26 23:52:55 rpki-01 krill[973419]: 2025-06-26 23:52:55 [ERROR] Could not get pending task from queue: unknown key

Using Krill 0.14.6

[partim]

Could you perchance send the content of the data/tasks directory to rpki-team@nlnetlabs.nl? That would help with finding out what goes wrong exactly.

After that, you can work around the issue by deleting the entire data/tasks directory and restart Krill. This will force it to reschedule the tasks.