Training hangs after a few steps

[melanibe]

Encountering similar issues to #395 with the latest version of ART (0.5.3), vllm 0.10.0. Running within local script (outside of collab / notebook).

Training runs fine for 2 batches and then suddenly hangs in the gather step of the next batch
Gather step 2: 83%|████████▎ | 5/6

Without any error message nor anything, just hangs.

[melanibe]

In my case, this is actually due to some vLLM error (exceeded context length). Unfortunately instead of failing and throwing an error the code just continues to run (and burn gpu hours) without doing anything without failing. The only way to see if is to dissect the vllm.log file.
This should not be happening, it should throw an error to the main script.