Add custom CI timeout to detect stuck jobs earlier

[kitsiosk]

Change Summary

Added custom timeout for Build job of the CI workflow based on historical data that could lead to resource savings and faster CI for the project.

More details

Over the last 568 successful runs, the Build job has a maximum runtime of 37 minutes (mean=2, std=2) across all matrix combinations.

However, there are failed runs that fail after reaching the threshold of 6 hours that GitHub imposes. In other words, these jobs seem to get stuck, possibly for external or random reasons.

One such example is this job run, that failed after 6 hours, while the full list of timed-out jobs is available below. With the proposed changes, a total of 11 hours would have been saved over the last six months retrospectively, clearing the queue for other workflows and speeding up the CI of the project, while also saving resources in general 🌱.

The idea is to set a timeout to stop jobs that run much longer than their historical maximum, because such jobs are probably stuck and will simply fail after GitHub's timeout of 6 hours.

Our PR proposes to set the timeout to max + 3*std = 43 minutes where max and std (standard deviation) are derived from the history of 568 successful runs. This will provide sufficient margin if the workflow gets naturally slower in the future, but if you would prefer lower/higher threshold we would be happy to do it.

We propose the same for the Update port sources job of the Update lockfiles workflow, that also has experienced timeouts.

Note that the timeout applies to all the matrix jobs, and not to their sum, overriding the default 6-hour timeout of github.

Click here to see all the recently timed-out runs.

05-Jan-2025 => timed-out run
05-Jan-2025 => timed-out run
18-Apr-2025 => timed-out run

Context

Hi,

We are a team of researchers from University of Zurich and we are currently working on energy optimizations in GitHub Actions workflows.

Thanks for your time on this and for your contribution to open-source software in general.

Feel free to let us know (here or in the email below) if you have any questions, and thanks for putting in the time to read this.

Best regards,
Konstantinos Kitsios
[email protected]