Compute worker - Avoid duplicating submission files #1874
Description
Original problem:
Solution:
But then it broke results submission competitions. We are still using the previous version of the compute workers to avoid the problem.
TODO
- Fix the current
developbranch (Put back copy of submission files into prediction output in compute worker #1923) - Perform the copy during scoring (instead of during ingestion)
- Improve the tests to catch problems with all types of submissions (code, results, etc.)
The current idea to fix this
Instead of doing the copy during ingestion program, we do it during scoring program. This way, the files are not saved in the storage as part of the ingestion output, because they are added to it after, only temporarily. No need for updating existing competitions. We may need to make changes in order to access the submission during scoring program execution.