UCC calling HCOLL in version 1.4.0

[bellenlau]

Hello,

we are running some OpenMPI/HPCX-MPI programs on a cluster using ucc/1.4.0. In this OpenMPI installation, UCC has the highest priority, but in some configurations an error related to some unsupported hcoll algorithm appears. To avoid this error we are considering disabling HCOLL communication and forcing UCC with:

export OMPI_MCA_coll_hcoll_enable=0
export OMPI_MCA_coll_ucc_enable=1

but I am not aware of potential consequences of this approach.

Indeed, from the call stack and the -mca coll_ucc_verbose output, we see that UCC is the main library used for collective communications, but -mca coll_hcoll_verbose shows that HCOLL is still used in the following communication:

../../../../../ompi/mca/coll/hcoll/coll_hcoll_rte.c:230 - send_nb() PML_ISEND: dest = 15: buf = 0x7fffe0f12060: size = 8: comm = 0x153c12d14e50

This seems to be related to some metadata, given the small buffer size? Further, by using nsys we noticed from the sampling points that ucc library calls hcoll in some bcast operations (hcoll_ml_progress, picture below). Is this behavior expected, and does it depend on the ucc version?

[Sergei-Lebedev]

Hello,

It's ok to disable hcoll or build mpi without it. hcoll is deprecated and we encourage users to switch to ucc instead.

Regarding the logs you see:

1. coll_hcoll_rte.c:230 - send_nb() PML_ISEND - this function is used as out of band communication to setup hcoll communicator. So yes, it's metadata.
2. ucc doesn't directly use hcoll and doesn't depend on hcoll. Due to MPI design, coll component is allowed to register a special progress function which will be called periodically (maybe even when the component is not used). This is needed to avoid some tricky deadlocks with blocking MPI collectives. And this is what you see in the backtrace in nsys - ucc calls opal_progress while doing broadcast which in turn calls hcoll_progress.