nyuv2 dataset ground truth problems

[puren]

Hello,

I try to use the pre-processed dataset that you prepared here to train a segmentation model.
However, we found out some samples with erroneous ground truth (wrong labels, mismatched labels between similar scenes).
Such data causes problems while training to converge. Did you have similar issues? If so, how did you clean the dataset?

[lorenmt]

Hello, this is the official version of NYUv2 dataset. Could you give an example of mismatched labels?

[puren]

Thanks a lot for your quick answer. We collected some of our observations in this pdf file.
NUYv2 Dataset.pdf

[lorenmt]

Thanks for providing these examples. Yes, I have to say that the label ambiguity exists in all segmentation datasets and is biased towards the labellers. I somehow studied this problem in my other project, and predicted this label similarity map in SUN RGBD dataset.

So unfortunately, there is no better way to avoid this, as the concept of "segmentation" is created and defined by humans, not a well-defined metric like depth estimation.

One potential solution to mitigate this is to do open-vocabulary segmentation, then we hope these ambiguous pairs of labels would have similar semantic embeddings: https://jerryxu.net/ODISE/

[puren]

Thanks a lot for the detailed answer. The graph is a very helpful visualization. I will check the referenced work for open-vocab segmentation as well.
One a bit out of scope question: Did you have any other experience with other datasets such as cityscape or ade20k dataset? Is there any chance that we get some more consistent labeling?

[lorenmt]

Cityscapes is very consistent, the object labels are very different from each other. ADE20K is hard to say... because it has 150 labels...

[puren]

Right. Thank you very much for all your answers.