Does the repo pick the weights that perform best in val dataset to evaluate in test dataset?

[pierowu]

Thank you for your solid work.
Does the repo implement the function that pick the model weights that perform best in val dataset to evaluate in test dataset?
From the code below, it seems that the repo directly choose the best results in test dataset as the final results?

Elevater_Toolkit_IC/vision_benchmark/evaluation/full_model_finetune.py

Lines 267 to 277 in 00d0af7

	train_one(train_dataloader, model, criterion, optimizer, epoch, config)
	# evaluate on validation set
	acc1, logits = validate(test_dataloader, model, criterion, epoch, config, return_logits=True)
	# remember best acc@1 and save checkpoint
	if acc1 > best_acc1:
	model_info['best_logits'] = logits
	best_acc1 = max(acc1, best_acc1)
	logging.info(f'=> Learning rate {config.TRAIN.LR}, L2 lambda {config.TRAIN.WD}: Best score: Acc@1 {best_acc1:.3f}')

[ChunyuanLI]

Thanks for the careful study.

When selecting the hyper-parameters, we sweep and validate on validation dataset:

Elevater_Toolkit_IC/vision_benchmark/evaluation/full_model_finetune.py

Line 173 in 00d0af7

best_score_ = train_task(train_dataloader, val_dataloader, config, sweep_run=True)

Once the best hyper-parameter is selected, and report the best results on the test set, the line of code indicates that the best weights /epoch are selected by looking at the best numbers on test set.

For the real applications where we are not allowed to leverage the labels in the test set to determine the training epochs, we recommend to fix the numbers of epochs in the hyper-parameter search stage.

[pierowu]

Thanks for the careful study.

When selecting the hyper-parameters, we sweep and validate on validation dataset:

Elevater_Toolkit_IC/vision_benchmark/evaluation/full_model_finetune.py

Line 173 in 00d0af7

best_score_ = train_task(train_dataloader, val_dataloader, config, sweep_run=True)

Once the best hyper-parameter is selected, and report the best results on the test set, the line of code indicates that the best weights /epoch are selected by looking at the best numbers on test set.

For the real applications where we are not allowed to leverage the labels in the test set to determine the training epochs, we recommend to fix the numbers of epochs in the hyper-parameter search stage.

Thank you for your reply. Would this way cause overfitting in the test set? For example, we can design a model with high variance which can perform well in serveral epochs in the test set. However, this model will deteriorate in other epochs.
When I want to compare with other methods in elevater benchmark, how can I make the comparison fair?