RF-DETR Nano huge differnce in training quality when trained locally vs trained on roboflow.com

[MichalTurek]

Search before asking

• I have searched the RF-DETR issues and found no similar bug report.

Bug

I am working on my Master's thesis related to object detection. One goal is to compare how different architectures perform when trained on limited hardware (8GB VRAM). The results on roboflow.com using DETR-NANO look impressive:

However when running training script:

I notice that when trained on roboflow.com the model immidietly starts from ~0.5 MAP and achives far better result:

When trained locally:

Environment

Python

Version: 3.10.0 (tags/v3.10.0:b494f59, Oct 4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)]
Executable: C:\Program Files\Python310\python.exe
Implementation: CPython
Prefix: C:\Program Files\Python310

System: Windows
Release: 10
Version: 10.0.26200
Machine: AMD64
Processor: Intel64 Family 6 Model 141 Stepping 1, GenuineIntel
Platform string: Windows-10-10.0.26200-SP0

RF-DETR

rfdetr module: <module 'rfdetr' from 'C:\Users\micha\AppData\Roaming\Python\Python310\site-packages\rfdetr\init.py'>
rfdetr (importlib.metadata): 1.3.0
rfdetr module file: C:\Users\micha\AppData\Roaming\Python\Python310\site-packages\rfdetr_init_.py

PyTorch / CUDA

torch.version: 2.9.0+cu130
torch.version.cuda: 13.0
torch.backends.cudnn.version(): 91200
CUDA available (torch.cuda.is_available()): True
Device count: 1

• Device 0: NVIDIA GeForce RTX 3070 Laptop GPU, capability (8, 6)

Key Python packages

rfdetr==1.3.0
torch==2.9.0+cu130
torchvision==0.24.0+cu130
torchaudio: not installed or not found
transformers==4.57.3
timm==1.0.22

pip freeze

absl-py==2.3.1
accelerate==1.12.0
annotated-types==0.7.0
anywidget==0.9.21
asttokens==3.0.1
bbox_visualizer==0.2.2
certifi==2025.10.5
charset-normalizer==3.4.4
colorama==0.4.6
comm==0.2.3
contourpy==1.3.2
cycler==0.12.1
Cython==3.2.2
decorator==5.2.1
defusedxml==0.7.1
einops==0.8.1
exceptiongroup==1.3.1
executing==2.2.1
fairscale==0.4.13
filelock==3.19.1
filetype==1.2.0
filterpy==1.4.5
fonttools==4.60.1
fsspec==2025.9.0
ftfy==6.3.1
gitdb==4.0.12
GitPython==3.1.45
grpcio==1.76.0
huggingface-hub==0.36.0
idna==3.7
ipython==8.37.0
ipywidgets==8.1.8
jedi==0.19.2
Jinja2==3.1.6
joblib==1.5.2
jupyter_bbox_widget==0.6.0
jupyterlab_widgets==3.0.16
kiwisolver==1.4.9
lap==0.5.12
Markdown==3.10
markdown-it-py==4.0.0
MarkupSafe==2.1.5
matplotlib==3.10.7
matplotlib-inline==0.2.1
mdurl==0.1.2
mpmath==1.3.0
networkx==3.3
ninja==1.13.0
norfair==2.3.0
numpy==2.2.6
open_clip_torch==3.2.0
opencv-python==4.12.0.88
opencv-python-headless==4.10.0.84
packaging==25.0
pandas==2.3.3
parso==0.8.5
peft==0.18.0
pi_heif==1.1.1
pillow==11.3.0
pillow-avif-plugin==1.5.2
polars==1.35.1
polars-runtime-32==1.35.1
polygraphy==0.49.26
prompt_toolkit==3.0.52
protobuf==6.33.1
psutil==7.1.3
psygnal==0.15.0
pure_eval==0.2.3
pycocotools==2.0.10
pydantic==2.12.5
pydantic_core==2.41.5
Pygments==2.19.2
pylabel==0.1.55
pyparsing==3.2.5
python-dateutil==2.9.0.post0
python-dotenv==1.2.1
pytz==2025.2
PyYAML==6.0.3
regex==2025.11.3
requests==2.32.5
requests-toolbelt==1.0.0
rf100vl==1.1.0
rfdetr==1.3.0
rich==14.2.0
roboflow==1.2.11
safetensors==0.7.0
scikit-learn==1.7.2
scipy==1.15.3
seaborn==0.13.2
six==1.17.0
smmap==5.0.2
stack-data==0.6.3
supervision==0.27.0
sympy==1.14.0
tensorboard==2.20.0
tensorboard-data-server==0.7.2
threadpoolctl==3.6.0
timm==1.0.22
tokenizers==0.22.1
torch==2.9.0+cu130
torchvision==0.24.0+cu130
tqdm==4.67.1
traitlets==5.14.3
transformers==4.57.3
typing-inspection==0.4.2
typing_extensions==4.15.0
tzdata==2025.2
ultralytics==8.3.223
ultralytics-thop==2.0.18
urllib3==2.5.0
wcwidth==0.2.14
Werkzeug==3.1.4
widgetsnbextension==4.0.15

Minimal Reproducible Example

`
def main():
    model = RFDETRNano()
    model.maybe_download_pretrain_weights()
    model.train(
        dataset_dir=DATASET_DIR,
        output_dir=OUTPUT_DIR,
        epochs=300,
        batch_size=16,
        grad_accum_steps=1,
        resolution=384,
        device="cuda",

        dataloader_num_workers=4,   
        # --- EARLY STOPPING ---
        early_stopping=True,
        early_stopping_patience=20,      
        early_stopping_min_delta=0.0,   
)


if __name__ == "__main__":
    import torch.multiprocessing as mp
    mp.freeze_support()  
    main()
` 

Additional

No response

Are you willing to submit a PR?

• Yes, I'd like to help by submitting a PR!

[isaacrob-roboflow]

Weird. The major difference is we use the o365 checkpoint on platform but this repo has coco. O365 should give slightly better results. What you're showing looks like a serious learning instability. How does your data look? Also, how did you generate those plots? Ideally you should be using the EMA weights

[MichalTurek]

hello, thanks for quick response! here is the dataset ive prepared:
. I generated the weights by opening this log.txt file that is being generated during the training session:

log.txt

[MichalTurek]

The data is a rader difficult one, in this particular training its a ball detection on football field. I know it is a small object and it maybe the reason but to be fair if training on roboflow site works properly why would it not work here ?

[MichalTurek]

Weird. The major difference is we use the o365 checkpoint on platform but this repo has coco. O365 should give slightly better results. What you're showing looks like a serious learning instability. How does your data look? Also, how did you generate those plots? Ideally you should be using the EMA weights
if that is the only difference, is there a way to use other checkpoint to test it out?

[isaacrob-roboflow]

Try plotting the EMA results

[MichalTurek]

the result is almost the same, maybe alittle smoother

[isaacrob-roboflow]

Super super interesting. Does it happen if you run it again?

[MichalTurek]

Started training again; here is a console log from the start:

`UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\Context.cpp:85.)
Using a different number of positional encodings than DINOv2, which means we're not loading DINOv2 backbone weights. This is not a problem if finetuning a pretrained RF-DETR model.
Using patch size 16 instead of 14, which means we're not loading DINOv2 backbone weights. This is not a problem if finetuning a pretrained RF-DETR model.
Loading pretrain weights
TensorBoard logging initialized. To monitor logs, use 'tensorboard --logdir C:\Users\micha\train\rfdetr_out2' and open http://localhost:6006/ in browser.
Not using distributed mode
fatal: not a git repository (or any of the parent directories): .git
git:
  sha: N/A, status: clean, branch: N/A

Namespace(num_classes=2, grad_accum_steps=1, amp=True, lr=0.0001, lr_encoder=0.00015, batch_size=16, weight_decay=0.0001, epochs=300, lr_drop=100, clip_max_norm=0.1, lr_vit_layer_decay=0.8, lr_component_decay=0.7, do_benchmark=False, dropout=0, drop_path=0.0, drop_mode='standard', drop_schedule='constant', cutoff_epoch=0, pretrained_encoder=None, pretrain_weights='rf-detr-nano.pth', pretrain_exclude_keys=None, pretrain_keys_modify_to_load=None, pretrained_distiller=None, encoder='dinov2_windowed_small', vit_encoder_num_layers=12, window_block_indexes=None, position_embedding='sine', out_feature_indexes=[3, 6, 9, 12], freeze_encoder=False, layer_norm=True, rms_norm=False, backbone_lora=False, force_no_pretrain=False, dec_layers=2, dim_feedforward=2048, hidden_dim=256, sa_nheads=8, ca_nheads=16, num_queries=300, group_detr=13, two_stage=True, projector_scale=['P4'], lite_refpoint_refine=True, num_select=300, dec_n_points=2, decoder_norm='LN', bbox_reparam=True, freeze_batch_norm=False, set_cost_class=2, set_cost_bbox=5, set_cost_giou=2, cls_loss_coef=1.0, bbox_loss_coef=5, giou_loss_coef=2, focal_alpha=0.25, aux_loss=True, sum_group_losses=False, use_varifocal_loss=False, use_position_supervised_loss=False, ia_bce_loss=True, dataset_file='roboflow', coco_path=None, dataset_dir='C:\\Users\\micha\\Downloads\\ball-detection.v3i.coco', square_resize_div_64=True, output_dir='C:\\Users\\micha\\train\\rfdetr_out2', dont_save_weights=False, checkpoint_interval=10, seed=42, resume='', start_epoch=0, eval=False, use_ema=True, ema_decay=0.993, ema_tau=100, num_workers=2, device='cuda', world_size=1, dist_url='env://', sync_bn=True, fp16_eval=False, encoder_only=False, backbone_only=False, resolution=384, use_cls_token=False, multi_scale=True, expanded_scales=True, do_random_resize_via_padding=False, warmup_epochs=0.0, lr_scheduler='step', lr_min_factor=0.0, early_stopping=True, early_stopping_patience=20, early_stopping_min_delta=0.0, early_stopping_use_ema=False, gradient_checkpointing=False, patch_size=16, num_windows=2, positional_encoding_size=24, mask_downsample_ratio=4, tensorboard=True, wandb=False, project=None, run=None, class_names=['ball'], run_test=True, segmentation_head=False, distributed=False)
number of params: 30147076
[544]
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
[544]
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
[544]
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
Get benchmark
Start training
Grad accum steps:  1
Total batch size:  16
LENGTH OF DATA LOADER: 145
UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\Context.cpp:85.)
UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\Context.cpp:85.)
UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4319.)
UserWarning: Using a non-tuple sequence for multidimensional indexing is deprecated and will be changed in pytorch 2.9; use x[tuple(seq)] instead of x[seq]. In pytorch 2.9 this will be interpreted as tensor index, x[torch.tensor(seq)], which will result either in an error or a different result (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\torch\csrc\autograd\python_variable_indexing.cpp:351.)
UserWarning: Using a non-tuple sequence for multidimensional indexing is deprecated and will be changed in pytorch 2.9; use x[tuple(seq)] instead of x[seq]. In pytorch 2.9 this will be interpreted as tensor index, x[torch.tensor(seq)], which will result either in an error or a different result (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\torch\csrc\autograd\python_variable_indexing.cpp:351.)
UserWarning: Using a non-tuple sequence for multidimensional indexing is deprecated and will be changed in pytorch 2.9; use x[tuple(seq)] instead of x[seq]. In pytorch 2.9 this will be interpreted as tensor index, x[torch.tensor(seq)], which will result either in an error or a different result (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\torch\csrc\autograd\python_variable_indexing.cpp:351.)
[BATCH 0] time: 3.5556 sec
Epoch: [0]  [  0/145]  eta: 0:37:51  lr: 0.000100  class_error: 0.00  loss: 47.4849 (47.4849)  loss_ce: 13.3135 (13.3135)  loss_bbox: 0.7085 (0.7085)  loss_giou: 1.8560 (1.8560)  loss_ce_0: 12.7621 (12.7621)  loss_bbox_0: 0.7135 (0.7135)  loss_giou_0: 1.8886 (1.8886)  loss_ce_enc: 13.6422 (13.6422)  loss_bbox_enc: 0.7329 (0.7329)  loss_giou_enc: 1.8675 (1.8675)  loss_ce_unscaled: 13.3135 (13.3135)  class_error_unscaled: 0.0000 (0.0000)  loss_bbox_unscaled: 0.1417 (0.1417)  loss_giou_unscaled: 0.9280 (0.9280)  cardinality_error_unscaled: 0.8125 (0.8125)  loss_ce_0_unscaled: 12.7621 (12.7621)  loss_bbox_0_unscaled: 0.1427 (0.1427)  loss_giou_0_unscaled: 0.9443 (0.9443)  cardinality_error_0_unscaled: 0.8125 (0.8125)  loss_ce_enc_unscaled: 13.6422 (13.6422)  loss_bbox_enc_unscaled: 0.1466 (0.1466)  loss_giou_enc_unscaled: 0.9338 (0.9338)  cardinality_error_enc_unscaled: 0.8125 (0.8125)  time: 15.6624  data: 12.1068  max mem: 6426`

[MichalTurek]

it is essentialy the same story with rerun...

[isaacrob-roboflow]

Wild. I don't have a great explanation, hard to make any guesses without seeing data.

Do you mind running more experiments? I'd try bs 2 and grad accum 8, and separately lr 1e-5

It is interesting that the peak score looks similar

[MichalTurek]

Namespace(num_classes=2, grad_accum_steps=2, amp=True, lr=1e-05, lr_encoder=0.00015, batch_size=8, weight_decay=0.0001, epochs=300, lr_drop=100, clip_max_norm=0.1, lr_vit_layer_decay=0.8, lr_component_decay=0.7, do_benchmark=False, dropout=0, drop_path=0.0, drop_mode='standard', drop_schedule='constant', cutoff_epoch=0, pretrained_encoder=None, pretrain_weights='rf-detr-nano.pth', pretrain_exclude_keys=None, pretrain_keys_modify_to_load=None, pretrained_distiller=None, encoder='dinov2_windowed_small', vit_encoder_num_layers=12, window_block_indexes=None, position_embedding='sine', out_feature_indexes=[3, 6, 9, 12], freeze_encoder=False, layer_norm=True, rms_norm=False, backbone_lora=False, force_no_pretrain=False, dec_layers=2, dim_feedforward=2048, hidden_dim=256, sa_nheads=8, ca_nheads=16, num_queries=300, group_detr=13, two_stage=True, projector_scale=['P4'], lite_refpoint_refine=True, num_select=300, dec_n_points=2, decoder_norm='LN', bbox_reparam=True, freeze_batch_norm=False, set_cost_class=2, set_cost_bbox=5, set_cost_giou=2, cls_loss_coef=1.0, bbox_loss_coef=5, giou_loss_coef=2, focal_alpha=0.25, aux_loss=True, sum_group_losses=False, use_varifocal_loss=False, use_position_supervised_loss=False, ia_bce_loss=True, dataset_file='roboflow', coco_path=None, dataset_dir='C:\\Users\\micha\\Downloads\\ball-detection.v3i.coco', square_resize_div_64=True, output_dir='C:\\Users\\micha\\train\\rfdetr_out2', dont_save_weights=False, checkpoint_interval=10, seed=42, resume='', start_epoch=0, eval=False, use_ema=True, ema_decay=0.993, ema_tau=100, num_workers=2, device='cuda', world_size=1, dist_url='env://', sync_bn=True, fp16_eval=False, encoder_only=False, backbone_only=False, resolution=384, use_cls_token=False, multi_scale=True, expanded_scales=True, do_random_resize_via_padding=False, warmup_epochs=0, lr_scheduler='step', lr_min_factor=0.0, early_stopping=True, early_stopping_patience=20, early_stopping_min_delta=0.0, early_stopping_use_ema=False, gradient_checkpointing=False, patch_size=16, num_windows=2, positional_encoding_size=24, tensorboard=True, wandb=False, project=None, run=None, class_names=['ball'], run_test=True, distributed=False) here is a paramter list, maybe there is something wrong here?

[MichalTurek]

you can take a look at dataset here: https://app.roboflow.com/yolov8-rb1a5/ball-detection-msnhg/browse?queryText=&pageSize=50&startingIndex=0&browseQuery=true. the dataset is soccer ball so it is a tiny element. However I managed to train yolov11 with it locally without a problem and also RF-DETR on server, so seems that dataset itself is not a problem.

In my opinion it looks like that the version of pretrained weight might simply work a lot better (o365 on roboflow server) vs the one shared for local training

[MichalTurek]

rerun with batch size=8 grad_accum=2 lr=1e-5 (on version 1.2.0) so there is no regression for sure

[MichalTurek]

One good news! this weird behaviour has been stopped by installing fresh python 3.13 :