Simulation for validation

[fugokidi]

Thank you for open-sourcing the VLA. I am new to robotic models and I do not have a physical robot. Is there any simulation platform to validate a VLA model? May I know how to evaluate a VLA model in the digital domain? Thank you again.

[mbreuss]

You can test models using the simpler env: https://github.com/simpler-env/SimplerEnv
The benchmark contains selected tasks from the Google Fractal dataset and the Berkley Bridge dataset that are part of OXE-dataset used for pretraining this policy.

[fugokidi]

@mbreuss Thanks a lot. I will check it out right away.

[siddk]

Thanks @mbreuss for getting here before we did! We don't have "fully vetted" results for the SimplerEnv environments yet - if either you or @fugokidi would be interested in making a PR with that integration, we would definitely welcome it!

Otherwise, @moojink @kpertsch or I will try to get to this as soon as we can!

[fugokidi]

@siddk Thank you for considering simulation. Although I am going to do it, the compute in my institution is under maintenance till July 12. By that time, if there is not any, I will try to contribute. If someone is faster and wants to contribute, please go ahead.

[DelinQu]

I just finished the openvla policy integration in simpler env and performed an evaluation. The performance is not too bad, and I might miss some implementation details, e.g., normalization and scales. Waiting for me to submit a PR. 🤗

[mbreuss]

@DelinQu very cool! Can you share your results for OpenVLA on the Simpler Benchmark?

[DelinQu]

Of course! I'm organizing the code and log.

[DelinQu]

The code is avaiable at SimplerEnv-OpenVLA. I need about one day to complete all the experiments and organize the data before reporting the results. The PR will be requested after all the validation.

[ZSL98]

@DelinQu Cool! Could you please share a link to your results? That would be helpful.

[ZSL98]

Oh! I found your PR here

[michaelwan11]

@moojink @siddk @kpertsch Thanks for the great work! I was wondering if you have OpenVLA success rates for the SimplerEnv environments? I took a look at the results obtained by @DelinQu here: simpler-env/SimplerEnv#10 and they seem a bit low compared to the real-world results you achieve in the paper, so I was wondering if you obtain similar results to what's in the PR.

[huangmz21]

Thank you very much for open-sourcing the VLA.I am new to this and I am trying to run the finetune.py on a single RTX 4090.I wonder if the parameter "max_steps" is sufficient for the module but not for various datasets,so we can stop the training manually.