WARNING: THIS SITE IS A MIRROR OF GITHUB.COM / IT CANNOT LOGIN OR REGISTER ACCOUNTS / THE CONTENTS ARE PROVIDED AS-IS / THIS SITE ASSUMES NO RESPONSIBILITY FOR ANY DISPLAYED CONTENT OR LINKS / IF YOU FOUND SOMETHING MAY NOT GOOD FOR EVERYONE, CONTACT ADMIN AT ilovescratch@foxmail.com
Skip to content

Conversation

@arcticfly
Copy link
Contributor

@arcticfly arcticfly commented Dec 19, 2025

Utility function to simplify comparison of constant model's metrics against those of newly trained models.

In the screenshot below, the golden line is the error rate of a comparison model.
Screenshot 2025-12-19 at 1 03 46 PM

Example:

await log_constant_metrics_wandb(
    model=model,
    num_steps=num_horizontal_run_steps,
    split_metrics={
        "val": metrics_dict,
        "val_failures": {
            k: v for k, v in metrics_dict.items() if "failed" in k
        },
    },
)

@arcticfly arcticfly requested a review from bradhilton December 19, 2025 21:20
@arcticfly arcticfly merged commit 8bfb8b8 into main Dec 19, 2025
2 checks passed
@arcticfly arcticfly deleted the horizontal-run branch December 19, 2025 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants