Confused about the store

[francisco-camargo]

Hey all,

Been staring and playing with the scikit-learn How-To. It help me quite a bit in understanding some basics about how to structure a project. However, it also brought into focus something that I'm realizing I have found confusing throughout the Tutorials and HowTo's that I've read: the store. Having read through the documentation, I still just don't have a mental model of how to think about the store, and I therefore don't really know how to use it.

• The "Configure and store classifiers" and the "Configure and store datasets" sections of the How-To make sense in that we want to add all these objects to the store.
• The "Task configuration" where store(make_config( is used kind of makes sense if the scope is just to set some default behavior. I suspect the syntax would be made clear if I reread the docs again
• The code within if __name__ == "__main__": is where the wheels fall off for me. Several functions, methods, and options are used and I don't know where to begin to understand what's happening there, nor do I have any real idea of the scope of what each line of code is trying to accomplish

In trying to play with the How-To code, I was thinking of trying to see if I could separate out the scikit-learn version of my_app.py into several bite-sized scripts. However this is how I realized I was not understanding the store. For example, do objects added to the store get written to disk somewhere such that I could call them later? Such that I could go to the command line and do something like

$ python put_classifiers_into_store.py
...
$ python put_datasets_into_store.py
...
$ python train_models.py

In other words, can I run each piece of the code separately? Or is it the case that I must have a single function that kicks-off putting objects into the store as well as running the task function?

I hope this makes some sense! Thanks!

[jgbos]

A store is only created "in-memory". You can store configs across multiple files, but you need to make sure to import those files to make sure the configs get loaded into memory, something like this:

# put_classifiers_into_store.py
from hydra_zen import store

sklearn_store = store(group="classifier")

# add classifier configs to store
sklearn_store(...)

# put_datasets_into_store.py
from hydra_zen import store

sklearn_dataset = store(group="dataset")

# add dataset configs to store
sklearn_dataset (...)

# train_models.py

# Need to load the files to add configs to store.
#
# Importing each module runs the code in said module. I.e.
# it adds the dataset/classifier configs to the store.
import put_datasets_into_store
import put_classifiers_into_store

from hydra_zen import store, make_config, zen

store(
    make_config(
        hydra_defaults=["_self_", {"dataset": "moons"}, {"classifier": "knn"}],
        dataset=None,
        classifier=None,
    ),
    name="config",
)

def task(dataset, classifier):
    # training task
    ...

if __name__ == "__main__":
    # store is only in-memory. You must add the configs to the 
    # Hydra's ConfigStore so Hydra "sees" them when you configure and launch your job
    store.add_to_hydra_store()
    zen(task).hydra_main(config_path=None, config_name="config")

If you do python train_models.py --info you should get the config:

dataset:
  _target_: sklearn.datasets._samples_generator.make_moons
  _partial_: true
  noise: 0.3
  random_state: 0
classifier:
  _target_: sklearn.neighbors._classification.KNeighborsClassifier
  n_neighbors: 3
  weights: uniform
  algorithm: auto
  leaf_size: 30
  p: 2
  metric: minkowski
  metric_params: null
  n_jobs: null

[francisco-camargo]

That makes sense, I will try this, thanks!

[francisco-camargo]

I implemented your suggestions, and works right away

One minor readability change that I made: I made function definitions such that in train_models.py the imported code is explicitly used

import add_classifiers_to_store
import add_datasets_to_store

# Define and save all classifiers of interest to the local hydra-zen store
add_classifiers_to_store.all_classifiers()

# Define and save all datasets of interest to the local hydra-zen store
add_datasets_to_store.all_datasets()

I can imagine some confused faces if things are imported but not explicitly used 😆

[rsokl]

Hi @francisco-camargo thanks for the feedback. As a temporary band aid, I added a little bit more inline explanation in the how-to:

That being said, it sounds like our docs need to do a much better job giving the reader more hands-on experience with and intuition for zen and store.

We really appreciate you taking the time to share your experience with us.

[rsokl]

To provide a little bit of insight into the config store business:

Hydra's config store
Hydra has a single, global config store. This "store" is just a dictionary that maps (config-group, config-name) -> config. It's purpose is to tell Hydra "here are a bunch of configs that I might reference by name (and perhaps by group) from the CLI". A user is responsible for populating this dictionary at runtime before configuring and running their Hydra job.

E.g. in the scikit how-to if I run python my_app.py dataset=circles, then I am telling Hydra to look in its config store, under (group=dataset, name=circles) and use that config for the dataset part of my job's config.

A major issue with Hydra's config store is that it is global and that it does not protect users from (inadvertently) overwriting its entries. That is, you might run a script that modifies Hydra's config store, and I might run your script without realizing that it is changing which configs are stored. Your script might silently modify the configs that get composed for my job. This is a recipe for disaster as you begin to create with larger scale Hydra-based applications.

How can we make Hydra's config store safer to use and easier to reason about?

hydra-zen's config store

hydra-zen's config store is local, and it prevents users from accidentally overwriting store entries. It is an abstraction, or intermediary, over Hydra's global store that exists for the purpose of ultimately populating Hydra's global config store.

Unlike Hydra's global config store, you can create multiple, independent config stores with hydra-zen. Each one ultimately corresponds to its own dictionary that maps maps (config-group, config-name) -> config, just like with Hydra's config store. But populating those independent dictionaries doesn't accomplish much until we add those entries to Hydra's global config store. This is typically the last thing that we do before generating the CLI to our Hydra application. This is where we call store.add_to_hydra_store().

Hopefully the reference docs (namely the example section) helps to clarify this. I will also work to improve the reference docs to be more self-sufficient, and to convey what I have written here.

[francisco-camargo]

Excellent, this is very insightful, thanks!

I agree, making the training material more self-sufficient may help quite a bit; it does feel like all the information need is written somewhere but it does require some hunting around various pages.

[addisonklinke]

@jgbos your example train_models.py has

# Importing each module runs the code in said module. I.e.
# it adds the dataset/classifier configs to the store.
import put_datasets_into_store
import put_classifiers_into_store

I've also relied on import side effects to trigger store usage in other files, but it has always felt a bit weird to me. Particularly when the imports aren't needed for anything other than storing. I'm curious whether you or @rsokl have found any better patterns, or is this still the recommended approach?

[addisonklinke]

If imports are unavoidable, maybe it's helpful to at least automate them? I could make a ZenStore subclass that's a singleton and then with each __call__ it tracks a list of callsites. Then during .add_to_hydra_store(), it runs a loop of import_lib.import_module() over the stored locations

[rsokl]

I think it was a mistake for me to define add_to_hydra_store on ZenStore. A better and safer design, which would avoid all of these issues, is to instead have included the following argument in zen: zen.hydra_main(*stores: ZenStore), which internally does the work of add_to_hydra_store for all of the stores it was passed.

This way hydra-zen apps are required to be passed all relevant stores at the point of cli creation:

if __name__ == "__main__":
    zen(task).hydra_main(store1, store2, config_name="my_app", config_path=None, version_base="1.2")

Rather than do auto-import magic, I think it is better to just introduce better abstractions that forces these use cases.

This would obviously be a major, compatibility-breaking change for me to introduce in-place in hydra-zen, but perhaps for your applications you can do this.

[rsokl]

Basically any global/non-local behavior should be avoided as much as possible. And the source of truth of what is getting added to hydra's store should be the place where the application's cli entry point is created.

[addisonklinke]

zen.hydra_main(*stores: ZenStore)

This is an interesting point and I understand the concern about changing the API

I think my question was actually slightly different. I'm not concerned about making store instances visible to zen.hydra_main() - I don't mind the store.add_to_hydra_store() all that much. Instead the trouble lies in populating the store. If I have a bunch of @store decorators throughout a codebase, each file has to get executed (i.e. import) otherwise nothing happens when you .add_to_hydra_store()