How should I handle json serialization of ListConfig objects?

[jeffliu-LL]

I've run into an issue with configs that contain lists if they get serialized to json. The process throws an error because the list remains a ListConfig, which doesn't json serialize.

Here's a minimum example:

from hydra_zen import builds, instantiate
import json

def json_serialize_func(list_obj):
    return json.dumps(list_obj)

json_config = builds(json_serialize_func, ['test', 'item'])
instantiate(json_config)

raises the following:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File /.../env/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py:92, in _call_target(_target_, _partial_, args, kwargs, full_key)
     91 try:
---> 92     return _target_(*args, **kwargs)
     93 except Exception as e:

Untitled-1.ipynb Cell 5 line 2
      [1](vscode-notebook-cell:Untitled-1.ipynb?jupyter-notebook#W4sdW50aXRsZWQ%3D?line=0) def json_serialize_func(list_obj):
----> [2](vscode-notebook-cell:Untitled-1.ipynb?jupyter-notebook#W4sdW50aXRsZWQ%3D?line=1)     return json.dumps(list_obj)

File /.../env/lib/python3.11/json/__init__.py:231, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    227 if (not skipkeys and ensure_ascii and
    228     check_circular and allow_nan and
    229     cls is None and indent is None and separators is None and
    230     default is None and not sort_keys and not kw):
--> 231     return _default_encoder.encode(obj)
    232 if cls is None:

File /.../env/lib/python3.11/json/encoder.py:200, in JSONEncoder.encode(self, o)
    197 # This doesn't pass the iterator directly to ''.join() because the
    198 # exceptions aren't as detailed.  The list call should be roughly
    199 # equivalent to the PySequence_Fast that ''.join() would do.
--> 200 chunks = self.iterencode(o, _one_shot=True)
    201 if not isinstance(chunks, (list, tuple)):

File /.../env/lib/python3.11/json/encoder.py:258, in JSONEncoder.iterencode(self, o, _one_shot)
    254     _iterencode = _make_iterencode(
    255         markers, self.default, _encoder, self.indent, floatstr,
    256         self.key_separator, self.item_separator, self.sort_keys,
    257         self.skipkeys, _one_shot)
--> 258 return _iterencode(o, 0)

File /.../env/lib/python3.11/json/encoder.py:180, in JSONEncoder.default(self, o)
    162 """Implement this method in a subclass such that it returns
    163 a serializable object for ``o``, or calls the base implementation
    164 (to raise a ``TypeError``).
   (...)
    178 
    179 """
--> 180 raise TypeError(f'Object of type {o.__class__.__name__} '
    181                 f'is not JSON serializable')

TypeError: Object of type ListConfig is not JSON serializable

I've gotten around it by manually converting the ListConfig to a list using OmegaConf.to_object() but that seems to kind of defeat the point of using automatic configs. Am I instantiating or making the configuration wrong in some way?

[jgbos]

Ah, this one gets me too. Unfortunately Hydra converts lists to a omegaconf.ListConfig object and does not convert it to list on instantiation. There are two options for you:

1. Convert the object to list: json.dumps(list(list_obj))
2. Use the convert all flag for Hydra instantiate: json_config = builds(json_serialize_func, ['test', 'item'], hydra_convert="all")

[jeffliu-LL]

Gotcha. The latter makes more sense for me since the serialization is actually happening inside a library function that I'd rather not edit. Are there any potential drawbacks from setting hydra_convert="all"? Do we lose any functionality in that case?

[rsokl]

You can also specify the parameter-conversion strategy via instantiate rather than in all of the configs. This can be nice as it makes the yamls cleaner: https://hydra.cc/docs/advanced/instantiate_objects/overview/#parameter-conversion-strategies

[jgbos]

@rsokl You can't do that with zen though right?

[rsokl]

The Zen class exposes an instantiate method that can be overridden for this sort of thing :)

from typing import Any
from hydra_zen.wrapper import Zen
from hydra_zen import zen, instantiate
from functools import partial

class CustomZen(Zen):
    @staticmethod
    def instantiate(__c: Any) -> Any:
        return instantiate(__c, _convert_="object")

my_zen = partial(zen, ZenWrapper=CustomZen)