opmd matplotlib.pylab crash (depending on import order)

[pgrete]

Describe the bug
Code crashes on import when openpmd is imported before matplotlib

To Reproduce
Compile-able/executable code example to reproduce the problem:

(py-oct25) pgrete@login10 ~ $ python -c "import openpmd_api as opmd; import matplotlib.pylab as plt; print('nope')"
Segmentation fault (core dumped)
(py-oct25) pgrete@login10 ~ $ python -c "import matplotlib.pylab as plt; import openpmd_api as opmd; print('yay')"
yay
(py-oct25) pgrete@login10 ~ $ python -c "import matplotlib as mpl; import openpmd_api as opmd; import matplotlib.pylab as plt; print('yay')"
yay

Expected behavior
No crash

Software Environment

• version of openPMD-api: current dev
• installed openPMD-api via: from source
• machine: Frontier
• name and version of Python implementation: CPython 3.11

[franzpoeschel]

I remember having had a very similar issue a while ago with Lammps and openPMD-api (mala-project/mala#476 (comment)). The same error pattern: Depending on import order, either a segfault or not.
In the related issue, the reason for the error was mainly a packaging error: Both packages were compiled against incompatible implementations of the C++ stdlib. Since the ABI of the stdlib is in parts identical, the loader jumped from lib A to lib B, or from B to A, depending which it saw first (hence the error only if you import things in a certain order).

The shortterm solution is probably to ensure a consistent build environment for both packages.
@ax3l We should maybe think of a way to not reexpose stdlib symbols?

[pgrete]

I see. Thanks for the quick pointer.

This helped me to work around the problem. I didn't use the python binding from my openpmd install (from source), but now installed the openpmd version from pip.

But practically, how would I fix this if I depend on features only enable/disabled in source builds (assuming I don't want to build an entire python/matplotlib/... stack from source)?

[franzpoeschel]

But practically, how would I fix this if I depend on features only enable/disabled in source builds (assuming I don't want to build an entire python/matplotlib/... stack from source)?

Since it seems to be openPMD injecting itself into Matplotlib in this case, probably not very easily..
(If things were reversed, i.e. a crash in openPMD that jumps into Matplotlib, recompiling with -fvisibility=protected may be an option since that prevents symbols from being resolved externally, as far as I understand it).
As things are, I only see the solution of compiling openPMD with -fvisibility=hidden so that Matplotlib does not even get to see our internal symbols, but that means we will need to manually export those of our symbols that should remain default-visible. TLDR: Can be done, but requires some work on our side.
I'll need to discuss with Axel what solution would be best.

[pgrete]

Potentially related question: does the pypi version of openpmd come with an internal version of adios2?
I just tried to load compressed data (in python) and got

ValueError: [Tue Oct 14 08:23:32 2025] [ADIOS2 EXCEPTION] <Operator> <OperatorFactory> <MakeOperator> : ADIOS2 didn't compile with sz library, operator not added

on series.flush().
However, the environment I'm running in uses an adios2 version with sz support (adios2/2.10.2-mpi on Frontier) which I used minutes before to generate the data.

[franzpoeschel]

Potentially related question: does the pypi version of openpmd come with an internal version of adios2?

Yes, it comes with a static build of ADIOS2 included.
Heavily WIP, but #1793 is an attempt at fixing the original issue. But you're currently likely to get undefined-symbol errors when using that branch. Current state works with some versions of Clang and g++.

[ax3l]

Potentially related question: does the pypi version of openpmd come with an internal version of adios2?

Correct, we compile and self-contain the PyPI wheels. SZ is missing, we can add it here to the library_builders.*: https://github.com/openPMD/openPMD-api/tree/wheels

[ax3l]

Thank you for the report!

I am surprised this is an issue, because we do hide the symbols of the Python library we expose:

openPMD-api/CMakeLists.txt

Lines 612 to 613 in 21adf5a

	set_target_properties(openPMD.py PROPERTIES CXX_VISIBILITY_PRESET "hidden"
	CUDA_VISIBILITY_PRESET "hidden")

Is there anything special with your from-source build, e.g., did you build in Debug (which disables symbol hiding)?

[ax3l]

To debug this further, can you please run gdb on the segfaulting logic?

gdb --ex r --ex bt python -c "import openpmd_api as opmd; import matplotlib.pylab as plt; print('nope')"

How did you install matplotlib?
Are both installed against the same Python module on Frontier?

[ax3l]

My guess is that the build on Frontier mixes stdlibs/compilers from where Python, matplotlib and openPMD-api are based on. (I do not think that #1793 will help here, unfortunately.)

This can be controlled via environment variables, e.g., loading a consistent set of modules and setting CC/CXX. Here is an example how to build a consistent C++ HIP stack with Python on Frontier (see profile file and dependencies install script): https://warpx.readthedocs.io/en/latest/install/hpc/frontier.html

If you like to debug (and post here) the dynamic libraries loaded by matplotlib and openPMD-api, try https://github.com/haampie/libtree on the dynamic libraries / compiled module files -- it's like ldd but recursive and more informative.

[pgrete]

Based on the discussion I would not be surprised that it's a mix with using the Cray programming env (and cray-python) combined with pip installed modules.
For completeness, the full info is below.

I'll report back shortly with a clean HIP stacked based on the warpx pointers.

PS: Based on the warpx doc on large scale IO with OpenPMD/ADIOS2 (-> don't use for checkpoints), do you have any advice for large scale IO (target checkpoint size is ~320TB) on GPFS filesystems if I want/have to write through OpenPMD/ADIOS2/BP5?
The target storage system (ExaStore in Juelich) has 20 storage blocks online.

Backtrace and environment

To debug this further, can you please run gdb on the segfaulting logic?

gdb --ex r --ex bt python -c "import openpmd_api as opmd; import matplotlib.pylab as plt; print('nope')"

Here's the backtrace

#0  0x00007fffffff0b8e in ?? ()
#1  0x00007fffffff0800 in ?? ()
#2  0x00007fffdc6d5240 in ?? () from /lib64/libgcc_s.so.1
#3  0x0000000000e01eb0 in ?? ()
#4  0x00007fffe9795d4e in _Unwind_GetCFA () from /opt/cray/pe/cce/19.0.0/cce-clang/x86_64/lib/libunwind.so.1
#5  0x00007fffdc6cb325 in ?? () from /lib64/libgcc_s.so.1
#6  0x00007fffdc6cbe1d in _Unwind_Resume () from /lib64/libgcc_s.so.1
#7  0x00007ffbc6c36818 in ft2font__getattr__(std::basic_string<char, std::char_traits<char>, std::allocator<char> >) [clone .lto_priv.0] [clone .cold] ()
   from /autofs/nccs-svm1_proj/ast146/proj-shared/software/py-oct25/lib/python3.11/site-packages/matplotlib/ft2font.cpython-311-x86_64-linux-gnu.so
#8  0x00007ffbc6c76606 in pybind11::cpp_function::initialize<pybind11::object (*&)(std::basic_string<char, std::char_traits<char>, std::allocator<char> >), pybind11::object, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, pybind11::name, pybind11::scope, pybind11::sibling>(pybind11::object (*&)(std::basic_string<char, std::char_traits<char>, std::allocator<char> >), pybind11::object (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> >), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) ()
   from /autofs/nccs-svm1_proj/ast146/proj-shared/software/py-oct25/lib/python3.11/site-packages/matplotlib/ft2font.cpython-311-x86_64-linux-gnu.so
#9  0x00007ffbc6c4ab50 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /autofs/nccs-svm1_proj/ast146/proj-shared/software/py-oct25/lib/python3.11/site-packages/matplotlib/ft2font.cpython-311-x86_64-linux-gnu.so
#10 0x00007fffed633cfd in cfunction_call (func=0x7ffbc6d65440, args=<optimized out>, kwargs=<optimized out>) at Objects/methodobject.c:542
#11 0x00007fffed60d53b in _PyObject_MakeTpCall (tstate=0x7fffed984b58 <_PyRuntime+166328>, callable=0x7ffbc6d65440, args=0x7fffffff0e68, nargs=1, keywords=0x0) at Objects/call.c:214
#12 0x00007fffed60e398 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7fffffff0e68, callable=0x7ffbc6d65440, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90
#13 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x7fffffff0e68, callable=0x7ffbc6d65440, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:77
#14 PyObject_CallOneArg (func=0x7ffbc6d65440, arg=<optimized out>) at Objects/call.c:376
#15 0x00007fffed6351cb in _PyObject_LookupAttr (v=v@entry=0x7ffbc6d46700, name=<optimized out>, result=result@entry=0x7fffffff0ef0) at Objects/object.c:961
#16 0x00007fffed5a9c5a in PyImport_ImportModuleLevelObject (name=name@entry=0x7ffbc7d87b70, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7ffbc65e0340, level=1) at Python/import.c:1913
#17 0x00007fffed6716af in import_name (level=0x7fffed95c508 <_PyRuntime+872>, fromlist=0x7ffbc65e0340, name=0x7ffbc7d87b70, frame=<optimized out>, tstate=<optimized out>) at Python/ceval.c:7424
#18 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3946
#19 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41dcf0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#20 _PyEval_Vector (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, func=func@entry=0x7ffbc65e93a0, locals=locals@entry=0x7ffbc6f36680, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6434
#21 0x00007fffed6e79ba in PyEval_EvalCode (co=co@entry=0x7ffbc6770b30, globals=globals@entry=0x7ffbc6f36680, locals=locals@entry=0x7ffbc6f36680) at Python/ceval.c:1148
#22 0x00007fffed6e6ab5 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ffbc6f36680, globals=0x7ffbc6f36680, source=0x7ffbc6770b30) at Python/bltinmodule.c:1077
#23 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#24 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038f90, args=0x7ffbc67904d8, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#25 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffbc65f4d00, callargs=0x7ffbc67904c0, func=0x7fffed038f90, tstate=<optimized out>) at Python/ceval.c:7324
#26 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#27 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41d9f8, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#28 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffce0, locals=<optimized out>, args=0x7fffffff1410, argcount=2, kwnames=<optimized out>) at Python/ceval.c:6434
#29 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffff1410, callable=0x7fffecfffce0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#30 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffce0, vargs=0x7fffffff1498) at Objects/call.c:819
#31 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#32 0x00007fffed5a9791 in import_find_and_load (abs_name=0x7ffbc678d700, tstate=0x7fffed984b58 <_PyRuntime+166328>) at Python/import.c:1748
#33 PyImport_ImportModuleLevelObject (name=name@entry=0x7ffbc678d700, globals=globals@entry=0x0, locals=locals@entry=0x0, fromlist=fromlist@entry=0x0, level=0) at Python/import.c:1847
#34 0x00007fffed668b4f in builtin___import___impl (level=<optimized out>, fromlist=0x0, locals=0x0, globals=0x0, name=0x7ffbc678d700, module=<optimized out>) at Python/bltinmodule.c:277
#35 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:78
#36 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038b30, args=0x7ffbc673a038, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#37 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffbc6f365c0, callargs=0x7ffbc673a020, func=0x7fffed038b30, tstate=<optimized out>) at Python/ceval.c:7324
#38 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#39 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41d8a0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#40 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffe20, locals=<optimized out>, args=0x7fffffff18a0, argcount=3, kwnames=<optimized out>) at Python/ceval.c:6434
#41 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7fffffff18a0, callable=0x7fffecfffe20, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#42 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffe20, vargs=0x7fffffff1928) at Objects/call.c:819
#43 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#44 0x00007fffed5a9ca8 in PyImport_ImportModuleLevelObject (name=name@entry=0x7fffed0aaaf0, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7ffbc6970a40, level=0) at Python/import.c:1918
#45 0x00007fffed6716af in import_name (level=0x7fffed95c4e8 <_PyRuntime+840>, fromlist=0x7ffbc6970a40, name=0x7fffed0aaaf0, frame=<optimized out>, tstate=<optimized out>) at Python/ceval.c:7424
#46 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3946
#47 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41d830, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#48 _PyEval_Vector (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, func=func@entry=0x7ffbc67c9da0, locals=locals@entry=0x7ffbc66dff40, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6434
#49 0x00007fffed6e79ba in PyEval_EvalCode (co=co@entry=0x7ffbc6ae6550, globals=globals@entry=0x7ffbc66dff40, locals=locals@entry=0x7ffbc66dff40) at Python/ceval.c:1148
#50 0x00007fffed6e6ab5 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ffbc66dff40, globals=0x7ffbc66dff40, source=0x7ffbc6ae6550) at Python/bltinmodule.c:1077
#51 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#52 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038f90, args=0x7ffbc6790958, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#53 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffbc6971000, callargs=0x7ffbc6790940, func=0x7fffed038f90, tstate=<optimized out>) at Python/ceval.c:7324
#54 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#55 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41d538, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#56 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffce0, locals=<optimized out>, args=0x7fffffff1f70, argcount=2, kwnames=<optimized out>) at Python/ceval.c:6434
#57 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffff1f70, callable=0x7fffecfffce0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#58 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffce0, vargs=0x7fffffff1ff8) at Objects/call.c:819
#59 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#60 0x00007fffed5a9791 in import_find_and_load (abs_name=0x7ffbc694bf00, tstate=0x7fffed984b58 <_PyRuntime+166328>) at Python/import.c:1748
#61 PyImport_ImportModuleLevelObject (name=name@entry=0x7ffbc69781b0, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7ffbc68dd280, level=1) at Python/import.c:1847
#62 0x00007fffed6716af in import_name (level=0x7fffed95c508 <_PyRuntime+872>, fromlist=0x7ffbc68dd280, name=0x7ffbc69781b0, frame=<optimized out>, tstate=<optimized out>) at Python/ceval.c:7424
#63 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3946
#64 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41d470, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#65 _PyEval_Vector (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, func=func@entry=0x7ffbc6ae3560, locals=locals@entry=0x7ffbc68deb40, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6434
#66 0x00007fffed6e79ba in PyEval_EvalCode (co=co@entry=0xdcad30, globals=globals@entry=0x7ffbc68deb40, locals=locals@entry=0x7ffbc68deb40) at Python/ceval.c:1148
#67 0x00007fffed6e6ab5 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ffbc68deb40, globals=0x7ffbc68deb40, source=0xdcad30) at Python/bltinmodule.c:1077
#68 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#69 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038f90, args=0x7ffbc695b518, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#70 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffbc69270c0, callargs=0x7ffbc695b500, func=0x7fffed038f90, tstate=<optimized out>) at Python/ceval.c:7324
#71 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#72 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41d178, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#73 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffce0, locals=<optimized out>, args=0x7fffffff2640, argcount=2, kwnames=<optimized out>) at Python/ceval.c:6434
#74 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffff2640, callable=0x7fffecfffce0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#75 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffce0, vargs=0x7fffffff26c8) at Objects/call.c:819
#76 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#77 0x00007fffed5a9791 in import_find_and_load (abs_name=0x7ffbc68cf9b0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at Python/import.c:1748
#78 PyImport_ImportModuleLevelObject (name=name@entry=0x7ffbc68cf9b0, globals=globals@entry=0x0, locals=locals@entry=0x0, fromlist=fromlist@entry=0x0, level=0) at Python/import.c:1847
#79 0x00007fffed668b4f in builtin___import___impl (level=<optimized out>, fromlist=0x0, locals=0x0, globals=0x0, name=0x7ffbc68cf9b0, module=<optimized out>) at Python/bltinmodule.c:277
#80 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:78
#81 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038b30, args=0x7ffbc6aef2c8, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#82 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffbc68dd500, callargs=0x7ffbc6aef2b0, func=0x7fffed038b30, tstate=<optimized out>) at Python/ceval.c:7324
#83 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#84 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41d020, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#85 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffe20, locals=<optimized out>, args=0x7fffffff2ad0, argcount=3, kwnames=<optimized out>) at Python/ceval.c:6434
#86 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7fffffff2ad0, callable=0x7fffecfffe20, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#87 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffe20, vargs=0x7fffffff2b58) at Objects/call.c:819
#88 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#89 0x00007fffed5a9ca8 in PyImport_ImportModuleLevelObject (name=name@entry=0x7fffed0aaaf0, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7ffbc6ad0b40, level=0) at Python/import.c:1918
#90 0x00007fffed6716af in import_name (level=0x7fffed95c4e8 <_PyRuntime+840>, fromlist=0x7ffbc6ad0b40, name=0x7fffed0aaaf0, frame=<optimized out>, tstate=<optimized out>) at Python/ceval.c:7424
#91 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3946
#92 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41cf60, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#93 _PyEval_Vector (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, func=func@entry=0x7ffbc6ae25c0, locals=locals@entry=0x7ffbc68ceb40, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6434
#94 0x00007fffed6e79ba in PyEval_EvalCode (co=co@entry=0xdc4540, globals=globals@entry=0x7ffbc68ceb40, locals=locals@entry=0x7ffbc68ceb40) at Python/ceval.c:1148
#95 0x00007fffed6e6ab5 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ffbc68ceb40, globals=0x7ffbc68ceb40, source=0xdc4540) at Python/bltinmodule.c:1077
#96 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#97 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038f90, args=0x7ffbc68cf458, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#98 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffbc68cec80, callargs=0x7ffbc68cf440, func=0x7fffed038f90, tstate=<optimized out>) at Python/ceval.c:7324
#99 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#100 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41cc68, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#101 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffce0, locals=<optimized out>, args=0x7fffffff31a0, argcount=2, kwnames=<optimized out>) at Python/ceval.c:6434
#102 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffff31a0, callable=0x7fffecfffce0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#103 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffce0, vargs=0x7fffffff3228) at Objects/call.c:819
#104 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#105 0x00007fffed5a9791 in import_find_and_load (abs_name=0x7ffbc6adad80, tstate=0x7fffed984b58 <_PyRuntime+166328>) at Python/import.c:1748
#106 PyImport_ImportModuleLevelObject (name=name@entry=0x7ffbc6adad80, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7ffbc6abbfd0, level=0) at Python/import.c:1847
#107 0x00007fffed6716af in import_name (level=0x7fffed95c4e8 <_PyRuntime+840>, fromlist=0x7ffbc6abbfd0, name=0x7ffbc6adad80, frame=<optimized out>, tstate=<optimized out>) at Python/ceval.c:7424
#108 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3946
#109 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41cbf8, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#110 _PyEval_Vector (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, func=func@entry=0x7ffbc6ae1300, locals=locals@entry=0x7ffbc6b0eb40, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6434
#111 0x00007fffed6e79ba in PyEval_EvalCode (co=co@entry=0xcff9b0, globals=globals@entry=0x7ffbc6b0eb40, locals=locals@entry=0x7ffbc6b0eb40) at Python/ceval.c:1148
#112 0x00007fffed6e6ab5 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ffbc6b0eb40, globals=0x7ffbc6b0eb40, source=0xcff9b0) at Python/bltinmodule.c:1077
#113 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#114 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038f90, args=0x7ffbc68cc718, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#115 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffbc6b0ec80, callargs=0x7ffbc68cc700, func=0x7fffed038f90, tstate=<optimized out>) at Python/ceval.c:7324
#116 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#117 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41c900, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#118 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffce0, locals=<optimized out>, args=0x7fffffff3870, argcount=2, kwnames=<optimized out>) at Python/ceval.c:6434
#119 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffff3870, callable=0x7fffecfffce0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#120 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffce0, vargs=0x7fffffff38f8) at Objects/call.c:819
#121 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#122 0x00007fffed5a9791 in import_find_and_load (abs_name=0x7ffbc6b064c0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at Python/import.c:1748
#123 PyImport_ImportModuleLevelObject (name=name@entry=0x7ffbc6b064c0, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7fffed869860 <_Py_NoneStruct>, level=0) at Python/import.c:1847
#124 0x00007fffed6716af in import_name (level=0x7fffed95c4e8 <_PyRuntime+840>, fromlist=0x7fffed869860 <_Py_NoneStruct>, name=0x7ffbc6b064c0, frame=<optimized out>, tstate=<optimized out>) at Python/ceval.c:7424
#125 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3946
#126 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41c828, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#127 _PyEval_Vector (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, func=func@entry=0x7ffbc6a9cc20, locals=locals@entry=0x7ffbc6ace200, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6434
#121 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#122 0x00007fffed5a9791 in import_find_and_load (abs_name=0x7ffbc6b064c0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at Python/import.c:1748
#123 PyImport_ImportModuleLevelObject (name=name@entry=0x7ffbc6b064c0, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7fffed869860 <_Py_NoneStruct>, level=0) at Python/import.c:1847
#124 0x00007fffed6716af in import_name (level=0x7fffed95c4e8 <_PyRuntime+840>, fromlist=0x7fffed869860 <_Py_NoneStruct>, name=0x7ffbc6b064c0, frame=<optimized out>, tstate=<optimized out>) at Python/ceval.c:7424
#125 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3946
#126 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41c828, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#127 _PyEval_Vector (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, func=func@entry=0x7ffbc6a9cc20, locals=locals@entry=0x7ffbc6ace200, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6434
--Type <RET> for more, q to quit, c to continue without paging--
#128 0x00007fffed6e79ba in PyEval_EvalCode (co=co@entry=0xde6cb0, globals=globals@entry=0x7ffbc6ace200, locals=locals@entry=0x7ffbc6ace200) at Python/ceval.c:1148
#129 0x00007fffed6e6ab5 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ffbc6ace200, globals=0x7ffbc6ace200, source=0xde6cb0) at Python/bltinmodule.c:1077
#130 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#131 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038f90, args=0x7ffbc6ae8ad8, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#132 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffbc6acdfc0, callargs=0x7ffbc6ae8ac0, func=0x7fffed038f90, tstate=<optimized out>) at Python/ceval.c:7324
#133 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#134 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41c530, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#135 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffce0, locals=<optimized out>, args=0x7fffffff3f40, argcount=2, kwnames=<optimized out>) at Python/ceval.c:6434
#136 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffff3f40, callable=0x7fffecfffce0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#137 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffce0, vargs=0x7fffffff3fc8) at Objects/call.c:819
#138 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#139 0x00007fffed5a9791 in import_find_and_load (abs_name=0x7ffbc7d81e30, tstate=0x7fffed984b58 <_PyRuntime+166328>) at Python/import.c:1748
#140 PyImport_ImportModuleLevelObject (name=name@entry=0x7ffbc7d81e30, globals=globals@entry=0x0, locals=locals@entry=0x0, fromlist=fromlist@entry=0x0, level=0) at Python/import.c:1847
#141 0x00007fffed668b4f in builtin___import___impl (level=<optimized out>, fromlist=0x0, locals=0x0, globals=0x0, name=0x7ffbc7d81e30, module=<optimized out>) at Python/bltinmodule.c:277
#142 builtin___import__ (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:78
#143 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038b30, args=0x7fffed0733b8, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#144 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffbc6e03080, callargs=0x7fffed0733a0, func=0x7fffed038b30, tstate=<optimized out>) at Python/ceval.c:7324
#145 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#146 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41c3d8, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#147 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffe20, locals=<optimized out>, args=0x7fffffff43d0, argcount=3, kwnames=<optimized out>) at Python/ceval.c:6434
#148 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=3, args=0x7fffffff43d0, callable=0x7fffecfffe20, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#149 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffe20, vargs=0x7fffffff4458) at Objects/call.c:819
#150 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#151 0x00007fffed5a9ca8 in PyImport_ImportModuleLevelObject (name=name@entry=0x7fffed0aaaf0, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7fffeced41c0, level=0) at Python/import.c:1918
#152 0x00007fffed6716af in import_name (level=0x7fffed95c4e8 <_PyRuntime+840>, fromlist=0x7fffeced41c0, name=0x7fffed0aaaf0, frame=<optimized out>, tstate=<optimized out>) at Python/ceval.c:7424
#153 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3946
#154 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41c378, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#155 _PyEval_Vector (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, func=func@entry=0x7fffece70ae0, locals=locals@entry=0x7fffece57280, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6434
#156 0x00007fffed6e79ba in PyEval_EvalCode (co=co@entry=0xcd47e0, globals=globals@entry=0x7fffece57280, locals=locals@entry=0x7fffece57280) at Python/ceval.c:1148
#157 0x00007fffed6e6ab5 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7fffece57280, globals=0x7fffece57280, source=0xcd47e0) at Python/bltinmodule.c:1077
#158 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at Python/clinic/bltinmodule.c.h:465
#159 0x00007fffed633ba6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7fffed038f90, args=0x7ffbc7d87ad8, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52
#160 0x00007fffed672b86 in do_call_core (use_tracing=<optimized out>, kwdict=0x7fffed0aaa80, callargs=0x7ffbc7d87ac0, func=0x7fffed038f90, tstate=<optimized out>) at Python/ceval.c:7324
#161 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:5376
#162 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41c080, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#163 _PyEval_Vector (tstate=0x7fffed984b58 <_PyRuntime+166328>, func=0x7fffecfffce0, locals=<optimized out>, args=0x7fffffff4aa0, argcount=2, kwnames=<optimized out>) at Python/ceval.c:6434
#164 0x00007fffed60e016 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffff4aa0, callable=0x7fffecfffce0, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92
#165 object_vacall (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, base=base@entry=0x0, callable=0x7fffecfffce0, vargs=0x7fffffff4b28) at Objects/call.c:819
#166 0x00007fffed60f0ef in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at Objects/call.c:878
#167 0x00007fffed5a9791 in import_find_and_load (abs_name=0x7fffed050440, tstate=0x7fffed984b58 <_PyRuntime+166328>) at Python/import.c:1748
#168 PyImport_ImportModuleLevelObject (name=name@entry=0x7fffed050440, globals=<optimized out>, locals=<optimized out>, fromlist=fromlist@entry=0x7fffed869860 <_Py_NoneStruct>, level=0) at Python/import.c:1847
#169 0x00007fffed6716af in import_name (level=0x7fffed95c4e8 <_PyRuntime+840>, fromlist=0x7fffed869860 <_Py_NoneStruct>, name=0x7fffed050440, frame=<optimized out>, tstate=<optimized out>) at Python/ceval.c:7424
#170 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3946
#171 0x00007fffed66a1eb in _PyEval_EvalFrame (throwflag=0, frame=0x7fffed41c020, tstate=0x7fffed984b58 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#172 _PyEval_Vector (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, func=func@entry=0x7fffed08e2a0, locals=locals@entry=0x7fffed0aa980, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6434
#173 0x00007fffed6e79ba in PyEval_EvalCode (co=co@entry=0x7fffed030f30, globals=globals@entry=0x7fffed0aa980, locals=locals@entry=0x7fffed0aa980) at Python/ceval.c:1148
#174 0x00007fffed6fc123 in run_eval_code_obj (tstate=tstate@entry=0x7fffed984b58 <_PyRuntime+166328>, co=co@entry=0x7fffed030f30, globals=globals@entry=0x7fffed0aa980, locals=locals@entry=0x7fffed0aa980) at Python/pythonrun.c:1710
#175 0x00007fffed6fc0aa in run_mod (mod=mod@entry=0x483108, filename=filename@entry=0x7fffed9616b0 <_PyRuntime+21776>, globals=globals@entry=0x7fffed0aa980, locals=locals@entry=0x7fffed0aa980, flags=flags@entry=0x7fffffff4fd0, 
    arena=arena@entry=0x7fffecfcb8b0) at Python/pythonrun.c:1731
#176 0x00007fffed6fc019 in PyRun_StringFlags (str=<optimized out>, start=<optimized out>, globals=0x7fffed0aa980, locals=0x7fffed0aa980, flags=0x7fffffff4fd0) at Python/pythonrun.c:1601
#177 0x00007fffed6fbf88 in PyRun_SimpleStringFlags (command=0x7fffecfdc590 "import openpmd_api as opmd; import matplotlib.pylab as plt; print('nope')\n", flags=flags@entry=0x7fffffff4fd0) at Python/pythonrun.c:487
#178 0x00007fffed703f0e in pymain_run_command (command=<optimized out>) at Modules/main.c:255
#179 pymain_run_python (exitcode=0x7fffffff4fa4) at Modules/main.c:592
#180 Py_RunMain () at Modules/main.c:680
#181 0x00007fffed703bd7 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:734
#182 0x00007fffed148e6c in __libc_start_call_main () from /lib64/libc.so.6
#183 0x00007fffed148f35 in __libc_start_main_impl () from /lib64/libc.so.6
#184 0x000000000040066a in _start () at ../sysdeps/x86_64/start.S:120

How did you install matplotlib? Are both installed against the same Python module on Frontier?

Module environment is/was

module load PrgEnv-cray/8.6.0
module load cpe/25.03
module load craype-accel-amd-gfx90a
module load rocm/6.4.1
module load cmake ninja
module unload darshan-runtime
module load hdf5/1.14.5-mpi
module load adios2 cray-python

export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}
export MPICH_GPU_SUPPORT_ENABLED=1

and python env installed via

python3 -m pip install --upgrade pip
python3 -m pip install --upgrade virtualenv
python3 -m pip cache purge
python3 -m venv /ccs/proj/ast146/proj-shared/software/py-oct25
source /ccs/proj/ast146/proj-shared/software/py-oct25/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade build
python3 -m pip install --upgrade packaging
python3 -m pip install --upgrade wheel
python3 -m pip install --upgrade setuptools
python3 -m pip install --upgrade h5py
python3 -m pip install --upgrade scipy
MPICC=cc python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py
python3 -m pip install --upgrade matplotlib
python3 -m pip install --upgrade jupyter
python3 -m pip install --upgrade yt
python3 -m pip install --upgrade unyt
python3 -m pip install --upgrade cmasher
python3 -m pip install --upgrade palettable
python3 -m pip install --upgrade adios2

and then OpenPMD via

cmake -Bbuild-oct25-21adf5a -DCMAKE_INSTALL_PREFIX=/ccs/proj/ast146/proj-shared/openpmd-api/oct25-21adf5a -DopenPMD_USE_PYTHON=ON -DPython_EXECUTABLE=$(which python3)

and then I used

export PYTHONPATH=/ccs/proj/ast146/proj-shared/openpmd-api/oct25-21adf5a/lib64/python3.11/site-packages:$PYTHONPATH
export LD_LIBRARY_PATH=/ccs/proj/ast146/proj-shared/openpmd-api/oct25-21adf5a/lib64:${LD_LIBRARY_PATH}

before running the scripts.