polars.exceptions.ComputeError: failed to determine supertype of list[extension] and datetime[μs]

[bschilder]

I encountered an error when running the meds_etl_omop command:

meds_etl_omop LOCAL_OMOP/lrWGS lrWGS/MEDS_v1

Output:

Generating metadata from OMOP `concept` table
1it [00:12, 12.77s/it]
Decompressing OMOP tables, mapping to MEDS Flat format, writing to disk...
0it [00:00, ?it/s]
 10%|████▍                                       | 1/10 [00:21<03:11, 21.28s/it]
Traceback (most recent call last):
  File "/home/jupyter/.local/bin/meds_etl_omop", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/jupyter/.local/lib/python3.12/site-packages/meds_etl/omop.py", line 750, in main
    process_table_parquet(task)
  File "/home/jupyter/.local/lib/python3.12/site-packages/meds_etl/omop.py", line 424, in process_table_parquet
    write_event_data(
  File "/home/jupyter/.local/lib/python3.12/site-packages/meds_etl/omop.py", line 315, in write_event_data
    event_data.sink_parquet(fname, compression="zstd", compression_level=1, maintain_order=False)
  File "/home/jupyter/.local/lib/python3.12/site-packages/polars/_utils/unstable.py", line 59, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jupyter/.local/lib/python3.12/site-packages/polars/lazyframe/frame.py", line 2228, in sink_parquet
    return lf.sink_parquet(
           ^^^^^^^^^^^^^^^^
polars.exceptions.ComputeError: failed to determine supertype of list[extension] and datetime[μs]

This error occurred with the following context stack:
	[1] 'select' failed
	[2] 'sink' input failed to resolve

Environment versions:

Python 3.12
meds                     0.1.3
meds_etl                 0.1.3
polars                   0.20.31

$ pip list
Package                  Version
------------------------ ---------------
accelerate               1.5.2
aenum                    3.1.15
aiobotocore              2.21.1
aioitertools             0.12.0
aliyun-python-sdk-core   2.16.0
aliyun-python-sdk-kms    2.16.5
annotated-types          0.7.0
botocore                 1.37.1
cairocffi                1.7.1
CairoSVG                 2.7.1
clickclick               20.10.2
connexion                2.14.2
cryptography             44.0.2
cssselect2               0.8.0
datasets                 3.4.1
etcd-distro              3.5.1
Flask                    2.2.5
fsspec                   2024.12.0
graphscope               0.29.0
graphscope-client        0.29.0
graphscope-flex          0.31.0
gremlinpython            3.7.0
grpcio                   1.71.0
grpcio-tools             1.71.0
gs-apps                  0.29.0
gs-coordinator           0.29.0
gs-engine                0.29.0
gs-include               0.29.0
gs_interactive           0.3
hdfs3                    0.3.1
huggingface-hub          0.29.3
icecream                 2.1.3
inflection               0.5.1
itsdangerous             2.2.0
lazy-import              0.2.2
makefun                  1.15.6
meds                     0.1.3
meds_etl                 0.1.3
mpmath                   1.3.0
mypy-protobuf            3.6.0
neo4j                    5.10.0
nest-asyncio             1.6.0
networkx                 2.8
nptyping                 2.4.1
nvidia-cublas-cu12       12.4.5.8
nvidia-cuda-cupti-cu12   12.4.127
nvidia-cuda-nvrtc-cu12   12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12        9.1.0.70
nvidia-cufft-cu12        11.2.1.3
nvidia-curand-cu12       10.3.5.147
nvidia-cusolver-cu12     11.6.1.9
nvidia-cusparse-cu12     12.3.1.170
nvidia-cusparselt-cu12   0.6.2
nvidia-nccl-cu12         2.21.5
nvidia-nvjitlink-cu12    12.4.127
nvidia-nvtx-cu12         12.4.127
oss2                     2.19.1
pip                      25.0.1
polars                   0.20.31
proto-plus               1.24.0.dev1
protobuf                 3.20.0
py4j                     0.10.9.5
pyarrow                  19.0.1
pycryptodome             3.22.0
pydantic                 2.8.2
pydantic_core            2.20.1
pygal                    3.0.5
pynndescent              0.5.13
pyOpenSSL                25.0.0
pysimdjson               5.0.2
pyspark                  3.3.4
s3fs                     2025.3.0
safetensors              0.5.3
schedule                 1.2.1
simple-parsing           0.1.7
swagger_ui_bundle        1.1.0
sympy                    1.13.1
tokenizers               0.21.1
torch                    2.6.0
torch-geometric          2.7.0
transformers             4.49.0
treelib                  1.7.1
triton                   3.2.0
types-protobuf           5.29.1.20250315
typing_extensions        4.12.2
umap-learn               0.5.7
vineyard                 0.24.2
vineyard-bdist           0.24.2
vineyard-io              0.24.2
Werkzeug                 2.2.3
xformers                 0.0.29.post3
xxhash                   3.5.0
zstd                     1.5.6.6

Thanks!
Brian

[bschilder]

Here's what the concepts table looks like:

import pandas as pd
concepts = pd.read_parquet("LOCAL_OMOP/lrWGS/concept.parquet")
concepts

[treena908]

having same problem.

using : meds==0.1.3 meds_etl==0.1.3, polars==0.20.31
error: polars.exceptions.ComputeError: failed to determine supertype of list[extension] and datetime[μs]

I have 'birth_datetime' column in 'person' table in the format ''2018-08-03 22:20:52+00:00'

Using meds_etl==0.1.3 is mandatory to use femr==0.2.3 package which converts the dataset in the form:

Dataset({
features: ['patient_id', 'events'],
num_rows: 6732
})

the updated meds_etl version like meds_etl==0.3.9 doesn't create dataset in the desired format that is accepted in femr. Can you suggest a wayaround with this.

mY DATASET (ALL of us) IS omop FORMAT.

[huangy6]

I believe this is due to an error on this line which exists in 0.1.3 but was corrected in subsequent versions:

meds_etl/src/meds_etl/omop.py

Line 118 in d93fe27

return (meds_etl.flat.parse_time(pl.col(column), OMOP_TIME_FORMATS),)

This line should instead say return meds_etl.flat.parse_time(pl.col(column), OMOP_TIME_FORMATS) (without the tuple)