Fix: expand dict transformation by Shamik-07 · Pull Request #9561 · marimo-team/marimo

Shamik-07 · 2026-05-15T22:27:25Z

📝 Summary

Using narwahls to convert all backend to polars and then using the unnest function of polars for expanding the dict and then convert it back to the original backend.
Closes #4583

Screen.Recording.2026-05-15.at.18.07.461.mov

📋 Pre-Review Checklist

For large changes, or changes that affect the public API: this change was discussed or approved through an issue, on Discord, or the community discussions (Please provide a link if applicable).
Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
Video or media evidence is provided for any visual changes (optional).

✅ Merge Checklist

I have read the contributor guidelines.
Documentation has been updated where applicable, including docstrings for API changes.
Tests have been added for the changes made.

…mes.

vercel · 2026-05-15T22:27:30Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
marimo-docs	Ready	Preview, Comment	Jul 7, 2026 10:28pm

cubic-dev-ai

1 issue found across 3 files

Architecture diagram

sequenceDiagram
    participant User as User (Marimo UI)
    participant DFPlugin as Dataframe Plugin
    participant Handler as ExpandDict Handler (handlers.py)
    participant Narwhals as Narwhals Layer
    participant Polars as Polars Engine
    participant Backend as Original Backend (pandas/Ibis/other)

    Note over User,Backend: Expand Dict Transformation Flow

    User->>DFPlugin: Trigger expand dict on column
    DFPlugin->>Handler: handle_expand_dict(df, transform)

    Handler->>Narwhals: collect_and_preserve_type(df)
    Narwhals->>Backend: Collect actual data from original backend
    Backend-->>Narwhals: Data as native type
    Narwhals-->>Handler: Collected DataFrame + undo function

    Handler->>Polars: collected_df.to_polars()
    Note over Handler,Polars: Convert to Polars for unnest support

    Polars->>Polars: polars_df.unnest(column_id)
    Note over Polars: Handles null dict values correctly

    Polars-->>Handler: Unnested Polars DataFrame

    Handler->>Narwhals: nw.from_native(unnested)
    Narwhals->>Handler: Narwhals wrapper

    Handler->>Handler: undo(narwhals_df)
    Note over Handler: Convert back to original backend type

    Handler-->>DFPlugin: Transformed DataFrame
    DFPlugin-->>User: Updated table with expanded columns

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Re-trigger cubic}

mscolnick

need to take an optional dep on polars

… rows with the create test dataframes instead.

refactor: adding None == NaN in assert frame equal with nans method to use it in expand_dict test.

Shamik-07 · 2026-05-19T22:30:16Z

need to take an optional dep on polars

Done.
Added allow_none_equals_nan for assert_frame_equal_with_nans as None!=NaN, which was causing the use of assert_frame_equal_with_nans to fail for test_expand_dict.

Shamik-07 · 2026-05-19T23:12:49Z

There are some pandas CI errors that i am looking into.

… errors for mixed object columns.

Shamik-07 · 2026-05-19T23:30:00Z

There are some pandas CI errors that i am looking into.

This is happening because of data conversion mismatch between pandas and narwahls with mixed data types

"Could not convert '3' with type str: tried to convert to double"

for

test_print_code_result_matches_actual_transform_pandas(
    transform=ExpandDictTransform(
        type=TransformType.EXPAND_DICT,
        column_id='strings',
    ),
)

So my only option is to fallback to pandas backend processing separately for the unnest.
This should fix it.

cubic-dev-ai

1 issue found across 1 file (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

… json normalise following the handlers code.

codecov · 2026-05-20T21:22:55Z

Bundle Report

Bundle size has no change ✅

Affected Assets, Files, and Routes:

view changes for bundle: marimo-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
`assets/dist-*.js`	-81 bytes	102 bytes	-44.26%
`assets/dist-*.js`	119 bytes	256 bytes	86.86% ⚠️
`assets/dist-*.js`	-14 bytes	169 bytes	-7.65%
`assets/dist-*.js`	-68 bytes	335 bytes	-16.87%
`assets/dist-*.js`	-5 bytes	164 bytes	-2.96%
`assets/dist-*.js`	17 bytes	177 bytes	10.62% ⚠️
`assets/dist-*.js`	-76 bytes	183 bytes	-29.34%
`assets/dist-*.js`	90 bytes	259 bytes	53.25% ⚠️
`assets/dist-*.js`	-8 bytes	169 bytes	-4.52%
`assets/dist-*.js`	-231 bytes	104 bytes	-68.96%
`assets/dist-*.js`	-139 bytes	137 bytes	-50.36%
`assets/dist-*.js`	283 bytes	387 bytes	272.12% ⚠️
`assets/dist-*.js`	33 bytes	137 bytes	31.73% ⚠️
`assets/dist-*.js`	-111 bytes	276 bytes	-28.68%
`assets/dist-*.js`	299 bytes	403 bytes	287.5% ⚠️
`assets/dist-*.js`	46 bytes	183 bytes	33.58% ⚠️
`assets/dist-*.js`	-152 bytes	104 bytes	-59.38%
`assets/dist-*.js`	58 bytes	160 bytes	56.86% ⚠️
`assets/dist-*.js`	-60 bytes	104 bytes	-36.59%
`assets/ttcn-*.js`	7 bytes	64 bytes	12.28% ⚠️
`assets/ttcn-*.js`	-7 bytes	57 bytes	-10.94%
`assets/__vite-*.js`	-5 bytes	93 bytes	-5.1%
`assets/__vite-*.js`	5 bytes	98 bytes	5.38% ⚠️

Shamik-07 · 2026-06-02T21:24:56Z

please address copilot review. We can do a manual round of reviews after that.

Thanks, fixed.

kirangadhave

Requesting a major change which should remove hard dependency on polars for other backends which support structs.

Also please address other nits.

kirangadhave · 2026-06-30T06:14:44Z

+            expanded.index = result_df.index
+            return undo(nw.from_native(result_df.join(expanded)))
+
+        DependencyManager.polars.require(


non pandas dataframes are forced to go through polars conversion here. For duckdb or ibis, that means forcing polars installation. We should not do that.

Narwhals has struct.field, so we can do:

schema = df.collect_schema() fields = [f.name for f in schema[col].fields] df.with_columns( [nw.col(col).struct.field(f).alias(f) for f in fields] ).drop(col)

This approach also stays lazy. Pandas approach with json_serialize is correct.

Are you sure that all backends, which support struct schema would necessarily have struct.field?

struct.field is part of the narwhals' public api, so it should translate okay. We support pandas, polars, ibis, pyarrow and duckdb backends. Except pandas all others have struct.field support.

It might be worth adding tests though. Thoughts?

Yes, you are correct about narwahl's api.

create_test_dataframes is used for generating the dataframes for the expand dict tests.

To support duckdb, should i create a new test for expand dict or it doesn't matter as the tests are only supported for pandas, polars, ibis, pyarrow?

kirangadhave · 2026-06-30T06:18:01Z

+            # older versions of pandas running on py310 otherwise CI will fail
+            expanded = pd.json_normalize(
+                result_df.pop(transform.column_id).map(
+                    lambda value: {} if value is None else value


Also check for NaN here

kirangadhave · 2026-06-30T06:20:01Z

+            import pandas as pd
+
+            result_df = native_df.copy()
+            # max_level=0 was used so that pandas doesn't recursively unnest dicts


the comment is narrating the code, simplify to explain the why instead.

kirangadhave · 2026-06-30T06:22:52Z

+            why="to expand dict/struct columns for non-pandas backends"
+        )
+        polars_df = collected_df.to_polars()
+        unnested = polars_df.unnest(transform.column_id)


Duplicate column names after unnest will throw error here. Handle it gracefully.

feat: raising duplicate columns error test: adding necessary tests for duplicate columns

Shamik-07 · 2026-07-03T15:32:57Z

@kirangadhave

Thanks for the review comments.
I have addressed your review comments.
I still have a question, are you sure that all backends, which support struct schema would necessarily have struct.field?

kirangadhave

Thanks again for the changes. We will be good to merge after this set of changes. Especially the test related ones are must do.

kirangadhave · 2026-07-07T15:44:01Z

+            allow_none_equals_nan=True,
+        )
+
+    @pytest.mark.xfail(


xfail is used to mark tests where failure is expected. Raising error is the expected pass case here, so don't mark as xfail.

kirangadhave · 2026-07-07T15:44:46Z

+            # pandas and other backends agree on expand-dict behaviour.
+            expanded = pd.json_normalize(
+                result_df.pop(transform.column_id).map(normalise_empty_dict),
+                # type: ignore[arg-type]


type ignore comment on new line doesn't do anything

kirangadhave · 2026-07-07T15:45:49Z

+
+        schema = collected_df.collect_schema()
+        dtype = schema.get(transform.column_id)
+        fields = getattr(dtype, "fields", None)


use isinstance(dtype, nw.Struct) here?

kirangadhave · 2026-07-07T15:47:18Z

+                .select(expanded_columns)
+            )
+
+        raise nw.exceptions.InvalidOperationError(


Let's add a test for this

fix: checking if schema is of dtype struct for handle expand dict. tests: add separate tests for expand dict error. fix: remove xfail from expand dict test.

Shamik-07 · 2026-07-07T22:12:28Z

Thanks again for the changes. We will be good to merge after this set of changes. Especially the test related ones are must do.

Thank you again for the detailed review.

kirangadhave

🚀

kirangadhave · 2026-07-08T15:56:18Z

Thanks @Shamik-07 for your contribution and addressing multiple review rounds

addressed in later commits

Shamik-07 added 4 commits May 15, 2026 17:52

fix: using polars unnest for all backend types.

838d75f

fix: update the expand dict print statement.

851e5b1

tests: enabling the expand dict test and adding the necessary datafra…

0ea6758

…mes.

tests: removing ibis skip in expand dict test.

294e21e

Shamik-07 mentioned this pull request May 15, 2026

New dataframe transform: Polars Native Expand Dict Transformation #4583

Closed

vercel Bot deployed to Preview May 15, 2026 22:28 View deployment

cubic-dev-ai Bot reviewed May 15, 2026

View reviewed changes

Comment thread marimo/_plugins/ui/_impl/dataframes/transforms/handlers.py Outdated

vercel Bot deployed to Preview May 15, 2026 22:31 View deployment

mscolnick reviewed May 18, 2026

View reviewed changes

Comment thread tests/_plugins/ui/_impl/dataframes/test_handlers.py

mscolnick previously requested changes May 18, 2026

View reviewed changes

Shamik-07 added 3 commits May 19, 2026 17:45

Merge branch 'main' into fix/expand_dict_transformation

0d1e52f

fix: removing additional polars dataframe creation and using the none…

24f672b

… rows with the create test dataframes instead.

fix: using polars as optional in test handlers.

4e82164

refactor: adding None == NaN in assert frame equal with nans method to use it in expand_dict test.

vercel Bot deployed to Preview May 19, 2026 22:29 View deployment

Shamik-07 requested a review from mscolnick May 19, 2026 22:30

fix: processing pandas backend separately to not cause arrow coercion…

88ce223

… errors for mixed object columns.

vercel Bot deployed to Preview May 19, 2026 23:30 View deployment

cubic-dev-ai Bot reviewed May 19, 2026

View reviewed changes

Comment thread marimo/_plugins/ui/_impl/dataframes/transforms/handlers.py

Shamik-07 added 2 commits May 19, 2026 19:39

fix: mypy error.

1f722b8

fix: changing the expand dict print code function for pandas to using…

ff063e5

… json normalise following the handlers code.

vercel Bot deployed to Preview May 19, 2026 23:50 View deployment

Merge branch 'main' into fix/expand_dict_transformation

eb5ed5f

vercel Bot deployed to Preview May 20, 2026 21:22 View deployment

Merge branch 'main' into fix/expand_dict_transformation

6e16445

vercel Bot deployed to Preview May 21, 2026 21:14 View deployment

Merge branch 'main' into fix/expand_dict_transformation

12f341d

vercel Bot deployed to Preview June 2, 2026 22:12 View deployment

Merge branch 'main' into fix/expand_dict_transformation

de0e372

vercel Bot deployed to Preview June 3, 2026 21:10 View deployment

kirangadhave requested changes Jun 30, 2026

View reviewed changes

Merge branch 'main' into fix/expand_dict_transformation

65176f2

vercel Bot deployed to Preview July 2, 2026 20:04 View deployment

Shamik-07 added 2 commits July 3, 2026 11:17

fix: removed polars dependency in expand dict

2a349d4

feat: raising duplicate columns error test: adding necessary tests for duplicate columns

Merge branch 'main' into fix/expand_dict_transformation

992ad9f

vercel Bot deployed to Preview July 3, 2026 15:20 View deployment

fix: raising an error at the end of handle expand dict.

be23303

vercel Bot deployed to Preview July 3, 2026 15:32 View deployment

kirangadhave self-requested a review July 7, 2026 15:41

kirangadhave requested changes Jul 7, 2026

View reviewed changes

fix: review comments.

db9db85

fix: checking if schema is of dtype struct for handle expand dict. tests: add separate tests for expand dict error. fix: remove xfail from expand dict test.

vercel Bot deployed to Preview July 7, 2026 22:12 View deployment

Merge branch 'main' into fix/expand_dict_transformation

d06a797

vercel Bot deployed to Preview July 7, 2026 22:28 View deployment

kirangadhave enabled auto-merge (squash) July 8, 2026 15:55

kirangadhave approved these changes Jul 8, 2026

View reviewed changes

kirangadhave self-requested a review July 8, 2026 15:56

kirangadhave approved these changes Jul 8, 2026

View reviewed changes

kirangadhave merged commit dbcb600 into marimo-team:main Jul 8, 2026
40 checks passed

Shamik-07 deleted the fix/expand_dict_transformation branch July 8, 2026 19:50

kirangadhave mentioned this pull request Jul 8, 2026

fix: align expand-dict transform behavior across dataframe backends #10110

Merged

Uh oh!

Conversation

Shamik-07 commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Summary

📋 Pre-Review Checklist

✅ Merge Checklist

Uh oh!

vercel Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mscolnick left a comment

Choose a reason for hiding this comment

Uh oh!

Shamik-07 commented May 19, 2026

Uh oh!

Shamik-07 commented May 19, 2026

Uh oh!

Shamik-07 commented May 19, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle Report

Affected Assets, Files, and Routes:

Assets Changed:

Uh oh!

Shamik-07 commented Jun 2, 2026

Uh oh!

kirangadhave left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shamik-07 Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Shamik-07 commented Jul 3, 2026

Uh oh!

kirangadhave left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shamik-07 commented Jul 7, 2026

Uh oh!

kirangadhave left a comment

Choose a reason for hiding this comment

Uh oh!

kirangadhave commented Jul 8, 2026

Uh oh!

Shamik-07 commented May 15, 2026 •

edited

Loading

vercel Bot commented May 15, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

codecov Bot commented May 20, 2026 •

edited

Loading

Shamik-07 Jul 6, 2026 •

edited

Loading