Skip to content

Unboxed Internal list/tuple/dict/containers representation for use within CPython #729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Fidget-Spinner opened this issue May 9, 2025 · 1 comment

Comments

@Fidget-Spinner
Copy link
Collaborator

So it seems it's not possible to make ob_item in lists and tuples use _PyHeapRef or store an untagged int directly without breaking a whole lot of extensions capi-workgroup/decisions#64.

A possible alternative that maintains compatibility is to use an internal representation of a list, perhaps called _PyInternalListObject. This would be exactly the same as PyListObject, except it uses a _PyHeapRef * for ob_item.

When passing the object to external C code, we convert it to a normal PyListObject by simply swapping out the unboxed ints within the PyListObject to boxed ones, then passing it to the external code. This means _PyInternalListObject has a compatible memory layout with PyListObject. Conversion is only done once and in-place. So this is fast and efficient.

We can minimize the number of conversions required by relying on the specializing interpreter to tell us when we're calling out to unknown C functions.

To keep stackref conversion quick, we should reserve one bit meaning an "internal container". It seems 10 is still unused right now, and it's perfect for that. x1 represents "skip the refcount", which we don't want (we do want to refcount internal containers!). So the 10 bit can now mean "internal container" (meaning list, tuples, or dicts). After conversion, the tag just becomes 00.

@Fidget-Spinner Fidget-Spinner changed the title Internal list/tuple representation for use within CPython Unboxed Internal list/tuple representation for use within CPython May 9, 2025
@Fidget-Spinner Fidget-Spinner changed the title Unboxed Internal list/tuple representation for use within CPython Unboxed Internal list/tuple/dict/containers representation for use within CPython May 9, 2025
@Fidget-Spinner
Copy link
Collaborator Author

@markshannon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant