Skip to content

gh-136421: Load _datetime static types during interpreter initialization #136583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Jul 21, 2025
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Include/internal/pycore_pylifecycle.h
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ extern PyStatus _Py_HashRandomization_Init(const PyConfig *);

extern PyStatus _PyGC_Init(PyInterpreterState *interp);
extern PyStatus _PyAtExit_Init(PyInterpreterState *interp);
extern PyStatus _PyDateTime_InitTypes(PyInterpreterState *interp);

/* Various internal finalizers */

Expand Down
19 changes: 19 additions & 0 deletions Lib/test/datetimetester.py
Original file line number Diff line number Diff line change
Expand Up @@ -3651,6 +3651,25 @@ def test_repr_subclass(self):
td = SubclassDatetime(2010, 10, 2, second=3)
self.assertEqual(repr(td), "SubclassDatetime(2010, 10, 2, 0, 0, 3)")

@support.cpython_only
def test_concurrent_initialization(self):
# Run in a subprocess to ensure we get a clean version of _datetime
script = """if True:
from concurrent.futures import InterpreterPoolExecutor

def func():
import _datetime
print('a', end='')

with InterpreterPoolExecutor() as executor:
for _ in range(8):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's up to you, but range(10) is probably better than range(8) for the Windows x64 CI to reproduce the issue.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, I think this is fine. Many other systems have 8 cores, not 10.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, but it sounds like you are talking about the max_workers argument rather than the submit count here.

Copy link
Member Author

@ZeroIntensity ZeroIntensity Jul 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

max_workers just chooses the number of threads on the system when not set. We could try to submit os.cpu_count() number of futures, but we don't need to overcomplicate this; we just need something that stresses several subinterpreters trying to import _datetime concurrently.

executor.submit(func)
"""
rc, out, err = script_helper.assert_python_ok("-c", script)
self.assertEqual(rc, 0)
self.assertEqual(out, b"a" * 8)
self.assertEqual(err, b"")


class TestSubclassDateTime(TestDateTime):
theclass = SubclassDatetime
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix crash when initializing :mod:`datetime` concurrently.
2 changes: 2 additions & 0 deletions Modules/Setup.bootstrap.in
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ posix posixmodule.c
_signal signalmodule.c
_tracemalloc _tracemalloc.c
_suggestions _suggestions.c
# needs libm and on some platforms librt
_datetime _datetimemodule.c

# modules used by importlib, deepfreeze, freeze, runpy, and sysconfig
_codecs _codecsmodule.c
Expand Down
3 changes: 0 additions & 3 deletions Modules/Setup.stdlib.in
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,6 @@
@MODULE_CMATH_TRUE@cmath cmathmodule.c
@MODULE__STATISTICS_TRUE@_statistics _statisticsmodule.c

# needs libm and on some platforms librt
@MODULE__DATETIME_TRUE@_datetime _datetimemodule.c

# _decimal uses libmpdec
# either static libmpdec.a from Modules/_decimal/libmpdec or libmpdec.so
# with ./configure --with-system-libmpdec
Expand Down
17 changes: 5 additions & 12 deletions Modules/_datetimemodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#include "pycore_object.h" // _PyObject_Init()
#include "pycore_time.h" // _PyTime_ObjectToTime_t()
#include "pycore_unicodeobject.h" // _PyUnicode_Copy()
#include "pycore_initconfig.h" // _PyStatus_OK()

#include "datetime.h"

Expand Down Expand Up @@ -7329,13 +7330,9 @@ clear_state(datetime_state *st)
}


static int
init_static_types(PyInterpreterState *interp, int reloading)
PyStatus
_PyDateTime_InitTypes(PyInterpreterState *interp)
{
if (reloading) {
return 0;
}

// `&...` is not a constant expression according to a strict reading
// of C standards. Fill tp_base at run-time rather than statically.
// See https://bugs.python.org/issue40777
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR does not address the possible races in PyDateTime_*.tp_base = &PyDateTime_*Type; below, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess not. I don't think there's an easy way to do this here, because atomically storing tp_base will continue to race with all the non-atomic reads elsewhere.

Do we even need to load it at runtime like this? We have other examples of directly storing it in the PyTypeObject structure:

.tp_base = &PyCFunction_Type,

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not needed now because the module is statically linked, that issue happens only with dynamic loaded modules so you can define it statically now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess a _Py_IsMainInterPreter() check will suffice in this PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not needed now because the module is statically linked, that issue happens only with dynamic loaded modules so you can define it statically now.

Ah, TIL. That's definitely the best option here then.

I guess a _Py_IsMainInterPreter() check will suffice in this PR?

For what?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For ensuring tp_base is set only once after Py_Initialize(). But I missed the Kumar 's comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove PyDateTime_DateTimeType.tp_base = &PyDateTime_DateType; as well.

Expand All @@ -7347,11 +7344,11 @@ init_static_types(PyInterpreterState *interp, int reloading)
for (size_t i = 0; i < Py_ARRAY_LENGTH(capi_types); i++) {
PyTypeObject *type = capi_types[i];
if (_PyStaticType_InitForExtension(interp, type) < 0) {
return -1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not for this PR but as a follow up I think it would be better to now remove _PyStaticType_InitForExtension and just use _PyStaticType_InitBuiltin for it, there's a lot of special casing that could be removed.

return _PyStatus_ERR("could not initialize static types");
}
}

return 0;
return _PyStatus_OK();
}


Expand Down Expand Up @@ -7379,10 +7376,6 @@ _datetime_exec(PyObject *module)
}
/* We actually set the "current" module right before a successful return. */

if (init_static_types(interp, reloading) < 0) {
goto error;
}

for (size_t i = 0; i < Py_ARRAY_LENGTH(capi_types); i++) {
PyTypeObject *type = capi_types[i];
const char *name = _PyType_Name(type);
Expand Down
1 change: 1 addition & 0 deletions PCbuild/_freeze_module.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@
</ItemGroup>
<ItemGroup>
<ClCompile Include="..\Modules\atexitmodule.c" />
<ClCompile Include="..\Modules\_datetimemodule.c" />
<ClCompile Include="..\Modules\faulthandler.c" />
<ClCompile Include="..\Modules\gcmodule.c" />
<ClCompile Include="..\Modules\getbuildinfo.c" />
Expand Down
5 changes: 5 additions & 0 deletions Python/pylifecycle.c
Original file line number Diff line number Diff line change
Expand Up @@ -760,6 +760,11 @@ pycore_init_types(PyInterpreterState *interp)
return status;
}

status = _PyDateTime_InitTypes(interp);
if (_PyStatus_EXCEPTION(status)) {
return status;
}
Comment on lines +763 to +766
Copy link
Contributor

@neonene neonene Jul 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
status = _PyDateTime_InitTypes(interp);
if (_PyStatus_EXCEPTION(status)) {
return status;
}
if (!_Py_IsMainInterpreter(interp)) {
status = _PyDateTime_InitTypes(interp);
if (_PyStatus_EXCEPTION(status)) {
return status;
}
}

Reply from #136620 (comment)

Can you run test_concurrent_initialization() with this change? Based on the crashes that come from the change, I said this PR and my example are "almost equivalent." I'm not sure right now what this PR ensures.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, what are you trying to achieve here? This will just break the types for the main interpreter.

Copy link
Contributor

@neonene neonene Jul 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will just break the types for the main interpreter.

Note that test_concurrent_initialization() does not load the _datetime in the main inter interpreter at all.

Correction: Run the script of the test without running test_datetime.


return _PyStatus_OK();
}

Expand Down
Loading