libobs: Use RTLD_NOW to load modules #12575

gxalpha · 2025-09-02T09:41:57Z

Description

RTLD_LAZY means that symbols will only be resolved when first used, while RTLD_NOW tries to resolve them immediately. This means that if there are missing symbols (e.g, because a function got removed from libobs), dlopen with RTLD_LAZY will happily open that module but we get a runtime crash when the module tries to use that symbol, while with RTLD_NOW we instead get a (nicer) error on dlopen.

Motivation and Context

Crashes like the #12565 are annoying. It'd be nicer if we didn't load broken plugins in the first place.
Fixes #12565

How Has This Been Tested?

Tested on macOS 26. Testing from Linux contributors would be appreciated.

Tested with 0.9.4 of the audio-monitor plugin which still uses obs_frontend_add_dock.
Prior to this PR, we get a crash on startup (not ideal).
With this PR, the plugin fails to load, and we get an error message in the log:

error: os_dlopen(/Users/.../Library/Application Support/obs-studio/plugins/audio-monitor.plugin/Contents/MacOS/audio-monitor->/Users/.../Library/Application Support/obs-studio/plugins/audio-monitor.plugin/Contents/MacOS/audio-monitor): dlopen(/Users/.../Library/Application Support/obs-studio/plugins/audio-monitor.plugin/Contents/MacOS/audio-monitor, 0x0106): Symbol not found: _obs_frontend_add_dock
  Referenced from: <3BF6DC09-1B39-30D5-98B5-33D6A4D46655> /Users/.../Library/Application Support/obs-studio/plugins/audio-monitor.plugin/Contents/MacOS/audio-monitor
  Expected in:     <A6A272D8-83B5-3DA7-A67B-D7C2C92AE81B> /Users/.../obs-studio/build_macos/frontend/Debug/OBS.app/Contents/Frameworks/obs-frontend-api.dylib
warning: Module '/Users/.../Library/Application Support/obs-studio/plugins/audio-monitor.plugin/Contents/MacOS/audio-monitor' not loaded
debug: Failed to load module file '/Users/.../Library/Application Support/obs-studio/plugins/audio-monitor.plugin/Contents/MacOS/audio-monitor', file not found

Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

My code has been run through clang-format.
I have read the contributing document.
My code is not on the master branch.
The code has been tested.
All commit messages are properly formatted and commits squashed where appropriate.
I have included updates to all appropriate documentation.

tytan652 · 2025-09-02T10:29:38Z

Since has_qt5_dependency was not removed, you need to make module_has_qt5_check use dlopen with RTLD_LAZY rather than os_dlopen.

tytan652 · 2025-09-02T10:48:20Z

~~Also the plugin being not loaded does not show the plugin load error prompt, load_all_callback may need a change to MODULE_FILE_NOT_FOUND (mismatching macro name now) case to a goto.~~

Edit: But MODULE_MISSING_EXPORTS is a return too and need to stay like this…

gxalpha · 2025-09-02T11:01:28Z

Also the plugin being not loaded does not show the plugin load error prompt

This is currently expected and matches Windows. A proper notification (on all platforms) would be nice but is out-of-scope for this PR.

gxalpha · 2025-09-02T14:08:09Z

Since has_qt5_dependency was not removed, you need to make module_has_qt5_check use dlopen with RTLD_LAZY rather than os_dlopen.

Potentially dumb question, but is the Qt 5 check even needed with RTLD_NOW? I imagine plugins linking against Qt 5 should be rejected by dlopen now?
The behavioral change would be that the message no longer appears, but I think that'd be fine since there shouldn't be that many plugins anymore using Qt 5, so the impact would be rather small.

tytan652 · 2025-09-02T14:49:43Z

On Linux outside of our Flatpak (e.g. Ubuntu), you can end up with Qt 5 and 6 installed side by side.
And you must avoid loading both of them (this creates a mess and qt5ct was a good example) and this check is there because we did not break ABI through SOVERSION when switching to Qt 6.
If you want to remove the check you need #12566 first to fail finding an old libobs.

PatTheMav · 2025-09-02T15:24:19Z

While RTLD_LAZY is the preferred default, RTLD_NOW can indeed serve as an additional safety-net as we're loading unchecked 3rd party code. Initial loading will be slower (though the brute force speed of modern architectures will mask this quite a bit) though any consecutive symbol lookup will be faster.

RTLD_LAZY means that symbols will only be resolved when first used, while RTLD_NOW tries to resolve them immediately. This means that if there are missing symbols (e.g, because a function got removed from libobs), dlopen with RTLD_LAZY will happily open that module but we get a runtime crash when the module tries to use that symbol, while with RTLD_NOW we instead get a (nicer) error on dlopen.

gxalpha · 2025-09-02T18:18:27Z

@tytan652 I've changed os_dlopen to dlopen with RTLD_LAZY. Would be nice if you could take a look and/or test this as I have no way to (especially whether the .so addition from os_dlopen is required?).

tytan652 · 2025-09-02T18:37:22Z

Would be nice if you could take a look and/or test this as I have no way to (especially whether the .so addition from os_dlopen is required?).

The suffix check is mainly meant for os_dlopen cross-platform use outside of load_all_callback/obs_open_module, so it should be good.

RytoEX · 2025-09-02T19:39:19Z

Since has_qt5_dependency was not removed, you need to make module_has_qt5_check use dlopen with RTLD_LAZY rather than os_dlopen.

Potentially dumb question, but is the Qt 5 check even needed with RTLD_NOW? I imagine plugins linking against Qt 5 should be rejected by dlopen now? The behavioral change would be that the message no longer appears, but I think that'd be fine since there shouldn't be that many plugins anymore using Qt 5, so the impact would be rather small.

We had actually planned to remove the Qt 5 check for this release, and may still before RC/stable. It was put into place around OBS Studio 28 three years ago for the transition from Qt 5 to Qt 6. We expect that the number of users still having Qt5-based OBS Studio plugins should be incredibly small. cc @kkartaltepe

gxalpha · 2025-09-02T19:43:11Z

Should I just undo the changes made to the Linux Qt 5 check here then?

RytoEX · 2025-09-02T19:43:27Z

Context, we used to always use only RTLD_LAZY, but switched to RTLD_LAZY | RTLD_FIRST for macOS in #956 (a66ad7e). Will defer to @PatTheMav on the changes here.

PatTheMav · 2025-09-03T13:10:40Z

Context, we used to always use only RTLD_LAZY, but switched to RTLD_LAZY | RTLD_FIRST for macOS in #956 (a66ad7e). Will defer to @PatTheMav on the changes here.

As mentioned above, using RTLD_NOW is fine, indeed as the man page mentions:

Note: With chained-fixups (the default mach-o format since macOS 12 and iOS 15) using RTLD_LAZY or RTLD_NOW has no effect, as all symbols are immediately bound.

So for any binary built with the macOS 12 deployment target (which plugins won't all be) the effect will be the same as if RTLD_NOW had been used.

RTLD_FIRST influences the behaviour of dlsym to ensure that a symbol is only resolved within the image loaded initially by dlopen and to ignore any matching symbols that might have come into existence since then, which also seems correct for loading plugins.

PatTheMav

macOS changes should be fine, Linux changes probably as well but defer to Linux maintainers to approve that.

tytan652

Tested on my machine

RytoEX · 2025-09-03T19:34:00Z

Should I just undo the changes made to the Linux Qt 5 check here then?

We can just leave it in, and deal with it in the revert/removal.

gxalpha force-pushed the rtld_now branch from 835e420 to 4f0ba48 Compare September 2, 2025 18:16

RytoEX requested review from kkartaltepe and PatTheMav September 2, 2025 19:41

RytoEX self-assigned this Sep 2, 2025

RytoEX added the Bug Fix Non-breaking change which fixes an issue label Sep 2, 2025

RytoEX added this to the OBS Studio 32.0 milestone Sep 2, 2025

PatTheMav approved these changes Sep 3, 2025

View reviewed changes

tytan652 approved these changes Sep 3, 2025

View reviewed changes

RytoEX merged commit 6242913 into obsproject:master Sep 3, 2025
15 checks passed

gxalpha deleted the rtld_now branch September 3, 2025 20:33

PatTheMav mentioned this pull request Sep 4, 2025

OBS 32 w/ Streamdeck - Segfault on mac #12576

Open

gxalpha mentioned this pull request Sep 4, 2025

libobs: Improve plugin load failure handling #12588

Open

6 tasks

Uh oh!

libobs: Use RTLD_NOW to load modules #12575

libobs: Use RTLD_NOW to load modules #12575

Uh oh!

Conversation

gxalpha commented Sep 2, 2025

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

Uh oh!

tytan652 commented Sep 2, 2025

Uh oh!

tytan652 commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gxalpha commented Sep 2, 2025

Uh oh!

gxalpha commented Sep 2, 2025

Uh oh!

tytan652 commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PatTheMav commented Sep 2, 2025

Uh oh!

gxalpha commented Sep 2, 2025

Uh oh!

tytan652 commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RytoEX commented Sep 2, 2025

Uh oh!

gxalpha commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RytoEX commented Sep 2, 2025

Uh oh!

PatTheMav commented Sep 3, 2025

Uh oh!

PatTheMav left a comment

Choose a reason for hiding this comment

Uh oh!

tytan652 left a comment

Choose a reason for hiding this comment

Uh oh!

RytoEX commented Sep 3, 2025

Uh oh!

Uh oh!

Uh oh!

tytan652 commented Sep 2, 2025 •

edited

Loading

tytan652 commented Sep 2, 2025 •

edited

Loading

tytan652 commented Sep 2, 2025 •

edited

Loading

gxalpha commented Sep 2, 2025 •

edited

Loading