Support ldc2.conf as a directory #4954

the-horo · 2025-06-16T10:43:31Z

Short summary of changes:

ldc2.conf can now be a directory
cmake files have been changed to take advantage of this during the build
=> easier to configure independent sections of settings
Support installing the config files as a directory
Split the settings across compiler and runtime config files
Compiler contains the -I includes, wasm flags and compiler-rt libdir
Runtime contains phobos&druntime libdir and rpath
=> The runtime and the compiler can now be installed as separate projects without their config files colliding/being skipped
Change the runtime config files for the native target to also include the triple like for multilib (e.g. x86_64-.*-linux-gnu instead of "default").
=> ldc-build-runtime can now install cross-builds alongside their configuration files (it skipped the config previously to avoid overwriting the host one), removing the need for post-install interaction from the user.

kinke · 2025-06-16T16:45:48Z

driver/configfile.cpp

+
+    llvm::sort(configFiles, [](const std::string &a, const std::string &b) {
+      auto sa = llvm::StringRef(a), sb = llvm::StringRef(b);
+      return sa.compare_numeric(sb) < 0;


Thx for the (lot of) work! - This was the 1st thing I was curious about, how to define the order. Such a 'smart' sort with a default number prefix sounds good to me. 👍

kinke · 2025-06-16T16:47:59Z

The current CI errors suggest an incomplete config for the built (non-installed) compiler. Let me know if you need help.

the-horo · 2025-06-16T16:54:17Z

The current CI errors suggest an incomplete config for the built (non-installed) compiler. Let me know if you need help.

Yup. I didn't think of testing that so I guess I did forget about something... It did work when installing so surely it can't be that broken.

the-horo · 2025-06-16T17:55:54Z

runtime/CMakeLists.txt

+if(PHOBOS2_DIR)
+    set(build_defaultlib "phobos2-ldc,druntime-ldc")
+else()
+    set(build_defaultlib "druntime-ldc")
 endif()


Is there any situation in which PHOBOS2_DIR doesn't exist? To fix the android failures I would need to slightly adjust this and I want to know if I can simplify the code a little bit?

I guess we can drop that special case of a Phobos-less build. I guess the most likely scenario is that someone forgot to initialize the git submodule before running CMake, not intentionally excluding Phobos from the build.

the-horo · 2025-06-16T18:02:33Z

FAILED: etc/ldc2_install.conf D:/a/ldc/bootstrap-ldc/etc/ldc2_install.conf 
C:\Windows\system32\cmd.exe /C "cd /D D:\a\ldc\bootstrap-ldc\etc\ldc2_install.conf.d && sh -c "output=\"${1}\"; shift; cat \"${@}\" > \"${output}\"" script D:/a/ldc/bootstrap-ldc/etc/ldc2_install.conf 30-ldc-runtime-lib.conf 35-ldc-compiler.conf 40-ldc-wasm.conf"
cat: 30-ldc-runtime-lib.conf: No such file or directory

Are those cmake quotes interfering with my command's quotes or is there some other reason it's failing?

kinke · 2025-06-16T18:12:58Z

Are those cmake quotes interfering with my command's quotes or is there some other reason it's failing?

Not sure, I'd need to check locally. In general, using the sh and cat tools isn't ideal on Windows, as those are usually NOT in PATH (but bundled with e.g. git).

Edit: I guess the error is earlier, when trying to invoke a .sh script to finalize the 30-ldc-runtime-lib.conf.in to .conf IIUC. AFAIK, that requires an explicit bash dir/file.sh … invocation on Windows, at least with a default cmd.exe shell.

JohanEngelen · 2025-06-16T18:28:53Z

FYI: instead of many separate small test files, you can use split-file. See: https://llvm.org/docs/TestingGuide.html#extra-files
That would make it much more clear what is being tested (no need to open 4 different files to see the full test case).

the-horo · 2025-06-16T18:32:32Z

FYI: instead of many separate small test files, you can use split-file. See: https://llvm.org/docs/TestingGuide.html#extra-files That would make it much more clear what is being tested (no need to open 4 different files to see the full test case).

Ohh, that looks useful

the-horo · 2025-06-16T18:52:57Z

Edit: I guess the error is earlier, when trying to invoke a .sh script to finalize the 30-ldc-runtime-lib.conf.in to .conf IIUC. AFAIK, that requires an explicit bash dir/file.sh … invocation on Windows, at least with a default cmd.exe shell.

You're right:

[148/212] Generating etc/ldc2_install.conf
 Volume in drive D is Temporary Storage
 Volume Serial Number is 0C26-1B92
 Directory of D:\a\ldc\bootstrap-ldc\etc\ldc2_install.conf.d
06/16/2025  06:47 PM    <DIR>          .
06/16/2025  06:47 PM    <DIR>          ..
06/16/2025  06:47 PM               279 30-ldc-runtime-lib.conf.in
06/16/2025  06:47 PM               180 35-ldc-compiler.conf
06/16/2025  06:47 PM               237 40-ldc-wasm.conf
               3 File(s)            696 bytes
               2 Dir(s)  153,611,042,816 bytes free

It seems that cmake is content to let the fill-multilib-section POST_BUILD command silently fail on windows

the-horo · 2025-06-16T20:11:32Z

Ok, one test failure for tomorrow me to go through. Other things that I thought of that could be useful:

Add back the help text in the config files as now they're just settings
Hook up the android cross-builds to use CONF_PREFER_DIR
Not sure, I'd need to check locally. In general, using the sh and cat tools isn't ideal on Windows, as those are usually NOT in PATH (but bundled with e.g. git).

I think I change the post build commands to depend only on cmake but we're still left with fill-multilib-section.sh which uses bash, grep and sed. I know that some of the druntime integration tests use the same commands so maybe it's fine as a dependency?
Maybe it's worth adding a -dumpmachine switch like clang has that only prints the target triple to ease the processing that is carried out by the .sh script.

kinke · 2025-06-16T20:30:31Z

Add back the help text in the config files as now they're just settings

Ah yeah that'd be good.

I think I change the post build commands to depend only on cmake but we're still left with fill-multilib-section.sh which uses bash, grep and sed. I know that some of the druntime integration tests use the same commands so maybe it's fine as a dependency?

Hmm; I think the subset of people running tests is way smaller than those just building. And for those druntime integration tests, we set PATH using a hardcoded assumption:

ldc/dmd/osmodel.mak

Lines 60 to 66 in 3acd761

    
           # set up bash shell on Windows 
        
           ifeq (windows,$(OS)) 
        
             # Note: setting SHELL to an absolute path to bash.exe does NOT suffice. 
        
             # The GNU tools like {rm,cp,mkdir}.exe need to be in PATH. 
        
             ifeq (,$(findstring C:\Program Files\Git\usr\bin,$(PATH))) 
        
               export PATH:=C:\Program Files\Git\usr\bin;$(PATH) 
        
             endif

Maybe it's worth adding a -dumpmachine switch like clang has that only prints the target triple to ease the processing that is carried out by the .sh script.

I still need to check your rationale for switching to explicit triples for the default target. The main problem I saw back then is that there's no clear 1:1 mapping to the normalized triple (the one with the vendor) - e.g. the vendor seems uninteresting for all but Apple targets (and can be anything from empty to none and unknown...), a trailing OS version suffix should be masked out, similarly a subarch (aarch64v9a), ... So converting the default normalized triple to a suitable pattern seems hard.

Edit: Another example: the (libc) environment. IIRC, one can use the prebuilt Alpine package (targeting x86_64-linux-musl by default and bundling those libs) with -mtriple=x86_64-linux-gnu to 'cross'-compile+link to glibc on a glibc host, linking static druntime and Phobos prebuilt for musl. At least for a simple test that I did. So IMO the least surprising behavior is to fall back to the default target libs for all triples unless the config has an according overriding section.

the-horo · 2025-06-17T10:00:39Z

Edit: Another example: the (libc) environment. IIRC, one can use the prebuilt Alpine package (targeting x86_64-linux-musl by default and bundling those libs) with -mtriple=x86_64-linux-gnu to 'cross'-compile+link to glibc on a glibc host, linking static druntime and Phobos prebuilt for musl. At least for a simple test that I did.

I don't agree with this point because:

statically linking against musl does not solve every ABI compatibility your programs/libraries may have
The stdlibs are not even linked statically, what you're describing is substituting the libc undefined symbols in the druntime&phobos of the alpine build with the hopefully existent functions from the glibc you're building against, and hoping that they have the same arguments (otherwise you get UB) and have the same semantics (otherwise you get logic errors).

I couldn't find an example of functions having different signatures but for different behavior just pick something from https://wiki.musl-libc.org/functional-differences-from-glibc.html. Here's an example with basename:

// b.c
#include <libgen.h>
#include <string.h>

int always_true(void) {
	char arr[] = "/lib/foo/";
	char *result = basename(arr);
	return strcmp(result, "foo") == 0;
}

// a.c
#include <unistd.h>
#include <stdio.h>

extern int always_true(void);

int main (int argc, char **argv) {
	printf("1 == %d\n", always_true());
}

If you compile both of those files on either musl or glibc you get the correct result:

musl /tmp # gcc -o a a.c b.c && ./a
1 == 1
glibc /tmp # gcc -o a a.c b.c && ./a
1 == 1

If you compile b.c on musl and try to link it on glibc you get the wrong results:

glibc /tmp # gcc -o a a.c b-musl.o && ./a
1 == 0

With the comment that this behavior in not specified so it can change at anytime (and even break).

For this reason, at least when it comes to the cenv part of the triple, it is not safe to normalize it away.

I have to think about the version components of the OS and ARCH fields and see if I can draw a conclusion, otherwise

So IMO the least surprising behavior is to fall back to the default target libs for all triples unless the config has an according overriding section.

I'll go with your suggestion here. It's not that important to encode the triple, it only makes the code a little shorter and marginally helps ldc-build-runtime as the cross-config file can be installed automatically. If the target triple were to stay default then ldc-build-runtime would just need to replace the string accordingly so no biggie.

kinke · 2025-06-17T12:51:18Z

Yeah I didn't mean to encourage anything like that, just saying that some guys might already use such stuff, regardless of whether by accident or knowing what they are doing. The env can be versioned too - androideabi29 (32-bit), android30 (64-bit)… it's all very fuzzy.

the-horo · 2025-06-17T16:48:34Z

Ok, I added back the documentation (though perhaps a little bit too much for a config file) and made the cat portable on windows. Still unsure what to do with the default triple situation, it sounds most reasonable to skip the last commit and keep "default".

the-horo · 2025-06-18T08:11:35Z

Changed the documentation file to only be installed with the compiler, not with every runtime build.

the-horo · 2025-06-18T13:17:47Z

Tests passed but I'm looking over the artifacts and checking that the config files aren't broken. The android one seems to be so I'll have to fix it

the-horo · 2025-06-18T15:26:44Z

The android config now looks fine and I've changed it to be a directory in the release artifacts. The test failure is a timeout

cmake/Modules/LdcCommon.cmake

CMakeLists.txt

kinke · 2025-06-21T10:09:47Z

driver/configfile.d

+ - rpath
+/
+void validateSettingNames(const GroupSetting group, const char* filePath) {
+    static fail(const Setting setting, const char* filePath) {


[Omitting the return type is rather unusual.]

kinke · 2025-06-21T10:11:50Z

driver/configfile.d

+void validateSettingNames(const GroupSetting group, const char* filePath) {
+    static fail(const Setting setting, const char* filePath) {
+        string fmt(Setting.Type type) {
+            import std.traits : EnumMembers;


We try to avoid using Phobos in the compiler. This should be doable via __traits(allMembers, T) too.

kinke · 2025-06-21T10:15:51Z

driver/configfile.d

            const rpath = findScalarSetting(sections, "rpath");
-            this.rpathcstr = rpath.length == 0 ? null : rpath.replacePlaceholders(cfgPaths).toCString.ptr;
+            if (rpath.length)


So one cannot unset it anymore with an empty value in a later section?

Fixed this (in the D code as well as the cmake code) and I've added a test case that checks that rpath = "" resets the rpath.

ldc2.conf.skel

kinke · 2025-06-21T10:22:16Z

runtime/CMakeLists.txt


+list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR} "${PROJECT_PARENT_DIR}/cmake/Modules")
+set(LDC_SOURCE_ROOT "${PROJECT_PARENT_DIR}")


Seems unused, only defined in 2 locations?

kinke · 2025-06-21T10:26:02Z

runtime/CMakeLists.txt


 # Only have either shared or static libs?
 # Then explicitly default to linking against them via default LDC switch.
 if(${BUILD_SHARED_LIBS} STREQUAL "ON")
-    set(ADDITIONAL_DEFAULT_LDC_SWITCHES "${ADDITIONAL_DEFAULT_LDC_SWITCHES}\n        \"-link-defaultlib-shared\",")


Okay so no more support for ADDITIONAL_DEFAULT_LDC_SWITCHES - I did use it in the past sometimes, mainly for workarounds for some CI platforms IIRC. Should possibly be mentioned in the changelog entry.

You can get the same behavior with:

if(ON_DUMMY_PLATFORM) makeConfSection(NAME "80-fix-platform" SECTION default SWITCHES ${ADDITIONAL_DEFAULT_LDC_SWITCHES}) endif()

or, if doing it in the CI scripts:

cat <<EOF > etc/ldc2.conf/80-fix-platform.conf .... EOF

I'll mention this in the CHANGELOG

[Heh not exactly - those were previously part of the switches in the first default section, so could still be overridden for specific targets later. But yeah, adding a file would be the new way of doing this.]

Edit: Just a matter of choosing a proper order number prefix for the new file, np at all.

runtime/ldc-build-runtime.d.in

tests/driver/config_diag.d

kinke · 2025-06-21T10:43:14Z

Note that I'd be okay with generating the ldc2.conf directory exclusively now via CMake, so no more support for the single file (as part of the CMake build - but still supported by the compiler). There are breaking changes already, such as the now unused *.conf.in files and dropped ADDITIONAL_DEFAULT_LDC_SWITCHES, so I don't see why we couldn't switch to the directory too.

Not sure how much that would simplify. I can take care of the CI packages, would want to e.g. redo the macOS package configs anyway, now somewhat easier by adding separate files instead of appending multiple sections.

the-horo · 2025-06-21T10:58:07Z

Note that I'd be okay with generating the ldc2.conf directory exclusively now via CMake, so no more support for the single file (as part of the CMake build - but still supported by the compiler).

As in remove the CONF_PREFER_DIR=OFF code?

kinke · 2025-06-21T10:58:54Z

CMakeLists.txt

+# LLD 8+ requires (new) `--export-dynamic` for WebAssembly (https://github.com/ldc-developers/ldc/issues/3023).
+list(APPEND wasm_switches -L--export-dynamic)
+
+makeConfSection(NAME "40-ldc-wasm"


I think we should decrease the order of this - these are just some wasm defaults. The main 'problem' is that we disable linking druntime and Phobos via -defaultlib= - but that actually only applies to naked wasm, not e.g. for emscripten environments, where we would ideally have and link druntime at least.

With the much greater flexibility now, we could add 2 wasm sections, and e.g. restrict the -defaultlib= (and empty lib-dirs) to a ^wasm.*-(none|unknown)$ section or so.

[just an idea for a follow-up PR perhaps, not here]

If ldc2.conf is a directory go through each file (directly) inside it, ordered by name and apply the settings in it. It would be similar to appending all the files and treating it like a normal config file. The specific ordering is the one from llvm::StringRef::compare_numeric. The previous behavior was to always error out if ldc2.conf was a directory, being it passed through `-conf` or being automatically picked up. To account for this change the following restrictions that affect config files have been lifted: - at least one section must match the current triple This is undesirable for cross-compile setups in which one config file is dedicated to one triple - both switches and post-switches must be defined This is undesirable for config files that simply append a CLI switch The above have been replaced by a check that all settings represent valid keys (so switches, post-switches etc.) and that they are of the correct type. Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>

kinke · 2025-06-21T11:16:12Z

As in remove the CONF_PREFER_DIR=OFF code?

Exactly. Again, not sure how much that would simplify the code; you already found a way for the portable concat of all files to a single one. But I suspect it'd be worth it to reduce CMake complexity. In case a package maintainer prefers the single file, he/she can presumably insert the cat one-liner somewhere in the build process.

kinke · 2025-06-21T12:38:14Z

I can take care of the CI packages, would want to e.g. redo the macOS package configs anyway, now somewhat easier by adding separate files instead of appending multiple sections.

I'll prepare a PR where I'm switching to CONF_PREFER_DIR=ON by default (so incl. CI builds), and adapt the CI packaging accordingly. So that CI won't be a blocker to remove CONF_PREFER_DIR altogether. And we should e.g. immediately see simplifications for the existing configs-merging of the windows-multilib package.

the-horo · 2025-06-21T12:44:09Z

I'll prepare a PR where I'm switching to CONF_PREFER_DIR=ON by default (so incl. CI builds), and adapt the CI packaging accordingly. So that CI won't be a blocker to remove CONF_PREFER_DIR altogether. And we should e.g. immediately see simplifications for the existing configs-merging of the windows-multilib package.

I've tried to guess-adapt the windows and macos merge actions, let's see what fails

kinke · 2025-06-21T13:34:27Z

.github/actions/merge-macos/action.yml

+        # 2) arm64 ios config
+        cp ../ldc2-arm64/etc/ldc2.conf/31-ldc-runtime-lib-ios-arm64.conf .
+        # 3) x86_64 & arm64 macos
+        for arch in x86_64 arm64; do


We should probably remove the 30-ldc-runtime-lib.conf file - AFAIK, that contains default: … with the x86_64 macOS libs. The special thing about the macOS universal package is that it's used by 2 compilers - the native x86_64 and native arm64 ones, merged into the universal bin/ldc2 executable. That's why we don't have a clear default target, and have these x86_64/arm64 sections instead.

Yup, I've accidentally removed the rm line that I've added while editing the file :(

I'm somewhat unsatisfied with the 30-ldc-runtime and 31-ldc-runtime inconsistent naming scheme but I guess I can't do anything unless the triple situation changes

We can tweak the naming scheme anytime. E.g., the current 30 config part is more than just the runtime libs, at least for the iOS configs - when cross-compiling, we normally need to tweak the used C compiler for linking and preprocessing too, either via -gcc or adding extra -Xcc flags. So I'd see this as target-specific group, where I think coupling both OS and arch makes sense (i.e., no need for a 30 OS group, and a separate 40 arch group or so).

E.g.:

// leaving space for user pre-init files... 30-compiler.conf: default switches, post-switches (mainly `-defaultlib` and `-I` with bundled druntime/Phobos modules) 50-target.conf: default target settings (lib-dir, rpath, extra switches) 55-target-wasm.conf 55-target-ios-arm64.conf …

The compiler-rt path would logically belong to the target group, but is only known when building the compiler. We could e.g. create a 70-compiler-rt.conf as part of the compiler build, appending the compiler-rt libdir for default (i.e., all targets). So that a user can append a better one earlier in the 50-target group, and/or fix up the section name in 70-compiler-rt.conf to only apply to the default target (by matching its triple).

Or what would also work: 51-target-compiler-rt.conf, appending the lib-dir for default. And the overrides in 55 then reset lib-dirs anyway.

kinke · 2025-06-21T13:39:11Z

runtime/ldc-build-runtime.d.in

@@ -159,7 +160,6 @@ void runCMake() {
        "-DLDMD_EXE_FULL=" ~ ldmdExecutable,
        "-DDMDFE_MINOR_VERSION=@DMDFE_MINOR_VERSION@",
        "-DDMDFE_PATCH_VERSION=@DMDFE_PATCH_VERSION@",
-        "-DLDC_WITH_LLD=@LDC_WITH_LLD@",
        "-DINCLUDE_INSTALL_DIR=@INCLUDE_INSTALL_DIR@",


I think we don't need this anymore.

yeah, and INCLUDE_INSTALL_DIR can be taken out of runtime/CMakeLists.txt since only the compiler uses it.

kinke · 2025-06-21T13:39:30Z

runtime/ldc-build-runtime.d.in

@@ -3,6 +3,7 @@ module ldcBuildRuntime;
 import core.stdc.stdlib : exit;
 import std.algorithm;
 import std.array;
+import std.conv;


[nit: now superfluous]

the-horo · 2025-06-21T14:12:49Z

If everything works, possible followups include:

Creating different configs for wasn and naked-wasm
Adding a --configTriple option to ldc-build-runtime (and either require the user to pass the triple or the d code can pick it up from ldc2 -v) in order to automate the config installation
Removing MULTILIB and using ldc-build-runtime in its place
Adding a -m (or a better named) flag to ldc-build-runtime that's an alias for --cflags=-m<> --ldflags=-m<> --installWithSuffix=<>
Fix LDC_DYNAMIC_COMPILE being disabled on solely-runtime builds
Fix the compiler testsuite depending on variables declared in the runtime CMakeLists.txt (I only got failures about two things)

When building, always generate separate files for each ldc2.conf section. This allows creating a full ldc2.conf, part by part, without carrying about the statements' order in the cmake file. When installing, `cat` all the configured files and install it as a single ldc2.conf file. Added the CONF_PREF_DIR cmake option for installing the conf files as a directory, without concatenating them. This is useful when building the runtime as a separate project as this setting would allow generating a (partial) ldc2.conf, enough to support the runtime libraries, which could latter be merged with the config generated by the root ldc project to get a fully functional ldc2. This is a requirement when cross-compiling and installing the libraries on another device or when performing multilib builds outside of -DMULTILIB=ON Also add a more detailed documentation about the configuration. Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>

Some settings like the default wasm switches or the compiler-rt lib-dir don't have anything to do with the runtime and, on top of that, some are conditioned by build settings that only affect the compiler. Moving them out of the runtime directory improves the support for cross-compiling because, in the scenario, some variables may be unset in the runtime directory as there is no longer a higher-level project to set them. Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>

Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>

kinke reviewed Jun 16, 2025

View reviewed changes

the-horo force-pushed the ldc2-conf branch from 2425bec to 30d32eb Compare June 16, 2025 17:26

the-horo commented Jun 16, 2025

View reviewed changes

the-horo force-pushed the ldc2-conf branch from 30d32eb to 7eda93a Compare June 16, 2025 18:29

the-horo force-pushed the ldc2-conf branch 2 times, most recently from 14e8a43 to f1f7373 Compare June 16, 2025 19:14

the-horo force-pushed the ldc2-conf branch from f1f7373 to 0b457f5 Compare June 17, 2025 16:44

the-horo force-pushed the ldc2-conf branch from 0b457f5 to 19e8429 Compare June 18, 2025 08:09

the-horo force-pushed the ldc2-conf branch 2 times, most recently from 3447df9 to b15cbe9 Compare June 18, 2025 11:58

the-horo force-pushed the ldc2-conf branch 2 times, most recently from 3d82ec7 to 3c35ebb Compare June 18, 2025 14:02