Skip to content

Custom cache directory name #543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 8, 2024
Merged

Custom cache directory name #543

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6aa6888
Custom cache directory name
gerhardols Mar 4, 2024
db51694
fixup! review comment CUSTOM_CACHE_DIR -> CUSTOM_CACHE_KEY
gerhardols Apr 8, 2024
7b6b1ee
run cmake-format
TheLartians Apr 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,9 @@ Note that passing the variable as a configure option to CMake will always overri

You can use `CPM_SOURCE_CACHE` on GitHub Actions workflows [cache](https://github.com/actions/cache) and combine it with ccache, to make your CI faster. See the [wiki](https://github.com/cpm-cmake/CPM.cmake/wiki/Caching-with-CPM.cmake-and-ccache-on-GitHub-Actions) for more info.

The directory where the version for a project is stored is by default the hash of the arguments to `CPMAddPackage()`.
If for instance the patch command uses external files, the directory name can be set with the argument `CUSTOM_CACHE_DIR`.

### CPM_DOWNLOAD_ALL

If set, CPM will forward all calls to `CPMFindPackage` as `CPMAddPackage`.
Expand Down
6 changes: 5 additions & 1 deletion cmake/CPM.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,7 @@ function(CPMAddPackage)
GIT_SHALLOW
EXCLUDE_FROM_ALL
SOURCE_SUBDIR
CUSTOM_CACHE_DIR
)

set(multiValueArgs URL OPTIONS DOWNLOAD_COMMAND)
Expand Down Expand Up @@ -710,7 +711,10 @@ function(CPMAddPackage)
string(TOLOWER ${CPM_ARGS_NAME} lower_case_name)
set(origin_parameters ${CPM_ARGS_UNPARSED_ARGUMENTS})
list(SORT origin_parameters)
if(CPM_USE_NAMED_CACHE_DIRECTORIES)
if(CPM_ARGS_CUSTOM_CACHE_DIR)
# Application set a custom unique directory name
set(download_directory ${CPM_SOURCE_CACHE}/${lower_case_name}/${CPM_ARGS_CUSTOM_CACHE_DIR})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the cache directory is a hash of all used arguments to guarantee that the dependency is actually present in precisely the requested version. How would we avoid conflicts here if users could set custom directories? E.g. say I have two projects that both use foo as a custom cache dir, but they use different versions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Version must be handled manually (like with a combination of the download version and a manual sha of the patch_command files).
The current sha will not work when e.g. patch_command refers to external files, so it is not fool proof either.
Is the description in the readme clear enough?

My use case with some more details:

  • Shared toolchain that are several GB and may be used in many application repos. The patch command contains external files adjusting the installation (licenses that may be updated more often than the toolchain).
    • As paths are in the patch_command, the sha will differ and the paths will not be shared
    • As there are external files in the patch_command, the sha is not reliable, requires change to the arguments too.
    • The sha makes it hard to see what is in the cache - I may want to clean certain test versions only
    • Using Docker is inconvenient, the toolchain has a few branches

I would have used SOURCE_DIR for this change if not compatibility had been broken...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true, patch commands using external files will definitely create issues. I'm still unsure about how I feel about applying patches in the cache as it can create very hard to track issues. In the JavaScript world using npm or yarn, dependencies are copied from the cache to a project-specific location and any patches are applied there. That way the original cache remains valid no matter how the patch is applied. Maybe a mechanism like that would make sense as well here?

I'm also not completely against this approach, just the name could be confusing as it seems to imply an absolute path. How about calling it something like CUSTOM_CACHE_KEY instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true, patch commands using external files will definitely create issues. I'm still unsure about how I feel about applying patches in the cache as it can create very hard to track issues. In the JavaScript world using npm or yarn, dependencies are copied from the cache to a project-specific location and any patches are applied there. That way the original cache remains valid no matter how the patch is applied. Maybe a mechanism like that would make sense as well here?

Maybe, even if my several GB packages should not be copied.
That would men to intercept and handle the fetch content arguments.
A bigger feature, I believe this can coexist without maintenance issues.

I'm also not completely against this approach, just the name could be confusing as it seems to imply an absolute path. How about calling it something like CUSTOM_CACHE_KEY instead?

Fine, I pushed a change.

elseif(CPM_USE_NAMED_CACHE_DIRECTORIES)
string(SHA1 origin_hash "${origin_parameters};NEW_CACHE_STRUCTURE_TAG")
set(download_directory ${CPM_SOURCE_CACHE}/${lower_case_name}/${origin_hash}/${CPM_ARGS_NAME})
else()
Expand Down
14 changes: 14 additions & 0 deletions test/unit/cache.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,3 +137,17 @@ execute_process(

assert_equal(${ret} "0")
assert_not_exists("${CPM_SOURCE_CACHE_DIR}/fibonacci")

# Use custom cache directory

set(FIBONACCI_PACKAGE_ARGS "CUSTOM_CACHE_DIR my_custom_unique_dir GIT_TAG e9ebf168ca0fffaa4ef8c6fefc6346aaa22f6ed5")
set(FIBONACCI_VERSION 1.1)
update_cmake_lists()

execute_process(
COMMAND ${CMAKE_COMMAND} -E env "CPM_SOURCE_CACHE=${CPM_SOURCE_CACHE_DIR}" ${CMAKE_COMMAND}
"-S${CMAKE_CURRENT_LIST_DIR}/remote_dependency" "-B${TEST_BUILD_DIR}" RESULT_VARIABLE ret
)

assert_equal(${ret} "0")
assert_exists("${CPM_SOURCE_CACHE_DIR}/fibonacci/my_custom_unique_dir")