`join_paths`' interaction with `File`s that are *really* directories is confusing

[adamnovak]

The new WDL 1.2 join_paths() function comes in overloads of:

File join_paths(File, String)
File join_paths(File, Array[String]+)
File join_paths(Array[String]+)

But this isn't a urljoin()-like operation where dirname/file1.txt and file2.txt gives dirname/file2.txt. Instead, the File-type argument for the first two overloads "must specify a directory", which I'm having trouble interpreting. Specifically I am confused about how the rules about localizing and de-localizing File values as they move in and out of tasks are applied to them, and how the behavior of one of these directory-specifying File values differs from that of a Directory initialized with the same string value.

The description of the File type doesn't really explain what it means for a File to actually specify a directory and not a file. It does say that a string literal for a non-optional File pointing to a nonexistent path is an error, but it doesn't say what happens if the path exists but does not point to a file (and instead points to, e.g., a directory).

The example for join_paths manages to construct a non-null File that points to a non-file, in the task inputs' default values:

File abs_file = "/usr"

These values seem mildly cursed and raise a lot of unanswered questions. Do these work identically to Directory type values, and bring along the directory contents recursively when passed between workflows and tasks, or do they not?

Even if they don't bring the directory contents with them, do they follow the same localization rules about required relationships to sibling Files and Directorys that share the same parent, and about whether the execution engine is allowed to adjust the text of the path as seen within a task, that would be applied to File values referring to files and Directory values referring to directories?

Specifically, if I make a call like:

call some_task {
inputs:
    some_directory_input="/usr/bin", # This input has a Directory type
    some_other_directory_input="/usr/share", # This input has a Directory type
    some_file_input="/usr/bin" # This input has a File type
}]

My reading of the localization rules is that I should have the contents of the host /usr/bin available in some_directory_input, and that the paths I get for some_directory_input and some_file_input must be the same path. But that would really suggest that join_paths() on some_file_input should be reading the host version of that directory, while the description of the function seems written like I should expect to see the container version.

[claymcleod]

First, just want to point out that the spec is a bit loose around when and how paths are translated between host and guest, and that's something we've been struggling with on the Sprocket project. So there is room for improvement. That being said, here's how I understand the answer to some of your questions.

---

Specifically I am confused about how the rules about localizing and de-localizing File values as they move in and out of tasks are applied to them,

My understanding is that join_paths operates exclusively on host paths. From the docs of the standard library function (emphasis mine):

Joins together two or more paths into an absolute path in the host filesystem.

Further, the spec dictates in various places that Files are in the context of the host filesystem and the translation to a guest path is deferred as long as possible.

A File declaration can have a string value indicating a relative or absolute path on the local file system.

1

A File or Directory declaration may have have a string value indicating a relative or absolute path on the local file system.

2

Thus, it's relatively clear that this translation should happen on the host side of things before substitution.

---

and how the behavior of one of these directory-specifying File values differs from that of a Directory initialized with the same string value.

From the examples, it was clearly the intention to allow folders to be represented as Files. I was not around for the drafting of WDL v1.2, but my understanding has always been that File can represent any file on the filesystem (including a directory) whereas the Directory type was added as a facility for bringing sets of files into the top-level WDL without needing to, say, zip the files to a bundle.

In Sprocket, we have ensure_path_exists which does check that files are actual files and fails otherwise. My understanding is that we actually did have to update this example (link) to make it pass.

In other words, I think the example and the wording is probably just confusing, and it is going to need some work. At the very least, we may need to support Directory types as the first argument, and we may also need to modify the language to include clarified behavior for File. You can see that we do things like require the second path to be relative, which is technically not to spec.

---

The description of the File type doesn't really explain what it means for a File to actually specify a directory and not a file. It does say that a string literal for a non-optional File pointing to a nonexistent path is an error, but it doesn't say what happens if the path exists but does not point to a file (and instead points to, e.g., a directory).

The rest of the comment calls into question what the behavior of a File that points to a directory should be, but the quote above succinctly summarizes the problem.

I would summarize the answer to all of these by just saying you're right: a File attempting to bind to a directory should probably just be an error. Unsure if that's a breaking change, in practice it almost certainly is, but whether implementations should have allowed it in the first place is not clear to me.