UW-657 fs makedirs

docs/index.rst

@@ -71,22 +71,19 @@ This tool helps transform legacy configuration files templated with the atparse
 
 | :any:`CLI documentation with examples<cli_template_translate_examples>`
 
-File Provisioning
-^^^^^^^^^^^^^^^^^
+File/Directory Provisioning
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-This tool helps users define the source and destination of files to be copied or linked, in the same UW YAML language used by UW drivers.
-
-| :any:`CLI documentation with examples<cli_file_copy_examples>`
+This tool helps users define the source and destination of files to be copied or linked, or directories to be created, in the same UW YAML language used by UW drivers.
 
+| :any:`CLI documentation with examples<cli_file_mode>`
 
 There is a video demonstration of the use of the ``uw file`` tool available via YouTube.
 
 .. raw:: html
 
    <iframe width="560" height="315" src="https://www.youtube.com/embed/b2HXOlt-Ulw?si=rLWatBFu4mvNR65S" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
 
-|
-
 Rocoto Configurability
 ^^^^^^^^^^^^^^^^^^^^^^
 

docs/sections/user_guide/cli/tools/file.rst

@@ -1,15 +1,15 @@
 ``file``
 ========
 
+.. _cli_file_mode:
+
 The ``uw`` mode for handling filesystem files.
 
 .. literalinclude:: file/help.cmd
    :emphasize-lines: 1
 .. literalinclude:: file/help.out
    :language: text
 
-.. _cli_file_copy_examples:
-
 ``copy``
 --------
 
@@ -52,8 +52,6 @@ The ``--target-dir`` option is optional when all destination paths are absolute,
 .. literalinclude:: file/copy-exec-no-target-dir-err.out
    :language: text
 
-.. _cli_file_link_examples:
-
 ``link``
 --------
 
@@ -95,3 +93,43 @@ The ``--target-dir`` option is optional when all linkname paths are absolute, an
    :emphasize-lines: 1
 .. literalinclude:: file/link-exec-no-target-dir-err.out
    :language: text
+
+``makedirs``
+------------
+
+The ``makedirs`` action creates directories. Any ``KEY`` positional arguments are used to navigate, in the order given, from the top of the config to the :ref:`makedirs block <makedirs_yaml>`.
+
+.. literalinclude:: file/makedirs-help.cmd
+   :emphasize-lines: 1
+.. literalinclude:: file/makedirs-help.out
+   :language: text
+
+Examples
+^^^^^^^^
+
+Given ``makedirs-config.yaml`` containing
+
+.. literalinclude:: file/makedirs-config.yaml
+   :language: yaml
+.. literalinclude:: file/makedirs-exec.cmd
+   :emphasize-lines: 2
+.. literalinclude:: file/makedirs-exec.out
+   :language: text
+
+The ``--cycle`` and ``--leadtime`` options can be used to make Python ``datetime`` and ``timedelta`` objects, respectively, available for use in Jinja2 expression in the config. For example:
+
+.. literalinclude:: file/makedirs-config-timedep.yaml
+   :language: yaml
+.. literalinclude:: file/makedirs-exec-timedep.cmd
+   :emphasize-lines: 2
+.. literalinclude:: file/makedirs-exec-timedep.out
+   :language: text
+
+The ``--target-dir`` option is optional when all directory paths are absolute, and will never be applied to absolute paths. If any paths are relative, however, it is an error not to provide a target directory:
+
+.. literalinclude:: file/makedirs-config.yaml
+   :language: yaml
+.. literalinclude:: file/makedirs-exec-no-target-dir-err.cmd
+   :emphasize-lines: 1
+.. literalinclude:: file/makedirs-exec-no-target-dir-err.out
+   :language: text

docs/sections/user_guide/cli/tools/file/copy-exec-no-target-dir-err.out

@@ -1 +1,3 @@
-[2024-07-26T21:51:23]    ERROR Relative path 'foo' requires the target directory to be specified
+[2024-08-14T15:19:59]     INFO Validating config against internal schema: files-to-stage
+[2024-08-14T15:19:59]     INFO 0 UW schema-validation errors found
+[2024-08-14T15:19:59]    ERROR Relative path 'foo' requires the target directory to be specified

docs/sections/user_guide/cli/tools/file/help.out

@@ -14,3 +14,5 @@ Positional arguments:
       Copy files
     link
       Link files
+    makedirs
+      Make directories

docs/sections/user_guide/cli/tools/file/link-exec-no-target-dir-err.out

@@ -1 +1,3 @@
-[2024-07-26T21:51:23]    ERROR Relative path 'foo' requires the target directory to be specified
+[2024-08-14T15:19:57]     INFO Validating config against internal schema: files-to-stage
+[2024-08-14T15:19:57]     INFO 0 UW schema-validation errors found
+[2024-08-14T15:19:57]    ERROR Relative path 'foo' requires the target directory to be specified

docs/sections/user_guide/cli/tools/file/makedirs-config-timedep.yaml

@@ -0,0 +1,8 @@
+config:
+  makedirs:
+    - foo/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/bar
+    - baz/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/qux
+yyyymmdd: "{{ cycle.strftime('%Y%m%d') }}"
+hh: "{{ cycle.strftime('%H') }}"
+nnn: "{{ '%03d' % (leadtime.total_seconds() // 3600) }}"
+validtime: "{{ (cycle + leadtime).strftime('%Y-%m-%dT%H') }}"

docs/sections/user_guide/cli/tools/file/makedirs-config.yaml

@@ -0,0 +1,4 @@
+config:
+  makedirs:
+    - foo
+    - bar

docs/sections/user_guide/cli/tools/file/makedirs-exec-no-target-dir-err.cmd

@@ -0,0 +1 @@
+uw file makedirs --config-file makedirs-config.yaml config

docs/sections/user_guide/cli/tools/file/makedirs-exec-no-target-dir-err.out

@@ -0,0 +1,3 @@
+[2024-08-14T15:19:58]     INFO Validating config against internal schema: makedirs
+[2024-08-14T15:19:58]     INFO 0 UW schema-validation errors found
+[2024-08-14T15:19:58]    ERROR Relative path 'foo' requires the target directory to be specified

docs/sections/user_guide/cli/tools/file/makedirs-exec-timedep.cmd

@@ -0,0 +1,4 @@
+rm -rf makedirs-parent-timedep
+uw file makedirs --target-dir makedirs-parent-timedep --config-file makedirs-config-timedep.yaml --cycle 2024-05-29T12 --leadtime 6 config
+echo
+tree -F makedirs-parent-timedep

docs/sections/user_guide/cli/tools/file/makedirs-exec-timedep.out

@@ -0,0 +1,29 @@
+[2024-08-12T04:35:49]     INFO Validating config against internal schema: makedirs
+[2024-08-12T04:35:49]     INFO 0 UW schema-validation errors found
+[2024-08-12T04:35:49]     INFO Directories: Initial state: Not Ready
+[2024-08-12T04:35:49]     INFO Directories: Checking requirements
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Initial state: Not Ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Checking requirements
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Requirement(s) ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Executing
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Final state: Ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Initial state: Not Ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Checking requirements
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Requirement(s) ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Executing
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Final state: Ready
+[2024-08-12T04:35:49]     INFO Directories: Final state: Ready
+
+makedirs-parent-timedep/
+├── baz/
+│   └── 20240529/
+│       └── 12/
+│           └── 006/
+│               └── qux/
+└── foo/
+    └── 20240529/
+        └── 12/
+            └── 006/
+                └── bar/
+
+11 directories, 0 files

docs/sections/user_guide/cli/tools/file/makedirs-exec.cmd

@@ -0,0 +1,4 @@
+rm -rf makedirs-parent
+uw file makedirs --target-dir makedirs-parent --config-file makedirs-config.yaml config
+echo
+tree -F makedirs-parent

docs/sections/user_guide/cli/tools/file/makedirs-exec.out

@@ -0,0 +1,21 @@
+[2024-08-12T04:35:49]     INFO Validating config against internal schema: makedirs
+[2024-08-12T04:35:49]     INFO 0 UW schema-validation errors found
+[2024-08-12T04:35:49]     INFO Directories: Initial state: Not Ready
+[2024-08-12T04:35:49]     INFO Directories: Checking requirements
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/foo: Initial state: Not Ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/foo: Checking requirements
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/foo: Requirement(s) ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/foo: Executing
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/foo: Final state: Ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/bar: Initial state: Not Ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/bar: Checking requirements
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/bar: Requirement(s) ready
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/bar: Executing
+[2024-08-12T04:35:49]     INFO Directory makedirs-parent/bar: Final state: Ready
+[2024-08-12T04:35:49]     INFO Directories: Final state: Ready
+
+makedirs-parent/
+├── bar/
+└── foo/
+
+3 directories, 0 files

docs/sections/user_guide/cli/tools/file/makedirs-help.cmd

@@ -0,0 +1 @@
+uw file makedirs --help

docs/sections/user_guide/cli/tools/file/makedirs-help.out

@@ -0,0 +1,29 @@
+usage: uw file makedirs [-h] [--version] [--config-file PATH]
+                        [--target-dir PATH] [--cycle CYCLE]
+                        [--leadtime LEADTIME] [--dry-run] [--quiet]
+                        [--verbose]
+                        [KEY ...]
+
+Make directories
+
+Optional arguments:
+  -h, --help
+      Show help and exit
+  --version
+      Show version info and exit
+  --config-file PATH, -c PATH
+      Path to UW YAML config file (default: read from stdin)
+  --target-dir PATH
+      Root directory for relative destination paths
+  --cycle CYCLE
+      The cycle in ISO8601 format (e.g. 2024-08-12T00)
+  --leadtime LEADTIME
+      The leadtime as hours[:minutes[:seconds]]
+  --dry-run
+      Only log info, making no changes
+  --quiet, -q
+      Print no logging messages
+  --verbose, -v
+      Print all logging messages
+  KEY
+      YAML key leading to file dst/src block

docs/sections/user_guide/yaml/files.rst

@@ -3,7 +3,7 @@
 File Blocks
 ===========
 
-File blocks define files to be staged in a target directory as copies or symbolic links. Keys in such blocks specify destination paths relative to the target directory, and values specify source paths.
+File blocks define files to be staged in a target directory as copies or symbolic links. Keys in such blocks specify either absolute destination paths, or destination paths relative to the target directory. Values specify source paths.
 
 Example block:
 

docs/sections/user_guide/yaml/index.rst

@@ -5,9 +5,10 @@ UW YAML
    :maxdepth: 1
 
    components/index
-   platform
    execution
    files
-   updating_values
+   makedirs
+   platform
    rocoto
    tags
+   updating_values

docs/sections/user_guide/yaml/makedirs.rst

@@ -0,0 +1,22 @@
+.. _makedirs_yaml:
+
+Directory Blocks
+================
+
+Directory blocks define a sequence of one or more directories to be created, nested under a ``makedirs:`` key. Each value is either an absolute path, or a path relative to the target directory, specified either via the CLI or an API call.
+
+Example block with absolute paths:
+
+.. code-block:: yaml
+
+   makedirs:
+     - /path/to/dir1
+     - /path/to/dir2
+
+Example block with relative paths:
+
+.. code-block:: yaml
+
+   makedirs:
+     - /subdir/dir1
+     - ../../dir2

src/uwtools/api/file.py

@@ -1,5 +1,5 @@
 """
-API access to ``uwtools`` file-management tools.
+API access to ``uwtools`` file and directory management tools.
 """
 
 import datetime as dt
@@ -8,7 +8,7 @@
 
 from iotaa import Asset
 
-from uwtools.file import Copier, Linker
+from uwtools.file import Copier, Linker, MakeDirs
 from uwtools.utils.api import ensure_data_source as _ensure_data_source
 
 
@@ -33,15 +33,15 @@ def copy(
     :param stdin_ok: OK to read from ``stdin``?
     :return: ``True`` if all copies were created.
     """
-    copier = Copier(
+    stager = Copier(
         target_dir=Path(target_dir) if target_dir else None,
         config=_ensure_data_source(config, stdin_ok),
         cycle=cycle,
         leadtime=leadtime,
         keys=keys,
         dry_run=dry_run,
     )
-    assets: list[Asset] = copier.go()  # type: ignore
+    assets: list[Asset] = stager.go()  # type: ignore
     return all(asset.ready() for asset in assets)
 
 
@@ -66,16 +66,49 @@ def link(
     :param stdin_ok: OK to read from ``stdin``?
     :return: ``True`` if all links were created.
     """
-    linker = Linker(
+    stager = Linker(
         target_dir=Path(target_dir) if target_dir else None,
         config=_ensure_data_source(config, stdin_ok),
         cycle=cycle,
         leadtime=leadtime,
         keys=keys,
         dry_run=dry_run,
     )
-    assets: list[Asset] = linker.go()  # type: ignore
+    assets: list[Asset] = stager.go()  # type: ignore
     return all(asset.ready() for asset in assets)
 
 
-__all__ = ["Copier", "Linker", "copy", "link"]
+def makedirs(
+    config: Optional[Union[Path, dict, str]] = None,
+    target_dir: Optional[Union[Path, str]] = None,
+    cycle: Optional[dt.datetime] = None,
+    leadtime: Optional[dt.timedelta] = None,
+    keys: Optional[list[str]] = None,
+    dry_run: bool = False,
+    stdin_ok: bool = False,
+) -> bool:
+    """
+    Make directories.
+
+    :param config: YAML-file path, or ``dict`` (read ``stdin`` if missing or ``None``).
+    :param target_dir: Path to target directory.
+    :param cycle: A datetime object to make available for use in the config.
+    :param leadtime: A timedelta object to make available for use in the config.
+    :param keys: YAML keys leading to file dst/src block.
+    :param dry_run: Do not link files.
+    :param stdin_ok: OK to read from ``stdin``?
+    :return: ``True`` if all directories were made.
+    """
+    stager = MakeDirs(
+        target_dir=Path(target_dir) if target_dir else None,
+        config=_ensure_data_source(config, stdin_ok),
+        cycle=cycle,
+        leadtime=leadtime,
+        keys=keys,
+        dry_run=dry_run,
+    )
+    assets: list[Asset] = stager.go()  # type: ignore
+    return all(asset.ready() for asset in assets)
+
+
+__all__ = ["Copier", "Linker", "MakeDirs", "copy", "link", "makedirs"]