S3Download fails unless flattenFolders is used

[sfratini]

Describe the bug

I have a bucket that we use for cache build files. We are currently using ccache so the folder structure has a tree, which we need to keep in place in the agent.

If we use the default values, we get an issue and the process fails, unless we use flattenFolders, which messes up the tree structure we need for ccache.

To reproduce

- task: S3Download@1
            displayName: Cache Download
            inputs:
              awsCredentials: 'AWS DevOps'
              regionName: 'us-east-1'
              sourceFolder: ccache
              bucketName: 'build-cache'
              globExpressions: '**'
              #flattenFolders: true
              targetFolder: $(Agent.TempDirectory)

Structure in the bucket is

/ccache/app/ios/0/2/38428937428947 (example cache file)

Expected behavior

Task downloads S3 bucket respecting tree structure.

Screenshots

This is the issue we get right after the download queue:

Queueing download of ccache/app/ios/f/f/eak4snpi3jd2838irr7etbr759b5rbm1W
Queueing download of ccache/app/ios/f/f/eak4snpi3jd2838irr7etbr759b5rbmR
Queueing download of ccache/app/ios/f/f/edija9200gj5mr373ijd1rusuqa7se4M
Queueing download of ccache/app/ios/f/f/f32dt44pm56o3hjujjtm3ato9vdsi920W
Queueing download of ccache/app/ios/f/f/f32dt44pm56o3hjujjtm3ato9vdsi92R
Queueing download of ccache/app/ios/f/stats
Queueing download of ccache/app/ios/tmp/
Queueing download of ccache/app/ios/tmp/inode-cache.v1
##[error]Error: ENOENT: no such file or directory, open '/Users/runner/work/_temp/ccache/'

Your Environment

• Cloud agent macos
• 1.13.0 toolkit

Additional context

We tried setting up different folders, assuming it could be a permission issue but the fact that the flattenFolders does something turns out it is not. At the same time, with flattenFolders we get that some files are being found and overridden in subsequent runs. Most likely the issue is that we are getting the same agent again and the temp folder is not clean. Not a problem for us, since it is overridden, but it is indeed saving files so it should not be a problem with the permissions.

Most likely issue is that it is not correctly creating all the intermediate folders for download.

[JadenSimon]

Most likely issue is that it is not correctly creating all the intermediate folders for download.

Yeah I'm thinking this is the issue as well.