Update bash ex-scripts to be linter compliant

[WalterKolczynski-NOAA]

Description

Updates the ex-scripts to be compliant with both shellcheck and shfmt. This does not necessarily enforce some standards not checked by shellcheck. Export statements are added to some variables to avoid 'not used' errors. In the future, a more careful analysis should be done to see if these variables are used downstream by executables or if they are truly unused, in which case they should be removed.

Scripts employing MPMD are also updated to use run_mpmd.sh. This resulted in some retooling of those scripts.

Refs: #397

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

• Full test suite on Ursa

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[JessicaMeixner-NOAA]

Can someone point me to where 4 spaces instead of 2 is the standard for the g-w or some linter check? I was unaware that was a thing, since 2 spaces is very widely used across the g-w.

This PR needs to be tested on WCOSS2 to ensure that all wave AWIPS, etc jobs are still running as expected as this changes those scripts and I think most of the changes were spaces, but I'm still going through to confirm as github isn't showing all of them as such. Additionally, wave jobs should be checked that the contents is the same before and after this PR as there were a few places where commands were changed.

[DavidHuber-NOAA]

Can someone point me to where 4 spaces instead of 2 is the standard for the g-w or some linter check? I was unaware that was a thing, since 2 spaces is very widely used across the g-w.

@WalterKolczynski-NOAA I think that this comes from the change you made to .editorconfig:
https://github.com/NOAA-EMC/global-workflow/blame/5fa67e3339e300c9d380e32c98dba541948fbfcc/.editorconfig#L25-L38

[JessicaMeixner-NOAA]

Can someone point me to where 4 spaces instead of 2 is the standard for the g-w or some linter check? I was unaware that was a thing, since 2 spaces is very widely used across the g-w.

@WalterKolczynski-NOAA I think that this comes from the change you made to .editorconfig: https://github.com/NOAA-EMC/global-workflow/blame/5fa67e3339e300c9d380e32c98dba541948fbfcc/.editorconfig#L25-L38

So this is a new thing. This even seems inconsistent in .editorconfig as well. Is .editorconfig where we should be looking for these sorts of standards in the future?

[WalterKolczynski-NOAA]

Can someone point me to where 4 spaces instead of 2 is the standard for the g-w or some linter check? I was unaware that was a thing, since 2 spaces is very widely used across the g-w.

@WalterKolczynski-NOAA I think that this comes from the change you made to .editorconfig: https://github.com/NOAA-EMC/global-workflow/blame/5fa67e3339e300c9d380e32c98dba541948fbfcc/.editorconfig#L25-L38

So this is a new thing. This even seems inconsistent in .editorconfig as well. Is .editorconfig where we should be looking for these sorts of standards in the future?

Yes. I will note before this point, no standard was enforced for bash scripts, which is why different scripts had different indentation. This PR is part of applying a standard to all the legacy scripts so enforcement can begin. Four spaces is a compromise between space usage and visibility, and is consistent with the python scripts.

[TerrenceMcGuinness-NOAA]

Gaea C6 Runner was of line this morning and I didn't get it back until this evening (working remotely in other time zone) it is running now.

[TerrenceMcGuinness-NOAA]

@DavidHuber-NOAA The rebuild on Gaea C6 for this PR had some inexplicable errors (I have not pursued this yet -- see below) While the restart from the nightly worked fine. We'll see in the morning after the nightly runs again.

pipeline is set to checkout PR_NUMBER=4151 from https://github.com/NOAA-EMC/global-workflow on machine Gaeac6
GraphQL: Projects (classic) is being deprecated in favor of the new Projects experience, see: https://github.blog/changelog/2024-05-23-sunset-notice-projects-classic/. (repository.pullRequest.projectCards)
Running after_script
$ echo "After script is running for job ${CI_JOB_NAME} on ${machine} with status ${CI_JOB_STATUS}" # collapsed multi-line command
After script is running for job build-gaeac6 on gaeac6 with status failed
Job failed, performing cleanup actions for Gaeac6
Updating GitHub labels for PR 4151 failure on Gaeac6
GraphQL: Projects (classic) is being deprecated in favor of the new Projects experience, see: https://github.blog/changelog/2024pipeline is set to checkout PR_NUMBER=4151 from https://github.com/NOAA-EMC/global-workflow on machine Gaeac6
GraphQL: Projects (classic) is being deprecated in favor of the new Projects experience, see: https://github.blog/changelog/2024-05-23-sunset-notice-projects-classic/. (repository.pullRequest.projectCards)
Running after_script
00:00
Running after script...
$ echo "After script is running for job ${CI_JOB_NAME} on ${machine} with status ${CI_JOB_STATUS}" # collapsed multi-line command
After script is running for job build-gaeac6 on gaeac6 with status failed
Job failed, performing cleanup actions for Gaeac6
Updating GitHub labels for PR 4151 failure on Gaeac6
GraphQL: Projects (classic) is being -05-23-sunset-notice-projects-classic/. (repository.pullRequest.projectCards)
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit status 1

[aerorahul]

Looks good.

[DavidHuber-NOAA]

@TerrenceMcGuinness-NOAA I recall these GraphQL errors coming up before. It required an updated github-cli IIRC. I'm going to go ahead and launch this on C6 manually.

[DavidHuber-NOAA]

Ursa passed on the second run through, so I suspect the prepiodaobs failure (noted 5 days ago here) was transient. Once the shopt option is added in and the conflicts are resolved, I will merge the changes locally on C6 and start the tests there. Once those pass, we can merge this PR.

[WalterKolczynski-NOAA]

Ursa passed on the second run through, so I suspect the prepiodaobs failure (noted 5 days ago here) was transient. Once the shopt option is added in and the conflicts are resolved, I will merge the changes locally on C6 and start the tests there. Once those pass, we can merge this PR.

What shopt option? Is this something I need to do?

[DavidHuber-NOAA]

Ursa passed on the second run through, so I suspect the prepiodaobs failure (noted 5 days ago here) was transient. Once the shopt option is added in and the conflicts are resolved, I will merge the changes locally on C6 and start the tests there. Once those pass, we can merge this PR.

What shopt option? Is this something I need to do?

@WalterKolczynski-NOAA Yes, this was your suggestion to an issue I raised: #4151 (comment)

[WalterKolczynski-NOAA]

Ursa passed on the second run through, so I suspect the prepiodaobs failure (noted 5 days ago here) was transient. Once the shopt option is added in and the conflicts are resolved, I will merge the changes locally on C6 and start the tests there. Once those pass, we can merge this PR.

What shopt option? Is this something I need to do?

@WalterKolczynski-NOAA Yes, this was your suggestion to an issue I raised: #4151 (comment)

Ah. Find wasn't showing it for some reason. Change made. Put in preamble for want of a better location.

[DavidHuber-NOAA]

Great, thanks @WalterKolczynski-NOAA! I'll start up testing on C6.