Could not run Jenkins workflow manually

[uturuncoglu]

@janahaddad I could not run Jenkins workflow manually with my account. I am not sure why but I could not see all options in the menubar to trigger workflow manually.

[uturuncoglu]

@kbooker79 @janahaddad Here is my screenshot. There is no option to run the workflow manually. Even if I modify and work on it in the past, I could not see its configuration now.

[uturuncoglu]

I think there is an issue in GitHub and I could not upload my screenshot.

[uturuncoglu]

@janahaddad @kbooker79 The manual Jenkins workflow (https://jenkins.epic.oarcloud.noaa.gov/job/NOS/job/Coastal-Test-WM-under-development/) is still not showing status correctly. Here is the output from recent run,

REGRESSION TEST RESULT: FAILURE
******Regression Testing Script Completed******
rt.sh finished
rt.sh: Cleaning up...
rt.sh: Exiting.
Pipeline Completed Regression Tests on Hercules-NOS! status=0

In the Jenkins side, it says it is passed but actually one test failed. I checked the actual log files under /work/noaa/nosofs/role-nosofs/jenkins/workspace/hercules/coastal/ufs-weather-model/nightly-test-build-coastal-wm/tests/run_dir/compile_atm2sch2ww3_gnu/err directory and it seems that WW3 is failed to compile with following error,

f951: Fatal Error: Cannot rename module file '../../include/kind_par.mod0' to '../../include/kind_par.mod': No such file or directory
compilation terminated.

If you check the previous run in the Jenkins, the same test is compiled without any issue (/work2/noaa/nosofs/UFS-Coastal-Jenkins-RT/stmp/role-nosofs/FV3_RT/rt_706469/compile_atm2roms_gnu/err) and run (/work2/noaa/nosofs/UFS-Coastal-Jenkins-RT/stmp/role-nosofs/FV3_RT/rt_706469/coastal_ike_shinnecock_atm2sch2ww3_gnu) but bit-to-bit check is failed. So, the error is not consistent even if we did not change anything in the model (same source and hash).

BTW, to double check I have just run this case manually yesterday on Hercules and it was fine (build, run and bit-to-bit identical with baseline). I think there is something wrong with the role account that affects both manual Jenkins workflow and also the one that is connected to the model level PRs. Maybe it would be nice to open an issue in the Hercules IT side. I am not sure how they could help but it would be nice to let them know.

[uturuncoglu]

@saeed-moghimi-noaa I am not sure what can be done but I could not able to edit or trigger Jenkins workflows. I could do it in the past so maybe something is changed in permission wise. As you can see from the following screenshot there is no any option in the left panel to trigger and configure.

Also, one test was failing without any reason and I need to look at it more detailed way. If you look at the last run, it says that it is passed but actually one test is failed. So, we need help from EPIC to resolve this issue. Maybe if I could get my configure permission again and have permission to look at UFS WM one, I could solve it by myself.

Another thing is that we have also hook to PR but the runs are failing with random errors. So, my thinking is that this is related with something in the role account. The owner of that account needs to one a ticket. Also, since Jana is left, someone needs to maintain role caoount on Hercules and regular clean the runs.

[saeed-moghimi-noaa]

@mansurjisan

Hi Mansur, Would you be able to look into this issues? Thanks

[mansurjisan]

Thanks @saeed-moghimi-noaa , @uturuncoglu — I'm looking into this issue and the related Jenkins permission and test run problems. Will report back once I have more clarity.

[mansurjisan]

Just checked Jenkins on my end as well and I’m seeing the same issue Ufuk reported; no option to configure or trigger workflows. This was previously available to me, so looks like something has changed permission-wise.

@kbooker79, could you take a look or point us to the right contact at EPIC who can help with this?

[uturuncoglu]

@mansurjisan Thanks for checking. So, it seems something change and affecting all of us.
@kbooker79 It would be also nice to have permission to check UFS WM workflow. Just having read access is fine. So, I could compare with the UFS Coastal one to solve false pass issue.

[kbooker79]

@uturuncoglu @mansurjisan, let me take a look at this.

[kbooker79]

@uturuncoglu @mansurjisan, please logout of Jenkins and try again.

[uturuncoglu]

@kbooker79 JFYI, still same for me. I could not see the configuration.

[mansurjisan]

@kbooker79 ,

Thank you, Kris ; I just logged out and back in, but I still don’t see any changes on my end. Let me know if there’s anything else I should try or if additional access needs to be granted.

[kbooker79]

@uturuncoglu @mansurjisan, try it again now, I think I found the issue.

[uturuncoglu]

@kbooker79 I could see the configure in the menu now. Thanks for your help. I wonder if I could see the recent one used by the UFS WM too.