Initial ICI changes#238

Draft

hannahnesser wants to merge 143 commits intodevfrom

Commits on Mar 19, 2024

Creating default config, python environment, and GCC environment settings
Hannah Nesser
committed

Commits on Mar 20, 2024

Changing DataPathTROPOMI to DataPathObs (and adding Species to config_required for sanitize_input_yaml)
Hannah Nesser
committed
Changes to generalize supporting other clusters, including adding Species, UseScheduler, SchedulerType, DataPathObs, and PythonEnvType/PythonEnvPath
Hannah Nesser
committed
Default species for Harvard should be CH4
Hannah Nesser
committed
Generalizing scheduler options
Hannah Nesser
committed

Commits on Mar 21, 2024

Moving conda environment activation into environment files
Hannah Nesser
committed
Removing conda environment specification from config file
Hannah Nesser
committed
Removing conda environment specification from config file
Hannah Nesser
committed
Moving conda environment activation into environment files
Hannah Nesser
committed
Removing UseSlurm option and replacing with UseScheduler/SchedulerType
Hannah Nesser
committed
Switching from UseSlurm to UseScheduler/SchedulerType
Hannah Nesser
committed

Commits on Mar 22, 2024

Adding PythonEnv option
Hannah Nesser
committed
Adding PythonEnv to variables required for local simulation
Hannah Nesser
committed
Removing python loading from GEOS-Chem environment
Hannah Nesser
committed
Creating separate environment file for python loading
Hannah Nesser
committed
Adding separate statement for loading Python env
Hannah Nesser
committed
Switching conda activate to source
Hannah Nesser
committed
Handling python environment loading for AWS (now hardcoded?) and local versions
Hannah Nesser
committed
Added reference to PythonEnv since htat's now specified along with GEOSChemEnv for local simulations
Hannah Nesser
committed

Commits on Apr 2, 2024

Commits on Apr 3, 2024

Modified to handle tmux
hannahnesser
committed

Commits on Apr 11, 2024

Switched to standard SBATCH headers
Hannah Nesser
committed

Commits on Apr 12, 2024

Adding in corrections for PBS
Hannah Nesser
committed
Changes to SBATCH to PBS conversion
Hannah Nesser
committed
Moved SBATCH to PBS conversion to common.sh
Hannah Nesser
committed
Added units to mem to avoid confusion in conversion from SBATCH to PBS
Hannah Nesser
committed
Added units to mem to avoid confusion in conversion from SBATCH to PBS
Hannah Nesser
committed
Switched condaenv options to PythonEnv
Hannah Nesser
committed

Commits on Jun 7, 2024

Minor syntax fix
Hannah Nesser
committed
Mostly changes to get replacement of SBATCH to PBS working
Hannah Nesser
committed
Adding print statement to check
Hannah Nesser
committed
Continued bug fixes to awk
Hannah Nesser
committed

Commits on Jun 8, 2024

Continued debugging
Hannah Nesser
committed

Commits on Jun 24, 2024

Continued bug fixes
Hannah Nesser
committed
Adding node request to sbatch
Hannah Nesser
committed
Removing nodes options, which will now be passed with sbatch -N directly
Hannah Nesser
committed

Commits on Jun 25, 2024

Changing default format of time request to HH:MM:SS for consistency between PBS/SBATCH requirements; changed default resource request to include a unit term (gb) for compatibility between PBS/SBATC…
Hannah Nesser
committed

Commits on Jun 26, 2024

Commits on Jun 27, 2024

Switched HourlyCH4 to HourlySpecies, changed ch4_run.template to run.template, and added in new arguments for jacobian.py (species and name changes for variables)
Hannah Nesser
committed
Changed HourlyCH4 to HourlySpecies and ch4_run.template to run.template
Hannah Nesser
committed
Changed tropomi_cache --> satellite_cache
Hannah Nesser
committed
Moving away from specific TROPOMI/CH4 references (mostly in variable names, but
Hannah Nesser
committed
Changes mostly to variable names to move away from TROPOMI-specific and methane-
Hannah Nesser
committed
Removing TROPOMI specific references
Hannah Nesser
committed
- Added species argument to average_satellite_observations
Hannah Nesser
committed
Adding functions: mixing_ratio_conv_factor, species_molar_mass, read_tropomi,
Hannah Nesser
committed
Change BlendedTROPOMI to SatelliteProduct, change HourlyCH4 to HourlySpecies, and add UseBCsForRestart
Hannah Nesser
committed
While this script is implicitly very TROPOMI-specific, we still remained TROPOMI and
Hannah Nesser
committed
- Change ch4_run.template to use the species variable from the config file
Hannah Nesser
committed
Changed data directory name from data_TROPOMI to data_satellite; there's still a reference to the integrated_methane_inversion in one of the AWS functions, but I'm not sure what to do with this
Hannah Nesser
committed
Changed ch4_run.template to <species>_run.template
Hannah Nesser
committed
Changed ch4_run.template to <species>_run.template
Hannah Nesser
committed
Move UseBCsForRestart from setup.sh to config file
Hannah Nesser
committed
Switched from HourlyCH4 to HourlySpecies and from ch4_run.template to <species>_run.template
Hannah Nesser
committed
Changed name of TROPOMI data directory from data_TROPOMI to data_satellite
Hannah Nesser
committed
Changed HourlyCH4 to HourlySpecies and ch4_run.template to <species>_run.template
Hannah Nesser
committed

Commits on Jun 28, 2024

Added species argument to main options
Hannah Nesser
committed
Changed tropomiCache to satelliteCache and addded Species and SatelliteProduct arguments to the call to jacobian.py
Hannah Nesser
committed
Removed bug check print statements and fixed a few PBS specific bugs
Hannah Nesser
committed
Corrected calls to write_boundary_conditions to include new arguments
Hannah Nesser
committed
Changed tropomiCache to satelliteCache, data_TROPOMI default storage to data_satellite, and added if statements to the TROPOMI download section to only download TROPOMI if it is a methane inversion
Hannah Nesser
committed
Changed HourlyCH4 to HourlySpecies and BlendedTROPOMI to SatelliteProdcut
Hannah Nesser
committed
Changed HourlyCH4 to HourlySpecies
Hannah Nesser
committed
Changed <species>_run.template to run.template
Hannah Nesser
committed
Generalizing name of run template
Hannah Nesser
committed
Changed ch4_run.template to just run.template
Hannah Nesser
committed
Local changes
Hannah Nesser
committed
Changing back to default config files
Hannah Nesser
committed
Merging dev and ICI branches before moving the ICI branch to a sub branch of dev
Hannah Nesser
committed
Moving sbatch requests to use the submit_job function
Hannah Nesser
committed
Adding False argument for the new viz Boolean
Hannah Nesser
committed
Adding species arguments
Hannah Nesser
committed
Removing specific CH4 references and replacing with {species} and doing the same for TROPOMI > satellite
Hannah Nesser
committed

Commits on Jul 1, 2024

Fixing comment string
Hannah Nesser
committed
Updating to include new flags from dev branch
Hannah Nesser
committed
Removing condaEnv/condaFile refs and replacing with PythonEnv per the PBS requirements
Hannah Nesser
committed
Adjusting for SBATCH options that were previously not caught by convert_sbatch_to_pbs
Hannah Nesser
committed
Adding a check to see if the PBS -l site=needed option has been previously added
Hannah Nesser
committed
Printing out any SBATCH options not caught by the conversion script
Hannah Nesser
committed
Removed print statement that didn't really work
Hannah Nesser
committed

Commits on Jul 2, 2024

Removing activation of python, which we get instead from a designated python environment
Hannah Nesser
committed
Switched tabs to spaces for readability
Hannah Nesser
committed
Changed tabs to spaces and changed hard coded GEOSCHEM_VERSION to the variable specified in run_imi.sh
Hannah Nesser
committed
Changes to allow submit_job to take a SaveOutput true/false boolean
Hannah Nesser
committed
Added SaveOut option to submit_job
Hannah Nesser
committed
Added SaveOut option to submit_job
Hannah Nesser
committed
Looks like I accidentally deletd a fi?
Hannah Nesser
committed
Added a missing fi
Hannah Nesser
committed
Updated config for Pleiades
Hannah Nesser
committed

Commits on Jul 3, 2024

Resolving merge conflicts with bugfix/jacobian-perturbation-fix
Hannah Nesser
committed
Switching from Simulation CPUs/Memory to Requested CPUs/Memory
Hannah Nesser
committed
Removing commented out vestige
Hannah Nesser
committed
Removing lingering references to TROPOMI
Hannah Nesser
committed
Updating options to better match bugfix/jacobian-perturbation-fix config files
Hannah Nesser
committed
UseEmisSF and UseOHSF seem to have been removed as options
Hannah Nesser
committed
- Bug fix for BCs being used in all simulations, not just regional simulations
Hannah Nesser
committed
Beginning to generalize HEMCO tracer changes for not just methane
Hannah Nesser
committed
Generalized function to be a function of species
Hannah Nesser
committed
Changed from tabs to spaces for readability and removed portions of code (UseEmisSF/UseOHSF if statemetns and also default analyticalInv setting) that were removed in the jacobian bug fix branch
Hannah Nesser
committed
Updating
Hannah Nesser
committed
Merge resolution with dev branch
Hannah Nesser
committed

Commits on Nov 2, 2024

Resolving merge conflicts
Hannah Nesser
committed

Commits on Jun 2, 2025

Commits on Jun 3, 2025

Commits on Jun 6, 2025

For whatever reason, we now need to source bash first so that we can use the module command
Hannah Nesser
committed
Bug fixes
Hannah Nesser
committed
Bug fix for slurm to pbs conversion
Hannah Nesser
committed
Bug fix for slurm to pbs conversion
Hannah Nesser
committed