|
| 1 | +====================== |
| 2 | +eScience 2025 Tutorial |
| 3 | +====================== |
| 4 | + |
| 5 | +This directory contains the materials for the eScience 2025 tutorial. The following subsections go over the contains of the material. |
| 6 | + |
| 7 | +-------- |
| 8 | +Contents |
| 9 | +-------- |
| 10 | + |
| 11 | +^^^^^^^^^^^^^ |
| 12 | +Tutorial Code |
| 13 | +^^^^^^^^^^^^^ |
| 14 | + |
| 15 | +The code elements of this tutorial (e.g., Jupyter notebooks, command-line scripts, Markdown/RST instruction files) can all be found in the :code:`tutorial-code` subdirectory. If materials are actually stored in other git repositories, they can be accessed from this subdirectory |
| 16 | +via a git submodule. |
| 17 | + |
| 18 | +^^^^^^ |
| 19 | +Slides |
| 20 | +^^^^^^ |
| 21 | + |
| 22 | +The slides used in presenting this tutorial can be found in the :code:`slides` subdirectory. |
| 23 | + |
| 24 | +^^^^^^ |
| 25 | +Docker |
| 26 | +^^^^^^ |
| 27 | + |
| 28 | +The Docker definition files (i.e., Dockerfiles) for all the necessary containers can be found in the :code:`docker` subdirectory. There are currently 5 definition files: |
| 29 | + |
| 30 | +1. :code:`Dockerfile.caliper`: builds Caliper and Adiak on top of the :code:`ubuntu/noble` image from DockerHub |
| 31 | +2. :code:`Dockerfile.thicket`: build Thicket on top of the image produced by :code:`Dockerfile.caliper` |
| 32 | +3. :code:`Dockerfile.benchpark`: download and bootstrap Benchpark on top of the image produced by :code:`Dockerfile.benchpark` |
| 33 | +4. :code:`Dockerfile.spawn`: download tutorial materials, download any remaining necessary packages, and do other setup work on top of the image produced by :code:`Dockerfile.benchpark` |
| 34 | +5. :code:`Dockerfile.init`: ensure user permissions are correct using the super-minimal :code:`alpine/git` image from DockerHub |
| 35 | + |
| 36 | +""""""""""""""""""""""""""""""""""""""" |
| 37 | +Testing the Builds of the Docker Images |
| 38 | +""""""""""""""""""""""""""""""""""""""" |
| 39 | + |
| 40 | +To enable automated testing of the Docker images, all edits to the Dockerfiles above should be done in a branch with an open PR. When a PR is open, a GitHub Actions CI will |
| 41 | +run and ensure that the images can be built. To properly configure the CI, edit the :code:`github_ci_matrix.json` file in the root of this repository as follows: |
| 42 | + |
| 43 | +1. Edit the "tag" field to be the tag (i.e., version) of the Docker images you will be generating |
| 44 | +2. Edit the "tutorial_dir" field to the name of this directory |
| 45 | + |
| 46 | +The CI reads :code:`github_ci_matrix.json` to get values shared by the matrices of all GitHub Actions jobs. |
| 47 | + |
| 48 | +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" |
| 49 | +Pushing the Docker Images to GitHub Container Registry (GHCR) |
| 50 | +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" |
| 51 | + |
| 52 | +Before trying to push to GHCR, someone with the necessary permissions should make sure this repo can push to these images in GHCR (**change names when we decide on appropriate ones**): |
| 53 | + |
| 54 | +* ghcr.io/llnl/caliper |
| 55 | +* ghcr.io/llnl/thicket |
| 56 | +* ghcr.io/llnl/benchpark |
| 57 | +* ghcr.io/llnl/reproducible-benchmarking-spawn |
| 58 | +* ghcr.io/llnl/reproducible-benchmarking-init |
| 59 | + |
| 60 | +If these images do not yet exist, your first push will properly set the permissions. If these images do exist, follow the instructions |
| 61 | +`here <https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility#ensuring-workflow-access-to-your-package>`_ |
| 62 | +to add this repository to each package. Make sure to grant "Write" permissions to the repository while doing this. |
| 63 | + |
| 64 | +After ensuring this repository has the necessary permissions, to push the Docker images to GHCR, follow these steps: |
| 65 | + |
| 66 | +1. Make sure all changes to the Dockerfiles have been merged into the :code:`main` branch |
| 67 | +2. From the GitHub webpage, navigate to the "Actions" tab |
| 68 | +3. On the left of the resulting page, click on "Build containers and push to GHCR" |
| 69 | +4. Click on the "Run workflow" button to the right of the page |
| 70 | +5. In the popup menu that appears, select the "main" branch and fill out the requested information |
| 71 | +6. Click the green "Run workflow" button to start the process and building and pushing images |
| 72 | + |
| 73 | +^^^^^^^^^^^^^^ |
| 74 | +Infrastructure |
| 75 | +^^^^^^^^^^^^^^ |
| 76 | + |
| 77 | +All the infrastructure needed to deploy the tutorial to a Kubernetes cluster with JupyterHub is contained in the :code:`infrastructure` subdirectory. |
| 78 | +This infrastructure is generated by the tool `here <https://lc.llnl.gov/gitlab/lumsden1/hpcic-k8s-configurer>`_. |
| 79 | +The infrastructure can be regenerated as-is using :code:`infrastructure/config.toml`. |
| 80 | + |
| 81 | +---------------------------- |
| 82 | +Testing the Tutorial Locally |
| 83 | +---------------------------- |
| 84 | + |
| 85 | +To test the tutorial locally, you first need to build all the Docker images except the init image. Before building, |
| 86 | +keep in mind the following dependencies between images: |
| 87 | + |
| 88 | +.. code-block:: |
| 89 | +
|
| 90 | + ghcr.io/llnl/caliper --> ghcr.io/llnl/thicket --> ghcr.io/llnl/benchpark --> ghcr.io/llnl/reproducible-benchmarking-spawn |
| 91 | +
|
| 92 | +Because of these dependencies, the first thing you should figure out is which (if any) images you need to build locally. |
| 93 | +If a Dockerfile has changes that are **not** on GHCR, you will need to build that image *and all downstream images (based on the flowchart above)* |
| 94 | +locally before testing. To build an image locally, run the following from this directory (**not the** :code:`docker` **directory**): |
| 95 | + |
| 96 | +.. code-block:: bash |
| 97 | +
|
| 98 | + $ docker build -t <image_name> -f ./docker/<dockerfile_for_image> . # Note the trailing "." |
| 99 | +
|
| 100 | +In the command above, :code:`<image_name>` should be one of the GHCR URLs above, followed by a colon, followed by a tag. It could look something |
| 101 | +like :code:`ghcr.io/llnl/benchpark:escience-2025`. Note that :code:`<iamge_name>` **must** match the value of the :code:`FROM` directive |
| 102 | +for the dependent image. For example, to get the :code:`<image_name>` field for :code:`ghcr.io/llnl/benchpark`, look for the :code:`FROM` directive |
| 103 | +in :code:`./docker/Dockerfile.spawn`. |
| 104 | + |
| 105 | +If all the changes to the corresponding Dockerfiles in :code:`docker` have already been pushed to GHCR, you do not need to build locally. |
| 106 | +Instead, you should just pull the spawn image using: |
| 107 | + |
| 108 | +.. code-block:: bash |
| 109 | +
|
| 110 | + $ docker pull ghcr.io/llnl/reproducible-benchmarking-spawn:<tag> |
| 111 | +
|
| 112 | +You should replace :code:`<tag>` in the command above with the GHCR tag of the image you want to pull. |
| 113 | + |
| 114 | +After you have a built spawn image (either by building locally or by pulling from GHCR), you can run the spawn image locally |
| 115 | +by running the following command: |
| 116 | + |
| 117 | +.. code-block:: bash |
| 118 | +
|
| 119 | + $ docker run --rm -it --entrypoint <entrypoint> --name reproducible_benchmark_tutorial_local -p 8888:8888 <spawn_image_name> |
| 120 | +
|
| 121 | +In the command above, :code:`<spawn_image_name>` is the name of the built spawn image. If you built that image locally, this argument |
| 122 | +should match the value you passed to the :code:`-t` flag of :code:`docker build` when building the spawn image. If you pulled the image |
| 123 | +from GHCR, this argument should be :code:`ghcr.io/llnl/reproducible-benchmarking-spawn:<tag>`. |
| 124 | + |
| 125 | +The :code:`<entrypoint>` field in the command above dictates what command runs within the container immediately after startup. |
| 126 | +It can be one of three values: |
| 127 | + |
| 128 | +1. :code:`/local-entrypoint.sh`: this entrypoint script will start a JupyterLab instance and make it available from outside the container. |
| 129 | +2. :code:`/entrypoint.sh`: this entrypoint script will run :code:`jupyterhub-singleuser`. It is intended for use in the cloud JupyterHub deployment and should not be used locally. |
| 130 | +3. :code:`bash`: by specifying :code:`bash` (or any other shell installed in the container), you will get command-line access to the container, instead of a Jupyter environment. |
| 131 | + |
| 132 | +At this point, you should either have a Jupyter URL that you can use to access Jupyter, or you should have shell access to the container. |
| 133 | +You can now do whatever local testing you want of the image. |
| 134 | + |
| 135 | +------------------------------------ |
| 136 | +Deploying the Tutorial to Kubernetes |
| 137 | +------------------------------------ |
| 138 | + |
| 139 | +TBA |
0 commit comments