Skip to content

Add documentation about the Docker judge #479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions en/references/judges/docker-judge/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
---
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bmesuere what's your opinion on adding these docs here?

For context, most judges except python and TESTed have their documentation in their respective github repo. With a link from our documentation.
Having it here will probably make it more discoverable, but might also create clutter for inexperienced users, as they probably won't need this more niche judge.
So I am a bit in doubt how best to proceed

title: "Docker judge"
description: "Docker judge"
order: 7
---
# Docker judge

The Docker judge has 4 main functions:
1. Check usage of Docker instructions like `USER` or `WORKDIR` using [dodona-containerfile-evaluator](https://github.com/Bond-009/dodona-containerfile-evaluator).
2. Static analysis (linting) using [Hadolint](https://github.com/hadolint/hadolint) to annotate the code and optionally fail the solution if it contains errors.
3. Build the submitted Dockerfile using [kaniko](https://github.com/GoogleContainerTools/kaniko).
4. Check the existence and/or contents of files and directories in the image that is build from the submitted Dockerfile.

Items 1 and 2 above are configurable.


## Configuring Docker instructions

Create a file called `judge.json` inside of the `evaluation` directory.
Using this file it's possible to verify the base image, the `from` object requires the `image` to contain the image name, optionally a `tag` or `hash` property can be added to check the used tag or digest.
It's also possible to check if the `USER` or `WORKDIR` instructions are used with the desired arguments.
Another feature is the ability to check if the comments contain certain strings using the `comments` array.
This file also contains a `files` property that is a JSON array of JSON objects.
Each object contains a `path` property with the desired location in the resulting image.
The object represents either a file or a directory, this is specified by the `type` property.
Additionally objects representing files can also contain a `compare` or `regex` property.
`compare` should be the name of a file inside of the `workdir` with which the file at `path` should be compared with.
`regex` contains a ("extended") "regular expression that should match on the content of the file at `path`.

An example `judge.json`:
```json
{
"from": {
"image": "alpine",
"tag": "3:20"
},
"user": "runner",
"workdir": "/course",
"comments": [ "docker run" ],
"files": [
{ "type": "directory", "path": "/course" },
{ "type": "file", "path": "/environment.yml", "compare": "environment.yml" },
{ "type": "directory", "path": "/usr/miniconda3/envs/pipeline-tools-1.0.0" },
{ "type": "file", "path": "/etc/os-release", "regex": "^ID=\"ubuntu\"$" }
]
}
```

## Configuring Hadolint

The behavior of Hadolint can be modified by creating a [Hadolint configuration file](https://github.com/hadolint/hadolint#configure) with one of the following names inside of the `evaluation` directory: `.hadolint.yml`, `.hadolint.yaml`, `hadolint.yml` or `hadolint.yaml`.
Using this it's possible to change the severity of annotations, fail when a certain severity is hit, check if labels exist and more.

An example configuration file looks as follows:
```yml
no-fail: false
failure-threshold: error
label-schema:
org.opencontainers.image.title: text
org.opencontainers.image.description: text
org.opencontainers.image.source: text
org.opencontainers.image.revision: text
org.opencontainers.image.licenses: text
org.opencontainers.image.vendor: text
override:
error:
- DL3049
ignored:
- DL3004
```

`no-fail` specifies if the judge should fail the solution if the linter fails.
`failure-threshold` sets the threshold for which rules will cause a fail, valid values are `error`, `warning`, `info` and `style`.
`label-schema` is a dictionary where the key is the label name and the value of a label can be either of `text`, `url`, `semver`, `hash`, `rfc3339`, `spdx` or `email`.
`override` is used to override the severity of rules, valid severities are the same as for `failure-threshold`.
`ignored` is a list or rules that should be ignored.
More configuration options can be found in the [Hadolint README](https://github.com/hadolint/hadolint#configure)


## Limitations
Due to the design of the judge it isn't possible to test the existence of files if no `COPY`, `RUN` or `ADD` instructions are present in the final stage.
For example, if the following solution is submitted, a test for the existence of the file `/etc/os-release` which is present in the alpine base image will fail.
```docker
FROM alpine:3.21

USER runner

WORKDIR /home/runner

LABEL org.opencontainers.image.title="test"
```
But once you add one of the instructions that can modify the filesystem this same test will succeed.
6 changes: 6 additions & 0 deletions en/references/judges/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,12 @@ C is a judge that uses the GTester framework to run tests on C exercises.\
**Get started** [Documentation](https://github.com/mvdcamme/C-Judge), [examples](https://github.com/mvdcamme/C-Judge/tree/master/example_exercises) \
**Created by:** [Maarten Vandercammen](mailto:mvdcamme@vub.ac.be)

## Docker
Docker is a judge that uses dodona-containerfile-evaluator, Hadolint, and Kaniko to evaluate Dockerfile exercises.
**Programming languages**: Dockerfile
**Get started** [Git repo](https://github.com/dodona-edu/judge-docker), [examples](https://github.com/Bond-009/dodona-docker-exercises) \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should probably also add a link to the documentation provided in this pr

**Created by:** [Lode Willems](mailto:bits@vib.be)

## Haskell
Haskell is a judge that uses HUnit to test Haskell exercises. \
**Programming languages:** Haskell\
Expand Down