-
Notifications
You must be signed in to change notification settings - Fork 5
Add documentation about the Docker judge #479
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Bond-009
wants to merge
8
commits into
dodona-edu:main
Choose a base branch
from
Bond-009:docker-docs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
9bc87ed
Add docs on how to create exercises for the Docker judge
Bond-009 f459dbd
Add links to used projects
Bond-009 4246719
Add documentation for checking the FROM instruction
Bond-009 3808deb
Fix typo
Bond-009 40a7a88
Add info about limitations and the checking of comments
Bond-009 71f5c85
Change formatting and ordering of the documentation
MaybeJustJames 4e5b495
Add Docker to the list of judges
MaybeJustJames c6e99a3
Fix typo
Bond-009 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
--- | ||
title: "Docker judge" | ||
description: "Docker judge" | ||
order: 7 | ||
--- | ||
# Docker judge | ||
|
||
The Docker judge has 4 main functions: | ||
1. Check usage of Docker instructions like `USER` or `WORKDIR` using [dodona-containerfile-evaluator](https://github.com/Bond-009/dodona-containerfile-evaluator). | ||
2. Static analysis (linting) using [Hadolint](https://github.com/hadolint/hadolint) to annotate the code and optionally fail the solution if it contains errors. | ||
3. Build the submitted Dockerfile using [kaniko](https://github.com/GoogleContainerTools/kaniko). | ||
4. Check the existence and/or contents of files and directories in the image that is build from the submitted Dockerfile. | ||
|
||
Items 1 and 2 above are configurable. | ||
|
||
|
||
## Configuring Docker instructions | ||
|
||
Create a file called `judge.json` inside of the `evaluation` directory. | ||
Using this file it's possible to verify the base image, the `from` object requires the `image` to contain the image name, optionally a `tag` or `hash` property can be added to check the used tag or digest. | ||
It's also possible to check if the `USER` or `WORKDIR` instructions are used with the desired arguments. | ||
Another feature is the ability to check if the comments contain certain strings using the `comments` array. | ||
This file also contains a `files` property that is a JSON array of JSON objects. | ||
Each object contains a `path` property with the desired location in the resulting image. | ||
The object represents either a file or a directory, this is specified by the `type` property. | ||
Additionally objects representing files can also contain a `compare` or `regex` property. | ||
`compare` should be the name of a file inside of the `workdir` with which the file at `path` should be compared with. | ||
`regex` contains a ("extended") "regular expression that should match on the content of the file at `path`. | ||
|
||
An example `judge.json`: | ||
```json | ||
{ | ||
"from": { | ||
"image": "alpine", | ||
"tag": "3:20" | ||
}, | ||
"user": "runner", | ||
"workdir": "/course", | ||
"comments": [ "docker run" ], | ||
"files": [ | ||
{ "type": "directory", "path": "/course" }, | ||
{ "type": "file", "path": "/environment.yml", "compare": "environment.yml" }, | ||
{ "type": "directory", "path": "/usr/miniconda3/envs/pipeline-tools-1.0.0" }, | ||
{ "type": "file", "path": "/etc/os-release", "regex": "^ID=\"ubuntu\"$" } | ||
] | ||
} | ||
``` | ||
|
||
## Configuring Hadolint | ||
|
||
The behavior of Hadolint can be modified by creating a [Hadolint configuration file](https://github.com/hadolint/hadolint#configure) with one of the following names inside of the `evaluation` directory: `.hadolint.yml`, `.hadolint.yaml`, `hadolint.yml` or `hadolint.yaml`. | ||
Using this it's possible to change the severity of annotations, fail when a certain severity is hit, check if labels exist and more. | ||
|
||
An example configuration file looks as follows: | ||
```yml | ||
no-fail: false | ||
failure-threshold: error | ||
label-schema: | ||
org.opencontainers.image.title: text | ||
org.opencontainers.image.description: text | ||
org.opencontainers.image.source: text | ||
org.opencontainers.image.revision: text | ||
org.opencontainers.image.licenses: text | ||
org.opencontainers.image.vendor: text | ||
override: | ||
error: | ||
- DL3049 | ||
ignored: | ||
- DL3004 | ||
``` | ||
|
||
`no-fail` specifies if the judge should fail the solution if the linter fails. | ||
`failure-threshold` sets the threshold for which rules will cause a fail, valid values are `error`, `warning`, `info` and `style`. | ||
`label-schema` is a dictionary where the key is the label name and the value of a label can be either of `text`, `url`, `semver`, `hash`, `rfc3339`, `spdx` or `email`. | ||
`override` is used to override the severity of rules, valid severities are the same as for `failure-threshold`. | ||
`ignored` is a list or rules that should be ignored. | ||
More configuration options can be found in the [Hadolint README](https://github.com/hadolint/hadolint#configure) | ||
|
||
|
||
## Limitations | ||
Due to the design of the judge it isn't possible to test the existence of files if no `COPY`, `RUN` or `ADD` instructions are present in the final stage. | ||
For example, if the following solution is submitted, a test for the existence of the file `/etc/os-release` which is present in the alpine base image will fail. | ||
```docker | ||
FROM alpine:3.21 | ||
|
||
USER runner | ||
|
||
WORKDIR /home/runner | ||
|
||
LABEL org.opencontainers.image.title="test" | ||
``` | ||
But once you add one of the instructions that can modify the filesystem this same test will succeed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,6 +20,12 @@ C is a judge that uses the GTester framework to run tests on C exercises.\ | |
**Get started** [Documentation](https://github.com/mvdcamme/C-Judge), [examples](https://github.com/mvdcamme/C-Judge/tree/master/example_exercises) \ | ||
**Created by:** [Maarten Vandercammen](mailto:mvdcamme@vub.ac.be) | ||
|
||
## Docker | ||
Docker is a judge that uses dodona-containerfile-evaluator, Hadolint, and Kaniko to evaluate Dockerfile exercises. | ||
**Programming languages**: Dockerfile | ||
**Get started** [Git repo](https://github.com/dodona-edu/judge-docker), [examples](https://github.com/Bond-009/dodona-docker-exercises) \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You should probably also add a link to the documentation provided in this pr |
||
**Created by:** [Lode Willems](mailto:bits@vib.be) | ||
|
||
## Haskell | ||
Haskell is a judge that uses HUnit to test Haskell exercises. \ | ||
**Programming languages:** Haskell\ | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bmesuere what's your opinion on adding these docs here?
For context, most judges except python and TESTed have their documentation in their respective github repo. With a link from our documentation.
Having it here will probably make it more discoverable, but might also create clutter for inexperienced users, as they probably won't need this more niche judge.
So I am a bit in doubt how best to proceed