You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/2024/ci-scanner/index.md
+31-18Lines changed: 31 additions & 18 deletions
Original file line number
Diff line number
Diff line change
@@ -24,24 +24,37 @@ CI Scanner Improvements
24
24
25
25
## What's the project about?
26
26
27
-
To be able to easily and continuously scan packages with
28
-
fossology checks in CI pipelines, a docker image
29
-
(fossology/fossology:scanner) capable of running license checks (using
30
-
nomos or ojo) and keyword and copyright scans is available.
31
-
This project aims to improve the CI Scanner Image in
32
-
various aspects and numerous quality of life improvements, like
33
-
highlighting the exact location of violation, ability to customize the
34
-
keywords used by the scanner, and improving user experience – allow
35
-
whitelisting from a custom location and ability to download and scan
36
-
dependencies
27
+
To be able to easily and continuously scan packages with
28
+
fossology checks in CI pipelines, a docker image [fossology/fossology:scanner](https://hub.docker.com/layers/fossology/fossology/scanner/images/sha256-a625b1b10832b98d47429387c18b4fb042f7b09f912b50da14da61fddb11a2ff?context=explore) capable of running license checks (using nomos or ojo) and keyword and copyright scans is available.
29
+
30
+
The main aims of this projects is to improve the CI pipeline with various quality of life improvements like:
31
+
- Highlight the exact location of violations in the results.
32
+
- Enable customization of keywords used by the scanner.
33
+
- Allow whitelisting from a custom location.
34
+
- Provide the ability to download and scan dependencies.
35
+
37
36
38
37
## What should be done?
39
38
40
-
1. Highlight the exact location (line number) of a violation during
41
-
reporting
42
-
2. Allow users to customize keyword scanning using their own
43
-
keyword.conf
44
-
3. Allow users to store allowlist.json file elsewhere (currently, it is
45
-
required to be present at the root of the project)
46
-
4. Allow users to download and scan dependencies by providing a path
47
-
at CI/CD pipeline trigger.
39
+
### Reporting line numbers for violations
40
+
- For calculating the previous and new line number from the diff scan output, an algorithm has to be made.
41
+
- The line number start byte and end byte information is spit out by all scanners except nomos in json output. That has to be fixed.
42
+
- Add the line number calculated to the finding log information as well as write it in results file.
43
+
44
+
### Keyword scanning using custom keyword.conf
45
+
- Currently, the keyword scanner uses a predefined set of keywords stored at `/usr/local/share/fossology/keyword/agent/keyword.conf.`
46
+
- To support this, we also need to document the regex-like format used for specifying these keywords.
47
+
- Decision to be made : Should custom `keyword.conf` overwrite the previous one?
48
+
49
+
### Providing allowlist.json from a different path
50
+
- Currently, the `allowlist.json` is located at the root of the project.
51
+
- We want to allow users to optionally specify a different path, using a CLI argument, like --allowlist
52
+
53
+
### Allow users to download and scan dependencies
54
+
- Currently, the project only scans the source code of the project either in repo/diff manner.
55
+
- We additionally want to allow the functionality to scan and dependencies of the project.
56
+
57
+
#### Steps to achieve this:
58
+
- With the [CycloneDX](https://cyclonedx.org/tool-center/) tool center, we can generate SBOM which contains the dependency download url.
59
+
- The SBOM format specifies the package URL (purl) for each dependency.
60
+
- Using the [python-packageurl](https://github.com/package-url/packageurl-python#purl-to-url) tool, we can extract the download url from the purl for this purpose.
- Set up a Virtual Machine because fossology does not work on Mac.:pensive:
34
+
- Faced challenges installing some python packages which were not available for aarch64 architecture.
35
+
Worked around the issue by commenting [this out](https://github.com/fossology/fossology/blob/6e6b00c2ded6a1db7647d0da9e97c78ed9ffddf8/install/fo-postinstall.in#L261-L263).
36
+
37
+
# Community Bonding Week 2
38
+
39
+
*(May 16, 2024 - May 23, 2024)*
40
+
41
+
### Meeting 3
28
42
29
43
*(May 16, 2024)*
30
44
31
45
### Discussion:
32
-
- Discussed a problem in setting up my development environment on mac.
33
46
- Contributors shared their weekly updates.
34
47
- Discussed and decided time for weekly project specific meetings.
48
+
- Mentors talked about the importance of open communication in open source.
49
+
50
+
### Work Done:
51
+
- Played around with fossology and scanned a few repositories.
52
+
- Tried to understand how scanners work internally.
53
+
- Talked with other contributors about the project.
35
54
36
-
## Meeting 4
55
+
# Community Bonding Week 3
56
+
57
+
*(May 23, 2024 - May 30, 2024)*
58
+
59
+
### Meeting 4
37
60
38
61
*(May 23, 2024)*
39
62
40
63
### Discussion:
41
-
- Everyone had final discussions on projects.
42
-
- Made sure everyone was on the same page.
43
-
- Understood the fossology codebase and asked few doubts on how to approach the line number task to mentors.
64
+
- Had final discussions on projects.
65
+
- Finalized any changes to the project milestones.
66
+
67
+
### Work Done:
68
+
- Tried building the fossology scanner image locally.
69
+
- Played around a while with the image, trying to understand what it does.
70
+
- Faced an issue of UI freezing in the VM. Solved it by using [SSH Remote Tunneling](https://code.visualstudio.com/docs/remote/ssh)
71
+
- Started to theorize how the line number algorithm would work.
72
+
73
+
74
+
### Meeting 5
75
+
*(May 30, 2024)*
76
+
- Discussed the project updates from the mentors and contributors.
77
+
- Got clear understanding of how to document our progress during the whole program.
78
+
- Had discussions with the mentors about how to approach the algorithm for line numbers.
79
+
80
+
***This summarizes my community bonding period at Fossology***
0 commit comments