From 24da380195a36ff832ef831becb7066b0e27298f Mon Sep 17 00:00:00 2001 From: rajuljha Date: Wed, 5 Jun 2024 14:52:52 +0530 Subject: [PATCH 1/3] chore(report) : Add community bonding report --- docs/2024/ci-scanner/index.md | 26 +++++++++++++--- docs/2024/ci-scanner/updates/2023-05-30.md | 36 ++++++++++++++++------ 2 files changed, 48 insertions(+), 14 deletions(-) diff --git a/docs/2024/ci-scanner/index.md b/docs/2024/ci-scanner/index.md index d0acb50500..1f8c81390d 100644 --- a/docs/2024/ci-scanner/index.md +++ b/docs/2024/ci-scanner/index.md @@ -6,7 +6,7 @@ slug: /2024/ci-scanner/ ## Author @@ -15,8 +15,8 @@ SPDX-FileCopyrightText: 2024 Rajul Jha ## Contact info -- [Email](mailto:email.here) -- [LinkedIn](https://linkedin.com/in/my-user) +- [Email](mailto:rajuljha49@gmail.com) +- [LinkedIn](https://linkedin.com/in/rajuljha) ## Project title @@ -24,8 +24,24 @@ CI Scanner Improvements ## What's the project about? -Insert Text Here +To be able to easily and continuously scan packages with +fossology checks in CI pipelines, a docker image +(fossology/fossology:scanner) capable of running license checks (using +nomos or ojo) and keyword and copyright scans is available. +This project aims to improve the CI Scanner Image in +various aspects and numerous quality of life improvements, like +highlighting the exact location of violation, ability to customize the +keywords used by the scanner, and improving user experience – allow +whitelisting from a custom location and ability to download and scan +dependencies ## What should be done? -What are the plans for the project? +1. Highlight the exact location (line number) of a violation during +reporting +2. Allow users to customize keyword scanning using their own +keyword.conf +3. Allow users to store allowlist.json file elsewhere (currently, it is +required to be present at the root of the project) +4. Allow users to download and scan dependencies by providing a path +at CI/CD pipeline trigger. \ No newline at end of file diff --git a/docs/2024/ci-scanner/updates/2023-05-30.md b/docs/2024/ci-scanner/updates/2023-05-30.md index 4af3bece4a..00a06f2d8b 100644 --- a/docs/2024/ci-scanner/updates/2023-05-30.md +++ b/docs/2024/ci-scanner/updates/2023-05-30.md @@ -5,21 +5,39 @@ author: Rajul Jha -# Meeting 1 +## Meeting 1 -*(May 30,2024)* +*(May 7, 2024)* -## Attendees: +### Discussion: +- Could not attend due to family emergency. -## Discussion: -# Meeting 2 +## Meeting 2 -*(May 18,2023)* +*(May 9, 2024)* -## Attendees: +### Discussion: +- I gave my introduction in the meeting. +- Got to know my mentors and colleagues. -## Discussion: +## Meeting 3 + +*(May 16, 2024)* + +### Discussion: +- Discussed a problem in setting up my development environment on mac. +- Contributors shared their weekly updates. +- Discussed and decided time for weekly project specific meetings. + +## Meeting 4 + +*(May 23, 2024)* + +### Discussion: +- Everyone had final discussions on projects. +- Made sure everyone was on the same page. +- Understood the fossology codebase and asked few doubts on how to approach the line number task to mentors. From d47aacef8c5ded0ecf713945abaf580823cd4461 Mon Sep 17 00:00:00 2001 From: rajuljha Date: Thu, 6 Jun 2024 10:25:08 +0530 Subject: [PATCH 2/3] chore(report) : Update report date --- docs/2024/ci-scanner/updates/{2023-05-30.md => 2024-05-07.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/2024/ci-scanner/updates/{2023-05-30.md => 2024-05-07.md} (100%) diff --git a/docs/2024/ci-scanner/updates/2023-05-30.md b/docs/2024/ci-scanner/updates/2024-05-07.md similarity index 100% rename from docs/2024/ci-scanner/updates/2023-05-30.md rename to docs/2024/ci-scanner/updates/2024-05-07.md From 990fa2a1fa4c1e4edd2cd3b9d09c4915d1f8d14d Mon Sep 17 00:00:00 2001 From: rajuljha Date: Thu, 6 Jun 2024 18:37:15 +0530 Subject: [PATCH 3/3] chore(report) : Add detailed report for community bonding and elaborated project info --- docs/2024/ci-scanner/index.md | 49 ++++++++++++-------- docs/2024/ci-scanner/updates/2024-05-07.md | 53 ++++++++++++++++++---- 2 files changed, 76 insertions(+), 26 deletions(-) diff --git a/docs/2024/ci-scanner/index.md b/docs/2024/ci-scanner/index.md index 1f8c81390d..4cd845e015 100644 --- a/docs/2024/ci-scanner/index.md +++ b/docs/2024/ci-scanner/index.md @@ -24,24 +24,37 @@ CI Scanner Improvements ## What's the project about? -To be able to easily and continuously scan packages with -fossology checks in CI pipelines, a docker image -(fossology/fossology:scanner) capable of running license checks (using -nomos or ojo) and keyword and copyright scans is available. -This project aims to improve the CI Scanner Image in -various aspects and numerous quality of life improvements, like -highlighting the exact location of violation, ability to customize the -keywords used by the scanner, and improving user experience – allow -whitelisting from a custom location and ability to download and scan -dependencies +To be able to easily and continuously scan packages with +fossology checks in CI pipelines, a docker image [fossology/fossology:scanner](https://hub.docker.com/layers/fossology/fossology/scanner/images/sha256-a625b1b10832b98d47429387c18b4fb042f7b09f912b50da14da61fddb11a2ff?context=explore) capable of running license checks (using nomos or ojo) and keyword and copyright scans is available. + +The main aims of this projects is to improve the CI pipeline with various quality of life improvements like: +- Highlight the exact location of violations in the results. +- Enable customization of keywords used by the scanner. +- Allow whitelisting from a custom location. +- Provide the ability to download and scan dependencies. + ## What should be done? -1. Highlight the exact location (line number) of a violation during -reporting -2. Allow users to customize keyword scanning using their own -keyword.conf -3. Allow users to store allowlist.json file elsewhere (currently, it is -required to be present at the root of the project) -4. Allow users to download and scan dependencies by providing a path -at CI/CD pipeline trigger. \ No newline at end of file +### Reporting line numbers for violations +- For calculating the previous and new line number from the diff scan output, an algorithm has to be made. +- The line number start byte and end byte information is spit out by all scanners except nomos in json output. That has to be fixed. +- Add the line number calculated to the finding log information as well as write it in results file. + +### Keyword scanning using custom keyword.conf +- Currently, the keyword scanner uses a predefined set of keywords stored at `/usr/local/share/fossology/keyword/agent/keyword.conf.` +- To support this, we also need to document the regex-like format used for specifying these keywords. +- Decision to be made : Should custom `keyword.conf` overwrite the previous one? + +### Providing allowlist.json from a different path +- Currently, the `allowlist.json` is located at the root of the project. +- We want to allow users to optionally specify a different path, using a CLI argument, like --allowlist + +### Allow users to download and scan dependencies +- Currently, the project only scans the source code of the project either in repo/diff manner. +- We additionally want to allow the functionality to scan and dependencies of the project. + +#### Steps to achieve this: + - With the [CycloneDX](https://cyclonedx.org/tool-center/) tool center, we can generate SBOM which contains the dependency download url. + - The SBOM format specifies the package URL (purl) for each dependency. + - Using the [python-packageurl](https://github.com/package-url/packageurl-python#purl-to-url) tool, we can extract the download url from the purl for this purpose. \ No newline at end of file diff --git a/docs/2024/ci-scanner/updates/2024-05-07.md b/docs/2024/ci-scanner/updates/2024-05-07.md index 00a06f2d8b..58537ac32d 100644 --- a/docs/2024/ci-scanner/updates/2024-05-07.md +++ b/docs/2024/ci-scanner/updates/2024-05-07.md @@ -8,7 +8,7 @@ SPDX-License-Identifier: CC-BY-SA-4.0 SPDX-FileCopyrightText: 2024 Rajul Jha --> -## Meeting 1 +### Meeting 1 *(May 7, 2024)* @@ -16,7 +16,11 @@ SPDX-FileCopyrightText: 2024 Rajul Jha - Could not attend due to family emergency. -## Meeting 2 +# Community Bonding Week 1 + +*(May 9, 2024 - May 15, 2024)* + +### Meeting 2 *(May 9, 2024)* @@ -24,20 +28,53 @@ SPDX-FileCopyrightText: 2024 Rajul Jha - I gave my introduction in the meeting. - Got to know my mentors and colleagues. -## Meeting 3 +### Work Done: +- Tried to setup the coding environment. +- Set up a Virtual Machine because fossology does not work on Mac.:pensive: +- Faced challenges installing some python packages which were not available for aarch64 architecture. +Worked around the issue by commenting [this out](https://github.com/fossology/fossology/blob/6e6b00c2ded6a1db7647d0da9e97c78ed9ffddf8/install/fo-postinstall.in#L261-L263). + +# Community Bonding Week 2 + +*(May 16, 2024 - May 23, 2024)* + +### Meeting 3 *(May 16, 2024)* ### Discussion: -- Discussed a problem in setting up my development environment on mac. - Contributors shared their weekly updates. - Discussed and decided time for weekly project specific meetings. +- Mentors talked about the importance of open communication in open source. + +### Work Done: +- Played around with fossology and scanned a few repositories. +- Tried to understand how scanners work internally. +- Talked with other contributors about the project. -## Meeting 4 +# Community Bonding Week 3 + +*(May 23, 2024 - May 30, 2024)* + +### Meeting 4 *(May 23, 2024)* ### Discussion: -- Everyone had final discussions on projects. -- Made sure everyone was on the same page. -- Understood the fossology codebase and asked few doubts on how to approach the line number task to mentors. +- Had final discussions on projects. +- Finalized any changes to the project milestones. + +### Work Done: +- Tried building the fossology scanner image locally. +- Played around a while with the image, trying to understand what it does. +- Faced an issue of UI freezing in the VM. Solved it by using [SSH Remote Tunneling](https://code.visualstudio.com/docs/remote/ssh) +- Started to theorize how the line number algorithm would work. + + +### Meeting 5 +*(May 30, 2024)* +- Discussed the project updates from the mentors and contributors. +- Got clear understanding of how to document our progress during the whole program. +- Had discussions with the mentors about how to approach the algorithm for line numbers. + +***This summarizes my community bonding period at Fossology*** \ No newline at end of file