Mime type reload #13620
Mime type reload #13620
Conversation
File magic is known to have performance issue. When looking for an alternative, I've been pointed to the tree_magic_mini crate that output the mime type of a binary stream. This is different from magic but it has multiple advantages as it is a standard so it can be correlated with other tools. So instead of replacing magic, this patch adds a new mime type output to fileinfo events. This patch also adds the `file.mimetype` sticky buffer. Mime type has 2 advantages over file magic. First it is really faster and second, the result are easier to use as the MIME type are well defined. But it provides less information than magic for example with regards to the size of images. Ticket: 7816
This patch adds a flag `--disable-gpl-mimetype` to skip the usage of `tree_magic_db` that is GPL license and will not allow double licensing. Other potential usage is to use the fallback method if the user wants to tune or update the DB without rebuilding Suricata.
As file data may have not been inspected yet, this is going to miss in the event even if it could be available. Let's compute it before logging the file. Ticket: 7816
This can be used with the requires keyword to avoid signatures load failure. Ticket: 7816
regit
requested review from
jasonish,
jufajardini,
victorjulien and
a team
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #13620 +/- ##
========================================
Coverage 83.69% 83.69%
========================================
Files 1011 1014 +3
Lines 275071 275231 +160
========================================
+ Hits 230210 230356 +146
- Misses 44861 44875 +14
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
The CI/Fuzz failures seem related to a Rust library and some interaction with fuzz (thus unrelated to this PR) |
There was a problem hiding this comment.
I think we should break this into 2 different tasks.
First, the feature this PR is about. As it uses GPL data, it should be opt-in, not opt-out. This is how we deal with other GPL libraries such as NFQ.
Then a second task of adding the opinionated license checks. I know some work already exists to just warn on this, as I'm not sure we should actually fail the build. So discussion is still required around that. And it should be complete when we add it. For example, ndpi and nfqueue bring in license issues. Maybe others?
| if test "$gpl_license_only" = "yes" and "$proprietary_license_only" = "yes"; then | ||
| echo "ERROR: The build options mix GPL and proprietary licenses." | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
Should we indicate in the error message what could be changed to fix the situation?
|
Closing in favor of updated PR. |
Implement mime type computation and matching using a rust crate. It also updates the configure output to specify the license compatibility and potentially trigger a fail if license are incompatible (like building Napatech and including mimetype GPL data).
Contribution style:
https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html
Our Contribution agreements:
https://suricata.io/about/contribution-agreement/ (note: this is only required once)
Changes (if applicable):
(including schema descriptions)
https://redmine.openinfosecfoundation.org/projects/suricata/issues
Link to ticket: https://redmine.openinfosecfoundation.org/issues/7816
Describe changes:
SV_BRANCH=OISF/suricata-verify#2606