Mime type reload v2 #13647

regit · 2025-07-24T18:43:45Z

Update of #13620 addressing comments.

Contribution style:

I have read the contributing guide lines at
https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html

Our Contribution agreements:

I have signed the Open Information Security Foundation contribution agreement at
https://suricata.io/about/contribution-agreement/ (note: this is only required once)

Changes (if applicable):

I have updated the User Guide (in doc/userguide/) to reflect the changes made
I have updated the JSON schema (in etc/schema.json) to reflect all logging changes
(including schema descriptions)
I have created a ticket at
https://redmine.openinfosecfoundation.org/projects/suricata/issues

Link to ticket: https://redmine.openinfosecfoundation.org/issues/

Describe changes:

GPL code inclusion is now optional (so MIT licensed crate is added)
Remove proprietary code detection in configure
Remove GPL warning in configure
Address comments

SV_BRANCH=OISF/suricata-verify#2606

regit · 2025-07-24T18:46:13Z

Setting to draft to see how the CI is going. Will unset if green "enough".

codecov · 2025-07-25T07:41:23Z

Codecov Report

❌ Patch coverage is 56.56566% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.69%. Comparing base (6bbba95) to head (eb76905).
⚠️ Report is 24 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #13647   +/-   ##
=======================================
  Coverage   83.69%   83.69%           
=======================================
  Files        1011     1014    +3     
  Lines      275071   275149   +78     
=======================================
+ Hits       230230   230296   +66     
- Misses      44841    44853   +12

Flag	Coverage Δ
fuzzcorpus	`62.85% <53.53%> (+0.01%)`	⬆️
livemode	`19.01% <30.30%> (-0.17%)`	⬇️
pcap	`44.76% <47.47%> (+0.01%)`	⬆️
suricata-verify	`65.05% <56.56%> (-0.03%)`	⬇️
unittests	`59.17% <44.44%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jufajardini

CI failures: dependency issues? https://github.com/OISF/suricata/actions/runs/16516991164/job/46711126442?pr=13647#step:10:35

etc/schema.json

regit · 2025-07-25T15:19:59Z

CI failures: dependency issues? https://github.com/OISF/suricata/actions/runs/16516991164/job/46711126442?pr=13647#step:10:35

Yes, got an optional crate and need to make sure it is available everywhere.

Following commit will add a new value that will overflow the actual size. Ticket: 7816

File magic is known to have performance issue. When looking for an alternative, I've been pointed to the tree_magic_mini crate that output the mime type of a binary stream. This is different from magic but it has multiple advantages as it is a standard so it can be correlated with other tools. So instead of replacing magic, this patch adds a new mime type output to fileinfo events. This patch also adds the `file.mimetype` sticky buffer. Mime type has 2 advantages over file magic. First it is really faster and second, the result are easier to use as the MIME type are well defined. But it provides less information than magic for example with regards to the size of images. Ticket: 7816

This patch adds a flag `--enable-gpl-mimetype` to force the usage of `tree_magic_db` that is GPL license and will not allow double licensing. This will allow the built Suricata to have mime type identification not dependant of the operating system database. Ticket: 7816

This can be used with the requires keyword to avoid signatures load failure. Ticket: 7816

As file data may have not been inspected yet, this is going to miss in the event even if it could be available. Let's compute it before logging the file. Ticket: 7816

Ticket: 7816

catenacyber · 2025-07-28T11:19:18Z

Why is CI still red ?

doc/userguide/rules/file-keywords.rst

rust/src/lib.rs

catenacyber · 2025-07-28T11:23:28Z

src/suricata.c

    strlcat(features, "HAVE_JA4 ", sizeof(features));
 #endif
    strlcat(features, "HAVE_LIBJANSSON ", sizeof(features));
+    strlcat(features, "HAVE_MIMETYPE ", sizeof(features));


no ifdef around this ?

catenacyber · 2025-07-28T11:26:19Z

src/detect.h

+#define FILE_SIG_NEED_MD5         BIT_U16(4)
+#define FILE_SIG_NEED_SHA1        BIT_U16(5)
+#define FILE_SIG_NEED_SHA256      BIT_U16(6)
+#define FILE_SIG_NEED_SIZE        BIT_U16(7)


Maybe you can start by cherry-picking #13601 ;-)

Show off :P

jufajardini · 2025-07-29T18:24:02Z

Why is CI still red ?

Dependencies -- libraries not found, still 🤔

jasonish · 2025-07-29T18:42:22Z

rust/Cargo.toml.in

 der-parser = { version = "~9.0.0", default-features = false }
 kerberos-parser = { version = "~0.8.0", default-features = false }

+tree_magic_mini = { version = "~3.1.6", features = [@WITH_GPL_DATA@] }


So the support is always compiled in? What happens if the data is not there?

you can load the mime type def from a file on disk. I'll document that.

jasonish · 2025-07-29T18:46:24Z

Would it make more sense to only compile in the mime-type crate if the feature is enabled, so we're not left in the situation where a user starts relying on the operating system files? I know the license always allows linking in, but practically, it might not make sense to.

Tests could then rely on the HAVE_MIMETYPE feature flag and know they are always testing against the same database. Likewise for rule writers.

regit · 2025-07-29T18:55:59Z

Why is CI still red ?

Dependencies -- libraries not found, still 🤔

2 issues:

dependencies are still a problem
mime type definition file may also be a fix

I'll work on it this week.

regit · 2025-07-29T19:06:34Z

Would it make more sense to only compile in the mime-type crate if the feature is enabled, so we're not left in the situation where a user starts relying on the operating system files? I know the license always allows linking in, but practically, it might not make sense to.

Tests could then rely on the HAVE_MIMETYPE feature flag and know they are always testing against the same database. Likewise for rule writers.

I'm fine with it but it means proprietary license user can't get the feature. Should we do:

enable feature with GPL flag: (most common use case)
flag on top of previous to not get the GPL data but get the feature

So GPL build does --enable-mimetype and proprietary build does --enable-mimetype --disable-gpl-mimetype

This will simplify the CI a lot if we get the MIME_TYPE feature conditional doing so.

jasonish · 2025-07-29T19:35:23Z

So GPL build does --enable-mimetype and proprietary build does --enable-mimetype --disable-gpl-mimetype

This will simplify the CI a lot if we get the MIME_TYPE feature conditional doing so.

For the purposes of getting CI working, I think we should have:

--enable-mimetype: enables the feature
--enable-bundled-gpl-mimetype: to turn on the feature for the bundled data

And HAVE_MIMETYPE feature flags for each I suppose, for easy testing with S-V.

I'm not 100% sure that would be the best for a release, but it gets us somewhere, with an opt-in behavior for now.

I took a quick look, and there does not appear to be any other more permissively licensed MIME database out there.

regit · 2025-07-29T19:49:16Z

So GPL build does --enable-mimetype and proprietary build does --enable-mimetype --disable-gpl-mimetype
This will simplify the CI a lot if we get the MIME_TYPE feature conditional doing so.

For the purposes of getting CI working, I think we should have:

--enable-mimetype: enables the feature

--enable-bundled-gpl-mimetype: to turn on the feature for the bundled data

I fear bug report with that approach: I did enable the feature and it is not working. Do we really risk something with proprietary license owner ? They will need to update their compilation to get the feature and should see the second flag. So they should be safe enough ?

jasonish · 2025-07-29T20:28:53Z

So GPL build does --enable-mimetype and proprietary build does --enable-mimetype --disable-gpl-mimetype
This will simplify the CI a lot if we get the MIME_TYPE feature conditional doing so.

For the purposes of getting CI working, I think we should have:

--enable-mimetype: enables the feature

--enable-bundled-gpl-mimetype: to turn on the feature for the bundled data

I fear bug report with that approach: I did enable the feature and it is not working. Do we really risk something with proprietary license owner ? They will need to update their compilation to get the feature and should see the second flag. So they should be safe enough ?

I think enabling anything that pulls in GPL license code/data needs to be opt-in at this time. The only other feature I think that pulls in GPL code is nfqueue, and its opt-in as well.

regit · 2025-07-29T20:32:02Z

So GPL build does --enable-mimetype and proprietary build does --enable-mimetype --disable-gpl-mimetype
This will simplify the CI a lot if we get the MIME_TYPE feature conditional doing so.

For the purposes of getting CI working, I think we should have:

--enable-mimetype: enables the feature

--enable-bundled-gpl-mimetype: to turn on the feature for the bundled data

I fear bug report with that approach: I did enable the feature and it is not working. Do we really risk something with proprietary license owner ? They will need to update their compilation to get the feature and should see the second flag. So they should be safe enough ?

I think enabling anything that pulls in GPL license code/data needs to be opt-in at this time. The only other feature I think that pulls in GPL code is nfqueue, and its opt-in as well.

Yes but --enable-mimetype is disabled by default so it will result in the feature being opt-in.

catenacyber · 2025-07-30T12:21:26Z

CI is green, cool :-) , will you do a new clean PR ?

rust/src/filemimetype.rs

regit · 2025-08-04T19:12:24Z

CI is green, cool :-) , will you do a new clean PR ?

Yes, fixing the last request from Jason and doing that.

regit requested review from jasonish, jufajardini, victorjulien and a team as code owners July 24, 2025 18:43

regit force-pushed the mime-type-reload-v2 branch from 8c68f72 to c104e61 Compare July 24, 2025 18:45

regit marked this pull request as draft July 24, 2025 18:45

regit force-pushed the mime-type-reload-v2 branch 2 times, most recently from c3123c5 to 44c063f Compare July 24, 2025 18:54

regit force-pushed the mime-type-reload-v2 branch from e843bee to a01a3ef Compare July 25, 2025 08:00

jufajardini reviewed Jul 25, 2025

View reviewed changes

etc/schema.json Outdated Show resolved Hide resolved

regit added 7 commits July 25, 2025 17:20

detect: convert file_flags to a uint16_t

49e17be

Following commit will add a new value that will overflow the actual size. Ticket: 7816

file: add HAVE_MIMETYPE feature

748e259

This can be used with the requires keyword to avoid signatures load failure. Ticket: 7816

eve/alert: compute mimetype if needed

9d382e1

As file data may have not been inspected yet, this is going to miss in the event even if it could be available. Let's compute it before logging the file. Ticket: 7816

doc/userguide: add information about file.mimetype

15f14c6

Ticket: 7816

configure: add display about mimetype db inclusion

d1fe8a8

regit force-pushed the mime-type-reload-v2 branch from a01a3ef to 6f251a4 Compare July 25, 2025 15:38

catenacyber reviewed Jul 28, 2025

View reviewed changes

doc/userguide/rules/file-keywords.rst Show resolved Hide resolved

catenacyber reviewed Jul 28, 2025

View reviewed changes

rust/src/lib.rs Show resolved Hide resolved

catenacyber reviewed Jul 28, 2025

View reviewed changes

jasonish reviewed Jul 29, 2025

View reviewed changes

regit force-pushed the mime-type-reload-v2 branch from 6f251a4 to 7b456ce Compare July 29, 2025 20:00

regit added 2 commits July 29, 2025 22:03

detect/mimetype: optional build

c73b656

ci: add a build with mimetype for S-V

f37f5ab

regit force-pushed the mime-type-reload-v2 branch from 7b456ce to f37f5ab Compare July 29, 2025 20:03

ci: gpl mimetype in rust check

eb76905

jasonish reviewed Jul 31, 2025

View reviewed changes

rust/src/filemimetype.rs Show resolved Hide resolved

regit mentioned this pull request Aug 5, 2025

Mime type reload v2.1 #13689

Closed

5 tasks

regit closed this Aug 5, 2025

Mime type reload v2 #13647

Mime type reload v2 #13647

Uh oh!

Conversation

regit commented Jul 24, 2025

Contribution style:

Our Contribution agreements:

Changes (if applicable):

Uh oh!

regit commented Jul 24, 2025

Uh oh!

codecov bot commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jufajardini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

regit commented Jul 25, 2025

Uh oh!

catenacyber commented Jul 28, 2025

Uh oh!

Uh oh!

Uh oh!

catenacyber Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

catenacyber Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

regit Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

jufajardini commented Jul 29, 2025

Uh oh!

jasonish Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

regit Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

jasonish commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

regit commented Jul 29, 2025

Uh oh!

regit commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasonish commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

regit commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasonish commented Jul 29, 2025

Uh oh!

regit commented Jul 29, 2025

Uh oh!

catenacyber commented Jul 30, 2025

Uh oh!

Uh oh!

regit commented Aug 4, 2025

Uh oh!

Uh oh!

codecov bot commented Jul 25, 2025 •

edited

Loading

jasonish commented Jul 29, 2025 •

edited

Loading

regit commented Jul 29, 2025 •

edited

Loading

jasonish commented Jul 29, 2025 •

edited

Loading

regit commented Jul 29, 2025 •

edited

Loading