Mime type reload v2.4 #13727

regit · 2025-08-17T18:58:11Z

Update of #13715 addressing comments from @jufajardini. As it seems it was not an in depth review. I'm reproducing here the details of changes from previous PR.

Contribution style:

I have read the contributing guide lines at
https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html

Our Contribution agreements:

I have signed the Open Information Security Foundation contribution agreement at
https://suricata.io/about/contribution-agreement/ (note: this is only required once)

Changes (if applicable):

I have updated the User Guide (in doc/userguide/) to reflect the changes made
I have updated the JSON schema (in etc/schema.json) to reflect all logging changes
(including schema descriptions)
I have created a ticket at
https://redmine.openinfosecfoundation.org/projects/suricata/issues

Link to ticket: https://redmine.openinfosecfoundation.org/issues/7816

Describe changes:

Address comments from @jufajardini
Don't embed GPL data by default (it needs to be explicitly set)
Load mime type files from disk
Embed mime type files in code
make install-conf install mime type files
make upgrade-data update mime type files

SV_BRANCH=OISF/suricata-verify#2606

Following commit will add a new value that will overflow the actual size. Ticket: 7816

File magic is known to have performance issue. When looking for an alternative, I've been pointed to the tree_magic_mini crate that output the mime type of a binary stream. This is different from magic but it has multiple advantages as it is a standard so it can be correlated with other tools. So instead of replacing magic, this patch adds a new mime type output to fileinfo events. This patch also adds the `file.mimetype` sticky buffer. Mime type has 2 advantages over file magic. First it is really faster and second, the result are easier to use as the MIME type are well defined. But it provides less information than magic for example with regards to the size of images. This patch is using version 3.2.0 of tree_magic_mini that provides capabilities to load mime files from a give directory. This patch is embedding the GPL database if an explicit flag is passed to configure (--enable-bundled-gpl-mimetype). In this case the mime type files comes from `tree_magic_mini` dependencies. The patch adds the mime definition in Suricata source code. This means that if the user uses this data, then we have no dependency in term of results on the operating system and on the evolution of the crate. Updating the mime type definition on the target system will be done by installing the files present in Suricata code. Installation of the mime type files is done via `make install-conf` to the mimetype/ directory below data directory. This directory is used by default even if the suricata.yaml is not updated with the new `mimetype-dir` config variable. The new Makefile target `upgrade-data` takes care of upgrading the file. Users will be able to use it when there is a new version available. The `mimetype-dir` YAML variable can be used to set up a different directory with data under control by the user. Ticket: 7816

Ticket: 7816

Add some builds with mimetype activated and suricata-verify so we can test feature. Test with and without embedded GPL data are done.

codecov · 2025-08-17T19:27:53Z

Codecov Report

❌ Patch coverage is 83.25581% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.72%. Comparing base (b93a277) to head (f322fa6).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #13727      +/-   ##
==========================================
+ Coverage   82.99%   83.72%   +0.72%     
==========================================
  Files        1001     1014      +13     
  Lines      272978   275266    +2288     
==========================================
+ Hits       226556   230461    +3905     
+ Misses      46422    44805    -1617

Flag	Coverage Δ
fuzzcorpus	`62.92% <63.51%> (+0.04%)`	⬆️
livemode	`19.00% <36.48%> (?)`
pcap	`44.74% <55.40%> (+0.05%)`	⬆️
suricata-verify	`65.09% <83.98%> (+0.01%)`	⬆️
unittests	`59.17% <56.75%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

catenacyber · 2025-08-26T20:13:16Z

.github/workflows/rust.yml

      - run: ./scripts/bundle.sh
      - run: ./autogen.sh
-      - run: ./configure --enable-warnings
+      - run: ./configure --enable-warnings --enable-mimetype


I wonder : could this be a plugin ?

To rephrase, what are we missing in Suricata so that it could be a plugin ?..

catenacyber · 2025-08-26T20:15:07Z

src/detect-engine-file.c

        }

+        if ((s->file_flags & FILE_SIG_NEED_MIMETYPE) && file_size == 0) {
+            SCLogDebug("sig needs file content, but we don't have any");


debug log message is bad copy/paste

catenacyber · 2025-08-26T20:16:55Z

src/detect-file-mimetype.c

+    int list_id;
+    const MpmCtx *mpm_ctx;
+    const DetectEngineTransforms *transforms;
+} PrefilterMpmFileMimetype;


Why not reuse PrefilterMpmListId ?

catenacyber · 2025-08-26T20:20:03Z

doc/userguide/rules/file-keywords.rst

+
+  file.mimetype; content:"application/vnd.microsoft.portable-executable";
+
+``file.mimetype`` supports multiple buffer matching, see :doc:`multi-buffer-matching`.


needs to be added as well in doc//userguide/rules/multi-buffer-matching.rst see #13752

catenacyber · 2025-08-27T19:36:33Z

etc/schema.json

                },
+                "mimetype": {
+                    "type": "string",
+                    "description": "The MIME type of the file (e.g., application/pdf, image/png, etc.)"


maybe say how it is found (not from headers, but from content) ?

catenacyber · 2025-08-27T19:38:10Z

doc/userguide/output/eve/eve-json-output.rst

+hashes.
+
+The type of the file can also be determined by doing an analysis of the beginning of the content
+of the file. This is done by using libmagic and/or libmimetype. Magic is slower than mimetype, but 


nit: space at end of line

catenacyber

CI : ✅
Code : need to check
Commits segmentation : ok (I would squash the doc commits)
Commit messages : good
Git ID set : looks fine for me
CLA : you already contributed
Doc update : good (some comments)
Redmine ticket : ok
Rustfmt : 🟠 rustfmt rust/src/filemimetype.rs has a diff
Tests : cool
Dependencies added: ⚠️ I do not know what needs to be checked for this new dependency

catenacyber

Now needs a rebase

Could you do the rustfmt ?

regit added 5 commits August 10, 2025 20:14

detect: convert file_flags to a uint16_t

408b85a

Following commit will add a new value that will overflow the actual size. Ticket: 7816

doc/userguide: add information about file.mimetype

38e2b20

Ticket: 7816

doc/eve: add fileinfo configuration information

7118cb9

ci: add builds with mimetype support

f322fa6

Add some builds with mimetype activated and suricata-verify so we can test feature. Test with and without embedded GPL data are done.

regit requested review from jasonish, jufajardini, victorjulien and a team as code owners August 17, 2025 18:58

catenacyber reviewed Aug 26, 2025

View reviewed changes

catenacyber reviewed Aug 27, 2025

View reviewed changes

catenacyber requested changes Sep 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Mime type reload v2.4 #13727

Mime type reload v2.4 #13727

Uh oh!

regit commented Aug 17, 2025

Uh oh!

codecov bot commented Aug 17, 2025

Uh oh!

catenacyber Aug 26, 2025

Uh oh!

catenacyber Aug 27, 2025

Uh oh!

catenacyber Aug 26, 2025

Uh oh!

catenacyber Aug 26, 2025

Uh oh!

catenacyber Aug 26, 2025

Uh oh!

catenacyber Aug 27, 2025

Uh oh!

catenacyber Aug 27, 2025

Uh oh!

catenacyber left a comment

Uh oh!

catenacyber left a comment

Uh oh!

Uh oh!


		file.mimetype; content:"application/vnd.microsoft.portable-executable";

		``file.mimetype`` supports multiple buffer matching, see :doc:`multi-buffer-matching`.

Mime type reload v2.4 #13727

Are you sure you want to change the base?

Mime type reload v2.4 #13727

Uh oh!

Conversation

regit commented Aug 17, 2025

Contribution style:

Our Contribution agreements:

Changes (if applicable):

Uh oh!

codecov bot commented Aug 17, 2025

Codecov Report

Uh oh!

catenacyber Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

catenacyber Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

catenacyber Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

catenacyber Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

catenacyber Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

catenacyber Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

catenacyber Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

catenacyber left a comment

Choose a reason for hiding this comment

Uh oh!

catenacyber left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!