Skip to content

feat: add file type source feature flags #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 29, 2025
Merged

feat: add file type source feature flags #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 150 additions & 1 deletion FILETYPES.md
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,16 @@ assert_eq!(file_type.extensions(), vec!["png"]);
assert_eq!(file_type.media_types(), vec!["image/png"]);
```

## Feature flags

| Name | Description | Default? |
|------------|-----------------------------------------------------------------------------------------------------------------|----------|
| `custom` | Enables custom file types | Yes |
| `httpd` | Enables [Apache HTTPD](https://github.com/apache/httpd/blob/trunk/docs/conf/mime.types) file types | No |
| `linguist` | Enables [Linguist](https://github.com/github-linguist/linguist/blob/main/lib/linguist/languages.yml) file types | No |
| `pronom` | Enables [PRONOM](https://www.nationalarchives.gov.uk/PRONOM) file types | Yes |
| `tokio` | Enables using tokio for async | No |

## Supported File Types

[List of supported file types](https://github.com/theseus-rs/file-type/blob/main/FILETYPES.md)
Expand Down
21 changes: 0 additions & 21 deletions export/httpd/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@

use anyhow::Result;
use file_type::format::{DocumentIdentifier, ExternalSignature, FileFormat, SignatureType};
use file_type::FileType;
use quick_xml::se::Serializer;
use reqwest::Client;
use serde::Serialize;
Expand Down Expand Up @@ -114,13 +113,6 @@ fn process_mime_types(mime_types: HashMap<String, Vec<String>>) -> Result<Vec<Fi
let mut file_formats = Vec::new();

for (mime_type, extensions) in mime_types {
let exists = extensions
.iter()
.any(|extension| mime_type_exists(&mime_type, extension));
if exists {
continue;
}

let external_signatures = extensions
.iter()
.enumerate()
Expand Down Expand Up @@ -164,19 +156,6 @@ fn process_mime_types(mime_types: HashMap<String, Vec<String>>) -> Result<Vec<Fi
Ok(file_formats)
}

/// Check if a mime type and extension already exist in the file type data.
fn mime_type_exists<S: AsRef<str>>(mime_type: S, extension: S) -> bool {
let mime_type = mime_type.as_ref();
let extension = extension.as_ref();
for file_type in FileType::from_media_type(mime_type) {
if file_type.extensions().contains(&extension) {
return true;
}
}

false
}

async fn store_file_format(file_format: &FileFormat, output_dir: &Path) -> Result<()> {
let file_name = format!("{}.xml", file_format.puid().replace('/', "-"));
info!("{file_name}: {}", file_format.name());
Expand Down
17 changes: 0 additions & 17 deletions file_type/CHANGELOG.md
View file
Open in desktop

This file was deleted.

6 changes: 5 additions & 1 deletion file_type/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,11 @@ tokio = { workspace = true, features = ["full"] }
walkdir = { workspace = true }

[features]
default = []
default = ["custom", "pronom"]
custom = []
httpd = []
linguist = []
pronom = []
tokio = [
"dep:tokio",
]
Expand Down
10 changes: 10 additions & 0 deletions file_type/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,16 @@ assert_eq!(file_type.extensions(), vec!["png"]);
assert_eq!(file_type.media_types(), vec!["image/png"]);
```

## Feature flags

| Name | Description | Default? |
|------------|-----------------------------------------------------------------------------------------------------------------|----------|
| `custom` | Enables custom file types | Yes |
| `httpd` | Enables [Apache HTTPD](https://github.com/apache/httpd/blob/trunk/docs/conf/mime.types) file types | No |
| `linguist` | Enables [Linguist](https://github.com/github-linguist/linguist/blob/main/lib/linguist/languages.yml) file types | No |
| `pronom` | Enables [PRONOM](https://www.nationalarchives.gov.uk/PRONOM) file types | Yes |
| `tokio` | Enables using tokio for async | No |

## Supported File Types

[List of supported file types](https://github.com/theseus-rs/file-type/blob/main/FILETYPES.md)
Expand Down
17 changes: 17 additions & 0 deletions file_type/data/httpd/httpd-10114909409051067364.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<FileFormat>
<FormatID>10114909409051067364</FormatID>
<FormatName>sun xml writer</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10114909409051067364</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>application/vnd.sun.xml.writer</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>sxw</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
17 changes: 17 additions & 0 deletions file_type/data/httpd/httpd-10214872156167243065.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<FileFormat>
<FormatID>10214872156167243065</FormatID>
<FormatName>rtf</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10214872156167243065</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>application/rtf</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>rtf</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
17 changes: 17 additions & 0 deletions file_type/data/httpd/httpd-10233703363409414994.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<FileFormat>
<FormatID>10233703363409414994</FormatID>
<FormatName>adobe photoshop</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10233703363409414994</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>image/vnd.adobe.photoshop</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>psd</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
17 changes: 17 additions & 0 deletions file_type/data/httpd/httpd-10420792781717195325.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<FileFormat>
<FormatID>10420792781717195325</FormatID>
<FormatName>wap wml</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10420792781717195325</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>text/vnd.wap.wml</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>wml</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
22 changes: 22 additions & 0 deletions file_type/data/httpd/httpd-10466185904492655952.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<FileFormat>
<FormatID>10466185904492655952</FormatID>
<FormatName>vrml</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10466185904492655952</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>model/vrml</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>wrl</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
<ExternalSignature>
<ExternalSignatureID>1</ExternalSignatureID>
<Signature>vrml</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
27 changes: 27 additions & 0 deletions file_type/data/httpd/httpd-10581694100561422952.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
<FileFormat>
<FormatID>10581694100561422952</FormatID>
<FormatName>ms powerpoint</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10581694100561422952</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>application/vnd.ms-powerpoint</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>ppt</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
<ExternalSignature>
<ExternalSignatureID>1</ExternalSignatureID>
<Signature>pps</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
<ExternalSignature>
<ExternalSignatureID>2</ExternalSignatureID>
<Signature>pot</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
17 changes: 17 additions & 0 deletions file_type/data/httpd/httpd-10691704915181776398.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<FileFormat>
<FormatID>10691704915181776398</FormatID>
<FormatName>gnumeric</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10691704915181776398</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>application/x-gnumeric</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>gnumeric</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
22 changes: 22 additions & 0 deletions file_type/data/httpd/httpd-10692450451458080727.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<FileFormat>
<FormatID>10692450451458080727</FormatID>
<FormatName>basic</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10692450451458080727</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>audio/basic</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>au</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
<ExternalSignature>
<ExternalSignatureID>1</ExternalSignatureID>
<Signature>snd</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
22 changes: 22 additions & 0 deletions file_type/data/httpd/httpd-10814198890374693472.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<FileFormat>
<FormatID>10814198890374693472</FormatID>
<FormatName>djvu</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10814198890374693472</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>image/vnd.djvu</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>djvu</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
<ExternalSignature>
<ExternalSignatureID>1</ExternalSignatureID>
<Signature>djv</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
17 changes: 17 additions & 0 deletions file_type/data/httpd/httpd-1090270817345578466.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<FileFormat>
<FormatID>1090270817345578466</FormatID>
<FormatName>ttf</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/1090270817345578466</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>font/ttf</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>ttf</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
17 changes: 17 additions & 0 deletions file_type/data/httpd/httpd-10965205291161609515.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<FileFormat>
<FormatID>10965205291161609515</FormatID>
<FormatName>ms htmlhelp</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10965205291161609515</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>application/vnd.ms-htmlhelp</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>chm</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
17 changes: 17 additions & 0 deletions file_type/data/httpd/httpd-10990209224482984948.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<FileFormat>
<FormatID>10990209224482984948</FormatID>
<FormatName>ogg</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/10990209224482984948</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>video/ogg</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>ogv</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
22 changes: 22 additions & 0 deletions file_type/data/httpd/httpd-1099749789536492620.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<FileFormat>
<FormatID>1099749789536492620</FormatID>
<FormatName>stardivision writer</FormatName>
<FileFormatIdentifier>
<Identifier>httpd/1099749789536492620</Identifier>
<IdentifierType>PUID</IdentifierType>
</FileFormatIdentifier>
<FileFormatIdentifier>
<Identifier>application/vnd.stardivision.writer</Identifier>
<IdentifierType>MIME</IdentifierType>
</FileFormatIdentifier>
<ExternalSignature>
<ExternalSignatureID>0</ExternalSignatureID>
<Signature>sdw</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
<ExternalSignature>
<ExternalSignatureID>1</ExternalSignatureID>
<Signature>vor</Signature>
<SignatureType>File extension</SignatureType>
</ExternalSignature>
</FileFormat>
Loading