Home

MMSA-Feature Extraction Tool

MMSA-Feature Extraction Tool extracts multimodal features for Multimodal Sentiment Analysis Datasets. It integrates several commonly used tools for visual, acoustic and text modality. The extracted features are compatible with the MMSA Framework and thus can be used directly. The tool can also extract features for single videos.

1. Installation

$ pip install MMSA-FET

2. Quick Start

from MSA_FET import FeatureExtractionTool

# initialize with config file
fet = FeatureExtractionTool("config.json")

# extract features for single video
feature = fet.run_single("input.mp4")
print(feature)

# extract for dataset & save features to file
feature = fet.run_dataset(dataset_dir="~/MOSI", out_file="output/feature.pkl")

For more details, please read Wiki.

3. Config File

The tool comes with a few example configs which can be used like below.

# Every tool has an example config
fet = FeatureExtractionTool(config="openface")
fet = FeatureExtractionTool(config="bert")
fet = FeatureExtractionTool(config="librosa")
...

For customized features, you'll have to provide a config file which is in the following format.

{
  "audio": {
    "tool": "librosa",
    "sample_rate": null,
    "args": {
      "mfcc": {
        "n_mfcc": 20,
        "htk": true
      },
      "rms": {},
      "zero_crossing_rate": {},
      "spectral_rolloff": {},
      "spectral_centroid": {}
    }
  },
  "video": {
    "tool": "openface",
    "fps": 25,
    "average_over": 3,
    "args": {
      "hogalign": false,
      "simalign": false,
      "nobadaligned": false,
      "landmark_2D": true,
      "landmark_3D": false,
      "pdmparams": false,
      "head_pose": true,
      "action_units": true,
      "gaze": true,
      "tracked": false
    }
  },
  "text": {
    "model": "bert",
    "device": "cpu",
    "pretrained": "models/bert_base_uncased",
    "args": {}
  }
}

4. Supported Tools & Features