Fahrsimulator_MSY2526_AI/project_report.md

Project Report: Multimodal Driver State Analysis

1) Project Scope

This repository implements an end-to-end workflow for multimodal driver-state analysis in a simulator setup.
The system combines:

• Facial Action Units (AUs)
• Eye-tracking features (fixations, saccades, blinks, pupil behavior)

Apart from this, several machine learning model architectures are presented and evaluated.

Content:

• Dataset generation
• Exploratory data analysis
• Model training experiments
• Real-time inference with SQlite, systemd and MQTT
• Repository file inventory
• Additional nformation

2) Dataset generation

2.1 Data Access, Filtering, and Data Conversion

Main scripts:

• dataset_creation/create_parquet_files_from_owncloud.py
• dataset_creation/parquet_file_creation.py

Purpose:

• Download and/or access dataset files (either download first via EDA/owncloud_file_access.ipynb or all in one with dataset_creation/create_parquet_files_from_owncloud.py
• Keep relevant columns (FACE_AUs and eye-tracking raw values)
• Filter invalid samples (e.g., invalid level segments): Make sure not to drop rows where NaN is necessary for later feature creation, therefore use subset argument in dropNa()!
• Export subject-level parquet files
• Before running the scripts: be aware that the whole dataset contains 30 files with around 900 Mbytes each, provide enough storage and expect this to take a while.

2.2 Feature Engineering (Offline)

Main script:

• dataset_creation/combined_feature_creation.py

Behavior:

• Builds fixed-size sliding windows over subject time series (window size and step size can be adjusted)
• Uses prepared parquet files from 2.1
• Aggregates AU statistics per window (e.g., FACE_AUxx_mean)
• Computes eye-feature aggregates (fix/sacc/blink/pupil metrics)
• Produces training-ready feature tables = dataset
• Parameter MIN_DUR_BLINKS can be adjusted, although this value needs to make sense in combination with your sampling frequency
• With low videostream rates, consider to reevaluate the meaningfulness of some eye-tracking features, especially the fixations
• running the script requires a manual installation of pygaze Analyser library from github

2.3 Online Camera + Eye + AU Feature Extraction

Main scripts:

• dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py
• dataset_creation/camera_handling/eyeFeature_new.py
• dataset_creation/camera_handling/db_helper.py

Runtime behavior:

• Captures webcam stream with OpenCV
• Extracts gaze/iris-based signals via MediaPipe
• Records overlapping windows (VIDEO_DURATION=50s, START_INTERVAL=5s, FPS=25)
• Runs AU extraction (py-feat) from recorded video segments
• Computes eye-feature summary from generated gaze parquet
• Writes merged rows to SQLite table feature_table

Operational note:

• DB_PATH and other paths are currently code-configured and must be adapted per deployment.

3) EDA

The directory EDA provides several files to get insights into both the raw data from AdaBase and your own dataset.

• EDA.ipynb - main EDA notebook: recreates the plot from AdaBase documentation, lists all experiments and in general serves as a playground for you to get to know the files.
• distribution_plots.ipynb - This notebook aimes to visualize the data distributions for each experiment - the goal is the find out, whether the split of experiments into high and low cognitive load is clearer if some experiments are dropped.
• histogramms.ipynb - Histogram analysis of low load vs high load per feature. Additionaly, scatter plots per feature are available.
• researchOnSubjectPerformance.ipynb - This noteboooks aims to see how the performance values range for the 30 subjects. The code creates and saves a table in csv-format, which will later be used as the foundation of the performance based split in model_training/tools/performance_based_split
• owncloud_file_access.ipynb - Get access to the files via owncloud and safe them as .h5 files, in correspondence to the parquet file creation script
• login.yaml - used to store URL and password to access files from owncloud, used in previous notebook
• calculate_replacement_values.ipynb - fallback / median computation notebook for deployment, creation of yaml syntax embedding

General information:

• Due to their size, its absolutely recommended to download and save the dataset files once in the beginning
• For better data understanding, read the AdaBase publication

4) Model Training

Included model families:

• CNN variants (different fusion strategies)
• XGBoost
• Isolation Forest*
• OCSVM*
• DeepSVDD*

* These trainings are unsupervised, which means only low cognitive load samples are used for training. Validation then also considers high low samples.

Supporting utilities in model_training/tools:

• scaler.py: Functions to fit, transform, save and load either MinMaxScaler or StandardScaler, subject-wise and globally - for new subjects, a fallback scaler (using mean of all subjects scaling parameters) is used
• performance_split.py: Provides a function to split a group of subjects based on their performance in the AdaBase experiments, based on the results created in researchOnSubjectPerformance.ipynb. To split into three groups for train, validation & test, call the function twice
• mad_outlier_removal.py: Functions to fit and transform data with MAD outlier removal
• evaluation_tools.py: Especially used for Isolation Forest, Functions for ROC curve as well as confusion matrix

4.1 CNNs

4.2 XGBoost

4.3 Isolation Forest

To start with unsupervised learning techniques, IsolationForest.ipynbwas created to research how well a simple ensemble classificator performs on the created dataset. The notebook comes with one class grid search for hyperparameter tuning as well as a ROC curve that allows manual fine tuning. Overall, our experiments have shown, that this approach is not sufficient, with the following results:

Metric / Model	Isolation Forest
Best Balanced Accuracy	0.57
Best AUC	0.61

In detail, the classificator tends to classify the majority of samples as low load and is therefore not sufficient to be used for later deployment.

4.4 One Class SVM with Autoencoder

The training of an On Class SVM on the data from the dataset resulted in every sample was predicted as an anomaly. In the next step, an autoencoder is pretrained to learn representation of the data. Afterwards, the encoder is used for preprocessing, which leads to OCSVM training on encoder output. The training includes hyperparameter tuning through gridsearch cv. Encoder output is visualized with print statements and plots that show the encoded data for both low and high load samples. We see that the encoder struggles to represent the unseen high load samples differently. As a consequence, the One Class SVM also does not achieve sufficient performance.

Metric / Model	One Class SVM
Best Balanced Accuracy	0.62

When the notebook is run completely, both the trained encoder and svm are saved for later use given the save paths are set correctly.

4.5 Deep SVDD

Similar to the OCSVM training, an autoencoder is used to preprocess the data before the actual Deep SVDD training. Nevertheless, the usage is partialy different. The Dee SVDD uses a pretrained encoder to fine tune it by apllying a different loss function (which results from the theoretical concept behind Deep SVDD). This means that the encoder weights are still modified in the actual Deep SVDD training. Also, this approach includes hybrid fusion of modalities. Instead of putting all features into the same input layer, the neural network is divided into two branches, that process action units and eye-tracking features separately. Then, after two Dense layers each, the branches are fusioned by concatenation. From there, another two Dense layers process the data. The decoder is not exactly similar, as the split of the modalities happens are the very end. To compute the total loss, loss from both modalities is combined by sum. Users are able to change loss weights. Training includes 2x2 phases, both autoencoder and later Deep SVDD are first trained with larger learning rate, then fine tuned with a smaller learning rate.

Metric / Model	Deep SVDD
Best Balanced Accuracy	0.60
Best AUC	0.57

4.6 General information on unsupervised approaches

As described above, the approachs didn't meet the requirements in terms of prediction performance. For all models, both MinMax-Scaling as well as Standard-Scaling was done. Also, both subjectwise and globally. Unfortunately, the differences were not that large, which may explain why preprocessing wasn't mentioned above.

Future research should always keep in mind while subject-wise scaling might be better for training, it makes deployment on new subjects more difficult. Our solution, as implemented in model_training/tools/scaler.py calculates a fallback scaler (using mean of all subjects scaling parameters).

5) Real-Time Prediction and Messaging

Main script:

• predict_pipeline/predict_sample.py

Pipeline:

• Loads runtime config (predict_pipeline/config.yaml)
• Pulls latest row from SQLite
• Replaces missing values using fallback map from config file - if more than 50% of values need to be replaced, the sample is dropped and "valid=False"
• Optionally applies scaler (.pkl/.joblib) - set via config file
• Loads model (.keras, .pkl, .joblib) and predicts
• Publishes JSON payload to MQTT topic

Expected payload form:

{
  "valid": true,    # false only if too many signals are invalid
  "_id": 123,       # this is the sample ID from the database
  "prediction": 0   # 0 for low load, 1 for high load
}

5.1 Scheduled Prediction (Linux)

Files:

• predict_pipeline/predict.service
• predict_pipeline/predict.timer

Role:

• Run inference repeatedly without manual execution
• Timer/service configuration can be customized

More information on how to use and interact with the system service and timer can be found in predict_service_timer_documentation.md

5.2 Runtime Configuration

Primary config file:

• predict_pipeline/config.yaml

Sections:

• database: SQLite location + table + sort key
• model: model path
• scaler: scaler usage + path
• mqtt: broker and publish format
• sample.columns: expected feature order
• fallback: default values for NaN replacement

Important:

• The repository currently uses environment-specific absolute paths in some scripts/configs to ensure functionality on Ohm-UX driving simulator.

5.3 Data and Feature Expectations

Prediction expects SQLite rows containing:

• _Id
• start_time - this is not yet used for either predictions or messages
• All configured model features (AUs + eye metrics)

Common feature groups (similar to own dataset):

• FACE_AUxx_mean columns
• Fixation counters and duration statistics
• Saccade count/amplitude/duration statistics
• Blink count/duration statistics
• Pupil mean and IPA

5.4 Create database from scratch

To (re-)create the custom database for deployment, use fill_db.ipynb. Enter the path to your dataset, drop unnecessary columns and insert a subset of data with tool functions from tools/db_helpers

6) Installation and Dependencies

Due to unsolvable dependency conflicts, several environemnts need to be used in the same time.

6.1 Environemnt for camera handling

TO DO

6.2 Environment for predictions

If you want to use the existing deployment on Ohm-UX driving simulator's jetson board, activate conda environment p310_FS_TF, a python 3.10 environment including tensorflow and all other packages required to run predict_sample.py Otherwise, as described in readme.md: Setup, you can use prediction_env.yaml, to create a new environment that fulfills the requirements.

7) Repository File Inventory

Root

• .gitignore - Git ignore rules
• readme.md - minimal quickstart documentation
• project_report.md - full technical documentation (this file)
• requirements.txt - Python dependencies

Dataset Creation

• dataset_creation/parquet_file_creation.py - local files to parquet conversion
• dataset_creation/create_parquet_files_from_owncloud.py - ownCloud download + parquet conversion
• dataset_creation/combined_feature_creation.py - sliding-window multimodal feature generation
• dataset_creation/maxDist.py - helper/statistical utility script for eye-tracking feature creation

AU Creation

• dataset_creation/AU_creation/AU_creation_service.py - AU extraction service workflow
• dataset_creation/AU_creation/pyfeat_docu.ipynb - py-feat exploratory notes

Camera Handling

• dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py - current camera + AU + eye online pipeline
• dataset_creation/camera_handling/eyeFeature_new.py - eye-feature extraction from gaze parquet
• dataset_creation/camera_handling/db_helper.py - SQLite helper functions (camera pipeline)
• dataset_creation/camera_handling/camera_stream_AU_and_ET.py - older pipeline variant
• dataset_creation/camera_handling/camera_stream.py - baseline camera streaming script
• dataset_creation/camera_handling/db_test.py - DB test utility

EDA

• EDA/EDA.ipynb - main EDA notebook
• EDA/distribution_plots.ipynb - distribution visualization
• EDA/histogramms.ipynb - histogram analysis
• EDA/researchOnSubjectPerformance.ipynb - subject-level analysis
• EDA/owncloud_file_access.ipynb - ownCloud exploration/access notebook
• EDA/calculate_replacement_values.ipynb - fallback/median computation notebook
• EDA/login.yaml - local auth/config artifact for EDA workflows

Model Training

CNN

• model_training/CNN/CNN_simple.ipynb
• model_training/CNN/CNN_crossVal.ipynb
• model_training/CNN/CNN_crossVal_EarlyFusion.ipynb
• model_training/CNN/CNN_crossVal_EarlyFusion_Filter.ipynb
• model_training/CNN/CNN_crossVal_EarlyFusion_Test_Eval.ipynb
• model_training/CNN/CNN_crossVal_faceAUs.ipynb
• model_training/CNN/CNN_crossVal_faceAUs_eyeFeatures.ipynb
• model_training/CNN/CNN_crossVal_HybridFusion.ipynb
• model_training/CNN/CNN_crossVal_HybridFusion_Test_Eval.ipynb
• model_training/CNN/deployment_pipeline.ipynb

XGBoost

• model_training/xgboost/xgboost.ipynb
• model_training/xgboost/xgboost_groupfold.ipynb
• model_training/xgboost/xgboost_new_dataset.ipynb
• model_training/xgboost/xgboost_regulated.ipynb
• model_training/xgboost/xgboost_with_AE.ipynb
• model_training/xgboost/xgboost_with_MAD.ipynb

Isolation Forest

• model_training/IsolationForest/iforest_training.ipynb

OCSVM

• model_training/OCSVM/ocsvm_with_AE.ipynb

DeepSVDD

• model_training/DeepSVDD/deepSVDD.ipynb

MAD Outlier Removal

• model_training/MAD_outlier_removal/mad_outlier_removal.ipynb
• model_training/MAD_outlier_removal/mad_outlier_removal_median.ipynb

Shared Training Tools

• model_training/tools/scaler.py
• model_training/tools/performance_split.py
• model_training/tools/mad_outlier_removal.py
• model_training/tools/evaluation_tools.py

Prediction Pipeline

• predict_pipeline/predict_sample.py - runtime prediction + MQTT publish
• predict_pipeline/config.yaml - runtime database/model/scaler/mqtt config
• predict_pipeline/fill_db.ipynb - helper notebook for DB setup/testing
• predict_pipeline/predict.service - systemd service unit
• predict_pipeline/predict.timer - systemd timer unit
• predict_pipeline/predict_service_timer_documentation.md - Linux service/timer guide

Generic Tools

• tools/db_helpers.py - common SQLite utilities used to get newest sample for prediction

8) Additional Information

• Several paths are hardcoded on purpose to ensure compability with the jetsonboard at the OHM-UX driving simulator.
• Camera and AU processing are resource-intensive; version pinning and hardware validation are recommended.