Fahrsimulator_MSY2526_AI/project_report.md

Project Report: Multimodal Driver State Analysis

1) Project Scope

This repository implements an end-to-end workflow for multimodal driver-state analysis in a simulator setup.
The system combines:

• Facial Action Units (AUs)
• Eye-tracking features (fixations, saccades, blinks, pupil behavior)

Apart from this, several machine learning model architectures are presented and evaluated.

Content:

• Dataset generation
• Exploratory data analysis
• Model training experiments
• Real-time inference with SQlite, systemd and MQTT
• Repository file inventory
• Additional nformation

2) Dataset generation

2.1 Data Access, Filtering, and Data Conversion

Main scripts:

• dataset_creation/create_parquet_files_from_owncloud.py
• dataset_creation/parquet_file_creation.py

Purpose:

• Download and/or access dataset files (either download first via EDA/owncloud_file_access.ipynb or all in one with dataset_creation/create_parquet_files_from_owncloud.py
• Keep relevant columns (FACE_AUs and eye-tracking raw values)
• Filter invalid samples (e.g., invalid level segments): Make sure not to drop rows where NaN is necessary for later feature creation, therefore use subset argument in dropNa()!
• Export subject-level parquet files
• Before running the scripts: be aware that the whole dataset contains 30 files with around 900 Mbytes each, provide enough storage and expect this to take a while.

2.2 Feature Engineering (Offline)

Main script:

• dataset_creation/combined_feature_creation.py

Behavior:

• Builds fixed-size sliding windows over subject time series (window size and step size can be adjusted)
• Uses prepared parquet files from 2.1
• Aggregates AU statistics per window (e.g., FACE_AUxx_mean)
• Computes eye-feature aggregates (fix/sacc/blink/pupil metrics)
• Produces training-ready feature tables = dataset
• Parameter MIN_DUR_BLINKS can be adjusted, although this value needs to make sense in combination with your sampling frequency
• With low videostream rates, consider to reevaluate the meaningfulness of some eye-tracking features, especially the fixations
• running the script requires a manual installation of pygaze Analyser library from github

2.3 Online Camera + Eye + AU Feature Extraction

Main scripts:

• dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py
• dataset_creation/camera_handling/eyeFeature_new.py
• dataset_creation/camera_handling/db_helper.py

Runtime behavior:

• Captures webcam stream with OpenCV
• Extracts gaze/iris-based signals via MediaPipe
• Records overlapping windows (VIDEO_DURATION=50s, START_INTERVAL=5s, FPS=25)
• Runs AU extraction (py-feat) from recorded video segments
• Computes eye-feature summary from generated gaze parquet
• Writes merged rows to SQLite table feature_table

Operational note:

• DB_PATH and other paths are currently code-configured and must be adapted per deployment.

3) EDA

TO DO

• EDA/EDA.ipynb - main EDA notebook
• EDA/distribution_plots.ipynb - distribution visualization
• EDA/histogramms.ipynb - histogram analysis
• EDA/researchOnSubjectPerformance.ipynb - subject-level analysis
• EDA/owncloud_file_access.ipynb - ownCloud exploration/access notebook
• EDA/calculate_replacement_values.ipynb - fallback/median computation notebook
• EDA/login.yaml - local auth/config artifact for EDA workflows

4) Model Training

Location:

• model_training/ (primarily notebook-driven)

Included model families:

• CNN variants (different fusion strategies)
• XGBoost
• Isolation Forest
• OCSVM
• DeepSVDD

Supporting utilities:

• model_training/tools/scaler.py
• model_training/tools/performance_split.py
• model_training/tools/mad_outlier_removal.py
• model_training/tools/evaluation_tools.py

5) Real-Time Prediction and Messaging

Main script:

• predict_pipeline/predict_sample.py

Pipeline:

• Loads runtime config (predict_pipeline/config.yaml)
• Pulls latest row from SQLite (database.path/table/key)
• Replaces missing values using fallback map
• Optionally applies scaler (.pkl/.joblib)
• Loads model (.keras, .pkl, .joblib) and predicts
• Publishes JSON payload to MQTT topic

Expected payload form:

{
  "valid": true,
  "_id": 123,
  "prediction": 0
}

5.1 Scheduled Prediction (Linux)

Files:

• predict_pipeline/predict.service
• predict_pipeline/predict.timer
• predict_pipeline/predict_service_timer_documentation.md

Role:

• Run inference repeatedly without manual execution
• Timer/service configuration can be customized per target machine

5.2 Runtime Configuration

Primary config file:

• predict_pipeline/config.yaml

Sections:

• database: SQLite location + table + sort key
• model: model path
• scaler: scaler usage + path
• mqtt: broker and publish format
• sample.columns: expected feature order
• fallback: default values for NaN replacement

Important:

• The repository currently uses environment-specific absolute paths in some scripts/configs.
• Paths should be normalized before deployment to a new machine.

5.3) Data and Feature Expectations

Prediction expects SQLite rows containing:

• _Id
• start_time
• All configured model features (AUs + eye metrics)

Common feature groups:

• FACE_AUxx_mean columns
• Fixation counters and duration statistics
• Saccade count/amplitude/duration statistics
• Blink count/duration statistics
• Pupil mean and IPA

6) Installation and Dependencies

Due to unsolvable dependency conflicts, several environemnts need to be used in the same time.

6.1 Environemnt for camera handling

TO DO

6.2 Environment for predictions

Install base requirements:

pip install -r requirements.txt

Typical key packages in this project:

• numpy, pandas, scikit-learn, scipy, pyarrow, pyyaml, joblib
• opencv-python, mediapipe, torch, py-feat, pygazeanalyser
• paho-mqtt
• optional data access stack (pyocclient, h5py, tables)

7) Repository File Inventory

Root

• .gitignore - Git ignore rules
• readme.md - minimal quickstart documentation
• project_report.md - full technical documentation (this file)
• requirements.txt - Python dependencies

Dataset Creation

• dataset_creation/parquet_file_creation.py - local source to parquet conversion
• dataset_creation/create_parquet_files_from_owncloud.py - ownCloud download + parquet conversion
• dataset_creation/combined_feature_creation.py - sliding-window multimodal feature generation
• dataset_creation/maxDist.py - helper/statistical utility script

AU Creation

• dataset_creation/AU_creation/AU_creation_service.py - AU extraction service workflow
• dataset_creation/AU_creation/pyfeat_docu.ipynb - py-feat exploratory notes

Camera Handling

• dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py - current camera + AU + eye online pipeline
• dataset_creation/camera_handling/eyeFeature_new.py - eye-feature extraction from gaze parquet
• dataset_creation/camera_handling/db_helper.py - SQLite helper functions (camera pipeline)
• dataset_creation/camera_handling/camera_stream_AU_and_ET.py - older pipeline variant
• dataset_creation/camera_handling/camera_stream.py - baseline camera streaming script
• dataset_creation/camera_handling/db_test.py - DB test utility

EDA

• EDA/EDA.ipynb - main EDA notebook
• EDA/distribution_plots.ipynb - distribution visualization
• EDA/histogramms.ipynb - histogram analysis
• EDA/researchOnSubjectPerformance.ipynb - subject-level analysis
• EDA/owncloud_file_access.ipynb - ownCloud exploration/access notebook
• EDA/calculate_replacement_values.ipynb - fallback/median computation notebook
• EDA/login.yaml - local auth/config artifact for EDA workflows

Model Training

CNN

• model_training/CNN/CNN_simple.ipynb
• model_training/CNN/CNN_crossVal.ipynb
• model_training/CNN/CNN_crossVal_EarlyFusion.ipynb
• model_training/CNN/CNN_crossVal_EarlyFusion_Filter.ipynb
• model_training/CNN/CNN_crossVal_EarlyFusion_Test_Eval.ipynb
• model_training/CNN/CNN_crossVal_faceAUs.ipynb
• model_training/CNN/CNN_crossVal_faceAUs_eyeFeatures.ipynb
• model_training/CNN/CNN_crossVal_HybridFusion.ipynb
• model_training/CNN/CNN_crossVal_HybridFusion_Test_Eval.ipynb
• model_training/CNN/deployment_pipeline.ipynb

XGBoost

• model_training/xgboost/xgboost.ipynb
• model_training/xgboost/xgboost_groupfold.ipynb
• model_training/xgboost/xgboost_new_dataset.ipynb
• model_training/xgboost/xgboost_regulated.ipynb
• model_training/xgboost/xgboost_with_AE.ipynb
• model_training/xgboost/xgboost_with_MAD.ipynb

Isolation Forest

• model_training/IsolationForest/iforest_training.ipynb

OCSVM

• model_training/OCSVM/ocsvm_with_AE.ipynb

DeepSVDD

• model_training/DeepSVDD/deepSVDD.ipynb

MAD Outlier Removal

• model_training/MAD_outlier_removal/mad_outlier_removal.ipynb
• model_training/MAD_outlier_removal/mad_outlier_removal_median.ipynb

Shared Training Tools

• model_training/tools/scaler.py
• model_training/tools/performance_split.py
• model_training/tools/mad_outlier_removal.py
• model_training/tools/evaluation_tools.py

Prediction Pipeline

• predict_pipeline/predict_sample.py - runtime prediction + MQTT publish
• predict_pipeline/config.yaml - runtime database/model/scaler/mqtt config
• predict_pipeline/fill_db.ipynb - helper notebook for DB setup/testing
• predict_pipeline/predict.service - systemd service unit
• predict_pipeline/predict.timer - systemd timer unit
• predict_pipeline/predict_service_timer_documentation.md - Linux service/timer guide

Generic Tools

• tools/db_helpers.py - common SQLite utilities used to get newest sample for prediction

8) Additional Information

• Several paths are hardcoded on purpose to ensure compability with the jetsonboard at the OHM-UX driving simulator.
• Camera and AU processing are resource-intensive; version pinning and hardware validation are recommended.