Fahrsimulator_MSY2526_AI/project_report.md

# Project Report: Multimodal Driver State Analysis

## 1) Project Scope

This repository implements an end-to-end workflow for multimodal driver-state analysis in a simulator setup.  
The system combines:
- Facial Action Units (AUs)
- Eye-tracking features (fixations, saccades, blinks, pupil behavior)

Apart from this, several machine learning model architectures are presented and evaluated.

Content:
- Dataset generation
- Exploratory data analysis
- Model training experiments
- Real-time inference with SQlite, systemd and MQTT
- Repository file inventory
- Additional nformation 


## 2) Dataset generation

### 2.1 Data Access, Filtering, and Data Conversion

Main scripts:
- `dataset_creation/create_parquet_files_from_owncloud.py`
- `dataset_creation/parquet_file_creation.py`

Purpose:
- Download and/or access dataset files (either download first via ```EDA/owncloud_file_access.ipynb``` or all in one with ```dataset_creation/create_parquet_files_from_owncloud.py```
- Keep relevant columns (FACE_AUs and eye-tracking raw values)
- Filter invalid samples (e.g., invalid level segments): Make sure not to drop rows where NaN is necessary for later feature creation, therefore use subset argument in dropNa()! 
- Export subject-level parquet files
- Before running the scripts: be aware that the whole dataset contains 30 files with around 900 Mbytes each, provide enough storage and expect this to take a while.


### 2.2 Feature Engineering (Offline)

Main script:
- `dataset_creation/combined_feature_creation.py`

Behavior:
- Builds fixed-size sliding windows over subject time series (window size and step size can be adjusted)
- Uses prepared parquet files from 2.1
- Aggregates AU statistics per window (e.g., `FACE_AUxx_mean`)
- Computes eye-feature aggregates (fix/sacc/blink/pupil metrics)
- Produces training-ready feature tables = dataset
- Parameter ```MIN_DUR_BLINKS``` can be adjusted, although this value needs to make sense in combination with your sampling frequency
- With low videostream rates, consider to reevaluate the meaningfulness of some eye-tracking features, especially the fixations
- running the script requires a manual installation of pygaze Analyser library from [github](https://github.com/esdalmaijer/PyGazeAnalyser.git)

### 2.3 Online Camera + Eye + AU Feature Extraction

Main scripts:
- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py`
- `dataset_creation/camera_handling/eyeFeature_new.py`
- `dataset_creation/camera_handling/db_helper.py`

Runtime behavior:
- Captures webcam stream with OpenCV
- Extracts gaze/iris-based signals via MediaPipe
- Records overlapping windows (`VIDEO_DURATION=50s`, `START_INTERVAL=5s`, `FPS=25`)
- Runs AU extraction (`py-feat`) from recorded video segments
- Computes eye-feature summary from generated gaze parquet
- Writes merged rows to SQLite table `feature_table`

Operational note:
- `DB_PATH` and other paths are currently code-configured and must be adapted per deployment.

## 3) EDA
The directory EDA provides several files to get insights into both the raw data from AdaBase and your own dataset.

- `EDA.ipynb` - main EDA notebook: recreates the plot from AdaBase documentation, lists all experiments and in general serves as a playground for you to get to know the files. 
- `distribution_plots.ipynb` - This notebook aimes to visualize the data distributions for each experiment - the goal is the find out, whether the split of experiments into high and low cognitive load is clearer if some experiments are dropped.
- `histogramms.ipynb` - Histogram analysis of low load vs high load per feature. Additionaly, scatter plots per feature are available.
- `researchOnSubjectPerformance.ipynb` - This noteboooks aims to see how the performance values range for the 30 subjects. The code creates and saves a table in csv-format, which will later be used as the foundation of the performance based split in ```model_training/tools/performance_based_split```
- `owncloud_file_access.ipynb` - Get access to the files via owncloud and safe them as .h5 files, in correspondence to the parquet file creation script 
- `login.yaml` - used to store URL and password to access files from owncloud, used in previous notebook
- `calculate_replacement_values.ipynb` - fallback / median computation notebook for deployment, creation of yaml syntax embedding

General information: 
- Due to their size, its absolutely recommended to download and save the dataset files once in the beginning
- For better data understanding, read the [AdaBase publication](https://www.mdpi.com/1424-8220/23/1/340)


## 4) Model Training

Included model families:
- CNN variants (different fusion strategies)
- XGBoost
- Isolation Forest*
- OCSVM*
- DeepSVDD*

\* These trainings are unsupervised, which means only low cognitive load samples are used for training. Validation then also considers high low samples.  


Supporting utilities in ```model_training/tools```:
- `scaler.py`: Functions to fit, transform, save and load either MinMaxScaler or StandardScaler, subject-wise and globally - for new subjects, a fallback scaler (using mean of all subjects scaling parameters) is used
- `performance_split.py`: Provides a function to split a group of subjects based on their performance in the AdaBase experiments, based on the results created in `researchOnSubjectPerformance.ipynb`. To split into three groups for train, validation & test, call the function twice
- `mad_outlier_removal.py`: Functions to fit and transform data with MAD outlier removal
- `evaluation_tools.py`: Especially used for Isolation Forest, Functions for ROC curve as well as confusion matrix

### 4.1 CNNs
### 4.2 XGBoost
### 4.3 Isolation Forest
To start with unsupervised learning techniques, `IsolationForest.ipynb`was created to research how well a simple ensemble classificator performs on the created dataset. 
The notebook comes with one class grid search for hyperparameter tuning as well as a ROC curve that allows manual fine tuning. 
Overall, our experiments have shown, that this approach is not sufficient, with the following results: 
| Metric / Model | Isolation Forest | 
|----------------|---------|
| Best Balanced Accuracy |0.57|
| Best AUC | 0.61|

In detail, the classificator tends to classify the majority of samples as low load and is therefore not sufficient to be used for later deployment.


### 4.4 One Class SVM with Autoencoder
The training of an On Class SVM on the data from the dataset resulted in every sample was predicted as an anomaly. 
In the next step, an autoencoder is pretrained to learn representation of the data. Afterwards, the encoder is used for preprocessing, which leads to OCSVM training on encoder output. 
The training includes hyperparameter tuning through gridsearch cv. 
Encoder output is visualized with print statements and plots that show the encoded data for both low and high load samples. 
We see that the encoder struggles to represent the unseen high load samples differently. As a consequence, the One Class SVM also does not achieve sufficient performance. 
| Metric / Model | One Class SVM | 
|----------------|---------|
| Best Balanced Accuracy |0.62|

When the notebook is run completely, both the trained encoder and svm are saved for later use given the save paths are set correctly. 
### 4.5 Deep SVDD
Similar to the OCSVM training, an autoencoder is used to preprocess the data before the actual Deep SVDD training. Nevertheless, the usage is partialy different. The Dee SVDD uses a pretrained encoder to fine tune it by apllying a different loss function (which results from the theoretical concept behind Deep SVDD). This means that the encoder weights are still modified in the actual Deep SVDD training. 
Also, this approach includes **hybrid fusion of modalities**. Instead of putting all features into the same input layer, the neural network is divided into two branches, that process action units and eye-tracking features separately.
Then, after two Dense layers each, the branches are fusioned by concatenation. From there, another two Dense layers process the data. 
The decoder is not exactly similar, as the split of the modalities happens are the very end. 
To compute the total loss, loss from both modalities is combined by sum. Users are able to change loss weights. Training includes 2x2 phases, both autoencoder and later Deep SVDD are first trained with larger learning rate, then fine tuned with a smaller learning rate. 

| Metric / Model | Deep SVDD | 
|----------------|---------|
| Best Balanced Accuracy |0.60|
| Best AUC | 0.57|

### 4.6 General information on unsupervised approaches
As described above, the approachs didn't meet the requirements in terms of prediction performance. For all models, both MinMax-Scaling as well as Standard-Scaling was done. Also, both subjectwise and globally. Unfortunately, the differences were not that large, which may explain why preprocessing wasn't mentioned above. 

Future research should always keep in mind while subject-wise scaling might be better for training, it makes deployment on new subjects more difficult. Our solution, as implemented in `model_training/tools/scaler.py` calculates a fallback scaler (using mean of all subjects scaling parameters). 

## 5) Real-Time Prediction and Messaging

Main script:
- `predict_pipeline/predict_sample.py`

Pipeline:
- Loads runtime config (`predict_pipeline/config.yaml`)
- Pulls latest row from SQLite
- Replaces missing values using `fallback` map from config file - if more than 50% of values need to be replaced, the sample is dropped and "valid=False"
- Optionally applies scaler (`.pkl`/`.joblib`) - set via config file
- Loads model (`.keras`, `.pkl`, `.joblib`) and predicts
- Publishes JSON payload to MQTT topic

Expected payload form:
```json
{
  "valid": true,    # false only if too many signals are invalid
  "_id": 123,       # this is the sample ID from the database
  "prediction": 0   # 0 for low load, 1 for high load
}
```

### 5.1 Scheduled Prediction (Linux)

Files:
- `predict_pipeline/predict.service`
- `predict_pipeline/predict.timer`

Role:
- Run inference repeatedly without manual execution
- Timer/service configuration can be customized

More information on how to use and interact with the system service and timer can be found in `predict_service_timer_documentation.md`

## 5.2 Runtime Configuration

Primary config file:
- `predict_pipeline/config.yaml`

Sections:
- `database`: SQLite location + table + sort key
- `model`: model path
- `scaler`: scaler usage + path
- `mqtt`: broker and publish format
- `sample.columns`: expected feature order
- `fallback`: default values for NaN replacement

Important:
- The repository currently uses environment-specific absolute paths in some scripts/configs to ensure functionality on Ohm-UX driving simulator.


## 5.3 Data and Feature Expectations

Prediction expects SQLite rows containing:
- `_Id`
- `start_time` - this is not yet used for either predictions or messages
- All configured model features (AUs + eye metrics)

Common feature groups (similar to own dataset):
- `FACE_AUxx_mean` columns
- Fixation counters and duration statistics
- Saccade count/amplitude/duration statistics
- Blink count/duration statistics
- Pupil mean and IPA

## 5.4 Create database from scratch
To (re-)create the custom database for deployment, use `fill_db.ipynb`. Enter the path to your dataset, drop unnecessary columns and insert a subset of data with tool functions from `tools/db_helpers`

## 6) Installation and Dependencies
Due to unsolvable dependency conflicts, several environemnts need to be used in the same time. 
### 6.1 Environemnt for camera handling
TO DO


### 6.2 Environment for predictions
If you want to use the existing deployment on Ohm-UX driving simulator's jetson board, activate conda environment `p310_FS_TF`, a python 3.10 environment including tensorflow and all other packages required to run `predict_sample.py`
Otherwise, as described in `readme.md: Setup`, you can use `prediction_env.yaml`, to create a new environment that fulfills the requirements.

## 7) Repository File Inventory

### Root

- `.gitignore` - Git ignore rules
- `readme.md` - minimal quickstart documentation
- `project_report.md` - full technical documentation (this file)
- `requirements.txt` - Python dependencies

### Dataset Creation

- `dataset_creation/parquet_file_creation.py` - local files to parquet conversion
- `dataset_creation/create_parquet_files_from_owncloud.py` - ownCloud download + parquet conversion
- `dataset_creation/combined_feature_creation.py` - sliding-window multimodal feature generation
- `dataset_creation/maxDist.py` - helper/statistical utility script for eye-tracking feature creation 

#### AU Creation
- `dataset_creation/AU_creation/AU_creation_service.py` - AU extraction service workflow
- `dataset_creation/AU_creation/pyfeat_docu.ipynb` - py-feat exploratory notes

#### Camera Handling
- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py` - current camera + AU + eye online pipeline
- `dataset_creation/camera_handling/eyeFeature_new.py` - eye-feature extraction from gaze parquet
- `dataset_creation/camera_handling/db_helper.py` - SQLite helper functions (camera pipeline)
- `dataset_creation/camera_handling/camera_stream_AU_and_ET.py` - older pipeline variant
- `dataset_creation/camera_handling/camera_stream.py` - baseline camera streaming script
- `dataset_creation/camera_handling/db_test.py` - DB test utility

### EDA

- `EDA/EDA.ipynb` - main EDA notebook
- `EDA/distribution_plots.ipynb` - distribution visualization
- `EDA/histogramms.ipynb` - histogram analysis
- `EDA/researchOnSubjectPerformance.ipynb` - subject-level analysis
- `EDA/owncloud_file_access.ipynb` - ownCloud exploration/access notebook
- `EDA/calculate_replacement_values.ipynb` - fallback/median computation notebook
- `EDA/login.yaml` - local auth/config artifact for EDA workflows

### Model Training

#### CNN
- `model_training/CNN/CNN_simple.ipynb`
- `model_training/CNN/CNN_crossVal.ipynb`
- `model_training/CNN/CNN_crossVal_EarlyFusion.ipynb`
- `model_training/CNN/CNN_crossVal_EarlyFusion_Filter.ipynb`
- `model_training/CNN/CNN_crossVal_EarlyFusion_Test_Eval.ipynb`
- `model_training/CNN/CNN_crossVal_faceAUs.ipynb`
- `model_training/CNN/CNN_crossVal_faceAUs_eyeFeatures.ipynb`
- `model_training/CNN/CNN_crossVal_HybridFusion.ipynb`
- `model_training/CNN/CNN_crossVal_HybridFusion_Test_Eval.ipynb`
- `model_training/CNN/deployment_pipeline.ipynb`

#### XGBoost
- `model_training/xgboost/xgboost.ipynb`
- `model_training/xgboost/xgboost_groupfold.ipynb`
- `model_training/xgboost/xgboost_new_dataset.ipynb`
- `model_training/xgboost/xgboost_regulated.ipynb`
- `model_training/xgboost/xgboost_with_AE.ipynb`
- `model_training/xgboost/xgboost_with_MAD.ipynb`

#### Isolation Forest
- `model_training/IsolationForest/iforest_training.ipynb`

#### OCSVM
- `model_training/OCSVM/ocsvm_with_AE.ipynb`

#### DeepSVDD
- `model_training/DeepSVDD/deepSVDD.ipynb`

#### MAD Outlier Removal
- `model_training/MAD_outlier_removal/mad_outlier_removal.ipynb`
- `model_training/MAD_outlier_removal/mad_outlier_removal_median.ipynb`

#### Shared Training Tools
- `model_training/tools/scaler.py`
- `model_training/tools/performance_split.py`
- `model_training/tools/mad_outlier_removal.py`
- `model_training/tools/evaluation_tools.py`

### Prediction Pipeline

- `predict_pipeline/predict_sample.py` - runtime prediction + MQTT publish
- `predict_pipeline/config.yaml` - runtime database/model/scaler/mqtt config
- `predict_pipeline/fill_db.ipynb` - helper notebook for DB setup/testing
- `predict_pipeline/predict.service` - systemd service unit
- `predict_pipeline/predict.timer` - systemd timer unit
- `predict_pipeline/predict_service_timer_documentation.md` - Linux service/timer guide

### Generic Tools

- `tools/db_helpers.py` - common SQLite utilities used to get newest sample for prediction

## 8) Additional Information

- Several paths are hardcoded on purpose to ensure compability with the jetsonboard at the OHM-UX driving simulator.
- Camera and AU processing are resource-intensive; version pinning and hardware validation are recommended.