280 lines
11 KiB
Markdown
280 lines
11 KiB
Markdown
# Project Report: Multimodal Driver State Analysis
|
|
|
|
## 1) Project Scope
|
|
|
|
This repository implements an end-to-end workflow for multimodal driver-state analysis in a simulator setup.
|
|
The system combines:
|
|
- Facial Action Units (AUs)
|
|
- Eye-tracking features (fixations, saccades, blinks, pupil behavior)
|
|
|
|
Apart from this, several machine learning model architectures are presented and evaluated.
|
|
|
|
Content:
|
|
- Dataset generation
|
|
- Exploratory data analysis
|
|
- Model training experiments
|
|
- Real-time inference with SQlite, systemd and MQTT
|
|
- Repository file inventory
|
|
- Additional nformation
|
|
|
|
|
|
## 2) Dataset generation
|
|
|
|
### 2.1 Data Access, Filtering, and Data Conversion
|
|
|
|
Main scripts:
|
|
- `dataset_creation/create_parquet_files_from_owncloud.py`
|
|
- `dataset_creation/parquet_file_creation.py`
|
|
|
|
Purpose:
|
|
- Download and/or access dataset files (either download first via ```EDA/owncloud_file_access.ipynb``` or all in one with ```dataset_creation/create_parquet_files_from_owncloud.py```
|
|
- Keep relevant columns (FACE_AUs and eye-tracking raw values)
|
|
- Filter invalid samples (e.g., invalid level segments): Make sure not to drop rows where NaN is necessary for later feature creation, therefore use subset argument in dropNa()!
|
|
- Export subject-level parquet files
|
|
- Before running the scripts: be aware that the whole dataset contains 30 files with around 900 Mbytes each, provide enough storage and expect this to take a while.
|
|
|
|
|
|
### 2.2 Feature Engineering (Offline)
|
|
|
|
Main script:
|
|
- `dataset_creation/combined_feature_creation.py`
|
|
|
|
Behavior:
|
|
- Builds fixed-size sliding windows over subject time series (window size and step size can be adjusted)
|
|
- Uses prepared parquet files from 2.1
|
|
- Aggregates AU statistics per window (e.g., `FACE_AUxx_mean`)
|
|
- Computes eye-feature aggregates (fix/sacc/blink/pupil metrics)
|
|
- Produces training-ready feature tables = dataset
|
|
- Parameter ```MIN_DUR_BLINKS``` can be adjusted, although this value needs to make sense in combination with your sampling frequency
|
|
- With low videostream rates, consider to reevaluate the meaningfulness of some eye-tracking features, especially the fixations
|
|
- running the script requires a manual installation of pygaze Analyser library from [github](https://github.com/esdalmaijer/PyGazeAnalyser.git)
|
|
|
|
### 2.3 Online Camera + Eye + AU Feature Extraction
|
|
|
|
Main scripts:
|
|
- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py`
|
|
- `dataset_creation/camera_handling/eyeFeature_new.py`
|
|
- `dataset_creation/camera_handling/db_helper.py`
|
|
|
|
Runtime behavior:
|
|
- Captures webcam stream with OpenCV
|
|
- Extracts gaze/iris-based signals via MediaPipe
|
|
- Records overlapping windows (`VIDEO_DURATION=50s`, `START_INTERVAL=5s`, `FPS=25`)
|
|
- Runs AU extraction (`py-feat`) from recorded video segments
|
|
- Computes eye-feature summary from generated gaze parquet
|
|
- Writes merged rows to SQLite table `feature_table`
|
|
|
|
Operational note:
|
|
- `DB_PATH` and other paths are currently code-configured and must be adapted per deployment.
|
|
|
|
## 3) EDA
|
|
The directory EDA provides several files to get insights into both the raw data from AdaBase and your own dataset.
|
|
|
|
- `EDA.ipynb` - main EDA notebook: recreates the plot from AdaBase documentation, lists all experiments and in general serves as a playground for you to get to know the files.
|
|
- `distribution_plots.ipynb` - This notebook aimes to visualize the data distributions for each experiment - the goal is the find out, whether the split of experiments into high and low cognitive load is clearer if some experiments are dropped.
|
|
- `histogramms.ipynb` - Histogram analysis of low load vs high load per feature. Additionaly, scatter plots per feature are available.
|
|
- `researchOnSubjectPerformance.ipynb` - This noteboooks aims to see how the performance values range for the 30 subjects. The code creates and saves a table in csv-format, which will later be used as the foundation of the performance based split in ```model_training/tools/performance_based_split```
|
|
- `owncloud_file_access.ipynb` - Get access to the files via owncloud and safe them as .h5 files, in correspondence to the parquet file creation script
|
|
- `login.yaml` - used to store URL and password to access files from owncloud, used in previous notebook
|
|
- `calculate_replacement_values.ipynb` - fallback / median computation notebook for deployment, creation of yaml syntax embedding
|
|
|
|
General information:
|
|
- Due to their size, its absolutely recommended to download and save the dataset files once in the beginning
|
|
- For better data understanding, read the [AdaBase publication](https://www.mdpi.com/1424-8220/23/1/340)
|
|
|
|
|
|
## 4) Model Training
|
|
|
|
Included model families:
|
|
- CNN variants (different fusion strategies)
|
|
- XGBoost
|
|
- Isolation Forest
|
|
- OCSVM
|
|
- DeepSVDD
|
|
|
|
Supporting utilities:
|
|
- `model_training/tools/scaler.py`
|
|
- `model_training/tools/performance_split.py`
|
|
- `model_training/tools/mad_outlier_removal.py`
|
|
- `model_training/tools/evaluation_tools.py`
|
|
|
|
## 5) Real-Time Prediction and Messaging
|
|
|
|
Main script:
|
|
- `predict_pipeline/predict_sample.py`
|
|
|
|
Pipeline:
|
|
- Loads runtime config (`predict_pipeline/config.yaml`)
|
|
- Pulls latest row from SQLite (`database.path/table/key`)
|
|
- Replaces missing values using `fallback` map
|
|
- Optionally applies scaler (`.pkl`/`.joblib`)
|
|
- Loads model (`.keras`, `.pkl`, `.joblib`) and predicts
|
|
- Publishes JSON payload to MQTT topic
|
|
|
|
Expected payload form:
|
|
```json
|
|
{
|
|
"valid": true,
|
|
"_id": 123,
|
|
"prediction": 0
|
|
}
|
|
```
|
|
|
|
### 5.1 Scheduled Prediction (Linux)
|
|
|
|
Files:
|
|
- `predict_pipeline/predict.service`
|
|
- `predict_pipeline/predict.timer`
|
|
- `predict_pipeline/predict_service_timer_documentation.md`
|
|
|
|
Role:
|
|
- Run inference repeatedly without manual execution
|
|
- Timer/service configuration can be customized per target machine
|
|
|
|
## 5.2 Runtime Configuration
|
|
|
|
Primary config file:
|
|
- `predict_pipeline/config.yaml`
|
|
|
|
Sections:
|
|
- `database`: SQLite location + table + sort key
|
|
- `model`: model path
|
|
- `scaler`: scaler usage + path
|
|
- `mqtt`: broker and publish format
|
|
- `sample.columns`: expected feature order
|
|
- `fallback`: default values for NaN replacement
|
|
|
|
Important:
|
|
- The repository currently uses environment-specific absolute paths in some scripts/configs.
|
|
- Paths should be normalized before deployment to a new machine.
|
|
|
|
## 5.3) Data and Feature Expectations
|
|
|
|
Prediction expects SQLite rows containing:
|
|
- `_Id`
|
|
- `start_time`
|
|
- All configured model features (AUs + eye metrics)
|
|
|
|
Common feature groups:
|
|
- `FACE_AUxx_mean` columns
|
|
- Fixation counters and duration statistics
|
|
- Saccade count/amplitude/duration statistics
|
|
- Blink count/duration statistics
|
|
- Pupil mean and IPA
|
|
|
|
## 6) Installation and Dependencies
|
|
Due to unsolvable dependency conflicts, several environemnts need to be used in the same time.
|
|
### 6.1 Environemnt for camera handling
|
|
TO DO
|
|
|
|
|
|
### 6.2 Environment for predictions
|
|
Install base requirements:
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
Typical key packages in this project:
|
|
- `numpy`, `pandas`, `scikit-learn`, `scipy`, `pyarrow`, `pyyaml`, `joblib`
|
|
- `opencv-python`, `mediapipe`, `torch`, `py-feat`, `pygazeanalyser`
|
|
- `paho-mqtt`
|
|
- optional data access stack (`pyocclient`, `h5py`, `tables`)
|
|
|
|
## 7) Repository File Inventory
|
|
|
|
### Root
|
|
|
|
- `.gitignore` - Git ignore rules
|
|
- `readme.md` - minimal quickstart documentation
|
|
- `project_report.md` - full technical documentation (this file)
|
|
- `requirements.txt` - Python dependencies
|
|
|
|
### Dataset Creation
|
|
|
|
- `dataset_creation/parquet_file_creation.py` - local source to parquet conversion
|
|
- `dataset_creation/create_parquet_files_from_owncloud.py` - ownCloud download + parquet conversion
|
|
- `dataset_creation/combined_feature_creation.py` - sliding-window multimodal feature generation
|
|
- `dataset_creation/maxDist.py` - helper/statistical utility script
|
|
|
|
#### AU Creation
|
|
- `dataset_creation/AU_creation/AU_creation_service.py` - AU extraction service workflow
|
|
- `dataset_creation/AU_creation/pyfeat_docu.ipynb` - py-feat exploratory notes
|
|
|
|
#### Camera Handling
|
|
- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py` - current camera + AU + eye online pipeline
|
|
- `dataset_creation/camera_handling/eyeFeature_new.py` - eye-feature extraction from gaze parquet
|
|
- `dataset_creation/camera_handling/db_helper.py` - SQLite helper functions (camera pipeline)
|
|
- `dataset_creation/camera_handling/camera_stream_AU_and_ET.py` - older pipeline variant
|
|
- `dataset_creation/camera_handling/camera_stream.py` - baseline camera streaming script
|
|
- `dataset_creation/camera_handling/db_test.py` - DB test utility
|
|
|
|
### EDA
|
|
|
|
- `EDA/EDA.ipynb` - main EDA notebook
|
|
- `EDA/distribution_plots.ipynb` - distribution visualization
|
|
- `EDA/histogramms.ipynb` - histogram analysis
|
|
- `EDA/researchOnSubjectPerformance.ipynb` - subject-level analysis
|
|
- `EDA/owncloud_file_access.ipynb` - ownCloud exploration/access notebook
|
|
- `EDA/calculate_replacement_values.ipynb` - fallback/median computation notebook
|
|
- `EDA/login.yaml` - local auth/config artifact for EDA workflows
|
|
|
|
### Model Training
|
|
|
|
#### CNN
|
|
- `model_training/CNN/CNN_simple.ipynb`
|
|
- `model_training/CNN/CNN_crossVal.ipynb`
|
|
- `model_training/CNN/CNN_crossVal_EarlyFusion.ipynb`
|
|
- `model_training/CNN/CNN_crossVal_EarlyFusion_Filter.ipynb`
|
|
- `model_training/CNN/CNN_crossVal_EarlyFusion_Test_Eval.ipynb`
|
|
- `model_training/CNN/CNN_crossVal_faceAUs.ipynb`
|
|
- `model_training/CNN/CNN_crossVal_faceAUs_eyeFeatures.ipynb`
|
|
- `model_training/CNN/CNN_crossVal_HybridFusion.ipynb`
|
|
- `model_training/CNN/CNN_crossVal_HybridFusion_Test_Eval.ipynb`
|
|
- `model_training/CNN/deployment_pipeline.ipynb`
|
|
|
|
#### XGBoost
|
|
- `model_training/xgboost/xgboost.ipynb`
|
|
- `model_training/xgboost/xgboost_groupfold.ipynb`
|
|
- `model_training/xgboost/xgboost_new_dataset.ipynb`
|
|
- `model_training/xgboost/xgboost_regulated.ipynb`
|
|
- `model_training/xgboost/xgboost_with_AE.ipynb`
|
|
- `model_training/xgboost/xgboost_with_MAD.ipynb`
|
|
|
|
#### Isolation Forest
|
|
- `model_training/IsolationForest/iforest_training.ipynb`
|
|
|
|
#### OCSVM
|
|
- `model_training/OCSVM/ocsvm_with_AE.ipynb`
|
|
|
|
#### DeepSVDD
|
|
- `model_training/DeepSVDD/deepSVDD.ipynb`
|
|
|
|
#### MAD Outlier Removal
|
|
- `model_training/MAD_outlier_removal/mad_outlier_removal.ipynb`
|
|
- `model_training/MAD_outlier_removal/mad_outlier_removal_median.ipynb`
|
|
|
|
#### Shared Training Tools
|
|
- `model_training/tools/scaler.py`
|
|
- `model_training/tools/performance_split.py`
|
|
- `model_training/tools/mad_outlier_removal.py`
|
|
- `model_training/tools/evaluation_tools.py`
|
|
|
|
### Prediction Pipeline
|
|
|
|
- `predict_pipeline/predict_sample.py` - runtime prediction + MQTT publish
|
|
- `predict_pipeline/config.yaml` - runtime database/model/scaler/mqtt config
|
|
- `predict_pipeline/fill_db.ipynb` - helper notebook for DB setup/testing
|
|
- `predict_pipeline/predict.service` - systemd service unit
|
|
- `predict_pipeline/predict.timer` - systemd timer unit
|
|
- `predict_pipeline/predict_service_timer_documentation.md` - Linux service/timer guide
|
|
|
|
### Generic Tools
|
|
|
|
- `tools/db_helpers.py` - common SQLite utilities used to get newest sample for prediction
|
|
|
|
## 8) Additional Information
|
|
|
|
- Several paths are hardcoded on purpose to ensure compability with the jetsonboard at the OHM-UX driving simulator.
|
|
- Camera and AU processing are resource-intensive; version pinning and hardware validation are recommended.
|
|
|