kurzti88066/Fahrsimulator_MSY2526_AI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cleaned readme, added project report

Browse Source
This commit is contained in:
Michael Weig 2026-03-09 20:45:24 +01:00
parent 0b2c629d16
commit fb0d39c668
2 changed files with 276 additions and 257 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

253
project_report.md Normal file
View File

@ -0,0 +1,253 @@
# Project Report: Multimodal Driver State Analysis
## 1) Project Scope
This repository implements an end-to-end workflow for multimodal driver-state analysis in a simulator setup.
The system combines:
- Facial Action Units (AUs)
- Eye-tracking features (fixations, saccades, blinks, pupil behavior)
It covers:
- Data ingestion and conversion
- Sliding-window feature generation
- Exploratory analysis
- Model training experiments
- Real-time inference from SQLite
- MQTT publishing
- Optional Linux `systemd` scheduling
## 2) End-to-End Workflow
### 2.1 Data Ingestion and Conversion
Main scripts:
- `dataset_creation/create_parquet_files_from_owncloud.py`
- `dataset_creation/parquet_file_creation.py`
Purpose:
- Read source recordings (`.h5` and/or ownCloud-fetched files)
- Keep relevant simulator/physiology columns
- Filter invalid samples (e.g., invalid level segments)
- Export subject-level parquet files
### 2.2 Feature Engineering (Offline)
Main script:
- `dataset_creation/combined_feature_creation.py`
Behavior:
- Builds fixed-size sliding windows over subject time series
- Aggregates AU statistics per window (e.g., `FACE_AUxx_mean`)
- Computes eye-feature aggregates (fix/sacc/blink/pupil metrics)
- Produces training-ready feature tables
### 2.3 Online Camera + Eye + AU Feature Extraction
Main scripts:
- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py`
- `dataset_creation/camera_handling/eyeFeature_new.py`
- `dataset_creation/camera_handling/db_helper.py`
Runtime behavior:
- Captures webcam stream with OpenCV
- Extracts gaze/iris-based signals via MediaPipe
- Records overlapping windows (`VIDEO_DURATION=50s`, `START_INTERVAL=5s`, `FPS=25`)
- Runs AU extraction (`py-feat`) from recorded video segments
- Computes eye-feature summary from generated gaze parquet
- Writes merged rows to SQLite table `feature_table`
Operational note:
- `DB_PATH` and other paths are currently code-configured and must be adapted per deployment.
### 2.4 Model Training
Location:
- `model_training/` (primarily notebook-driven)
Included model families:
- CNN variants (different fusion strategies)
- XGBoost
- Isolation Forest
- OCSVM
- DeepSVDD
Supporting utilities:
- `model_training/tools/scaler.py`
- `model_training/tools/performance_split.py`
- `model_training/tools/mad_outlier_removal.py`
- `model_training/tools/evaluation_tools.py`
### 2.5 Real-Time Prediction and Messaging
Main script:
- `predict_pipeline/predict_sample.py`
Pipeline:
- Loads runtime config (`predict_pipeline/config.yaml`)
- Pulls latest row from SQLite (`database.path/table/key`)
- Replaces missing values using `fallback` map
- Optionally applies scaler (`.pkl`/`.joblib`)
- Loads model (`.keras`, `.pkl`, `.joblib`) and predicts
- Publishes JSON payload to MQTT topic
Expected payload form:
```json
{
"valid": true,
"_id": 123,
"prediction": 0
}
```
### 2.6 Scheduled Prediction (Linux)
Files:
- `predict_pipeline/predict.service`
- `predict_pipeline/predict.timer`
- `predict_pipeline/predict_service_timer_documentation.md`
Role:
- Run inference repeatedly without manual execution
- Timer/service configuration can be customized per target machine
## 3) Runtime Configuration
Primary config file:
- `predict_pipeline/config.yaml`
Sections:
- `database`: SQLite location + table + sort key
- `model`: model path
- `scaler`: scaler usage + path
- `mqtt`: broker and publish format
- `sample.columns`: expected feature order
- `fallback`: default values for NaN replacement
Important:
- The repository currently uses environment-specific absolute paths in some scripts/configs.
- Paths should be normalized before deployment to a new machine.
## 4) Data and Feature Expectations
Prediction expects SQLite rows containing:
- `_Id`
- `start_time`
- All configured model features (AUs + eye metrics)
Common feature groups:
- `FACE_AUxx_mean` columns
- Fixation counters and duration statistics
- Saccade count/amplitude/duration statistics
- Blink count/duration statistics
- Pupil mean and IPA
## 5) Installation and Dependencies
Install base requirements:
```bash
pip install -r requirements.txt
```
Typical key packages in this project:
- `numpy`, `pandas`, `scikit-learn`, `scipy`, `pyarrow`, `pyyaml`, `joblib`
- `opencv-python`, `mediapipe`, `torch`, `py-feat`, `pygazeanalyser`
- `paho-mqtt`
- optional data access stack (`pyocclient`, `h5py`, `tables`)
## 6) Repository File Inventory
### 6.1 Root
- `.gitignore` - Git ignore rules
- `readme.md` - minimal quickstart documentation
- `project_report.md` - full technical documentation (this file)
- `requirements.txt` - Python dependencies
### 6.2 Dataset Creation
- `dataset_creation/parquet_file_creation.py` - local source to parquet conversion
- `dataset_creation/create_parquet_files_from_owncloud.py` - ownCloud download + parquet conversion
- `dataset_creation/combined_feature_creation.py` - sliding-window multimodal feature generation
- `dataset_creation/maxDist.py` - helper/statistical utility script
#### AU Creation
- `dataset_creation/AU_creation/AU_creation_service.py` - AU extraction service workflow
- `dataset_creation/AU_creation/pyfeat_docu.ipynb` - py-feat exploratory notes
#### Camera Handling
- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py` - current camera + AU + eye online pipeline
- `dataset_creation/camera_handling/eyeFeature_new.py` - eye-feature extraction from gaze parquet
- `dataset_creation/camera_handling/db_helper.py` - SQLite helper functions (camera pipeline)
- `dataset_creation/camera_handling/camera_stream_AU_and_ET.py` - older pipeline variant
- `dataset_creation/camera_handling/camera_stream.py` - baseline camera streaming script
- `dataset_creation/camera_handling/db_test.py` - DB test utility
### 6.3 EDA
- `EDA/EDA.ipynb` - main EDA notebook
- `EDA/distribution_plots.ipynb` - distribution visualization
- `EDA/histogramms.ipynb` - histogram analysis
- `EDA/researchOnSubjectPerformance.ipynb` - subject-level analysis
- `EDA/owncloud_file_access.ipynb` - ownCloud exploration/access notebook
- `EDA/calculate_replacement_values.ipynb` - fallback/median computation notebook
- `EDA/login.yaml` - local auth/config artifact for EDA workflows
### 6.4 Model Training
#### CNN
- `model_training/CNN/CNN_simple.ipynb`
- `model_training/CNN/CNN_crossVal.ipynb`
- `model_training/CNN/CNN_crossVal_EarlyFusion.ipynb`
- `model_training/CNN/CNN_crossVal_EarlyFusion_Filter.ipynb`
- `model_training/CNN/CNN_crossVal_EarlyFusion_Test_Eval.ipynb`
- `model_training/CNN/CNN_crossVal_faceAUs.ipynb`
- `model_training/CNN/CNN_crossVal_faceAUs_eyeFeatures.ipynb`
- `model_training/CNN/CNN_crossVal_HybridFusion.ipynb`
- `model_training/CNN/CNN_crossVal_HybridFusion_Test_Eval.ipynb`
- `model_training/CNN/deployment_pipeline.ipynb`
#### XGBoost
- `model_training/xgboost/xgboost.ipynb`
- `model_training/xgboost/xgboost_groupfold.ipynb`
- `model_training/xgboost/xgboost_new_dataset.ipynb`
- `model_training/xgboost/xgboost_regulated.ipynb`
- `model_training/xgboost/xgboost_with_AE.ipynb`
- `model_training/xgboost/xgboost_with_MAD.ipynb`
#### Isolation Forest
- `model_training/IsolationForest/iforest_training.ipynb`
#### OCSVM
- `model_training/OCSVM/ocsvm_with_AE.ipynb`
#### DeepSVDD
- `model_training/DeepSVDD/deepSVDD.ipynb`
#### MAD Outlier Removal
- `model_training/MAD_outlier_removal/mad_outlier_removal.ipynb`
- `model_training/MAD_outlier_removal/mad_outlier_removal_median.ipynb`
#### Shared Training Tools
- `model_training/tools/scaler.py`
- `model_training/tools/performance_split.py`
- `model_training/tools/mad_outlier_removal.py`
- `model_training/tools/evaluation_tools.py`
### 6.5 Prediction Pipeline
- `predict_pipeline/predict_sample.py` - runtime prediction + MQTT publish
- `predict_pipeline/config.yaml` - runtime database/model/scaler/mqtt config
- `predict_pipeline/fill_db.ipynb` - helper notebook for DB setup/testing
- `predict_pipeline/predict.service` - systemd service unit
- `predict_pipeline/predict.timer` - systemd timer unit
- `predict_pipeline/predict_service_timer_documentation.md` - Linux service/timer guide
### 6.6 Generic Tools
- `tools/db_helpers.py` - common SQLite utilities used by prediction path
## 7) Known Technical Notes
- Several paths are hardcoded for a specific runtime environment and should be parameterized for portability.
- Camera and AU processing are resource-intensive; version pinning and hardware validation are recommended.

280
readme.md
View File

@ -1,279 +1,45 @@
# Multimodal Driver State Analysis # Multimodal Driver State Analysis
This repository contains a full workflow for multimodal driver-state analysis in a simulator setting, from raw recording data to trained models and real-time inference. Short overview: this repository contains the data, feature, training, and inference pipeline for multimodal driver-state analysis using facial AUs and eye-tracking signals.
It combines two modalities: For full documentation, see `project_report.md`.
- Facial Action Units (AUs)
- Eye-tracking features (fixations, saccades, blinks, pupil dynamics)
## What This Project Covers ## Quickstart
- Data extraction from raw simulator files (`.h5` / ownCloud)
- Conversion to subject-level Parquet files
- Sliding-window feature engineering (AU + eye tracking)
- Exploratory data analysis (EDA) notebooks
- Model training experiments (CNN, XGBoost, Isolation Forest, OCSVM, DeepSVDD)
- Real-time prediction from SQLite + MQTT publishing
- Optional Linux `systemd` deployment (`predict.service` + `predict.timer`)
## Repository Structure
```text
Fahrsimulator_MSY2526_AI/
|-- dataset_creation/
| |-- parquet_file_creation.py
| |-- create_parquet_files_from_owncloud.py
| |-- combined_feature_creation.py
| |-- maxDist.py
| |-- AU_creation/
| | |-- AU_creation_service.py
| | `-- pyfeat_docu.ipynb
| `-- camera_handling/
| |-- camera_stream_AU_and_ET_new.py
| |-- eyeFeature_new.py
| |-- db_helper.py
| `-- *.py (legacy variants/tests)
|-- EDA/
| `-- *.ipynb
|-- model_training/
| |-- CNN/
| |-- xgboost/
| |-- IsolationForest/
| |-- OCSVM/
| |-- DeepSVDD/
| |-- MAD_outlier_removal/
| `-- tools/
|-- predict_pipeline/
| |-- predict_sample.py
| |-- config.yaml
| |-- predict.service
| |-- predict.timer
| |-- predict_service_timer_documentation.md
| `-- fill_db.ipynb
|-- tools/
| `-- db_helpers.py
`-- readme.md
```
## End-to-End Workflow
## 1) Data Ingestion and Conversion
Main scripts:
- `dataset_creation/create_parquet_files_from_owncloud.py`
- `dataset_creation/parquet_file_creation.py`
Purpose:
- Load simulator recordings from ownCloud or local `.h5` files.
- Select relevant columns (`STUDY`, `LEVEL`, `PHASE`, `FACE_AU*`, `EYE_*`).
- Filter invalid rows (for example `LEVEL == 0`).
- Save cleaned subject-level Parquet files.
Notes:
- These scripts contain placeholders for paths and credentials that must be adapted.
- ownCloud download uses `pyocclient` (`owncloud` module).
## 2) Feature Engineering (Offline Dataset)
Main script:
- `dataset_creation/combined_feature_creation.py`
Behavior:
- Processes all Parquet files in an input directory.
- Applies sliding windows:
- Window size: 50 seconds (`25 Hz * 50 = 1250 samples`)
- Step size: 5 seconds (`125 samples`)
- Groups data by available context columns (`STUDY`, `LEVEL`, `PHASE`).
- Computes:
- AU means per window (`FACE_AUxx_mean`)
- Eye-tracking features:
- Fixation counts and duration stats
- Saccade count/amplitude/duration stats
- Blink count/duration stats
- Pupil mean and IPA (high-frequency pupil activity)
Output:
- A combined Parquet dataset (one row per window), ready for model training.
## 3) Camera-Based Online Feature Extraction
Main scripts:
- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py`
- `dataset_creation/camera_handling/eyeFeature_new.py`
Behavior:
- Captures webcam stream (`OpenCV`) at ~25 FPS.
- Computes eye metrics with `MediaPipe`.
- Records 50-second overlapping segments (new start every 5 seconds).
- Extracts AUs from recorded clips using `py-feat`.
- Extracts eye features from saved gaze parquet.
- Writes combined feature rows into an SQLite table (`feature_table`).
Important:
- Script paths and DB locations are currently hardcoded for the target environment and must be adapted.
## 4) Model Training
Location:
- `model_training/` (mostly notebook-driven)
Includes experiments for:
- CNN-based fusion variants
- XGBoost
- Isolation Forest
- OCSVM
- DeepSVDD
Utility modules:
- `model_training/tools/scaler.py` for fitting/saving/applying scalers
- `model_training/tools/mad_outlier_removal.py`
- `model_training/tools/performance_split.py`
- `model_training/tools/evaluation_tools.py`
## 5) Real-Time Prediction and Messaging
Main script:
- `predict_pipeline/predict_sample.py`
Runtime behavior:
- Reads latest row from SQLite (`database.path`, `database.table`, `database.key`).
- Applies NaN handling using fallback medians from `config.yaml`.
- Optionally scales features using a saved scaler (`.pkl` or `.joblib`).
- Loads model (`.keras`, `.pkl`, or `.joblib`) and predicts.
- Publishes JSON message via MQTT (topic/host/qos from config).
Message shape:
```json
{
"valid": true,
"_id": 123,
"prediction": 0
}
```
(`prediction` key is configurable via `mqtt.publish_format.result_key`.)
## 6) Automated Execution with systemd (Linux)
Files:
- `predict_pipeline/predict.service`
- `predict_pipeline/predict.timer`
Current timer behavior:
- first run after 60s (`OnActiveSec=60`)
- then every 5s (`OnUnitActiveSec=5`)
Detailed operation and commands:
- `predict_pipeline/predict_service_timer_documentation.md`
## Installation
Install dependencies from the tracked requirements file:
### 1) Setup
Activate the conda-repository "".
```bash ```bash
pip install -r requirements.txt conda activate
``` ```
Make sure, another repository that fulfills requirement.txt is available, matching with predict_pipeline/predict.service - See `predict_pipeline/predict_service_timer_documentation.md`
## Python Version
Recommended: ### 2) Camera AU + Eye Pipeline (`camera_stream_AU_and_ET_new.py`)
- Python `3.10` to `3.12`
## Core Dependencies 1. Open `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py` and adjust:
- `DB_PATH`
- `CAMERA_INDEX`
- `OUTPUT_DIR` (optional)
2. Start camera capture and feature extraction:
```bash ```bash
pip install numpy pandas scipy scikit-learn pyarrow pyyaml joblib paho-mqtt matplotlib python dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py
``` ```
## Computer Vision / Eye Tracking / AU Stack 3. Stop with `q` in the camera window.
```bash
pip install opencv-python mediapipe torch moviepy
pip install pygazeanalyser
pip install py-feat
```
## Data Access (optional) ### 3) Predict Pipeline (`predict_pipeline/predict_sample.py`)
```bash 1. Edit `predict_pipeline/config.yaml` and set:
pip install pyocclient h5py tables - `database.path`, `database.table`, `database.key`
``` - `model.path`
- `scaler.path` (if `use_scaling: true`)
- MQTT settings under `mqtt`
## Notes 2. Run one prediction cycle:
- `tensorflow` is required for `.keras` model inference in `predict_sample.py`.
- `py-feat`, `mediapipe`, and `torch` can be platform-sensitive; pin versions per your target machine.
## Configuration
Primary runtime config:
- `predict_pipeline/config.yaml`
Sections:
- `database`: SQLite path/table/key
- `model`: model file path
- `scaler`: scaling toggle + scaler path
- `mqtt`: broker connection + publish format
- `sample.columns`: expected feature order
- `fallback`: median/default feature values used for NaN replacement
Before running prediction, verify all absolute paths in `config.yaml`.
## Quick Start
## A) Build Training Dataset (Offline)
1. Set input/output paths in:
- `dataset_creation/parquet_file_creation.py`
- `dataset_creation/combined_feature_creation.py`
2. Generate subject Parquet files:
```bash
python dataset_creation/parquet_file_creation.py
```
3. Generate combined sliding-window feature dataset:
```bash
python dataset_creation/combined_feature_creation.py
```
## B) Run Prediction Once
1. Update paths in `predict_pipeline/config.yaml`.
2. Run:
```bash ```bash
python predict_pipeline/predict_sample.py python predict_pipeline/predict_sample.py
``` ```
## C) Run as systemd Service + Timer (Linux) 3. Use `predict_service_timer_documentation.md` to see how to use the service and timer for automation.
1. Copy unit files to `/etc/systemd/system/`.
2. Adjust `ExecStart` and user in `predict.service`.
3. Enable and start timer:
```bash
sudo systemctl daemon-reload
sudo systemctl enable predict.timer
sudo systemctl start predict.timer
```
Monitor logs:
```bash
journalctl -u predict.service -f
```
## Database and Table Expectations
The prediction script expects a SQLite table with at least:
- `_Id`
- `start_time`
- all model feature columns listed in `config.yaml` under `sample.columns`
The camera pipeline writes feature rows into `feature_table` using helper utilities in:
- `dataset_creation/camera_handling/db_helper.py`
- `tools/db_helpers.py`
## License
No license file is currently present in this repository.
Add a `LICENSE` file if this project should be shared or reused externally.