From fb0d39c66880b465892d0758a70d7b8596e0a156 Mon Sep 17 00:00:00 2001 From: Michael Date: Mon, 9 Mar 2026 20:45:24 +0100 Subject: [PATCH] cleaned readme, added project report --- project_report.md | 253 +++++++++++++++++++++++++++++++++++++++++ readme.md | 280 ++++------------------------------------------ 2 files changed, 276 insertions(+), 257 deletions(-) create mode 100644 project_report.md diff --git a/project_report.md b/project_report.md new file mode 100644 index 0000000..5da6ccb --- /dev/null +++ b/project_report.md @@ -0,0 +1,253 @@ +# Project Report: Multimodal Driver State Analysis + +## 1) Project Scope + +This repository implements an end-to-end workflow for multimodal driver-state analysis in a simulator setup. +The system combines: +- Facial Action Units (AUs) +- Eye-tracking features (fixations, saccades, blinks, pupil behavior) + +It covers: +- Data ingestion and conversion +- Sliding-window feature generation +- Exploratory analysis +- Model training experiments +- Real-time inference from SQLite +- MQTT publishing +- Optional Linux `systemd` scheduling + +## 2) End-to-End Workflow + +### 2.1 Data Ingestion and Conversion + +Main scripts: +- `dataset_creation/create_parquet_files_from_owncloud.py` +- `dataset_creation/parquet_file_creation.py` + +Purpose: +- Read source recordings (`.h5` and/or ownCloud-fetched files) +- Keep relevant simulator/physiology columns +- Filter invalid samples (e.g., invalid level segments) +- Export subject-level parquet files + +### 2.2 Feature Engineering (Offline) + +Main script: +- `dataset_creation/combined_feature_creation.py` + +Behavior: +- Builds fixed-size sliding windows over subject time series +- Aggregates AU statistics per window (e.g., `FACE_AUxx_mean`) +- Computes eye-feature aggregates (fix/sacc/blink/pupil metrics) +- Produces training-ready feature tables + +### 2.3 Online Camera + Eye + AU Feature Extraction + +Main scripts: +- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py` +- `dataset_creation/camera_handling/eyeFeature_new.py` +- `dataset_creation/camera_handling/db_helper.py` + +Runtime behavior: +- Captures webcam stream with OpenCV +- Extracts gaze/iris-based signals via MediaPipe +- Records overlapping windows (`VIDEO_DURATION=50s`, `START_INTERVAL=5s`, `FPS=25`) +- Runs AU extraction (`py-feat`) from recorded video segments +- Computes eye-feature summary from generated gaze parquet +- Writes merged rows to SQLite table `feature_table` + +Operational note: +- `DB_PATH` and other paths are currently code-configured and must be adapted per deployment. + +### 2.4 Model Training + +Location: +- `model_training/` (primarily notebook-driven) + +Included model families: +- CNN variants (different fusion strategies) +- XGBoost +- Isolation Forest +- OCSVM +- DeepSVDD + +Supporting utilities: +- `model_training/tools/scaler.py` +- `model_training/tools/performance_split.py` +- `model_training/tools/mad_outlier_removal.py` +- `model_training/tools/evaluation_tools.py` + +### 2.5 Real-Time Prediction and Messaging + +Main script: +- `predict_pipeline/predict_sample.py` + +Pipeline: +- Loads runtime config (`predict_pipeline/config.yaml`) +- Pulls latest row from SQLite (`database.path/table/key`) +- Replaces missing values using `fallback` map +- Optionally applies scaler (`.pkl`/`.joblib`) +- Loads model (`.keras`, `.pkl`, `.joblib`) and predicts +- Publishes JSON payload to MQTT topic + +Expected payload form: +```json +{ + "valid": true, + "_id": 123, + "prediction": 0 +} +``` + +### 2.6 Scheduled Prediction (Linux) + +Files: +- `predict_pipeline/predict.service` +- `predict_pipeline/predict.timer` +- `predict_pipeline/predict_service_timer_documentation.md` + +Role: +- Run inference repeatedly without manual execution +- Timer/service configuration can be customized per target machine + +## 3) Runtime Configuration + +Primary config file: +- `predict_pipeline/config.yaml` + +Sections: +- `database`: SQLite location + table + sort key +- `model`: model path +- `scaler`: scaler usage + path +- `mqtt`: broker and publish format +- `sample.columns`: expected feature order +- `fallback`: default values for NaN replacement + +Important: +- The repository currently uses environment-specific absolute paths in some scripts/configs. +- Paths should be normalized before deployment to a new machine. + +## 4) Data and Feature Expectations + +Prediction expects SQLite rows containing: +- `_Id` +- `start_time` +- All configured model features (AUs + eye metrics) + +Common feature groups: +- `FACE_AUxx_mean` columns +- Fixation counters and duration statistics +- Saccade count/amplitude/duration statistics +- Blink count/duration statistics +- Pupil mean and IPA + +## 5) Installation and Dependencies + +Install base requirements: +```bash +pip install -r requirements.txt +``` + +Typical key packages in this project: +- `numpy`, `pandas`, `scikit-learn`, `scipy`, `pyarrow`, `pyyaml`, `joblib` +- `opencv-python`, `mediapipe`, `torch`, `py-feat`, `pygazeanalyser` +- `paho-mqtt` +- optional data access stack (`pyocclient`, `h5py`, `tables`) + +## 6) Repository File Inventory + +### 6.1 Root + +- `.gitignore` - Git ignore rules +- `readme.md` - minimal quickstart documentation +- `project_report.md` - full technical documentation (this file) +- `requirements.txt` - Python dependencies + +### 6.2 Dataset Creation + +- `dataset_creation/parquet_file_creation.py` - local source to parquet conversion +- `dataset_creation/create_parquet_files_from_owncloud.py` - ownCloud download + parquet conversion +- `dataset_creation/combined_feature_creation.py` - sliding-window multimodal feature generation +- `dataset_creation/maxDist.py` - helper/statistical utility script + +#### AU Creation +- `dataset_creation/AU_creation/AU_creation_service.py` - AU extraction service workflow +- `dataset_creation/AU_creation/pyfeat_docu.ipynb` - py-feat exploratory notes + +#### Camera Handling +- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py` - current camera + AU + eye online pipeline +- `dataset_creation/camera_handling/eyeFeature_new.py` - eye-feature extraction from gaze parquet +- `dataset_creation/camera_handling/db_helper.py` - SQLite helper functions (camera pipeline) +- `dataset_creation/camera_handling/camera_stream_AU_and_ET.py` - older pipeline variant +- `dataset_creation/camera_handling/camera_stream.py` - baseline camera streaming script +- `dataset_creation/camera_handling/db_test.py` - DB test utility + +### 6.3 EDA + +- `EDA/EDA.ipynb` - main EDA notebook +- `EDA/distribution_plots.ipynb` - distribution visualization +- `EDA/histogramms.ipynb` - histogram analysis +- `EDA/researchOnSubjectPerformance.ipynb` - subject-level analysis +- `EDA/owncloud_file_access.ipynb` - ownCloud exploration/access notebook +- `EDA/calculate_replacement_values.ipynb` - fallback/median computation notebook +- `EDA/login.yaml` - local auth/config artifact for EDA workflows + +### 6.4 Model Training + +#### CNN +- `model_training/CNN/CNN_simple.ipynb` +- `model_training/CNN/CNN_crossVal.ipynb` +- `model_training/CNN/CNN_crossVal_EarlyFusion.ipynb` +- `model_training/CNN/CNN_crossVal_EarlyFusion_Filter.ipynb` +- `model_training/CNN/CNN_crossVal_EarlyFusion_Test_Eval.ipynb` +- `model_training/CNN/CNN_crossVal_faceAUs.ipynb` +- `model_training/CNN/CNN_crossVal_faceAUs_eyeFeatures.ipynb` +- `model_training/CNN/CNN_crossVal_HybridFusion.ipynb` +- `model_training/CNN/CNN_crossVal_HybridFusion_Test_Eval.ipynb` +- `model_training/CNN/deployment_pipeline.ipynb` + +#### XGBoost +- `model_training/xgboost/xgboost.ipynb` +- `model_training/xgboost/xgboost_groupfold.ipynb` +- `model_training/xgboost/xgboost_new_dataset.ipynb` +- `model_training/xgboost/xgboost_regulated.ipynb` +- `model_training/xgboost/xgboost_with_AE.ipynb` +- `model_training/xgboost/xgboost_with_MAD.ipynb` + +#### Isolation Forest +- `model_training/IsolationForest/iforest_training.ipynb` + +#### OCSVM +- `model_training/OCSVM/ocsvm_with_AE.ipynb` + +#### DeepSVDD +- `model_training/DeepSVDD/deepSVDD.ipynb` + +#### MAD Outlier Removal +- `model_training/MAD_outlier_removal/mad_outlier_removal.ipynb` +- `model_training/MAD_outlier_removal/mad_outlier_removal_median.ipynb` + +#### Shared Training Tools +- `model_training/tools/scaler.py` +- `model_training/tools/performance_split.py` +- `model_training/tools/mad_outlier_removal.py` +- `model_training/tools/evaluation_tools.py` + +### 6.5 Prediction Pipeline + +- `predict_pipeline/predict_sample.py` - runtime prediction + MQTT publish +- `predict_pipeline/config.yaml` - runtime database/model/scaler/mqtt config +- `predict_pipeline/fill_db.ipynb` - helper notebook for DB setup/testing +- `predict_pipeline/predict.service` - systemd service unit +- `predict_pipeline/predict.timer` - systemd timer unit +- `predict_pipeline/predict_service_timer_documentation.md` - Linux service/timer guide + +### 6.6 Generic Tools + +- `tools/db_helpers.py` - common SQLite utilities used by prediction path + +## 7) Known Technical Notes + +- Several paths are hardcoded for a specific runtime environment and should be parameterized for portability. +- Camera and AU processing are resource-intensive; version pinning and hardware validation are recommended. + diff --git a/readme.md b/readme.md index 063e485..415cbbd 100644 --- a/readme.md +++ b/readme.md @@ -1,279 +1,45 @@ # Multimodal Driver State Analysis -This repository contains a full workflow for multimodal driver-state analysis in a simulator setting, from raw recording data to trained models and real-time inference. +Short overview: this repository contains the data, feature, training, and inference pipeline for multimodal driver-state analysis using facial AUs and eye-tracking signals. -It combines two modalities: -- Facial Action Units (AUs) -- Eye-tracking features (fixations, saccades, blinks, pupil dynamics) +For full documentation, see `project_report.md`. -## What This Project Covers - -- Data extraction from raw simulator files (`.h5` / ownCloud) -- Conversion to subject-level Parquet files -- Sliding-window feature engineering (AU + eye tracking) -- Exploratory data analysis (EDA) notebooks -- Model training experiments (CNN, XGBoost, Isolation Forest, OCSVM, DeepSVDD) -- Real-time prediction from SQLite + MQTT publishing -- Optional Linux `systemd` deployment (`predict.service` + `predict.timer`) - -## Repository Structure - -```text -Fahrsimulator_MSY2526_AI/ -|-- dataset_creation/ -| |-- parquet_file_creation.py -| |-- create_parquet_files_from_owncloud.py -| |-- combined_feature_creation.py -| |-- maxDist.py -| |-- AU_creation/ -| | |-- AU_creation_service.py -| | `-- pyfeat_docu.ipynb -| `-- camera_handling/ -| |-- camera_stream_AU_and_ET_new.py -| |-- eyeFeature_new.py -| |-- db_helper.py -| `-- *.py (legacy variants/tests) -|-- EDA/ -| `-- *.ipynb -|-- model_training/ -| |-- CNN/ -| |-- xgboost/ -| |-- IsolationForest/ -| |-- OCSVM/ -| |-- DeepSVDD/ -| |-- MAD_outlier_removal/ -| `-- tools/ -|-- predict_pipeline/ -| |-- predict_sample.py -| |-- config.yaml -| |-- predict.service -| |-- predict.timer -| |-- predict_service_timer_documentation.md -| `-- fill_db.ipynb -|-- tools/ -| `-- db_helpers.py -`-- readme.md -``` - -## End-to-End Workflow - -## 1) Data Ingestion and Conversion - -Main scripts: -- `dataset_creation/create_parquet_files_from_owncloud.py` -- `dataset_creation/parquet_file_creation.py` - -Purpose: -- Load simulator recordings from ownCloud or local `.h5` files. -- Select relevant columns (`STUDY`, `LEVEL`, `PHASE`, `FACE_AU*`, `EYE_*`). -- Filter invalid rows (for example `LEVEL == 0`). -- Save cleaned subject-level Parquet files. - -Notes: -- These scripts contain placeholders for paths and credentials that must be adapted. -- ownCloud download uses `pyocclient` (`owncloud` module). - -## 2) Feature Engineering (Offline Dataset) - -Main script: -- `dataset_creation/combined_feature_creation.py` - -Behavior: -- Processes all Parquet files in an input directory. -- Applies sliding windows: - - Window size: 50 seconds (`25 Hz * 50 = 1250 samples`) - - Step size: 5 seconds (`125 samples`) -- Groups data by available context columns (`STUDY`, `LEVEL`, `PHASE`). -- Computes: - - AU means per window (`FACE_AUxx_mean`) - - Eye-tracking features: - - Fixation counts and duration stats - - Saccade count/amplitude/duration stats - - Blink count/duration stats - - Pupil mean and IPA (high-frequency pupil activity) - -Output: -- A combined Parquet dataset (one row per window), ready for model training. - -## 3) Camera-Based Online Feature Extraction - -Main scripts: -- `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py` -- `dataset_creation/camera_handling/eyeFeature_new.py` - -Behavior: -- Captures webcam stream (`OpenCV`) at ~25 FPS. -- Computes eye metrics with `MediaPipe`. -- Records 50-second overlapping segments (new start every 5 seconds). -- Extracts AUs from recorded clips using `py-feat`. -- Extracts eye features from saved gaze parquet. -- Writes combined feature rows into an SQLite table (`feature_table`). - -Important: -- Script paths and DB locations are currently hardcoded for the target environment and must be adapted. - -## 4) Model Training - -Location: -- `model_training/` (mostly notebook-driven) - -Includes experiments for: -- CNN-based fusion variants -- XGBoost -- Isolation Forest -- OCSVM -- DeepSVDD - -Utility modules: -- `model_training/tools/scaler.py` for fitting/saving/applying scalers -- `model_training/tools/mad_outlier_removal.py` -- `model_training/tools/performance_split.py` -- `model_training/tools/evaluation_tools.py` - -## 5) Real-Time Prediction and Messaging - -Main script: -- `predict_pipeline/predict_sample.py` - -Runtime behavior: -- Reads latest row from SQLite (`database.path`, `database.table`, `database.key`). -- Applies NaN handling using fallback medians from `config.yaml`. -- Optionally scales features using a saved scaler (`.pkl` or `.joblib`). -- Loads model (`.keras`, `.pkl`, or `.joblib`) and predicts. -- Publishes JSON message via MQTT (topic/host/qos from config). - -Message shape: -```json -{ - "valid": true, - "_id": 123, - "prediction": 0 -} -``` -(`prediction` key is configurable via `mqtt.publish_format.result_key`.) - -## 6) Automated Execution with systemd (Linux) - -Files: -- `predict_pipeline/predict.service` -- `predict_pipeline/predict.timer` - -Current timer behavior: -- first run after 60s (`OnActiveSec=60`) -- then every 5s (`OnUnitActiveSec=5`) - -Detailed operation and commands: -- `predict_pipeline/predict_service_timer_documentation.md` - -## Installation - -Install dependencies from the tracked requirements file: +## Quickstart +### 1) Setup +Activate the conda-repository "". ```bash -pip install -r requirements.txt + conda activate ``` +Make sure, another repository that fulfills requirement.txt is available, matching with predict_pipeline/predict.service - See `predict_pipeline/predict_service_timer_documentation.md` -## Python Version -Recommended: -- Python `3.10` to `3.12` +### 2) Camera AU + Eye Pipeline (`camera_stream_AU_and_ET_new.py`) -## Core Dependencies +1. Open `dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py` and adjust: +- `DB_PATH` +- `CAMERA_INDEX` +- `OUTPUT_DIR` (optional) +2. Start camera capture and feature extraction: ```bash -pip install numpy pandas scipy scikit-learn pyarrow pyyaml joblib paho-mqtt matplotlib +python dataset_creation/camera_handling/camera_stream_AU_and_ET_new.py ``` -## Computer Vision / Eye Tracking / AU Stack +3. Stop with `q` in the camera window. -```bash -pip install opencv-python mediapipe torch moviepy -pip install pygazeanalyser -pip install py-feat -``` -## Data Access (optional) +### 3) Predict Pipeline (`predict_pipeline/predict_sample.py`) -```bash -pip install pyocclient h5py tables -``` +1. Edit `predict_pipeline/config.yaml` and set: +- `database.path`, `database.table`, `database.key` +- `model.path` +- `scaler.path` (if `use_scaling: true`) +- MQTT settings under `mqtt` -## Notes -- `tensorflow` is required for `.keras` model inference in `predict_sample.py`. -- `py-feat`, `mediapipe`, and `torch` can be platform-sensitive; pin versions per your target machine. - -## Configuration - -Primary runtime config: -- `predict_pipeline/config.yaml` - -Sections: -- `database`: SQLite path/table/key -- `model`: model file path -- `scaler`: scaling toggle + scaler path -- `mqtt`: broker connection + publish format -- `sample.columns`: expected feature order -- `fallback`: median/default feature values used for NaN replacement - -Before running prediction, verify all absolute paths in `config.yaml`. - -## Quick Start - -## A) Build Training Dataset (Offline) - -1. Set input/output paths in: -- `dataset_creation/parquet_file_creation.py` -- `dataset_creation/combined_feature_creation.py` - -2. Generate subject Parquet files: -```bash -python dataset_creation/parquet_file_creation.py -``` - -3. Generate combined sliding-window feature dataset: -```bash -python dataset_creation/combined_feature_creation.py -``` - -## B) Run Prediction Once - -1. Update paths in `predict_pipeline/config.yaml`. -2. Run: +2. Run one prediction cycle: ```bash python predict_pipeline/predict_sample.py ``` -## C) Run as systemd Service + Timer (Linux) - -1. Copy unit files to `/etc/systemd/system/`. -2. Adjust `ExecStart` and user in `predict.service`. -3. Enable and start timer: -```bash -sudo systemctl daemon-reload -sudo systemctl enable predict.timer -sudo systemctl start predict.timer -``` - -Monitor logs: -```bash -journalctl -u predict.service -f -``` - -## Database and Table Expectations - -The prediction script expects a SQLite table with at least: -- `_Id` -- `start_time` -- all model feature columns listed in `config.yaml` under `sample.columns` - -The camera pipeline writes feature rows into `feature_table` using helper utilities in: -- `dataset_creation/camera_handling/db_helper.py` -- `tools/db_helpers.py` - - - - -## License - -No license file is currently present in this repository. -Add a `LICENSE` file if this project should be shared or reused externally. +3. Use `predict_service_timer_documentation.md` to see how to use the service and timer for automation. \ No newline at end of file