{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kategorisierung der Ohrbilder\n",
    "\n",
    "Mit diesem Jupyter-Skript werden alle notwendigen Funktionen zur Umsetzung und Auswertung der Kategorisierung von Ohrbildern zusammengefasst.  \n",
    "Das Jupyter-Skript bezieht sich auf die Masterarbeit: \"Verwendung des menschlichen Ohrs zur Personenauthentifizierung an IT-Systemen mittels CNNs\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bibliotheken importieren\n",
    "\n",
    "Zunächst werden alle notwendigen Bibliothken importiert\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Import Libearies ##\n",
    "%matplotlib inline\n",
    "import cv2\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "# Import TORCHVISION with popular datasets, model architectures, \n",
    "# and common image transformations for computer vision\n",
    "from torchvision import transforms\n",
    "import torchvision\n",
    "import torchvision.utils\n",
    "import torchvision.models as models\n",
    "import torchsummary\n",
    "\n",
    "# Import Debugging method for set_trace\n",
    "from IPython.core.debugger import set_trace\n",
    "import logging\n",
    "\n",
    "# Import Time Features\n",
    "import datetime\n",
    "import time\n",
    "\n",
    "# Import TORCH for Data-Structures for multi-dimensional tensors \n",
    "# and mathematical operations\n",
    "import torch\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "import torch.optim as optim\n",
    "import torch_utils\n",
    "#import python_utils\n",
    "#import utils\n",
    "#from ignite.metrics import EpochMetric \n",
    "\n",
    "# Import Winsound to play a *.wav-File\n",
    "# and Counter for counting numbers in an array\n",
    "import winsound\n",
    "from collections import Counter\n",
    "\n",
    "# Import Random Libery\n",
    "import random\n",
    "from random import shuffle\n",
    "\n",
    "# Import IPYWIDGET for interactive HTML widgets for Jupyter notebooks\n",
    "import ipywidgets as wg\n",
    "from IPython.display import display\n",
    "\n",
    "# Import PYTORCH LIGHTNING as a PyTorch research framework\n",
    "from pytorch_lightning.metrics.classification import ROC, F1 \n",
    "from pytorch_lightning.metrics.functional.classification import multiclass_auroc\n",
    "#from pytorch_lightning.metrics import Precision, functional, Recall\n",
    "from pytorch_lightning.metrics.functional.classification import auc, auroc, multiclass_roc, f1_score\n",
    "from  pytorch_lightning.metrics.utils import to_onehot\n",
    "\n",
    "from pytorch_lightning.metrics.functional import roc\n",
    "#from pytorch_lightning.metrics.functional import precision_recall_curve\n",
    "from pytorch_lightning.metrics import Accuracy\n",
    "from pytorch_lightning.metrics.classification import AveragePrecision\n",
    "\n",
    "\n",
    "# Import ScikitLearn for AUC-Score\n",
    "#from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Festlegung: Prozessor oder Graphikkarte\n",
    "\n",
    "Falls NVIDIA-Graphikkarte und CUDA vorhanden ist, dann wird die GPU für Berechnungen genutzt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "# Read Device: Choose CUDA, if NVIDIA GPU is avaiabel\n",
    "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Voreinstellungen - Hilfsfunktionen\n",
    "\n",
    "* **secs_to_HMS():** Berechnte Sekunden in HH:MM:SS um\n",
    "* **WaitTime_Finished():** Gibt eine Audio-Datei aus, wenn Training und Testen beendet ist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Convert Seconds to Hours, Minutes and Seconds ##\n",
    "def secs_to_HMS(secs):\n",
    "    if (secs < 3600):\n",
    "        return datetime.datetime.fromtimestamp(secs).strftime('%M:%S'), \"[MM:SS]\"\n",
    "    else:\n",
    "        return datetime.datetime.fromtimestamp(secs).strftime('%H:%M:%S'), \"[HH:MM:SS]\"\n",
    "\n",
    "## Acustic Sound if Model-Trainings finished ##\n",
    "def WaitTime_Finished():\n",
    "    for i in range(2):\n",
    "        files=os.listdir(\"Sound/\")\n",
    "        file=random.choice(files)\n",
    "        winsound.PlaySound(\"Sound/\"+str(file), winsound.SND_FILENAME)\n",
    "        time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Funtionen um Datensätze festzulegen, einzulesen und anzuzeigen\n",
    "\n",
    "* **ChooseDir():**             return: Datensatz-Verzeichnis und Datensatz-String  \n",
    "* **Show_Dataset_Picture():**  Zeigt erstes Bild eines Datensatzes\n",
    "* **Create_Dataset():**        Erzeugt Datensatz-Arrays für Training und Testen\n",
    "* **Create_Dataset():**        Erzeugt erweiterte Datensatz-Arrays für Training und Testen\n",
    "\n",
    "\n",
    "Hinweis: Nur 20 Probanden des AMI-Datensatzes sind in den Datenätzen vorhanden. Mehr konnte bei Git nicht hochgeladen werden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Indirect Path for Linux and Windows## \n",
    "def ChooseDir(datadir):\n",
    "    if(datadir == 'CP'):\n",
    "        return \"Datensaetze/CP\", \"CP\"\n",
    "    if(datadir == 'AMI'):\n",
    "        return \"Datensaetze/AMI\", \"AMI\"\n",
    "    if(datadir == 'AWE'):\n",
    "        return \"Datensaetze/AWE\", \"AWE\"\n",
    "    if(datadir == 'EarVN'):\n",
    "        return \"Datensaetze/EarVN_1_0\", \"EarVN_1_0\"\n",
    "    if(datadir == 'UERC'):\n",
    "        return \"Datensaetze/UERC\", \"UERC\"\n",
    "\n",
    "## Show first Picture of choosen Dataset    \n",
    "def Show_Dataset_Picture(datadir, categories):\n",
    "    for category in categories:\n",
    "        path = os.path.join(datadir, category)\n",
    "        for img in os.listdir(path):\n",
    "            img_array = cv2.imread(os.path.join(path, img))\n",
    "            plt.imshow(cv2.cvtColor(img_array, cv2.COLOR_BGR2RGB))\n",
    "            plt.show()\n",
    "            break\n",
    "        break   \n",
    "        \n",
    "\n",
    "## Create normal Dataset \n",
    "def Create_Dataset(datadir, categories):\n",
    "    ## Transformer for Images ##\n",
    "    ## For PyTorch Models use: mean = [0.4318, 0.4660, 0.5889], std = [0.1752, 0.1893, 0.2096]\n",
    "    transformer = transforms.Compose(\n",
    "        [transforms.ToTensor(),\n",
    "         transforms.Normalize((0.4318, 0.4660, 0.5889), (0.1752, 0.1893, 0.2096))\n",
    "        ])\n",
    "\n",
    "    ## Define Size of Images; \n",
    "    ## For PyTorch Models use: IMG_Size = 224 \n",
    "    IMG_SIZE = 224\n",
    "\n",
    "    ## Create normal dataset for all Databases and Categories \n",
    "    for category in categories:\n",
    "        ## Get path of each Category\n",
    "        path = os.path.join(datadir, category)\n",
    "        \n",
    "        ## Create class numbers for each Category\n",
    "        class_num = categories.index(category)\n",
    "        \n",
    "        ## Calculate train_share for Train- and Testdataset\n",
    "        count_train_share = (len(os.listdir(path)))*train_share\n",
    "        counter = 1\n",
    "        \n",
    "        ## Get all Images of current Category\n",
    "        os_listdir = os.listdir(path)\n",
    "        ## Sort the Pictures-Path of each category randomly\n",
    "        random.shuffle(os_listdir)\n",
    "\n",
    "        for img in os_listdir:\n",
    "            try:\n",
    "                ## Read, Resize, and Transform image\n",
    "                img_array = cv2.imread(os.path.join(path, img))\n",
    "                img_array = cv2.resize(img_array, (IMG_SIZE, IMG_SIZE))\n",
    "                img_array = transformer(img_array)\n",
    "                \n",
    "                ## Split Images to Train- and Test-Dataset-Array\n",
    "                if(counter <= count_train_share):\n",
    "                    dataset_train.append([img_array, class_num])\n",
    "                    counter += 1\n",
    "                else:\n",
    "                    dataset_test.append([img_array, class_num])\n",
    "            except Exception as e:\n",
    "                 pass\n",
    "    \n",
    "    ## Rondomly sortig dataset_train and dataset_test Arrays\n",
    "    random.shuffle(dataset_train)\n",
    "    random.shuffle(dataset_test)\n",
    "\n",
    "\n",
    "\n",
    "## Create expand Dataset    \n",
    "def Create_Dataset_Expand(datadir, categories):\n",
    "    ## Transformer for Images: ToTensor() ad Normalize \n",
    "    ## For PyTorch Models: to Normalize images use: mean = [0.4318, 0.4660, 0.5889], std = [0.1752, 0.1893, 0.2096]\n",
    "    transformer = transforms.Compose(\n",
    "        [transforms.ToTensor(),\n",
    "         transforms.Normalize((0.4318, 0.4660, 0.5889), (0.1752, 0.1893, 0.2096)),\n",
    "        ])\n",
    "\n",
    "    ## Define Size of Images; \n",
    "    ## For PyTorch Models use: IMG_Size = 224 \n",
    "    IMG_SIZE = 224\n",
    "\n",
    "    ## Create expand  dataset for all Databases (except for: EarVN1.0 Database)\n",
    "    for i in range(4):\n",
    "        for category in categories:\n",
    "            ## Get path of each Category\n",
    "            path = os.path.join(datadir, category)\n",
    "            \n",
    "            ## Create class numbers for each Category\n",
    "            class_num = categories.index(category)\n",
    "            \n",
    "            ## Calculate train_share for Train- and Testdataset\n",
    "            count_train_share = round((len(os.listdir(path)))*train_share)\n",
    "            counter = 1\n",
    "            \n",
    "            ## Get all Images of current Category\n",
    "            os_listdir = os.listdir(path)\n",
    "            \n",
    "            ## Sort the Pictures-Path of each category randomly\n",
    "            random.shuffle(os_listdir)\n",
    "            \n",
    "            for img in os_listdir:\n",
    "                    ## Read, Resize, and Transform image\n",
    "                    img_array = cv2.imread(os.path.join(path, img))\n",
    "                    img_array = cv2.resize(img_array, (IMG_SIZE, IMG_SIZE))\n",
    "                     \n",
    "                    ## Rotate Image\n",
    "                    rows,cols,ch = img_array.shape\n",
    "                    rotate_numbers = [0, 90, 180, 270]\n",
    "                    rotate_number = rotate_numbers[i]\n",
    "                    M = cv2.getRotationMatrix2D(((cols-1)/2.0,(rows-1)/2.0),rotate_number,1) ## cols-1 and rows-1 are the coordinate limits.\n",
    "                    img_array = cv2.warpAffine(img_array,M,(cols,rows))\n",
    "                    \n",
    "                    ## Transform the img-Array\n",
    "                    img_array = transformer(img_array)\n",
    "                    \n",
    "                    ## Split Images to Train- and Test-Dataset-Array\n",
    "                    if(counter <= count_train_share):\n",
    "                        dataset_train.append([img_array, class_num])\n",
    "                        counter += 1\n",
    "                    else:\n",
    "                        dataset_test.append([img_array, class_num])\n",
    "    \n",
    "    ## Randomly sortig dataset_train and dataset_test Arrays\n",
    "    random.shuffle(dataset_train)\n",
    "    random.shuffle(dataset_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Funktionen um vortrainiertes Netzwerk festzulegen\n",
    "\n",
    "* **Network_Choice():** Gibt Netzwerk und Netzwerk-String zurück"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "## All pretrained Networks from PyTorch ##\n",
    "\n",
    "network_name = 'nix'\n",
    "\n",
    "# Function to choose network\n",
    "# return: pretrained Network, Name of Network\n",
    "def Network_Choice(netw):    \n",
    "    if(netw == 'vgg11'):\n",
    "        return models.vgg11(pretrained=True).to(DEVICE), \"VGG11\"\n",
    "    if(netw == 'vgg11bn'):\n",
    "        return models.vgg11_bn(pretrained=True).to(DEVICE), \"VGG11bn\"\n",
    "    if(netw == 'resnet18'):\n",
    "        return models.resnet18(pretrained=True).to(DEVICE), \"ResNet18\"\n",
    "    if(netw == 'resnet34'):\n",
    "        return models.resnet34(pretrained=True).to(DEVICE), \"ResNet34\"\n",
    "    if(netw == 'alexnet'):\n",
    "        return models.alexnet(pretrained=True).to(DEVICE), \"AlexNet\"\n",
    "    if(netw == 'squeezenet1_0'):\n",
    "        return models.squeezenet1_0(pretrained=True).to(DEVICE), \"SqueezeNet-1-0\"  \n",
    "    if(netw == 'GoogLeNet'):\n",
    "        return models.googlenet(pretrained=True).to(DEVICE), \"GoogLeNet\" \n",
    "    if(netw == 'shufflenet_v2_x0_5'):\n",
    "        return models.shufflenet_v2_x0_5(pretrained=True).to(DEVICE), \"Shufflenet-v2-x0_5\"  \n",
    "    if(netw == 'resnext101_32x8d'):\n",
    "        return models.resnext101_32x8d(pretrained=True).to(DEVICE), \"Resnext101-32x8d\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Funktionen für die Ausgabe von Ohrbildern und dazugehöriger Labels\n",
    "\n",
    "* **imshow():** Zeigt Bild\n",
    "* **matplotlib_imshow():** Ordnet 4 Ohrbilder mit dazugehörigem Label an"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions to show an image\n",
    "\n",
    "# Inverse the transformation of the images\n",
    "invTrans = transforms.Compose([ transforms.Normalize(mean = [ 0., 0., 0. ],\n",
    "                                                     std = [ 1/0.1752, 1/0.1893, 1/0.2096 ]),\n",
    "                                transforms.Normalize(mean = [ -0.4318, -0.4660, -0.5889 ],\n",
    "                                                     std = [ 1., 1., 1. ]),\n",
    "                               ])\n",
    "\n",
    "def imshow(img):\n",
    "    img = invTrans(img)\n",
    "    npimg = img.numpy()\n",
    "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
    "    plt.show()\n",
    "    \n",
    "# helper function to show an image\n",
    "# (used in the `plot_classes_preds` function below)\n",
    "def matplotlib_imshow(img, one_channel=False):\n",
    "    #set_trace()\n",
    "    img = invTrans(img)      # unnormalize\n",
    "    if one_channel:\n",
    "        img = img.mean(dim=0)\n",
    "    npimg = img.numpy()\n",
    "    #set_trace()\n",
    "    if one_channel:\n",
    "        plt.imshow(npimg)        \n",
    "    else:\n",
    "        plt.imshow(np.transpose(npimg, (1, 2, 0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Klasse zur Berechnung von Top-1 und Top-5 Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Class of Top-1 and Top-5 Accuracy\n",
    "class AvgrageMeter(object):\n",
    "\n",
    "    def __init__(self):\n",
    "        self.reset()\n",
    "\n",
    "    def reset(self):\n",
    "        self.avg = 0\n",
    "        self.sum = 0\n",
    "        self.cnt = 0\n",
    "\n",
    "    def update(self, val, n=1):\n",
    "        self.sum += val * n\n",
    "        self.cnt += n\n",
    "        self.avg = self.sum / self.cnt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy jeder Kategorie in Log-File (TXT-File)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Write Accuracy of every Categorie to Log-Files (TXT-File) ##\n",
    "def Write_Each_Categorie_Acc_To_Editor(database, net, acc_per_categorie):\n",
    "    logfile_path = './Log_Files/' + database + '/' + str(current_time) + '_' + net + '.txt'\n",
    "    datei = open(logfile_path, 'a')\n",
    "    datei.write(acc_per_categorie)\n",
    "    datei.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Konfiguration\n",
    "\n",
    "Auswahl von: \n",
    "* Datensatz\n",
    "* Netzwerk\n",
    "* Normaler/Erweiterter Datensatz\n",
    "* Trainshare\n",
    "* Batch Size Train\n",
    "* Batch Size Test\n",
    "* Learning Rate\n",
    "* Momentum\n",
    "\n",
    "Hinweis: Konfiguration ausführen, Auswahl festlegen und weiter zum nächsten Abschnitt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "03acf4dac0f945c6b10f19c9efda3ed6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Dataset:', index=1, options=('CP', 'AMI', 'AWE', 'EarVN', 'UERC'), value='AMI')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4c28430bc0714174bf0b1de6587f8f3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Network:', options=('vgg11', 'vgg11bn', 'resnet18', 'resnet34', 'alexnet', 'squeezenet1_…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "33ae1c0ce7ba43948f1a5160cc709d64",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Normal/Exp:', options=('Normal_Network', 'Expand_Network'), value='Normal_Network')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d71b8c9fbc4f42b9849f82d148e22656",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedFloatText(value=0.8, description='Trainshare:', max=1.0, step=0.1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6792a002f8ff43d794f7833ecbb0eddf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedIntText(value=4, description='Batch_Train:', min=1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99ed5d89518c49acb526ea652ffbf8e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedIntText(value=4, description='Batch_Test:', min=1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72cfe0750b3f4adcafad0cd48d4ad42d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedFloatText(value=0.001, description='Learning_Rate', max=1.0, step=0.0001)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16ffa6aca9b84fb4852949a9b1ba39ad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedFloatText(value=0.8, description='Momentum: ', max=1.0, step=0.1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Choose Dataset\n",
    "datadir_choose = wg.Dropdown(\n",
    "     options=['CP', 'AMI', 'AWE', 'EarVN', 'UERC'],\n",
    "     value='AMI',\n",
    "     description='Dataset:',\n",
    "     disabled=False,\n",
    "     button_style=''\n",
    ")\n",
    "\n",
    "## Choose Network\n",
    "network_string_choose = wg.Dropdown(\n",
    "     options=['vgg11','vgg11bn','resnet18','resnet34','alexnet','squeezenet1_0','GoogLeNet','shufflenet_v2_x0_5','resnext101_32x8d', 'siamese'],\n",
    "     value='vgg11',\n",
    "     description='Network:',\n",
    "     disabled=False,\n",
    "     button_style=''\n",
    ")\n",
    "\n",
    "## Choose Normal or expand network\n",
    "normal_expand_string_choose = wg.Dropdown(\n",
    "     options=['Normal_Network', 'Expand_Network'],\n",
    "     value='Normal_Network',\n",
    "     description='Normal/Exp:',\n",
    "     disabled=False,\n",
    "     button_style=''\n",
    ")\n",
    "\n",
    "## Choose Trainshare\n",
    "trainshare_choose = wg.BoundedFloatText(\n",
    "     value=0.8,\n",
    "     min=0,\n",
    "     max=1,\n",
    "     step=0.1,\n",
    "     description='Trainshare:',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "## Choose Batch_Size for Training\n",
    "batch_train = wg.BoundedIntText(\n",
    "     value=4,\n",
    "     min=1,\n",
    "     max=100,\n",
    "     step=1,\n",
    "     description='Batch_Train:',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "## Choose Batch_Size for Testing\n",
    "batch_test = wg.BoundedIntText(\n",
    "     value=4,\n",
    "     min=1,\n",
    "     max=100,\n",
    "     step=1,\n",
    "     description='Batch_Test:',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "## Choose Learning-Rate\n",
    "learning_rate_choose = wg.BoundedFloatText(\n",
    "     value=0.001,\n",
    "     min=0,\n",
    "     max=1,\n",
    "     step=0.0001,\n",
    "     description='Learning_Rate',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "## Choose \n",
    "momentum_choose = wg.BoundedFloatText(\n",
    "     value=0.8,\n",
    "     min=0,\n",
    "     max=1,\n",
    "     step=0.1,\n",
    "     description='Momentum: ',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "\n",
    "display(datadir_choose, network_string_choose, normal_expand_string_choose, trainshare_choose, batch_train, batch_test, learning_rate_choose, momentum_choose)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datensatz generieren\n",
    "\n",
    "Mit den bereits beschriebenen Funktionen werden alle Einstellungen ausgelesen und ein Datensatz, sowie die Kategorien eingelesen\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Arrays for Dataset and Categories ##\n",
    "dataset_train = []\n",
    "dataset_test = []\n",
    "CATEGORIES = []\n",
    "img_array = []\n",
    "\n",
    "## Read all choosen \n",
    "datadir = datadir_choose.value\n",
    "network_string = network_string_choose.value\n",
    "normal_expand = normal_expand_string_choose.value\n",
    "train_share = trainshare_choose.value\n",
    "batch_size_train = batch_train.value\n",
    "batch_size_test = batch_test.value\n",
    "learning_rate = learning_rate_choose.value\n",
    "momentum = momentum_choose.value\n",
    "\n",
    "\n",
    "## Choose Datadirectory to the Dataset ##\n",
    "DATADIR, Database = ChooseDir(datadir)\n",
    "\n",
    "## Read and list all Categories of a Dataset \n",
    "CATEGORIES = os.listdir(DATADIR)\n",
    "\n",
    "## Get Length of Categories for chosen Dataset \n",
    "len_classes = len(CATEGORIES)\n",
    "\n",
    "## Create Dataset for Training and Testing\n",
    "if(normal_expand == 'Normal_Network'):\n",
    "    Create_Dataset(DATADIR, CATEGORIES)\n",
    "elif(normal_expand == 'Expand_Network'):\n",
    "    Create_Dataset_Expand(DATADIR, CATEGORIES)\n",
    "else:\n",
    "    print('False: Choosen Normal/Expand')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Einstellungen Anzeigen\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 97404 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datensatz: AMI\n",
      "\n",
      "Anzahl der Bilder insgesamt:\t 140\n",
      "Anzahl der Trainingsdatensätze:\t 100\n",
      "Anzahl der Testdatensätze:\t 40\n",
      "\n",
      "Kategorien: ['000', '001', '002', '003', '004', '005', '007', '008', '009', '010', '011', '012', '013', '014', '018', '019', '020', '026', '027', '028']\n",
      "20\n"
     ]
    }
   ],
   "source": [
    "## Show first picture of the choosen Dataset\n",
    "Show_Dataset_Picture(DATADIR, CATEGORIES)\n",
    "\n",
    "print(f'Datensatz: {Database}')\n",
    "print(f'\\nAnzahl der Bilder insgesamt:\\t {len(dataset_train)+len(dataset_test)}')\n",
    "print(f\"Anzahl der Trainingsdatensätze:\\t {len(dataset_train)}\")\n",
    "print(f\"Anzahl der Testdatensätze:\\t {len(dataset_test)}\")\n",
    "print(f'\\nKategorien: {CATEGORIES}')\n",
    "print(f'{len(CATEGORIES)}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train- und Testloader festlegen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create Train-DataLoader\n",
    "train_loader = DataLoader(dataset_train, batch_size=batch_size_train, shuffle=True)\n",
    "\n",
    "## Create Test-DataLoader\n",
    "test_loader = DataLoader(dataset_test, batch_size=batch_size_test, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Netzwerk festlegen\n",
    "\n",
    "Legt Netzwerk fest und zeigt die Übersicht der Netzwerk-Struktur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VGG11\n",
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "            Conv2d-1         [-1, 64, 224, 224]           1,792\n",
      "              ReLU-2         [-1, 64, 224, 224]               0\n",
      "         MaxPool2d-3         [-1, 64, 112, 112]               0\n",
      "            Conv2d-4        [-1, 128, 112, 112]          73,856\n",
      "              ReLU-5        [-1, 128, 112, 112]               0\n",
      "         MaxPool2d-6          [-1, 128, 56, 56]               0\n",
      "            Conv2d-7          [-1, 256, 56, 56]         295,168\n",
      "              ReLU-8          [-1, 256, 56, 56]               0\n",
      "            Conv2d-9          [-1, 256, 56, 56]         590,080\n",
      "             ReLU-10          [-1, 256, 56, 56]               0\n",
      "        MaxPool2d-11          [-1, 256, 28, 28]               0\n",
      "           Conv2d-12          [-1, 512, 28, 28]       1,180,160\n",
      "             ReLU-13          [-1, 512, 28, 28]               0\n",
      "           Conv2d-14          [-1, 512, 28, 28]       2,359,808\n",
      "             ReLU-15          [-1, 512, 28, 28]               0\n",
      "        MaxPool2d-16          [-1, 512, 14, 14]               0\n",
      "           Conv2d-17          [-1, 512, 14, 14]       2,359,808\n",
      "             ReLU-18          [-1, 512, 14, 14]               0\n",
      "           Conv2d-19          [-1, 512, 14, 14]       2,359,808\n",
      "             ReLU-20          [-1, 512, 14, 14]               0\n",
      "        MaxPool2d-21            [-1, 512, 7, 7]               0\n",
      "AdaptiveAvgPool2d-22            [-1, 512, 7, 7]               0\n",
      "           Linear-23                 [-1, 4096]     102,764,544\n",
      "             ReLU-24                 [-1, 4096]               0\n",
      "          Dropout-25                 [-1, 4096]               0\n",
      "           Linear-26                 [-1, 4096]      16,781,312\n",
      "             ReLU-27                 [-1, 4096]               0\n",
      "          Dropout-28                 [-1, 4096]               0\n",
      "           Linear-29                 [-1, 1000]       4,097,000\n",
      "================================================================\n",
      "Total params: 132,863,336\n",
      "Trainable params: 132,863,336\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.57\n",
      "Forward/backward pass size (MB): 125.37\n",
      "Params size (MB): 506.83\n",
      "Estimated Total Size (MB): 632.78\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "## Get choosen Network and Network-Name\n",
    "network, network_name = Network_Choice(network_string)\n",
    "print(network_name)\n",
    "\n",
    "## Show Network-Archticture with all Layers and Number of Parameters\n",
    "torchsummary.summary(network, (3, 224, 224), device=DEVICE.type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Festlegung des eindeutigen Stempels\n",
    "\n",
    "Eindeutiger Stempel mit Uhrzeit, um CSV-, TXT- und Log-Files wiederzuerkennen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021_02_27-21_22_49\n"
     ]
    }
   ],
   "source": [
    "## Use actual Time for individual Stamp ##\n",
    "current_time = datetime.datetime.now().strftime(\"%Y_%m_%d-%H_%M_%S\")\n",
    "print(current_time)\n",
    "\n",
    "## Unique Stamp for the identification of networks, graphs, train- and Testdata\n",
    "stamp = current_time + '_' + Database + '_' + network_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Festlegung 'writer' für Tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "writer = SummaryWriter('runs/' + current_time + '_' + Database + '_' + network_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Festlegung von Optimizer und Loss-Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Choose SGD (Stochastic Gradient Descent) as optimizer\n",
    "optimizer = optim.SGD(network.parameters(), lr=learning_rate, momentum=momentum)\n",
    "## Choose CrossEntropyLoss as Loss\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Anzeige von 4 Ohren des gewählten Datensatzes mit jeweiligem Label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 86852 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  014   009   011   004\n"
     ]
    }
   ],
   "source": [
    "## Get random images from Trainloader\n",
    "dataiter = iter(train_loader)\n",
    "images, labels = dataiter.next()\n",
    "\n",
    "## show images\n",
    "imshow(torchvision.utils.make_grid(images))\n",
    "## print labels\n",
    "print(' '.join('%5s' % CATEGORIES[labels[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Durchführung des Trainings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start Training um:  21:22:52\n",
      "\n",
      "Epoch 1:\n",
      "[1, 2]: loss: 9.865, \t Running_Correct: 0.000\n",
      "[1, 4]: loss: 7.968, \t Running_Correct: 0.000\n",
      "[1, 6]: loss: 5.690, \t Running_Correct: 0.000\n",
      "[1, 8]: loss: 5.712, \t Running_Correct: 0.000\n",
      "[1, 10]: loss: 5.761, \t Running_Correct: 0.000\n",
      "[1, 12]: loss: 4.737, \t Running_Correct: 0.000\n",
      "[1, 14]: loss: 3.476, \t Running_Correct: 0.500\n",
      "[1, 16]: loss: 5.120, \t Running_Correct: 0.000\n",
      "[1, 18]: loss: 5.714, \t Running_Correct: 0.500\n",
      "[1, 20]: loss: 4.876, \t Running_Correct: 0.000\n",
      "[1, 22]: loss: 3.751, \t Running_Correct: 0.000\n",
      "[1, 24]: loss: 4.292, \t Running_Correct: 1.500\n",
      "\n",
      "Epoch 2:\n",
      "[2, 2]: loss: 4.078, \t Running_Correct: 0.500\n",
      "[2, 4]: loss: 2.458, \t Running_Correct: 1.500\n",
      "[2, 6]: loss: 2.992, \t Running_Correct: 1.500\n",
      "[2, 8]: loss: 2.615, \t Running_Correct: 0.500\n",
      "[2, 10]: loss: 3.510, \t Running_Correct: 1.500\n",
      "[2, 12]: loss: 4.087, \t Running_Correct: 0.000\n",
      "[2, 14]: loss: 2.451, \t Running_Correct: 1.500\n",
      "[2, 16]: loss: 3.023, \t Running_Correct: 0.500\n",
      "[2, 18]: loss: 2.056, \t Running_Correct: 2.000\n",
      "[2, 20]: loss: 3.019, \t Running_Correct: 0.000\n",
      "[2, 22]: loss: 2.386, \t Running_Correct: 1.000\n",
      "[2, 24]: loss: 2.218, \t Running_Correct: 1.000\n",
      "\n",
      "Epoch 3:\n",
      "[3, 2]: loss: 2.589, \t Running_Correct: 1.000\n",
      "[3, 4]: loss: 2.130, \t Running_Correct: 1.500\n",
      "[3, 6]: loss: 1.970, \t Running_Correct: 1.500\n",
      "[3, 8]: loss: 1.701, \t Running_Correct: 1.000\n",
      "[3, 10]: loss: 2.226, \t Running_Correct: 1.500\n",
      "[3, 12]: loss: 2.392, \t Running_Correct: 1.500\n",
      "[3, 14]: loss: 1.832, \t Running_Correct: 1.000\n",
      "[3, 16]: loss: 1.451, \t Running_Correct: 2.000\n",
      "[3, 18]: loss: 1.218, \t Running_Correct: 1.500\n",
      "[3, 20]: loss: 1.221, \t Running_Correct: 2.500\n",
      "[3, 22]: loss: 1.429, \t Running_Correct: 2.500\n",
      "[3, 24]: loss: 1.765, \t Running_Correct: 1.500\n",
      "\n",
      "Epoch 4:\n",
      "[4, 2]: loss: 0.289, \t Running_Correct: 4.000\n",
      "[4, 4]: loss: 0.272, \t Running_Correct: 3.500\n",
      "[4, 6]: loss: 0.624, \t Running_Correct: 3.000\n",
      "[4, 8]: loss: 0.885, \t Running_Correct: 3.000\n",
      "[4, 10]: loss: 0.775, \t Running_Correct: 2.500\n",
      "[4, 12]: loss: 1.037, \t Running_Correct: 2.500\n",
      "[4, 14]: loss: 1.357, \t Running_Correct: 2.500\n",
      "[4, 16]: loss: 0.908, \t Running_Correct: 2.500\n",
      "[4, 18]: loss: 0.882, \t Running_Correct: 3.000\n",
      "[4, 20]: loss: 0.604, \t Running_Correct: 3.500\n",
      "[4, 22]: loss: 0.439, \t Running_Correct: 3.500\n",
      "[4, 24]: loss: 0.416, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 5:\n",
      "[5, 2]: loss: 0.538, \t Running_Correct: 3.500\n",
      "[5, 4]: loss: 0.313, \t Running_Correct: 3.500\n",
      "[5, 6]: loss: 0.286, \t Running_Correct: 4.000\n",
      "[5, 8]: loss: 0.367, \t Running_Correct: 3.500\n",
      "[5, 10]: loss: 0.350, \t Running_Correct: 3.500\n",
      "[5, 12]: loss: 0.999, \t Running_Correct: 2.500\n",
      "[5, 14]: loss: 0.664, \t Running_Correct: 3.000\n",
      "[5, 16]: loss: 0.100, \t Running_Correct: 4.000\n",
      "[5, 18]: loss: 0.754, \t Running_Correct: 3.000\n",
      "[5, 20]: loss: 0.759, \t Running_Correct: 3.000\n",
      "[5, 22]: loss: 0.711, \t Running_Correct: 2.000\n",
      "[5, 24]: loss: 0.471, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 6:\n",
      "[6, 2]: loss: 0.281, \t Running_Correct: 4.000\n",
      "[6, 4]: loss: 0.529, \t Running_Correct: 3.500\n",
      "[6, 6]: loss: 0.416, \t Running_Correct: 3.500\n",
      "[6, 8]: loss: 0.218, \t Running_Correct: 3.500\n",
      "[6, 10]: loss: 0.917, \t Running_Correct: 3.500\n",
      "[6, 12]: loss: 1.952, \t Running_Correct: 2.500\n",
      "[6, 14]: loss: 0.185, \t Running_Correct: 4.000\n",
      "[6, 16]: loss: 0.771, \t Running_Correct: 3.000\n",
      "[6, 18]: loss: 0.558, \t Running_Correct: 3.500\n",
      "[6, 20]: loss: 0.315, \t Running_Correct: 3.500\n",
      "[6, 22]: loss: 0.167, \t Running_Correct: 4.000\n",
      "[6, 24]: loss: 0.531, \t Running_Correct: 3.500\n",
      "\n",
      "Epoch 7:\n",
      "[7, 2]: loss: 0.215, \t Running_Correct: 4.000\n",
      "[7, 4]: loss: 0.131, \t Running_Correct: 4.000\n",
      "[7, 6]: loss: 0.250, \t Running_Correct: 3.500\n",
      "[7, 8]: loss: 0.140, \t Running_Correct: 4.000\n",
      "[7, 10]: loss: 0.141, \t Running_Correct: 4.000\n",
      "[7, 12]: loss: 0.274, \t Running_Correct: 3.500\n",
      "[7, 14]: loss: 0.018, \t Running_Correct: 4.000\n",
      "[7, 16]: loss: 0.213, \t Running_Correct: 3.500\n",
      "[7, 18]: loss: 0.039, \t Running_Correct: 4.000\n",
      "[7, 20]: loss: 0.145, \t Running_Correct: 4.000\n",
      "[7, 22]: loss: 0.024, \t Running_Correct: 4.000\n",
      "[7, 24]: loss: 0.013, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 8:\n",
      "[8, 2]: loss: 0.029, \t Running_Correct: 4.000\n",
      "[8, 4]: loss: 0.537, \t Running_Correct: 3.500\n",
      "[8, 6]: loss: 0.206, \t Running_Correct: 3.500\n",
      "[8, 8]: loss: 0.108, \t Running_Correct: 4.000\n",
      "[8, 10]: loss: 0.076, \t Running_Correct: 4.000\n",
      "[8, 12]: loss: 0.019, \t Running_Correct: 4.000\n",
      "[8, 14]: loss: 0.848, \t Running_Correct: 3.000\n",
      "[8, 16]: loss: 1.830, \t Running_Correct: 2.500\n",
      "[8, 18]: loss: 1.012, \t Running_Correct: 3.500\n",
      "[8, 20]: loss: 0.529, \t Running_Correct: 3.500\n",
      "[8, 22]: loss: 1.194, \t Running_Correct: 2.500\n",
      "[8, 24]: loss: 0.835, \t Running_Correct: 3.500\n",
      "\n",
      "Epoch 9:\n",
      "[9, 2]: loss: 0.199, \t Running_Correct: 4.000\n",
      "[9, 4]: loss: 0.114, \t Running_Correct: 4.000\n",
      "[9, 6]: loss: 0.208, \t Running_Correct: 3.500\n",
      "[9, 8]: loss: 0.101, \t Running_Correct: 4.000\n",
      "[9, 10]: loss: 0.197, \t Running_Correct: 3.500\n",
      "[9, 12]: loss: 0.178, \t Running_Correct: 4.000\n",
      "[9, 14]: loss: 0.021, \t Running_Correct: 4.000\n",
      "[9, 16]: loss: 0.116, \t Running_Correct: 4.000\n",
      "[9, 18]: loss: 0.101, \t Running_Correct: 4.000\n",
      "[9, 20]: loss: 0.032, \t Running_Correct: 4.000\n",
      "[9, 22]: loss: 0.170, \t Running_Correct: 3.500\n",
      "[9, 24]: loss: 0.245, \t Running_Correct: 3.500\n",
      "\n",
      "Epoch 10:\n",
      "[10, 2]: loss: 0.029, \t Running_Correct: 4.000\n",
      "[10, 4]: loss: 0.011, \t Running_Correct: 4.000\n",
      "[10, 6]: loss: 0.048, \t Running_Correct: 4.000\n",
      "[10, 8]: loss: 0.010, \t Running_Correct: 4.000\n",
      "[10, 10]: loss: 0.018, \t Running_Correct: 4.000\n",
      "[10, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[10, 14]: loss: 0.042, \t Running_Correct: 4.000\n",
      "[10, 16]: loss: 0.033, \t Running_Correct: 4.000\n",
      "[10, 18]: loss: 0.009, \t Running_Correct: 4.000\n",
      "[10, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[10, 22]: loss: 0.009, \t Running_Correct: 4.000\n",
      "[10, 24]: loss: 0.002, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 11:\n",
      "[11, 2]: loss: 0.007, \t Running_Correct: 4.000\n",
      "[11, 4]: loss: 0.019, \t Running_Correct: 4.000\n",
      "[11, 6]: loss: 0.004, \t Running_Correct: 4.000\n",
      "[11, 8]: loss: 0.011, \t Running_Correct: 4.000\n",
      "[11, 10]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[11, 12]: loss: 0.004, \t Running_Correct: 4.000\n",
      "[11, 14]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[11, 16]: loss: 0.006, \t Running_Correct: 4.000\n",
      "[11, 18]: loss: 0.032, \t Running_Correct: 4.000\n",
      "[11, 20]: loss: 0.055, \t Running_Correct: 4.000\n",
      "[11, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[11, 24]: loss: 0.016, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 12:\n",
      "[12, 2]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[12, 4]: loss: 0.041, \t Running_Correct: 4.000\n",
      "[12, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[12, 8]: loss: 0.008, \t Running_Correct: 4.000\n",
      "[12, 10]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[12, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[12, 14]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[12, 16]: loss: 0.004, \t Running_Correct: 4.000\n",
      "[12, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[12, 20]: loss: 0.005, \t Running_Correct: 4.000\n",
      "[12, 22]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[12, 24]: loss: 0.002, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 13:\n",
      "[13, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[13, 4]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[13, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[13, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[13, 10]: loss: 0.010, \t Running_Correct: 4.000\n",
      "[13, 12]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[13, 14]: loss: 0.006, \t Running_Correct: 4.000\n",
      "[13, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[13, 18]: loss: 0.007, \t Running_Correct: 4.000\n",
      "[13, 20]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[13, 22]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[13, 24]: loss: 0.008, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 14:\n",
      "[14, 2]: loss: 0.008, \t Running_Correct: 4.000\n",
      "[14, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[14, 6]: loss: 0.009, \t Running_Correct: 4.000\n",
      "[14, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[14, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[14, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[14, 14]: loss: 0.013, \t Running_Correct: 4.000\n",
      "[14, 16]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[14, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[14, 20]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[14, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[14, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 15:\n",
      "[15, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[15, 4]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[15, 6]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[15, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[15, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[15, 12]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[15, 14]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[15, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[15, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[15, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[15, 22]: loss: 0.004, \t Running_Correct: 4.000\n",
      "[15, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 16:\n",
      "[16, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[16, 4]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[16, 6]: loss: 0.004, \t Running_Correct: 4.000\n",
      "[16, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[16, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[16, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[16, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[16, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[16, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[16, 20]: loss: 0.005, \t Running_Correct: 4.000\n",
      "[16, 22]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[16, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 17:\n",
      "[17, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[17, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[17, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[17, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[17, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[17, 12]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[17, 14]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[17, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[17, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[17, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[17, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[17, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 18:\n",
      "[18, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[18, 4]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[18, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[18, 8]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[18, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[18, 12]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[18, 14]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[18, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[18, 18]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[18, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[18, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[18, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 19:\n",
      "[19, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[19, 4]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[19, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[19, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[19, 10]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[19, 12]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[19, 14]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[19, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[19, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[19, 20]: loss: 0.004, \t Running_Correct: 4.000\n",
      "[19, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[19, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 20:\n",
      "[20, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[20, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[20, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[20, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[20, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[20, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[20, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[20, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[20, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[20, 20]: loss: 0.006, \t Running_Correct: 4.000\n",
      "[20, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[20, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 21:\n",
      "[21, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[21, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[21, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[21, 8]: loss: 0.006, \t Running_Correct: 4.000\n",
      "[21, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[21, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[21, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[21, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[21, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[21, 20]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[21, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[21, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 22:\n",
      "[22, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[22, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[22, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[22, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[22, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[22, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[22, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[22, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[22, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[22, 20]: loss: 0.007, \t Running_Correct: 4.000\n",
      "[22, 22]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[22, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 23:\n",
      "[23, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[23, 4]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[23, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[23, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[23, 10]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[23, 12]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[23, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[23, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[23, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[23, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[23, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[23, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 24:\n",
      "[24, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[24, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[24, 6]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[24, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[24, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[24, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[24, 14]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[24, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[24, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[24, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[24, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[24, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 25:\n",
      "[25, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[25, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[25, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[25, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[25, 10]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[25, 12]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[25, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[25, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[25, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[25, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[25, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[25, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 26:\n",
      "[26, 2]: loss: 0.006, \t Running_Correct: 4.000\n",
      "[26, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[26, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[26, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[26, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[26, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[26, 14]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[26, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[26, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[26, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[26, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[26, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 27:\n",
      "[27, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[27, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[27, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[27, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[27, 10]: loss: 0.005, \t Running_Correct: 4.000\n",
      "[27, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[27, 14]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[27, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[27, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[27, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[27, 22]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[27, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 28:\n",
      "[28, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[28, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[28, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[28, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[28, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[28, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[28, 14]: loss: 0.005, \t Running_Correct: 4.000\n",
      "[28, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[28, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[28, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[28, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[28, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 29:\n",
      "[29, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[29, 4]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[29, 6]: loss: 0.004, \t Running_Correct: 4.000\n",
      "[29, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[29, 10]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[29, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[29, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[29, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[29, 18]: loss: 0.006, \t Running_Correct: 4.000\n",
      "[29, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[29, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[29, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 30:\n",
      "[30, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[30, 4]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[30, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[30, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[30, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[30, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[30, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[30, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[30, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[30, 20]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[30, 22]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[30, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 31:\n",
      "[31, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[31, 4]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[31, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[31, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[31, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[31, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[31, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[31, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[31, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[31, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[31, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[31, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 32:\n",
      "[32, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[32, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[32, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[32, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[32, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[32, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[32, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[32, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[32, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[32, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[32, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[32, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 33:\n",
      "[33, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[33, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[33, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[33, 8]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[33, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[33, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[33, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[33, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[33, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[33, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[33, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[33, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 34:\n",
      "[34, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[34, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 35:\n",
      "[35, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[35, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[35, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[35, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[35, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[35, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[35, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[35, 16]: loss: 0.005, \t Running_Correct: 4.000\n",
      "[35, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[35, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[35, 22]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[35, 24]: loss: 0.019, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 36:\n",
      "[36, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[36, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[36, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[36, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[36, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[36, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[36, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[36, 16]: loss: 0.003, \t Running_Correct: 4.000\n",
      "[36, 18]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[36, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[36, 22]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[36, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 37:\n",
      "[37, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[37, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[37, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[37, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[37, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[37, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[37, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[37, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[37, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[37, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[37, 22]: loss: 0.004, \t Running_Correct: 4.000\n",
      "[37, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 38:\n",
      "[38, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[38, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[38, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[38, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[38, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[38, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[38, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[38, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[38, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[38, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[38, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[38, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 39:\n",
      "[39, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[39, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[39, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[39, 8]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[39, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[39, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[39, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[39, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[39, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[39, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[39, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[39, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 40:\n",
      "[40, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[40, 4]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[40, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[40, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[40, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[40, 12]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[40, 14]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[40, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[40, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[40, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[40, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[40, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 41:\n",
      "[41, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[41, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 42:\n",
      "[42, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 6]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[42, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[42, 24]: loss: 0.002, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 43:\n",
      "[43, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[43, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[43, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[43, 8]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[43, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[43, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[43, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[43, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[43, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[43, 20]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[43, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[43, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 44:\n",
      "[44, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[44, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[44, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[44, 8]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[44, 10]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[44, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[44, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[44, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[44, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[44, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[44, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[44, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 45:\n",
      "[45, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[45, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[45, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 46:\n",
      "[46, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 16]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[46, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[46, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 47:\n",
      "[47, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[47, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 48:\n",
      "[48, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[48, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[48, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[48, 8]: loss: 0.004, \t Running_Correct: 4.000\n",
      "[48, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[48, 12]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[48, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[48, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[48, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[48, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[48, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[48, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 49:\n",
      "[49, 2]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 10]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[49, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 20]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[49, 24]: loss: 0.000, \t Running_Correct: 4.000\n",
      "\n",
      "Epoch 50:\n",
      "[50, 2]: loss: 0.001, \t Running_Correct: 4.000\n",
      "[50, 4]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[50, 6]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[50, 8]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[50, 10]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[50, 12]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[50, 14]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[50, 16]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[50, 18]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[50, 20]: loss: 0.002, \t Running_Correct: 4.000\n",
      "[50, 22]: loss: 0.000, \t Running_Correct: 4.000\n",
      "[50, 24]: loss: 0.001, \t Running_Correct: 4.000\n",
      "Ende  Training um:  21:23:57\n",
      "Dauer Training:  01:05   [MM:SS]  \n",
      "\n"
     ]
    }
   ],
   "source": [
    "## Number of Epochs for training\n",
    "epoch_range = 50\n",
    "\n",
    "##Start Trainingtime\n",
    "print(\"Start Training um: \", time.strftime(\"%H:%M:%S\"))\n",
    "start_time = time.time()\n",
    "\n",
    "## Variables for Training\n",
    "write_loss_epoch = ''\n",
    "running_correct = 0\n",
    "cancel_train = 0\n",
    "predicted_roc = []\n",
    "\n",
    "## Train Network\n",
    "network.train()\n",
    "for epoch in range(epoch_range):  # loop over the dataset multiple times\n",
    "    \n",
    "    ## Reset for every epoch\n",
    "    running_loss = 0.0\n",
    "    running_correct = 0\n",
    "    print(f'\\nEpoch {epoch+1}:')\n",
    "    \n",
    "    for i, data in enumerate(train_loader):\n",
    "        ## Get the inputs and labels from data; data is a list of [inputs, labels]\n",
    "        inputs, labels = data\n",
    "        inputs, labels = inputs.to(DEVICE), labels.to(DEVICE)\n",
    "\n",
    "        ## Zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        ## Calculate outputs\n",
    "        network.to(DEVICE)\n",
    "        outputs = network(inputs)\n",
    "        \n",
    "        ## Calculate loss,  Backpropagation and Optimize\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        ## Calulate running loss\n",
    "        running_loss += loss.item()\n",
    "        \n",
    "        ## Get predicted value\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        \n",
    "        running_correct += (predicted == labels).sum().item()\n",
    "        mini_batches = round(len(train_loader)/10)\n",
    "        \n",
    "        if i % mini_batches == (mini_batches-1):\n",
    "            print(f'[{epoch + 1}, {i + 1}]: loss: {(running_loss/mini_batches):0.3f}, \\t Running_Correct: {(running_correct/mini_batches):0.3f}')#, \\n\\t\\t Top1-Acc: {top_1.avg}, \\n\\t\\t Top5-Acc: {top_5.avg}')\n",
    "           \n",
    "            ## Collect losses in Array\n",
    "            write_loss_epoch += '['+ str(epoch+1)+ ', ' + str(i+1) + '] \\t loss: ' + str(running_loss/mini_batches) + '\\n'\n",
    "            \n",
    "            \n",
    "            ## Write Training Loss and Training Accuracy log the running loss\n",
    "            writer.add_scalar('training loss', running_loss / (mini_batches), epoch * len(train_loader) + i)\n",
    "            writer.add_scalar('accuracy', (running_correct / mini_batches/4), epoch * len(train_loader) + i)\n",
    "    \n",
    "            ## Reset\n",
    "            running_loss = 0.0\n",
    "            running_correct = 0\n",
    "\n",
    "\n",
    "## Finished Training: Calculate Training Time\n",
    "print(\"Ende  Training um: \",time.strftime(\"%H:%M:%S\"))\n",
    "stop_time = time.time()\n",
    "time_dif, time_format = secs_to_HMS(stop_time-start_time)\n",
    "print(\"Dauer Training: \", time_dif, \" \", time_format, \" \\n\")\n",
    "\n",
    "## Acoustic Sound to call attention for finished Training\n",
    "WaitTime_Finished()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trainiertes Netzwerk speichern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Path for saving trained Network\n",
    "PATH = './Netzwerke/'+ current_time + '_' + Database + '_' + network_name + '_Train' + '.pth'\n",
    "\n",
    "## Save trained Network\n",
    "torch.save({\n",
    "            'epoch_range': epoch_range,\n",
    "            'model_state_dict': network.state_dict(),\n",
    "            'optimizer_state_dict': optimizer.state_dict(),\n",
    "            'running_loss': running_loss,\n",
    "            }, PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataiter = iter(train_loader)\n",
    "images, labels = dataiter.next()\n",
    "img_grid = torchvision.utils.make_grid(images)\n",
    "\n",
    "# write to tensorboard\n",
    "writer.add_image((current_time + '_' + Database + '_' + network_name), img_grid)\n",
    "writer.add_graph(network.cpu(), images)\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Durchführung Testen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset: AMI, \tNetzwerk: VGG11\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\wurhofer\\anaconda3\\envs\\Test\\lib\\site-packages\\pytorch_lightning\\utilities\\distributed.py:49: UserWarning: Metric `AveragePrecision` will save all targets and predictions in buffer. For large datasets this may lead to large memory footprint.\n",
      "  warnings.warn(*args, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy    of the network on the 40 test images: 92.50 %\n",
      "Acc_PyLig     of the network on the 40 test images: 92.50 %\n",
      "F1            of the network on the 40 test images: 92.50 \n",
      "F1_PyLig      of the network on the 40 test images: 92.5\n",
      "AUC_PyLig:    of the network on the 40 test images: 96.05\n",
      "Top1          of the network on the 40 test images: 92.50 %\n",
      "Top5          of the network on the 40 test images: 100.00 %\n",
      "\n",
      "Overview: Accuracy of each class:\n",
      "Accuracy of   000 :  0 %\n",
      "Accuracy of   001 : 100 %\n",
      "Accuracy of   002 : 100 %\n",
      "Accuracy of   003 : 100 %\n",
      "Accuracy of   004 : 100 %\n",
      "Accuracy of   005 : 100 %\n",
      "Accuracy of   007 : 100 %\n",
      "Accuracy of   008 : 100 %\n",
      "Accuracy of   009 : 100 %\n",
      "Accuracy of   010 : 100 %\n",
      "Accuracy of   011 : 100 %\n",
      "Accuracy of   012 : 100 %\n",
      "Accuracy of   013 : 100 %\n",
      "Accuracy of   014 :  0 %\n",
      "Accuracy of   018 : 100 %\n",
      "Accuracy of   019 : 50 %\n",
      "Accuracy of   020 : 50 %\n",
      "Accuracy of   026 : 100 %\n",
      "Accuracy of   027 : 100 %\n",
      "Accuracy of   028 : 100 %\n"
     ]
    }
   ],
   "source": [
    "## Define global Varibales\n",
    "correct = 0\n",
    "total = 0\n",
    "running_loss_test = 0\n",
    "class_correct = list(0. for i in range(len(CATEGORIES)))\n",
    "class_total = list(0. for i in range(len(CATEGORIES)))\n",
    "acc_single_categorie = ''\n",
    "class_probs = []\n",
    "class_preds = []\n",
    "pred_all = torch.tensor([], dtype=int)\n",
    "labels_all = torch.tensor([], dtype=int)\n",
    "top_1 = AvgrageMeter()\n",
    "top_5 = AvgrageMeter()\n",
    "precision_sum = 0\n",
    "recall_sum = 0\n",
    "f1_sum = 0\n",
    "top1_sum = 0\n",
    "top5_sum = 0\n",
    "it = 0\n",
    "len_classes = len(CATEGORIES)\n",
    "\n",
    "## Test Network\n",
    "network.eval()\n",
    "with torch.no_grad():\n",
    "    print(f'Dataset: {Database}, \\tNetzwerk: {network_name}')\n",
    "    \n",
    "    for i, data in enumerate(test_loader, 0):\n",
    "        \n",
    "        it += 1\n",
    "        ## Load images from Test-Loader and hand over to DEVICE\n",
    "        images, labels = data\n",
    "        images, labels = images.to(DEVICE), labels.to(DEVICE)\n",
    "        \n",
    "        ## Calculate the predicted outputs\n",
    "        network.to(DEVICE)\n",
    "        outputs = network(images)\n",
    "        class_probs_batch = [F.softmax(el, dim=0) for el in outputs]\n",
    "        \n",
    "        ## Calculate the Test-Loss\n",
    "        loss = criterion(outputs, labels)\n",
    "        running_loss_test += loss.item()\n",
    "        \n",
    "        ## Get predicted values\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        \n",
    "        ## Calculate the Total and Correct Test-Results\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted == labels).sum().item()\n",
    "        c = (predicted == labels).squeeze()\n",
    "        \n",
    "        \n",
    "        ## Create a Tensor-Array with all Predicted-Values\n",
    "        pred_all = torch.cat([pred_all, predicted.cpu()])\n",
    "        ## Create a Tensor with all Label-Values\n",
    "        labels_all = torch.cat([labels_all, labels.cpu()])\n",
    "        \n",
    "        ## Create a Tensor-Array with all Predicted-Values for Tensorboard\n",
    "        class_probs.append(class_probs_batch)\n",
    "        class_preds.append(predicted) \n",
    "        test_probs = torch.cat([torch.stack(batch) for batch in class_probs])\n",
    "        test_preds = torch.cat(class_preds)\n",
    "        \n",
    "        \n",
    "        ## Calculate and update the Top-1 and Top-5 Accuracy\n",
    "        prec1, prec5 = torch_utils.accuracy(output=outputs, target=labels, top_k=(1, 5))\n",
    "        top_1.update(prec1.data, inputs.size(0))\n",
    "        top_5.update(prec5.data, inputs.size(0))\n",
    "\n",
    "        \n",
    "        ## Calculate F1-Score without OneHot \n",
    "        f1_score_t = F1(num_classes=len_classes, average='micro')\n",
    "        f1_score = f1_score_t(predicted.cpu(), labels.cpu())\n",
    "        f1_sum += f1_score \n",
    "        \n",
    "        \n",
    "        for i in range(3):\n",
    "            label = labels[i]\n",
    "            class_correct[label] += c[i].item()\n",
    "            class_total[label] += 1\n",
    "\n",
    "        \n",
    "        ## Write Test-Accuracy and Test-Loss to Tensorboard\n",
    "        writer.add_scalar('Test Accuracy',\n",
    "                          100 * correct / total,\n",
    "                          len(test_loader)+i)\n",
    "        writer.add_scalar('Test Loss',\n",
    "                           running_loss_test,\n",
    "                           len(test_loader) + i)\n",
    "\n",
    "        \n",
    "        \n",
    "        \n",
    "## Calculate the Test-Acuracy with all Predicted-Values and Labels\n",
    "accuracy = Accuracy()\n",
    "acc_all_b = accuracy(pred_all, labels_all)\n",
    "acc_all = acc_all_b * 100\n",
    "\n",
    "## Calculate the F1-Score\n",
    "## with PyTorch-Lightning\n",
    "f1_score_t = F1(num_classes=len_classes, average='micro')\n",
    "f1_score = f1_score_t(pred_all.cpu(), labels_all.cpu())\n",
    "f1_score_all = 100 * f1_score\n",
    "\n",
    "\n",
    "## OneHot Function for all Predicted-Values\n",
    "preds_onehot = (F.one_hot(pred_all)).float()\n",
    "\n",
    "\n",
    "## Calculate the Mean-Value of the Precision\n",
    "## with PyTorch-Lightning\n",
    "average_precision = AveragePrecision(num_classes=len_classes)\n",
    "prec_avg = average_precision(preds_onehot, labels_all)\n",
    "prec_mean_t = torch.mean(torch.stack(prec_avg))\n",
    "prec_mean = 100 * prec_mean_t\n",
    "\n",
    "## Calculate the AUC-Value\n",
    "## with PyTorch-Lightning\n",
    "auroc = multiclass_auroc(pred=preds_onehot, target=labels_all, num_classes=len_classes)\n",
    "auroc_1 = 100 * auroc\n",
    "\n",
    "\n",
    "# Calculate finale Values without OneHot\n",
    "acc = 100 * correct / total\n",
    "f1_f = 100 * f1_sum /it\n",
    "top_1_f = float((top_1.avg).cpu().numpy())\n",
    "top_5_f = float((top_5.avg).cpu().numpy())\n",
    "acc = 100 * correct / total\n",
    "\n",
    "\n",
    "print(f\"\\nAccuracy    of the network on the {len(dataset_test)} test images: {acc:.2f} %\")\n",
    "print(f'Acc_PyLig     of the network on the {len(dataset_test)} test images: {acc_all:.2f} %')\n",
    "print(f\"F1            of the network on the {len(dataset_test)} test images: {f1_f:.2f} \")\n",
    "print(f'F1_PyLig      of the network on the {len(dataset_test)} test images: {f1_score_all}')\n",
    "print(f'AUC_PyLig:    of the network on the {len(dataset_test)} test images: {auroc_1:.2f}')\n",
    "print(f\"Top1          of the network on the {len(dataset_test)} test images: {top_1_f:.2f} %\")\n",
    "print(f\"Top5          of the network on the {len(dataset_test)} test images: {top_5_f:.2f} %\")\n",
    "\n",
    "\n",
    "## Overview: Accaurcy of every Categorie\n",
    "print(f\"\\nOverview: Accuracy of each class:\")\n",
    "for i in range(len(CATEGORIES)):\n",
    "    if(class_total[i]==0):\n",
    "        class_total[i] = 0.1\n",
    "    print('Accuracy of %5s : %2d %%' % (CATEGORIES[i], 100 * class_correct[i] / class_total[i]))\n",
    "    acc_single_categorie = 'Accuracy of\\t'+ CATEGORIES[i] + ':\\t\\t' + str(100 * class_correct[i] / class_total[i]) + '%\\n'\n",
    "    Write_Each_Categorie_Acc_To_Editor(Database, network_name, acc_single_categorie)\n",
    "    \n",
    "WaitTime_Finished()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Funktion und Ausführung um weitere Daten und Parameter in Log-File zu schreiben"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./Log_Files/AMI/2021_02_27-17_31_53_AlexNet.txt\n"
     ]
    }
   ],
   "source": [
    "## Write to Log-Files (TXT-File) ##\n",
    "def Write_Training_Loss_To_Editor(database, net, train_num, test_num, accuracy, loss_epoch):\n",
    "    logfile_path = './Log_Files/' + database + '/' + str(current_time) + '_' + net   + '.txt'\n",
    "    #logfile_path = './Log_Files/Test.txt'\n",
    "    print(logfile_path)\n",
    "\n",
    "    datei = open(logfile_path, 'a')\n",
    "    datei.write(\"Datensatz: \")\n",
    "    datei.write(Database + \"\\n\\n\")\n",
    "    datei.write(\"Netzwerk: \" + net + \"\\n\\n\")\n",
    "    datei.write(\"Anzahl der Trainingsdatensaetze: \\t\" + str(train_num) + \"\\n\")\n",
    "    datei.write(\"Anzahl der Testdatensaetze: \\t\\t\" + str(test_num) + \"\\n\\n\")\n",
    "    datei.write(\"Accuracy des \" + net + \"-Netzwerks \" + \"bei \" + str(test_num) + \" Testbilder:  \" + str(accuracy) + \"%\\n\\n\")\n",
    "    datei.write(loss_epoch)\n",
    "    datei.close()\n",
    "\n",
    "Write_Training_Loss_To_Editor(Database, network_name, len(dataset_train), len(dataset_test), acc, write_loss_epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Funktion um PR-Kurve für jede Kategorie in Tensorboard zu übertragen\n",
    "\n",
    "Hinweis: Diese Funktion kann ausgeführt werden, wird aber in der Arbeit nicht weiter analysiert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_pr_curve_tensorboard(class_index, test_probs, test_preds, global_step=0):\n",
    "    '''\n",
    "    Takes in a \"class_index\" from 0 to 9 and plots the corresponding\n",
    "    precision-recall curve\n",
    "    '''\n",
    "    tensorboard_preds = test_preds == class_index\n",
    "    tensorboard_probs = test_probs[:, class_index]\n",
    "\n",
    "    writer.add_pr_curve(CATEGORIES[class_index],\n",
    "                        tensorboard_preds,\n",
    "                        tensorboard_probs,\n",
    "                        global_step=global_step)\n",
    "    writer.close()\n",
    "\n",
    "# plot all the pr curves\n",
    "for i in range(len(CATEGORIES)):\n",
    "    add_pr_curve_tensorboard(i, test_probs, test_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Schreiben der Daten ins Logbuch\n",
    "\n",
    "Schreibt alle gesammelten Daten und berechneten Parameter in ein CSV-File\n",
    "\n",
    "* **Write_Data_To_CSV():** \n",
    "    Schreibt alle Accuracy-Werte aller genutzten Netzwerke in ein CSV-File\n",
    "* **Write_Data_To_CSV_expand():**\n",
    "    Schreibt alle berechnten Parameter (Top1-/Top5-Accuracy, AUC, F1-Score) in ein CSV-File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Function to write Data to CSV-File\n",
    "def Write_Data_To_CSV(database, net, train_num, test_num, batchSize_Train, batchSize_Test, learning_rate, momentum, epoch, running_loss, train_time, time_format, accuracy, stamp):\n",
    "    CSV = pd.read_csv(\"CSV_Files/Logbuch_CSV_corr.csv\", sep=';')\n",
    "    CSV_df = pd.DataFrame(CSV)\n",
    "\n",
    "    Names = [{   \n",
    "                \"Datensatz\":database,\n",
    "                \"Netzwerk\":net,\n",
    "                \"Anzahl Trainingsbilder\": train_num,\n",
    "                \"Anzahl Testbilder\": test_num,\n",
    "                \"Batch_Size Training\": batchSize_Train,\n",
    "                \"Batch_Size Test\": batchSize_Test,\n",
    "                \"Learning Rate\": learning_rate,\n",
    "                \"Momentum\": momentum,\n",
    "                \"Epochen\": epoch+1, \n",
    "                \"Running Loss\": running_loss, \n",
    "                \"Dauer Training\":train_time, \n",
    "                \"Zeitformat\":time_format,\n",
    "                \"Accuracy\": accuracy, \n",
    "                \"Stempel\": stamp\n",
    "             }]\n",
    "    Names_df = pd.DataFrame(Names)\n",
    "    df_neu = CSV_df.append(Names_df, ignore_index=True)\n",
    "    df_neu.to_csv(\"CSV_Files/Logbuch_CSV.csv\", sep=';', index=False, header=True)\n",
    "    return pd.read_csv(\"CSV_Files/Logbuch_CSV.csv\", sep=';')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Write_Data_To_CSV_expand(database, net, train_num, test_num, batchSize_Train, batchSize_Test, learning_rate, momentum, epoch, running_loss, train_time, time_format, accuracy, Acc_all, Top_1_f, Top_5_f, AUC, F1_f, F1_score_all, stamp):\n",
    "    CSV = pd.read_csv(\"CSV_Files/Logbuch_CSV_erweitert.csv\", sep=';')\n",
    "    CSV_df = pd.DataFrame(CSV)\n",
    "\n",
    "    Names = [{   \n",
    "                \"Datensatz\":database,\n",
    "                \"Netzwerk\":net,\n",
    "                \"Anzahl Trainingsbilder\": train_num,\n",
    "                \"Anzahl Testbilder\": test_num,\n",
    "                \"Batch_Size Training\": batchSize_Train,\n",
    "                \"Batch_Size Test\": batchSize_Test,\n",
    "                \"Learning Rate\": learning_rate,\n",
    "                \"Momentum\": momentum,\n",
    "                \"Epochen\": epoch+1, \n",
    "                \"Running Loss\": running_loss, \n",
    "                \"Dauer Training\":train_time, \n",
    "                \"Zeitformat\":time_format,\n",
    "                \"Accuracy\": accuracy, \n",
    "                \"Accuracy_PyLig\": Acc_all.numpy(),\n",
    "                \"Top1\": Top_1_f,\n",
    "                \"Top5\": Top_5_f,\n",
    "                \"AUC\": AUC.numpy(),\n",
    "                \"F1_my\": F1_f.numpy(),\n",
    "                \"F1_PyLig\": F1_score_all.numpy(),\n",
    "                \"Stempel\": stamp\n",
    "             }]\n",
    "    Names_df = pd.DataFrame(Names)\n",
    "    df_neu = CSV_df.append(Names_df, ignore_index=True)\n",
    "    df_neu.to_csv(\"CSV_Files/Logbuch_CSV_erweitert.csv\", sep=';', index=False, header=True)\n",
    "    return pd.read_csv(\"CSV_Files/Logbuch_CSV_erweitert.csv\", sep=';')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Datensatz</th>\n",
       "      <th>Netzwerk</th>\n",
       "      <th>Anzahl Trainingsbilder</th>\n",
       "      <th>Anzahl Testbilder</th>\n",
       "      <th>Batch_Size Training</th>\n",
       "      <th>Batch_Size Test</th>\n",
       "      <th>Learning Rate</th>\n",
       "      <th>Momentum</th>\n",
       "      <th>Epochen</th>\n",
       "      <th>Running Loss</th>\n",
       "      <th>Dauer Training</th>\n",
       "      <th>Zeitformat</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Accuracy_PyLig</th>\n",
       "      <th>Top1</th>\n",
       "      <th>Top5</th>\n",
       "      <th>AUC</th>\n",
       "      <th>F1_my</th>\n",
       "      <th>F1_PyLig</th>\n",
       "      <th>Stempel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Neu</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CP</td>\n",
       "      <td>Resnext101-32x8d</td>\n",
       "      <td>68.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.005421876907348633</td>\n",
       "      <td>02:51</td>\n",
       "      <td>[MM:SS]</td>\n",
       "      <td>97.06</td>\n",
       "      <td>97.06</td>\n",
       "      <td>97.22</td>\n",
       "      <td>100.0</td>\n",
       "      <td>98.44</td>\n",
       "      <td>97.22</td>\n",
       "      <td>97.06</td>\n",
       "      <td>2021_02_03-18_10_22_CP_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMI</td>\n",
       "      <td>Resnext101-32x8d</td>\n",
       "      <td>500.0</td>\n",
       "      <td>200.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.026825428009033203</td>\n",
       "      <td>21:03</td>\n",
       "      <td>[MM:SS]</td>\n",
       "      <td>96.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>97.98</td>\n",
       "      <td>96.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>2021_02_03-20_49_49_AMI_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AWE</td>\n",
       "      <td>Resnext101-32x8d</td>\n",
       "      <td>800.0</td>\n",
       "      <td>200.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33:36:00</td>\n",
       "      <td>[MM:SS]</td>\n",
       "      <td>51.5</td>\n",
       "      <td>51.5</td>\n",
       "      <td>51.5</td>\n",
       "      <td>73.0</td>\n",
       "      <td>75.51</td>\n",
       "      <td>51.5</td>\n",
       "      <td>51.5</td>\n",
       "      <td>2021_02_03-22_11_34_AWE_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>UERC</td>\n",
       "      <td>Resnext101-32x8d</td>\n",
       "      <td>2192.0</td>\n",
       "      <td>548.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.6218500137329102</td>\n",
       "      <td>02:32:02</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>30.29</td>\n",
       "      <td>30.29</td>\n",
       "      <td>30.29</td>\n",
       "      <td>49.63</td>\n",
       "      <td>65.02</td>\n",
       "      <td>30.29</td>\n",
       "      <td>30.29</td>\n",
       "      <td>2021_02_03-22_47_50_UERC_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>CP_e</td>\n",
       "      <td>Resnext101-32x8d_e</td>\n",
       "      <td>340.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.009591817855834961</td>\n",
       "      <td>14:22</td>\n",
       "      <td>[MM:SS]</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2021_02_04-17_32_38_CP_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AMI_e</td>\n",
       "      <td>Resnext101-32x8d_e</td>\n",
       "      <td>2400.0</td>\n",
       "      <td>400.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>02:40:59</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2021_02_04-17_51_14_AMI_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>AWE_e</td>\n",
       "      <td>Resnext101-32x8d_e</td>\n",
       "      <td>3200.0</td>\n",
       "      <td>800.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>03:14:34</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>98.375</td>\n",
       "      <td>98.375</td>\n",
       "      <td>98.375</td>\n",
       "      <td>99.5</td>\n",
       "      <td>9.917.928</td>\n",
       "      <td>98.375</td>\n",
       "      <td>98.375</td>\n",
       "      <td>2021_02_04-21_54_54_AWE_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>UERC_e</td>\n",
       "      <td>Resnext101-32x8d_e</td>\n",
       "      <td>8768.0</td>\n",
       "      <td>2192.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.004006385803222656</td>\n",
       "      <td>07:07:53</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>9.630.474.452.554.740</td>\n",
       "      <td>9.630.474</td>\n",
       "      <td>9.630.474.090.576.170</td>\n",
       "      <td>9.945.255.279.541.010</td>\n",
       "      <td>9.814.559</td>\n",
       "      <td>9.630.474</td>\n",
       "      <td>9.630.474</td>\n",
       "      <td>2021_02_05-10_11_05_UERC_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>EarVN_1_0</td>\n",
       "      <td>Resnext101-32x8d</td>\n",
       "      <td>22681.0</td>\n",
       "      <td>5731.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>29.751.343.727.111.800</td>\n",
       "      <td>17:26:41</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>92.23</td>\n",
       "      <td>92.235.214</td>\n",
       "      <td>9.223.656.463.623.040</td>\n",
       "      <td>9.738.311.004.638.670</td>\n",
       "      <td>9.600.976</td>\n",
       "      <td>92.236.565</td>\n",
       "      <td>92.235.214</td>\n",
       "      <td>2021_02_05-18_04_07_EarVN_1_0_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>AMI</td>\n",
       "      <td>AlexNet</td>\n",
       "      <td>100.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.00015020370483398438</td>\n",
       "      <td>00:22</td>\n",
       "      <td>[MM:SS]</td>\n",
       "      <td>95.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.368416</td>\n",
       "      <td>95.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2021_02_27-17_31_53_AMI_AlexNet</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Datensatz            Netzwerk  Anzahl Trainingsbilder  Anzahl Testbilder  \\\n",
       "0         Neu                 NaN                     NaN                NaN   \n",
       "1          CP    Resnext101-32x8d                    68.0               34.0   \n",
       "2         AMI    Resnext101-32x8d                   500.0              200.0   \n",
       "3         AWE    Resnext101-32x8d                   800.0              200.0   \n",
       "4        UERC    Resnext101-32x8d                  2192.0              548.0   \n",
       "5        CP_e  Resnext101-32x8d_e                   340.0               68.0   \n",
       "6       AMI_e  Resnext101-32x8d_e                  2400.0              400.0   \n",
       "7       AWE_e  Resnext101-32x8d_e                  3200.0              800.0   \n",
       "8      UERC_e  Resnext101-32x8d_e                  8768.0             2192.0   \n",
       "9   EarVN_1_0    Resnext101-32x8d                 22681.0             5731.0   \n",
       "10        AMI             AlexNet                   100.0               40.0   \n",
       "\n",
       "    Batch_Size Training  Batch_Size Test  Learning Rate  Momentum  Epochen  \\\n",
       "0                   NaN              NaN            NaN       NaN      NaN   \n",
       "1                   4.0              4.0          0.001       0.8     50.0   \n",
       "2                   4.0              4.0          0.001       0.8     50.0   \n",
       "3                   4.0              4.0          0.001       0.8     50.0   \n",
       "4                   4.0              4.0          0.001       0.8     50.0   \n",
       "5                   4.0              4.0          0.001       0.8     50.0   \n",
       "6                   4.0              4.0          0.001       0.8     50.0   \n",
       "7                   4.0              4.0          0.001       0.8     50.0   \n",
       "8                   4.0              4.0          0.001       0.8     50.0   \n",
       "9                   4.0              4.0          0.001       0.8     50.0   \n",
       "10                  4.0              4.0          0.001       0.8     50.0   \n",
       "\n",
       "              Running Loss Dauer Training  Zeitformat               Accuracy  \\\n",
       "0                      NaN            NaN         NaN                    NaN   \n",
       "1     0.005421876907348633          02:51     [MM:SS]                  97.06   \n",
       "2     0.026825428009033203          21:03     [MM:SS]                   96.0   \n",
       "3                      0.0       33:36:00     [MM:SS]                   51.5   \n",
       "4       0.6218500137329102       02:32:02  [HH:MM:SS]                  30.29   \n",
       "5     0.009591817855834961          14:22     [MM:SS]                  100.0   \n",
       "6                      0.0       02:40:59  [HH:MM:SS]                  100.0   \n",
       "7                      0.0       03:14:34  [HH:MM:SS]                 98.375   \n",
       "8     0.004006385803222656       07:07:53  [HH:MM:SS]  9.630.474.452.554.740   \n",
       "9   29.751.343.727.111.800       17:26:41  [HH:MM:SS]                  92.23   \n",
       "10  0.00015020370483398438          00:22     [MM:SS]                   95.0   \n",
       "\n",
       "   Accuracy_PyLig                   Top1                   Top5        AUC  \\\n",
       "0             NaN                    NaN                    NaN        NaN   \n",
       "1           97.06                  97.22                  100.0      98.44   \n",
       "2            96.0                   96.0                   99.0      97.98   \n",
       "3            51.5                   51.5                   73.0      75.51   \n",
       "4           30.29                  30.29                  49.63      65.02   \n",
       "5           100.0                  100.0                  100.0      100.0   \n",
       "6           100.0                  100.0                  100.0      100.0   \n",
       "7          98.375                 98.375                   99.5  9.917.928   \n",
       "8       9.630.474  9.630.474.090.576.170  9.945.255.279.541.010  9.814.559   \n",
       "9      92.235.214  9.223.656.463.623.040  9.738.311.004.638.670  9.600.976   \n",
       "10           95.0                   95.0                  100.0  97.368416   \n",
       "\n",
       "         F1_my    F1_PyLig                                         Stempel  \n",
       "0          NaN         NaN                                             NaN  \n",
       "1        97.22       97.06         2021_02_03-18_10_22_CP_Resnext101-32x8d  \n",
       "2         96.0        96.0        2021_02_03-20_49_49_AMI_Resnext101-32x8d  \n",
       "3         51.5        51.5        2021_02_03-22_11_34_AWE_Resnext101-32x8d  \n",
       "4        30.29       30.29       2021_02_03-22_47_50_UERC_Resnext101-32x8d  \n",
       "5        100.0       100.0         2021_02_04-17_32_38_CP_Resnext101-32x8d  \n",
       "6        100.0       100.0        2021_02_04-17_51_14_AMI_Resnext101-32x8d  \n",
       "7       98.375      98.375        2021_02_04-21_54_54_AWE_Resnext101-32x8d  \n",
       "8    9.630.474   9.630.474       2021_02_05-10_11_05_UERC_Resnext101-32x8d  \n",
       "9   92.236.565  92.235.214  2021_02_05-18_04_07_EarVN_1_0_Resnext101-32x8d  \n",
       "10        95.0        95.0                 2021_02_27-17_31_53_AMI_AlexNet  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Write_Data_To_CSV_expand(Database, network_name, len(dataset_train), len(dataset_test), batch_size_train, batch_size_test, learning_rate, momentum, epoch, running_loss, time_dif, time_format, acc, acc_all, top_1_f, top_5_f, auroc_1, f1_f, f1_score_all, stamp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Datensatz</th>\n",
       "      <th>Netzwerk</th>\n",
       "      <th>Anzahl Trainingsbilder</th>\n",
       "      <th>Anzahl Testbilder</th>\n",
       "      <th>Batch_Size Training</th>\n",
       "      <th>Batch_Size Test</th>\n",
       "      <th>Learning Rate</th>\n",
       "      <th>Momentum</th>\n",
       "      <th>Epochen</th>\n",
       "      <th>Running Loss</th>\n",
       "      <th>Dauer Training</th>\n",
       "      <th>Zeitformat</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Stempel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AWE</td>\n",
       "      <td>VGG11</td>\n",
       "      <td>800</td>\n",
       "      <td>200</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>25</td>\n",
       "      <td>0.018202782</td>\n",
       "      <td>04:11</td>\n",
       "      <td>[MM:SS]</td>\n",
       "      <td>24.13</td>\n",
       "      <td>2020_10_13-13_47_08_AWE_VGG11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CP</td>\n",
       "      <td>VGG11</td>\n",
       "      <td>68</td>\n",
       "      <td>34</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>49</td>\n",
       "      <td>0.001134396</td>\n",
       "      <td>00:42</td>\n",
       "      <td>[MM:SS]</td>\n",
       "      <td>94.12</td>\n",
       "      <td>2020_10_13-13_47_08_CP_VGG11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AMI</td>\n",
       "      <td>VGG11</td>\n",
       "      <td>500</td>\n",
       "      <td>200</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>49</td>\n",
       "      <td>0.002470016</td>\n",
       "      <td>05:06</td>\n",
       "      <td>[MM:SS]</td>\n",
       "      <td>78.00</td>\n",
       "      <td>2020_10_13-13_47_08_AMI_VGG11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>EarVN_1_0</td>\n",
       "      <td>VGG11</td>\n",
       "      <td>22681</td>\n",
       "      <td>5731</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>04:50:23</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>68.14</td>\n",
       "      <td>2020_10_13-13_47_08_EarVN_1_0_VGG11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>EarVN_1_0_e</td>\n",
       "      <td>VGG11_e</td>\n",
       "      <td>22681</td>\n",
       "      <td>5731</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>04:50:23</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>68.14</td>\n",
       "      <td>2020_10_13-13_47_08_EarVN_1_0_VGG11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>151</th>\n",
       "      <td>WPUT_e_w</td>\n",
       "      <td>GoogLeNet_e_w</td>\n",
       "      <td>9076</td>\n",
       "      <td>2904</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50</td>\n",
       "      <td>4.961.103.916.168.210</td>\n",
       "      <td>03:17:00</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>99.44</td>\n",
       "      <td>2020_11_01-11_07_43_WPUT_GoogLeNet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>WPUT_e_w</td>\n",
       "      <td>Shufflenet-v2-x0_5_e_w</td>\n",
       "      <td>9076</td>\n",
       "      <td>2904</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50</td>\n",
       "      <td>43.149.518.966.674.800</td>\n",
       "      <td>03:09:59</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>99.35</td>\n",
       "      <td>2020_11_01-13_56_51_WPUT_Shufflenet-v2-x0_5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153</th>\n",
       "      <td>WPUT_e_w</td>\n",
       "      <td>Resnext101-32x8d_e_w</td>\n",
       "      <td>9076</td>\n",
       "      <td>2904</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50</td>\n",
       "      <td>0.841792107</td>\n",
       "      <td>07:52:04</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>100.00</td>\n",
       "      <td>2020_11_01-16_10_09_WPUT_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154</th>\n",
       "      <td>UERC_w</td>\n",
       "      <td>Resnext101-32x8d_w</td>\n",
       "      <td>2192</td>\n",
       "      <td>548</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50</td>\n",
       "      <td>0.597034216</td>\n",
       "      <td>02:36:52</td>\n",
       "      <td>[HH:MM:SS]</td>\n",
       "      <td>44.13</td>\n",
       "      <td>2020_10_23-10_11_30_UERC_Resnext101-32x8d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>AMI</td>\n",
       "      <td>AlexNet</td>\n",
       "      <td>100</td>\n",
       "      <td>40</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.8</td>\n",
       "      <td>50</td>\n",
       "      <td>0.00015020370483398438</td>\n",
       "      <td>00:22</td>\n",
       "      <td>[MM:SS]</td>\n",
       "      <td>95.00</td>\n",
       "      <td>2021_02_27-17_31_53_AMI_AlexNet</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>156 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Datensatz                Netzwerk  Anzahl Trainingsbilder  \\\n",
       "0            AWE                   VGG11                     800   \n",
       "1             CP                   VGG11                      68   \n",
       "2            AMI                   VGG11                     500   \n",
       "3      EarVN_1_0                   VGG11                   22681   \n",
       "4    EarVN_1_0_e                 VGG11_e                   22681   \n",
       "..           ...                     ...                     ...   \n",
       "151     WPUT_e_w           GoogLeNet_e_w                    9076   \n",
       "152     WPUT_e_w  Shufflenet-v2-x0_5_e_w                    9076   \n",
       "153     WPUT_e_w    Resnext101-32x8d_e_w                    9076   \n",
       "154       UERC_w      Resnext101-32x8d_w                    2192   \n",
       "155          AMI                 AlexNet                     100   \n",
       "\n",
       "     Anzahl Testbilder  Batch_Size Training  Batch_Size Test  Learning Rate  \\\n",
       "0                  200                    4                4          0.001   \n",
       "1                   34                    4                4          0.001   \n",
       "2                  200                    4                4          0.001   \n",
       "3                 5731                    4                4          0.001   \n",
       "4                 5731                    4                4          0.001   \n",
       "..                 ...                  ...              ...            ...   \n",
       "151               2904                    4                4          0.001   \n",
       "152               2904                    4                4          0.001   \n",
       "153               2904                    4                4          0.001   \n",
       "154                548                    4                4          0.001   \n",
       "155                 40                    4                4          0.001   \n",
       "\n",
       "     Momentum  Epochen            Running Loss Dauer Training  Zeitformat  \\\n",
       "0         0.8       25             0.018202782          04:11     [MM:SS]   \n",
       "1         0.8       49             0.001134396          00:42     [MM:SS]   \n",
       "2         0.8       49             0.002470016          05:06     [MM:SS]   \n",
       "3         0.8       49                       0       04:50:23  [HH:MM:SS]   \n",
       "4         0.8       49                       0       04:50:23  [HH:MM:SS]   \n",
       "..        ...      ...                     ...            ...         ...   \n",
       "151       0.8       50   4.961.103.916.168.210       03:17:00  [HH:MM:SS]   \n",
       "152       0.8       50  43.149.518.966.674.800       03:09:59  [HH:MM:SS]   \n",
       "153       0.8       50             0.841792107       07:52:04  [HH:MM:SS]   \n",
       "154       0.8       50             0.597034216       02:36:52  [HH:MM:SS]   \n",
       "155       0.8       50  0.00015020370483398438          00:22     [MM:SS]   \n",
       "\n",
       "     Accuracy                                      Stempel  \n",
       "0       24.13                2020_10_13-13_47_08_AWE_VGG11  \n",
       "1       94.12                 2020_10_13-13_47_08_CP_VGG11  \n",
       "2       78.00                2020_10_13-13_47_08_AMI_VGG11  \n",
       "3       68.14          2020_10_13-13_47_08_EarVN_1_0_VGG11  \n",
       "4       68.14          2020_10_13-13_47_08_EarVN_1_0_VGG11  \n",
       "..        ...                                          ...  \n",
       "151     99.44           2020_11_01-11_07_43_WPUT_GoogLeNet  \n",
       "152     99.35  2020_11_01-13_56_51_WPUT_Shufflenet-v2-x0_5  \n",
       "153    100.00    2020_11_01-16_10_09_WPUT_Resnext101-32x8d  \n",
       "154     44.13    2020_10_23-10_11_30_UERC_Resnext101-32x8d  \n",
       "155     95.00              2021_02_27-17_31_53_AMI_AlexNet  \n",
       "\n",
       "[156 rows x 14 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Write_Data_To_CSV(Database, network_name, len(dataset_train), len(dataset_test), batch_size_train, batch_size_test, learning_rate, momentum, epoch, running_loss, time_dif, time_format, acc, stamp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PR-Kurve mit Tensorboard\n",
    "\n",
    "Generiert für alle Kategorien eine PR-Kurve in Tensorboard. Diese Funktion wird schlußendlich nicht zur Bewertung der Datensätze und Netzwerke verwendet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. gets the probability predictions in a test_size x num_classes Tensor\n",
    "# 2. gets the preds in a test_size Tensor\n",
    "# takes ~10 seconds to run\n",
    "\n",
    "class_probs = []\n",
    "class_preds = []\n",
    "with torch.no_grad():\n",
    "    for data in test_loader:\n",
    "        images, labels = data\n",
    "        network.to(DEVICE)\n",
    "        network.cpu()\n",
    "        output = network(images)\n",
    "        class_probs_batch = [F.softmax(el, dim=0) for el in output]\n",
    "        _, class_preds_batch = torch.max(output, 1)\n",
    "\n",
    "        class_probs.append(class_probs_batch)\n",
    "        class_preds.append(class_preds_batch)\n",
    "\n",
    "test_probs = torch.cat([torch.stack(batch) for batch in class_probs])\n",
    "test_preds = torch.cat(class_preds)\n",
    "\n",
    "# helper function\n",
    "def add_pr_curve_tensorboard(class_index, test_probs, test_preds, global_step=0):\n",
    "    '''\n",
    "    Takes in a \"class_index\" from 0 to 9 and plots the corresponding\n",
    "    precision-recall curve\n",
    "    '''\n",
    "    tensorboard_preds = test_preds == class_index\n",
    "    tensorboard_probs = test_probs[:, class_index]\n",
    "\n",
    "    writer.add_pr_curve(CATEGORIES[class_index],\n",
    "                        tensorboard_preds,\n",
    "                        tensorboard_probs,\n",
    "                        global_step=global_step)\n",
    "    writer.close()\n",
    "\n",
    "# plot all the pr curves\n",
    "for i in range(len(CATEGORIES)):\n",
    "    add_pr_curve_tensorboard(i, test_probs, test_preds)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
