MA-Ohranalyse_mit_CNN_PyTorch/OneShot_Ears.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One Shot Learning\n",
    "\n",
    "Mit diesem Jupyter-Skript werden alle notwendigen Funktionen zur Umsetzung und Auswertung des OneShot-Learnings (Siamese Networks) von Ohrbildern zusammengefasst.  \n",
    "Das Jupyter-Skript bezieht sich auf die Masterarbeit: \"Verwendung des menschlichen Ohrs zur Personenauthentifizierung an IT-Systemen mittels CNNs\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bibliotheken importieren\n",
    "\n",
    "Zunächst werden alle notwendigen Bibliothken importiert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Import Libearies ##\n",
    "%matplotlib inline\n",
    "import cv2\n",
    "from PIL import Image\n",
    "import PIL.ImageOps \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "## Import TORCHVISION with popular datasets, model architectures, \n",
    "## and common image transformations for computer vision\n",
    "from torchvision import transforms\n",
    "import torchvision\n",
    "import torchvision.utils\n",
    "import torchvision.models as models\n",
    "import torchvision.datasets as dset\n",
    "import torchsummary\n",
    "\n",
    "# Import Debugging method for set_trace\n",
    "from IPython.core.debugger import set_trace\n",
    "import logging\n",
    "\n",
    "## Import Time Features\n",
    "import datetime\n",
    "import time\n",
    "\n",
    "## Import TORCH for Data-Structures for multi-dimensional tensors \n",
    "## and mathematical operations\n",
    "import torch\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader, Dataset\n",
    "import torch.optim as optim\n",
    "import torch_utils\n",
    "from torch.autograd import Variable\n",
    "\n",
    "# Import Winsound to play a *.wav-File\n",
    "# and Counter for counting numbers in an array\n",
    "import winsound\n",
    "from collections import Counter\n",
    "\n",
    "## Import Shutil to copy or delete Files\n",
    "import shutil\n",
    "from shutil import copyfile\n",
    "\n",
    "## Import Random Libery\n",
    "import random\n",
    "from random import shuffle\n",
    "\n",
    "## Import IPYWIDGET for interactive HTML widgets for Jupyter notebooks\n",
    "import ipywidgets as wg\n",
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Festlegung: Prozessor oder Graphikkarte"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Funktion zur Gegenüberstellung zwei Ohren\n",
    "\n",
    "Zwei Ohren werden gegenübergestellt und dazu der Constative-Loss Wert angezeigt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build a rectangle in axes coords\n",
    "left, width = 5, .5\n",
    "bottom, height = .25, .5\n",
    "right = left + width\n",
    "top = bottom + height\n",
    "\n",
    "## Function to show Ear images and the Constrative Loss\n",
    "def imshow(img,text=None,should_save=False):\n",
    "    npimg = img.cpu().numpy()\n",
    "    plt.axis(\"off\")\n",
    "    if text:\n",
    "        plt.text(75, 8, text, style='italic',fontweight='bold',\n",
    "            bbox={'facecolor':'white', 'alpha':0.8, 'pad':10})\n",
    "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
    "    plt.show()   \n",
    "\n",
    "## Function to show Learning-Curve with matplotlib\n",
    "def show_plot(iteration,loss):\n",
    "    plt.plot(iteration,loss)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datensätze festlegen, anzeigen und einlesen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Indirectr Path for Linux and Windows## \n",
    "def ChooseDir(datadir):\n",
    "    if(datadir == 'CP'):\n",
    "        return \"Datensaetze/CP\", \"CP\"\n",
    "    if(datadir == 'AMI'):\n",
    "        return \"Datensaetze/AMI\", \"AMI\"\n",
    "    if(datadir == 'AWE'):\n",
    "        return \"Datensaetze/AWE\", \"AWE\"\n",
    "    if(datadir == 'EarVN'):\n",
    "        return \"Datensaetze/EarVN_1_0\", \"EarVN_1_0\"\n",
    "    if(datadir == 'UERC'):\n",
    "        return \"Datensaetze/UERC\", \"UERC\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Voreinstellungen - Hilfsfunktionen\n",
    "\n",
    "* **secs_to_HMS():** Berechnte Sekunden in HH:MM:SS um\n",
    "* **WaitTime_Finished():** Gibt eine Audio-Datei aus, wenn Training und Testen beendet ist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Convert Seconds to Hours, Minutes and Seconds ##\n",
    "def secs_to_HMS(secs):\n",
    "    if (secs < 3600):\n",
    "        return datetime.datetime.fromtimestamp(secs).strftime('%M:%S'), \"[MM:SS]\"\n",
    "    else:\n",
    "        return datetime.datetime.fromtimestamp(secs).strftime('%H:%M:%S'), \"[HH:MM:SS]\"\n",
    "\n",
    "## Acustic Sound if Model-Trainings finished ##\n",
    "def WaitTime_Finished():\n",
    "    for i in range(2):\n",
    "        files=os.listdir(\"Sound/\")\n",
    "        file=random.choice(files)\n",
    "        winsound.PlaySound(\"Sound/\"+str(file), winsound.SND_FILENAME)\n",
    "        time.sleep(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021_02_27-21_19_36\n"
     ]
    }
   ],
   "source": [
    "## Use actual Time for individual Tag ##\n",
    "current_time = datetime.datetime.now().strftime(\"%Y_%m_%d-%H_%M_%S\")\n",
    "print(current_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "## All Networks from PyTorch wich can be pretrained ## \n",
    "class SiameseNetwork(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SiameseNetwork, self).__init__()\n",
    "        self.cnn1 = nn.Sequential(\n",
    "            nn.ReflectionPad2d(1),\n",
    "            nn.Conv2d(1, 4, kernel_size=3),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.BatchNorm2d(4),\n",
    "            \n",
    "            nn.ReflectionPad2d(1),\n",
    "            nn.Conv2d(4, 8, kernel_size=3),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.BatchNorm2d(8),\n",
    "\n",
    "\n",
    "            nn.ReflectionPad2d(1),\n",
    "            nn.Conv2d(8, 8, kernel_size=3),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.BatchNorm2d(8),\n",
    "        )\n",
    "\n",
    "        self.fc1 = nn.Sequential(\n",
    "            nn.Linear(8*100*100, 500),\n",
    "            nn.ReLU(inplace=True),\n",
    "\n",
    "            nn.Linear(500, 500),\n",
    "            nn.ReLU(inplace=True),\n",
    "\n",
    "            nn.Linear(500, 5))\n",
    "    def forward_once(self, x):\n",
    "        output = self.cnn1(x)\n",
    "        output = output.view(output.size()[0], -1)\n",
    "        output = self.fc1(output)\n",
    "        return output\n",
    "    def forward(self, input1, input2):\n",
    "        output1 = self.forward_once(input1)\n",
    "        output2 = self.forward_once(input2)\n",
    "        return output1, output2\n",
    "\n",
    "\n",
    "def Network_Choice(netw):    \n",
    "    if(netw == 'vgg11'):\n",
    "        return models.vgg11(pretrained=True).to(DEVICE), \"VGG11\"\n",
    "    if(netw == 'vgg11bn'):\n",
    "        return models.vgg11_bn(pretrained=True).to(DEVICE), \"VGG11bn\"\n",
    "    if(netw == 'resnet18'):\n",
    "        return models.resnet18(pretrained=True).to(DEVICE), \"ResNet18\"\n",
    "    if(netw == 'resnet34'):\n",
    "        return models.resnet34(pretrained=True).to(DEVICE), \"ResNet34\"\n",
    "    if(netw == 'alexnet'):\n",
    "        return models.alexnet(pretrained=True).to(DEVICE), \"AlexNet\"\n",
    "    if(netw == 'squeezenet1_0'):\n",
    "        return models.squeezenet1_0(pretrained=True).to(DEVICE), \"SqueezeNet-1-0\"  \n",
    "    if(netw == 'GoogLeNet'):\n",
    "        return models.googlenet(pretrained=True).to(DEVICE), \"GoogLeNet\" \n",
    "    if(netw == 'shufflenet_v2_x0_5'):\n",
    "        return models.shufflenet_v2_x0_5(pretrained=True).to(DEVICE), \"Shufflenet-v2-x0_5\"  \n",
    "    if(netw == 'resnext101_32x8d'):\n",
    "        return models.resnext101_32x8d(pretrained=True).to(DEVICE), \"Resnext101-32x8d\"\n",
    "    if(netw == 'siamese'):\n",
    "        return SiameseNetwork().to(DEVICE), 'Siamese_Network'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contrastive Loss\n",
    "\n",
    "Constrive-Loss Berechnung für ein Siamese-Network "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ContrastiveLoss(torch.nn.Module):\n",
    "    def __init__(self, margin=2.0):\n",
    "        super(ContrastiveLoss, self).__init__()\n",
    "        self.margin = margin\n",
    "\n",
    "    def forward(self, output1, output2, label):\n",
    "        euclidean_distance = F.pairwise_distance(output1, output2, keepdim = True)\n",
    "        loss_contrastive = torch.mean((1-label) * torch.pow(euclidean_distance, 2) +\n",
    "                                      (label) * torch.pow(torch.clamp(self.margin - euclidean_distance, min=0.0), 2))\n",
    "        return loss_contrastive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Konfiguration\n",
    "\n",
    "Auswahl von: \n",
    "* Datensatz\n",
    "* Netzwerk\n",
    "* Trainshare\n",
    "* Batch Size Train\n",
    "* Batch Size Test\n",
    "* Learning Rate\n",
    "* Momentum\n",
    "\n",
    "Hinweis: Konfiguration ausführen, Auswahl festlegen und weiter zum nächsten Abschnitt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97cf4f37c7ed4036a2879dbb139818c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Dataset:', index=1, options=('CP', 'AMI', 'AWE', 'EarVN', 'UERC'), value='AMI')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b5002f1761e64a5bbbaec4ecd034c112",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Network:', options=('vgg11', 'vgg11bn', 'resnet18', 'resnet34', 'alexnet', 'squeezenet1_…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0eb51fd4b5f744e98edad81c104e7f43",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedFloatText(value=0.8, description='Trainshare:', max=1.0, step=0.1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48e119312d5e429ea553518c6d3df1aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedIntText(value=4, description='Batch_Train:', min=1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "21770422b0304a63907e69a750f6bea2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedIntText(value=4, description='Batch_Test:', min=1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "14af8e2f1cc0414d8ce34c0bd93ddd7c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedFloatText(value=0.001, description='Learning_Rate', max=1.0, step=0.0001)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d8b9466f0d994762bd04d858d213ce0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BoundedFloatText(value=0.8, description='Momentum: ', max=1.0, step=0.1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "datadir_choose = wg.Dropdown(\n",
    "     options=['CP', 'AMI', 'AWE', 'EarVN', 'UERC'],\n",
    "     value='AMI',\n",
    "     description='Dataset:',\n",
    "     disabled=False,\n",
    "     button_style=''\n",
    ")\n",
    "\n",
    "network_string_choose = wg.Dropdown(\n",
    "     options=['vgg11','vgg11bn','resnet18','resnet34','alexnet','squeezenet1_0','GoogLeNet','shufflenet_v2_x0_5','resnext101_32x8d', 'siamese'],\n",
    "     value='vgg11',\n",
    "     description='Network:',\n",
    "     disabled=False,\n",
    "     button_style=''\n",
    ")\n",
    "\n",
    "trainshare_choose = wg.BoundedFloatText(\n",
    "     value=0.8,\n",
    "     min=0,\n",
    "     max=1,\n",
    "     step=0.1,\n",
    "     description='Trainshare:',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "batch_train = wg.BoundedIntText(\n",
    "     value=4,\n",
    "     min=1,\n",
    "     max=100,\n",
    "     step=1,\n",
    "     description='Batch_Train:',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "batch_test = wg.BoundedIntText(\n",
    "     value=4,\n",
    "     min=1,\n",
    "     max=100,\n",
    "     step=1,\n",
    "     description='Batch_Test:',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "learning_rate_choose = wg.BoundedFloatText(\n",
    "     value=0.001,\n",
    "     min=0,\n",
    "     max=1,\n",
    "     step=0.0001,\n",
    "     description='Learning_Rate',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "momentum_choose = wg.BoundedFloatText(\n",
    "     value=0.8,\n",
    "     min=0,\n",
    "     max=1,\n",
    "     step=0.1,\n",
    "     description='Momentum: ',\n",
    "     disabled=False\n",
    ")\n",
    "\n",
    "\n",
    "display(datadir_choose, network_string_choose, trainshare_choose, batch_train, batch_test, learning_rate_choose, momentum_choose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Define Variables, Arrays for Dataset and Categories ##\n",
    "dataset_train = []\n",
    "dataset_test = []\n",
    "CATEGORIES = []\n",
    "#img_array = []\n",
    "network_name = 'nix'\n",
    "\n",
    "datadir = datadir_choose.value\n",
    "network_string = network_string_choose.value\n",
    "train_share = trainshare_choose.value\n",
    "batch_size_train = batch_train.value\n",
    "batch_size_test = batch_test.value\n",
    "learning_rate = learning_rate_choose.value\n",
    "momentum = momentum_choose.value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pfade zum OneShot-Learning\n",
    "\n",
    "Pfade zu den AMI-Bildern für das Trainieren und Testen der Siamese/PyTorch-Netzwerke"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Config():\n",
    "    ## Define Pathes for Train- and Test-Images\n",
    "    path_OneShot_Train = \"./Datensaetze/AMI_OneShot/OneShot_Train/\"\n",
    "    path_OneShot_Test = \"./Datensaetze/AMI_OneShot/OneShot_Testing/\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Netzwerke und Datensätze einlesen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Get Network and Network-Name\n",
    "network, network_name = Network_Choice(network_string)\n",
    "## Define Constrative-Loss as Loss-Function\n",
    "criterion = ContrastiveLoss()\n",
    "## Define Adam as optimizer\n",
    "optimizer = optim.Adam(network.parameters(),lr = 0.0005 )\n",
    "\n",
    "## Choose Datadirectory to the Dataset ##\n",
    "DATADIR, Database = ChooseDir(datadir)\n",
    "## Read and list all Categories of a Dataset \n",
    "CATEGORIES = os.listdir(DATADIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Erstellung oder Löschen des Dataset-Folder für SiameseDataset_Ear\n",
    "\n",
    "* **Create_OneShot_File_Images():**\n",
    "Falls keine Bilder in *./Datensaetze/AMI_OneShot_Train'* oder *./Datensaetze/AMI_OneShot_Test'* vorhanden sind können diese mit *Create_OneShot_File_Images()*.  \n",
    "\n",
    "* **Delete_OneShot_File_Images():**\n",
    "Falls andere Datensätze als der AMI-Datensatz gewünscht sind, kann der AMI-Datensatz durch *Delete_OneShot_File_Images():* gelöscht werden "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create Dataset-Images for OneShot Learning\n",
    "def Create_OneShot_File_Images():\n",
    "    for category in CATEGORIES:\n",
    "            path = os.path.join(DATADIR, category)\n",
    "            class_num = CATEGORIES.index(category)+1\n",
    "            count_train_share = (len(os.listdir(path)))*train_share\n",
    "            counter = 1\n",
    "            os_listdir = os.listdir(path)\n",
    "            try:\n",
    "                ## Create Folders in Train- and Test-Directories\n",
    "                os.mkdir(Config.path_OneShot_Train+str(class_num))\n",
    "                os.mkdir(Config.path_OneShot_Test+str(class_num))\n",
    "            except OSError:\n",
    "                print (\"Creation of the directory %s failed\" % Config.path_OneShot_Train)\n",
    "            else:\n",
    "                print (\"Successfully created the directory %s \" % Config.path_OneShot_Test)\n",
    "\n",
    "            for img in os_listdir:\n",
    "                try:\n",
    "                    ## Copy all Images in Train- and Test-Directories\n",
    "                    if(counter <= count_train_share):\n",
    "                        shutil.copy(path+'/'+img, Config.path_OneShot_Train+str(class_num))\n",
    "                        counter += 1\n",
    "                    else:\n",
    "                        shutil.copy(path+'/'+img, Config.path_OneShot_Test+str(class_num))\n",
    "                except Exception as e:\n",
    "                     pass\n",
    "                    \n",
    "## Delete Dataset-Images for OneShot Learning                   \n",
    "def Delete_OneShot_File_Images():\n",
    "    for category in CATEGORIES:\n",
    "            class_num = CATEGORIES.index(category)#1\n",
    "            count_train_share = (len(os.listdir(path)))*train_share\n",
    "            counter = 1\n",
    "            os_listdir = os.listdir(path)\n",
    "            ## Delete all Directories and Images of One Shot Learning\n",
    "            try:\n",
    "                shutil.rmtree(Config.path_OneShot_Train+str(class_num))\n",
    "                shutil.rmtree(Config.path_OneShot_Test+str(class_num))\n",
    "            except OSError:\n",
    "                print (\"Deletion of the directory %s failed\" % Config.path_OneShot_Train)\n",
    "            else:\n",
    "                print (\"Successfully deleted the directory %s\" % Config.path_OneShot_Train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n",
      "Successfully created the directory ./Datensaetze/AMI_OneShot/OneShot_Testing/ \n"
     ]
    }
   ],
   "source": [
    "#Create_OneShot_File_Images()\n",
    "#Delete_OneShot_File_Images()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Klasse zur Erstellung der Bildpaare\n",
    "\n",
    "* **SiameseDataset_Ears()**: Erstellt die identische Anzahl an gleichen und ungleichen Bildpaaren und transformiert die Bilder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create Dataset of Ear-Images for OneShot-Learning\n",
    "class SiameseDataset_Ears(Dataset):\n",
    "    \n",
    "    def __init__(self,imageFolderDataset,transform=None,should_invert=True):\n",
    "        self.imageFolderDataset = imageFolderDataset    \n",
    "        self.transform = transform\n",
    "        self.should_invert = should_invert\n",
    "        \n",
    "    def __getitem__(self,index):\n",
    "        ## Get random Picture from Test-File\n",
    "        img0_tuple = random.choice(self.imageFolderDataset.imgs)\n",
    "        \n",
    "        #we need to make sure approx 50% of images are in the same class\n",
    "        should_get_same_class = random.randint(0,1) \n",
    "        if should_get_same_class:\n",
    "            while True:\n",
    "                #keep looping till the same class image is found\n",
    "                img1_tuple = random.choice(self.imageFolderDataset.imgs) \n",
    "                if img0_tuple[1]==img1_tuple[1]:\n",
    "                    break\n",
    "        else:\n",
    "            while True:\n",
    "                #keep looping till a different class image is found \n",
    "                img1_tuple = random.choice(self.imageFolderDataset.imgs) \n",
    "                if img0_tuple[1] !=img1_tuple[1]:\n",
    "                    break\n",
    "\n",
    "        ## Load images and convert to RGB if PyTorch-Network is Choosen\n",
    "        ## Load images and convert to Gray if Siamese-Network is Choosen\n",
    "        if(network_name == ('VGG11' or 'VGG11bn' or 'ResNet18' or 'ResNet34' or 'alexnet' or 'Squeezenet1_0' or 'GoogLeNet' or 'Shufflenet_v2_x0_5' or 'Resnext101_32x8d')):\n",
    "            img0 = Image.open(img0_tuple[0]).convert('RGB')\n",
    "            img1 = Image.open(img1_tuple[0]).convert('RGB')    \n",
    "        elif(network_name == ('Siamese_Network')):\n",
    "            img0 = Image.open(img0_tuple[0]).convert(\"L\")\n",
    "            img1 = Image.open(img1_tuple[0]).convert(\"L\")\n",
    "            #img0 = img0.convert(\"L\")\n",
    "            #img1 = img1.convert(\"L\")\n",
    "        else:\n",
    "            print('False Network choosen')   \n",
    "           \n",
    "        ## Invert loaded PIL images\n",
    "        if self.should_invert:\n",
    "            img0 = PIL.ImageOps.invert(img0)\n",
    "            img1 = PIL.ImageOps.invert(img1)\n",
    "        \n",
    "        ## Transform \n",
    "        if self.transform is not None:\n",
    "            img0 = self.transform(img0)\n",
    "            img1 = self.transform(img1)\n",
    "        \n",
    "        return img0, img1 , torch.from_numpy(np.array([int(img1_tuple[1]!=img0_tuple[1])],dtype=np.float32))\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.imageFolderDataset.imgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definition des Trainings- und Test-Folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Folder for Train-Images\n",
    "folder_dataset_train = dset.ImageFolder(root=Config.path_OneShot_Train)\n",
    "\n",
    "## Folder for Test-Images\n",
    "folder_dataset_test = dset.ImageFolder(root=Config.path_OneShot_Test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definition des Transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer = transforms.ToTensor()\n",
    "\n",
    "## Define Transformer for PyTorch-Networks or Siamese Network\n",
    "if(network_name == ('VGG11' or 'VGG11bn' or 'ResNet18' or 'ResNet34' or 'alexnet' or 'Squeezenet1_0' or 'GoogLeNet' or 'Shufflenet_v2_x0_5' or 'Resnext101_32x8d')):\n",
    "    ## Transformer for PyTorch-Network\n",
    "    transformer = transforms.Compose([\n",
    "    transforms.Resize(256),\n",
    "    transforms.CenterCrop(224),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.4318, 0.4660, 0.5889), (0.1752, 0.1893, 0.2096)),\n",
    "])\n",
    "\n",
    "elif(network_name == ('Siamese_Network')):\n",
    "    ## Transformer for Siamese-Network\n",
    "    transformer=transforms.Compose([transforms.Resize((100,100)),\n",
    "                                transforms.ToTensor()\n",
    "                                ])\n",
    "else:\n",
    "    print('False Network choosen')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Laden von Train- und Testdatensätzen mit Transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "## For PyTorch-Networks\n",
    "siamese_dataset_train = SiameseDataset_Ears(imageFolderDataset=folder_dataset_train,\n",
    "                                        transform=transformer,\n",
    "                                        should_invert=False)\n",
    "\n",
    "## Create Test-Dataset\n",
    "siamese_dataset_test = SiameseDataset_Ears(imageFolderDataset=folder_dataset_test,\n",
    "                                        transform=transformer,\n",
    "                                        should_invert=True)\n",
    "\n",
    "\n",
    "\n",
    "## Create Training-DataLoader\n",
    "train_dataloader = DataLoader(siamese_dataset_train, batch_size=batch_size_train, shuffle=True,)\n",
    "\n",
    "## Create Test-DataLoader\n",
    "test_dataloader = DataLoader(siamese_dataset_test, batch_size=batch_size_test, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualisierung der Bildpaare des TrainLoaders\n",
    " \n",
    "Bildpaare:  *Proband1 oben, Proband2 unten*  \n",
    "0: Gleiches Paar  \n",
    "1: Ungleiches Paar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAABiCAYAAADz0wB7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOy9f4zU55Xu+fF6quXpwpNqZilGdCO38XZAIDBrzAjsFXhkGF3jewV/GK8GZ4W9mxBNxnMje5PrzNw40WSiTH7cmyiKk1WckexIY67WRBqsO8Z3B7xjWBm8A/hiEAiHa2gLGk03ulAeU1yrWpb3j+d96rxVXdVd1d2QzCxHapqu+v58f5z3nOc857y3fPLJJ9yUm3JTbspNuTHy3/2qH+Cm3JSbclP+/yQ3le5NuSk35abcQLmpdG/KTbkpN+UGyk2le1Nuyk25KTdQbirdm3JTbspNuYFyU+nelJtyU27KDZTfmOzLW2655Z81n2xz+r0e2A0c6PC8fqAKVK7HQ/0TlYeAzwPLgQtADTgLnAbmp7/7gFGgJ/1cAY4Dr93gZ+1F/QdAESgBdwMX02cX0wFVeAn4zA14pm8DS1AbbaazsbUcOHE9H2oK+SPgOf9RApYBC4BL6bMzwAiMo/7+dZEHUZ++B7xCZ23YC9yFxvN4B8d/8sknt7T77pbJeLr/nJXuPcBCNOeuEBP/HmAOMICUxvz0+zyNE6GczuukA66HHEeTLpcTwBrg2g1+ln7gj4G1SKk+RyxgvZM8zzeQgj4CvH2dn9GyBy0QbATmAYPpixHgg/TZz5ESuQM4rK8XAee6uE83CrEMPI3G03eavtuMxuiJLq53I6QGFIrAamAparczwCY0KarowQeBf0O9HdtqoiYpAwXULbMpD6Jx2gN8rcX3dwIrgEPoHStN33U6BiZTupNauv/cZT6yLPanv7chq/dRpNT2AmOo80FjyINgDM3Lbq3dfiYOpFaftZNRNCDZAnwK+C/pdw8s362xfhS4t8vnmoncBawCXgZ+nD57COm1RWjB6kFtCnqHfcALqK0XIgviei8Wf5Wei5Xpp4BMnU3Iut0rpfcSUB6DjWPwTDr3LLJE3+3wXt0oyCdQ++QK10ZBH1LGv07yCcAQmhBVNBGWIgv3M5obP0X9vwbgfuDLwB745GRn7djXwTHdSi+a31Xgm03fLUfKuICMrkK6fw96H+hu0Z1MbqjSLaHVY5wbayEuRg1oq6uEVrMBNKBHgMfRYLCSADX+2fQzSjS+ZSHdK91WyrWdwl2Tnq+IdOyzoAG8FI2K1UgDDwHDyDL7BawakRJb0uWzTUe2IWWxDync/vSsT6P2q6Xn70ET6Wz6/xLgLeBq+nx++ny2J1ouj5EergocSw9WA76n7xahcfEsavfR9P/7gIf6Yc8IrEuX+PGEq0vKTBwnk8k6YAfwZPbZGmBDus+J9Bx7urjm9ZQH/Z+NyLLdi8bjt+ThPIra4NnsnKNvwqo30QudgdM1fX8XWnBayfUYB+vT7yNNn+9ASncOGof70Ti9THd92ancEHjhcfTC89GLHEBW0Y3ARO9Ekxo0195Fg3oFmvjH0YDeggZAETX4fqS4TtN+hbvemNpXkTJ7i2RdDALDoRQgYWpfRHhawtB4Bzip951zHZ/vETTJ5gBPocXqCQQ1LM6Oq6B2HkVW+Pn0/9fT94tRHw0guOGt6/Csn4BWhO1ogdqpz19K72BvpoIWAuPNV9BisCN9/wNkMb+IxvVMYZGvAnPRIgVSagvT332o/1/mxuPe7eQTgK3IaroEvClnYTOoof7j7XDpQ8EJ/Whx26l5dxrYPJSOO6lLXAH6huDQJVi0CXbs1Du3MspmMt/6UcxhPzHuQGPvMdTmNTRn/grpgJnop8nghevOXngBDaANyJKoogl3o4JQi5ByXUisngNI+RcIS3IgHXuFUArHaa1wewllfg/J3Z+B9Db9/RCygOx+f4IG4XPDsirnowH0XDHd/Ido5aggq+MOoEfv9VczfLbJZD5avIqobTciF62v6birhLdQQ5MqH/hF1PbG0mdbFoPa6bOosQ/q8zHgse1Q+Bxyf9KvUYTpzUfvsyO71hPpZ3kZ3npAAdjHs+/7u3y2uTRaXuPIQFmIFqABBN38Okjd2i6gsfamFtHNAF8H3hqC1cv03TJkBe/Un1dQe3IGBdtWx0JXuA/WfQ6+v1OLS5mJcwJmZuA8jJ7V466Q7rM+/SxPz3iQ66+friu8sAcNWpCSO4IG6Y1YtRejibwKNebZ9Lk78y5CsT6KGr6GOuYImnS5219GC/wP0HsdT8c7mtnshhSYHoTyTHqu55H1uBagCIWq3mF9umfRN/nXKAB0mFC685DiPaNV/AqyPmdbqkihvpz+XoQmlhehsfT90fT3hfR/t9U6QiHfm16nj4CDZktOg9qkiCb9sD4vb0dacgFwCnhTn+9D42Y9UFiJmA2ngGtQOqkFb98YvDBWN5jZhtrjley+U42BEpoXp7PP+rLvrqI2O910XoFQEueYXmxhOvIQ6EVL1ANjq3pQQ31pFcz5bRh7Rwr3J0jppsPX5BcqIEx9KZQvQfXnMn5XIRbHkw/AnDdm77l3oLbM+2YdWuSfQPPtCmrnX8zebdvKdbN0f4QasYAa/SywixvnJg2gOTaXoCdB0FeWoMlv62sAraTn07NeaLreOqQY9iGFOx912hL0nnanS8Qq2qlcI1b9DQgyKAzp9xqAqiCZ7yCr4pkhWLQMzbRTSME+gpSHo4KluH473GwmUkYD+QRaSEvIOlvbdJxxMluPo2jhewT4czQeViCFDeqz+dN8pnWTfbkAtZfNtb8ZFKa0ACneYX18Nj3LBqCwHUE3nxsUtFPQAxbK+v6lfr3PaWJRyy3dSZ+HoM6ZvuaYh2l3C9Mtq03nldE4NmY/3fbqRurDaRzBV2fS3zXg/1wFcwrAP0J1VNbKcNMFtgKXfx/+eq6CljUUi1gGxW9C4bvw2FZ45gEov6FT3H4lMiy5S+lF1LCd2WeOO2xKf59G68Pead6jW7kuSncbeiErnnGkcF9pe8bsynKkID3Za4TStftqfPkammQXkMI9jqyx3EK5B1nGtsRqhKLuIYJBft9iOv+hDp93G1LmjwMb+hOn0dHhJOvS/flT4Jf/B/zHQbl0h9NLzAO+dKse7GLjuUW0CM6mLEFtfATNsbWoHX3bCkHHu0K00UA6dgXhhSxHj70Q9csAsQh1I6141qMQJNEKcClZ2meGNRP7kU+ZOnyMmIxsBTYvhepwcE8XpJ+ifpdXwuJSvOcIcCQNhENTPO9VYvxALDhe8M1prjWdN47apxsDppW73o3s83+qwHDm2X0d6Psd4Dfh3FuhlPMJ9E3g5a3Q969h/TJ9dzH9zENadR6KWBagukwW77MEktG88HQqa2n0FHoRDHYWjb8jaN6fYHIGUWmS77qV66J0HyUsF1ADPn89btRC7kn3vooGsFktHiSL0OR3ZLKX4JceR4Pc48VW+kKkCOagVfNA+v84mhBzCFy4SFi/85FCbScFYH+PvLN1W7T6sh4KX08PsjQdaAI/wJ+s0Jl3fha2D0oZHEQD/f2P4a9/T8dVUdAoyVpmPvFyuYraazeyxhfSmi0xmn6qqN2Nr89Pny1Cyrsv/fQwu9huGeQJlJB1VoPysvT39vTZSRrwoR4QSwSg8IFmfS8xMKwgKsii69EC+zSa4Cs+pcOmosD5cp7sCwnr1xzx00xMLBijMYDXrJSbpZyuPRNZ5QtdAkZSu64EvrAK3nwVTh4Abpd1VWPiqvnJB8AxuHRMBsU16otg3dy/BAzB2El4ZjssKSbGCdMPrnohtDgOsYTweucwkdFgMV+4guI4syGzrnTvJKLQ4ygY1IqEfL2kihp0CWrQZpigDzWyA2br099WDldojLz7nAvIM30WKd0aUjyeOJ4YcwmrzUomD8Tk8jywbjXs+DrwB7B5O3JjK8SABE1yy5njKCQMjP9X4WeX0ovXgJF34G+36PtL1JX1CmY3s2oJUgwj6D3X0thuJYKbO0ojg2QRkZjioOW9xNy7nI7pNijVVq6hdn0n/X0SNeE4Iu5nfKy6sn8T9cFfjsD76ZiT6bsiwYMrIYW8WpDD4u3w8pnG4NpkMpr93+9eIyh29xJUp3bSLturhNpwPrMEQTQDxxeByjkYnAvf+hgOf6g2rhDwA2gcHv5b/X/vh2r7N4nMv2PpmHnIe3gOWA0D22cGjd3JxPnvxWcJMeev0n6BHKM7r7UTmfVA2ufRYKkieHGqAE4h/cwWMb6HcClsPeUD2zCDcduvENavv7+KFEgPcjtebvF8rVw/00560OStERZ/c7Djp8DjX0eTd166wTJixQdZUQaJ08vdcg988rnn4PmlcOxDHT8PYburgeJvwrlD8Dmk7R8A3tDpufcxU7lAuHzrUZtXyNywIhyqqj0PEW3lH1NzjLUdIXTZfURCxVxmgZZXQEqgEIlSDQ87SB2DtMEwAPAZIQy7vpxdqz9db5i4WMKLi/cDG+GxIuz5ydSPVSYSRkBjysHIIoK53uvg9c5P8t0cBE056Gna5LSkRKOfPQbsuazP9iPttBQtUDYUetJ3JeDgVxUdXo3ab5g606bBwJgH/AWwfuYeTx7gXoOgrCWEjbKIqY3CxUTQcjZk1i3dFYQV+f0pjrUuyRVaP2qcZmuzE+knkhvmo8YtEMwFCNfVabSrEF51gqCzmRjvyd5qQRhtui4Eyd+43Fw0Ducg1oPlTmDHSjTZNyBLyT/GZ6vIfSsSlhpS1gM/Ax4+FTSxU9l5h0fg8Kg+30hd4RaYvfx3R8/9Lj0EFGM5UY32fA096muIsrMTBQV3ZtdzW1qPmUa2mOnjaQ3BzBqNfuYDQO+t6pihxvPs2fQAu4ZQW/4pAnsLSMkah1yQfg8h6+0asF4W2lS4tLNlLUU0HovIaDD0cnziqQ3Sanz2p/ewQVFEwwGmDvC1lWaKziDq2Cqa7J9FUNcQWq22IuZHEdECnkp85B6EpS1N39l/H08v48BAPyz60+k+7EQc2P16moAFn2k+qYWsYuo+6EZm1dItEaD1fiYH+vuRUjqN2tw4qF3zK+mzPXROvTJWaCUwSmNyQDl9b3rIo+nY0wSma2t0MsvbCucqEQzJrV6vzl5JHbW37PMMqKaXKyEXdhwp3hFkDXj1L1CP+O5A7TS2B8obkSJIVhwlIkBhaMEaqzJzPrFlHA3eUcJymNt0jJNLfpydk8s2pGiriJHhdFfPN+OboPd9ne7FeCDD8QD1+OKCW4GP9cHeCPwdR7riMZKCKCLl4OBkL2HdjhOBoBLC0nqBMqxYTZ1W1U5qNI4Lw2EXCGvXCRrdSJmoL2BvAWQ5+5rdSt0xuNTiyzNoQr9PeBWGGM4I5r0APFWE76RCQtxBBBnchhAr1XPpuH0ywpqD253I1aa/zVrywtbD1On3j6BxOpsp6rOqdCtooTvC1GmLxj43oYawInMRmT7UOJuQ0rrC5OyHHWgAFwmL09lEdtGsDE+n75wldxwpiXYdUELKcxMKxC4mlPk46lx3oif1XELxz6fRtX+3Cov2Al9Ak7lAFFfxRDbAWcp+RvR7VQVZaAXkhnnw7iVqCowgi8PW867ZSwt2YoiDlQUaYZYK8h6a+2s5arv7iEj9caRgnWZdIPrOOOQc1BTdBFPWIKirLvOIDt4EjH+sz97QR8cJhkeDB/MIWgCPpRdbkD4voz7qJfqqmv4egpcOQ3Ur9Oxq/4xW9BZXYxulcewsYXIDJucDGwe/izAIfJ0V6f/NOGcn8jmkPBvMd8MEoLG2m4aJvx6tQ1tLCDrrhcJ+ZP3msYhCevBK+v8xNI4rusi+7ylN+sUunjevZ1JAFu0cghLqIPBkspyJfP120g1XetYx3X1oQrZ6gF4C97yMBsQRZFmYz3mcxuBUH+o8B6X20jr10i7Ao+lvK4JCuucIUnyX0YQ2ZWmyVN9eoijOJjToTB8zDa0vXf8qwWLwc1v5v0ejRbMbeMi0BpfCu4RGyimCE/opNPhyImcvEawoIT/9HeB7yLLwYCW7ZiIgWok9w8RqVt3IFQK7NsfWVkUZYbXNdKkCau/cC6iiTKFFNOK2tki8mLm/oHPF+1wJFm9DJH1o5I1uQYPsFHWM40fpWR7fRgr8IAji3y6FkbMw/JGUxcp07hmksI1BLkCKp3wbDH/EU8/R2ipsEo9NCM/pPBo7rg8yFV0qtwBt3R5CyJXlCGrTItOrJ1BPGsjf6SJ65x7UXu/EV08D+79ABIW3pmN/iNp1QTre8APEiw4h78KWM/BsGV7s4sFz72ActcccFMM5nf6eCqPtJpbQTXLKrCtdt3Er8cQ53XTMAYSBrdkO/FwKajdhjS5Hg3B5EQaqwrlGabSkjqbjvLq79OJRNFFN/Sqi1XdLOs7fN8tiwhoz7QzCOnbQw9QTLwhz0r2d8VZDVq+VzTrgpw8QrPdTaNaZm3I3UczmDMF5tJU1hCb+ELDx92Dv38F3fwQbvgxPfqSw+f0oSy2PICdZw8wtXtNsXB3MFCdDBPtonNiL0aLVQ/CdQfPQllghnWtYYRGxUNmrcJGcTmSVXX0r3XzAFdJ339Cfu1C/Pv5ltNBdQgvfZ4G/PSXl0otM9CGg51YY+ViDahwNkKF03vsf6foOik4hOYfUFugWZDDmJQbbSXOBnXNEAZc8aNyXXXO6cgJYPpx9MIzaa176fyLVvIYSX1iA2u6L82HlUjj1d2po83TvTufmq4bbzGP3GBQ/Bzt+1t2z5nDAg+i916M5epzrU9+jU7lhO0eU0CR6i4lK+fn0He/DmiEpKLMO3iOSK8bH4cmhGFQ/Qt4fRMDAVayMF3riNpfIM22sXXTY2LB5uy8hnM9JAKsQLGt6msn/rqIG4X7bul4D7PKqbs1j7DVxOykihQsaOcPAnyEL4YcIvymQIIheePgp4Ldh9b/SQD5GDFhb0AlbtPUz01KBY+kxFxN1Abw4uVCMxZjvWkS3c2af2xGize5N37sAji29GlLWZdT3k/GNF5MCuLuAi0mRDTYdZAWZ2vxl4MkvAH/ycGBEj6D2fwVZw+8gr2H8VuhZGRbuEIGn+/8XgRfh2X85yYMmqdKYNg1atKpoDE+FY7Yy/o6n852w4VKaK4BX0zHT4Zy+5P94fCa4q44rEqyA4jfTTbffCr//WRj/R3WcYZ69qG1NkO1HENsg0Qc9aCwvhX0zyE7wQm7dMN1EC4unbjn99NNdvOSGKd0KWoUXo/HczHv7BtTdY09gN5Bd07M1ePcMLOmR0bGEiZQ05/4vohFHdaDrUHZNK+ZmuYcoSTia7vMk6jxTeuyq2Xp20KdIcHjvQmPKjIangfKXkAXlaN/qdFKZGMyDt0P/fCmGYVnjT/pZd6LB/AvgpVeB3wb+KxTHg65zTOc1k9SPIyV0laZc+GnIfGTBe3Kbs2xPw1JLz+4sLy+Otoj3IE/HfTw/O8aB0EVIefcQVcyaxWjNE2iRYyVwLHkYnrCeGXuQYuzXM2/08Zf+QVzcn6DyX08SNQS+hRTLtY/h74+qb+x9DBKm1X23SrF8Dv7865M0YBJnl1nmoDb0YmVqYjfi+XOeCOIOIA9vhKg70K3Uiyf5gQbT7zJ1+OYlEq/43/5v8M25sOVfASdhwe/Iq3sHteNO1LbDqC3HiCLy992uhhlGCSzzyNIEpydOWhqg+1IEy9EiVSY8xcWonZ1V2Y0hc0Pr6S5GLqWJ8GaSQMJpR4CL4U6ZJF8lMpgAxmuh3OYjj9o0rSXpdzH9ttXllN2FKBLeDh5yQM+TfxMR6DmOFMRBhP+5BKAVxJX0jIYSPGGqaAJs/SYxmHrQhH0TwQFjxCQeH4ee31B77NEkGQGK2wmmw7F04aGvwu/+AfTcC1t3a9eDrcgkdHAiRRXOoslm7Hkm4qI7e9DAW0QEoPJAVA1N9qME/v1qevTdNHo9mwi+dAGNlzGi77ei9n+SiZj0PmRIbSJzx51k74jVUiJadwcwrPfYQLrB/3y07iK3ZB58i+D+XUN9uR/4A6QYCrfCxY91n2Eak1omkXwsXk2PNxe1k4Nf3YiTKp4hFLfjIHem6xvW6Ka4UD2g9Knsw3E0Zj/QnxuA0v3AubNw/3bgv8HIyzByWRf4FuoXW0HHCAy3zrr5MH12G1z6qE4r+z5RArMbGSDKuL40xbG5LCcw9auE0jatcQ7TK75/QzemNFNqA+rs/5UWabLDoajmEi4rNBKl3ZAO4Fj5GRYwLcyL8lkiUFEmosTNK9RypEAKaMX2dY4iq+EbaKF4GY03p7gayjAfNoc1jA1f+A/IOrDPOI4srAXppArqwYMfwdg/wjv6+FGgZG7jIBr0S4lJf/JvgHORSGECLGgyXIo2uJye5y5mJjkh3wueoZhcqum7ven+PyUSJqpM5GM3c1s9uBen452y3Sx5FuCqLyDtuxq10xukkmHp4IOoXYoxZniKusKtLwRDwN8AL0gxXjiMrDUnR/jgU0B5MCn2uVC6NaiAHYiZgiDleBfBNYfuU3hPo3H+ecIKy9u5eZuprqGGfUSRoF4C2yXR044B/+7v0sHnYfdlGQPvkOpZAP8X8F2iluVFNLbnof5ZcDuc+UiXmAeshKde6PZBG/np8+kuOGbuvr1mU04Nf5p11S2X/IYqXQfADqDOWYxcxadQ4xwFLowEX9Ppow5k5TzORURjDhBWpTFFByAMMZxDk7cHNVy7LJ5RNMjnIAvtZYQd/3ukyzwhXyOgByvcCwSGdhdR+IZ034HNaEA5+aECPHyb/j5J0GWWAj//EHbr/CdASsS1GBxBt6Vw8EO48jrMu00KvEow4e/WcRWEpxsW2dvm/TuVE0TAy5xciCpZljHC29hDKNz5CCNfguAcaO3ymjtvpW7WQ7NUEIw74ADWwXSwZ4MDXqCXP0N9gffkqqT7lHzj7wIPr4LHH6a8JSm/GvC7q/RQpjxtBH45DPTCxcsKsjk7q0PJaxAfIsZnje751W+ld7IB4oSgB9OjrkfeUz9By5xKGnDLGkGd+wB5XvdlBxdR21z9a3jxVSnbi+m7+xC89sDvwZf/QIr2ziG11UmiPOktv6NF0zUvXqEjNkizrM1+d+sxQCzK9qDPE5TWq9k1u8n2vGFK12wAaNwVtowsuaeR2+gJ4AlmpVUjgPAamsyu0+rorFNMXV+hRiMn8TSB51op5wrC/Wur9jhRcLsVV8/BM+fLk843TewKMVCvQtS6NSPhEvDjj+AvUbR7mLCi0oT13k0ALOiHbbfLAhtK5yTskr3DcPgjzQ5bwSPRgOcJr84wyExkjFC6VojniSpiuVTQQnsatekqNOlN2+tBO8uWYQKA6aytGhEgapdRVfM/y4jcck/Ui6jtjZtX9XeZoPS9R6b8PgusHwI+DfxjKJlLwNgxNealdI95t8Knb4Pah7DgNg30Cgp8dij5+LqKjBL3U7eBz17UrlYIY6j9ryBIx8ybhdTzDzp6vlW0iPqP0BDZvAzBSPhfhjW2D6fPbAhwW7r7b6udRs5AKdn1BdSWfFoeyqV0jwV0VHqu1PT/DelO+5ial9tKlhDcXm9Wa9qddYsX7k7lhindIlJk3ln3PI3FxVcR+ysVaSwT6II0S9I55ZJe1AEBK9u8hOAqIghjMVRwL5ortkjdUdfScx1AONc5JuffOfliRdP9XdfBGUaQcLv9yBU10f5upDzfQKP/IlrRf0E9GmyogyFgzjqoXEt0sdtg9W0ROT+I8LJegiG/NV7AE8vKdjYKoBi+cdbUWHr3LS2OtaG5AbX5EtR/Gwk6D1C3yvMgp2EL0/HaLRi7SDc5idrBKbr+bISwPn9C3fry9e0F8OX0YV9i9L/6/8C8xPtfCZz6WNcyP3Lnx3Duo3ShcVnSl6jjnN3KOTRUjB1221dLkILpIfDaccLLg0hMOk4jn3cqqePlnpyGXIb051xQWx8kcPHBdLNPAdt/H/Z/BL/8a7j6f0cG5vCoJuD/CLz6Efzy1WCHDCJu78Gpny/3GJwoZe+3W5rYU+h9TyDdZUPjOBGYM/TTTbbcDVO6byMltz/97CEKoRSQdfQYmpC5orQy82d9wNi1cDMLqDGcXJFPVvM+LecJZoEzfgyQW7oparGIWP3GaHR9jRfnBXCoEC7vOJFrvgwpyp8jK/d71DN77M4IQ/wlfPo++PROWLYa7lgY13mHcOv2IkXeCxzTM32VCA5eQe1tt366Ys+j+bP9BJUvl/lowdtCpGOuKTYaMA5aWMwisafU6p6W86B2cjv4ArZSm0HnYf3KC6gDcoE3O3n2HzX5X0mMm3nAHbepD4+law+jfvwZ6uNNwBcH0z4205N3CYjBu2l0Kt5MtTkr1FZaDfVDDRk6U9X9tdjbAPT+9hqGEU1ndWZpjmUnDQF3z4VHvwl9n4LtCZ75i1NaxObdKuyuCjBfF/n0w4kdcavOP0NHwGm+QF1BCvMy0/PsHIQ8QXhrhm68tc8P6L4g0w3FdDcRyUAvpZ8XCIupDJQ2QrFHK6YJ+E67tcKkFviui2BfJrLCzCow1utJfY3ADctEnnseY+lWKsSz+HnsDjYzBMZdbWkpkZFzjLpyrNdRKMuD/Up6vgLp878/Cp/8A2q534DBlREhr6KopDPVeqhnCB1K725Xybhzt1SkZvGAnk8YIaaM5dv2WDyIHRwbA8arOvYlwqs4nb5zuxbShbyPXbu+6oPGPP4zCG8cTLr2jfS5X/wYcH/U/IB0w38D7LwMb/wn4Ly8Dz/cMFD5KDDKIdLCebuURAl9NzI8/UGVxOVRDeN0KhX0yM3KwAEh14WGRoNmKrGRdAKk0d2OFWTVDjWdMIIWvwLQ81vAO/Cfd8ErR+HihzIyzgD7P1abnQF+OZrG9D9A3xoofByL292TP59zASzXiLHUvOXRVGLKXnMbevHvQQbidAyXG0YZ+z6yrspIyW0hah6cRsq1DFRPaYvm02iS5Zho7hoVCJdznDD9rWxdRSgPZoGUxH6k1J3RZiv7xRbPXSAs2rzoilOavXWPlZlTgv0MLkS9HiisRwPrILKm+mnc2yfxdvcdzuCv9Jybj6XzfnIGxlP2w+r0Mqmea3R6sX0AACAASURBVF0x7E1/H5Z18yyRcGDGhQOViU02bdmX3u1oendTnOYjt9Vuci1rI291bpz+XqR0HbF3P9fZKslMeZf2Ft830rUYIgoJOfFkQdJ/bp8hojbum/q1PH1dqqI+6gEWfASH34I98OKw4JBFzxBRN0iV3YCTHyYc/eO4d4eUsXZii/8QGqP7mVnhFdP8VhClM+v77XUou0m86RpqQ1uhI2iB8yLkQTWMrNjxYbh7OHYI3gVnR2DRT4jU9aWoLR+7HcZSxMZlIovU+6qduHZFLuaAdyMFZP+4GFcuhmueT78nS9RpJzdE6XoFMi5bKEFfJVx8u0JV4PxIBMacslglMshyxWvrtpB9ZvdpI8HJy1MlTTvqQ6vYKBqAdmGbOYsFQnHn1zH+20fUi7ByWZue0/xeVxqbfzFZbnbLKghKeB4NqH7dbMNh3es76d4NgaktaNW/hjCyIpGCeYpwq9NK4zTnh1Ib5gHHRUS8bbpyFFmKLmRuhevn3kjAOnOJkp+fR0rVKdVrif3JvAgb0zR9ZTIX+9sICq/ugeI2NFEdPCsSNXOz1OgxsuBdL5QqRKT2JFpRlgL3weMleOUNeGVEAd/6ZLQpmpQ7l3QtzhDWcBeSj7G30dgyLnkvrbcksnSygNowcR8VmLqgjsX8+npgr4KMB7fpm8T4HEeTOnHv+Tn1XYBZAKyERXtkzZ89ButK6QXeAVZ+qOsNEenyBaasam9DolkuMHGH6slkbXr0q7SHG11yoEb3C+ENUbpLiIHzHrCwElSWIlrBFyJY6CiakLaEoLH2gd1ip/w68ryQ4DdeQFQvB+TWElatYdUqGuBvEUG3JUxUum5Q4745kX0X4SafJQazsdwakbECcLUCV8/AwDjhitqk9Wzrj/e0tcoDxF7ReQ9bY+auryVxTp8h+IUeJMbJbYXPRMbTNcqoGt8OYtPOq+j9NxC1TM+nz14irN4+QjH7s7y0oxv9BI1sk1yukRbIZUTltkFiUVotY6twhjqNo+wB5jC3OdMVItz9RtxjM2LZOMMLkGJYAPTfDnwoL2MIDeqcRtWhNCft/AC5sG7Pe1D7vsZEI8HB5XaKuZLOuxPpxrNo+E1VEdByAMUGTPtcBxMXlsQKYSna1NPj8300YX6WHVuCWkXjZx0EPFFDyneEsKJ83TZSQm3UHPjuR311iNiYoEDrolm5tMNpH0R9ZCrrdObPDcF0zWmz8jyLFOyrRFTwLeS+ryAm1gARJ7IB4pe18jDJP9+hAAIHs7ufixvNFqQZFEVSgXEUCNqMOuptworO5TyxQ4UVmyOmK4jAj2sD16ow5w6iBOA8pEiPIcvKCnVbuunXUfnHR4iKMb3ImjqJLNv+dJ0HsgdLmOYraKV2ei1MpCDNtA4DxC4SA8QEdoDSWT0XUFtuSJ9vSufsRArYZHPzIusZzPen97l/6ij+KETgxoqvRJ3y0AAx5BPYmm4B6rRl6UE/N/Eefw4M9BNZPcPp+v/5w8gSvJResBvGfCbNWPjbNMYKakhJPUlGtUN9PcrkHsGdRCnnhaiJml3ydtKLvJUlZDjwMRrHXiH9vE/sf7YSLUBfQG0L9XZ21mc9OWgeYn1U0dh2fOIMk1IEbEw1u/tmEJmeuBVZ7JNJu/3SHP8uE/v5TcZuaic3ROmaUXCWCGydJqykvC3LxIv5b9PArHjnEORvDxivYFeyzxxlbK6sdILAjO9Bq7+VhPP8H0ZKeSta3V4kLDqLaWdeDC4TAT/XDXB22lVg/jjUzqQTL6Ieu5ga530iwDZEEMoHiVKCw+lcK4cCUSTkDSKiXNGln0nv4mpgc9IzmsfcbYCmnZgN4QWvysSBnSsCQy/eGWI92r/NNEG3JxCwy5uTu4jfRgvwiTNEJGk1spZMq8stqWaFWEJpp/OoZ0DV6xpbVirIC6jNN6Z7eePL4XROAbnTuyZ54EnEi38u3u3gAlojbJjci5JnHNB5l9ZpvYZD8gSj9wgKVCdyDSUKWZHV73OQcP33oLa4j6iPa8tpNUqZfgAtWncDX4CBr6eXGiTKl5q3Pob6sEMfvvkwB28NZblU62RIRatbFQjFbm92FOmGboNpN4y9cIXY4tz47XEaMagCmjgeUFUa8VyTkvcTeLC5vucJZkLOmQU1UvPqfwUNnC1ogO8mLFJTq9aiQf0ZZPWeI4yicnYdM5HuJTAyJ4BALBjjpbTbg/mjEFzEzai3t6X/D6fv7yB63PSa8u2xr1raBoUjaKAnbuRX0aRYiwbc8qz9bJmT2mwKqGxKGUPeuDN3jhDKHQLvNgyzG3klA0SVMheWh6C1jYMm4UlgMBRHu8pay4dg+QNElliBSIh4gHhR04/yaPtPUFnGYWLVvkRjVtk8ZJZtT9fcmH4uor7oIRbTKhOj+V1Ic3ZeH1EY3wulDY3laOGaLKhjw6aC2qpMGCjdYJLfISChOq59B1Kgg+nvJwmr34GOBbdqbA4iL8KA/XbUxp9L11iAJvinCAPEeHmGx7eSZq+tl+CG96CFvl0Cz1TSR0A7rpXt4P5UUEWz3BClO4om1QVkZZ5F0NcVmjiaG7WS2ir22M9xSLMTQA2Z78hrfNdwhHGwViu5eZDGEMeJQseup9BDDOxmCpQnvl1313VwJpEDaANIGV8BTp+EC2fQ4JlHvYoY40hhHiN4pvOIXQlWI61WAeb994qUHyNI+P3AH1KfPW+hbXLs2kMUjTFVzoVk7B3MVDyYvWidJaV108g3PZQ+v0ps257jzRazPy5UCFx2EjlBwnPHiXKDZjHY2h0izLxWEacfkmpfpJ+9NE50992f3CYt54AmhOV8Xzp3M9PzPdHiXqPR2vV4c3DWLBQvVpPt9txs1Pel8x6mtVU9mRxH8IqNmBOgNjE1zAvND4n3/wB45WN5Y3uRV+cC5e+knz8Efndu7D13imjbUzTtetpanHCTU0THUFe5BIF3Iemmylo/kRlqZT2AhtB0Nk29IUq3QgyQ82jSGWrIoYXi3nDZnYJrvNWZXQ7IWaG6KI6tyiKRE32cCLq1cqMNV7hQumlN+4jATiE7rhlT3JbOXYQsdLvUDuo50ns6nbtmI/TY+nJkfRxN0goaXO8ji6CIGqOGBvR60qAdlet8hih4PpSu8YYu5+3QnbHnSm1Wiv4xzDDVFt+dyDlkwZp4b2/BmXmH0vd5ZpSLFFnx2hq21DH0nNne07oWwTofdzeRLGK3FtROhmUgUni3EhbwGyirr0JsPul7fw5ZZBtQ5asCssgWEEVfbL1tS9efBnvB8i5qk1whOgvKP3m9j1dR/7ZSoPnYzxdZ08a6KahTBd4dUbPOJYodqegPMhBWovG5B43lGvDIbfIS9qFg2meJso1LkWL++8tR/aeA2vRi+t5UwEmkQFS8A3XfY+n9BojU/DrfvwPpRwua61gcIWov9Kbrdksbu2Hwwjk0kN5Gq4MDkh7vBRr3MDtAZK8dT9/PSb/t2tgKtcXqZAeICe38dbN6cnmLUOAriOI1nk/O9TfdLS+Sk8MbjmBa8deyH8gqpY0nhWHgskIURhknTeh0kq2zi9lnZ9KD2U2z7KZewWYxUSFtLUF3M8Th4icOWF1Nzz9TiAHCk4FokwG0yD5H0MMMKeQY/Bxis8v6Qjyo83OD8UKtdTxlvi/4DgErGJapoLYrIkwR1LEj6WaD2YWG0zWGkSKBCIo5sHMYAeZ7CMu6Nzv/FI2lw6YpbxFjHBr321uF+jePH+T5CpZeGt3fcQJT9xzsxtNxHxaGIkbZA9GeJTSel6UHcvzi/Y8iuPbddDEHkCsEy2E3URnc5+6nIZvQi25zwp8hS/PPPZ+9wJtXb/piJ91jAoVhTLOqHJzbR7R5gdYGQbPc0Iy0XEYIjByiahUo/uC9y+whGkowTGTX38kReR1WKz8rY2MwrVb0o8hIXEHgn4YYzmY/Zl54AgwQrp0x5R7C0iZ7tgukQiDQuENvHkwbTJ/nytiutcGp99Ox9xGZT5n7u4TYrmUTkRtuT8CWkmGF8+lSNWbG1bW8SyS75B5JjahDatjAfegtjxYRbJA61zItNm6ud7Pzm6UPqH6A2qqCMMH3qe/OyzwCd7w/O/ENIrnErvEuBIrfhyzcQTI6BRowp4hI/UXCBMqgkAudVJKZREronZekVziXbmMu8xwEKZsqWaM15bFZudgwGU+P3yl7gXTsEke7B/VZnVs8TDBqtqff30Lt9WfIMPgpaufx9HBlooMvAf8hXWyYCFIOoXF+R2SKLUcOX67kDAuuJTahvEy0DURdFGdpTiUPIcW6Nt0zp65WiboV/TTh3JPIr0zpWvyQ5sj2okYaI2AJ/3hFdiaNJ6gnuBWlseK8UpODac2N4kFqRWkCuAN2vrczW0wasGJ4OZ27kXDjzel11NQDZewYVHJFaUywQFRUMn7lCexJPZz+b5x3Pwr+HNM73IIG1jaE7bl6l933jUT01jvD3kvscDFNdtMEeQvNLbM6IGou2Pr1FuNelGxJmOlxFH1YbXInjSO2sszWp+uYeF8vjA2ypt4nrOB7CX50hcjga26EO4jyjcYsnfQwTGwaand4AaGIT8H8GeI2FWIXk3XEVuQniNrBZgJ5zOVGN6jNWkX0q2iezaXzIuagtP3RCmqva+mGxUz5vUFYvF7EfkbQI4fTMSfT//dmDw5q38H0/17U1nek44t69wMEC2pt9r5XkBI0N34usZHBHqJS2Hk6x2Lno7igjbwraOq5PvdrhKFuI3KqOhm/cqXb7Cpeo9GdvIoazHnorhjkAjcQuJRpUKaiXUYdc5nINW+1KaM70S5XmYD+DHHYYu5J3xu3nUPUP3WWlzNgbJU5AHi6AiUPyB40kXMC8Ps0uqZWvlU0EAfTC+2lXqfuCWLl/woi73thKRK8YSeKGFawtZvznmdLdiLrwAGzHGf39td+ZUM4fgYXRaLaPj21mTr2VWDRSoK6BY1lHfPAmnmjm2ksJGsYIpf/kn6bJ+qAkQOgXjyvEWbjYeoW2vGDbV6gC7lAbDJpbvluwgjxApT3q+eFx8FmGrFeGyqmUj7e4bMUUIGcopXhemKj1Lyz9qBBYAMBqG9GOYzay8FiX2OcwH/tLbhNbT0fDhhsXfo6h/zOEXVctqbPjtOYzHCBxnT+qcQGXX0OEwFyS7uEnXZyQ7frma44o8lR1yJqTBcydwCsh9iWJ8+6cr97XJRpbPgxgvdrCMJZLFYIawlaWp6KvATRzlzJyGyIIuHiGE9aM5Q9XBUNvtWEdhwhqvEPEivSEBrIb1A3w98ialnsIJRvO4vVt12EdILpLvsJS/Q9ZgdmAPGa3W45plwk+NVuM6cmu5/MfFhBY01iyFJ3k3wjHXNlGIpOP12QLjaE2tIcaBBkMA8o3AY9H8VmnjuZqOVHgP+BwIPfp16AfhwoDKfPDAn1EgXmd8Kq+5iSdTGVjKFx+DCxwelypHjXEmnwV5p+CkTpTBeAckqr062NBXda3NsJQIs2ojaxJ9KLlKcLbVTQWDVkc4lIFzYWYS/Pgc9jiPFQSsf1EwvhA9QZP2M7lStkiORCegdb864NZyjQWbDj6Z07tepLxFZdXgfOEgFMZ7guzn772vl2Va3k117pOvLvgIJhANc+NVaZT1onPlhnOVDTQ0AOzSmUe4mt3h1dv4uoq2DF6ToNF4hNKg1BWHlUs88hy4pLmVFjI1A2SOzRMYIG5jAyE216emnPtOF3kFW7DkVnHyOR9icxV9OtgYCVT6NJaQx2srz+6cg+1O6GOozvrkjP8iphJbguhJWMXedcwVoR597RHhLxfR6R9vspovj1A4RlOkjQ9fgoGCHH0sWao+OXUFT9DmT1OiBXU5stdyKFgw5VXavyEygZd58FGUH6bBFaHI+k306D91hzvVinu5pTmu9fB7HYGc7pNJC2BMUPV+1KOLMZHg4uLkDtZQrKYYSfLyUw2tVEMf9hItPMQcsH0nf2TECTcx7ajh3NPRejGaPRirdRVkQBxBxqaTe+7QjlspYoDGRmjff3y8ffu0xMvW4V6M3l117p5uwBK1jXAvXAepvIT88HG0QWVu5C22XI5RqKn2wlqDAnCEUKYSlCUMxyV97RZU+C3GXuAw68CStKyjevp919gKwBww2+kSVTtn1ocCxGLvV6sgLUHeADNuTGiUisF7HT6Z2mwztsJ+8S/Zb3mSf7BWTBuSaxceB8UcvFKd+u//oIauuHgf1nYIOt2wpqmDME73mYgGqOETmdf0pgWm803dBVr5YSwF3671Fg+SZC6Q5Tn+GlzcBFGJ8FeMHyLuqf+WihPEKMd0M2ICjhKmrvASIbMbdmzVG1a95p/YDXkVc1CqwbJOhcFWQWDqcHOkgo3jeJQuRD2cXOoLY1DHQmFU/aTMQuBtNxx5AHsTJce8MpDnLbij9PlDDtTZefrEa2CS65lJH3mq/Bfci7eJ0gV3hqWuFaeU+F6f7aK90KsUq7epcl37zvLJFaWkDzwPV1F6bvvY13IZ03TqOSeRd1vDmshgicquwAhI2anPNoiyFPWc6t9LPp+S9XYNEQQQkbI9JS+9Hg3Q8XqloEXiaU1ecJKljdiCoS1ZisPMx8aKOITc+zK2p3fjp7SE0lLxKL4L3pcU8Q8IYz5YzVm/KTp25XUF/bKvZ2PfehCTBQhrNmwQ+j2fYBUWi8H2mqU9kFi6Q9uG6DL32kiT9EVkyb2PkgW/jeQtbOn0NkFv4lEWSDOtWpMEhY0bMgv0Bj1PUrThBt10fEZO8iFK6ht2bFmhduv0JnWVUOJK8HqhUoXiKgnIuonasEdxnCmr2E2nNv+nsjDbj4GKnP82AlREAuBTLXpnd5lDByXiAMhnNoDC0hPKrzNAYex5H3tYRgIL1Eo2dsi9msqSMEpOHtuB4ikrjOEdDkVPbPLZ988kn7L2+5pf2Xv0biydiJlbYuHW+K2AWUtz9Kd1Hcm9Iof4QsTtdTqCAPI2/T08gIOkp3O3S0k81ERNr0sx6kFFy/wVmo9kRsHZbTuY82XbOAlJpdeRc494SypfcVAs46CHxzFt4HpKs+n+6xc5aueVMCgjD8Zyrq9ZJPPvnklnbf3XBLtxV+MlN5j0ZaTC/tOXhLiD26nLXm6vKdSKuauzdSepExvKvN98+hHZZvpPwVUnQ5jm4vw/jiBQLKeAJBBJ3UcJ1MXJzI9/QzQEAYRaLOgKENJ4y8lI41bmwo6QIRJLWHUu4BemF3Rd+vQtbuerS4bCaCK9OVfkT3280/PQNgOcG1ttfXTU2H6ynLiWSq88y+/ulWbojSXUxUJtyLBuohut8orp1cA7b1w86RALXrhO1M7iGYBd577Eg67hFkKU812IuEF9Vt59k9az6vE0W+BrXXCFDqR1jaYPrAacUX4cmT4hW2XWZnWf6Ixtq3hdTwA8SOzS8j1xjUBnbT/E7TFVc2s3IdIBSqLZo8E89QkCPRV5GCK6JFzFsPnSBqG9QzvErUB9QAUrgvpOMdMZ+pLEHt809J4S4n6o840G0e9mx4MzOVO4lA4iizx86ZiVxXpbsO8UaN6dWQ65YH52dDliOFu60HdqYLe9Ln4sDRVmLfJEMSxo3bWeIGzn2Ns8TknEzuJAafodZmaTfJSgS2+TQp7XE7UrDjRLaOOb0fpINPwidosF3Pgf8gkT7tJKVCavQDKPmo2UUeJxTtZmYWvHOqdV4ZzljcKMHYyI+zMrZCPYIWjU3A5tVw9LAYEa4PYpZA2Vy19H7eimg36psak3tYnci9XF+X93rIZmREuaCVWSi/DguH4wX2sDr1Zq+3XDel+1UC93KEv4IsXA/82ZLFg3BiGPbXYl40BwbuQYNhIxFA2o8Gh5MEbCE1XDt7hxE0cU09qzL54CoTxTLeovv9yJ5A8Mfr6VkB9v08Fq35b+iY8lakNS6hGVAC3pSyuJ4Wr7nBZh1cQFDPbsKynUxeQRjpdF0+BzStcB0gdW1k471ziACdz9lDBFt8/vhhWW0uQekiTa4YVzY/l9hp5Gj6cTR9uovcunSP/VMdmMlMlfxM5WXCgHEq/H5+9e67xcHCK8iL+XVZ0K5LRtoOhCvmFKWjyPL5PnLLZnMlXL8atg1JobWzoJ3f7/TTGkGzKRBFePJBbDggl1UEn7ddxhRoQixHcMp8lJ7bTuE2F8l4CJW4PIsCNI5mPosU7iLga8Azg1A2wPvDdJALvSR6w8uTPONM5I8Iloez8B5CmGQ7hbsZ+OTt29ldFg78II31YbuVvuy3qT9lgvDvxd7FkHKr9zWCe+2aGxZHtp3s4oIpbItrOnDnWhObmHpni6nexZgzdFaMpZvqYLMtTyFaVQXNo6NE9dFfB+klmEQOqHYi/VMfMmOZdaX7DMIUPSFqSMneiyzfd5n9jjk9DGffb825sywh0gadoXMauf8uvt0spozliQV5ZlU7KaEB2YcU0NNMrlhyzvCD6dgnUNKDmRkX0jM6nf88RPGbr6PMnj0EHzKNnq10vgdWN7IEWThVYqfn9cDoRv39CMJsS+kddqB2OHHPhywZ0/n/O/HunVRnapYVyM138RHT6Jx3AgEleCslCHdzN1EqARoVqgn25rIOgBa3UlxjIepnL+StxlCnYkvZspagxrWTmSj5mYh5rLWeKIX6PL8+7jvIeLqL8Hw60Tm93BjMd1aV7mKinKCtx6cRBeZ6yo8PA72Tuw81IstkIbJAryGrZhUaSM051LlyNeZ7gHBRe5goxmF7EJTxKoJU7mpxrGUcWY6gtnoOKN4vhen7jyIl/NAy2LENlm9LL3UkHXR/uuGfEeTyJA8xewVtoDEhZB+yujcDf16C8hdiuyMXH9lK0LDMpV5CcElbwTqdSJkoKNRD48TKiw75Wb0bhRVyJT0/Wxqv681NvWEqpP6vAKVGq8mJMK3GQjeyh9iaKpepiPa/CtkBrOsHamrT3b/i52klaxCksITOC5b30X0dhenIrCrdLYR1WEGu7fOTHH8nsuwsMyk/+tYkS9liYvvv80jhepKsRxNrIZpohaZzqzTu4/YejUVsWol5wMaQN9HovjbLN5Bl+31gaz9R2GYwjnENWv4EeOkpeOn3tFXsQUILfPY24Rj7idqvSZ6b5P7dihVaDfWx981cvhm4FEW25yAPZxGNyQ7VHrX/HiKt2tSu6YgTVfy6rjjnZsmZDJdpVJpfAZ7fTYNfaUVtvLfBq1kd2XS5N5fvETcdcbKDx5nfabI8/uuRzNKJLATo1f0PMTlU6Dm9mMYqjtdbzK12okMn0s1uEjORWVO6OwhX6yzC7CYjjJfQoHXhmceR4nmc7idfAXhkkt70xLFVdRfSS3cSWG6RiXn985Flm7scjoC3yll3IAcad2nwvVvJ3h6ds+oBeKqETMbtxLYlJSI1BmDzUjTF/wVsGoyK68PAwY/gm78vDXiYKLWElF8327JMJt62xFsvbQR2fIH65owDG9WHTlOeTxQkKgDHa0pI+Uq6nrP2bO0211xoJ6btVbMfiAi6s+5O0FhvoBka+jxwIHWyA2OkZzeMULfEd0UBn1x555XTpiNX0YJ+nuCUOjuynZyf5LvrJXeihJJ3z6jvpwr85VtCOQZwI8SGlCvbwdRG3RTlS2ZNZk3p5tbPeaYm6PfQyDBYQVjKq1qe0V6KwC8q7XfltGvpAQ2xjY0LsBj7yaUVlckR63aWbr5xnQtztJM/Ajashh1fRxbqdiKd11XIkktbHzH7TyEi1jmoDkf1rCqy1s4cgu/+vjDeY9SVdR9a1GZDLhN7n+0g7RtVIbaDr0TFtxxSqKG2ewpBEpZa02/n17eqf5yLUzSdymyl5zWqh+iLMQIGKDJxAXocOQ4HiGw2Q1H1iZhpVbMarPhqCP6ZrnjvLgic2ayKdsriV8Fc2ELQ8hwXKaO+uoeJluw5tEgZ783njOvLXw8pEim+48gbmwpz92aT11tmTem6QvsFxFIANeg9tG7YHHT/KlEA3HsuTTbZmsWTol3+uIMULnTz0+ycfPWdKsL5bSIw4+QKw6d3pv+7YLgVjTOmmrG5vcBzzxNWbS8BCQwD12D8JOH2Jm1xy78E5u+GKy8Lz3WRnHfSwxV/C355RA9xGClfYj+02RDvBTeKFPnCHgKYTWX5Bvqjupvx/V1obXE/9aZrObnCNbHzLWnWEa5ps+RFkFys3oyD5uvkBY+Mt68jdt0xVHI8nWe82dSzVVDf2dPlKc1+8fM/3+Y5O5HzxA7S14i95VznuVuZ4U5BbcXJRS4Ofo6YXytoHzAuMTGgtQK1czsdMV0pEXCSIZgtNEJczXInsbsJaMz1o7G7mNZYbxOC17HMitI1Mf40Yiq8TmzaNlXZuDUoAA+BlS1HXvaODu8/1YrvFcwdsYpwd58htvxu5a6VgfP36/95YoQxtz4EV5h6lFcwsxU2QGPm1SPA+pXEpnymeZl4Oaj/F5YRs6dXN90DrB0DllyW5h4nFG8BODwCp9IGf9uob3KZR+lnKm7PPmS5lmrpGbwzw1JgO8xJYO0hpNCeRlDN6UG1q5X3VdQX/cildzUtF+124O1BGi1UV0qDgHGMDbt/DDW4yL3hggLq72+vhIe2CepItYY4TlTusmI33no2WQumnnkB3p/uO1332VzikXSdGlEUqFuXtxe1h9kj7TzA6YqhNS8GDkDlKFizNEN3fqaFaBwsZ2rPplOpEEXzj6L2fJ6p4Z/TxKKwitjqynPYC4OfcYTpMbFmLTnCK4rd6XsJxdYK8C+jh3+MxpJ/DjrZcLrM1ET7qbKaxomSiw6K5YraUEErS3kJsPBNDZIisYvEVaQMvBuDS0gOELxOd9JpGiGTrwGF9USPeWk27WMEKa4isU1MUqwPpfvV95FyCUO/qEf2cHrJlenaZ2YPT3NtUReXuQAMnCF2uADYBVfSIvEysej0AjuGI4Ehr4OcU7BOEHCAEzBM19uEtpjvQ4qq1Xs5HdgTbTGN15nVmgAAIABJREFUheZXoMVg7TF47pj4xVaku5CyH0ULqlOX7b28i8a2sxKdMDOHqQtYt5MaEZAqEBl0q9JzdJJ0USLGo6vsVdIz3kNnlcSmEhsuDjC7Yl2+E8V5JlKvTtO4eNiLgDAIXPXrLDPPpBwj6l/fhdrQmZOtqG05vLMDvZODsRVi6y3vQjETetysKF2v+M7QeoiwVM4TCq4XYb1HCG7swvT/MrChnFaRXhivqHzcH6djfjDJ/adKI7Xy8y4OOc7qbYDaWROulekanaPEvmLeqM7Fju8jFECZ2KWhWRb1I6vQluz7hLKFgBQWpN+fQtbwSWA1rDmMGmRoPuwZjdJ5W9Bqtx99Zmt6RH+btjbTokNXieDjcdJkGyHqys6Ds2dEJfpKdp7pdM7UW4X6whsIrgPWl2FOL8y/CIdqoZSXIC/KW96D2t7/d7t7zfFEdmUxVxfzZCkiiOGnqI7ClvQ8c9Iz/4iw5mx5+56uL2DMGNobF53KaHq3E6gp70z38q6zU9WoyBUuRPLOQ2hxme5i0EpcZL6I2uZewtNwtmaz0nW796IFzrVYXHTqLlKWJeGVwMxSxM+i8fBset5D6fqtFGYRBTLvJGCJC0jRnkt/u4reGGEYuFTk4+k9FjJ14aNZgRd6CGVWIMxyK2PLejSwH0WWhKOLL6MGKhSoB2IKlVhFtyA8dbriDvTq6p15Ha3sJHDnRrYrvIlIVjDudplIh6wSpXJdSBo0OR4fIbbhvYZqKDiqUEDK1gVtlqK6r3cj/Pc+xMmtAJdG1eNfT3+/SeyxNo/IGEi1SW31zZQ3XUOTzfvD9fjiY9T3EDtK7DkIQjo2EVmBW4iqT6+jSXoBuDAGpZWwaHVYVLaEN9EIA7kv3GwW/98ejq3S4wRDYG663iZiMb6CEns2AUfvb4x+55F6ezoOulgRjzL9VFPXfbVYaVxl6iQJCFjLi4GhiU6SebqR3OOw9e9rz0ELxGT3cp2LJcj7cEajr+t32Yp0xGKmh5vWCzChtrRH0o73/BaaiguJsZDTPEeIfRYXpWfbTFjrLyJP4vQk97DMitK1gh0jAlOO+nrF20zKYiHM+COIo/oC6oizI+k/pdi6fC56+ceQq9opzpvLWYLEbsvWYr7lZMkLjxP7s40TmVDOOrOr7aIfrqJla9D7Wi1O57zowNM+ZJG+k/7+FLEV+31I4S4gNqwsp+8/i8y0zavgS0vhsb+B/3epcltdgczbofghkrgGxUzEVpMnemmQxowDomYExMaHXysq8PYZpPiv0ggdLUd9XdkNo282Kjen3XpPOojsvPHsxwo3LzJ/BS0OryBYYjdRL2I9wZq5kn6Xe+DsmxpzAzTuvOBiOmOEC+8MuJlQuDzeLSPpfeeke02meO4hgoZW/n5m78Dx8AyeDSJQvB/NSXPVfZ/lhEcwmdL1Ma59bOvc2X8OyhnGsILuNmnB/O+FqL+fXKZ5OxUscCCdt4mJOsE7DPs5B5DB9iChaPMs0nYyK0r3HOF+eWIsoTHF00G1Q8SguBdNhAMEf5akeAtplI2iF+1BL/gssqA6Wf0thgWa9E/9c+/k20rWEFt1nE7Pv44IcDh4lgdo8q3gbanZJXlhExT/ACnTTYRl258aYZDG7cHfAb4H4zv1m73EtuB9vwOf/qpOHPy0buLK/OPp9ylkASNFuImZK90ThOs9Hxo1wjF4d1cM7mfIxkMVBlJQbzeRsNGwtXp/YK95UkMBjQMrRwjFmv9Y3M/u17yi3GsoEcULrjcbrVOKaoHxeUPHs9mPla1LBY4TdZlnIs1BJFtbB5kcDrKF5biIn+M1whOZLElpKnF5xHFis1Uzjbwges3tJJjoYLT70VmC/nGZTkNLhu+64Zkbn+9B+uXoSbVFc5JSKwV5FI3tMo00s7cJr9ZeTS393oIMy58ydZLMrChd8/d7CUjA7hGEJXCasAg+Azy+CVZ9WXtN/cgX24eU0GBj5NYrYQ1YWBQO9yM6S5O09XmcKFANatALhBtnsafvfdBeQA2e71NVJDBru1uepKNI9x2n0Vrb/0XkN21K71hBFq1B1pXpAebdDoP96s2Dcl2+4YfbgyI9+4BX/xPw18A/QPlBKev9CE5w9RfjwoSyrCKsb7ripAiv9tVjjd8vJqy+99Ixu1DCzHNVKeJniE39PLB/DAwsCMaBMVMXIC9kn0OMu1ZiJeDg6Qst3uFHqLkuE7CHxTFNL6QQu004eu8kCsNos5G339v0Ow9QTSariO1prmXPMo48SmjM/uxUjBX3IKVyiOgPzxnzmt0XkymdA4TSspJ18NnXstXvzxx07XQDTVAbOIllDM1FU1ot7QLwjjmtZ+IO2yYH5B6OK8SBPIqp+mtWlG4FTRQXmDAFy6uz3W8rqs8Aa+5HQF8BWAZPlpIiGEN1YY+Fkq0BxcEEspf0e90yrS7PdvB8pv3k2TEQBP48ctmLdOIWdP+3abQyDhEbIy7qaVwYTKOCwNdyLifL0sX2E9t5HyYoY8PpAS59qJPOwNmTUrhfA/gi2khxMLXT/o/hjV1I8QJfvFUvMoSU8jDBZUIKMFdeMxEvqD2ExVolJozbZD5aJ0wveyk9+hbEz95Co/UxdhjKg42ehwuQu9+s0CdzFT3xdyOcthlrLaO+fQlNvOM0pt0WgSf7YXdR59tVvYAWVMNUJSZO5unKSPbc19Ci5DKeU/FYX6d9YaVr6fzpBKUc3HKB9XFiPuXz6Ep6Xnsm7WQMLXariuHBQCjsKlFM3vfpYXrFfcxYAI0DxyIs7drjGqEv8kV+Oepne982CKqEFX2ZqSv7zWrtBeNdzZxXU79AA/ch0DKyH420O4DBlNFTAnaGy1F/6XnQV4TCSijNA3phYHBCrZKWMoYU4jgTI87mQzYP9udp7dJdQwrwhfRiy4mykbZ4rDDyIOIVkLtvpVjTOzFE7ExrM+UwMPwPcFgL2aOkHRm2EMtuD2q7vcCbf6unLn2sUbyLcDuuqQGqhJU23UGcy35CiZuD6Wh0D8HhtCdxhJgEW9OrPExEvy3lZdR3jXWwxjQiL45WtqZGWS5k31mB7maiwrU35r3cDA0cIu0CUSboekPwtZ6gARry8IR19brZ4kA3w1yO8ndSwexl2ltZY0zvGfPEjCtEuzrw5bR/K2BvhzWZ/BhtvLq4HIGrHAJ0H+bJLDW6Z9xcJXYFdk2YThdH3ztP1LJXmxtS64iNP08gw2KqJKRZU7o2t+cSEy3PirGyW56+44fAG2g27tMxC4FKhbpv5ElVSlyU4mrqjOTxU8A1KA519nxOW4VwHb1dc3NudicWwTf92KWI5Nu69eS00nB71Au5OupzCth4q5TvrnSCqV6/+Lge/n8MIji2AGVXLCCs41PAyC4YSddygsUl6mnC4+m9bEHMtNzjawT+Via8EiteW1xHiPZ16rUzkdaggOQAavtvgxTuymjD0ey3FzG7tY5M+/3q7mdaQffQugLWOqTMnkZexCLUl6YD1mtWbtCBpZTV5+CRn8Opuq14qdOVnD/uvd3W0lnGlveha3Xs4+i1ppP5ZTgBNCf30wgP2NL1nOrEi3oZGBvTeQ4W5klHXryvMrFIUacyks57FAVsf0rnAbnn0vE5A8e1tBcRC7VrPCxBw6WPqZNZZk3p5mmLfekh7kWTyRMIsg6ZhxTMSTQzjmny/ZTARg0tjDuq4uyrQSjMS/d1YsAUYtzJVp6t8tNICbjaWDeyB/GJGVJnGKsyTQz0Tua1Mogw3BE0uy4BP/8Y/h0CzDxKzlBfARyZZh6wbCk8NqSHLwEPIDrZm8DuUTjysSCMeUwI558nsDkX+Z6p7Ce8kZyqZJcLJlqmhpiWbwG2Btd2RfphCKhEokGBsHRMsM8tieYgzBVgrBb4datsxYXIyl6CFNqj6R5nySyhEvCnt8IpWHs4auYeQYG1AlGKcTaghVYyhqzv03TGPrhC+zTcFwlebTeSM3BMwXPCRQ2NqyOEN+IgY54F5wz33NI2tbKGPCZ/9152DceF7NJPp+yijUGIgGg7yRekV9BCvJBYUF3Q3nIIKWXHhGzlTkXPmzWl65x1l0+06e0MozE0sE+QsjyGJ17D9BFTd0jH1nqgejH9kXDRyjCMj8FAT+RJTyZ5gRIHJ7xKu8M7gSpy+QHJ2n1fKa8e8CbiL8r+3wdabk8iy7SCFO0gKkwDClP/EPgJYi0QPFRWA3waro7D6vmwfRCG5sbsP4zYDTVEPbOllnyyPTQGPmajAPZOwrr0BPT/BxCMZJfQC1090n03MB6LYB/JKv4eVA/rWk7HvUyj0jVURfq/rWAzCs6m982L6ljyovVm2qxHkIehjHfRg4/9Tx9zYXc8+3sEnm/WQ45LzoY010wo0FgPYDKxB9MK63ayznw6V15lArLpIwJGJWIeO9vL/eMxlit/x3qWE+jYi0hJOTCX/3hrHb+3s/XeY3LpJ+0jmImZHQ4ct1O6i5nYbh6DDxIxAM9nP5sTKBy/cbnLyWTWlO41ouhFngq8HnEz96Og1+PpuFdR5s9rRAeC9IRfDLLKUXkIcUgE+sJG4A511AiT55hXCd5iFQ2AK+m5XPt2fduz28urwIEaHKnGfa4Srqet67rL8QGB6R5GPT2UXnRnOmYXdUVcP+8N4MoBmPNbUP5j6FsHSz8ta7mGFPYQ6ohT6fx+3WMctfXc9N5X07s+Mo33bZb9qP/uJSaoF7OFaJK5qtgTqK2PAI/9GVST738Fufu707lnjwVGbM7mGI1UMYhJYmvbFvwR2ifT2LoFGNgWKawOFHlxrqBmfYkg7z+fnqmXwPINw8+WXKPRgHAxHph6fDrVvZVYoR2lcy/HzqU56jaG3Ab+e2G67nh2/VyBeS54kbM8hxTUCzT2rWMwTlpx+vJU7WxqWfPilGecHqe1jKZ3yhc9J0vMIRYqK37/bYMq33o+j1G0klkNpB1HjXWaKPbtjBNASmEjlMuyKrdkD+pG90paoIXCtUlxjQhCXYxzJ8sttzIoEp2/CCmMQ2glu0z3NWd3o4QNB2JsFdxFU1QegiJWSQ8yjBRkP1Ke22HfsJ7zwGFNsvo7fQC8cxk++W8IEfsd4DdD0S5D1rCJo0Xq+YiOpq5Ck8K42XRTLHPZQ2SWQaO1WUYKbguwpQgbe7SoeiF+nKDV5VmNhqmsBGsEfutti6CxvoI9GKd7tgq6rEn3dXEWdlJf3fNo9Xm0mJ5ACnwJcu8XE6VAjdxMB2ucSnL45BpRq3cqi/qvkBHTSgyndVO60AmN9lz8bH1EhN/vP0oopGa+sqGIcRqzwk4gi3cOwfnOt9IaJ6CcTryJ40Q6ci7vpXuso7XiLqDxYgjpQTRWILi6VqRz0PjdSJShHUPj3GNiqiSZWVW6Dq7YOjG+a8uCM4hDujQe2Nv71AfaSqA/MMJFEODuqXThYdRKO4FC56mXxt58L1PbTAGq0v0+VyPp0Z4nLAi7UyAFYFxprIAs0noZRtQe86iP3g3bpBg3IE7jQyTd+SbS8J85AzuOw2Pfhb/4u0jpWkBszX4MBdDGdI2niS1tzJM2PjWdFMtcxoj6qsb+jN3aCjgOjFahVoO+QRgowaKSlPGTyCo9ROOGkleJhdLwdHNaue9nt7mPgBZyWYys+qeRwpybfzlPGWjOsnL0/Arw7wnvbXe6tqvJlYmFYrYld4FLqF1Op9+TwWiTWYJFGrmwk0meRl1DEJoxXY/lEmonJzLk7Bi3Jdl1Xkfj+EUmuvg2hMypNRcWpCgdvGqGXnIxfLA2vWd+7AiN2G6z9NFYDtJGRG/624X4bQQUid1n8nBTX3r3qcbErG9M6Qi2XZ1FQKGHmN2X0u+UCODJdRmgCNVTUB2J7Joi6dweZNUVqAdbmJd+OhQHV7wSbUQNtQ8pBjdgs7U7VcR3PsKSvJnghnTOqvT5hpXJerP55RFRBX6W3uVg3MgJA0dQuuxmUEOsRFzdP7lNdRj+MH3ZixR5hRgN6V6nUft6UJfRomAsshPi/VRykEberqlh9nJsAdSA6jDMH1cAcn1RVsO+9NiLEMxgtsd7RP0KB9DyIKWt6XdppB25mcvIYtmEgmXrieiys8v4uo59iKh65lKNjxKF300r9DF30Ugdm00x/5f0LueJTLCp0lhbyZ0EXWqMqalX48QQNZ6d90EOYZxN1xxI/zc808oDcDAtN5LKNCq1nCrqH2jky7aTucSCu7bpu8kMM7fJWHp219hwiv8cIthn3HYRQT30sZ16j7NW2tHyOpEaa04jNaQovTS8kWqTElSuyyhN1JN3E1B0YdGcG3QR8Xo/EOWk3EIjWkc78prTeRzFN44Lggeq2XOvIErLPUi4He8R9QRyqaIOf4JIFtuAOmD5IIwdS6nC43DhTRX4ppR+tqb3m4d6PVG8jheBR2DVz1ESyX0o6DYPbcszDyVYXEKNuYDgt5SAMxpIn0eD3Ukc/rmCFIezlWYiv0jXcg0Dcy+t3O8iYJcrQE81wRDVGLgLkVIcXwkvHZNlaQXjknpObLElNoaU6lmCxuMsMX//GLF7RRlZW3vJMpx2RG0O08EKRME212TNay+YdtcOH5wNyRXj6wTs4qpX3cg5FO+wMnda+mRiJWXP9RCxk4Yt07cJa7eeEk7suN1LWN/3EArb9EXS+7hoVLOizqlnV1Cftau2Zg/LVeGOonHvuX8gPcODTJzD/cSYLSKd5PN+QNAOc0/CRaz8zFeyc6Za1GZd6YJecB3SE/XVqUakvgK1auA3ECumd1voKabjvewavLNVV0iGYYvRs4Koh2pKjzvL+fZevYtoYr5IY8Fir7qulepVdw4TS7e9h6xaY8VmbxSTz+0BcX4kFfY2SFVEUfyDSHnaJ38AcecuEtupJ65o3cK/SChrmyW7shcjBsu9RLk6W6K2YGbLPT6S2sATzzSaJem3LTXTy0BKax8BJe0HfnSsUak4YAOxC4UVoK+VW0k5jl5Ck6+I2uKnqK8/T1htp1G/zSUWoFVIEdSI6DpE0M3e3ExrvsJEyy+XvPbryBTHTiU9hAfXSZ9bQecZnK+heV1AbVdCw9G7RxiOcEJBD7EbiLFgL1qX0zu9jYYtxCLm9FoI2GiqBc5e8VUiON5sGXvc3Elj332eoHodT+cdRePhWtOxBaKYl7PzoLu6G9dN6fahyV53RS6ln2vUTRWni5rYbkx1fhkKn0K9ZWWURl3lMJSWpWuebLyvLbqrqMG2oMlvq9f1E04j68vbrx8lJuhmgm8IYanZQrSLkXeEmRuVdD3jpYVxbbmzeBmMnYQND8CFN9BqdInGpf0aUcKxhkaOZ9lw+n8ptYWVLsTM+CFS0B8AY3rPJ5AlOIAWBmNtxuAuowE0G8rjdYIZYFjJQdKFNGb9eY8183wLRPQ7lzJRXMgLV04lsktri6iAxtw3CAz2AMHWeJaopWAF4fHiCPUeIsBq7NuBGHOci0xNC+pU2inRO5moHJuPfQi9w4EW3zWL1+JulHalxf9dvtX96s9tEUP0vaujueDRZWLRzN/teRprPC9EfWuanpOM2iVd9KK+WkX06Wk0FvOxPULEW/LPDxFerj1CG1nNOLkV7Zzs+FEa8eKp6KvXRemCrMEN+QdZD47Vwuqy5VUjNiQsPoTezoplGLx7QmkI6uXcM1NgMVI0IyinH9TAnlwriKIkJmYbmdiEJt6P0QAeQG6IK4OtIhaIVkR4RywX98DVWlQdu1CDNSmdtFyCs4dh0UYUEFxNzID70s/7BIxgqGBluvER4Eu3wchHULwVej+W4t6DzMVCumYyb59Kp+eDaQ6i52wl8OtyYzPOSH5MKMZVqG+9d553f+hByn+cqGVgF7VGrMmbiIntWsXjSDE6EPb/tXf+MHJVVxz+KHYLD1LWK2VXwrZiQJFXSAYJXHgbQEpoNkVIAQVJkwopIkU6mqRIRxMqU6BIKEUooIhonAIo1lJIYYyIUZBXK+ONbEfyrGQPErOKZhQ5xbnfnjtvZ3bWMN4Q5R1pNTt/33v33XvuOb/zO+cYkbeYPKTif470dJYJZVwHDXU2Nkj0yuP1SYrUV2R5Ufv4XSfZffdLBsRG0fSqIBWUlv4LxG3fL5AmPNAMYu1nPR9hVPEukHP7GbIVTpcYx2YCku2RxMFrvLN5rh5ngaxUpiGmBzwpCWWH0U3RuX2CvVat+Hj9+iX21lnQmh8nGlXGLT4uxxTOmLaWZh5Iq+UtYjI42EMy3a+WX98tROt5OP1sefEku33BdlvPAGxGMIZeJEc4YWx1ooWgy2ngqC7d92dGMSTbpb9Euo7zhOI4w2imlTt0TWHRumWQN+0KcHwJhiXw1+vB8kOw8TcSgx0Sd+nzcr1/J+6gF/WdckIny+e2/gXHFuHTfwckUdKneYiosXsxnr5brvEsqbRcLKdJl8gMtVmS+98krBMtorr2qqnSKmIjys3W3AYlxHqXyMDsbrIIMUxu7CIzfl+Y6jkS4mhibTL3jMpDBtk8Z1GtNUJpCTk0ZZZjCGkcjBOvQ6/lBFkNdJLskB2Ta7dbxG7Sd2C0KI0F4A0q1hae1L86nVfrk/LeBrlWm3KEvMd6NAauplHzVOpXiDFZZG+LIEgo5HHyumuY0+vRO5sk1psw9d9xPcZ0b+K+Kt1PiDbjGmPrZGQQsqTbyw8EpvrOAIYfQf8P7Lac6Q+JVaaZspD1FubmY1CfJlytY8Rik34k3ahuBe4E+oCMti6SysnWKGK7YoQC7S7Gplhlyd1yBTj+BPR7cR0L5bxPfZ9MgRb4+idhtW6Wk75I3MUvq/efLa+/fTsGdIdkczwG/B64GZvai4TC1cqtC8MY8FgmsamvkxQySbpEXYo6Rdh7YRCzphQ5vipl3VEpONYtfZS0PprVxiAXuji19Y2FlupUdMVsNs/bz5iqvEIyH6RArrM3EDPXeJyVNNNpa/kBqTyvcrAOE7rC9Tj0mJ50oCK6yWhXmCVGEwd6xHoyXmBrG5kl0yL7O2T8xM34RyS18/bkr+5a2XfKcW6TxlNTrhLjIPd2h/AobOfjuhn3XXFx55SwxoOEQp/GsID7rHQhLuhVMvvE/HhI9/DN8nwVuDKInWPjLzDchOHNUt+gl1/qbxYO5yDpSNZPWCbxn6PkgMgjtTXPBTL6eJXRjBkH1uI9A7ImroEArWjIqGfNU70BXHo/8Oeu4Fan1J79HllZbJuwVPuEYt0iC9lA1mOQ4/spYRk/RljC24SvezNxtVNlLFbJyaFCMMLrdSyS1KtZyjkCIXEROonfJylXtXVbZyDKx3XDdJ7AKGXMpJseiQPWVrMWa40D11aINDrvpWPiIjegVm/UF9gr8ogPSx4mxk/IRohmWjBHRXav0iPGoA7E9ctxTzGa8ivFTd6u8+8gwTtDFkfJdVmzRhYnfA8yyPo0mVghltyUm+Tm83D5O0JsqK8Q3vkkzN4NocModOf1fSuUrvIeoXjXSWvCifJbwrW/A5xYKmX3Xkiy9S5BfieedDqpJBbmc2BXCOW7RtaA8Mbbv6mObl8mC5u74FTUV0iloQKX7mIhFbGbOv1SPMpId3e7uEtbwGbUBaZcxy5neZusqetzgTsfh+Vxm1KHgVDCn8e/w3KNR4gAmgGtp0gX+gzZwfhEGRuzs2aB6TblHOkeaom4MVh7WMy9S3ZwlZxeK1qxWQvkQypSnSAtHa+3tjxV3E1r1N/Qy9GireEE01vX2TtO9e9JtJnVBnatnM+4LEmtdPnCB+Hw1nUy7kW0D5w7q2RCihvg2erzPZITLw1vnJV7jNFr08pVgXpv6hoMk0TGkJXsII2pccXbrQbodTxKFi2X9TPueHaANqA2JONSlMeX9jlPuI+BtHHSJYsJSx86Dfz0T/DAT4o72i2NAd/NwsAvHimTv1sW2BK7d2I4SCzJgJmWVG3dSR2Sx/cZgXtqFRmBrKlUFrFYI7Gm5uQ+Qrp1bhA1brbUK4ZqBzY+jXY1nT7Z/+wioUTN1hNkU7lajQxG6+6+wW7r9VeB1wiIZY3sznqb5DeK3xrMEg90YdScxlnK6+VYq4SlskLSf+oiJ5RzqClZ16v3VMaQ7q7Ph9Wjc8v7IB1thZhPzWIvjxDWa60U9JhcSMJi4ziiUqt6xG2re5pN42seRL4gPRbvzzVinpvBVyeETJI5Mhi1wr21Y9fSvUNyos+U56bezrG3E4MpvOO8gLPkOjOg5X2t6ZpuopfYH2M17uJcs8zlJJrYkJgnGkc1LGhwbNx66JXfvUroGD1bO8pcIhzPP+5zrodm6Spm775FLIJXCIULYQ1fLu+9RfSx6gPXi4WnBbObFF7+NSrrgnucUDiSsecZXfSnSRD9bdIag8SCThCTS2K8Lm/T7ThKZuF8QZIOjJJ2B9ApFLc7QGdIWLVfkpEcSFz3o/L8u6Tp1CETI/SDisJ9hFS4vyRocrX1pRVe42Qq4TrD6yBu0deVc2RQo8bZxHxVrHXxa93XW9Xn66SI+jlkgk2fjGTX1plFiJrKyZCB0NIqOS6fEfB5Mxah1MeXi7pM4LCSa76pDIkFvsIoFWmdVIIHcd09Vz29ez23IZkqf4MY6+PkOJuy38SgJyncPnG/a4aAdC/vr+vN1Nv9RG/XOIBp2vVf08vZIXs2dqq/p8rvTRojOwsvE/fmPJmdqALeTw7V0q3lE8bvtucaz+9QqFvF1r8FHO9neihE+4/nSVdHEF9lU9eBuEUsrD5Jx3mb4Of2y/euMwoZfEAStAekwt4hFLkW/HPEJFonXaXlkpJ7ow9nn4XhNvAPmHuCULQ/jOc8QaR2eWGPEYq1RxSzOU/guUVrvEZYuE+S6aouAheWhrHKznwMo/UW+ZGbeL+sXYjg2q9IZQ+J93VIC+IrkiYEowEwv6d30icrmhmIk3Pr7xpDEKu9TW6s1jp+lFzUHs9qZZeIRT8u2DQkxqyu21CzROaYDWxzkzAktPx2yI4XblzjjlOJLBnvAAAD2UlEQVRbd7IXpO/VGaMHwaMXSOpkzYCx5XqtFHW7x0ExVmaz/2BtvdYKWC+t9jgeZ/z89HxqQ6Kur2Iy0DiPYIfQQ3dI+NHW8uOugfLam2TzArNqzVKbBt/815TuQUWKkNSd40vQ7xZ+4DwcHYTSgSTg36o+b6AERiueicnoMr5HuicmdohHCTvU7IY5slqSbmydqnmCQq0ZRDDw+Dx0t2BJa9WMss3ypW2iKou1dC9WP15VcOkRlsIGgVX9jOBpdiaQbeWjKiphW4svljF5kem9nb6pvE5UFhPWcLJqyT7/C/jNG6PfEVMdEEpG+AjSla0jzWLuS+wNbrj5ahXV2Ww+h1C018vjOJ6scoy9DBk3AbtRzUpMLdeavsbklFiI65dxo6IxNRuSPnbQAGCPGI810qhxY4PMCFsm03pvMEoPWyKxebm0NRzxVfW+96yusT1JZOBofdcZi3rAnuMkuUbCNcIN+xki3XJ9Nb59nRin/UrMwv+A0jWz5EFKjdNuWjCdQbr14q5ShVbJoiYuCMgEB4urQE7eD0l3EzJZQ6tVvqcdDWxqqWL+gKzT6yT5cXl9dQC/24IXtoryXyIoX3W+pVJvlQW7vkBG0tcIK/c0ycY4qEmlEr5B1hbQsptkScxSzpPFSExK8NSbChdiUapoJdm70JfIlFM3yN36DtVvDAiFMUcueLF7h93Fb+PJv7I/7qnCNaqusleJi/XNStx4dHuPTjk/IZWa3yqVztKUB8GcjzGqALXknimvi2uKnwq36W0ZwJSEs8hoRbJa6c9X78kt9zcneQ3SRF2DQh16d3oeHdKKHcdAgZgDQk2ez36ywWjlNr2KaZ1EvvVK12LGujB1hSN3tJcJbOUykVuvFSvdSBdTrHaeDACII2nRXCAG3hJt7tryW1eIwNkGMeHcmb2xWhBaSmukxfHz8tmFk4Rlu1VOwNn/EBH6vAi9zcC118msnnfIGg+7cpJR7LdPWNFj2OQqCcjgohbkvRS3/ibSJZSasM800jvExuZibSoyrUuxaZWd7qT3Q06piw9GM89snnmD0oZpwrmcIgNALu7aOnN8683kXqUuFFPLh4SXI1OlGRxSnmSU46yIcV/Z5xhNGRJjb21koZvzhOX9Mck2EOJxXa1Un3dj7zNZ6dVUwAerzwsfNUWqqMeD3JTrymzOGb3QU4zWIKmtbVk0D5ObwH5ymZyDZsVOu+8P3L17d8pHWmmllVZamZUcOnuhlVZaaeX/WVql20orrbRyiNIq3VZaaaWVQ5RW6bbSSiutHKK0SreVVlpp5RClVbqttNJKK4co/wFDV3daOHI8oAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.]\n",
      " [1.]\n",
      " [0.]\n",
      " [1.]\n",
      " [1.]\n",
      " [0.]\n",
      " [1.]\n",
      " [1.]]\n"
     ]
    }
   ],
   "source": [
    "vis_dataloader = DataLoader(siamese_dataset_train, shuffle=True, batch_size=8)\n",
    "\n",
    "dataiter = iter(vis_dataloader)\n",
    "\n",
    "#example_batch = next(dataiter)\n",
    "example_batch = dataiter.next()\n",
    "concatenated = torch.cat((example_batch[0],example_batch[1]),0)\n",
    "imshow(torchvision.utils.make_grid(concatenated))\n",
    "print(example_batch[2].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start Training um:  21:20:32\n",
      "Epoch number 0\n",
      " Current loss 308.1065979003906\n",
      "\n",
      "Epoch number 0\n",
      " Current loss 1.138925552368164\n",
      "\n",
      "Epoch number 0\n",
      " Current loss 0.964200496673584\n",
      "\n",
      "Epoch number 1\n",
      " Current loss 1.052968978881836\n",
      "\n",
      "Epoch number 1\n",
      " Current loss 0.9594541788101196\n",
      "\n",
      "Epoch number 1\n",
      " Current loss 0.4107712507247925\n",
      "\n",
      "Epoch number 2\n",
      " Current loss 1.5126802921295166\n",
      "\n",
      "Epoch number 2\n",
      " Current loss 1.127558946609497\n",
      "\n",
      "Epoch number 2\n",
      " Current loss 3.7104902267456055\n",
      "\n",
      "Epoch number 3\n",
      " Current loss 0.7310527563095093\n",
      "\n",
      "Epoch number 3\n",
      " Current loss 0.7123371362686157\n",
      "\n",
      "Epoch number 3\n",
      " Current loss 1.3404734134674072\n",
      "\n",
      "Epoch number 4\n",
      " Current loss 1.0322986841201782\n",
      "\n",
      "Epoch number 4\n",
      " Current loss 1.4310665130615234\n",
      "\n",
      "Epoch number 4\n",
      " Current loss 0.8324761390686035\n",
      "\n",
      "Epoch number 5\n",
      " Current loss 0.8502787351608276\n",
      "\n",
      "Epoch number 5\n",
      " Current loss 1.3122851848602295\n",
      "\n",
      "Epoch number 5\n",
      " Current loss 0.921744167804718\n",
      "\n",
      "Epoch number 6\n",
      " Current loss 1.1442584991455078\n",
      "\n",
      "Epoch number 6\n",
      " Current loss 1.011681318283081\n",
      "\n",
      "Epoch number 6\n",
      " Current loss 0.9135341048240662\n",
      "\n",
      "Epoch number 7\n",
      " Current loss 1.2509589195251465\n",
      "\n",
      "Epoch number 7\n",
      " Current loss 0.7557617425918579\n",
      "\n",
      "Epoch number 7\n",
      " Current loss 0.8743131160736084\n",
      "\n",
      "Epoch number 8\n",
      " Current loss 0.7078773975372314\n",
      "\n",
      "Epoch number 8\n",
      " Current loss 2.043950080871582\n",
      "\n",
      "Epoch number 8\n",
      " Current loss 1.0420727729797363\n",
      "\n",
      "Epoch number 9\n",
      " Current loss 0.9620883464813232\n",
      "\n",
      "Epoch number 9\n",
      " Current loss 0.9625381827354431\n",
      "\n",
      "Epoch number 9\n",
      " Current loss 1.1636301279067993\n",
      "\n",
      "Epoch number 10\n",
      " Current loss 1.5617934465408325\n",
      "\n",
      "Epoch number 10\n",
      " Current loss 1.029129147529602\n",
      "\n",
      "Epoch number 10\n",
      " Current loss 1.2446420192718506\n",
      "\n",
      "Epoch number 11\n",
      " Current loss 1.0907390117645264\n",
      "\n",
      "Epoch number 11\n",
      " Current loss 0.9430091977119446\n",
      "\n",
      "Epoch number 11\n",
      " Current loss 1.0374553203582764\n",
      "\n",
      "Epoch number 12\n",
      " Current loss 1.2946057319641113\n",
      "\n",
      "Epoch number 12\n",
      " Current loss 0.8857178092002869\n",
      "\n",
      "Epoch number 12\n",
      " Current loss 0.9587574005126953\n",
      "\n",
      "Epoch number 13\n",
      " Current loss 0.446734756231308\n",
      "\n",
      "Epoch number 13\n",
      " Current loss 0.9940989017486572\n",
      "\n",
      "Epoch number 13\n",
      " Current loss 0.9222986698150635\n",
      "\n",
      "Epoch number 14\n",
      " Current loss 0.8071630597114563\n",
      "\n",
      "Epoch number 14\n",
      " Current loss 0.9962530136108398\n",
      "\n",
      "Epoch number 14\n",
      " Current loss 0.9935855269432068\n",
      "\n",
      "Epoch number 15\n",
      " Current loss 1.2180464267730713\n",
      "\n",
      "Epoch number 15\n",
      " Current loss 1.0259199142456055\n",
      "\n",
      "Epoch number 15\n",
      " Current loss 0.8863251209259033\n",
      "\n",
      "Epoch number 16\n",
      " Current loss 1.0609502792358398\n",
      "\n",
      "Epoch number 16\n",
      " Current loss 0.9913638234138489\n",
      "\n",
      "Epoch number 16\n",
      " Current loss 1.1235753297805786\n",
      "\n",
      "Epoch number 17\n",
      " Current loss 1.1986947059631348\n",
      "\n",
      "Epoch number 17\n",
      " Current loss 0.7384413480758667\n",
      "\n",
      "Epoch number 17\n",
      " Current loss 0.8233901262283325\n",
      "\n",
      "Epoch number 18\n",
      " Current loss 1.227494716644287\n",
      "\n",
      "Epoch number 18\n",
      " Current loss 0.9361811876296997\n",
      "\n",
      "Epoch number 18\n",
      " Current loss 0.19040855765342712\n",
      "\n",
      "Epoch number 19\n",
      " Current loss 0.5063803195953369\n",
      "\n",
      "Epoch number 19\n",
      " Current loss 1.4189283847808838\n",
      "\n",
      "Epoch number 19\n",
      " Current loss 0.782526969909668\n",
      "\n",
      "Ende  Training um:  21:21:26\n",
      "Dauer Training:  00:53   [MM:SS]  \n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAagklEQVR4nO3dbYxcV53n8e/v3nrodtuO7aQTTBxwAt6ZgLRxslYmDPvAkJkhRKsNaGAVtIJolJF5EVagRdols9LOjLRIjJYhM2h3o81MsmRWLJDlYYlQBGRD0GheTEInhJDEk4mBPBgbuyG2Y7vd1fXw3xf3VHV1d8Vuu7vdXbd/H6l0q07d6jrndvWv/3XqVJUiAjMzK5dstTtgZmbLz+FuZlZCDnczsxJyuJuZlZDD3cyshCqr3QGASy65JHbu3Lna3TAzGypPPPHELyNifNB1ayLcd+7cycTExGp3w8xsqEh66fWu87SMmVkJOdzNzErI4W5mVkIOdzOzEnK4m5mVkMPdzKyEHO5mZiU01OH+/C9O8NnvPM+vTjZWuytmZmvKUIf7TyZP8l8f3c+kw93MbI6hDvdqXnS/2fIXjpiZ9TtruEsakfS4pB9JelbSn6T2KyU9JukFSV+RVEvt9XR5f7p+50p1vlYpuj/Tbq/UXZiZDaXFVO4N4N0RcQ2wG7hJ0g3AnwJ3RcQu4Chwe9r/duBoRLwVuCvttyJqqXJvtDordRdmZkPprOEehZPpYjWdAng38NXUfj/wvnT+lnSZdP2NkrRsPe7Tq9wd7mZmcyxqzl1SLukp4AjwMPAT4FhEtNIuB4DL0/nLgVcA0vXHgYsH/My9kiYkTUxOTp5X57uVe7PtOXczs36LCveIaEfEbmAHcD1w9aDd0nZQlb4gfSPinojYExF7xscHfhzxWblyNzMb7JxWy0TEMeD7wA3AFkndz4PfARxM5w8AVwCk6y8CXl2Ozs7nF1TNzAZbzGqZcUlb0vlR4LeBfcCjwAfSbrcB30znH0yXSdd/LyJWZN6kmhdPErwU0sxsrsV8E9N24H5JOcU/gwci4luSngO+LOk/Az8E7k373wv8L0n7KSr2W1eg38Bs5d5oe1rGzKzfWcM9Ip4Grh3Q/lOK+ff57dPAB5eld2dRz3PAc+5mZvMN9TtUu5V705W7mdkcQx3u3Tl3V+5mZnMNdbhX8oxMDnczs/mGOtyhmJqZ8bSMmdkcQx/u1Txz5W5mNs/Qh3vdlbuZ2QJDH+41V+5mZgsMfbhXK5mXQpqZzTP04e7K3cxsoeEP94rD3cxsvnKEu6dlzMzmGPpw91JIM7OFhj7cvRTSzGyhoQ93v6BqZrbQ0Id7NfdSSDOz+YY+3L1axsxsIYe7mVkJDX24V/OMmba/Q9XMrN/Qh3u9kjHTaq92N8zM1pShD3e/icnMbKHhD3cvhTQzW2Dow72aZ3QC2h3Pu5uZdQ19uNcqxRBcvZuZzTpruEu6QtKjkvZJelbSx1P7H0v6uaSn0unmvtvcKWm/pOclvWclB+BwNzNbqLKIfVrAJyPiSUmbgCckPZyuuysiPtu/s6S3AbcCbwfeCPw/Sf8oIlZkSUstF4BfVDUz63PWyj0iDkXEk+n8CWAfcPkZbnIL8OWIaETEz4D9wPXL0dlBepW7w93MrOec5twl7QSuBR5LTR+T9LSk+yRtTW2XA6/03ewAZ/5nsCSeljEzW2jR4S5pI/A14BMR8RpwN/AWYDdwCPiz7q4Dbr5gKYukvZImJE1MTk6ec8e7qnkxBH94mJnZrEWFu6QqRbB/MSK+DhARhyOiHREd4C+ZnXo5AFzRd/MdwMH5PzMi7omIPRGxZ3x8/LwHUMtduZuZzbeY1TIC7gX2RcTn+tq39+32fuCZdP5B4FZJdUlXAruAx5evy3N1p2UaDnczs57FrJZ5J/Bh4MeSnkptfwh8SNJuiimXF4GPAkTEs5IeAJ6jWGlzx0qtlAHPuZuZDXLWcI+Iv2XwPPpDZ7jNp4FPL6Ffi1bznLuZ2QJ+h6qZWQmVJ9xduZuZ9Qx9uHsppJnZQkMf7t05d6+WMTObNfThXvecu5nZAkMf7t05d0/LmJnNGvpwr/odqmZmCwx9uHsppJnZQkMf7pVMSF4KaWbWb+jDXRLVPHO4m5n1GfpwB6jnmadlzMz6lCLcaxWHu5lZv1KEezXPvBTSzKxPKcLdlbuZ2VzlCXdX7mZmPeUI9zxjprXga1rNzNatUoR71ZW7mdkcpQj3Yinkin2Tn5nZ0ClFuPsFVTOzuUoR7tVcNNueczcz6ypFuLtyNzObqyThnvsFVTOzPqUI92ouV+5mZn3OGu6SrpD0qKR9kp6V9PHUvk3Sw5JeSNutqV2SPi9pv6SnJV230oOoeymkmdkci6ncW8AnI+Jq4AbgDklvAz4FPBIRu4BH0mWA9wK70mkvcPey93qemj8V0sxsjrOGe0Qciogn0/kTwD7gcuAW4P602/3A+9L5W4C/jsLfAVskbV/2nvepVfzBYWZm/c5pzl3STuBa4DHgsog4BMU/AODStNvlwCt9NzuQ2lZM1ZW7mdkciw53SRuBrwGfiIjXzrTrgLYFi9Al7ZU0IWlicnJysd0YqFbJaHWCTsdr3c3MYJHhLqlKEexfjIivp+bD3emWtD2S2g8AV/TdfAdwcP7PjIh7ImJPROwZHx8/3/4DfV+S7akZMzNgcatlBNwL7IuIz/Vd9SBwWzp/G/DNvvaPpFUzNwDHu9M3K6WWO9zNzPpVFrHPO4EPAz+W9FRq+0PgM8ADkm4HXgY+mK57CLgZ2A9MAb+/rD0eoFe5e97dzAxYRLhHxN8yeB4d4MYB+wdwxxL7dU56lbvD3cwMKMk7VLuVu5dDmpkVShHuVVfuZmZzlCLcu5V7w+FuZgaULNy9WsbMrFCOcE/TMk1X7mZmQFnC3ZW7mdkc5Qh3v6BqZjZHKcK9u1rGSyHNzAqlCHevljEzm6sU4V73xw+Ymc1RinCffYeqP/LXzAxKEu6z71Btr3JPzMzWhlKEu5dCmpnNVY5w91JIM7M5ShHu1bz4ROIZz7mbmQElCXdJ1Pwl2WZmPaUIdyjm3R3uZmaF0oR7NZffoWpmlpQm3F25m5nNKle4u3I3MwPKFO65w93MrKs04V71ahkzs57ShHvdc+5mZj2lCXe/oGpmNuus4S7pPklHJD3T1/bHkn4u6al0urnvujsl7Zf0vKT3rFTH56vmmZdCmpkli6ncvwDcNKD9rojYnU4PAUh6G3Ar8PZ0m/8uKV+uzp6JV8uYmc06a7hHxN8Ary7y590CfDkiGhHxM2A/cP0S+rdo/vgBM7NZS5lz/5ikp9O0zdbUdjnwSt8+B1LbApL2SpqQNDE5ObmEbhSqrtzNzHrON9zvBt4C7AYOAX+W2jVg34Ef1RgR90TEnojYMz4+fp7dmFV35W5m1nNe4R4RhyOiHREd4C+ZnXo5AFzRt+sO4ODSurg4Xi1jZjbrvMJd0va+i+8HuitpHgRulVSXdCWwC3h8aV1cnFrFq2XMzLoqZ9tB0peAdwGXSDoA/BHwLkm7KaZcXgQ+ChARz0p6AHgOaAF3RMQF+WJTv0PVzGzWWcM9Ij40oPneM+z/aeDTS+nU+fBSSDOzWeV5h2qe0WwHnY6/as/MrDzhXimG0uy4ejczK0+458VQPO9uZlamcK843M3MukoX7s2259zNzEoT7lVPy5iZ9ZQm3HvTMu0LsqzezGxNK0+49yp3T8uYmZUn3CvFZ5b5jUxmZmUK97z4ThDPuZuZlSncvRTSzKynNOFezYtpGX8ypJlZicK9W7k3XLmbmZUn3Ou9pZAOdzOz0oR79wXVpit3M7PyhHvVSyHNzHpKE+7+VEgzs1nlCXcvhTQz6ylNuPc+OMzTMmZm5Ql3T8uYmc0qTbhnmajmcuVuZkaJwh2KqRkvhTQzK1m41yqZK3czMxYR7pLuk3RE0jN9bdskPSzphbTdmtol6fOS9kt6WtJ1K9n5+Wp55jl3MzMWV7l/AbhpXtungEciYhfwSLoM8F5gVzrtBe5enm4ujit3M7PCWcM9Iv4GeHVe8y3A/en8/cD7+tr/Ogp/B2yRtH25Ons2rtzNzArnO+d+WUQcAkjbS1P75cArffsdSG0LSNoraULSxOTk5Hl2Y65axeFuZgbL/4KqBrQN/FLTiLgnIvZExJ7x8fFluXNPy5iZFc433A93p1vS9khqPwBc0bffDuDg+Xfv3FTzzF/WYWbG+Yf7g8Bt6fxtwDf72j+SVs3cABzvTt9cCJ5zNzMrVM62g6QvAe8CLpF0APgj4DPAA5JuB14GPph2fwi4GdgPTAG/vwJ9fl21SsbUVOtC3qWZ2Zp01nCPiA+9zlU3Dtg3gDuW2qnzVcy5D5ziNzNbV8r1DtU8Y6bVXu1umJmtunKFu1fLmJkBZQv3PKPZ8rSMmVmpwr1a8Uf+mplBycK9ludeCmlmRtnC3R8/YGYGlC3c0zcxFSsyzczWr3KFe6UYTtNr3c1snStluPtFVTNb78oV7nmq3D3vbmbrXKnCverK3cwMKFm4dyt3r5gxs/WuXOHuyt3MDChbuLtyNzMDyhbuFYe7mRmUNdw9LWNm61ypwr3qpZBmZkDJwr1buTdcuZvZOleucPcLqmZmQMnCvd77bBmHu5mtb6UK96ordzMzoGTh7qWQZmaFUoa7p2XMbL2rLOXGkl4ETgBtoBUReyRtA74C7AReBP51RBxdWjcXpzst03Dlbmbr3HJU7r8VEbsjYk+6/CngkYjYBTySLl8Qdb+JycwMWJlpmVuA+9P5+4H3rcB9DOSlkGZmhaWGewDflfSEpL2p7bKIOASQtpcOuqGkvZImJE1MTk4usRuFLBOVTJ5zN7N1b0lz7sA7I+KgpEuBhyX9/WJvGBH3APcA7NmzZ9m+9LSaZ67czWzdW1LlHhEH0/YI8A3geuCwpO0AaXtkqZ08F7WKw93M7LzDXdKYpE3d88DvAs8ADwK3pd1uA7651E6ei1olY6a9bE8EzMyG0lKmZS4DviGp+3P+d0R8W9IPgAck3Q68DHxw6d1cvJqnZczMzj/cI+KnwDUD2n8F3LiUTi1FUbk73M1sfSvVO1ShW7m3V7sbZmarqnThXq2IpufczWydK124e87dzKyM4e6lkGZmZQz33C+omtm6V75wz+XK3czWvfKFu5dCmpmVMNzzzB8cZmbrXunC3R8cZmZWwnD3ahkzM4e7mVkplS/cc7+gamZWvnBPq2Ui/BEEZrZ+lS/c84wIaHUc7ma2fpUv3CvFkLwc0szWs9KFezUvhuQXVc1sPStduHcrd4e7ma1n5Q13T8uY2TpWvnD3tIyZWQnD3ZW7mVkJw92Vu5lZ+cK96qWQZmblC/du5d5YQ5X7dLPtd8ya2QVVWakfLOkm4C+AHPiriPjMSt1Xv7WyFPLgsdN8+5lf8O1nf8HEi6+y85Ixfu+6Hbz/2st545bRVe1b2XQ6QZZptbthtqasSLhLyoH/BvwOcAD4gaQHI+K5lbi/fvUU7t959jA/P3aaapZRrYhKlhEUQdCJoJ22QmSZyDPIJDKJ0WrORRuqXDRaZctolc2jVQBeeXWKl341xUuvTvHyr04xebLBSDVnrFZhQ73YtjvB958/wo8OHAfg1y7bxB/8s6t46uVj/JfvPM9nv/s8v/mWi/m963Zw9fbN6Xil44Y43Wxz9NQMr56a4ehUsT3dbDNazYtTLadezalm4tRMm6lGi5MzLU41Wkw3O2waqbBtQ40tYzW2bqiyZbRGnokgPXMICOD0TJsTjSYnplu9E8BYLWesXmGsnrOhVqGai0arw0yrw0y72LY7QSUTeZ4V20y9babifJbOTzfbnJ5pMzXTYqrZZrrZYaSasWmkyuaRCptHq2weqVLJRKtT/F66pyCo5hnVvPj9VfOM080WLxw+yT8cPskLR07wwuGTHD4xzaWb6uzYuoEdW0fZsXWUN24ZpV7JyQR5JiSRS71j3VV8VEWHZjuYaXVopjFON9u943qq0eZko0VEsHm0eFx0T2O1Su+4dLetdocsE9V0fCp9Y6jkmm3LREDvsdjuBK1O0GoHjVabRqtDo9mh0WoTwMZ6pff72VivMFLNCYJ2BzoRvWeHG+tF3zaPVtg8UmXjSIWpmTbHp5ocP93k2OkZjp9u0u4Embq/N3rHKM9nf595OmDNdhTHpl0co/aAj/eIKIqqRvd4pOOZZ6KWjkG1kvV+p3lWHIM8HatOFPfRbEfxO2kVx6X7WMpU/I0C6bFS7Ns9btH7uy6OB8CGWs6mkSob6xU2j1TYOFJBqDje0b0NnG62OdVocbLR/Z23iICRas5INaOe/v42jlQY31hnfFOdi8dqVPKFkx+dTnBqpsUvjk9z8Pg0h46d5uDxaSZPTDNarbBtrMq2sTrbxqps3VDjzReP8YaLRs45685mpSr364H9EfFTAElfBm4BVjzcL91cZ6Sa8aXHX17R+9lYr3DZ5jqNVoepmeKB0Z0KuuaKLfyHm36d97z9Mq4a39i7zcu/muJrTx7g6z88wL974EeLup9KJkaqOY1Wm2Z78NROrZKxsV6hXsl47XSTUzPtcx5P74XoIXmtYqSa8dZLN/Kbb7mYN1w0wpETDX5+9DRPvnyUbz19aGD4nKtaJZsTppLYd+gEr51ucqLRWoZRLE41lz8vaQ2SYNuGGptHqzSabU6n03Rz4d9Qd9/pZnvB3+dH/8VV3Pneq5e/fysxFyzpA8BNEfEH6fKHgd+IiI/17bMX2Avwpje96Z+89NJLy3b/0802UzNtWqnSaKWqQ+k/f7fCzDIREXQ60E7/9SOiqHJOd6ucJq+lKudN2zbwpos38OZtG9g2VkPzysBmuq/RWn7G/nU6wQ9fOcrkiQYxW1ADxTOPbWM1to3V2DpWY1MKle7Pn04PoFY7es8YqvOqh5lWh2NTMxydanJsaoZuJsw+Q4DRVNFsGqmwaaRCvZL3bjs1U1QwUzNtmu0O9UpGLc+pVTLqlYwsE51ULbU70avk2hF00rbVjlT5ZIxUczbUimcCI9WMRqvDa9NNXjvdStsmnQjyrKh0M4lKN9DaHZqdKLbtDrVKxlvHN7Fj6+jrTsW02h1+eXKm169OdCvjwb+PPBP1voqyGGfem+J7vfs4Md3i1EyLWp5Rq6RTnlHJMzqdoNkpHg+tdt/5vm2zHUiQp8dinh6b3ePc7UOextlotTnVmK0wp5vt3uO5+9iOgJONFsfT4/a16eLZ2YZa3nu2sWVDjc2jleLZbBSV7tzjNFsNd/9J1vqPT148BgYd/e5xqKfHSzUX7Yii8k+VfCM9+2t1Or1nKt0Kv3iGlp715MVjoXhWQuofRBTP6LrPGCt5lo4hc45HBEzNtDk53eodh+4zsDybPeaZxEg1Y+NIhbFa8Y98rF5BKrJkutlJ2zavTTeZPDHD5MkGvzzRYPJkgxPTLUYqGaO1orofqeaM1XMu2zzC9otG2X7RCJdtHuk9nqab7d6z8qOnmrzhojpvvXTT6z7WzkTSExGxZ+B1KxTuHwTeMy/cr4+Ifzto/z179sTExMSy98PMrMzOFO4rtVrmAHBF3+UdwMEVui8zM5tnpcL9B8AuSVdKqgG3Ag+u0H2Zmdk8K/KCakS0JH0M+A7FUsj7IuLZlbgvMzNbaMXWuUfEQ8BDK/Xzzczs9ZXuHapmZuZwNzMrJYe7mVkJOdzNzEpoRd7EdM6dkCaBxbxF9RLglyvcnQupTOMp01jA41nLyjQWWNp43hwR44OuWBPhvliSJl7v3VjDqEzjKdNYwONZy8o0Fli58XhaxsyshBzuZmYlNGzhfs9qd2CZlWk8ZRoLeDxrWZnGAis0nqGaczczs8UZtsrdzMwWweFuZlZCQxPukm6S9Lyk/ZI+tdr9WQxJ90k6IumZvrZtkh6W9ELabk3tkvT5NL6nJV23ej1fSNIVkh6VtE/Ss5I+ntqHbjySRiQ9LulHaSx/ktqvlPRYGstX0sdVI6meLu9P1+9czf6/Hkm5pB9K+la6PLTjkfSipB9LekrSRGobuscagKQtkr4q6e/T3887LsRYhiLcNfuF2+8F3gZ8SNLbVrdXi/IF4KZ5bZ8CHomIXcAj6TIUY9uVTnuBuy9QHxerBXwyIq4GbgDuSL+DYRxPA3h3RFwD7AZuknQD8KfAXWksR4Hb0/63A0cj4q3AXWm/tejjwL6+y8M+nt+KiN19a8CH8bEG8BfAtyPi14FrKH5HKz+WSN8AvpZPwDuA7/RdvhO4c7X7tci+7wSe6bv8PLA9nd8OPJ/O/w/gQ4P2W4sn4JvA7wz7eIANwJPAb1C8S7Ay/zFH8b0E70jnK2k/rXbf541jRwqJdwPfoviq3GEez4vAJfPahu6xBmwGfjb/+F6IsQxF5Q5cDrzSd/lAahtGl0XEIYC0vTS1D80Y09P4a4HHGNLxpCmMp4AjwMPAT4BjEdFKu/T3tzeWdP1x4OIL2+Oz+nPg3wPdrwG/mOEeTwDflfSEpL2pbRgfa1cBk8D/TFNmfyVpjAswlmEJ90FftF62NZxDMUZJG4GvAZ+IiNfOtOuAtjUznohoR8Ruior3euDqQbul7Zoei6R/CRyJiCf6mwfsOhTjSd4ZEddRTFPcIemfn2HftTyeCnAdcHdEXAucYnYKZpBlG8uwhHuZvnD7sKTtAGl7JLWv+TFKqlIE+xcj4uupeWjHAxARx4DvU7yOsEVS99vJ+vvbG0u6/iLg1Qvb0zN6J/CvJL0IfJliaubPGd7xEBEH0/YI8A2Kf8DD+Fg7AByIiMfS5a9ShP2Kj2VYwr1MX7j9IHBbOn8bxdx1t/0j6dXyG4Dj3adta4EkAfcC+yLic31XDd14JI1L2pLOjwK/TfEi16PAB9Ju88fSHeMHgO9FmhBdCyLizojYERE7Kf42vhcR/4YhHY+kMUmbuueB3wWeYQgfaxHxC+AVSb+Wmm4EnuNCjGW1X3A4hxcmbgb+gWJu9D+udn8W2ecvAYeAJsV/5Nsp5jYfAV5I221pX1GsCPoJ8GNgz2r3f95Y/inF08OngafS6eZhHA/wj4EfprE8A/yn1H4V8DiwH/g/QD21j6TL+9P1V632GM4wtncB3xrm8aR+/yidnu3+vQ/jYy31bzcwkR5v/xfYeiHG4o8fMDMroWGZljEzs3PgcDczKyGHu5lZCTnczcxKyOFuZlZCDnczsxJyuJuZldD/B3qLmaKH3NGNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epoch_range = 20\n",
    "counter = []\n",
    "loss_history = [] \n",
    "iteration_number= 0\n",
    "\n",
    "\n",
    "## Start Training\n",
    "print(\"Start Training um: \", time.strftime(\"%H:%M:%S\"))\n",
    "start_time = time.time()\n",
    "\n",
    "for epoch in range(epoch_range):\n",
    "    for i, data in enumerate(train_dataloader, 0):\n",
    "        img0, img1 , label = data\n",
    "        img0, img1 , label = img0.to(DEVICE), img1.to(DEVICE), label.to(DEVICE)\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        ## Calculate Ouputs and LossConstrative\n",
    "        if(network_name == ('VGG11' or 'VGG11bn' or 'ResNet18' or 'ResNet34' or 'alexnet' or 'Squeezenet1_0' or 'GoogLeNet' or 'Shufflenet_v2_x0_5' or 'Resnext101_32x8d')):\n",
    "            output1 = network(img0)\n",
    "            output2 = network(img1)\n",
    "            loss_contrastive = criterion(output1,output2,label)\n",
    "           \n",
    "        elif(network_name == ('Siamese_Network')):\n",
    "            output1, output2 = network(img0, img1)\n",
    "            #outputs = network(img0, img1)\n",
    "            loss_contrastive = criterion(output1,output2,label)\n",
    "        else:\n",
    "            print('False Network choosen')\n",
    "        \n",
    "        \n",
    "        ## Backpropagation and Optmizer\n",
    "        loss_contrastive.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        ## Get loss\n",
    "        if i %10 == 0 :\n",
    "            print(\"Epoch number {}\\n Current loss {}\\n\".format(epoch,loss_contrastive.item()))\n",
    "            iteration_number += 10\n",
    "            counter.append(iteration_number)\n",
    "            loss_history.append(loss_contrastive.item())\n",
    "\n",
    "            \n",
    "## Finished Training\n",
    "print(\"Ende  Training um: \",time.strftime(\"%H:%M:%S\"))\n",
    "stop_time = time.time()\n",
    "time_dif, time_format = secs_to_HMS(stop_time-start_time)\n",
    "print(\"Dauer Training: \", time_dif, \" \", time_format, \" \\n\")            \n",
    "\n",
    "show_plot(counter,loss_history)\n",
    "WaitTime_Finished()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZmUlEQVR4nO3df5Ak5X3f8fdnZveW3+YQK+rMge9QHU6Ack5og0kpkolJxA+rBHLFyVEVQ2SlTpKhyoqSiiGqiogTqhzZRCrKMqqTdQFiCYSEKVEqFOlMbJFUhNCedDoOIcQCJ7PcFbcyMr+O29vd+eaPfmZndnr218zs7OrZz6tqqnue6Zn+Ts/sZ599urdbEYGZma0PldUuwMzM+sehb2a2jjj0zczWEYe+mdk64tA3M1tHBla7gMWceeaZsWXLltUuw8zs58bevXt/GhHD7R5b86G/ZcsWRkdHV7sMM7OfG5J+Mt9jiw7vSNot6YikA01tX5K0L90OStqX2rdIerPpsc82Pecdkp6QNCbpDknq9o2ZmdnyLKWnfxfwJ8A99YaI+Jf1eUm3A680Lf9sRGxv8zp3AjuBx4CHgSuBry+/ZDMz69SiPf2IeBR4ud1jqbf+L4B7F3oNSZuA0yLi21H8C/A9wLXLL9fMzLrR7dE77wJeiohnmtq2Svq+pG9JeldqOxsYb1pmPLW1JWmnpFFJoxMTE12WaGZmdd2G/nXM7eUfBs6NiLcDHwO+KOk0oN34/bwn/YmIXRExEhEjw8Ntd0CbmVkHOj56R9IA8JvAO+ptETEJTKb5vZKeBc6n6Nlvbnr6ZuBQp+s2M7POdNPT/6fAjyJidthG0rCkapo/D9gGPBcRh4HXJF2a9gNcD3y1i3WbmVkHlnLI5r3At4FfljQu6YPpoR2Ud+C+G9gv6QfAV4APR0R9J/BHgD8DxoBnWeEjd+545Bm+9WPvDzAza7bo8E5EXDdP+79u0/YA8MA8y48CFy2zvo796V+Pcf0/2sKvne99AmZmddmee0cIXyDGzGyufENf4Mw3M5sr39BngWNCzczWqXxDX3JP38ysRb6hD4T7+mZmc2Qb+nhM38ysJNvQ93mbzczK8g19+ZBNM7NWGYe+j94xM2uVb+jjMX0zs1b5hr7ko3fMzFrkG/q4p29m1irf0PeYvplZSbahD/6PXDOzVtmGvgTu65uZzZVv6OMxfTOzVvmGvk/DYGZWkm/o40M2zcxa5Rv67umbmZXkG/p4N66ZWatFQ1/SbklHJB1oartV0ouS9qXb1U2P3SJpTNLTkq5oar8ytY1Jurn3b6VUt3v6ZmYtltLTvwu4sk37pyJie7o9DCDpAmAHcGF6zp9KqkqqAp8BrgIuAK5Ly64oj+mbmc01sNgCEfGopC1LfL1rgPsiYhJ4XtIYcEl6bCwingOQdF9a9ofLrniJ5PEdM7OSbsb0b5K0Pw3/bExtZwMvNC0zntrma29L0k5Jo5JGJyYmOirOp2EwMyvrNPTvBN4GbAcOA7en9nYXrIoF2tuKiF0RMRIRI8PDwx0VKHwRFTOzVosO77QTES/V5yV9DvhaujsOnNO06GbgUJqfr31FuKdvZlbWUU9f0qamu+8H6kf2PATskDQkaSuwDXgc+C6wTdJWSRsodvY+1HnZS6gRH6dvZtZq0Z6+pHuBy4AzJY0DnwAuk7SdojN9EPgQQEQ8Kel+ih2008CNETGTXucm4BtAFdgdEU/2/N3Mrds9fTOzFks5eue6Ns2fX2D524Db2rQ/DDy8rOq6UPT0HftmZs2y/Y9cPKZvZlaSbej7dPpmZmX5hr4vjG5mVpJv6OOjd8zMWuUb+j61splZSb6h74uomJmV5Bv67umbmZVkG/rgg3fMzFplG/q+iIqZWVm+oQ+4r29mNle+oe8xfTOzkrxDf7WLMDNbY/INfV9ExcysJN/Qd0/fzKwk39DHY/pmZq2yDX0zMyvLN/R95Swzs5JsQ99XzjIzK8s39LXaFZiZrT35hj7ekWtm1mrR0Je0W9IRSQea2v5I0o8k7Zf0oKTTU/sWSW9K2pdun216zjskPSFpTNId0sr2xX3lLDOzsqX09O8Crmxp2wNcFBG/AvwYuKXpsWcjYnu6fbip/U5gJ7At3Vpfs6fc0zczK1s09CPiUeDllrZvRsR0uvsYsHmh15C0CTgtIr4dxd7Ve4BrOyt5aXzuHTOzsl6M6f8O8PWm+1slfV/StyS9K7WdDYw3LTOe2tqStFPSqKTRiYmJjorylbPMzMq6Cn1JHwemgS+kpsPAuRHxduBjwBclnUb9TMdzzZvIEbErIkYiYmR4eLjD4tzTNzNrNdDpEyXdALwXuDwN2RARk8Bkmt8r6VngfIqeffMQ0GbgUKfrXlJ9+Nw7ZmatOurpS7oS+H3gfRFxtKl9WFI1zZ9HscP2uYg4DLwm6dJ01M71wFe7rn7BGnHqm5m1WLSnL+le4DLgTEnjwCcojtYZAvakIy8fS0fqvBv4A0nTwAzw4Yio7wT+CMWRQCdS7ANo3g/Qc8WYfm0lV2Fm9nNn0dCPiOvaNH9+nmUfAB6Y57FR4KJlVdcFH71jZlaW73/k+nz6ZmYl+Ya+r5xlZlaSb+i7p29mVpJt6IPH9M3MWmUb+vJFVMzMSvINfXBX38ysRb6h7zF9M7OSfEMfd/TNzFrlG/q+iIqZWUm+oY97+mZmrfINfZ+GwcysJNvQBx+yaWbWKtvQL3r6jn0zs2b5hv5qF2BmtgblG/oe0zczK8k39H1hdDOzknxD3z19M7OSvEN/tYswM1tj8g19X0TFzKwk29DHPX0zs5Ilhb6k3ZKOSDrQ1HaGpD2SnknTjaldku6QNCZpv6SLm55zQ1r+GUk39P7tNNUMTn0zsxZL7enfBVzZ0nYz8EhEbAMeSfcBrgK2pdtO4E4ofkkAnwB+FbgE+ET9F8VK8EVUzMzKlhT6EfEo8HJL8zXA3Wn+buDapvZ7ovAYcLqkTcAVwJ6IeDkifgbsofyLpGeKE6459s3MmnUzpn9WRBwGSNO3pvazgRealhtPbfO1l0jaKWlU0ujExERHxfnoHTOzspXYkdvuDAixQHu5MWJXRIxExMjw8HDHRbijb2Y2Vzeh/1IatiFNj6T2ceCcpuU2A4cWaF8RvoiKmVlZN6H/EFA/AucG4KtN7deno3guBV5Jwz/fAN4jaWPagfue1LYi3NM3MysbWMpCku4FLgPOlDROcRTOHwL3S/og8DfAb6XFHwauBsaAo8AHACLiZUn/BfhuWu4PIqJ153Dv+DQMZmYlSwr9iLhunocub7NsADfO8zq7gd1Lrq4L8smVzcxKsv2PXF9ExcysLN/Qx4dsmpm1yjf0PaZvZlaSb+j7IipmZiX5hr57+mZmJXmH/moXYWa2xmQb+iD39M3MWmQb+vIJ9c3MSvINfTymb2bWKt/Q95i+mVlJvqHvC6ObmZXkG/ru6ZuZleQb+nhM38ysVb6hLw/vmJm1yjb0wcM7Zmatsg19+TSbZmYl+YY+cuabmbXIN/R9ERUzs5J8Qx+P7piZtco39H1qZTOzko5DX9IvS9rXdHtV0kcl3Srpxab2q5uec4ukMUlPS7qiN29h3vp8ERUzsxYDnT4xIp4GtgNIqgIvAg8CHwA+FRF/3Ly8pAuAHcCFwC8Cfynp/IiY6bSGhfifs8zMyno1vHM58GxE/GSBZa4B7ouIyYh4HhgDLunR+st8GgYzs5Jehf4O4N6m+zdJ2i9pt6SNqe1s4IWmZcZTW4mknZJGJY1OTEx0VJCc+mZmJV2HvqQNwPuAL6emO4G3UQz9HAZury/a5ultYzkidkXESESMDA8Pd1gXHtM3M2vRi57+VcD3IuIlgIh4KSJmIqIGfI7GEM44cE7T8zYDh3qw/rY8pm9mVtaL0L+OpqEdSZuaHns/cCDNPwTskDQkaSuwDXi8B+tvy6dWNjMr6/joHQBJJwH/DPhQU/MnJW2nyNyD9cci4klJ9wM/BKaBG1fqyB3wRVTMzNrpKvQj4ijwlpa2315g+duA27pZ51K5p29mVpbvf+TiMX0zs1bZhn5xbmUzM2uWbejXI9/j+mZmDfmGfkp9Z76ZWUO+oZ/6+s58M7OGfEN/tqfv2Dczq8s39NPUkW9m1pBv6HtM38ysJOPQr4/pO/XNzOqyDf069/TNzBqyDX3/b5aZWVm+oV8/ZNM9fTOzWfmGfn1Hrsf0zcxm5Rv6aeqevplZQ76hP9vTNzOzunxDf3ZM37FvZlaXb+i7p29mVpJt6Ne5o29m1pBt6MsH6puZlWQb+rPc0zczm9V16Es6KOkJSfskjaa2MyTtkfRMmm5M7ZJ0h6QxSfslXdzt+uetK019nL6ZWUOvevr/JCK2R8RIun8z8EhEbAMeSfcBrgK2pdtO4M4erb/EZ9k0MytbqeGda4C70/zdwLVN7fdE4THgdEmbVqIAn0/fzKysF6EfwDcl7ZW0M7WdFRGHAdL0ran9bOCFpueOp7Y5JO2UNCppdGJioqOiZk+t7K6+mdmsgR68xjsj4pCktwJ7JP1ogWXbHVJTSuWI2AXsAhgZGekotX2cvplZWdc9/Yg4lKZHgAeBS4CX6sM2aXokLT4OnNP09M3AoW5raMfn3jEzK+sq9CWdLOnU+jzwHuAA8BBwQ1rsBuCraf4h4Pp0FM+lwCv1YaCe85WzzMxKuh3eOQt4MI2fDwBfjIj/Jem7wP2SPgj8DfBbafmHgauBMeAo8IEu1z+v2XEkZ76Z2ayuQj8ingP+QZv2vwUub9MewI3drHOpPKZvZlaW7X/k+spZZmZl+Ya+r5xlZlaSb+inqXv6ZmYN+Ya+x/TNzEryDX1fOcvMrCTb0McnXDMzK8k29H0JFTOzsnxDXz5k08ysVb6hn6Y+ZNPMrCHf0PeYvplZSf6hv7plmJmtKfmGvg/ZNDMryTf03dM3MyvJNvTr3NE3M2vINvTrh2y6r29m1pBv6Kepe/pmZg35hr7H9M3MSvINfV9ExcysJN/Q90VUzMxK8g39NHVP38ysoePQl3SOpL+S9JSkJyX9Xmq/VdKLkval29VNz7lF0pikpyVd0Ys3MH99xdShb2bWMNDFc6eBfxcR35N0KrBX0p702Kci4o+bF5Z0AbADuBD4ReAvJZ0fETNd1LCANKbv4R0zs1kd9/Qj4nBEfC/NvwY8BZy9wFOuAe6LiMmIeB4YAy7pdP2LcU/fzKysJ2P6krYAbwe+k5pukrRf0m5JG1Pb2cALTU8bZ55fEpJ2ShqVNDoxMdFZTR09y8wsb12HvqRTgAeAj0bEq8CdwNuA7cBh4Pb6om2e3rYfHhG7ImIkIkaGh4c7rSu9VkdPNzPLUlehL2mQIvC/EBF/ARARL0XETETUgM/RGMIZB85pevpm4FA361+wtjT1mL6ZWUM3R+8I+DzwVET896b2TU2LvR84kOYfAnZIGpK0FdgGPN7p+hevr5i6p29m1tDN0TvvBH4beELSvtT2H4HrJG2nGLo5CHwIICKelHQ/8EOKI39uXLkjd3waBjOzdjoO/Yj4v7Qfp394gefcBtzW6TqXwxdRMTMry/Y/cnFP38ysJNvQ92kYzMzK8g19X0TFzKwk39BPU/f0zcwa8g19j+mbmZXkG/q+iIqZWUm+oT/7z1lOfTOzunxDP00d+WZmDdmGPj4Ng5lZSbahL19ExcysJN/Q9/iOmVlJvqGfps58M7OGfEPfF1ExMyvJOPSLqcf0zcwa8g39NHVP38ysId/Q92kYzMxKsg19fBEVM7OSbEPfPX0zs7J8Q78+49Q3M5uVb+jL/5FrZtaq76Ev6UpJT0sak3Tziq0nTT2kb2bW0NfQl1QFPgNcBVwAXCfpgpVZVzFtDf2/fX2S145Nzfu8bnb8Ts/Uut5x7B3PK2dqpsaxqZnVLsP6aKYWHPzpG9Rq/rmqG+jz+i4BxiLiOQBJ9wHXAD/s9YoGq8Xvs39zzyhnnLyBDdUKUzM1Xj56HBXr5tQTBqjVgjeOz/ALJw5Si+DVN6c44+QNRMBMBLVaUAuoRTBTC2oRRMBJG6qcMFjl+EyNyakak9Mz1AI2VCucesIAQfGcxUQUYTRdCzZUKxw9Ps3JQwNEwAmD1TnrhcZfMJIa5xdahuU+RctciSi2/RvHp2e3c/0lZmrBm8dnGBqoMFCtMFARlYqYStuwWB+N56UXbH6dSmpXerC+fH07AVQrFQarYmigwuuT00xO1ZDg1WPTSPCWkzdQkYrXErOfaVD80q3Pz7a3+Rybt0vrJmrdYgttw0rL+yhqKK+ztZ6BavEeZ2rB1EzjvbfbZjMzwXQtOGlDddmf59rU+JmExnudmq5xfKY2+z4rErUIXn7jONWKOHGwSrUi3jw+w2BVnDBY5cQNVSLg1TenCJj9jlXq3zU1votS8dnU5vlOLP9dpGlE03wxlWD4lCH2fOzXul5Pq36H/tnAC033x4FfbV1I0k5gJ8C5557b0YrOP+tU/uu1FzF25HUmp2vM1GoMVisMnzrETK34IXnt2DQVwclDA7zy5hQSnHbCID87OkVFUK1oNhzq9+s/SEcnpzk2VWNosMLQQIWhgSqD1QpHp6Z5/dh04wuzhFo3DFSoVMTx6RonDlZ57dg01Yo4PlMr1iuVflg7+dIt9xmdfK9nIjg+XeOUoYH0Go0XkcRJG6pMTteYTr/oahEMVottWF/nbPARszXUfzAi/bDX50nLVCqimrbRTART0zWOpTqGBipEBGecPEQtgonXJ4kIarXiteb8cKfPrN5WWSQk24XzQtswiDlngK2HSP19tIZM/TWav0+SmK7VmJoOqlUxWBHVSmXOa9Zrq0XxvR2siqPHZ7LZw1Vp+mzq35XBaiV1nGYIgpla0aE6/6xTeOXNKY5NFd+7EzcMMD1T49j0DG9MziDgtBMHqVY0+4u1FpFu5V/Clcrcz6cTs591+i40Z4XSL6uTNqxMPPc79NttptL3MCJ2AbsARkZGOvqeViviX136S5081cwsW/3ekTsOnNN0fzNwqM81mJmtW/0O/e8C2yRtlbQB2AE81OcazMzWrb4O70TEtKSbgG8AVWB3RDzZzxrMzNazfo/pExEPAw/3e71mZpbxf+SamVmZQ9/MbB1x6JuZrSMOfTOzdURr/VwvkiaAn3Tw1DOBn/a4nF5wXcuzVuuCtVub61qeHOv6pYgYbvfAmg/9TkkajYiR1a6jletanrVaF6zd2lzX8qy3ujy8Y2a2jjj0zczWkZxDf9dqFzAP17U8a7UuWLu1ua7lWVd1ZTumb2ZmZTn39M3MrIVD38xsHcky9Pt18fUF1n9Q0hOS9kkaTW1nSNoj6Zk03ZjaJemOVOt+SRf3sI7dko5IOtDUtuw6JN2Qln9G0g0rVNetkl5M22yfpKubHrsl1fW0pCua2nv6OUs6R9JfSXpK0pOSfi+1r+o2W6CuVd1mkk6Q9LikH6S6/nNq3yrpO+m9fymdRh1JQ+n+WHp8y2L19riuuyQ937S9tqf2vn3302tWJX1f0tfS/f5ur+KaoPncKE7Z/CxwHrAB+AFwQZ9rOAic2dL2SeDmNH8z8N/S/NXA1ymuKnYp8J0e1vFu4GLgQKd1AGcAz6XpxjS/cQXquhX4922WvSB9hkPA1vTZVlficwY2ARen+VOBH6f1r+o2W6CuVd1m6X2fkuYHge+k7XA/sCO1fxb4SJr/XeCzaX4H8KWF6l2Buu4C/nmb5fv23U+v+zHgi8DX0v2+bq8ce/qzF1+PiONA/eLrq+0a4O40fzdwbVP7PVF4DDhd0qZerDAiHgVe7rKOK4A9EfFyRPwM2ANcuQJ1zeca4L6ImIyI54Exis+4559zRByOiO+l+deApyiu67yq22yBuubTl22W3vfr6e5gugXw68BXUnvr9qpvx68Al0vSAvX2uq759O27L2kz8BvAn6X7os/bK8fQb3fx9YV+QFZCAN+UtFfFRd4BzoqIw1D8EANvTe39rne5dfSzvpvSn9e760Moq1VX+lP67RS9xDWzzVrqglXeZmmoYh9whCIUnwX+LiKm26xjdv3p8VeAt/Sjroiob6/b0vb6lKSh1rpa1r8Sn+Ongf8A1NL9t9Dn7ZVj6C/p4usr7J0RcTFwFXCjpHcvsOxaqBfmr6Nf9d0JvA3YDhwGbl+tuiSdAjwAfDQiXl1o0X7W1qauVd9mETETEdsprnd9CfD3F1jHqtUl6SLgFuDvAf+QYsjm9/tZl6T3AkciYm9z8wLrWJG6cgz9Vb/4ekQcStMjwIMUPwwv1Ydt0vRIWrzf9S63jr7UFxEvpR/UGvA5Gn+u9rUuSYMUwfqFiPiL1Lzq26xdXWtlm6Va/g74a4ox8dMl1a/K17yO2fWnx3+BYpivH3VdmYbJIiImgf9B/7fXO4H3STpIMbT26xQ9//5ur253Sqy1G8UlIJ+j2MFR31l1YR/XfzJwatP8/6MYB/wj5u4M/GSa/w3m7kR6vMf1bGHuDtNl1UHRI3qeYkfWxjR/xgrUtalp/t9SjFkCXMjcnVbPUeyQ7PnnnN77PcCnW9pXdZstUNeqbjNgGDg9zZ8I/B/gvcCXmbtj8nfT/I3M3TF5/0L1rkBdm5q256eBP1yN73567cto7Mjt6/bqWbispRvF3vgfU4wvfrzP6z4vfSA/AJ6sr59iLO4R4Jk0PaPpC/iZVOsTwEgPa7mX4s/+KYrewQc7qQP4HYqdRWPAB1aorv+Z1rsfeIi5gfbxVNfTwFUr9TkD/5jiz+T9wL50u3q1t9kCda3qNgN+Bfh+Wv8B4D81/Qw8nt77l4Gh1H5Cuj+WHj9vsXp7XNf/TtvrAPDnNI7w6dt3v+l1L6MR+n3dXj4Ng5nZOpLjmL6Zmc3DoW9mto449M3M1hGHvpnZOuLQNzNbRxz6ZmbriEPfzGwd+f/ZNg2T2GuwXQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_plot(counter,loss_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Speichern der trainierten Netzwerke"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Path to save Network \n",
    "PATH = './Netzwerke_OneShot/' + current_time + '_' + Database + '_' + network_name + '_Train' + '.pth'\n",
    "\n",
    "## Save Network\n",
    "torch.save({\n",
    "                'epoch_range': epoch_range,\n",
    "                'model_state_dict': network.state_dict(),\n",
    "                'optimizer_state_dict': optimizer.state_dict(),\n",
    "                'loss': loss_contrastive\n",
    "            }, PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder_dataset_test = dset.ImageFolder(root=Config.path_OneShot_Test)\n",
    "\n",
    "siamese_dataset = SiameseDataset_Ears(imageFolderDataset=folder_dataset_test,\n",
    "                                        transform=transformer\n",
    "                                       ,should_invert=False)\n",
    "\n",
    "test_dataloader = DataLoader(siamese_dataset,batch_size=1,shuffle=True)\n",
    "\n",
    "## Test Function for Siamese-Network\n",
    "def evaluation(model, test_loader):\n",
    "    with torch.no_grad():\n",
    "        model.eval()\n",
    "        correct = 0\n",
    "        count = 0\n",
    "\n",
    "        for mainImg, imgSets, label in test_loader:\n",
    "            mainImg, imgSets, label = mainImg.to(DEVICE), imgSets.to(DEVICE), label.to(DEVICE)\n",
    "            predVal = 2.1\n",
    "            pred = -1\n",
    "            count += 1\n",
    "            \n",
    "            ## Determine which category an image belongs to\n",
    "            for i, testImg in enumerate(imgSets):\n",
    "                testImg = testImg.to(DEVICE)\n",
    "\n",
    "            if(network_name == ('VGG11' or 'VGG11bn' or 'ResNet18' or 'ResNet34' or 'alexnet' or 'Squeezenet1_0' or 'GoogLeNet' or 'Shufflenet_v2_x0_5' or 'Resnext101_32x8d')):\n",
    "                output1 = network(mainImg)\n",
    "                output2 = network(imgSets)\n",
    "                \n",
    "            elif(network_name == ('Siamese_Network')):\n",
    "                output1, output2 = network(mainImg, imgSets)\n",
    "                \n",
    "            else:\n",
    "                print('False Network choosen')\n",
    "\n",
    "      \n",
    "            euclidean_distance = F.pairwise_distance(output1, output2)\n",
    "            euclidean_distance = euclidean_distance.cpu().numpy()\n",
    "            \n",
    "\n",
    "            if(((euclidean_distance < predVal) and (label==0)) or ((euclidean_distance > predVal) and (label==1))):\n",
    "                correct += 1\n",
    "            print('Accuracy: {}'.format(correct/count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n",
      "Accuracy: 0.5\n",
      "Accuracy: 0.3333333333333333\n",
      "Accuracy: 0.25\n",
      "Accuracy: 0.4\n",
      "Accuracy: 0.3333333333333333\n",
      "Accuracy: 0.2857142857142857\n",
      "Accuracy: 0.375\n",
      "Accuracy: 0.3333333333333333\n",
      "Accuracy: 0.3\n",
      "Accuracy: 0.2727272727272727\n",
      "Accuracy: 0.25\n",
      "Accuracy: 0.3076923076923077\n",
      "Accuracy: 0.35714285714285715\n",
      "Accuracy: 0.4\n",
      "Accuracy: 0.375\n",
      "Accuracy: 0.35294117647058826\n",
      "Accuracy: 0.3888888888888889\n",
      "Accuracy: 0.42105263157894735\n",
      "Accuracy: 0.45\n",
      "Accuracy: 0.42857142857142855\n",
      "Accuracy: 0.4090909090909091\n",
      "Accuracy: 0.391304347826087\n",
      "Accuracy: 0.375\n",
      "Accuracy: 0.4\n",
      "Accuracy: 0.38461538461538464\n",
      "Accuracy: 0.37037037037037035\n",
      "Accuracy: 0.39285714285714285\n",
      "Accuracy: 0.3793103448275862\n",
      "Accuracy: 0.36666666666666664\n",
      "Accuracy: 0.3870967741935484\n",
      "Accuracy: 0.375\n",
      "Accuracy: 0.3939393939393939\n",
      "Accuracy: 0.4117647058823529\n",
      "Accuracy: 0.42857142857142855\n",
      "Accuracy: 0.4166666666666667\n",
      "Accuracy: 0.40540540540540543\n",
      "Accuracy: 0.42105263157894735\n",
      "Accuracy: 0.4358974358974359\n",
      "Accuracy: 0.45\n"
     ]
    }
   ],
   "source": [
    "evaluation(network, test_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}