BachelorarbeitCozmo/Notebooks/getting-started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unity ML-Agents Toolkit\n",
    "## Environment Basics\n",
    "This notebook contains a walkthrough of the basic functions of the Python API for the Unity ML-Agents toolkit. For instructions on building a Unity environment, see [here](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Getting-Started-with-Balance-Ball.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Set environment parameters\n",
    "\n",
    "Be sure to set `env_name` to the name of the Unity environment file you want to launch. Ensure that the environment build is in `../envs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_name = \"../envs/Bachelorarbeit-Cozmo\"  # Name of the Unity environment binary to launch\n",
    "train_mode = True  # Whether to run the environment in training or inference mode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Load dependencies\n",
    "\n",
    "The following loads the necessary dependencies and checks the Python version (at runtime). ML-Agents Toolkit (v0.3 onwards) requires Python 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python version:\n",
      "3.6.7 |Anaconda, Inc.| (default, Oct 28 2018, 19:44:12) [MSC v.1915 64 bit (AMD64)]\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import sys\n",
    "\n",
    "from mlagents.envs import UnityEnvironment\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "print(\"Python version:\")\n",
    "print(sys.version)\n",
    "\n",
    "# check Python version\n",
    "if (sys.version_info[0] < 3):\n",
    "    raise Exception(\"ERROR: ML-Agents Toolkit (v0.3 onwards) requires Python 3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Start the environment\n",
    "`UnityEnvironment` launches and begins communication with the environment when instantiated.\n",
    "\n",
    "Environments contain _brains_ which are responsible for deciding the actions of their associated _agents_. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:mlagents.envs:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of Training Brains : 1\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: CozmoLearningBrain\n",
      "        Number of Visual Observations (per agent): 1\n",
      "        Vector Observation space size (per agent): 0\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: discrete\n",
      "        Vector Action space size (per agent): [2, 2]\n",
      "        Vector Action descriptions: forward, rotation\n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name=env_name)\n",
    "\n",
    "# Set the default brain to work with\n",
    "default_brain = env.brain_names[0]\n",
    "brain = env.brains[default_brain]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Examine the observation and state spaces\n",
    "We can reset the environment to be provided with an initial set of observations and states for all the agents within the environment. In ML-Agents, _states_ refer to a vector of variables corresponding to relevant aspects of the environment for an agent. Likewise, _observations_ refer to a set of relevant pixel-wise visuals for an agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Agent state looks like: \n",
      "[]\n",
      "Agent observations look like:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD8CAYAAADzEfagAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAEI9JREFUeJzt3VuMnOV9x/Hvv+aQKqACJSDLdgtEvgiVKmJZFClRRE8JcGMihcq9wYqQXLUgJVJ74TRSQ+/aSkkl1JbIUVBMlXJoEoQv0jbIpaI3HOyUg4lL2CQUNrZsRTSENlJS4N+LeQaG9ezuvDPznma+H2k1M8++u/v3493f/p/nfWc2MhNJ0mR+oe0CJKlPDE1JqsDQlKQKDE1JqsDQlKQKDE1JqqC20IyIGyLihYhYiYgDdX0dSWpS1HGdZkRsAb4L/C6wCjwF/H5mfmfuX0ySGlRXp3ktsJKZ38/MnwP3A3tq+lqS1Jhzavq824BXRh6vAr+x3sER4dOSJLXtR5n5vs0Oqis0Y8zYu4IxIvYD+2v6+pJU1X9NclBdobkK7Bh5vB04OXpAZh4EDoKdpqT+qGtP8ylgZ0RcGRHnAXuBwzV9LUlqTC2dZma+ERF3AP8CbAHuyczn6/haktSkWi45qlyEy3NJ7TuWmbs3O8hnBElSBYamJFVgaEpSBXVdciRJ6+rCuZS1IsZdXn42Q1NaUm0G16QB1UWGptQDNb2wztw/56z6EOSGptSSKgHRVsA1HWJdDPK1DE2pBpOEzTwDoq5w60OINc3QlGY0LrDmFTaThqHh1hxDUxqxUUitF0wbBdZ6n2/SkDMMu8fQ1FJbG2rThNQ0Qav+MjS18OYdavMIWvWXoamFU0eojX5OQ3K5GZrqrVn3C6t8XoNSQ4amemcYavPuIIcMSG3E0FRvzCMs3Y/UrAxNdd6s+4nuR2qeDE112qzd5TyX8hIYmuqoeXSHBqbqYGiqczJzLkFnWKoOvnK7OmUegdnFF7jV4jA01Rl2mOoDQ1OSKjA0tbBcpqsOhqY6w2W1+sDQlKQKDE1JqsDQVCfMuv+YmW9/Di9qV528uF2tmPdJmtGAjIi5Xb4krWVoqnFrA22eAWqXqboZmmrcMNDqevUhA1N1MjTVKgNOfeOJIEmqwNBUK3y2jvrK0FSjRi8NqmNpPjxzLtXFPU01pu4z24almjBTaEbES8DrwJvAG5m5OyIuAR4ArgBeAn4vM/97tjK1KOoITP8GkJo0j+X5b2bmNZm5uzw+ABzJzJ3AkfJYS67Oi82Hn9fAVBPq2NPcAxwq9w8BN9fwNSSpFbOGZgLfiohjEbG/jF2emacAyu1lM34NLQC7QC2KWU8EfSgzT0bEZcAjEfGfk35gCdn9mx6o3qv7eeA+z1xNmqnTzMyT5fYM8BBwLXA6IrYClNsz63zswczcPbIXqgXkGW0tmqlDMyLeGxEXDu8DHwWOA4eBfeWwfcDDsxap/hlejxkRtXeBXpupJs2yPL8ceKj8QJwD/ENm/nNEPAU8GBG3AS8Dt8xepvqkrVcacpmuJkQXfkNHRPtFaG6aCq+137sGpmZ0bJLtQp8RpN7xYna1yeeea66a6DK9mF1tMjQlqQJDU3PT5F6mXabaYmhKUgWGpuai7e6vC1eBaDkYmpJUgaGpmbXdZYJn0tUcQ1O9NPpnM0bvS3Xz4nb1ztq/m26XqSbZaWomdnhaNoamZmanp2ViaEpSBYamZmKXqWVjaGpq7mdqGRmaklSBoSlJFRiamkoXngUktcHQlKQKDE1VZpepZWZoSlIFhqYkVWBoqhKX5lp2hqYmZmBKhqYkVWJoSlIFhqYm4tJcGjA0JakCQ1OSKjA0tSmX5tI7DE1JqsDQ1IbsMqV3MzQlqQJDU+uyy5TOZmhqLP/+jzTeOW0XoO6xw5TWZ6epd7HDlDa2aWhGxD0RcSYijo+MXRIRj0TEi+X24jIeEXFXRKxExLMRsavO4lUPu0xpfZN0ml8BblgzdgA4kpk7gSPlMcCNwM7yth+4ez5lSlI3bBqamfkY8Oqa4T3AoXL/EHDzyPi9OfA4cFFEbJ1XsaqXe5nS5qbd07w8M08BlNvLyvg24JWR41bL2FkiYn9EHI2Io1PWoCVmwKst8z57Pu67eOyZhcw8CBwEiAjPPrSsTyHkySq1adpO8/Rw2V1uz5TxVWDHyHHbgZPTlyeN15eA1+KZNjQPA/vK/X3AwyPjt5az6NcBrw2X8equPnWZYGCqXZsuzyPiPuB64NKIWAU+B/wF8GBE3Aa8DNxSDv8mcBOwAvwU+GQNNUtSa6IL+0Puabanb12mVKNjmbl7s4N8RpAkVWBoLjG7TKk6Q1OSKjA0JakCQ3NJuTSXpmNoSlIFhuYS6nOX2YVL5LTcDE1JqsDQXDJ97jKlLvBvBC2J4bLWwJRmY6e5BAxMaX4MzQVnYErz5fJ8QY2eZTYwpfkxNBeMYSnVy9BcQIalVB/3NNUbXtiuLrDTXCCLGipuOahLDM0Fs2ih4sX46hqX5wtk0cJl2GEuagetfrLTXBCL1JG5HFeXGZrqHINSXebyfAEs2vJ10f49WiyG5oKwO5OaYWhKUgWGZs+5lJWaZWgugEVami/SVQBaTIZmj9llSs0zNHvK18mU2mFo9piBKTXP0Owhl+VSewzNnrLLlNphaPaMXabULkOzh+wypfYYmpJUgaHZIy7NpfYZmj2zyEtznw2kPtg0NCPinog4ExHHR8bujIgfRsTT5e2mkfd9JiJWIuKFiPhYXYUvG7tMqRsm6TS/AtwwZvyvM/Oa8vZNgIi4GtgL/Fr5mL+LiC3zKnZZ+ewfqTs2Dc3MfAx4dcLPtwe4PzN/lpk/AFaAa2eoT4WBKXXDLHuad0TEs2X5fnEZ2wa8MnLMahmTpIUwbWjeDbwfuAY4BXy+jI9rh8ZuxkXE/og4GhFHp6xhadhlSt0xVWhm5unMfDMz3wK+xDtL8FVgx8ih24GT63yOg5m5OzN3T1PDsvAEkNQtU4VmRGwdefhxYHhm/TCwNyLOj4grgZ3Ak7OVKEndsemf8I2I+4DrgUsjYhX4HHB9RFzDYOn9EvAHAJn5fEQ8CHwHeAO4PTPfrKd0SWpedGH5FxHtF9FBXmokNerYJNuFPiOo4wxMqVsMzY7qwgpA0tkMzQ5yWS51l6HZUQam1E2GZse4LJe6zdDsILtMqbsMzQ6xy5S6z9DsmGXuMv2loT4wNNUJvmq7+sLQ7Ai7LKkfDM0OsdOSus/Q7AC7TKk/DM2W+eyfgWX/96s/DM0OMDDsttUfhqYkVWBotswuU+oXQ7NFLkml/jE01TovbFefGJotscuU+snQbJHdldQ/hqYkVWBotsA9PKm/DM2GuZcp9Zuh2QK7TKm/DM0G2WVK/WdoNswuU+o3Q7MhdpnSYjA0G2SXeTavJFDfGJoNsMuUFoeh2RC7KWkxGJo1s8ucjPOkvjA0G2CXKS0OQ7NGnuSYnPOkvjA0JakCQ7Mm7tFJi8nQrJFLTmnxbBqaEbEjIh6NiBMR8XxEfKqMXxIRj0TEi+X24jIeEXFXRKxExLMRsavuf4T6yT1f9dEkneYbwB9n5geA64DbI+Jq4ABwJDN3AkfKY4AbgZ3lbT9w99yr7gHDQFpMm4ZmZp7KzG+X+68DJ4BtwB7gUDnsEHBzub8HuDcHHgcuioitc6+8w9zPlBZXpT3NiLgC+CDwBHB5Zp6CQbACl5XDtgGvjHzYahmTpN47Z9IDI+IC4OvApzPzJxssP8e946zWKyL2M1i+LxS7TGmxTdRpRsS5DALzq5n5jTJ8erjsLrdnyvgqsGPkw7cDJ9d+zsw8mJm7M3P3tMV3zTAw3c+sxl806pNJzp4H8GXgRGZ+YeRdh4F95f4+4OGR8VvLWfTrgNeGy/hlYGBOzrlSH8Vmv+Uj4sPAvwPPAW+V4T9lsK/5IPArwMvALZn5agnZvwFuAH4KfDIzj27yNXrfathlVuclR+qYY5OsfDcNzSYsSmgaANU4Z+qYiULTZwSpFQam+srQnAMDQFoehqYkVWBozqgLe8KSmmNozoFLc2l5GJozsMucnXOovjE0p+R1mdJyMjRnYGDOzjlU3xiaU3BJOT/Opfpm4lc50jvsjmbjda3qMzvNijLT7khaYobmFOyS5se5VN8YmhX5Qz4/duzqI0NzCv6wS8vL0JyC3eZ8OI/qI0OzAjvM+XI+1UeG5oT8AZcEhmYlLiclGZoTsMuUNGRoTsgucz58NpD6ztDchF1mfQxP9ZGhOQF/uCUNGZobcCkpaS1DU5IqMDTX4V6mpHEMzQ24NK+Pv5TUV4bmBvzBlrSWr9w+hieAJK3HTnMNA7M+zq0WgaE5wuW4pM0YmmsMOyEDVNI4XdnT/BHwv+W2NaNLxwmXkZfScs1TaK3mKeZ3lHPdjGWu+VcnOSi60lFFxNHM3N12HVVYc3P6WLc1N6Ppml2eS1IFhqYkVdCl0DzYdgFTsObm9LFua25GozV3Zk9TkvqgS52mJHVe66EZETdExAsRsRIRB9quZz0R8VJEPBcRT0fE0TJ2SUQ8EhEvltuLO1DnPRFxJiKOj4yNrTMG7ipz/2xE7OpQzXdGxA/LfD8dETeNvO8zpeYXIuJjLdW8IyIejYgTEfF8RHyqjHd2rjeouetz/Z6IeDIinil1/3kZvzIinihz/UBEnFfGzy+PV8r7r5hrQZnZ2huwBfgecBVwHvAMcHWbNW1Q60vApWvG/go4UO4fAP6yA3V+BNgFHN+sTuAm4J+AAK4DnuhQzXcCfzLm2KvL98n5wJXl+2dLCzVvBXaV+xcC3y21dXauN6i563MdwAXl/rnAE2UOHwT2lvEvAn9Y7v8R8MVyfy/wwDzrabvTvBZYyczvZ+bPgfuBPS3XVMUe4FC5fwi4ucVaAMjMx4BX1wyvV+ce4N4ceBy4KCK2NlPpO9apeT17gPsz82eZ+QNghcH3UaMy81Rmfrvcfx04AWyjw3O9Qc3r6cpcZ2b+T3l4bnlL4LeAr5XxtXM9/D/4GvDbMccXPWg7NLcBr4w8XmXj/8Q2JfCtiDgWEfvL2OWZeQoG35DAZa1Vt7H16uz6/N9RlrL3jGx9dK7msvz7IIMOqBdzvaZm6PhcR8SWiHgaOAM8wqDr/XFmvjGmtrfrLu9/DfjledXSdmiOS/+uns7/UGbuAm4Ebo+Ij7Rd0Bx0ef7vBt4PXAOcAj5fxjtVc0RcAHwd+HRm/mSjQ8eMtVL3mJo7P9eZ+WZmXgNsZ9DtfmDcYeW21rrbDs1VYMfI4+3AyZZq2VBmniy3Z4CHGPzHnR4uscrtmfYq3NB6dXZ2/jPzdPlBeQv4Eu8sCztTc0ScyyB8vpqZ3yjDnZ7rcTX3Ya6HMvPHwL8x2NO8KCKGr58xWtvbdZf3/xKTb/9squ3QfArYWc6Cncdg0/ZwyzWdJSLeGxEXDu8DHwWOM6h1XzlsH/BwOxVuar06DwO3ljO71wGvDZeWbVuz3/dxBvMNg5r3ljOkVwI7gSdbqC+ALwMnMvMLI+/q7FyvV3MP5vp9EXFRuf+LwO8w2I99FPhEOWztXA//Dz4B/GuWs0Jz0fSZsDFnxm5icBbve8Bn265nnRqvYnAW8Rng+WGdDPZJjgAvlttLOlDrfQyWWP/H4DfubevVyWAZ87dl7p8Ddneo5r8vNT1bfgi2jhz/2VLzC8CNLdX8YQZLvmeBp8vbTV2e6w1q7vpc/zrwH6W+48CflfGrGIT4CvCPwPll/D3l8Up5/1XzrMdnBElSBW0vzyWpVwxNSarA0JSkCgxNSarA0JSkCgxNSarA0JSkCgxNSarg/wHuebcd/X63rwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Reset the environment\n",
    "env_info = env.reset(train_mode=train_mode)[default_brain]\n",
    "\n",
    "# Examine the state space for the default brain\n",
    "print(\"Agent state looks like: \\n{}\".format(env_info.vector_observations[0]))\n",
    "\n",
    "# Examine the observation space for the default brain\n",
    "for observation in env_info.visual_observations:\n",
    "    print(\"Agent observations look like:\") \n",
    "    \n",
    "    #change to numpy array instead of list\n",
    "    #observation = cv2.Canny(observation,100,100)\n",
    "    observation = np.array(observation)\n",
    "    #slice1Copy = np.uint8(observation)\n",
    "    #edges = cv2.Canny(slice1Copy,100,100)\n",
    "    \n",
    "    \n",
    "    #if edges.shape[3] == 3:\n",
    "    #plt.imshow(observation[0])\n",
    "    #else:\n",
    "       # plt.imshow(edges[0,:,:,0])\n",
    "        \n",
    "    if observation.shape[3] == 3:\n",
    "        plt.imshow(observation[0,:,:,:])\n",
    "    else:\n",
    "        plt.imshow(observation[0,:,:,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Take random actions in the environment\n",
    "Once we restart an environment, we can step the environment forward and provide actions to all of the agents within the environment. Here we simply choose random actions based on the `action_space_type` of the default brain. \n",
    "\n",
    "Once this cell is executed, 10 messages will be printed that detail how much reward will be accumulated for the next 10 episodes. The Unity environment will then pause, waiting for further signals telling it what to do next. Thus, not seeing any animation is expected when running this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n",
      "Total reward this episode: 1.0\n"
     ]
    }
   ],
   "source": [
    "for episode in range(1000):\n",
    "    env_info = env.reset(train_mode=train_mode)[default_brain]\n",
    "    done = False\n",
    "    episode_rewards = 0\n",
    "    while not done:\n",
    "        action_size = brain.vector_action_space_size\n",
    "        if brain.vector_action_space_type == 'continuous':\n",
    "            env_info = env.step(np.random.randn(len(env_info.agents), \n",
    "                                                action_size[0]))[default_brain]\n",
    "        else:\n",
    "            action = np.column_stack([np.random.randint(0, action_size[i], size=(len(env_info.agents))) for i in range(len(action_size))])\n",
    "            env_info = env.step(action)[default_brain]\n",
    "        episode_rewards += env_info.rewards[0]\n",
    "        done = env_info.local_done[0]\n",
    "    print(\"Total reward this episode: {}\".format(episode_rewards))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Close the environment when finished\n",
    "When we are finished using an environment, we can close it with the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}