18_reinforcement_learning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dK5SmJpnMM89"
      },
      "source": [
        "**18장 – 강화학습**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u9vrGAroMM9B"
      },
      "source": [
        "_이 노트북은 18장에 있는 모든 샘플 코드를 담고 있습니다._"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B1XZUYIUMM9C"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/rickiepark/handson-ml2/blob/master/18_reinforcement_learning.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />구글 코랩에서 실행하기</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R7Ch9S54MM9C"
      },
      "source": [
        "# 설정"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NYj6MAH1MM9D"
      },
      "source": [
        "먼저 몇 개의 모듈을 임포트합니다. 맷플롯립 그래프를 인라인으로 출력하도록 만들고 그림을 저장하는 함수를 준비합니다. 또한 파이썬 버전이 3.5 이상인지 확인합니다(파이썬 2.x에서도 동작하지만 곧 지원이 중단되므로 파이썬 3을 사용하는 것이 좋습니다). 사이킷런 버전이 0.20 이상인지와 텐서플로 버전이 2.0 이상인지 확인합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I6OcREmCMM9D"
      },
      "outputs": [],
      "source": [
        "# 파이썬 ≥3.5 필수\n",
        "import sys\n",
        "assert sys.version_info >= (3, 5)\n",
        "\n",
        "# 코랩에서 실행하고 있나요?\n",
        "IS_COLAB = \"google.colab\" in sys.modules\n",
        "\n",
        "if IS_COLAB:\n",
        "    !apt update && apt install -y libpq-dev libsdl2-dev swig xorg-dev xvfb\n",
        "    %pip install -U tf-agents==0.13.0 pyvirtualdisplay\n",
        "    %pip install -U gym~=0.21.0\n",
        "    %pip install -U gym[box2d,atari,accept-rom-license]\n",
        "    %pip install pyglet==1.5.27\n",
        "\n",
        "# 사이킷런 ≥0.20 필수\n",
        "import sklearn\n",
        "assert sklearn.__version__ >= \"0.20\"\n",
        "\n",
        "# 텐서플로 ≥2.0 필수\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "assert tf.__version__ >= \"2.0\"\n",
        "\n",
        "if not tf.config.list_physical_devices('GPU'):\n",
        "    print(\"감지된 GPU가 없습니다. GPU가 없으면 LSTM과 CNN이 매우 느릴 수 있습니다.\")\n",
        "    if IS_COLAB:\n",
        "        print(\"런타임 > 런타임 유형 변경 메뉴를 선택하고 하드웨어 가속기로 GPU를 고르세요.\")\n",
        "\n",
        "# 공통 모듈 임포트\n",
        "import numpy as np\n",
        "import os\n",
        "\n",
        "# 노트북 실행 결과를 동일하게 유지하기 위해\n",
        "np.random.seed(42)\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "# 깔끔한 그래프 출력을 위해\n",
        "%matplotlib inline\n",
        "import matplotlib as mpl\n",
        "import matplotlib.pyplot as plt\n",
        "mpl.rc('axes', labelsize=14)\n",
        "mpl.rc('xtick', labelsize=12)\n",
        "mpl.rc('ytick', labelsize=12)\n",
        "\n",
        "# 부드러운 애니메이션을 위해\n",
        "import matplotlib.animation as animation\n",
        "mpl.rc('animation', html='jshtml')\n",
        "\n",
        "# 그림을 저장할 위치\n",
        "PROJECT_ROOT_DIR = \".\"\n",
        "CHAPTER_ID = \"rl\"\n",
        "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
        "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
        "\n",
        "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
        "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
        "    print(\"그림 저장\", fig_id)\n",
        "    if tight_layout:\n",
        "        plt.tight_layout()\n",
        "    plt.savefig(path, format=fig_extension, dpi=resolution)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HXIRyZZ8MM9F"
      },
      "source": [
        "# OpenAI 짐 소개"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vuke9TNMM9G"
      },
      "source": [
        "이 노트북은 강화학습 알고리즘을 개발하고 평가하는 훌륭한 도구인 [OpenAI 짐(gym)](https://gym.openai.com/)을 사용합니다. 학습 에이전트가 상호작용하기 위한 환경을 많이 제공합니다. 먼저 `gym`을 임포트합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "btRLkc6iMM9H"
      },
      "outputs": [],
      "source": [
        "import gym"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_7k2ardmMM9H"
      },
      "source": [
        "가능한 환경 목록을 확인해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OWKEgVSnMM9I",
        "outputId": "7385a9ab-2e43-490a-e703-7716829ac08f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "dict_values([EnvSpec(Copy-v0), EnvSpec(RepeatCopy-v0), EnvSpec(ReversedAddition-v0), EnvSpec(ReversedAddition3-v0), EnvSpec(DuplicatedInput-v0), EnvSpec(Reverse-v0), EnvSpec(CartPole-v0), EnvSpec(CartPole-v1), EnvSpec(MountainCar-v0), EnvSpec(MountainCarContinuous-v0), EnvSpec(Pendulum-v0), EnvSpec(Acrobot-v1), EnvSpec(LunarLander-v2), EnvSpec(LunarLanderContinuous-v2), EnvSpec(BipedalWalker-v3), EnvSpec(BipedalWalkerHardcore-v3), EnvSpec(CarRacing-v0), EnvSpec(Blackjack-v0), EnvSpec(KellyCoinflip-v0), EnvSpec(KellyCoinflipGeneralized-v0), EnvSpec(FrozenLake-v0), EnvSpec(FrozenLake8x8-v0), EnvSpec(CliffWalking-v0), EnvSpec(NChain-v0), EnvSpec(Roulette-v0), EnvSpec(Taxi-v3), EnvSpec(GuessingGame-v0), EnvSpec(HotterColder-v0), EnvSpec(Reacher-v2), EnvSpec(Pusher-v2), EnvSpec(Thrower-v2), EnvSpec(Striker-v2), EnvSpec(InvertedPendulum-v2), EnvSpec(InvertedDoublePendulum-v2), EnvSpec(HalfCheetah-v2), EnvSpec(HalfCheetah-v3), EnvSpec(Hopper-v2), EnvSpec(Hopper-v3), EnvSpec(Swimmer-v2), EnvSpec(Swimmer-v3), EnvSpec(Walker2d-v2), EnvSpec(Walker2d-v3), EnvSpec(Ant-v2), EnvSpec(Ant-v3), EnvSpec(Humanoid-v2), EnvSpec(Humanoid-v3), EnvSpec(HumanoidStandup-v2), EnvSpec(FetchSlide-v1), EnvSpec(FetchPickAndPlace-v1), EnvSpec(FetchReach-v1), EnvSpec(FetchPush-v1), EnvSpec(HandReach-v0), EnvSpec(HandManipulateBlockRotateZ-v0), EnvSpec(HandManipulateBlockRotateZTouchSensors-v0), EnvSpec(HandManipulateBlockRotateZTouchSensors-v1), EnvSpec(HandManipulateBlockRotateParallel-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v1), EnvSpec(HandManipulateBlockRotateXYZ-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v1), EnvSpec(HandManipulateBlockFull-v0), EnvSpec(HandManipulateBlock-v0), EnvSpec(HandManipulateBlockTouchSensors-v0), EnvSpec(HandManipulateBlockTouchSensors-v1), EnvSpec(HandManipulateEggRotate-v0), EnvSpec(HandManipulateEggRotateTouchSensors-v0), EnvSpec(HandManipulateEggRotateTouchSensors-v1), EnvSpec(HandManipulateEggFull-v0), EnvSpec(HandManipulateEgg-v0), EnvSpec(HandManipulateEggTouchSensors-v0), EnvSpec(HandManipulateEggTouchSensors-v1), EnvSpec(HandManipulatePenRotate-v0), EnvSpec(HandManipulatePenRotateTouchSensors-v0), EnvSpec(HandManipulatePenRotateTouchSensors-v1), EnvSpec(HandManipulatePenFull-v0), EnvSpec(HandManipulatePen-v0), EnvSpec(HandManipulatePenTouchSensors-v0), EnvSpec(HandManipulatePenTouchSensors-v1), EnvSpec(FetchSlideDense-v1), EnvSpec(FetchPickAndPlaceDense-v1), EnvSpec(FetchReachDense-v1), EnvSpec(FetchPushDense-v1), EnvSpec(HandReachDense-v0), EnvSpec(HandManipulateBlockRotateZDense-v0), EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v1), EnvSpec(HandManipulateBlockRotateParallelDense-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v1), EnvSpec(HandManipulateBlockRotateXYZDense-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v1), EnvSpec(HandManipulateBlockFullDense-v0), EnvSpec(HandManipulateBlockDense-v0), EnvSpec(HandManipulateBlockTouchSensorsDense-v0), EnvSpec(HandManipulateBlockTouchSensorsDense-v1), EnvSpec(HandManipulateEggRotateDense-v0), EnvSpec(HandManipulateEggRotateTouchSensorsDense-v0), EnvSpec(HandManipulateEggRotateTouchSensorsDense-v1), EnvSpec(HandManipulateEggFullDense-v0), EnvSpec(HandManipulateEggDense-v0), EnvSpec(HandManipulateEggTouchSensorsDense-v0), EnvSpec(HandManipulateEggTouchSensorsDense-v1), EnvSpec(HandManipulatePenRotateDense-v0), EnvSpec(HandManipulatePenRotateTouchSensorsDense-v0), EnvSpec(HandManipulatePenRotateTouchSensorsDense-v1), EnvSpec(HandManipulatePenFullDense-v0), EnvSpec(HandManipulatePenDense-v0), EnvSpec(HandManipulatePenTouchSensorsDense-v0), EnvSpec(HandManipulatePenTouchSensorsDense-v1), EnvSpec(Adventure-v0), EnvSpec(Adventure-v4), EnvSpec(AdventureDeterministic-v0), EnvSpec(AdventureDeterministic-v4), EnvSpec(AdventureNoFrameskip-v0), EnvSpec(AdventureNoFrameskip-v4), EnvSpec(Adventure-ram-v0), EnvSpec(Adventure-ram-v4), EnvSpec(Adventure-ramDeterministic-v0), EnvSpec(Adventure-ramDeterministic-v4), EnvSpec(Adventure-ramNoFrameskip-v0), EnvSpec(Adventure-ramNoFrameskip-v4), EnvSpec(AirRaid-v0), EnvSpec(AirRaid-v4), EnvSpec(AirRaidDeterministic-v0), EnvSpec(AirRaidDeterministic-v4), EnvSpec(AirRaidNoFrameskip-v0), EnvSpec(AirRaidNoFrameskip-v4), EnvSpec(AirRaid-ram-v0), EnvSpec(AirRaid-ram-v4), EnvSpec(AirRaid-ramDeterministic-v0), EnvSpec(AirRaid-ramDeterministic-v4), EnvSpec(AirRaid-ramNoFrameskip-v0), EnvSpec(AirRaid-ramNoFrameskip-v4), EnvSpec(Alien-v0), EnvSpec(Alien-v4), EnvSpec(AlienDeterministic-v0), EnvSpec(AlienDeterministic-v4), EnvSpec(AlienNoFrameskip-v0), EnvSpec(AlienNoFrameskip-v4), EnvSpec(Alien-ram-v0), EnvSpec(Alien-ram-v4), EnvSpec(Alien-ramDeterministic-v0), EnvSpec(Alien-ramDeterministic-v4), EnvSpec(Alien-ramNoFrameskip-v0), EnvSpec(Alien-ramNoFrameskip-v4), EnvSpec(Amidar-v0), EnvSpec(Amidar-v4), EnvSpec(AmidarDeterministic-v0), EnvSpec(AmidarDeterministic-v4), EnvSpec(AmidarNoFrameskip-v0), EnvSpec(AmidarNoFrameskip-v4), EnvSpec(Amidar-ram-v0), EnvSpec(Amidar-ram-v4), EnvSpec(Amidar-ramDeterministic-v0), EnvSpec(Amidar-ramDeterministic-v4), EnvSpec(Amidar-ramNoFrameskip-v0), EnvSpec(Amidar-ramNoFrameskip-v4), EnvSpec(Assault-v0), EnvSpec(Assault-v4), EnvSpec(AssaultDeterministic-v0), EnvSpec(AssaultDeterministic-v4), EnvSpec(AssaultNoFrameskip-v0), EnvSpec(AssaultNoFrameskip-v4), EnvSpec(Assault-ram-v0), EnvSpec(Assault-ram-v4), EnvSpec(Assault-ramDeterministic-v0), EnvSpec(Assault-ramDeterministic-v4), EnvSpec(Assault-ramNoFrameskip-v0), EnvSpec(Assault-ramNoFrameskip-v4), EnvSpec(Asterix-v0), EnvSpec(Asterix-v4), EnvSpec(AsterixDeterministic-v0), EnvSpec(AsterixDeterministic-v4), EnvSpec(AsterixNoFrameskip-v0), EnvSpec(AsterixNoFrameskip-v4), EnvSpec(Asterix-ram-v0), EnvSpec(Asterix-ram-v4), EnvSpec(Asterix-ramDeterministic-v0), EnvSpec(Asterix-ramDeterministic-v4), EnvSpec(Asterix-ramNoFrameskip-v0), EnvSpec(Asterix-ramNoFrameskip-v4), EnvSpec(Asteroids-v0), EnvSpec(Asteroids-v4), EnvSpec(AsteroidsDeterministic-v0), EnvSpec(AsteroidsDeterministic-v4), EnvSpec(AsteroidsNoFrameskip-v0), EnvSpec(AsteroidsNoFrameskip-v4), EnvSpec(Asteroids-ram-v0), EnvSpec(Asteroids-ram-v4), EnvSpec(Asteroids-ramDeterministic-v0), EnvSpec(Asteroids-ramDeterministic-v4), EnvSpec(Asteroids-ramNoFrameskip-v0), EnvSpec(Asteroids-ramNoFrameskip-v4), EnvSpec(Atlantis-v0), EnvSpec(Atlantis-v4), EnvSpec(AtlantisDeterministic-v0), EnvSpec(AtlantisDeterministic-v4), EnvSpec(AtlantisNoFrameskip-v0), EnvSpec(AtlantisNoFrameskip-v4), EnvSpec(Atlantis-ram-v0), EnvSpec(Atlantis-ram-v4), EnvSpec(Atlantis-ramDeterministic-v0), EnvSpec(Atlantis-ramDeterministic-v4), EnvSpec(Atlantis-ramNoFrameskip-v0), EnvSpec(Atlantis-ramNoFrameskip-v4), EnvSpec(BankHeist-v0), EnvSpec(BankHeist-v4), EnvSpec(BankHeistDeterministic-v0), EnvSpec(BankHeistDeterministic-v4), EnvSpec(BankHeistNoFrameskip-v0), EnvSpec(BankHeistNoFrameskip-v4), EnvSpec(BankHeist-ram-v0), EnvSpec(BankHeist-ram-v4), EnvSpec(BankHeist-ramDeterministic-v0), EnvSpec(BankHeist-ramDeterministic-v4), EnvSpec(BankHeist-ramNoFrameskip-v0), EnvSpec(BankHeist-ramNoFrameskip-v4), EnvSpec(BattleZone-v0), EnvSpec(BattleZone-v4), EnvSpec(BattleZoneDeterministic-v0), EnvSpec(BattleZoneDeterministic-v4), EnvSpec(BattleZoneNoFrameskip-v0), EnvSpec(BattleZoneNoFrameskip-v4), EnvSpec(BattleZone-ram-v0), EnvSpec(BattleZone-ram-v4), EnvSpec(BattleZone-ramDeterministic-v0), EnvSpec(BattleZone-ramDeterministic-v4), EnvSpec(BattleZone-ramNoFrameskip-v0), EnvSpec(BattleZone-ramNoFrameskip-v4), EnvSpec(BeamRider-v0), EnvSpec(BeamRider-v4), EnvSpec(BeamRiderDeterministic-v0), EnvSpec(BeamRiderDeterministic-v4), EnvSpec(BeamRiderNoFrameskip-v0), EnvSpec(BeamRiderNoFrameskip-v4), EnvSpec(BeamRider-ram-v0), EnvSpec(BeamRider-ram-v4), EnvSpec(BeamRider-ramDeterministic-v0), EnvSpec(BeamRider-ramDeterministic-v4), EnvSpec(BeamRider-ramNoFrameskip-v0), EnvSpec(BeamRider-ramNoFrameskip-v4), EnvSpec(Berzerk-v0), EnvSpec(Berzerk-v4), EnvSpec(BerzerkDeterministic-v0), EnvSpec(BerzerkDeterministic-v4), EnvSpec(BerzerkNoFrameskip-v0), EnvSpec(BerzerkNoFrameskip-v4), EnvSpec(Berzerk-ram-v0), EnvSpec(Berzerk-ram-v4), EnvSpec(Berzerk-ramDeterministic-v0), EnvSpec(Berzerk-ramDeterministic-v4), EnvSpec(Berzerk-ramNoFrameskip-v0), EnvSpec(Berzerk-ramNoFrameskip-v4), EnvSpec(Bowling-v0), EnvSpec(Bowling-v4), EnvSpec(BowlingDeterministic-v0), EnvSpec(BowlingDeterministic-v4), EnvSpec(BowlingNoFrameskip-v0), EnvSpec(BowlingNoFrameskip-v4), EnvSpec(Bowling-ram-v0), EnvSpec(Bowling-ram-v4), EnvSpec(Bowling-ramDeterministic-v0), EnvSpec(Bowling-ramDeterministic-v4), EnvSpec(Bowling-ramNoFrameskip-v0), EnvSpec(Bowling-ramNoFrameskip-v4), EnvSpec(Boxing-v0), EnvSpec(Boxing-v4), EnvSpec(BoxingDeterministic-v0), EnvSpec(BoxingDeterministic-v4), EnvSpec(BoxingNoFrameskip-v0), EnvSpec(BoxingNoFrameskip-v4), EnvSpec(Boxing-ram-v0), EnvSpec(Boxing-ram-v4), EnvSpec(Boxing-ramDeterministic-v0), EnvSpec(Boxing-ramDeterministic-v4), EnvSpec(Boxing-ramNoFrameskip-v0), EnvSpec(Boxing-ramNoFrameskip-v4), EnvSpec(Breakout-v0), EnvSpec(Breakout-v4), EnvSpec(BreakoutDeterministic-v0), EnvSpec(BreakoutDeterministic-v4), EnvSpec(BreakoutNoFrameskip-v0), EnvSpec(BreakoutNoFrameskip-v4), EnvSpec(Breakout-ram-v0), EnvSpec(Breakout-ram-v4), EnvSpec(Breakout-ramDeterministic-v0), EnvSpec(Breakout-ramDeterministic-v4), EnvSpec(Breakout-ramNoFrameskip-v0), EnvSpec(Breakout-ramNoFrameskip-v4), EnvSpec(Carnival-v0), EnvSpec(Carnival-v4), EnvSpec(CarnivalDeterministic-v0), EnvSpec(CarnivalDeterministic-v4), EnvSpec(CarnivalNoFrameskip-v0), EnvSpec(CarnivalNoFrameskip-v4), EnvSpec(Carnival-ram-v0), EnvSpec(Carnival-ram-v4), EnvSpec(Carnival-ramDeterministic-v0), EnvSpec(Carnival-ramDeterministic-v4), EnvSpec(Carnival-ramNoFrameskip-v0), EnvSpec(Carnival-ramNoFrameskip-v4), EnvSpec(Centipede-v0), EnvSpec(Centipede-v4), EnvSpec(CentipedeDeterministic-v0), EnvSpec(CentipedeDeterministic-v4), EnvSpec(CentipedeNoFrameskip-v0), EnvSpec(CentipedeNoFrameskip-v4), EnvSpec(Centipede-ram-v0), EnvSpec(Centipede-ram-v4), EnvSpec(Centipede-ramDeterministic-v0), EnvSpec(Centipede-ramDeterministic-v4), EnvSpec(Centipede-ramNoFrameskip-v0), EnvSpec(Centipede-ramNoFrameskip-v4), EnvSpec(ChopperCommand-v0), EnvSpec(ChopperCommand-v4), EnvSpec(ChopperCommandDeterministic-v0), EnvSpec(ChopperCommandDeterministic-v4), EnvSpec(ChopperCommandNoFrameskip-v0), EnvSpec(ChopperCommandNoFrameskip-v4), EnvSpec(ChopperCommand-ram-v0), EnvSpec(ChopperCommand-ram-v4), EnvSpec(ChopperCommand-ramDeterministic-v0), EnvSpec(ChopperCommand-ramDeterministic-v4), EnvSpec(ChopperCommand-ramNoFrameskip-v0), EnvSpec(ChopperCommand-ramNoFrameskip-v4), EnvSpec(CrazyClimber-v0), EnvSpec(CrazyClimber-v4), EnvSpec(CrazyClimberDeterministic-v0), EnvSpec(CrazyClimberDeterministic-v4), EnvSpec(CrazyClimberNoFrameskip-v0), EnvSpec(CrazyClimberNoFrameskip-v4), EnvSpec(CrazyClimber-ram-v0), EnvSpec(CrazyClimber-ram-v4), EnvSpec(CrazyClimber-ramDeterministic-v0), EnvSpec(CrazyClimber-ramDeterministic-v4), EnvSpec(CrazyClimber-ramNoFrameskip-v0), EnvSpec(CrazyClimber-ramNoFrameskip-v4), EnvSpec(Defender-v0), EnvSpec(Defender-v4), EnvSpec(DefenderDeterministic-v0), EnvSpec(DefenderDeterministic-v4), EnvSpec(DefenderNoFrameskip-v0), EnvSpec(DefenderNoFrameskip-v4), EnvSpec(Defender-ram-v0), EnvSpec(Defender-ram-v4), EnvSpec(Defender-ramDeterministic-v0), EnvSpec(Defender-ramDeterministic-v4), EnvSpec(Defender-ramNoFrameskip-v0), EnvSpec(Defender-ramNoFrameskip-v4), EnvSpec(DemonAttack-v0), EnvSpec(DemonAttack-v4), EnvSpec(DemonAttackDeterministic-v0), EnvSpec(DemonAttackDeterministic-v4), EnvSpec(DemonAttackNoFrameskip-v0), EnvSpec(DemonAttackNoFrameskip-v4), EnvSpec(DemonAttack-ram-v0), EnvSpec(DemonAttack-ram-v4), EnvSpec(DemonAttack-ramDeterministic-v0), EnvSpec(DemonAttack-ramDeterministic-v4), EnvSpec(DemonAttack-ramNoFrameskip-v0), EnvSpec(DemonAttack-ramNoFrameskip-v4), EnvSpec(DoubleDunk-v0), EnvSpec(DoubleDunk-v4), EnvSpec(DoubleDunkDeterministic-v0), EnvSpec(DoubleDunkDeterministic-v4), EnvSpec(DoubleDunkNoFrameskip-v0), EnvSpec(DoubleDunkNoFrameskip-v4), EnvSpec(DoubleDunk-ram-v0), EnvSpec(DoubleDunk-ram-v4), EnvSpec(DoubleDunk-ramDeterministic-v0), EnvSpec(DoubleDunk-ramDeterministic-v4), EnvSpec(DoubleDunk-ramNoFrameskip-v0), EnvSpec(DoubleDunk-ramNoFrameskip-v4), EnvSpec(ElevatorAction-v0), EnvSpec(ElevatorAction-v4), EnvSpec(ElevatorActionDeterministic-v0), EnvSpec(ElevatorActionDeterministic-v4), EnvSpec(ElevatorActionNoFrameskip-v0), EnvSpec(ElevatorActionNoFrameskip-v4), EnvSpec(ElevatorAction-ram-v0), EnvSpec(ElevatorAction-ram-v4), EnvSpec(ElevatorAction-ramDeterministic-v0), EnvSpec(ElevatorAction-ramDeterministic-v4), EnvSpec(ElevatorAction-ramNoFrameskip-v0), EnvSpec(ElevatorAction-ramNoFrameskip-v4), EnvSpec(Enduro-v0), EnvSpec(Enduro-v4), EnvSpec(EnduroDeterministic-v0), EnvSpec(EnduroDeterministic-v4), EnvSpec(EnduroNoFrameskip-v0), EnvSpec(EnduroNoFrameskip-v4), EnvSpec(Enduro-ram-v0), EnvSpec(Enduro-ram-v4), EnvSpec(Enduro-ramDeterministic-v0), EnvSpec(Enduro-ramDeterministic-v4), EnvSpec(Enduro-ramNoFrameskip-v0), EnvSpec(Enduro-ramNoFrameskip-v4), EnvSpec(FishingDerby-v0), EnvSpec(FishingDerby-v4), EnvSpec(FishingDerbyDeterministic-v0), EnvSpec(FishingDerbyDeterministic-v4), EnvSpec(FishingDerbyNoFrameskip-v0), EnvSpec(FishingDerbyNoFrameskip-v4), EnvSpec(FishingDerby-ram-v0), EnvSpec(FishingDerby-ram-v4), EnvSpec(FishingDerby-ramDeterministic-v0), EnvSpec(FishingDerby-ramDeterministic-v4), EnvSpec(FishingDerby-ramNoFrameskip-v0), EnvSpec(FishingDerby-ramNoFrameskip-v4), EnvSpec(Freeway-v0), EnvSpec(Freeway-v4), EnvSpec(FreewayDeterministic-v0), EnvSpec(FreewayDeterministic-v4), EnvSpec(FreewayNoFrameskip-v0), EnvSpec(FreewayNoFrameskip-v4), EnvSpec(Freeway-ram-v0), EnvSpec(Freeway-ram-v4), EnvSpec(Freeway-ramDeterministic-v0), EnvSpec(Freeway-ramDeterministic-v4), EnvSpec(Freeway-ramNoFrameskip-v0), EnvSpec(Freeway-ramNoFrameskip-v4), EnvSpec(Frostbite-v0), EnvSpec(Frostbite-v4), EnvSpec(FrostbiteDeterministic-v0), EnvSpec(FrostbiteDeterministic-v4), EnvSpec(FrostbiteNoFrameskip-v0), EnvSpec(FrostbiteNoFrameskip-v4), EnvSpec(Frostbite-ram-v0), EnvSpec(Frostbite-ram-v4), EnvSpec(Frostbite-ramDeterministic-v0), EnvSpec(Frostbite-ramDeterministic-v4), EnvSpec(Frostbite-ramNoFrameskip-v0), EnvSpec(Frostbite-ramNoFrameskip-v4), EnvSpec(Gopher-v0), EnvSpec(Gopher-v4), EnvSpec(GopherDeterministic-v0), EnvSpec(GopherDeterministic-v4), EnvSpec(GopherNoFrameskip-v0), EnvSpec(GopherNoFrameskip-v4), EnvSpec(Gopher-ram-v0), EnvSpec(Gopher-ram-v4), EnvSpec(Gopher-ramDeterministic-v0), EnvSpec(Gopher-ramDeterministic-v4), EnvSpec(Gopher-ramNoFrameskip-v0), EnvSpec(Gopher-ramNoFrameskip-v4), EnvSpec(Gravitar-v0), EnvSpec(Gravitar-v4), EnvSpec(GravitarDeterministic-v0), EnvSpec(GravitarDeterministic-v4), EnvSpec(GravitarNoFrameskip-v0), EnvSpec(GravitarNoFrameskip-v4), EnvSpec(Gravitar-ram-v0), EnvSpec(Gravitar-ram-v4), EnvSpec(Gravitar-ramDeterministic-v0), EnvSpec(Gravitar-ramDeterministic-v4), EnvSpec(Gravitar-ramNoFrameskip-v0), EnvSpec(Gravitar-ramNoFrameskip-v4), EnvSpec(Hero-v0), EnvSpec(Hero-v4), EnvSpec(HeroDeterministic-v0), EnvSpec(HeroDeterministic-v4), EnvSpec(HeroNoFrameskip-v0), EnvSpec(HeroNoFrameskip-v4), EnvSpec(Hero-ram-v0), EnvSpec(Hero-ram-v4), EnvSpec(Hero-ramDeterministic-v0), EnvSpec(Hero-ramDeterministic-v4), EnvSpec(Hero-ramNoFrameskip-v0), EnvSpec(Hero-ramNoFrameskip-v4), EnvSpec(IceHockey-v0), EnvSpec(IceHockey-v4), EnvSpec(IceHockeyDeterministic-v0), EnvSpec(IceHockeyDeterministic-v4), EnvSpec(IceHockeyNoFrameskip-v0), EnvSpec(IceHockeyNoFrameskip-v4), EnvSpec(IceHockey-ram-v0), EnvSpec(IceHockey-ram-v4), EnvSpec(IceHockey-ramDeterministic-v0), EnvSpec(IceHockey-ramDeterministic-v4), EnvSpec(IceHockey-ramNoFrameskip-v0), EnvSpec(IceHockey-ramNoFrameskip-v4), EnvSpec(Jamesbond-v0), EnvSpec(Jamesbond-v4), EnvSpec(JamesbondDeterministic-v0), EnvSpec(JamesbondDeterministic-v4), EnvSpec(JamesbondNoFrameskip-v0), EnvSpec(JamesbondNoFrameskip-v4), EnvSpec(Jamesbond-ram-v0), EnvSpec(Jamesbond-ram-v4), EnvSpec(Jamesbond-ramDeterministic-v0), EnvSpec(Jamesbond-ramDeterministic-v4), EnvSpec(Jamesbond-ramNoFrameskip-v0), EnvSpec(Jamesbond-ramNoFrameskip-v4), EnvSpec(JourneyEscape-v0), EnvSpec(JourneyEscape-v4), EnvSpec(JourneyEscapeDeterministic-v0), EnvSpec(JourneyEscapeDeterministic-v4), EnvSpec(JourneyEscapeNoFrameskip-v0), EnvSpec(JourneyEscapeNoFrameskip-v4), EnvSpec(JourneyEscape-ram-v0), EnvSpec(JourneyEscape-ram-v4), EnvSpec(JourneyEscape-ramDeterministic-v0), EnvSpec(JourneyEscape-ramDeterministic-v4), EnvSpec(JourneyEscape-ramNoFrameskip-v0), EnvSpec(JourneyEscape-ramNoFrameskip-v4), EnvSpec(Kangaroo-v0), EnvSpec(Kangaroo-v4), EnvSpec(KangarooDeterministic-v0), EnvSpec(KangarooDeterministic-v4), EnvSpec(KangarooNoFrameskip-v0), EnvSpec(KangarooNoFrameskip-v4), EnvSpec(Kangaroo-ram-v0), EnvSpec(Kangaroo-ram-v4), EnvSpec(Kangaroo-ramDeterministic-v0), EnvSpec(Kangaroo-ramDeterministic-v4), EnvSpec(Kangaroo-ramNoFrameskip-v0), EnvSpec(Kangaroo-ramNoFrameskip-v4), EnvSpec(Krull-v0), EnvSpec(Krull-v4), EnvSpec(KrullDeterministic-v0), EnvSpec(KrullDeterministic-v4), EnvSpec(KrullNoFrameskip-v0), EnvSpec(KrullNoFrameskip-v4), EnvSpec(Krull-ram-v0), EnvSpec(Krull-ram-v4), EnvSpec(Krull-ramDeterministic-v0), EnvSpec(Krull-ramDeterministic-v4), EnvSpec(Krull-ramNoFrameskip-v0), EnvSpec(Krull-ramNoFrameskip-v4), EnvSpec(KungFuMaster-v0), EnvSpec(KungFuMaster-v4), EnvSpec(KungFuMasterDeterministic-v0), EnvSpec(KungFuMasterDeterministic-v4), EnvSpec(KungFuMasterNoFrameskip-v0), EnvSpec(KungFuMasterNoFrameskip-v4), EnvSpec(KungFuMaster-ram-v0), EnvSpec(KungFuMaster-ram-v4), EnvSpec(KungFuMaster-ramDeterministic-v0), EnvSpec(KungFuMaster-ramDeterministic-v4), EnvSpec(KungFuMaster-ramNoFrameskip-v0), EnvSpec(KungFuMaster-ramNoFrameskip-v4), EnvSpec(MontezumaRevenge-v0), EnvSpec(MontezumaRevenge-v4), EnvSpec(MontezumaRevengeDeterministic-v0), EnvSpec(MontezumaRevengeDeterministic-v4), EnvSpec(MontezumaRevengeNoFrameskip-v0), EnvSpec(MontezumaRevengeNoFrameskip-v4), EnvSpec(MontezumaRevenge-ram-v0), EnvSpec(MontezumaRevenge-ram-v4), EnvSpec(MontezumaRevenge-ramDeterministic-v0), EnvSpec(MontezumaRevenge-ramDeterministic-v4), EnvSpec(MontezumaRevenge-ramNoFrameskip-v0), EnvSpec(MontezumaRevenge-ramNoFrameskip-v4), EnvSpec(MsPacman-v0), EnvSpec(MsPacman-v4), EnvSpec(MsPacmanDeterministic-v0), EnvSpec(MsPacmanDeterministic-v4), EnvSpec(MsPacmanNoFrameskip-v0), EnvSpec(MsPacmanNoFrameskip-v4), EnvSpec(MsPacman-ram-v0), EnvSpec(MsPacman-ram-v4), EnvSpec(MsPacman-ramDeterministic-v0), EnvSpec(MsPacman-ramDeterministic-v4), EnvSpec(MsPacman-ramNoFrameskip-v0), EnvSpec(MsPacman-ramNoFrameskip-v4), EnvSpec(NameThisGame-v0), EnvSpec(NameThisGame-v4), EnvSpec(NameThisGameDeterministic-v0), EnvSpec(NameThisGameDeterministic-v4), EnvSpec(NameThisGameNoFrameskip-v0), EnvSpec(NameThisGameNoFrameskip-v4), EnvSpec(NameThisGame-ram-v0), EnvSpec(NameThisGame-ram-v4), EnvSpec(NameThisGame-ramDeterministic-v0), EnvSpec(NameThisGame-ramDeterministic-v4), EnvSpec(NameThisGame-ramNoFrameskip-v0), EnvSpec(NameThisGame-ramNoFrameskip-v4), EnvSpec(Phoenix-v0), EnvSpec(Phoenix-v4), EnvSpec(PhoenixDeterministic-v0), EnvSpec(PhoenixDeterministic-v4), EnvSpec(PhoenixNoFrameskip-v0), EnvSpec(PhoenixNoFrameskip-v4), EnvSpec(Phoenix-ram-v0), EnvSpec(Phoenix-ram-v4), EnvSpec(Phoenix-ramDeterministic-v0), EnvSpec(Phoenix-ramDeterministic-v4), EnvSpec(Phoenix-ramNoFrameskip-v0), EnvSpec(Phoenix-ramNoFrameskip-v4), EnvSpec(Pitfall-v0), EnvSpec(Pitfall-v4), EnvSpec(PitfallDeterministic-v0), EnvSpec(PitfallDeterministic-v4), EnvSpec(PitfallNoFrameskip-v0), EnvSpec(PitfallNoFrameskip-v4), EnvSpec(Pitfall-ram-v0), EnvSpec(Pitfall-ram-v4), EnvSpec(Pitfall-ramDeterministic-v0), EnvSpec(Pitfall-ramDeterministic-v4), EnvSpec(Pitfall-ramNoFrameskip-v0), EnvSpec(Pitfall-ramNoFrameskip-v4), EnvSpec(Pong-v0), EnvSpec(Pong-v4), EnvSpec(PongDeterministic-v0), EnvSpec(PongDeterministic-v4), EnvSpec(PongNoFrameskip-v0), EnvSpec(PongNoFrameskip-v4), EnvSpec(Pong-ram-v0), EnvSpec(Pong-ram-v4), EnvSpec(Pong-ramDeterministic-v0), EnvSpec(Pong-ramDeterministic-v4), EnvSpec(Pong-ramNoFrameskip-v0), EnvSpec(Pong-ramNoFrameskip-v4), EnvSpec(Pooyan-v0), EnvSpec(Pooyan-v4), EnvSpec(PooyanDeterministic-v0), EnvSpec(PooyanDeterministic-v4), EnvSpec(PooyanNoFrameskip-v0), EnvSpec(PooyanNoFrameskip-v4), EnvSpec(Pooyan-ram-v0), EnvSpec(Pooyan-ram-v4), EnvSpec(Pooyan-ramDeterministic-v0), EnvSpec(Pooyan-ramDeterministic-v4), EnvSpec(Pooyan-ramNoFrameskip-v0), EnvSpec(Pooyan-ramNoFrameskip-v4), EnvSpec(PrivateEye-v0), EnvSpec(PrivateEye-v4), EnvSpec(PrivateEyeDeterministic-v0), EnvSpec(PrivateEyeDeterministic-v4), EnvSpec(PrivateEyeNoFrameskip-v0), EnvSpec(PrivateEyeNoFrameskip-v4), EnvSpec(PrivateEye-ram-v0), EnvSpec(PrivateEye-ram-v4), EnvSpec(PrivateEye-ramDeterministic-v0), EnvSpec(PrivateEye-ramDeterministic-v4), EnvSpec(PrivateEye-ramNoFrameskip-v0), EnvSpec(PrivateEye-ramNoFrameskip-v4), EnvSpec(Qbert-v0), EnvSpec(Qbert-v4), EnvSpec(QbertDeterministic-v0), EnvSpec(QbertDeterministic-v4), EnvSpec(QbertNoFrameskip-v0), EnvSpec(QbertNoFrameskip-v4), EnvSpec(Qbert-ram-v0), EnvSpec(Qbert-ram-v4), EnvSpec(Qbert-ramDeterministic-v0), EnvSpec(Qbert-ramDeterministic-v4), EnvSpec(Qbert-ramNoFrameskip-v0), EnvSpec(Qbert-ramNoFrameskip-v4), EnvSpec(Riverraid-v0), EnvSpec(Riverraid-v4), EnvSpec(RiverraidDeterministic-v0), EnvSpec(RiverraidDeterministic-v4), EnvSpec(RiverraidNoFrameskip-v0), EnvSpec(RiverraidNoFrameskip-v4), EnvSpec(Riverraid-ram-v0), EnvSpec(Riverraid-ram-v4), EnvSpec(Riverraid-ramDeterministic-v0), EnvSpec(Riverraid-ramDeterministic-v4), EnvSpec(Riverraid-ramNoFrameskip-v0), EnvSpec(Riverraid-ramNoFrameskip-v4), EnvSpec(RoadRunner-v0), EnvSpec(RoadRunner-v4), EnvSpec(RoadRunnerDeterministic-v0), EnvSpec(RoadRunnerDeterministic-v4), EnvSpec(RoadRunnerNoFrameskip-v0), EnvSpec(RoadRunnerNoFrameskip-v4), EnvSpec(RoadRunner-ram-v0), EnvSpec(RoadRunner-ram-v4), EnvSpec(RoadRunner-ramDeterministic-v0), EnvSpec(RoadRunner-ramDeterministic-v4), EnvSpec(RoadRunner-ramNoFrameskip-v0), EnvSpec(RoadRunner-ramNoFrameskip-v4), EnvSpec(Robotank-v0), EnvSpec(Robotank-v4), EnvSpec(RobotankDeterministic-v0), EnvSpec(RobotankDeterministic-v4), EnvSpec(RobotankNoFrameskip-v0), EnvSpec(RobotankNoFrameskip-v4), EnvSpec(Robotank-ram-v0), EnvSpec(Robotank-ram-v4), EnvSpec(Robotank-ramDeterministic-v0), EnvSpec(Robotank-ramDeterministic-v4), EnvSpec(Robotank-ramNoFrameskip-v0), EnvSpec(Robotank-ramNoFrameskip-v4), EnvSpec(Seaquest-v0), EnvSpec(Seaquest-v4), EnvSpec(SeaquestDeterministic-v0), EnvSpec(SeaquestDeterministic-v4), EnvSpec(SeaquestNoFrameskip-v0), EnvSpec(SeaquestNoFrameskip-v4), EnvSpec(Seaquest-ram-v0), EnvSpec(Seaquest-ram-v4), EnvSpec(Seaquest-ramDeterministic-v0), EnvSpec(Seaquest-ramDeterministic-v4), EnvSpec(Seaquest-ramNoFrameskip-v0), EnvSpec(Seaquest-ramNoFrameskip-v4), EnvSpec(Skiing-v0), EnvSpec(Skiing-v4), EnvSpec(SkiingDeterministic-v0), EnvSpec(SkiingDeterministic-v4), EnvSpec(SkiingNoFrameskip-v0), EnvSpec(SkiingNoFrameskip-v4), EnvSpec(Skiing-ram-v0), EnvSpec(Skiing-ram-v4), EnvSpec(Skiing-ramDeterministic-v0), EnvSpec(Skiing-ramDeterministic-v4), EnvSpec(Skiing-ramNoFrameskip-v0), EnvSpec(Skiing-ramNoFrameskip-v4), EnvSpec(Solaris-v0), EnvSpec(Solaris-v4), EnvSpec(SolarisDeterministic-v0), EnvSpec(SolarisDeterministic-v4), EnvSpec(SolarisNoFrameskip-v0), EnvSpec(SolarisNoFrameskip-v4), EnvSpec(Solaris-ram-v0), EnvSpec(Solaris-ram-v4), EnvSpec(Solaris-ramDeterministic-v0), EnvSpec(Solaris-ramDeterministic-v4), EnvSpec(Solaris-ramNoFrameskip-v0), EnvSpec(Solaris-ramNoFrameskip-v4), EnvSpec(SpaceInvaders-v0), EnvSpec(SpaceInvaders-v4), EnvSpec(SpaceInvadersDeterministic-v0), EnvSpec(SpaceInvadersDeterministic-v4), EnvSpec(SpaceInvadersNoFrameskip-v0), EnvSpec(SpaceInvadersNoFrameskip-v4), EnvSpec(SpaceInvaders-ram-v0), EnvSpec(SpaceInvaders-ram-v4), EnvSpec(SpaceInvaders-ramDeterministic-v0), EnvSpec(SpaceInvaders-ramDeterministic-v4), EnvSpec(SpaceInvaders-ramNoFrameskip-v0), EnvSpec(SpaceInvaders-ramNoFrameskip-v4), EnvSpec(StarGunner-v0), EnvSpec(StarGunner-v4), EnvSpec(StarGunnerDeterministic-v0), EnvSpec(StarGunnerDeterministic-v4), EnvSpec(StarGunnerNoFrameskip-v0), EnvSpec(StarGunnerNoFrameskip-v4), EnvSpec(StarGunner-ram-v0), EnvSpec(StarGunner-ram-v4), EnvSpec(StarGunner-ramDeterministic-v0), EnvSpec(StarGunner-ramDeterministic-v4), EnvSpec(StarGunner-ramNoFrameskip-v0), EnvSpec(StarGunner-ramNoFrameskip-v4), EnvSpec(Tennis-v0), EnvSpec(Tennis-v4), EnvSpec(TennisDeterministic-v0), EnvSpec(TennisDeterministic-v4), EnvSpec(TennisNoFrameskip-v0), EnvSpec(TennisNoFrameskip-v4), EnvSpec(Tennis-ram-v0), EnvSpec(Tennis-ram-v4), EnvSpec(Tennis-ramDeterministic-v0), EnvSpec(Tennis-ramDeterministic-v4), EnvSpec(Tennis-ramNoFrameskip-v0), EnvSpec(Tennis-ramNoFrameskip-v4), EnvSpec(TimePilot-v0), EnvSpec(TimePilot-v4), EnvSpec(TimePilotDeterministic-v0), EnvSpec(TimePilotDeterministic-v4), EnvSpec(TimePilotNoFrameskip-v0), EnvSpec(TimePilotNoFrameskip-v4), EnvSpec(TimePilot-ram-v0), EnvSpec(TimePilot-ram-v4), EnvSpec(TimePilot-ramDeterministic-v0), EnvSpec(TimePilot-ramDeterministic-v4), EnvSpec(TimePilot-ramNoFrameskip-v0), EnvSpec(TimePilot-ramNoFrameskip-v4), EnvSpec(Tutankham-v0), EnvSpec(Tutankham-v4), EnvSpec(TutankhamDeterministic-v0), EnvSpec(TutankhamDeterministic-v4), EnvSpec(TutankhamNoFrameskip-v0), EnvSpec(TutankhamNoFrameskip-v4), EnvSpec(Tutankham-ram-v0), EnvSpec(Tutankham-ram-v4), EnvSpec(Tutankham-ramDeterministic-v0), EnvSpec(Tutankham-ramDeterministic-v4), EnvSpec(Tutankham-ramNoFrameskip-v0), EnvSpec(Tutankham-ramNoFrameskip-v4), EnvSpec(UpNDown-v0), EnvSpec(UpNDown-v4), EnvSpec(UpNDownDeterministic-v0), EnvSpec(UpNDownDeterministic-v4), EnvSpec(UpNDownNoFrameskip-v0), EnvSpec(UpNDownNoFrameskip-v4), EnvSpec(UpNDown-ram-v0), EnvSpec(UpNDown-ram-v4), EnvSpec(UpNDown-ramDeterministic-v0), EnvSpec(UpNDown-ramDeterministic-v4), EnvSpec(UpNDown-ramNoFrameskip-v0), EnvSpec(UpNDown-ramNoFrameskip-v4), EnvSpec(Venture-v0), EnvSpec(Venture-v4), EnvSpec(VentureDeterministic-v0), EnvSpec(VentureDeterministic-v4), EnvSpec(VentureNoFrameskip-v0), EnvSpec(VentureNoFrameskip-v4), EnvSpec(Venture-ram-v0), EnvSpec(Venture-ram-v4), EnvSpec(Venture-ramDeterministic-v0), EnvSpec(Venture-ramDeterministic-v4), EnvSpec(Venture-ramNoFrameskip-v0), EnvSpec(Venture-ramNoFrameskip-v4), EnvSpec(VideoPinball-v0), EnvSpec(VideoPinball-v4), EnvSpec(VideoPinballDeterministic-v0), EnvSpec(VideoPinballDeterministic-v4), EnvSpec(VideoPinballNoFrameskip-v0), EnvSpec(VideoPinballNoFrameskip-v4), EnvSpec(VideoPinball-ram-v0), EnvSpec(VideoPinball-ram-v4), EnvSpec(VideoPinball-ramDeterministic-v0), EnvSpec(VideoPinball-ramDeterministic-v4), EnvSpec(VideoPinball-ramNoFrameskip-v0), EnvSpec(VideoPinball-ramNoFrameskip-v4), EnvSpec(WizardOfWor-v0), EnvSpec(WizardOfWor-v4), EnvSpec(WizardOfWorDeterministic-v0), EnvSpec(WizardOfWorDeterministic-v4), EnvSpec(WizardOfWorNoFrameskip-v0), EnvSpec(WizardOfWorNoFrameskip-v4), EnvSpec(WizardOfWor-ram-v0), EnvSpec(WizardOfWor-ram-v4), EnvSpec(WizardOfWor-ramDeterministic-v0), EnvSpec(WizardOfWor-ramDeterministic-v4), EnvSpec(WizardOfWor-ramNoFrameskip-v0), EnvSpec(WizardOfWor-ramNoFrameskip-v4), EnvSpec(YarsRevenge-v0), EnvSpec(YarsRevenge-v4), EnvSpec(YarsRevengeDeterministic-v0), EnvSpec(YarsRevengeDeterministic-v4), EnvSpec(YarsRevengeNoFrameskip-v0), EnvSpec(YarsRevengeNoFrameskip-v4), EnvSpec(YarsRevenge-ram-v0), EnvSpec(YarsRevenge-ram-v4), EnvSpec(YarsRevenge-ramDeterministic-v0), EnvSpec(YarsRevenge-ramDeterministic-v4), EnvSpec(YarsRevenge-ramNoFrameskip-v0), EnvSpec(YarsRevenge-ramNoFrameskip-v4), EnvSpec(Zaxxon-v0), EnvSpec(Zaxxon-v4), EnvSpec(ZaxxonDeterministic-v0), EnvSpec(ZaxxonDeterministic-v4), EnvSpec(ZaxxonNoFrameskip-v0), EnvSpec(ZaxxonNoFrameskip-v4), EnvSpec(Zaxxon-ram-v0), EnvSpec(Zaxxon-ram-v4), EnvSpec(Zaxxon-ramDeterministic-v0), EnvSpec(Zaxxon-ramDeterministic-v4), EnvSpec(Zaxxon-ramNoFrameskip-v0), EnvSpec(Zaxxon-ramNoFrameskip-v4), EnvSpec(CubeCrash-v0), EnvSpec(CubeCrashSparse-v0), EnvSpec(CubeCrashScreenBecomesBlack-v0), EnvSpec(MemorizeDigits-v0)])"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "gym.envs.registry.all()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QR-DxVNOMM9J"
      },
      "source": [
        "Cart-Pole은 매우 간단한 환경으로 왼쪽과 오른쪽으로 움직이는 카트와 그 위에 수직으로 놓여 있는 막대로 구성됩니다. 에이전트는 카트를 왼쪽이나 오른쪽으로 움직여 막대가 바로 서 있도록 만들어야 합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DauG0fsUMM9K"
      },
      "outputs": [],
      "source": [
        "env = gym.make('CartPole-v1')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "22Z8jxZBMM9K"
      },
      "source": [
        "`reset()` 메서드를 호출해 환경을 초기화합니다. 이 메서드는 관측을 반환합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ln5prFuAMM9K"
      },
      "outputs": [],
      "source": [
        "env.seed(42)\n",
        "obs = env.reset()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hAKkSMyZMM9K"
      },
      "source": [
        "관측은 환경에 따라 다릅니다. 이 경우 4개의 실수로 구성된 1D 넘파이 배열입니다. 카트의 수평 위치, 속도, 막대의 각도(0=수직), 각속도를 나타냅니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3MooBNyqMM9L",
        "outputId": "4f1e7271-119a-47e2-ef1e-b172d9dd0535"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([-0.01258566, -0.00156614,  0.04207708, -0.00180545])"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "obs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YFnea-CUMM9L"
      },
      "source": [
        "환경은 `render()` 메서드를 호출하여 시각화할 수 있습니다. 그리고 렌더링 모드(환경에 따른 렌더링 옵션)를 선택할 수 있습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B3xZQ77_MM9L"
      },
      "source": [
        "**경고**: (Cart-Pole을 포함해) 일부 환경은 화면 접근 권한이 필요합니다. `mode=\"rgb_array\"`로 지정하더라도 별도의 윈도우를 엽니다. 일반적으로 이 윈도우를 무시할 수 있습니다. 하지만 주피터를 백엔드(headless) 서버로 실행한다면 예외가 발생합니다. 이를 피하는 한 가지 방법은 [Xvfb](http://en.wikipedia.org/wiki/Xvfb) 같은 가짜 X 서버를 설치하는 것입니다. 데비안이나 우분투에서는 다음과 같이 설치합니다:\n",
        "\n",
        "```bash\n",
        "$ apt update\n",
        "$ apt install -y xvfb\n",
        "```\n",
        "\n",
        "그다음 `xvfb-run` 명령으로 주피터를 실행합니다:\n",
        "\n",
        "```bash\n",
        "$ xvfb-run -s \"-screen 0 1400x900x24\" jupyter notebook\n",
        "```\n",
        "\n",
        "또는 Xvfb를 감싼 [pyvirtualdisplay](https://github.com/ponty/pyvirtualdisplay) 파이썬 라이브러리를 설치할 수 있습니다:\n",
        "\n",
        "```bash\n",
        "%pip install -U pyvirtualdisplay\n",
        "```\n",
        "\n",
        "그다음 다음 코드를 실행합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IRCSe2PCMM9M"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "    import pyvirtualdisplay\n",
        "    display = pyvirtualdisplay.Display(visible=0, size=(1400, 900)).start()\n",
        "except ImportError:\n",
        "    pass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_sZ-HW8dMM9M",
        "outputId": "745dfe7c-f163-4372-8f0b-9df0972820e3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.render()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ecaApWqyMM9M"
      },
      "source": [
        "이 예에서는 `mode=\"rgb_array\"`로 지정해 환경 이미지를 넘파이 배열로 받을 것입니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nv1aNcK_MM9M",
        "outputId": "b00724e1-d49b-4daf-c403-e5f282e930af"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(400, 600, 3)"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "img = env.render(mode=\"rgb_array\")\n",
        "img.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HmRfeGldMM9N"
      },
      "outputs": [],
      "source": [
        "def plot_environment(env, figsize=(5,4)):\n",
        "    plt.figure(figsize=figsize)\n",
        "    img = env.render(mode=\"rgb_array\")\n",
        "    plt.imshow(img)\n",
        "    plt.axis(\"off\")\n",
        "    return img"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CO1XfyXcMM9N",
        "outputId": "450d1e22-3f7c-410c-9db5-1ecf3586085c"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAASUAAADICAYAAACuyvefAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAF4klEQVR4nO3dzY8bdx3H8e/Y3sfulhSiKlSoIr2UWy9BKDlHAokD/wF3cuN/yYkLh4gr/wRCWiEFISHoAVVERXS3PKQJWbRee8fDgQjqbHcDnk39sfN63Wb8oK+0o7e8Y89vmq7rCiDFYNkDAHyeKAFRRAmIIkpAFFECoogSEGX0ksf9XgB4FZqLHvBJCYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBktewBWX9d1Vd3sc3uaqqappmmWNhOrS5TobfzkqD7+xc+qex6m4eZ23fjgu7X/zvtLnoxVJEr0NpuO6/jTj+Y+Le2/8y1RYiHOKdFbOx2f2zfc2F7CJKwDUaK38WdH8+eUmqa23/r68gZipYkSV6B7YdsJbhYnSkAUUQKiiBK9dF33798pwRURJXo7+fuf5raHG1u1+ca15QzDyhMlepudTee2m+GoBn4SwIJECYgiSkAUUQKiiBL9dF21k5O5Xc1gaIUAFiZK9DJrp3X67K9z+7b2r9dwc2dJE7HqRImr1wzKpSYsSpSAKKIERBElIIoo0cvZ+FmdnRzP7du+dqPKt28sSJToZXY2rVk7f5nJaGvXTwJYmCgBUUQJiCJKQBRRopeundaLa3QPRpvLGYa1IEr0Mn5yVF17Nrdv56vfWNI0rANRopcvXArXN2/0IEpAFFECoogSPbmTCVdLlOjl5PGf57ab4UZt7H5lSdOwDkSJXtrJeG67GQxrtL23pGlYB6IERBElIIooAVFEiYV1XVez6encvqYZWLaEXkSJhXWztsZPDuf2be695UQ3vYgSV6sZPL+bCSzG0QNEESUgiigBUUSJhbWTk5qePJvbt/Xm9WoGDisW5+hhYV07rdl0/jKT0fZeNU5004OjB4giSkAUUQKiiBIL62btuTW6m8FoSdOwLkSJhY2f/uXctW+7199d0jSsC1Ficd2sXlwO1zdv9OUIAqKIEhBFlIAozRfe4fS/3D/nNdS2bT148KCOjo4ufd77107r3cEn/9k+a2f1y0/3a7L99qWvu3v3bt26detKZmVlXbgSoO9vOadt27p//349fPjw0uf96Affrh9+7zv12eTt2hyc1nYd1f2f/LT+ePjk0tft7OyIEhcSJRZ22u7WwePv19Pp12rYtPXNrYPqup8veyxWnHNKLOyT8Xv1dHq9qgbVdhv10T8/qMlsZ9ljseJEiYV1L5wWmHXDarvhkqZhXYgSC7v55se1P3pcVV0N6qzee+M3tT08XvZYrLhLzynNZrMvaw6C/K9/91/99nf14aMf1+PJjdocjGu3Duvwb/946eu6rnNsveYGlywEeGmUDg4OrnwY8k2n0zo+fvknnl//4bCqDqvq9//X+z969Mix9Zq7c+fOhY9dGqXLXsj6mkwmtbf36u7ddvPmTccWF3JOCYgiSkAUUQKiiBIQRZSAKK5945zhcFj37t2rw8PDV/L+t2/ffiXvy3qwdAmwDBcuXeLfNyCKKAFRRAmIIkpAFFECoogSEEWUgCiiBEQRJSCKKAFRRAmIIkpAFFECoogSEEWUgCiiBEQRJSCKKAFRRAmIIkpAFFECoogSEEWUgCiiBEQRJSCKKAFRRAmIIkpAFFECoogSEEWUgCiiBEQRJSCKKAFRRAmIIkpAFFECoogSEGX0ksebL2UKgOd8UgKiiBIQRZSAKKIERBElIIooAVH+Bc6iAHz8aqUYAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 360x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_environment(env)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xx5PVNLmMM9N"
      },
      "source": [
        "환경과 상호작용하는 방법을 알아 보죠. 에이전트는 \"행동 공간\"(가능한 행동의 집합)에서 하나의 행동을 선택해야 합니다. 이 환경의 행동 공간을 다음처럼 확인해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EqjV6_ESMM9N",
        "outputId": "6397e2dd-fcdc-40b7-d5f0-4e6510899b50"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Discrete(2)"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.action_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oIvYOWJIMM9N"
      },
      "source": [
        "네 단 두 개의 행동이 가능합니다: 왼쪽 또는 오른쪽으로 가속합니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IUM_t8GpMM9O"
      },
      "source": [
        "막대가 오른쪽으로 기울어져 있기 때문에(`obs[2] > 0`), 카트를 오른쪽으로 가속해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0uXGZvqxMM9O",
        "outputId": "85c1d33f-51fa-4a8a-96fe-6863fbca6411"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([-0.01261699,  0.19292789,  0.04204097, -0.28092127])"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "action = 1  # 오른쪽으로 가속\n",
        "obs, reward, done, info = env.step(action)\n",
        "obs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1GFdgA3nMM9O"
      },
      "source": [
        "이제 카트가 오른쪽으로 움직였습니다(`obs[1] > 0`). 막대가 여전히 오른쪽으로 기울어져 있습니다(`obs[2] > 0`). 하지만 각속도가 음수이므로(`obs[3] < 0`) 다음 스텝에서는 왼쪽으로 기울 것 같습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M-gO_rXbMM9O",
        "outputId": "96a3fb1b-741a-4e15-9508-16b1f3ddcd52"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Saving figure cart_pole_plot\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAADvCAYAAADM8A71AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAGHklEQVR4nO3dzWpcBRjH4fdMkklMG2s1+BFF0aIWP6s7wYXQ4kLoBbjpRYilvQGhdyBFvAbrSqwrN5aCGyuIINUK1o9qqmlsYhsnx40Uy8Q2mbbz76HPswrvDHPexeHHMDlzpmnbtgAYv156AYA7lQADhAgwQIgAA4QIMEDI5HUed4kEwI1rNhp6BwwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAymV4ARtW27ZW/m6YJbgKjEWA668fPP6wLP3xVD730RvWmZqqqqun1avsDu6rpTYS3g+sTYDrr8vL5Wvn1+zp9/N0rs97UTD3/5js1ObM9uBlsjs+AAUIEGCBEgAFCBBggRIABQgQYIESAAUIEGCBEgAFCBBggRIABQgSYThpcXq211aWh+fTcfW7EQ2cIMJ3019IvtXz266H5fU+9UhP9uwIbwdYJMECIAAOECDBAiAADhAgwQIgAA4QIMECIAAOECDBAiAADhAgwQIgA00mXls4NzZqJqZratjOwDYxGgOmkX7/6dGg2Nbujdj7+UmAbGI0AA4QIMECIAAOECDBAiAADhAgwQIgAA4QIMECIAAOECDBAiAADhAgwQIgA0zl//ny6Vs//ODSf3/1qVeOUpjucrXTO2uqFGlxeGZrP3PNgNU0T2AhGI8AAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAkyntG1bf69eGJr3Jvs10Z8JbASjE2C6pV2vX059MjSenX+05hZ2BxaC0QkwndO27YZzd0KjawQYIESAAUIEGCBEgAFCBBggRIABQgQYIGQyvQCsrKzUYDDY1HPb9UG16+tD88FgUMvLy5t6jX6/X9PT01vaEW6F5v8uav/XNR+Em2H//v114sSJTT13otfU+2+9Xg/du/2q+Zff/VZvv/fppl7j0KFDdfDgwS3vCTdgw28JeQdM3NLSUi0uLm7quRO9plbWpuvcpUeqqqqptu7t/1Rrf69t+jVWV1dH3hVuJgGmU9pq6os/Xqv++rNXJk9sO1Xr7c/RvWAU/glHp7z89GM1OfPgfyZNffvnC/XByd9jO8GoBJhOuf/+J2t6+urPf9uqOnX6XGYhuAECTKfs7J+rqd6lq2Z3TVwcmkEX+AyYTjl/4WKtLR6vsxefqYfn5+qe7f167u7PatvE8E3a4XYnwHTKRye/qY9OHqmqqhd3PVAL83P1cVWdXdzcNcBwO7nmdcBHjhxxHTC33NGjR+vMmTNjO96+fftq7969YzseHD58eOvXAR84cODWbAP/cezYsbEGeM+ePc5tbgvXDPDCwsK49uAO1u/3x3q8ubk55za3BVdBAIQIMECIAAOECDBAiAADhPgiBnE7duyo+fn5sR1vdnZ2bMeCa3FDduK28osYN4NfxCBgwy9iCDDArbdhgH0GDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIZPXebwZyxYAdyDvgAFCBBggRIABQgQYIESAAUIEGCDkH6bUxi6unNOlAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 360x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_environment(env)\n",
        "save_fig(\"cart_pole_plot\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7tsgxZK_MM9O"
      },
      "source": [
        "요청한 대로 실행되는 것 같습니다!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6nS0v0NiMM9P"
      },
      "source": [
        "환경은 이전 스텝에서 얼마나 많은 보상을 받는지 에이전트에게 알려 줍니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WHa5a1p7MM9P",
        "outputId": "e2b9da6c-28c3-4011-e409-61d7d35c81ea"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "1.0"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "reward"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sttrrhw9MM9P"
      },
      "source": [
        "게임이 끝나면 환경은 `done=True`를 반환합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fjignPnfMM9P",
        "outputId": "6466a7d0-d8aa-4767-dbc2-df8e39450ce1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "done"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ysEdYE7MM9P"
      },
      "source": [
        "마지막으로 `info`는 훈련이나 디버깅에 유용한 추가적인 정보를 담은 환경에 특화된 딕셔너리입니다. 예를 들어 일부 게임에서는 얼마나 많은 에이전트의 생명이 몇 개가 남아 있는지 나타낼 수 있습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NaW2h9CoMM9Q",
        "outputId": "bd2439d8-808a-483a-a62d-9d3e4be4e6c8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{}"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "info"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3tktQI60MM9Q"
      },
      "source": [
        "환경이 재설정된 순간부터 종료될 때까지 스텝 시퀀스를 \"에피소드\"라고 합니다. 에피소드 끝에서 (즉, `step()`이 `done=True`를 반환할 때), 계속하기 전에 환경을 재설정해야 합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Hb5bmiIwMM9Q"
      },
      "outputs": [],
      "source": [
        "if done:\n",
        "    obs = env.reset()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wPl8DcS8MM9R"
      },
      "source": [
        "그럼 어떻게 막대를 똑바로 유지할 수 있을까요? 이를 위해 정책을 정의해야 합니다. 에이전트가 매 스텝마다 행동을 선택하기 위해 사용할 전략입니다. 어떤 행동을 선택할지 결정하기 위해 지난 행동과 관측을 모두 사용할 수 있습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1MuS0SmWMM9R"
      },
      "source": [
        "# 간단한 하드 코딩 정책"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z6xtsBnHMM9R"
      },
      "source": [
        "간단한 정책을 하드 코딩해 보죠. 막대가 왼쪽으로 기울어지면 카트를 왼쪽으로 움직이고 오른쪽으로 기울어지면 반대로 움직입니다. 어떻게 작동하는지 확인해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fIxTVS-JMM9R"
      },
      "outputs": [],
      "source": [
        "env.seed(42)\n",
        "\n",
        "def basic_policy(obs):\n",
        "    angle = obs[2]\n",
        "    return 0 if angle < 0 else 1\n",
        "\n",
        "totals = []\n",
        "for episode in range(500):\n",
        "    episode_rewards = 0\n",
        "    obs = env.reset()\n",
        "    for step in range(200):\n",
        "        action = basic_policy(obs)\n",
        "        obs, reward, done, info = env.step(action)\n",
        "        episode_rewards += reward\n",
        "        if done:\n",
        "            break\n",
        "    totals.append(episode_rewards)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4-ZUL55bMM9R",
        "outputId": "385eb081-57df-4caa-b7c9-5fedb00bc2e6"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(41.718, 8.858356280936096, 24.0, 68.0)"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "np.mean(totals), np.std(totals), np.min(totals), np.max(totals)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XjK9VwcYMM9S"
      },
      "source": [
        "예상대로 이 전략은 너무 단순합니다. 최대로 막대를 유지한 스텝 횟수가 68입니다. 이 환경은 에이전트가 막대를 200 스텝 이상 유지해야 해결된 것으로 간주합니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "21BN-tYuMM9S"
      },
      "source": [
        "하나의 에피소드를 시각화해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cQpd8b7WMM9S"
      },
      "outputs": [],
      "source": [
        "env.seed(42)\n",
        "\n",
        "frames = []\n",
        "\n",
        "obs = env.reset()\n",
        "for step in range(200):\n",
        "    img = env.render(mode=\"rgb_array\")\n",
        "    frames.append(img)\n",
        "    action = basic_policy(obs)\n",
        "\n",
        "    obs, reward, done, info = env.step(action)\n",
        "    if done:\n",
        "        break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gos2Se6AMM9S"
      },
      "source": [
        "애니메이션을 출력합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cBPZOwgaMM9T"
      },
      "outputs": [],
      "source": [
        "def update_scene(num, frames, patch):\n",
        "    patch.set_data(frames[num])\n",
        "    return patch,\n",
        "\n",
        "def plot_animation(frames, repeat=False, interval=40):\n",
        "    fig = plt.figure()\n",
        "    patch = plt.imshow(frames[0])\n",
        "    plt.axis('off')\n",
        "    anim = animation.FuncAnimation(\n",
        "        fig, update_scene, fargs=(frames, patch),\n",
        "        frames=len(frames), repeat=repeat, interval=interval)\n",
        "    plt.close()\n",
        "    return anim"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DVYZ7NU6MM9T"
      },
      "outputs": [],
      "source": [
        "plot_animation(frames)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xTqme9K4MM9T"
      },
      "source": [
        "확실히 이 방법은 불안정해서 약간 흔들리면 막대가 너무 기울어져 게임이 끝납니다. 이 보다는 더 똑똑한 전략이 필요합니다!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gjAWzygUMM9T"
      },
      "source": [
        "# 신경망 정책"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zroA9QpAMM9T"
      },
      "source": [
        "관측을 입력으로 받고 각 관측에 대해 선택할 행동의 확률을 출력하는 신경망을 만들어 보죠. 행동을 선택하기 위해 신경망은 각 행동의 확률을 추정합니다. 이 추정된 확률에 따라 랜덤하게 행동을 선택합니다. Cart-Pole 환경의 경우 두 개의 가능한 행동이 있습니다(왼쪽과 오른쪽). 따라서 하나의 출력 뉴런만 있으면 됩니다. 이 뉴런은 행동 0(왼쪽)의 확률 `p`를 출력합니다. 물론 행동 1(오른쪽)의 확률은 `1 - p`가 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zwTSL6KcMM9U"
      },
      "outputs": [],
      "source": [
        "keras.backend.clear_session()\n",
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)\n",
        "\n",
        "n_inputs = 4 # == env.observation_space.shape[0]\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Dense(5, activation=\"elu\", input_shape=[n_inputs]),\n",
        "    keras.layers.Dense(1, activation=\"sigmoid\"),\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3cDCANQlMM9U"
      },
      "source": [
        "이 환경에서는 지난 행동과 관측을 무시할 수 있습니다. 각 관측이 완전한 환경의 상태를 담고 있기 때문입니다. 은닉 상태가 있다면 환경의 은닉 상태를 추정하기 위해 지난 행동과 관측을 고려해야 할 수 있습니다. 예를 들어, 이 환경이 카트의 위치만 제공하고 속도를 알려 주지 않는다면, 현재 속도를 추정하기 위해 현재 관측 뿐만 아니라 지난 관측도 고려해야 합니다. 또 다른 예는 관측에 잡음이 있는 경우입니다. 가장 가능성 있는 현재 상태를 추정하기 위해 지난 몇 개의 관측을 사용할 수 있습니다. 이 문제는 매우 간단합니다. 현재 관측에 잡음이 없고 환경의 모든 상태가 담겨 있습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a4cbqL1oMM9U"
      },
      "source": [
        "정책 네트워크가 출력한 확률 중에서 가장 높은 확률을 가진 행동을 선택하지 않고 랜덤한 행동을 선택하는 이유가 궁금할지 모릅니다. 이 방법은 에이전트가 새로운 행동을 탐험하는 것과 잘 동작하는 행동을 활용하는 것 사이에 밸런스를 찾도록 합니다. 비유를 들어 보죠. 한 음식점에 처음 방문했다고 가정해 보죠. 모든 음식에 대한 선호도가 동일하다면 랜덤하게 하나를 선택합니다. 이 음식이 좋다고 느낀다면 다음 번에 이 음식을 주문할 확률을 높일 수 있습니다. 하지만 이 확률을 100%로 높여서는 안됩니다. 그렇지 않으면 다른 음식을 시도해 볼 수 없습니다. 어쩌면 다른 음식이 이번에 먹은 것보다 훨씬 더 좋을 수도 있습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h5RiuKZ4MM9U"
      },
      "source": [
        "모델을 실행하여 한 에피소드를 플레이하고 애니메이션을 위한 프레임을 반환하는 함수를 작성해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cUSmxiEhMM9U"
      },
      "outputs": [],
      "source": [
        "def render_policy_net(model, n_max_steps=200, seed=42):\n",
        "    frames = []\n",
        "    env = gym.make(\"CartPole-v1\")\n",
        "    env.seed(seed)\n",
        "    np.random.seed(seed)\n",
        "    obs = env.reset()\n",
        "    for step in range(n_max_steps):\n",
        "        frames.append(env.render(mode=\"rgb_array\"))\n",
        "        left_proba = model.predict(obs.reshape(1, -1))\n",
        "        action = int(np.random.rand() > left_proba)\n",
        "        obs, reward, done, info = env.step(action)\n",
        "        if done:\n",
        "            break\n",
        "    env.close()\n",
        "    return frames"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ku2fsJ7kMM9V"
      },
      "source": [
        "랜덤하게 초기화된 정책 네트워크가 얼마나 잘 수행하는지 확인해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C4xD0Cq5MM9V"
      },
      "outputs": [],
      "source": [
        "frames = render_policy_net(model)\n",
        "plot_animation(frames)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uwIOKbamMM9V"
      },
      "source": [
        "음.. 아주 나쁘군요. 이 신경망은 더 배워야 합니다. 먼저 앞에서 사용한 기본적인 정책을 학습할 수 있는지 확인해 보죠. 막대가 왼쪽으로 기울면 왼쪽으로 움직이고, 오른쪽으로 기울면 오른쪽으로 움직이도록 합니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lGamEjtPMM9V"
      },
      "source": [
        "같은 신경망으로 동시에 50개의 다른 환경을 플레이할 수 있습니다(이렇게 하면 각 스텝마다 다양한 훈련 배치를 얻을 수 있습니다). 그리고 5000번 반복 동안에 훈련합니다. 게임이 종료되면 환경을 재설정합니다. 사용자 정의 훈련 루프를 사용하여 모델을 훈련하기 때문에 훈련 스텝마다 환경에 앞서 예측을 쉽게 만들 수 있습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NNwa-__yMM9V",
        "outputId": "e2e20d61-1171-4192-f0fd-228d87c211cd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration: 4999, Loss: 0.094"
          ]
        }
      ],
      "source": [
        "n_environments = 50\n",
        "n_iterations = 5000\n",
        "\n",
        "envs = [gym.make(\"CartPole-v1\") for _ in range(n_environments)]\n",
        "for index, env in enumerate(envs):\n",
        "    env.seed(index)\n",
        "np.random.seed(42)\n",
        "observations = [env.reset() for env in envs]\n",
        "optimizer = keras.optimizers.RMSprop()\n",
        "loss_fn = keras.losses.binary_crossentropy\n",
        "\n",
        "for iteration in range(n_iterations):\n",
        "    # if angle < 0, we want proba(left) = 1., or else proba(left) = 0.\n",
        "    target_probas = np.array([([1.] if obs[2] < 0 else [0.])\n",
        "                              for obs in observations])\n",
        "    with tf.GradientTape() as tape:\n",
        "        left_probas = model(np.array(observations))\n",
        "        loss = tf.reduce_mean(loss_fn(target_probas, left_probas))\n",
        "    print(\"\\rIteration: {}, Loss: {:.3f}\".format(iteration, loss.numpy()), end=\"\")\n",
        "    grads = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
        "    actions = (np.random.rand(n_environments, 1) > left_probas.numpy()).astype(np.int32)\n",
        "    for env_index, env in enumerate(envs):\n",
        "        obs, reward, done, info = env.step(actions[env_index][0])\n",
        "        observations[env_index] = obs if not done else env.reset()\n",
        "\n",
        "for env in envs:\n",
        "    env.close()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1hyqRPv2MM9V"
      },
      "outputs": [],
      "source": [
        "frames = render_policy_net(model)\n",
        "plot_animation(frames)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "du_XBPffMM9W"
      },
      "source": [
        "정책을 잘 학습한 것 같군요. 이제 스스로 더 나은 정책을 학습할 수 있는지 확인해 보겠습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Krl3BwxCMM9W"
      },
      "source": [
        "# 정책 그레이디언트"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Is92qcXZMM9W"
      },
      "source": [
        "이 신경망을 훈련하려면 타깃 확률 `y`를 정의해야 합니다. 행동이 좋으면 해당 확률을 증가시키고 반대로 나쁘면 감소시켜야 합니다. 하지만 행동이 좋은지 나쁜지 어떻게 알까요? 대부분 행동의 효과가 지연되어 나타나기 때문에 한 에피소드에서 점수를 얻거나 잃을 때 어떤 행동이 이 결과에 기여했는지 명확하지 않다는 것이 문제입니다. 마지막 행동일까요? 아니면 마지막에서 10번째 행동일까요? 아니면 50 스텝 이전의 행동일까요? 이를 _신용 할당 문제_ 라고 부릅니다.\n",
        "\n",
        "_정책 그레이디언트_ 알고리즘은 이 문제를 해결하기 위해 먼저 여러 개의 에피소드를 플레이하고 그다음 좋은 에피소드에 있는 행동의 가능성을 조금 더 높이고, 나쁜 에피소드에 있는 행동의 가능성을 조금 낮춥니다. 먼저 플레이해보고 다시 돌아가서 수행한 작업을 생각해 보겠습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qFuuHNucMM9W"
      },
      "source": [
        "이 모델을 사용해 하나의 스텝을 플레이하는 함수를 만듭니다. 지금은 선택한 행동이 모두 좋다고 가정하고 손실과 그레이디언트를 계산합니다(그레이디언트를 저장하고 나중에 행동이 좋은지 나쁜지에 따라 수정하겠습니다):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "twr__keJMM9W"
      },
      "outputs": [],
      "source": [
        "def play_one_step(env, obs, model, loss_fn):\n",
        "    with tf.GradientTape() as tape:\n",
        "        left_proba = model(obs[np.newaxis])\n",
        "        action = (tf.random.uniform([1, 1]) > left_proba)\n",
        "        y_target = tf.constant([[1.]]) - tf.cast(action, tf.float32)\n",
        "        loss = tf.reduce_mean(loss_fn(y_target, left_proba))\n",
        "    grads = tape.gradient(loss, model.trainable_variables)\n",
        "    obs, reward, done, info = env.step(int(action[0, 0].numpy()))\n",
        "    return obs, reward, done, grads"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TqLY1B30MM9W"
      },
      "source": [
        "`left_proba`가 높으면 `action`이 `False`가 될 가능성이 높습니다(0~1 사이에서 균등 분포로 난수를 샘플링하면 `left_proba`보다 높지 않을 가능성이 높기 때문에). 그리고 `False`를 숫자로 바꾸면 0이므로 `y_target`은 1 - 0 = 1입니다. 다른 말로 하면 타깃을 1로 지정하는 것은 왼쪽일 확률을 100%로 가정한다는 의미입니다(따라서 올바른 행동을 선택했습니다)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yUXNSzmiMM9X"
      },
      "source": [
        "이제 `play_one_step()` 함수를 사용해 여러 개의 에피소드를 플레이하고 에피소드와 스텝마다 모든 보상과 그레이디언트를 반환하는 또 다른 함수를 만들어 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FAldzAamMM9X"
      },
      "outputs": [],
      "source": [
        "def play_multiple_episodes(env, n_episodes, n_max_steps, model, loss_fn):\n",
        "    all_rewards = []\n",
        "    all_grads = []\n",
        "    for episode in range(n_episodes):\n",
        "        current_rewards = []\n",
        "        current_grads = []\n",
        "        obs = env.reset()\n",
        "        for step in range(n_max_steps):\n",
        "            obs, reward, done, grads = play_one_step(env, obs, model, loss_fn)\n",
        "            current_rewards.append(reward)\n",
        "            current_grads.append(grads)\n",
        "            if done:\n",
        "                break\n",
        "        all_rewards.append(current_rewards)\n",
        "        all_grads.append(current_grads)\n",
        "    return all_rewards, all_grads"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4vI4svXMM9X"
      },
      "source": [
        "정책 그레이디언트 알고리즘은 모델을 사용해 여러 번 에피소드를 플레이합니다(예를 들어 10번). 그다음 모든 보상을 할인하고 정규화합니다. 이를 위한 함수를 만들어 보죠. 첫 번째 함수는 할인된 보상을 계산합니다. 두 번째 함수는 여러 에피소드에 걸쳐 할인된 보상을 정규화합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EQF-In9vMM9Y"
      },
      "outputs": [],
      "source": [
        "def discount_rewards(rewards, discount_rate):\n",
        "    discounted = np.array(rewards)\n",
        "    for step in range(len(rewards) - 2, -1, -1):\n",
        "        discounted[step] += discounted[step + 1] * discount_rate\n",
        "    return discounted\n",
        "\n",
        "def discount_and_normalize_rewards(all_rewards, discount_rate):\n",
        "    all_discounted_rewards = [discount_rewards(rewards, discount_rate)\n",
        "                              for rewards in all_rewards]\n",
        "    flat_rewards = np.concatenate(all_discounted_rewards)\n",
        "    reward_mean = flat_rewards.mean()\n",
        "    reward_std = flat_rewards.std()\n",
        "    return [(discounted_rewards - reward_mean) / reward_std\n",
        "            for discounted_rewards in all_discounted_rewards]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nV2JnT8mMM9Y"
      },
      "source": [
        "3개의 행동을 수행하고 각 행동의 보상이 10, 0, -50이라고 가정해 보죠. 80%의 할인 계수를 사용하면 세 번째 행동은 -50(마지막 보상의 100%)를 받지만 두 번째 행동은 -40(마지막 보상의 80%)만 받습니다. 그리고 첫 번째 행동은 -40의 80%(-32)에 첫 번째 보상(+10)의 100%를 받습니다. 따라서 할인된 보상의 합은 -22가 됩니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cEiOHjbDMM9Y",
        "outputId": "0502b10f-8545-465e-9e88-0a292d6c7bc7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([-22, -40, -50])"
            ]
          },
          "execution_count": 32,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "discount_rewards([10, 0, -50], discount_rate=0.8)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "97_H6aE3MM9Y"
      },
      "source": [
        "전체 에피소드에 대해 모든 할인된 보상을 정규화하기 위해 전체 할인된 보상의 평균과 표준 편차를 계산합니다. 그리고 할인된 보상에서 평균을 빼고 표준 편차를 나눕니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "7pMp9pWzMM9Y",
        "outputId": "606ed469-2126-44f4-bcbd-e3fccfa2fae7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[array([-0.28435071, -0.86597718, -1.18910299]),\n",
              " array([1.26665318, 1.0727777 ])]"
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "discount_and_normalize_rewards([[10, 0, -50], [10, 20]], discount_rate=0.8)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KYl9ZKkqMM9Z"
      },
      "outputs": [],
      "source": [
        "n_iterations = 150\n",
        "n_episodes_per_update = 10\n",
        "n_max_steps = 200\n",
        "discount_rate = 0.95"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3De70JDfMM9Z"
      },
      "outputs": [],
      "source": [
        "optimizer = keras.optimizers.Adam(learning_rate=0.01)\n",
        "loss_fn = keras.losses.binary_crossentropy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-jypGQqYMM9Z"
      },
      "outputs": [],
      "source": [
        "keras.backend.clear_session()\n",
        "np.random.seed(42)\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Dense(5, activation=\"elu\", input_shape=[4]),\n",
        "    keras.layers.Dense(1, activation=\"sigmoid\"),\n",
        "])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l8X7hRqGMM9Z",
        "outputId": "4d0c3f24-e253-4a2a-9df2-06960a89cda6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration: 149, mean rewards: 199.6"
          ]
        }
      ],
      "source": [
        "env = gym.make(\"CartPole-v1\")\n",
        "env.seed(42);\n",
        "\n",
        "for iteration in range(n_iterations):\n",
        "    all_rewards, all_grads = play_multiple_episodes(\n",
        "        env, n_episodes_per_update, n_max_steps, model, loss_fn)\n",
        "    total_rewards = sum(map(sum, all_rewards))                     # Not shown in the book\n",
        "    print(\"\\rIteration: {}, mean rewards: {:.1f}\".format(          # Not shown\n",
        "        iteration, total_rewards / n_episodes_per_update), end=\"\") # Not shown\n",
        "    all_final_rewards = discount_and_normalize_rewards(all_rewards,\n",
        "                                                       discount_rate)\n",
        "    all_mean_grads = []\n",
        "    for var_index in range(len(model.trainable_variables)):\n",
        "        mean_grads = tf.reduce_mean(\n",
        "            [final_reward * all_grads[episode_index][step][var_index]\n",
        "             for episode_index, final_rewards in enumerate(all_final_rewards)\n",
        "                 for step, final_reward in enumerate(final_rewards)], axis=0)\n",
        "        all_mean_grads.append(mean_grads)\n",
        "    optimizer.apply_gradients(zip(all_mean_grads, model.trainable_variables))\n",
        "\n",
        "env.close()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B63GZYuLMM9Z"
      },
      "outputs": [],
      "source": [
        "frames = render_policy_net(model)\n",
        "plot_animation(frames)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DGkNh4UrMM9Z"
      },
      "source": [
        "# 마르코프 연쇄"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DHyyVS37MM9a",
        "outputId": "16c14253-c256-469e-b23a-97211c9394db"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "States: 0 0 3 \n",
            "States: 0 1 2 1 2 1 2 1 2 1 3 \n",
            "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 3 \n",
            "States: 0 3 \n",
            "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 3 \n",
            "States: 0 1 3 \n",
            "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 ...\n",
            "States: 0 0 3 \n",
            "States: 0 0 0 1 2 1 2 1 3 \n",
            "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 3 \n"
          ]
        }
      ],
      "source": [
        "np.random.seed(42)\n",
        "\n",
        "transition_probabilities = [ # shape=[s, s']\n",
        "        [0.7, 0.2, 0.0, 0.1],  # from s0 to s0, s1, s2, s3\n",
        "        [0.0, 0.0, 0.9, 0.1],  # from s1 to ...\n",
        "        [0.0, 1.0, 0.0, 0.0],  # from s2 to ...\n",
        "        [0.0, 0.0, 0.0, 1.0]]  # from s3 to ...\n",
        "\n",
        "n_max_steps = 50\n",
        "\n",
        "def print_sequence():\n",
        "    current_state = 0\n",
        "    print(\"States:\", end=\" \")\n",
        "    for step in range(n_max_steps):\n",
        "        print(current_state, end=\" \")\n",
        "        if current_state == 3:\n",
        "            break\n",
        "        current_state = np.random.choice(range(4), p=transition_probabilities[current_state])\n",
        "    else:\n",
        "        print(\"...\", end=\"\")\n",
        "    print()\n",
        "\n",
        "for _ in range(10):\n",
        "    print_sequence()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "51NUkpFaMM9a"
      },
      "source": [
        "# 마르코프 결정 과정"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JNr2w5rlMM9a"
      },
      "source": [
        "전이 확률, 보상, 가능한 행동을 정의해 보죠. 예를 들어, 상태 s0에서 행동 a0가 선택되면 0.7의 확률로 상태 s0로 가고 +10 보상을 받습니다. 그리고 0.3의 확률로 상태 s1으로 가고 보상이 없습니다. 상태 s2로는 이동하지 않습니다(따라서 전이 확률은 `[0.7, 0.3, 0.0]`이고 보상은 `[+10, 0, 0]`입니다):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0fKqxeBvMM9a"
      },
      "outputs": [],
      "source": [
        "transition_probabilities = [ # shape=[s, a, s']\n",
        "        [[0.7, 0.3, 0.0], [1.0, 0.0, 0.0], [0.8, 0.2, 0.0]],\n",
        "        [[0.0, 1.0, 0.0], None, [0.0, 0.0, 1.0]],\n",
        "        [None, [0.8, 0.1, 0.1], None]]\n",
        "rewards = [ # shape=[s, a, s']\n",
        "        [[+10, 0, 0], [0, 0, 0], [0, 0, 0]],\n",
        "        [[0, 0, 0], [0, 0, 0], [0, 0, -50]],\n",
        "        [[0, 0, 0], [+40, 0, 0], [0, 0, 0]]]\n",
        "possible_actions = [[0, 1, 2], [0, 2], [1]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HVNgfWe0MM9a"
      },
      "source": [
        "# Q-가치 반복"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rew91PtsMM9a"
      },
      "outputs": [],
      "source": [
        "Q_values = np.full((3, 3), -np.inf) # 불가능한 행동은 -np.inf\n",
        "for state, actions in enumerate(possible_actions):\n",
        "    Q_values[state, actions] = 0.0  # 모든 가능한 행동에 대해"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzX_h6NgMM9b"
      },
      "outputs": [],
      "source": [
        "gamma = 0.90  # 할인 계수\n",
        "\n",
        "history1 = [] # 책에는 없음\n",
        "for iteration in range(50):\n",
        "    Q_prev = Q_values.copy()\n",
        "    history1.append(Q_prev) # 책에는 없음\n",
        "    for s in range(3):\n",
        "        for a in possible_actions[s]:\n",
        "            Q_values[s, a] = np.sum([\n",
        "                    transition_probabilities[s][a][sp]\n",
        "                    * (rewards[s][a][sp] + gamma * np.max(Q_prev[sp]))\n",
        "                for sp in range(3)])\n",
        "\n",
        "history1 = np.array(history1) # 책에는 없음"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PbyItG4pMM9b",
        "outputId": "f4b300c1-79f2-4b57-b163-21f512fccba8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[18.91891892, 17.02702702, 13.62162162],\n",
              "       [ 0.        ,        -inf, -4.87971488],\n",
              "       [       -inf, 50.13365013,        -inf]])"
            ]
          },
          "execution_count": 43,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "Q_values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xHsBvKMpMM9b",
        "outputId": "4ddf59f8-2bdf-4891-8136-7468cd38c3b3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0, 0, 1])"
            ]
          },
          "execution_count": 44,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "np.argmax(Q_values, axis=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F1z-sffDMM9b"
      },
      "source": [
        "할인 계수 0.9를 사용했을 때 이 MDP의 최적 정책은 상태 s0에서 행동 a0를 선택하고, 상태 s1에서 행동 a0를 선택하고, 마지막으로 상태 s2에서 행동 a1(선택 가능한 유일한 행동)을 선택하는 것입니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4UDoc4sxMM9b"
      },
      "source": [
        "할인 계수 0.95로 시도해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xUNa27J3MM9b"
      },
      "outputs": [],
      "source": [
        "Q_values = np.full((3, 3), -np.inf) # 불가능한 행동에 대해서는 -np.inf\n",
        "for state, actions in enumerate(possible_actions):\n",
        "    Q_values[state, actions] = 0.0  # 모든 가능한 행동에 대해서"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GwPPi5GgMM9c"
      },
      "outputs": [],
      "source": [
        "gamma = 0.95  # 할인 계수\n",
        "\n",
        "for iteration in range(50):\n",
        "    Q_prev = Q_values.copy()\n",
        "    for s in range(3):\n",
        "        for a in possible_actions[s]:\n",
        "            Q_values[s, a] = np.sum([\n",
        "                    transition_probabilities[s][a][sp]\n",
        "                    * (rewards[s][a][sp] + gamma * np.max(Q_prev[sp]))\n",
        "                for sp in range(3)])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IjMO4FG4MM9c",
        "outputId": "aecd24fe-1183-494f-9cb6-d9682b9861e4"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[21.73304188, 20.63807938, 16.70138772],\n",
              "       [ 0.95462106,        -inf,  1.01361207],\n",
              "       [       -inf, 53.70728682,        -inf]])"
            ]
          },
          "execution_count": 47,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "Q_values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TZ74tFImMM9c",
        "outputId": "9ac08a10-fd7a-46a6-d0b7-0c74455effbf"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0, 2, 1])"
            ]
          },
          "execution_count": 48,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "np.argmax(Q_values, axis=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bPZSt8aWMM9c"
      },
      "source": [
        "이제 정책이 바뀌었습니다! 상태 s1에서 불 속으로 들어가는 것을 선택합니다(행동 a2). 할인 계수가 크기 때문에 에이전트가 미래에 더 많은 가치를 두기 때문에 미래 보상을 얻기 위해 당장의 불이익을 감내합니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "byyrVmOOMM9d"
      },
      "source": [
        "# Q-러닝"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k6NvMNmhMM9d"
      },
      "source": [
        "Q-러닝은 에이전트의 (예를 들면, 랜덤한) 플레이를 보고 점진적으로 Q-가치 추정을 향상합니다. 정확한 (또는 충분히 가까운) Q-가치 추정을 얻으면 최적의 정책은 가장 높은 Q-가치를 가진 행동을 선택하는 것입니다(즉, 그리디 정책)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvxJeIjUMM9d"
      },
      "source": [
        "환경을 돌아다니는 에이전트를 시뮬레이션해야 합니다. 따라서 행동을 선택하고 새로운 상태와 보상을 받는 함수를 정의해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R2bGoGb4MM9d"
      },
      "outputs": [],
      "source": [
        "def step(state, action):\n",
        "    probas = transition_probabilities[state][action]\n",
        "    next_state = np.random.choice([0, 1, 2], p=probas)\n",
        "    reward = rewards[state][action][next_state]\n",
        "    return next_state, reward"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Q2HWVhNMM9d"
      },
      "source": [
        "또한 탐험 정책도 필요합니다. 가능한 모든 상태를 여러번 방문한다면 어떤 정책도 가능합니다. 상태 공간이 매우 작기 때문에 랜덤한 정책을 사용하겠습니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qlonNWVgMM9e"
      },
      "outputs": [],
      "source": [
        "def exploration_policy(state):\n",
        "    return np.random.choice(possible_actions[state])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u5yJmXHEMM9e"
      },
      "source": [
        "이제 앞에서와 같이 Q-가치를 초기화하고 Q-러닝 알고리즘을 실행해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7C5MYH2NMM9e"
      },
      "outputs": [],
      "source": [
        "np.random.seed(42)\n",
        "\n",
        "Q_values = np.full((3, 3), -np.inf)\n",
        "for state, actions in enumerate(possible_actions):\n",
        "    Q_values[state][actions] = 0\n",
        "\n",
        "alpha0 = 0.05 # 초기 학습률\n",
        "decay = 0.005 # 학습률 감쇄\n",
        "gamma = 0.90 # 할인 계수\n",
        "state = 0 # 초기 상태\n",
        "history2 = [] # 책에는 없음\n",
        "\n",
        "for iteration in range(10000):\n",
        "    history2.append(Q_values.copy()) # 책에는 없음\n",
        "    action = exploration_policy(state)\n",
        "    next_state, reward = step(state, action)\n",
        "    next_value = np.max(Q_values[next_state]) # 다음 스텝의 그리디 정책\n",
        "    alpha = alpha0 / (1 + iteration * decay)\n",
        "    Q_values[state, action] *= 1 - alpha\n",
        "    Q_values[state, action] += alpha * (reward + gamma * next_value)\n",
        "    state = next_state\n",
        "\n",
        "history2 = np.array(history2) # 책에는 없음"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2pA3UavRMM9e",
        "outputId": "62542fdc-c754-4e1a-8e3c-5c030138ac23"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[18.77621289, 17.2238872 , 13.74543343],\n",
              "       [ 0.        ,        -inf, -8.00485647],\n",
              "       [       -inf, 49.40208921,        -inf]])"
            ]
          },
          "execution_count": 52,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "Q_values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P2K_DxamMM9e",
        "outputId": "215116b7-af9f-442b-fda9-535c3073ea9d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0, 0, 1])"
            ]
          },
          "execution_count": 53,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "np.argmax(Q_values, axis=1) # 각 상태에 대한 최적의 행동"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mZyBRX1zMM9f",
        "outputId": "6aba482f-9bf2-462a-f754-a66473c47259"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Saving figure q_value_plot\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAEYCAYAAABBfQDEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XecXFX5x/HPk82md5JAEpIAMSEISvwRaVKigAqioKGJKIJ0UIoogpRIEWlSBJRqkCJNECMo0gIoLaEkkACBBEgnPdmWtnl+f5w77N2Z2TK7s3tnZ7/v1+u+ZuaeuXees7Nz55lzzznX3B0REREREQk6JB2AiIiIiEghUYIsIiIiIhKjBFlEREREJEYJsoiIiIhIjBJkEREREZEYJcgiIiIiIjFKkEVERKRgmNlWZuZmNjbpWKT9UoIsRcPMLjWzt5KOIylmtm/0pdIn6VhEpHWY2RAzu9XM5pvZejNbYGa3mdmWDWw32cxubK04czQPGAS02+O5JE8JsuRFMw7S083s9jrKDogSvlEtE3XTmNlxZraqrsetFMN8MzsjbfULhC+V1a0Zi4gkw8y2BqYCOwBHA58DjgK2B6aY2VaJBZeFmXVqzPPcvdrdF7v7xpaOSaQuSpCl2Zp5kL4DONzMumcpOxZ40d1n5TXgAmVBaVO3d/f10ZeKLo8p0j7cBGwC9nX3Z9x9rrs/B+wbrb+pqTuOGj3uN7OV0fK4mY2MlY8ws8fMbLGZVZjZG2Z2YNo+PjazCWZ2Z9SIcG+s+8R4M3vKzCrNbKaZ7RfbrlYXCzMbFz3ex8xejbaZamb/l/Z6x5rZ3Kh8kpmdYmY6HkqTKEGWfGjOQfpuoBQ4LL7SzAYA3wFujx6XRgfZj8ysysxmmdnZZmZ17djM7jGzv6ety+iGEbUAv2tma83sfTP7WX37Tdt2X+A2oHd0AHczOz8q62xmV0Wt6RVm9lr0/M+2jZ7/TTObCqwD9jGzkWb2DzP71MzKzex1M9s/tt1/gSHAtdH2G9P21yf23EPM7B0zWxd9cZwbr1vUEn2umd1uZmvMbJ6ZndWYuotIcsysH/BN4CZ3r4yXRY9vBvY3s75N2Hc34DlgLbA3sBuwCHg6KgPoAfwL2A/YEfgb8IiZjU7b3VnAe8BY4LzY+suAG6JtpwD3m1mPBkK7HPgV8H/AckLCbVHMuxG+L24CxgD/AH6TU8VFYpQgS7M09yDt7iuAvxNai+N+CFQBD0ePS4C5hER6O+DCaPlRM+M/GbgYOD/a7y+j+yc2chcvAD8H1hC6NwwCro3K/gJ8BTgC+CJwL/C4me2Qto8rgHOB0YSW+J7A44QfGF8CHgMei7XefIfwZXVh9HpD6qjbzsCD0fIF4NdR3U5Ke+rZwBuEL53fA9dE24pI4RoJGPBuHeUzo/KRdZTX54ho22Pcfbq7v0c4JvYADgRw92nu/id3f9vdP3T3ywjHkUPS9vW8u18ZPeeD2Ppr3X1StO48oB8hsa3PBe7+XBTPxYRjZur49zPgP+5+hbvPcvfbgEebUHcRQAmyNF8+DtK3A3tY7b7GxwL3pZJud1/r7hPcfYq7f+zu9wO3At9vZvznAz9397+5+0fu/hhwJXBKYzZ29/WE5Nij7g2L3b0iqsuhwCHu/qK7z3b364GngBPSdnOhuz/l7nPcfZm7v+Hut0RfPB+4+8XAdGB89JorCC3zZdHrfVpHeD8Hnnb3i6MvjLsJyfs5ac97wt1vjr7ArgU+Br7WmPqLSOLq6kKQOlO0X3QmKrXs2Yh97gRsDZSltiOMbegLjAAws+5mdmXUPWJl9JyxwLC0fU2t4zWmx+4vjG4HNhBXfduMBl5Le/6rDexPpE4dkw5AikZDB+n1ZvYD4JZY2f7u/iLwDPARISn+lZntQui/fHStHZmdChwDDAe6ErpmzG5qwGY2CBgM3GFmt8WKOgLVTd1vZCdC3Wel9dboDPwn7bm1vkCi04wTgG8RWog7Al3IPPg3ZDvCac+4/wK/NrNusRb/6WnPWUjDX1QikqwPCMfd7Qln4dJtB2wEbgQeiK1f0Ih9dyDMIHFElrIV0e3VhLOHZ0exVBLOmqUPxKuo4zU2pO64u0fHyYYa7TbE7qe+c1LbGHV/D4nkTAmyNFdjD9IfEZLZ+C/6BfDZwfHPwMlm9mvgJ8A0d3899cQoub6a0Cr6CqHV9mfAAfXEtomaBD0lPggudWA9nsyWhuYeaDsQkuydyEy2K9Mep3+BXEtowf0F8GH0/HvJ/OJpSF1fGOnrNmQp19klkQLm7ivM7N/AKWZ2bbyLW9RP+FTgUXdfTe4z27xBODu3zN3rmqFnD+Av7v636DW7EFqXkxpU/S6Q3jVMXcWkyZQgS7PkeJAGKKtjV38mtJoeSmi1OC+tfA/gJXe/Obb/zzUQ3lLCabe4eB+3hcCnwDbufm8D+6rPekIf6bg3onUDo1byXOwBTHT3R+Czv+M2wNsNvGa6mdG+0vf9SXp/cRFpk04FXiYMnjuf0GAxgjAAbgOhEaE+/c0svd/vEsIP8rMJYx8uJIz/GAocBPwp6jc8C/iumT0WvdZFhDNdSbkB+K+Z/YLQWLMX8N0E45E2Tq1Ekg+nEpK1p83sa2Y21MzGEfrbNuYgjbvPB54kDOorJRyg42YBY83sG9EsDxMIA+Dq82y0zdFm9jkzOxfYJfaaTkjKzzWz081sWzPbIXp+ej/d+nwM9Ijq3t/Murr7u4TTmn+xMJ3R1mb2ZTP7pZkd3MD+ZgHfM7MvmVlqcF/nLK+5l4WpmDarYz/XAPua2QVmNsrMfgicSehjLSJtnLt/ROj3O4MwI9DHhNknNgFj3H1xA7s4HHgzbTkr+gG9FzAHeIgwC8VdhD7IK6NtzyIk0y8SZrN4JbqfCHd/mXA28GeEbmMHEwZAr00qJmnj3F2LlmYvwJaE6c4WELoUOOFg2TeHfXwv2u7eLGWdCa3MqwgH6NsIU/h8GHvOpcBbadtdAiwmnGK8kXDATH/OUYQvhrWE/nUvAofVE+dxwKrYYyMMGFwWxX9+tL4TYaT1HEKL7yLCjBRfisr3jZ7fJ23/WxGS+0rCFaXOBP4N3B57zlcILcrrgI117Y8wovyd6PXnEmbLsFj5fOCMtNf/L3Bd0v9TWrRoyX0Bfhp93g9KOpakF0J3tbeTjkNL21zMXX3aJf/M7KeEFsxDPcwMISIircDMDiN0tbjO3auSjqe1RN0rngLKCQ0G1wLneZidRyQnSpClxbTXg7SIiLQ+M3sAGAf0JgwMvwW43pXoSBMoQRYRERERidEgPRERERGRmKKa5q1///6+1VZbJR2GiEizvP7668vcfUC+96tjpIgUg5Y6RsYVVYK81VZbMXVqXVe1FBFpG8zsk5bYr46RIlIMWuoYGacuFiIiIiIiMUqQRURERERilCCLiIiIiMQoQRYRERERiVGCLCIiIiISowRZRERERCRGCbKIiIiISIwSZBERERGRGCXIIiIiIiIxSpBFRERERGKUIIuIiIiIxChBFhERERGJUYIsIiIiIhKjBFlEREREJEYJsoiIiIhIjBJkEREREZEYJcgiIiIiIjFKkEVEREREYpQgi4iIiIjEKEEWEREREYlRgiwiIiIiEqMEWUREREQkRgmyiIiIiEiMEmQRERERkRglyCIiIiIiMUqQRUREErZpE6xdG+5v3AgrVoT7ZWXw4YfJxSXSXnVMOgAREZH2zB0GDYIlS+p+zpQpMHYszJ0LCxbANtvA5pu3Xowi7Y1akEVERFrJ7ruDWVieeCKsmzSp/uQY4MtfDtsMHx72scUW4fH3vw+VlS0ft0h7owRZRESkFbzzDrz8cs3jb30rJLkHHdT0fd5/P3TvDvPnNz8+EamhBFlERKSFrVkD559f/3MeeCD0RXYPS1kZzJkT1vfqVf+2Q4eGbUQkPxJJkM2ss5ndYWafmFmZmb1pZvvHyvcxs/fMrNLMnjOz4UnEKSIi0hzXXx9aiXv3hsceC+tuvBFeeqn28x55BA47LDw3pUcP2HrrsH716pAAL1oUkui5c+G552r3Q+7QAfbdV10uRPIhqUF6HYF5wN7AXOAA4EEz+wJQDjwCHAdMAi4BHgB2TSZUERGRxquqgj/8ARYvhmuvzSw/8kjo2xeWL4fTT4cDD4Tvfrdx+95ii3A7dGhYFi0KiXHKM8+ELhdlZSHBFpGmSSRBdvcKYEJs1T/N7CNgJ2AzYIa7PwRgZhOAZWY22t3fa+1YRUREGuuJJ0Lf4rpMmBCSY4B+/eDuu5v3emawfj106lR7fc+eIVneYw+48EI4/ngYMaLu/TzzTOgf/ctfZu5LpD0qiD7IZrY5MAqYAWwPTEuVRcn07Gi9iIhIXk2aBIMHh2Tz1VfDujPOCInlpk2N20dlJfziF9mT48suCy297nDRRfmLO6W0NMT57rvwwx/WrN9nH+jcGa64Aj73OXj22TBbxqOPwimnQEVFeN7f/x66ZlxwAXTrBg8/3Ph6ixQr84R79ZtZKfAvYLa7n2hmdwBL3f1Xsef8D7jN3Sdm2f4E4ASAYcOG7fTJJ5+0TuAiIi3EzF5397F52peOkfV4+mnYb7/a626/HY47rubxW2/BjjvWvY+NG8Mguqqq2uunTYOPP4bvfCdv4TZow4bcWoBLS8M26UaNgvffz19cIvmUz2NkXRJtQTazDsDdwHrgtGh1OZA+XrcXUJZtH+5+q7uPdfexAwYMaLFYRUTaIh0jM334ITz4YGjRPeGEzPJ4cgwwZgy89lrd+9t779rJsVlogf3iF1s3OYaQ8LqHxDylvguKZEuOAWbNCvU46aT8xPXqq3DvvfnZl0hrSCxBNjMD7gA2B8a7e+pjOgPYMfa87sCIaL2IiEizjBwJhx8eBrd99FFYt+229W+zyy4hYUy1qq5bF/r+nnpqzYwU220X+iBXVdWejSIJw4eH5HfTpjBY8LXXQqKfzU47hVkyKitDd5O4W24JdVm4sPb6l18OXTXiVq+GPfesuRBKfNl1VzjqqJrHxx0HH3wAP/lJmMv52mtDH+n07b7+9drdPaqrwwDEs88OyXt6q71IviTWxcLM/gSMAfZ19/LY+gHAh8CxwOPAb4C93b3BWSzGjh3rU6dObaGIRURaR0udPmyPx0j3kGi5w8UXw4oVcMMNtZ8zcGDoI7xuXeiDC3DmmXD11VBS0vjXmj07XAK6kC1YEOpbWhpadJcsCf2t4wn94sXh0tfZDBwYBhmmd7/45jfh3/9uubjrs912cNNNob/1NtvAJZfAzJmhL/VmmyUTk7Ss1uhikcgsFtG8xicC64DFVvPJPNHd7zWz8cCNwD3Aq8ARScQpIiJtV6p/cefOYcaIRYtql5eWwsknwzXXhNbkrl1DIl1dXZMYu4dp2B5/vP7Xuuuuwk+OAYYMqbn/gx9kf84WW4R6z50bWqLjlizJflnshpLj00+HPn3gN7/JLd7GePdd+NrXMtf371/78XvvNXymoDnKy2HGDOjSpf4+69I2JNLFwt0/cXdz9y7u3iO23BuVP+3uo929q7uPc/ePk4hTRETartTgu3XrMpPj558PXSSuvx46pjUVpbca//OfMG9e3a9TUQE/+lHz4y00w4aFRPmf/8wsiyfacb/6Vc2VAOPLddeFKe7cw6DGo4+GceNg2bLQLaW8PHObbBc86dQJ9t8/zDP91FO51Wf06JquGy+8kNu2ELpzLF6cuf6++2D8+DC13q67hq4sqdd5991Q3zff1JUO25rEZ7HIp/Z4+lBEio+6WDTfzJmwfR2Tg26xRehq0CHHJqKXXw6ty+vXh32bhZbp9AS7WLmHvsiDBtX+2338cbji3157hR8eLWHTprrfrzlzwnv697+HH0KnnRZauQ84AN55p+F9/+53Yf7n1Mns+fPhz38OU/L94Q9hf3XZe+/c6/zCC6Hf94YN4SzG3/8eYu3TJ7f9tIS1a0O//Ntvh3POCd1pSkpy/6y0tNboYqEEWUSkwChBbr7DDoOHHqq9bocdQjLTq1f7SWoleP11GNvAJ6pDh+bN/7zHHmHg4pQpIeHNVdeuobV5xowwkLQhS5eGbbp3r0nun3gidAk67riQ4NfV0p/NJ5+EVv34DCjpzML84IceWvcP0NagBDlHPXv29J122qnWusMOO4xTTjmFyspKDsjyH/vjH/+YH//4xyxbtoxDDjkko/zkk0/m8MMPZ968efwwPgN75Oc//znf/va3ef/99znxxBMzys8//3z23Xdf3nrrLc4444yM8t/+9rfsvvvuvPTSS5x33nkZ5ddddx1jxozh6aef5tJLL80ov+WWW9h2222ZNGkS11xzTUb53XffzRZbDGXixEe5/faH2bixO9XVndm0qQubNnXmtNPOoaSkJ8899yqvvPIm7qVs2lSKe0c2bSrlW986GPdS3n77XT76aC7uJbh3iG5L2G23PaiuhtmzP2LJkuVAB9zDT80OHTry+c/vwKZNMG/efFavXgMYYLgbHTt2ZOutR0StEgupqEidTwuf9I4dOzF06FAAFi1aRFXV2lp169SpM4MHDwbC9uvXr69V3qVLFzbffIto/wvYuHFjrfKuXbsyYMBAABYsmE91dfVnZe5G9+7d2Gyz0Ilt/vx5bNpU+7PSo0d3+vULI0Dmzp2b8bfv1asnffr0xX0T8+bNzyjv3bs3vXv3prp6IwsWLMwo79u3Dz179mLjxg0sXLgoo7xfv3706NGD9evXsXjxpxnl/ftvRrdu3Vm3bi2ffprZaXDAgP507dqNqqpKli5dllG++eYD6dy5C5WVFSxbtjyjfIstNqdTp86Ul5ezYsWKjPLBgwfRsWMpZWVrWLlyVUb5kCGDKSnpyOrVq1m9enVG+dChW2LWgVWrVrJmTeYsj8OGDQNgxYrllJdX1Crr0MHYcsvwv7N8+bLY/1ZQUlLCkOibY+nSJRn/W6WlHRk0KPxvLVnyKWvXrqtV3qlTJ7aIrvm7ePHiLP97nRk4MMyttWjRQjZsSP/f68KAAQM54gh4++3xLF9e++/7/PPPt8jBvz0cI8Op+a2YOnUiABdc8ClHH705l1wykw8//DkdO9ae9uDuu+9m6NChPPDAA/zxj3/M2P/DDz9M//79mThxIhMnTswof+KJJ+jWrRs333wzDz74YEb55MmTAbj66qv5Z1pfha5du/Kvf/0LgEsuuYRnnnmmVvlmm23G3/72NwDOPfdcXn755VrlW265Jffccw8AZ5xxBm+99Vat8lGjRnHrrbcCcMIJJzBr1qxa5WPGjOG6664D4KijjmL+/NrHqd12243LL78cgPHjM/9P99lnHy644AIA9t9/f6rSppQ48MADOfvsswEYN24c6ZL636usHMqUKY2/hOHWW1dSXf0wm232CiUlFXzwwVmsXh06Ge+4I2yzzVyWLj2OkpLax4FrrrmNSZNGsnr1m7z88m9ZsOB7zJ///Ua95oABzzFz5hfo378/d9xxF3fe+RClpeWUl4/g9dfvaHTsAIcf/gSvvTaf+fMPYYst/s2gQf+ke3fnmWf+ihmccspt3HrrD6iu7pbTflN69lzB2rVQXd2VTZu6AtCnz0K+/vXBVFXB9OlvsnHjHDp3XsLatYNYsuSrdOmyiS9/uS/dusHChffRs+cdVFVtQXn5KHr3ns64caO4/PLLqaqCww47mrKy2vO3t9QxMk6/odsod2PdugG89lpX3ngDnn56G2bPPpn16/uxfv1mrF/fl40bezB69GZRP67vRkttJ5+curdLtNQWHVuB7aKltprj/dbRUtv//pe6t2XWerz5Zure4KzlNXlT5pDqykpY9Vnelbl9ZWUYsR5k/oyurISa431mfFVVoX9cMDRr+dKlqUfDspZ/+imErv7Zy0N/to51lgelWcsXLEjd65y1vKbPZJes5TU5fbes5TWtCN2jpbbU9FjQI1pqmz07da8XmVObh7log97RUlvNd3nfaKmtZhT9ZtFSV3n/jLLa5QMzyqqqYM2a1KPMSWSrquL/m1tkLV+5MvUo83+zqir8b36a+btGmqGsbCRvvHFbrXX77VfGiBGbM378bK65RnOCCXTrNo+99x732Y+zq676iEsuWUtZWfiOGzHiRnr3nsbvf38se+21Gy+//BbnnXfnZ9uPGXM6EP9xNotLL12f8To9emxiwgSYNGk+b765lBEjbmHLLf/GRx8dz8aN3fn977dl6NABvPzyU1x00S5s3FhznFy69KsMGBDOerzzztHA0U2u7wMP1Pz4mDfvSObNOxKId5s4/rPyXr3eYYcdzqVfvz785jd3M2UK/OMfD/P22+uZP/9Q+vV7jbKybVm/vua4WlbWL+M1V60aTM3vxS9FS43ycnjuudSjI6OlxvLlM5g5Mwy0ray8i9LSFWza1ImuXRfQtesCYJ9c/ww5K6oW5GI8fege5oqcNi109n/vvbC8/372AQzZdOgQ+jb16QO9e4fTMd261V66dg0jvTt3DoMgOnUK90tLw/2SknBKMrWkHqf6JqXfdugQTsXE7zdmgcz7KXXdj8t1fX2SnsdUil+fPqHvZDp1sWi81ICuRYtgyyy/w4voK07agYoK6JHZ3pDhq18N3Tj23LPm/q9/HfqC9+kTGjB22in+Q71hF14YBlI25rtv6dJwhcmpUyH9xM4pp4R5twcNCvNql5SEvvsQBli++mrILZrXQFCk07xJ3dxD8jt5cugrN3ly9lGzEK6ONGJE6GM0aFDNMnhwKOvbN3xQevRQsicixSd+WeUsvTe47LLWjUekubp3D41fRx4ZBu9B+BF9xx3hO71Xr8y+v9kay7beOpyl2rQJHnkE9t035AOhK0QYiHjggeFM33/+A/vsk1ueMGBAmCVmv/3g3HMbfn59gyznzw/9rvfaKzTWPfAA3HhjeP6pp4aBsHfdFR6/+27o490a1IJcANxDIvznP4d/1PRfVQMHhqs4bbddWEaPDnM59s086ywiRUAtyI1zxhlhmrZsNm7M7SIfItJ2FO2FQiT49FOYODFMp1LTHzO0/u69dxhNOm5czdyNIiISuGdPjk8+OcxgoeRYRJpDCXIrcw+dzm+5BR57LLRyQOg7d+yxcMQRSohFRLKpqAhdzrbZJvS7zObmm1s3JhEpTkqQW9HHH4f+NE88ER6XlMB3vgMnnBCuY68WDxGR7MrKQv/LbP70JzjpJLjpptaNSUSKlxLkVrBxY7jM5kUXhc70vXvD2WeHFuPB2Wc3ExGRmN13z77+6afDAKNsg/RERJpKCXILmzIFjj8+TNMGcPjhIVnONrWTiIjUVl0d5uzOdsng884LybGISL4V2NW1i0dlJfzsZ2H2iWnTYPjw0LXi/vuVHIuINEZZWZhvfZttatbFL8iW5cJ6IiJ5oRbkFlBRAd/+drhKTEkJnHVW6F7RPfNiZCIiUodrrslc9+yzYfaf3XfXMVVEWo4S5DyrqAiTb0+eHFqKH38c/u//ko5KRKTtefHF2o+/9a0ww88xxyQTj4i0HzknyGbWGRgMdAWWuvvSvEfVRsWT40GDQgvyttsmHZWISNvz5JOhtRjCVbSGDAnzw4uItIZG9UE2s55mdrKZvQCsBj4E3gEWm9k8M7vNzL7ckoEWuoqK0Lqh5FhEpPH+8Y8wRuPZZ2HmTNhhh3CV0G9+s+Y5u+4aBuN11DlPEWklDR5uzOxM4HxgDvAP4DJgIVAF9AN2APYEnjKzV4CfuvsHLRZxASovD8nxCy+Eadueew5GjUo6KhGRwrZmDRx0ULi/zz7wxS/CjBmZzxs6tHXjEhFpzO/x3YG93T3LJDsAvAbcaWYnAT8B9gbaTYKs5FhEJDfV1WEA82mn1V4/fXrmc885B7p2bZ24RERSGkyQ3f3QxuzI3dcB7e4in6ecUpMcT54MI0cmHZGISOG68074yU8a99x774Ujj2zZeEREslGPrmZ46im4+27o0gWeeUbJsYhIfRYubFxy/Lvfwfe/D8OGtXxMIiLZNDlBNrPNgQOBIYAT+iU/7u6L8xRbQaushJNOCvcnTIDRoxMNR0Sk4A0Zkn39kUfCffeF+zfdFM7MiYgkqUlX0jOzk4Fnga2AJcBSYGvgGTNrF4e2iy+GOXPCoJKzzko6GhGRtqlbtzCNm4hIIWlqC/IZwBh3r4qvNLPLgGkUeV/kadPg6qvDhPW33QalpUlHJCJS2Natq7l/9tnhGArwm9+E6dseeADuuQeOPz6Z+ERE4pqaIDvQlzDVW1y/qKxoVVeHA3h1Nfz0p7DzzklHJCJSeGbPhgEDoFev8HjZspqyq66CQw+F+fNrpnk77LCwiIgUgqYmyGcBk81sBjAvWjcM+DxwZj4CK1Q33QRTpsCWW8JllyUdjYhI4Zk5E7bfvubxmWfWJMJf+EK43XlnNTCISOFqUoLs7k+Y2ZPAzoTLThuwAHjN3avzGF9BmTsXzjsv3L/pJujZM9l4REQKzYYNNd0nUq69NiwQWpVFRApdTgmymXUm9D8eDcwH3gLecvfZLRBbQXEPk9pXVMD48fCd7yQdkYhIYXnqKfj61+t/zqBBrROLiEhz5NqC/CdgP+Bx4BxgLdDdzMqA6e6+V57jKxiPPAKTJoX+dDfckHQ0IiKFo7IS3nyz4eQYas7CiYgUslynefsW8CN3PxFYB3wZOBaoBF7Jc2wFwx0uvDDcv/zycNU8EREJjjgC9tij7vLjjqu5rznjRaQtyLUFuSvwQXR/PdDB3e8ys57A5/IaWQF56aUw6GTzzTUFkYhIukmTsq/32JxGv/sd9OkDHZo0+76ISOvK9VA1h3DlPAiD8raM7v8LOCJfQRWa224Lt8ccozmPRUTibq5j1vsuXWo/3mwzKClp+XhERPIh1wT5QSDVy2wy8JPo/heALtk2aOtWrgwT2EPt04QiIu3dI4/AqafWXjd2bJgG8403kolJRCQfcupi4e7xmX+vBKaY2QqgB3BLPgMrFPfeC2vXwr77wogRSUcjIpK8f/4TRo0KM/rE/ec/sN9+ycQkIpJPTe4N5u7zge2B04GD3f2nuWxvZqeZ2VQzW2dmE2PrtzIzN7Py2HJBU+NsDne49dZw/4QTkohARKSwTJ0K3/42bLvOwCVNAAAe70lEQVRt7fX9+ys5FpHi0dQr6QHg7iuAu5u4+ULgUuAbhMF/6fq4+8amxpYPr74Kb78dJrZPXQVKRKQ9mzEjc90uu4RWZRGRYtGsBLk53P0RADMbS81gv4KSaj0+5hjo1CnZWERECkF5eea6V4p2kk8Raa8KecKdT8xsvpn92cz6t/aLr14N998f7mtwnohIsDHtvF56P2QRkWJQiAnyMsIFSIYDOwE9gXvrerKZnRD1ZZ66dOnSvAVx771QVQVf+xqMHJm33YqItKp8HyPLymrun3gi/PGPzd6liEjByVuCbGbDzKzZ+3P3cnef6u4b3f1T4DTg62bWq47n3+ruY9197IABA5r78tE+4ZZoTg4NzhORtizfx8gLoiHTV14Jf/pTGKMhIlJs8tmC/DEwzcz2yuM+AVLXYrI877dOU6bA9OlhVPbBB7fWq4qIFJ5Vq2DJknB/ypSa9W+/nUw8IiKtIZ+D9I4FtgauAnZp6Mlm1jF6/RKgxMy6ABsJ3SpWES5p3Re4AZjs7qvzGGu9UoPzfvxj6Ny5tV5VRKSwuEPfvuF+RQU8/3xN2TbbJBOTiEhryFuC7O4To7sXNXKT89OeexTwG+B94LfAQGAN8BTw/fxE2bA1a+Cvfw33jz++tV5VRKSwuMP++9c8vvhi6BA753j66a0fk4hIa2lygmxmmwMHAkMI3SAWAo+7++LGbO/uE4AJdRT/talxNdd990FlJYwbF64UJSLSHk2fDk8+WfP4iitq+h8ffHBNy7KISDFqUh9kMzsZeBbYClgCLCV0r3jGzE7JW3QJuPPOcKvBeSLSno0Zk7nupZfC7bhxrRqKiEira2oL8hnAGHeviq80s8uAacDNzQ0sCStWhMuoduqkK+eJSPvlnrnu6KNDqzJAly6tG4+ISGtraoLshAF0VWnr+1Ez60Sb8+KL4Yth112hW7ekoxERScb69Znr7rqr5v4uDQ7DFhFp25qaIJ8FTDazGcC8aN0w4PPAmfkILAmTJ4dbnT4UkfasKtb08cc/wskn1y4fPrx14xERaW1NSpDd/QkzexLYGRhMmKN4AfCau1fnMb5WlUqQv/rVRMMQEUlUKkEeOBBOOikzQdYAPREpdjklyGb2EPBUdGWmajNbAWxJmKc4f9d5TsCKFTBtWpj3eNddk45GRCQZBx8Mc+aE+127JhuLiEhScm1B3otwIRDMbDPgVULr8Toz28fd2+y1lV54oab/sQagiEh7VF0Njz1W87ikJLlYRESSlOs0bz2BRdH98YTLS28G3AZclr+wWp/6H4tIe1dRUftxqiX5179u/VhERJKUa4I8FxgR3T8E+Iu7bwQmAm26Y4L6H4tIe5eeIKdccknN/TfeaJ1YRESSlGsXizuBm8zsCeCrwEmx/bTZidFWrAjze3burOmLRKT9Wb8eysvh8MOzl5tlnxtZRKRY5ZQgu/uVZgbwDeBsd49OwLEz8EmeY2s1zz8fDv677ab+xyLS/my9NSxcmLk+PvexiEh7kvM0b+5+JXBl2urNgfvzElEC1P9YRNqzbMkxwI9+1LpxiIgUiqZeKKSWKGlus9T/WETaqxUrsq9Xciwi7VmDg/TMbOvG7syCoc0LqXUtXx76H3fpAjvvnHQ0IiKta/787Ouvuqp14xARKSSNmcXiZTO7w8x2q+sJZtbXzE4GZgIH5S26VvD88+FW/Y9FpD2aPbv240suCbNZDByYTDwiIoWgMV0sRgO/Bh43s2rgdcJcyGuBvsDnge2A14Az3P3JFoq1Raj/sYi0Z++9V3N/p53g/POTi0VEpFA02ILs7qvc/RfAEOBk4D2gD7A1sBG4C/iSu3+lrSXHoP7HItK+bdoUbseNgylTEg1FRKRgNHqQnrtXAQ9HS1FYtgzeflv9j0Wk/Vq/PtzuuWeY71hERHK/kl5RSfU/3n33cJEQEZH2Zu3acNutzV7qSUQk/3JOkM1sfzN73MxmpmasMLPjzGyf/IfXstT/WETau3Xrwq0aCUREauSUIJvZD4AHgVmEPsilUVEJ8Mv8htbylCCLSHt3/fXhVgmyiEiNXFuQfwkc7+5nEgbopbwCjMlbVK1g6VJ45x3o2lX9j0VEVq1KOgIRkcKRa4I8Eng5y/pyoFfzw2k96n8sIiIiItnkmiAvBEZlWb8XMDvL+oKl7hUi0t5VV9fc798/uThERApNrgnyrcANZvaV6PFQMzsauBL4Y14ja2FKkEWkvduwoeb+kUcmF4eISKFp9DzIAO5+pZn1Bp4CugDPAeuAq939phaIr0WUl8OMGdCpk/ofi0j7lUqQe/QIi4iIBDklyADu/mszu4xwiekOwEx3L897ZC0odWnVUaNCkiwi0h6lLhJSWlr/80RE2pucEmQz+0cd6wFw9+/kIaYWl0qQR49ONg4RkSSlWpCVIIuI1JZrC/LytMelwI7AUOCRvETUCt59N9xut12ycYiIJCnVgqwzaSIiteXaB/mYbOvN7BqgLC8RtQK1IIuIwJo14Xb+/GTjEBEpNDlfaroOtwCn5GlfLU4tyCIicPHFSUcgIlKY8pUgb5un/bS4DRvgww/D/VHZZnQWEWknXnst6QhERApTroP0bkhfBQwC9gfuzFdQLWnOnJAkDx8O3bsnHY2ISHJ6tanrn4qItJ5cB+l9Ie3xJmApcCZtJEFW/2MRkeCQQ+Dtt6FLl6QjEREpLLkO0vtqSwXSWlIJsvofi0h7V1ISbs88M9k4REQKTb76IOfMzE4zs6lmts7MJqaV7WNm75lZpZk9Z2bD8/W6qQF6akEWkfZu+vRwqxZkEZHaGmxBruviINnkeKGQhcClwDeArrHX60+YU/k4YBJwCfAAsGsO+66TWpBFRIKnngq3qZZkEREJGtPFIv3iIHnh7o8AmNlYYMtY0feAGe7+UFQ+AVhmZqPd/b3mvaZakEVEUnr1gpUrYf/9k45ERKSwNJgg13VxkBa0PTAt9voVZjY7Wp+RIJvZCcAJAMOGDat3x4sXh4nx+/aFAQPyGrOISEGq7xiZupLeFlu0dlQiIoUtsT7I9egBrE5btxrome3J7n6ru49197EDGsh64xcIMWt+oCIiha6+Y+S6deFWl5oWEakt12neMLOOwM7AMKDWYdXd/5KHmMqB9Nk5e5GHS1lrijcRkRqpFmQlyCIiteV6oZDRhIFzWxMuElId7WMDsA7IR4I8Azg69prdgRHR+mbRJaZFRGqkEuTOnZONQ0Sk0OTaxeI64HWgN1AJbAeMBd4CxueyIzPraGZdgBKgxMy6RK3TjwI7mNn4qPxCYHpzB+iBWpBFRFLcaxLk0tJkYxERKTS5JshfBi519wrCVfQ6uvsbwC+Ba3Lc1/lAFfAr4Kjo/vnuvpSQbF8GrAR2AY7Icd9ZqQVZRCTYsCHclpZCh0IcjSIikqBc+yAboeUYwiWmhwDvA/OBz+WyI3efAEyoo+xpIK/tvGVlsGBBOJW41Vb53LOISNuzOhoKrf7HIiKZck2Q3wF2BOYArwHnmFk1cDzwYZ5jy6tU94pRozQpvojIrFnhtqIi2ThERApRrgnyZUD36P75wD+B54BlwGF5jCvv1P9YRKRGqovFbrslG4eISCFqVIJsZvu4+zPu/mRqnbvPAT5vZv2Ale7uLRVkPugS0yIiNVID9Hr0SDYOEZFC1NihGU+Z2Rwz+7WZDY4XuPuKQk+OQZeYFhGJ0xzIIiJ1a2yCvD3wCPBT4BMze9zMDjazNtObVy3IIiI1lCCLiNStUQmyu7/r7mcDWwKHAw48BCwwsyvMbNsWjLHZNmyADz4Il5ceNSrpaEREkqcEWUSkbjnNfunuG939EXc/EBgO3AB8D5hpZi+0RID5MGcObNwIw4dDt25JRyMikrzHHw+377yTbBwiIoWoydPDu/tC4GZCkrwK+Eq+gso39T8WEantnnvC7YwZycYhIlKIcp3mDQAz2xc4FjgYWAv8Fbg9j3HllaZ4ExEREZHGanSCbGbDgGOAHxO6V7wAnAA87O5rWyS6PNElpkVEauvWDSor4XM5XQNVRKR9aFQXCzN7inD1vBOB+4FR7j7O3e8p9OQY1IIsIpLu4IPD7UUXJRuHiEghamwLchVhMN7j7l7dgvHknbtakEVE0q1bF247d042DhGRQtSoBNndv5O+zsy+Akx193V5jyqPFi2CsjLo1w/69086GhGRwqAEWUSkbk2exQL4FzAkX4G0lHjrsVmysYiIFIr//S/cKkEWEcnUnAS5TaSb6n8sIpJp5Mhwq4YDEZFMzUmQ2wRdYlpEJNPaaHj15psnG4eISCFqToJ8IvBpvgJpKbpIiIhIplSC3KVLsnGIiBSinC8UYma9gZHAO7SBFmi1IIuIZFqzJtwqQRYRydToBNfMhpnZJGA58CrwJrDMzP5qZgNjzyuYIR9r1sCCBWEQyvDhSUcjIlI4Fi8OtxqkJyKSqVEtyGY2BHgF2ARcCMwkDNL7PHAK8IqZfQnYK1p3RYtEm6OPPw6322wDJSWJhiIiUlA6dw5TvfXpk3QkIiKFp7FdLC4CPgL2dfeq2PpHzexa4D/AP4BdgKPyG2LTLVwYbocU/GR0IiKtx71mHuROnZKNRUSkEDU2QT4A+EFacgyAu1ea2fnAs8DP3P3hfAbYHIsWhdtBg5KNQ0SkkGzcGG47doQOBT+SRESk9TX20DgAmF1P+YdAtbvf2PyQ8ifVgjx4cLJxiIgUErUei4jUr7EJ8hLgc/WUjwQWNz+c/FILsohIpk+jCTorK5ONQ0SkUDU2Qf4XcGm2GSrMrAtwCfBEPgPLB7Ugi4hk+uEPk45ARKSwNbYP8gRgKvChmd0IvAc4sD1hFosS4LCWCLA51IIsIpLp5ZeTjkBEpLA1KkF294VmtjtwM/BbwhRvEJLkfwOnuvvClgmx6VItyEqQRURERKSxGn0lPXf/GDjAzPoS+hwDfODuK1sisOZyr5kIXwmyiEimX/0q6QhERApTzpeajhLi11oglrxasQLWr4fevaFbt6SjEREpDKtW1dwfMya5OEREClnRzoCpAXoiIpkmTKi5379/YmGIiBS0ok2QNUBPRCTT9dfX3N977+TiEBEpZEWbIKsFWUSkfh1z7mQnItI+FG2CrBZkEREREWmKok2Q1YIsIlK3v/wl6QhERApXwSbIZjbZzNaaWXm0vJ/L9mpBFhHJNGpUuN1552TjEBEpZAWbIEdOc/ce0bJtLhuqBVlEJNPGjeG2pCTZOEREClmhJ8hNphZkEZFMc+aEWw3QExGpW6EnyJeb2TIz+5+Zjcv2BDM7wcymmtnUpUuXAuEqerrMtIhI7WPkp58u/2y95kAWEalbISfI5wDbAEOAW4FJZjYi/Unufqu7j3X3sQMGDABg5cpwFb1evaB791aNWUSkoMSPkf36bQaE7hU9eiQcmIhIASvYBNndX3X3Mndf5+53Af8DDmjMtup/LCKSadOmcLvllsnGISJS6Ao2Qc7CAWvME9X/WEQkk3u47dQp2ThERApdQSbIZtbHzL5hZl3MrKOZ/QDYC3iyMdurBVlEJFMqQe7cOdk4REQKXaGOYy4FLgVGA9XAe8DB7t6ouZDVgiwikkktyCIijVOQCbK7LwW+3NTt1YIsIpJpw4Zwu3ZtsnGIiBS6guxi0VxqQRYRyWTRKI6qqmTjEBEpdEWZIKsFWUQkU6qLxfbbJxuHiEihK8oEWS3IIiKZVq4Mt+qDLCJSv6JLkHUVPRGR7JYvr30rIiLZFV2CvGoVrFsHPXvqSlEiItk8/3zSEYiIFLaiS5DV/1hEREREmqPoEmT1PxYRqd/NNycdgYhIYSu6BFktyCIi2aWmeTv22GTjEBEpdEWXIKsFWUQku9Q0byUlycYhIlLoii5B1gwWIiL1U4IsIlK/okuQUy3I6mIhIpJdqquFiIhkV3QJslqQRUTqptZjEZGGFV2CrBZkEZG6KUEWEWlY0SXIakEWEalbh6I76ouI5F9RHSqrq2Ht2nAFvZ49k45GRKTwqAVZRKRhRZUgb9gQbtV6LCKSXUVF0hGIiBS+okyQ1f9YRERERJqqKBNktSCLiIiISFMVVYK8fn24VQuyiEh2AwcmHYGISOErqgRZLcgiIvU788ykIxARKXxFmSCrBVlEJDsdH0VEGlaUCbJakEVEshszJukIREQKX1EmyGohERHJrmPHpCMQESl8RZkgqwVZRCS70tKkIxARKXxFlSBv2gTdu+sqeiIidVGCLCLSsKJKkCG0HpslHYWISGFSgiwi0rCiS5DV/1hEpG7qgywi0rCiS5DV/1hEpG5qQRYRaVjRJchqQRYRqVuvXklHICJS+IouQVYLsohI3dTFQkSkYUWXIKsFWUQkuw5Fd8QXEWkZRXe4VAuyiEh2muFHRKRxii5BVguyiEh2SpBFRBqn6BJktSCLiIiISHMUbIJsZv3M7FEzqzCzT8zsyIa26dBBI7RFROqiFmQRkcYp5PHMNwHrgc2BMcDjZjbN3WfUtUFpqb4ARETqouOjiEjjFGQLspl1B8YDF7h7ubv/F/gH8MP6ttME+CIidVOCLCLSOIXagjwKqHb3WbF104C9059oZicAJ0QP15nZO60QXyHpDyxLOogEtMd6q87tx7b52pGOke32f6g91lt1bj/ydoysS6EmyD2A1WnrVgM905/o7rcCtwKY2VR3H9vy4RWO9lhnaJ/1Vp3bDzObmq996RjZ/uoM7bPeqnP7kc9jZF0KsosFUA6kD7frBZQlEIuIiIiItCOFmiDPAjqa2cjYuh2BOgfoiYiIiIjkQ0EmyO5eATwCXGxm3c3sK8BBwN0NbHpriwdXeNpjnaF91lt1bj9aqt7t8e/ZHusM7bPeqnP70eL1Nndv6ddoEjPrB9wJ7AcsB37l7vclG5WIiIiIFLuCTZBFRERERJJQkF0sRERERESSogRZRERERCSmKBJkM+tnZo+aWYWZfWJmRyYdU76Z2WlmNtXM1pnZxLSyfczsPTOrNLPnzGx4QmHmlZl1NrM7ove0zMzeNLP9Y+XFWu97zGyRma0xs1lmdlysrCjrnGJmI81srZndE1t3ZPQ/UGFmf4/GJxQFM5sc1bc8Wt6PleWt3sVwjGzO8SDa9s7oM7XYzM5K23fBf65y/Ww09J63hc+VmR1hZu9GMc42sz2j9UX5XpvZVmb2hJmtjGK/0cw6RmVjzOz1KO7XzWxMbDszsyvMbHm0XGlWc93M+rZtbdbEXKY572tD29bJ3dv8AvwVeIBwgZE9CBcV2T7puPJcx+8BBwN/BCbG1veP6nso0AW4Cngl6XjzVOfuwARgK8KPuQMJc2FvVeT13h7oHN0fDSwGdirmOsfq/h/gReCe2N+iDNgr+nzfB9yfdJx5rO9k4Lg6/gfyVu9iOEY253gAXB79X/UFtos+U9+MytrE5yrXz0Z973lb+FwRBuh/Auwavd9DoqVo32vgCWBiFNsWwNvAz4BO0d/iTKBztO4ToFO03YnA+8CW0d9oJnBSVFbvtgnUsUm5THPe1/q2rTfWpP8h8vDH7g6sB0bF1t0N/C7p2Fqovpem/VOdALyU9veoAkYnHWsL1X86ML691JtwOc1FwGHFXmfgCOBBQhKUSgJ+C9wXe86I6PPeM+l481TnyWRPkPNW72I+Rjb2eAAsAL4eK7+EKCFsC5+rXD8bDb3nbeFzBbwE/CTL+qJ9r4F3gQNij68CbgG+HtXLYmVzqUkQXwJOiJX9hChBbGjbBOuaUy7TnPe1vm3rW4qhi8UooNrdZ8XWTSP8Qm4PtifUF/hsDunZFGH9zWxzwvs9gyKvt5ndbGaVwHuEBPkJirjOZtYLuBj4eVpRep1nE33xt150Le5yM1tmZv8zs3HRunzWuyiPkY09HphZX2BwvJza9S/oz1UTPxsNvecF/bkysxJgLDDAzD40s/lRd4OuFPF7DVwPHGFm3cxsCLA/8G9CfNM9yu4i06mjXmTWub5tC0WLvK+N2LZOxZAg9yA0rcetJvyKbg/aRf3NrBS4F7jL3d+jyOvt7qcQ6rIn4aI56yjuOl8C3OHu89LWF3OdAc4BtiGcFr0VmGRmI8hvvYvub5jj8aBH7HF6GQ1sWwia8tloqE6FXufNgVLgEMIxcAzwJeB8ivu9fp6QuK0B5gNTgb+T+/u5GugR9UMu9DqntNT72tC2dSqGBLkc6JW2rhehf1V7UPT1N7MOhNOD64HTotVFX293r3b3/xL6lZ1MkdY5GjCyL3BtluKirHOKu7/q7mXuvs7d7wL+BxxAfutdVH/DJhwPymOP08sa2jZRzfhsNFSngq1zpCq6/YO7L3L3ZcDvafiz0Zbf6w7Ak4QGke6EfrV9gSvI/f3sBZRHrcYFW+c0LfW+NrRtnYohQZ4FdDSzkbF1OxJOu7UHMwj1BcDMuhP6kxVF/aNfwHcQWhTGu/uGqKio652mIzV1K8Y6jyMMtJprZouBs4HxZvYGmXXehjDQZFbmboqCA0Z+6100x8imHA/cfSWhm9KOsV3F61/In6txNO2z0dB7XtCfq+g9m0/4PKQr1ve6HzAUuDH6wbwc+DPhR8EM4IvxmSmAL1JHvcisc33bFooWeV8bsW3dkuykncfO3vcTRux2B75CGxyh3Yg6diSMzryc0HrSJVo3IKrv+GjdFRTQqNw81PtPwCtAj7T1RVlvYCBhQE4PoAT4BlABHFTEde5GGLGdWq4GHo7qmzrduGf0+b6HAhtt34x694ne39Rn+QfRe71tvutdLMfIph4PgN8RTl/3JcwMs4iaAU4F+7lqzmejvve8LXyuCP2up0THxL6EWQguKdb3OopvDvCr6HjQB3iU0JUoNRPF6YQfMqdRexaLkwgD/IYQ+tvOIHMWi6zbJlDHJuUyzXlf69u23liT/ofI0x+8H6GfTgVhdOaRScfUAnWcQPg1HV8mRGX7EgZzVRFGxW+VdLx5qvPwqJ5rCadJUssPirXe0Qf9eWAV4QvsbeD4WHnR1TnL32AC0Uj96PGR0ee6AngM6Jd0jHl8r6cQTvWtIiR++7VEvYvhGNmc4wEhMbgz+kx9CpyVtu828bnK5bPR0Hte6J8rQh/km6PPxmLgBqBLMb/XhL7Wk4GVwDLgIWBgVPYl4PUo7jeAL8W2M+BKYEW0XEntWSvq3Dah/+Gcc5nmvK8NbVvXYtHGIiIiIiJCcfRBFhERERHJGyXIIiIiIiIxSpBFRERERGKUIIuIiIiIxChBFhERERGJUYIsIiIiIhKjBFmkmcxsgpm9k3QcIiKFSMdIaYs0D7K0KWY2Eejv7gfG77fSa28FfAR82d2nxtb3ADp7uDSoiEhidIwUyY+OSQcgkjQz6whUexN/Lbp76opeIiJFR8dIaY/UxULaJDObABwNfMvMPFrGRWVDzOx+M1sZLY+b2cj4tmb2jpn92MxmA+uA7mb2TTN7MdpmhZk9aWbbxV72o+h2SvR6k+P7i+2/g5ldYGbzzGydmb1tZgfFyreKth9vZk+ZWaWZzTSz/WLPKTWzG8xsYbSPeWb2u7z/IUWkKOkYKdI8SpClrboaeBB4GhgULS+ZWTfgOWAtsDewG7AIeDoqS9kaOBI4FNgxen534DpgZ2AcsBqYZGadom12jm6/Gb3e9+qI7XTgF8A5wBeAR4FHzGxM2vMuA26IXn8KcH90KhLgZ8B3gSOAkcDhwPsN/1lERAAdI0WaRV0spE1y93IzqwLWufvi1HozOwow4JjU6UAzOxFYAhxI+MIA6AT80N0/je32b/HXMLNjgDWEg/5/gaVR0fL4a2ZxNnC1u98XPb7QzPaK1h8Ve9617j4peq3zgB8BY6LXGg7MAl6M6jEXeKn+v4qISKBjpEjzqAVZis1OhJaPMjMrN7NyQitHX2BE7Hnz0w78mNkIM7vPzGab2RrgU8JnZFhjX9zMegGDgf+lFf0X+Hzauumx+wuj24HR7UTCF8EsM7vJzL5lZvq8ikhz6Rgp0ghqQZZi0wF4i3DaLd2K2P2KLOWTgAXAidHtRmAmoSUlV9kGs6Sv2/BZgbubGUQ/Wt39DQsjwr8JfA24C5hmZvu5+6YmxCMiAjpGijSKEmRpy9YDJWnr3gC+Dyxz91WN3ZGZbQZsB5zq7s9F6/6P2p+R9dFt+mt+xt3XmNlCYA/g2VjRHoQvkkZz9zLgIeAhC9M1vQJ8jnBaUUSkITpGijSREmRpyz4G9jezbYHlhNOE9xL6sT1mZhcS+qUNBQ4C/uTuH9Sxr5XAMuB4M5sHDAGuIrSQpCwBqoBvmNnHwFp3X51lX1cBF5vZB8DrhD51exJObTaKmZ1FGDjzFqEV5UhCX7/5jd2HiLR7H6NjpEiTqL+OtGW3Ae8CUwmDQ77i7pXAXsAcQsvCe4RTb30JB/isolNyhwNfBN4BbgIuIExvlHrORsLI6eMI/eEeq2N3NxC+AK6M9vVdYLy7v5VD3coIo7xfI7T4jAH2j+onItIYOkaKNJGupCciIiIiEqMWZBERERGRGCXIIiIiIiIxSpBFRERERGKUIIuIiIiIxChBFhERERGJUYIsIiIiIhKjBFlEREREJEYJsoiIiIhIzP8D281N+Dou0q8AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "true_Q_value = history1[-1, 0, 0]\n",
        "\n",
        "fig, axes = plt.subplots(1, 2, figsize=(10, 4), sharey=True)\n",
        "axes[0].set_ylabel(\"Q-Value$(s_0, a_0)$\", fontsize=14)\n",
        "axes[0].set_title(\"Q-Value Iteration\", fontsize=14)\n",
        "axes[1].set_title(\"Q-Learning\", fontsize=14)\n",
        "for ax, width, history in zip(axes, (50, 10000), (history1, history2)):\n",
        "    ax.plot([0, width], [true_Q_value, true_Q_value], \"k--\")\n",
        "    ax.plot(np.arange(width), history[:, 0, 0], \"b-\", linewidth=2)\n",
        "    ax.set_xlabel(\"Iterations\", fontsize=14)\n",
        "    ax.axis([0, width, 0, 24])\n",
        "\n",
        "save_fig(\"q_value_plot\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hgzBnxQLMM9f"
      },
      "source": [
        "# 심층 Q-네트워크"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MO8UzaisMM9f"
      },
      "source": [
        "DQN을 만들어 보죠. 상태가 주어지면 가능한 모든 행동에 대해서 행동을 플레이한 후 (하지만 결과를 보기 전에) 기대할 수 있는 할인된 미래 보상의 합을 추정합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F8sJomGpMM9f"
      },
      "outputs": [],
      "source": [
        "keras.backend.clear_session()\n",
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)\n",
        "\n",
        "env = gym.make(\"CartPole-v1\")\n",
        "input_shape = [4] # == env.observation_space.shape\n",
        "n_outputs = 2 # == env.action_space.n\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Dense(32, activation=\"elu\", input_shape=input_shape),\n",
        "    keras.layers.Dense(32, activation=\"elu\"),\n",
        "    keras.layers.Dense(n_outputs)\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cpvvM-jWMM9f"
      },
      "source": [
        "이 DQN을 사용해 행동을 선택하려면 가장 큰 예측 Q-가치를 가진 행동을 선택하면 됩니다. 하지만 에이전트가 환경을 탐험하려면 `epsilon` 확률로 랜덤한 행동을 선택합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VwRuzE5XMM9f"
      },
      "outputs": [],
      "source": [
        "def epsilon_greedy_policy(state, epsilon=0):\n",
        "    if np.random.rand() < epsilon:\n",
        "        return np.random.randint(n_outputs)\n",
        "    else:\n",
        "        Q_values = model.predict(state[np.newaxis])\n",
        "        return np.argmax(Q_values[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5fZ-3GVxMM9g"
      },
      "source": [
        "재생 메모리도 필요합니다. 여기에는 에이전트의 경험이 담겨 있습니다. 형식은 `(obs, action, reward, next_obs, done)`와 같습니다. `deque` 클래스를 사용할 수 있습니다(더 강력한 경험 재생의 구현을 위해 딥마인드의 [Reverb 라이브러리](https://github.com/deepmind/reverb)를 참고하세요):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zWp-e3BOMM9g"
      },
      "outputs": [],
      "source": [
        "from collections import deque\n",
        "\n",
        "replay_memory = deque(maxlen=2000)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ISMBLBd9MM9g"
      },
      "source": [
        "그리고 재생 메모리에서 경험을 샘플링하는 함수를 만듭니다. 이 함수는 5개의 넘파이 배열 `[states, actions, rewards, next_obs, dones]`을 반환합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CqeAa6CkMM9g"
      },
      "outputs": [],
      "source": [
        "def sample_experiences(batch_size):\n",
        "    indices = np.random.randint(len(replay_memory), size=batch_size)\n",
        "    batch = [replay_memory[index] for index in indices]\n",
        "    states, actions, rewards, next_states, dones = [\n",
        "        np.array([experience[field_index] for experience in batch])\n",
        "        for field_index in range(5)]\n",
        "    return states, actions, rewards, next_states, dones"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "djWVPLqCMM9g"
      },
      "source": [
        "이제 DQN을 사용해 한 스텝을 플레이하는 함수를 만들고 경험을 재생 메모리에 기록할 수 있습니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2AgsJA6kMM9g"
      },
      "outputs": [],
      "source": [
        "def play_one_step(env, state, epsilon):\n",
        "    action = epsilon_greedy_policy(state, epsilon)\n",
        "    next_state, reward, done, info = env.step(action)\n",
        "    replay_memory.append((state, action, reward, next_state, done))\n",
        "    return next_state, reward, done, info"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ANVUmkAqMM9h"
      },
      "source": [
        "마지막으로 재생 메모리에서 약간의 경험을 샘플링하고 훈련 스텝을 수행하는 함수를 만들어 보죠:\n",
        "\n",
        "**노트**:\n",
        "* 2판의 처음 세 번의 릴리스에는 `target_Q_values`를 열 벡터로 변환하는 `reshape()` 연산이 빠져있습니다(`loss_fn()`에서 필요합니다).\n",
        "* 이 책은 학습률 1e-3을 사용하지만 아래 코드에서는 훈련이 크게 좋아지기 때문에 1e-2를 사용합니다. 또한 여러 가지 DQN의 학습률을 튜닝했습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IE95b_C3MM9h"
      },
      "outputs": [],
      "source": [
        "batch_size = 32\n",
        "discount_rate = 0.95\n",
        "optimizer = keras.optimizers.Adam(learning_rate=1e-2)\n",
        "loss_fn = keras.losses.mean_squared_error\n",
        "\n",
        "def training_step(batch_size):\n",
        "    experiences = sample_experiences(batch_size)\n",
        "    states, actions, rewards, next_states, dones = experiences\n",
        "    next_Q_values = model.predict(next_states)\n",
        "    max_next_Q_values = np.max(next_Q_values, axis=1)\n",
        "    target_Q_values = (rewards +\n",
        "                       (1 - dones) * discount_rate * max_next_Q_values)\n",
        "    target_Q_values = target_Q_values.reshape(-1, 1)\n",
        "    mask = tf.one_hot(actions, n_outputs)\n",
        "    with tf.GradientTape() as tape:\n",
        "        all_Q_values = model(states)\n",
        "        Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)\n",
        "        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))\n",
        "    grads = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(zip(grads, model.trainable_variables))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HDzA6K8hMM9h"
      },
      "source": [
        "이제 모델을 훈련해 보죠!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OoEEkDHuMM9h"
      },
      "outputs": [],
      "source": [
        "env.seed(42)\n",
        "np.random.seed(42)\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "rewards = [] \n",
        "best_score = 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "XYdtvldOMM9h",
        "outputId": "2a3f5c5a-32d7-4851-f4db-03f7e1014439"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Episode: 599, Steps: 200, eps: 0.010"
          ]
        }
      ],
      "source": [
        "for episode in range(600):\n",
        "    obs = env.reset()    \n",
        "    for step in range(200):\n",
        "        epsilon = max(1 - episode / 500, 0.01)\n",
        "        obs, reward, done, info = play_one_step(env, obs, epsilon)\n",
        "        if done:\n",
        "            break\n",
        "    rewards.append(step) # Not shown in the book\n",
        "    if step >= best_score: # Not shown\n",
        "        best_weights = model.get_weights() # Not shown\n",
        "        best_score = step # Not shown\n",
        "    print(\"\\rEpisode: {}, Steps: {}, eps: {:.3f}\".format(episode, step + 1, epsilon), end=\"\") # Not shown\n",
        "    if episode > 50:\n",
        "        training_step(batch_size)\n",
        "\n",
        "model.set_weights(best_weights)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EOy_doHlMM9i",
        "outputId": "c5b7f261-0654-4677-bba1-8e7ddfdb4bb5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Saving figure dqn_rewards_plot\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAEYCAYAAABRMYxdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAACJEElEQVR4nO2dd5wkVbn3f09Vp4mb88IG0sKSWSSDCIiIXr2CGbOiXrnGq+JrQuUqGK8YL4pg4KKgmEBQQBABQZbswhJ3Fzbn2Ykdqs77R9WpOnXqVHV1z/TM7Ozz/Xxmt7viqTPh/PqJJIQAwzAMwzDMRMIa6wEwDMMwDMOMNCxwGIZhGIaZcLDAYRiGYRhmwsECh2EYhmGYCQcLHIZhGIZhJhy5sR5AK5k+fbpYuHDhWA+DYRiGYZgW8cADD2wVQszQt09ogbNw4UIsX758rIfBMAzDMEyLIKI1pu3somIYhmEYZsLBAodhGIZhmAkHCxyGYRiGYSYcLHAYhmEYhplwsMBhGIZhGGbCMWoCh4iKRHQFEa0hol4ieoiIzlL2n0ZEK4logIhuJ6IFyj4iokuJaJv/9VUiotEaO8MwDMMwuxejacHJAXgBwCkAJgH4LIBriWghEU0HcL2/bSqA5QB+pZx7PoBXAzgMwKEAXgHgvaM2coZhGIZhditGrQ6OEKIfwEXKphuIaBWAowBMA7BCCHEdABDRRQC2EtESIcRKAG8D8A0hxFp//zcAvAfAD0dr/AzDMAzD7D6MWaE/IpoFYH8AKwC8H8Ajcp8Qop+IngWwFMBK//9HlNMf8beZrns+PIsP9t5775aMnWGY4XPej+/DlI4CvvPGI8ZsDP9x9QPoKzv42TtfZNz/wvYBvO3Kf+KX7zkWM7tLAID/99vHcP+q7bAtwiXnHIrD95psPPey257GN295CmcunYU/r9iEf3zqJTj3B//AhWctwYPP78DV9z0fOb5oWzhywRRs2jWEmz98clPP8+sH1uKTv3kU9/2/0/CKy+7C9oEKAOCDL9kXcya14WPXPYJ3n7gIn3nFQQCAEy/9K1571F7oKNq4+MYn8IpD5+DOp7ZgqObCJsLXXnsoVm7oxXdvfwZf/vdD8OyWPvz96S34y0dOAQC89Sf/RHcph1ceNhfv+8UDeO1R8/HVcw8DALz/Fw/gtpWbAQCHzJuE37z/+GCcr/3hPXhkbQ9mdRfxxwtOxCu/exc27Srj8PmTce37jgMA/Oc1D6FnsBr73lzzz+dx0R9WQCjbXnvUfPz3vx+C79/xDL5685N41eFz8deVm1GuuTjjoFm455mt6K84kescufdkrNk2gG39FRy9cAq++KqD8Zrv34MZXUW0F2ys3Nibac7b8jZO2m86/vL4pti+rmION3zwRLzrquV4fvsAXnzAjNhxOYtwxkGz8OcVG+GK2CVSmTuphMntBTy+YVfmc2Z0FrHvzE7847ltxv0E4KT9puPWJzZj35mdOHHf6ThwThc+/4cVmcd3xoGzcOfTWzCjq4hpHQU8+PxOfO9NR+JlB8/OPM6RYEwEDhHlAVwN4KdCiJVE1Algi3ZYD4Au/3Wn/17d10lEJISITLkQ4nIAlwPAsmXLGvxxYRhmtLjrma0AMKYC50+PbUzd//TmXjy3pR/Pbx8IBM7/KcJk5YZdiQLn4Rd2AgD+vMJb0K5bvhbrdg7iw796GEctmILpHQX82+HzAADlmoMr716Nvz3l/Rl0XAHbajzM8Mt/egKOK/D0pj5s3DWEMw6ahQfW7MC/1u3CjoEqAGDF+nAxXLtjEN+69Sm85ghvHDc8ugEA8MYX7YVfP7AWj67twVObvIX+sXU9uOafUVF2pz/e/Wd1QQjg0bXhn+nH1vVg0bQOzOgq4q5ntqJ3qIquUh6DFQf3r96Bye15vLB9EM9u6cML2wcxpT2Pf67ejqGqg1Lexh8fWW98xhXre0AEvOP4RQCAGx9bj3/5z7Rinff/Hx5ZDyE88XGj/0znHDkfM7qKAICnN/UG4mv/WZ149IUerN7aj57BKnoGvXk69YAZOGB2d+p8b+8v49rla3HDoxswvbOIc4+aH+zb0DOI3z+8Hs9u7g8EiH6cEAL/e+dz+P3D61HIWXjnCYtS76eyc6CCX97/ArBtAGcfMgd7TW2ve86qrX3484pNWLdzEAfP68aJ+8a6G+Bn/1iNW5/w5uaZzX14ZnMfzjlyPgq2hTcdsyB2vM7vH16HGx/z5rx3qIbntvTDtggLp9cf30gz6gKHiCwAPwdQAXCBv7kPgP6T1A2gN2F/N4A+XdwwDMOMJJWa9yfGSfjoWkv5SNtfrkXe/2tdT3gtAew9rR0XnrUEAFB1XFx59+rg2M29Q5gzqa3h8br+n0T5/zlHzsfaHYOouQJVx/XuL49Rxq4/xwdP2w+3PbEZPQPV4NmHqlELiErNv7aK4wocOn8STjtwJu56ZivWbBvAwfMmYX3PIABg6dxu3P3MNgxVXf/9JNz1zFZs6BnCoukdKfcSmNSWD+buyY27sKWvDADBM8qVYVZ3Eau3DQAA3nfKYuw3y/vMvLFnCH+95DYcMKsLpxwwA1fevRrlWvQZ3v/iffGiRVMTxwEAq7f249rlawEAC5XvJwA8unYnfv/wemzrL0fO2XtqW+S4n9y9ClVHYGp7IbK9Hq4r8LentmB7fwWXnHMIukr5uufc8+zWQHC/eP+Z+K8zD4gdc+Nj6zGwfTCybXPvEPaZ2ZlpfI+t24kNPUORbR8/8wAsqSMWW8Gopon7mU9XAJgF4BwhRNXftQJeALE8rgPAPv722H7/9QowDMO0kJrrLXpJOiZJ+ADAgOYSUS0n63YOwlISQfO2hc5i+Hlz3Y7oApMVubBX/IU+bxPyNqHmuqHA8cc8qAgW/TnaCzlMastj11A1uOZgJUXguHEh6LgCOZuw0Bcrq7b2AwDW7/SebW/f4iCvu/c07/2TG3fhU9c/mniviuMiZ4VLl21ZkPpKF2rqol/K28Hr2ZNK+NRZS3DhWUtQzNmo1FyUa9HnO2BWF+rRVrCNrwGgw/9+bu2rRLar41CP6yhGt9fDsgiffcVB+OwrDsokbgBgakcheD1FeR0ZXy4+jo09Q5jVVcp0D9P56s/2aDLadXB+AOBAAK8UQqi/wb8FcDARnUNEJQCfA/CoH2AMAD8D8FEimkdEcwF8DMBVozhuhmH2QEKLQBMWnErUgrNuZ/gnb+OuIeiFLia15Y3HNoK0ypSrUuBYyFmEmiNQdaIiRBVgUshJOgo2utvy6BmsGgWRTiBwlHlyhYBFFAiZNds8gSPF295TOyLXXegLnC/d8ASu+ecLyfdyBAo5VeCEz61/P7rbwoVVFyDnn7wPXnzATBT9a/UNed+vgm1h7qQSJrXXFw0RgaMJl85A4EQtOMVcdNntKOQixzfCyw+Zg/OOre82kkxtD0XN1A7z8+nzBACbdg1hVncx0z2K+bis6CqNjcAZtbv6dW3eC6AMYKNSxua9QoiriegcAN8F8AsA9wF4g3L6/wJYDOAx//2P/W0MwzAto1qLL9wqJteMZKCcLAhcIUCIKpwpHflA2NQTOE9t6sWabQM446BZAID7ntsGy6LANSWtETmbkLMtVJ3QgiOPGVAEmGp5KeYs5GwLk9ry2LRrKLAwpFpwnLjbS8YRtRdymNVdxKqtnqto/c5BWATMn+K54KTA2WtKO4jqP3vVcZFT4pNsi4Lvj/796FYsG7oAkRRsb0Hu9V2Kbzt+QeS8NNRrJlpwenWBox9nR45vJarVZmqHWbCYLDC7hmpBDFo99OcDxs6CM5pp4msAJEbNCSFuBWB08PmxNp/wvxiGYUaFah0XVSMWHMCLCdm0qwwhELPgTG4LF5/1dRb5l37rTgDA6kvOBgC8/vJ7AYQLrm7BKdfiLippwbEo+hxyoe0u5fD05ira/YU73YITje/xtoWB0rO7S0GczLqdQ5jVXQpcNWX/uqWCjVldJWzcFY3f0Kk6Anlbd1FJgaNZcLIIHN+i0utbcD542n6ZXT6BhcwVseu3+++39esuKs2CE7ioWr8c520LXaUceodqEWtOZHwGCw4AzMoocPTnA/YcFxXDMMxuQ9UPPHUThExSDI4QIhZkDABHLwyDVvVi7KpLZJsWt5EVaZ0ZkhYcy7Pg1Bw3FjAtLTiFnBV5DiloJrXlI0HGWVxUqqfLdQVs/xnz/hgAYEtfGTO7ioEVRl43b1lGN8hgxcF9Skpz1XGRtxULDoXPpLvapGukmLNgJWSlFQOB44WEFnKNLYtS2OixNZZF6CjYBheV2ZU1WiJAxuFMSXBRlRKeP7OLymTBGSMXFQschmGYBGTcittgDE655hqtPscoWTn6cjtZicFJC15OQ56lWnDyFqHqiEQLTjFnRywfMiZkUlseveVa8IzpLirv2qrAcERowcnZFNxjsFJDeyGHnC9SZBZVziYUDVaWT13/KF5/+b14YftAcA/VgmNZpAic8DksCq0iprgSiW7BKdiNLYvS4mG6R0cxFxOreoyKnO9Gg4ybZYpvuZmaEGScNFdzJmV1UbEFh2EYZtxT10WVEINjst4AwKLpncFr3aAwWbHgJAmqegRBxr4FJ29bnrhQs6hEVOAUclZEmLT7C213Wx5CALv8ujDpaeLSOhRuc1wRWE3ythXM5WDVQVvBDsRPYMGxrZgVBACe2OBVC5Euv2pNBOII8KxUUuBUFaGWs61gsU5yT8nnBzyBU8hZMctaPeS1TffoLOYC15wkFmQ8ii4qAJjWUUApb6G9YL5f0lwtmJacuq8in0+Nk2KBwzAMM86o1qmDk7Rdigf5h32/mZ149KKXQsluji2kU5SYiGYtOIGLSrGKeC6q0ILjai6qYs6KxuD4C1+3b1GSBQIzuagUYea4IljkbEu14HgCR1phpGUob1OiewQIU+CrmgVHDTJ2FKFWsK1U8SGRLpXeoarR+lCPtHt0FHOoaPV1dBHX6QvKzgTBMdLsNbUd86ckF90ziUwAkTlPQ1rh2gt24O4cKxfVmLVqYBiGGe80myYurQ1TOwroK9fQWcqhu5SPZE7pdgIZxDmzqwinOX0TWJoCC47lu6hcN0wTN1hwTDE4MkBXVvbNFGTsX0cIAVcgqPWTs0IRNVR10ZYPLTjSMpSzLKOLSurAQOA4mouKKEwTj1hwKDE+RkW14DQjcKSLyhScq7qdbN/SNNYWnP868wD8x4v3SdxvmqtG3Hby+Yp5GyV43xNTXM5owBYchmGYBOq5qJIsLf1+iriMc5Cpt6rRRrfgnHXwbPzuAydg3pS2REGVFd2C40QsON4xMo294Ft4pEchyKJqiy64+pDUMepp4nJe7MBFRYE7b7DqoC1vI++bs4aq2Sw4kmpNRIKMZRYTEH6/vO1WanyMJEgTH6o1tRDLbKn2BBeVRNY5iqeJj26QcWcxl5rybcqCOuWAeEuHJKRILeYsdJZyY2a9AdiCwzAMk0i9OjjVhBicAcWCA4SBpWr1Yj3UI2dbOHyvybCJGnJRmcSQWgcnbxOqSqsGaW0Jg4wtDFVddLflsXOgGlgd0qwZrisizS5rWpCvnK8wyNiKBCu3FewgjkZahnIJMTjBcyIUMTktyDgQVooFp6BYcLLF4FQxvTNbppBKW50gY8nktjy291diQcado2zBqYc+Vz8870icvH8DAkdacHJedW49dX80YQsOwzBMAvVcVMkWnKjAMVpwEu6pZgXVQwgR66EEINhWsC3PPeQoLippwamGgdA11w0sDDIGx0oJtnWFiIg7XdhIQ0pgwbEIVceF6woM+s00wzTxsK1EmqhSXVSqy8SmMAan6qouKiuTiypIEy/XGk4RB+rH4EhkGQC9kF6zrRpahSrUiIAzl85ODEg2UVLmvKuUH7MAY4AtOAzDMInosSXx/ekuqmlS4Pif2lXJkKQfbKJYPZckHDdB4AQuKitI0ZbBrkElY3+MNVfAcUUQcyMXs7Ru5o4QkWevaQHMcvyyDo4MMpZjbcvbgRVmqJKeRaU3EK05IlrJ2CYMVBy89of3YItSNThvE9oK3j3SXFRS4AiRbrVKopQiouTiXsiFYituwbEjx441qgDrKOQazipTLTj/+ZJ9MWT4+RwtxseMMgzDjENkcbxGY3Cki0qWxi8GFpxwsUiykFgW4CbH80aouSLWJBKIFvqTKdpJdXAc1+tTJWNupCVBHV9nMYc+JfXddaPuoHoWHOmiku6otrwVK/SXs8wWHN39VXVc5HNRCw4A3L96R+Q8VTC1GeJKJKrVpikLToqImtnlubwqNTey8Kscv+90vOekRThk/qSG790K1GDp9hRhmET4nDaOWTxtxMbVDCxwGIZhElD7N/3krlU4ZvHUyP4kC44M8pVun8CCEwkyNt/TaiAGxxUisNZE76/UwQmabUatLIO+i0pacOZMasN5x+6NF/sBpWqtmY6iHRU4QkQCemuBK8+7vh6DIzuaSzETLfSXHoMj50KtdZPXelGZyDeYJq6/zkraPRYqtWNKgQUnelx3KY9Pn31Qw/dtFTLQ+4i9J+PMpbMbPz+waI19BAwLHIZhmASkq8V1Bb54w+Px/QlBxnLxl24H+Uc/4qJKiMKxlaaZ9XASLDjlmgsi71oyy0jG4Kjp2vIaNdfr0H3xqw8Jx0GqwMnB65Ps31eISPCoKvQ895V3bVnoz4sDEkHNm5Kx0B8ZF0VHsdzI//U0cRM5mwJ3W1tKDIlqtRnpOjgLp4f1ZpIsOOMNaYk6+5A5ePdJixs+X7XgjDXje6YZhmHGEOmiSsqWSrLgyMW/U+mFBKRnUUkayaJyXBEIFZVyNRQBMtZFCokgIFfJqnLcaIduAJHeTXp8iHCjbRlUseO4InRRUWjBqTpuYK0xp4lbxkVRTz2vOSKSRaWPO7QaZbPgDN9FldwOQi2oJ59tPCz8aWQJzE4jeM5xYMEZ+xEwDMOMU6QIMIkIICX42IlacKRbIq0OjsSyKDHmx3R/cxaVE7hxpCtIjlUXDI4jIp2/JaoFRxcIcQtOOAZXhC6qSCXjSAyODVumiVeUGBzDoqjG4AghUHFcFBT3md5EU441bxM6ijaWzu3GQXO7Y9eVFIdpwVk6txtLZncF7kgVVSTIZxvvFpy0oOksjKfnZBcVwzBMAnrNGJ1tfWU8t6UPi2d0RrZXXeEXrosuFmmVjCUWJXcv13FcgbKhwnC5FtaKyVvRhcZRspGAMAYnyRKijl/iKm4oeQ11THL8lhJk7LgimMe2ghVacGqe9YiI0mNwHBG8zmmtGlRKeS9eyOvDZeHGD54Uu6aKd28vfqgZC87J+89IrROzeHoH2ov2sIXDaDFnUgmdxRwWz8jWe0pnPD3n2EsshmGYcYos9JfUpuCRtT14yTf+Fttec1y/9UD006xqtEnKwlZ7K9XDSaiDM1R1g2q/arBwMWcFgcBVNxqDY2tCSBUO+qdx19XTxIWyLxQ8UodIa1LvkNf2oaS0anDcsHmm6VN/IMiUOKJ8iotKZjXlrGzLGxEFdXVa4T667WOn4Ib/PGm3icGZ1lnEv75wJo7ce0pT54+n5xz7ETAMw4wBWawklcBFlTFv26fqeIv21I4CchYFfaYyuaiU3kr1SAwyrjrBAq9aO2SciCNCa0gWC46e+RNzUUWK/rlKq4boGHqHvEysNqXQHxBamYwWHCd0qUlRprZq0IOMpYuqkMtev6XQwkVZfp/DHk0Te9ndY4OMiegCIlpORGUiukrZ/mYi6lO+BohIENFR/v6LiKiqHdN4eDfDMIxPFiuJdMMMJrio0s7L2xamdxbxt0+citOWzASQzUXVeBaVuZJxPhdWEZZIl5mjWEMcKXDsFIGjW3BENPBaHYMjRDB+NcgYCC047YUcLIsCK5asa2MSOGEMjotqTQqcZBeVDPrNasHxnk+KotYtiVM6CijY1rgp6Ncq2vI2SnkrqAE1loz2TK8HcDGAMwG0yY1CiKsBXC3fE9HbAXwWwIPKub8SQpw3OsNkGGaikyVTSXdRyViNeqjVdudNDv7UQV1z0yw4mV1UrjBalyqOG1hFVAuOTMN2hQisLmWlKKCKGmSsp2+7rojMX7nmBt2yXTd0Wclb24GLKrTgyLFVamEGV6qLygndYqkCxx+rekw98oGLrHVWh3OOnI9lC6Y21PZgdyRnW7jhP0/EvMnt9Q9uMaNqwRFCXC+E+B2AbXUOfRuAn4nhttRlGGZcMh5+tTMJHCcaZJzWnyl6njAusBELTkqhv4ydGrxCfwml8KVFRnXnSAuJowiUqhN1JwXjiFhw4kHGVcVFpVbqjVhwElxU0k0jrUtyrlQLzntP9oz0ataXbDehWpviAifMosrK1r6yf//WLYmlvI0DZne17PrjiX1ndqW2xxgtxp0zkIgWADgZwM+0Xa8kou1EtIKI3p9y/vm+G2z5li1bWjpWhmGaYxzom0xWEhnzIa0kdkaBU3PdmMsHiAYWJ7uosokv7z7mSsZA6KJRXTVShLguIpWIveOSXVS668YLTA7PrzhucIxq3dGDjGULCzmOsJWD978UGF3FHF60aGrwjPJ/+VpvtqkiF9ZGLDhSrJ118JzM5zDjn/FoK3srgL8LIVYp264FcDmATQCOAfAbItophLhGP1kIcbl/LJYtWzYO/owyDKOTNcakpWNowkVlWQAyhOPoDSElWVo1NBqDkxQAbcqikhaSmutGgoTlfVWiQcDRfa7WbBMIRYsqRKTFS1pwBioO8jYF7rmgGKEVdREJw3gcpZ9WmgWnFLi/sltwrnjbMrQVbOw9bezdKszIMV4FzpfVDUIItUb6PUT0bQDnAogJHIZhxj9ZC9m1kta6qNwEC0L9ZptEww8yBuLiAVBcVAaBogsCdXwyIFie4grEBFJRCWB2g0J/Mh4mLOqnWpRszUUlj/NaTETnz0sTzxBkLLOoGrDgnHbgrMzHMrsP40rgENEJAOYC+HWdQwWSLbwMw4xzxoMFJ4uLSk8Tz+6iimclAZqLaoRaNeiuJknOjlpPACXI2I330YpVMlYbWhJ5sUEijIfRzw/cX0oKutQoduCiciKxMaGw8f7vLObwvlP2wb8dNhc9g9XI9X9+75pAzOVTKhnLDtiNWHCYiclop4nniKgEwAZgE1GJiFSR9TYAvxFC9GrnvYqIppDHiwB8EMDvR2/kDMOMJONB4GQJ5JVWCpkmri+mSVT9Qn860cwp87VkNlIWXCG8CrwGa0UgHpIsOJoFRq94rD6qZ8EJNxhdVL54UgOYpSCUczFYdSLxPHoMDhHhwrOW4KC53TGB8tyWflxy08rIs3nXTgoyHnchpswoM9o/AZ8BMAjgQgDn+a8/AwC+8HkdgJ8aznsDgGcA9MILPr5UCGE6jmGY3YDx4KKqZVA4oYvKC47NaMDxXVSGGBzldZJW8iwl2e5Tc7y2CKZ75SyDBcd3I8lKxKo40C04qhiziCLP7rrx+ZMi6//uex595VrkmnJ8Q1UnKk5kppdBDOrjiT5beLzu6is1EWTMTExG1UUlhLgIwEUJ+4YATE7Y98aWDYphmFFnd7DgCMVKIZttZnV61BLSxLN0E/diXbK3anCEgEWEtrwdaSkhhY1qCZEZRjc+tgEVx0Wb37dJP07HtqJjd7Q0cSCMwfn5vWvw15Wb/fNMQcZx64vp3qYgbYlapXgk0sSZiQlLXIZhRh2Rsc5LK6kXgxOp8+JbcrLKsmpCDE4ki2okXFSul3JvWYSZ3cXIvkLgogr/zHcUvcX/kptWwnFFpO5LmsXEIopYnNQ4G4nagmDdzsHINfMJMThhEHLzFpwkF1UjlYyZiQn/BDAMM+qMBwtOPRFRdeIqLGuPqJrj1s3iSXRRNZAmXnNduELAImBGZ1Tg5Axp4vOnRNOg1QJ+aRYTi7QYnJQgY/08IBQrg5Wa2UVltOAkz596DT0uKqiDMw6aPTJjC/8EMAwz6owHgVNvDCaBo7tlJHpl5pqTkEWlLMZJrRoayaKSlhSLwoaeEj1FGwAmt+UjxxQjFpzk5cC2tBgcEZ+LgqHNgZ7JNajH4EgXVYMxOBEXlTaP0zqKKOSsmOBj9jzGVZo4wzB7BuMhyLi+BSe+f8jQuRvwnkfVM1XXjQT3SrJEhaj1ZsKxmOvq1BwBV3hiaUZXdEEPCv1ZydaOUmYLTvRcNVNKYrLg6M02XQGji6rRGBy1n5MuhKZ1FnDXJ0/F9A4WOHs6bMFhGGbU2R16UZksOEnD1q1BNUfEqv8C0RicpEJ/UkhId9i/1vVgv0/fhDue3Gy8rxACtgUcvtfkyD6Ti0oXDaoFJy3IWE8TFyJef8foojJYaEwuKpM7L82C01FMFjg5izCzq5Q5pZ+ZuLDAYRhm1MnaLXskuOy2p/Heny+Pj6EJgZOE/jg1x2zByZJFJa0eco4eemEnAOAvj2+KHeu48GNwCK86fC6ufvcxsc7YUlDYFqVacNIEhU3RkGhTHR1TJ24pqFSrjd7XCgDmKh3Xg3NTBFdHQR23fh4va4wH/yQwDDPqjKaL6slNvVi5sTe2vZEsqnroFpxqQm2a+mX+QquHFGDSMmJqqllzXTiurFNDOGHf6YFQkRlSUkgdPG9SLF5FzaJKC+q1LIrEDDlucqG/yHlakDEQteBs3DUEAFhg6AGVJLhKeSsiYmJd0Nlww/hwDA7DMKNO1mykkUAIYQworjeGYVtwTIJBqw5sQi7s8pqy+rAp/ke6qEy3khaVmV0lfPXcQ3HakpkxoVfMaMHR08SFiLd6SHMzqaJGFX5rd3jp5Iumd8TOTRJccj6Ce2iiLWuANjPxYYHDMMyoM5ohOI4rjEX9RtJFZYrBMfeiUlxUCdeSQkK3MJX9In5q/JLqolLvDURjYl63bC//2n2Ra0YtOA0U+stowdFbMXivw+Mqfm+pBdPiAidJcOnj1HUQ6xtGwi4qhmFGndFME3eF+X4j6aLSj6y65qwnSnwTIoWEFGBS2MhGk6owc1wXroiKj5obFzgSXXSp1pB6Fhx1vytEIE4kphgc2xBkbLL0TO8sxMeaMB49OFsd13tPXox9ZsTFErNnwhYchmFGnVEVOIaUZm97+nnDtuAYs6hUC066i0q60KSwkTE4jmbBcYQwBizrrhwgLg5UEZSaRaX3ohIiqO4saTPcT08T11//4M1H4slNvcaaQFktOOr7T738wMRnYPY8WOAwDDPqjKYbwRXCeL/6FpwGYnCUQ2UPK3MWlfm1SiBwhOyDJS043v+qMHNc10sTNwiENJeRRBVB5u7nnjvR1tLEf3n/CzGh0VmKLycyzigpyPisQ+bgrEPmxM7zxpOeRh+8z9oBldnjYIHDMMyoM5p1cJwkF1UdE46eBp2GUJxU0kVkrIOjWG2S1mXS0sQDC47/vxux4HjxRaZF3ugySrPg1HFRqafe8eSW2DFdxfhykjMGGWeLjEiy4Ojb01xrzJ4Nx+AwDDPqqNaTVoudpCyqegYa3QWjctkbj8CP3roseK9aiKQwMtZjIfVlwgJO0kXlvZfCRnYKj7ioRLKLylx4TzsmYwyObsExYbLgmLKoTHVwTBCRUXSxwGGywgKHYZhRx424dFp7L1NbAbk9Dd1Fpa7vNhHOOGgWvviqpQCiVhVZ4ddUByebi8ofnxDY3DuEtTsGAAC7BqvevbQgY6FlUUnMhfcs7ZjwfafBAiOvapHZ4qSKC9P5ah0cWZzPNC9JmFLpdSsUCxwmCXZRMQwz6qiCwBUCVqYuTc3fyySi6gU6S0tMIWehUnPRlrcxUPGsKFKESHeSeqmqb3GpF2Sc5KOyAguOwIsu+WuwfddQDUIILYvK74NluFfJGIOjHxOKoCkd8Uwm8oNw9FYNkra8jb5yDYBZ4Khz0FnKob/ipBYUNJ1fiT2DJnA4BodJgC04DMOMOqogaHU0jusmxeCk31m6qKSVQ80SkkJFrrWqm03G4NRrtplcBycaZKyOd7DqRFxUru9+MxkxTBYcXaR0+93F33XiooTReNhaoT+JKpDaCob7KSfJ+cvqogKAdsM12UXFZKUpgUNEbUR0OhEtaPC8C4hoORGViegqZftCIhJE1Kd8fVbZT0R0KRFt87++Sqa8QoZhdgt0C06r72VME8+YRSWFgrqAS6uBjKNRLy/PM7uoyPhaRS7YSW41ocX7OK4wpllnyaJaNK0Dt3zkZHzmbHN6deiiSrDgFJSWCXX+JEsx1IiLqsNgFdKbirLAYZLI5KLyxcg/hRDfJ6ICgH8CWAqgQkT/LoS4KeP91gO4GMCZAOLd1YDJQoiaYfv5AF4N4DB4H/huAfAcgB9mvC/DMOMINxJk3Np7OQkuqnoWHL0icJshIDew4KhZVP55xkJ/9T1UgdXjJd/4W2yfK7RCf/6zmRZ5Y5CxdtOcTdhvVpd5IJExwSii1Dmp1707FDiNWHCiS9TvPnACDpwTHS93DWeSyPqTdiaAe/3X/wagC8BsABf5X5kQQlwvhPgdgG2ZR+jxNgDfEEKsFUKsA/ANAG9v8BoMw4wTVG3RaoHjusJY80ZvNaATWHB8S4hqwZFrfehOUq/rx+DUWciTluVUS4gucFw30UVlKvSnF/OrZ02RQ0lyUbUpAiQtzdwbj+XfM7vA0VPPD99rcsz1Vu++zJ5L1p+0KQA2+69fBuA3QojNAH4J4KARHM8aIlpLRFcS0XRl+1IAjyjvH/G3xSCi83032PItW+K1GhiGGXvEqLqozPeo12yzormojG0NKH4t2eLBVAcn4qJKbEWQPCYBodXBQbKLymDBiWcg1RNhFIzVHGSsdvXOZsExtWpIoqMYF2k6XOiPSSLrT9pGAAcTkQ3PmnOrv70TQHUExrEVwNEAFgA4Cp6F6GplfyeAHuV9D4BOUxyOEOJyIcQyIcSyGTNmjMDQGIYZaSIWnBbfS8at6PV26lUy1l1UqjXBDoKM44trWh2cLGtxmstFd1HJDLHMQcYpbQ5Sx0RkbMUQcVHVi8Hxx5PWEkKn3RCDo8MxOEwSWQXOTwD8CsC/ADgAbvO3HwNg5XAHIYToE0IsF0LUhBCbAFwA4KVE1O0f0gegWzmlG0CfGM1yqAzDjBj6It1K5PV1g009C07VcWFR6MbpUgrZWVoMjvoMFcdLJTdZUCJZVEl1cFKEgtDaTtQcz6KTNQZHv3YjVYX/5w2H47xj945sjwReZ3RRNdKmQ4rKsw+dg0vPOcQ8NrbgMAlk+ukWQnwRwDsBXA7gRCGELE1QA3BpC8YlfwXkT+4KeAHGksP8bQzD7Iaon01E9pZPTSHFhx5UnCVNXLXCyJRqILRWmGJwZK0cY9p0A802TQjEM9DchEJ/JkuQvq2u9UOJwZk7uQ0XnhXNtipFLDjpl5LzIXtrZUFmUR0+fzJef/TexmM4yJhJInOhPyHEbwzbftrIzYgo59/TBmATUQmeSDoKwE4AT8OL97kMwB1CCOmW+hmAjxLRn+D9jn8MwHcauTfDMOOHqIuq9TE43v+6iyr9vGpNoGBbwfmqBScs9Be/9qAUOAaXjqpDktblNIOEnvLuuAKOMGc4mdBdUnWDjLUx6dYSU22gJKQYakjg+KKov2JKrmWYdBIFDhG9NetFhBA/y3joZwB8Xnl/HoAvAHgSwJcBzASwC14a+BuV4/4XwGIAj/nvf+xvYxhmFHFdgd89vA6vOnzesGIfolaIkRhZyr1c6aISxu1J1FwXeZuC8zqLcQuOqZKx7BllymJSRUCiiyo9yjgicGqu8LuJpz5K4rWTrEhJ5+ljNhXiS0LOh+ytlQVpwekvs8BhGifNgvM97X0BQB6A/Om04AUYl+FZWOoihLgIyWnl16ScJwB8wv9iGGaMuHb5C7jw+sewvb+Cd5+0uOnrROvgjE0MjpomLkQ8E6nqu6ikEFIr8IYuqvB8ibRQmFxUgCcShKjfbNOEQFRMuW6yi8qEflzSGNWxAqHA0QWSyUoFAK9bNh93PxOtBvLKQ+fiB3c8i5csmZlprEAocPrqCBzbIrz/lH0yX5fZM0gUOEKIoJoSEZ0NT5h8GMB9/uZjAHwTwJdaNzyGYcYT2we88LstfeVhXSeySI9CoT8gHnOjFxvUNULFd1HJ81V3TmDRMFQyli6q9oTFn+AJlXqF/ky4IlrTp+YKuG52F5UqUFZfcnbd44M0ca3uj6SUIJC+eu5hsW0Hze3OdE+VYxdPAwCcfuCs1OOe/fLLG7ous2eQNQbn6wDeKYT4h7LtbiL6MICrANwwwuNiGGYcElgdhilKIuIi5WJDVQe9QzXM6Co2fy/f5hxLE1ctOIbzAheVf77aJDJmwVGuMFj1Tki24HgmnCRRkhpkbEgT1wv97TW1DS9sHzSeL48796j5ifcwnxd9XokUcW980V7BtlMPGLnyHPvO7MSqr7w8s4BjGJWsAmchgH7D9gEA5tB2hmEmHHKBK9dcDFUdY5xJFrJWMn7LFffh/tU7Gv7kH71X/SwqT/wkuKj889UA3TAmxbfgKGElMgbHlKYNeHPoxO4W3a/y1XMPheMKfOr6x2JZVI4bTxO/9aOnJM4pEeFfXzgz0bWUhP68klLejlxvxRfOTHzuZmFxwzRL1p/E+wBcRkTz5Ab/9bcQtnBgGGaCIz/JX3XPaiz57M1NXydrHZz7V+9o+h769XVXmGzFAJgtOJWaQF4VOIqLKihkbMiiGqo6aMvbiQuz7vbR0d1AU9sLgbhy3WgWVdVx4YroOcWcnSo8O4u5zAHieksKnULOilyvo5ir26KCYUaLrD+J7wYwDcBqIlpNRKsBrIaX9fSe1gyNYZjxxkh9mBZa/EsrkTpGF1IRgWMYQ811UbApEEbqwh0W+otPyGDFSQ/eDcRRNhdVMW9FjlWfY1tfBa4rRuz7oiMvmxQXxFWEmfFMVoGzHsCRAM6GF1j8LQAvB3CIEOKZFo2NYZhxxki5C1RrSqsrGYvAgqMLnPQ4ID2LSu0tZWsxKa/4zl34v/ueB+C5qNJcQLr1J75fEzg5W8nWCt1hcyaVsKl3qKEsqmbhasHM7khdgeP3n+oBsJ8Q4i9CiMuEEN8WQtzCrRIYZs9ipJY5PYOplSRlUVXqWHCqNRGpg6NacEx1Yb55y1MAPIFTyif/aZUuqiSxqIuVQs6KuMLk88yZVMKmXUNwElo1jCR1enIyzLik7o+tEMIBsAZeHRyGYfZgRuqDfKMCp15Rvizn6vep1ik4V3VdPwbHe68GGZPRzeQdWM9FFZybsD/mospZgSgSCJ9nzuQ2DFVd9AxUW+eiSmkqCrRenDLMcMiqy78E4BIimt7KwTAMM74ZqXVUNOiiqtf5Ow0pUHQLTr0YnKrjCRx5Xs6mQOTYhhgceY3BSj0XlbkqsESP0VUtOEJp1TB3UgkAsGuo1jIXlbxqkouq1W02GGY4ZE0T/y8AiwCsI6K10FLGhRCHjvTAGIYZf4xUY8NoHZz6OK5AkxnpShZVYzE4Q1UXxZwVxPDkLAvFnIVaxQkWfHU25PUHq06kMadOELib0UVVzIVBxq4I7zN7UltwTKtjZNiCw+yOZBU4v27pKBiGaZgNPYO47YnNOO/YBaN2z5GLwVFf118lhxOILC0e+jXqxeBIV5NaybiUt9FfcYKJMC38Q1UHs7pTChPWcVGZgowVO1HwHNKCA7QwRoa0/xWOXTwVpx+UXmGYYcaSTAJHCPGFVg+EYZjGeMeV92Plxl68dOkszOwq1T9hBBi5LKrGYnBqw4jBkddvtA7OYNVBe8EO0sxzthUUsZPWH9WgJZTzhueiigcZW4oFR45nulLdueXF8AwT9Mvzj2vtPRlmmHBsPMPspmzv9/pCudmbMw+bEQsyjlURzn58oziJLirVghO//kClhra8rbioCP915gEAgGkdXs5FpD6N22iQcYKLyhRkrKSJy+cpqHV5WqRvLjxrCQCgvdikf5BhxpBMFhwiKgD4NIA3wmvNEHEwCyH4p59hxojRLFGStCg3SqRVQ4bj9QDhxu5lThOv1pLjgFxXYKjqoq2Qi1Qyfs2R8/GaI8M+TpRgwUmrJFzXgqOnidtWMOtPb+7Fdr/RqdrdvFUxOG8+ZgHefMzouUAZZiTJGoPzJQCvB/AVeEX+Pg6vP9UbAHy2JSNjGCaVsYjvbEWaeKYsqiYFjhAicFHpt0mLwRmq+R3BC3aYRWUIdLE0hVNzXPSXa+gsJv9pDcJaEoOMtfcWBcde8H8PBdtVgcP9mhgmTlYX1esAvE8I8b/w+sT9XgjxQQCfB3BGqwbHMEx9RnNpGylXiOoSyuJiazZNXNVFsUJ/ah0c7fIDlVDgyFvn7fjD6zE463YOwhXAXlPbE8dUrw6OFCuT2vJY+aWXRc5RUeN8Wl3JWGfe5Lb6BzHMGJPVgjMLwOP+6z4Ak/3XNwO4dITHxDBMBsYiRbc1LqrWWXDSmnpGg4yj+wZ9gVPKh1lUpiaSEQOOEFi11augsWh6R+KYguJ5CR8v5TgLOStwdemz3l6wMbu7hLxNqDoiVjunlTz8uTMi1iOGGa9k/Sl9HsBc//UzAM70Xx8HYDDrzYjoAiJaTkRlIrpK2X4sEd1CRNuJaAsRXUdEc5T9FxFRlYj6lK/FWe/LMMzIMFaVjJsVOGmusLRCf4PV0ILjKkHGOqprSABYs20AALBgWooFJ/jfPJl5X60cNn9SsE230Ow9tR2WRcaCg61mcnsB7YWsn40ZZuzIKnB+C+A0//W3AXyBiFYBuArAjxu433oAFwP4ibZ9CoDL4cX1LADQC+BK7ZhfCSE6la/nGrgvwzAJnH3Z3/HRax/OdGwrmm2OnsCJ7osW+ouiuqhMrRokeiXjVVv70VGwMaMzuQ5O2ObBvH9qRwG/ft9x+PYbjoidI5GWnbxvBuIYHIaJk7UOzqeU178mohcAnADgKSHEDVlvJoS4HgCIaBmA+cr2m9TjiOi7AP6W9boMs2cilH+bY9OuIaxYvwsr1u/CN193eN3js8TgvO/nD6CzlMPXX3tY4jGiwSDjZgv9qcIovdlmdN9ApQbAExJu0KrB4KJSXgsIrNnWj72ndaQKjjCLKvmYZQunRu+jHSpdYLYd7WrOMExIU45UIcR9QohvNiJuGuRkACu0ba/0XVgriOj9SScS0fm+G2z5li1bWjQ8hhk/DCcW58ZHNwAAZnalVN5VyGIouHnFRvz6gbWpx6TFxpiPr39fE2kVk6uOG9aX0c4bClxUYZq4Ocg4asHpGawGNXKSSCkOnHyOcp+8Tbjo35YCCDO7Wt1NnGF2RzIJHCJ6ioj+l4jeqMbGtAIiOhTA5+ClokuuBXAggBkA3gPgc0T0RtP5QojLhRDLhBDLZsyY0cqhMsyYElbobV7hbNw1BABYmBIUa7rncGm0Dk6tyWqGaoFA/RLVmhsUy9Ofy+yiqhNk7J+XVgPHOye9Do7xHOX1O05YhEl+ryvpNmMXFcPEyWrB+RqADgBfBbCWiJ5sheAhon0B3ATgQ0KIv8vtQojHhRDrhRCOEOIeeHFA547UfRlmd2Y4mkMKgKyVgodRb0+7TqOVjId/H1OzTZkNpGdRSYGjpmKbg4yVN8Kz/LSnVDFWz2kkMFgVMOpZYZBx5ksxzB5D1hicHwH4ERCIkBfDq3/zU3giadgh9US0AMCtAL4khPh5vSFhdMt/MMy4Qy7JrWhjkHjPETLhiEazqJqNwVHOczRRVXFcdLfl0AvEVKJME28r2Pi/9xyD3zywzthJXRUprhB1+1AB9evgmFBvHRE7TYglhtlTyCxMiMgCcDQ8cfMSeEHG6wDc0cA1cv49bQA2EZUA1ODV2fkrgO8JIX5oOO9VAO4EsNMfwwcB/L+s92UYxkxSI8p6xw+XaGxM/eOdJk046nhVUSWbdwYuKu08NU38+H2m4/h9phuvrwuLgTp9qIAwPbwxF1V4sCp25P05Bodh4mTtRXUjgBMBbIOX3XQNgPOFEGsavN9n4FU/lpwH4Avw/r4sBvB5Igr2CyE6/ZdvgJdaXgSwFsClQoifNnhvhplQiAatLyZksG9Wy0yWonxZyNKqQR1Ts0HG0SyqcLusgRO4qBJicEq5bNYYwPsjNlStL3BCLdKIi8r82rYaF0sMs6eQ1YJzBjzryU0AbgdwhxBia6M3E0JcBOCihN1fSDnPGFDMMMzwrCpuIJKyHt/8vZKukzR+tU7NSBf6k402k2JwBis1lPKW0S2lou52XAEHyOCiajxuhgxWG3U7u6gYJk7WIONJAN4EYAeAD8MLNH6MiC4jon9v1eAYhqnPcCw4Ujdkj8Fp+lbadeoHGauZU00LHFd9HV6jUseCM1R16woVwJy9VDfIOOXc5HNUUaO6q3wXFQschomRNch4EF4A8K1AEGj8aQDvB/ABeDE1DMOMASOSRZXZgjPyLqqkK0YsOE032zRXMg5cVAkxOFXHDVompGGSFfXTxJPPrXeOfp60ArG+YZg4WWNwZsILLj7V/39/AJsB/Aaey4phmBHEcQWEEMbquRK5KA8ns0kKgOwxOOlEOnSnoMbDJImmmnJQs5liSVlUcpyhBSeeQp5F4JhcQ5ldVA2UWbUMVhv1NbuoGCZO1hicjf7XnfBq0NwhhFjZslExzB7O2Zf9HSs39mL1JWfXPXZ4MTje/1ldQLoQEEIEC/ZNj23A+69+sOHrJI2/poyp1qTASXKFhUHGtnEMNddFzlC5WMckLDK7qEYgyNhqIp6HYfYUsgqcg1jQMMzosXJjb91jGk3xNhEGGTcXg+MKQOqAW57Y1PB90+6tdvtuNgZHtRQ5hhicYs5sRqk5wljYT8dkOCnVzaJq3EdlcksBoRWI08QZJk4mI6kUN0S0jIheT0Qd/vsOv7YNwzBjxHBSt0MXVWPHm97r1ow0UZIli6rm1BdB9UiOwdGyqLTLZ47BMeiK9owxOE1XMja4qLhVA8PEyRqDMwvAH+AV2RMA9gPwHIBvAhgC8KFWDZBhGDNBHZwma8QAw8+iUs/TM3mqjgvbMi/2qvhJEmhqFlWzLqpIU0/ltbQOFW1zmnjNFU27qOrVwZE0HWRsqGrMMTgMEydrmNu34MXgTAMwoGy/DsBLR3pQDMNkZ1gWnGFmUcm3X//zk/j9I+si+yop1fkilpWEw9QsqmaDjNXhRuvgpKeJVx3X2FxTp5kg49DqUvfyxvtEX3v/ZzA2McweR1b30mkAThNC7NBMoc8C2HvER8UwTF3CLKrmr9FoDE7S+d+9/ZnYvmpKRlXUglP/mKZjcBJcVNIiJK00+tVrjkA+gwXHJFLqtmpoxkWlvDa1amAXFcPEyar72wBUDNtnwHNRMQwz2vir8kgInKzXMAUZJ6FaYJLuq7+Onq8EGY9ADI5juKd0qwkhcN9z2/CHR9YD8LOoMlhwjAKn5XVwwjc2u6gYJpGsAudOAG9X3gsisgF8EsBtIz0ohmGyMxwXldQQWS04aUHGOtUUF1XEgpNYyXj4Fhw34T6BwLHCQn+vv/xefPCahwB44qxVMTjNZFGZ2jOorzmJimHiZHVRfQLA34joaHgNL78BYCm8Fg4ntGhsDMNkYDhp4o027NSPEikBzukCRx1D/fOb70Wl3jMe9yNjV0x1cJquZFyvQaf/f7NWF1M8DqeJM0ycrGnijwM4BMA9AP4CoAQvwPgIIcSzrRsewzD1GE4lY+m2ydqtuzELTlYXlfmYkUgTj4gaQ8CxbKZ519NbYvfOUgdHFylZGnRKs8tIZFFJLxrH4DBMnLoWHCLKA7gLwFuFEJ9v/ZAYhsmCXK+HV+jPv9YIpInrZHZRZUkTTxFLaUTcUm5cMEkRc9EfH4+cl7UOji5w2gv1jeJh/6jsoqR+q4bMl2KYPYa6v8FCiCqARRheTz+GYVpG87+aDbuoYhac5GPT0sSTsptUqiNhwUkIZpb3TOrCnTUGRzfDZOpALv9vpJIx18FhmIbJGmT8UwDvaeVAGIZpjFCcNH8Np8E6OLrOSLP8pKWJu64I4kYSg4zVbuIjEYMj4tezEzKlapnr4ETfl/JZMq8at7qomVPRSsbyfxY4DKOTNci4A8CbiegMAA8A6Fd3CiE+ONIDYxgmG6NZB0fXGc2miTuuF+PidU03H1NxnOB1s5WMo1lUyvYgi8p8XtXNVgdnOC6qRqJworVvwtfSAsX6hmHiZLXgHAjgQQA7ACyGF3Asvw7OejMiuoCIlhNRmYiu0vadRkQriWiAiG4nogXKPiKiS4lom//1VeKoOmYPJ4zBGUYl4wZr6ejxMmkp6mkxOK4QQYxL0vjL1fD8ZisZuwarDRA+b1JAcK3JXlTZXFSNi5KkOjj8Z5BhkslkwRFCnDpC91sP4GIAZ8IrHggAIKLpAK4H8G4AfwTwJQC/AnCsf8j5AF4N4DB4f9dvgdcL64cjNC6G2W0ZlgXHbZ0FJzUGJ+KiMh9Trg2/0J9jCCxWtydlStWarIOTqQ9VU26luFtKfT2cnwGGmaiMagcTIcT1QojfAdim7XoNgBVCiOuEEEMALgJwGBEt8fe/DcA3hBBrhRDr4NXhefvojJphxicisL4Mx4LTmMDRV9I0y0pa5pMjELiAEi04tdBF1XwMTnoWVZLIqGatg9OEBSfIoqp7ZPwc73U8i2o4VjyGmaiMlxZtSwE8It8IIfrh9blaatrvv14KA0R0vu8GW75lyxbTIQwzoRjO0hZ0E89cB0e7d2oMTsYg44RjVBfVSAQZq6/luJMK5DVbByeLBac5F1XERxUgx88Ch2HijBeB0wmgR9vWA6ArYX8PgE5THI4Q4nIhxDIhxLIZM2a0ZLAMM54YXgxOo5WMsxf6q+eikllKSRaocs2FbREKOWtEXFSRLCqtDo6KEAI1VyDXRCXjTAJn2M02420bhpNJxzATlaxZVK2mD0C3tq0bQG/C/m4AfWI4tnmGmSCMbRZV84X+coGLynxMueagmPNEhtNkob+kpp56JWMVmbGVb8aCk8lF1XhgcETUGLbzn0KGiZP4EYWI/kpEk/3XbyWiYgvHsQJeALG8dweAffztsf3+6xVgmD0YaU0ZlgUnaLaZ8Z6NpImn1MFxROgCSgsyLuYs2EQj0k08EoMj6+AYxIaMHcpkwdFOb2/AgtNsFpVankdqsGZdeAwzkUn7DT4BQLv/+kp4jTWHBRHliKgEwAZgE1GJiHIAfgvgYCI6x9//OQCPCiFW+qf+DMBHiWgeEc0F8DEAVw13PAwz3kn7ZB4EGQ/j+qoAyGIF0I9JLfRXpw5OljTxQs7r7dR0mriisUzxOKYYnKp/UpY6OLqXvJTBgiMZyWabrG8YJk6ai2olgC8T0e3wrKKvI6JdpgOFED/LeL/PAPi88v48AF8QQlxEROcA+C6AXwC4D8AblOP+F179ncf89z/2tzHMhMYVQL11diSyqDLfK3Z+8rFpMTiuCF1UiUHGNQfFnI2aI5qPwUmogxMW+kux4DTR4KkRF1Uj+iapXg+xi4phEkkTOO8H8G0Ar4L3N+gSmP8WCXgWlroIIS6ClwJu2ncrgCUJ+wSAT/hfDDOh0dOZ7YSEYnnU8GJw1NfJ9wru2VA38fQYHGnBSQsyLuYsDFapaReMSLBQha0awuc9ab/puPe5baj5487iotJpyEXVQKJ4UpCxHD5nUTFMnESBI4S4B8DRAEBELoDFQojNozUwhtlTSWoQmcRIFPrLei9TkHGSQElzKzmuQFs+QwxO3oI91LzAUTWWExE73v+qwLH91hFVGWScpdmmRrY0cf//ZmNwKDpmgF1UDGMi60eURQC4qAzDjAIRV0pafRpZw2aEXFRZLhNvtpl8XppbSXVRpRX6K+ZsX3jUH1vSfcLX8e1qkHHOsuAKhBacDM02dRpxUTXiATOlhgPAUQumAAD2ndmZ/WIMs4eQtVXDGiKaRUQfAHAQvD+tjwP4vhBiUysHyDB7GrUGrSrD+fCeFKOShD4eV4jEMaaJkmiQsfmYisyisghO1kqEhmsAnjVGtSjJ51YtONJiUw1cVC2y4DTRbDPqogpfn3vUfBy7eBr2mtoeO4dh9nQyfUQhohMAPAPgTQAGAQwBeDOAp4nouNYNj2H2PNSaL9lcVMNPE896r9j5wixQiNKv5woovajSY3Bsi9BkGRys2zmIUt7C1I6C0Vqlu6jkfQFkatWgM6ktn+GoJjqAk/kNEbG4YZgEshb6+zqAawC8TwjhAgARWfCaXX4DwPGtGR7D7Hk4CW6VJIYTgyMavJcuWkSCBaeYs9J7UUUqGZuPKVfdwEXVbJr46q39WDitA71DtaiLyo0X+stpAqfRLKor33E0DplXv5qGvGwjaeKmwGKGYdLJ+hHlcHjNLoPPe/7rbwI4ogXjYpg9lppiVkmtgxMU+mv+XtHA28YDmt2EGJyCbeG3D63Fi/77VqPryxM4Mk08JQYn7xX6qzXpolq9zRM4RDC6qHIRC47357DSpAXn1ANmxurimAizqLKTlEXFMEwyWX+De+AFGussArBzxEbDMHsQ1y1/Af913SOx7dF6LfWvkyQQspDUjDL5+GwWHNsi7BioYnNvGUNVJ7Y/GmRsvpd0UVlNBhk7rsAL2wexYHq7ZwUyWKssiltwmhU4WRlus03WNwyTjay/wb8EcAURvZmIFhHRQiI6D8CP4LmuGIZpkI//+lH8+oG1se2RBpEpqkOu18Ox4ERdVI0HNLvCHORs18lAylTJuOa5qHJNBhmv3zmIiuNi0bQOWBSN43FdAYuilhHbjgqcZoKMsyCnpjEXlfqaFQ7DZCFrDM4n4P0t+IlyThXADwBc2IJxMcweiypqmmmf0Oi9ZP2XZgKaTVlUHz1jf1x935rwHMN1XCGUIGPzvcpVr9lm3qZIZllWegarAICpHQVYWtCzK4QnFBStEFhwnOytGpqhkQJ/pnNY3zBMNjJZcIQQFSHEhwBMgRePcwSAqUKIjwghKi0cH8PscdQyuqhGqpJxPaEhuf3Jzbj7mW3a+QJCMa4cPK8bHzxtv0h9maQYHCkg0rKoCjkLxZyNcrVxC07VCV1NFpFW1NALMFaFg625qJqpg5OFZpptqpooS5wPwzDZLTgAACHEAMJ+UAzDtABTz6Q0hheD4wX7VlC/Ds47rrw/fm9hHqOanWQSMGoWlSm+pua4qLkCxZyNYt7Cjv7GP0dJoZizyRCD47uoTBacFruoKCj016yLaqRHxDATk9Z8RGEYpmmyChwpHJpMMPLPDbOZUuvWJIgf3UVlqi9jEk6uCIN4TfE10k1UzFso5iwMDcOCk7MsEEUDlV1XwKaos0jGDZWdVgcZ+/83G2TchIuLYfZEWOAwzDgjGoNT//jhVDJWhUbavTbsGko836R9VIFj2u/F/iAxvka6pIo5C6W8jXItnolVj6AruE2wrXjNH4soIhxiFpwWmUrkZZtvtjmy42GYiQoLHIYZA9ICg9UFv5n2CY2gBvumXWfN1v7E803PosbgmK7rCAHL8lxHJoEjLTheDI4VFN9rBDl3OYv8LKqoZYw0F5WcB5nWXsi1KgZnuL2oWOEwTBZY4DDMGDBQCS0SukBQXTapLqrYi8ZxRf2eUACwetuAeQxCRM4zuahMzyBdRHnLXPFYrUVTzNlNCZxYkHGk5o8n7FSpEFQylgKnxS6qRjxNqqZhfcMw2cgcZExEBQAHA5gJTRgJIf40wuNimAlN71AteO0KQI1njcSKjEKhvywWnLU7QoEzb3IbLn71wXjHVffDdROCjOtlUfkCw7bNVYqlOCnYMganCReVEmRsaZWMZZp4xILjfxOGaqH1qBVIC0xDLiqug8MwDZNJ4BDRGQB+Dk/c6AgA9VvoMgwT0DtUDV7LWjSSzK0aRqDQn1pROO1eqgUlbxNmdBWD800CR81A0ncLISD8GJhcgotKbsvbMgZneEHGehaV40qhoXQTl0HGvphqWZBxE2niqhjiGByGyUbW3+DvAbgBXmuGdgBtyteItLIloj7tyyGi7/j7FhKR0PZ/diTuyzBjwa6IBUd3UTXYqqFJgSOFRphFlXysOkYiCqwIei8q+TLNgiPf2xYhZ1mR7ukSNVW7mLPguAK1Bvs1BEHGFvlZVNECiraVFIPTYguO/38jlhh2UTFM42R1Uc0B8GUhxJq6RzaJEKJTviaiDgCbAFynHTZZCFEDw+zm6BYclUbr4DQbZCxvI9Ojs6aJE4XtBoDkXlRJ45PBvrYfZFw1uKikBadgWyjmfctKzUWuAauKo7iobE3gOK7volKOl1YnmbHVuiwq6aJq/ByAg4wZJitZ/1rcAOD4Vg5E41wAmwH8fRTvyTCjhhqD46RYcLJkUTXroXICNxDVvZc6RgISLTiSaBZVdJ/UMxYR8jYZ7xu4l2xCKe95wBuNw5HCKW9bsCxoWVTxNHHVglPIWS0TEs25qMyvGYZJJqsF530AriaiowD8C14fqgAhxM9GeFxvA/AzEQ8KWENEAsAtAD4uhNiqn0hE5wM4HwD23nvvER4Ww4wMakyJXkSvFnGl1L9Ws72opGVFWirSLqN6hyyiIA4kKQbHZMH584qNeHpTL95+wiL/GO84UxZVNZJFFVpwGkF1UXlZVFEXlWXBnEVVc1qWQQWoAoeDjBmmlWQVOGcCOA3AywEMIPqhUQAYMYFDRHsDOAXAu5TNWwEcDeBhANPgxQRd7Y8rghDicgCXA8CyZcuGUwONYVqGmgo+XBdV8zE43v+5DC4qEYnBCRfnbIX+vANuemwD7l+9A285biEAacGxzFlUSpBxMedZcBoVOKEVKN6LyjFlUfnzIC04rSLIomqykjELHIbJRtbf4q8D+C6ALiFEpxCiS/nqHuExvRXAXUKIVXKDEKJPCLFcCFETQmwCcAGAlxLRSN+bYUYF1SKS5qJqZQyOvK+MPUnzhqljspQgY6EV+pOv1V5U8lxHeK9dJci4vgWHFAtOYy6qIE3c8tPEI3Vw4LdqiFcyLteclnUSB5RWDc2ez/qGYTKRVeBMBvBDIYS5nOnI8lYAP61zjPxTxb/qzG5JpJifq+8bnSwqVwn2Vd+b0EVY1EUVP17VB2E6u4Djf8n75mwrIU08dFGFMTiNWXAiQcYWxYRjciXjVltwvP+btcSwwGGYbGR1Uf0GwOkAnm3hWEBExwOYBy17ioiOAbATwNMApgC4DMAdQoieVo6HYVpFTXOXJO3LEl/TrAVH+HohjMHJlkWlWnCSCv3ZYZpVICxc33ojrxXWwTE12wwDoAMLTqNBxrKSsd9sM9JN3M+iUolacLILnOMWT8PC6dmrZVhNuKhM5zMMk05WgfMcgP8mopMBPIp4kPE3R2g8bwNwvRCiV9u+GMCX4RUa3AUvyPiNI3RPhhl1ItYEzYIRiRUZThW/emMIXFT1WzWo+1TLR3KQsXqu76Jy4xacTEHG+eaDjC3y3GW2LnBkqwY1eFex4HSVMhd5xzXnH9vQuEIXFVtwGKaVZP0tfieAXnip4nq6uAAwIgJHCPHehO3XALhmJO7BMOOBtFTwWoMuqubr4ESzqHShpaJamdT0apGUJm4IMnaF8ESOjMHx08TLBteTtOrkhhNk7IZ1cywrOpeyknG0QrC04LiYNs6CjFXYgsMw2cgkcIQQi1o9EIYZr/QMVFHMh7EgI4EqGOJBxtlaNYTHNDcGV7Pg6OOIHKsX+qtjwVEXYXmqE7io/GMsgm1ZqLpx15Pqoir5Fpw12xoLAaw5IhBvehaVEJ51J5p+7f1fro5WmniT54/cUBhmQsPdxBmmDod98S949ffuHtFrqu0J0urgZLPgNDcGNxaDk3ysExE40VYN9bqJR7KoIi4qIG9RRNBJapFmm56wvPjGJ3Dfc9syP5/jagKnnouKZLPNxmJwGkVajZp3UbHEYZgsZG22eVnafiHEB0dmOAwzPlm5UQ8LGx5pIqbhOjhN1jKOuajSLDhqDA6yxOAYXFS+9UY+n0UpMTiO6qIKxcbWvkqGJwuvIYWK12wz3OeIuItKPlPVEaOSRdW8i2rkxsIwE5msMTiHaO/zAJb45z84oiNimD2AaGfr6AKvtiRoZaE/3UWVvdkmEuvgLJ7RAUBr1eAbaGSQsZqenk9IE6+qWVSKa9BuYHWvOWGndKLoPAshYCekiXv3bZ3AkbdpPk2cFQ7DZCFrDM6p+jYiKgG4AtwvimEappZipRnMIHBMxfUaRXdRpdbBSUoTV4KM33XiInzkjP0BmC04UtyoQcaeBcdQyVhJ8UY+3N5Isb+q6wZVmm2iyDy5It5sUxUcxdEIMm7yfLbgMEw2mv4tFkIMAfhvAJ8eueEwzJ5BWhbVYCW5CKBE1SJf/8tTuPLuVQ2PQS/0l1oHJ9ZsU44jLPT3kiUz0Vn0PjNZCS4qIRQXlSXr4JhdVLZFsCxCMWfjwrOWAGis4abjhhYciyga2O3GWzWor0elkjFnUTFMSxnux5QZADpHYiB7Aq4r8P5fPIAH1mwf66HsUWzuHcLbr/wndg5kj9+QtKoOTUTgNGPB0d5/4Y+P44+PrMelN6/MPgYRuoEA4P7VO/CBqx80pour41DTxL0gY2+fuu7mEiw4APCeny0H4FlVcrY5BqfmiIjIeO1R8wE0Vs04kkWlxeC4wksdhyFNHMAo9aJiocIwrSTTbzERfVT7+hgRfR3ALwH8qbVDnDhs7S/jpn9txHt/zmFLI0HvUBUX/uZR9A5VU4/74R3P4Y4nt+C65WsbvkfV4D4ZCVSBs6O/ggt/8ygGK56wicbgmM83CZ//vOYh/OCO7MXGRWDB8f4M/OCOZ3HjYxuw3SAEHS3K2JQmrgoEyxCDI4XThp4h/75emrjJglNxXM895SNT9AcbsOCoQcYWwZAmbs6iAlobg6O3iGgUi31UDJOJrEHG/6m9dwFsAXAlgK+M6IgmMM2mhTJmfnrPavzy/hcwe1IJHz59/8TjdvkCaFJbPvGYJCqjIHC+9ucnsXJjL5bOm4S3HLsgEDpAstuo2eJ+0Wt4/+vuGJOo011lYZBx6C5TBUIkTVyz4ATXsLxCf+Y0cYF8Li5wGnFR1RQXlZdFFXVRFXN6DE74uqUWHAyvlg3/FWGYbHChvzGhNW6PPQ35SbZehdueQU/gdLdlL78vqTZYPTcrqtVCWiWkO2Ww6qC9YGOg4iRacEZA34TBvppFoGJ45og4EVCCjFULTniIek0p0nTdFAYZm2NwVDeX7YuhRlxU3jWkBUdvtun9/KhuIholF9Vx+0zD5t5y0+dzDA7DZKPxv/gAiCgHoCSE6Bvh8Uxo5B/6FrYX2qOQ1WZNC7LKLl/g5KzGF62qYfEdCVSrhWxVIN0insDJYaDiJFYXHhkLjozBic6LSUToRfLCOjjhzzMlWXA0F5XEspCYJl5R3EuSUt5uPMhYKfSnTpkrKxmr41EtOC10UZ203wyctN+Mps9nDxXDZCP1t5iITiOi12nbLgTQB2AnEd1MRJNbOL4JRVopfKZxZCpvPYHTO1QDAONCWo/WxeCEr4f81GfpKhqqOugsei6ZZBfV8McgL53TVkxTnEskfgUZLDiRVg1mF1VgwUlwUelWlFLebihNXK2DY1H0/m6dGJxWCpxhwwKHYTJR77f4QgDz5RsiehG8rt4/B/AJAIeB08RTeX7bQJC9I03xzdYtYaLks1pw/BicZjKiGm3wmBXVgiOtEoEFp+JZcIBkS02aBSfrz1eSi8pkJVHFgezjJF+LQOAoQcYJzTZVbIuQT0kT14VXKW815qJy9UrGisBx/WywhCyqfAtdVMOFXVQMk416v8WHAPib8v61AO4RQrxHCPFNAB8E8G+tGtxE4OSv3Y6zL7sLQPQP/K6hahAbwjSH/IRfLxBYuqhMloJ6tMqCoy7qQwYXVYdvwUmsg5MyrKxCTm/VIDFZcNRpiFpwwjFGgowNFpy4i8rLolJr40iqjoi7qHKhi2rtjoG6Qk5NEyeiyFwGLqqEOjjj2YLDAodhslHvt3gygM3K+xMA3Ky8vx/AvBEe04Rj3c5BAOGiJgB88teP4mPXPjx2g5oAyDWxrsCRLqom4mlaJXBMFhi5YA9VHXQUm7fgZHXFycNy2mJeruOicoW5F1WkDo6SmSWnUHdRCREep4tPL8Vbt+B4Aueh53fgxEtvx7XLX0h9vqrjBs9mW/E4It1ytbtYcFjeMEw26v0WbwCwDwAQURHAEQD+oezvAtB8OsAER18c5SLhugJbesvY0kDjQCaOLPFfz0UlacZFlfXajWISW3J8AxUHHb6LKknHpAmcrKKsEQtO5H5CaIX+vM1J8SyhBSd6zd6hanBvfT6qxiBjz0X12LoeAMCja3vSHi8WZByNwUE8Bke5XaGFlYyHC1twGCYb9QTOTQC+SkQvAXApgH5Ee08dCuCZFo1tt6fPtxxI1E/WVcdtWQrynkLVn880EaIu9s0EGbeyDo4eRFtzvXgWmSYOJAempz1JZhdVQgyO2ioiuKYmDgAvcNezOsVjcEz30cdVytvBvfXvTc3kosrbGKw6QdB4Vym9rpFXBydME/dq9oQfMryCewkxOOPYRcUmHIbJRr3f4s8BGAJwK4B3AniPEEI1O7wTwC0jNRgiuoOIhoioz/96Utl3GhGtJKIBIrqdiBaM1H1bRa8mcBzFRVWuuS1bPPcUHH/+0iwWapCwqaBcPVqWJi5ELM7DcQXKNRdCYFguqqxjDgv96Wni9bKoQkHjKr2oEgWOv18VSe85aRGOXTwtuLcufiqOG3FzAUDRj8HpK3u/VzLTLImq4yKvWHDUsUgXVVKzzfEscDhNnGGykfpbLITYKoQ4GcAUAFOEEL/VDnktgC+O8JguEEJ0+l8HAAARTQdwPYDPApgKYDmAX43wfUecXVoLgWBREr4FhwXOsKhlsOConaqbESutsrKZLDiOKwJxEQQZN1HoL2swdeCisuu7qCIWHCWoWO1FlbTwmoKMTz9wFoDQeqR3FK+5bkwAlvIWyjUX/YHASS/jpaaJy0uFGV0GF9UoNdscLuyiYphsZK1kbHR2CyFGq2vkawCsEEJcBwBEdBGArUS0RAiRvbvgKCMFTptfZl4NMq46IvZHnWkMKVjSLGHqvqZicFrpojJYcKS4aA9icJoIMs5swZExOFmCjMPX8upEntAyFfoTihPNVAdHCo+8bXZRVWsiJrxkkLF0/erB0To1NwwylmNzXIG87f1PhOQ08XFswWF9wzDZGI+/xV8hoq1EdDcRvdjfthTAI/IAIUQ/gGf97RGI6HwiWk5Ey7ds2TIa401EuqhkPIX6CbbquKi0yP2xpyBdTukxOOEcD7fQ3+1PbsY3b3mq4WuYqLkC+RzFtsk+VB2GnxmVtEdpOIsqY5CxFGRq3ZtoHZyE+xhicGSDT/l/liDjNl/g9PoWnHqCtaYFGXtjR/AMdkqhv3riaSxhCw7DZGO8/RZ/EsBieKnnlwP4IxHtA6ATgG5F6oGXxRVBCHG5EGKZEGLZjBnNl0MfCQKBU9QsOEKgUnNRaaAq6+7GP1dtx6/uf76hcx5b24Or7l6V+fjAgpPRRdVMDI567XdceT8uu+3phq9hwmzBcUMLThCDYz4/SfgAcXdPEkGQcRYXlSsCa0vYXDO5m7hqYJLaRR2zrJOTT0oTd5OzqPoyVqb26uCEaeLeWEJrklfoL0TVDePZRcX6hmGyMa4EjhDiPiFErxCiLIT4KYC7AbwcXmuIbu3wbgC9oz3GRuj1XVQy5ddVXFQVx21ZAOt44Jf/fB7/c2tjYuCV370LF/3x8czHy0VxoJIsFIebRdW6XlQChVw0SLbmiqDoX2D1S8qiGhELjndc3tKDjM29qKRVIxpkbC70Fx1r3EUlY2+SsqiqNWGug1Nz0FuWlanThZxaS0dtLeH976eFR9LEdxMXFadRMUwmxu9vsYeA9ydoBby2EAAAIuqAV59nxRiNKxPSglPKR1N+xR4QZFx1RdPiIGurAbkoDlRqicdUaoqLqonxNGpl+91D64IsnzQ8Cw7FtsmfiVIuXeCMRAyOvJce7Gx2UYVWDbXuTVKhPxXpSlL1iIyvSaqDUzNYcIo5r+rxjv6qP/7051TdXIHAUayo8VYN4bnjWeBwFhXDZGPc/BYT0WQiOpOISkSUI6I3AzgZwJ8B/BbAwUR0DhGV4KWvPzqeA4yBsEWA/MMfBhl7LqqaK1JdDbszNcdtqjUCkN1qIhfFNAuOOoaRsuAkCbB1Owfx4V89jJse21D3urWELCrpEpOiOGnI/1yVHN9fzTjv/WVv3vRsJFOQseeiisbgFHIWegaruONJL9bNSlh5TWni0nIjXUj6z0qlZhI43pxs768EY0rCcYUvyqTA0cbipjfb1OOSxhPEPiqGyUSmLKpRIg/gYgBLADgAVgJ4tRDiSQAgonMAfBfALwDcB+ANYzTOzEgLTvgJNvwkq7YZKFnp9Tx2R2qugNOkBadcc2KLv/EevgWi5ls+TJ+6q8ONwTFY2Wpu3H0CAIO+JclUR0bHdeOF7GqKBaeY9/YltXT4xG8eTbx21mwxafnqKkX/DCR1Ew8Ejr9tUlse1z+4LjimXpp4JMjYX6TthCwq0xzLnwk5vjTBKucxTBOPu6j0OjiRXlTjuFXDONZeDDOuGDcCRwixBcDRKftvhSd+dhv6KtEeSEHdlkhtFjf4tD6RqDluoiWhZ7CKmuNiWmfRuL9cc+PR46Z7aFlpJoETcVGNUKsGr8pu/FhZVDBLB/Ka66ZbcHxrhclYlMU1k4V+3/KlCxxTDI4jFMHh335yewFecXOPaJCxWjcnbqmUgiOfkkWlZzLFKj+nPKf8Xhe0NPHgQ4ZIr2TMFhyG2f0ZNwJnIlL2FwrdgqMyUQONa65IjAU54ot/gSuA1Zecbdyftf+Tev2keYwEGY9Qs82K46INcYUjx52ldo4r4h2ra64Izg0sOIafmXrXz/qc/eUaLALaCtFnGTS4/FwRWnCkFWRyW7RVQtKy6wgRazmh18FRv+dCCGM38aJBECYhCzTGLTj+87jxLKrdpQ4OwzDZ4N/iFlKuSVN66ErRSVvMswbbjkeqjhv0VtKpZ0jJYgEBorEmSVaL4cfgmCw4rvG5AoGT0YKjd6x2DRYcUy+qpOvL9Tmri6q/7DX1tDWLQNkQWO26oVgIXFTtmsBJadWgu9rkPT0rELBzMOwAE1pfzC4qSdqHA/mzocfgOLqLKrGSMf9pZJjdHf4tbiFyoXYUs7hO0sK8cuMuHPCZm/HC9oHWDbCF6G65RjAtsCYczUVlQnVRjVQvqsc3eN+b57dFvzfSspJF4DiOQNEYg+PdL4zBiZ+bdH25KGd1UQ1Uamgv2rH0bpPAdIwWnELkGFUgqD/qnosqej1pUZnS4YmkHf2hwAnjZzQXlaFuUBJyHmNp4rLooIhXMqaIBYfdQAyzu8MCp4XIhUIu8qZP1knuhhe2D6LiuNjQM9S6AbaQtGeuR9kQA2K8h1qlOIuLqimxFR/LM5v7UHFcvPjrt+M137872N6IBUcVDME2Nyz+WLAtpVu3PiazAJQCILMFp+JZcHTDi1HgRLKovG1TNAtOJAZH2e6aXFR+7M0U34KzvT/s2xaKkzoxOBmKHcbSxINSDZ6LCgkWnPFcyZhhmGzwb3ELkem2csFpxEUlt++utXKka6iZ8Wd2USnXVoXi6/73H/j6n5+MjANoTmyZxi+z41wBPPj8znAMDcTgOG6815JqwcnnrKBbt06yBce7XjVrFlXZs+AQUUQ86NeXIkuvZDw5ReCouCI+93YQg2Ohu5TDjoG4BUe3ohT1wohpLirNChSLwRGItWqwFYWjW4sYhtn94N/iFlLRLDjmIGPzYlVtwN2Rhee3DeDsy/6ObX3lEblePQIXVROBvc24qN7z0+X4/cNeyvI/V23Hd29/BoBXERfw+hjpY7n4hsfxu4fWIQ2zwKkajmzMRaX2SZJcefdqfPmmJwB4C2zNFfje7c/iiQ27IsclCUC5mGdt1dBfqQVNPduUtDB9/uU863VwJrVHXVSk/DVRA4JdYciiUpTF1I4CthlcVM1acG57YhPe+KP7vHN8kaTHJ3l1cJAYZKyLT4Zhdj9Y4LQQPQbH9Ac5OXYkuzUgCz+5exVWrN+F3z+8fkSuVw/5XFmLzqlkdVGplorntvbjQ798OHaMnL+2gh0rJvfHR9fjtpWbU+9hqmmza9BcqbjciIvKFcbCeNI6olov9AafST8T0uqQ1RXXX3aCIn9RgRMNopbuJelWknv0rCZVILzt+IX4wKn7wLYIrht3UanWkikdhUgMTi3BRRXPojLPw7t+uhxbesuRMcv7RZqDWhSJu1GtOeM5TZxhmGywwGkhQRaVvyCZ3Q3mxUguYiPlopK9jdLaGowkcpFtzoKTNQYng6VECpy8HXOTlGsu+uu0VRg0iC3ZC0m/hxQ25YRxrdraj6/9eSWEEJ6LKmERLdhWZOGdqllKpABcNL0jel7OXFMmCc+C4/1cqKniXisRtY6NvH68I7eK+jilvI2Pn7kE7XkbjsGCoz771PZCUJ0YCH/24y4qLYsqg5CTmWpSfDlCBG6qtDRxrjXDMLs/LHBaiF4Hp5Eg45GOwZELmSzutnbHQEOduxtFLrJpcS8RK4FyXFYXlclSoS+6cqEu5a3Y8ZWaW7dv1JChJoyMwZHs8t/XCzL+wNUP4nu3P4vV2wbguAK2Zf710xf2KR1RgSN/Zr7+2kMj24O+ThmtZgN+mjgQtoaQ91a/B65mwZHvT9l/JpbMDksymmJwLIsgRDzdXbVeTe0oRGJwkiw4scKIGYRc3opmUa1YtwvXLX/B34bEVg0Mw+z+sMBpIVmyqKoJi2HWGJzfP7wOWzPE1chYiwF/QX/nVffjoj8+3rKYnCxBxqq4UxfU7IX+4tWAdcEYcVE5AjXHxc/vXYOBSg2VTBYcg4tKFzh+z7F6MThSGPSXa77ACfeV8uEbvT5Od1u0Hqe8fsGOBt3m7SYsOEXvGlIAy58T1YrmiKjgkD/GbQUbP37bsuA4kz6wyPu5TxO6Uzs8C44Up0GAsGbhaiSLShJacLz3H7vuEVx4/WPeNiu52SbDMLs/LHBahKtUpU2z4NSPwUn+I94zWMWHfvlw3UBZIAyalI0pt/b5DQtbVEwwSx0c1Q2itgfI7KJyRbAwJ50rx1HKeTE4V9y1Cp/93b/wy3++gJorMgmcLq0ZpR5k3CMFTh0LjnQD7RqsoqZZcLpKYUaSnsGjf4ukGCzmo8fJxTyLBUcIgYGKasHxzpUxOeo8SveSdFGpSeCq6DBZQGyL/CDj5LHMmVRCuRaWRAhcVJqgUeelkLMyPacUSbZBvRRzdkSUsVuKYSYWLHCGwQNrtieneTvyU7YVVPQ1ponXyaJKsvAAYQBsveaOz23pw9odgwBCgSOtCVkDehulmiGGSJ071YKT2UXliEhwLADc++y22Dgs8hZExxX4rS8G5YLXV06/12DFifVqiruoNIGT8MxyrFt8q5maSaSKKN01o/+MhRYcXQD4aeIZLDjlmgvHFYEFR7qfpGBUO4rHs6jUe6YLHPJT3dOE9FELpgIA7l+9HeWaE3RK15+vqHyvOwrxmCoAeOSFnZH3eh0clVnd5l5oDMNMDLgXVZM8t6UP5/zgHzjv2L1x8asPie2XwqG9aKMy4Hrl6huogyMDVdXFsmeginLNwczuEgBV4LiBm2q6oYHlS77xt+B1vx9kLMeSVUw0SprVSqKKn4gFJ2uhP9eNCZzzf/5A7B5524JtEQarAis39gIIrTBJFpzntw1gelcBg1UH0zsLQE+4L9GCU8dFFQgcmeGjxNpELCHaxw5dJAYCR7NwSJGSpd6PjD2SFhspBNr99wMVB89s7sO+MzsTs6jU8wBzLyqbCK6bPqYD53Sho2Bj+eoduOvprbjugbX+/TQXlXKvjmIu5oobqjp41ffujmwLBI7BgjOru2R0qzEMMzFggdMkMg7j0bU9xv1SOHQUctg5UEXNdRPSxBMq8PrZVaoF5/hLbkN/xcHqS85GueYEC2W55mDZxbcCSG5gKRnwLRbyA7Wpc/RIIDNc9OdTRZ4qBFQrVPYsKhFrFBnd76LiuCjYFvK2FbnfjgFPlAxWHT8eJrrSnfy123HIvEkYrISp1OFYo+N7elMfBisOtvluv75yDT0D1VivJulSCiw4yj3V17rnJSZwZENOXeDYXuG6LNllO/3nn+Q3zJTBxR3+fH75T0/gnme34c6PnxqMWx6jZgOqIispBmf7QCXVypizLbxo0VT86bENke+97qJSg687CrmYi2qXoT5R2Kohft9ZXSVQYotQhmF2d9hF1STy02WiBcbfLk3+jisS0sQd4zUqjrcgqItbv5LR86Yf3Ydzf/iPyL2yIBcBOZZ67q1mkYusvtiqdXGiQcZmd1XqPdy4i0ploOqg5ngVg22LIgugmrXTr6XOyzE/tq7Hi8Hx42OSPu1/+7anceDnbsZvHvQsD6u29uOwL/4ldpwsCSCFqeqiUgWOtHY88rmXopS3YiJRWrh0Cw4A5C0rU/p0j9/cUja7lJaODl/M3eO7+rb2l1NdVKqVxRTDYlmEWx7fhLdfeX/qeD551hL0DFYjWW15zZSlXr+9aAcfGIaqDoQQ6BuKW+PkmPWGogAws7vIFhyGmcCwwGmCquMGcRhJpvdA4PgLRi0hk+S7tz+L/T9zU+DmCO7hL4ZJQcYPrNkR3kuxKNQLmpX3kUNpRBxlxXXDWiO61UpdrIdvwXFTLTiDFSdwUeUsQs9AOMfqa33OBjTR11mKF8PLgp6yLp8xEDiq1UY5Vs7ZpPY8Oov5xMwwXeAcOKcbtkWZXFTSgjPZt+DkNAtO+AzxGBx1rPUCc2XsS71MvyWzu7FQq+uTzyVfu7OYg+MK9AxWcdSXbsGfV2wypvzLMZvGWcrbbL9hmAkMC5wm+PCvHsYbf3QvgOQsIWmFaPcXRccxCxz5h3/9zsHI9rR4jtjCqVg81mgdrvW4H5mOm2TBueeZrVh44Y1YtbXf+FxZUK000gLlugILL7wR3771qdg+fRyNVDJOEx395RoqSgyOmt4dseDoAkcLPO7UasWonHfs3on311PM5XuZwabG4KgaJuICsikWaF42BBl/701H4uNnHoCcTYmB3TsHKlh28a148PkdgYtONruU19IF40ClFtbBkeNtIPGukf5fcye3Rd7nEuoEAZ57ruoIrN0xgP6Kg/tWbTNacOSYTVlUAGdOMcxEZtwIHCIqEtEVRLSGiHqJ6CEiOsvft5CIBBH1KV+fHauxqrEPSYuJXIQ6/CyVJAuORC1VD6RXMtYtHKogWL2tP/3YmoutfZXEGJxf3u8VQXvo+dBCdOFvHsVbrrgvcew66nPK19Iq8qO/h8UFVfG2aVfYNT1rewrHTY/BGah4LqpCzooFrO5ULDh6VpTuspKWkpLBJXT6gbMS769b5QZ9F+Nm/1nVzB617YDq1svnLGOQcSEXrXZ87OKpyPuxRkl1cNbtHMTWvjKe2tiLnb7Ak3FCSdlG/eVaYI0rGCw49WjEBTpvcinyPq3hZc6y4LhuIBZXbuhFb4oFJ6nGDcsbhpm4jKcg4xyAFwCcAuB5AC8HcC0RqSlKk4UQo9NrIAX1k3yii0pmUfmf/h03niauuhM29Q5F9qVVMtZN8eWag1LewlDVjQkcU6G61dv6g8wYPd5FtnJQLSNS9GRFdUPJ1wOGxafiuLjq7lW497ntWL5mO5bM7kKl5hpjcK64axVWrOvBN19/uHLteBaVymDVc1HlLAoaUUqiFhxtDrT3cpE0WXBMWWuSXYM1zJkUHQ+AoLFkzhB3A4TuQ3nvWAxOzUFRex5bqfeSZFWUJQL6Kw56BquwKExPD2JVNCXQV3aCsUlrSCOVkxoTOFELTpqLKmd7z7nVd/et3LgrJlSBMMhYFYMvP2Q23nfKPv72zMNjGGY3Y9xYcIQQ/UKIi4QQq4UQrhDiBgCrABw11mPTKeXChS4pCyrIogosOG7sk6+anbOxJxqjkFZTRXeplGtuEJC5tTdqCTIKnK39iosqen252A+nyadqgZCZLv2GlgeVmouL/vg4bl6xEVv7KnjXiYvQUcwZM7u+dMPjuF4raFgvi6q/XIvE4EgsAnYq1pW+cg2uK/DRax/G8tXbYxYcmUWk18MBvO+hns0kSbLgBOOwCD887yj86vxjI6JGzQ7K21Y8BqcWr+AshUneosQsKvlzM1CuYedAFZPa8kH6dJhtZLLgJAcZ12OogRivyVrPrTQXVc4i1BwRuHh3DFSxamsfAOAVh84JjjMJtzMOmoVD508GwC4qhpnIjBuBo0NEswDsD2CFsnkNEa0loiuJaHrCeecT0XIiWr5ly5aWjE2tIJtUTTXMovIWRdeNxut0FGxMVtKIVRcNkN6qQbfgDFWdoG7OzgFN4CiL6uLpHchZhNXb+hUXldmC0ztUQ1+5hk/5Ze3r8eTGXlx688pYQUPpLjEFP+vicGpHwbdEZfvU77gCxVy9IGOBfM6KLHAzuoqRRbq/XMOuoSquf3Adzv3hP2INSY9dPA3vOWkRvvumI2P3aCvYQeaRzi5d4FQd7D21PXg/b3IbXnbwbByzeFrk50i15hQMMTWVmhsTVfL5cn5hyQef3xE09pSoFpydg9WIoMgFlo7oM/T5bSW8scg6ONkVTiMxOLo1Ls1FZVte4cZtimt3+WrPrXrxqw8OtklhqxqmOovR9H2GYSYm41LgEFEewNUAfiqEWAlgK4CjASyAZ9Hp8vfHEEJcLoRYJoRYNmPGjJaMT7XgJMU7BDE4BcWCo/yxn9pZiCwwm3uHsLWvjG/d8hT++Mj6IE3X5KLSXSpDVTcQQju1RVUVC6W8jflT2rB6axiIrMfoSEtLX7mGK/6+Ctf883nj8wHAExt24ef/WA0AeOOP7sUP7ngWPYPVyJjl64EEC47K5PY8SnnbeKxEtU5UXTfWmFJlQGZRWRSx4MzoirqV+iu1iHtDnR/A+x5++uyDsNfU9lgsRylvx9pFSFQLzs/+sRo9g1UcOCdsTnnE3pOD12rjSFUgei6qeByVbsGRlpec76L6+HWP4Hu3P4snNvSGz+mLzDuf2oI/PrI+IrDTY3CiLqoGNEtDnHXIbJytWF9yKd/bvE2ouS629pYDy9oDa3agYFsRy6i00KjPpdc1YhhmYjLuBA4RWQB+DqAC4AIAEEL0CSGWCyFqQohN/vaXElH3WIxRbYyYGGTsCwuZJq43HJzWUQxSdAFg064yPnX9Y/j2bU/jP695SLl+fDXRrSG95XAhjVlwFIGTz1nYb1YXHl23M9imW0uk1aFvqIaNmlVJz95644/uxWd/vwL95VoQ09JfcYxBxrrbx3s2XeAU0F6wUy04Utw5roAQ6W6MHQOV0EWlWANmaHEzfb4FR/LFGx6P7Fdjb3SXRlveTlwwf/T357CxZwg1x8Xnfu8ZIvebGQocad0Doj3B1GnO21ZQMmCo6uCKu1ZhsOoYqhh74+os5dAzWA0sRb9/OHTrSeH4+IZdAIBnNvdF7gPEY3D6y7Wg8GBQtbhFAqe9kMP3FCuZ3rJCxfZdVFv6ylg8vQMzu4qouQKdpVws3gqIChyTq5FhmInHuBI45K0eVwCYBeAcIUS8NKmH/BM7Jg501T2QnCYeLfSnZ1FN7yxEPkGv2zGIdTuiqeJAaOVQz9VdVLsGw/e6BUeP+zh64RS8sD28j5pi7rhhTENfuRbrNK6LLWl1eHJTb7Aoe3EvSpCxzKIy9HyKWXDa8mgv5NBXruH3D6/zRYzAbx9aGxwjxZyMcTIVuwM8V8uW3nLgolItOHpgcH+5FszhofMnQUeN85HfhyWzuzCzq4i8TYkWnJUbe/Gx6x6OfE+6Sjm8ZMlMfOJlB0SOTQp5yueswP34nb8+jS/d8DhueXxTYgzO3Elt2LBzMGjkedvKzeFzaiLzA6fuG7yWcSvnHjU/InL6yo6hm3iLFI6GyTr3umXz8dKDZgWWqq19FUzvLGLJHO+zTpLYVJ9pUhu7qBhmT2C8fZT5AYADAZwuhAhWYSI6BsBOAE8DmALgMgB3CCHMfRJaTJYsKuny6Par4DputOHgtI5ixBK0cdeQMbZBBpiqVg3dgiNdIUTR9OdyzcFflQUOQmDZwqmRc9UU8+39lcD98NALO7FGy8gaqkUtB3Mml9C7qQ9PbgzdIL1DtSCwGghdSqYYHD14dlKb56Jau2MQH/rlw9jRX8GSOd34yK8eCY6R4k7Ob9Kn8XmT27Bp1xAGKw5mdhWDBa5gW+jWFrj+shP0lzpm0dSg/ca0jgK29VeMsSAfPWN/vHTpbABIjMEBgB391UgJgPaCjZ+8/ejYcU5CLJdaB0cVpnrskbQszZ3chjuf3hLUt3lmcx+291cwtaMQEZkn7Tc9yCQCgAXTOoI2H6qA6VdicPJNZFENB1MA8FfPPQyAF3TuuAI7+is4eG43ZnWXcOdTWxItqqphavakkvEYAOgu5YLvK8MwuzfjxoJDRAsAvBfA4QA2KvVu3gxgMYCbAfQC+BeAMoA3jtVY9XThoNZLpYZnt3hm/w09g+gu5YJPizUtTXxqZwGT/EVojv8Hd9OueLVX+QdbjZUxVWwFvN46OwcqQazP125+ElfdszrY7wrg4LlRC4Wakr1dWYgfeWFnRCwBwJBmDZItDFb6Lg85NjUuKQgyNrioVAtOV9FzLajWkOVrdsRElizmJgWJKlZWfeXlwevZ3SVs2lXG1r4ypnUWAwtOe9GOVOvtKNjoK4cxOEtmh17P956yGABiPaWAaFG6JAsO4MWRqPNqavoIRC2Bbzl2QfBajcFRU9uThN3cySUMVBw8v30AU/xxL1/tdedWf270jCUVNdi3v1ILXJPS9fP24xcmnjta5Cwv+Lpn0MsGk7E7G3qGjMer857m+nr0ojPx9dceNrKDZRhmTBg3FhwhxBqku5yuGa2x1EPPYBmo1NBVyuOD1zyEW5/YjKf/+yys2zGIuZPbAsuBowUZn7TfdDzmWwr2ntqe+Ie5arDgmOp9AMCs7iI27hpCX6WG7lIeTykxFoD3ybyQs3DgnG484YuSIc2Ck4aevi1FxjNbwvv0DdXQrSy+VdfFUNWJVVhWnw0IRYQqFm54dANueHSDds8a1u8cRM9g3IKjfuKf1V3CivU92D5QwYzOQmBlGSg7aFNiX6Z0FNBfrgXPcsDsMEbmPSctxrtOXGysgqvWbOnwr3flO47Gi/efAVcASz57E6qOwJptAxFhoqfxS+TPxs0fPgkHzArHoAqcpzeF83z4XpON15Hj2rhrCP922Fzc9K8NePD5nXjp0tmR7LDJKW6aS885NIgDU7OochbhuS+/fFzUjrEtCkT/pLY8DvRdVIclzIsePM0wzMRn3Fhwdid0C44M3rzzqa0AgI09Q1i3cxDzJrcFloOa36phyewuPPjZM3D8PtODhXP+lHYkIa0cqsBJEiIzujxLkOyzpAsxqa/OP3lRsK1c8xoVqoHC0iVzgRKjAcRr6si4lVVbQitLX7kaDTJ2BC74v4ciliTT9aRLxVRMT+XHdz2H4y/5K+59zssy606wZMzsLmL1tgEIAUzrLOK0A2cC8Nxinco5U9oLEQvOvjM7g31ElFjiX42favddct2lXHDOw597Kf7rpfujZ7CKZ5X5WTDN/L2W7stSzo4ItYLfkmBbXzkS9P0izdUomTclFF4zuorYZ0YnVm70xKxai2iKwSoleeVhc7H8M6fjzKWzPBeVPzaLCJZF46J2jBpILK14//rCmfjV+ccajzc129zdSIo3YxjGDP/GNIEuHGQMjIypeWH7gCdwpqgWHE/g2BZhaoe3mEvxo8asqJ/egTCwV3VRyWaNOrO6veBZ6VrS2xNIV8O/HzEft370ZBw8rxtDVRc/v3cNln7+z4FVR8ZgHLH35Ejsgi5w5HOvV6xPvUPxIONbn9hkHK/qAptssOCYuPsZT9hIt4t0k0lknMjs7jDOYnpnEfsq2UvzFRHQ3ZYL6uCU8lZdgSVRF3lpwVGzojqKORzgu7sefmEnAOCmD52EVx0+13g9KQrVGkve83iF/pb7zVXl/ByupJirqK6z7lIeS2Z3BTFSajXpSSkuKsCbsyntBazfORRY35LE3liQMwQNdxZzid8/+e2a1mF+7iWzu4zbxxP3f/p0LP/M6WM9DIbZbRg3LqrdiaL2R/Sl37oTZy6dhbaCjV1DNbzpx17fprmT24JaHjU/yFj9wzx7Ultw3E0fOgkzuor4+HWP4MlNYdCuyYLz9OZwv4pc1Lf7lpgdWsq4Gjy678wulHJeSva1y71WDP9a57nM5Cf2hdM7YBHFGnMe9aVbsGzhFAxWHXQU7IhloK9cixStS6qqC0TrxMiFJ0ng/PxdL8Jbrvhn8H6jH6+kx6Lcc+Fp6CvX8IgvKgAvY83b95JYy4yOQg5beyvoHarFxFJWpPtLz+CRgvPJjb3oKNiBG8WEXkxPIgv93b9qOwo5C3d+4lRs3lWOiCkVdQGf1JbDkjnd+N3D69EzUI18n9JcVJK3Hb8Qv3lwbVDsMSV0ZdRRxVZ3W/0/Y/JH/9jF02L7/vKRkzGrOznweLzA2V8M0xgscJqglI//pf/zik2RKrWAt8DJdF1pwVGDHc9cOguXv+UovGTJzMDkPrMr+oe26rh4YfsA/v379wTXVF0eKnv77o+NPV62zbY+XeBEj3eECAoKAsDz2wfQVcxhSkcBz28fwPwpbd54/RMHqw7KNQfb+iv48wrPKrPvrK6ImHhmcx/+59ang/dJafQA8Fu/9cI7T1iEd5ywEECyi+qEfaKFq5/03S66KJnRVcSMrmJEWE3z08KldUON/eks5gIXlXR33frRUxKz4/70wZNiIkyKLD2bSt73+e0D2GtqtM+STtDQMhe34FRrLu5fswOHz5+M6Z3F1P5XRIS5k0pY3zOE7rY8FviC54mNuyIxOFlqwRw4pxu/eu9xeI3/s5fkmrr7wpfESgoMh7s+eSp29CdViPBQU8izLPx7T2vHlW8/GsftExc4+88a/9YbhmEahwVOEyS1B1AXkPedsg/OXDobz/liRNbBUWMBiCiWkvq+F++DxTM68JWbVgLwYkbufDpsOXHA7G5s2mVuQbFgWgeIgHU7PZfRVm3R0Qv1rdXq7jy7pR8LprXj6ncfgyc29KKYsyPjLVcdPLs5Kq72n9kZCJwZXUXc/K+Nkf2m9HAA+NgZ++MbtzwFADhm8VTs5YvDJMuEZVEgRgBPEFiESEaUihpLoxf2U7NoOoo59Fc8F5UUS+q5OgfNjVthXn34PEztKASuR4lqTZlSxyUk0X+28jkLQzUXKzfsimRXpTFnchvW9wyhq5THYX7PpQfW7IikietWyCSO3HsKijkL5ZqbGMcyb3JbrFHmcJg/pR3zp6QfYysFHrsNlrc/XnBirAzBqUtmjsj4GIbZPRhHRufdB5MFBwC2KhaTT77sALQXcpEsqpofg5PGoukdePkhYbn6as0FKcllabECnUUbs7pKWLdjEFXHxY6BKk4/cBb+48VevRO9Ptt33nhEzGIwpb2A+VPaccZBswAgFoPz5KZdkeP3mxWKge5SLmaxufo+c6uHtyqpxqpFRH39ny+JBjnrLqCuUj7RqqBuT3NhdBS9GJwtSsn/RpnSUcCrDp8X217K28E1505KFwBBs0ytuF3e9noulWtuJMMrjUXTOwB47s2pHQXsO7MT9z63LVJwMKlBqAlp9UgSq2lc+97jcOU7jk4Nam4GUwyOyiHzJ+GoBXVUEsMwExq24DSB6ka57n3H4bLbnsbfn/YyqI5eOAXnHbsgWGCDLCpXwHVFpkyI+VPa8KHT9sOTG3tx28pN2NwbBvHqQcgqxZyNeVPasH7nID79Wy9u4pQDZuC4xdPw/TuejVWgPXbxNHzizANw8Y1PBNt0K4TqUvvIrx7Badqn4P2U8XSmxLC88UV7Y0tvOQg4VmNNVKuNOrcfOm0/fOevzwTvD5zTFckkkuLhirctM/av+s37j8NDz+80iqBfvOsYrN85iM29Q6g6Ais39uK1y/ZKHH+zTO0ooHeohoW+6EjijxeciLue2RIba0ERPGkxPCqffvmB6CrlgsyxoxdODXqKXfTKg7BxVxlHJ2RhmfjGaw/D925/NlYkMgsvWuSdM7WjgB0DVZx/8mK8+IDh94iLBopzbArDMHFY4DSB+un36IVT8d+vPgQnf+12AN5Crn6al5/M73xqC5av2YGT9jM2QY9ARPjIGfvj2uUv4OYVGyNZSAfPCwv1nX3oHMyb3IbL73wuGNfcyW244dH1+MdznkXlhH2mBYumKaxEbzw5d3I0Bki3OKml/wFg6ZxuvOOEhXjFoXPwrVuehm1RLH7lpP2m4zNnH4jbVm4OnqWUt1DIWajU3EQLTs62cMGp++J434JwxkGzcfuToXtusx9ofNqBs+IPBuCoBVNx1ALzonyi/3248u5VALzYplYUsJNViBcmpIdLDprbbXR/qe60NNeZypSOAj7/yqXB+9ctm49VW/tw3OLpePsJi1LONDOts4jPvfKghs9T+dFbl+HyO5/DJ1+2ZESysVTrTNbMN4Zh9izYRdUE+h9UtfaIHmshm0Feu9zrp7TZUK04iTOXzkbBtvCvdaFbaEpH+Gn1/afsgw+dtl/wvpizMXdyKXBF3fThk7F4RmcQO2FqBXH6gbMin6jVSr4A8IM3H4UT9g0DMw+ZNwnffsPhwfuZ3SV8/pVLcdSCqThz6Sy875TFsXiMn7/rGHQUcyj5wnB2dwlEhJm+uFLdDXoA73+deQCO39cTI68+Yi5OP3AmrvRbHegxFs0g2yicdfCclqRB9/punQXT0i04SUiBM6u72PRCfsTeU/DL84/Dh07fr/7BLWLxjE5ccs6hIzbHaZWYGYZhABY4TaGXerctwrXvPQ4HzOqKuRGmd3kxEHIRV1PA6zGpLY9Tl4TiY/GMDkxpLwRpzxZRJHOnmLdw/D7TMb2zgFcfPjcQGrMmFTGru4jPnh3/FN5RzOGqd7woeH/gnKgL7Lh9puEX7zomeP+HC07AQf4z6u62txy3EB8/cwm+8ppDMG9yG15+yOwg/gcI2zVIK9RXXnMI9praFil02JayiLcXcvjx247GqUtm4t+PmIePnbF/4rFZOeuQOdhnRkfQlmGkkUHRi+q4qJKQgiCpcvGezIdO2w//fkQ89olhGAZgF9WI8aJFU/Hnj5wc295eyOHWj54CAHjZ/9yJE/et76JSefXh8/DnFZvw+mV74dJzDwUAXPzqQ/C+XzwQs5QUbAun7D8Dyz9zRmR7MWfjvv+XrUCYKWWWtMyv+VPaUchZuPScQ4zXOHn/Gbj7wpfEth+xl+dW+MCpnug5ab8Z+PsnosfJzt16kUKdb73+8NT9WTlwTjdu+9iLR+RaJj73ioPwlZtWBtaqRpG9zQ7fyxwwe/ahc/DUxuyieSLxkREQuAzDTFxY4IwiN384LoDqceqSmZg7qRSJz3jZwbODzs+A1/zwqntWJzZyzMI5R87Hbx5cm1rs7tV+Fd62go2nLj6r4XssnN4RGbcJGXD8fsXyszvzjhMW4R1NxL1ITth3On72jzV4+SHmDtffe9ORTV+bYRhmIkN6bZSJxLJly8Ty5ctbcu2FF96IZQum4NfvP74l11dx6qSXy+/hcHsECSESrzFS92Aax9UKRDIMwzAhRPSAEGKZvp0tOE3yxBdfFrRhaDX1AjNHSnSkXYeFzdjB4oZhGKZxWOA0SVudppAMwzAMw4wdnEXFMAzDMMyEgwUOwzAMwzATjt1G4BDRVCL6LRH1E9EaInrTWI+JYRiGYZjxye4Ug/M9ABUAswAcDuBGInpECLFiTEfFMAzDMMy4Y7ew4BBRB4BzAHxWCNEnhLgLwB8AvGVsR8YwDMMwzHhktxA4APYH4AghnlK2PQJgqX4gEZ1PRMuJaPmWLVv03QzDMAzD7AHsLgKnE0CPtq0HQKyvgBDiciHEMiHEshkzZui7GYZhGIbZA9hdBE4fgG5tWzeAPbMJD8MwDMMwqewuQcZPAcgR0X5CiKf9bYcBSA0wfuCBB7YS0ZoWjWk6gK0tuvZEheescXjOGofnrHF4zhqD56txWjlnC0wbd5teVET0SwACwLvhZVH9CcDxY5VFRUTLTb0vmGR4zhqH56xxeM4ah+esMXi+Gmcs5mx3cVEBwH8AaAOwGcA1AN7PKeIMwzAMw5jYXVxUEEJsB/DqsR4HwzAMwzDjn93JgjPeuHysB7AbwnPWODxnjcNz1jg8Z43B89U4oz5nu00MDsMwDMMwTFbYgsMwDMMwzISDBQ7DMAzDMBMOFjgMwzAMw0w4WOA0CBFNJaLfElE/Ea0hojeN9ZjGGiK6wO//VSaiq7R9pxHRSiIaIKLbiWiBso+I6FIi2uZ/fZWIaNQfYJQhoiIRXeH//PQS0UNEdJayn+fMABH9gog2ENEuInqKiN6t7OM5S4GI9iOiISL6hbKN58wAEd3hz1Wf//Wkso/nLAEiegMRPeGvjc8S0Un+9rGbMyEEfzXwBa8Gz6/g9cc6EV5PrKVjPa4xnpPXwEvh/wGAq5Tt0/35eS2AEoCvAbhX2f9eAE8CmA9gHoDHAbxvrJ9nFOarA8BFABbC+5DxCnhtRxbynKXO21IARf/1EgAbARzFc5Zp7v4C4O8AfuG/5zlLnqs7ALzbsJ3nLHnOzgCwBsCx/t+0ef7XmM7ZmE/M7vTlL0wVAPsr234O4JKxHtt4+AJwsSZwzgdwjzZ/gwCW+O/vAXC+sv9d6g//nvQF4FEA5/CcZZ6vAwBsAPA6nrO6c/UGANfCE9VS4PCcJc9XksDhOUues3sAvGu8zRm7qBpjfwCOEOIpZdsj8D5ZMnGWwpsfAIAQoh/AswjnK7Ife+hcEtEseD9bK8BzlgoRfZ+IBgCshCdw/gSes0SIqBvAFwF8TNvFc5bOV4hoKxHdTUQv9rfxnBkgIhvAMgAziOgZIlpLRN8lojaM8ZyxwGmMTnjmNpUeAF1jMJbdgXrzpe/vAdC5h/mt8wCuBvBTIcRK8JylIoT4D3hzcRKA6wGUwXOWxpcAXCGEeEHbznOWzCcBLIbnMrkcwB+JaB/wnCUxC0AewLnwfi8PB3AEgM9gjOeMBU5j9AHo1rZ1w4ufYOLUmy99fzeAPuHbKic6RGTBc3FWAFzgb+Y5q4MQwhFC3AXPb/9+8JwZIaLDAZwO4FuG3TxnCQgh7hNC9AohykKInwK4G8DLwXOWxKD//3eEEBuEEFsBfBPjYM5Y4DTGUwByRLSfsu0weK4FJs4KePMDACCiDgD7IJyvyH7sQXPpf0K5At6nn3OEEFV/F89ZdnII54bnLM6L4QWuP09EGwH8F4BziOhB8Jw1ggBA4DkzIoTYAWAtvHnSGds5G+vgpN3tC8Av4WVSdQA4AZxFBXgLTQnAV+BZJEr+thn+/Jzjb7sU0Qj69wF4Ap4peK7/g72nZB38EMC9ADq17Txn5vmaCS9YthOADeBMAP0AXsVzljhn7QBmK19fB/Brf754zsxzNtn/2ZJ/w97s/5wdwHOWOm9fBHC//3s6BV7G3pfGes7GfGJ2ty8AUwH8zv+hfx7Am8Z6TGP9BS87Q2hfF/n7TocXEDoILzthoXIeAfgqgO3+11fh90ebyF8AFvhzNATPRCu/3sxzljhnMwD8DcBOALsAPAbgPcp+nrP6c3gR/CwqnrPUn7P74blQdsL7EHIGz1ndecsD+L4/ZxsBXAagNNZzxs02GYZhGIaZcHAMDsMwDMMwEw4WOAzDMAzDTDhY4DAMwzAMM+FggcMwDMMwzISDBQ7DMAzDMBMOFjgMwzAMw0w4WOAwDLNbQESCiM5t4fWX+fdY2Kp7MAwzerDAYRim5RDRVb540L/ubeAycwD8sVVjZBhmYpEb6wEwDLPHcCuAt2jbKllPFkJsHNnhMAwzkWELDsMwo0VZCLFR+9oOBO6nC4joRiIaIKI1RHSeerLuoiKiz/nHlYloIxH9TNlXJKL/IaJNRDRERPcS0Yna9V5GRCv9/X8HsL8+YCI6noj+5o9pHRH9gIj07sgMw4xDWOAwDDNe+AKAPwA4HMDlAH5GRMtMBxLROfC6Y/8HgP0AvALAP5VDvgrg9QDeCeAIeL2rbiaiOf75e8HrKXeLf7/v+Oeo9zgEwF/8MR0G4DX+sT8Z3mMyDDMacC8qhmFaDhFdBeA8eA1GVb4nhPgkEQkAPxZCvEc551YAG4UQ5/nvBYDXCiF+TUQfBfBeAAcLIaravToA7ADwbiHEz/xtNoCnAFwjhPgMEX0ZwLkADhCy6x/RZ+B1QF4khFjtW4SqQoh3Kdc+HMBDAGYJITaPyOQwDNMSOAaHYZjR4k4A52vbdiqv/6Ht+weAsxOudR2ADwFYRUR/BnAzgD8IIcoA9oHX3fhuebAQwiGifwA4yN90IIB7RfQTnn7/owDsS0SvV7aR//8+AFjgMMw4hgUOwzCjxYAQ4pmRuJAQ4gUiOgDAaQBOB/ANAJ8nomMQihCTeVpuI8M+HQvAjwF8y7BvXWMjZhhmtOEYHIZhxgvHGt4/kXSwEGJICHGjEOIjAI4GsBTACQCegZedFQQV+y6q4wA87m96HMAxRKQKHf3+DwJYKoR4xvA12MTzMQwzirAFh2GY0aJIRLO1bY4QYov/+jVEdD+AO+DFx5wG4BjThYjo7fD+ft0HoA9eQHEVwNNCiH4i+gGAS4hoK4BVAD4CYBaA7/uX+CGAjwH4HyL6PoBDALxPu82lAO4loh8C+F8AvQCWAHilEOK9jT8+wzCjCQschmFGi9MBbNC2rQMw3399EYBzAFwGYAuAdwgh7k+41k4AnwTwdXjxNo8DeI0QYpW//5P+/1cCmAwvMPhlQogNACCEeJ6IXgPgm/CClR8AcCGAX8gbCCEeJaKTAVwM4G8AbADPAfhtY4/NMMxYwFlUDMOMOWqG1FiPhWGYiQHH4DAMwzAMM+FggcMwDMMwzISDXVQMwzAMw0w42ILDMAzDMMyEgwUOwzAMwzATDhY4DMMwDMNMOFjgMAzDMAwz4WCBwzAMwzDMhOP/AzU9PXQwEZfwAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 576x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.figure(figsize=(8, 4))\n",
        "plt.plot(rewards)\n",
        "plt.xlabel(\"Episode\", fontsize=14)\n",
        "plt.ylabel(\"Sum of rewards\", fontsize=14)\n",
        "save_fig(\"dqn_rewards_plot\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ao-N4W5GMM9i"
      },
      "outputs": [],
      "source": [
        "env.seed(42)\n",
        "state = env.reset()\n",
        "\n",
        "frames = []\n",
        "\n",
        "for step in range(200):\n",
        "    action = epsilon_greedy_policy(state)\n",
        "    state, reward, done, info = env.step(action)\n",
        "    if done:\n",
        "        break\n",
        "    img = env.render(mode=\"rgb_array\")\n",
        "    frames.append(img)\n",
        "    \n",
        "plot_animation(frames)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jcs3uX_QMM9i"
      },
      "source": [
        "나쁘지 않네요! 😀"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FwKGEv9nMM9i"
      },
      "source": [
        "## 더블 DQN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sPnMAa4OMM9i"
      },
      "outputs": [],
      "source": [
        "keras.backend.clear_session()\n",
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Dense(32, activation=\"elu\", input_shape=[4]),\n",
        "    keras.layers.Dense(32, activation=\"elu\"),\n",
        "    keras.layers.Dense(n_outputs)\n",
        "])\n",
        "\n",
        "target = keras.models.clone_model(model)\n",
        "target.set_weights(model.get_weights())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TLAntgw1MM9i"
      },
      "outputs": [],
      "source": [
        "batch_size = 32\n",
        "discount_rate = 0.95\n",
        "optimizer = keras.optimizers.Adam(learning_rate=6e-3)\n",
        "loss_fn = keras.losses.Huber()\n",
        "\n",
        "def training_step(batch_size):\n",
        "    experiences = sample_experiences(batch_size)\n",
        "    states, actions, rewards, next_states, dones = experiences\n",
        "    next_Q_values = model.predict(next_states)\n",
        "    best_next_actions = np.argmax(next_Q_values, axis=1)\n",
        "    next_mask = tf.one_hot(best_next_actions, n_outputs).numpy()\n",
        "    next_best_Q_values = (target.predict(next_states) * next_mask).sum(axis=1)\n",
        "    target_Q_values = (rewards + \n",
        "                       (1 - dones) * discount_rate * next_best_Q_values)\n",
        "    target_Q_values = target_Q_values.reshape(-1, 1)\n",
        "    mask = tf.one_hot(actions, n_outputs)\n",
        "    with tf.GradientTape() as tape:\n",
        "        all_Q_values = model(states)\n",
        "        Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)\n",
        "        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))\n",
        "    grads = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(zip(grads, model.trainable_variables))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n0Qhf1P7MM9j"
      },
      "outputs": [],
      "source": [
        "replay_memory = deque(maxlen=2000)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3xQNo7M1MM9j",
        "outputId": "8a29f368-626e-4bc2-ac2f-1c8e8f6e64ae"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Episode: 599, Steps: 55, eps: 0.0100"
          ]
        }
      ],
      "source": [
        "env.seed(42)\n",
        "np.random.seed(42)\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "rewards = []\n",
        "best_score = 0\n",
        "\n",
        "for episode in range(600):\n",
        "    obs = env.reset()    \n",
        "    for step in range(200):\n",
        "        epsilon = max(1 - episode / 500, 0.01)\n",
        "        obs, reward, done, info = play_one_step(env, obs, epsilon)\n",
        "        if done:\n",
        "            break\n",
        "    rewards.append(step)\n",
        "    if step >= best_score:\n",
        "        best_weights = model.get_weights()\n",
        "        best_score = step\n",
        "    print(\"\\rEpisode: {}, Steps: {}, eps: {:.3f}\".format(episode, step + 1, epsilon), end=\"\")\n",
        "    if episode >= 50:\n",
        "        training_step(batch_size)\n",
        "        if episode % 50 == 0:\n",
        "            target.set_weights(model.get_weights())\n",
        "    # Alternatively, you can do soft updates at each step:\n",
        "    #if episode >= 50:\n",
        "        #target_weights = target.get_weights()\n",
        "        #online_weights = model.get_weights()\n",
        "        #for index in range(len(target_weights)):\n",
        "        #    target_weights[index] = 0.99 * target_weights[index] + 0.01 * online_weights[index]\n",
        "        #target.set_weights(target_weights)\n",
        "\n",
        "model.set_weights(best_weights)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NKBNVII0MM9j",
        "outputId": "5f2e25a2-07b8-468b-c6b6-3a134991de07"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Saving figure double_dqn_rewards_plot\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAEYCAYAAABRMYxdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAACiO0lEQVR4nO2dd5wkVdX3f6e6e8JO2JyB3SXsLixRlhwkgyLKI+ojYg6YMD8qvgZQMafnMYsJMYtiRFAJoiBpySwsS9hdNufJoUPd94+qW3XvrVuhZ7qnZ3bPl88w0xVvVff2PXXO75xDQggwDMMwDMPsSTiNHgDDMAzDMEytYQOHYRiGYZg9DjZwGIZhGIbZ42ADh2EYhmGYPQ42cBiGYRiG2ePIN3oA9WTGjBli4cKFjR4GwzAMwzB14v77798hhJhpLt+jDZyFCxdixYoVjR4GwzAMwzB1gojW2ZZziIphGIZhmD0ONnAYhmEYhtnjYAOHYRiGYZg9DjZwGIZhGIbZ42ADh2EYhmGYPY4xM3CIqJmIfkhE64iol4geJKIXKOvPJKJVRDRARLcR0QJlHRHRF4hop//zRSKisRo7wzAMwzATi7H04OQBrAfwfACTAXwcwG+IaCERzQBwvb9sGoAVAH6t7HspgAsBHAHgcAAvAvDWMRs5wzAMwzATijGrgyOE6AdwpbLoL0S0BsDRAKYDWCmEuA4AiOhKADuIaKkQYhWA1wH4ihBig7/+KwDeAuC7YzV+hmEYhmEmDg0r9EdEswEsBrASwNsBPCzXCSH6iegZAMsArPJ/P6zs/rC/zHbcS+F5fLDffvvVZewMwzDMyPj8javw3dufwRcuOgz/fUz6d/SNj27G23/+AFZ+8ly0NXtT1hV/fAw7+oq44dHNOOvg2bj5ia340ssOx8uX76vtW6q4WPKxG3HIvE48va0Pqz79Am39w+u78Oof3oPhsou3Pf8AvP/sxfjS31ahVBH4fy88GN+89Sl8+e+r8ZIj5+HWVdswXHaDfZfM7sCf33Vy6vjf/JMV2NozhHU7+zF5UgFzJ7fi4fVd+MVbjsfRC6Zq2573v//C+l0DWDKnAw4Rfvv2E/HjO9fgk39+PLjOOz58Os7+6r/wofOWYPdACV+/5SlcdeGhKJZdfP6mVSg4hO++5micclCksC/uXbMLb7zmPhQrbmS7257chm/e+jSue+sJcJxkBciX/rYK3//3GrTkHZy+dBb++NAmXP6CpXjb8w+A6wos/fhNuOLFh2BbzzC6B0u48sXW6bruNMTAIaICgJ8D+IkQYhURtQPYbmzWDaDD/7vdf62uayciEkIIdSchxNUArgaA5cuXa+sYhmGYxvLAc7sBAKu29Gba/v9ueQoA8NyuARw8txMAsGLdbqzc1AMAuPmJrQCAJy3H6x8uwxXAYxu9bYUQUOWba3b0o3eojLxDeNw/3gPrulCseIbM45u9ZTc+ugXFiouLj90Pk1sLuG/truA60pDjA4CeoTLW7xoEAKzd0R8xcOQ9eeC5rmDZE5v167zjqR0YLFV8o2eWv18PhksuhBDoL7p4eluf1cB5Znsf+obLePnR++C6+zfgGWW7xzf14P51u1GsuGhxconXtHJTD1xXoGeojBsf3aKNs1hxUay4uOovT+CYRdPQNVDMdJ/qwZhnURGRA+CnAIoALvMX9wHoNDbtBNAbs74TQJ9p3DAMwzDjm57BEgCg2m9vV9nBtq/tcATdE2HuJ/y9Wgs5yOlEQIR/+9tX/D/e/vwDcPkLluKkA6ZXPX4T1zhA3HRmLla9K3KdK7zrbynkgtdJx3rTKYsi25nXnIQQ3j0Dwntju7dCiFHfp9EwpgaOn/n0QwCzAVwkhCj5q1bCExDL7doAHOAvj6z3/14JhmEYZkLRO1QGED+hm0iPi7q5aRyY68Od9Zfmfq4fcXIcCgwkIaD9DQAV3xKQzp9wTCOfvc094w5lGis5xQPlBsaFgCsEck7yuKRBl/e3U++HPI/t3kbHJAJDS96buP2E1fQcG8bag/MdAAcDuEAIMags/z2AQ4noIiJqAfAJAI/4AmMAuBbA+4loPhHNA/ABANeM4bgZhmGYGtA75HtwMm6fIgcJyDYx2/fJOaQYC/GTfWjgINh2pJhGSNyhTAPBcdR14TiECA2XNGPJsVRZkftkvaScY/eOmYao66JhjGUdnAXwUruPBLCFiPr8n0uEENsBXATgMwB2AzgOwCuV3b8H4M8AHgXwGIAb/GUMwzDMBMF1BfqGPQ9OFoMECI0J3duQcRqOCUmZq3MOKRO8CGZp8yzScyNDX6PxTZiXEOsBMUNUmgcn3FcoHpzY+6MYdOZ28u/MHhzDSJL3Vr3HqjesEYxlmvg6RByG2vqbASyNWScAfMj/YRiGYSYg/cVyMClntVECY0LTi0S3s4VlojoX+z45sntwosaF/tvbf2Q1Z+O8SSbmdUnjhChc5/pjluGrWA2O/9uxbGeG5ZIQAsgZ7hHpqTHfp0ZKZblVA8MwDDMmSP0NUH2ISt3eqsGx7BvR3ERee79VD44rROiNMENUCI0Ldf+RkGZ8mWOUqBocdcyqLibNG2Tz4IQi42wenFyMB0c7JvYikTHDMAyz96IZOJldOJbJ2LJZFqMn6sHxfuccUkIs8XoUJ9DgyBBV7UTG8SJdnTBMpoSF/FBQPkVkrGqO5H7BearwrAkB5HK6geMa90x6wvYmkTHDMAyzlyIFxkD2EFUYDgqX2UNU0WXpHpxwwldDLGGIyjhoHUXGcd6guHRyonDMUoMTenDizun9DkJUyobVaHCEQNSDYzOQxOju0WhhA4dhGIYZE3oUAyezyNj/rRoEWUNU5kJz4g81KUrKtXIuc3vHFBmPJkTl2o2tCMbiskxZh+7Bcd30LCq5OGcxhITxO3HsijEV7G8JcQmI7ILwOsAGDsMwDDMm6CGqbPtQQkqzviy6MOLJiISoQg+OMJZZNg+MrcCDU8MQlYhJpzYNBLUmj6kbcizhPO0cZojKyHhK2tcce1SDox/HO1Zjs6jYwGEYhmHGhB7fwGnKOZkFuoEHR1mWtdBfaojKH4RDpFXydS3eCLmd9zv+nFmJepOShcGSwMABaWEhV3jGoEN2Y089lhy/rZJxFotELSqoLlN3DzxhHKJiGIZh9nSkBqezNZ/Z+xFkLCmzcVYNTooDJ3idz5Eiko3P/AkK/SGl3kwGsmpwzPskDRyQnrkkhGe4EFHqsWyGkPwri+HpncsuMjbF4ByiYhiGYfZ4SmVvsmvO5zI/2YcZSyE2D4XNYErTuQRp4ooHx3XVyV7fPshgsqSuV4t5CXGGgGlwlDVhcLiv10hU1xPFnZMgDSHlWK70wGQIUQmBfC5Og6Ofj0NUDMMwzB5PxU/7UVsjpCGnUc2DY9kui+chLiPJcVQPjhqiMsYS6UWVfs6sY8layVjeQ4IuhpbVhZM9OB6O78GxiYwzeXAQ9eCEES7V08Z1cBiGYZi9gIqv3XAo+5O9nEcrqrdhhBqcqMjY+53TNDih28HcPcyiQrjtCImEy+KMEmNFWREZq+nsAukaHHk/iCwenBjdUdxxIr2ogpumLuMQFcMwDLMXUHE9YyLJy2AijYpKmgbHFqKKhIHM194CNYvKO5a+XhLJohoLD47xWhMZy22ELzKGL5iOOad6CocQCSeZ28Rhq4MT1uTRt2MPDsMwDLPH49VP0fsopWFvthndLkvqeCRE5f9WQ2ay7YG6XhJmUdWj2aZ9u+Q08XDMQgg45I3N1B6ZEOmZY9547GE5+5j0ruaAUpNHDVE1VIHDBg7DMAwzRpQrAnnH8fUj2fYJPTjqUluIKt2DY24RV8k4OFSsBkfffySY401L7ZZohf4CkbH3N5GX3xVfydjXHBF5hpBFg5NVZBxNE4+O13U5RMUwDMPsBbi+l4GIqn66r1iyh1SyBHhMz0agwVEL/UFPv1YJ08T1/UdCxPjK6MGJa68Q3tv0jCwvi8r0igltmyRsaeKqNylYBg5RMQzDMHsBFTcUGbsxlXtNZMaSrfO1il1knHzsoLKvmiYuROjNMccSdBMffbPNrBock1IQotIL/UkPjuNQqjdIenrsGpwMHhxYRMamB0dI8TN7cBiGYZg9HJlF5Qlks018ch5N8+DYDIT0Zpv+OdS0dW3S17d3jBDVaLwTI9fguMEYTA+OFBmnFvqDcc3K+bOl24d9r+LGCYTi50bBBg7DMAwzJri+B0fto5RGUAcnzYNj2TfNiJDr847iDVGOH/Hg1LDZZpoA2hyjRGqRyNhGho0civcshR4caQip28UIj2LGHglRWa4jqSr0WDCmBg4RXUZEK4homIiuUZZfQkR9ys8AEQkiOtpffyURlYxt9h/LsTMMwzCjo+yKqtPEKRAZqwaOZcMMXp04o8KJZFHJ9frxIh6cUYWo9NdJtWtUW6KixPZCkbHfbNNJadUgwvBWXJp4Vg1OFpGx9/feE6LaBOAqAD9SFwohfi6EaJc/AN4B4FkADyib/VrdRgjx7NgNm2EYhhktrivg+BqcrBOfLURlt2/SdTlxRoVj6lliKv1JY6sWzTbN8cYbJUBzPpyqAw+O4oGRoSACaRWOo+fUqzGr54yr3hx3HMcwcOSOoR9I+IX+0o9XL/JjeTIhxPUAQETLAeyTsOnrAFwrRlMmkmEYhhlXBBocqmbii4qMrXobi2jZ3CySmg34tWOMiTkqx9G8KLVotpk1i8ozcHIYKnkXqGpw1LCQ9PR4dXDijyUvI9JsU/EGpY9dRAv9WfaX9XkaxbjT4BDRAgCnArjWWHUBEe0iopVE9PaE/S/1w2Artm/fXtexMgzDMNkJs6jiM31Mwpoz4TLbrvZKxlGDxlwf1o7RvSHm/tp0PqaVjAWaFA+OWgdHDasFWVRJzTYhtGKFNpFxJg+OJUQVFPozZD2N9FKMOwMHwGsB/FsIsUZZ9hsABwOYCeAtAD5BRBfbdhZCXC2EWC6EWD5z5sz6j5ZhGIbJREVqcJDdgyOnUT2LKj0cBcQX9lP3Car/aiGa6GStimpNgW1WtN0i4bN4YbAeogp1NPIYXqG/sMZQfEaWXstHL/QXGktpSG+Rej2h10gPJaZVVa4n49XA+Ym6QAjxuBBikxCiIoT4D4D/A/CyhoyOYRiGGRHSg4OEfkkmtqrBtn2zpI7bNDkEAhTBrdoBW91eD1HZj59GIRdOueq+967Zhc/c8IR1H1cIzcCxeXAEZLiN4DjJWVRqLR89My37dciMLdVes2l4ZBPQRjGmGpw0iOgkAPMA/DZlUwHDY8gwDMOMb1w/vTip47WJvdmmXWYcWZISBhKKbkXV3aihHwkpFs5I6uDkHELeIRSDc4frXvG9u6z7SAOhOZ8Lr8EithZCr4MTq+dBmJHlOFFjBMhmtAmh1twx77HxdwMtnLFOE88TUQuAHIAcEbUQkWpkvQ7A74QQvcZ+LyGiqeRxLIB3A/jj2I2cYRiGGS3Sg1NNLyo5Iad2E8+wLPIaYe0YTc9iG4fy90iabTqk61aypmO7Apje3oQT9p8OQPHgmIX+XET0RNEDhvfT1EFVpcFB2M8q3F96k0JPjoCo2stVS8Y6RPUxAIMALgfwav/vjwGAb/i8AkZ4yueVAJ4G0AtPfPwFIYRtO4ZhGGacUlZFxhnNA1urBqsGx7JvWqaS6/paEkTbHpjbOxYPTjWTt0MUG6KKw/UHk3MI//vKIwGohl44ZtcNU8CTKhl7Xh5VZByuE8o2WcZFBM3qs90z0WCR8ViniV8J4MqYdUMApsSsswqKGYZhmImDq6aJZ+1F5f9Wu4nbjZkRhKgQ6lZUPYvpjQAMgXBw/LTRhzhEgQfHzECKwxVSJxSev6ymiQciY+GfI7nZplBFxjHNNrNckpqxZe5v6noa6MAZXxochmEYZs9FZlEhoZ2Aib3ZZnQ72zLTk2FLzSYj80gouhF1f3sWVfbZW2pwAKCQi2pXbAj/P0/Qq2uR1IJ+Mhwkw0ZJhf7UNHEzpRuIGoU2ZDaWKjMO9UDKIcXeFaJiGIZh9lJc1xO3VpMmblYyTpq8I8ssHht9ve8dUdaqmT/q/qrPxVabJw1Vg1PIORnFvPC1NdH7oBZLlMUJTcG0iRQiy+uxenAyXZMICiSqY9W3CH8aBRs4DMMwzJhQdl3kHcczELKKjP3fcmJP67OkLUvZRgjZOkLx4MAerrFVMq4qROWEGpxCzslcUM/T1oSC3jBNPNQxeaEsv2hhSohK3lAzAyoMd6WPS6bX20TGkWM20MJhA4dhGIYZEyoCgUGRNURlil9jPTjWEJVp0Jjr4ac7q+GeuDo4aohKjq2KEJWvwZGenGyGhAhSspOyyWTml5OhkWngwSH9+gSS76+KLCqYJjKWLSQaBRs4DMMwzJjgugI50sMrqfv426V6cKytGtJei4hR4Ar7ZK+GY4IQVUahtLePp8HJO05iQ0wVAaWonlEPSAtRiTDcllRjSHqsvOsxm23qv5Nwhe5V8pZFvV6NDlGxyJhhGIYZE4JeVG72XlRyu4olBKJvF7+vxJZFZYZ1PGFsuF6ienBC1U41ImNfaJyjRJ2MOV6zLYIqMgb0exK2nYg7ntJs09HvT+CByXBNYWPPcJmZ0SWXcbNNhmEYZtzw2b8+ga/+Y3XNjyvTxL2/s+0T1npJ3iFLf6ro67CSMVSjxjJZq5N5EKKqYu7OESGfc5Ru6hk8OG60qF5Y6E/1OmVvtikNtagGpwqRsa/BUY0++/4i8/tcD9iDwzAMw2jcs2YXOpprPz3ohf6yIT0Ksg5OVR4c4yxRkbFfO0Y9rhaiUrdWC/1VLzIOQ1SU6GUxx+/6sSczi0q9Hi/TKpryHjme4sExM9mEsk0arrBkURl/qZ6wRsEeHIZhGEbDdesjDnVdqXnJHrpwDcPGtpvigLHuG7y2aHAIBMfRvSG2tgWaBsf/XV2Iivwfxx9vllAQAKnBgenBMQwUEbadSEqll8aZ2WzTlgUVh/DPr3pwwv3VMUW1TGMJGzgMwzCMRrlOBk5FVN+LSo4jFBlHdyx4LbRj95WYRoUrPTiKweEJe6MHUyU4jj9zpl2DKVKWHhwzgylpf1m7JhQ2hwaO2iAzzLZKaLYplGabZhaVRXcURyDO1o6t/1aP1ShPDhs4DMMwjIbnwan9cStu9b2owklSaK9VvLRriwbHeG3rTUW+d0T12qjeHInWiwrR6srWsaseIMfT4IQi42wenNBb4i0rK24pVdjrKh6cxFYNyvWo70FVHhwBP2ymenDkOuWYKcUZ6w0bOAzDMIxGxfcI1Py4fquGanpRCWPiFZb98jm7psdW2M98LbOBzPN468NtVW+FfJF2h9Rj5TJqcPJKLEzW5FFFxtKTpeqYZGq740RDT/r16t3Ere9BJs9SaEypYzV3F8bvsYYNHIZhGEaj3h4cqkZkLMJ9Abt2xasMbPHgWDw22muExoMZovFaNoQ76IX+somM1dWy2aYM0cUZIWojToGwvUK0krEttV16cOLGI4LjxDfbzKLBCcNh5rVq12XxhI0lbOAwDMMwGvXS4LjCKzSXtdCd3AdQs6ii28RVBs7ebFPPopL7as02ldkyEBmnXIMW4nIIB8/pwMFzO2NF0YDuwVHTv82WFd75w7HK6sJJ4S+tDo6h1QmOlcGzJsNhqi7JvH/qnw2ybzhNnGEYhtGp1NGDI0M02UXG8ne8niPv2D1CWVo1RCsZh+fRQ1Rqmrh/vJSxm1lY7z9nCQDg7K/eHmuE5HMOgEqwvxAxhf6UUJTnbaLUZpvSWJLXYPfgpGOOSb1WNSzVyDYNABs4DMMwjIHMyqk1ZSVNPOvkJ7dKatWQz9nTkswlUYPHn6iV17puRPHAaGniGUNUyno19JSkQYp4cCAzvYxWDcrovEXp91Zo1xPXbDP9fUlq1TDSY9YDNnAYhmEYjUod6+BUXehPhqgSNCIFx8kkMrZmUUHX1KhiY3V7a7PNtCwqw6BQ/47TuuRzqsg4rNUjz1tRz6kZEJTebFOEobZIHR3lnInXJEIDy9FcONH97QUTx44RaXCIqJWIziKiBVXudxkRrSCiYSK6Rlm+kIgEEfUpPx9X1hMRfYGIdvo/XyS9MQjDMAxTIyquqKqRZObjCr9VQzUenEAbEj9ZxqaJRxYZHhzf6yENFukxkfuqBoylFVVqGE/T8KgeoMQsqnBa9sYQGiUOEcoVGaLSix/KsSYV+tONJd3IzFqUT6426+BYm21GZTljSiYPjm+M3CuE+DYRNQG4F8AyAEUi+i8hxI0Zz7cJwFUAzgXQalk/RQhRtiy/FMCFAI6Ad6/+AeBZAN/NeF6GYRgmIxVRLw+OMjFm1uD4HpwUA8e2PCoqjhmPoqmxOEgA6GniQYgq5SJUY0ELUSHekFA9OKEXKdTNyDo4aiZaUJGZklPQBdQ0cX0MWY0Rud7U4NhqBzU6RJXVg3MugLv9v18MoAPAHABX+j+ZEEJcL4T4A4CdmUfo8ToAXxFCbBBCbATwFQCvr/IYDMMwTAYqrqhLWKHsuqHIOOM+prbDWsk458QYOPrrqMg47N8E6BlKMoNJ4lhCVGkXERvicuLDNrlI6rUIzkdK7Ro1TVx6nmRNn6yF/kbSqsHsXB6ONWohTZQQ1VQA2/y/zwPwOyHENgC/AnBIDcezjog2ENGPiWiGsnwZgIeV1w/7yyIQ0aV+GGzF9u3bazg0hmGYvYN69KKStVqcKrppe/t5v8M6OFGyFvqLtm7QxbKm90F9rXorKNg+bfDhn6rh4lVOtu+cc6IeHLkrwV7JWI5VipFjs6gQGiWqsaQONatwmiJp4vI4atjMOPgYk9XA2QLgUCLKwfPm3OwvbwdQqsE4dgA4BsACAEfD8xD9XFnfDqBbed0NoN2mwxFCXC2EWC6EWD5z5swaDI1hGGbvoh51cOQEmKPq0sQDA8fQ4qjkHXvtF5vHRl+v93nSaswE//PQWjUEIa1s3g5AN1ySivF5aeLh+L36NqFuxg0MDGjbqf2h4t472Zlc7q97cKJjTromIrvIWDWaqmn/UA+yZlH9CMCv4WloKgBu8ZcfB2DVaAchhOgDsMJ/uZWILgOwmYg6hRA9APoAdCq7dALoE41qcMEwDLMHY4ZnaoE0HnIOEidh21gAu2EjyarBia73wkWByNiYnOP2DrOoEg+va3gMD1DoMdEPYrZq0IvzKcdWdnOFAAkK+kPFjss4lrZdlW84gazdxLVDGr/HmkwGjhDiU0S0EsB+AK4TQhT9VWUAX6jDuAIdk/97JTyB8b3+6yP8ZQzDMEyNqUeauDxeznESwygmcrPEbuI5x+pNMZfYPDhOxhCV5q3I2GwzzoNDisC3YhhuushY1uoJw0q2a3OFgCN8kbGTXAeHFG+Q3mwzOuaka3KU+kHqeKzC5XHuwYEQ4neWZT+p5mRElPfPmQOQI6IWeEbS0QC6ADwFT+/zdQD/FELIsNS1AN5PRH+Fdx8/AOAb1ZybYRiGSUd6DWpdybisenASUplNgiwqJSXaJB/bqsHQ4Ljmen2SNmvMqC91D0ywSSLxIuXQwCubBo7Wi0q2aoiOQZJzKDBIpQYnSWQsD29mW2UVBIcaHL19RVIl5HpUxc5CrIFDRK/NehAhxLUZN/0YgCuU168G8EkATwL4LIBZAHrgpYFfrGz3PQD7A3jUf/0DfxnDMAxTQ5I8JbU4rtSJZD16kGqcMK6ck7HZprkeusg4kkWVUKjPesDI+eP2D68jauCEVoPr+mOUaeKWY+eIUIHQUsXj3jq1Dk5Eg5PQ68s8hrweLYsqwVuTpYFnPUjy4HzLeN0EoABA2sAOPIHxMDwPSypCiCsRn1b+y4T9BIAP+T8MwzBMnUjylIwGN/DgVCsy1g0b225xvahSs6iECFKrzfUCZpq38nfM8SLnV/7Oac06Qy9LuaK7lbRKxgg9M4CXgWYe2/FbV7lCpm7H212qNyjSbDPYJvs1ydFIL5LZv8u60xgSm0UlhOiQPwBeCeARAKcAaPF/TgHwEIBXjcE4GYZhmDEgfJKvsQfHP16+yjRxaWTILCpzAp4/pdXvRRW/b4Aw18v+TVEPjhBC217VvwQOnIzeDiCahRUXoso5ulfENErUdYCefo6ULCp1qVkvR2Q0bIUbjkXeEzkGV9htmUaFqLKmiX8ZwLuFEHcKIcr+z50A3guv6B7DMAyzB1AxPCY1O64MUTmkZRGlIcMbtlYNczpbcOflZ0SK1gX7pnpwQq+HOkZvWz20ooaHgt5VaWNXNThamriiwanoR8mbBg7CNHF1DEGoyEg/j7sX8nhaHRybIDitOrO/Xq1kLI0yT781fkJUWQ2chQD6LcsH4GVWMQzDMHsAlYo0cGp8XBmi8gvEZRYZu/r+trBRXFgm0prB4sFR051N0a0uErYdP/kakjw4oQbHCFGpGhxf8B3WrgmPIe9H3jCczAJ+KjIkF4xBK/SX7X2X61UNjjRwXGH3ADUoiSqzgXMPgK8T0Xy5wP/7awhbODAMwzATnFCDUz8PTqQGSwJys1CDEzUa4tLO5aIfvX55ZF8gDP/YCv1BmJWMqw9RqesVaY3WqsL04OSMXlTQvC7hdqqmKRgXkpttCtjDXd65sl5T6MFxFAPTO77dVzPee1G9GcB0AGuJaC0RrQWwFl7W01vqMzSGYRhmrLF5SmqBnOTyDiW2KjAx68XYvBNxuhO576IZ7f4YzPW6J8Ks7GtqVsK/5YuUcE5MiCrJg1MwhMSelyk6BmmIatlZTkqzTaF2E7drcLIUR/SugYJjhSGqmCyqBnlwstbB2QTgeQBOB7AU3ufpcQA3czVhhmGYPYd6ldcvKx4HSsj0iRuPTRsURHNiPEJBcUGpmbFocvJEgQGht2rQQ1QE3UDx9k8ee1yaOSlGSFRkrLZqEFr/KPUYMvnKLCCYXOjPrINj0+CkXJP04MCmwWmcMWMj1cDx+091AzhCCPF3AH+v+6gYhmGYhlAOPCW1nank8aTHJHuauL6/SuiNsAhklH2lzWCri6N6cEyRsXYuLU08nNCTUI+hZjs5FO6cJjKWHc+986rHtoWoCEC8gNtVYlREZqE/ec40kbG8BpsGxx6kGrchKiFEBcA6eHVwGIZhmD0YW7ZSLagYE3K1vaisHhxl4rdNzEGDz5hzSuMh9MgoBo4bak3U3+p5UzOObN4myJCa93fEgxNp1RDqf2y9n8wmnokaHCXcZQq9A41TqtEW3pfAg6OE+GwhxPEuMv40gM8T0Yx6DoZhGIZpLGNRydipIkYlhyGzu7SwkSK+tTmcpAEShqiix6aYOjhmeMvWTTxrxpG5v9oHKlLoz4kaMYEGR5mxK27UwKEgTTx+TGrmmebBkZ6yNA+O9IqREjqTISrY39ZGRa2yanD+B8AiABuJaAOMlHEhxOG1HhjDMAwz9oSekuTtHt3QjRXrduENJy3KdlwlrbmaQn9yq7LF8ArFt2T1pgTF8GI8OF6zTXslYzXrC67QPDgIQlSpipXgL90QCVO5k1s16EJiVQcUZFEZhpMpHtZGI3Q9jzsaDw4oqsFx4zxpjTFxsho4v63rKBiGYZhxQVYPzgXfvAMAqjZwcjJNPON4ghCVDJ0p69TwkVVkbHg5zG1ks81Qg6OvA5I9OOljD/+OrYNjanCUEJU0NlWvi7nONJw848l+d/WMLKNVQ5UeHC9E5WtwgsKH9lYN4zqLSgjxyXoPhGEYhmk8lTppcNTKu0kdr03kZuVgXIoHJzAaYnpR+b/DEIrhwYFs1eC9jrRqgL3Any2tPGns6j7mMc00cT1TSh+DrsHxfkfr4MQLuGXdH+9Y9jTxNMIQFUV6ZJnVn5W9Mh271mTV4DAMwzB7AfXT4Hi/w0rG2faLeHA0o8H7HVcZWa2947021ru6J0ILUSkGmXcu3ZAwx5I0dnWs8lhxHhx1u1DQG2qNwrHr4Su5b5K8SUAEx4pvtpl8TWqrBnnufOAhs3twxnUvKiJqIqJPEtFqIhoioor6U+9BMgzDMGNDverghJoWT79RbZq4tVWDUrQuaWKN87h4WVShJ8KmwZEeEi1NXGZRpRkDynpTDCzHZo5J3U4ahbbqw7YQlawSHffeyZCcdyyzsGG29129p2q4S66z+m/Gs4EDL4vqdfAaa7oAPgjgWwB2AnhHfYbGMAzDjDVliyFRC9SsJM/LkDUkYnpw1BCV/zuu9ouRBm6baB0KDSU9RIVgvN52upgXSA+86EUJda1MXOVgvd+UZ+HI8anRMlNALdcnNduEFqIaZaE/VYOjhAAnYrPNVwB4mxDiewAqAP4ohHg3gCsAnF2vwTEMwzBji2vRoWTdPnE7xeNQTS+qsF6Mq70GzDRxW4hKFxHbKhkTQg+OauBoWVTQjQvzmrIwvS0sJaeGh4wscS0rKmyLEO5nYhUZx9o3YasGM7Ve/llVqwYzi0rEib0TD1k3sho4s+G1ZgCAPgBT/L9vAnBOjcfEMAzDNIikar42sk7y8riyh1H2NPHQo+S6eqVcte2ANXtHiojla2MbIfyQmRJiCcZr1MGxNdtMc0zIa/zCRYfhNScsCPdHfEhIy5SKiIyj58gZ40ou9BfW0jFrEcl90t8WEVyDYxg4XiVj2x7j24PzHIB5/t9PAzjX//sEAINZT0ZElxHRCiIaJqJrlOXHE9E/iGgXEW0nouuIaK6y/koiKhFRn/Kzf9bzMgzDMNmoCNXASZ+YKlkNFSWkUl2auH4u9XTqhB9XyVhvpmmuF5onQjPuEoyLMESVfBVySDM7mtGczyn7x3tM1FCWWnPGG0OyB0dea5xhKj1WcgzWEFUVGpygVYNSSHE8NdvMauD8HsCZ/t//B+CTRLQGwDUAflDF+TYBuArAj4zlUwFcDWAhgAUAegH82Njm10KIduXn2SrOyzAMw2TAVs03iayTl+aNSEhljh5fDxtpBo4SbrEdzhVejCqsPKxvJYwQVlKISvWsZK9krBso4f6hB6tiHMSWJm6rgxNsb7RqiMsoA7x7pIa7RlLoT6+DI8cQHmMi1sH5iPL3b4loPYCTAKwWQvwl68mEENcDABEtB7CPsvxGdTsi+iaA27Mel2EYhqkNNqFt1u2TCOrgKFlLQgirV0JFHUPZ1UWsYS+qGINJyEnfLgoWxnj0Cd/7HWRRqd3EMzbblKvNSyQKdSnm/Ssohf7KSljPdhxAb+3gVRdO0OAY+iW7Bse+L4L1oaEaERnHeXDGeYhKQwhxjxDiq9UYN1VyKoCVxrIL/BDWSiJ6e9yORHSpHwZbsX379joNj2EYZs/E9lSfRNYQlZ5ebA8ZpY2nUjGmyiDDKb4Ojp5KbhEZx3h4Ir2olNmy2mabphGnemnMYb/4iHl408mLvDEYYTKbyNiskKwaj5HxKNub2wVNVjOG3QBSjhUaONaeYOM5ROXXv/keEV2samPqAREdDuAT8FLRJb8BcDCAmQDeAuATRHSxbX8hxNVCiOVCiOUzZ86s51AZhmH2OGztCpIQGTNkwkrGatp2hhCYOjYR48GJCVEJoRsHthCVQ8nNNtVU9ICMISp5OjO0pIqMTQNxVmcLXr58H20dKfuZaB6cBL2RNx4RuR653Yg8OP4yNU08xpHWELJ6cL4EoA3AFwFsIKIn62HwENGBAG4E8B4hxL/lciHE40KITUKIihDiP/B0QC+r1XkZhmEYj4qS01tbD07oEQk9IBn2cwWact5UVXZdbadg4o+p/SJFxt42UU+C9OAkaXCshf4QWGjJY5d6FcM00dPEo8cIjA8zVT1VZGxvHCqRmiO5rbpdsHnG91MtkKhWip5wzTaFEN8H8H0gMEJOg1f/5ifwjKSsTTtjIaIFAG4G8GkhxE/ThgS7McswDMOMAtWDk8U7M6I08ZjKwjaE8BpQFiveMXQPTrzxEhw/8PJEtSne6tD8sIWownPohgSQvSie6cFxnOTKwWZdHtMo0Y+lZ1El3VutVYPRgDQsPJh8TaoHxwxRTUiRMQAQkQPgGHjGzRnwRMYbAfyzimPk/XPmAOSIqAVAGV6dnVsBfEsI8V3Lfi8B8C8AXf4Y3g3g/2U9L8MwDJONatPEsxb6U9PEs7Y6ADwjopBzAFRQrtjTxGMrGUMxUBDVlwghtIlaC8/J3lnWLCrdwxJHsDpimITGlu0YQdq69ABZjCxJThtXctVmNWRn6o7CUFU2DQ4RguvSRMbjKEiVVYNzA4DdAH4JYKn/e5kQYpEQ4g1VnO9j8OrmXA7g1f7fHwPwZgD7A7hCrXWj7PdKePV3egFcC+ALQoifVHFehmEYJgOuRYeSuL2xSf9wGbeu2hrZTk0Tz5qFJMcgM4tMD46WXRTjwVHToqMhKt0oUI07s9eTalrIv1M9ONC9QBLvkFKDE93PNKCypomrGhy7RyscvbmdWlAxCbUBqBPxBo2vZptZPThnw/Oe3AjgNgD/FELsqPZkQogrAVwZs/qTCftZBcUMwzBMbSlrBk769qYG58O/ewR/eWQzbvnA83HAzHblWOFknzULSe5XCDQ4+h6q+NauwRFa1pDpLZHrA2+GtdCfbhCof2euGWMsV4vx2Tw4gUfJDJNZlBn5qjQ4ImIsRTw4adfk/yblGEE3cdiNvnGdRQVgMoBXwfPivBee0PhRIvo6Ef1XvQbHMAzDjC2j7UX17PZ+AMBgsaJvF2RRKQZFlhCVQGDgeIX+VA+O99she4hKD8lEtxHQPRG2NhWOakVB/zvNwxVkURmuF6JkDY7c2tTg2Org6M021Swq+9hCPY+xnaHFiUMoRpc0uHLKseIMq0aQVWQ8CE8AfDMQCI0/CuDtAN4JT1PDMAzDTHB0DU769pHUa/+3ORkHhfOUiTFTmrgvMga8LCp1F11kbPPgAHJKV42K8Nh+HRzLtZghKrPeTBbCSsY6WhaVVWSsh6hsXiSJmUVl66slkWnx+hj9df47lxp2UzQ4QV+rtGab4zlERUSz4ImLT/d/LwawDcDv4IWsGIZhmD2Aals1mGnOQXE7Y1rXNDgZs5C8bQQK/kzquvpkGYqM444llGrH0clXenjkBG3vRRXV4GQOUQXjTPDgJIqM9RiX1YNTRaE/U5OkbpcUMtOPoe4f9eDYaFQl46wanC3+z7/g1aD5pxBiVd1GxTAMwzSEag2cuPnQnIyD0IYTpjJnS0MHCnnFg6N1E5feGXurBtdVvB8O2SsZI9QE2TLIrB4cY5v4sesi4XB/tQ5OdD9TE5TkwTFFxuHYoscVyrFDQ0j+zurBCb1SQRFFxYNjNZDGswcHwCFs0DAMw+z5VNuLypzk4yb1ijJZVysyzjuhBsc2f6qVkVVviemxiHYTl5O0HhKS6+R+5vVk9kAZx5CobSPsdXCkQZdBg2OIn9WMpshwhNK6IijOZ3hwUo02OZZQ75NXQ1QJ+4w1mUTG0rghouVE9N9E1Oa/bvNr2zAMw4xbeoZKWHj5DbjliWj6MqNTbS8qW/sDIBqiUkW7SV4GEyGgVDLWRcZmdlEkBAU9xGTX4CgeHK0OTuhxAsxCf9lCVHEaHC9EFW4T8fAYIuwgFGfT4CiFcEjZJs2Do26n3tP0zLDQgJXHyinGkrX+zjivgzObiO4BcC+AX8ArzAcAXwXwlTqNjWEYpiY8tbUXAPDN254e0/P2DZfx/C/dhgee2z2m5x0NtkyirNsD8V4NNYtK9TJs6xnCy7/7H9y/LnqP5GSaV+rgWAv9BQaB3ZvkbWPJohJSt+If3yYyTtC/ZM6iinhwwtYSFVdoXhh1e7Oasq0OjubBcVKabYrQsFG3UzdNzaJSxhhptgn7PRnvaeJfg6fBmQ5gQFl+HYBzaj0ohmGYWmKGG8aKh57rwrqdA/jy354c0/OOhnKVGhxzk7j0Z3WyVj04H/7dI7hv7W78beUWbfuf3r0O3/7nMwCg1cHRjBb/d2zrBKFk+lB08pYeHmsdnIgGJ9yv6iyqiIcmNLZcYUkj93+H7S305SpVN9v0/1a3UzdNe8fVawo9OOE6e4hqfIuMzwRwphBit+EiewbAfjUfFcMwTA2JK7hW9/PCPsGNZ6qtgxNpFhmj5dDSxIMnfoHVW72i9b1DJW37j//hseDvsA5OXJq4PWSkFvrzUslt6+0enGgWlRoKite5qMi1thCUqsGJdBs36vKQ4SlR0UTGSCn0h6ix5AqBr/w9NMCze6UQ0fMIAau7pkEOnMwenFYARcvymQCGajcchmGY2iNinqTrf17vt60C7Xil2jo4Zh2XwDNhZAep3ohQFOx3CAewsSt+KpGtGsoVs1WDvp3pP3CFXtjO5lWiGA+O2U1cNUJCIyJ2yN54YlLmHcXYsoWogjTxEYiMg07kMaEiM6xUrojAUya3SSI8bqhdygUGpl0E3igLJ6uB8y8Ar1deCyLKAfgwgFtqPSiGYZhaEj5Jj62hEfcEP56pthdVJOwTE6KSy3MOaaLgst+MaVPXYOw58r4HxwyBaL2okEVkbI7dN7gsRkE0i0oPBdnOZxJ4O4yZ1tTgmCGq2FYNqWni8ffCuyYRMZaKFb3idNo7HmpwwnPrrRqiRxjvIaoPAbidiI4B0AxPWLwMXguHk+o0NoZhmJoQl80yZuedQBaOpsHJUKfGrOMidzc9O3I7M01cnm/j7sFImrek4O9gZlGZehKrHkjJGooYQMJvb+BEr0UVRXv7h+vkn2nZQUEWlPHJU40tIYRWjdi7Hn//DM029UrGyc02hQgHL7cbKrnGNmkhqvAzHdx/mUXlxmRRjWcPjhDicQCHAfgPgL8DaIEnMD5KCPFM0r4MwzB7LQ3S/owGW7G7JKKtGrzXtqJ6gF7J2BWeB4MIGCxV0DWg63Akei+qcLlZZThicKghGccmMtY1OFqrBhmiCjQruqfEu0brcLXjy2tWCT1AAhVFJxSsh67BcSxjkOQND46jCH5thLoZ73WxYho4SVdkaHD8ccsQ1c7+Ih7Z0B3dJ/mQdSPVg0NEBQB3AHitEOKK+g+JYRimxhjhhrE77UQXGVe3vbpP1LPj931SRcZCoFRxcdCsdqze2odHN3bj1MUzI+cIKhlXdI1HNE08ek7VOLAJn9WsLmuzTWsWVTaRsVnHRqJ6WSpu9HNJ0qOk6F1UCjlCqaKHr+Q1JjXbFMr9kNsVy/oblbnQn6rB8a3Ad/z8gZh9GmPipHpwhBAlAIvQOCOMYRhmVMRNNPWmUdlbo8EWpknCpmux7euKUExLyrYVV+DUg2aiozmPPz28yXoOtZKxrVWD2VdJPWfgsaDoJKYaXYDde2WrZCxfp+pVYkKUat0eL0RlHFuOwfDgSEF2k7JDXin0l9Zs0xXR6zANnHQPTmi0B2niKf+wxnWICsBPALylngNhGIapF43ypAQGzgRy4VQU4U2mZptxImNjhlU9FaqXoewKTGrO46xDZuO2Vdus52jKq3VwlBXGbTVHK5QJ3daqwVsfeiKsWVQxKdpqR/A44gxctW5PxY2GqEKRsb+/zHjyx9RcyEW2lduFxqPFgwPV4PN+lyqmByf5mtSHBXkMU0MUZXyLjNsAXEJEZwO4H0C/ulII8e5aD4xhGKZWhLqBsQ5ReUwc86b6NPFIiCpmXyFEoPugwCPhbZR3CDPamzBQ1DN6JIWgkrE+GYcZQXEiY8W4JLMlQSg8N+vOePvqImPzTSSkG4ChBsf04IQGnk2DExiApgfHt3hUD44uMtarCkfGYxh8gMWDkzGPSg3tpf27Gte9qAAcDOABALsB7A9PcCx/Ds16MiK6jIhWENEwEV1jrDuTiFYR0QAR3UZEC5R1RERfIKKd/s8XaSI9EjEM01AapQFoVP2d0aA+0Gcp9BfXiyqaRSUi4R7pPcjnCPmcE4RgTGSIyvTgRETGFm+SqjlRV6tp4NZu4kEdHP1ckiwhKnk5tvCWNz7vPKYHJFIHx7hfzQXFwDE8OOmF/nRNUbUiY9WDQxk9OI0KUWXy4AghTq/R+TYBuArAufCKBwIAiGgGgOsBvBnAnwF8GsCvARzvb3IpgAsBHAHvPfoHgGcBfLdG42IYZg8m8KSMsaURTsYTx8KxddROItqLyg9RxQh6Ab3IHOB5cPIOaSnqKgWlFxUpj+Vm+nR6HZyoB0etyKuFqGTdHsOICs+dIUQFOT67h0YI777EGjiGhkfe6+a83YOjGh02W1GIsLGn3G646jTx8BrCDLUUA2ech6hqghDiegAgouUA9lFWvRTASiHEdf76KwHsIKKlfifz1wH4ihBig7/+K/A0QWzgMAyTijqZjSVxvYjGM9X2olJtkpd95z/Y2jPsLXdNAyfakkB6D3KOg7zjBKJjc8LPK2niqrEQadUQGZvQQjI2D44qlq1Y1odZVIYRgixZVPb1pKyv+GP84ztPUtoo6AaX3L7kv26KMXCcNA+OCI/V4nuBeofLxpgTL0mrKZVVZNyoENWYGjgJLAPwsHwhhOgnomf85avM9f7fy2wHIqJL4Xl8sN9+3CaLYZgw7DLWdkbZmKAmAno13+pCVCuUjuDWlG3DWLjk+/cA8Dw0Mhuo7LrIOTltX7XZZl5ZFWpwvN/REJVe7Vjz4ATCc9IK1QXjNXtRWcJMqXfHMJIkWvdtv1XDEftOUdZ7v00PTlmGqJSb0NqU0/ZTO6/bhiOvo63Zm/67B0uRbTJckhYOy6eGqBpj4WTV4NSbdgDdxrJuAB0x67sBtNt0OEKIq4UQy4UQy2fOjNZTYBhm78NsWjh253X9847paUeFOjFmqoPjbxRX2E99nTOMhcGSJyrO+SEqIAxbqTQpk7ZWydjw4CTWwTGyqNRQi3x71Gs3e1GZnx3PI5TNgxMNb4XrXWHT4OhGSmDwWEJUHS2hn4JAgTFoZkcBUpPkHWySbxj1GAZOerPN0CsZ6HnG2jWakfFi4PQB6DSWdQLojVnfCaBPNMosZBhmQtGoVg2h52h8TgA2qq1kLCddU6xqSxOP64qddyiY5G06nJwiMlaHpCRIAYhqPdSQjHf46LWp6c7qtf/gjjXaWCNGCjI02wzObTdghKvfFwTbG2P0zy6NFjVE1dlSUI4bervM9wPQm4+2x3hw0lw4mmEoQ1QplsS4K/RHRLcS0RT/79cSUXMdx7ESnoBYnrsNwAH+8sh6/++VYBiGyYCZjTJ2552AHpyKagSkby8nL7OnkS1NXE6E5u3IO04YhrJMzA55RlDFdfVu4vJ3aOHo54SeuWX34ER7P6nEZ1Gli4zjNFhqLy7XVugvxoMjjb+4NHHVwClZPGFqr69JTXYDJ72ScfiwYIrG4xiPhf5OAjDJ//vH8BprjgoiyhNRC4AcgBwRtRBRHsDvARxKRBf56z8B4BFfYAwA1wJ4PxHNJ6J5AD4A4JrRjodhmL2DMPQwtpZGuUGG1WiouheVb48Ml/UaNlnSxCX5XLIHh8ibyMuu7qMJvSsxISpXr/siLNdGoOBjYbteM6ymjiktOygs9KgvDwsd+qJqyweESMmEkgaOb7Sohf7UXR2iwPgplS0hKuXvtmbvGBENThUenFCDk+zCGY9p4qsAfJaIboN3e19BRD22DYUQ12Y838cAXKG8fjWATwohriSiiwB8E8DPANwD4JXKdt+DV3/nUf/1D/xlDMMwqTTOg6OHGCYCei8q+8ykLpeGTFq6sZomboZkcg4FqeB2A8czgCoVoXtwAuPFP6cZokLosYhocJRjmJWDVWy9qACZRRXdXju/sL//pgbHpmFxiIJ7G6TVu1Jk7GjbqX8X/FndrsEJt2/J50AEdA0UtW0ye3AovK7xGqJKMnDeDuD/ALwE3mfh84gpjgjPw5KKEOJKAFfGrLsZwNKYdQLAh/wfhmGYqggnirE9byCYnTj2jV9Z1zNIYuruGaEeGaIyPDi2NPGEEFWgs7GGqHwDR5gaHN27YqtkrBYi1rKo3PDYgYFjueA4o4wyiIxDDY6+XK287FoqGcvxhgYy/PH5ISrFwFH3JAJyFK/BUevgOA5hUiGH7kE9TTzNFFENQ9U7lmWfsSbW7hJC/EcIcYwQYiq8e7i/EKLD8mOKgxmGYcYVjfKkmBNUVoQQeG7nQE3HMlSqYGvPUOp2FVcEdWfinrz1bCN5/GQNjppFZRMZJ3pwIDU4ZhaVXO8bDMZ+QujNNs3xyGPIVWu298MkzKLSl9uad5oEdWxiRMRCiNgQldc7yzfMHSkyjmZRkebBCfU5Vg0O9M/ipOY8ugd1D056ob/wmuT7qDb8tO+UvLpeZM2iWgRgez0HwjAMUy8aFaIqx0xwafzugY049Uu34e5nd9ZsLG/6yX047rO3pG5XcQUKsi5MzMRkq5VjanCiWVTxGpxcjsJ2DJaJ2XG8TCoziypMAbefUwgEXiPVYAD0DCc5rk3dUQPQ1PlIyDiejVgPjqIZct1wjPrxw3shd7fVwVHvJRGhkJfGkD1EpW7f1pSLGELZNTjZPTjjLotKRQixDsAsIvoUEf2WiK4jok8S0ew6j49hGGbUBE/CDaqDU21o7MHndgMAntrWV7Ox3Pl0NmOp4goUfA9B3NO8zcCJenBsxobdg1NwHEVkHJ2YCV6dHE+Doy8H4g1IV/HgaKJdGB6chJkwzKIyx1RF3ybDODKzqKwhKorWb7JVMnYorCFEQGI2mnkumUllbpPlmlTDMLUXVeLa+pHJwCGikwA8DeBVAAYBDAG4BMBTRHRC/YbHMAwzehqlhRmPlYyzFKeT3pQ4D47WdduN0eBYs6j8F6YHRw1RWTw4Mouq5LqakNhs9G1emoASxiLS9g0znJIDl/GVjCldryKNqEgauPfbFd59shkIQyUXT27t1bYPPTi6BkdWJXaIlDo49hCVerGyFk5km6RrQviZDurgjNM08aytGr4M4JcA3iaEJ80iIgdeL6ivADixPsNjGIYZPY0r9Cf881e3Xz3nA1cASZKJsisCj0Dc07zqCZHz6HA5XYMTVzQvLU3cm7gJZdODYxgfSQ0+yRiT2p8sKc05rtYLUfbGlOb1hg0xBVzX7sGxjUGO38yiamvKoXuwBNI0OLZum7o3aVJzLrJJei+q8BqyVjIe1yEqAEfCa3YZ3DH/768COKoO42IYhqkZjWrVICdr05vRSGwhIBXXFYFoNNbAUUNUMR6caLPNeGPBExmHoRXTcCDywjKlimt1B6i9nVS0rCHSVcFq+Ki1KYfLTj/Qeq1x4ZcsISrp7ai2m7jtXCq6yDj04AyX3cATZq+Dozc8bbOEqFIbBCihPWkIF1JExuM6RAWv99Miy/JFALpqNhqGYZg60KhQUeDBaVQ7ZQu2JozaeiECL0DcXGcrBjhkiowj3hQlTdx4I/KKBqfiisgYZeilVHE1D4MZPrI12wyMKkcfd2h8eK8vOGKe9Vpju4lTFRoc43pb/UJ9l1//CB7d2J2q0TLP3WQYODPavUYDQ6UKco7XQsHmwXENkfGkpqgHJ+s1OUQ4d9kcfPrCQzGroyV5p3FYB0flVwB+SEQfAvAfeAbZyfBq4/yyTmNjGIapCWG67tieV+pJ0oyKOOoxXFsISKXiIt2Do6aJxxT6qyZNXNXglFwR8XhJ8exzuwZw8xPbwuXGDTJH64mMPfKOg7JbUdbpx4hLdY6rZGxmZdlQq/6qyAaZ/3lmp3W9ie3c4TrC/118JH5293M4bP5kL5PKcewaHEV0DYSeH22bFH+LGu6d2taE1xy/ABt2J5c0aJR9n9XA+RC86/mRsk8JwHcAXF6HcTEMw9SMivKlPKbn9cNBjdIg2LCJeFUqrpsqMrb1dIp4cCxp4oFmxjheIUcoVpzg/GYUra05j6acg4e26lll0vMRhnz0c5YrYbitOe9gWAmjyZ5bssBgU0w53theVMhQByfmfVc7gHvnyGbgXP+OE3H7k9sjtW9mdbTg/WcvDpYVcmRPE4duLLXZNDjJEUyr0ZYW+r3iTytx6PxOHL1gWvLBa0wmA0cIUQTwHiL6CLwmmATgaSFEbStRMQzD1IFGhYgCDc44ClGlaXAqrkBrIdmDo3pYKoEGJz1NPCyaF/XgSD1HqRL14MyZ3BLUd1FJq2Q8XK6gxQ8HNRdyKCq6lJJ/H6TnKM6DY55DXZ7Vbo16cAqJ6yNj8E3C5+03Fc/bbyp+d/+GyDqVgtQrGXh1cMLtzXEA2T046mkrKUYzAOzsK6ZuU2uyanAAAEKIASHEo0KIR9i4YRhmoiANjbG2M+Tkn+H7f8xIM7ZcEdZSia2Do6aJxxT6S0oTt2lwpIFh0+DM7mgJxqSSVsl4qOQGgtzmvKNlekkDQB43LpMqDFFZNDhpxkBMaLSzVfct2LKQzj4kLDMXNa7sf0s8vZJpYEa9mJ02Ayez0Rb+bXrvbKi6obFi7M/IMAwzxsjJdqxDRdKwSs1MGUPSQ1RqFlX8NhI5iUebbYZ/r9rSg6e398VnUSmVjEsVN+Jx62zN2w0c2D04w+UKbn58K4bLlaDqr2fghBOxvA/yuPEhKjlm49wZRMZytXm9pmFhcx59/7XLg7/jsrDkOEyaclEPTlj3J1xmhsrU7eIICySGB5o3pRUA8N6zDordT62+PFawgcMwzLhie+8wnt1euwq+QDgJj7WdURlhiKqe40zNonJFMOlnSROXxmNSs83z/vff2N47HFs0L28U+ouIjImsBkiYAq6P6yt/X403X7sCuwdKaClID05OM8JkM8p8SohKelcirRqQ3mwzrv5Ss+HNSKsjk+jBsYWoLBocYdneauCkeKXUVg2S9uY81n7+/NhMNABoLoy9uZFVZMwwDDMmnPT5W1GsuFj7+fNrdswwRNUYD8740uBk8OBU1YvK+x0t9BfdOdDgWJaraeI2zZSt1ko4yUqRsfdKbVQaeHAKeohKenCk4RRr4BhGlLo87V21eU281/EemaQxZF1XsHpwRGR7mwYnc6E/i2GVTxioadSNBezBYRhmXFG0VWAdJW6DDJzxmEWVpQ5OmgZHfYuyFvoD9LYJKvmcE5yz5LrWwoiZQlSQmVHh8eXE2pRzUFTCX9IACIrVpWpwzGuhVGNAWMI5SecwCY26pBCVzYPjoFjWB2erydNpDVGleHBkqwbLkJOywRoRosrswSGiJgCHApgFwzASQvy1xuNiGIapGaEGZ2zPO+o6OHXIa7eW8Fdw1RBVzLhtzTYHTQPHsmuYRaUvzxseHPV+HTK3EwCCBqAq0TRx/7Vq4MgQVUH2aHLR4uRCkbF/3LgwURCisqWJZ+gmnqXRaty5Wws5lCplq/4n2NfqwbGFqKLGVmfrCDw4rvQE2Tw48T6TRnhwMhk4RHQ2gJ/CM25MBICxN80YhmEy0qiKwhMxi6pcpchYzqPdgyV9m4SWCrYQVSEQGYugFsu7zzwIl566PwAllOQQ/vDOk/Cib9wRzPTyeDZ9SJAm7nsQhksuWgq5IMsoznMjkdV+ZfVhidH5wYorRKr3Bghr7Zi0NuXQM1S2GFdk/VtiD1FFj2/X4CQzXHbhkD1kmOjBaYAGJ+sZvwXgL/BaM0wC0Kr8TKrFQIioz/ipENE3/HULiUgY6z9ei/MyDLPnU2lQiKrcIMMqiTQNjuuq3cSze3C6B0qx20jk/GcaWYWcg1yQJh6GqBbNmBR0vJYT6tS2Juw3fZJ2vEiISjEI1DRxIExnLwcenGQDZHZnC37z1hNw7rI52nKvDo7Az+5eh4WX34DBYjRVulwRqX2agHhR+SS/V5R5L1U7wuxUDtgNnHDfcGfTaPPGkvz5GCx6tYVshluyBmf8ZlHNBfBZIcQ6IcSQEGJY/anFQIQQ7fIHwGwAgwCuMzabomz36Vqcd29k+VX/wE/vWtvoYTDMmCG/6xtWB6fqE9dvoJl6UeX1kE/SMeTk2+V3sw6Wx3QFB6I6K6+jt7euVAlDVOpkrKZzh54ge5q4rsEJ08SBUAwdZFGleHAIwLGLpkXquJB/vu/88xkAwI6+6FRYrLhW7ZBJXOq+9D6ZxpOeRRXFK/RnanCi2hmbkZL2EDBYqlgNIwCBkWpjPIuM/wLgxHoOxOBlALYB+PcYnnOvQAiBHX1FfPyPKxs9FIYZM2ot9r3sFw/gF/c8l7jND/79LO54ekdNz1sLslQyTvfg6Nu7rkDXQBHT25qs20ikYWJO6ERhJeOKK4LzqoZKYODknWBSDz04MqQW3S9IE/cnZWngmFlUccRFmBy/knFSBKpUcWOP/4/3nYpTDpoRbGdjZofXRDPqdUsWGTfZNDhB9lMyaXWSBkthdWiTJA9O2n2uB1lFxm8D8HMiOhrAY/D6UAUIIa6t8bheB+BaEfWVrSMiAeAfAD4ohNhh7khElwK4FAD222+/Gg9r4jOe0lUZZqyQ39m1sjP+8shm/OWRzXjVcfHfMVfd8ETw93gycLLUwUnT4Jhp4n3FMlwBTJ3UhB1+SX65jfo1ngu8NNEJXa4rV9xgjGqoSXpQiKJdxAMNjv+bKMmDU9HGEJceHmJfTyQ1Nv65LfeqVBaxHpyDZnfg/MPm4t9P7YjNHPzaK47AL+99DkfsMzlybonNM2LV4Fj2VXn3mQfh8U3dWLOj376Bz3DJRaulCzmQrMFJq/VTD7IaOOcCOBPACwEMQPefCgA1M3CIaD8AzwfwJmXxDgDHAHgIwHR4mqCf++PSEEJcDeBqAFi+fPn4+VYZJ6TF3xlmT6SWaeJmS4IsjKcHi9Q6OCK90J9Zybir33vmnaZ6cIJ07HBbObma4RNvnefFKStZVI7mwQkFxWa6ufwtAkM2PH5Ug+NN/CX/HGkhpKR5WSAMk9k8Y6WKm6jxkcLbOK/J9PZmXHZGtDqwLXSnktSqwcx++uorjkBLIYcXHjYX7/v1Q9b3RiUpRJUW7htrsho4XwbwTQBXCiGSzbvR81oAdwgh1sgFQog+ACv8l1uJ6DIAm4moUwjRU+fx7FGMpydJhhkryjUMUXUZYtosjNS+qYddlBSCEEJAiNCYiO1FZYiMuwY9r8309miISvUkSEMkLkyWz3kGThBqskzkRITmvINXHbcfTj7QC/E4gRfF209tqtmsVDIGwpYSpbLebDOOuCwoGaKS2AyDNA2OHFNa2DAyppT1Xh0cs/CifduXPm8fZT97F3KVwWK8gdMAJ00iWc2tKQC+OwbGDeAZOD9J2SbwttV5LHsc7MFhJgq1zDyqpch4V3/1XZFH6sGpR/ZVJWEyleMMNThx41L2ESIw+qZOatKWA7qxIc99yoEzsWhGW+S4BcfxWjXIEJVFg+N5cAif/a/DcMS+U/xlvkHmb6tWLFYrGXvr/CyqoJt4usjYupw8gyr0Stk9OEnak5agNk9173Na5rnVUJEanISdk7KvJIOlSmzKd5aU+LEkq4HzOwBn1XMgAEBEJwKYDyN7ioiOI6IlROQQ0XQAXwfwTyFEd73HtKcxntJVGSYJWx2VkWLTg4wUaeDEPcUKIfCXRzZpy0Zs4NTB45r0kCPXSb1L3LbqeyOEl0EFQBMZy3utTpjSyzF5UgG3/c9pkePmcoRipYI/PezdPzVE1ZQLNTgm0Wab4TnDXlRGiMofS5oGJ66NgqyDI9fadDSlSrwGBwCact5nqFSuzoOT1trBrsHxs6hS9jM9PyZDCSGq8UbWENWzAD5DRKcCeARRkfFXazSe1wG4XgjRayzfH8Bn4RUa7IEnMr64Rufcq2APDjNR8Jo+1u5YQG09OLYiaQBw02NbcNkvHtSWjdRQqYd2J+mYrqLTKOQoqBUT2U4r9CfQPeDdk9OXzsLXb31aO8+w5sFJvp684+Bnd4fZaVqIKkHLEhb6871GFYsHJ69nUcmJPK3QX3IWVVjIzxb6K5bdxBBY0GC0yhBVWuzCrsHxfieFkZos6eUmg6VKrMh4vJHVwHkjgF54qeJmurgAUBMDRwjx1pjlvwTwy1qcY29nPIkdmfHP5u5BtBZymKKEHsaKWn5Wa1nob/dAsoFjVvRVz5+VUCxb3diykKTBCUNUhLwTH64wu4n3DJUBAAfP7cTaz5+Pkz5/q1WDYx7v2EXTcO+aXcFrM81YtT3CEFV0hg7TxL3XRUUIHhEZl8IQVc6hEWf3kH8+ubftXqVpcGSbiDSjwnZuID71upAniwbH9+AkhqhGp8EZb2QycIQQi+o9EGZsYAOHqYYTPncr2ppyWPmp88b83LX0NtayVcNOPw26rdn+9Zm3TDoj9uDUwcJJ9OD4c5vjkD/ZpWdRCSGCybRZSeWW91r1ppjG1a/ecryWkmuGi+wi4+h4zErGmganoKeJD5Uq2NYz5IeP0o2bWAOIyDubv9oeonKDSsw2pIGSZlRExuRfcEuMFqYln0Ox4moeprQ0ccC7x2W/rlHcdQ8l1MEZb4yvnC6m7rCBw1RLv6UE/VhQUw9ODZttSg9OnNFiK3Y2UTQ48j7lSIYrMnhwXOGlQ+comExzDgXblJSu1mYoxlGabALRe5ezaHBsBFsFHhw1RKVnUX3yz4/j2M/egh29w6nhKe3YBo4UGfuvbTqaUooHRxp0acX1ImPyTxoXKgoF1eGYshT6k2ONq8sDAEMJdXDGG1mbbX49ab0Q4t21GQ5Tb1iDw4xnTG1HrahtiMoLQakTt4qt2Nl4yKLysn6S9R5yXS4lRBU0qswRXBGdyB2ioLhisVKJ7BeH6f1ybFlUFheE3E4e3WrgyJoz/j3d3jds7VBuEufxkK0aJLZr8wr9JWcteftWmybuHTMuVGQ2FgXs3cRNVI+SzUtTrrgoVtw9K0QF4DDjdQHAUn//B2o6IqausAeHGc+U3OyC1GqopchY9gUqxRgKqoFz/mFzkc8Rbn1i24jOVct/rgTPAEjyFshLyjkOCvn0EFVTzoHriki2kKzyCwDFBA+OScSDY6tkbNlPLpPnVD0XckI3PUA7+4qJrQXCY8dlUREEwhBQbJp4QpPJDj98tdCSMp84Jn9IcaEivWpzAYDiwUkMUclrsb/vQ/59jQuNjTeyanBON5cRUQuAH4L7RU0o2MBh6sWjG7pxwTfvwL8/dDr2nTZpRMdQv1jHa5q4dN/HPXWrXtJD5nVid3+x6muRm9fy3yv5LpykYwYhKie5Jopc3pR34Arhi2nDmTNHlEmDYxLR4NgqGWdIE7elOjsOoSkfpkHv7B/O1AgzVoLjny8pTdy8LyazOltw7RuPxZH7TUkdh3bulBBVi9F3C1BDVAkeHN8Yi3vfpXE/UTw4IzbDhBBDAD4D4KO1Gw5Tb+QT1Dirx8QA+NZtT+Obtz7V6GGMGFn75Q8PbhzxMVQdQ6VKXUIScmKtRYhKZujEhahKRngk51DEqPjMDY/jyj/FN7yV4YRaGGQS+U8+KUwtjRKHCIWEEJU8RnM+h4rwrtkMUYUaHDeyXxyml0Wt9yKNHft3lxGiqrg45aAZ+NHrl2tbHTZ/cvD3zr5ipEO45Fuvep556AjyGkdT6A8ATl08E50thcRtTOTnKT5EpRc1BNQQVfxxpTEWVwtnyM9A21tExjMBtNdiIHsDrivw9p/dj/vX7UrfuG5jaNipmRRueWIr/vnk9kYPY8TsN93z2jyb0qwvCXWSqIcHpxYOEfnlHxduUa+huZCDowhuJd//9xpc85+1seeQh6hPFlWSBkd6cCgxRCXr40xqzqFYrkQ1OA6FKduaByf5C8j0SKgenCQPhNwsDIu5WDy7A2csna1tJ1s7AN61xoWozlk2OyhaGOvxIF2DY/NOpRX6GynSkzIpTmQcZIx59/sX9zyHr9/iPTyl1cEB4j040sDZ00TG7zcXAZgL4BIAf631oPZUdvQP48bHtuC+tbux4mN1LwxtJfDgNOTs44tv3fY0DpnbidOXzmr0UAAAgyXX2hl4oiCftkdj4KiTYdJEXC0jERn/4p7nMHVSAS84bK623CwUZ1JSrKjmnIMcRT04adTSIJPIJ/fELCrVwEkMUXnbdTTnMVhyIynXjpImbqtkHEdL3jRwwr+T0pwD4ay/0XC5YvXOXHrq/tjUNYjr7t8AIL5NAyEULsd9ZsIQVYIGp5ycRTVSBn1DoznOgxOEqLzt/t/vHw3WJYWoQtGz/ZrleasJUX3g7MU4ZfHMzNvXkqwi43cZr10A2wH8GMDnajqiPZikD9ZYwRocjy3dQ/jS354EAKz9/PkNHo3HcKmCOnwXjhlywn92e9+Ij6FpcGrobZSf+2ocInJSMD8fRaPUv4kWoio4gTdDrUmSRmDg1PDfqzxUlkrGaQaOPEZ7Sx47+4oRD46aJj6shaiS31Qz9KGGqGS4LklkLCAC0bMtNNTWnMcXX3Y4/vDQxsQ6OI7f2Vy9VttYuwbCvmSxGpyECswjRTYMjTM0WoKihpb7nVIHB6itBudNpyzCpKaspkZt4UJ/DaFxRgYbOB5/9nvdzOpobvBIQgZL9qfOiYKc+Hv9qrYjQf1irbp8fQJhHZzaiYwzhajyTpAJ5AqvvozKQLFs/fKvZVo74BkHlcCjksGD47dqGLJNkAgzyNqb89iwe9DPFlKzqMI08VIVImOziaMaojp4bieO2m8KrrhgWWQ/VWQs35+4f0tEhM6WAnb2F+M9OBSeO+47s7O1gHU7+wOjzKbJyqLBGQkDRe/fWGyIyiIyliTXwfE1OHEGTornCADecsoiTG4t4Mt/X+2fr3EP9iO680SUJyLW3lSJqIPbuVrkP9bx1vV1rBBC4A8PbsQDz+0GgMQqo2PNUKky4QzQvuEybnhkM4Dk4mBZKVbRt6ga5LFGckxzH/lUXKoIqwhYDQE153OBV8527rjO5GEWVdXDtaKeOks3ccf34MRpZqSh0tFSwFCpEtGa5EjpC1VWQ1Qj9+C0FHL4/TtOwpF+B3Hbdq4IJ/WkcO/kVk/UG9dok4jwsfMPwZRJBcyMeQia3JpHz1BZMRz1aytXXLgivVv5SDjrkNloyju45LgF1vVq1WaTpEadQR2cFJFxkgfno+cfgsvOOCh43cipJvHOE9GZRPQKY9nlAPoAdBHRTUQ0pY7j26Ooh2CwWvb2Qn+3PLEN7/31Q7jxsS0AgB19ww0eUchgqTKu3p8sGTwfuf5RvPMXD2DVlh4jJXVk16Fef60MnBVrd2FT16A/rur3396rf0aKKZoSs8hckpZjd3+0bxUweg/OYxu7sa13KHI8IJsGJ+8bOMUUkXF7cx6DxUokHdpRdEeacDzlPY1qcKqbHW1tI2x0+AaOaXx88Nwlwd/nHToHD33inNiMoc6WAroHS8H9NA2csBhi7Q2cfaZOwuqrXoAlczqs621p4pLELKqU3liDIxAZj1sDB8DlAPaRL4joWHhdvX8K4EMAjgCniSfy3M6BIE4rn3pqmfpZLZWEOPbewK4B/Ym5Z6ispVI2CiEEhkpuTcMyoyWLgbFx9wAAoG+oXBPvSzWTYVZ+dvc6TG4t4IWHzYkYDNt6hrCleyhmT4+NvnEkKaZoSvQsKkfxLlg8OAN2D85oQ2ov+sYdOOsrt4fHy2g4yvOGvajihdSFHKGlkMNQ2Y3Joop6cF5/0sLEcZsF5LIaOLLx6ebuoeDfc3NCgb3JMQbOO08/MLMmb3Jrweuk7jdXNT2Y8nWWfle1xpYmLknrRQUkaXCStT82xnOI6jAAtyuvXw7gP0KItwghvgrg3QBeXK/B7Qmc+qXbcP7X7wCgf1n1DJWsXYfrjawtspdGqKzyJ9k8sZHIJ61qe9LUE9XjGDcpqjqFYhX1TuIo1SFENVCsYFZHCzpbChGD4aN/eAwf/O3DifvbDBw5gdh0F5EQFcVrOXb12z2IogaaoR5FC6W+l1nq4HganGSRcc4htBS8wnlDJbMOTliSQnqB1nzuhfjguUsTx5wUokpiwfQ2LJndgT89vCkUwiZ4GWb4KeCjMT46fSNJhhmjHpxkLVA9CQ0cmwYnKYsqWYOTJUQlufjY/QAkp6XXm7Q7PwWAWmP8JAA3Ka/vAzC/xmPa45BfkPKLRQD48G8fwQd+89CYj2U8hMkaibBYOOPBwJFfyuMpRKVOyHEp0XICqghh9BwamSeqWAcPTtEXwBJRRP+2u79o1cGoXtatiofHdb2qvVK7Zc2ciQtRudFj70oJUdVKg6N5cBKMaK0OTs6JNbhLFRcFxwkmur7hkl7J2PDgNOWcTLq/kXpwAOCCI+bi/nW7scH/vo0T4ALAEb6OZ1vvyEPU0gskMe9VKfDgNMDA8d8XmwYn6W2QGpzdMdowGaJqaUq/pqsuPBSPXnlOpL/YWJJ25s0ADgAAImoGcBSAu5T1HQDGj4hhnGF+ycunI9cV2N47jO0NmFgDkfFeGqRS7Tv53TkedDhDvis5rRDaWJLFwMkpk7fmwRmhJ6oerRqKZc/Akd2fVYbLrvUpVzVcVDe/XN7mGzhx1WslTXknyJyS16Ne47Yee3hMblKrcHZWDY6rGTgU+yRfrgjk/RAVAPQMli3NNkMDJ6unxPQM5KpwNS+e7elRVm/pBYDE1OTlC6cCAB5a35X5+CZm9WHzXknvXiMMHDVN3PwMJXnF5Fgvv/5RXP/Ahsj6oVIFRMmd3SU5h9BRZYXmWpM2yhsBfJGIzgDwBQD90HtPHQ7g6TqNbcLTZ6TLql8spYobq1SvJ+PJQ9AI1Kuf3dkCYHwYOOPdgzNcseuUghCVqFGIqjL6Y5hID4Jj8eAMlytWnYJ6LarQ1jRwTEPunK/djp/f81zw2iGKpBur53vwuS7rmOWkNBIvlq12jm7gJGRRZayDU3Zd5HOhB6d3qKRNeqq3bKBYxqSM2Ypm+rFThW0wf2orAOCpbV4dpiQPztI5nQCAeZNbs5/AwPTgXP/ARrzv1w8FrxupwcnnvBYhNgNe6pVsqN3VbVXVB4sVtBZyEyYLN+3j8wkAQwBuBvBGAG8RQqhuhzcC+EetBkNE/ySiISLq83+eVNadSUSriGiAiG4jInt+3DjCrAdSUUJUw2W3Jmm11VLLwmETEfVhRoYZbE/wY42sNzKuNDhVhKjKFf3zPFKxtDqh1uqzKkNUDkU1LUMl11rrRfdGKd6cksweygXHVlm9VS9y2GTJopLHbso5eGhDl9XAGk33c1uXczejBidIE6e0EJVAwSG0+EaEmQ6dUyoZ9w2XM5djMDU41YSo5k/RDZy25ngDJ+cQ/nTZSbjubSdkPr5JZ2v0mn6v9GGTn+VGVSdvzjsYLlciYaqpk5pi91GNMVvH8MFSZcI02gRSDBwhxA4hxKkApgKYKoT4vbHJywF8qsZjukwI0e7/LAEAIpoB4HoAHwcwDcAKAL+u8XlrTs+QHl8PvmSE78FpgIETfLlNDAO85qhf9PLLtFZhoZ6hEpZf9Q/c/ezOqveVsW05wXz+xlW49NoVNRnXSMlm4ITraxGiqoUXyHbMQINj1rQpVzBs0SkMx4wjS4gKAKa1NeHblzwPsztbIiJjeYyTDpyOYtnFYxu7I/u7oxAZq/deeoK09Psshf4cQj4hRFVxBXI5CkIhALSKvWqzzf7hcqKxodJiGANZRcaA51GZ1JTDM76B05pSPffwfaZg3pTaeXBMGqnBAbzvt+Fy1ICfMil+3KoXzpYeP1iqTJhGm0DGQn9CiG4hRORbQAixy/Do1IuXAlgphLjO72J+JYAjiChZkt9gpIEjLV5VZFyqiIaEqCrciypAPlnVaiJ9dEM3dvQV8b83r656X/mUJT0f3739Gfz98a01GddIUTUwcRNdzo8hFCu6K3ykxrveqqF2Bk5z3gtRmfZCrAZHDVFZ/o4LUUlmtjfjhX4PK8cIUcljLJrh1Uq1idylE2a0Bs6AH/p0tRBV/DHl/c8RoSmxF5UvMlbCQHHNNvuHK2jLWKp/pFlUgBcWmz+lFX3DfpXfOk/EafqSRhs4zXkHQ6VK8PAkmdIa78FRM77ylvjgUKkyYRptAqPvJl4PPkdEO4joTiI6zV+2DECQyymE6AfwjL9cg4guJaIVRLRi+/bGdmaWISoZC1a/ZEoVN7aIVj0ZRxrWhqDe8RbD8Bwt8jgj+UKTBo4rxk8YUZ0o40XG3u/hklsT78sGv64OUDsDZ7gcH6IaLnkGjinEjAu3yWtsb4rPogJ0b0bOqIMjDSoZ4hiy3NvKKDQ4aohKlqLQCyjGfwnc/exONOUcLJgxCYWcAyHsYzBFxgAsaeKjC1F95AVLq06xVj0ykzJ6jUaKTJOPo9hAkTEgQ1RuJESVqMFRxirbQagMldw9J0TVAD4MYH94qedXA/gzER0AoB2A6cfthpfFpSGEuFoIsVwIsXzmzMZ0MJUEBk6z4cHxBZnFBhSYCzw4e6sLRwtR+R6cGll98jjVVl8FoD1ljRehsWoMpGlwhssV3SgYofH+5JbeQAdQyzRxma6teqWEEEH2munFiQu3Sb1MWohKffoNq8O62rFlFo4tRCbv/UiSqNTxSi9yliyqiivwl0c24cyDZ6GzpZBY9K3susgraeIAYtPE+4vl4H6lIf9NTm9rwluff0CmfVTm+IkDOYfq0gPKxMykUpGfraY6NNvMQnM+h+FS1MBxEr6f8sq63uGogSNFxhOFcWXgCCHuEUL0CiGGhRA/AXAngBfCaw3RaWzeCaB3rMdYDb3+l4t0z7pKiKpYcROb3tWL8TJ5Ngr18mXGRq3eB3lvba7dNNQ4+XipZqxpT+IMHEcaOKYHZ2TXsGpLL5bNmwygtiEqrw6L/v57vaS8v00Dx5YaLo8FhCLjOENONQ5bg5okrnYu+SRtC5HJ74oReXCU8XYPlCLjiRPVb9g9gB19RZy2xHswTCr6Jjtxqx6MgpFFVdE0ONV5cEZa4HBGhxd+GatMH5s3pFTxPIJX3/4s2ppyWDC9re7jsNFSkCLj7P8W1XtmZgH/88ltuOvZnZGGqOOZ8T5SAU8ushJeWwgAABG1wavPs7JB48qE9ODIf7QV5amsUSLjidbMsdaok3bB8VJ4a2VQyMluJGmh49KDo6WJx4SoSDdw2ppGbjT2DJWwsWsQy+Z5zzI1NXD8rt5Cm+gr1r+913YPjqnBUf8NixiPlzQC5HsszyU1HDaDYzStGtQxyWrGmhfK4jECwoKk+06dBCA0WGxGXFjJ2B6iyil6Jy9ElVVkHGZljYTpbV5jzDiDvNbYdDhdAyXs6i/irmd34m3PPwAz2u3NOutNcyGHgWIl8CRVS5/hwXn9j+8DUF2bhkYzbgwcIppCROcSUYvfrfwSAKcC+BuA3wM4lIguIqIWeOnrjwghVjVyzGn0+PFvaRSHImMvRFV2xZjrLcZrob9SxcWtq+ovqlXDUTnHqxVRq9RsOXGNpHLnUFExcBqUKt49WMJdz4QZYJk8OGoWVcUNMlfMsN+tq7amGvQbd3sT7P4zPfFtzQr9BWnieh0c9cl22HjKjfNGDRsGzk1+01ZAN+pUr0erUVW2aGpwbCGq0YiMlYuU30HyOIUcRUSnEnn/ZT2ZpBBVqeLVwVENnCZDg1NxBcoVL4un2hDVSAsczvA7f49VCQ6bB6d7sBhon/aZNvIsrdHS3pxHf7Eca9DGIcN8qgdHfT9YZDwyCgCuArAdwA4A7wJwoRDiSSHEdgAXAfgMgN0AjgPwykYNNCvSgxPUtAh+h08oY10LZ7x4B0y+cctTeOM1K/Cv1fUVhqvXn3cIBYdqdk9kxkp+BBqcIc2D05gQ1Tt+fj8u/v7dQWg1S5q4nNMDD05zVLh91zM78cZrVuArf0/OLhs2wz81eF8qrkDFFWjK5SIi46wenGJZRJZL0eyvV6zHmh39AKA9KatGgTQCZDFHeS87muM9OGGaeLbrVFHP3V/Uv4PamvOxBs6mLq+q8pzJ3gSXlyEqy/jKfrPNOA1OU97rUdXvX3NWkbFskDlS23ZGe3yGUD2Q/ai++LLDce0bjwXgeXCk5ywtlbyetDXn0T9cXYgKAO7+f2fiZUfvo3lwegbDv8dTra40sn3qxgDfiDkmYf3NAMZ1WrhJX1F3D8svbNWoKVXcMa0rII2s8SYyXrfLy57ZGdN8sFaoX/65HCGfc2omMpZGykgMHHXSaVQYceWmHgChJyKLgSPv3XC5gmLZxRS/iJhqnOz2O2av9Q2BOOQ5Ah1GDe5DUFQv72Co5GluhBAgIs2wMCcBuV9rIadnURl1cIDQOFKNVLUJZ2DgBCEqvwBcwQmKsZlIA2cknwV1ApKTlHw/2pvzQUdok01dg5jZ0RwYGU1JIuOKi3xzXg9RKRlPUyc1YfdAEf3++bN6cKR34OXL9820vcnMMQ4HdfoeHO+z7xkzXQOl4F4kiZDrTXtzDn3DZe1zedbBszLumw8edAC94ezWmPYi45FxY+DsiUi3t+nBURlrofF49eBIk6DevUDVL+u832+nVGsPzgg0OJrI2CjUNtZl0eU9ylIHR35+i4YGRzUag/fW1spdQRoVsodQLT6rqoEj/xbCM/DVsFRcFlVbc96qwZmueArkMvV4ugfH0dbL7ZvzvoFjecIOKxmPToMjDQx5nPbmPLoGBq37beoeDKoBA4oGx5Ym7grkHdIyBlUNztS2JgyXXWz3m1lmNXCa8g6e+NR5I67+O9Z6F6nB6R0qBxWCd/UXA0OtoR6cpjz6h8tBde27PnJGZgOwoyWPvuFy8P2jGjhbJpCBM55CVHsc8slMPgHaviiSxHC1arSnMpoKqfVETuL1HpY6WeUcQt6pnQdHGjgjuYY4kfFYenOCjCLDMAfsOhFvmzAzaLjiBj2HVMNd2mdp90V2I5dhj6R6LVmRPbRkHRwg/OyrISVTpxBqbXLWxpvzp7Ticy89DEBoUKj3SP133Wp4cOTxmvIOmgu5GA+OPtZqUO99/7BeIVuGqGzfLZu6BjFvSkvwOjFE5dfBAbyUbkDX4EzzJ/v1fl2jrCJjwPPiJKUyJzHWBkWH/3nvHSphVmcziDxDUWpwOhscohooVvCjO9cA8LxqWfWB7c15uCL8zG5SDBzZiX0iwAZOHZFfkklPY3HCy1VberDkYzdh/a4B6/qRUg68STU97KgJn/Lri/rlL59AaxVTHjImsGpQJ1jV4GqEx01OuKqBE9eQVI5PanAmBcUT1XvgG68p5w3CQk3SwKl25PHHbM6pPaG8dckeHL/eTVNeez/UPlKLZrRp+2qiZVVk3GQXGTflHC+V1+LBCR5ERnAP1HsvQ1QVJURVcYX1M9o9WMZkpcptUoiq5DfbBICZvrBX9VxO9Y2e9bu8iTFrJePRIt/jtjESwh402xPEy9DezPZmbOoaDOoPNdKDo+qerrrw0KqkENIztdsvM/Ds9j405x38432n4ksvO7y2A60jbODUEfnFJycB29N43GS4ftcgihUXm7tr6w6UYxgvtVYCgqf8+k7o6nXnHAeFXC1Fxt5kMpKwY5wHpxZju/nxrTj7q7enZjHJe28a5gCwtcdu4MhjylDIJEt9GHnctLd2WNG9eOcf/WdUDVGR4cFJEhmHWpuc9h5IL11rUy6osivPob6HWogqb2pwQq9Scz5nTxOX3xmjbNXQbxo4vmZkyKLD6R0qBZoSICVE5TfbBBQDR/G6TGvzJsjndnm6q6whqlrw13efglv/57QxOde5y+bg2jcei9eesBCAV0l5Y9cgegbLaMo5DWu0Cej3XNY2ysrBc70aug+v70LFFbhp5RacctAMHDS7IwghTwTYwKkj8qk8NCqyh6jk8lrXylG7FNfbmKgGyviUP1pMD04+59TM2JOT30h6jGkGjjLGWoTPntzai6e29QXji0Oe1W7g2A1tuY1MKW2z6GdCIz6rBie7B2eoVMF/fftOPPjcbvsxlXCQY7RMUD0ucSLjtua89m90oFhG3iE0553Aw1EMPDjh/VX/aTkOoSnvhCGqQIOTixcZK1XPqyXsgUS48bEteOXVd4UGjv/+mJlUw+UKhsuulvacHKIKPTizOrywlgzLAGHH6ic2e7VYZ3e2YKw4ZF7nmJ2PiHDq4pmBFmn+1FZs6hpC92AJna2FMdfPqagNTmV9oKwcOn8yWgs53LtmF+5ftxtbe4bxkiPn13qIdYcNnDpSNDw4dpFxnHhTFyRWw7V3rcWHf/uI9ctRnbTinAO3rdqGD173sH1lnaAxilFpWVQOIe9QzYTecoIbiVEalyZei7HJ8aSOS5YukAaO//mZ1dGMbbEeHG8b6ZKfZBEZS4MpXYOjZ1Fl8eDs6BvGg891WTtyq9fSlFM1OHJc6Wnik5p0D07/cAWTmrwquU1BCwZpMIXHMNsEtOQdfO/2Z/HDO9YEYyrkyG+I6OK9v3oQ161YH2wvTzmiLCp/HxluuvvZXZHsL9PAkSUtVM2IWb/HPIf02Ozr13qReh8gnFBXbupGU94JdDp7OvMDD04pqHPUKNQQVbW1awo5B0ftNwUr1u3C45u8f1vH7z+9puMbC9jAqSPmk7DNgxM36cgvwZHoOT771yfw6xXr8RvlC1OiTp5xX553PL0D1z+4serzjoaxes5RJ14vi6r2IuORvGeDJTd4CtRDVKMfW1YDJ/Tg+J5Hf+KeN6UVW3vtHhw5vi4/Vj/ZT5VVM9MCAydlnGo4KW/0jYojyOKKMQTVY5oeHDU0ZMuias47ERG62nYg8OD4QmbZNPN1JyzAn991snY8OcF8+i+PY3tfEZ0teRB5lYCHyxX84aFN+OBvHwm2DysZp96CCPJ9lmnLAHDrE9vQlHeC0MNg0W7gqB4ceZ39Fs9f2Q1Fxm97/gF4x2kH4OJj9wvWd7TkkfMfHuZPaR2xaHiiMW9yC4plF09s6Wmo/gYYfVhw4Yw2bOkewtqdA2hryo15jaFawAZOHQmyqPwvHJvIWC0ipi3P+tRt4ZC5nQCAFWujbnv1cHEZGsWyG1QhHWvqnd2lTry5nGzVUNs08ZG8Z8OlipJirYaoauHB8Y5RivmsSSIaHCENnBZ0DZTsT/L+saUIWYYmNA9OSWaXZQtRNfmNMbO8L2nGm83AEf6mmoFjhKhkB/JCztG8aAPFSuClMjU48v68+ZT9sWSO3gdYFXj+4cGNQcXd5rxjDR0GRUFHkUU1VTFwfr1iPc5cOguz/NBN1IPjGaiy+CAQhjgGLE0XSxU36LnWUsjhQ+ct1bwEjkPB+dXMrD2dkw+agUKO8Oz2/obWwAGyF1eMo6M5j96hMtbu7MeC6W0NDbeNFDZw6oiZbluNyHg0GhzpKu63tLuvZPDgDFs6LLuuwLduezpo3lcvalWTJo6oB6d2WVRmpdqq9i1VgswF9X35+i1PYXd/cVTjqtYbWDQ8j/MmeyEIWdNEpWx4J0MDJ6rByerBafY9OFkK/QX/TmLuucxmsqWJq5lrpvG2tWcIM9qbvTpJRmVg+WRcMDQ48ni2bBX1fgyWKkG9luZ8Drss7+9oyjnIz7g5wZ6+dFZs2MnmwZFiUqsHx2+2mcRBszwjT35+9gYOnNWBL1x0OA6d34kXHDqnoWMZrQenvTmP4bKLp7b2BRmDEw02cOpIliyquC/mrBqcPz60MZLCK1ND+4btrmVJXAhAjls997+f3oEv/e1JXPGnxxLHM1Lkw8FIBLrVoNfB8UMQNRIZDwYanOonpaFSJXjiKinjue7+DfjI9Y+OalyZNTg+ZmhVlu63pYqbXr5pbdFKxra6OjaKFRdEfvo+1diDo6WJp4eoVm3pxZLZHcgbWXYDwxYPTqDBkRqi6Neq2bhQuvtbCo7VcAwfiqyXlYh8SDD/fe8/oy2syaMYLd0DJfzwjjUA9OaR8jr7LR6cspImHscLD/Mm+N11figab7z0efvgL+86Ba9UQnaNoK2K2kM2ZMbdxq5BLJg+qRZDGnPYwKkTrlJrIsmDk67Bif+S7x4s4T2/egh/MPQy0t1s+2JSn4rjnpDlhKR+6UvDQ82UqCUyi6reHdaLhgcnn6uNyLhYdrHN16mM5BoGi5XgC8UMl3QNjs6DE7QKSblOudb04EivTM+QLVShH1OmB2u1YzIaWMWyi6acAyJCLkeZBLZZNTjNeSdwscvDDhYrcChsSigZLFawdmc/ls7t8EJUyr+D/mI5yBRrjglR2Tw45r9F1YNjM+SkbTKSLCp5700vzcIZbWht0jubA8Dnb1qFW1dtA6B7cAo5B015x+oJVkXGcbzo8HmYO7kFbzxpYdXXwIye0Yao1P33mcoGzl7H/et2xad5V8Inx7IrIISwp4mnZFEleTTkF5ieniqCp0X7k5ew/q0iQ1TqtfnhdqvoUQiBu5/dOaq088CDU2cDx6xkXKhRmvjKTd0Y8oXCIxEZD5Xc4AvFzOgZbdQu8Aam1sGBdn5pYEivjM24NY0Q2YuqZPHgpDX9k7oXAMhRVgOnGg2Ot0x+TvuGy2hvzmNya0FrJrh6ay+EAJbO8Q0c5fMxUKzEhqjW7OjH5NaC1rJAIv+tySfhwMCxeHuA0PsykiwqeS9Mr9T0tqZIXyxAN6LMsFZbUw4Dhie44no9vaQGJ46pbU246yNn4sQDZ1R9Dczokd6695510Ij2V43d6RNQYAywgTNint3eh4u+cxc+9ZeV1vXyS10WPnOF3WMSZyANWyal7oEStin1SEIDx8WOvmHs6BvGYKkSTIimWxww0sRjDRz5BRl+scmn3+7BEjZ3671svFobd+OX90aztp7a2luVWDnJY1ULVGOmFpWM1+8awECxjPvW7gIAnHjA9KqNNFlZVnpwHt1gpDyP8pZk9aDIXlERD06CgWMah50tBeQdMjw4UUM8bpzSK5Jzshk4addmr4PjresdKqOjpYCOlnyQ5g4Aa3d6xekOmNnuX4uaJl4OXP85//NTrFQwWKzgbyu34LxlybqLfaZ6ehQ5YcQVghtdLypvn5ccMU9bTkRWDc5cRSPT3qI/9U9qykc8OPJej6TnGjN2EBHWfv58vPesxSPav10RnE+boGn+bOCMEOmuf8ScjHyGlVLvgDcR2NPE7V9gMuNF9eCc+PlbcOxnbwmOL2P3w+UKll91M5ZfdXNQcI3I7sFRJ400DY76BCgnvYfWd+GEz92qbb/Fr7b85JYebfmu/iLO/tq/8KHfPoK+4XJQ6dd6vTLTp84enFLEgzO6LKpTvngb3vDj+/Doxh7sM7UV8ya3pmYrmcjJRva1+YGvh5CMpJqtSuY08cCDo2dRyb5CPRYDp1QRWvpoU96JZKaZTSbjkCEqwDM+M3lwUsT4UvjbnM9FRMZ9w6XAg6Mab/LftvTGSA8s4P2bUiu5NvlZVnev2Yn+YgUvOmJu4njNAnRx5fPle5F2C3b2DUfukzTIXnPCQqz69HkAgIW+50iOXdXgVLTq3rrR0tYc9eDI9zZNZMxMbFRjV4apJxps4IwQGX+O9cBEqrKKmDTxivUY8qlXz+AIv2he9f178LLv3qWdCwB6faNmTmeLVnhLohk4VWRRJT19twRPhfp1yAZt1z+4Ee/6xQP48O/ixbJBQ9IqDZxqM5Y0D05udM025f27Z80u9AyWML2tCYU8VW2kBQZOiz1mPtrU+azGY9hXqqK9bm3yKu7aDJyKKyKTtldbSDFwLJV+bRSVEFVBqfybRNjN3H6Puv3Qk6w7A6gGThntLXl0tha0a5MPBm3N+WASL1UEXFdgQEnnB8Iu5Wu2e14fWaLBZKmfNn7comkAgAXTvKwU1YOjGgyVDFlU3QMlHH3Vzfji31Zpy8uuJ9bOOV6dnYevOAc3vufU4Hx5h7TWG0n3ua056sGR/17SQlTMxEbV4LAHZy+iVHGDtMp4I0GGqMLS9bZtv3nbM1j8sRsj7n/pBYgL2dy/bnd4LsWwkCX1Z3e2oFhxI3oO1RsQJz2xPXGbwld1spJf0kPGubYpxeHufGYnnt3eZz8hwqfOagS/q7f24pBP3ISnt/Vm3kf1ruQcZ1QiY/XeygJwhZxTtQZHTjCqS1ilZhqcBM9SueIGn0/5vssQZt4hzwgYsnlwXMydrBs4XuZR1Ps3lMWD43+WFs1owzO+0ZC4j+VBQKVrsIiOljzyOSVE5W/aN6RqcHQDh8h7OMkH/ZhcDJUrECL8Nw14xtxw2cXanf3oaMnHTgTXve0E3Hn5GXjF8n1x03tPwckHeboU1YOjVj9OSkwAPN3M7gFPfH7DI5u1daWKQEExPia3FoIaNY5DOOWgGbjpsc3B+ztYqqCQI9x5+RmR87Q15SN1euS/F/bg7NmoD1yNLlo4UtjAGQHv/fVDuPj7dwNIF+rK7sqVit3Akam3ajt6INQO2DwUpphXNSwe3+SFieb4T9WmFycuRHXln1bi4I/f5I89qsExjZe7nt2J5336H9jWMxRMZtFaIuFTYrHsWut9SLIKYVWe3taHsivw9LboRHjbqm1YePkNkW7sJUODUxhFmrjq5u/zDRwvZFGtB8fb3tQ/SEbbMyxLTSVbyrT8bDsORcI4krLFg5N39OJ4gUcwgwZHGjhL5nTg6W29qfcyCOXGbNc9UAoq+pohql7fg2NeW9+wlylFRIFguFQWwb8l1YPT7Htw1u4cwMKEYmgdLQXMn9IKIsLSOaGXR/X4yH+a6vsd99Z/4LqHcdqX/2ndxusTFW98XHjUfGzqHsJDG7oAeJ+/WR0tmD8lWq9mUlMuEuruUzxczJ6L6sExQ5cThXFj4BBRMxH9kIjWEVEvET1IRC/w1y0kIkFEfcrPxxs1VtWtHBv7D5r1+dVpYzw4ErOYW5J40syOUL0rj/o9eWTtEvPLSQ0dqOO55j9rMViqoG+4HAlRfeT6R/GJP+pi6vvW7MKu/iKe2zUQTNCDhpfHbNC4s78YO1nLybSaOjjSYLIZTr++zxM8mxopM4sqN4pCf6prv7/oeQPMyrdZGAo8OHY9xuhDVOkGjmqcXnvXOnznn8/oHpyWfMTAEcL7TJt9hrziidE08UweHN+gOHhOJ0oVgTU7kr04aSLj3QNFTPF7MuWMOjh9Q2V0NOfR2VJAf7ESjFkVEgchKtcNNGSaBifvGbRrd/Rj4QiKoR2535Tg7+FyJbinkrjSEtc/EN9KpVRxrZlckuftNxUAsHqL5/kcLFWstXsAz4gxPThB1eMGV+pl6sukKvtXjUfGjYEDIA9gPYDnA5gM4OMAfkNEC5Vtpggh2v2fTzdgjAB0t3JsiEpmUflfhhU3miauWsVmr5+kp24zO2q4HH5ByaaD8qna3Fb12qhjl2NZvbU3Uujvl/c+FxmD9Dj1FyvB5Gg+oW81GjQWy641swuovhjdV/7+JL54k6c9kK56FZkRZD5QqxNvIUcojKJVg6o56h+uoK05h0LOQSXFmDVJDVGNUncdVBtOMLxMo/kLN60K9sv5Hhw1lVo9rjmZxomM1fYfv71/A17/43u1/dQQlWx18MRmXbhuEnr+7NfWNRh6cKR35eN/fAzfvPUpJU3c+zcqxcXee+ktkzqTckVgmy/qVwuoFXKEPz28Cc/tGsCCadXXClGNJVf4Wh/lUmzG7X+e2am9Nh8aSm5yleF5U1rRlHOwxs8WGy5VYsXOk5pykeQAW9VjZs9jIrZmMBk3n1AhRD+AK5VFfyGiNQCOBnB/QwYVQ0s+/DKIe1oPsqgCD44b+bJqbw6fird0R40BwB6yMb0yw2UXBcfBENzgiXfO5GbrtuqTujoJL5g2Cc/u6Meqzb3Bubf2DOGtP11hvb6NvoEzMFwOJnrzCX1bzxCmTCoEjRgBz9tie/KrVoPz+wc3BhOS6sH508ObsGZ7f+C2N/+Jar2oHAf5UTTbVO+lDFEV8mHBwpyT7Qko8ODUSWScpVWDaeCo5835GhxTEyPfs3zOwTcuPiroOSQzj2zHHiq7aM85+B9Lt/rhiovJTd5n48BZ7ZjUlMODz3Vhe+8wOlry+O9j9ovsk1YvqnugFIReZJbanU/vxJ1Pe0aCFBkDXpbYtLamwPDxriV8P6+64QlMmVTA8oXTguM3Kd7ci47exzqGNH7xluPwlb+vxv3rdmO4XNEMRtt7b3ZON7coK32ibOQcwr7TWrFuhxe+HSxVgvRxk7bmfOShJPTgjJvpg2GsjCcPjgYRzQawGIAaG1lHRBuI6MdEZK0eRUSXEtEKIlqxffv2uoxNLc4Vp98Is6i8LwHX1fU6bU05rduvGc5JatVgfuEMlSpB3Rx5Clnbwty2f7gchAHUL08pjly1pScY+3dvfxZ/W7nVen2burzx9hcrgT6n1whhbO0dwv6G235Hn12HI+9jFg1OxRVBajqgh/fe/csH8bWbVwdf+uZDiOohCioZj9iDExo4xbKL9qZ8cG+r0eFs2O0ZizYNBDCy7uQq1YaoJFIfliPPg7N7oKgZxfI9yzuEC46Yh6MXTAteayEq5TO8u7+Ij/7+Ue31R65/BF0DRS1EVcg5OGq/Kbh3zS5cdcMTsRl4cRlif3lkE357/wZ0DZaCFFdbsTIpMgaA9/z6Ibiu8EJUTXoxv/vX7cbD67vwwXOXBEX6gFAY/NZT9x9xv54TD5iBC4+aD8DzCmq1qiwfzXU7dUMzqsERqTVqFk5vC+r9DBYrWqNMlUlNuciY5INFo5tJMvXnM/91KH7yxmMbPYwRMy4NHCIqAPg5gJ8IIVYB2AHgGAAL4Hl0Ovz1EYQQVwshlgshls+cObMu41M9OHFu/0CD06R4cJQviWntTUHVV8DLONrRN4yv/WM1/vzwpsANbZuUTOHwUMnVJpHOlnxgPJnbDhQrwZNXxfKUvX7XQLDc1ntIIj04f3xoI/6+cgsALxzwq3ufw6MbuiGEwNodA1hqpM3GCY3lRJXFm7Ktd0gzFnf6x9TFmTLlWT9epJJxhjTx392/AQ8852Wt/WbFety+2jOczfRamUWlXk8WVm3uQXtzPqhVYjJkaXZYDVnSxG0enJWbekDkiYyXL5yG3qEyfnznmmB96MHRJ9N8LioyltHYt/70fvz8njDkeeczO/DLe9fjbyu3oFiuaPq2YxZOwxNbkkNUcaHcy37xIP7nuofRNVAM/i2ohomkoyUfeBQfXt+Frb1DgTdOvbbf3r8BTTkHLzpcL54njc/RVnoNMhFLFe3BwxbqXLtjILJMxQtRJX+1L5jehnU7ByCEwFDJRXPebuDIe9OntOnoZQNnr+GS4xbg+YvrM4+OBePOx0hEDoCfAigCuAwAhBB9AGSsZCsRXQZgMxF1CiGSvwHrgCrISyswJlNKTV3G9LZmLfVua88wPnL9o/jH47rHxDZRmmGn3mHdczKtrSn48jFTe/uHy5jW1oSd/UVNjyPj7Nssjf+S+PdTO4K/uwaKuNxvDPnvD52OvuEyDp03Wdt+Z4zRJL0BWQwDM+NManC2KF4weatVAbYQQssMk5WMZZVpJyZT4AN+OOVnbzoOH/rtIzh+/2l4/uKZWhYV4HkD5D2txoOzaksvFs9uj21eOGAYUnc/uxOuEDjxgGwl8EMPTvy9HbBoox58bjdyvgvsgsPn4vv/ehZ/f3wr3nzK/t7xpAfHGHfBkia+eHYHVm3pxeOGpkYWq7xv7W4tiwoADt9nsuadEEJEdAFpGhxXhCmudg9OAUtmdwSve4fKvmBcioy98axYtwvPWzAlki4rdUk246kapAZmuOwGDyVNeccaolq7sx8z2puDBxBhBKlKZTe1T9T+M9swWKpgY9cghkrxHpwp/vV2DRYx2TcUZYgqLqTKMOOFceXBIe/b64cAZgO4SAgR19kxiECMycAM1KfM+DRxGaKyZ1HNaG/SQlQbdw9i42594gaiZfOBaNjJFH9ObWsKQk6qx0QI4XtwvPOqHiWpo7F1Ns6KeitW+RkaS+d2aNvsjPHgSG+AfCLe2TeMW56wh8c2GPdJXqM8JxDeI9XLMlx2rZWMgej7ePvq7ZGWFB+47iEAwGY/PGZqjlQPTtYChEIIPLm1N/B02br2mlksr7z6brzq+/dkFjJn6UxvGsL7z2hDqSIC8TkRYXZnc/D0DoSfSXMybSmEqcUrN3VjU/cQFs/WPwcSOUnft3aXFqICgPlT9HvRZelKHV5bvJdLekqb87mIbqSjJY/Jkwq45g3HAPAmb1VkLD8fQyU3kg4PhPdt1AaO4sGR4ep5k1si7VT6h8vY1juMw+ZH08slqgcqjoP9f5dPbunFUKkSnN9Efkep9753qIy2ptyETR1m9h7GlYED4DsADgZwgRAimF2I6DgiWkJEDhFNB/B1AP8UQtj7JNSZLFlUphu34grNYzK9rTl4OgI878PO/qhxEaTYqinJhoEjhcry4XbaJK+p3qSmnKZPKVa8dhGdrdEQlTQEqvXgxLHKf1I3J7b4EJUearjkB/fgTT9ZEfGSAKH+RyKvcZ2SUiwLt5lCYCA0OvO+yBjQtVTD5Qpe96N78V/f+o+2/9aeYRy3aBo2dw3BdUUkdNTWnMM8Pz3/Tw9vsl6nSc9gGV0DpUCrdPsHT49sUyy71s/ZPc/ujCx7amtvtKZShjo48jN0xD6TccDMNhzjC2kdxWPS0VIInt4BJURlTHSzO1uCz9Hn/upluh2lpEOr7Oj137udA9jaM6x5cKRoWWL7bJYyiNNlh3MAmGkYInIC7wg8nmVNZKxmOdmMGHnfRh2iCjw4oYEzZ3JLxHiRSQSHzQ89o+bncFPXYKyeSyL/Xf7y3vXY1D0U78HxjUM1U7FnsMQp4syEYNwYOES0AMBbARwJYItS7+YSAPsDuAlAL4DHAAwDuLhRYzVTKuXkM1As4xm/Wu/m7kF0toQCxrKRJj6tvQmT/S8PWQnWTKsG7J2B41KtZ3d4x5FfSlMnNWGX8sUke8pIo+tRJRvDZkjEkSV7cM2Ofsyb3KIViwLiQ1SmTkR6Y7oGowbRpq7B4LgtBQf9xQp6hkrYbTxlAp7hJoTAYxu7g2XSsMw5FEzO6gT59DbvPdzSM6R5LIiAc5bNQbHiNTc1ix+2N+dxwgHTcdbBs3D1v56NvzkK8vrSer1IA1Q1dP799I7Idmd/7V848fO3asuyaHCkF/Bnbz4Ot3zgNBy9cKp2XsDTdqn3Qx7P1HvM7mjG1p4hPLqhG+t3D+DcZbPxhpMWBesPmtUe/L3d+DyoXk1zEjWF+EC2RqLzlMneNFJky4RO37Ozu98TO0sPyLzJ8fsCocDXNJyqRXpQhksutvqG3LzJrZE+ZDJt/mglk6u/WA40Z0IIbOwajBiHJvLe3ux7SeOyqOT7odZA8pqUcniKGf+MGwNHCLFOCEFCiBal1k27EOLnQohfCiEWCSHahBBzhRCvFUJsadRYzQ7AUr/y7l8+iDO/cjtKFRcbdw9i3pTWwI1bMUTGpxw0IxAg75dQP6Nk8eCok4zK7E7vS1Y+BU9ra8Lu/iJ29xfRP1wODCP55XTVDU+gd6gEIUSmvj+So/adEllmFoXa0V/EzA79S7+jOW8NUe3uLwaeAbNRZddAKZI1talrEPtOm4SnP/MCfOtVzwPgFS1Tv4Rl6GOo5OLHd67Fi75xB/7xuPeRkYZlPhdWqt3ZNxyMYdXmMNSlFpqb3tYUCIE3dg1GjMK2Zq/67bGLpqF7sGRtbWAiDYvOlFLo8jOmevm2dkcnfMlwuRK0ysiib+oeLMGhsDmsradSR0sBfcPliIC7yfj3MLuzBUMlFxd88w6s8yv8AsCx/qQ8R2ntsKNvWDOYzzkkvhu3NHB29A0H/x5szTbN9iSqN8P0tEw2PDhSxyUNnNmTm2P3VZk6yl49QT+3cgXbeobgkHefyhX9e+PJLb1ozjs4dJ4eopL/fnf1FzFcdlM9OABw1sGzg7+b4wycVj1ENVyu4LldA6mfV4YZD4wbA2ciYXpwpEbiX6u9J+ot3UPY6LuJpYeg7LdqWDqnAw98/GyceMCMwPjZZ2q8gRP08VEMkLgwjxT9yQljalsTdvUXcdSn/4ELvnFHME5VS7Bh92AwUc2xaAxMHvz42Th8nymR5WZGxYZdA5Ev/X2nTcJOS5r4xd+/O/C+lCouuhVPTNdACT/497M4/nO3YO2OfvQNl8N7m3MC7cqqLb3oUrxVaoPHH/mZP1K7o3pw5HtwxlduxzGfuRkA8OTW0MD51+qw1EBLIRd4AzZ2DUaMQulVktuYoSIhRBBe7PcNBWmUpfV6kYbQNsXLZxaHVLPI3vnzB3DsZ27BQLGcyYPTPVhCZ2shEFofqHhZJB0teVRcEXyOpF7LNGRndeqv5f249k3H4tErz9FCWtt7h9HWlA8+e4fOjxpWkk1dQxgolrH8qpvxxmvu065J1ReZmYOqJ2jBdHsqtwzbys+IfAhQs4tsXprD9/FCRWlZS2nI0hObu4ewtWcIMzuaMbvTC1GpXtgnt/Zi8eyOiIEhH15kduO8DAbONy4+Cqf6GTJxHpzJioFTrri45Pv34PHNPZEwOcOMR9jAGQGmB0dOUjK7av2uAW8Snqp6cDwDJ+dQIACWX/RqZdQlhmZFTk5qiCpOCCy/ZOX8Mb2tCZv8p/xnd/QHXYGft99UfPFlhwPwJmHpiZg9Od3AmdrWFJnQAGCS0Wbg2R39mGaEXfabNgm7+vVaKsWyi9WKQVGsuHjY75EDAN2DxeD1Xx/bjEOv+BtWbenFfN8FP29yCzpa8li1pQddg6XIe9MzWAomLandmeprMvKKyBjwvD1rd/TjofVdgdhXHUtT3sH8qd7EsX7XYKR7epqB853bn8GyK/6GlZu6seyKv+FX960PvDyqgWPTbsqGovKY8ya3REKa6nhufmIbAOCQT/wtWFaq2LU8gCeWVY1UW2XbDiMzL2js2qF/bkwxrvQmtBRy6GgpIKcUodveO4xJTTnc+J5TcN9Hz4pkSUlDa05nC7528+rgesIyClENTl+MhxMA3nXGgfjt206ILG8teKLZlX4vt30tDx22ENUv33I87v7ImbHny4q89x/9/WP4zYoNmN3ZEnhk1dDcat/AMQ0qadRtqsLAaW3KaeFCG/mcg47mPLoGi/jjQ5uwwm/yO1G7SzN7F2zgjADTnXvO1/6Ft/50RSDUe9UP7kHvUBnzprQGdTTKvshYfXqd48f3501pxY3vOQUrPnZWJHZu8+A8FdM9W34py5DX1ElNmjEkn7ramvM4bYn35LZJ8UTM6cymI5hhcdXbJs6oB6cVW3qGcMgnbgqaYG7YPaAJKcsVgb88Egp0dw+UMKfTu0+/8ftLAeEXuNe8sAOrt/Rh90AJ+0zVv9jXKc02N+z2/p6s9CYyK76e9uV/4t41u3DhkfMxra1J0yk153PobClgdmczntraGymOJ/UK8wMvj+5hkf2x/vSQd303PbYlMI6lBwFAZExEnofqsY3duPSnXlHvw/aZHNGk2JphqtyzZhcWf+xG/PupaAHM7sFSqhdJejVO+NytuG/trkD0a3psTAPHnGzVubnsCrQ152MN59++7QT87b2nWrPLvLT/aIjKLJug0tacx8G+1+8YX2cEeJ+jjpY8HvUN2oUzLAZOR/Rz39ac10JuI2XelFb88HXLA6NmTmcLZvn3UXrtShUX23qHI59xIEzdTisaaSI9Z9t648OdU9q8auR3PrMD09ua8Pf3nYr//e8js10YwzQQNnBGgK0x3d9Wbo0Uy5rd2Rw8rUoPjlpr5dxls3H1a47Gm09ehIPndmJGezNmGU/DpYqL9bsG8F/f/k9wTLNkvuTCo+bhe685Gm862atTomaPAMBrfuj1/pnUlMOMtmY05Rxs6Bq0hq4kqrZGGmfT26IT0XApGv6QT3lyYpTXNlx28dE/PAYAQTVVyWCpghsf24IXHuZpMboGSsGX99qdobEyX/mS32fqJGzuGUT3QDES7lOrvgYhqknSg+OgoHh8jlWEmxceNR8Lpk/S0mP3n+mFN5bO6cSqLbqBQxT2bpnZ3oxCjrBx9yCuf2ADXvzNO1BxRVAg8t61uwB4qegf8esGqcaFWThv4fQ2rNrcqzUOXTZvMnqHylqfoDTNz7qdXhHHvz66ObKuZ7CkGVkAcOflZ+AP7zwpeK2GRX55z3PY2jOEya2FiLdntmHwmJOtmV6c1NRvyqQmLJnTEeh4VLb3DQeGTdkVgVZF9eDYxLBtzXn87E3H4QevO0Zb3tGShys8D63qldp3mjd+2+e+lpx58Gxc/46TcMUFh+DDL1ga/Hvc2jOEPz60EQd99EYIYf93KktMrN7ai+ltTZk1QfIz3ZQQYpvS2oSugSJWrN2N5QunYvHsjsD4YpjxDEvhR0Bc1U91snnb8w/Aucvm4FnfGJF1cHKKC56IcM4yXVT5ttMOwP4z2/C5G7302mLFxb+UJ+4lczqxtcfegqI5n8O5yvHivuTamvNwHMLcKS3Y1DUUTNS2L0452jefvAivPn5B7HFt7QSkgfPXd5+C1Vt7tSynf63ejk1dg5GqrNILccbS2bj5iW3oGiyiy+KZUA2CWZ3N2NozjNZCDiceqE+mamsIqVOYKg2cHGGW4jW45Pj98JVXHIH71u7CohltWDS9DQ8+1wUAuOKCQ4JeQ0vndOCuZ3Zi0cxw0lU7ajsOYe7kVqzfNYAnt/TgkQ3dXr0RX/wqj6miaiDMtOslszuwcnM3lvnC0i+//IjgfdnWM4yFM/LavUvjl/euR89gGR84ZzHe9+uH8MPXH4PuwVLEEzF/SqtmnKjGwubuIXQW8xFjBvBSq799yfOw79RJWLerPxDySszwRltT+tfQXEtW0JNbejXPTcl10ezkgvf53WcehJfH9Ic6+aBokcSO5gKAQSyc3qY9iPzmrSfg4fVdETF1PZg/pTXIOAt7wg3j+/8Os/LkPf/VpcejkHNw0Xf+EzTOfHJLb6T2VBJnLJ2FL77scLzo8Lmx20yZVMATm3uxpWcIrz1hQdXXxDCNgj04I8DmwQH0yfTD5y3BpKa8lkVVdkVqcaxFM9rwwsPCL5tS2QUp9QyXzon/8jLHdfYhs/H6ExfinacfoC2Xmp/5U1rx54e9nj1AaOA05R1c+8Zj8fM3Hxd4Jc47dA4W+rVabE/Fw6UKfvPWE/Cj1y8PlsnU532nTcKZB88OnrBleOzPD2/SPCzq+JfO6cCU1gK6B0roHihpWoEPnrtEq+I7u6MFxbKL7sESZrQ1Wd8fGVbLO4QLj5qPKy44BDPam7X+QTPbm7HvtEl46fO8SVEVpL72hIWBTmLp3A4UKy5ueCT0hJiT9vIFU3H76u2BVuSuZ3dGChSqqNoTqa+44oJDcMO7T8ZJB07H+l2D+NdT2zFlUgEvO3qfwBj5v1uewv/evBpAWPsnCfne3fDoZnz/38/i4Q3duG/NLvQMlVNL73cq7/ujG7uxZke/1SgGgBceNheH7TM50toAAD583lJ86Lwlwftk6rds2B4qnts1oBk4l/v9qqSBc+GR87BvFR2+pYbKDIfNndyK8w6NNwDqRVPewfS2JmztHdK8hfKeH7//dBy9YCpmtDdj3Q7PO/fk1l4snRMv1DYhIrxi+b5avR+T8w+biy09Q+hsyVvfT4YZr7CBMwJUl/x1bzsBpyhPg8csnIr/e+WRwYQVZFH5LvQs1T/3mdqK95x5EM7za66o8XFThKxiTgKzOlpw5YuX4QNnL8HbTwuNHPnE/PoTFwIAfuH3Blo4fRLeftoBuOk9p+DUxTNx0oEzAtNKveYDZ7bjXWcciG9f8jx84OzFwfUdu2gazlg6O/BGmJP++YfPxRtOWoj/e+VRWDqnA3c8vQNrlLCT+sV84Kx2dLYW8Kv71uPetbtwwMx2fPLFy3DtG4/FO08/ULuP6iQ7eVITCoaGpZCjIMTR3pLHrI6W4ClZ9eDMMDQgRy8INRrq+Z6/eBbOXDpL29YMX7zkqPnoGy4HGpHfP7gBFVcEhQCTkAbOCw6di2XzJuP8w+ch7xDuW7s7CJ1IndXvH9yI/735KQDZPDj/74UH44IjvEnqNys8w/aJzT0ZNTjh+v5iGau39kVCqlnoaCngHacdiPZm73hpVXcB4NXH7xeELSWbuga1Fg2/f3AjtvWGtYuqbSXwrF8S4AXGeRrJrM4WrN81oNXQMjVPC6dPwpqd/Vi3sx9DJRdLEh6CRsJ/H7Mv/uecxfjua46uid6IYcYKNnBGgNkQ8DMXHha8vvjY/fCSI+cHr+XE+K/V27Fi3e5MBg4R4X1nL8YZB89CqSKCYlwAcKhSwfT8w+fi0lP3t45LxXEIHz5vafBaGiDnLJuDFx8xLwgvtTbl8OHzlmL/mUpmhT9ctdKp4xA+cM4SvPCwubjsjAPx5pMX4VeXHh+sl1/AbcaTeVtzHldcsAyTWwtYvnAqHli3G89s6wvS2lWBdUshFxTcA7zMp9eduDBIa1VRwyRTJxVw2D56/6tZHS1B8UOz8KDqOZluGGQnHDA9ci7AM9x++Ppj8LHzD8Zy3wiaZgivTzpgOl58xDyctmQmjl00DY9t9LJz3n3mQdZjqvzkjcfitScsCIyvaW1NOHaRpw+S93afqa3atfQOlQID55xDZuMv7zrZeuzjFk3DNy4+Cs9fPDMQhv96xXoUyy4OSjCeAd1zd9WFh+LYhdPwoiNG7tmQ2pu2BA1OeO4CvqYIW6dMKuA3Kzbg4fVd2nb3r90dGDgdzdXVavnki5fhnacfgAuVf7+N5pSDZmj93oCoMb1wRhvW7ujHrau8zLnlimFeC4gIl51xUObeZwwzXmADZwSYokpV8GpWpJUZMfJpeZulWnEc5y6bg6acE0yOQJjiDABvf/4BeI8yYcZpgyRXv+ZovOzofTR9gfq0Z6uFIUv1x9XJICJ87EWH4Kj9wi/Vb73qeThz6SxrLRXJMQunob/oNfuT2pK+4Qred9ZifPxFhwAAPuH/BsLMJxuqB2fRjDZ83a/vsXh2u7++ORAWmwaOivne5RzCpy88FG97/gHW7d98yv740RuOwXGLpuFD5y7R1uVzDr5+8VG45g3HaoURzzx4Ni45br/YMQDee/KplxyqvU/SwJFGAREF1wd4AmJZK+fblzxPM4RVZEbThUeFoYatPcNozjs4d9ls6z6S1kIOLz5iHq55wzG45LgF+M3bTsDpS2Yl7pOEvJak8IiK+vlePKvDWi7h3rW78OBzuzGrozk2lBzH+YfPxQfPXRpJVW8k7z97Mc45ZHbwbwSIirQXz27Htt5hXHvXOhy+z2T9AYVh9mLYwBkBZg2KnEP4zVtPwJLZHUEKqmRGRxMOnNUehKrUInJpTG4t4PSlocdi/5ltmDqpKdCTOESae7855Qv9nGVz8OWXH6EtUzU9tswI+V0f5x2ycej8yfjh649JNLhO2D/0jsi+Or1DJbznrIPwppO98NEbT14U/B1nYAF6obkj952CGe3NuPaNxwYGy2HzJwehIpv7/soLDsFJB063dhN/zfELcPkLlkaWSzpbCvj1W0+ILSAHINAutTXlMKO9CZ/5r8Nw9iGhMfGeDF4dGS5TdTxLlc/a2p396B4sob05H/TXepXFkJLG+TmHzMFR+03BK5bvg2ltTXj18QtS+wsREb5+8VE4bRRGjYr0GMVV5k7CllIOANc/sBG3PbkNFxwxb1wZKiOlpZDD1a9djhvefQreefoBeOlRUe/Scj/777ldA5r3mGH2dtjAqRHHLpqGv73v1EiMelJTHje///l4+rMvxNI5HXjzyYtijmBHusv/e/m+uPUDp6GQc3CVHxIz02+TUj3jOHLfKZjV0YyvvuIIq3fjg75notbN9VRjSk72b7LcG1mnY0dMDyvAmwSWzunAu844UJvUXnTEPMzsaMYHzl2Csw6ehamTCvjo+QdH9n/9SYvw8zcfH1leK6RodeGMtmB8Uhz7/dcux/t8HVMSR+03FTmHNG/SaYtnBpqeh9d34cbHNmui6c/+12H4x/tOBeBpt1Rjtq05j9+/4yR88WVH4IGPnx14zcaSj59/MCa3FgLReTWonwepgXvPmQehe7CEUkXgoufZs6cmMh88dym+aqk/c+i8yWgpOHAIuGAUIUOG2dPgNPEx5Kb3nlr1PqcvnYV5k1twiOKiPu/QOVj7+fOD168/cSGu+c9aqwcijentzbj3o2fFrr/kuAW45Lj6pIZ++sJD8fE/PIbn7TdVux6V05fOwmf++gTOODjZa2C7t685fgFe46e2v+TI+Q17upUCZ7WWy8uO3gf/fHK79r4m0d6cxzOffaG27Jxlc3DOsjk448v/xA/uWIMcEb73mqO1bQ6a3RF7bxvNrM4WPHzFOVXtc+j8TuzqK+Lly/fFPWt2YcXHzgoqDPcOlfCTu9bi4mP3y3xf9wSa8g5OW+z9+xiJ6Jth9lRICHvp9j2B5cuXixUrVtTl2AsvvwHLF0zFb99+Yl2Or1JJyb6S7+FEdMmnXRvgXd9EvDaJ6woc97lb8KaTF8XqeUbDnU/vwOt+dC/+59wldTn+eMU1CmcCXu2YsahXM94QQkAIjOghh2EmOkR0vxBieWQ5GzgjY7BY0bpRM0wSvUMltBZygT6mHsevdRiRYRhmIhBn4HCIaoS0ZkhtZRhJvY0PNm4YhmF02P3AMAzDMMweBxs4DMMwDMPscUwYA4eIphHR74mon4jWEdGrGj0mhmEYhmHGJxNJg/MtAEUAswEcCeAGInpYCLGyoaNiGIZhGGbcMSE8OETUBuAiAB8XQvQJIe4A8CcAr2nsyBiGYRiGGY9MCAMHwGIAFSHEamXZwwCWmRsS0aVEtIKIVmzfvn3MBsgwDMMwzPhhohg47QC6jWXdACKNhYQQVwshlgshls+cWX0JeIZhGIZhJj4TxcDpA2DWXu8EkL1zJcMwDMMwew0TRWS8GkCeiA4SQjzlLzsCQKLA+P77799BROvqNKYZAHbU6dh7KnzPqofvWfXwPasevmfVwfereup5z6wNEydMqwYi+hUAAeDN8LKo/grgxEZlURHRCltpaCYevmfVw/esevieVQ/fs+rg+1U9jbhnEyVEBQDvANAKYBuAXwJ4O6eIMwzDMAxjY6KEqCCE2AXgwkaPg2EYhmGY8c9E8uCMN65u9AAmIHzPqofvWfXwPasevmfVwferesb8nk0YDQ7DMAzDMExW2IPDMAzDMMweBxs4DMMwDMPscbCBwzAMwzDMHgcbOFVCRNOI6PdE1E9E64joVY0eU6Mhosv8/l/DRHSNse5MIlpFRANEdBsRLVDWERF9gYh2+j9fJCIa8wsYY4iomYh+6H9+eonoQSJ6gbKe75kFIvoZEW0moh4iWk1Eb1bW8T1LgIgOIqIhIvqZsozvmQUi+qd/r/r8nyeVdXzPYiCiVxLRE/7c+AwRneIvb9w9E0LwTxU/8Grw/Bpef6yT4fXEWtbocTX4nrwUXgr/dwBcoyyf4d+flwNoAfAlAHcr698K4EkA+wCYD+BxAG9r9PWMwf1qA3AlgIXwHjJeBK/tyEK+Z4n3bRmAZv/vpQC2ADia71mme/d3AP8G8DP/Nd+z+Hv1TwBvtiznexZ/z84GsA7A8f532nz/p6H3rOE3ZiL9+BNTEcBiZdlPAXy+0WMbDz8ArjIMnEsB/Me4f4MAlvqv/wPgUmX9m9QP/970A+ARABfxPct8v5YA2AzgFXzPUu/VKwH8Bp5RLQ0cvmfx9yvOwOF7Fn/P/gPgTePtnnGIqjoWA6gIIVYryx6G92TJRFkG7/4AAIQQ/QCeQXi/tPXYS+8lEc2G99laCb5niRDRt4loAMAqeAbOX8H3LBYi6gTwKQAfMFbxPUvmc0S0g4juJKLT/GV8zywQUQ7AcgAziehpItpARN8kolY0+J6xgVMd7fDcbSrdADoaMJaJQNr9Mtd3A2jfy+LWBQA/B/ATIcQq8D1LRAjxDnj34hQA1wMYBt+zJD4N4IdCiPXGcr5n8XwYwP7wQiZXA/gzER0AvmdxzAZQAPAyeP8ujwRwFICPocH3jA2c6ugD0Gks64Snn2CipN0vc30ngD7h+yr3dIjIgRfiLAK4zF/M9ywFIURFCHEHvLj928H3zAoRHQngLABfs6zmexaDEOIeIUSvEGJYCPETAHcCeCH4nsUx6P/+hhBisxBiB4CvYhzcMzZwqmM1gDwRHaQsOwJeaIGJshLe/QEAEFEbgAMQ3i9tPfaie+k/ofwQ3tPPRUKIkr+K71l28gjvDd+zKKfBE64/R0RbAPwPgIuI6AHwPasGAYDA98yKEGI3gA3w7pNJY+9Zo8VJE+0HwK/gZVK1ATgJnEUFeBNNC4DPwfNItPjLZvr35yJ/2RegK+jfBuAJeK7gef4He2/JOvgugLsBtBvL+Z7Z79cseGLZdgA5AOcC6AfwEr5nsfdsEoA5ys+XAfzWv198z+z3bIr/2ZLfYZf4n7MlfM8S79unANzn/zudCi9j79ONvmcNvzET7QfANAB/8D/0zwF4VaPH1OgfeNkZwvi50l93FjxB6CC87ISFyn4E4IsAdvk/X4TfH21P/gGwwL9HQ/BctPLnEr5nsfdsJoDbAXQB6AHwKIC3KOv5nqXfwyvhZ1HxPUv8nN0HL4TSBe8h5Gy+Z6n3rQDg2/492wLg6wBaGn3PuNkmwzAMwzB7HKzBYRiGYRhmj4MNHIZhGIZh9jjYwGEYhmEYZo+DDRyGYRiGYfY42MBhGIZhGGaPgw0chmEYhmH2ONjAYRhmQkBEgoheVsfjL/fPsbBe52AYZuxgA4dhmLpDRNf4xoP5c3cVh5kL4M/1GiPDMHsW+UYPgGGYvYabAbzGWFbMurMQYktth8MwzJ4Me3AYhhkrhoUQW4yfXUAQfrqMiG4gogEiWkdEr1Z3NkNURPQJf7thItpCRNcq65qJ6H+JaCsRDRHR3UR0snG884holb/+3wAWmwMmohOJ6HZ/TBuJ6DtEZHZHZhhmHMIGDsMw44VPAvgTgCMBXA3gWiJabtuQiC6C1x37HQAOAvAiAPcqm3wRwH8DeCOAo+D1rrqJiOb6++8Lr6fcP/zzfcPfRz3HYQD+7o/pCAAv9bf90eguk2GYsYB7UTEMU3eI6BoAr4bXYFTlW0KIDxORAPADIcRblH1uBrBFCPFq/7UA8HIhxG+J6P0A3grgUCFEyThXG4DdAN4shLjWX5YDsBrAL4UQHyOizwJ4GYAlQnb9I/oYvA7Ii4QQa32PUEkI8Sbl2EcCeBDAbCHEtprcHIZh6gJrcBiGGSv+BeBSY1mX8vddxrq7AJwfc6zrALwHwBoi+huAmwD8SQgxDOAAeN2N75QbCyEqRHQXgEP8RQcDuFvoT3jm+Y8GcCAR/beyjPzfBwBgA4dhxjFs4DAMM1YMCCGersWBhBDriWgJgDMBnAXgKwCuIKLjEBohNve0XEaWdSYOgB8A+Jpl3cbqRswwzFjDGhyGYcYLx1tePxG3sRBiSAhxgxDifQCOAbAMwEkAnoaXnRWIiv0Q1QkAHvcXPQ7gOCJSDR3z/A8AWCaEeNryMziC62MYZgxhDw7DMGNFMxHNMZZVhBDb/b9fSkT3AfgnPH3MmQCOsx2IiF4P7/vrHgB98ATFJQBPCSH6ieg7AD5PRDsArAHwPgCzAXzbP8R3AXwAwP8S0bcBHAbgbcZpvgDgbiL6LoDvAegFsBTABUKIt1Z/+QzDjCVs4DAMM1acBWCzsWwjgH38v68EcBGArwPYDuANQoj7Yo7VBeDDAL4MT2/zOICXCiHW+Os/7P/+MYAp8ITB5wkhNgOAEOI5InopgK/CEyvfD+ByAD+TJxBCPEJEpwK4CsDtAHIAngXw++oum2GYRsBZVAzDNBw1Q6rRY2EYZs+ANTgMwzAMw+xxsIHDMAzDMMweB4eoGIZhGIbZ42APDsMwDMMwexxs4DAMwzAMs8fBBg7DMAzDMHscbOAwDMMwDLPHwQYOwzAMwzB7HP8fzvECH//G7xMAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 576x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.figure(figsize=(8, 4))\n",
        "plt.plot(rewards)\n",
        "plt.xlabel(\"Episode\", fontsize=14)\n",
        "plt.ylabel(\"Sum of rewards\", fontsize=14)\n",
        "save_fig(\"double_dqn_rewards_plot\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "gsE640k5MM9j"
      },
      "outputs": [],
      "source": [
        "env.seed(43)\n",
        "state = env.reset()\n",
        "\n",
        "frames = []\n",
        "\n",
        "for step in range(200):\n",
        "    action = epsilon_greedy_policy(state)\n",
        "    state, reward, done, info = env.step(action)\n",
        "    if done:\n",
        "        break\n",
        "    img = env.render(mode=\"rgb_array\")\n",
        "    frames.append(img)\n",
        "   \n",
        "plot_animation(frames)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RnHiUB4KMM9k"
      },
      "source": [
        "# 듀얼링 더블 DQN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "96KQOB0RMM9k"
      },
      "outputs": [],
      "source": [
        "keras.backend.clear_session()\n",
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)\n",
        "\n",
        "K = keras.backend\n",
        "input_states = keras.layers.Input(shape=[4])\n",
        "hidden1 = keras.layers.Dense(32, activation=\"elu\")(input_states)\n",
        "hidden2 = keras.layers.Dense(32, activation=\"elu\")(hidden1)\n",
        "state_values = keras.layers.Dense(1)(hidden2)\n",
        "raw_advantages = keras.layers.Dense(n_outputs)(hidden2)\n",
        "advantages = raw_advantages - K.max(raw_advantages, axis=1, keepdims=True)\n",
        "Q_values = state_values + advantages\n",
        "model = keras.models.Model(inputs=[input_states], outputs=[Q_values])\n",
        "\n",
        "target = keras.models.clone_model(model)\n",
        "target.set_weights(model.get_weights())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y_-4S5UmMM9k"
      },
      "outputs": [],
      "source": [
        "batch_size = 32\n",
        "discount_rate = 0.95\n",
        "optimizer = keras.optimizers.Adam(learning_rate=7.5e-3)\n",
        "loss_fn = keras.losses.Huber()\n",
        "\n",
        "def training_step(batch_size):\n",
        "    experiences = sample_experiences(batch_size)\n",
        "    states, actions, rewards, next_states, dones = experiences\n",
        "    next_Q_values = model.predict(next_states)\n",
        "    best_next_actions = np.argmax(next_Q_values, axis=1)\n",
        "    next_mask = tf.one_hot(best_next_actions, n_outputs).numpy()\n",
        "    next_best_Q_values = (target.predict(next_states) * next_mask).sum(axis=1)\n",
        "    target_Q_values = (rewards + \n",
        "                       (1 - dones) * discount_rate * next_best_Q_values)\n",
        "    target_Q_values = target_Q_values.reshape(-1, 1)\n",
        "    mask = tf.one_hot(actions, n_outputs)\n",
        "    with tf.GradientTape() as tape:\n",
        "        all_Q_values = model(states)\n",
        "        Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)\n",
        "        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))\n",
        "    grads = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(zip(grads, model.trainable_variables))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QRit2vBkMM9k"
      },
      "outputs": [],
      "source": [
        "replay_memory = deque(maxlen=2000)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jEWYxRQZMM9l",
        "outputId": "9be7270a-84be-4b8c-f132-b74357c798aa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Episode: 599, Steps: 200, eps: 0.010"
          ]
        }
      ],
      "source": [
        "env.seed(42)\n",
        "np.random.seed(42)\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "rewards = []\n",
        "best_score = 0\n",
        "\n",
        "for episode in range(600):\n",
        "    obs = env.reset()    \n",
        "    for step in range(200):\n",
        "        epsilon = max(1 - episode / 500, 0.01)\n",
        "        obs, reward, done, info = play_one_step(env, obs, epsilon)\n",
        "        if done:\n",
        "            break\n",
        "    rewards.append(step)\n",
        "    if step >= best_score:\n",
        "        best_weights = model.get_weights()\n",
        "        best_score = step\n",
        "    print(\"\\rEpisode: {}, Steps: {}, eps: {:.3f}\".format(episode, step + 1, epsilon), end=\"\")\n",
        "    if episode >= 50:\n",
        "        training_step(batch_size)\n",
        "        if episode % 50 == 0:\n",
        "            target.set_weights(model.get_weights())\n",
        "\n",
        "model.set_weights(best_weights)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "41U4UdTxMM9l",
        "outputId": "11ea93c3-dfb5-45b9-f7c3-4f4bbf7b6363"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAENCAYAAADzFzkJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABz0klEQVR4nO1dd7wdVbX+1pxzbk3CTbkJkEBCgBBqQIJBmgUUewELKnbBhuXpe4pP1ChYQEXFZ8OHIog8C+B70os0RcEA0kMJEEhIb7fl3nvOmfX+mNkze/bsPWfm9Jvs7/dL7pwpe6+ZM2evvda31trEzLCwsLCwsEgDp9UCWFhYWFhMHFilYWFhYWGRGlZpWFhYWFikhlUaFhYWFhapYZWGhYWFhUVq5FstQCMxY8YMnjdvXqvFsLCwsJhQuPfeezcyc7/u2A6tNObNm4dly5a1WgwLCwuLCQUiWmk6Zt1TFhYWFhapYZWGhYWFhUVqWKVhYWFhYZEaVmlYWFhYWKRG05QGEXUS0UVEtJKIBonofiJ6jXT8eCJaTkQjRHQrEc2VjhERnUtEm/x/5xERNUt2CwsLCwsPzbQ08gCeB/BSALsA+DKA3xPRPCKaAeBKf980AMsA/E669nQAbwawCMAhAF4P4CNNk9zCwsLCAkATQ26ZeRjAUmnX1UT0DIDDAUwH8Agz/wEAiGgpgI1EtJCZlwN4H4DvMfMq//j3AJwG4GfNkt/CwsLCooWcBhHNArAAwCMADgTwgDjmK5gV/n6ox/3tA6EBEZ1ORMuIaNmGDRsaIbrFBMeqLSO4dfn6VosBAFg/OIrL73kO1z+8tiV93/BI9n7veWYzfn77CoyX3Irnrtw0jDufzPY7HBor4X//tRrbx8u48r5VEMs3FMsufr/sebhu9uUcrntoDTYPj2e+TqCavi+7e6Xxe904NKY9dteKjXh6w1Bs//3PbcEjL2wDALgu4/fLnkexrH/+T60fxPk3PYEHnt+aWtYsaElyHxEVAFwG4NfMvJyIJgFQ36xtACb725P8z/KxSURErCwIwswXArgQABYvXmwXC7GI4c0/vgsbh8bw7Ldf12pRcOp/340n1nmDxENLX4XJXYWm9f3OC/+BFRuG8eQ3XoNCLv388SOXLsOWkSJeNHcqjpg3LfHcl37nNgDI9Ky/dNVD+N9/vYBFc3bBA6u2YdcpXThqnxm48I6n8Z0bHgcBeNviPVK3t2loDB+77D68aM8+XPnxo1NfJ+Ont63A+Tc9gUKO8JbD5qTq80tXPQxAf+9X3rcK37puOR77+qvRVcgF+9/1i7u117zlJ3cF+6+4bxU+/8cHsWFwDJ94+T6xti/667O4/J7nMHNyJxbt0Zf6HtOi6ZYGETkALgUwDuAMf/cQgCnKqVMADBqOTwEwpCoMC4s02Dg0BgAoGWZqzcRzm0eC7Som0DXh2U0jlU/SYMtIEQCMM91asWbbKADg6Y3DAIDBsRIAYNOQZyls217M1F7Jf7CrtmyvWqZN/juzbSRd38Vy8pdZLDOYAbeKIWyrL4PJchorlTG7rxunHjlXe7xWNFVp+BFPFwGYBeBkZhbfwCPwSG5xXi+Avf39seP+9iOwsKgCjh93N1Ist1YQBU6T4wGrGbAi1zdI5waPYQJPCSs9WzHfreUrML0uZZdRyDXuZWq2pfFTAPsDeAMzy2r/KgAHEdHJRNQF4CsAHvRJcAC4BMBniWg2Ee0O4HMALm6i3BY7EHo6PK/syFjrlQZJP/1mR5GLAavagavcIEO/UY+hmTqoXMFsFI+uFsVturJUZuQaOANpZp7GXHhhsocCWEtEQ/6/dzPzBgAnA/gGgC0AlgA4Rbr85wD+DOAhAA8DuMbfZ2GRGcKHPDJearEk0QGyVYlHXOVwWg0hXQ1q1U3iuTbTmV2pL/HoqhGpklItlt1MHFVWNDPkdiUSfhfMfDOAhYZjDODz/j8Li5rQ0yGURjtYGiEmmjemVvdW09ACf1clK0wo6moeYaVryi4jvwO5pywsWo52UhrtgKrdUw2yNKjONlc92st6p5UUamBpNEDxFl1Gzmnc0G6VhsVOh25faQy3hXsqHNAmWjBg8yyN+vTTzMdb0XVXAxFeyT1VKrso7AichoVFu6Db5zS2t4GlMZHdU42KWBaDYr2eRz2I9axNVNIZ4ngjFG+pbN1TFhZ1hXBPDY+13tJoGfstodpxq1GWhmmQr5awD69vHmTXnc6CDDiNBvRdcl3krXvKwqJ+6BYht21gacholXeq6uipCeZOq8X9V+lKNelPfjY6q6MeIbcmlCwRbmFRX3QXvNe+HZRG5KfdKqXRZkS4CiFftYR2o3Xb8rUDWPT1G/H7Zc8H+2RloHtOwa4GyFYss7U0LCzqCcf3f7TDTDlChE8wVqPR0VNB1nSN7dXDFZSkrh5f61U7uuOJsHye/Gh075mQqZZHaHp9y66LvCXCLSzqh3ZavqsdZKl23GqUzq37M2mSLpYnAFFOI35ukI3fAOEsEW5h0SC0W4hryziNDB3L5zaqjEi9UQ8pk9oI3WfyvuTnJI7XYmmYlGvRbWxGuFUaFjst2mHMm2ght/Ig1yxOo16ox/etqw8mrAX5kPxsdO6peiT3Gd1TO0rtKQuL9oHvM2+xFEDzixTqkOU5lKTStrVyQj+/fQXmnXmNscR6wBXX6MrhOgzQYVs6q8H7K3+Tsj5lze3VWiwyCcUdrMqthUXboO0sjRYJlKXbyAy6RkvjglueBACMKSsAqoq0VuXUyJwIQFIaktxuJfdUDbWnKqFUtnkaFhYNQbtFK7WXNHqUXHkwrK2ttJeLAbgdQm717in/mLQvmqdhtk4aQoS71j1lYdEQtIOl0RbIYmmU62dpCJiGN/H91MqdcGyj8ahkkdWDCDehVLbuKQuLHRbyxHUiZISXKhC89YB4JG6dB9a6RFElRELB8F3q5G9klduS6yJvo6csLOqJxvq4s2FiJfeVI+6pxsoblNqo1dJotJz+X9l9VtE9VYfkPhNKLu84VW6J6AwiWkZEY0R0sbT/3dJKfkNENEJETESH+8eXElFROWd+M2W32AHRbv6pCVBGJBI91aiMcH+8E4NtrcqpntFT2mi3gAgPd0WUq9Y9pVxcJ5RdBjN2qPU0XgBwDoBfyjuZ+TJmniT+Afg4gKcB3Ced9jv5HGZ+unliW+xICEnI1qMNIm4zPYfoYNiY/lX3lOizXS2xIE9D2hcJuU1wT9Vb74rw5UZmhDdtuVcAYOYrAYCIFgOYk3Dq+wBcwu2WsmuxQ6CRMfJZMdGS++rJaYRrd+vbqZffv56ThMQ8DenLrOSeClykNQilU6JCwe5URDgRzQVwHIBLlENvIKLNRPQIEX0s4frTfRfYsg0bNphOs7Boi5lrWxDhGTqulOmcqV/DftUFJPqsddnWRj3fSpyGzr0mvHz1DiYo+dFtO5J7Kg3eC+BOZn5G2vd7APsD6AdwGoCvENE7dRcz84XMvJiZF/f39zdeWosJh0YmVk1EZHkMcvZ2vcqImNxTQT8154PU74vW5mn4zcvjdOpFmGoQTadEi7422qksDXhK49fyDmZ+lJlfYOYyM98F4IcA3toS6SwmPNqK07DRUxUHTrdGTqORiXSAbC3I672Hx3XcTz2We01yT+00GeFEdDSA3QH8scKpjLZYKNNiIqMdLI32cE+lP7cUmUHXS4Dkw41yg9ULgcrIULCwJgsjIXpCfD8NTNNoeshtnoi6AOQA5Iioi4hkMv59AK5g5kHlujcR0VTy8GIAnwLwv82T3GJHAgd/W6812mHmk+U5mEJJ//bURlzz4JqG9C8smurLiDTYHRnIFyJtnkY1MiVxUMIqcxoYltdsS+MsANsBnAngVH/7LADwlcnbobimfJwC4CkAg/AI8nOZWXeehUVFcKg12gptJo4WpbJeabz7v+/GJ357n+6SilDHQHW8q9k9VdVV2ds3Rk8lVLmtNxEe8CsNVBrNDrldCmCp4dgogD7DMS3pbWFRDRpd9TQLIsu9tq6OSGpUGz3FzEa3SrwVNXoqdTfJMtSnmXi7IuQ2Ej0FaVvnnqr+HUxyT4m+GkhptBenYWHRTLRbGtAE0BkR8rtmriHl5XWrcltLTkTCteI9MmaEJyzCVBMRrrlU9LUjuacsLFoPEU3TBjqjLTLCMzyHSP5BBgsgzaArEHNP1fxFNfaLjsdORe9JH3IrjtVZFqs0LCzqjzalNCYGZLdLhjyNpDMrtVJrGZF6hNwmjcHiMchuo6gbT3eNIMLrK5PoyyoNC4sGoN0sjYlQGl0+t1l5GjUXLKzp6srQKc+Ka6lz5E9V0Ne0EpZGDQ1XgFUaFjsdQhKy9Vqj1tIY9UCWMTm6TkQ2Itx4TPke1CeiI5qrQU11nhKuDUKCM9SeCkqjV5FVn/QUgpIr1tKwsKgfGuVPrgYRS6MNlFglRKKC6uSeUg+q41393FONga42lhvhNOLXiDDcamRKdPX5B+1yrxYWOwEmQvSUbDFkqQmVSIRXuLb25V5rf7CJnIYbtzTkIAHtehrBIkz1/dKte8rCogGo56I8taIdSqNneQ7ymTWvqGfIilbdUPUKuW3U9x1mrIeo5J4KHl0VIiU9BUuEW1g0AK1XFSEa6XtuBLjCYGi8LsNTr3fIbT10RVIbWiI8Ze2pei/CVNZYPfWGVRoWOy3aQXmYYvubiWqJ8Cxuo2T3VHI7ZTfdeZXaT3t12WU8tX6w8onifBaupnBflPvRyNSgYAzRruU0LCzqiIYXsKsSbSaOFkLGvEN188enLY3e6H4EfnjzEzjh/Dvw5LpQcSTN3MuaBZUqZc43KhjDuqcsLBqA0J3cBsP0xPJORaJzGrYIkxo9Va8yIilx73NbAADrBsZStRFOQji2D0iuPVWN4k26wrqnLCwagYAYba0YgOqeao0M1ZQR8SyN+vShuuXUc90GuXLqBTFQmxL69Bnh3t+qQm6Tcl5sGRELi8ahHYagKBHeIk4jU0a4h1xG91RSHyYlEXxWRt2swQNZlXHS+bquyxqlVikjPHRP6a2TaiG6ajtOg4i6iegEIppbb4EsLBoNu0Z49RADWz7n1M09pUJtVs0HyTq4ZrVQggx0zbirT9SLWxpp3VMcuSadfEnKum3yNIjoYiL6uL/dAeAeADcCeJyIXtM48Sws6o/wN9d6rTHR3FMCWTmNaiO0ACl5rkkEkFAyaXsLLI1IBWDZgtD0UUPIbbqSJq23NE4E8A9/+40AJgPYFd6CSkvTdkZEZxDRMiIaI6KLpf3ziIiJaEj692XpOBHRuUS0yf93Hk20AHeLtkM7WBrRMiLNQ8QtkuE6MZPNEWXLJE+URf2scBxByGyVIbd1fLBa95SInpJCayu5p3RVbtOKmfwsG89ppF25byqA9f72q+Gt472eiP4HwJcy9PcCgHPgKaFuzfE+Zi5p9p8O4M0AFsF7ZjcBeBrAzzL0bWEBQM4Qbq0cQHT23Ex5qu1LXOcQMmmbLAUL45xG+n707Wc8X1yQctwN3VN6l1Ta5L60brfkREPvb64NLI21AA4iohy8Af9mf/8kAMW0nTHzlcz8JwCbsggJ4H0AvsfMq5h5NYDvAXh/xjYsLADUPnPd0ZCpjEjg76e6Pb+YpaEcr72MSFYOBMb+dE3pk/squKdqWNUlDafRDiG3vwTwOwAPAygDuMXfvwTA8jrKs5KIVhHRr4hohrT/QAAPSJ8f8PfFQESn+y6wZRs2bKijaBY7GtrC0mhRlVs2bFdC4J5yKBtPkeGY6s0Jw1OrdE9VdVX6gVdYGiZOI3m513CfSc4sSs9tgnsqldJg5q8D+CCACwEcw8zj/qESgHPrIMdGAEcAmAvgcHicyWXS8UkAtkmftwGYpOM1mPlCZl7MzIv7+/vrIJrFjobAPdVaMWJornsqeSZsvM7/m3VMypankfy5XnTmXl+8Bhfc8qRGIPM1SSG3sgVQlMrcai0DjYvU9IwqcT4ygozwBiZTpG6ama9g5u8z8ypp36+Z+X9rFYKZh5h5GTOXmHkdgDMAvIqIpvinDAGYIl0yBcAQt0OZUosJh8Ax0AZvT6viOaq+9YDTyEaEJ51sckeZjmd2NyUMxuff9IRGnozRU5qQ24dXD2C/WZO9/QlEeJpcl/j9m8+VAxUaBSMRTkTvTdsIM19SH3HCJoUY/t9H4JHg9/ifF/n7LCyqRrtxGq0jwjNwGmJApdqT0dLMrLsKTs2cRlYVKfM2SbIJqApgcLSIx9YO4H0vmYfH1w0mJ/dF9unlVJ9zUpiubr3yeiMpeurHyucOAAUAwu5y4JHgYwBSKQ0iyvt95gDkiKgLnovrcABbATwJL1LrAgC3MbNwSV0C4LNEdC285/w5AD9K06eFhQrW/WJbhGg+eDM5jercU0HGceaQW/PZSRnhC3edUjunUeVjTTvuBisL+v1s214EMzBnqhcgql0iXBdya1Kisc8J7ilXcBopBK8SRvcUM08W/wCcAuBBAMcC6PL/HQvgXwDelaG/swBsB3AmgFP97bMAzAdwPYBBeGT7GIB3Stf9HMCfATzkH7/G32dhUQVY+r99MLFCbjMS4YnnmmfSTj0smjqer1MkIantbQgl0pl3Ip9116S5tTinIf6a3V7tkKfxXQAfZOa/S/v+RkSfAXAxgKvTNMLMS2FOBrw84ToG8Hn/n4VFTUj60TUb7ZCiWo3FQFQ/pRv7GiJKg2p2T+kzsitLn7Y3tWJtSSiNQi7yOXJN8DcNp6HneHRXtlPtqXkAhjX7RwDsWTdpLCyaiNarjPZANRaD45Ea6a/LcEx2TxGFCWv1dN8l8QJCCaQOueXo31JZtTTi2YmBonHlfSZ59Ncm8SvtkKdxN4ALiGi22OFvfx9heRELiwmBdlIWkTyNCeGe8qNzHD2nYZrBJ5fzjn4WA9/lpx0Jong13exVbs1unHpALQlS8jWBUBpaSyPgaaR9Kd/M8FrNfQWcRustjQ8DmA7gWSJ6loieBfAsgJkATmuMaBYWjUE7rdzXrCJ8KiJEeBWl0R2DoZGWzDXJIs49dt8ZeMne071+Yn1k++L0bpwEJZap9fCeVUujkHPgUHLtqTTKKx6CLK7Vnev9baR7Ki2n8QKAFwF4OYCF8Nx9jwK42eZKWExUtMOL27KMcNZvp72OiMAcd7u4zHBSKkJTty6HM2WHKHDv1JfTqHx+2ueiKgBhWeQcQt7Rl5DXRfCld0+Zz2+Ge6qi0vDrTW0DsIiZb4RXEt3CYsIi+L222Xynqe6pKq+Ty4gUy+nbTTNIh585CBn1iHDRtrmRW5evx4ahMbx98R4amRIGbZ08yt9KfauDuFASeccxlpAXeyI1qirIIxA2p7uvxrunKioNZi4T0Up4eRoWFhMeOn/yzoaqy4gERHjyTDd2XcY8DcFbEKVz4Xzg4n8CgFZp6JAqE7taS8MvIZLPEfIOGTgN30Waon1TqXhd9d9yG3EaZwP4tlJE0MJiYqMNtEbETdTMfiPb2TkNMiT3VWMtxTgNhtbSqBoZlFsggHJOkmssJMK9z6XA0iA4JkuDo9d6+0wKV79DS4QLTqMN8jT+HcBeAFYT0Soo4bfMfEi9BbOwaBRC90PrtUY0M7s1nEa268RMNmO7GdxTLiNiaTQiuS8x5NYglwmm5L58zvEtDT33o/aR1rWXJF/AaTSwYGFapfHHxolgYdFctFP0VCROv5kd10iEexnhupluytlyYh8czOt1mefZQ271fVQ6P22EmZrcJyrc5h2qyGmkUoiqUtUUSFRlb3lGODN/rWESWFi0CO2gNBhAR87BeLnG5emaBDF4mgYlo9LIRITL0VOakNPMIbdmN06FC1NBTe4rR6KnKAjBjV4T5zTM4coqp6HfD4Rl2ltSe8rCYkdHq91T1z20BusGRoPwyOZGT1XXWRhya5jBZxNCC5c5WA+CJE6jWWVEdHkQ6TgNnwj3LyzkCLkcaRdhCngJju+Ly6r/nOSeajkRTkQdRPQ1InqCiEaJqCz/a5h0FhYNQNY4/Eag7DI+dtl92Dw8Lv3AW8NpZHJP+X8dw3KvmtQN/7rKg7SAHD0lFyxsXhkRvVyV2gqS+3yfY85xzHkawbWV+1DPCJYr1lzbDPdUluip98Fbm9sF8B/wSqdvAvDxxohmYdEYhGuEtw7yQNJIV4IJ0Qlu+idRabnXTO4pg4XFjAinUXMZEa08KQbrCEmdntMQ7qi8Q3BIX0ZET4Sbnp3qnhPnx9HS0ugK3g7go8z8c3hrhP8vM38KwFcBvLJRwllYNBKttDTkgVDMCpvqnqqys0iehu646bqkNjWfnUiehipDRk5DS9hXvi5tfaowfNb7G0ZP+RnhGk5DH3JrkkO91kyEh5xG6y2NWfDKhgDe0qt9/vb1AF5VZ5ksLBqK8MfZOq2hVnIFmitNGgIWAJavHcDqrdtj+ylj9FSiLMo1LocZ4aZ+MrWv2ZdYe0ozkxecRpJ1JeQsSkR4zpDcl+U5GYnwBGXotEFp9OcA7O5vPwXgRH/7JfAWUkoFIjqDiJYR0RgRXSztP5KIbiKizUS0gYj+QES7SceXElGRiIakf/PT9mthIaNdOA0B8QNvrqUhbSec9+of3Imjv/2X4LNwf5hW7jNGACUSz1FEOY3aQ26zJvfp5hSJq+UpVkM5CLl1kM+RoTR6XA5jDyYiXHeqpHAbhbRK4yoAx/vbPwTwNSJ6Bt4CTP+dob8XAJwD4JfK/qkALoS3bsdceCv4/Uo553fMPEn693SGfi0sYmglpyGPI410JZhQbVKhONNxoH2A5tLoCW3qOI0gI7z2kNus8gTnJNR20u0T32lJck+ZLA1drlDajHDVslGPNfp9Spun8UVp+49E9DyAowE8wcypVu3zr70SAIhoMYA50v7r5POI6L8A3J62XQuLLEiKPmkW5DBMJyCEW6nG0iEMudVbGplKfhitkmiV25pDbrWDf8L5ihKIthWHqcpt3s/T0Fk1HFxrlsMkKxv2A0DZbfwkJG1GeATMfDe8hZkaheMAPKLsewMRbQawBsB/MfNPdRcS0ekATgeAPfe0iwpaxJFk3jcLsntKuFuaKk9K91T8spBo1c66q7qLOKchhj25YGGtuSVqH1mkSuY0ov3IVW4d0if36cJ6TRLpanOZZGEpx6VRSJun8QQR/ZyI3ilzDY0AER0C4CvwwnoFfg9gfwD98BZ9+goRvVN3PTNfyMyLmXlxf39/I0W1mOBon+ip5veflghXERCthugp08w5q3vKqSOnkVVphIOyPKAncRpRS0MuI+JxGmYiPE2+THx/tD+13UZbGml10ncA9AI4D8AqInq8EUqEiPYBcB2ATzPznWI/Mz/KzC8wc5mZ74LHq7y1Xv1a7Fxg5W8rEM3TaC0RXs2FpmHJNWiN5OQ+pQ1ptkyoRxmROFK5hdLuExaGIMJdBpEX4JBzHD2nEVxbWTHFno+r3w9EF7BqFNJyGr8A8AsgGNhfBi8/49fwFE9Vbi4ZRDQXwM0AzmbmSyuJBPN7a2GRjDagDrR5Gs3MCFfS+9JfJ3MwdZJFaUeucquzNOrTZ5ISM3NeSWHGYiJQchkFX+vljaXR45ZGWllD+eLnCoXVSKQe7InIAXAEPIXxCnhE+GoAt2VoI+/3mQOQI6IuACV4eSB/AfBjZv6Z5ro3AbgDwFZfhk8B+M+0/VpYyGgHIlxHsjYTadwiOohwWDKUEamuYGHckggywp3qcj+S2geS1WTWkGw1E7xUdoM1us3RU+LaykLp3HfeX70yauT64EBKpUFE1wA4Bl7ZkNsBXA7gdGZembG/s+BlkQucCuBr8B7XfABfJaLgODNP8jdPgRem2wlgFYBzmfnXGfu2sGgbRKKnhJO4me6paq9jz8QnmIjY7P2pxzxrRmSExxdhyqpD9G6cyu6ytN2IpgSXUXIZeaE0SJ+noSP3U/eXcH7buKfguaK2wuMbbgVwGzNvzNoZMy8FsNRw2Fh+nZm1pLeFRTVot+S+XAuip6K+9AzXwc+hyLrca4ZIKzkjXC5YGEQwZXxSWjlTWHpplaLYJyyKssvI53ylYSDCxZ5qQm4r52lUbrMWpCXCdwHwLgBbAHwGHhn+EBFdQERvaZRwFhaNQDhTa6F7SsdptIgIz9Iv+3yDKWeimqVZY4OiG80IrzXkVqcWk6On2HhOkktOWBrFMiNXkdNQN8zfQ0wOnWtLOjdzxnxGpCXCt8MjqW8GAjL8SwA+BuAT8DgKC4sJhXaxNCZSSIfgG8zLsBosjaQ2NbWVxLjnRU+pMqQUNuH8RI5F+VsJKqdRLLso5JI5jTBMV+433bNLdE+5jV0fHEjPacyER4C/3P+7AMB6AFfAc1dZWEwYtMNyr9qQ2xZZPlnLiBD5nIbmeDV5GrraSjKnEY8eqh2pkvu0RLOuLe+vWEdj8/A4pvZ0ADBbGuIm0nznukABk3zlJrin0nIaa/1/d8DLkbiNmZc3TCoLiwai3dxTuVa7pwznyDkX7Ls9PEuDjCv3VVXlVtNGyGnUHnKb3h6qsq3APcVgZqwfHMXMKZ0AYMzT0FoaJoUbu9Z8/tBoCZO6as6ASETa1g+wSsJiR0MrLY2Id6olGeGVb16O8Cq5jEKOfCvAI6Wz1XRK4hCin9WV++LJfRVFr3h+onJj8zlJlgbgWZDrBsZwwG5TAAA5B4lEeBrlbaA0tM9/6/Zx9PlWTqOQiggXCoOIFhPRO4io1//c6+deWFhMGATRUy2UodW1p9IQ4bKM4yXP9SIS77JaGsljdFwpBFVuHU3IbdboKa1yS1BikhxpIN/zaMnFpqExzJrSBcCrP1Uqm0ujp3MNmtxT8TO3jhTR111IJ3iVSFt7ahYR3Q3gHgC/hZeMBwDnw1sC1sJiwoBjG82HrvZUM5MNObKt71eWUUQGMSQiXNduShdL4rkRTqNRlkZ112mjp6TG1g2MwmVg5mTPPVXIEYqagoVhRJjcX2WFyxxKoFPQW0eK6OtpA6UB4PvwOI3pAEak/X+AXbnPYoKilZxGJE/DaYWlkcI9pbE0QitAzzVUo/e07il/mwz91ApTjSxPHsE3pHNPMYff4daRcQDA5C5v4O7IOxjXWRoaOdIq3ERLownuqbSupeMBHM/MW5QY4BUAbP1xi4mFNoiekgeLVkTcsvFDCDkBbixQGsl5ANW5p+KfnQROIyt0VydZGqz8rQSXGR05B9vdMgZHSwCAroI3H+/I5VB2GWU3Wt6DNZaGUR7FlWhyr44WyxgtutilHdxTALoBjGv29wMYrZ84FhaNR9ZBoRGQSeZgEG6RQKZuZRnHA/eUH3JL+iuNZG4iER49FoueqnB+JWRd94MzviAMz6IAgKExT2l0FrzUtULeu5GiYm2EnEaa9iVrJHJt9OJt24sA0DZK4w4A75c+MxHlAHwBwC31FsrCopHIRkI2BtE8De9vU6vcZiTCxTZzcu2pLJaGSVcyhwfrYWmklSd2TkoC3WUOlYawNPKe0ujIefuFpSZfI/9NkinOaegt5bXbvPm7IOEbhbTuqc8DuJ2IjoBXNPB7AA6EV17k6AbJZmHRULTS0pB/8K1YIzzN3csDWqA04C3yYybCqyE14tcLRUp+nganGFyNzWdQbp44YkBP21aoHISlIdxTnb4yGVeUhvbZpQhIYEku9exVW7YDAGb3dWvbqRfShtw+CuBgAHcBuBFAFzwS/DBmXtE48Sws6o+k9Qia0j8zvn71o8HnlteeMgxWOkvDZZERbljuNcVsOXZM6j9cGZAif9PkM6RpX+1He36C68ikKFX3VJfvnhL7VTJctJ0uMz26bbp29VYvRmn21MYqjYqWBhEVAPwVwHuZ+asNlcbCoglodZ7GWMnFMxuHg8+Bm6aZSkPezuieAsyWhrGMSMpEQFexNMTfNG6cNHBd9nM/KnMaES4hoc+IpTFqUBqSpWGymtLfl37Ss2rLdkzuyree02DmIoC90Fpr3sKi/mhl+JSEVrin0ty6PLCGpTA4rD2V0e2TRhZxPUl5Gt7+zM0mtp9mNEsbUhzhNBT3VCGnUxoaeRJEUq1C8VnNNB8cLTU8RwNIT4T/GsBpjRTEwqJZaLWloQ48YhGmZsqTZvEfeVAK13Dwy4hoCgmK49r+UriD5G2SOI24vNmelNy+iAjLWrCQDMEKzN4gLpTG4JieCJejp9IoikgfisWjlmKX223GBCSt0ugFcDoR/YuILvLX0Qj+pe2MiM4gomVENEZEFyvHjiei5UQ0QkS3+muGi2NEROcS0Sb/33nU6KLxFjssWm1fqANW6LdvnmRqRI4OOiLcS7wzJyNWcw86V1kip5GVCJe2Re5JqoxwjVyxc/z9ldxTcvRUtF3Wbuv6UK9XCyE2Y9U+IH301P4A7vO35yvHsnyFLwA4B8CJ8HI/AABENAPAlQA+DODPAM4G8DsAR/qnnA7gzQAW+f3dBOBpALH1xC0s0qJV3ilVabS69pQJ8kQ2EnIr8jRSEsWV+pMHy8A95X/WcRpZIbefxtJIyrhW94l2Ynka/mcdp2HiZ4zPTulfXBPP/eCmFL9MuwjTy+vRGTNfCXiFDwHMkQ6dBOARZv6Df3wpgI1EtNAvlvg+AN9j5lX+8e/Bc5dZpWGRGWE2bmu0hjrLbfT6B5WQxj0Vhtx6s1mvym0ctZZGF9uqpeFKg2Ut31rSUqkxuXTut1h73l85T6Mj78Dxv9ROTfSUidNIK0dgaSg1rZibU10grXuq0TgQwAPiAzMPwytRcqDuuL99IDQgotN9F9iyDRs2NEhcix0BLePBY0oj7oLZNlLEb+9+rmEuq4jCNHRhck8B5pX7skRPBcc0g2jIacRlyfrFRd1T4j4qn+/qHpHSt87S6MqHw2pHznNTqXkauuaMLrDINgfPPRbGi/biNBqNSQC2Kfu2AZhsOL4NwCQdr8HMFzLzYmZe3N/f3xBhLXYMtIt7KrQ0wv2fv+IB/OdVD+Hh1QMNkSGdeyquNCDcUwCGx8u477ktSrvp/PJ+U/F9/jgor9wn7zddl9i/tCsaOmxqI95PpefV6XMag6PFgM8A9GVE5O8/qrwqPzvZPaWWXHfd5qzN0i5KYwjAFGXfFACDhuNTAAxxK+tAWExYtDp6ykSEy9g45JV6GyuVGy6PMblPwwUwogPTST+5K9pWitly0lHV0pBLrKRzKelaz8hpBHkQccdZ3D3l7emf0om8QxgYLUXyJDoqhNymc5FyZEtco5Zc39ksjUfgkdwAvMWdAOzt748d97cfgYVFFdAPCs2D6hrp7vBmps0UJ41bxNUm94VlRLTX1JinkchpVJBXvt7YPlcvp65v8Yhm9HbidYfsBgDYu39ScLwSEY4030PkHA4iwOIhtyluoA4wKg0i+gsR9fnb7yWizlo7I6I8EXUByAHIEVGXv/LfVQAOIqKT/eNfAfCgtMTsJQA+S0SziWh3AJ8DcHGt8ljsnGi1faoqq27fndFMsVIt92oqIwKpMq/arsnSSJzZh0jKCA8tRHNblZRBFveULspJ7Vu2jI6cP93rQ7ouCLkt60Nusyb3ydfoKue22tI4GkCPv/0reMUJa8VZALYDOBPAqf72Wcy8AcDJAL4BYAuAJQBOka77ObxQ3IcAPAzgGn+fhUXVaJXyULsVPvB2szQi7ikpeoqIjFE6xiq3KWUJrlc4jbSWQSXuRCX0tW1o2jIqQ4mDOXbfGQCAty/eIzjeqSHC0/IzYd+KeypQalHF3g4ht8sBfJOIboU3uXg7EWlZOWa+JE1nzLwUwFLDsZsBLDQcY3iVdj+fph8LiyQEg0LLQm6j/crEabMQ99bH4WrzNPxV9QyDUzVPVBfJJa+n4fUruxVTtiX2aZRS5uQ+Q9+yZTRnag+e+dZrI1aYtvZUhMPRy2mUg+PL8Oac0FJtdXLfxwD8EMCbfHm+DYPLEJ77yMJiQiApeasZiHEagXtKM+A1SIZUy73qLA0Oq9wKyGR9lugp3bF4lVuxX3ZPpWsr2KcZpBPdZQm8h7onUBpO1DISKOTi0VNRC0a2IlI8O47KIGeFu622NJj5LgBHAAARuQDmM/P6xotkYdEctIraUNen7u7wZqMtc5cZOo4Q4UH0lLfcqzw4jYyFSsM8g9dZAPF9aka4rmBh1uckn5+G04DGqqhEUps4nnzOgUMpM8KNlkZUsUQsjZLrrXAEUba+faKn9gJgM+UsdgiEPuvWjNJqt0nuqUYNAWncU5VW7hMQpTPEcW1/iS6l+HYsT4NTBqdW4DR0K+ZlaUM9pi4apUNH3olmhEfazeYn89xT4eeiK1sw3JTqAmnLiKwkollE9AkAB8C7jUcB/ISZ1zVSQAuLuiOFm6Ox3es5jWbKk0ZfmtxTasjt8HioNKoLuZVcLH4/pOM0gu8tYcCv8BTlKLAscpmjp6Jy6tCRc4yWhswbpc1xkeWSczWEQm80UlkaRHQ0gKcAvAtexNMogHcDeJKIXtI48SwsGoh24zSaavnoXSQydHkawm8ucxrDEfeUwS+fVqoETiONttd2Lw/SKSyN8FypCUPfaoiwDh35HMZKLtYNjHrchq5dmBWemtYRWeekHCXYWx1yK+O7AC4HsICZ38PM7wGwAMD/wFsv3MJiwoCVv81GW0RPRUTQPwlTyC0QzQofkSyNdP2Zj8UzwsPkvjTQ6gxpW8zs0+SNpNHh4fNIsjQIKzYMYck3b8FPb1sRuZd0UVyygueIdRIpT9KkMiJpS6MfCuD9zGGEMTO7RHQ+gPsbIZiFRaMQRk+1itOI9tvd6pBbw2Mo64hwjudpmOoqRfpLHKTl2baHkNMI200TPaWNeJJ2yeVQjPJIpL8qmHqdsMYS3VN5B/c8sxkA8PzmEUUJ6OU0ya9aGhH3lB+k0GiktTS2wSPDVewFYGvdpLGwaAJab2lEP7cieirVLFoebKXlXh0vJTw4VirrB8FKIImvEIhXuZWIcDFwZyWxNRaTGsEWOV/TVrgvep34mDRUd0hVb6f2dqSy8mTooq3yjq4QYhtxGvDcUBcR0buJaC8imkdEpwL4BTy3lYXFhEPrQlyjn7sS8jQaJ0N8dq9CXxo97gKJ5goY+kuQQZe3QAqnkWZGbuxI02diaXSNcjLl9oR5Gub2ZKUxViwnVLk1yKN8dpmDdToiRQu59cl9Mj4PT4n9UrqmCOCn8EqCWFhMGKSJwmkk2oHTkFEpBwFQM8KrcU9lk0XNCPcKFlZupFKCZJoyIsFCTSnkDYnw5OgpgbGSa07uM3UY0QsMlz1FNDxejj37fLsoDWYeB/BpIvoivOqzBOApZh5ppHAWFo1Aq5SFgBhovvPWQ7DvrMnIadw0jUaWARFQV+6LWhvlaJhRTbI0JCNcx2mksjTiMsZn/d7fRCJcsjRGi2VtBrzXtkHhKtyKZ2nkABSjrkG0l6UBAPCVxEMNksXCoqlotXuqr6cDh+7Rh5WbhpsuD6cZrGSlIbt1KGpryAOXOeQ2iYeIX+8onIbLYQuJA36FfeLadJZG3ApQL0uT3FdQLQ0N8Z8EVXkxh4pItTR2pkWYLCyaBtMAILDgrOtwztWPNqz/WKmMptCXUZgieCLnyDNipWChPDjJWclGH33KgT4czKPraaR3b2ncUwmhw7prdEUNQ0tDPRcROXUYLZYj21ndU2oJFZc5IMJVkrydoqcsLHYYVBqAxksu/vuvzzS8f5U8barhk6IzebCSyW6x3KtApDy30cWSThaV0xD9pF1Po9Jt6Vbui5Pb+v06pEnuG9ge5rGMlVxz7akKfXjneLWnck7I9YRtcVtFT1lY7HDQzUrlWWGjEIaVRnMRtLPkBsnAhm0ZWk6D42VEihH3VDWyxAfRgNNwwv1pSqNXyvR23Xgb6jW6PI1K0VNJM/yB0SIAT7F47qn08sp9h58hKY3os29G7SmrNCx2Wuh+ruIH3kiYXBpaf3yDtIbJRRI9J6403MA9FcoeLfttaiudLKY8DdfTGpWhOUdHPMvWkaropJSUpGYjbSe5pwa2e+/UnKk9vnvKYGkYHpLqnmIG8n7JdVex8tqKCCeiDgAHAZgJRdkw87W1CkJEQ8qubngFET9JRPMAPANgWDp+LjOfXWu/FjsfktbTEK6ERmZppyFPBapdy7qiDKnCVz04pGaER88rlvRrRVTqT6skgz6jnEZtZUTiyi/JpaatT2W6L8WdpsPwuGe9zu7rxoahMaOVk849JTgNxz8mnddOZUSI6JUALoWnMFQwvDW/awIzB6uxE1EvgHUA/qCc1sfMlQvdWFikgG4gG/QtjUldmQILMyEI00TUPVVpllxPpPLX+4Lmcw7K5dBlQ1DcU8pCQJllkftULA1Hct0FBkBCH5VCbnUKIVppVs48l2Xk4LiMNHkahRyhWGbMmtKJ57eMaAl2k+xeH1H5jEQ42osI/zGAq+GVDemBZwWIfz0J11WLtwJYD+DOBrRtsZMjHHzixwZGvTnJpM7GKQ3V0ghKZWhn441iNWR59PvFYNWRc2KWhhzxZVqVLk0f6rF4RriU3BdwDQltVZHcJxdcrGRcqPtUJafDnz95DL510sHo7sjFQ24zchpe9JSe02g3Inw3AN9k5pXMPMrMY/K/Bsj1PgCXcPyJriSiVUT0KyKaobuQiE4nomVEtGzDBrtulEUcYRROHML/3EiloSaEJf3QG2ZppDhHDEj5HEUIZFKS+4qGtSLS9qdbjjVUqMntxtqqcJpoR9Jz2Lo95LF0Vo/crinSKmmGv3DXKXjni/dEZz6nCbmNSJ8oc3hNGD2lttVOpdGvBnBUIwURIKI9AbwUwK+l3RvhLT07F8DhACYDuEx3PTNfyMyLmXlxf39/o8W1mIBIisIRRHhvZ/M5jUqulUbIAFS2ZvIOBSG3gXtKOi6H45otjXQuJZVYFj3VlNxXwT21daQYO26SK962/rvUobPgYKzoxlxKlfqQ3WeejKboqRavEa7gowAuI6LDATwMr+5UAGa+pI4yvRfAX5n5Gan9IQDL/I/riOgMAGuIaAozD9Sxb4udCvFf6VjR+4V6ZRoag2A27Sght9pzG6M10vnSvQMFxT2l5peMKwsB1SKLidMAyxZikgJKdvEJUWUifNv28Vj/pjaqSe4T6Mg5KLmuVoklQVVkHOE0ZBnbq4zIiQCOB/BaACNQ3jkA9VYa365wjui/+am0FhMeJlcD0LhBWtdHmozwhkmTomExIOVzFBDhXshtlAgvlStnhCd2p8yWgfCZONLgmMrSqGCtuZroKdnSqBQCG3dPVeY0BIjI52b07ZluSyXkPUtDRE8pz66N8jS+C+C/AExm5knMPFn6N6VewhDRUQBmQ4maIqIlRLQfETlENB3ABQBuY+Zt9erbYucBK38jxxqvM8IZj5qnUScldueTG/A/9zxXQYbKLqXA0nCcyOJFKhGepvZU6sgwhdNwGsRpyAphi8E9peZHSOLFzk8zwxeFKUuuPq+l0vcgzpGjp1jROu3EafQB+BkzD1c6sUa8D8CVzDyo7J8P4HoAg/DcY2MA3tlgWSx2cOhmkpXKYg+MFvGt6x7DeMk1nFEZ8aJ8vjzakTV7+++56B6ceWVyXdEsM1xBhG8eHsf9z20FEJ1Zj6eJnkpZ+iPuupM4jSqjp2QI5VdmRl9PAUTAthHZPaVvq9IzSqU0/NE2miNSGar1xgzk/OQ+mdAXiZeNRlr31BUATgCwooGygJk/Yth/OexiTxZ1QlL0VKUksvNvfAIX3/Us5s/oxTuO2LPK/qOz08Bt30R3WZpmhZyFnIOSy7j/uS0AvMxmGWkywtPKorru5IKF1bunJOshcE8BecdBTyEXJN/J/attmaOnohOAJAgFKJSGQ/GQWR30yX36PI1mlBFJqzSeBvANIjoOwIOIE+Hn11swC4vGQbgo4kcqDdJjvoVRqqRdEiC8E8HsNOGH3hR3mXGw8v7mcw5c5qDG1KlH7ol/PL05OC/Ncq/JA3185q2GI6d2T1XoW17uNed49yZzMiwn+qXoQbSdJqlORDyJdyfnUEqLL6pY5IKFaln5ZiT3pVUaH4TnGjoK8dBbBmCVhsWEQ6Uy2o2CiTzVWz4NsjQq9Cv3XXAIpTIHA24h50T03HgaIjxJaWj6VFfuY5ZPTHB1aaOn5Pa9v2Vm5Ii8bG1DRnu0Lf1EI4ulEXIa4vunVOVR1Agpl6GNnmqrMiLMvFejBbGwaBZqcU+pbVTVv/83NDSCWhnGc+sNlUDVn+P9zTmEMnNA4OYcNXpKP+imlyUuV+C6k4jwNFVuK1kaQZ6Gy3AcQp4dJfor2T1lajsNpyFOKYvnSFQVEQ7WR0+llaNW2Cq3Fjsdkoa1SoNePX6TpoEx6dx6I02rzF6yWD5HKLuhpZF31OQ+OU8je38RS8BvihRLQw1VNcusaz/cKZcRyTmEnG9FBf0bFIWJT8kUPSXcU2WTeyrZTSjOinIaUVnahggnoguSjjPzp+ojjoVFE6HlNPxDhhGqHmN425VGTxisHCLkHAdltxzxxcu+8/EIp2F6btk4ifh6GpVn5NEW9BD3UPbvrZBTM9plfkVn+UX3hWVEErsFEOc0HKoc7aXK5PWp5zSY2yu572DlcwHAQv/6++oq0QTCw6u3oauQwz4zJ1U+2aJtkBS6mVQ2vV6IhdwGfevObZQU6dwiBCDnR/mIGXLecepbe0ohc4FkSyNpoNU9L1kkEenlugyHAHKciKUUnbnrZYz2F5U3CaRwGjlH4TRSuqeinEb02bUTp/FydR8RdQG4CDtxJdrX/+ivAIBnv/26FktikQWhqyH+K9WupSChHj9KdXYaVLnVchrVaw3ht9chlasHoaXhEeHe4JrPJbinalRyQZVbUTbe35+a09C6pzzkHQqURtn1ZusOkbLyoIHTMLSvuhqTIIhw1w2v0UWOqdApMpGnoZLk7VQaPQZmHgXwDQBfqp84FhaNRxIRLjjRRpYTUct/J6EWS6OoVrqTZTBsR/v2Zq45x7c0JE5D1p5plntNO9AHxLI/MkWT+8xtBNcnpJ4Xck4ga5nZd08pIbeGtkzvTJbaUyK5Tygux6GUyX1xTiZHcUuDm2Rp1EqE9wOwvhmLCQntrDSwNCpcW4d+Y8l92nOr70kmeE0yqNvqOZ6l4VW5LUtuFRlJq+BJrZllkY6pg7DOUFJbqsR3eDNwbzEkkckvyovncxThNFx16q5s1hJyqyb3edFT8n3or9NFfwXuKaXCcNsk9xHRZ9Vd8NbYeDeAmpd6tbBoJkJOw+yeamS+hrGMSJ27TFIaaeC6wtJw4Lphcl/eieZplN3kQTtpv3rMlBGeVBo9zcBLADryTpBTItxTBccxLiKVrgKt336m2lOh8k1Tol5VDICXlCj3L+RNKn5ZL6Qlwj+pfHYBbADwKwDfqqtEFhYNxOBoMVidL4l4rmRp1PLTVGfTST/0rG4y2dVSSnRPVR6sAk6DPHdO2ZSnIRPJVfjT5CtCYllYGmJGnaCQItsaXsjf1ZFzAtK+zF4fnhWlJ/J1RLjafpb1NISFFpQRcRJCfJkl11y4X9TO0q7cl1KOWmGT+zLCdRlPbxy2EVMTFG/72d+Dbd0YVIkIT7o2LbJkhGe1PsYjSqM291TIaXhEuMxpyIpOLpqX3TmlR9LKfaaB29vW9e0NwAXJ0nBdRs7PQdlelBWFftt0I6qrMQmOEnKbIwqUQKwb1t+76iKMZoQ3p4xIVZwGEeWJaKccNX96+wqccP7teGxN+6/9NDRWwubh8con7kRYvlYqoKzlNLy/pvG2Hj/JGBFO0f0ysk7c5eq7stslJkOKtkJOwxu4yn6YqqNYGmXDTF1tK6kf9fowT0MqIyKkVgfuFPdBEES44p7KJYXcxpWR2leW5D6hCMXzchyKrMpnVLgaReaQn+ehyNhyIpyIjieityv7zgQwBGArEV1PRH0NlK/t8I+nNwEA1g2MtliSyjj23L/gRWff1Gox2ha6Aa4Sp1EP2mHjkKfI1dm0tr+MpsaYpDSSifCoW0OHiKXhepZG3g9riobcVuY0khAhwpWM8EjIrWHgrmQ1sd9eR87BeMk7QURP5ZWMcNbM6pOQKblP5TRi66nov5OIe0oK13WINO6p1lsaZwKYIz4Q0YsBfBPApQA+D2ARdrKQW/GjbORyoPWCvLiMRRwll2ODclr3VLUYLZbxnRseB5DuB57ZPVVKx2mk6SRiabiMUtkNM5Gl86JErUnZpnSV+X8dhdNgJMzEI9FXek6DEHVPiegp2frwrg+vi7rd9JOJYOafgkwQ55SlMiJmy8zgMpMEdJSCh80qI1JJaRwM4Hbp89sA3MXMp/nl0D8F4I31EoaIbiOiUSIa8v89Lh07noiWE9EIEd1KRHPr1W8WBEqjYMt27QhQ/f4BEW4Yb2v9UYo1yIH4bNrkj8/UfinOaQyNlWKDXRpl5LLnjhK+d8/SkN1F0X68a/RtpVV+pjXCk9ad0CkdHTpyFBLhriHk1jBYm62c0F1UCY5iaThqyK3hOl1yn0Oei1CVMY3yqhWVRr4+AOulz0fDW0FP4J/wlmetJ87wl5SdxMz7AQARzQBwJYAvA5gGYBmA39W531QYK3oLthQcqzR2BKgunDBPozGWhtxuWLBQzKZr5zRkl0qpzNg4NIaDvnoDfnJbdP20NKvSuSwijByU/dLo+Vx8UCobBt1of2ao60UA0noakTIinKmtsG+P1OjIS5yGf295n+TXyZ8mlDhwT6WYTojkPrlasGvQGnJ3Zk4jqnTaxdJYA2BvACCiTgCHAfi7dHwyvKVXG42TADzCzH/wM9GXAlhERAub0HcEQZx3A+P4LZqHcYUsFhZGxa+3yu9fntWS8rce3ciulmLZxfoB7+f55wdeyNwH+4NQzkFgaYiS3LLSiXAa2cSN9R/OpKN/mRPSBivN1jkkwmPRU0rIbaU8DXVXluS+mKWRkBFukkOMO45DcCi+dGw7RE9dB+A8InoFgHMBDCNaa+oQAE/VWaZvEdFGIvobEb3M33cggAfECf5a5Sv8/U2FcC80ssyEReMhflslVWk02NKQB6hYlVtNl1nliFgaLhvdW2maDTkNjwgvl0P3lPmayj76JKghpWGehnnWryv3ET0uMsKdgPOJuKfSWBqC01DazpLcp95LTo1+MiX3yQrEDS2xGBHO7VFG5CsARgHcDG/1vtOYWY7h/CCAeobnfAHAfHgurwsB/JmI9oZXqmSbcu42eJZOBER0OhEtI6JlGzZsqKNoHsZKnnuqmiSmRuPVP7gDR3/7L60WY0JARK4UFfdUmNxX4fut8tcpD1Dqehq6HrO+ZSXFPWVCNOLINFj5nIZPhBddiQhPwV0kKQrXYJ2Ea3b4UVoBp5HEK0Q+aWTyMqVl95TLYe0pMxGu4TRifEp6S0M8u6JpPY2IxZSsyETos3p9y5P7mHkjgOOIaBcAQ8xcVk55G7zw27qAme+WPv6aiN4J4LV+H1OU06fAW4JWbeNCeAoHixcvrvvIPibNVNoNkRwEi0SIekpqLkPq0uj1cE+JonwJDqqsIbdRS6PWgoUhpyFqTwlOw3xNOsvGFKZbUiyNkO8J+4wN3An9CAQht5LSyDl+yG0aIlzfbKRibSWIc8qStZDK5aZ133nuqVhp9CawGqnYXGbeplEYYObNiuVRbzA8d+Qj8MJ7AQBE1AuPa3mkgX1rESgN656a0AhnfdncU7Wa/7I7LMk9Ve26HrKikC0N1X2SZlEj4e4IrTI3dE8ZrRP5erOcpgzvoPx64J4KZam0poXXlu4+vL+FHKEo8jT8svH5nKPkaYTX6SaGJvdUluQ+OU/DVIo92kdckREQd0+hOZZG24QAEVEfEZ1IRF1+xvm7ARwH4AYAVwE4iIhO9tfx+AqAB5l5ebPlDKtkNrtni3pCTbQSEONHo6rcaonwwD0Vb7UWTiNaiM88OzfBG4QoiPoZL7mB28h4TcTFIm9He0xracgFC02yR2bihoFezQh3WazcR5ES8rK7qcxKI5rOxUdKMZKGtafC6Kl0IbdxpeGF3FJcSbcBEd5MFACcA68Q4kZ4RRLfzMyPM/MGACfDW79jC4AlAE5plaBAe7qn6oE/3b8a+3/5+kiS2I4IEc+u3mdaIrxaTkvHaQjIXYYLM2VsXyXCjf4P7WYEIafhDRNjJX1yX6TZlO4pNeonkNl/PgVfU8kr9xkFrnAv7LvZYlVuyeNOmOW1w71r8jlHS1KbCxamT+4T9+gkJveF2zqehZQyIlm4lVqRtsptw+ErhiMSjt8Mb4nZtsCO6p4655pHsb1Yxtbt45g5uavV4jQM6nrNAmndQtXOGWT3UZpJYWZLQ1JK8hoYMfdUJOLI7GqSLY2xoqvN0zDJm8THlA1+LHn9bE/usF2jG61i9JQXOtyhRE85fvSU16+LnJML5C84lDJPI/1grYbc5ghRhZci8ky2NBwKa1dlyRepFe1kaUwoyDPNI75xM9Zua/9aVGkgXuhmvHythPiRxzgN/2MlS7LakFxZSanRU/VANHrKDT5XmxGOiKVR1maER9o1bSdYGtH9Hm+ilkZnTprty33G2y27jFwuGinFzMj5taeAcPYfLHKUc7TWkHofWTgN4RKVy5ub3G66elxAGHnlOIgQ4VmUV62wSqNKyO/8hsExXPPQmtYJU0eIH8qO6n4TcCRyV0Zq91S1SiPinvL+CgWtzWbO2I2aEV7T9ygsDV/OsRSchokIV6Uwuqf8/AmBwNJIzNMwffBQLDMKOQcdeQeu74oqc1h7CohzlYWc3nUUVxqhu6gSxKMTwRCOEj1ljpgKPwg5VU5DnNIOZUQsDCgr4YwVrPaWIkvYpjg1qaz2jgA1Zl4gzNNIvr4+7qnKlkbmRZik9ouum+CekrYTXC8OATkxsMoFC1O4UpL2y+7dSMSSkkAYKVjI8fPVtnW9F8suOnJOREGUXW+A7ciH9wZIloajWBrCbWmwctKtER4NuVWjn0yQ3zXxuyQifxGndJOcesIqjSqhDjaN0vDnXr8cNz26rqY2ssw2xbk7utII/MvGPI3kZ1btDD7qnooe0/vjs0GWq+yaLY207inP0vAElTkN0+WVXEWBbHKYq1KSJBdRGqEsqVxiWkvDk7vgyz5edgOFGCiNUhhVBXiLM0WsJsN9BBnahuMyYpyGEj0l9xK9p7ilQYiH7Mp9NBJtQ4RPNKiJU/Wu+fLYmgH88q/P4LqH12LT0BheecCsqtsqM6f+osUMMGnVtx0B1eZpCFS7hrgud0K8OboWs1sa4fnFMhsT/NIs9yqaCohwKXrKBFPeQcw9ZbI0XA7WvwaiBQtN8lZSVMWyi0LOQaevIIpl14+eomDfWClqaRRUTsNg5WTL01AsjaSMcA35LWQXbcml0bO4yWqFVRpVImZp1PnL+sil9+K5zSMAgFG/3tVFf30Gc6f14IQKCkReXxgwl/nWwd2BLY27VmwMtqt1Twn+oVqdqro1geQJR22chpsqpNM4gxechu+MHy2W0eEP6EnX6A+ocrraQ6qlAYShpeGaFmrTZr4DCDkN2T3lshc91RHjNIR7irRh52rzWQbrnEK65yjZGgv7CLdDTgOR0uihm6yyHLXCuqcyQNb+al0fdRWuWiE3J+pdnX31o/jwJcsqXqu6JCqFB59/4+N4ePW2yLnqYLoj4F2/CKvUGKOnUloa1bqnkp6r1j2V1dJQ1gg31Z9K0yozw3FCS2NkvITOQvLiY+bS6NH9puQ+ET0lQ/j+0ygkXf8epxGS3sWy61e51XAa/uPL55xUZURE1ny6goXeX6Olob+lKBFeFkqK/NLo0ffVhty2GaKJU9HBpt6+RLm10WK2Wb+6Ip14Se95ZjMu+fuzkXPHSy4u+MtTOOkndwEIf8Cqr39HgzrrEzC5IQQqLQdbCTp3UeieireZtZuoe8o1K7cUA2LAafiWhstAV96pcI22ixhMcuksDW9GLYW9KteYBlsBj9NwAgVRLLtB9JRYgXO8pBLh0TwNBN97tG2Ry5IGIafhZ4QnlBFRt1WLSKynEZTy98+17qk2gzwrjZmuGb8sZsZPbluB1x+yG+ZO7008d7QYK/uViPFydLYmXE5v/7m3FErOIbx7yVwA4QusriuxI1oaMsRAGFtPo4KlkTa6Sgdmxg9vfhIAcNTe04P9QRkRTZtZ+4kR4caZvyyXvi2XQ8JVQKxYGS6WpF5f2R0Wl9ON7C/konNZUle4U91TFbRGscQoSJbGWMmF64ZZ4oCJCNcpVmWSAU7tElKjp3IJ62nIcJmDbPbxsjcWqCv3cbC2urU02gryQBpzAWX8da8fHMN3bngcH7j4n9rj8uxlNGNJj1KZIzNOdeD40lUPB9sm5ZB6fekJipwSMy9QSWmIwbKaigAbBsfw7CaPp/re2xcF+xM5jYzxUyoRXkueBsOTracjdEl15aPuKdWVZCKsVSnKipwCSZxG2Eh84NZtB+27rp+nEZaO8SwNKXrKH4zF91tQQm6D9jWWRtqBWrsIk+F5qS63wEoqiWuja4SLay2n0WYoKv5iGVndOWJms33cYEXInEZGS0POBAaS6ySZ5Jbv9eoHX8CDq7ZmkqHdESb3RZ9NJUuilrj4otSoLkkuOvsX/WTrI1yPgrz3QPjA1b5Mg5Uig0PA9EkdwT5haQjE62fp3S0mOYHou6Zb6CngNFJYMaaQ246cg46cp/Buf2IDyi5jz2k9cSLcvyafo5jVpoMI3U2DMLnPtzSCbPe46yu6ngYi4cKAx104Tvw9aUa6mFUaBqwfGMXl9zwX2Sf7v1X3VFZ3jnA5rdk2ijXbtieeO6b09d93Ph1TBJFY7rIbUQZJs2JTaK18P2f89n688b/+lijjRIMY7J7dNIw/3rsq2C+eq4mzEI+1GkpDniBoV8DTuUMyE+He+V2FXKT2VLyryoO74DRmTOoM9gkOQFyjWgUpKBQA8cz1YNuNh/XKM2pdW5VcbcUSR/I07nxyI4iANx06O5jBqyG3HhEetiFkVJsXEWZpkFM5DccciaeG36qEPfmcRlmZxNiM8BbitEvvxRevfChSUypqaUQH8qzunBFpAPnAr6IuqusfXoNVm0NFonIa51zzGB5QZv7qC16qMEsSg4apmm1Qi2cHzdco5LxFbC67+zn8+x8eiKzo5v3VXxe4p6p4LhGloZQQUMedarmTsut6Cw7lHZRcNxWnYYLn949aGl2KpaFGDaappeTJqbc0Sm7c0hC++yBIQZEzTUZ4Ieeg4A+827YX0VPIoauQC/I0VE5DLVgo16yK3K/LmYlwOSNcbtP0nbgcEuHFCmVErKXRQmwaGgMQHVQjL7diWcgzc9dl/O6fzwWhsjoMj5eC7Y1+XwIf/c19EYJ2rOTGXlbVvRFZfMd1I/Lp9Jl4cU2Whmhv6/ai8R4mMpiBqT3hYCivswB4z+dP96+OXVeLe2p7UbY0kt1TQbROFZxGXqxIp0weNM0nguHl+3RKPEZgafhy5RLq5yRZM7Iyk991sXa3DPKvN4mszspVFF3hngqVRrfP06gzeHF9zqGIjGLNDbV1j9MwCKYgKI0eEOFhG6rscj8ep5GLyClqgqkVDCwR3kIEkQ7yi1PWbwPAH+9dFczKb3x0Lb5wxUP4gR8po4M865R/JLqXfrRYjv34hQvqqvu9flViUVYiZWas3DQcuV7Ib+I0hLLcPDymPT7R4TJjWm+oNNTkLgD4zO/+hceVJXTVmV0WjEgThZilobQpNqvhNEQhvmKZUTZ8v1GXjslfHxKr3X5+RhZLIwlqaHCwv8wxhSrWnUhT4kX3vIR7qiOvURo51dLwGigo62kIAjreH6ee3etqTwGmUGtp0sdAh/++iHEjJMKjVkozyohYpaHBv57fipV+lIv8Qie5p57ZOIw/3Ps8gHBG+cJWM1chu6fkH54aAgr4SiNm2bi48M6n8W+/ewB/fvCF2I+wpER6vfQ7t0WvN4TahvfnXb9pyFvNt0MKg3x49TY8vaFuS8O3BGVGRGmMKe4Jge2Ka1D8SKtxT40WzZyGt160PFCIATJbH56l4fhroLswUW1p8kyYQ9fL5C4vOr8ipyG9TlH3FJTz9JyGztJw1JDbmJyRT7H7CNxTkoIQSjAWchsk90XdU+L3Hg/35dQ8grrcq7iu0lfBfk6JQ8DQmDfx6OnIK+tpCEsjlSg1oW2UBhF1EtFFRLSSiAaJ6H4ieo1/bB4RMRENSf++3ChZ7nlmU7A9VpQVhX52JLBlxHPliCiNpNXv5FmnbOKrpDfgDWTq4FUsu1jn8y1bhscjP7yi4pbQZsn6/URIyMiMT1gantIQgwYAvP5Hf8Urvne78d50eGHrdty7cnOma+oJdb0T1+WIr16daZpQi3tKnihUciME/vuM/YhBN58jvzR6Za4tMU/DF3OS//0LS0Ncoio/Y0kPpRPxfsqr6Xn74ws9iXUjRAtxItzsBgO8iZGnNMJ2uzvyQf+ALrnPI8JV7k/nnqqW0wijp+Kyq+4ph7z1zAdHhdLI2TIi8BINnwfwUgC7APgygN8T0TzpnD5mnuT/O7tRgsjJRaMSLxG1OuJvp3gJ1BdRh5FxvX97zJD9PTRainwult1gVvzc5u2RmX+p7EaUkm5WvG5gDA+u2hqxmGTFJMoVbPM5jZ5OffmIZzYOx1xfOnzy8vtx8k//jnUDrVms6uXfvS3yWXVPmSyNOPEJ7XlpoCp+Gap7Koy/z4aSX4aj4HhEeLColjKYJM3cQxlkS6MAABF+A4hH60SeS0IfwtLoyjvRkFttRjj5A7hezkr3Uip70UfidwkA3b7yE0rv9ic2AAgtbxFaLO7HxA1lCblVi2SKZxe6mPTKz3U9hZN3CAOjRV/+XMQC2ynLiDDzMDMvZeZnmdll5qsBPAPg8GbLIisNeRAvambiMkJz05+dJORuyEpDfulM5LkwSwXGS2Hc+i//9gze+rO/h7K5HHGN6ZTGOy78O974X3+LWDaDkmIS9yeOi/UF/vls1Fp4+Xdvi7m+dBD3laXM+5PrBoOAhFoRdzNFiXAdpwHEn11gadQYPaWCSM8z1GxpiFDRhGaSBmNhEU3xLQ01e10d4NPKKwbhrkIultwX4zQo2m7MPaXIrKLoV0iQXaw9HeJ+PPmXrdyCzcPj2OJb1iLMWF0qILYCItKTz3mHkHMoeA8KGt5UB5e9+lZ5h4LfaHdHLrKeRhA9tZNZGhEQ0SwACwA8Iu1eSUSriOhXRDTDcN3pRLSMiJZt2LChqr7ll0v2Q0eIcM2gIaxfoSxMVgOgkKKypWGwTlSlIVsaKsZLLlZtCZWGzpUiXj7ZbSNmMUD4oxaDvUPABbc8ibdJyml1AmejYlpvZ6yPSnjl9+/AK79/R+rzs8B1ORg4gPgiPMF+5fsIQ2GrVxpPnPOa2DF1hhjOILP1USqH0VNFN6mMSFpOw9ue1Ok9q+GxqOJTifDorF/PWwDh8+vuyMUsDXNyXyiXri0gzjUye65amdMQ/apYs207Ng6NY3JnPnDDCQJe/PbV711+RpVARNhtl67gtyVkEGOM6dmx7wKTy7X3dOSiRHigNHYiS0MGERUAXAbg18y8HMBGAEcAmAvP8pjsH4+BmS9k5sXMvLi/v7+q/gt5Pceg8/krcgMIB5qxsostw+NaQlzwH0DUxDe5tIY1SsM0qxsZL+PqB8PlZ5NI21uWrw+2H3h+a6R9IFR8DhEeWzMQufbob//F2K6Mx9cOBs9LdbNVguBU6g2XOXBRAPE4fYExwyJNWQfzgdEinlg/hIIUxaNCN2hk1U0PrNoK8v3fJalgYXywi/elwuUw7v81B+8GAJjf3xu5RrY08uqa11KzqqUXJCHm45aGGsZLviwm607uR81pEm135MP1NIAwGgwAfvvhJQC80j6bh8cxbVJHuJ63G+UH1Sgq4TpKi9l93aEMQmmMa0j2iKvSqzYscz1d+Vw0T6OJZUTarmAhETkALgUwDuAMAGDmIQCiJvg6IjoDwBoimsLMA/qWqkfBaGmY8zTk4+LveMnFUd/+C7YXy3j226+LnLtifchByPXZTJbGoFZp6OVftnIz/iUpgKRZ8TWScvnCFQ/F7mVMIgCTBsqhsVIwGxVYsWEIrss48Qd3RM6rJ4RSkfkJGQOjRQxock3KrFgaBiJctRbVDFwdtm0vYqxUxszJXcG+U37+Dzy6ZgC7dBf0F1F08K7GolmxYQhPrPPeqzlTuyN5GurEoZJLR/QtBsQ3LtodL9uvH1N8bkPnnprUlTfaL6prTjzHroKDYemYztIQBQvNhTvDXkfG4wEjgJfMmc85mNpTwJaRYqSe1h7TegAAGwZ8pdHbEQm5l3/rqvtYfkZpMGdqD+5+xnPxChlGE/K55D6ER6Kr4MDxo6nE+yGUslrssRFoK6VB3lT9IgCzALyWmU2+DPEtNkSvRjiNSHJfsntKnBtYGqWykfx8SlIasnVhqjOlWhrjZTYOKPeu3AIA+I8T98N3bngcOmqlr6eArSNmV5H4oYgfiWcpmAewx9cO4PC50zAwWkS5zFi9dTte/6O/Yv6MaAXftJZG2pDWF519EwDElLLAa35wp9aN5rqMLmngCAvWRc+LV8EVf83yHXferdi2vRiR6VHfSpvSrf/JxWtDZXd/yVZZIedgZLwUzM5jSiNF8zKnASBQGDIiSqMzrxQsDCEns8rydBZykQRSbRkRv8aSUAhJ96IqDfGbFL/pWVO6sGWkGLE0+id7rtO7VmzEpuFxzO7rChSB6zLG2Y21J5AluQ8A5k3vCba7C967IBSqwdAIiiIKS0NMdvKOE4wdT2/wglH2mpFcMbseaDf31E8B7A/gDcwc/NKJaAkR7UdEDhFNB3ABgNuYeVsjhOiIKI2ytC0pEI1FIAZ8VXmoeGLdIDYNj+M/TtwPxy3oj6yXYeQ01OipkpnTeHDVNnTkHCya0wcA2DIyHgk3BCoPGtuVeymWOdHSOOeaxwAAS75xCw47+yas8KO5nt4YjaxSLSYTspaDN8HEu7gM9EgDh9nSiMohu6dMkWDbErLoE6Nb5AG3CktDvCO/+sARfp6GZGnECFwzsSyQJjJIHuAndxWULPBwO2ZpCL9+IRf5LSUVLBTv5LDSliz/iKKcxHchLLxe3xqWreKuQg6TOvP4079ewGNrBjBjUmckEU92Rau/T85oabx18RwAnuXT60ckbg84Db1rT/Am4rkIhTdjUgc2+nlUT64bAhGwd/+k1LJUi7ZRGkQ0F8BHABwKYK2Uj/FuAPMBXA9gEMDDAMYAvLNRskTdUx53sEWKrAD0taZGpQEW0Ptgl68dwKt8cnfu9B7svktXMECWXTYOROpgu3V7MTEaZ96MnkBRfOTSezFjUif27g9nIarlomKNT5AL90wShyJkB8IfgOk+0loaaZSG6f7T8CAuc4QMVQvWCQhLY3C0iNFiOfiR3vDwWiz55i3465MbUQ+IQV4gJH3TXT9aLAfPfI+p3SjkKFIafbRYjgQhJOVQyLtN46FcbkNgcmfeKK9qAQi5ugpOxGof9xdMkiEKForve0gJppD7VIn6rSPe9yUi5Z73l1BetEdf5Lzvvm1RsH3Ynn0Bz1iWSHAgPonIEnILALvt0o2/fO6luPHfXhoosErvesn1VhkU7inx3s6c3IlNw2MolV2s2DCE2X3dWoK/3mgbpcHMK5mZmLlLysWYxMyXMfPlzLwXM/cy827M/F5mXtsoWQqRZLsyLrzjaRx29k145IVtKOQIkzvz2jwN8UKJWWtZmXVtHh7Hq39wZ7Bv5uQudBVyGC2WMTxWwpeuegj/8ccHI21e8bGjAMQH+QtueRLXPLQGJuw5rTfygx4vuThy/vTgx2GKOwe8cEMRfRW4p9xkS0N98b957XLteWk5jTRriBzw1etj+5avHcCLzr4JF//tmcR1zl03qjTUjGABoTQPXnojFn75+iAYQCiTh18wG7uDo8VgcO2t8GPu6466C7MmEb78u7fhzCs9TmpSZwF5xyPCRXTcuoExHLL0xlRtyTJUmkXL0VO9nTljwULVAgiVRi4Szrp1pIipPVE3GJF3vpiQqO+QbDWpEwnh+trFb/OIedMAAIfPnRo575UHzAoU5Evmzwjuy3VDXqSr4GjdU1lLd8zvn4S9ZvQGFkMQPWW4p4HRIqZ0FwL3lLhu5pQuMAObhsexdmAUu+/SjWagrTiNdkEhH7U07vRnkw+s2oZpvR0YLbqBpXHmaxbi29ctD84FQv94hKsoubEZ8MzJnegq5DA0VsKBX71BK8t0n+DNGnU0Z2p3JCprrOSiI+9g35mVzde9+3vx6JoBDI2VJPeUm5iclnRMxqA0S3R994kumkhWQuMlFy77oaTSLFQ3nj7ru8OW/vnRSN6JCo8IlzmN0NLozIeDw3jZxfqEhMSkWebBS2/EN99yMN61ZE/0dOYxPF7Gkr2mac+dNqkjUudL3Fta59QaKXR6Ulce+Rxh0/A4nlxfeW5l6oNhHhB1RHjOiZYTT+IaAqWRD91TA9tLKLkcyZ8BPL5ny0j421F/CxFLQ3VP+Yq4z3dPnffWQ/DZVy0IZvmh7IQbP3McBkaL2HN6D3JP+3IyB0pjUmc+tvSyy1w1s9pViLqndO5JANg6UkRfTyF493skSwPwrPoNg2M4YPcp1QmSEW1jabQTZE5jeKwUfF65aRjTej1/pyCKD9p9F9x15ivQkXcwWiqj7HK4wJI08A2PlWLWwswpnZjSnTfO4BfuOjl4sYbGsysNeRY4NFZCZz4XK9Ggw76zJmFwtISDvnpDQNgPjpZwzzPmMiDbxytbBoC3ep2YDX71/x7BgrOu05LestJYcNZ1WPjl6xPX9CiWXbiKAvrJbSuM5x8ypy9ChsqcxnSlkOF9z201tlNplnn9I96g3duRw/z+Xpz95oO0503r7YxMKuS1yLMkEhJ5XE1nPqd10+mSGHXKV/RrdE/5f2WloSbhyVB5CGGFe3ka3vZGX2nK5V0AYMWGYdy1wivtM623A8Pj5cg7I1s34t1yXU9+4Z7q8xVRb2fe6Pffd9ZkHD53mn8vIREurPLeznwseoqrsDQExPuX9NsplV0MjpbQ192BTn8cmucvDz1zihedt35gDOsHRgMl0mhYpaGBzGlc+o+V+PvT3gvrslcy3SEKSxQ7wO593Zg7rQf/+68XsPd/Xotf3PkMAERmJYefc3Ns5tvTkUdftz5U9LenLcHlpx0ZJBlVY2moUSgdeSeiEE046UVzgu0n11cuTJhzKNXqgsct6MfweBlXP/gCAO/ZAsDe/3ltZLAZGC1qyeRH1wwEC1apCviYc/+CI75xMz548bJgn2r9/PL9i/G704/E704/Ej885VC9e4qBqZHyImVsSqj0WymZSpCXI+NlvHjetGASoGJ6b0hqAuFAfvUDazD/P68NfPGV0JXPwXEIrztk12DfGxftHmwHkToV9NBeX7wWT28cNt6fyDeYL/Fk6nrhUbeRKXrKQdF1A/ctECaC6iAGRtmikHkMsf+Tl9+P+f95beCemtKVzaki14kS70ZvRz62hG5WTkNG3D0Vd+0N+L/7vp4Cdt3FUxLz/Agp8Sye3TSM4fEyZk0JQ7wbCas0NFAjjWRM7srDoXh5Y9NgIOPUi+4GAFzywRfjz2ccA8B7GXQ4dI8+TO3tCGr9VCKuVRziR07J6Mw7MZJRh72m9+KKjx1lzinQtJvGPXXcvjPQkXOwwg8PlKNkBrZ793fpP1bikKU34l2/uFvbxppto3jg+a0xd966gTFsqkCAH7NPP5bMn44l86ejpyOvtTSYozWpfnzrisDlpYNa7kO1moTi3j5eTiQpp/V2YP1g3A0mAiAeXBVyJ+Oa9VUExAx+yV7Tg33H7z8z2B4peu2Z/OcqTAPiSS+ajUs++GJ89Q0HBvvUNSjkZ6ES1AFPkM+B2TtXVFSeruTc/OidhwXbIjxWnkSJPJxpvR3YMlyE63LA9z27cRhTuvKp3nsZwiKXI9BE0UbZ7VyLpSHqW4nfTtS1530ILaUCDtuzDwCwm688xLMQOVnW0mghkhJkLv7AiyPuqVBppH+Ue83oxcFzdgFgVhoiFltksWZNitu9rzumaDrz0UqfrzxglvbarkIOh8+dioNmp/ORduYdlFxOJJ4B75527+vCqi0jQT8Cm4bHwMz4ya1PGTOmAeDGR9bhTT9Ov/TsF169MNhW25WT++QfrupTvz/BPTVaLOMLf3wQr73gr2DmmE+d/TIUw+Ml9HaYZ7v9kztRLDN+frvnUlOj88Rs9JEXtmHBWddFAiZ0a1nLPvvF86bhZN96FIO3GNu9BDFg3pnX4NzrlwcyC5gGRCLCcQv6Ucg5uPmzL8U1nzoGfT0dEZfYc/7yAnOn98Sey7qBMUzv7QizoksuNgzp3VNH7R0qwEBpSO+2iAqb3deNx9cNRt6PP/3rBbzmoN2095CEKf6Eadv2YoTTAKJKw6sLVaXSyDsgCr9bnXW9VQoZfu9L5uFnpx4eWI5inBLVHw7xx5RGwyoNDZIGrT2m9UTcU0K/qANNEuQYcZN7SsBxvLITwrX1Hyful3j+1Z88JrBiVPeUpzTCe5vd143dd4mbtGFIXzpzVwz+J/3krsTzZk7uRHdHHlc/uAbzzrwm8sN/xfdux/rBMazZNorXHWz+kV/mu7TS4kPH7GU8lnMIv/nQEvR25PCc7/5xmWMWlroQk4ztxTJ+t+x5PLZmAMeed2ssQun5zdux1xevhcv6ekcCbzvcG9S/dd1yvP5Hd2LdQNQl9rk/PIDrH16L113wVwDAzY+FhR/lWffuffHvrK+7gFcf5LmrwkQyf/bcmQ9CdX/qc0CyW9U0qZGxz8xJOHD3XTBzssfLiEFVuDaP2nsGto4UcdFfn8Hic27CPv95LS6/5znMmdoduLme2zSC+5/bgmm9HdhVcbP0Sb8toTTkAVb8Nr78+gPwniPn4qHV0Yi2k140u+I9qBDWzubh8ZjSGFMqX1frniIidBdygfyywhV6W1gau3QXkHMIrz5oV+P6Hc3I0QCs0tBCHVgBz5Vy67+/DIAfUx8s8O59gersKAnyLFD3o1T9r115J1gS9vWHhAPqx1+2d+zag2bvElgxh8+dik8dv29wrDOfi9TfcZlx1SeOxg9POVQr58wpUXO3U1Gm4t0VSkP+sc6aEjeV50zrTnT9ifIKL9vPXDMsKVRYh468g1s+91Jc+qEXa48fs+8MHDl/eqDIto4UY5ZmUkLij28NyXa5SKTA4+tChZMUdjt9UmcwU3x4tRfWu8e0aAjlN699LNiWgxzk/Itj9ok/u56OXBBxMzhaxNHf/kuwquTU3g48+kK0Es/gWNhe2omDfK54V59YN4jejhyOmOeFt5599aPYODQefIezp3Zj31neQLd87QDuemoTluw1LTZzlyc/+82aDAA47ZJl+NJVXoixcE8dtmcfvvbGA3H2mw7EO1+8R3CNmpORBtMCpTEWEPW9gdIIXZkPrR5IFZFowoJZk4PfzaaheCDEhkHvWfZXcD1dftqRTSlWCFiloYU8sH3g6HkAPNNPpOg7FBYcFD9edZ0BFbKfXLZkVKXxw1MOxbWfPjayr7OQw1jJxV4zejFnaliG4Jh9tIV+AxARPi0pjZfsPR19PR04dl/vuo1DY5g1pQsnHhiSprKbbZYyYLz50OiMTZSPnqFRmMfuGx+8Zvd14wfvONRoLX3q8vsBRP3xKtKG9l7/mWPx29O8QnR790/SyiOwz6zoj/7UI/fExR84An/53Etj575sv3586hX7xPZ/9pULYvvU53LsguQCmnLE2CsPmIV3vXhu5Ljs/hPPYfXW7Tj2vFsBAC9d0I+vvuGAWLtEFCiNFRuGIlnyfT0dWO5bUuK9lC0XnfI3QZwrLJd/PrsFi/boi7yzMuZM7cFeM7x8os//8UGsHRjFWw5LtgpOPHBXdOQcbB0p4k/3r0ap7GJg1CsLUsh5NZne85J5+OZbDsbSNxyAn7z7Ran4RhXTfTJ+0/B48DzEQmTC0li+dhAbh8awZL75fa2EI+dPx4OrtmK0WI6EXK/2JyDrB5KVxtWfPAaXn3YkXrJ39TJkhc3T0ECeab5sv36cc81jEXNYTvBxAqXhXXPU3tOD8EDAG4RHiy6O3mcG/vzAC7G+ejry+P47FmHPaT1YtWU73nRo/EezcNfJ2DA4hk+8fB/kHMKhe/ThX89vTZX9mXMI//uJo7F66/agMNvHX7YP7nxyYzCLkSOqbvxMOFC+ftFu2Dg0hn89vxV3rdiEuTN6cOmHXozv3vA4Hli1DR956d7IO4TJXXn889ktwXV79/fi6286EH+8d1VElsldBUzuKuATL98Hq7aM4PJ7nscpR+yB//nn88E5B+4+Bbvu0oUfnnIofnjzk0EZkrPffBC+/KeHg/OO3XdGkD/z2oN3xcJdp+D8m56Qnln6mPVTl8zF4GgJv737OUzuzGN+/yTM9039b510ML54ZVjI8TUH7YpXHbArLvjLU5E29tt1cuTzv79qAd531Dwc7LurvvDqhRXdBzMmdQYFBw+ZvUvMRy2+L8CLxpp35jWR40vfeGBkgLzhM8cF/JHgb0T7h+3Zh1OXzMX/Se+kcL/IbsO0wRBAaGmsGxjFtu1FLF87gM8cvwBzpoYW0+sO3i0gqY9fOBOd+RzOPfkQPL52AP2TO40826UfejG6Cjn0duYxv78Xy9cOYni8jH2+dB2AuHIjIrz/aLNrshLEb+u86x/Hwl0nY8akThw02/s+Tjg/LMDZXcjhVQaZ0+Dg2bugWGYcvPQGzJnag4W7TsbytYO44r7V+PrVj2LfWZPR11MwTkqFTM2EVRoayEpjn5mT8f13LMKCWeGg8OXXH4CPX3YfgDDET1gSx+7bj2P2nYEZkzrx+T8+iGP37ceSvabh7UfsgQ8fs1fgO5fxlsM8f/bhc2OHAADfe9siXHHfarz5UI8Au/gDR+Dye57XRkjpsGiPvoiJLkL3hDnrOITz374Ii+dOw55SQbWZk7vw+VcvxBevfAh3rdiEfWdOxrH79uPah9bggVXb0Jl3cOqRc3HZ3VGe4cPHzo+QzNd86hg8vznqujlh/1m4/J7nMTJexqeP3xc/vMVzl3z8Zd4s/k2HzsbMyV145y/+AQB4z5FzI0rjF+9djF/c8TQch3D6cfPhMmO0WE7MzTBhj2k9+OZbDsaL9pwaG6hPOWKPiNKYMakTU3s78Mv3L46E97543jT84aMvCdYbOeMV+0baERZrEr7/jkOx5Ju3APByZWS3R0feSVwJEogWwwM8RSaUmbA0RJjz999+KObN6MUdT4ZrzmweHsdR37oFL8iJgp3ph4i9Z3qW+Ed/cx/OfM1CMHsW+m4Sb3b6cfPxuVctwF0rNgUz9LcePkfbngzZUpwztTuwjgQaWd11+dpBnPW6/bXh6v9x4n6YPqn6qCXhniuWGc9sHMabD90dq7duxxX3rQr6XjCrOVxFWliloYFKIItBXeC1B++GB5e+Cr/+27NY7JcjeN9R8zBWcvGBo+ehq5DD9Q97s6npvR348LHzAcQH77SYOaULH5P4i76ejuDzRe9bDIcIH7j4n6nbmze9B199wwERt5Scm6HiP07cD3tO68HxC73QTTHrESGravKZWj33wN13wYG7RwfjVyycibNetz9O2H8WduvrCpSGPCsVIYYCP37Xi/C3FRvxhkN2R1chh08eHx2YP//qhXjpgv4gtj0rdIMXEeG/3nUYlj27BRff9WzgknvFwnB2eeF7DsfU3g4c0TsNv//ISyJuBoE0LpJZU7oCC2rO1J6IS6J/UmfgVnrNQbviuoejmd4XvufwRJ+2mgG9m0+YT1b4M5kzesOi3YOyG2kgTxRElYR9Zk4CEeH/zjga/3p+a/D+z6+BtN1v18m4+bFwHZgPHbNX7F2pB354yqH49P/8CwDw7iVzwWD82wkL8P2bQ4s2zWQgCSJRT+CYffuxaI8+fO3Pjwb7svBKzYBVGlViSlchMmh1FXIR0vmE/WfhU6/YBx/yFUajcPz+3uD1y/cvTl3cjojwgQym+7TejojS+tyrFmBKVx5v9C2fty3eA+sGxtDdkcN3bng88Lf/4aMviZS3UGX4sObZzJaURlchh/NOPiRIIHvdIbvhdYckh0/W4l824fWH7I5j9+nHlO4CDpRKNVz0vsUgiiqQFytlQq742FGR9dsr4dyTD8GV963CAbtNARHh++9YhDlTe7B87WBgaX33bYuCRK4XzZ2KjpyDV0kTAB2m9hTwbycswPK1A1i0R1+g+LcMh8lvA6Ml/Pur9sMXr3oIZZfx769aYIzUMeGyDy/Bb+95LlinRQSSHDKnL7VlXAkff9k+YAYO2H0KHCK8NiHarha86dDZcJnR1xOGBn/6hH2x/26Tcctj6/Gy/fprJp878g7OefNB2KW7gOVrB/CmQ3dHIeegtyOPz1/hhVUftU/z+Io0oGrq9k8ULF68mJctW1b5RA1+fdezfq5C832GExWjxTLOv+kJnPGKfbRrLyThsTUDuO7htfi3E/ZtWhTIRMPv/vkc+id3RpRUrXhy3SD+/MALeNNhs3Hlfavw2Vfuh9VbtuOye1biCycuzKw0BO54YgNWbhrGe14yr26y7my47O6VePD5bfjKGw6IWYqNBhHdy8yLtces0rCwsLCwkJGkNGzIrYWFhYVFalilYWFhYWGRGhNGaRDRNCK6ioiGiWglEb2r1TJZWFhY7GyYSNFTPwYwDmAWvCVhryGiB5j5kZZKZWFhYbETYUJYGkTUC+BkAF9m5iFm/iuA/wPwntZKZmFhYbFzYUIoDQALAJSZ+Qlp3wMADlRPJKLTiWgZES3bsGGDetjCwsLCogZMFKUxCcA2Zd82AJPVE5n5QmZezMyL+/uTC8RZWFhYWGTDRFEaQwDUCnRTAJgXOrCwsLCwqDsmChH+BIA8Ee3LzE/6+xYBSCTB77333o1ElG3VnhAzAGys8tp2g72X9oS9l/bDjnIfQG33YiifOoEywonof+AtbfxheNFT1wI4qlHRU0S0zJQROdFg76U9Ye+l/bCj3AfQuHuZKO4pAPg4gG4A6wFcDuBjNtzWwsLCormYKO4pMPNmAG9utRwWFhYWOzMmkqXRbFzYagHqCHsv7Ql7L+2HHeU+gAbdy4ThNCwsLCwsWg9raVhYWFhYpIZVGhYWFhYWqWGVhoWFhYVFaliloWAilWAnojP8OltjRHSxcux4IlpORCNEdCsRzZWOERGdS0Sb/H/nUQvXWCWiTiK6yH/eg0R0PxG9Rjo+Ye7Fl+k3RLSGiAaI6Aki+rB0bELdiwAR7UtEo0T0G2nfhLoXIrrNv4ch/9/j0rEJdS++XKcQ0WP+WLWCiI719zf2XpjZ/pP+wcsB+R28elfHwKtxdWCr5TLIehK8MOSfArhY2j/Dl/ttALoAfAfAP6TjHwHwOIA5AGYDeBTAR1t4H70AlgKYB28i83p4JWLmTbR78WU6EECnv70QwFoAh0/Ee5FkuxHAnQB+MxHfMV+m2wB8WLN/It7LKwGsBHCk/5uZ7f9r+L20/GVsp3/+4DUOYIG071IA3261bBXkPkdRGqcDuEu5r+0AFvqf7wJwunT8Q/KL1Q7/ADwIrxz+hL4XAPsBWAPg7RP1XgCcAuD38BS7UBoT7l4SlMZEvJe7AHyoFfdi3VNRpC7B3uY4EJ7cAABmHgawAuF9RI6jze6RiGbB+y4ewQS9FyL6CRGNAFgOT2lciwl4L0Q0BcDXAXxOOTTh7sXHt4hoIxH9jYhe5u+bUPdCRDkAiwH0E9FTRLSKiP6LiLrRhHuxSiOK1CXY2xyV7kM9vg3ApDbx0xYAXAbg18y8HBP0Xpj54/BkPBbAlQDGMDHv5WwAFzHz88r+iXgvXwAwH55b5kIAfyaivTHx7mUWgAKAt8J7vw4FcBiAs9CEe7FKI4odpQR7pftQj08BMMS+vdoqEJEDzx04DuAMf/eEvBcAYOYye6tMzgHwMUyweyGiQwGcAOD7msMT6l4AgJnvZuZBZh5j5l8D+BuA12Li3ct2/++PmHkNM28EcD6adC9WaUQRlGCX9lUswd6GeASe3ACC5XL3RngfkeNog3v0ZzoXwZtFnczMRf/QhLsXDfIIZZ5I9/IyeMEIzxHRWgD/DuBkIroPE+9edGAAhAl2L8y8BcAqePKraPy9tJLMacd/AP4HXgRVL4Cj0d7RU3l4ERLfgjdD7/L39ftyn+zvOxfRCIqPAngMnpm+u//StDoa5GcA/gFgkrJ/Qt0LgJnwiONJAHIATgQwDOBNE/BeegDsKv37LoA/+vcx0e6lz/8uxG/k3f73st9Euxdfpq8D+Kf/vk2FF9l2djPupWU33a7/AEwD8Cf/hXoOwLtaLVOCrEvhzTbkf0v9YyfAI2G3w4samSddRwDOA7DZ/3ce/DpkLbqPub7so/DMZ/Hv3RPwXvoB3A5gK4ABAA8BOE06PmHuxfC+/WYi3ov/vfwTnptmK7wJyisn4r34MhUA/MS/l7UALgDQ1Yx7sQULLSwsLCxSw3IaFhYWFhapYZWGhYWFhUVqWKVhYWFhYZEaVmlYWFhYWKSGVRoWFhYWFqlhlYaFhYWFRWpYpWFhUWcQERPRWxvY/mK/j3mN6sPCwgSrNCwsJBDRxf6ArP77R4ZmdgPw50bJaGHRSuRbLYCFRRviZgDvUfaNp72YmdfWVxwLi/aBtTQsLOIYY+a1yr/NQOB6OoOIrvGX01xJRKfKF6vuKSL6in/eGBGtJaJLpGOdRPQDIlrnL0X6DyI6Rmnv1f7ynaNEdCe8tUagnHMUEd3uy7SaiH7qr4VhYVFXWKVhYZEdXwPwf/DWMbgQwCVEtFh3IhGdDK867McB7AtvKdt7pFPOA/AOAB+EtybCQwCuJ6Ld/Ov3gFcL7Sa/vx/518h9HAxvOdb/g1e19CT/3F/WdpsWFnHY2lMWFhKI6GIAp8Irnijjx8z8BSJiAP/NzKdJ19wMYC0zn+p/ZgBvY+Y/EtFn4a3LfBCH5d7Fdb0AtsBbgvQSf18OXon+y5n5LCL6JrzFdvZjUXGO6Cx4FU33YuZnfculyMwfkto+FMD9AGYx8/q6PBwLC1hOw8JChzvgrbUsY6u0/Xfl2N8BvM7Q1h8AfBrAM0R0A4DrAfwfM4/BW+egAG8xIADewk1E9HcAB/i79odX2lqe3an9Hw5gHyJ6h7RPrMS2NwCrNCzqBqs0LCziGGHmp+rREDM/T0T7ATgeXsnq7wH4KhEtQTiw68x9sS/NMpwOgP+GfoW91dkktrBIhuU0LCyy40jN58dMJzPzKDNfw8z/BuAIAAfCW+DrKXhRWQHx7bunXgLgUX/XowCWKGs4q/3fB2+hsKc0/7bDwqKOsJaGhUUcnUS0q7KvzMwb/O2TiOif8Ba4eSs8K2KJriEiej+839nd8BaWegeAIoAnmXmYiH4K4NtEtBHAMwD+Dd6Stz/xm/gZgM8B+AER/QTAwfBWX5NxLoB/ENHPAPwc3kJDCwG8gZk/kv32LSzMsErDwiKOEwCsUfatBjDH314KbznNCwBsAPABZv6noa2tAL4Ab6nUAjzL4SRmfsY//gX/76/gLUl6P4BXM/MaAGDm54joJADnwyPU7wVwJoDfiA6Y+UEiOg7AOfBWDcwBeBrAVdlu28KiMmz0lIVFBsiRUa2WxcKiFbCchoWFhYVFalilYWFhYWGRGtY9ZWFhYWGRGtbSsLCwsLBIDas0LCwsLCxSwyoNCwsLC4vUsErDwsLCwiI1rNKwsLCwsEiN/wcSr9d7zWy/1QAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.plot(rewards)\n",
        "plt.xlabel(\"Episode\")\n",
        "plt.ylabel(\"Sum of rewards\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "K2CK44D5MM9l"
      },
      "outputs": [],
      "source": [
        "env.seed(42)\n",
        "state = env.reset()\n",
        "\n",
        "frames = []\n",
        "\n",
        "for step in range(200):\n",
        "    action = epsilon_greedy_policy(state)\n",
        "    state, reward, done, info = env.step(action)\n",
        "    if done:\n",
        "        break\n",
        "    img = env.render(mode=\"rgb_array\")\n",
        "    frames.append(img)\n",
        "    \n",
        "plot_animation(frames)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F_0zgUNNMM9l"
      },
      "source": [
        "매우 안정적인 에이전트같습니다!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1PSFnlTiMM9l"
      },
      "outputs": [],
      "source": [
        "env.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vag2RJSiMM9m"
      },
      "source": [
        "# TF-Agents를 사용해 브레이크아웃 게임하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OzHU4KP2MM9m"
      },
      "source": [
        "TF-Agents를 사용해 브레이크아웃 플레이를 학습하는 에이전트를 만들어 보죠. 심층 Q-러닝 알고리즘을 사용하겠습니다. 따라서 이전 구현과 구성 요소를 쉽게 비교할 수 있습니다. 하지만 TF-Agents에는 다른 (그리고 복잡한) 알고리즘을 많이 구현되어 있습니다!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ejGbHM0MM9m"
      },
      "source": [
        "## TF-Agents 환경"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UfK1JXGwMM9m"
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(42)\n",
        "np.random.seed(42)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HdxvM9JeMM9o",
        "outputId": "2e35aa44-c936-4f58-9460-fe566dd7c48a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tf_agents.environments.wrappers.TimeLimit at 0x7fe46bf261d0>"
            ]
          },
          "execution_count": 79,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from tf_agents.environments import suite_gym\n",
        "\n",
        "env = suite_gym.load(\"Breakout-v4\")\n",
        "env"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HSy8OORDMM9p",
        "outputId": "7cc47c0c-0dec-4c74-b7a8-600e9ccbf197"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<gym.envs.atari.atari_env.AtariEnv at 0x7fe46bdbba50>"
            ]
          },
          "execution_count": 80,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.gym"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "N7m8k3asMM9p",
        "outputId": "ce7a862c-efe8-43ea-a068-107a8fc5034c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TimeStep(step_type=array(0, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       ...,\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]]], dtype=uint8))"
            ]
          },
          "execution_count": 81,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.seed(42)\n",
        "env.reset()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DJQ1sDUFMM9p",
        "outputId": "f6f7f9f9-be2f-4f66-9017-2163dd10c3e3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TimeStep(step_type=array(1, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       ...,\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]]], dtype=uint8))"
            ]
          },
          "execution_count": 82,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.step(1) # Fire"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z47Lx1BJMM9q",
        "outputId": "dcd62dc0-78ce-4aa3-95b1-c3e863cb681f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Saving figure breakout_plot\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAIpCAYAAAD3tqwgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAASjElEQVR4nO3dW29c533v8f+aGZ5EKhEpKrLlpLHRGE2yN2IlKbCDHApHCFDkRfZl9K5RbV0IAWoDzcbeiHNAvRNBkWmLlCNKlKg5PPvCjRK3NknHwzU/Sp8P4JvRwjyPxuL6ch3mWV1rrQAgzWDREwCATyJQAEQSKAAiCRQAkQQKgEgCBUCk0VF/2HWde9ABOFWtte6TXncEBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARDpyLX46E/XdXXp0qXa3t6urvvEZalOzWQyqZ2dnfrwww+P3XZ7e7suXbpUg0G/v9vMZrPa2dmpvb29Y7fd3Nysy5cv13A47GFmf9Zaqw8++KDu3r1brZ39ZSw3NjbqxRdfrOXl5d7HvnfvXu3s7NR0Ou197EUZDod1+fLl2tzcPHbb3d3dev/992s2m/Uws8URqBDD4bC++c1v1ve+973ed6wPHjyo69evHxuowWBQX/va1+pHP/pR7zutx48f140bN+revXvH7vxffvnl+vGPf1xra2s9ze4j4/G4bt68WXt7e8/EjvXy5cv1k5/8pLa2tnodt7VWv/jFL+r69ev16NGjXsdepOXl5Xrttdfq29/+9pG/pM5ms3rrrbfqzTffrCdPnvQ4w/4JVIiu6+rcuXO1tbVVo1G//1uWl5drZWXlRNuura3V1tbWibefl0ePHtXq6uqJtl1dXa3Nzc1aX18/5Vl93Hg8rnPnzvU65mlaWlqqCxcu1MWLF3sdt7VWGxsbvR+lL1rXdbW+vl4XL148NlDr6+u9n2lZhOfrXwAAZ4ZAARDJKb4zprVWDx8+rL29vZpMJkduu7KyUhcvXjzxqbF5Ojg4qN3d3RqPx0dut7y8XBcvXuz9elHVR6cN9/b26vDw8MjtRqNRXbx4sfdThmfFeDyu3d3dOjg4mNt73r1795m4jsfnI1Bn0K1bt+rNN9+shw8fHrndCy+8UNeuXasXX3yxp5n92Z07d+qNN9449saLra2tunbtWn31q1/tZ2J/YXd3t65fv14ffPDBkdudP3++Xn/99Xr11Vefi/P+n9X+/n7dvHmz3n333bm958OHD5/5GwA4nkCdQQcHB3Xnzp3a398/crvBYLCwH/LHjx/Xe++9V7u7u0duNx6P6/Hjxz3N6uMODw9rZ2en7ty5c+R2Fy5ceK7uJvusJpNJ3b17t27fvr3oqfCMcQ0KgEgCBUAkp/iAz2U0Gj1dvWNeHj16VPv7+8/Eihz89QQK+Fw2Njbq+9//fl29enUu79daq3feead+/vOfL+z6JBkECvhclpeX66WXXprb+7XW6t69e70v+UUe16AAiCRQAERyig/4RJPJpO7fvz+3xYu7rqvV1dVaWVnxhWdORKCAT7Szs1M/+9nP5rZy/Wg0qqtXr9Y3vvENgeJEBAr4RPv7+/XOO+/M7f1WVlbqpZdeqq9//etze0+eba5BARBJoACI5BTfGTQajWptbe3Yx22srq4u7Kmkw+GwVldXj32Mxurq6sK+7zIYDE40x7W1tefyOzmDwaCWl5fndr1oZWWllpaWXH/ixATqDLpy5Updu3bt2GctbWxs1IULF/qZ1H/xpS99qV5//fVjVwJYW1ur7e3tnmb1cVtbW/XDH/7w2OcYrays1AsvvPDc7VgvXbpUr732Wp0/f34u7zccDusrX/nKc/c58tcTqDOm67ra3t6u7e3tY9cpW+SOYHNzszY3N6Pn+IUvfKGuXr0aPcdF2tzcrO985ztz+wXief0c+esJ1Bn0px/05B94c3w2dF23sNPE4F8eAJEcQQWZzWY1mUx6f8TAZxnzT3Ps+7fqyWRSs9nsRNv+aY7HXaObt/F4XNPptNcxT1NrbSGfY1U9U5/jZzGbzWo8Hh95VN9aO/HPwlknUCGm02n99re/XcjO//Dw8NjHnld99IPxu9/9rt54443e72obj8d169atE4X09u3bdePGjVpaWuphZn82nU7r3XfffWZ2Hru7u3Xz5s1aX1/vddzWWv3hD3+oJ0+e9Druoo3H4/rlL39Z+/v7xwbq97///bF38T4LuqN+4Luu87SwHg0Gg4VdD5nNZifa+S/ymsSzNMezYJGf4/N0lPCXTroPeNY+n9baJ/6lBQqAhfq0QLlJAoBIR16DcvstAItyZKB+8IMf9DUPAPiYIwP105/+tK95AMDHHBmo4xbRBIDT4iYJACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEijvgZqrdXBwUE9fPiwWmt9DQvAnHRdV+vr63Xu3Lnquu7Ux+stULPZrH7961/X22+/XdPptK9hAZiT0WhU3/3ud+tb3/rWsxWo1lrt7u7Wb37zmxqPx30NC8CcLC8v1yuvvNLbWTDXoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEijvgbqqmp1OKzN5eUad11fwwIwJ8vLy7U6HPY2Xm+BGnRd/f3WVv3PV1+tNp32NSwAc9KNRrW6tVWDng4yeg3U354/X1++cqUGAgVw5sxGo7q1sVF3qqr1MF5vgfpLnVN8ABzDTRIARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkRayFl/rZZlBAM6yfgO1Mqs6P66aWc0c4Kxpw1a1POttvP4C1VW1tUm1i4fVmkABnDWtm1StTT56Am0P+j2C6v6zwM0pPoAzp2vVerxzwU0SAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIPS4W26p1raaDWbXW33LtAMzHrBtU6/p7ol9vgWpVNR5N69HquLo26WtYAOZk1s1qPOrvcUm9Pm5jNmg1Hk5rUI6gAM6aWc1q1vX3uCTXoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkXpdLLZ1VdVV9bdYOwDz8qd9eF/6C1RXNVmb1aO1aXXV33LtAMxHq0FNejzA6PUIarLa6vG5aXUetwFw5rSa1uRgVvW4n/F6DVR1f/EfAGdLq173326SACCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABE6m2x2FZVD2pU99tqlcdtAJw9bVTDGvW25nePgerqvbZat9tmzQQK4MwZtGF9ua3WC/WMBaqq6rAGdb9GNXNmEeDMGdawDnvcfysFAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiBSr4vF1mS56vF6tW7W67AAfH6tDaomS72N11+g2qBm779c071XazrrY6F2AOZq0Gq2Na7anPTyvI0eA9VV+/ByTf/fl2s2c2YR4KzphrNqw1tVF+5Ude3Ux+v3FN9TjqAAOJpDGQAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIC1mLr7XTX2QQgDnred/dW6Bam9WD/d/We3/43zWZ9DUqAPOyNKq68qVzVbVafSz63WugPtx7u/7jNzdq/EShAM6a5ZWleuVv/qFa+15VDU99vB5P8bWaTh/X+Mkfazwe9zcsAHPRdcs1mx32Np6bJACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRRoueAKQbVNX6aFQrw+HT11pr9XA6rcfT6eImBs84gYJjnF9aqn+8cqX+xxe/+PS1R9Np/cudO/XW3t4CZwbPNoGCY6wOh3V1c7N+fPny09f2J5P61f379fbeXrUFzg2eZQIFn0HXdYueAjw33CQBQCSBAiCSQAEQSaAAiCRQAEQSKAAiuc0cjtGqatpajVurah9962k8m9Ws+QYUnCaBgmMcTCZ1Y2enbj18+PS1w9msfr2/70u6cIoECo7xYDKpf93ZqRvvv//0tVZVE0dQcKoECk5g0pogQc/cJAFAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQKTRUX+4P5jObaDpcFaHXas2t3cE5m3QdXVxebk2lpYWPZVPNWut7j15UvfH40VP5fnTWi0dHta5Bw9qMBye+nBHBur/rD2Z20Cz6aR2liYCBcHWh8P6xytX6n9tb1e36Ml8ikfTaf3z7dv15s6O/UnPBq3V1gcf1Mu/+lUNB6d/Au7IQO0uze8IatZN62DQqnX+SUGq0WBQr2xs1N9vbVXXZSbqwXhc/7a7W12VQPWsa61WDw7qi3t7NeohUK5BARBJoACIJFAARBIoACIJFACRBAqASAIFQKQjvwcFPF9aa/VoOq3743Hs96AeTib1ZDZb9DTogUABTz2aTuuN996r/9jfX/RUPtWktfq/H37oS7rPAYECnjqczeqtvb16e29v0VM5kjg9HwQK+G8EgARukgAgkiMoAE6kVdW0tXoym9W0nf5xtkABcCKT2aze3turx9NpDeZ4l+c/fMrrAgXAiUxaq7f39urf53wTzT99yutHP1H31ntzm0CbTuvJH/erfH0B4MyatdbbbrxrR5xH3Py7l+d2krG1Vof37tfj3T9W9XDuEoCzobX2iecLjwxU13n8LQCn69MC5TZzACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkbrW2qLnAAD/jSMoACIJFACRBAqASAIFQCSBAiCSQAEQ6f8DiyfmfwULrTYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x576 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "img = env.render(mode=\"rgb_array\")\n",
        "\n",
        "plt.figure(figsize=(6, 8))\n",
        "plt.imshow(img)\n",
        "plt.axis(\"off\")\n",
        "save_fig(\"breakout_plot\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jtx1u0ITMM9s",
        "outputId": "4f30bddd-6911-469d-ed0e-0901f27e051e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TimeStep(step_type=array(1, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       ...,\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]],\n",
              "\n",
              "       [[0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        ...,\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0],\n",
              "        [0, 0, 0]]], dtype=uint8))"
            ]
          },
          "execution_count": 84,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.current_time_step()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "12x24B2PMM9s"
      },
      "source": [
        "## 환경 스펙"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kk38Rk_rMM9t",
        "outputId": "00928a64-3623-445d-f4c2-17f0fa8b63a6"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "BoundedArraySpec(shape=(210, 160, 3), dtype=dtype('uint8'), name='observation', minimum=0, maximum=255)"
            ]
          },
          "execution_count": 85,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.observation_spec()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i9WO1wqXMM9t",
        "outputId": "6dc6bf2e-1b89-40bc-b052-0d883bf32aee"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "BoundedArraySpec(shape=(), dtype=dtype('int64'), name='action', minimum=0, maximum=3)"
            ]
          },
          "execution_count": 86,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.action_spec()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6okHHwEFMM9t",
        "outputId": "a4c2bc5a-72ad-40c2-f947-1b0383b559a2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TimeStep(step_type=ArraySpec(shape=(), dtype=dtype('int32'), name='step_type'), reward=ArraySpec(shape=(), dtype=dtype('float32'), name='reward'), discount=BoundedArraySpec(shape=(), dtype=dtype('float32'), name='discount', minimum=0.0, maximum=1.0), observation=BoundedArraySpec(shape=(210, 160, 3), dtype=dtype('uint8'), name='observation', minimum=0, maximum=255))"
            ]
          },
          "execution_count": 87,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.time_step_spec()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xDpYjirOMM9t"
      },
      "source": [
        "## 환경 래퍼"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmdUZvGDMM9t"
      },
      "source": [
        "TF-Agents 래퍼로 TF-Agents 환경을 감쌀 수 있습니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "THk6YyUqMM9t",
        "outputId": "7df6c899-1b04-4c04-812b-24cc7c1575da"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tf_agents.environments.wrappers.ActionRepeat at 0x7fe46872cad0>"
            ]
          },
          "execution_count": 88,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from tf_agents.environments.wrappers import ActionRepeat\n",
        "\n",
        "repeating_env = ActionRepeat(env, times=4)\n",
        "repeating_env"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B64H12U2MM9u",
        "outputId": "141b8e0e-5ad6-4b0d-e2d1-893727d9652f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<gym.envs.atari.atari_env.AtariEnv at 0x7fe46bdbba50>"
            ]
          },
          "execution_count": 89,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "repeating_env.unwrapped"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Xkqpth6MM9u"
      },
      "source": [
        "가능한 래퍼 목록은 다음과 같습니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pQ3yQ-mdMM9u",
        "outputId": "aae08213-dd80-4c78-8202-95a8fa04337a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "ActionClipWrapper           Wraps an environment and clips actions to spec before applying.\n",
            "ActionDiscretizeWrapper     Wraps an environment with continuous actions and discretizes them.\n",
            "ActionOffsetWrapper         Offsets actions to be zero-based.\n",
            "ActionRepeat                Repeates actions over n-steps while acummulating the received reward.\n",
            "FlattenObservationsWrapper  Wraps an environment and flattens nested multi-dimensional observations.\n",
            "GoalReplayEnvWrapper        Adds a goal to the observation, used for HER (Hindsight Experience Replay).\n",
            "HistoryWrapper              Adds observation and action history to the environment's observations.\n",
            "ObservationFilterWrapper    Filters observations based on an array of indexes.\n",
            "OneHotActionWrapper         Converts discrete action to one_hot format.\n",
            "PerformanceProfiler         End episodes after specified number of steps.\n",
            "PyEnvironmentBaseWrapper    PyEnvironment wrapper forwards calls to the given environment.\n",
            "RunStats                    Wrapper that accumulates run statistics as the environment iterates.\n",
            "TimeLimit                   End episodes after specified number of steps.\n"
          ]
        }
      ],
      "source": [
        "import tf_agents.environments.wrappers\n",
        "\n",
        "for name in dir(tf_agents.environments.wrappers):\n",
        "    obj = getattr(tf_agents.environments.wrappers, name)\n",
        "    if hasattr(obj, \"__base__\") and issubclass(obj, tf_agents.environments.wrappers.PyEnvironmentBaseWrapper):\n",
        "        print(\"{:27s} {}\".format(name, obj.__doc__.split(\"\\n\")[0]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "614L79fnMM9u"
      },
      "source": [
        "`suite_gym.load()`는 TF-Agents 환경 래퍼와 짐 환경 래퍼로 환경을 만들고 래핑합니다(후자가 먼저 적용됩니다)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f3Yr4xBAMM9u"
      },
      "outputs": [],
      "source": [
        "from functools import partial\n",
        "from gym.wrappers import TimeLimit\n",
        "\n",
        "limited_repeating_env = suite_gym.load(\n",
        "    \"Breakout-v4\",\n",
        "    gym_env_wrappers=[partial(TimeLimit, max_episode_steps=10000)],\n",
        "    env_wrappers=[partial(ActionRepeat, times=4)],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sCEYP6X_MM9u",
        "outputId": "b0b0e284-28f5-49d7-fc0d-8fcb8f97d421"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tf_agents.environments.wrappers.ActionRepeat at 0x7fe4686ff550>"
            ]
          },
          "execution_count": 92,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "limited_repeating_env"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oGEO08eyMM9v",
        "outputId": "3e503dee-0d1f-41e3-e956-ea86aa8a328f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<gym.envs.atari.atari_env.AtariEnv at 0x7fe3de8b6c90>"
            ]
          },
          "execution_count": 93,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "limited_repeating_env.unwrapped"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wdgTTSshMM9v"
      },
      "source": [
        "아타리 브레이크아웃 환경을 만들고 기본 아타리 전처리 단계를 적용합니다:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eGxzv4XlMM9v"
      },
      "source": [
        "**경고**: 브레이크아웃은 게임 시작과 죽을 때마다 FIRE 버튼을 눌러야 합니다. 처음에는 FIRE 버튼을 누르는 것이 빨리 지는 것처럼 보이기 때문에 에이전트가 이를 배우는데 매우 오랜 시간이 걸릴 수 있습니다. 훈련 속도를 높이려면 `AtariPreprocessing` 래퍼 클래스를 상속하여 `AtariPreprocessingWithAutoFire`를 만들고 사용합니다. 이 클래스는 게임 시작과 말이 죽을 때마다 자동으로 FIRE(즉 플레이 행동 1)를 누릅니다. 일반적인 `AtariPreprocessing` 래퍼를 사용한 책의 코드와 다른 점입니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ptRx_mCTMM9v"
      },
      "outputs": [],
      "source": [
        "from tf_agents.environments import suite_atari\n",
        "from tf_agents.environments.atari_preprocessing import AtariPreprocessing\n",
        "from tf_agents.environments.atari_wrappers import FrameStack4\n",
        "\n",
        "max_episode_steps = 27000 # <=> 108k ALE frames since 1 step = 4 frames\n",
        "environment_name = \"BreakoutNoFrameskip-v4\"\n",
        "\n",
        "class AtariPreprocessingWithAutoFire(AtariPreprocessing):\n",
        "    def reset(self, **kwargs):\n",
        "        obs = super().reset(**kwargs)\n",
        "        super().step(1) # FIRE to start\n",
        "        return obs\n",
        "    def step(self, action):\n",
        "        lives_before_action = self.ale.lives()\n",
        "        obs, rewards, done, info = super().step(action)\n",
        "        if self.ale.lives() < lives_before_action and not done:\n",
        "            super().step(1) # FIRE to start after life lost\n",
        "        return obs, rewards, done, info\n",
        "\n",
        "env = suite_atari.load(\n",
        "    environment_name,\n",
        "    max_episode_steps=max_episode_steps,\n",
        "    gym_env_wrappers=[AtariPreprocessingWithAutoFire, FrameStack4])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WorYqLXtMM9v",
        "outputId": "8194b3ab-178c-428e-d371-0f52233df0b1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tf_agents.environments.atari_wrappers.AtariTimeLimit at 0x7fe46bf46510>"
            ]
          },
          "execution_count": 95,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o8Lr8XZwMM9v"
      },
      "source": [
        "몇 개의 스텝을 플레이하고 어떻게 동작하는지 확인합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XQt-IN-wMM9v"
      },
      "outputs": [],
      "source": [
        "env.seed(42)\n",
        "env.reset()\n",
        "for _ in range(4):\n",
        "    time_step = env.step(3) # 왼쪽"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zJYOoAhzMM9w"
      },
      "outputs": [],
      "source": [
        "def plot_observation(obs):\n",
        "    # 컬러 채널이 3개이기 때문에 4 프레임을 출력할 수 없습니다.\n",
        "    # 따라서 현재 프레임과 다른 프레임의 평균 값을 뺀 차이를 계산합니다.\n",
        "    # 그다음 이 차이를 현재 프레임의 빨강과 파랑 채널에 더해서 보라 색을 구합니다.\n",
        "    obs = obs.astype(np.float32)\n",
        "    img = obs[..., :3]\n",
        "    current_frame_delta = np.maximum(obs[..., 3] - obs[..., :3].mean(axis=-1), 0.)\n",
        "    img[..., 0] += current_frame_delta\n",
        "    img[..., 2] += current_frame_delta\n",
        "    img = np.clip(img / 150, 0, 1)\n",
        "    plt.imshow(img)\n",
        "    plt.axis(\"off\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Sc2BjqzsMM9w",
        "outputId": "095a4c8f-76ac-4993-b834-1e201f7ca2a5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Saving figure preprocessed_breakout_plot\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAGoCAYAAADiuSpNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADJ9JREFUeJzt3U+InOUBx/F3Jlmwi+AW/6QExMtiKggGeuplCaUiueToXQQFoaBIiUJ7KgQ8CEK9tFBE8KCeRZHmENJCT5LsQU08KVpipVFZZDAsO28vPWTmeZcdJ+87729mPp/b+/Ds+z7ZnZ1vXt533h3UdV0BQJJh3wsAgGniBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHGO93HQwWDgk78AVHVdD5rGnTkBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECIE4vH8KlPSdOnCjGPvvssyO/7sqVK0fOefTRR4uxjY2Nie2dnZ1izu7ubjH29ttvT2yfPXu2mPPFF18UY99+++3EdtO/9+TJkxPbb7zxRjHn+eefL8aeeOKJie133323mDMajYqx69evT2xPf0+qqvl7N+2ee+45ck6il156qRg7f/78xPb0z62qqurLL7+c63jT3++qqqpnn312rn0tg9dee60Ye+qppya2L1y4UMx55ZVXOltTH5w5ARBHnACII04AxBEnAOK4IWJNnTlz5sg5n376aTE2ffNBm1599dVi7M0335zYbroY//LLL3e2pqaL8dPfu3lvSlll165dK8aabjiZxc2bN+90OSwhZ04AxBEnAOKIEwBxXHMCfpIPPvigGLtx48Zc+5r+sPIzzzxTzGn6wPh777031/FYHs6cAIgjTgDEEScA4ogTAHHcELGmLl26dOSc+++/v/uF3ObFF18sxqafxtz0gdcunTp1qhib/t41PZV8lZ0+fboYm/45zWpZn8xO95w5ARBHnACII04AxBEnAOIM6rpe+EFPnz69+IMCEOfq1auDpnFnTgDEEScA4ogTAHF6uea0t7e30IOOx+OJ7eGwbPL0nMPmtXH8Wffd5Zq61Pe6+z7+Oprld2yVzfqaa+u9qK05CWva2tpyzQmA5SBOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQJzjfS/gTly/fr0YG41GPawEYHVtbm4WY6dOner0mM6cAIgjTgDEEScA4ogTAHEGdV0v/KB7e3utHHRnZ6cY293dbWPXAPzfY489Voxdvnx5Yns8HhdzhsPhkXO2trYGTcd05gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBECc430v4E489NBDxdhoNCrGxuPxxPZwWDZ5es5h8+Yx7767XFOX+l5338dfR7P8jq2yWV9zbb0XtTVn1jU9+OCDxVjX1usVBMBSECcA4ogTAHGW+prT+fPnizHXnPrX97r7Pv46cs1pta85bW5uFmNdW69XEABLQZwAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgzlJ/CPe+++4rxvb393tYCcDq2tjYWPgxnTkBEEecAIgjTgDEEScA4iz1DRFNF+m6fBLwvDyV3FPJV52nkq/2U8mPHTtWjHVtvV5BACwFcQIgjjgBEEecAIiz1DdENF24Ozg4mGnePHPmNe++l/Wict/r7vv462jdv+ez/vvbei9q8z2t7/fHQ4+58CMCwBHECYA44gRAnJW75tT0YbG6rie2B4PBkXMOmzePeffd5Zq61Pe6+z7+Oprld2yVzfqaa+u9qK05s67JNScAqMQJgEDiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHGW+gkRDzzwQDHW9IQIf6Z9sfped9/HX0f+TPtq/5n2pr/2cOvWrWKsTev1CgJgKYgTAHHECYA44gRAHHECII44ARBHnACII04AxFnqD+HevHmzGGv6sBgA82t6uMHdd9/d6TGdOQEQR5wAiCNOAMRZ6mtOe3t7xdj+/n4xVtf1xPZgMDhyzmHz5jHvvrtcU5f6Xnffx19Hs/yOrbJZX3NtvRe1NWfWNW1sbBRjrjkBsHbECYA44gRAHHECIM5S3xAxGo2KsR9//LGHlQCsrrvuumvhx3TmBEAccQIgjjgBEEecAIiz1DdEfPLJJ8VY05PKx+PxxPZwWDZ5es5h8+Yx7767XFOX+l5338dfR7P8jq2yWV9zbb0XtTVn1jXde++9xdj29nYx1qb1egUBsBTECYA44gRAHHECII44ARBHnACII04AxBEnAOIs9Ydw33rrrWLs448/Lsb8mfbF6nvdfR9/Hfkz7av9Z9ofeeSRYuzcuXPFWJucOQEQR5wAiCNOAMQRJwDiLPUNEV9//XUx9tVXX/WwEoDV1fRU8q45cwIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACIc7zvBUCy3zaM/Xpq+0+LWAisGWdOAMQRJwDiiBMAcVxzYn1tTm3/vJzym3+XY8c6WQxwO2dOAMQRJwDiiBMAccQJgDhuiGB9/WFqe6Oc8uffl2N7nSwGuJ0zJwDiiBMAccQJgDiuObGCdhrGflYO/fDh5PZH5ZQbrawH+KmcOQEQR5wAiCNOAMQRJwDiuCGCFfS7hrGGux0ufFiOARGcOQEQR5wAiCNOAMQRJwDiuCGCaINqMLH9evV6MWe32p3Y/mv1l4Y9/bPNZQEdc+YEQBxxAiCOOAEQxzUnog2n/v/0cPVwMeed6p2pkcsdrghYBGdOAMQRJwDiiBMAccQJgDhuiCDaQXUwsf149XhPKwEWyZkTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARDneB8HHY1GrexnPB63sp/DDKrBxPbZ6uxc+6mr+sh9t/l1s3i/utYw+stW9l01rLtqad2z+Xs5dGa/HNvsfiVdOHFjcvtXV/pZx+2mf+Kz/rT/M7X9UQtroX37++XvzzfffNPKvre2thrHnTkBEEecAIgjTgDEEScA4vRyQ8T333/fyn4ODg5a2c9hhlPtfqF6Ya79jKvyxo3pfbf5dbN4v/pbw+jTrey7alj3Yv8f9I9y6OmGGyJ+0f1KurB9eXL7hYAbIqZ/4rP+tP81te2GiEy3bt0qxj7//POJ7aYb1IbD4ZFztre3G4/pzAmAOOIEQBxxAiCOOAEQp5cbIuq64ckHgxmemNDwdV06qCZvuHiuem6u/STeEFFV/20Ya+tydN83RPxQDv2xYdpG5wvpxNW9ye35XpXtmveGiIafFFRV5cwJgEDiBEAccQIgzmDR13GqqqqefPLJVg568eLFYuy7775rY9cALEBd1403HDhzAiCOOAEQR5wAiCNOAMTp5YaIwWCw+IMCEMcNEQAsDXECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4gzquu57DQAwwZkTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA4/wPd8aiZkkg/HgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.figure(figsize=(6, 6))\n",
        "plot_observation(time_step.observation)\n",
        "save_fig(\"preprocessed_breakout_plot\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m5LkCvqDMM9w"
      },
      "source": [
        "파이썬 환경을 TF 환경으로 변환합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wQ-_-OGOMM9w"
      },
      "outputs": [],
      "source": [
        "from tf_agents.environments.tf_py_environment import TFPyEnvironment\n",
        "\n",
        "tf_env = TFPyEnvironment(env)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fPzGxdg0MM9w"
      },
      "source": [
        "## DQN 만들기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lat01T6IMM9w"
      },
      "source": [
        "관측을 정규화하는 작은 클래스를 만듭니다. 이미지를 0~255 사이의 바이트로 저장하는 것이 램을 적게 사용하지만 신경망에는 0.0~1.0 사이의 실수를 전달해야 합니다:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qewdBtjYMM9w"
      },
      "source": [
        "Q-네트워크를 만듭니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x0KEhXLPMM9x"
      },
      "outputs": [],
      "source": [
        "from tf_agents.networks.q_network import QNetwork\n",
        "\n",
        "preprocessing_layer = keras.layers.Lambda(\n",
        "                          lambda obs: tf.cast(obs, np.float32) / 255.)\n",
        "conv_layer_params=[(32, (8, 8), 4), (64, (4, 4), 2), (64, (3, 3), 1)]\n",
        "fc_layer_params=[512]\n",
        "\n",
        "q_net = QNetwork(\n",
        "    tf_env.observation_spec(),\n",
        "    tf_env.action_spec(),\n",
        "    preprocessing_layers=preprocessing_layer,\n",
        "    conv_layer_params=conv_layer_params,\n",
        "    fc_layer_params=fc_layer_params)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dO8912T2MM9x"
      },
      "source": [
        "DQN 에이전트를 만듭니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YZ7_mYSDMM9x"
      },
      "outputs": [],
      "source": [
        "from tf_agents.agents.dqn.dqn_agent import DqnAgent\n",
        "\n",
        "train_step = tf.Variable(0)\n",
        "update_period = 4 # run a training step every 4 collect steps\n",
        "optimizer = keras.optimizers.RMSprop(learning_rate=2.5e-4, rho=0.95, momentum=0.0,\n",
        "                                     epsilon=0.00001, centered=True)\n",
        "epsilon_fn = keras.optimizers.schedules.PolynomialDecay(\n",
        "    initial_learning_rate=1.0, # initial ε\n",
        "    decay_steps=250000 // update_period, # <=> 1,000,000 ALE frames\n",
        "    end_learning_rate=0.01) # final ε\n",
        "agent = DqnAgent(tf_env.time_step_spec(),\n",
        "                 tf_env.action_spec(),\n",
        "                 q_network=q_net,\n",
        "                 optimizer=optimizer,\n",
        "                 target_update_period=2000, # <=> 32,000 ALE frames\n",
        "                 td_errors_loss_fn=keras.losses.Huber(reduction=\"none\"),\n",
        "                 gamma=0.99, # discount factor\n",
        "                 train_step_counter=train_step,\n",
        "                 epsilon_greedy=lambda: epsilon_fn(train_step))\n",
        "agent.initialize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7FDZnZydMM9x"
      },
      "source": [
        "재생 버퍼를 만듭니다(램을 많이 사용하기 때문에 메모리 부족 에러가 나오면 버퍼 크기를 줄이세요):"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BIR_XItoMM9x"
      },
      "source": [
        "**경고**: (책과 달리) 1,000,000이 아니고 100,000 크기의 재생 버퍼를 사용합니다. 대부분의 경우 메모리 부족 에러가 나기 때문입니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uTnd-iELMM9x"
      },
      "outputs": [],
      "source": [
        "from tf_agents.replay_buffers import tf_uniform_replay_buffer\n",
        "\n",
        "replay_buffer = tf_uniform_replay_buffer.TFUniformReplayBuffer(\n",
        "    data_spec=agent.collect_data_spec,\n",
        "    batch_size=tf_env.batch_size,\n",
        "    max_length=100000) # OOM 에러가 나면 줄이세요\n",
        "\n",
        "replay_buffer_observer = replay_buffer.add_batch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kH1cPdVPMM9x"
      },
      "source": [
        "호출 횟수를 카운트하고 출력하는 간단한 사용자 정의 옵저버를 만듭니다(하나의 스텝으로 카운트하지 않는 두 에피소드 사이의 경계는 제외합니다):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GtlRfjVVMM9x"
      },
      "outputs": [],
      "source": [
        "class ShowProgress:\n",
        "    def __init__(self, total):\n",
        "        self.counter = 0\n",
        "        self.total = total\n",
        "    def __call__(self, trajectory):\n",
        "        if not trajectory.is_boundary():\n",
        "            self.counter += 1\n",
        "        if self.counter % 100 == 0:\n",
        "            print(\"\\r{}/{}\".format(self.counter, self.total), end=\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eJ0eNLtOMM9y"
      },
      "source": [
        "훈련 측정 지표를 추가해 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u3G1rCtKMM9y"
      },
      "outputs": [],
      "source": [
        "from tf_agents.metrics import tf_metrics\n",
        "\n",
        "train_metrics = [\n",
        "    tf_metrics.NumberOfEpisodes(),\n",
        "    tf_metrics.EnvironmentSteps(),\n",
        "    tf_metrics.AverageReturnMetric(),\n",
        "    tf_metrics.AverageEpisodeLengthMetric(),\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h-wWJZddMM9y",
        "outputId": "6ac4c76c-58b5-4c71-d482-efc86541594f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(), dtype=int64, numpy=0>"
            ]
          },
          "execution_count": 105,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "train_metrics[0].result()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8SXxCde5MM9y",
        "outputId": "0daa0786-045f-4072-a237-a236d82a83a9"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 0\n",
            "\t\t EnvironmentSteps = 0\n",
            "\t\t AverageReturn = 0.0\n",
            "\t\t AverageEpisodeLength = 0.0\n"
          ]
        }
      ],
      "source": [
        "from tf_agents.eval.metric_utils import log_metrics\n",
        "import logging\n",
        "logging.getLogger().setLevel(logging.INFO)\n",
        "log_metrics(train_metrics)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YlXRcaauMM9y"
      },
      "source": [
        "수집 드라이버를 만듭니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Wj-UwBSDMM9z"
      },
      "outputs": [],
      "source": [
        "from tf_agents.drivers.dynamic_step_driver import DynamicStepDriver\n",
        "\n",
        "collect_driver = DynamicStepDriver(\n",
        "    tf_env,\n",
        "    agent.collect_policy,\n",
        "    observers=[replay_buffer_observer] + train_metrics,\n",
        "    num_steps=update_period) # collect 4 steps for each training iteration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jA7YyShHMM9z"
      },
      "source": [
        "훈련 전에 초기 경험을 수집합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CyBfjSfeMM9z",
        "outputId": "6fe75dec-4296-4db4-d419-1c3d0304d8a3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "20000/20000"
          ]
        }
      ],
      "source": [
        "from tf_agents.policies.random_tf_policy import RandomTFPolicy\n",
        "\n",
        "initial_collect_policy = RandomTFPolicy(tf_env.time_step_spec(),\n",
        "                                        tf_env.action_spec())\n",
        "init_driver = DynamicStepDriver(\n",
        "    tf_env,\n",
        "    initial_collect_policy,\n",
        "    observers=[replay_buffer.add_batch, ShowProgress(20000)],\n",
        "    num_steps=20000) # <=> 80,000 ALE frames\n",
        "final_time_step, final_policy_state = init_driver.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kVsKhQqQMM9z"
      },
      "source": [
        "3개의 스텝을 가진 2개의 서브 에피소드를 샘플링해서 출력해 보죠:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FXwAvr6CMM9z"
      },
      "source": [
        "**노트**: `replay_buffer.get_next()`는 deprecated 되었습니다. 대신 `replay_buffer.as_dataset(..., single_deterministic_pass=False)`를 사용해야 합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C08ZmQKvMM9z"
      },
      "outputs": [],
      "source": [
        "tf.random.set_seed(9) # 에피소드 끝에서 경로 샘플을 보여주기 위해\n",
        "\n",
        "#trajectories, buffer_info = replay_buffer.get_next( # get_next() is deprecated\n",
        "#    sample_batch_size=2, num_steps=3)\n",
        "\n",
        "trajectories, buffer_info = next(iter(replay_buffer.as_dataset(\n",
        "    sample_batch_size=2,\n",
        "    num_steps=3,\n",
        "    single_deterministic_pass=False)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cnYnHDN8MM90",
        "outputId": "f02f60a9-aa0a-40a3-abe7-31431c2d44cf"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "('step_type',\n",
              " 'observation',\n",
              " 'action',\n",
              " 'policy_info',\n",
              " 'next_step_type',\n",
              " 'reward',\n",
              " 'discount')"
            ]
          },
          "execution_count": 110,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trajectories._fields"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aychI9z6MM90",
        "outputId": "baddbb8b-9b6f-48af-fdc1-cc338d6f9306"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TensorShape([2, 3, 84, 84, 4])"
            ]
          },
          "execution_count": 111,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trajectories.observation.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "16Er6LrOMM90",
        "outputId": "8f4d3800-63ba-49a4-fc83-eb956b5af3a2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TensorShape([2, 2, 84, 84, 4])"
            ]
          },
          "execution_count": 112,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from tf_agents.trajectories.trajectory import to_transition\n",
        "\n",
        "time_steps, action_steps, next_time_steps = to_transition(trajectories)\n",
        "time_steps.observation.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mn5k4vzBMM90",
        "outputId": "b3d7f23a-b7e6-4c08-89f4-a081142945d7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[1, 1, 1],\n",
              "       [1, 1, 1]], dtype=int32)"
            ]
          },
          "execution_count": 113,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trajectories.step_type.numpy()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ixiR9LUgMM91",
        "outputId": "07a7695c-3a1e-4850-81e2-9fcb070a8b8f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Saving figure sub_episodes_plot\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAHbCAYAAADWAWzeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABGMUlEQVR4nO3dW3Mb94Hn/W+jcSRAUiRF0QfJji0/WY+ddRzP7sU8NamaC9fu1lzM9da8hXlj8wIy11tPTbm2JpmKk3V2HNmOEh1sWaJI8YBzo7ufiybApkSCIAUSoPT9sGjJEgg12fihf939738HaZoiSZIkKVOY9QJIkiRJ88SCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUk5x3F8GQeAccNKMpWkaTPpYMyvN3qSZNa/S7J2UV48gS5IkSTkWZEmSJCln7BCLV0EQBIRhSK1Wo1g8+u2maUqSJC88vlAovPC4brdLFEVHvqZUKlEulykUCqPP4eOHdygc/hoEAUEQjH4PkCQJg8GANE2JoogoigBGzxWGIaVS6YXlOU6hUBg979BgMKDdbhPH8ek/qAtSLBYJw5BCoTD6WSVJQhzHpGlKv98ffd/zIr+sYRiOXjdRFBHHMUmSEEXRC68dTYeZNbNnZWZnx7ya17O6Knl9pQtypVKhXC5z48YNPv/8c27fvj36uzRN2d/fp9lskr/ddqVSYXFxkWKxOAry/v4+X3zxBXfu3KHX67G7u8tgMOD27dv84he/oF6vs76+zsrKCoPBgE6nw2AwYDAYjF6Y5XKZcrlMGIZUKhWKxSJbW1vcv3+fVqvFnTt3+PbbbwnDkNXVVRYWFtjY2ODjjz+m0Wic+D0O3xTq9TqNRuNI0P/yl7/wq1/9ivv371/AT/d0pVKJW7dusb6+zvXr1/nkk0+4du0aOzs7bG5u0mw2+fLLL/nmm2+Yp1ueLy8v88knn3D9+nXW19e5efMmQRBw9+5dHjx4wN7eHt988w3b29uzXtRXjpk1s+dhZmfDvJrX87gqeX1lC3IQBJRKJWq1Gm+++SZ///d/z9/+7d+O/j5JEp48ecLm5uaRvZRGo8HGxgaVSmW0d/PkyRNarRZbW1vs7+/TarWI45hbt27xd3/3d6ytrfHBBx9w8+ZNoihiZ2eHXq9Hr9ej0+kAUK/XWVhYoFQqsbi4SLlc5s9//jO//e1v2draot1u86c//YkwDLl27RorKyv81V/9Ff/tv/031tfXT/weh3u1a2trrK+vE4bh6O9//etf82//9m8zDe+bb77J7du3uX37Nv/wD//ArVu3ePjwId9++y1Pnz5lc3OTb7/9dq7Cu7i4yCeffMJ7773HBx98wGeffUYQBHzxxRd8+eWX/Pjjjzx69Gjm4X3VmFkze15m9vKZV/N6Xlclr69sQYbDPb/hKaD86Z/BYMDOzg537949cnpkY2ODlZWVI+Ednr7In8IZPv/wVE2xWKRYLNLv99nb26PZbLK/vz9awevr66yurlKr1Wg0GqPHD79++LxpmtLr9Wi322xvb3Pv3j329vaOnGKqVquj00LDZWs0Gi98j2EYvnBK6LINf0bDZRueDhp+znr5jpN/zeTXf/51oIthZmefCTOrSZnX2efBvF6cV7ogjzMYDPjNb37DP//zP9Pr9UZ//tlnn7G8vMytW7eOHVN1mr29Pb788ku+//57Hjx4wB//+EfSNOXTTz/lww8/ZG1tjVqtRr1eP/broyjiyZMnbG9vs7m5yXfffTc6dVSpVKhWq7z77rtcv36dMAwpl8sUi0XSNOXtt98+8/JKV4WZla4O86qr7rVd00mS8OjRI37/+9+PTtEALC0t0Ww2GQwG5xp43+/3efLkCQ8ePODbb7/l97//PWma0mg0WF5eHg2aH7dc7XYbyN4IHj16BECtVqNarY5CH0URxWJxNAZsf39/rk6hSNNmZqWrw7zqqnOaN0mSJCnHgixJkiTlvLZDLCC7AnRhYeHIgPBqtToa2H6egeLDK3vL5TLVapWFhQXSNB0N+h9ObTPOcJD68MKEIAioVqvUajVqtRrlcnn0XMNf81fWSq8qMytdHeZVV9lrW5CLxSI///nP+cd//Mcjk2h/8MEHbGxsUKvVKJVKZ37eRqPBxx9/zMbGBrdv3+ajjz4iTVPef/99bt26xdLSEouLi2OXa2VlhYWFBW7cuMF/+k//6cgVuaVSibW1tSNX1IZhyBtvvGGA9Uozs9LVYV511b3WBfnjjz9mbW3tyByNS0tLrK+vU6lUzjVFSr1e56c//Snvvvsu3W6XVqsFZPP+1et1SqXSiVfXDpdrdXWVlZUVPvzwQ/77f//vozka83cJGn4O94SHV9xKryozK10d5lVX3diCXKlULms5pq5QKIxOjQwnLP/LX/4y+vskSdje3mZ7e/tIeLvdLnEcH7m95dbW1iiEw1skQjaNzbNnz0jTlHq9Prqzz/7+Pv1+n36/T7fbBaDdbo+mtNnd3aVUKvH999+zvb09umvQ8O4/wzv19Pt9dnd3T7wNZj68vV6Pbrd75LGPHz8mTdOZrcdyuUySJHS7Xfb29vj++++J45gff/yRzc1Nnj17RhzHVCqVmd9SMi8MQ9rtNjs7Ozx58oT79+8TBAGbm5vs7+/T7XZHd2uaN/O4TJMys2b2vK5qZudtec7CvJrX87oqeQ3GTVvy/vvvX9k5TfKTUNfrdW7fvs3q6uro74dTweTnZwRGKyU/aXmv1+P+/fs8ffqUOI7pdrskScL169d58803R3us1Wp1dM/3JElG90OHw/ulB0EwCmi73WZ3d5d+v8/W1hZbW1sEQTA69dRoNLhx48bozeKk7xOysV6VSuXI3vjOzg537txhZ2dnij/ZyYVhyPLyMgsLC9TrdTY2NqhWq6O9/n6/z6NHj9ja2pqr6XNqtRpvvPEGCwsLLCwssLy8DMCzZ8/Y3d2l1+uN7sx0Ge7evTvxIRYza2Zfhpmdjkkza17N68swr9NxUl7HFuRPP/10fn6i0mvqd7/73cQF2cxKszdpZs2rNHsn5XXsEItf/vKXF7M0ki6EmZWuDvMqza+xR5D/4z/+w71bacY++uijiY8gm1lp9ibNrHmVZu+kvI49gnzz5s2LWRpJF8LMSleHeZXml3fSkyRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUs7YW01PKkkSkiQBIE29tbx0kiDIbvleKBQoFGa3f2pmpcnMQ2bNqzSZaeZ1KgW51+uxv79PHMckSUIcx9N4WumVEoYhhUKBMAxZXFykVqvNbFnMrHS6ecmseZVON+28TqUgR1FEs9kkiiKiKDK80jGKxSLFYpFSqTTTcgxmVprEvGTWvEqnm3Zep1KQ9/f3uXv3Lu12m263S7fb9TSQlBMEAdVqlWq1Sr1ep1qtsri4OLPlMbPSePOUWfMqjXcReX3pgpymKffu3eNf/uVf2NzcZGtri2fPnhleKScIAlZXV1ldXeXGjRvU63XeeuutmSyLmZVONy+ZNa/S6S4ir1M5gtxqtXj48CGPHj3i8ePHbG5uGl4pJwgC1tfX2djYoN/v02q1Zro8ZlYab54ya16l8S4ir07zJkmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVJOcRpPUigUKBaLo89SqUSaptN4aumVEAQBpVJplJFCYbb7pmZWGm+eMmtepfEuIq9TKcjlcpmlpSW63S6DwQDA8Eo5QRCwtrbGysoKy8vLlMvlmS6PmZXGm6fMmldpvIvI61QKchiG1Go1FhYWqNfr9Ho9wyvlBEFAvV5nYWGBarVKGIYzXR4zK403T5k1r9J4F5HXly7IQRDQaDR49913WVpaYm1tjb29vZdeMOlVEgQBS0tLLC8vs7q6SqPRmOmymFlpvHnJrHmVTncReZ3KEeSNjQ3+5m/+hk6nQ6/Xo9frTeNppVdKpVKhUqlQq9W4cePGTJfFzEqnm5fMmlfpdNPO61QKcqVSYWVlhXq9ThRFozFSkg4NL64pl8tUKpWZLouZlU43L5k1r9Lppp1Xp3mTJEmScizIkiRJUs5UhlgEQUChUCAMQ5IkIU1Tr7CVcoIgIAzD0WcQBDNfHjMrnWyeMmtepfEuIq9TKcjFYpGFhQXiOGYwGDg+SjpGfqL/YnEq0XupZTGz0njzklnzKp1u2nmd2jzIlUqFOI4pFovEcTyNp5VeKcM921nflWu4LGZWGm9eMmtepdNNO69THWIxPO3jqR/pRYVCgUKhQBAEczPEwsxKJ5uXzJpX6XTTzutUC3L+/w2wdCg/hnAY4HlYnvz/m1np0Dxl1rxK411EXqc2qGrY2POfkg7NWzbMrDTePGXDvErjTTsbU7/qwOBKx5vXXJhZ6XjzmAvzKh1v2rlwHmRJkiQp50LmrXHvVrpazKx0dZhX6eJdyBhkSS+at4zM2/JI82aeMjJPyyLNo2lnZOpDLAyvdLxhNuZtIzdPyyLNk3nM7LwshzRvpp1XxyBLkiRJORZkSZIkKWcqBdmxUdJk5iUjZlaazDxkxLxKk5lmRqZykd7wPvGAt8GUTpDfyIVhONNlMbPS6eYls+ZVOt208zqVglwsFqlUKu7hSqcYbtziOCZJkpkth5mVJjMPmTWv0mSmmdepFOThPbA9DSSdbBjcNE1nWo7BzEqTmJfMmlfpdNPOqxfpSZIkSTkWZEmSJClnKkMshuM98pM0SzpqeGHNPFxkY2al081LZs2rdLpp53UqBXkwGDAYDGa+0ZeugiAIKBaLFAqzO4FjZqXJzTqz5lWa3LTyOpWCnCQJURQdGSAt6aj8BTaFQmGmBdnMSqebl8yaV+l0087r1ArycEqNJEkMsPSc50M761kszKw03jxl1rxK411EXqdWkPv9PnEcj0JseKVDw+CGYUgYhpTL5Zkuj5mVxpunzJpXabyLyOtUL9Ibfg4GgxMf+/zFBV5sMB/yb7anvfG6Dg89/7Ma97MrFrO4BUEw842bmb36zOz5XMXMmterz7yezyzzOpWC3O/32dvbI4oi+v3+iRcTDAdOF4vF0a0AZzkOU4cGgwH9fn80wfZJpydch0cNT32maTr2QpogCCiVSpRKJcrlMtVqlVqtNoMlzpjZq8/Mns9VzKx5vfrM6/nMMq9TKchRFLG/v0+326XX641eBM8LgoBKpUK1WqVQKFAul0eNX7PV6/VoNptHjlIcx3V41PBNL0mS0ev/pNf+MLSVSoVr165d/sLmmNmrz8yez1XMrHm9+szr+cwyr1P5qXe7XZ4+fUq73abdbtPpdI59XBAENBoNFhcXKRaLLCwsUKlUprEIeglpmtJsNnn69ClRFBFF0Ymn8FyHh9I0pd/v0263GQwG7O/v02w2Tzy1U6vVWFhYoF6vs7GxcclLe5SZvdrM7Plc1cya16vNvJ7PrPP60gU5TVN++OEH/tf/+l88ffqU7e1tdnZ2jj19EIYh77zzDu+88w61Wo2NjQ2Wl5dfdhE0Bffu3eP//J//Q6vVotVq0W63j30Rug6P2t3d5fHjx3Q6He7du8eDBw+OPTJQKBRYWVlhZWWF9fV1VlZWePvtt2ewxGb2VWFmz+eqZda8vhrM6/nMMq9TOYK8u7vLd999x6NHj3j8+DGbm5snrvhWq0WapjQaDcIwnPnFSsr88MMPfP311+zu7rK3t8f+/v6xj3MdHvX06VMePHhAq9Xi66+/5s6dO8duuIIgYH19nY2NDXZ3d9nd3Z3B0h4ys1efmT2fq5hZ83r1mdfzmWVeX9+R35IkSdIxLMiSJElSzqVfGpmf4uS0+Rx1eYbTqExyKsd1eOj5u1u9ilzf88nMns+rnlnX9Xwyr+czy7xeakFO05Td3V0ePHhApVJhd3eXRqNxmYugEzx69Ijd3V3a7TZRFJ34ONfhUc1mk62tLbrdLru7u6/cBtf1Pb/M7Pm8ypl1Xc8v83o+s8zrpRbkJEnY2dmh1+sRhiEPHjygVCpd5iLoBO12m93dXQaDwYnzM4Lr8HlRFNHr9Yjj+MSrkq8y1/f8MrPn8ypn1nU9v8zr+cwyr5c+xCKOY6IoOnJnFM3e8AU4yYvPdXhoMBgQRdHYOyNdda7v+WRmz+dVz6zrej6Z1/OZZV4vvSAPv9EgCCgUCq/1PcbnSZIko1s4nhZg1+Gh4VixNE3HHhW4ylzf88nMns+rnlnX9Xwyr+czy7xeekF+VffaXyeuw9eL6/vqcx2+PlzXV5/rcD44zZskSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJOcVZL4CmIyRbmSkwAJLZLo4kSdKVZUG+wgpkKzAEbgHvAzHwR+DBDJdLkiTpKrMgX2EBUAZKwLvA/wtEwA7wkOxosiRJks7GgnyFlYAloAo0gApZaXZguSRJ0vlZkK+wZeA/AyvATbKS3MKVKkmS9DLsUldYBXgD2CAryWWgR3YUWZIkSedjQb4qCrnPKlCCOIJmCypxVoz3gDawj+OPpVk7JrJEZGd54hkulyTpdBbkq6JIdoi4DKwDS9Dbg80H0GtnG97ewefjGS6mpMwxkWWPbIaZ9gyXS5J0OgvyVVEgOwRVBurAYnYEuVvIVmKHbKPbJyvJkmbrmMgS4UW0knQVWJCvggBYBd4CFsgmPV6D7gP44SFUmtmGt0926rY7uyWVxImR5QHZFIzN2S2aJGkCFuSrYLi1/YDsMNR7wDp0ivDDvx99aIrjj6VZOyGyFIF/H/N1kqT5YEG+CgKyc7U1sqkrErLDxREk3lNamjtjIutt4CXpCrAgXxULwHWy+0o/BX4gO1freAppLhlZSbq6LMhXwfCe0g2y8RObwBbZVncww+WSdCwjK0lXmwV57oRk17xXGJ2kTQuwvQ3fbQMpPCGbL2qX7JytpNkxstKVcUxcKQDbB59ew6MhC/LcKQPvkF3zvgzcgrgI3/07PPo1EGVb2AFOWSHNAyMrXRnHxHV08exBXCXAgjyHQrLRi8tk18HfAErQakDLm0hLc8fISlfGCXGlQTY0ShqyIM+dErBBNjFUSHYLkCbZcSdP/khzx8hKV4Zx1aS8qdPcKQNvAz8lmzm1TTYyqo3xleaQkZWuDOOqSXkEeYYCAkJCAAoUCAhIKTMgJSEiu2l06+CzP8MllZQJ4CCz2fGFANIUBgOIEiMrzRPjqpdgQZ6hKlWWWaZIkRo1qlTpUeURj9mnR3bN+w9kJ4Ge4C0GpFmrko1eLJJd/16Fbg/uPoLBvpGV5olx1UuwIM9QmTJLLFGhwuLBRxvYYYd9toB94Eey0VE9jK80a2VgiWySqMXsM2rDjzsQ7RtZaZ4YV70EC/IM1aixwQYLLFChQpkyEBGyT3bZQJvsxM9wgihJs1Uju8RngWyrW862qu0wezc1stL8MK56CRbkGbrOdf4r/5VllunQoUuXbXao8i1wnyyyEdmlA+7bSrN3HfivZOdtO0AX+tvwYzW7TZ6RleaHcdVLsCDPUJUqa6yxyiq77LLHHm3ahPTJjiBLmi9VslsMrJKNYNyDtA3dcPyXSbp8xlUvwYI8QwkJERFduvzIj9znPrvs0rQcSzNXpDiaZSY4uIVATMiAASldstGL98m2vGZWmq0ih1NWHNzyI4khGkA3Na46MwvyDMXE9OjRps097vFbfkuXLrvsznrRpNdaQECJEhUqBAcfAH2KxPRJaQP3gN+SXeJjZqXZCchuAVI5+P1BQY770IuhnRpXnZkFeYYGDGjRAqBJkxYt+vSJvVxAmqmAgDJl6tRHf5YCKQUCOmQjFpsczqBqZqXZCcimrKjn/iyFQXp4v3fjqjOyIM/Qj/zIv/KvFCnyhCe0aBEffEianSJF3uM9bnOblJQ2bQYMeMyAb/nfxAzIZk5tkW1tzaw0O0Wym0ffJtuVbQMD+PEx/Ou3UIyNq87MgjxDW2yxzTYAqTe5lOZCQECRIm/yJj/jZ8TE7LBDly4Jd/kz/5dsBlUzK81eQFZl3gR+RtZ+d4AubCWw/efsYcZVZ2RBnjGLsTRfhmOOq1Rp0KBHj4iIFi169EiJcWsrzYvhmOMq0CC75UdEdri4l91bWjqHwqwXQJLmxbAcFymyyirv8i7Xuc4uu9zjHptsMmAw68WUBByW4yLZXG7vkk1+vEt2Vd4mmFedkwVZknICAgoUqFChQYMqVXr02GWXNm0SbykgzYlhQS6QzWDRIDuS3CMryW28BYjOyyEWkpSTkhIRcY97/IbfsMMOP/CDBVmaAxUqrLJKhSoxCQMSEsrs85Q2/w48A37AgqyXZUGWpAMpKQkJPXr8gT/wmMf06PE939OkSULiLDPSDC2wwAd8wBprdA/uI9Aj4QEP6PDng5v4fE82r1uCU1bovCzIkvSchIQWLUJC+vTp0iUimvViSa+14fUBDRoss0xxdEYnIqQJ7JENr+iCedVLsiBLUs7wKHKLFgMGxMRERM44I83I8MY9JUqsscZP+Sm3uMX3fM8dviGgRzacYpvsorwIZ5rRy7IgS9JzUlI6Bx+SZisgoEKFGjVWWOE93uMDPiAm5i53ycpwh2zcsWOONR3OYiFJkuba8MxOTEyHDk2atGiNfo3o41FjTZNHkCVJ0txKSRkwICJijz3+xJ9o0+ZrvuYbvqFJkw4dh0FpqizIkiRprqWkxMT06LHNNgUKbLLJFlu0ac968fQKsiBLkqS5NSzHAQEtWnzP9+yx550tdaEsyJIkaa4NDj4iIlq0KFAgOviQLoIFWZIkzbXh+OJhUZYumrNYSJIkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJElXWkAw1ecrjvvLwWAw0ZMkSUKapqNPaRIBsABUyfbUwoM/CwkpUaRAgSJFwtHfTCYup/QXU5IiJEGJOCiRf1XGFOgRElMgPfj/lJQBA2ImeM0PBrC3B73e5N/sgTRNSZKEKIrodrsTfc3S0tLEz29mdVlCQurUKVIkoEKW5gIpIekZj72EhZhKqU+hkJBUEpJqDMFpr8sU6AA9IAEGB7++6CUie6GZNa+6PCFQB4pQDqEWQhhkfzy2Cb6oAJQOfq2QbcOPbqFTsjymBHSBDkEKlU6ZUq9MkEBhUIDk+O16TECHEoODbfQkAgLWWGMtXeNaco1CVHjpvI79sURRdOoTp2nKYDAYBViaVAisAtfJXojV0a8lGjQoUmSBBWrUzrRf2Kmn7L6TEC1AL6zTCxtHNthdimxTo0fIgJA+BWJSWrSI6Zz+DzSb8Kc/Qb8PZ3jNDze0cRzT7XZpNpsEwXT3eM2sLkuZMje4QZ06AasUeBMoM6BMTJmz7NTWih1WG7uUShHRWo/oeg8Kp702B8AT4BkQAU1OKsjnjOyFZ9a86vKUgRtAHRYqsFGDciHrzC823LGKQIOsJK+RbcOP7hIPgC4wIGCLgB8JBymrT5ZYfLZIISpQbBYpJMfvSHcp8iMNWpSAYKKSHBCwlC6xnCyzEq8QdsOXzuvYgpwkx7/Z5A3fQKSzCsj2PhtkL8QFssAtUGCZIkVK1KmwQI2zpLdTTCnWY6JF6IQNOuEyaXAYxBIlOgd70gVCUkIKJBQokb2JTKB4EJ0gOHNJTtOUOI4nPnp0FmZWlyUkpEr1oCAvUWANqDCgymDSHB2oFzosl4qUyxFRrUN/qQPhabmKgH2yYpxy2nvEOSN7oZk1r7o8IVkTrkOxCrU6VMNsA1znTAV5eAS5DNSApYNnPxSRbdUHBDQJCChGsLZfYrlZpZCGlIISwQlnmjqU6LFIkQqQHYc+TUBAgwb1tE4trUE8+Rmak4wtyL0JzkelaUoURSRJ4h6uziQE3gA+JIvttdGvZd7gGmWqVFinxOqZxhZ1lxJ2PxwQXYe98gY75TdJckHcoca3rLJPhQ5FmpSJiOnxmDbP4LT91ceP4eFD2NrKtrQTvuaHR6KiKKLVarG7uzvx9zQpM6vLUqXKLW6xwRuEvEvIz4EFeoT0znjOdq3c5f2VXRYWIrof7NH5cIekeFop7JFtphOgdfDZP/aR54zshWfWvOryVIFbwAY0FuEn16ERwjrZqdwzFOQysEJ2UOsDsm340cT3gGcE9A6GTf5AqZfyVmmNteQtCq0ihVaNoB8+/9QAtKhyjzfYpU56cAT5tFd9QEA5KVOJK1SiCsVW8aXzOtWC7PgonUUIbAA/JduB3SAL3AZlfsIyFRYIeevg1O3kYxp7izHN/yciuglb1Xd5UnufJDh8qT+mTp832WKBJmXKVOkSscM94MfT/4E//xn+9//ODkWdQf4o1HBjO+0hFmZWl6VChbd5m/d4j5CPKfFLUhbpwCQDlY54s9Tj02tNFpciWu9tsf/pE9LKaQW5BewCO2TvJpsnPvKckb3wzJpXXZ4K8DbwHiyuwjtvwUo568xn28RSIjugtZQ9G58ePPuhFvAYaFPgGSElyq2En+yusLFzkyAsweZwkMaLmtS5wU94xjVSApJJ23sKQRzAAAqtAnu7e2cq/s8bW5DPEkRDq7MaXmKzx+EJmQ4QkFChT5mQkDYF9k88FXOcXhTT2hsQbcOz6g5blS3i4HBP9RkdmhToUKVDmR4V+gyIeQJsceq+6s5ONpjxJVzUhs7M6rLExLRps8ceIVsUeUTKHl0CumfcKtWSHpu9Fp1uRHv/Ga3tLZJSfMpXdcg2xD2yd5CTX89TiOyFZNa86vLEQBvYg6gAzSqExWwI/9kuGSAhS12XbJDTNs9X3Q7ZjmuHkA4FEsqdlGqrR9xrEUQlSFOCEwpyiwGbPGOH+ODo8WTjkENCCgcfC+kC5bT8UjNbnPHaRWl6IuBbsk1ckdH1tdRps8yPhBQpsUuRe5wlvdGThPa/JsQN6BS/pV1cPnKRXpsSj2nQpUif8GBGi4QWewdLc4q9veycbZKcbTDjGaVpOvUjzNK0NGnyFV9xn/sE3KHA78gu0gsZnHEWi3on4v8+alPejok6HfoP2qQTjUH+nuzIce/g83iXFFkzqznWBL4C7sPTKnzZgGoBFsnGIZ9BB3hEVow7wAOOG4PcBiICHhHQJYxg5fuHLG62CHoFgl6JgOOHWPQp8ZQ/0KYycTkOCKhRo37w8TN+xk1ujv2a0/JqQdbMxMBTsoDlp40p0afCHgUKlGhTPmEv8yRRE7p3U+IyDIIyUaFCvmBHFGhTIqZATHAwlUxKnz5ZsE/R62WXxV/wER03tJpnPXo84hHbbAOPCQ42kwlFUooTT88EUI4iftjrUghjku6AeDuaYJ84Idvot8neTU4+4nxJkTWzmmM9RrW2FcL3ZSgG2dHjs21iicjO/IZkR5G3eT6uycGjhhkdUEhgoblDtd0liAPCuHDi0d2IkH3K9A+meJ14FouDj2tc4yf85PSvOSWvYwvynTt3JlgsePjwIa1Wi263SxzHngrSRFKyS2oKB5/D34ekdIkJSCgCxROmbjpJHKf0O5BEEAcxcTA4MvhwQECPIsnB2KbsM5sJedxGdiSKsolVYeItbpqm9Pt9Wq0WxWKRr7/+eqIpngA+//zziR4HZlaXJyGhT59kNEowGc2BnJ5wZOgkcRoTD/oUkoS0F5ME8YQFucfh/Mcnv4bPEdlLyax51eVJyLayCcQh9CIYZON1z3irgdEsx8MEBi98eUK2LR0W5ZQggag3oDPoQwKF9OSCnM2DPCAanYmaYOFSSPoJ/VafTrHDV19/xU60M9H3c1Jeg3FB+6d/+qeJUnj//n2++uorms0m7XabTuesl2jodTUsx3AYgSxsw/+efQRRGkB6EPjs9MzzzxAcbE4P/zwb5zTJtbJkW9g4PvPhqEKhQBiGhGHI4uIilUploiNO9+/fn/hHYGZ1mULCUVazJGe/P2t9C0gpBOnBl6ekp86BPDQsxuOze87IXnhmzasu18GGMQAKwdGN7hkFuc/jB1QNX9oJHFxmFyQBQVrI7gE05pU/3OU+6/tIWAgphAUKYYGFxQXKlfJL5XXsEeTHjx9PtFA7Ozv0er3RZObSpBJOmtp/wrJ60pdOf4rhlzacpuksd+Q6KzOryxRPcsZlUvnIT/FpX8ZFZ9a86nIdBCvlpTN2vrhO4R8eI0qi0UHrZrf50s83tiB/8803Ez3J/v4+rVaLKIqI4zl5Z5NeQ2ZWujrMqzS/xg6xKJVKEx3Cc35G6eKkaTrxCTAzK83epJk1r9LsnZTXsQU5CALTKM3YWQqymZVmb9LMmldp9k7K69kmq5QkSZJecRZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5QRp6q3gJUmSpCGPIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTnFcX8ZBIG32ZNmLE3TYNLHmllp9ibNrHmVZu+kvHoEWZIkScqxIEuSJEk5FmRJkiQpZ+wY5FdBEASEYUitVqNYPPrtpmlKkiQvPL5QKLzwuG63SxRFR76mVCpRLpcpFAqjz+Hj0zQd/X74vEEQjH4PkCQJg8GANE2JoogoigAIw5BCoUCxWKRSqYwen3/u4fMOFQqFI48DGAwGtNtt4jg+40/t7AqFwmi5wzAkDMMXlicv/7PI/zyGP4tut/vC93hZCoXCaL2GYTh63URRRBzHJElCFEUvvHY0HWbWzJ6VmZ0d82pez+qq5PWVLsiVSoVyucyNGzf4/PPPuX379ujv0jRlf3+fZrN55EVSqVRYXFykWCyOgry/v88XX3zBnTt36PV67O7uMhgMuH37Nr/4xS+o1+usr6+zsrLCYDCg0+kwGAwYDAajQJbLZcrlMmEYUqlUKBaLbG1tcf/+fVqtFnfu3OHbb7+lWCyysbFBo9HgnXfe4b/8l//C8vIycRyPPlutFr1eDzh88dfrdRqNxpE3nr/85S/86le/4v79+xf+s15cXOSdd96hXq/zxhtv8Pbbb7/wZjk0DEShUGBhYYFqtUq/32d3d5der8cf//hHvvjiC1qt1oUv93GWl5f55JNPuH79Ouvr69y8eZMgCLh79y4PHjxgb2+Pb775hu3t7Zks36vMzJrZ8zCzs2Fezet5XJW8vrIFOQgCSqUStVqNN998k7//+7/nb//2b0d/nyQJT548YXNz88heSqPRYGNjg0qlMtq7efLkCa1Wi62tLfb392m1WsRxzK1bt/i7v/s71tbW+OCDD7h58yZRFLGzs0Ov16PX69HpdACo1+ssLCxQKpVYXFykXC7z5z//md/+9rdsbW3Rbrf505/+RKlUYn19nfX1df76r/+a//k//ydvvfXWaO83iqLRcgzfXIIgYG1tjfX1dcIwHH0vv/71r/m3f/u3SwlvvV7nvffeY21tjb/6q7/ik08+oVwuH/vY4RtZsVhkZWWFpaUl2u02P/zwA81mk4WFBX7729/OLLyLi4t88sknvPfee3zwwQd89tlnBEHAF198wZdffsmPP/7Io0ePZh7eV42ZNbPnZWYvn3k1r+d1VfL6yhZkONzzG54Cyu9tDQYDdnZ2uHv37pHTIxsbG6ysrBwJ7/C0Rv5UxfD5h6d9isUixWKRfr/P3t4ezWaT/f390QpeX19ndXWVWq1Go9EYPX749c+fKsk/bxiGo1MPvV6PR48e8fjx4yN/H4YhGxsbR77H007BTFMURezt7REEAT/88AONRmMU0OHyDU+pVKvV0RvZtWvXRutm+NjnT79dtvxrJr/+868DXQwza2bPw8zOhnk1r+dxVfL6ShfkcQaDAb/5zW/453/+59GpFIDPPvuM5eVlbt26deyYqtPs7e3x5Zdf8v333/PgwQP++Mc/kqYpn376KR9++CFra2vUajXq9fqZnrfT6bC9vc2TJ0/41a9+xW9+8xtKpRILCwtUKhX+x//4H/zkJz+hVCqd6XmnZXd3lz/84Q+USiX+8Ic/0Gg0KJVKLC8vj/bs33jjDWq1GktLS6yurrKwsECtVmN1dXUmy6yrxcxOl5nVRTKv02VeL99rW5CTJOHRo0f8/ve/H52iAVhaWqLZbDIYDM418L7f7/PkyRMePHjAt99+y+9//3vSNKXRaLC8vEyapvT7/TM953BQfafTYW9vj++++44vv/yScrnM4uIitVqNTz/99FIuFDhJv9/n6dOnR/6sWCxy/fp1lpaWWF5eZjAYsLi4OHqz7Pf7R944pXHM7HSZWV0k8zpd5vXyOc2bJEmSlGNBliRJknJe2yEWwGh8UX5AeLVaHQ28P89A8eGVveVyeTRQPk1TqtUqpVJpNLXNeZ53OKi9UqmwsLBAuVymVqtRq9UolUozHdg+XLbhRRXDiwKGyzecDmj4M8hfQCFNysxOj5nVRTOv02NeL99rW5CLxSI///nP+cd//MfRPIoAH3zwARsbG6NAnFWj0eDjjz9mY2OD27dv89FHH5GmKe+//z63bt1iaWmJxcXFMz1nEASjgfZhGPL555/z7rvvjoJcKpX4+c9/PrOLByCbtuXmzZvU63Vu3LjBm2++eeQCh3K5zPLyMuVymYWFBZaWlqhWq2f+Wej1ZWany8zqIpnX6TKvl++1Lsgff/wxa2trR+ZoXFpaYn19nUqlcq4pXOr1Oj/96U9599136Xa7o3kGFxcXqdfrlEqlM19dGwTBaEqcWq3GL3/5S37+858fmQLn+vXrJ86JeBkajQa3b99mbW2NDz/8cDRH4/NT9gRBQLlcHk3k3mg0ZrbMulrM7HSZWV0k8zpd5vXyjS3IlUrlspZj6gqFwuhUw3DC8r/85S+jv0+ShO3tbba3t4+Et9vtEsfxkdtbbm1tjUI4vEUiZNPYPHv2jDRNqdfrozv77O/v0+/36ff7dLtdANrt9mhKm93dXUqlEt9//z3b29ujuwYN9wIhm/Nwf3+fhw8f0u/3R889/DdbrdaRScyjKKLf7x85nfL48WPSNL2U9VgsFonjmH6/T6vVYnt7+8S97eHpsTAMabVaNBoNut0uT548od1u02w2R7cAnYUwDGm32+zs7PDkyRPu379PEARsbm6yv79Pt9sdHVmYN/O4TJMys2b2vK5qZudtec7CvJrX87oqeQ3G3Yv7/fffn82NuqcgPwl1vV7n9u3bR+YCHE4F8/wUKMOVkp+0vNfrcf/+fZ4+fUocx3S7XZIk4fr166PTHPV6nWq1OpouJkkSkiQZTQsznKA7CILRuKB2u83u7i79fp+trS22trYoFArU63UqlQpLS0u8/fbbVKvV0T3Uh/coHwwGo+8TskA8f0/5nZ0d7ty5w87OzgX/tLNxZdeuXaNUKtFoNFhaWjpx7FN+r3z4Bjv8ucZxzObmJvfv3z/zVD3TUqvVeOONN1hYWGBhYYHl5WUAnj17NrpV5/DOTJfh7t27Ex9iMbNmdlJm9uJMmlnzal4nZV4vzkl5HVuQP/300ysbXulV8bvf/W7igmxmpdmbNLPmVZq9k/I6dojFL3/5y4tZGkkXwsxKV4d5lebX2CPI//Ef/+HerTRjH3300cRHkM2sNHuTZta8SrN3Ul7HHkG+efPmxSyNpAthZqWrw7xK88sZpCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpQz9lbTk0qShCRJAEhTby0vnSQIslu+FwoFCoXZ7Z+aWWky85BZ8ypNZpp5nUpB7vV67O/vE8cxSZIQx/E0nlZ6pYRhSKFQIAxDFhcXqdVqM1sWMyudbl4ya16l0007r1MpyFEU0Ww2iaKIKIoMr3SMYrFIsVikVCrNtByDmZUmMS+ZNa/S6aad16kU5P39fe7evUu73abb7dLtdj0NJOUEQUC1WqVarVKv16lWqywuLs5secysNN48Zda8SuNdRF5fuiCnacq9e/f4l3/5FzY3N9na2uLZs2eGV8oJgoDV1VVWV1e5ceMG9Xqdt956aybLYmal081LZs2rdLqLyOtUjiC3Wi0ePnzIo0ePePz4MZubm4ZXygmCgPX1dTY2Nuj3+7RarZkuj5mVxpunzJpXabyLyKvTvEmSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUU5zGkxQKBYrF4uizVCqRpuk0nlp6JQRBQKlUGmWkUJjtvqmZlcabp8yaV2m8i8jrVApyuVxmaWmJbrfLYDAAMLxSThAErK2tsbKywvLyMuVyeabLY2al8eYps+ZVGu8i8jqVghyGIbVajYWFBer1Or1ez/BKOUEQUK/XWVhYoFqtEobhTJfHzErjzVNmzas03kXk9aULchAENBoN3n33XZaWllhbW2Nvb++lF0x6lQRBwNLSEsvLy6yurtJoNGa6LGZWGm9eMmtepdNdRF6ncgR5Y2ODv/mbv6HT6dDr9ej1etN4WumVUqlUqFQq1Go1bty4MdNlMbPS6eYls+ZVOt208zqVglypVFhZWaFerxNF0WiMlKRDw4tryuUylUplpstiZqXTzUtmzat0umnn1WneJEmSpBwLsiRJkpQzlSEWQRBQKBQIw5AkSUjT1CtspZwgCAjDcPQZBMHMl8fMSiebp8yaV2m8i8jrVApysVhkYWGBOI4ZDAaOj5KOkZ/ov1icSvRealnMrDTevGTWvEqnm3ZepzYPcqVSIY5jisUicRxP42mlV8pwz3bWd+UaLouZlcabl8yaV+l0087rVIdYDE/7eOpHelGhUKBQKBAEwdwMsTCz0snmJbPmVTrdtPM61YKc/38DLB3KjyEcBngelif//2ZWOjRPmTWv0ngXkdepDaoaNvb8p6RD85YNMyuNN0/ZMK/SeNPOxtSvOjC40vHmNRdmVjrePObCvErHm3YunAdZkiRJyrmQeWvcu5WuFjMrXR3mVbp4FzIGWdKL5i0j87Y80ryZp4zM07JI82jaGZn6EAvDKx1vmI1528jN07JI82QeMzsvyyHNm2nn1THIkiRJUo4FWZIkScqZSkF2bJQ0mXnJiJmVJjMPGTGv0mSmmZGpXKQ3vE884G0wpRPkN3JhGM50WcysdLp5yax5lU437bxOpSAXi0UqlYp7uNIphhu3OI5JkmRmy2FmpcnMQ2bNqzSZaeZ1KgV5eA9sTwNJJxsGN03TmZZjMLPSJOYls+ZVOt208+pFepIkSVKOBVmSJEnKmcoQi+F4j/wkzZKOGl5YMw8X2ZhZ6XTzklnzKp1u2nmdSkEeDAYMBoOZb/SlqyAIAorFIoXC7E7gmFlpcrPOrHmVJjetvE6lICdJQhRFRwZISzoqf4FNoVCYaUE2s9Lp5iWz5lU63bTzOrWCPJxSI0kSAyw95/nQznoWCzMrjTdPmTWv0ngXkdepFeR+v08cx6MQG17p0DC4YRgShiHlcnmmy2NmpfHmKbPmVRrvIvI6tYv0hnu2wz1dSS8ahnjWzKw0mXnIrHmVJjPNvE51DHIcx6OLCSQdVSwWKRaLM7/pAJhZaRLzklnzKp1u2nmdWkEeDAZEUWR4pRPkxw/OQ0E2s9J485JZ8yqdbtp5neoY5H6/PwqwpKOKxSKlUmk0p+ksmVnpdPOSWfMqnW7aeZ1KQe52u2xubtLtdul2u/R6vWk8rfRKqVQqVKtVarUaS0tLLC8vz2xZzKx0unnJrHmVTjftvE6lIEdRRKvVot1u0+l06Ha7XmEr5QRBQLVaJYqi0enSWTKz0njzlFnzKo13EXmd/eX0kiRJ0hyxIEuSJEk5U5sHeTiBufeMl14UBMEoI3EczzwfZlYab54ya16l8S4ir1MpyHt7e3z33Xfs7Oywt7fH/v6+4ZVygiBgcXGR5eVlrl27xq1bt2a6PGZWGm+eMmtepfEuIq8vXZDTNGVnZ4dvvvmGJ0+e8PTpU7a3tw2vlBMEAWtra6ytrbGxscFnn302s2Uxs9Lp5iWz5lU63UXkdWrzIPd6PXq9Ht1ul06nY3ilnCAIRtMz9fv9ubhRiJmVTjZPmTWv0ngXkdepTvO2v7/P3t4eu7u7My8A0jwpFAqUSiUqlQqLi4tEUTTT5TGz0njzlFnzKo13EXmdSkGO4/jInm273XbvVsoJgmA0f2m32535nfTMrDTePGXWvErjXUReneZNkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyimO+8vBYDDRkyRJQpqmpGlKIU0pAWXgOlAHusAeMDj4feelFvnVExCwcPAREFCgQEAw0VcCpATEVBhQGf3ZJAZBTKfUYVCIshVW47ldphRIDj77QPPg9y8KgcrBr8P1f+KSxGQvhoTsxdA7fVkTstfO4OCzf7B0V02apiRJQhRFdLvdib5maWlp4uc/T2bT9Cr+JF9OQECNGmXKFChQokThTMcLAgZUiA9e6elzX5sC3YOPF16pwcHnAtDgLJElYECJJgUiyiTUSM4b2TM5R2RfGReZ2Vc9r2XKlCgRElKhQnF85RgjIKZMQpGUAgkhJwUnJqFPn5iYhISYGII0y9vCwZcVjvvyzsFnyuGr/aXimj1NE4g4zGXOBcX1tTbNvI59tUZRNNHCDAYDkiSBg3JcA5aBvwZuApvAn4DWwe+P2WS81gICVlnlTd4kJBy9oZwmHW2YC3S4RpvrpBN83VA37PBj7UcGpRZcA94ga7cjCdmmMAaeHfz++E1jGVg5+HWJ7OlOrBtdoE32pvHjiU95RA94evBlbbL3nfj0L5srw+DGcUy326XZbBIEZ3q7PdVZM3uVNrbTVKDAtYOPEiUWWaR09MU/VkqBDit0WSElJCE8UpJTEp6yRZ9NkvwmLyDbiwzJjiDc4pR34aNCWtS4T4k9rjHgDSJK+XfTySN7JueI7CvhojP7Kud1uBO6zDJlyqyySp36uZ4rJaTLEhF1EooMqJKcEJyIPs/YoUuXARFdeqRBAqvAmxwexTmyqUzI2skTsld5B+i/bFyz0nOfwyOEEVxCXF9b087r2HWdJKfvy6RpSpImo92sIMheTMOitEa2Z1Qle20M9/uuTswvXkBAmTKLLFKkSInSRHvaWUHONsxF1ghYP9iznkwatCkUe1AqZitokeww8EjM4XHbLuNG5BTI1nmV7KzBEhy/JCnZm1NA9k4wYSfpkr3XDN9jplsrL8/wCFAcxxMfPTqLSTObJsnoyMhV/Vm+jICAEiUWWBhlr3L0xT/WMHMF1o4tyAnpQXHt88IxoeEPvUIWlMl7OQHBwTtEgSqFl4nsxM4Z2VfGRWZ24m3sBI+bR0WKlClTpcriwcd5JISUWKXHIglFImonbut69CkREtEhoU9AJyvIZbJtXJHsRXxkE5uQbWGKZK/4w3fFl4hr9oUlODiO9YILiOtrb5p5HdvCer3T92XSNCUiIqknpIsJ5b2URgmWEliP4Y00W/HnPbHyOihQ4A3e4Gf8jAoVatQmOpqVbZiLJITssM5T3iQ+w0/6Wdjk6eKf2a/vwrvAz8nOI43EwD7ZZvEu2a7w8QNkKsA62RvITeAnjFnnO2SHg1sHT/+EU/eY9sne354A2wdPMf16ebGGe7ZRFNFqtdjd3Z36vzFpZoki6knCYpKwd3DmJyFb46/DzmtIyDrrvM/71KnzJm+ycPTFP1ZKyBZv8Yy3SCgQUyDJbVSz07zf85iHJPlzHQWyDWYJeAv4Bdle5cTL/YRFtqnTPohsn4X8Gps8smeyw5kj+0q46MxOvI2NIpIkuVJHkQMCGjR4gzdo0OCn/JQb3DjXcyWU2OUtWlxnQIEuJQYn1MkWfRKeUaBLmzZddqEwyM6Q/oxsY1XjuaYbkwWxyeEgot7LxvVwg9U++P/nxgZeUFxfW9PO6/QK8kJCWk8pV1NKJViK4XqSFeQdTjiaKCAryBts8DEfU6M28dGshJCYCgkhj9ngIW+fqSD/GO7zdeM6XHuWtdpPyMbGjERk6c5ON2UV9XjDgrwKfAD855MenZK9aTwEdoE/T7as22RvXZAV46v4esrv1Q7DO+0hFmcpyAtJQj1NqR4U5JisJF+Nze/LCQm5znV+wk9YZpnb3Gb56It/rJiQR/yEH3mHmEJuxGJmQMIT7hOwwZHBQAHZu26Z7HTvp3CWs84h92jwB67xhJskfEJw3shO7JyRfSVcdGbPWpCv0hjkYUHeYIMVVviIj3iXd8/1XAPKPOV9dniLiIAW2Uv9ODtEPGOHPj1i9gh4CoUINoCPycrxC6deBmQt9j7Z1mXv4Ht4qbjCPeAPZAFKeOF03QXE9bU27byObVMTB7EM6UoKCcTPIKlAN4KdATxJsrE1bQ5PJVyNeF+elJQuXXYOxk316VOeICop4cGFCyHPgH0KZyrIrbRJHG1Cbyc7NLTNc+86A7Ldm+EAh5NP8w1PFbXJ3lq2GHMq6tnBg/bJ3hUmsHOwBMM3kqv+GrqoDd2kz1kGVtKUhGx1VMhW/fNF71U1zFyTJgEB22wTnbjJfVF21qbGHkWSUUE+fCMekNDjMdlx1+dGyw+vzOmQBeUMh4xSdojo0SOmRcI26XkjeybniOwr5yIye5bnuyrFOG/AgA4dypTZYefcY5BjyjxjgT2y96k2AYMTBoe1iOizy4AeMU1gG9Ioy8QO2a8vtNHhsdw+z78LvkRcs39vOMj4mKMPFxRXMZ28TmfkwwbwN5DuQbsI7SZ0OvD/xfD7Qda7/szhiYurF/OLFRPzLd8SEY3GbE1ykR4HV/OmBLRZYp9rL1xNP04r6rG3tQnNdrZidnjuTSM/d8QTDs8THfNcZOu4CjwGvmHMeKo2h+eVHjPRC6IL/EC2kR539OBVkqbp1I8wDx1Elj2yN4HhicXhbAWvuoiIv/AXWrQoU+YP/OHMY5BbrNDiGikBCQFpboOdkPKIZ8Rsc2SzN7wqZwB8ffDrGQY1RuyxxX2aNOkxYIf0vJE9k3NE9rV0kZm9ahISnvKUhIQKFTbZZInJZ+TJSwlpc40+i8QERBSP7JDm9RiwSYs2EX36JLQhTuBbsg3H8JDwCxfpfQ88IivJ3dGfvkRcszfY+2RvsMccHbyguGpCp+V1OgV5EXgX0jZEf4J24+BFFWavwzZXc8zoZUlI2GSTPn0KuY/TDTfKARE1+tTPVJAHcUSvuQedfrbCmjz3ppGf7mZ4Dfvx+mR710Wy07A/Mubir+jgC4Y77ZMs68Hi9TmcMedVd5Eb2oPI0iabYaZx8OdXcejKeSQkbLNNhw4hIVWqE+6UZlICIhYYsDCaTSY98opPadIipckLl60PIzW8YP4MV+bE9GmyQ4c+PRKaz89bM3lkz+QckX0tWY4PpaS0aBETExKyx96ZdkKPPlfAgCoxFVIKxJRO3NbFxHRoEzEgISZhAEmaTVLR5/CCuRfmR2ySvboPDxO8ZFyzf2+Hw2t1nyvIFxRXTei0vI4tyHfu3JnoH3nYe0ir3aLX6RJtxsSdlLQH3SR7qb0Kp8QvWkREm/ZoDuRJCnL2M81W8IA+A3qcZU6CQRqTJAeTpvXJDs2+sLUdXrY1fi0mHBbX4SmpY5dk+JTDd4UJ52qLOdyTv4oXkqVpSr/fp9VqUSwW+frrryea4gng888/n/jfmTSzvYcPabdadLpdNuOYTprS4/XY8YBs4z1gQI8eBQqjjfjkXw8xEQO65HdU888f0yM97oTsMCDDzJ1hi5sSZRt8EvqkLxPZM/yb54rslXcZmZ14G/vwIa1Wi263SxzHV2a4RUxMRERMTIECg3MeJsvm+u+TUDoye9NxEhIG9A7Kce4Vm43NOJwD+YWC3OW4sRAvEdej49aOWWUXENfX1kXkNRgXtH/6p3+aaH3dH9znq95XNKMmnbttOvc60IfCPgT9wxeBTjY8ajzZDULyDm8Wcpajxxx8VRwcJPeEaWiGj5tkLRZyv576XQyf8gxXhQ0fOvy8agqFAmEYEoYhi4uLVCqViY443b9/f+IXxaSZHdy/T++rr4iaTe6229zrdOjzeo0xHeZt+HFWaS40x/3QU5KjcyC/uADnOGSfDejg4AYhLxnZM/7LZ47slXfRmZ14G3v/Pl999RXNZpN2u02nczXmOgiO+TjvMx3uiHLw+5MNc5fmtxbDwJy4CPk7ebz47OeM67E3CDnuYfaklzftvI49gvz48eOJFmon3qEX9RgMBsTNhPRgnHv8uryLTkFy2sb0ogzXUcxLpzN57lcdNZym6Sx3+DmrSTMb7+wQ9bLMNpNkdGnK6xTZmeTt6AKcKyxTjKxOcdGZnXgbu7ND7yCvV2lO5PTgYy6cM29T+nJdgmnndWxB/uabbyZ6kv1kn1bcIkoi4t04u9rndToXJ82JSTOb7O8Tt1okUcRuHBtZaQYm3sbu79NqtYiiiDg2pdJlGDvEolQqTbTrl6TJ4Z7i8JSCpKlI03Ti85KTZjY9uDU8aWpkpSmbNLMTb2Ov2BzI0lVyUl7HFuQgCEyjNGNnKchmVpq9STNrXqXZOymv3vpbkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKSdIU28FL0mSJA15BFmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKU8/8DJK0ydWj7XvQAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x489.6 with 6 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.figure(figsize=(10, 6.8))\n",
        "for row in range(2):\n",
        "    for col in range(3):\n",
        "        plt.subplot(2, 3, row * 3 + col + 1)\n",
        "        plot_observation(trajectories.observation[row, col].numpy())\n",
        "plt.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0, wspace=0.02)\n",
        "save_fig(\"sub_episodes_plot\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4DcXRL7KMM91"
      },
      "source": [
        "이제 데이터셋을 만들어 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W1GCdTZZMM91"
      },
      "outputs": [],
      "source": [
        "dataset = replay_buffer.as_dataset(\n",
        "    sample_batch_size=64,\n",
        "    num_steps=2,\n",
        "    num_parallel_calls=3).prefetch(3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Zk3mXi6MM91"
      },
      "source": [
        "성능을 높이기 위해 메인 함수를 TF 함수로 변환합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pjQ1bgAKMM91"
      },
      "outputs": [],
      "source": [
        "from tf_agents.utils.common import function\n",
        "\n",
        "collect_driver.run = function(collect_driver.run)\n",
        "agent.train = function(agent.train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OfdoyCEVMM91"
      },
      "source": [
        "이제 메인 루프를 실행할 준비가 되었습니다!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-p7irXxjMM91"
      },
      "outputs": [],
      "source": [
        "def train_agent(n_iterations):\n",
        "    time_step = None\n",
        "    policy_state = agent.collect_policy.get_initial_state(tf_env.batch_size)\n",
        "    iterator = iter(dataset)\n",
        "    for iteration in range(n_iterations):\n",
        "        time_step, policy_state = collect_driver.run(time_step, policy_state)\n",
        "        trajectories, buffer_info = next(iterator)\n",
        "        train_loss = agent.train(trajectories)\n",
        "        print(\"\\r{} loss:{:.5f}\".format(\n",
        "            iteration, train_loss.loss.numpy()), end=\"\")\n",
        "        if iteration % 1000 == 0:\n",
        "            log_metrics(train_metrics)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cNlkxwaLMM92"
      },
      "source": [
        "다음 셀에서 에이전트를 50,000 스텝 동안 훈련합니다. 그다음 다음 셀을 실행하여 에이전트의 동작을 살펴 보겠습니다. 이 두 셀을 원하는만큼 많이 실행할 수 있습니다. 에이전트는 점점 향상될 것입니다! 에이전트가 어느정도 좋은 동작을 수행하려면 200,000 반복 정도 걸릴 것입니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UnCIGFW8MM92",
        "outputId": "337d2a2c-9a3c-4408-8fdb-628cbe281483"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /opt/conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201: calling foldr_v2 (from tensorflow.python.ops.functional_ops) with back_prop=False is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "back_prop=False is deprecated. Consider using tf.stop_gradient instead.\n",
            "Instead of:\n",
            "results = tf.foldr(fn, elems, back_prop=False)\n",
            "Use:\n",
            "results = tf.nest.map_structure(tf.stop_gradient, tf.foldr(fn, elems))\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /opt/conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201: calling foldr_v2 (from tensorflow.python.ops.functional_ops) with back_prop=False is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "back_prop=False is deprecated. Consider using tf.stop_gradient instead.\n",
            "Instead of:\n",
            "results = tf.foldr(fn, elems, back_prop=False)\n",
            "Use:\n",
            "results = tf.nest.map_structure(tf.stop_gradient, tf.foldr(fn, elems))\n",
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 0\n",
            "\t\t EnvironmentSteps = 4\n",
            "\t\t AverageReturn = 0.0\n",
            "\t\t AverageEpisodeLength = 0.0\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "998 loss:0.00008"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 24\n",
            "\t\t EnvironmentSteps = 4004\n",
            "\t\t AverageReturn = 1.7000000476837158\n",
            "\t\t AverageEpisodeLength = 184.1999969482422\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1998 loss:0.00181"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 48\n",
            "\t\t EnvironmentSteps = 8004\n",
            "\t\t AverageReturn = 1.7000000476837158\n",
            "\t\t AverageEpisodeLength = 182.39999389648438\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2998 loss:0.00005"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 73\n",
            "\t\t EnvironmentSteps = 12004\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<<244 more lines>>\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\t\t NumberOfEpisodes = 1003\n",
            "\t\t EnvironmentSteps = 176004\n",
            "\t\t AverageReturn = 5.099999904632568\n",
            "\t\t AverageEpisodeLength = 246.5\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "44998 loss:0.00165"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 1019\n",
            "\t\t EnvironmentSteps = 180004\n",
            "\t\t AverageReturn = 5.199999809265137\n",
            "\t\t AverageEpisodeLength = 256.6000061035156\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "45998 loss:0.00136"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 1035\n",
            "\t\t EnvironmentSteps = 184004\n",
            "\t\t AverageReturn = 4.599999904632568\n",
            "\t\t AverageEpisodeLength = 252.1999969482422\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "46998 loss:0.00100"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 1050\n",
            "\t\t EnvironmentSteps = 188004\n",
            "\t\t AverageReturn = 5.699999809265137\n",
            "\t\t AverageEpisodeLength = 276.5\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "47998 loss:0.00116"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 1063\n",
            "\t\t EnvironmentSteps = 192004\n",
            "\t\t AverageReturn = 5.900000095367432\n",
            "\t\t AverageEpisodeLength = 296.3999938964844\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "48998 loss:0.00049"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:absl: \n",
            "\t\t NumberOfEpisodes = 1077\n",
            "\t\t EnvironmentSteps = 196004\n",
            "\t\t AverageReturn = 7.800000190734863\n",
            "\t\t AverageEpisodeLength = 308.29998779296875\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "49999 loss:0.00073"
          ]
        }
      ],
      "source": [
        "train_agent(n_iterations=50000)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pLO9TGYzMM92"
      },
      "outputs": [],
      "source": [
        "frames = []\n",
        "def save_frames(trajectory):\n",
        "    global frames\n",
        "    frames.append(tf_env.pyenv.envs[0].render(mode=\"rgb_array\"))\n",
        "\n",
        "watch_driver = DynamicStepDriver(\n",
        "    tf_env,\n",
        "    agent.policy,\n",
        "    observers=[save_frames, ShowProgress(1000)],\n",
        "    num_steps=1000)\n",
        "final_time_step, final_policy_state = watch_driver.run()\n",
        "\n",
        "plot_animation(frames)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l0sfIwMrMM92"
      },
      "source": [
        "에이전트를 친구에게 보여주고 싶어서 애니메이션 GIF로 저장하고 싶다면 다음 방법을 사용하세요:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wniytrMWMM92"
      },
      "outputs": [],
      "source": [
        "import PIL\n",
        "\n",
        "image_path = os.path.join(\"images\", \"rl\", \"breakout.gif\")\n",
        "frame_images = [PIL.Image.fromarray(frame) for frame in frames[:150]]\n",
        "frame_images[0].save(image_path, format='GIF',\n",
        "                     append_images=frame_images[1:],\n",
        "                     save_all=True,\n",
        "                     duration=30,\n",
        "                     loop=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "78ons3n3MM92",
        "outputId": "651bd711-39c2-40ee-899b-87336adbb389"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<img src=\"images/rl/breakout.gif\" />\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%html\n",
        "<img src=\"images/rl/breakout.gif\" />"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ybGmShZDMM93"
      },
      "source": [
        "# 추가 내용"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "md6Ca2uYMM93"
      },
      "source": [
        "## Deque vs 로테이팅 리스트"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsOqt6KqMM93"
      },
      "source": [
        "`deque` 클래스는 추가(append)가 빠르지만 랜덤 접근은 느립니다(재생 메모리가 클 경우):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5eMTXxp7MM93",
        "outputId": "d7bc4176-f01c-42cb-db26-4baf1d61f70d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[121958, 671155, 131932, 365838, 259178]"
            ]
          },
          "execution_count": 122,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from collections import deque\n",
        "np.random.seed(42)\n",
        "\n",
        "mem = deque(maxlen=1000000)\n",
        "for i in range(1000000):\n",
        "    mem.append(i)\n",
        "[mem[i] for i in np.random.randint(1000000, size=5)]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "utM_wl_9MM93",
        "outputId": "01bcb5f1-f71e-4497-f56e-1c315a828e94"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "47.4 ns ± 3.02 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)\n"
          ]
        }
      ],
      "source": [
        "%timeit mem.append(1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WJjo2-_7MM93",
        "outputId": "726ae27e-5811-4ff1-dcf1-07c80c45b3a4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "182 µs ± 6.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
          ]
        }
      ],
      "source": [
        "%timeit [mem[i] for i in np.random.randint(1000000, size=5)]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9UkzfOjdMM93"
      },
      "source": [
        "또는 다음의 `ReplayMemory` 클래스 같은 로테이팅 리스트를 사용할 수 있습니다. 재생 메모리가 클 경우 랜덤 접근이 더 빠릅니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Sor4s3k0MM94"
      },
      "outputs": [],
      "source": [
        "class ReplayMemory:\n",
        "    def __init__(self, max_size):\n",
        "        self.buffer = np.empty(max_size, dtype=np.object)\n",
        "        self.max_size = max_size\n",
        "        self.index = 0\n",
        "        self.size = 0\n",
        "\n",
        "    def append(self, obj):\n",
        "        self.buffer[self.index] = obj\n",
        "        self.size = min(self.size + 1, self.max_size)\n",
        "        self.index = (self.index + 1) % self.max_size\n",
        "\n",
        "    def sample(self, batch_size):\n",
        "        indices = np.random.randint(self.size, size=batch_size)\n",
        "        return self.buffer[indices]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QEHf3W1QMM94",
        "outputId": "b9f66f83-c991-435d-df09-6f8da5177ae3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([757386, 904203, 190588, 595754, 865356], dtype=object)"
            ]
          },
          "execution_count": 126,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "mem = ReplayMemory(max_size=1000000)\n",
        "for i in range(1000000):\n",
        "    mem.append(i)\n",
        "mem.sample(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-b_IXtyhMM94",
        "outputId": "86b5047a-ac80-4121-d63c-3ba1b8d5473e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "519 ns ± 17.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n"
          ]
        }
      ],
      "source": [
        "%timeit mem.append(1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cZ47hlV0MM94",
        "outputId": "2683bf27-8f6c-49df-e008-04bf86467606"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "9.24 µs ± 227 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n"
          ]
        }
      ],
      "source": [
        "%timeit mem.sample(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8mqPd8eRMM95"
      },
      "source": [
        "## 사용자 정의 TF-Agents 환경 만들기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cQcOSOEcMM95"
      },
      "source": [
        "사용자 정의 TF-Agents 환경을 만들려면 `PyEnvironment` 클래스를 상속하는 클래스를 만들고 몇 개의 메서드를 구현해야 합니다. 예를 들어 다음과 같은 환경은 간단한 4x4 그리드를 표현합니다. 에이전트가 한쪽 코너 (0,0)에서 시작하여 반대쪽 코너 (3,3)으로 이동해야 합니다. 에이전트가 목적지에 도착하면 에피소드가 끝납니다(+10 보상을 받습니다). 또는 에이전트가 경계를 벗어나면 끝납니다(-1 보상). 행동은 위(0), 아래(1), 왼쪽(2), 오른쪽(3)이 가능합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mHtkxartMM95"
      },
      "outputs": [],
      "source": [
        "class MyEnvironment(tf_agents.environments.py_environment.PyEnvironment):\n",
        "    def __init__(self, discount=1.0):\n",
        "        super().__init__()\n",
        "        self._action_spec = tf_agents.specs.BoundedArraySpec(\n",
        "            shape=(), dtype=np.int32, name=\"action\", minimum=0, maximum=3)\n",
        "        self._observation_spec = tf_agents.specs.BoundedArraySpec(\n",
        "            shape=(4, 4), dtype=np.int32, name=\"observation\", minimum=0, maximum=1)\n",
        "        self.discount = discount\n",
        "\n",
        "    def action_spec(self):\n",
        "        return self._action_spec\n",
        "\n",
        "    def observation_spec(self):\n",
        "        return self._observation_spec\n",
        "\n",
        "    def _reset(self):\n",
        "        self._state = np.zeros(2, dtype=np.int32)\n",
        "        obs = np.zeros((4, 4), dtype=np.int32)\n",
        "        obs[self._state[0], self._state[1]] = 1\n",
        "        return tf_agents.trajectories.time_step.restart(obs)\n",
        "\n",
        "    def _step(self, action):\n",
        "        self._state += [(-1, 0), (+1, 0), (0, -1), (0, +1)][action]\n",
        "        reward = 0\n",
        "        obs = np.zeros((4, 4), dtype=np.int32)\n",
        "        done = (self._state.min() < 0 or self._state.max() > 3)\n",
        "        if not done:\n",
        "            obs[self._state[0], self._state[1]] = 1\n",
        "        if done or np.all(self._state == np.array([3, 3])):\n",
        "            reward = -1 if done else +10\n",
        "            return tf_agents.trajectories.time_step.termination(obs, reward)\n",
        "        else:\n",
        "            return tf_agents.trajectories.time_step.transition(obs, reward,\n",
        "                                                               self.discount)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ASFuy2U5MM95"
      },
      "source": [
        "행동과 관측 스펙은 일반적으로 `tf_agents.spec` 패키지에 있는 `ArraySpec`이나 `BoundedArraySpec`의 인스턴스입니다(이 패키지에 있는 다른 스펙도 살펴 보세요). 선택적으로 `render()` 메서드, 자원을 해제하기 위한 `close()` 메서드를 정의할 수도 있습니다. 또한 `reward`와 `discount`를 32비트 실수 스칼라로 사용하고 싶지 않다면 `time_step_spec()` 메서드를 정의할 수 있습니다. 베이스 클래스는 현재 타임 스텝을 추적하므로 `reset()`, `step()` 대신에 `_reset()`, `_step()`을 구현해야 합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8ikv6ppsMM95",
        "outputId": "3a07cfe3-ab81-4e1c-a580-c9b52e740e59"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TimeStep(step_type=array(0, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[1, 0, 0, 0],\n",
              "       [0, 0, 0, 0],\n",
              "       [0, 0, 0, 0],\n",
              "       [0, 0, 0, 0]], dtype=int32))"
            ]
          },
          "execution_count": 130,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "my_env = MyEnvironment()\n",
        "time_step = my_env.reset()\n",
        "time_step"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nyL2ivJOMM95",
        "outputId": "27944b75-d53d-4d93-c2bc-74c8e2001cab"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TimeStep(step_type=array(1, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[0, 0, 0, 0],\n",
              "       [1, 0, 0, 0],\n",
              "       [0, 0, 0, 0],\n",
              "       [0, 0, 0, 0]], dtype=int32))"
            ]
          },
          "execution_count": 131,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "time_step = my_env.step(1)\n",
        "time_step"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ZgnO0JSMM96"
      },
      "source": [
        "# 연습문제 해답"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k5WRiK2CMM96"
      },
      "source": [
        "## 1. to 7.\n",
        "\n",
        "부록 A 참조"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lzXGaVbIMM96"
      },
      "source": [
        "## 8.\n",
        "_연습문제: 정책 그레이디언트를 사용해 OpenAI 짐의 LunarLander-v2 환경을 해결해보세요. 이를 위해 Box2D 패키지를 설치해야 합니다(`%pip install -U gym[box2d]`)._"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5qzPiBWzMM96"
      },
      "source": [
        "먼저 LunarLander-v2 환경을 만들어 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J3QWnu4wMM96"
      },
      "outputs": [],
      "source": [
        "env = gym.make(\"LunarLander-v2\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w-QO2DYPMM96"
      },
      "source": [
        "입력은 8차원입니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7v8hpHyCMM96",
        "outputId": "d38cb675-0bfc-4127-a88c-d41843bee1f0"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Box(-inf, inf, (8,), float32)"
            ]
          },
          "execution_count": 241,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.observation_space"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "khWHDPkuMM96",
        "outputId": "77511fba-ad73-4e79-9dc8-0998a98426ed"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([-0.00499964,  1.4194578 , -0.506422  ,  0.37943238,  0.00580009,\n",
              "        0.11471219,  0.        ,  0.        ], dtype=float32)"
            ]
          },
          "execution_count": 242,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.seed(42)\n",
        "obs = env.reset()\n",
        "obs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FSwLy_zXMM97"
      },
      "source": [
        "[소스 코드](https://github.com/openai/gym/blob/master/gym/envs/box2d/lunar_lander.py)를 보면 8D 관측(x, y, h, v, a, w, l, r)이 각각 다음에 해당합니다:\n",
        "* x,y: 우주선의 좌표. (0, 1.4) 근처의 랜덤한 위치에서 시작하고 (0, 0)에 있는 목적지 근처에 내려야 합니다.\n",
        "* h,v: 우주선의 수평, 수직 속도. 랜덤한 적은 속도로 시작합니다.\n",
        "* a,w: 우주선의 각도와 각속도.\n",
        "* l,r: 왼쪽이나 오른쪽 다리가 땅에 닿았는지(1.0) 아닌지(0.0) 여부."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4J8l50gDMM97"
      },
      "source": [
        "행동 공간은 이산적이며 4개의 가능한 행동이 있습니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xs0R_U5TMM97",
        "outputId": "d886221b-af4e-414e-9e96-48be9c16792d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Discrete(4)"
            ]
          },
          "execution_count": 243,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env.action_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "alRmUUExMM97"
      },
      "source": [
        "[LunarLander-v2 설명](https://gym.openai.com/envs/LunarLander-v2/)을 보면 이 행동은 다음과 같습니다:\n",
        "* 아무것도 하지 않음\n",
        "* 왼쪽 방향 엔진을 켬\n",
        "* 주 엔진을 켬\n",
        "* 오른쪽 방향 엔진을 켬"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YDfJ1tdvMM97"
      },
      "source": [
        "(행동마다 하나씩) 4개의 출력 뉴런을 가진 간단한 정책 네트워크를 만들어 보죠:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gREhSkWHMM97"
      },
      "outputs": [],
      "source": [
        "keras.backend.clear_session()\n",
        "np.random.seed(42)\n",
        "tf.random.set_seed(42)\n",
        "\n",
        "n_inputs = env.observation_space.shape[0]\n",
        "n_outputs = env.action_space.n\n",
        "\n",
        "model = keras.models.Sequential([\n",
        "    keras.layers.Dense(32, activation=\"relu\", input_shape=[n_inputs]),\n",
        "    keras.layers.Dense(32, activation=\"relu\"),\n",
        "    keras.layers.Dense(n_outputs, activation=\"softmax\"),\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iq235HmbMM97"
      },
      "source": [
        "출력 층에 CartPole-v1 환경처럼 시그모이드 활성화 함수를 사용하지 않고 대신에 소프트맥스 활성화 함수를 사용합니다. CartPole-v1 환경은 두 개의 행동만 있어서 이진 분류 모델이 맞기 때문입니다. 하지만 두 개 이상의 행동이 있으므로 다중 분류 모델이 됩니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eraao9BuMM97"
      },
      "source": [
        "그다음 CartPole-v1 정책 그레이디언트 코드에서 정의한 `play_one_step()`와 `play_multiple_episodes()` 함수를 재사용합니다. 하지만 다중 분류 모델에 맞게 `play_one_step()`를 조금 수정하겠습니다. 그다음 수정된 `play_one_step()` 를 호출하고, 우주선이 최대 스텝 횟수 전에 랜딩하지 못하면 (또는 부서지면) 큰 페널티를 부여하도록 `play_multiple_episodes()` 함수를 수정합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zVxW3agtMM98"
      },
      "outputs": [],
      "source": [
        "def lander_play_one_step(env, obs, model, loss_fn):\n",
        "    with tf.GradientTape() as tape:\n",
        "        probas = model(obs[np.newaxis])\n",
        "        logits = tf.math.log(probas + keras.backend.epsilon())\n",
        "        action = tf.random.categorical(logits, num_samples=1)\n",
        "        loss = tf.reduce_mean(loss_fn(action, probas))\n",
        "    grads = tape.gradient(loss, model.trainable_variables)\n",
        "    obs, reward, done, info = env.step(action[0, 0].numpy())\n",
        "    return obs, reward, done, grads\n",
        "\n",
        "def lander_play_multiple_episodes(env, n_episodes, n_max_steps, model, loss_fn):\n",
        "    all_rewards = []\n",
        "    all_grads = []\n",
        "    for episode in range(n_episodes):\n",
        "        current_rewards = []\n",
        "        current_grads = []\n",
        "        obs = env.reset()\n",
        "        for step in range(n_max_steps):\n",
        "            obs, reward, done, grads = lander_play_one_step(env, obs, model, loss_fn)\n",
        "            current_rewards.append(reward)\n",
        "            current_grads.append(grads)\n",
        "            if done:\n",
        "                break\n",
        "        all_rewards.append(current_rewards)\n",
        "        all_grads.append(current_grads)\n",
        "    return all_rewards, all_grads"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZJ3-NqLoMM98"
      },
      "source": [
        "앞에서와 동일한 `discount_rewards()`와 `discount_and_normalize_rewards()` 함수를 사용합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wK2nO-T0MM98"
      },
      "outputs": [],
      "source": [
        "def discount_rewards(rewards, discount_rate):\n",
        "    discounted = np.array(rewards)\n",
        "    for step in range(len(rewards) - 2, -1, -1):\n",
        "        discounted[step] += discounted[step + 1] * discount_rate\n",
        "    return discounted\n",
        "\n",
        "def discount_and_normalize_rewards(all_rewards, discount_rate):\n",
        "    all_discounted_rewards = [discount_rewards(rewards, discount_rate)\n",
        "                              for rewards in all_rewards]\n",
        "    flat_rewards = np.concatenate(all_discounted_rewards)\n",
        "    reward_mean = flat_rewards.mean()\n",
        "    reward_std = flat_rewards.std()\n",
        "    return [(discounted_rewards - reward_mean) / reward_std\n",
        "            for discounted_rewards in all_discounted_rewards]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p-oWW-uDMM98"
      },
      "source": [
        "이제 몇 개의 하이퍼파라미터를 정의합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j62ORPXfMM98"
      },
      "outputs": [],
      "source": [
        "n_iterations = 200\n",
        "n_episodes_per_update = 16\n",
        "n_max_steps = 1000\n",
        "discount_rate = 0.99"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VTOAXopvMM98"
      },
      "source": [
        "여기서도 다중 분류 모델이기 때문에 이진 크로스 엔트로피가 아니라 범주형 크로스 엔트로피를 사용해야 합니다. 또한 `lander_play_one_step()` 함수가 클래스 확률이 아니라 클래스 레이블로 타깃을 설정하기 때문에 `sparse_categorical_crossentropy()` 손실 함수를 사용해야 합니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MHQ544cjMM98"
      },
      "outputs": [],
      "source": [
        "optimizer = keras.optimizers.Nadam(learning_rate=0.005)\n",
        "loss_fn = keras.losses.sparse_categorical_crossentropy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YV9y4MpRMM99"
      },
      "source": [
        "모델을 훈련할 준비가 되었네요. 시작해 보죠!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WaWxYxI0MM99",
        "outputId": "c55c97c9-a912-4870-bf33-406d5f43eb96"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Iteration: 200/200, mean reward: 134.2  "
          ]
        }
      ],
      "source": [
        "env.seed(42)\n",
        "\n",
        "mean_rewards = []\n",
        "\n",
        "for iteration in range(n_iterations):\n",
        "    all_rewards, all_grads = lander_play_multiple_episodes(\n",
        "        env, n_episodes_per_update, n_max_steps, model, loss_fn)\n",
        "    mean_reward = sum(map(sum, all_rewards)) / n_episodes_per_update\n",
        "    print(\"\\rIteration: {}/{}, mean reward: {:.1f}  \".format(\n",
        "        iteration + 1, n_iterations, mean_reward), end=\"\")\n",
        "    mean_rewards.append(mean_reward)\n",
        "    all_final_rewards = discount_and_normalize_rewards(all_rewards,\n",
        "                                                       discount_rate)\n",
        "    all_mean_grads = []\n",
        "    for var_index in range(len(model.trainable_variables)):\n",
        "        mean_grads = tf.reduce_mean(\n",
        "            [final_reward * all_grads[episode_index][step][var_index]\n",
        "             for episode_index, final_rewards in enumerate(all_final_rewards)\n",
        "                 for step, final_reward in enumerate(final_rewards)], axis=0)\n",
        "        all_mean_grads.append(mean_grads)\n",
        "    optimizer.apply_gradients(zip(all_mean_grads, model.trainable_variables))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "axnoL3jMMM99"
      },
      "source": [
        "학습 곡선을 그려 보겠습니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ol8ai_ZzMM99",
        "outputId": "433c404c-b364-404d-b70a-9e201fd0949f"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAENCAYAAADDmygoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABT3UlEQVR4nO2deXhjZ3W432PLlmxL3vfZ9zWZJDPZt0kTEgh7UiBAAvxYQktTaMvaEiAt0AItbYHShpQlQCEJ0CQQ0pCNOHsmmUky++7ZPON9lWxL1vL9/rj3yrIt75KtGZ/3ee5j+X7fvTq6ku7RWb5zxBiDoiiKoqSSrNkWQFEURTnzUOWiKIqipBxVLoqiKErKUeWiKIqipBxVLoqiKErKcc22AJlAeXm5Wbx48ZSO7e3tpaCgILUCpYhMlU3lmhwq1+TJVNnONLm2bdvWZoypSDpojJnz28aNG81Ueeqpp6Z8bLrJVNlUrsmhck2eTJXtTJML2GpGua+qW0xRFEVJOapcFEVRlJSjykVRFEVJOapcFEVRlJSTkcpFRG4Tka0iEhKRuxP2LxYRIyKBhO1LCeMiIt8UkXZ7+5aIyKy8CEVRlDlMpqYinwK+BlwH5CUZLzbGRJLsvxV4B7ABMMDjQD1wZ3rEVBRFUZKRkZaLMeZ+Y8yDQPskD/0g8G1jTIMx5iTwbeBDKRZPURRFGQcxGVxyX0S+Bsw3xnzI/n8xcATLsnEsk88aY9rs8W7gWmPMFvv/TcBTxhhfknPfimXpUFVVtfHee++dkoyBQACv1zulY9NNpsqmck0OlWvyZKps05Xr1eYIiwqzKMsb3y6IxAyurIlFBaYq11VXXbXNGLMp6eBoC2AyYcNyjd2d8L8X2ITlzqsCfgM8mjAeBVYn/L8CSwnJWM+jiyhnFpVrcqhckydTZZuOXH2hiFnyhd+brz60e9y5Oxu6zIov/p/52YtH0yoXZ8oiSmNMwBiz1RgTMcY0A7cB14pIoT0lABQmHFIIBOyLoCiKctpyuDVAzEBjdzC+78m9zZz7D4/R2TswZO7uU90MRGJ86cFd/PzFozMsqcVppVyS4CgNx/bbjRXMd9hg71MURTmtOdwaAKCpZ1C5PLyjkc6+MK83dA2Ze6KjnyyBDfOL+OXLJwD4+sN7+Mjdr8yYvBmpXETEJSIeIBvIFhGPve9CEVklIlkiUgZ8F6gzxnTbh/4M+BsRmScitcCngbtn5UUoipIWdjZ08+LhsXN9ukOG5w62zZBEQ3l4RyM/fLaeZw60MhWnSSQa4+8f2s3Rtl6MMfzr4wc42OzncIutXGzLxRjDc4es17iroXvIORo6+6gpyuPs+cU0dvcD8PLRTv64v4X2QGg6L2/CZKRyAW4H+oEvADfbj28HlgJ/APzALiAEvDfhuB8ADwE77fGH7X2KomQwLxxq46J/fBJ/MDzu3Nsf3MlXfrdrzDmPHBnglh9v4URH36hzjrT1ctNdL9IzgeecKCc6+vjLe17law/v5QM/fplP/3o7wXB0Uuc42BLgJ88f5f5XGzjR0c93nzzIj58/wiHbcmnuCRKNGQ62BGjxW4pi58mhyuVEZz8LSvOoLc6jqy9MbyjCyc4+jIFnZ0jpZqRyMcbcYYyRYdsdxph7jDFLjDEFxpgaY8wHjDFNCccZY8znjDGl9vY5jbcoSuazvaGbpp4gJzr6x5zX1TfAjpPddAyLMQD8/MWjvOcHLwJwwh/DGPjfVxtGPderxzp5qb6DfY3+6QmfwN0vHCVLhEf/6gr+6poV3P/qSf72/p2TOofj/tp9qoddpyyl8dyhNg639AJWFlh7IBS3zDYuKmHXcOXS0ceCknxqiz2ApUjbAtY1q9vfMvUXOAkyUrkoijK3aLNdNa3juGxePNyOMdDVFx7hcnpibwtbjnTQ1B2kIWCN/XprA7FY8t+XgZC1DjsxhjEdeoJh7nvlBG85u4ZV1T7+6pqVfPzKpTz4+kkO2S6tieAokV2nuuNK40RHPwda/CyvtNKFG7uDPHeojSXlBVy3ropT3cG4uysYjtLiDzG/JJ95xdYa9JePdABQkJvNMwfbCEdjo16XVKHKRVGUWSeuXPxjK5dn7RhDJGbiysFhX1MPYP0y7w4ZNiwo5mRXPy+MEp+JK5fuQWvpt6+f5MHXTk5I5oe2n6LFP6iYHnj1JIFQhI9ctjS+79bLl+JxZfP9pw5N6JwwaLk094So299KcX4OAMbAZcvLATjV1c/LRzq4ZFkZ6+cVAYOusZNd1utx3GIwqFzeed48OnoHOPcfHueSb/yRrUc7JizXZFHloijKrDOacglFovz8xaOEIlbc4vlDbTjrArv6BmMlHb0DNPdYx9631cqO+surlpPryuLpA8ndQP6go1wGn/OHzx7hJ88fGVfeFn+Qv7znNX703ODcR3Y1sqrKx1nzi+L7yrxubrl4Eb99/SQNnUPjP7GYYc+pnhEW2OHWAD6PVZlrT2MP16ypoqbIcm85yuWZg60EQhE2LS6JK5dBK8d6ngWl+VT63GRnCa/YSuSWixZz0dJS3ri+Gk9OFjfd9dKElelkUeWiKMqs027HA4Yrl2cPtPGl3+7m/ldPcrStl2PtfVxq32A7+wbjLo7Vkp0lvHa8C4CzFxRRW+QZsi4kkUDIUk7NCW6xFn+Qrv7xA/xOnGbHCeuG3tk7wMtHOrh2XdWIue+/cCExA4/tbo7v6+gd4P/d/QrXf/dZnj80aFnFYob61l7euK46vu+seUVcssx6zecvLiU3O4tH7XOdt7CEQk8O80vyONBsWTwnOm3LpSQfV3YW1YUe2nsHcGUJyyu93HvrxfzLuzbw29su443rq1lbm7g0MHWoclEUZdYZLebSbLud7nn5OD98rp6cbOHdmxYA0JlguTg3+6tXVwLgzYEKr5vqIk88dXc4AdtycVJ1ozFDqz80xCIaDUeZ7TzZTSxmeHJfCzED166tHjF3UVkBKyq9PL5nULl86t7XeOGw5eLb3zyYUNDYE6Q/HOWchcUsLssHYP28Qv7iqmX84zvPoig/h6oiNx29A5QW5LKw1JpT5nXHlWJDZx+52VlU+twA8aB+TbGH7IRyMEV5OfzH+85jZdWI6lgpQZWLoijsOtnNd544OCvPHY2ZePZXq3+oImixXV07Grq55+UT/OnGBaypsX5pd/UN8MrRDn6x5Rj7mnooK8jlmjWW5TDfl4WIUFuUN4blYikXx53W3hsiZqzAfHScYPe+Jn/8HPVtAR7d3URNkYf185JbAW9YW8XLRzvoDRu6+8K8cLidj12+FK/bNSRd2lnLsqzCy7raIkRgTU0hSyu8vO/ChQDUFFpxlPMWFuN0FCnKy6HbtuQaOvqZV5JHlq1InLjL/OL8MV9TqlHloigKD7x2kn974gC9oWSdLCbH7lPdk8pE6ugdwJnupMs6tAZCeN0uPDnWreoTm5fFA9xdfWF+9OwRvvjALh7Z1cTqGh8bFhQDMM9rza8u8sTXhbT6Q/QPDK45cWIuzT1BYjETV2TGQE8S11gsZvjXx/ZzoqOPfY3+eCbWk3tbeOZAK9eurWK09lFvWFtFNGbY3hql7kAL0ZjhmrVVLCzN51h7b3yeE8xfVuHlw5ct4YvXryE/d2hnlGo7/nLuwpL4vuK8HLody6Wrn/klg51KHOUyL2HfTKDKRVGUuOUw2q/8iXKsvZc3f/c5Ht3dNP5km/Ze66Ze7nWPiLm09ISYX5LHp9+wik9fu5IFpfkU51nKpbNvgFO2S8sfjLC6upDllV7ecU4tF9VYN+SaIg+RmKEtEOJt//Ec//zo/vi5HcslEjO09YaGxF6SxV32Nfn57h8P8e3H9nOoJcCb1ldTkJvNvz5+gHA0xgcvWTzqa9wwv5hKn5tnGsI8sbeFsoJcNswvtpRLguVyoNkK5pd7c9m4qISPXr50xLkc5XJegnIpysuJy9zmD1HhdcfHau3581W5KIoy07THlcvYixjH46QdTK5v6x1n5iBtfuu519T46O4PxzPDwLJcKnxuPnbFUj6xeTkAruwsfB4XXX1hTnX1c9HSUsq9uVyyrIzsLOHfbzqXFSXZANQUWTfU14530dgdZMuRweC5PxihINea19wdiq92B8vlNpw9jVac5bfbTzEQjbG2tpD184oIRWK889z5LK0YvWR9VpbwV9esZF9HjIe2n2Lzqkqys4RFZfk0dPQTixki0RhP7G3m4qVlo1pAMKioNiwYzEorzs+hpz9MLGbo7LPiMQ5xy6VYlYuiKDNMh209TNdycQLyDZ3jK6mdDd18/eE98bUiTiwl0TXW2hOkwucecWxJfi5N3UHaAgNctrycV754DVevGZmp5fzKd1al72/yx8uxBEKR+KLEpp7guJbLnlM9ZGcJTubw6upCNi0uISdb+NTVK8Z9ve+9YAEX11rK7Oo1VuLBwrJ8BqIxmnqCvHC4nVZ/iHeeO2/M87z57Bq2/N3VQ9xlRXk5xIyVGNE3EKUkQblsWFDMuQuLuXBJ2bgyppJMbXOsKMoM4qQCN3ZNU7nYv/6dhXyj0dU3wMd/vpVT3UFusG+ma2p88XPMK87DGENrIESlzzPi+OL8nLglUVucN+ov/Zq4cmkFLBfYnsYezltYQiAYYVml1yo9090/xHLpTpIxtrexh/XzisgWq1zNssoCPrF5Oe88dx4Ly8YPlosIH1rn5sZLV3DtWksROtlexzv6ePD1k/g8Lq6yM97GO1ciRbar8IhtMZYlKJdyr5sHPnHpuOdMNWq5KMocxxiTMreYY7mcHLZgsKtvgJvuepFnDlg3+b+9f2f8Zv7YnmZys7NYWm5ZEY6C6uoLE46apJZLcX4ux+1YRe0Y7p7SglxyXVk09QTjLrAdJ7oIRaIMRGMsLisgO0to6gnS0hOMK6POYW4xYyyltLamkH94+3r+8Z3rcbuyKXC7WF458VRed7bwpxvn48q2br2LSgsAy6J6dFcT16+vwZOTPeHzOQxXLomWy2yhykVR5ji9A1EGIjHAcou9VN/OTXe9yKvHOyd9Lid+crKrP77y3BjDZ3+zg5fqO3hybzP+YJhHdjXx0cuXsrSigEAoQpk3l8pCS4k4ysVRPpVJ3WI58cdjxRJEJK4wzl9SSoXPzY6G7vgal6K8HKp8bpq6QzT3hOJusuFrXU51B+nuD8fjLO85f+Gkr00ynLUn33nyIL0DUd5zwYIpnac431ImySyX2UKVi6LMcToSYhyN3f3c/2oDL9V38K47X+SRnY2TOldrvHhiLL7I8b5XTvD4nmZysoVDrYF4EceNi0q41F55XubNpaxguHKxXHTJlYt18xQZjKuMRnWhNb66upAN84t5vaErninmdbtYVull27EOmnuC1BblUehxxdN6Hfacslxwa2tSu5o9JzuLecV5dPQO8Jaza4ZkgE0Gx3KpV8tFUZTZYPepbh4blibspAIvKsunsSvItmOdXLS0lLKC3EmlFIOVBuuyF++d7LSsl7ueqWfDgmLeenYth1oCHLTLlKyo9HLpcivIXO51k+vKoiQ/J65UHCWTzC3m3EyrfB5ysse+jTmWy5oaHxvmF1Hf2htPXPB6XLxtQy1H2/to8YeoLHRTnJ87Iltsz6keRGB1depXsy8qy8ftyuJvr18z5XM4a3/UclEUZVa465l6Pv3r7RhjCEdjBEKReDB/fW0R/lCEw629XL6igsVlBZPOHmsNhFhtB+ZPdvWx5UgH9W293HLRIlZU+WjuCfHq8U7criwWlOZz0dIyRCzlAlb8xKmNFXeLFY60TBy3mFPaZCxqbLfZqmofK+xSJ06RR5/bxfVn1ZBvx2MqCz0U5+eMyBZ7an8Lq6p8FLhTnwP1+Teu5kcfPH9aqcKOsj3e3kd2llDoyRnniPSjykVR5hBdfWH8wQinuoN854mDvPHfn4kvoFyXULrk3IXFVNmr2ydKzC7jsmF+MWClI9/z8nF8HhdvPqsmHs94bE8zyyq8ZGcJxfm5fPH6NbznfCvWsKrKxwG7tEqrP0ReTnY8EJ+I4/YZK5jvcMWKCi5fUc6yCi8LSq35TqaZ1+OiwO3iTetrAKjyOZbLoHLZfqKL1090cdP5U4uHjMf6eUVctqJ8Wufw5GTjdmUxEI1Rkp8TL/0ym6hyUZQ5hNPSd39TD88daqOhs58dJ7sAy3IBq7LwhvnF1NgVhY0x3P38EZ5K0sGwuz8cT9vt7BsgGjOsqPSSn5vNtmOdPLKziRvOnUdebnZcuXT0DrCyanDB4UcvX8r5i0sBWFnto6nHCp47bqpkacZOAHsiv/YvXlbGzz9yITnZlrUEsNcudOm1LZGbL1qI1+1iVbWP4rycIW6xn714jILcbG7cOH/c55pNHOvFiUfNNqpcFOUM43j76H3jnZpZOxq640HqZw+24cnJYmmFlRa7psZy/1QVeghFYnT1hfmXxw7w4+dG9jn59K9e55P3vgYMBvMrfB5qi/N4ZFcT2VkSL2GyoCSPXDs+smKUSryO0jnY7KelJ5g0mA/ES8BMxHJJpNCTQ1FeDodabOVi9005d2EJO++4lkVlBUPcYu2BEA/tOMUN583HlwGuprFw4i6lGRBvAVUuinJG8eLhdq7456d49Xgn0ZjhV1tP8Mstx+Ml4nvsFNzfvm6VMAE41t5HWYGbqkIPOdnCRjtjyQmE7zzZTSAUiVfsTeRwa2/cxeSkIZd7c+MWxWeuWxW3FlzZWSwptxTYisrkpVKc8u+7T/Wwp7EnPn84yyu9XLKsLN7bZTIsKM0jHLXSpH3uQYXhWEhOEchYzHDf1hMMRGJ84OJFk36emcaxXFS5jIGI3CYiW0UkJCJ3Dxu7WkT2iUifiDwlIosSxkREviki7fb2LRmrSI+inGFsb+gCoG5fC3/c18LnfrODv3tgJ39z33Zg0HJxsoqcG1GZN5ec7Cx++uEL+Eu7lEmVHUh32gSf6g4OaS1sjKGpO0irP0TfQITWgBWfKfe5uWZtFW9cV82HhhVzdFxjo1ku84rzKMjN5n9eOoY/GImX0B9OgdvFLz92Ufx8k2FBiaXssrMkXm05kaL8XIyx3Hy/eOk4lywrG1XeTKIoz3ovVbmMzSnga8CPE3eKSDlwP/AloBTYCtyXMOVW4B3ABuBs4C3Ax9MvrqJkBk4w/PnD7Tyxpxmfx8Xbz6mlxR8iGI4SisTIdVlf+3KvmytXVgCDN6RLlpXHM7ccy+VFu6kVMMR66Y9Av12n61h7X9xyqfC5ueWiRdx5y8YhzakAzl9cQlWhO172ZDgiwooqHwdbAnhysrh8RcX0LkgSHEvK53Eljec4mWj3v3qSk139fODixSmXIR2o5TIBjDH3G2MeBNqHDd0A7DbG/NoYEwTuADaIyGp7/IPAt40xDcaYk8C3gQ/NjNSKMvs4XQ1fP9HF43ub2byqkvkleXT2DcQXBp5j9zw5Z0ER6+wWt8luSBU+NyKWW8zhUIJy6QwN9mw51t5LayBErisL3xjpuh+4eDHPfu5PRiidRFbZVsIVKyrIS5IpNl0W2KXnvaPI6cQu/uWx/cwrzuOaNePX+soEMi3mcroVrlwHbHf+Mcb0ishhe/++4eP243XJTiQit2JZOlRVVVFXVzclgQKBwJSPTTeZKpvKNTkmKlfMGPY39bHQl8Vxf4yO3gFqTTsdjYZozPDQk88DUJVlKSBfuJNIqxUv6etoSfocRblCV8hQWyA09xme3LqHMv8hAE519gGWkvjjyztpCBh8LsPTTz89rdebFbCU4MLszrR8LztbLdeeREJJ5zT0WNZYVR7cug6ee/aZKckwWbmmS0eTZTk2HTtMXfjYpI5Nh1ynm3LxAq3D9nUDvoTx7mFjXhER4xQ6sjHG3AXcBbBp0yazefPmKQlUV1fHVI9NN5kqm8o1OSYqV31rgMijT/Oxq9fy1d/vIRozfOIdV/LH/c3cs287JQtXwZbt3HD5OWxa18dbzq4hx5XFd157kkvPWcXmi0YGrRfueo6uhm42LKniUEuAAU8BmzdvAuC5XzwODCAC2cU1tHd3snp+Lps3XzSt17u0vY/O7L188sazp7wYcKxrNr8lwL9ue5rqsiI2b74k6ZxzzutidXVh3IWYKtL5GTvuPsoDh3Zz2fkbJu1OTIdcp5tyCQDDi/sUAv5RxguBwHDFoiiZRos/yJ119Ty6u4k73raON6xNHsgeiwO2S2zD/CKuWlVBNGYoys+h1K7ZddRup1uYlzOka+IfP3NlPM4ynOpCDzvoZmlFASKDNbYAOoPW12pNdSFb6tupb+vlTzdOvYSJw8KyfO68ZeO0zzMa88dxiwGcbS8EPZ1w1rc4Ndpmm4yMuYzBbqxgPQAiUgAss/ePGLcf70ZRMpy/u38nP3/pKP3hKH//0O4h3Rgnyv6mACJWRtb333ced95s3aCdOlNOhpgT+HWoKcobtT6XE9RfWu5leYWX4x198WZbnSFDUV4Oq6p98YKJm1elPgCfajw52dQUeUZch9OdN6yt4l/fvSHeF2e2yUjlIiIuEfEA2UC2iHhExAU8AKwXkRvt8S8DO4wx++xDfwb8jYjME5Fa4NPA3bPwEhRlwnT3h3n6QCsfumQx37npHBo6+/nZC5PzmYNluSwszSc/14UrOyveM8QplTJouUzcYVFlK5clFQWsqi4kZmDrUasUf1fQUF3oYZHdKGtecd6UUoNng/9437l86pqVsy1GSvHkZHPDefPHbJE8k2SkcgFuB/qBLwA3249vN8a0AjcCXwc6gQuBmxKO+wHwELAT2AU8bO9TlIzl8T3NhKOG68+q4XK7DtbX/28vb//+8+xt7Bn/BDZ7GnviixATKbXdJUfbrJX7k4ljXLqsnAuWlLK62sfVayqp8Ln5r6etgH5nyFBZ6GZxmbXQcfOqioy5sY3HxkWloy7QVFJDRioXY8wdxhgZtt1hjz1hjFltjMkzxmw2xhxNOM4YYz5njCm1t89pvEXJJA61BOKlRxz+b2cj84rz4inC33vvuXz2ulVsP9HFE3uaJ3Te9kCII229SfuB5OVmk5eTTSAUIdeVNalOhxsWFPOrj19Mfq4LT042t16+lOcPtbPtWCedtuWytrYQEXjj+uoJn1c588lI5aIoZyp/98BObn9wV/z/nmCYZw+2cv1Z1YPlR/Jz+YurlpObnUVgIDLaqQD4wv/u4OcvHmXbMctVtWlx8mZTztqH6ZZif9+FCynJz+Hbj+2nO2SoLvKwssrHlr+7Oi0LHpXTF1UuijKDnOzsp7N3sJz78fY+wlHDxkUjlUKBO5teu9zK0bbeeCDdYdfJbu595QTf++MhXj7SQU62cNa8oqTPW+a1lEvRJOItyShwu/jk1St44XA7hsFeK5W+8fuqKHMLVS6KMkNEY4bmnqH1uZzHySyKAreL3lAUYwxv/d5zfPKe10j08v70haOA1VTr3ldOsH5e0aguLydNtTAFGVK3XLQo3pGxOkkjL0UBVS6KMmO0B0JEYibeUwXAb1cpdkq/J+J1uwiEIgxEwR+K8NieZh62e9p39A7w2+2neNfG+RR6rHmbklg/DmUpcouBVd346+88i4o8iZePUZThqHJRlBnCaRkcCEXiFkggZCmaZL1CCtwu+gYi9EWsuVkCd/xuNwORGI/samQgEuMjly/hzWfXAiR1rTnEYy4pWtuxcVEJ/3xl/qT7qShzB1UuijJDOMrFGOgdsOInccslyWrxAreLQChKv+1Fu3pNFW2BAfY3+dl1spvi/BxWVfn48KWLuXJlBZeM0dukJG65nG5FOZTTFVUuijJDNHX3xx/7bdeYo1x8Sd1iVkC/37ZcLl1WBlhVinef6mFdbWG8RP1PP3zBmC6vshRbLooyHqpcFGWGaOoJxR8HbKXiD0bIzU6+9qQg12UrF+v/tbVFFHpcvH6ik31NftbWTDze4VguZ1rJEyVzUeWiKDNEouXitBsOhMJJg/nguMUGLRefx8X6eUX8YVcTA5EY62qTpx0nI5UBfUWZCKpcFGWGaOwOkmvX+0p0iyVzicHgOpe+BOVy1ryiuGJaO4lMrYVl+bhdWSyr0JInysyg0T1FmSGaeoIsrShgX5M/vr4lEIyMWvq9wO0iZqDH7vjoc+ewzl4k6XZlsXQStbEqfR52/f11o1Y/VpRUo580RZkGT+1vobknOO48YwyN3UFW2IUl/Qkxl9EsF0fpOH1TvLblArC62hevejxRVLEoM4l+2hRlikRjho/9dCs/fv7IuHM7+8IMRGKssEvSxwP6oQhed/I4SEGurVxChvzcbLKzhEWl+ZR7czk3SYFKRckk1C2mKFOkq2+ASMzQ3D2+5dJkz3E6Og7GXMIUepI3dypIsFycRZZZWcJvb7tMs76UjEeVi6JMkc6+AQBaA6FxZhJ3ndUU5eF1u+JBeX8wMmq2WNwtFjKUFw7Omaer4pXTAHWLKcoU6eyzrI+WnvGVS6vfmlPpc+OzU4yNMQRCY2eLgRXQT1YeRlEyGVUuijJFOnonbrk4cyp8bnyeHPzBMP3hKNGYGTXm4lguhuQr+BUlk1HloihTpMt2i3X1hQlForz/hy9x9yjB/VZ/CJ/b6ubos6sYB8Yo/QKDMZex5ihKpqLKRVGmSMewpl/PH2rnkV1NAPx66wlePtIRH2/1h6jwuQErpdgfjMTjLhNSLqNYN4qSqahyUZQp4gT0AV45arUZ3nmym/6BKLc/uIsfPlsfH28NhCi3lYvlFovEF1KOqlxyB+uNjRb0V5RMZVTlIiILJ7rNpMC2bHUiEhSRgL3tTxi7WkT2iUifiDwlIotmWj7lzCIUifLRn77C7lPdQ/Z39g4ql5ePtAPQNxDlV1tPEIrEhiyubPOHqPA6ysWyXJx05NFiLq7sLNyurPgxinI6MZblchQ4MsFtNrjNGOO1t1UAIlIO3A98CSgFtgL3zZJ8yhnC0bY+ntjbwjMH2obs7+wbiDfhciwXgLuesSyWxoT1L4luMZ/bhT8YHjfmAoNB/dFKxChKpjKWcjkfuMDebgZOAV8G3mBvXwZO2mOZwg3AbmPMr40xQeAOYIOIrJ5dsZRM5nh735jjp+xqxolVjcFKRV5e6UUETnb1U+FzU+hxcbLLmtcaCBGOxgiGo/hDkUHl4nERisRoty2fsRSHE3fRasbK6caon2pjzDbnsYj8K/DXxpjfJEz5o+2O+hRwT/pEHJV/EpFvAPuBLxpj6oB1wHZngjGmV0QO2/v3JR4sIrcCtwJUVVVRV1c3JSECgcCUj003mSpbJsl1sDPK17cE+fJFHipd/UnlevqE5b7aVd9AXd2g9XKytY+FhVl4c8A/AKWuMK5c2B2EIrfQHTL87rE6YlZpMNpPHqGuroFG+3xbdlje3B3btnA4R5LKZ8KW9XPs8H7qeg+n6mWnhEx6H4eTqbLNJbkmamtfAOxIsn8HsDF14kyYzwN7gAHgJuAhETkH8AKtw+Z2AyPqaxhj7gLuAti0aZPZvHnzlASpq6tjqsemm0yVLZPkatl6ArbsIFi8CC8NSeXa9th+2H2ISK6PzZsvje8PPvMYqxbX0kMH+5r8nLt8HuVeN7ufOsS7L1jMfz97hMVrz0UEeOYFLt+0gc2rK2nf1sAv9m4np7gSOMl1f3LlqEUoq/e9wAl/Jxdt3MDlKyrScxGmSCa9j8PJVNnmklwTzRY7Cnwiyf5PAMdSJs0EMcZsMcb4jTEhY8xPgeeB64EAMLzJRSHgn2kZldMDZ+X8toSYyXBOdVnWQ6JbLBozdPeHKSnIjbu7lpQX8M7z5nHT+Qt464ZawCr74jxHeUJAH6CxK0h+bvaY1Y0dt5iu0FdONyZqufw18ICIvBF4yd53IbAYK84x2xhAgN3AB52dIlIALLP3K8oI4srleCfvX5T8Bt7U0x+fG4nGcGVn0d0fxhgoyc+JK5fF5QUsq/DyjRvPji+wbOwO4smxlEfiOheA+rbAuIH6Ag3oK6cpE7JcjDF/AFZgZWIVAkX245XGmEfSJ95IRKRYRK4TEY+IuETk/cAVwKPAA8B6EblRRDxYSQc7jDH7xjqnMndxyrJ09YVp6jVJ5zTalkvMQFvAUhpO6ZfSglwqfR7AslwcivJy8ORk0dTdH1dgZV4rs2xdbRELSvNo7gmNm2LszXUC+qpclNOLcT+xIpIDPAd8wBjzd+kXaVxygK8Bq4EoVqD+HcaY/QAiciPwH8D/AFuwYjKKkpTWnhCVPjct/hAHu6Ijxo0xnOruZ2l5AfVtvTR291Nd5IlbJiX5uZy7sJilFQUsKsuPHyciVBd6aOoJUeiJUlqQG2/WVZSXw2N/dSX//Wx93FU2GnHLRZWLcpox7ifWGBMWkSVYrqdZxxjTipUmPdr4E1iKR1HGpTUQ4vwlpbxwqI1DnbER4119YYLhGOcsLKa+rTe+MNKxXEryc7lifhHXrasecWx1kYem7n4GIrmU21aLQ15uNp+8esW48i0szaPILeTlZI87V1EyiYkG9H8KfCydgijKbNDqtyyXy1ZUsK15sCSLg7PG5Ty786PT9KvLLrdfUjB6oL2mKI/G7iAnOvrHtVBG4+aLFvGNy/MQSZ6qrCiZykRt7QLg/SLyBmAb0Js4aIz5ZKoFU5R00zdgKZNKn4e3nzOPh7af4p4tx/nYFUvjcxxlsra2kNzsLBody6VvMOYyGlWFHho6+4F+vvSWtVOS0ZWdRZ5LFYty+jFRy2UN8CrQCSwFzkrY1qdHNEVJD72hCL/ccpzmnsEeK+csKGZNaRY/fK6eUGQw9nLKVi7zivOoLHTHWxo3dPbhc7vGdFfVFFmB/rPnF/HBi7XEnTK3mJDlYoy5Kt2CKMpM8ZttDXzld7v59BtWAoMpwtcvyeHb20L8cW8LbzqrBoDGrn5cWUK5101NkYcm23LZfaqHNbWFY7qrzppfRLnXzTdvPHvMtSyKciain3hlzuH0WXl4ZyNgtR4GWFOWjc/t4pmDg0UejrT1Ul3kITtLqCr00NQdJBoz7Gv0s652+HrdoZy3sIRXvng1a2rGnqcoZyITzm8UkauA9wILgSGOZmPMn6RYLkWZNtGYocdeRe9gjGGLrVz2NVmFGxzLxZUlXLK8jGcOtGGMIRw1PHewjettK2ZJeQGP7Gpie0MX/eEo62qLxpVBA/HKXGVClouIfAh4BKtG12as+l0lwHlYNb4UJeO495XjXPGtpwiGB2MoR9p6aQuEKMm3sryys4TS/EHlc8XKCk529VPf1ssrRzvwhyJcvaYSgKtWVxKNGf7zqUMArFWLRFFGZaJusc9g9U95LxAG/tYYcy7WQsVAuoRTlOmw/UQX/lAkXtoeBl1iH750CQDl3lyysgatiyvs4pDPHGjlib3N5LqyuGxFOQDnzC+m3Ovmib0t5GZnsaLKO1MvRVFOOyaqXJYCT9iPQ1jVh8FaCf+hFMukKCnhcKuVMd+V0I54y5EOyr1u3nP+AmDQJeawoDSfJeUF3PfKCR7d1cSly8rIt0uwZGUJ19hWzMpqb3zFvaIoI5not6OdwbL1JxlMPy4D8lItlKJMhr2NPTzwWsOI/YdbLaPaWfAIsO1YJ+cvLqGy0MOyigLmFY/8+H7q6hWc7OrnVHeQN6wduvL+DWurAFhXM368RVHmMhMN6D8LXAvsBH4FfNdeUHk18HiaZFOUCXHn04f5w64m3nHOvHgAvaN3IK5UOm3LJRiOcqKzjxvOmwfAjz90Pm7XyHUq7zh3Htetq+aVox1csqxsyNily8tZP6+Qa2wloyhKciaqXG4DPPbjfwIiwKVYiuZraZBLUSbMgeYAoUiMnv4IRXag3rFawGpHDHC0vRdjYGmF5dVdVFYw8mQ2ebnZXLFyZHMuT042v//Ly1MpvqKckUx0EWVHwuMY8M20SaQokyAaM3FF0uwPDiqXlkHl0mUH9A+3WDGYZRWjKxVFUVLDRFORfyAiN4lITboFUpTJcKy9l4GIVc3YqVgMluXidmWRl5NNV79ludTbSiix74qiKOlhom4xL/DPQK2IHALqnM0Y05ge0RRlfA4mWCgtdq0wsDLFlpQX4A9G4jGX+rZeaos88ewvRVHSx0Q7Ub7fGLMAq0/KP2NVSf4m0CAi2uVRmTUONvvjj5v9Qy2XZZVeivNz4oF9Z5+iKOlnson6h7EyxnZjdYCMAVNrVKEoKeBAc4B5xXn43K645WKMoaGzn0Wl+ZTk59LZN4AxhvrWXpaqS0xRZoSJxlw+KyL/B3QB9wCrgF8Cy40xS9InnqKMzYFmPyurvFY5fDvm0h+OEo0ZCvNyKM7PobsvTKs/RCAUUctFUWaIiTqfv4lVT+yrwN12q2FFmVUi0Rj1rb1cubKCUCRGi9+yXJxukgW52XHL5ZAdzF9arspFUWaCibrFrgX+G3g7cFxEdorI90TkBhEpG+dYRUkLp7qCDERjLKvwUlXoiVsufSGrUGWB22VZLv1hDjZbymVZpbrFFGUmmGhA/wljzO3GmMuAUuCvsTLI7gWa0ijfpBGRUhF5QER6ReSYiLxvtmVS0sPJLqu//fySPCp9blp6QhhjBi0Xt4vi/FxiBl6qb6c4P4fqQs9Yp1QUJUVMpp9LFVa5/c3AVcBKoBkrJTmT+D4wAFQB5wAPi8h2Y8zuWZVKSTmnbOVSW5xHZaGHgWiMrr4wvXG3mCteWv+Fw+2cNa9I+6soygwxIeUiInuwgvgtwNPAv2OtccmoNGQRKQBuBNYbYwLAcyLyO+AW4AuzKpySchzlUl3koarQSlps8YfoG3DcYtmURK1eLd394XE7RyqKkjrEGDP+JJE/IwOVyXBE5FzgBWNMXsK+zwBXGmPeOmzurcCtAFVVVRvvvffeKT1nIBDA683MIHGmypYquX68K8TrLVG++yf5HOiM8o9bgnxmk5veMPzX9hBfvyyP/ojhay9ZsZg/O9vNRbWj/546069XqslUuSBzZTvT5Lrqqqu2GWM2JR00xkxqw3I3ZU32uJnYgMuBpmH7PoalGEc9buPGjWaqPPXUU1M+Nt1kqmypkuvmH75k3va9Z40xxhxtC5hFn/+9+dUrx809W46ZRZ//vWno7DP1rdb+RZ//vTnU4p8RuVKNyjV5MlW2M00uYKsZ5b460XUuLhH5loj4sfq5LLb3f1NEPjFpdZc+AsBw30ch4E8yVznNOdXVT63dj6XSZwXqWwMhem23mDch5pKfm82SMaogK4qSWiaainwH8FbgZqxOlA4vk1mdKA8ALhFZkbBvA1ZFASUDuP/VBlr6YlM+PhozfP+pQ3T2DnCqKxhv9pWXm43blUV3QkA/351NoScHEVhTUziknbGiKOllotli7wU+bIx5WkQS7wy7sLLGMgJjTK+I3A/8g4h8FCtb7O3AJbMqmAJYLtjP/Ho7l81z8e5hY397/07W1hZyy0WLxjzH6yc6+edH99PVN0B/OBq3XACK8gbriOW6suJtiBeXFXDhktKUvhZFUcZmosqlFjg2yvGZVmL2E8CPsTLb2oE/N5qGnBFEY4aYgUNd0RFjD+84xYmOvhHKJRiOIkK8Y+ThVqsny6+2Wm2NE5VLcX4OXf0DuLIFr3vwY/nQX16G26X97hVlJpmoYtgNXAEcHbb/3cC2VAo0XYzV2Owdsy2HMpJw1MpMPBUwdPeF4429QpEoPcEIxzp6h8w3xnDLj7ZQWpDLD26xElLqbeXSbfdomZeoXPJy6e4Pk5/rosA92L44UdEoijIzTPRb9/fA/4jIAiAbeJeIrAbeB7w5XcIpZxYD0UGP6qsnOrlqVSUA7QGr38qpriDhaCzuzqo70MorRztZVJYfP+5wawBPThbBsHWu2uLBFfdF+Tmc6OjD58mhQHu2KMqsMtHyLw9hWSnXYpXZ/wqwAnirMeaJ9ImnnEmEE5XLsc7447aAlSMSjRlOdfUTikRp6g7ynScOAtDYHXTSyqlvDXD5igoqfW7crixKC3Lj5ynOs+qI9YYiFKi1oiizyrjfQBHJAb4OfN8Yc2X6RVLOVBKVy7YkygXgeEcfn/3NDl4+0gHA2ppC9jT20N47QFFeDsc7+rh2XTUrKr3sPNk9pJyL0xisdyBKUV7ODLwiRVFGY1zlYowJ22tZ/nMG5FHOYMIRy/rIc8HrJ7po6g5SXeShzT8Qn7O3sYetRzu4bl0VV6+pwud28ee/eJXGriA9/WHCUcPS8gLetWnBiPMX5+fSH47S2TtAbZEWqFSU2WSiKTSPAn+STkGUMx8n5rJ5QQ5ZItz4Xy9wtK2XVttyyckWHnztFDED779wEe/etID5JVa85WRXfzyYP1rDL8daaezuV7eYoswyE/0GPgn8o4icjZUdNiStxxhzf6oFU848HLfYsqIs/uIt5/OeH7zInU8fJi83G6/bRVWhmz2NPWQJnLuwGIAaO2Df2N0/ePwoDb+K7eyzcNRohpiizDIT/Qb+h/33k0nGDFYGmTLHiERj3PPycd59/oL4OpSxcJSDKwvWzyvirPlFHGj2U1ucR4XPzcLSfA639rKmphCfx1IUZQW55LqyaOwO0t0XpqwgN57CPJzEOEtiKrKiKDPPRLPFssbY9Fs8R3mxvp0v/XY3zx9qm9D8ROUCsKLSx8GWAG2BEOXeXBbZtb/OXzy4ml5EqC3ycKqrn+0NXayu8Y16/uK8wcyxfE1FVpRZRZctK1PmcIvVOrinPzKh+QN2QD/bzvBaXunFH4ywr8lPudeyXAA2LS4ZclxNUR67T/Wwr8nPJcvKRz1/cYJFo24xRZldVLkoU6a+zQq9+UMTUy7DLZfldmC+qy9MudfNZSvKOWdBMZcOUyA1xR6O2M918bKyUc+f6C7TgL6izC76DVSmzOFWy3LxB8MTmj/SLTYYmC/3ullZ5ePBv7h0xHG1RVaJF6/bxdnzikY9v8/tIjtLiMYMBbnqrVWU2UQtF2XKOKnBgeDkLJdse91jhc+Nz2P9vin35Y52WDxj7MIlpbiyR//Iikg8qK+Wi6LMLqpclCnRNxChsdtqH+yfoHIZsAtXuuy+KiISt17Kve5Rj3MqH4/lEnMoVuWiKBnBpJWLiBSLSGnilg7BlMzGsVoAAsNiLk4dsOGEI0PdYjAYdxlLuWxcVMJbN9Tytg2148rlxF00FVlRZpeJtjleJCKPiEgQq0dKq7212X+VMwinB/ZYOMH83OysIZbLvzy6n3fd+WLSY4a7xQBWVlmpxZW+0ZVLoSeH7733XCoLxy/pErdcNBVZUWaViX4DfwIUAx8GTmEtnFTOUK7/7nO85ewa/uKq5aPOqW8NIAKra3xDAvoHW/xsO95J/0CUvGFB9cGA/qB2eff5Cyjz5rKgNJ9U4MRcNBVZUWaXiX4DLwAuMsbsSqcwyuxjjOFgs58tR9z8xVVD9//vqyepKfJw6fJy6lt7qS3Ko8LrpqknGJ/XG4pijKVkzp5fPOTcgzGXwX2Fnhzeee78lMlfnG8lBuSrW0xRZpWJxlyOAKP7LZQzhv5wlEjMcKjZH99njOGfH93PZ369nX94aA8AOxq6WF3tw+txDXGLOWte9jX5GU7EsVxkxFDKOG9RCZsWlUyoHI2iKOljosrlU8A/icjofhLljMBRFKe6g/TaiuK+V07wn3WHWViaz/5mPy8cbuNoex9XrKzA53ENCegHbBfZ/iTKJR5zSWOO4ts21PKbP78kfU+gKMqEmOjX/LfAZmC/iPSJSE/ilj7xhiIidSISFJGAve0fNn61iOyzZXxKRBbNlGxnConxk8OtATp6B/jGH/Zx4ZJS7vrARgC++vu9AFy1qhKvOwd/MBxPAOgNRYHkysVxi2Wn0XJRFCUzmGjM5ba0SjE5bjPG/HD4ThEpB+4HPgo8BHwVuA+4aGbFO73pSXBxHWoJ8MstxwkEI3ztHetZXumlpsjD3sYellUUsLAsH5/HRThqCEVieHKy41ZMMrdYOBojNztrSPdIRVHOTCakXIwxP023ICngBmC3MebXACJyB9AmIquNMftmVbLTiMT4yZb6Dv731QZuvmgRK+yU4StXVnDvKye4alUlQHyFfSAUITc7i0Aogs/joi0Qoj0Qoixh/Uo4EiNHzRZFmRNMZRFltYgsTNzSIdgY/JOItInI8yKyOWH/OmC7848xphc4bO9XJkhPv+UWy83O4jevNhCJGd5/4eBbfM2aKuvvWuuvo1z8wQh9Ycsldt5Cq6rxcNdYOBojx6VFIRRlLjAhy0VEioDvAu8GkhWBmqnUnM8De4AB4CbgIRE5xxhzGPAyckFnN5C0AYiI3ArcClBVVUVdXd2UBAoEAlM+Nt1MRbZtJyzlMt8L9d2GpUVZnNy7jZNWmIVsY/jqpXkEj++k7jgcbbEsnaeff4kit2WVFEe7APjDC68x0DBYqfjYiRAmGiUQGMjIa5ap76XKNXkyVba5JNdEYy7/AmwA3oEV1/gwMA8ri+zTqRBEROqAK0cZft4Yc5kxZkvCvp+KyHuB64HvAQGgcNhxhcBI5z9gjLkLuAtg06ZNZvPmzVOSu66ujqkem26mItv+pw/D7n1sXr+I+ueP8JE/WcvmC0fPi3Afbuc7r77EynUbrBX0dU+zedNaHqrfTnHNIjZvXhmf+/vW7XgD7Xi9WRl5zTL1vVS5Jk+myjaX5JqocnkT8F5jzLMiEgW2GWPuE5FG4OPAb6YriDFm81QOAxwn/m7gg86AiBQAy+z9ygTxByNkZwlv2VDDrlPdvHWcel5xt1goQr4dzC/Ky6G0wE2rPzRkbjiqMRdFmStM1AFeDByzH3cDTnnaF4EZWVRgF8y8TkQ8IuISkfcDVwCP2lMeANaLyI0i4gG+DOzQYP7k6AmG8bpdnLewhF99/GIKPcn71Tskxlyc0vtedw4VvtGUi8ZcFGUuMNFv+mFgqf14L3CTWPmkNwAd6RAsCTnA1xgsmPmXwDuMMfsBjDGtwI3A14FO4EKsuIwyCfzBSFxhTASfrXwCwXA8DbnAnW0pl8BQ5TIQMapcFGWOMNG7yN3A2UAd8A3g91hrX7Kw4i5px1Ye548z5wlg9UzIc6biD4bHtVYScQpEBkKRuHLxuXOo8Lo53BIYMncwWyyaMnkVRclMJrrO5d8SHv9RRFYDm4CDxpid6RJOmXl6Jmm55LqycLussvtet5VpFrdc/CGMMfFFk9YiSo25KMpcYEp1yY0xx4HjKZZFyQB6+sPML5lc+Xufx4U/FKFwwLJIvB4XFT43A9EY3f3heKVijbkoytxhwt90EfmEiOy263Yttfd9QUTenT7xlHQTDEdp6OyL/+8PRijMm9xvDp8nB38wgj9ordJ3uyzLBRgS1B+IGlyqXBRlTjDRTpR/BdyOtS4k0a9xksyqO6ZMkp88f5Q3/fuz8XL4k425gBV3sQL64Xh74QrvSOUSjqhbTFHmChP9GflnwMeMMd8BEhumv4qWVzmtae4J4g9FaOwOYoyJ1wabDF631dOlNxTFax8bt1wSMsbULaYoc4eJftMXAcm6UIaBvNSJo8w0TobX8Y4+egeixAyTVi4lBTm09w7gD0biveuTucVUuSjK3GGi3/R64Lwk+6/HqvWlnAaEIlE+/vOtfOTuV/jJ80cA4g3BjrX3xYtWTtYttrzCy7H2Xjp6Q3HFVOhxkevK4nhHH++68wWeOdBKOKrrXBRlrjCZ2mL/ISL5WDGXi0XkFuBzWHXGlNOAbUc7eXR3M163i5fq2/l/ly4ZYrk45fZ9k1QuK6p8xAzsaezh4qVW8QYRocLr5oFXT+IPRXj1eKeVipzOHseKomQME/oZaYz5CXAH8I9APvBzrKZcnzTG3Jc26ZSUsuVIB1kC7960gN6BKNGYiVsuJzr64l0oJ+sWW2n3egmGYxS4B4+t8Lnx2+f3ByPqFlOUOcSEv+nGmP82xiwCKoFqY8wCY8yP0ieaMhmMMbx2vHPMOVuOtLO2tpDaYg8AATsID3CsozfBcpmccllSXoArS0Yc68RdnOdSt5iizB0m/U03xrQZY1rSIYwydR7f08w7//MFdjR0JR0PRaK8dryLCxaXxWMq/lCY3gHbLdbeR49tuRTmTc4tluvKYnF5AUA8oA9QU+QhNzuLcq+bQCjCgFouijJnGPMnqoj8biInMca8LTXiKFPl2YNtgBU7OXt+8YjxHQ3dhCIxLlxaSixmAOz04QhZYpV9aejsByZvuQCsqvJxqCUQT0UG+IurlvP2c2r5h9/vpScY1vIvijKHGO8u8hasUvt16RdFmQ4v1bcD0NQdHLLfGMP/vHSMP+6zjM3zF5ey+1Q3QHxtyuLyAupbe/nfbQ3k52ZTnJes2ejYrKjyws7BQpYAVYUeqgo9FHpcdPeHMQa1XBRljjCecvkX4Gasvik/Ae42xjSkXSplUrT6Qxy0KxAPVy77m/186bdWv7TLV5RTWpAbzwbr6A0xEI2xtqaQ+tZe6tt6+fwbV5M7hT73TlA/Ubk4eN0u6lt7AayqyGbSp1cU5TRjzLuIMeZzwALgr7GrIIvIIyLypyIyOce8kjYcqyU7S2jqGapc2gMDANzzsYv4+UcuBAbdXo4iWlNjdYdeWlHARy5bMiUZzp5fRE62sLB0ZNFLn8dFZ58lh1ouijI3GNe5boyJAr8Dfici1cAHsJp2/aeILDXGBMY8gTIlWnqC7GnsYfOqynHnvljfjtftYnW1j+ZhyqWj17qpl3kHXV2Ocmm051b43Nz+5jVctLRsSlYLwPySfLbe/gYKk8RrvO4c+uyKybnZMrSAkKIoZySTvZMUYLU89gIB1MGRNn724jE++tOt8YKSY/HykQ42LS5hfkle3HIxxnprHIuhJD9Bubgto7PZtly8bhcfvXwp6+cVTUvmoryceO+WRBITBNRyUZS5wbjfdBHJE5EPisgzwE6sOmMfNMYsNcb0pl3COUpH3wCRmIlbHqPR3RfmUEuATYtKqCry0Nwd4nBrgHVfeZQT/lj8+OL8QS+mJycLV5bQaCuXgiRxklSiykVR5h7jpSLfBbwHOAj8CHibMaZrBuSa8zgLGtsCA1QWekad97q9ruXchSUcaPYzEI3x0PZT9A1EOd4TZSA2QKHHNeSmLiL4PK64leO1y+SniyHKZYpuN0VRTi/G+8n6UayOk43Am4A3JXN76DqX1OMUkWxLKFmfjNeOdyJiBdSdYx7d3QxAV8gQzApTWjAytdjnyZkxy8XrHrSadJ2LoswNxvsZ+TPgKaANaB9jSwkicpuIbBWRkIjcnWT8ahHZZ3fDfEpEFiWMiYh8U0Ta7e1bkkwTniY4db7GVy5drKry4fPkUFVkWTh7G3sAS7l09g5QkkS5eN0uBiJWPCdxVX06ULeYosw9xryrGGM+NENyOJzCykS7jmF9YkSkHLgfy5p6CPgqcB9wkT3lVuAdwAasRIPHsVoF3DkDcqecnrhbbHTlEotZ9cTefHYNANXD3GedQUPfwAA1RSPdaok3/GRrU1KJd5hyGT9FQVGU052M+hlpjLnfGPMgya2hG4DdxphfG2OCWFWaN4jIanv8g8C3jTENxpiTwLeBD6Vf6vQw6BYbPaBf39ZLTzDCuQtKACul2LHV3K4sy3LpS265JJbVT7dbLDE92aVuMUWZE6T3rpJa1gHbnX+MMb0ictjev2/4uP141BbMInIrlrVDVVUVdXV1UxIqEAhM+dix6OqzLJZdh45TV9ecdM5LjZZ1E2o6SF3dYQAKc4XukGF1idDQE6EnHCTQ3jxCxr5u6/wugReeeybl8ifSGRy0VXbv2E5NTn9artl0Sdd7OV1UrsmTqbLNJblOJ+XiBVqH7esGfAnj3cPGvCIixln0kYAx5i7gLoBNmzaZzZs3T0mouro6pnrsaAxEYgz84REAXN4SNm++IOm8Q8/Ww/a9vPXqyyi217Es2vUcvaEIF66pYuez9cQMbFi9jM2blw059o/du3jh1DF8eTkpl384gVAE6h4F4IJNG+k8/Hran3MqpOO9TAUq1+TJVNnmklwzplxEpA64cpTh540xl41zigBQOGxfIeAfZbwQCCRTLJmOE8wHaPOPHnNpCwyQky0UJZTI/8y1qzDAwWY/dvFjSgtGVupxYi7pdokBFORmI4IWrlSUOcSMfdONMZuNMTLKNp5iAdiNFawHQEQKgGX2/hHj9uPdzADH2nvjvVD+6ZG9PPjayWmdz1nj4nZljRnQbw+EKCtwD1kVf8XKCq5cWUFVQnA/cXW+gxNzSXcwH6x1Nc7zaJtjRZkbZNTPSBFxiYgHyAayRcQjIs7d7wFgvYjcaM/5MrDDGLPPHv8Z8DciMk9EaoFPA3fPhNw33fUS33vyIAC/3HKcx/cmj5FMFEdRLSkvoL13IN5/ZThtgRDlvuTl8ROVS7J1Ls7NfiYsFyDeoEwtF0WZG2TaN/12oB/4Alap/357H8aYVuBG4OtAJ3AhcFPCsT/ASlHeCewCHrb3pZ2O3gGOtPXSG4rgD0bilsdUcY5fWlFANGbo6g8nndcWGKCswJ10rDKhxXDybLGZVS6OMlPloihzg4wK6Btj7sBKMR5t/Alg9ShjBvicvc0YxhhCkRiN3cF4OZVAMLkyADjR0cf8krykBR4dnDTkpeVewLJQklkf7YFQvI/KcCoLB5VLaRK3mGNJFOSmt/SLg6PMVLkoytxAv+nTJGxn2TZ2B+NVhgOh5JbLrpPdXP6tp/i3Jw6Oec5EtxgkD+obY2gLDIzqFsvPdZHngiyBwrzZDejD4ELKXFUuijIn0G/6NHGUS0fvAEfb+wAIjOIW299kJbZ998mD3Pvy8VHP6bjFllTYysWubHykrZdu26rpCUYYiMao8CZ3iwGUuIXi/Fyys0ZaSc7NfiYC+jCYQJCjAX1FmROocpkmA9HBYPvrJzoB8I9iuRxt7yVL4LLl5dzx0G5OdfUnndfTH0YEFpcNWi69oQhv/d5z8cQBJ4sssQnYcIo9MqTUfiLOzb4gzRWRHTTmoihzC/2mT5NwQqGs1453AZZbLNnymqPtfcwvyecbN56FMfCNR/aNmAOWVeJzuyjOy6EgN5sdDV38YVcTgVCEhk5LITnti8vHsFzesjSXz167KulYUV4OWZI8TTkdOCVgXEmsKEVRzjwyKqB/OhKODj4+2GJ1fDYG+gaiI+IZx9p7WVSWz/ySfD5+5TK+++RBPnjJYjYuKhkyrycYpjAvh6ws4T3nL+SnLx5lf7N17lbbYnEsl7GUy9qybDafVZN0zOt28YuPXsS6ecPXpaaHG86bT2WhZ8xEBkVRzhzUcpkm4VHWoAwP6htjONLWG3d1/dmVSynJz+G/7JpgifT0R+Juq49dsYQsGSyj3+K3kgYmolzG4+JlZfGssXSzqtrHRy5bMiPPpSjK7KPKZZoMDKsf76T2Dl/r0tUXxh+MsNjOAMvPdXHLxYt5Ym8zh1sDQ+b6g+G4G6mmKI8bzp0PwDVrKmn1h+KZYiJQMkpMRVEUZTZR5TJNBqJD/19eaa1NGW65HGnvBWBxWX583wcuXoTblcUPn60fMrcnGBmSPvzFt6zhlx+9kAuWlBIMxwiEItbal/xcXBogVxQlA9E70zRx3GLOivjlldaiRv+whZTHbOWyyHaLgeXSum5dNU/ubRkyt6c/PKSZV6Enh0uWl1NhP0eLP0SbPzQtl5iiKEo6UeUyTZyAvuPuWlFlWy7D3GJH2/oQgQWlQxpssri8gNZAiHB00L9mucVGursqfVa9sFZ/aMy6YoqiKLONKpdp4lguS2yLZHmFpVyGr3U51t5LbVEebtfQdSU1RR6MsawRsFoX+0ORpKvqHcul1R+iRS0XRVEyGFUu08SJuVy9ppILlpRy1vwiYKTlsr85wNKKguGHx/veN3UHMcbwrUf3YwwsKs0fMddZjV/f2ktDZ/+odcUURVFmG13nMk2cbLELl5Zx7bpqIrZ7KzGgHwhF2N/Uwxv+ZMWI46uLBpXL/7x0jDufPszNFy3knefOGzG3OD+HnGyh7oAVo1lbOzNrVBRFUSaLKpdp4rjF3C7LCHRlZ5GXkz1EuWw/0UXMMGKxJFhuMYDG7n5equ9gaUUBX337+qSLDUWECq+b1090AbC+tijVL0dRFCUlqFtsmjgBfUe5gFUUMnGdy7ZjnYjAOQuKRxxflJeDJyeLpu4gh1sDrK72jbmKvcLnxhgrO63CpzEXRVEyE1Uu02QgZimWRIXgc7uGWC7bjnWystI3pNe9g4hQU5TH0fY+jrX3xhMCRqPCzhhbP0+tFkVRMhdVLtMkHDV4coZmgHk9rnjDsFjM8OrxTs5L4hJzqC708PKRdmIGllWOp1wsa2WdxlsURclgVLlMk4EYeHKGXkZvguVysCWAPxhJGm9xqCny0GO70VZUjp0BNqhc1HJRFCVzUeUyTcIxM2Ltitc9GHN5ZFcjIlaRyNGosoP6IiRNV05kUWk+2VnC2fNVuSiKkrlottg0CUeTWC52QD8WM/xmWwOXLCtjXnHeKGcYzBhbUJI/wsU2nLedU8uGBUXUjnE+RVGU2SajLBcRuU1EtopISETuHja2WESMiAQSti8ljIuIfFNE2u3tWzIDzUOsgP5QheAE9F860k5DZz/v3rRgzHM4CylXjBNvAauT4/JxXGeKoiizTaZZLqeArwHXAaP9NC82xiTrI3wr8A5gA2CAx4F64M7UizlIOGrwDrNcfJ4cAqEIv3rlBD6Pi+vWVY95jpoi66Uun4ByURRFOR3IKMvFGHO/MeZBoH0Kh38Q+LYxpsEYcxL4NvChFIqXlHCMpNli0Zjht9tP8acb54/r6lpUnk+hx8WFS0vTKaqiKMqMIcl6vc82IvI1YL4x5kMJ+xYDR7CsG8cy+awxps0e7wauNcZssf/fBDxljEnqQxKRW7GsHaqqqjbee++9U5L19mcDlOe7+KuNnvi+Px4P87M9AxTkwDcvz8ebO753zhiT8hbAgUAArzfzrCGVa3KoXJMnU2U70+S66qqrthljNiUdNMZk3IblGrt72D4vsAnLlVcF/AZ4NGE8CqxO+H8FlhKS8Z5v48aNZqpc9A8Pm0/8YtuQfb97/aRZ9Pnfm5+/eHTK500FTz311Kw+/2ioXJND5Zo8mSrbmSYXsNWMcl+dMbeYiNTZAflk23PjHW+MCRhjthpjIsaYZuA24FoRcVYTBoDElYWFQMC+AGljIAqeYQH9q9dU8t33nst7L1iYzqdWFEXJWGYsoG+M2ZzqU9p/HV/Sbqxg/sv2/xvsfWklHAP3sIB+fq6Lt22oTfdTK4qiZCwZFdAXEZeIeIBsIFtEPCLisscuFJFVIpIlImXAd4E6Y0y3ffjPgL8RkXkiUgt8Grg73TKHY2aE5aIoijLXySjlAtwO9ANfAG62H99ujy0F/gD4gV1ACHhvwrE/AB4CdtrjD9v70ko4OtJyURRFmetk1DoXY8wdwB2jjN0D3DPGsQb4nL3NCNGYIWJGxlwURVHmOvqTexqEIlYzl+HlXxRFUeY6elecBqGw1dI4sVGYoiiKosplWgTjlou6xRRFURJR5TINgo7lom4xRVGUIehdcRrEYy4a0FcURRmCKpdp4Fgu6hZTFEUZiiqXaRAKW5aLBvQVRVGGonfFaRCMODEXtVwURVESUeUyDYJquSiKoiRF74rTIBTRmIuiKEoyVLlMA8dy0RX6iqIoQ9G74jRwLBe3piIriqIMQZXLNAip5aIoipIUvStOg4Wl+WyqytaYi6IoyjAyquT+6ca166rJbfWQk606WlEUJRG9KyqKoigpR5WLoiiKknJUuSiKoigpR5WLoiiKknJUuSiKoigpR5WLoiiKknJUuSiKoigpR5WLoiiKknLEGDPbMsw6ItIKHJvi4eVAWwrFSSWZKpvKNTlUrsmTqbKdaXItMsZUJBtQ5TJNRGSrMWbTbMuRjEyVTeWaHCrX5MlU2eaSXOoWUxRFUVKOKhdFURQl5ahymT53zbYAY5Cpsqlck0PlmjyZKtuckUtjLoqiKErKUctFURRFSTmqXBRFUZSUo8pFURRFSTmqXKaIiJSKyAMi0isix0TkfbMkh1tEfmTL4BeR10TkTfbYYhExIhJI2L40g7LViUgw4bn3J4xdLSL7RKRPRJ4SkUUzJFNg2BYVke/ZYzN6vUTkNhHZKiIhEbl72Nio10csviki7fb2LRGRdMslIheJyOMi0iEirSLyaxGpSRi/Q0TCw67f0lTJNY5sY753s3jN3j9Mpj5bzo32eFqv2Vj3B3s8bZ8zVS5T5/vAAFAFvB/4LxFZNwtyuIATwJVAEfAl4FcisjhhTrExxmtvX51h+W5LeO5VACJSDtxvy1oKbAXumwlhEmTxYr13/cCvh02bqet1Cvga8OPEnRO4PrcC7wA2AGcDbwE+nm65gBKsrKLFwCLAD/xk2Jz7Eq+xMaY+hXKNJZvDaO/drFwzY8wvhn3mPgHUA68mTEvnNRv1/pD2z5kxRrdJbkABlmJZmbDv58A3Zls2W5YdwI1YNwEDuGZJjjrgo0n23wq8MOx69gOrZ1i+D2J90Z2syVm5Xlg3pbsnen2AF4BbE8Y/AryUbrmSjJ8H+BP+vwP4n1m6ZmO+dxl0zZ4CvjIb1yzhOZ37Q1o/Z2q5TI2VQNQYcyBh33ZgNiyXIYhIFZZ8uxN2HxORBhH5if1rZSb5JxFpE5HnRWSzvW8d1vUCwBjTCxxm5q/fB4GfGfubk8BsXi8Y//oMGWf2PntXMPRzBvBW2222W0T+fBZkGu29m/VrZrucrgB+Nmxoxq7ZsPtDWj9nqlymhhfoHravG/DNgixxRCQH+AXwU2PMPqxCdOdjuTA2Ysn3ixkU6fPAUmAeljvlIRFZRgZcPxFZiOUq+GnC7tm+Xg7jXZ/h492AN5UxhPEQkbOBLwOfTdj9K2ANUAF8DPiyiLx3hkQa772b9WsGfAB41hhzJGHfjF2zJPeHtH7OVLlMjQBQOGxfIZYPelYQkSws19wAcBuAMSZgjNlqjIkYY5rt/deKyHDZ04IxZosxxm+MCRljfgo8D1xPZly/DwDPJX7RZ/t6JTDe9Rk+XggEklhgaUFElgOPAJ8yxjzr7DfG7DHGnDLGRI0xLwDfAf50JmSawHs3q9fM5gMM/TEzY9cs2f2BNH/OVLlMjQOAS0RWJOzbwEgXwYxg/5L4EVaA+kZjTHiUqc6HYiZ/rQ1/fsG6ThucnSJSACxjZq/fiC96Embreo13fYaMM4OfPdu18wTwVWPMz8eZ7rzfs8Hw927WrhmAiFwK1AK/GWdqyq/ZGPeH9H7OZjKQdCZtwL3APVhBsEuxTMZ1syTLncBLgHfY/guBVVg/IsqwMkGemiGZioHrAA9Wxsr7gV5bngr7et1oj3+TNARXx5DtElsW32xeL/u6eIB/wvpV6VyrMa8P8GfAXix3Y639hf+zGZBrHpZP/rOjHPd2rIwyAS4ATgIfnKFrNuZ7N1vXLGH8Lqz43mxcs9HuD2n9nKXlSzMXNqzUvQftm9Rx4H2zJMcirF87QSwz1tneD7wXOGLL2IgVSKyeIbkqgFewTOwu+8P9hoTxa4B9WNkpdcDiGbxmPwB+nmT/jF4vrEwhM2y7Y7zrY9+IvgV02Nu3sDPe0ikX8BX7ceLnLJBw3D1Au71/H/DJmbpm4713s3XN7DGP/R24Oslxab1mY90f0v0508KViqIoSsrRmIuiKIqSclS5KIqiKClHlYuiKIqSclS5KIqiKClHlYuiKIqSclS5KIqiKClHlYuizBJ2X4+0lUcRkU32cyxO13MoymioclGUKSAid9s37uHbS5M4TQ3wULpkVJTZxDXbAijKacwTwC3D9g1M9GBjTFNqxVGUzEEtF0WZOiFjTNOwrQPiLq/bRORhu4XsMRG5OfHg4W4xEfmyPS8kIk0i8rOEMbeI/LuINIvVOvolEbls2PneaLesDYrIs1h9Oxg25xIRedqW6aSI/NcsVH1W5gCqXBQlffw98DvgHOzChSKyKdlEEbkR+AxWG9wVWC1lX06Y8i3gPcCHgXOBncAfxO5hLyILsGrdPW4/3/fsYxKf4yzgMVumDcAN9tzRWgYrypTR2mKKMgVE5G7gZqyCgIl83xjzeRExwA+NMR9LOOYJoMkYc7P9vwHeZYz5jYj8DVZ/8vVmWMsEuxR6J1bL6J/Z+7KxWj/cY4y5XUT+EasPyCrjVB0UuR34KrDEGHPUtoTCxpiPJJz7HOA1oMoY05KSi6MoaMxFUabDM1h9yBPpSnj84rCxF4E3j3KuXwOfAo6IyKPAH4DfGWNCWD02crCarQFgjImKyIvAWnvXGqxy6Ym/Foc//0ZguYi8J2Gf0ztkGaDKRUkZqlwUZer0GWMOpeJExpgTIrIKuBqrDPq3ga+IyIUMKoBkbobJNDTLAn4I/FuSsZOTk1hRxkZjLoqSPi5K8v/e0SYbY4LGmIeNMX+N1Q9+HVYjukNYWWjxAL7tFrsY2GPv2gNcOKy/+fDnfxWrod2hJFv/FF6fooyKWi6KMnXcIlI9bF/UGNNqP75BRF7BasL0p1hWyYXJTiQiH8L6Pm7Baub0HiAMHDTG9IrIfwHfEJE2rKZYf43VtvY/7VPcCXwa+HcR+U/gLKxOgol8E3hJRO7EapjmB1YDbzXGfHzyL19RRkeVi6JMnWuwuh4mchKYbz++A6uF7HeBVuD/GWNeGeVcXcDngX/Biq/sAW4wxhyxxz9v//0JVgvp14A3GmMaAYwxx0XkBuBfsRIDtgFfAP7HeQJjzA4RuQL4GvA0kA3UAw9M7mUryvhotpiipIHETLDZlkVRZgONuSiKoigpR5WLoiiKknLULaYoiqKkHLVcFEVRlJSjykVRFEVJOapcFEVRlJSjykVRFEVJOapcFEVRlJTz/wGFnLfNOWo9PwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.plot(mean_rewards)\n",
        "plt.xlabel(\"Episode\")\n",
        "plt.ylabel(\"Mean reward\")\n",
        "plt.grid()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3VgVDzDfMM99"
      },
      "source": [
        "결과를 확인해 보죠!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IS0TjvFzMM99"
      },
      "outputs": [],
      "source": [
        "def lander_render_policy_net(model, n_max_steps=500, seed=42):\n",
        "    frames = []\n",
        "    env = gym.make(\"LunarLander-v2\")\n",
        "    env.seed(seed)\n",
        "    tf.random.set_seed(seed)\n",
        "    np.random.seed(seed)\n",
        "    obs = env.reset()\n",
        "    for step in range(n_max_steps):\n",
        "        frames.append(env.render(mode=\"rgb_array\"))\n",
        "        probas = model(obs[np.newaxis])\n",
        "        logits = tf.math.log(probas + keras.backend.epsilon())\n",
        "        action = tf.random.categorical(logits, num_samples=1)\n",
        "        obs, reward, done, info = env.step(action[0, 0].numpy())\n",
        "        if done:\n",
        "            break\n",
        "    env.close()\n",
        "    return frames"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pzfWaUVGMM-B"
      },
      "outputs": [],
      "source": [
        "frames = lander_render_policy_net(model, seed=42)\n",
        "plot_animation(frames)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RQW9sCXpMM-B"
      },
      "source": [
        "꽤 괜찮군요. 더 오래 훈련하거나 하이퍼파라미터를 튜닝하여 200을 넘을 수 있는지 확인해 보세요."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6xoce5bUMM-B"
      },
      "source": [
        "## 9.\n",
        "_연습문제: 알고리즘에 상관없이 TF-Agents를 사용해 SpaceInvaders-v4 환경에서 사람을 능가하는 에이전트를 훈련해보세요._"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S0Zm1na_MM-B"
      },
      "source": [
        "`\"Breakout-v4\"`를 `\"SpaceInvaders-v4\"`로 바꾸고 [TF Agents를 사용해 브레이크아웃 게임하기](#TF-Agents%EB%A5%BC-%EC%82%AC%EC%9A%A9%ED%95%B4-%EB%B8%8C%EB%A0%88%EC%9D%B4%ED%81%AC%EC%95%84%EC%9B%83-%EA%B2%8C%EC%9E%84%ED%95%98%EA%B8%B0) 절에 있는 단계를 따라해 보세요. 하지만 몇 가지를 바꾸어야 합니다. 예를 들어 스페이스 인베이더 게임은 게임을 시작할 때 FIRE 버튼을 누를 필요가 없습니다. 대신 플레이어의 레이저 캐논이 몇 초간 깜빡거린 다음 자동으로 게임이 시작됩니다. 성능을 높이려면 에피소드를 시작할 때와 죽을 때마다 깜빡임 단계(약 40 스텝 동안 지속됩니다)를 건너 뛸 수 있습니다. 사실 이 단계에서는 아무것도 할 수 없고 아무것도 움직이지 않습니다. 건너 뛰는 방법은 `AtariPreprocessingWithAutoFire` 래퍼 대신에 다음과 같은 사용자 정의 환경 래퍼를 사용하는 것입니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i9vUt5WJMM-B"
      },
      "outputs": [],
      "source": [
        "class AtariPreprocessingWithSkipStart(AtariPreprocessing):\n",
        "    def skip_frames(self, num_skip):\n",
        "        for _ in range(num_skip):\n",
        "          super().step(0) # NOOP for num_skip steps\n",
        "    def reset(self, **kwargs):\n",
        "        obs = super().reset(**kwargs)\n",
        "        self.skip_frames(40)\n",
        "        return obs\n",
        "    def step(self, action):\n",
        "        lives_before_action = self.ale.lives()\n",
        "        obs, rewards, done, info = super().step(action)\n",
        "        if self.ale.lives() < lives_before_action and not done:\n",
        "            self.skip_frames(40)\n",
        "        return obs, rewards, done, info"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ncMSSzA6MM-B"
      },
      "source": [
        "또한 전처리된 이미지가 게임 플레이에 관한 충분한 정보를 담고 있는지 항상 확인해야 합니다. 예를 들어, 낮은 해상도에도 불구하고 레이저 캐논과 에일리언에서 발사된 총알은 항상 보여야 합니다. 이 경우에 브레이크아웃에서 수행했던 전처리가 스페이스 인베이더에도 잘 맞습니다. 하지만 다른 게임에서는 항상 확인해봐야 합니다. 이를 위해 에이전트가 랜덤하게 플레이하게 잠시 놔두고 전처리된 프레임을 기롭한 다음 애니메이션을 플레이하여 게임 플레이가 잘 보이는지 확인하세요.\n",
        "\n",
        "좋은 성능을 얻으려면 에이전트를 꽤 오랜 시간 동안 훈련해야 합니다. 안타깝게도 DQN 알고리즘은 스페이스 인베이더에서 사람을 뛰어 넘는 수준을 달성할 수 없습니다. 사람은 이 게임에서 효율적인 장기 전략을 학습할 수 있지만 DQN은 매우 짧은 전략만 학습할 수 있습니다. 하지만 지난 몇 년간 많은 발전이 있었습니다. 이제는 많은 RL 알고리즘이 이 게임에서 전문가의 수준을 뛰어 넘을 수 있습니다. [State-of-the-Art for Space Invaders on paperswithcode.com](https://paperswithcode.com/sota/atari-games-on-atari-2600-space-invaders)를 참고하세요."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GdCKaaQGMM-B"
      },
      "source": [
        "## 10.\n",
        "_연습문제: 10만 원 정도 여유가 있다면 라즈베리 파이 3와 저렴한 로보틱스 구성품을 구입해 텐서플로를 설치하고 실행할 수 있습니다! 예를 들어 루카스 비월드의 재미있는 [포스트](https://homl.info/2)를 참고하거나, GoPiGo42나 BrickPi43를 둘러보세요. 간단한 작업부터 시작해보세요. 예를 들어 (조도 센서가 있다면) 로봇이 밝은 쪽으로 회전하거나 (초음파 센서가 있다면) 가까운 물체가 있는 쪽으로 움직이도록 해보세요. 그다음 딥러닝을 사용해보세요. 예를 들어 로봇에 카메라가 있다면 객체 탐지 알고리즘을 구현해 사람을 감지하고 가까이 다가가게 만들 수 있습니다. 강화 학습을 사용해 목표를 달성하기 위해 모터 사용법을 스스로 학습할 수도 있습니다._"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MHUiq7VhMM-C"
      },
      "source": [
        "이제 여러분 차례입니다. 도전적이고 창의적으로, 무엇보다도 인내심을 가지고 한 발씩 나아가세요. 여러분은 할 수 있습니다!"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}