{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"collapsed_sections": [
"1sJaj1WA_zdK",
"gAfP6ktBIAiG",
"zcuSyuzjKzoW",
"DjiwZ8a0K7mJ",
"XgykXOBHLLFK",
"K11Vd_W5LfZC",
"QY4Xiy9iLOVO",
"zyLopENcLSEq",
"LtSsZNJsLVk4",
"l3Jn6qx_MJd9",
"UhZjSRlSML0a",
"Yq8ZgeT7ND_4",
"vDylIggXBD6I",
"AkNeNzq1ukIB",
"10RO7OsDvF1A",
"iCP3xTLXvwt2",
"gfoT5fS2wFom",
"vvpEpTuOwbCn",
"7KPPO4U2PebH",
"2aa993e5",
"e64aabb0-25ef-455f-9494-b6f4ca3ecfc9",
"2bb2c4bf",
"c851ab41-4a7b-4df1-8140-48623daeae99",
"1-o680EtaRVN",
"BVYKFueVaUQk",
"9913bc18-8a26-4a1b-8abc-3bc8e672f191",
"27287f0e-56f0-458f-8c2e-9124a9739d48",
"fe958104"
]
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"source": [
"# **Seminar 5: Parameter-efficient fine-tuning (PEFT)**"
],
"metadata": {
"id": "XN_549l_BP-q"
}
},
{
"cell_type": "markdown",
"source": [
"## **1. Introduction**"
],
"metadata": {
"id": "1sJaj1WA_zdK"
}
},
{
"cell_type": "markdown",
"source": [
"\n"
],
"metadata": {
"id": "ZIRBQ3BDAGEk"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "o_sLj9DiBPg0"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "ET-RMQlqBWf1"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "M7LtOi16BZhb"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "eXaCVPHfCR1m"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "blRAs8UwBuSf"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "p_sN8qJaCaHB"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "j07kaHsRBsgc"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "lNfdOPW2B6Oh"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "5hpAFIF9CAd7"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "lOCjE-9QCFs8"
}
},
{
"cell_type": "markdown",
"source": [
"## **2.1 The Basics: Training an Adapter**\n",
"\n",
"In this notebook, we train an adapter for a **RoBERTa** ([Liu et al., 2019](https://arxiv.org/pdf/1907.11692.pdf)) model for sequence classification on a **sentiment analysis** task using the _[Adapters](https://github.com/Adapter-Hub/adapters)_ library and Hugging Face's _Transformers_ library.\n",
"\n",
"We train a **[bottleneck adapter](https://docs.adapterhub.ml/methods.html#bottleneck-adapters)** on top of a pre-trained model here. For training, we use the [movie review dataset by Pang and Lee (2005)](http://www.cs.cornell.edu/people/pabo/movie-review-data/). It contains movie reviews from Rotten Tomatoes which are either classified as positive or negative. We download the dataset via Hugging Face's [_Datasets_](https://github.com/huggingface/datasets) library.\n",
"\n",
"\n",
" \n",
" \n",
" Step \n",
" Training Loss \n",
" \n",
" \n",
" \n",
" \n",
" 200 \n",
" 0.491800 \n",
" \n",
" \n",
" 400 \n",
" 0.306300 \n",
" \n",
" \n",
" 600 \n",
" 0.292900 \n",
" \n",
" \n",
" 800 \n",
" 0.263600 \n",
" \n",
" \n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"text/html": [
"\n",
" \n",
" \n",
"
\n",
" [1602/1602 07:24, Epoch 6/6]\n",
"
\n",
" \n",
" \n",
" Step \n",
" Training Loss \n",
" \n",
" \n",
" \n",
" \n",
" 200 \n",
" 0.491800 \n",
" \n",
" \n",
" 400 \n",
" 0.306300 \n",
" \n",
" \n",
" 600 \n",
" 0.292900 \n",
" \n",
" \n",
" 800 \n",
" 0.263600 \n",
" \n",
" \n",
" 1000 \n",
" 0.251200 \n",
" \n",
" \n",
" 1200 \n",
" 0.234400 \n",
" \n",
" \n",
" 1400 \n",
" 0.236400 \n",
" \n",
" \n",
" 1600 \n",
" 0.220600 \n",
" \n",
" \n",
"
"
]
},
"metadata": {}
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"TrainOutput(global_step=1602, training_loss=0.28717051883612976, metrics={'train_runtime': 447.0915, 'train_samples_per_second': 114.473, 'train_steps_per_second': 3.583, 'total_flos': 2156331556195200.0, 'train_loss': 0.28717051883612976, 'epoch': 6.0})"
]
},
"metadata": {},
"execution_count": 10
}
],
"source": [
"trainer.train()"
]
},
{
"cell_type": "markdown",
"source": [
"#### **Loss**"
],
"metadata": {
"id": "l3Jn6qx_MJd9"
}
},
{
"cell_type": "code",
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"\n",
"logs = trainer.state.log_history\n",
"\n",
"train_steps = []\n",
"train_loss = []\n",
"eval_steps = []\n",
"eval_loss = []\n",
"\n",
"for entry in logs:\n",
" if \"loss\" in entry and \"epoch\" in entry:\n",
" train_steps.append(entry[\"step\"])\n",
" train_loss.append(entry[\"loss\"])\n",
"\n",
"plt.figure(figsize=(6, 3))\n",
"plt.plot(train_steps, train_loss, label=\"Train Loss\", marker='o', markersize=4, color='blue')\n",
"plt.xlabel(\"Step\")\n",
"plt.ylabel(\"Loss\")\n",
"plt.title(\"Training and Evaluation Loss\")\n",
"plt.legend()\n",
"plt.show()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 333
},
"id": "wi18LAHEFwR3",
"outputId": "a7d1980f-8ca2-4bbc-83dc-b449ffd1512e"
},
"execution_count": null,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiAAAAE8CAYAAAD5dQphAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAATyxJREFUeJzt3XlYVGX7B/DvADLDIosiqyiICyoqCopopiWKZu4lmgmuvblU5pa+JW4VastruWFWLplpJmKZuZGYluGKK+EKuAFqAgIKyjy/P85vBkYGRJY5LN/PdZ1rZp6zcD84ztw821EIIQSIiIiIDMhI7gCIiIio5mECQkRERAbHBISIiIgMjgkIERERGRwTECIiIjI4JiBERERkcExAiIiIyOCYgBAREZHBMQEhIiIig2MCQlQORo4cCTc3t1KdO3fuXCgUivINqJJJSEiAQqHA2rVr5Q6lSG5ubhg5cqQsP7sq/H6IyhsTEKrWFApFibbo6Gi5QyUA0dHRxf47bdq0Se4Qy2Tjxo1YsmSJ3GHoGDlyJCwtLeUOg2ogE7kDIKpI3333nc7r9evXY+/evYXKmzdvXqafs3r1aqjV6lKd+8EHH2DmzJll+vnVzdtvv4327dsXKvf395chmvKzceNGnD17FpMnT9Ypb9iwIR48eIBatWrJExiRDJiAULX2+uuv67z++++/sXfv3kLlT8rOzoa5uXmJf05ZvjhMTExgYsL/igV16dIFr7zyitxhGIxCoYBKpZI7DCKDYhcM1XjdunWDl5cXjh8/jueffx7m5ub473//CwDYvn07+vTpA2dnZyiVSnh4eGDBggXIy8vTucaTY0A0ffqffvopvvrqK3h4eECpVKJ9+/Y4evSozrn6xoAoFApMmjQJkZGR8PLyglKpRMuWLbFr165C8UdHR8PX1xcqlQoeHh5YtWpViceVHDx4EK+++ioaNGgApVIJV1dXvPvuu3jw4EGh+llaWuLGjRsYMGAALC0tUa9ePUybNq3Q7yItLQ0jR46EtbU1bGxsEBISgrS0tKfG8iy8vLzwwgsvFCpXq9VwcXHRSV4+/fRTdOrUCXXr1oWZmRl8fHzw008/PfVnFPU7XLt2LRQKBRISErRlJXmfdOvWDb/++isSExO1XUqa90xRY0B+//13dOnSBRYWFrCxsUH//v0RFxenN85Lly5h5MiRsLGxgbW1NUaNGoXs7Oyn1rOktmzZAh8fH5iZmcHOzg6vv/46bty4oXNMcnIyRo0ahfr160OpVMLJyQn9+/fX+V0dO3YMgYGBsLOzg5mZGdzd3TF69Ohyi5OqDv7ZRQTg7t276N27N4YOHYrXX38dDg4OAKQvG0tLS0yZMgWWlpb4/fffERoaioyMDHzyySdPve7GjRtx//59/Oc//4FCocDixYsxaNAgXLly5amtJocOHUJERAQmTJiA2rVr48svv8TgwYORlJSEunXrAgBOnjyJXr16wcnJCfPmzUNeXh7mz5+PevXqlajeW7ZsQXZ2NsaPH4+6deviyJEjWLp0Ka5fv44tW7boHJuXl4fAwED4+fnh008/xb59+/DZZ5/Bw8MD48ePBwAIIdC/f38cOnQIb775Jpo3b45t27YhJCSkRPFo3L9/H3fu3ClUXrduXSgUCgQFBWHu3LlITk6Go6Ojzu/s5s2bGDp0qLbsiy++QL9+/TB8+HDk5uZi06ZNePXVV7Fjxw706dPnmeIqSkneJ++//z7S09Nx/fp1/O9//wOAYsde7Nu3D71790ajRo0wd+5cPHjwAEuXLkXnzp1x4sSJQoOehwwZAnd3d4SFheHEiRP4+uuvYW9vj0WLFpVL/UaNGoX27dsjLCwMKSkp+OKLL/Dnn3/i5MmTsLGxAQAMHjwY586dw1tvvQU3NzekpqZi7969SEpK0r7u2bMn6tWrh5kzZ8LGxgYJCQmIiIgoc4xUBQmiGmTixIniybd9165dBQARHh5e6Pjs7OxCZf/5z3+Eubm5ePjwobYsJCRENGzYUPv66tWrAoCoW7eu+Pfff7Xl27dvFwDEL7/8oi2bM2dOoZgACFNTU3Hp0iVt2alTpwQAsXTpUm1Z3759hbm5ubhx44a27OLFi8LExKTQNfXRV7+wsDChUChEYmKiTv0AiPnz5+sc27ZtW+Hj46N9HRkZKQCIxYsXa8seP34sunTpIgCINWvWFBvP/v37BYAit1u3bgkhhIiPjy/0uxBCiAkTJghLS0udej1Zx9zcXOHl5SVefPFFnfKGDRuKkJAQ7Wt9/y5CCLFmzRoBQFy9erXInyGE/vdJnz59dN4nGpr3S8Hfj7e3t7C3txd3797Vlp06dUoYGRmJ4ODgQnGOHj1a55oDBw4UdevWLfSznhQSEiIsLCyK3J+bmyvs7e2Fl5eXePDggbZ8x44dAoAIDQ0VQghx7949AUB88sknRV5r27ZtAoA4evToU+Oi6o9dMEQAlEolRo0aVajczMxM+1zzV3mXLl2QnZ2Nf/7556nXDQoKgq2trfZ1ly5dAABXrlx56rkBAQHw8PDQvm7dujWsrKy05+bl5WHfvn0YMGAAnJ2dtcc1btwYvXv3fur1Ad36ZWVl4c6dO+jUqROEEDh58mSh4998802d1126dNGpy86dO2FiYqJtEQEAY2NjvPXWWyWKRyM0NBR79+4ttNWpUwcA0LRpU3h7e2Pz5s3ac/Ly8vDTTz+hb9++OvUq+PzevXtIT09Hly5dcOLEiWeKqThlfZ886datW4iNjcXIkSO1dQak90CPHj2wc+fOQufo+7e5e/cuMjIynvnnF3Ts2DGkpqZiwoQJOuNU+vTpA09PT/z6668ApN+BqakpoqOjce/ePb3X0rSU7NixA48ePSpTXFT1MQEhAuDi4gJTU9NC5efOncPAgQNhbW0NKysr1KtXTzuANT09/anXbdCggc5rTTJS1Ad0cedqztecm5qaigcPHqBx48aFjtNXpk9SUpL2S04zrqNr164ACtdPpVIV6topGA8AJCYmwsnJqVDXQrNmzUoUj0arVq0QEBBQaCv4bxQUFIQ///xTOw4hOjoaqampCAoK0rnWjh070LFjR6hUKtSpUwf16tXDypUrS/TvV1JlfZ88KTExEYD+31vz5s1x584dZGVl6ZSX5b1W2lg8PT21+5VKJRYtWoTffvsNDg4OeP7557F48WIkJydrj+/atSsGDx6MefPmwc7ODv3798eaNWuQk5NTphipamICQgTdv2A10tLS0LVrV5w6dQrz58/HL7/8gr1792r71Esy7dbY2FhvuRCiQs8tiby8PPTo0QO//vor3nvvPURGRmLv3r3agZBP1q+oeOQSFBQEIYR2rMqPP/4Ia2tr9OrVS3vMwYMH0a9fP6hUKqxYsQI7d+7E3r178dprrz3191jUIF59g27L+j4pDxX9fimJyZMn48KFCwgLC4NKpcLs2bPRvHlzbWuaQqHATz/9hMOHD2PSpEm4ceMGRo8eDR8fH2RmZhosTqocOAiVqAjR0dG4e/cuIiIi8Pzzz2vLr169KmNU+ezt7aFSqXDp0qVC+/SVPenMmTO4cOEC1q1bh+DgYG353r17Sx1Tw4YNERUVhczMTJ1WkPj4+FJfsyju7u7o0KEDNm/ejEmTJiEiIgIDBgyAUqnUHrN161aoVCrs3r1bp3zNmjVPvb6mBSEtLU3bdQDktwhoPMv7pKQr3jZs2BCA/t/bP//8Azs7O1hYWJToWmVVMJYXX3xRZ198fLx2v4aHhwemTp2KqVOn4uLFi/D29sZnn32GDRs2aI/p2LEjOnbsiI8++ggbN27E8OHDsWnTJowdO7biK0SVBltAiIqg+Yuy4F+Qubm5WLFihVwh6TA2NkZAQAAiIyNx8+ZNbfmlS5fw22+/leh8QLd+Qgh88cUXpY7ppZdewuPHj7Fy5UptWV5eHpYuXVrqaxYnKCgIf//9N7799lvcuXOnUPeLsbExFAqFTqtFQkICIiMjn3ptzfibP/74Q1uWlZWFdevWFfoZQMneJxYWFiXqknFycoK3tzfWrVunM4X57Nmz2LNnD1566aWnXqO8+Pr6wt7eHuHh4TpdJb/99hvi4uK0M4mys7Px8OFDnXM9PDxQu3Zt7Xn37t0r1CLj7e0NAOyGqYHYAkJUhE6dOsHW1hYhISF4++23oVAo8N133xm0Sftp5s6diz179qBz584YP3488vLysGzZMnh5eSE2NrbYcz09PeHh4YFp06bhxo0bsLKywtatW8s0ZqBv377o3LkzZs6ciYSEBLRo0QIRERHPPA7i4MGDhb7MAGkQZuvWrbWvhwwZgmnTpmHatGmoU6cOAgICdI7v06cPPv/8c/Tq1QuvvfYaUlNTsXz5cjRu3BinT58uNoaePXuiQYMGGDNmDKZPnw5jY2N8++23qFevHpKSkrTHPcv7xMfHB5s3b8aUKVPQvn17WFpaom/fvnp//ieffILevXvD398fY8aM0U7Dtba2xty5c4uN/Vk9evQIH374YaHyOnXqYMKECVi0aBFGjRqFrl27YtiwYdppuG5ubnj33XcBABcuXED37t0xZMgQtGjRAiYmJti2bRtSUlK006LXrVuHFStWYODAgfDw8MD9+/exevVqWFlZGTSpokpClrk3RDIpahpuy5Yt9R7/559/io4dOwozMzPh7OwsZsyYIXbv3i0AiP3792uPK2oarr4piQDEnDlztK+LmoY7ceLEQuc+OVVUCCGioqJE27ZthampqfDw8BBff/21mDp1qlCpVEX8FvKdP39eBAQECEtLS2FnZyfGjRunne5bcEpoUVM19cV+9+5dMWLECGFlZSWsra3FiBEjxMmTJ8tlGm7B35tG586dBQAxduxYvdf85ptvRJMmTYRSqRSenp5izZo1euPW97s9fvy48PPzE6ampqJBgwbi888/1zsNt6Tvk8zMTPHaa68JGxsbAUD7ntE3DVcIIfbt2yc6d+4szMzMhJWVlejbt684f/68zjGauty+fVunXF+c+mimWOvbPDw8tMdt3rxZtG3bViiVSlGnTh0xfPhwcf36de3+O3fuiIkTJwpPT09hYWEhrK2thZ+fn/jxxx+1x5w4cUIMGzZMNGjQQCiVSmFvby9efvllcezYsWJjpOpJIUQl+nOOiMrFgAEDcO7cOVy8eFHuUIiI9OIYEKIq7sll0y9evIidO3eiW7du8gRERFQCbAEhquKcnJwwcuRINGrUCImJiVi5ciVycnJw8uRJNGnSRO7wiIj04iBUoiquV69e+OGHH5CcnAylUgl/f398/PHHTD6IqFJjCwgREREZHMeAEBERkcExASEiIiKD4xgQPdRqNW7evInatWuXeOlkIiIiklYFvn//PpydnWFkVEw7h4xrkAghhFi2bJlo2LChUCqVokOHDiImJqbIYzUL6xTclEqlzjFqtVrMnj1bODo6CpVKJbp37y4uXLjwTDFdu3at2MWQuHHjxo0bN27Fb9euXSv2u1bWFhDNksTh4eHw8/PDkiVLEBgYiPj4eNjb2+s9x8rKSucGTU+2UCxevBhffvkl1q1bB3d3d8yePRuBgYE4f/48VCpVieKqXbs2AODatWuwsrIqZe2IiIhqnoyMDLi6umq/S4si6ywYPz8/tG/fHsuWLQMgdX24urrirbfewsyZMwsdv3btWkyePFnn5kwFCSHg7OyMqVOnYtq0aQCA9PR0ODg4YO3atdr7ETxNRkYGrK2tkZ6ezgSEiIjoGZT0O1S2Qai5ubk4fvy4zs2jjIyMEBAQgMOHDxd5XmZmJho2bAhXV1f0798f586d0+67evUqkpOTda5pbW0NPz+/Yq+Zk5ODjIwMnY2IiIgqjmwJyJ07d5CXlwcHBwedcgcHByQnJ+s9p1mzZvj222+xfft2bNiwAWq1Gp06dcL169cBQHves1wTAMLCwmBtba3dXF1dy1I1IiIieooqNQ3X398fwcHB8Pb2RteuXREREYF69eph1apVZbrurFmzkJ6ert2uXbtWThETERGRPrINQrWzs4OxsTFSUlJ0ylNSUuDo6Fiia9SqVQtt27bFpUuXAEB7XkpKCpycnHSu6e3tXeR1lEollErlM9aAiIhKSgiBx48fIy8vT+5QqIyMjY1hYmJS5mUqZEtATE1N4ePjg6ioKAwYMACANAg1KioKkyZNKtE18vLycObMGbz00ksAAHd3dzg6OiIqKkqbcGRkZCAmJgbjx4+viGqUSEQEMG8ecOEC0LQpMGcOMGiQbOEQERlUbm4ubt26hezsbLlDoXJibm4OJycnmJqalvoask7DnTJlCkJCQuDr64sOHTpgyZIlyMrKwqhRowAAwcHBcHFxQVhYGABg/vz56NixIxo3boy0tDR88sknSExMxNixYwFIU3InT56MDz/8EE2aNNFOw3V2dtYmOYYWEQEMHgwoFIAQwJkz0uutW5mEEFH1p1arcfXqVRgbG8PZ2RmmpqZc4LEKE0IgNzcXt2/fxtWrV9GkSZPiFxsrhqwJSFBQEG7fvo3Q0FAkJyfD29sbu3bt0g4iTUpK0qnYvXv3MG7cOCQnJ8PW1hY+Pj7466+/0KJFC+0xM2bMQFZWFt544w2kpaXhueeew65du0q8Bkh5mzcvP/kApEeFApg/nwkIEVV/ubm52iUWzM3N5Q6HyoGZmRlq1aqFxMRE5Obmlvr7lXfD1aM81wExMwMePixcrlIBDx6U6dJERJXew4cPcfXqVbi7u8v2hyCVv+L+XSv9OiA1RdOmUotHQQoF0KyZPPEQERFVBkxAKticOfndLhpCSOVEREQ1FROQCjZokDTgtHVrQDOcpX9/YOBAeeMiIiLDcnNzw5IlS+QOo9JgAmIAgwYBsbHAzz9Lr//4A+BsNCKiykmhUBS7zZ07t1TXPXr0KN54440yxdatWzdMnjy5TNeoLJiAGFCvXoC7O3DvHrBpk9zREBFVHRERQJs20sD+Nm2k1xXl1q1b2m3JkiWwsrLSKdPc7BTIX2CtJOrVq8eZQAUwATEgY2NgwgTp+dKl+VNziYhqCiGArKxn2zZulNZPOnNGmlWoWU9p48Znu05JP3MdHR21m7W1NRQKhfb1P//8g9q1a+O3336Dj48PlEolDh06hMuXL6N///5wcHCApaUl2rdvj3379ulc98kuGIVCga+//hoDBw6Eubk5mjRpgp81TeWltHXrVrRs2RJKpRJubm747LPPdPavWLECTZo0gUqlgoODA1555RXtvp9++gmtWrWCmZkZ6tati4CAAGRlZZUpnuIwATGw0aOlKbixsUAxN+glIqqWsrMBS8tn24YPl84tuJ4SIJU/y3XKs+t75syZWLhwIeLi4tC6dWtkZmbipZdeQlRUFE6ePIlevXqhb9++SEpKKvY68+bNw5AhQ3D69Gm89NJLGD58OP79999SxXT8+HEMGTIEQ4cOxZkzZzB37lzMnj0ba9euBQAcO3YMb7/9NubPn4/4+Hjs2rULzz//PACp1WfYsGEYPXo04uLiEB0djUGDBqFCV+oQVEh6eroAINLT0yvk+qNHCwEI8dprFXJ5IqJK48GDB+L8+fPiwYMHQgghMjOlzz85tszMZ49/zZo1wtraWvt6//79AoCIjIx86rktW7YUS5cu1b5u2LCh+N///qd9DUB88MEH2teZmZkCgPjtt9+KvGbXrl3FO++8o3ffa6+9Jnr06KFTNn36dNGiRQshhBBbt24VVlZWIiMjo9C5x48fFwBEQkLCU+slROF/14JK+h3KFhAZTJwoPW7ZAiQnyxsLEZEhmZsDmZnPtnl56V9PqVWrZ7tOeQ6/8PX11XmdmZmJadOmoXnz5rCxsYGlpSXi4uKe2gLSunVr7XMLCwtYWVkhNTW1VDHFxcWhc+fOOmWdO3fGxYsXkZeXhx49eqBhw4Zo1KgRRowYge+//157f542bdqge/fuaNWqFV599VWsXr0a9+7dK1UcJcUERAbt2gGdOgGPHgGrV8sdDRGR4SgUgIXFs23z5umup6S5vcW8ec92nfK8BY2FhYXO62nTpmHbtm34+OOPcfDgQcTGxqJVq1bIzc0t9jq1atV64vejgFqtLr9AC6hduzZOnDiBH374AU5OTggNDUWbNm2QlpYGY2Nj7N27F7/99htatGiBpUuXolmzZrh69WqFxAIwAZGNphVk1SopESEiIv0KrqekUkmPERGVaz2lP//8EyNHjsTAgQPRqlUrODo6IiEhwaAxNG/eHH/++WehuJo2bQpjY2MAgImJCQICArB48WKcPn0aCQkJ+P333wFIyU/nzp0xb948nDx5Eqampti2bVuFxSvrzehqsldeAaZMAW7cALZvl14TEZF+gwZV7ht4NmnSBBEREejbty8UCgVmz55dYS0Zt2/fRmxsrE6Zk5MTpk6divbt22PBggUICgrC4cOHsWzZMqxYsQIAsGPHDly5cgXPP/88bG1tsXPnTqjVajRr1gwxMTGIiopCz549YW9vj5iYGNy+fRvNmzevkDoAbAGRjakpMG6c9Hz5cnljISKisvn8889ha2uLTp06oW/fvggMDES7du0q5Gdt3LgRbdu21dlWr16Ndu3a4ccff8SmTZvg5eWF0NBQzJ8/HyNHjgQA2NjYICIiAi+++CKaN2+O8PBw/PDDD2jZsiWsrKzwxx9/4KWXXkLTpk3xwQcf4LPPPkPv3r0rpA4A74arV3neDbc4168Dbm5AXp40r93Lq8J+FBGRLHg33OqJd8Ot4urXz+/DZCsIERHVJExAZKYZjPrdd0B6uryxEBERGQoTEJl17Qq0bCktE7xundzREBERGQYTEJkpFMCkSdLz5cuBCho0TUREVKkwAakEXn8dsLICLlwAoqLkjoaIqPxxvkP1Uh7/nkxAKgFLS+D/Z0lh2TJZQyEiKlealT6zy/NOcCQ7zb/nkyu5PgsuRFZJTJgAfPkl8MsvQEKCND2XiKiqMzY2ho2Njfb+Jubm5lCU55roZFBCCGRnZyM1NRU2NjbaFVZLgwlIJdGsGdCjB7B3LxAeDixcKHdERETlw9HREQBKfZM1qnxsbGy0/66lxYXI9DDUQmRP+vlnoH9/oG5daZEyrtlDRNVJXl4eHvHmV1VerVq1im35KOl3KFtAKpE+fYCGDYHERGDzZiAkRO6IiIjKj7GxcZma7Kl64SDUSsTYGBg/Xnq+dKl0u2kiIqLqiAlIJTNmDKBUAsePA0eOyB0NERFRxWACUsnY2QFDh0rPeX8YIiKqrpiAVEKalVE3bwY4aJyIiKoj2ROQ5cuXw83NDSqVCn5+fjhSwn6HTZs2QaFQYMCAATrlI0eOhEKh0Nl69epVAZFXHF9foEMHIDcX+PpruaMhIiIqf7ImIJs3b8aUKVMwZ84cnDhxAm3atEFgYOBT54onJCRg2rRp6NKli979vXr1wq1bt7TbDz/8UBHhVyhNK0h4OPD4sbyxEBERlTdZE5DPP/8c48aNw6hRo9CiRQuEh4fD3Nwc3377bZHn5OXlYfjw4Zg3bx4aNWqk9xilUglHR0ftZmtrW1FVqDCvvgrUqwdcuyatjkpERFSdyJaA5Obm4vjx4wgICMgPxsgIAQEBOHz4cJHnzZ8/H/b29hgzZkyRx0RHR8Pe3h7NmjXD+PHjcffu3WJjycnJQUZGhs4mN5UKGDtWes7BqEREVN3IloDcuXMHeXl5cHBw0Cl3cHBAcnKy3nMOHTqEb775BqtXry7yur169cL69esRFRWFRYsW4cCBA+jduzfy8vKKPCcsLAzW1tbazdXVtXSVKmdvvgkYGUl3yI2LkzsaIiKi8iP7INSSun//PkaMGIHVq1fDzs6uyOOGDh2Kfv36oVWrVhgwYAB27NiBo0ePIjo6ushzZs2ahfT0dO127dq1CqjBs2vQQFqaHWArCBERVS+yJSB2dnYwNjZGSkqKTnlKSoreG9xcvnwZCQkJ6Nu3L0xMTGBiYoL169fj559/homJCS5fvqz35zRq1Ah2dna4dOlSkbEolUpYWVnpbJXFxInS47p1QCXoGSIiIioXsiUgpqam8PHxQVRUlLZMrVYjKioK/v7+hY739PTEmTNnEBsbq9369euHF154AbGxsUV2m1y/fh13796Fk5NThdWlIr34IuDpCWRmAt99J3c0RERE5UPWLpgpU6Zg9erVWLduHeLi4jB+/HhkZWVh1KhRAIDg4GDMmjULAKBSqeDl5aWz2djYoHbt2vDy8oKpqSkyMzMxffp0/P3330hISEBUVBT69++Pxo0bIzAwUM6qlppCkd8Ksnw57w9DRETVg6x3ww0KCsLt27cRGhqK5ORkeHt7Y9euXdqBqUlJSTAyKnmOZGxsjNOnT2PdunVIS0uDs7MzevbsiQULFkCpVFZUNSpccDAwa5Y0EHX/fqlVhIiIqCpTCMG/qZ+UkZEBa2trpKenV5rxIJMmSS0gAwcCERFyR0NERKRfSb9Dq8wsmJpuwgTpcft2IClJ3liIiIjKiglIFdGihdT1olYDq1bJHQ0REVHZMAGpQjT3h1m9GsjJkTcWIiKismACUoX07Qu4ugK3bwNbtsgdDRERUekxAalCTEyk5dkBYNkyeWMhIiIqCyYgVczYsYCpKRATAxw7Jnc0REREpcMEpIqxtweGDJGe8/4wRERUVTEBqYI0g1F/+AG4c0feWIiIiEqDCUgV1KED4OMjzYT55hu5oyEiInp2TECqIIUivxVk5UogL0/eeIiIiJ4VE5AqKigIqFsXSEwEfv1V7miIiIieDROQKsrMDBgzRnrOKblERFTVMAGpwsaPl7pj9u4F4uPljoaIiKjkmIBUYW5u0uqoALBihayhEBERPRMmIFXcxInS49q1QGamrKEQERGVGBOQKi4gAGjaFMjIADZskDsaIiKikmECUsUZGQETJkjPly0DhJA3HiIiopJgAlINhIQAFhbAuXPAH3/IHQ0REdHTMQGpBmxsgBEjpOeckktERFUBE5BqQjMYdds24Pp1eWMhIiJ6GiYg1YSXF9C1q7Qs+1dfyR0NERFR8ZiAVCOa+8N89RWQmytvLERERMVhAlKN9O8PODsDKSnA1q1yR0NERFQ0JiDVSK1awJtvSs85GJWIiCozJiDVzLhxUiLy11/AyZNyR0NERKQfE5BqxtEReOUV6fny5fLGQkREVBQmINWQZjDq998D//4rbyxERET6yJ6ALF++HG5ublCpVPDz88ORI0dKdN6mTZugUCgwYMAAnXIhBEJDQ+Hk5AQzMzMEBATg4sWLFRB55eXvD3h7Aw8fAt9+K3c0REREhcmagGzevBlTpkzBnDlzcOLECbRp0waBgYFITU0t9ryEhARMmzYNXbp0KbRv8eLF+PLLLxEeHo6YmBhYWFggMDAQDx8+rKhqVDoKRX4ryMqV0togRERElYmsCcjnn3+OcePGYdSoUWjRogXCw8Nhbm6Ob4v5sz0vLw/Dhw/HvHnz0KhRI519QggsWbIEH3zwAfr374/WrVtj/fr1uHnzJiIjIyu4NpXLsGGArS1w5Qqwa5fc0RAREemSLQHJzc3F8ePHERAQkB+MkRECAgJw+PDhIs+bP38+7O3tMWbMmEL7rl69iuTkZJ1rWltbw8/Pr9hr5uTkICMjQ2er6szNgdGjpeeckktERJWNbAnInTt3kJeXBwcHB51yBwcHJCcn6z3n0KFD+Oabb7B69Wq9+zXnPcs1ASAsLAzW1tbazdXV9VmqUmmNHy91x+zaBVy6JHc0RERE+WQfhFpS9+/fx4gRI7B69WrY2dmV67VnzZqF9PR07Xbt2rVyvb5cPDyA3r2l5ytWyBsLERFRQSZy/WA7OzsYGxsjJSVFpzwlJQWOjo6Fjr98+TISEhLQt29fbZlarQYAmJiYID4+XnteSkoKnJycdK7p7e1dZCxKpRJKpbIs1am0Jk0Cdu4E1qwBFiwALCzkjoiIiEjGFhBTU1P4+PggKipKW6ZWqxEVFQV/f/9Cx3t6euLMmTOIjY3Vbv369cMLL7yA2NhYuLq6wt3dHY6OjjrXzMjIQExMjN5r1gSBgVJLSFoasHGj3NEQERFJZGsBAYApU6YgJCQEvr6+6NChA5YsWYKsrCyMGjUKABAcHAwXFxeEhYVBpVLBy8tL53wbGxsA0CmfPHkyPvzwQzRp0gTu7u6YPXs2nJ2dC60XUlMYGQETJgBTp0qDUceOlcaFEBERyUnWBCQoKAi3b99GaGgokpOT4e3tjV27dmkHkSYlJcHI6NkaaWbMmIGsrCy88cYbSEtLw3PPPYddu3ZBpVJVRBWqhFGjgA8+AE6fBv78E3juObkjIiKimk4hhBByB1HZZGRkwNraGunp6bCyspI7nHLxxhvA6tVAUBCwaZPc0RARUXVV0u/QKjMLhspm4kTpcetW4NYteWMhIiJiAlJDtGkjdb08fgx89ZXc0RARUU3HBKQG0bSCrFoFPHokbyxERFSzMQGpQQYNAhwdpS6YbdvkjoaIiGoyJiA1iKkp8J//SM95fxgiIpITE5Aa5o03ABMT4OBBaVouERGRHJiA1DDOzlJXDAAsXy5vLEREVHMxAamBJk2SHjdsAO7dkzcWIiKqmZiA1EDPPQe0agVkZwNr18odDRER1URMQGoghSK/FWTFCuD/bypMRERkMExAaqjhwwFra+DSJWDPHrmjISKimoYJSA1lYSHdpA7glFwiIjI8JiA12IQJ0uPOncCVK/LGQkRENQsTkBqsSRMgMBAQAli5Uu5oiIioJmECUsNpBqN+8400K4aIiMgQmIDUcL17A+7u0nogmzbJHQ0REdUUpUpArl27huvXr2tfHzlyBJMnT8ZXvM97lWNsDIwfLz1ftkzqjiEiIqpopUpAXnvtNezfvx8AkJycjB49euDIkSN4//33MX/+/HINkCre6NGASgWcPAn8/bfc0RARUU1QqgTk7Nmz6NChAwDgxx9/hJeXF/766y98//33WMulNaucunWBYcOk55ySS0REhlCqBOTRo0dQKpUAgH379qFfv34AAE9PT9y6dav8oiOD0QxG3bIFSEmRNxYiIqr+SpWAtGzZEuHh4Th48CD27t2LXr16AQBu3ryJunXrlmuAZBjt2gH+/sCjR8Dq1XJHQ0RE1V2pEpBFixZh1apV6NatG4YNG4Y2bdoAAH7++Wdt1wxVPRMnSo/h4cDjx/LGQkRE1ZtCiNLNe8jLy0NGRgZsbW21ZQkJCTA3N4e9vX25BSiHjIwMWFtbIz09HVZWVnKHYzA5OUCDBkBqKvDTT8DgwXJHREREVU1Jv0NL1QLy4MED5OTkaJOPxMRELFmyBPHx8VU++ajJlErgjTek5xyMSkREFalUCUj//v2xfv16AEBaWhr8/Pzw2WefYcCAAVjJNb2rtP/8R1obJDoaOHtW7miIiKi6KlUCcuLECXTp0gUA8NNPP8HBwQGJiYlYv349vvzyy3INkAyrfn1gwADp+YoVsoZCRETVWKkSkOzsbNSuXRsAsGfPHgwaNAhGRkbo2LEjEhMTyzVAMjzNYNT164H0dHljISKi6qlUCUjjxo0RGRmJa9euYffu3ejZsycAIDU1tUYN2qyuunUDWrQAsrKAdevkjoaIiKqjUiUgoaGhmDZtGtzc3NChQwf4+/sDkFpD2rZt+0zXWr58Odzc3KBSqeDn54cjR44UeWxERAR8fX1hY2MDCwsLeHt747vvvtM5ZuTIkVAoFDqbZp0SKhmFIn9hsuXLAbVa3niIiKj6KfU03OTkZNy6dQtt2rSBkZGUxxw5cgRWVlbw9PQs0TU2b96M4OBghIeHw8/PD0uWLMGWLVuKnE0THR2Ne/fuwdPTE6amptixYwemTp2KX3/9FYGBgQCkBCQlJQVr1qzRnqdUKnWmCz9NTZ2GW9D9+4CLi/S4Zw/Qo4fcERERUVVQ0u/QUicgGpq74tavX/+Zz/Xz80P79u2x7P/nfKrVari6uuKtt97CzJkzS3SNdu3aoU+fPliwYAEAKQFJS0tDZGTkM8ejwQRE8vbbwNKlQL9+wPbtckdDRERVQYWuA6JWqzF//nxYW1ujYcOGaNiwIWxsbLBgwQKoS9hen5ubi+PHjyMgICA/GCMjBAQE4PDhw089XwiBqKgoxMfH4/nnn9fZFx0dDXt7ezRr1gzjx4/H3bt3i71WTk4OMjIydDYCJkyQHnfsABISZA2FiIiqmVIlIO+//z6WLVuGhQsX4uTJkzh58iQ+/vhjLF26FLNnzy7RNe7cuYO8vDw4ODjolDs4OCA5ObnI89LT02FpaQlTU1P06dMHS5cuRY8C/QO9evXC+vXrERUVhUWLFuHAgQPo3bs38vLyirxmWFgYrK2ttZurq2uJ6lDdeXoCAQHSGJDwcLmjISKi6qRUXTDOzs4IDw/X3gVXY/v27ZgwYQJu3Ljx1GvcvHkTLi4u+Ouvv7SDWAFgxowZOHDgAGJiYvSep1arceXKFWRmZiIqKgoLFixAZGQkunXrpvf4K1euwMPDA/v27UP37t31HpOTk4OcnBzt64yMDLi6utb4LhhA6noZMACoWxe4fh1QqeSOiIiIKrOSdsGYlObi//77r96Bpp6envj3339LdA07OzsYGxsj5Yl7v6ekpMDR0bHI84yMjNC4cWMAgLe3N+Li4hAWFlZkAtKoUSPY2dnh0qVLRSYgSqUSSqWyRHHXNC+/LN0fJikJ2LwZCAmROyIiIqoOStUF06ZNG+3A0YKWLVuG1q1bl+gapqam8PHxQVRUlLZMrVYjKipKp0XkadRqtU7rxZOuX7+Ou3fvwsnJqcTXpHzGxsD48dJz3h+GiIjKS6laQBYvXow+ffpg37592mTh8OHDuHbtGnbu3Fni60yZMgUhISHw9fVFhw4dsGTJEmRlZWHUqFEAgODgYLi4uCAsLAyANFbD19cXHh4eyMnJwc6dO/Hdd99p7z+TmZmJefPmYfDgwXB0dMTly5cxY8YMNG7cWDtNl57dmDHA3LnAsWPAkSNAhw5yR0RERFVdqVpAunbtigsXLmDgwIFIS0tDWloaBg0ahHPnzhVaGKw4QUFB+PTTTxEaGgpvb2/ExsZi165d2oGpSUlJuHXrlvb4rKwsTJgwAS1btkTnzp2xdetWbNiwAWPHjgUAGBsb4/Tp0+jXrx+aNm2KMWPGwMfHBwcPHmQXSxnUqwcEBUnP2QpCRETloczrgBR06tQptGvXrtgZJ1UB1wEp7OhRqeXD1BS4dg3Qs04cERFRxa4DQjVP+/ZSApKbC3zzjdzREBFRVccEhEpMc5fclSuBx4/ljYWIiKo2JiBUYkOGAHZ2UhfMjh1yR0NERFXZM82CGTRoULH709LSyhILVXIqFTB2LLBwoTQYdcAAuSMiIqKq6pkSEGtr66fuDw4OLlNAVLm9+SaweDEQFQXExQHNm8sdERERVUXPlIAUvMU91UwNG0p3x42MBFaskO6WS0RE9Kw4BoSemWYw6tq1AG8cTEREpcEEhJ5Z9+5As2ZAZibwDOvOERERaTEBoWemUACTJknPly8Hym8pOyIiqimYgFCpBAcDlpbSQNT9++WOhoiIqhomIFQqVlZSEgLw/jBERPTsmIBQqWkGo27fDiQlyRsLERFVLUxAqNRatABeeAFQq4FVq+SOhoiIqhImIFQmmsGoq1cDOTnyxkJERFUHExAqk379gPr1gdu3gS1b5I6GiIiqCiYgVCYmJtLy7AAHoxIRUckxAaEyGzcOMDUFYmKAY8fkjoaIiKoCJiBUZvb2wKuvSs+XL5c3FiIiqhqYgFC50AxG/eEH4M4deWMhIqLKjwkIlQs/P8DHR5oJ8+23ckdDRESVHRMQKhcKRf7CZCtWAHl58sZDRESVGxMQKjdDhwJ16gCJicDOnXJHQ0RElRkTECo3ZmbAmDHSc07JJSKi4jABoXI1frzUHbNnD3DhgtzREBFRZcUEhMqVu7s0GBWQ7hXTpg0QESFvTEREVPkwAaFyFRGRvxhZXh5w5gwweDCTECIi0mUidwBUvcybJ3XBCCG91jyOHi0lI15e0ubhIS3jTkRENRO/AqhcXbiQn3QUlJ4OzJ2b/1qpBJo3z09INFuDBlICQ0RE1ZvsXTDLly+Hm5sbVCoV/Pz8cOTIkSKPjYiIgK+vL2xsbGBhYQFvb2989913OscIIRAaGgonJyeYmZkhICAAFy9erOhq0P9r2rRwAqFQAM7OUitIhw6AhYW0YFlsLLBhAzBzJvDyy4CbG2BtDfj7S/eX+eILICoKSEnRn9QQEVHVpRBCvo/2zZs3Izg4GOHh4fDz88OSJUuwZcsWxMfHw97evtDx0dHRuHfvHjw9PWFqaoodO3Zg6tSp+PXXXxEYGAgAWLRoEcLCwrBu3Tq4u7tj9uzZOHPmDM6fPw+VSlWiuDIyMmBtbY309HRYWVmVa52ru4gIacyHphtG8xgRAQwcKB2jVktrhZw9q7vFxQGPHum/rp1d4daSli0BGxuDVY2IiEqgpN+hsiYgfn5+aN++PZb9/6IRarUarq6ueOuttzBz5swSXaNdu3bo06cPFixYACEEnJ2dMXXqVEybNg0AkJ6eDgcHB6xduxZDhw4t0TWZgJRNRAQwfz4QHw80awbMmZOffBTn0SPg0qXCicmlS1LSok/9+vnJiCYxadECMDcv3zoREVHJVPoEJDc3F+bm5vjpp58wYMAAbXlISAjS0tKwffv2Ys8XQuD3339Hv379EBkZiR49euDKlSvw8PDAyZMn4e3trT22a9eu8Pb2xhdffKH3Wjk5OcjJydG+zsjIgKurKxOQSuLBA+CffwonJklJ+o9XKIBGjQq3mDRtCpiaGjZ2IqKapqQJiGyDUO/cuYO8vDw4ODjolDs4OOCff/4p8rz09HS4uLggJycHxsbGWLFiBXr06AEASE5O1l7jyWtq9ukTFhaGefPmlbYqVMHMzIC2baWtoPR04Pz5wolJaipw+bK0FcxjTUykFpknExN3d8DY2LB1IiKq6arcLJjatWsjNjYWmZmZiIqKwpQpU9CoUSN069at1NecNWsWpkyZon2taQGhyk0zYNXfX7c8NRU4d65wYpKRIZWfOwds3px/vJmZ1G3zZGLi4sIZOUREFUW2BMTOzg7GxsZISUnRKU9JSYGjo2OR5xkZGaFx48YAAG9vb8TFxSEsLAzdunXTnpeSkgInJyedaxbsknmSUqmEUqksQ22oMrG3l7YXXsgvEwK4fr1wUnL+vNTFc/y4tBVkbV04KfHykgbEPikiQloD5cIFqatnzhxg0KCKrScRUVUmWwJiamoKHx8fREVFaceAqNVqREVFYdKkSSW+jlqt1o7fcHd3h6OjI6KiorQJR0ZGBmJiYjB+/PjyrgJVIQoF4Ooqbb1755fn5QFXrhROTOLjpS6eP/+UtoIcHHQTkjt3gFmz8mf8aFZ/3bqVSQgRUVFk7YKZMmUKQkJC4Ovriw4dOmDJkiXIysrCqFGjAADBwcFwcXFBWFgYAGmshq+vLzw8PJCTk4OdO3fiu+++w8qVKwEACoUCkydPxocffogmTZpop+E6OzvrDHQl0jA2Bpo0kbaCM3VycqTWjCcTkytXpHVJUlKkNUoKKrj6q0IhzQRiAkJEpJ+sCUhQUBBu376N0NBQJCcnw9vbG7t27dIOIk1KSoKRUf5aaVlZWZgwYQKuX78OMzMzeHp6YsOGDQgKCtIeM2PGDGRlZeGNN95AWloannvuOezatavEa4AQAdJKra1aSVtBmZnSeiUFk5I9ewqfL4Q01iQhQVpgjYiIdMm6DkhlxXVA6Fm0aSN1uxT1P+mFF4CRI6VuGQsLg4ZGRGRwJf0OlX0pdqKqbs6c/G4XIP+xVSvp+f79QEgI4OgIjBkDHDzIpeWJiJiAEJXRoEHSgNPWrQGVSnqMiABOnwauXgUWLJDu/puZCXz7LfD880DjxlJ5YqLc0RMRyYNdMHqwC4bKmxDSbJq1a6U1SDIz8/e9+CIwapSUyHAJeSKq6ir9UuyVGRMQqkhZWVILydq1wO+/55fXrg0MGSKNF+ncmYugEVHVxASkDJiAkKEkJADffSclI1eu5Jc3biyNGwkOBho0kCs6IqJnxwSkDJiAkKEJIQ1OXbsW+PFHqZUEkFpBuneXWkUGDmQXDRFVfkxAyoAJCMkpM1PqolmzBoiOzi+vXRsICpKSkU6d2EVDRJUTE5AyYAJClcXVq8D69cC6ddJzjSZNpERkxAhpeXkiosqCCUgZMAGhykatzu+i2bJFt4smICC/i8bMTM4oiYiYgJQJExCqzDIzgZ9+kpKRAwfyy62sgKFDpWSkY0d20RCRPJiAlAETEKoqrlyRumjWrtVd1Kxp0/wumvr15YqOiGoiJiBlwASEqhq1WmoNWbtWah3JzpbKjYyAHj2kZKR/f3bREFHFYwJSBkxAqCq7f19KQtaskcaNaFhb53fR+Pmxi4aIKgYTkDJgAkLVxaVL+bNokpLyyz09pUTk9dcBFxfZwiOiaogJSBkwAaHqRq2W1hTRdNE8eCCVGxkBPXvmd9GoVDIGSUTVAhOQMmACQtVZRoY0lXftWuDQofxyGxtg2DApGWnfnl00RFQ6TEDKgAkI1RQXL0rdM+vWAdev55c3b57fRePsLFt4RFQFMQEpAyYgVNPk5QH790utIlu3Ag8fSuVGRkCvXlIy0rcvu2iI6OlK+h1qZMCYiKiSMjaWVlTdsAFITgZWr5buN6NWAzt3AkOGSC0hEycCR49KSUqbNtK03jZtpHvXEBE9C7aA6MEWECLJhQtS98z69bpdNAUpFNLdfLduBQYNMmx8RFT5sAWEiMqsaVPgo4+AhARgzx7gtdcKD07V/Akzbhzw2WdSi8mVK1LrCRFRUdgCogdbQIiKplIBOTklO65ZM2lAq6en9Ni8uXQnX44lIaq+SvodamLAmIioGmjWDDhzJr/lA5BaReztga5dgbg4qevm4UPg1ClpK8jICHB3L5yYNG8uTQUmopqBLSB6sAWEqGgREcDgwfljPzSPERHAwIHSMXl5UrdNXFz+9s8/0mNaWtHXdnDIT0YKJicuLlyXhKiq4DTcMmACQlS8iAhg/nwgPl5qEZkzJz/5KI4QQEpKfjJSMDkpapArAFha6iYkmuceHkCtWuVXLyIqOyYgZcAEhMjw7t+XEpGCyck//0j3s3n8WP85JiZA48a63TientJmaWnY+IlIwgSkDJiAEFUeubnA5cu63Tia51lZRZ/n6qq/1cTent05RBWJCUgZMAEhqvyEkLptnhxjEhcHpKYWfZ6tbeExJp6egJubtCAbEZVNlVkHZPny5XBzc4NKpYKfnx+OHDlS5LGrV69Gly5dYGtrC1tbWwQEBBQ6fuTIkVAoFDpbr169KroaRGRgCoXUytGzJ/DOO8DKldIdf1NSgLt3gT//BL7+Gpg2DejTB2jUSDrn3j3gr7+Ab78Fpk8HXn5Z6saxtJRWdR06FJg7F9i8GTh9Ov/OwRERXP2VqDzJ2gKyefNmBAcHIzw8HH5+fliyZAm2bNmC+Ph42NvbFzp++PDh6Ny5Mzp16gSVSoVFixZh27ZtOHfuHFxcXABICUhKSgrWrFmjPU+pVMLW1rbEcbEFhKh6evhQmiL8ZKtJfHzRa5tophinpOiWcfVXIv2qRBeMn58f2rdvj2XLlgEA1Go1XF1d8dZbb2HmzJlPPT8vLw+2trZYtmwZgoODAUgJSFpaGiIjI0scR05ODnIKfPpkZGTA1dWVCQhRDZGXByQm6u/OuXev6PNq1wZmzAD8/ID27bmOCRFQBRYiy83NxfHjxzFr1ixtmZGREQICAnD48OESXSM7OxuPHj1CnTp1dMqjo6Nhb28PW1tbvPjii/jwww9Rt27dIq8TFhaGefPmla4iRFTlGRtLXTSNGkndNRpCSONJGjSQBsM+6f59YPbs/NeenlIy0rGj9NiqlTRTh4gKk60F5ObNm3BxccFff/0Ff39/bfmMGTNw4MABxMTEPPUaEyZMwO7du3Hu3Dmo/n9t502bNsHc3Bzu7u64fPky/vvf/8LS0hKHDx+GcREjzNgCQkTFadNG/+qvTk7S6q8xMdL9b55kZgb4+EjJiCYxqV+fs3Coeqv0LSBltXDhQmzatAnR0dHa5AMAhg4dqn3eqlUrtG7dGh4eHoiOjkb37t31XkupVEKpVFZ4zERUNc2Zo3/112XL8hdgu30bOHIE+PtvKSE5cgRITwcOHZI2DSen/ITEzw/w9ZW6cohqGtkSEDs7OxgbGyOl4MguACkpKXB0dCz23E8//RQLFy7Evn370Lp162KPbdSoEezs7HDp0qUiExAiouIMGiQNOC1u9dd69aTuG00XjlotDXiNicnfTp0Cbt0CIiOlDZDujdOiRX63jZ+f9JpTgqm6ky0BMTU1hY+PD6KiojBgwAAA0iDUqKgoTJo0qcjzFi9ejI8++gi7d++Gr6/vU3/O9evXcffuXTg5OZVX6ERUAw0a9GwzXoyM8ldlDQmRyrKzgRMndJOSpCTg7Flp+/pr6ThLS6llpGBLibNz+deJSE6yT8MNCQnBqlWr0KFDByxZsgQ//vgj/vnnHzg4OCA4OBguLi4ICwsDACxatAihoaHYuHEjOnfurL2OpaUlLC0tkZmZiXnz5mHw4MFwdHTE5cuXMWPGDNy/fx9nzpwpcTcLp+ESkaHcuqXbdXP0KJCZWfg4V1fdhMTHBzA3N3y8RE9TJcaABAUF4fbt2wgNDUVycjK8vb2xa9cuODg4AACSkpJgZJS/VtrKlSuRm5uLV155Rec6c+bMwdy5c2FsbIzTp09j3bp1SEtLg7OzM3r27IkFCxZwjAcRVUpOTkD//tIGSFOC4+LyW0j+/hs4dw64dk3afvpJOs7YWJplU7DrplkzqeWFqCrgUux6sAWEiCqTzEzg2DHdrpubNwsfZ20trUdSsKVEz5qORBWqSixEVlkxASGiyu769fxum5gYKUHRLBtfkLu7bkLSti1QYOIg/b+ICGDePGngcNOm0iBjrnJbOkxAyoAJCBFVNY8fSwNZNd02MTFSV86TatWS1jUp2HXTuHHNXpskIkL/NGsutV86TEDKgAkIEVUH6enSoNaCXTf67hRcpw7QoUN+QtKhA3DgQPm3CKjV0j13cnKklWX1PVbEvqcdk52tP15nZ+CHH6RWI67VUnJMQMqACQgRVUdCSPe8Kdh1c+JE0Tfie1KPHtKXcmkTgry8iq1fRVEopAG+vr75m7c3YGEhd2SVExOQMmACQkQ1RW4ucPq0btfNxYuG+dmmptKmVBZ+1FdW1n1FHdOnj3QDwieX2q9dWxrYe+1a4dg1C8j5+kpTon19pa4tMzPD/O4qMyYgZcAEhIhqMpVKf6uIiQnw0UflkxCYmlaecSdFjQGJiJBWu01JAY4flwb6arZbtwpfx9gY8PLSbSlp1Uqqc03CBKQMmIAQUU1W1M33WrcGYmNlC6tCRUQUv9T+k27e1E1Kjh6V7gf0pFq1pCSkYFLi5SWVV1dMQMqACQgR1WRPaxGgwoSQpkYfO6abmNy9W/hYpVJK8gp237RoIbUwVQdMQMqACQgR1XTP2iJAhWkG/Rbsujl2TJqd9CQzM2lga8GWkmbNquZNCZmAlAETECIiqghCAFeu6CYkx48D9+8XPtbCQpoCXDApadKk8i+3zwSkDJiAEBGRoajV0syjgt03J04AWVmFj61dW+q20XTd+PoCHh6VZ0AvwASkTJiAEBGRnPLypO6vgi0lsbH6l9u3sdFNSHx8ADc3+ZISJiBlwASEiIgqm8ePpeX1CyYlp07pnzJdp45u142vL1C/vm5SUlH3v2ECUgZMQIiIqCrIzQXOndOdeXP6NPDoUeFj7e3zW0gePQIWLqyY+98wASkDJiBERFRV5eRI67gUbCk5e/bpS+GX11ovJf0OrSazjomIiAiQ1hnRdLtoPHggtYxoEpJ163QXmgOk1/HxhouTCQgREVE1Z2aWf7djQJplo2+122bNDBdTJZ9NTEREROVtzpz8sR9A/hiQOXMMFwMTECIiohpm0CBpwGnr1tLNB1u3NvxS++yCISIiqoEGDSqfabelxRYQIiIiMjgmIERERGRwTECIiIjI4JiAEBERkcExASEiIiKD4ywYPTSr02dkZMgcCRERUdWi+e582p1emIDocf/+fQCAq6urzJEQERFVTffv34e1tXWR+3kzOj3UajVu3ryJ2rVrQ1Hw3sVlkJGRAVdXV1y7dq1G3OCuptUXqHl1rmn1BWpenVnf6q8i6iyEwP379+Hs7Awjo6JHerAFRA8jIyPUr1+/Qq5tZWVVY97YQM2rL1Dz6lzT6gvUvDqzvtVfede5uJYPDQ5CJSIiIoNjAkJEREQGxwTEQJRKJebMmQOlUil3KAZR0+oL1Lw617T6AjWvzqxv9SdnnTkIlYiIiAyOLSBERERkcExAiIiIyOCYgBAREZHBMQEhIiIig2MCUgZhYWFo3749ateuDXt7ewwYMADx8fE6xzx8+BATJ05E3bp1YWlpicGDByMlJUXnmKSkJPTp0wfm5uawt7fH9OnT8fjxY0NWpVQWLlwIhUKByZMna8uqY31v3LiB119/HXXr1oWZmRlatWqFY8eOafcLIRAaGgonJyeYmZkhICAAFy9e1LnGv//+i+HDh8PKygo2NjYYM2YMMjMzDV2Vp8rLy8Ps2bPh7u4OMzMzeHh4YMGCBTr3dKjq9f3jjz/Qt29fODs7Q6FQIDIyUmd/edXv9OnT6NKlC1QqFVxdXbF48eKKrppexdX30aNHeO+999CqVStYWFjA2dkZwcHBuHnzps41qkt9n/Tmm29CoVBgyZIlOuVVqb5AyeocFxeHfv36wdraGhYWFmjfvj2SkpK0+2X57BZUaoGBgWLNmjXi7NmzIjY2Vrz00kuiQYMGIjMzU3vMm2++KVxdXUVUVJQ4duyY6Nixo+jUqZN2/+PHj4WXl5cICAgQJ0+eFDt37hR2dnZi1qxZclSpxI4cOSLc3NxE69atxTvvvKMtr271/ffff0XDhg3FyJEjRUxMjLhy5YrYvXu3uHTpkvaYhQsXCmtraxEZGSlOnTol+vXrJ9zd3cWDBw+0x/Tq1Uu0adNG/P333+LgwYOicePGYtiwYXJUqVgfffSRqFu3rtixY4e4evWq2LJli7C0tBRffPGF9piqXt+dO3eK999/X0RERAgAYtu2bTr7y6N+6enpwsHBQQwfPlycPXtW/PDDD8LMzEysWrXKUNXUKq6+aWlpIiAgQGzevFn8888/4vDhw6JDhw7Cx8dH5xrVpb4FRUREiDZt2ghnZ2fxv//9T2dfVaqvEE+v86VLl0SdOnXE9OnTxYkTJ8SlS5fE9u3bRUpKivYYOT67mYCUo9TUVAFAHDhwQAgh/eeuVauW2LJli/aYuLg4AUAcPnxYCCG9cYyMjERycrL2mJUrVworKyuRk5Nj2AqU0P3790WTJk3E3r17RdeuXbUJSHWs73vvvSeee+65Iver1Wrh6OgoPvnkE21ZWlqaUCqV4ocffhBCCHH+/HkBQBw9elR7zG+//SYUCoW4ceNGxQVfCn369BGjR4/WKRs0aJAYPny4EKL61ffJD+vyqt+KFSuEra2tznv6vffeE82aNavgGhWvuC9kjSNHjggAIjExUQhRPet7/fp14eLiIs6ePSsaNmyok4BU5foKob/OQUFB4vXXXy/yHLk+u9kFU47S09MBAHXq1AEAHD9+HI8ePUJAQID2GE9PTzRo0ACHDx8GABw+fBitWrWCg4OD9pjAwEBkZGTg3LlzBoy+5CZOnIg+ffro1AuonvX9+eef4evri1dffRX29vZo27YtVq9erd1/9epVJCcn69TZ2toafn5+OnW2sbGBr6+v9piAgAAYGRkhJibGcJUpgU6dOiEqKgoXLlwAAJw6dQqHDh1C7969AVS/+j6pvOp3+PBhPP/88zA1NdUeExgYiPj4eNy7d89AtSmd9PR0KBQK2NjYAKh+9VWr1RgxYgSmT5+Oli1bFtpfHev766+/omnTpggMDIS9vT38/Px0umnk+uxmAlJO1Go1Jk+ejM6dO8PLywsAkJycDFNTU+1/ZA0HBwckJydrjyn4D6rZr9lX2WzatAknTpxAWFhYoX3Vsb5XrlzBypUr0aRJE+zevRvjx4/H22+/jXXr1gHIj1lfnQrW2d7eXme/iYkJ6tSpU+nqPHPmTAwdOhSenp6oVasW2rZti8mTJ2P48OEAql99n1Re9atq73ONhw8f4r333sOwYcO0NyarbvVdtGgRTExM8Pbbb+vdX93qm5qaiszMTCxcuBC9evXCnj17MHDgQAwaNAgHDhwAIN9nN++GW04mTpyIs2fP4tChQ3KHUmGuXbuGd955B3v37oVKpZI7HINQq9Xw9fXFxx9/DABo27Ytzp49i/DwcISEhMgcXfn78ccf8f3332Pjxo1o2bIlYmNjMXnyZDg7O1fL+lK+R48eYciQIRBCYOXKlXKHUyGOHz+OL774AidOnIBCoZA7HINQq9UAgP79++Pdd98FAHh7e+Ovv/5CeHg4unbtKltsbAEpB5MmTcKOHTuwf/9+1K9fX1vu6OiI3NxcpKWl6RyfkpICR0dH7TFPjjTWvNYcU1kcP34cqampaNeuHUxMTGBiYoIDBw7gyy+/hImJCRwcHKpVfQHAyckJLVq00Clr3ry5dvS4JmZ9dSpY59TUVJ39jx8/xr///lvp6jx9+nRtK0irVq0wYsQIvPvuu9oWr+pW3yeVV/2q2vtck3wkJiZi7969Ordlr071PXjwIFJTU9GgQQPtZ1hiYiKmTp0KNzc3ANWrvgBgZ2cHExOTp36OyfHZzQSkDIQQmDRpErZt24bff/8d7u7uOvt9fHxQq1YtREVFacvi4+ORlJQEf39/AIC/vz/OnDmj84bXfAA8+YaRW/fu3XHmzBnExsZqN19fXwwfPlz7vDrVFwA6d+5caGr1hQsX0LBhQwCAu7s7HB0ddeqckZGBmJgYnTqnpaXh+PHj2mN+//13qNVq+Pn5GaAWJZednQ0jI92PBWNjY+1fUdWtvk8qr/r5+/vjjz/+wKNHj7TH7N27F82aNYOtra2BalMymuTj4sWL2LdvH+rWrauzvzrVd8SIETh9+rTOZ5izszOmT5+O3bt3A6he9QUAU1NTtG/fvtjPMdm+q0o1dJWEEEKMHz9eWFtbi+joaHHr1i3tlp2drT3mzTffFA0aNBC///67OHbsmPD39xf+/v7a/ZqpTT179hSxsbFi165dol69epV2WuqTCs6CEaL61ffIkSPCxMREfPTRR+LixYvi+++/F+bm5mLDhg3aYxYuXChsbGzE9u3bxenTp0X//v31Ttts27atiImJEYcOHRJNmjSpNNNSCwoJCREuLi7aabgRERHCzs5OzJgxQ3tMVa/v/fv3xcmTJ8XJkycFAPH555+LkydPamd9lEf90tLShIODgxgxYoQ4e/as2LRpkzA3N5dlmmZx9c3NzRX9+vUT9evXF7GxsTqfYwVnNlSX+urz5CwYIapWfYV4ep0jIiJErVq1xFdffSUuXrwoli5dKoyNjcXBgwe115Djs5sJSBkA0LutWbNGe8yDBw/EhAkThK2trTA3NxcDBw4Ut27d0rlOQkKC6N27tzAzMxN2dnZi6tSp4tGjRwauTek8mYBUx/r+8ssvwsvLSyiVSuHp6Sm++uornf1qtVrMnj1bODg4CKVSKbp37y7i4+N1jrl7964YNmyYsLS0FFZWVmLUqFHi/v37hqxGiWRkZIh33nlHNGjQQKhUKtGoUSPx/vvv63wZVfX67t+/X+//25CQECFE+dXv1KlT4rnnnhNKpVK4uLiIhQsXGqqKOoqr79WrV4v8HNu/f7/2GtWlvvroS0CqUn2FKFmdv/nmG9G4cWOhUqlEmzZtRGRkpM415PjsVghRYIlDIiIiIgPgGBAiIiIyOCYgREREZHBMQIiIiMjgmIAQERGRwTEBISIiIoNjAkJEREQGxwSEiIiIDI4JCBERERkcExAiIiIyOCYgRCSb27dvY/z48WjQoAGUSiUcHR0RGBiIP//8EwCgUCgQGRkpb5BEVCFM5A6AiGquwYMHIzc3F+vWrUOjRo2QkpKCqKgo3L17V+7QiKiC8V4wRCSLtLQ02NraIjo6Gl27di20383NDYmJidrXDRs2REJCAgBg+/btmDdvHs6fPw9nZ2eEhITg/fffh4mJ9DeVQqHAihUr8PPPPyM6OhpOTk5YvHgxXnnlFYPUjYiejl0wRCQLS0tLWFpaIjIyEjk5OYX2Hz16FACwZs0a3Lp1S/v64MGDCA4OxjvvvIPz589j1apVWLt2LT766COd82fPno3Bgwfj1KlTGD58OIYOHYq4uLiKrxgRlQhbQIhINlu3bsW4cePw4MEDtGvXDl27dsXQoUPRunVrAFJLxrZt2zBgwADtOQEBAejevTtmzZqlLduwYQNmzJiBmzdvas978803sXLlSu0xHTt2RLt27bBixQrDVI6IisUWECKSzeDBg3Hz5k38/PPP6NWrF6Kjo9GuXTusXbu2yHNOnTqF+fPna1tQLC0tMW7cONy6dQvZ2dna4/z9/XXO8/f3ZwsIUSXCQahEJCuVSoUePXqgR48emD17NsaOHYs5c+Zg5MiReo/PzMzEvHnzMGjQIL3XIqKqgS0gRFSptGjRAllZWQCAWrVqIS8vT2d/u3btEB8fj8aNGxfajIzyP9L+/vtvnfP+/vtvNG/evOIrQEQlwhYQIpLF3bt38eqrr2L06NFo3bo1ateujWPHjmHx4sXo378/AGkmTFRUFDp37gylUglbW1uEhobi5ZdfRoMGDfDKK6/AyMgIp06dwtmzZ/Hhhx9qr79lyxb4+vriueeew/fff48jR47gm2++kau6RPQEDkIlIlnk5ORg7ty52LNnDy5fvoxHjx7B1dUVr776Kv773//CzMwMv/zyC6ZMmYKEhAS4uLhop+Hu3r0b8+fPx8mTJ1GrVi14enpi7NixGDduHABpEOry5csRGRmJP/74A05OTli0aBGGDBkiY42JqCAmIERU7eibPUNElQvHgBAREZHBMQEhIiIig+MgVCKqdtizTFT5sQWEiIiIDI4JCBERERkcExAiIiIyOCYgREREZHBMQIiIiMjgmIAQERGRwTEBISIiIoNjAkJEREQG93+VezOIL10tJQAAAABJRU5ErkJggg==\n"
},
"metadata": {}
}
]
},
{
"cell_type": "markdown",
"source": [
"#### **Evaluation**\n",
"\n",
"We can put our trained model into a Transformers pipeline to be able to make new predictions conveniently"
],
"metadata": {
"id": "UhZjSRlSML0a"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 141
},
"id": "hnct-N6dLvOd",
"outputId": "0b321308-9bac-4263-e4b1-66c70806acd9"
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"text/html": [
"\n",
" \n",
" \n",
"
\n",
" [34/34 00:04]\n",
"