diff --git a/learn/security/it-threat-detection.ipynb b/learn/security/it-threat-detection.ipynb
index 7a06c0a7..37f037bc 100644
--- a/learn/security/it-threat-detection.ipynb
+++ b/learn/security/it-threat-detection.ipynb
@@ -1,1822 +1,1793 @@
{
- "cells": [
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "[](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/security/it-threat-detection.ipynb) [](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/security/it-threat-detection.ipynb)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "b4Yv1jeGywpL"
- },
- "source": [
- "## IT Threat Detection With Similarity Search\n",
- "\n",
- "This notebook shows how to use Pinecone's similarity search as a service to build an application for detecting rare events. Such application is common in cyber-security and fraud detection domains wherein only a tiny fraction of the events are malicious. \n",
- "\n",
- "Here we will build a network intrusion detector. Network intrusion detection systems monitor incoming and outgoing network traffic flow, raising alarms whenever a threat is detected. Here we use a deep-learning model and similarity search in detecting and classifying network intrusion traffic.\n",
- "\n",
- "We will start by indexing a set of labeled traffic events in the form of vector embeddings. Each event is either benign or malicious. The vector embeddings are rich, mathematical representations of the network traffic events. It is making it possible to determine how similar the network events are to one another using similarity-search algorithms built into Pinecone. Here we will transform network traffic events into vectors using a deep learning model from recent academic work.\n",
- "\n",
- "\n",
- "We will then take some new (unseen) network events and search through the index to find the most similar matches, along with their labels. In such a way, we will propagate the matched labels to classify the unseen events as benign or malicious. Mind that the intrusion detection task is a challenging classification task because malicious events are sporadic. The similarity search service helps us sift the most relevant historical labeled events. That way, we identify these rare events while keeping a low rate of false alarms. \n"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "1OUSClcPhU4j"
- },
- "source": [
- "## Setting up Pinecone\n",
- "\n",
- "We will first install and initialize Pinecone. You can get your [API Key here](https://app.pinecone.io)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "251n1avKzCrm"
- },
- "outputs": [],
- "source": [
- "!pip install -qU pinecone-client==3.1.0"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "_cGTuY8dywpV",
- "outputId": "da0b57bf-bde2-401f-e502-6ad80f7fc6cc"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[]"
- ]
- },
- "execution_count": 1,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import os\n",
- "from pinecone import Pinecone\n",
- "\n",
- "# initialize connection to pinecone (get API key at app.pinecone.io)\n",
- "api_key = os.environ.get('PINECONE_API_KEY') or 'PINECONE_API_KEY'\n",
- "\n",
- "# configure client\n",
- "pc = Pinecone(api_key=api_key)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "a7ysNAlrjD_k"
- },
- "source": [
- "## Installing other dependencies"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "r3g-b61IywpQ"
- },
- "outputs": [],
- "source": [
- "!pip install -qU pip python-dateutil tensorflow==2.5 keras==2.4.0 scikit-learn matplotlib==3.1.0 seaborn"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "0uuHcP-WywpQ"
- },
- "outputs": [],
- "source": [
- "from collections import Counter\n",
- "import matplotlib.pyplot as plt\n",
- "import pandas as pd\n",
- "import seaborn as sns\n",
- "from tensorflow import keras\n",
- "from tensorflow.keras.models import Model\n",
- "import tensorflow.keras.backend as K\n",
- "from sklearn.metrics import accuracy_score, precision_score, recall_score\n",
- "from sklearn.metrics import confusion_matrix"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "lEwqF1osp83o"
- },
- "source": [
- "We will use some of the code from a recent [academic work](https://github.com/rambasnet/DeepLearning-IDS). Let's clone the repository that we will use to prepare data."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "FUDeovNiywpT"
- },
- "outputs": [],
- "source": [
- "!git clone -q https://github.com/rambasnet/DeepLearning-IDS.git "
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "mc4ERmwniO1H"
- },
- "source": [
- "## Define a New Pinecone Index"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from pinecone import ServerlessSpec\n",
- "\n",
- "cloud = os.environ.get('PINECONE_CLOUD') or 'aws'\n",
- "region = os.environ.get('PINECONE_REGION') or 'us-east-1'\n",
- "\n",
- "spec = ServerlessSpec(cloud=cloud, region=region)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Create the index:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "9_TIgYxBywpV"
- },
- "outputs": [],
- "source": [
- "# Pick a name for the new service\n",
- "index_name = 'it-threats'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import time\n",
- "\n",
- "# check if index already exists (it shouldn't if this is first time)\n",
- "if index_name not in pc.list_indexes().names():\n",
- " # if does not exist, create index\n",
- " pc.create_index(\n",
- " index_name,\n",
- " dimension=128,\n",
- " metric='euclidean',\n",
- " spec=spec\n",
- " )\n",
- " # wait for index to be initialized\n",
- " while not pc.describe_index(index_name).status['ready']:\n",
- " time.sleep(1)\n",
- "\n",
- "# connect to index\n",
- "index = pc.Index(index_name)\n",
- "# view index stats\n",
- "index.describe_index_stats()"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "IOP9jCo5ywpX"
- },
- "source": [
- "## Upload\n",
- "Here we transform network events into vector embeddings, then upload them into Pinecone's vector index. "
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "PzkJJd8ZYTNM"
- },
- "source": [
- "### Prepare Data"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "-N0bh6dUYTNN"
- },
- "source": [
- "The datasets we use consist of benign (normal) network traffic and malicious traffic\n",
- "generated from several different network attacks. We will focus on web attacks only. \n",
- "\n",
- "The web attack category consists of three common attacks: \n",
- "- Cross-site scripting (BruteForce-XSS), \n",
- "- SQL-Injection (SQL-Injection), \n",
- "- Brute force administrative and user passwords (BruteForce-Web)\n",
- "\n",
- "The original data was recorded over two days."
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "vuOLNAehYTNN"
- },
- "source": [
- "**Download data for 22-02-2018 and 23-02-2018**\n",
- "\n",
- "\n"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "pCrwPkDJYTNO"
- },
- "source": [
- "Files should be downloaded to the current directory. We will be using one date for training and generating vectors, and another one for testing."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "HshKyF0KywpR",
- "outputId": "d5b3ceee-b584-47e2-a428-43e5c835968f"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Thursday-22-02-2018 100%[===================>] 364.91M 3.07MB/s in 2m 6s \n",
- "Friday-23-02-2018_T 100%[===================>] 365.10M 3.07MB/s in 1m 53s \n"
- ]
- }
- ],
- "source": [
- "!wget \"https://cse-cic-ids2018.s3.ca-central-1.amazonaws.com/Processed%20Traffic%20Data%20for%20ML%20Algorithms/Thursday-22-02-2018_TrafficForML_CICFlowMeter.csv\" -q --show-progress\n",
- "!wget \"https://cse-cic-ids2018.s3.ca-central-1.amazonaws.com/Processed%20Traffic%20Data%20for%20ML%20Algorithms/Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\" -q --show-progress"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "gr-hay4rfk0d"
- },
- "source": [
- "Let's look at the data events first."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "bSSZ4YCZywpT",
- "outputId": "27d111b0-c0c1-4694-9e07-65965d2014f1"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "Benign 1048009\n",
- "Brute Force -Web 362\n",
- "Brute Force -XSS 151\n",
- "SQL Injection 53\n",
- "Name: Label, dtype: int64"
- ]
- },
- "execution_count": 60,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "data = pd.read_csv('Friday-23-02-2018_TrafficForML_CICFlowMeter.csv')\n",
- "data.Label.value_counts()"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "vYl82iXeywpT"
- },
- "source": [
- "**Clean the data** using a python script from the cloned repository."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "y-trCZOcywpT",
- "outputId": "c51b5e77-4e66-47a3-c4f6-d492a7dfaef8"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "cleaning Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\n",
- "total rows read = 1048576\n",
- "all done writing 1042868 rows; dropped 5708 rows\n"
- ]
- }
- ],
- "source": [
- "!python DeepLearning-IDS/data_cleanup.py \"Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\" \"result23022018\""
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "oKHL8HCjYTNQ"
- },
- "source": [
- "Load the file that you got from the previous step."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 278
- },
- "id": "uzH6_tBpywpU",
- "outputId": "017263dd-8e2a-45a4-9693-5b5b763197ce"
- },
- "outputs": [
- {
- "data": {
- "text/html": [
- "\n",
- "\n",
- "
\n",
- " \n",
- " \n",
- " Dst Port \n",
- " Protocol \n",
- " Timestamp \n",
- " Flow Duration \n",
- " Tot Fwd Pkts \n",
- " Tot Bwd Pkts \n",
- " TotLen Fwd Pkts \n",
- " TotLen Bwd Pkts \n",
- " Fwd Pkt Len Max \n",
- " Fwd Pkt Len Min \n",
- " ... \n",
- " Fwd Seg Size Min \n",
- " Active Mean \n",
- " Active Std \n",
- " Active Max \n",
- " Active Min \n",
- " Idle Mean \n",
- " Idle Std \n",
- " Idle Max \n",
- " Idle Min \n",
- " Label \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " 0 \n",
- " 22 \n",
- " 6 \n",
- " 1.519374e+09 \n",
- " 1532698 \n",
- " 11 \n",
- " 11 \n",
- " 1179 \n",
- " 1969 \n",
- " 648 \n",
- " 0 \n",
- " ... \n",
- " 32 \n",
- " 0.0 \n",
- " 0.0 \n",
- " 0 \n",
- " 0 \n",
- " 0.0 \n",
- " 0.000000e+00 \n",
- " 0 \n",
- " 0 \n",
- " Benign \n",
- " \n",
- " \n",
- " 1 \n",
- " 500 \n",
- " 17 \n",
- " 1.519374e+09 \n",
- " 117573855 \n",
- " 3 \n",
- " 0 \n",
- " 1500 \n",
- " 0 \n",
- " 500 \n",
- " 500 \n",
- " ... \n",
- " 8 \n",
- " 0.0 \n",
- " 0.0 \n",
- " 0 \n",
- " 0 \n",
- " 58786927.5 \n",
- " 2.375324e+07 \n",
- " 75583006 \n",
- " 41990849 \n",
- " Benign \n",
- " \n",
- " \n",
- " 2 \n",
- " 500 \n",
- " 17 \n",
- " 1.519374e+09 \n",
- " 117573848 \n",
- " 3 \n",
- " 0 \n",
- " 1500 \n",
- " 0 \n",
- " 500 \n",
- " 500 \n",
- " ... \n",
- " 8 \n",
- " 0.0 \n",
- " 0.0 \n",
- " 0 \n",
- " 0 \n",
- " 58786924.0 \n",
- " 2.375325e+07 \n",
- " 75583007 \n",
- " 41990841 \n",
- " Benign \n",
- " \n",
- " \n",
- " 3 \n",
- " 22 \n",
- " 6 \n",
- " 1.519374e+09 \n",
- " 1745392 \n",
- " 11 \n",
- " 11 \n",
- " 1179 \n",
- " 1969 \n",
- " 648 \n",
- " 0 \n",
- " ... \n",
- " 32 \n",
- " 0.0 \n",
- " 0.0 \n",
- " 0 \n",
- " 0 \n",
- " 0.0 \n",
- " 0.000000e+00 \n",
- " 0 \n",
- " 0 \n",
- " Benign \n",
- " \n",
- " \n",
- " 4 \n",
- " 500 \n",
- " 17 \n",
- " 1.519374e+09 \n",
- " 89483474 \n",
- " 6 \n",
- " 0 \n",
- " 3000 \n",
- " 0 \n",
- " 500 \n",
- " 500 \n",
- " ... \n",
- " 8 \n",
- " 4000364.0 \n",
- " 0.0 \n",
- " 4000364 \n",
- " 4000364 \n",
- " 21370777.5 \n",
- " 1.528092e+07 \n",
- " 41989576 \n",
- " 7200485 \n",
- " Benign \n",
- " \n",
- " \n",
- "
\n",
- "
5 rows \u00d7 80 columns
\n",
- "
\n",
- "\n",
- "
\n",
- " \n",
- " \n",
- " Dst Port \n",
- " Protocol \n",
- " Timestamp \n",
- " Flow Duration \n",
- " Tot Fwd Pkts \n",
- " Tot Bwd Pkts \n",
- " TotLen Fwd Pkts \n",
- " TotLen Bwd Pkts \n",
- " Fwd Pkt Len Max \n",
- " Fwd Pkt Len Min \n",
- " ... \n",
- " Fwd Seg Size Min \n",
- " Active Mean \n",
- " Active Std \n",
- " Active Max \n",
- " Active Min \n",
- " Idle Mean \n",
- " Idle Std \n",
- " Idle Max \n",
- " Idle Min \n",
- " Label \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " 0 \n",
- " 22 \n",
- " 6 \n",
- " 1.519288e+09 \n",
- " 20553406 \n",
- " 10 \n",
- " 7 \n",
- " 1063 \n",
- " 1297 \n",
- " 744 \n",
- " 0 \n",
- " ... \n",
- " 20 \n",
- " 1027304.0 \n",
- " 0.0 \n",
- " 1027304 \n",
- " 1027304 \n",
- " 1.952608e+07 \n",
- " 0.000000e+00 \n",
- " 19526080 \n",
- " 19526080 \n",
- " Benign \n",
- " \n",
- " \n",
- " 1 \n",
- " 34989 \n",
- " 6 \n",
- " 1.519288e+09 \n",
- " 790 \n",
- " 2 \n",
- " 0 \n",
- " 848 \n",
- " 0 \n",
- " 848 \n",
- " 0 \n",
- " ... \n",
- " 20 \n",
- " 0.0 \n",
- " 0.0 \n",
- " 0 \n",
- " 0 \n",
- " 0.000000e+00 \n",
- " 0.000000e+00 \n",
- " 0 \n",
- " 0 \n",
- " Benign \n",
- " \n",
- " \n",
- " 2 \n",
- " 500 \n",
- " 17 \n",
- " 1.519288e+09 \n",
- " 99745913 \n",
- " 5 \n",
- " 0 \n",
- " 2500 \n",
- " 0 \n",
- " 500 \n",
- " 500 \n",
- " ... \n",
- " 8 \n",
- " 4000203.0 \n",
- " 0.0 \n",
- " 4000203 \n",
- " 4000203 \n",
- " 3.191524e+07 \n",
- " 3.792787e+07 \n",
- " 75584115 \n",
- " 7200679 \n",
- " Benign \n",
- " \n",
- " \n",
- " 3 \n",
- " 500 \n",
- " 17 \n",
- " 1.519288e+09 \n",
- " 99745913 \n",
- " 5 \n",
- " 0 \n",
- " 2500 \n",
- " 0 \n",
- " 500 \n",
- " 500 \n",
- " ... \n",
- " 8 \n",
- " 4000189.0 \n",
- " 0.0 \n",
- " 4000189 \n",
- " 4000189 \n",
- " 3.191524e+07 \n",
- " 3.792788e+07 \n",
- " 75584130 \n",
- " 7200693 \n",
- " Benign \n",
- " \n",
- " \n",
- " 4 \n",
- " 500 \n",
- " 17 \n",
- " 1.519288e+09 \n",
- " 89481361 \n",
- " 6 \n",
- " 0 \n",
- " 3000 \n",
- " 0 \n",
- " 500 \n",
- " 500 \n",
- " ... \n",
- " 8 \n",
- " 4000554.0 \n",
- " 0.0 \n",
- " 4000554 \n",
- " 4000554 \n",
- " 2.137020e+07 \n",
- " 1.528109e+07 \n",
- " 41990741 \n",
- " 7200848 \n",
- " Benign \n",
- " \n",
- " \n",
- "
\n",
- "
5 rows \u00d7 80 columns
\n",
- "
"
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
+ },
+ "execution_count": null,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import os\n",
+ "from getpass import getpass\n",
+ "\n",
+ "from pinecone import Pinecone\n",
+ "\n",
+ "# initialize connection to pinecone (get API key at app.pinecone.io)\n",
+ "api_key = os.environ.get(\"PINECONE_API_KEY\") or getpass(\"Enter your Pinecone API key: \")\n",
+ "\n",
+ "# configure client\n",
+ "pc = Pinecone(api_key=api_key)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "a7ysNAlrjD_k"
+ },
+ "source": [
+ "## Installing other dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "r3g-b61IywpQ"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -qU pip python-dateutil tensorflow scikit-learn matplotlib seaborn"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "0uuHcP-WywpQ"
+ },
+ "outputs": [],
+ "source": [
+ "from collections import Counter\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "import pandas as pd\n",
+ "import seaborn as sns\n",
+ "import tensorflow.keras.backend as K\n",
+ "from sklearn.metrics import (\n",
+ " accuracy_score,\n",
+ " confusion_matrix,\n",
+ " precision_score,\n",
+ " recall_score,\n",
+ ")\n",
+ "from tensorflow import keras\n",
+ "from tensorflow.keras.models import Model"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "lEwqF1osp83o"
+ },
+ "source": [
+ "We will use some of the code from an [academic work on deep learning for intrusion detection](https://github.com/rambasnet/DeepLearning-IDS). Let's clone the repository that we will use to prepare data."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "FUDeovNiywpT"
+ },
+ "outputs": [],
+ "source": [
+ "!git clone -q https://github.com/rambasnet/DeepLearning-IDS.git "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "mc4ERmwniO1H"
+ },
+ "source": [
+ "## Define a New Pinecone Index"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/guides/get-started/understanding-organizations#regions)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from pinecone import ServerlessSpec\n",
+ "\n",
+ "cloud = os.environ.get(\"PINECONE_CLOUD\") or \"aws\"\n",
+ "region = os.environ.get(\"PINECONE_REGION\") or \"us-east-1\"\n",
+ "\n",
+ "spec = ServerlessSpec(cloud=cloud, region=region)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Create the index:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "9_TIgYxBywpV"
+ },
+ "outputs": [],
+ "source": [
+ "# Pick a name for the new service\n",
+ "index_name = \"it-threats\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "\n",
+ "# check if index already exists (it shouldn't if this is first time)\n",
+ "if not pc.has_index(index_name):\n",
+ " # if does not exist, create index\n",
+ " pc.create_index(name=index_name, dimension=128, metric=\"euclidean\", spec=spec)\n",
+ " # wait for index to be initialized\n",
+ " while not pc.describe_index(index_name).status[\"ready\"]:\n",
+ " time.sleep(1)\n",
+ "\n",
+ "# connect to index\n",
+ "index = pc.Index(index_name)\n",
+ "# view index stats\n",
+ "index.describe_index_stats()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IOP9jCo5ywpX"
+ },
+ "source": [
+ "## Upload\n",
+ "Here we transform network events into vector embeddings, then upload them into Pinecone's vector index. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "PzkJJd8ZYTNM"
+ },
+ "source": [
+ "### Prepare Data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-N0bh6dUYTNN"
+ },
+ "source": [
+ "The datasets we use consist of benign (normal) network traffic and malicious traffic\n",
+ "generated from several different network attacks. We will focus on web attacks only. \n",
+ "\n",
+ "The web attack category consists of three common attacks: \n",
+ "- Cross-site scripting (BruteForce-XSS), \n",
+ "- SQL-Injection (SQL-Injection), \n",
+ "- Brute force administrative and user passwords (BruteForce-Web)\n",
+ "\n",
+ "The original data was recorded over two days."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "vuOLNAehYTNN"
+ },
+ "source": [
+ "**Download data for 22-02-2018 and 23-02-2018**\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pCrwPkDJYTNO"
+ },
+ "source": [
+ "Files should be downloaded to the current directory. We will be using one date for training and generating vectors, and another one for testing."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "HshKyF0KywpR",
+ "outputId": "d5b3ceee-b584-47e2-a428-43e5c835968f"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Thursday-22-02-2018 100%[===================>] 364.91M 3.07MB/s in 2m 6s \n",
+ "Friday-23-02-2018_T 100%[===================>] 365.10M 3.07MB/s in 1m 53s \n"
+ ]
+ }
+ ],
+ "source": [
+ "!wget \"https://cse-cic-ids2018.s3.ca-central-1.amazonaws.com/Processed%20Traffic%20Data%20for%20ML%20Algorithms/Thursday-22-02-2018_TrafficForML_CICFlowMeter.csv\" -q --show-progress\n",
+ "!wget \"https://cse-cic-ids2018.s3.ca-central-1.amazonaws.com/Processed%20Traffic%20Data%20for%20ML%20Algorithms/Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\" -q --show-progress"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gr-hay4rfk0d"
+ },
+ "source": [
+ "Let's look at the data events first."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "bSSZ4YCZywpT",
+ "outputId": "27d111b0-c0c1-4694-9e07-65965d2014f1"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Benign 1048009\n",
+ "Brute Force -Web 362\n",
+ "Brute Force -XSS 151\n",
+ "SQL Injection 53\n",
+ "Name: Label, dtype: int64"
+ ]
+ },
+ "execution_count": null,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "data = pd.read_csv(\"Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\")\n",
+ "data.Label.value_counts()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "vYl82iXeywpT"
+ },
+ "source": [
+ "**Clean the data** using a python script from the cloned repository."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "y-trCZOcywpT",
+ "outputId": "c51b5e77-4e66-47a3-c4f6-d492a7dfaef8"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "cleaning Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\n",
+ "total rows read = 1048576\n",
+ "all done writing 1042868 rows; dropped 5708 rows\n"
+ ]
+ }
+ ],
+ "source": [
+ "!python DeepLearning-IDS/data_cleanup.py \"Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\" \"result23022018\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "oKHL8HCjYTNQ"
+ },
+ "source": [
+ "Load the file that you got from the previous step."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 278
+ },
+ "id": "uzH6_tBpywpU",
+ "outputId": "017263dd-8e2a-45a4-9693-5b5b763197ce"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " Dst Port \n",
+ " Protocol \n",
+ " Timestamp \n",
+ " Flow Duration \n",
+ " Tot Fwd Pkts \n",
+ " Tot Bwd Pkts \n",
+ " TotLen Fwd Pkts \n",
+ " TotLen Bwd Pkts \n",
+ " Fwd Pkt Len Max \n",
+ " Fwd Pkt Len Min \n",
+ " ... \n",
+ " Fwd Seg Size Min \n",
+ " Active Mean \n",
+ " Active Std \n",
+ " Active Max \n",
+ " Active Min \n",
+ " Idle Mean \n",
+ " Idle Std \n",
+ " Idle Max \n",
+ " Idle Min \n",
+ " Label \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " 0 \n",
+ " 22 \n",
+ " 6 \n",
+ " 1.519374e+09 \n",
+ " 1532698 \n",
+ " 11 \n",
+ " 11 \n",
+ " 1179 \n",
+ " 1969 \n",
+ " 648 \n",
+ " 0 \n",
+ " ... \n",
+ " 32 \n",
+ " 0.0 \n",
+ " 0.0 \n",
+ " 0 \n",
+ " 0 \n",
+ " 0.0 \n",
+ " 0.000000e+00 \n",
+ " 0 \n",
+ " 0 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ " 1 \n",
+ " 500 \n",
+ " 17 \n",
+ " 1.519374e+09 \n",
+ " 117573855 \n",
+ " 3 \n",
+ " 0 \n",
+ " 1500 \n",
+ " 0 \n",
+ " 500 \n",
+ " 500 \n",
+ " ... \n",
+ " 8 \n",
+ " 0.0 \n",
+ " 0.0 \n",
+ " 0 \n",
+ " 0 \n",
+ " 58786927.5 \n",
+ " 2.375324e+07 \n",
+ " 75583006 \n",
+ " 41990849 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ " 2 \n",
+ " 500 \n",
+ " 17 \n",
+ " 1.519374e+09 \n",
+ " 117573848 \n",
+ " 3 \n",
+ " 0 \n",
+ " 1500 \n",
+ " 0 \n",
+ " 500 \n",
+ " 500 \n",
+ " ... \n",
+ " 8 \n",
+ " 0.0 \n",
+ " 0.0 \n",
+ " 0 \n",
+ " 0 \n",
+ " 58786924.0 \n",
+ " 2.375325e+07 \n",
+ " 75583007 \n",
+ " 41990841 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ " 3 \n",
+ " 22 \n",
+ " 6 \n",
+ " 1.519374e+09 \n",
+ " 1745392 \n",
+ " 11 \n",
+ " 11 \n",
+ " 1179 \n",
+ " 1969 \n",
+ " 648 \n",
+ " 0 \n",
+ " ... \n",
+ " 32 \n",
+ " 0.0 \n",
+ " 0.0 \n",
+ " 0 \n",
+ " 0 \n",
+ " 0.0 \n",
+ " 0.000000e+00 \n",
+ " 0 \n",
+ " 0 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ " 4 \n",
+ " 500 \n",
+ " 17 \n",
+ " 1.519374e+09 \n",
+ " 89483474 \n",
+ " 6 \n",
+ " 0 \n",
+ " 3000 \n",
+ " 0 \n",
+ " 500 \n",
+ " 500 \n",
+ " ... \n",
+ " 8 \n",
+ " 4000364.0 \n",
+ " 0.0 \n",
+ " 4000364 \n",
+ " 4000364 \n",
+ " 21370777.5 \n",
+ " 1.528092e+07 \n",
+ " 41989576 \n",
+ " 7200485 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ "
\n",
+ "
5 rows × 80 columns
\n",
+ "
\n",
- "\n",
- "
\n",
- " \n",
- " \n",
- " type \n",
- " accuracy \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " 0 \n",
- " Benign \n",
- " 1.00 \n",
- " \n",
- " \n",
- " 1 \n",
- " Attack \n",
- " 0.58 \n",
- " \n",
- " \n",
- "
\n",
- "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " Dst Port \n",
+ " Protocol \n",
+ " Timestamp \n",
+ " Flow Duration \n",
+ " Tot Fwd Pkts \n",
+ " Tot Bwd Pkts \n",
+ " TotLen Fwd Pkts \n",
+ " TotLen Bwd Pkts \n",
+ " Fwd Pkt Len Max \n",
+ " Fwd Pkt Len Min \n",
+ " ... \n",
+ " Fwd Seg Size Min \n",
+ " Active Mean \n",
+ " Active Std \n",
+ " Active Max \n",
+ " Active Min \n",
+ " Idle Mean \n",
+ " Idle Std \n",
+ " Idle Max \n",
+ " Idle Min \n",
+ " Label \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " 0 \n",
+ " 22 \n",
+ " 6 \n",
+ " 1.519288e+09 \n",
+ " 20553406 \n",
+ " 10 \n",
+ " 7 \n",
+ " 1063 \n",
+ " 1297 \n",
+ " 744 \n",
+ " 0 \n",
+ " ... \n",
+ " 20 \n",
+ " 1027304.0 \n",
+ " 0.0 \n",
+ " 1027304 \n",
+ " 1027304 \n",
+ " 1.952608e+07 \n",
+ " 0.000000e+00 \n",
+ " 19526080 \n",
+ " 19526080 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ " 1 \n",
+ " 34989 \n",
+ " 6 \n",
+ " 1.519288e+09 \n",
+ " 790 \n",
+ " 2 \n",
+ " 0 \n",
+ " 848 \n",
+ " 0 \n",
+ " 848 \n",
+ " 0 \n",
+ " ... \n",
+ " 20 \n",
+ " 0.0 \n",
+ " 0.0 \n",
+ " 0 \n",
+ " 0 \n",
+ " 0.000000e+00 \n",
+ " 0.000000e+00 \n",
+ " 0 \n",
+ " 0 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ " 2 \n",
+ " 500 \n",
+ " 17 \n",
+ " 1.519288e+09 \n",
+ " 99745913 \n",
+ " 5 \n",
+ " 0 \n",
+ " 2500 \n",
+ " 0 \n",
+ " 500 \n",
+ " 500 \n",
+ " ... \n",
+ " 8 \n",
+ " 4000203.0 \n",
+ " 0.0 \n",
+ " 4000203 \n",
+ " 4000203 \n",
+ " 3.191524e+07 \n",
+ " 3.792787e+07 \n",
+ " 75584115 \n",
+ " 7200679 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ " 3 \n",
+ " 500 \n",
+ " 17 \n",
+ " 1.519288e+09 \n",
+ " 99745913 \n",
+ " 5 \n",
+ " 0 \n",
+ " 2500 \n",
+ " 0 \n",
+ " 500 \n",
+ " 500 \n",
+ " ... \n",
+ " 8 \n",
+ " 4000189.0 \n",
+ " 0.0 \n",
+ " 4000189 \n",
+ " 4000189 \n",
+ " 3.191524e+07 \n",
+ " 3.792788e+07 \n",
+ " 75584130 \n",
+ " 7200693 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ " 4 \n",
+ " 500 \n",
+ " 17 \n",
+ " 1.519288e+09 \n",
+ " 89481361 \n",
+ " 6 \n",
+ " 0 \n",
+ " 3000 \n",
+ " 0 \n",
+ " 500 \n",
+ " 500 \n",
+ " ... \n",
+ " 8 \n",
+ " 4000554.0 \n",
+ " 0.0 \n",
+ " 4000554 \n",
+ " 4000554 \n",
+ " 2.137020e+07 \n",
+ " 1.528109e+07 \n",
+ " 41990741 \n",
+ " 7200848 \n",
+ " Benign \n",
+ " \n",
+ " \n",
+ "
\n",
+ "
5 rows × 80 columns
\n",
+ "
"
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "# Create confusion matrix\n",
- "conf_matrix = confusion_matrix(y_true, y_pred_model)\n",
- "\n",
- "# Show confusion matrix\n",
- "ax = plt.subplot()\n",
- "sns.heatmap(conf_matrix, annot=True, ax = ax, cmap='Blues', fmt='g', cbar=False)\n",
- "\n",
- "# Add labels, title and ticks\n",
- "ax.set_xlabel('Predicted')\n",
- "ax.set_ylabel('Acctual')\n",
- "ax.set_title('Confusion Matrix')\n",
- "ax.xaxis.set_ticklabels(['Benign', 'Attack'])\n",
- "ax.yaxis.set_ticklabels(['Benign', 'Attack'])"
+ "data": {
+ "text/plain": [
+ "[Text(0, 0.5, 'Benign'), Text(0, 1.5, 'Attack')]"
]
+ },
+ "execution_count": null,
+ "metadata": {},
+ "output_type": "execute_result"
},
{
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "VJRwvXOvYtBL",
- "outputId": "9d45426b-6bd8-4374-dcf8-9ffecedd7176"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Accuracy: 0.871\n",
- "Precision: 1.000\n",
- "Recall: 0.287\n"
- ]
- }
- ],
- "source": [
- "# Calculate accuracy\n",
- "acc = accuracy_score(y_true, y_pred_model, normalize=True, sample_weight=None)\n",
- "precision = precision_score(y_true, y_pred_model)\n",
- "recall = recall_score(y_true, y_pred_model)\n",
- "\n",
- "print(f\"Accuracy: {acc:.3f}\")\n",
- "print(f\"Precision: {precision:.3f}\")\n",
- "print(f\"Recall: {recall:.3f}\")"
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEWCAYAAABliCz2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAbKUlEQVR4nO3debxVdb3/8debc5RBQEZxKBAcILUUtX5eS1IzE6+aM6JlGE455HXWMBwoLbNBrzl3zTmnLEwihyupZDeQxCEpUqIUQTgCKjId+Pz+WOvQ5sg5Z4t7nQ3n+34+HufBXsNe38+Gzfus/V3f/V2KCMzMrO1rV+0CzMysdTjwzcwS4cA3M0uEA9/MLBEOfDOzRDjwzcwS4cC3NkNSR0kPS1oo6f6PcJxjJD1aydqqQdJvJX2t2nXYusOBb61O0tGSJkt6T9KbeTB9rgKHPhzoA/SMiCPW9iARcVdE7FuBelYjaU9JIemXjdbvmK+fUOZxLpF0Z0v7RcTQiLhtLcu1NsiBb61K0lnAT4DLycK5L3Ad8OUKHL4f8LeIqK/AsYoyF9hdUs+SdV8D/lapBpTx/237AL8prNVI2hi4DDg1In4ZEYsiYnlEPBwR5+b7tJf0E0mz8p+fSGqfb9tT0uuSzpb0Vv7p4Lh826XAaGBY/slhZOMzYUlb5mfStfnyCEmvSXpX0gxJx5Ssf6bkebtLmpR3FU2StHvJtgmSxkiamB/nUUm9mvlrWAb8Cjgqf34NcCRwV6O/q6sl/UvSO5Kek7RHvn4/4Fslr3NqSR3flTQReB8YkK87Pt9+vaQHSo7/fUlPSFLZ/4C23nPgW2v6D6AD8FAz+4wCdgN2AnYEPgNcVLJ9U2BjYAtgJPBTSd0j4mKyTw33RkTniPhZc4VI2gi4BhgaEV2A3YHn17BfD+CRfN+ewI+ARxqdoR8NHAdsAmwInNNc28DtwLH54y8BLwOzGu0ziezvoAdwN3C/pA4RMb7R69yx5DlfBU4EugAzGx3vbOBT+S+zPcj+7r4WnlslKQ58a009gXktdLkcA1wWEW9FxFzgUrIga7A83748IsYB7wED17KelcAOkjpGxJsR8fIa9vlPYHpE3BER9RFxDzANOLBkn1sj4m8RsRi4jyyomxQRfwB6SBpIFvy3r2GfOyOiLm/zh0B7Wn6dP4+Il/PnLG90vPeBr5D9wroTOD0iXm/heNbGOPCtNdUBvRq6VJqwOaufnc7M1606RqNfGO8DnT9sIRGxCBgGnAy8KekRSYPKqKehpi1KlmevRT13AKcBe7GGTzx5t9UreTfSArJPNc11FQH8q7mNEfEn4DVAZL+YLDEOfGtNzwJLgIOb2WcW2cXXBn35YHdHuRYBnUqWNy3dGBG/i4gvApuRnbXfXEY9DTW9sZY1NbgDOAUYl599r5J3uZxP1rffPSK6AQvJghqgqW6YZrtnJJ1K9klhFnDe2pdu6ysHvrWaiFhIdmH1p5IOltRJ0gaShkq6Mt/tHuAiSb3zi5+jybog1sbzwBBJffMLxhc2bJDUR9JBeV/+UrKuoRVrOMY4YNt8KGmtpGHAdsBv1rImACJiBvB5smsWjXUB6slG9NRKGg10Ldk+B9jyw4zEkbQt8B2ybp2vAudJarbrydoeB761qoj4EXAW2YXYuWTdEKeRjVyBLJQmAy8ALwJT8nVr09ZjwL35sZ5j9ZBuR3YhcxbwNln4nrKGY9QBB+T71pGdGR8QEfPWpqZGx34mItb06eV3wG/JhmrOJPtUVNpd0/ClsjpJU1pqJ+9CuxP4fkRMjYjpZCN97mgYAWVpkC/Sm5mlwWf4ZmaJcOCbmSXCgW9mlggHvplZIpr7AkxVdRx8mq8m2zpp/qRrq12CWZM61NLk/Eg+wzczS4QD38wsEQ58M7NEOPDNzBLhwDczS4QD38wsEQ58M7NEOPDNzBLhwDczS4QD38wsEQ58M7NEOPDNzBLhwDczS4QD38wsEQ58M7NEOPDNzBLhwDczS4QD38wsEQ58M7NEOPDNzBLhwDczS4QD38wsEQ58M7NEOPDNzBLhwDczS4QD38wsEQ58M7NEOPDNzBLhwDczS4QD38wsEQ58M7NEOPDNzBLhwDczS4QD38wsEQ58M7NEOPDNzBLhwDczS4QD38wsEQ58M7NE1BbdgKQaoE9pWxHxz6LbNTOz1RUa+JJOBy4G5gAr89UBfKrIds3M7IOKPsM/AxgYEXUFt2NmZi0oug//X8DCgtswM7MyFH2G/xowQdIjwNKGlRHxo4LbNTOzRooO/H/mPxvmP2ZmViWFBn5EXFrk8c3MrHxFj9J5mGxUTqmFwGTgxohYUmT7Zmb2b0VftH0NeA+4Of95h2yI5rb5spmZtZKi+/AHR8SQkuWHJT0VEUMkvVxw22ZmVqLoM/zekvo2LOSPe+WLywpu28zMShR9hn828IykVwEB/YFTJG0E3FZw223aDRcfw9AhOzD37XfZ9YjLV63/xlGf5+RhQ6hfsZLxT7/EqKt/za7b9+Pabw8HQILv3jCOsU++QOdO7Xn8f85c9dwtNunGL8ZN4tyrHmz112PpGX3RhTz1+wn06NGTX/76N9UuJwmKaHxNtcINSO2BQWSBP63cC7UdB59WbGHruc/uvBWL3l/KLWOOXRX4Q3bdhvOP/xKHnH4Dy5bX07t7Z+bOf4+OHTZg2fIVrFixkk17deX/7r2QAfuOYsWKlasdc+Jd53HeDx9k4pRXq/GS1hvzJ11b7RLahOcmT6JTp06MuvB8B34FdahFTW0r5Axf0t4R8b+SDm20aYAkIuKXRbSbkolTXqXvZj1WW3fiEXtw1a2PsWx5PQBz578HwOIly1ft037DDVjTL/mt+vZmkx5dHPbWanbZ9dO88cbr1S4jKUV16Xwe+F/gwDVsC8CBX4Ct+23CZwdvxaWnHsiSZcu58EcP8dxfsolJP71DP2645Cv03awHIy+67QNn90futwsPPDqlGmWbWSspJPAj4uL8z+M+zPMknQicCFD7sT2p7bV9AdW1XbU17ejetRNDjr2KXbfvx51Xfp1PHHAJAJNemskuh3+Xgf37cMtlX+V3E//C0mX1q557xJd2YeRFt1epcjNrDUV/8ao9cBiwJavPh3/ZmvaPiJuAm8B9+GvjjTkL+NUTUwGY/PJMVq4MenXvzLy8awfgrzPmsGjxMrbfenOm5Gf/n9x2C2pravjzK/+qSt1m1jqKHpb5a+DLQD2wqOTHCvDwhBfY8zPbArB1303YcINa5s1/j36b96SmJvun7rtZd7bdsg8zZ/17xuoj99uF+8ZPrkrNZtZ6ih6W+bGI2K/gNpJ02xUj2GOXbejVrTN/Hz+GMTeM47ZfPcuNlxzD5Pu/xbLlKzh+9B0A7D54AOccty/L61ewcmVwxuX3Urfg3793D/vizhx8+vXVeimWqPPPOYvJk/7EggXz+eLeQ/jGqadz6GFHVLusNq3QYZmSbgL+OyJe/LDPdZeOras8LNPWZa0+LLPE54ARkmaQzYcvICLCtzg0M2tlRQf+0IKPb2ZmZSr0om1EzAQ+DuydP36/6DbNzGzNCg1fSRcD5wMX5qs2AO4ssk0zM1uzos+2DwEOIh+KGRGzgC4Ft2lmZmtQdOAvi2wYUADks2SamVkVFB3490m6Eegm6QTgcXynKzOzqij6JuZXSfoi2a0NBwKjI+KxIts0M7M1K3pYJnnAPyapF1DX0v5mZlaMQrp0JO0maYKkX0oaLOkl4CVgjiRPtWBmVgVFneFfC3wL2JhsXvyhEfFHSYOAe4DxBbVrZmZNKOqibW1EPBoR9wOzI+KPABExraD2zMysBUUFfuntlBY32uZJ0czMqqCoLp0dJb1DNllax/wx+XKHgto0M7NmFHWLw5oijmtmZmvPE5mZmSXCgW9mlggHvplZIhz4ZmaJcOCbmSXCgW9mlggHvplZIhz4ZmaJcOCbmSXCgW9mlggHvplZIhz4ZmaJcOCbmSXCgW9mlggHvplZIhz4ZmaJcOCbmSXCgW9mlogmb3EoaefmnhgRUypfjpmZFaW5e9r+sJltAexd4VrMzKxATQZ+ROzVmoWYmVmxmjvDX0XSDsB2QIeGdRFxe1FFmZlZ5bUY+JIuBvYkC/xxwFDgGcCBb2a2HilnlM7hwBeA2RFxHLAj0L7QqszMrOLKCfzFEbESqJfUFXgLGFBsWWZmVmnl9OFPltQNuBl4DngP+FOhVZmZWcW1GPgRcUr+8AZJ44GuEfFCsWWZmVmllXPRdsia1kXEU8WUZGZmRSinS+fckscdgM+Qde34i1dmZuuRcrp0DixdlvRx4MrCKjIzs0KszeRprwM7VLoQMzMrVjl9+P9NNncOZL8gdgKmFlmUmZlVXlnDMkse1wP3RMTEguoxM7OClBP43SLi6tIVks5ovM7MzNZtiojmd5CmRMTOjdb9OSIGF1nYrAXLmi/MrEqW1q+sdglmTerfq4Oa2tbcDVCGA0cD/SWNLdnUBairXHlmZtYamuvS+QPwJtCL1W+G8i7gb9qama1nmrsBykxgpqRjgFkRsQRAUkfgY8A/WqVCMzOriHLG4d8HlHZargDuL6YcMzMrSjmBXxsRyxoW8scbFleSmZkVoZzAnyvpoIYFSV8G5hVXkpmZFaGccfgnA3dJujZffh04triSzMysCOVMnvYqsJukzmTj9t8tviwzM6u0Frt0JF0uqVtEvBcR70rqLuk7rVGcmZlVTjl9+EMjYkHDQkTMB/YvriQzMytCOYFfI6l9w0I+Dr99M/ubmdk6qJyLtncCT0i6NV8+DrituJLMzKwI5Vy0vVLSC8A+gIDxQL+iCzMzs8oq945Xs8m+bXsY8AXglcIqMjOzQjQ3W+a2wFHAcLLZMe8lG5a5VyvVZmZmFdRcl8404GngwIj4O4CkM1ulKjMzq7jmunQOI+vKeVLSzZK+QNaHb2Zm66EmAz8iHoqIYcAgYAJwJtBH0vWS9m2l+szMrEJavGgbEYsi4q6IOIBsHvzngQsKr8zMzCqqxXvaVovvaWvrKt/T1tZlzd3TttxhmWZmtp5z4JuZJcKBb2aWCAe+mVkiHPhmZolw4JuZJcKBb2aWCAe+mVkiHPhmZolw4JuZJcKBb2aWCAe+mVkiHPhmZolw4JuZJcKBb2aWCAe+mVkiHPhmZolw4JuZJcKBb2aWCAe+mVkiCgt8SSPXsO57RbVnZmbNqy3w2IdLWhIRdwFIug5oX2B7ZmbWjCID/1BgrKSVwFDg7Yg4pcD2zMysGRUPfEk9ShaPB34FTAQuk9QjIt6udJtmZtYyRURlDyjNAAJQyZ8NIiIGlHOcWQuWVbYwswpZWr+y2iWYNal/rw5qalvFz/Ajon+lj2lmZh9dkaN0TpXUrWS5uyT34ZuZVUmR4/BPiIgFDQsRMR84ocD2zMysGUUGfjtJq/qSJNUAGxbYnpmZNaPIYZm/A+6TdAPZxduTgfEFtpes74/5Nn+c+BTduvfg1nseAuDnN1/HI79+kI27dQfg+G98k90+O4RXXn6RH15xKQARwYgTTmGPPb9QtdqtbZs7ZzY/GDOK+W/XIYn9v3w4Bx95DO++s5DLv30ec2bPos+mm/OtMT+gS9euLF++nGuuvIzp0/6C2rXj5DPOY8edP13tl9FmVHyUzqoDS+2Ak4AvkI3UeRS4JSJWlPN8j9Ip39Q/T6Zjx05ccemo1QK/Y8dODPvKiNX2XbJkMRvUbkBNbS118+Zy/FcO54HfPEFNbZG/+9sWj9IpX928ubxdN49tBn6C9xct4vSRRzH6ip/w2LixdOnalWFfHcm9d/yM9959h5GnnMnYB3/B9Gkvc/aoMSyYX8dFZ5/KNbfcTbt2ngWmXM2N0insbzEiVkbE9RFxeEQcFhE3lhv29uHsOHhXunbduKx9O3TouCrcly1bSpPvDLMK6NmrN9sM/AQAnTbaiI/3G0Dd3Ld49ukn2WfoQQDsM/Qg/vDUkwD88x+vsdOu/w+Abt170rlzF6ZPe7k6xbdBRY7S2UbSA5L+Ium1hp+i2rMPeuiBexh5zKF8f8y3efedhavW/+WlFxhx1MF8/ehDOfOC0T67t1Yx+803eHX6NAZu/0kWzH+bnr16A9kvhYULsu9jDth6W559egIr6uuZPet1pv/1FebOmVPNstuUIj8n3QpcD9QDewG3A3c09wRJJ0qaLGnynT+/pcDS2r6DDj2Sux4cx813PEDPXr257uqrVm3bbodP8fNf/Iobbv0Fd992C8uWLq1ipZaCxe+/z3dGnc1J3zyXjTbq3OR+X/rPg+nduw+njzyaG67+AdvtsCM1tTWtWGnbVuSpXceIeEKSImImcImkp4GLm3pCRNwE3ATuw/+oevTsterxAV8+jAvPPu0D+/TrP4AOHToy47W/M/AT27dmeZaQ+vrljBl1Fnvtuz+f23MfALp170HdvLn07NWbunlz2bhbNiNLTW0tJ51x7qrnnnnSsWz+sb5VqbstKvIMf0l+4Xa6pNMkHQJsUmB7VqJu3txVj5/+/RP0H7A1AG/Oep0V9fUAzH5zFv/65z/YdLPNq1KjtX0RwY+vuIS+/QZw2FHHrlq/2+f25PHfjgXg8d+O5T/22AvIBhUsWfw+AFP+9Cw1NTX0679V6xfeRhU5SufTwCtAN2AM0BW4MiL+r5zn+wy/fGMuOo/np0xi4YIFdO/RgxEnnsrU5ybx9+nTkMSmm23BWReMpmev3jw67mHuvv1n1NbW0q5dO44deRKf+7yHZX4YHqVTvpemTuGcU45jy622oZ2y88sRJ53OoO0/yeXfPpe35sxmkz6bMuo7V9Gl68bMfvMNRp35Ddq1a0fP3ptw5oWX0GdTn5B8GM2N0iky8I+IiPtbWtcUB76tqxz4ti6ryrBM4MIy15mZWSsoYj78ocD+wBaSrinZ1JVsxI6ZmVVBEaN0ZgGTgYOA50rWvwucWUB7ZmZWhiLmw58KTJXUJyJuK90m6Qzg6kq3aWZmLSuyD/+oNawbUWB7ZmbWjCL68IcDRwP9JY0t2dQVmFfp9szMrDxF9OH/AXgT6AX8sGR9AMMKaM/MzMpQRB/+TGAm8B+SdiI72z8SmAE8WOn2zMysPEV06WxL1n8/HKgD7iX7gtdelW7LzMzKV0SXzjTgaeDAiPg7gCQPxzQzq7IiRukcBswGnpR0s6SGO16ZmVkVVTzwI+KhiBgGDAImkH3Zqo+k6yXtW+n2zMysPEXe4nBRRNwVEQcAHwOeBy4oqj0zM2teYbNlflSeLdPWVZ4t09Zl1Zot08zM1iEOfDOzRDjwzcwS4cA3M0uEA9/MLBEOfDOzRDjwzcwS4cA3M0uEA9/MLBEOfDOzRDjwzcwS4cA3M0uEA9/MLBEOfDOzRDjwzcwS4cA3M0uEA9/MLBEOfDOzRDjwzcwS4cA3M0uEA9/MLBEOfDOzRDjwzcwS4cA3M0uEA9/MLBEOfDOzRDjwzcwS4cA3M0uEA9/MLBEOfDOzRDjwzcwS4cA3M0uEA9/MLBEOfDOzRDjwzcwS4cA3M0uEIqLaNVgrkHRiRNxU7TrMGvN7s/X4DD8dJ1a7ALMm+L3ZShz4ZmaJcOCbmSXCgZ8O95HausrvzVbii7ZmZonwGb6ZWSIc+GZmiXDgrwckrZD0vKSpkqZI2v0jHOsySftUsj4zAEmHSApJg/LlLSUdXbJ9J0n7f4Tj/0NSr0rUmioH/vphcUTsFBE7AhcCV6ztgSJidEQ8XrnSzFYZDjwDHJUvbwkcXbJ9J2CtA98+Ogf++qcrML9hQdK5kiZJekHSpfm6LSW9IulmSS9LelRSx3zbzyUdnj/eX9I0Sc9IukbSb/L1l0j6H0kTJL0m6ZtVeJ22HpHUGfgsMJJ/B/73gD3yT6fnA5cBw/LlYZI+I+kPkv6c/zkwP1aNpKskvZi/r09v1FZHSeMlndCKL7FNqK12AVaWjpKeBzoAmwF7A0jaF9gG+AwgYKykIcA/8/XDI+IESfcBhwF3NhxQUgfgRmBIRMyQdE+jNgcBewFdgL9Kuj4ilhf5Im29djAwPiL+JultSTsDFwDnRMQBAJLmALtGxGn5cley91993s14Odn79ESgPzA439ajpJ3OwC+A2yPi9lZ7dW2Ez/DXDw1dOoOA/YDbJQnYN//5MzCFLKS3yZ8zIyKezx8/R/bxutQg4LWImJEvNw78RyJiaUTMA94C+lTyBVmbM5wsiMn/HF7GczYG7pf0EvBjYPt8/T7ADRFRDxARb5c859fArQ77teMz/PVMRDybX7jqTXZWf0VE3Fi6j6QtgaUlq1YAHRsdSi001fj5fq/YGknqSfapcwdJAdQAAYxr4aljgCcj4pD8PTuh4ZD589dkIjBU0t3hLxF9aD7DX8/kIyBqgDrgd8DX8/5TJG0haZMyDzUNGJD/RwMYVuFSLR2Hk3Wx9IuILSPi48AMYCVZl2CDdxstbwy8kT8eUbL+UeBkSbUAjbp0RpO996+r6CtIhAN//dAxv9D1PHAv8LWIWBERjwJ3A89KehF4gNX/QzUpIhYDpwDjJT0DzAEWFlO+tXHDgYcarXuQ7OJtfT6c+EzgSWC7hou2wJXAFZImkp3ENLiF7DrUC5KmsvpIH4D/AjpIurKA19KmeWqFhEnqHBHv5dcDfgpMj4gfV7suMyuGz/DTdkL+qeFlso/XN7awv5mtx3yGb2aWCJ/hm5klwoFvZpYIB76ZWSIc+NZmlcwy+pKk+yV1+gjH2rNkrqGDJF3QzL7dJJ2yFm1cIumcta3RrCUOfGvLGqak2AFYBpxculGZD/1/ICLGRsT3mtmlG9l3HMzWKQ58S8XTwNYlM4leRzb/0Mcl7Svp2fxeA/eXfHN5v4bZRIFDGw4kaYSka/PHfSQ9lH+5aKqyexV8D9gq/3Txg3y/D8xqmq8fJemvkh4HBrba34YlyYFvbV7+Ff2hwIv5qoFkUwEMBhYBFwH7RMTOwGTgrHw20ZuBA4E9gE2bOPw1wO/zexXsTPadhguAV/NPF+c2mtV0J2AXSUMk7UL2bdTBZL9QPl3hl262Gk+IZW1Zw7TSkJ3h/wzYHJgZEX/M1+8GbAdMzL5wzIbAs2Szic6IiOkAku4km7a3sb2BYwEiYgWwUFL3RvuUzmoK2RS/25BNg/FQRLyftzH2I71asxY48K0tWxwRO5WuyEN9Uekq4LGIGN5ov51oesbGD6upWU3/q4JtmLXIXTqWuj8Cn5W0NYCkTpK2JZtNtL+krfL9mprf/QngG/lza/KbejSeFbKpWU2fAg7J7+DUhaz7yKwwDnxLWkTMJZua9x5JL5D9AhgUEUvIunAeyS/azmziEGcAe+WzlT4HbB8RdWRdRC9J+kFTs5pGxBSy2U+fJ5td8unCXqgZnkvHzCwZPsM3M0uEA9/MLBEOfDOzRDjwzcwS4cA3M0uEA9/MLBEOfDOzRPx/QQOreDywqpcAAAAASUVORK5CYII=",
+ "text/plain": [
+ ""
]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 112
- },
- "id": "TFbfVi_-W6GT",
- "outputId": "68facf43-ac41-44de-d59a-a489627cc893"
- },
- "outputs": [
- {
- "data": {
- "text/html": [
- "\n",
- "\n",
- "
\n",
- " \n",
- " \n",
- " type \n",
- " accuracy \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " 0 \n",
- " Benign \n",
- " 1.00 \n",
- " \n",
- " \n",
- " 1 \n",
- " Attack \n",
- " 0.29 \n",
- " \n",
- " \n",
- "
\n",
- "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " type \n",
+ " accuracy \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " 0 \n",
+ " Benign \n",
+ " 1.00 \n",
+ " \n",
+ " \n",
+ " 1 \n",
+ " Attack \n",
+ " 0.58 \n",
+ " \n",
+ " \n",
+ "
\n",
+ "
"
]
+ },
+ "metadata": {},
+ "output_type": "display_data"
}
- ],
- "metadata": {
- "accelerator": "TPU",
+ ],
+ "source": [
+ "# Create confusion matrix\n",
+ "conf_matrix = confusion_matrix(y_true, y_pred_model)\n",
+ "\n",
+ "# Show confusion matrix\n",
+ "ax = plt.subplot()\n",
+ "sns.heatmap(conf_matrix, annot=True, ax=ax, cmap=\"Blues\", fmt=\"g\", cbar=False)\n",
+ "\n",
+ "# Add labels, title and ticks\n",
+ "ax.set_xlabel(\"Predicted\")\n",
+ "ax.set_ylabel(\"Acctual\")\n",
+ "ax.set_title(\"Confusion Matrix\")\n",
+ "ax.xaxis.set_ticklabels([\"Benign\", \"Attack\"])\n",
+ "ax.yaxis.set_ticklabels([\"Benign\", \"Attack\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
"colab": {
- "collapsed_sections": [],
- "machine_shape": "hm",
- "name": "it_threat_detection.ipynb",
- "provenance": []
- },
- "environment": {
- "name": "tf2-gpu.2-3.m65",
- "type": "gcloud",
- "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m65"
- },
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.7 (main, Sep 14 2022, 22:38:23) [Clang 14.0.0 (clang-1400.0.29.102)]"
- },
- "vscode": {
- "interpreter": {
- "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
- }
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "VJRwvXOvYtBL",
+ "outputId": "9d45426b-6bd8-4374-dcf8-9ffecedd7176"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Accuracy: 0.871\n",
+ "Precision: 1.000\n",
+ "Recall: 0.287\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Calculate accuracy\n",
+ "acc = accuracy_score(y_true, y_pred_model, normalize=True, sample_weight=None)\n",
+ "precision = precision_score(y_true, y_pred_model)\n",
+ "recall = recall_score(y_true, y_pred_model)\n",
+ "\n",
+ "print(f\"Accuracy: {acc:.3f}\")\n",
+ "print(f\"Precision: {precision:.3f}\")\n",
+ "print(f\"Recall: {recall:.3f}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 112
+ },
+ "id": "TFbfVi_-W6GT",
+ "outputId": "68facf43-ac41-44de-d59a-a489627cc893"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " type \n",
+ " accuracy \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " 0 \n",
+ " Benign \n",
+ " 1.00 \n",
+ " \n",
+ " \n",
+ " 1 \n",
+ " Attack \n",
+ " 0.29 \n",
+ " \n",
+ " \n",
+ "
\n",
+ "