notebooks/TF-IDF with Spark MineWiki Search Engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "851e804e",
   "metadata": {},
   "source": [
    "# BASALT 2022 MineDojo Wiki Search Engine\n",
    "\n",
    "## Background\n",
    "I am working on potential solutions for https://www.aicrowd.com/challenges/neurips-2022-minerl-basalt-competition. The goal of this search engine is to quickly find relevant articles when looking at a certain item. This algorithm uses PySpark's TF-IDF algorithm for scoring word frequency and runs quite quickly.\n",
    "\n",
    "## Data Mining\n",
    "In order to get the data to run this script, I downloaded the MineDojo Wiki Dataset and crawled that for data.json files.\n",
    "\n",
    "## Code\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "18f30cda",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "import os\n",
    "from pyspark.sql.types import *\n",
    "from pyspark.ml.feature import HashingTF, IDF, Tokenizer\n",
    "from pyspark.sql.types import FloatType\n",
    "from pyspark.sql.functions import udf\n",
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffeed3b0",
   "metadata": {},
   "source": [
    "### Methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8f54c982",
   "metadata": {},
   "outputs": [],
   "source": [
    "def fast_scandir(dirname):\n",
    "    subfolders= [f.path for f in os.scandir(dirname) if f.is_dir()]\n",
    "    for dirname in list(subfolders):\n",
    "        subfolders.extend(fast_scandir(dirname))\n",
    "    return subfolders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b991b40",
   "metadata": {},
   "source": [
    "### Crawl Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ac1802df",
   "metadata": {},
   "outputs": [],
   "source": [
    "wikiPath = \"/Volumes/Extreme SSD/Extra Datasets/wiki_full/\"\n",
    "subfolders = fast_scandir(wikiPath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c9bb199c",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataFiles = []\n",
    "for folder in subfolders:\n",
    "    if os.path.exists(folder + \"/data.json\"):\n",
    "        dataFiles.append(folder + \"/data.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0537718b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6281\n",
      "/Volumes/Extreme SSD/Extra Datasets/wiki_full/.minecraft/data.json Failed to Load Texts\n",
      "/Volumes/Extreme SSD/Extra Datasets/wiki_full/.minecraft/path/data.json Failed to Load Texts\n",
      "/Volumes/Extreme SSD/Extra Datasets/wiki_full/Mods/TheGunMod/.GUN2_File_Format/data.json Failed to Load Texts\n"
     ]
    }
   ],
   "source": [
    "print(len(dataFiles))\n",
    "dataDict = {\"Location\": [], \"Text\": []}\n",
    "for file in dataFiles:\n",
    "    location = \"--\".join(file.split(\"/\")[5:-1])\n",
    "    fullText = \"\"\n",
    "    with open(file, 'r') as jsonFile:\n",
    "        data = json.load(jsonFile)\n",
    "        try:\n",
    "            for text in data['texts']:\n",
    "                fullText += text['text'] + \"\\n\"\n",
    "        except:\n",
    "            print(file, \"Failed to Load Texts\")\n",
    "        try:\n",
    "            for image in data['images']:\n",
    "                if not str(image['alt_text']) == \"null\":\n",
    "                    fullText += image['alt_text'] + \"\\n\"\n",
    "        except:\n",
    "            print(file, \"Failed to Load Images\")\n",
    "        try:\n",
    "            for table in data['tables']:\n",
    "                if not str(\"\\n\".join(table['headers']['text'])):\n",
    "                    fullText += \"\\n\".join(table['headers']['text'])\n",
    "                if not str(\"\\n\".join(table['cells']['text'])):\n",
    "                    fullText += \"\\n\".join(table['cells']['text'])\n",
    "        except:\n",
    "            print(file, \"Failed to Load Tables\")\n",
    "    dataDict[\"Location\"].append(location)\n",
    "    dataDict[\"Text\"].append(fullText.replace(\"\\n\", \" \").replace(\",\", \"\"))\n",
    "    \n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5a7e11a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "rawData = pd.DataFrame.from_dict(dataDict)\n",
    "rawData.to_csv(wikiPath + \"page_text.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a687ff0",
   "metadata": {},
   "source": [
    "### Open Text Data with PySpark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ac5e67cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22/07/30 15:52:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "22/07/30 15:52:19 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.\n"
     ]
    }
   ],
   "source": [
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .appName(\"MineWiki TF-IDF\") \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "51dbe292",
   "metadata": {},
   "outputs": [],
   "source": [
    "rawData = spark.read.csv(wikiPath + \"page_text.csv\")\n",
    "articles = rawData.toDF(\"Title\", \"Document\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "dcab3e7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Title: string (nullable = true)\n",
      " |-- Document: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "articles.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2c6e8baa",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+\n",
      "|               Title|            Document|\n",
      "+--------------------+--------------------+\n",
      "|            Location|                Text|\n",
      "|  Launcher_2.1.1350x|Launcher 2.1.1350...|\n",
      "|Bedrock_Edition_b...|\"Bedrock Edition ...|\n",
      "|Xbox_360_Edition_...|\"Xbox 360 Edition...|\n",
      "|     Launcher_1.6.19|Launcher 1.6.19 1...|\n",
      "|            Badlands|\"Badlands The bad...|\n",
      "|      Banner_Pattern|\"Banner Pattern B...|\n",
      "|Reinforced_Deepslate|\"Reinforced Deeps...|\n",
      "|                Well|Well Well may ref...|\n",
      "|Bedrock_Edition_1...|\"Bedrock Edition ...|\n",
      "|Bedrock_Dedicated...|Bedrock Dedicated...|\n",
      "|            Breaking|\"Breaking This ar...|\n",
      "| Java_Edition_18w03b|\"Java Edition 18w...|\n",
      "|Xbox_One_Edition_...|\"Xbox One Edition...|\n",
      "|Bedrock_Dedicated...|Bedrock Dedicated...|\n",
      "|    Pillager_Outpost|\"Pillager Outpost...|\n",
      "|     Water_Breathing|\"Water Breathing ...|\n",
      "|Java_Edition_1.10...|Java Edition 1.10...|\n",
      "|Bedrock_Edition_b...|\"Bedrock Edition ...|\n",
      "|     Adam_Martinsson|\"Adam Martinsson ...|\n",
      "+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "articles.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b7c74f3",
   "metadata": {},
   "source": [
    "### Clean Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "db5fad13",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles.filter(articles.Document.isNull()).count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e1dcc1be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleanedArticles = articles.filter(articles.Document.isNotNull())\n",
    "cleanedArticles.filter(cleanedArticles.Document.isNull()).count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "970aa7b5",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+--------------------+--------------------+\n",
      "|               Title|            Document|               words|         rawFeatures|\n",
      "+--------------------+--------------------+--------------------+--------------------+\n",
      "|            Location|                Text|              [text]|(262144,[143985],...|\n",
      "|  Launcher_2.1.1350x|Launcher 2.1.1350...|[launcher, 2.1.13...|(262144,[12524,27...|\n",
      "|Bedrock_Edition_b...|\"Bedrock Edition ...|[\"bedrock, editio...|(262144,[619,784,...|\n",
      "|Xbox_360_Edition_...|\"Xbox 360 Edition...|[\"xbox, 360, edit...|(262144,[3449,592...|\n",
      "|     Launcher_1.6.19|Launcher 1.6.19 1...|[launcher, 1.6.19...|(262144,[12524,27...|\n",
      "|            Badlands|\"Badlands The bad...|[\"badlands, the, ...|(262144,[238,535,...|\n",
      "|      Banner_Pattern|\"Banner Pattern B...|[\"banner, pattern...|(262144,[702,8254...|\n",
      "|Reinforced_Deepslate|\"Reinforced Deeps...|[\"reinforced, dee...|(262144,[702,1512...|\n",
      "|                Well|Well Well may ref...|[well, well, may,...|(262144,[5755,853...|\n",
      "|Bedrock_Edition_1...|\"Bedrock Edition ...|[\"bedrock, editio...|(262144,[542,1088...|\n",
      "|Bedrock_Dedicated...|Bedrock Dedicated...|[bedrock, dedicat...|(262144,[2366,214...|\n",
      "|            Breaking|\"Breaking This ar...|[\"breaking, this,...|(262144,[521,789,...|\n",
      "| Java_Edition_18w03b|\"Java Edition 18w...|[\"java, edition, ...|(262144,[874,921,...|\n",
      "|Xbox_One_Edition_...|\"Xbox One Edition...|[\"xbox, one, edit...|(262144,[1512,236...|\n",
      "|Bedrock_Dedicated...|Bedrock Dedicated...|[bedrock, dedicat...|(262144,[2366,825...|\n",
      "|    Pillager_Outpost|\"Pillager Outpost...|[\"pillager, outpo...|(262144,[1603,234...|\n",
      "|     Water_Breathing|\"Water Breathing ...|[\"water, breathin...|(262144,[702,1865...|\n",
      "|Java_Edition_1.10...|Java Edition 1.10...|[java, edition, 1...|(262144,[245,619,...|\n",
      "|Bedrock_Edition_b...|\"Bedrock Edition ...|[\"bedrock, editio...|(262144,[7019,840...|\n",
      "|     Adam_Martinsson|\"Adam Martinsson ...|[\"adam, martinsso...|(262144,[1275,328...|\n",
      "+--------------------+--------------------+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer(inputCol=\"Document\", outputCol=\"words\")\n",
    "wordsData = tokenizer.transform(cleanedArticles)\n",
    "\n",
    "hashingTF = HashingTF(inputCol=\"words\", outputCol = \"rawFeatures\")\n",
    "featurizedData = hashingTF.transform(wordsData)\n",
    "featurizedData.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdc519ce",
   "metadata": {},
   "source": [
    "### Starting IDF Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "314dd465",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "idf = IDF(inputCol=\"rawFeatures\", outputCol=\"features\")\n",
    "idfModel = idf.fit(featurizedData)\n",
    "rescaledData = idfModel.transform(featurizedData)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1e0b7235",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22/07/30 15:52:27 WARN DAGScheduler: Broadcasting large task binary with size 4.0 MiB\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------------+\n",
      "|               Title|            Document|               words|         rawFeatures|            features|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------------+\n",
      "|            Location|                Text|              [text]|(262144,[143985],...|(262144,[143985],...|\n",
      "|  Launcher_2.1.1350x|Launcher 2.1.1350...|[launcher, 2.1.13...|(262144,[12524,27...|(262144,[12524,27...|\n",
      "|Bedrock_Edition_b...|\"Bedrock Edition ...|[\"bedrock, editio...|(262144,[619,784,...|(262144,[619,784,...|\n",
      "|Xbox_360_Edition_...|\"Xbox 360 Edition...|[\"xbox, 360, edit...|(262144,[3449,592...|(262144,[3449,592...|\n",
      "|     Launcher_1.6.19|Launcher 1.6.19 1...|[launcher, 1.6.19...|(262144,[12524,27...|(262144,[12524,27...|\n",
      "|            Badlands|\"Badlands The bad...|[\"badlands, the, ...|(262144,[238,535,...|(262144,[238,535,...|\n",
      "|      Banner_Pattern|\"Banner Pattern B...|[\"banner, pattern...|(262144,[702,8254...|(262144,[702,8254...|\n",
      "|Reinforced_Deepslate|\"Reinforced Deeps...|[\"reinforced, dee...|(262144,[702,1512...|(262144,[702,1512...|\n",
      "|                Well|Well Well may ref...|[well, well, may,...|(262144,[5755,853...|(262144,[5755,853...|\n",
      "|Bedrock_Edition_1...|\"Bedrock Edition ...|[\"bedrock, editio...|(262144,[542,1088...|(262144,[542,1088...|\n",
      "|Bedrock_Dedicated...|Bedrock Dedicated...|[bedrock, dedicat...|(262144,[2366,214...|(262144,[2366,214...|\n",
      "|            Breaking|\"Breaking This ar...|[\"breaking, this,...|(262144,[521,789,...|(262144,[521,789,...|\n",
      "| Java_Edition_18w03b|\"Java Edition 18w...|[\"java, edition, ...|(262144,[874,921,...|(262144,[874,921,...|\n",
      "|Xbox_One_Edition_...|\"Xbox One Edition...|[\"xbox, one, edit...|(262144,[1512,236...|(262144,[1512,236...|\n",
      "|Bedrock_Dedicated...|Bedrock Dedicated...|[bedrock, dedicat...|(262144,[2366,825...|(262144,[2366,825...|\n",
      "|    Pillager_Outpost|\"Pillager Outpost...|[\"pillager, outpo...|(262144,[1603,234...|(262144,[1603,234...|\n",
      "|     Water_Breathing|\"Water Breathing ...|[\"water, breathin...|(262144,[702,1865...|(262144,[702,1865...|\n",
      "|Java_Edition_1.10...|Java Edition 1.10...|[java, edition, 1...|(262144,[245,619,...|(262144,[245,619,...|\n",
      "|Bedrock_Edition_b...|\"Bedrock Edition ...|[\"bedrock, editio...|(262144,[7019,840...|(262144,[7019,840...|\n",
      "|     Adam_Martinsson|\"Adam Martinsson ...|[\"adam, martinsso...|(262144,[1275,328...|(262144,[1275,328...|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rescaledData.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e550c88",
   "metadata": {},
   "source": [
    "### Testing With Search Term"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b820f52c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+\n",
      "| words|\n",
      "+------+\n",
      "|[tree]|\n",
      "+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "schema = StructType([StructField(\"words\", ArrayType(StringType()))])\n",
    "\n",
    "df = spark.createDataFrame(([[[\"tree\"]]]), schema).toDF(\"words\")\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d69aaed4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+--------------------+\n",
      "| words|         rawFeatures|\n",
      "+------+--------------------+\n",
      "|[tree]|(262144,[193711],...|\n",
      "+------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tree = hashingTF.transform(df)\n",
    "tree.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "de35748a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Row(rawFeatures=SparseVector(262144, {193711: 1.0}))]\n"
     ]
    }
   ],
   "source": [
    "featureVec = tree.select('rawFeatures').collect()\n",
    "print(featureVec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6b531794",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "193711\n"
     ]
    }
   ],
   "source": [
    "treeID = int(featureVec[0].rawFeatures.indices[0])\n",
    "print(treeID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5b6535a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22/07/30 15:52:29 WARN DAGScheduler: Broadcasting large task binary with size 4.1 MiB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 19:>                                                         (0 + 1) / 1]\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+--------------------+--------------------+--------------------+-----+\n",
      "|               Title|            Document|               words|         rawFeatures|            features|score|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------------+-----+\n",
      "|            Location|                Text|              [text]|(262144,[143985],...|(262144,[143985],...|  0.0|\n",
      "|  Launcher_2.1.1350x|Launcher 2.1.1350...|[launcher, 2.1.13...|(262144,[12524,27...|(262144,[12524,27...|  0.0|\n",
      "|Bedrock_Edition_b...|\"Bedrock Edition ...|[\"bedrock, editio...|(262144,[619,784,...|(262144,[619,784,...|  0.0|\n",
      "|Xbox_360_Edition_...|\"Xbox 360 Edition...|[\"xbox, 360, edit...|(262144,[3449,592...|(262144,[3449,592...|  0.0|\n",
      "|     Launcher_1.6.19|Launcher 1.6.19 1...|[launcher, 1.6.19...|(262144,[12524,27...|(262144,[12524,27...|  0.0|\n",
      "|            Badlands|\"Badlands The bad...|[\"badlands, the, ...|(262144,[238,535,...|(262144,[238,535,...|  0.0|\n",
      "|      Banner_Pattern|\"Banner Pattern B...|[\"banner, pattern...|(262144,[702,8254...|(262144,[702,8254...|  0.0|\n",
      "|Reinforced_Deepslate|\"Reinforced Deeps...|[\"reinforced, dee...|(262144,[702,1512...|(262144,[702,1512...|  0.0|\n",
      "|                Well|Well Well may ref...|[well, well, may,...|(262144,[5755,853...|(262144,[5755,853...|  0.0|\n",
      "|Bedrock_Edition_1...|\"Bedrock Edition ...|[\"bedrock, editio...|(262144,[542,1088...|(262144,[542,1088...|  0.0|\n",
      "|Bedrock_Dedicated...|Bedrock Dedicated...|[bedrock, dedicat...|(262144,[2366,214...|(262144,[2366,214...|  0.0|\n",
      "|            Breaking|\"Breaking This ar...|[\"breaking, this,...|(262144,[521,789,...|(262144,[521,789,...|  0.0|\n",
      "| Java_Edition_18w03b|\"Java Edition 18w...|[\"java, edition, ...|(262144,[874,921,...|(262144,[874,921,...|  0.0|\n",
      "|Xbox_One_Edition_...|\"Xbox One Edition...|[\"xbox, one, edit...|(262144,[1512,236...|(262144,[1512,236...|  0.0|\n",
      "|Bedrock_Dedicated...|Bedrock Dedicated...|[bedrock, dedicat...|(262144,[2366,825...|(262144,[2366,825...|  0.0|\n",
      "|    Pillager_Outpost|\"Pillager Outpost...|[\"pillager, outpo...|(262144,[1603,234...|(262144,[1603,234...|  0.0|\n",
      "|     Water_Breathing|\"Water Breathing ...|[\"water, breathin...|(262144,[702,1865...|(262144,[702,1865...|  0.0|\n",
      "|Java_Edition_1.10...|Java Edition 1.10...|[java, edition, 1...|(262144,[245,619,...|(262144,[245,619,...|  0.0|\n",
      "|Bedrock_Edition_b...|\"Bedrock Edition ...|[\"bedrock, editio...|(262144,[7019,840...|(262144,[7019,840...|  0.0|\n",
      "|     Adam_Martinsson|\"Adam Martinsson ...|[\"adam, martinsso...|(262144,[1275,328...|(262144,[1275,328...|  0.0|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "termExtractor = udf(lambda x: float(x[treeID]), FloatType())\n",
    "\n",
    "treeDF = rescaledData.withColumn('score', termExtractor(rescaledData.features))\n",
    "treeDF.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5ca0ee61",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22/07/30 15:52:30 WARN DAGScheduler: Broadcasting large task binary with size 4.1 MiB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 20:===================>                                      (2 + 4) / 6]\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------------------------------+---------+\n",
      "|                                  Title|    score|\n",
      "+---------------------------------------+---------+\n",
      "|                                   Tree|320.97998|\n",
      "|                Tutorials--Tree_farming|229.72096|\n",
      "|                        Tree--Structure|94.405876|\n",
      "|                                    Oak|94.405876|\n",
      "|                            Jungle_tree| 88.11215|\n",
      "|                                  Birch| 78.67156|\n",
      "|                                Sapling|62.937252|\n",
      "|                                 Spruce|59.790386|\n",
      "|                                 Acacia|56.643524|\n",
      "|                     Biome--Before_1.18|47.202938|\n",
      "|                     Configured_feature|37.762352|\n",
      "|                               Dark_oak|31.468626|\n",
      "|                            Azalea_tree|31.468626|\n",
      "|          Java_Edition_removed_features|28.321762|\n",
      "|       Tutorials--Best_biomes_for_homes|28.321762|\n",
      "|          Tutorials--Superflat_survival|28.321762|\n",
      "|                       Old_Growth_Taiga|  25.1749|\n",
      "|             Tutorials--Nether_survival|22.028038|\n",
      "|                    Tutorials--Shelters|22.028038|\n",
      "|Java_Edition_1.15--Development_versions|22.028038|\n",
      "+---------------------------------------+---------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "sortedResults = treeDF.filter(\"score > 0\").orderBy('score', ascending=False).select(\"Title\", \"score\")\n",
    "sortedResults.show(truncate=100)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "minedojo",
   "language": "python",
   "name": "minedojo"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}