polars-cheat-sheet.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/boyuan-li/BL-data-processing/blob/master/polars-cheat-sheet.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a413d91c-c585-4f2c-84c4-4a57b2cf4372",
      "metadata": {
        "id": "a413d91c-c585-4f2c-84c4-4a57b2cf4372"
      },
      "source": [
        "# [Polars](https://www.pola.rs/) Cheat Sheet\n",
        "\n",
        "[![Polars cheat sheet: download as PDF](https://img.shields.io/badge/Polars%20cheat%20sheet-Download%20as%20PDF-green)](https://franzdiebold.github.io/polars-cheat-sheet/Polars_cheat_sheet.pdf)\n",
        "\n",
        "Most examples were taken from the official [Polars user guide](https://pola-rs.github.io/polars-book/user-guide/).\n",
        "\n",
        "By Franz Diebold ([diebold.io](https://www.diebold.io)), inspired by [Pandas Cheat Sheet](https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8da10053-1ff7-49a6-bb81-3abd21ff6701",
      "metadata": {
        "id": "8da10053-1ff7-49a6-bb81-3abd21ff6701"
      },
      "source": [
        "### Install"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dfcf679b-a9f4-4d8d-9db8-d293078b55df",
      "metadata": {
        "id": "dfcf679b-a9f4-4d8d-9db8-d293078b55df"
      },
      "outputs": [],
      "source": [
        "!pip install polars"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c31fa6b4-f2e8-4846-bcd3-cfbf6607f926",
      "metadata": {
        "id": "c31fa6b4-f2e8-4846-bcd3-cfbf6607f926"
      },
      "source": [
        "### Import"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3d49a140-6a2d-420d-a74b-ca92c2e67e26",
      "metadata": {
        "pycharm": {
          "is_executing": true
        },
        "id": "3d49a140-6a2d-420d-a74b-ca92c2e67e26"
      },
      "outputs": [],
      "source": [
        "import polars as pl"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "19dddd3a-80e1-4309-8472-07d8fcb79c99",
      "metadata": {
        "id": "19dddd3a-80e1-4309-8472-07d8fcb79c99"
      },
      "source": [
        "### Creating/reading DataFrames"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c36161bf-1ad1-4966-a6d6-40cbb9dc6a6d",
      "metadata": {
        "id": "c36161bf-1ad1-4966-a6d6-40cbb9dc6a6d"
      },
      "outputs": [],
      "source": [
        "# Create DataFrame\n",
        "df = pl.DataFrame(\n",
        "    {\n",
        "        \"nrs\": [1, 2, 3, None, 5],\n",
        "        \"names\": [\"foo\", \"ham\", \"spam\", \"egg\", None],\n",
        "        \"random\": [0.3, 0.7, 0.1, 0.9, 0.6],\n",
        "        \"groups\": [\"A\", \"A\", \"B\", \"C\", \"B\"],\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2fdd20b6-729e-4780-babe-5b14d523c77a",
      "metadata": {
        "id": "2fdd20b6-729e-4780-babe-5b14d523c77a"
      },
      "outputs": [],
      "source": [
        "# Read CSV\n",
        "df = pl.read_csv(\"https://j.mp/iriscsv\", has_header=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0f21bef6-ee10-416e-b4a1-81762bc33f85",
      "metadata": {
        "id": "0f21bef6-ee10-416e-b4a1-81762bc33f85"
      },
      "outputs": [],
      "source": [
        "# Read parquet\n",
        "df = pl.read_parquet(\"path.parquet\", columns=[\"select\", \"columns\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7bc74df2-7977-4e62-b9eb-f211453739d2",
      "metadata": {
        "id": "7bc74df2-7977-4e62-b9eb-f211453739d2"
      },
      "source": [
        "### Expressions\n",
        "Polars expressions can be performed in sequence. This improves readability of code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3894dd14-c4f5-45a6-a240-1b69352f54e2",
      "metadata": {
        "id": "3894dd14-c4f5-45a6-a240-1b69352f54e2"
      },
      "outputs": [],
      "source": [
        "df.filter(pl.col(\"nrs\") < 4).groupby(\"groups\").agg(pl.all().sum())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ae4007ff-42c0-4d08-82d2-2e41acacb076",
      "metadata": {
        "id": "ae4007ff-42c0-4d08-82d2-2e41acacb076"
      },
      "source": [
        "### Subset Observations - rows"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "03595339-2609-48bd-a0e8-28505877060a",
      "metadata": {
        "id": "03595339-2609-48bd-a0e8-28505877060a"
      },
      "outputs": [],
      "source": [
        "# Filter: Extract rows that meet logical criteria.\n",
        "df.filter(pl.col(\"random\") > 0.5)\n",
        "df.filter((pl.col(\"groups\") == \"B\") & (pl.col(\"random\") > 0.5))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "db699607-6c83-44a1-8c86-38ea104f1904",
      "metadata": {
        "id": "db699607-6c83-44a1-8c86-38ea104f1904"
      },
      "outputs": [],
      "source": [
        "# Sample\n",
        "# Randomly select fraction of rows.\n",
        "df.sample(frac=0.5)\n",
        "\n",
        "# Randomly select n rows.\n",
        "df.sample(n=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3371e28b-4b09-4a3b-ae7a-29687c1b36f1",
      "metadata": {
        "id": "3371e28b-4b09-4a3b-ae7a-29687c1b36f1"
      },
      "outputs": [],
      "source": [
        "# Select first n rows\n",
        "df.head(n=2)\n",
        "\n",
        "# Select last n rows.\n",
        "df.tail(n=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "41fc6d50-9dbb-4f7c-9902-3e219fa2a3ee",
      "metadata": {
        "id": "41fc6d50-9dbb-4f7c-9902-3e219fa2a3ee"
      },
      "source": [
        "### Subset Variables - columns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "913e57a2-dd27-4ce2-921a-6b7304f96853",
      "metadata": {
        "id": "913e57a2-dd27-4ce2-921a-6b7304f96853"
      },
      "outputs": [],
      "source": [
        "# Select multiple columns with specific names.\n",
        "df.select([\"nrs\", \"names\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "969ec688-95a8-47e8-8f81-a1795e00b2b0",
      "metadata": {
        "tags": [],
        "id": "969ec688-95a8-47e8-8f81-a1795e00b2b0"
      },
      "outputs": [],
      "source": [
        "# Select columns whose name matches regular expression regex.\n",
        "df.select(pl.col(\"^n.*$\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b226c7a9-7e36-4929-909c-8c0fa682e55e",
      "metadata": {
        "id": "b226c7a9-7e36-4929-909c-8c0fa682e55e"
      },
      "source": [
        "### Subsets - rows and columns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "25e62d39-604d-48c6-9be2-849a7dfde2b4",
      "metadata": {
        "id": "25e62d39-604d-48c6-9be2-849a7dfde2b4"
      },
      "outputs": [],
      "source": [
        "# Select rows 2-4.\n",
        "df[2:4, :]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a87d3a1e-a2fc-4133-a837-cc510a2b72fa",
      "metadata": {
        "id": "a87d3a1e-a2fc-4133-a837-cc510a2b72fa"
      },
      "outputs": [],
      "source": [
        "# Select columns in positions 1 and 3 (first column is 0).\n",
        "df[:, [1, 3]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2512197f-8763-45b3-b2d0-41037b3f4688",
      "metadata": {
        "id": "2512197f-8763-45b3-b2d0-41037b3f4688"
      },
      "outputs": [],
      "source": [
        "# Select rows meeting logical condition, and only the specific columns.\n",
        "df[df[\"random\"] > 0.5, [\"names\", \"groups\"]]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b4e993b3-227d-40b0-877c-b64598470536",
      "metadata": {
        "tags": [],
        "id": "b4e993b3-227d-40b0-877c-b64598470536"
      },
      "source": [
        "### Reshaping Data – Change layout, sorting, renaming"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7a45bd7d-02b6-41a6-9773-184396771647",
      "metadata": {
        "id": "7a45bd7d-02b6-41a6-9773-184396771647"
      },
      "outputs": [],
      "source": [
        "df2 = pl.DataFrame(\n",
        "    {\n",
        "        \"nrs\": [6],\n",
        "        \"names\": [\"wow\"],\n",
        "        \"random\": [0.9],\n",
        "        \"groups\": [\"B\"],\n",
        "    }\n",
        ")\n",
        "\n",
        "df3 = pl.DataFrame(\n",
        "    {\n",
        "        \"primes\": [2, 3, 5, 7, 11],\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8e6cc724-3369-4c7e-9066-cc7ccbb74e85",
      "metadata": {
        "id": "8e6cc724-3369-4c7e-9066-cc7ccbb74e85"
      },
      "outputs": [],
      "source": [
        "# Append rows of DataFrames.\n",
        "pl.concat([df, df2])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d3115adf-1342-4a57-9da9-b998ae86d515",
      "metadata": {
        "id": "d3115adf-1342-4a57-9da9-b998ae86d515"
      },
      "outputs": [],
      "source": [
        "# Append columns of DataFrames\n",
        "pl.concat([df, df3], how=\"horizontal\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4c19acb6-4c53-48d9-9098-9d2dc6442bcb",
      "metadata": {
        "id": "4c19acb6-4c53-48d9-9098-9d2dc6442bcb"
      },
      "outputs": [],
      "source": [
        "# Gather columns into rows\n",
        "df.melt(id_vars=\"nrs\", value_vars=[\"names\", \"groups\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b8491178-47f3-477d-83e8-9642bae403b1",
      "metadata": {
        "id": "b8491178-47f3-477d-83e8-9642bae403b1"
      },
      "outputs": [],
      "source": [
        "# Spread rows into columns\n",
        "df.pivot(values=\"nrs\", index=\"groups\", columns=\"names\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e6bf22bb-7180-45ac-924b-8f8763d8a751",
      "metadata": {
        "id": "e6bf22bb-7180-45ac-924b-8f8763d8a751"
      },
      "outputs": [],
      "source": [
        "# Order rows by values of a column (low to high)\n",
        "df.sort(\"random\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0412baca-8146-4505-9ab3-32b051d03b45",
      "metadata": {
        "id": "0412baca-8146-4505-9ab3-32b051d03b45"
      },
      "outputs": [],
      "source": [
        "# Order rows by values of a column (high to low)\n",
        "df.sort(\"random\", reverse=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3eb2e067-111b-42fd-9e53-5164291ba8b9",
      "metadata": {
        "id": "3eb2e067-111b-42fd-9e53-5164291ba8b9"
      },
      "outputs": [],
      "source": [
        "# Rename the columns of a DataFrame\n",
        "df.rename({\"nrs\": \"idx\"})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f2b5466f-e9e2-44da-a574-532b5aa11d5c",
      "metadata": {
        "id": "f2b5466f-e9e2-44da-a574-532b5aa11d5c"
      },
      "outputs": [],
      "source": [
        "# Drop columns from DataFrame\n",
        "df.drop([\"names\", \"random\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bc33cbc5-7aeb-4885-abcd-3f7293964aef",
      "metadata": {
        "id": "bc33cbc5-7aeb-4885-abcd-3f7293964aef"
      },
      "source": [
        "### Summarize Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f0baaa52-d99a-446c-abc5-c7acc369c674",
      "metadata": {
        "id": "f0baaa52-d99a-446c-abc5-c7acc369c674"
      },
      "outputs": [],
      "source": [
        "# Count number of rows with each unique value of variable\n",
        "df[\"groups\"].value_counts()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1faeab9e-cdf3-4317-973f-1282d5570d54",
      "metadata": {
        "id": "1faeab9e-cdf3-4317-973f-1282d5570d54"
      },
      "outputs": [],
      "source": [
        "# # of rows in DataFrame\n",
        "len(df)\n",
        "# or\n",
        "df.height"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dc8e7874-0a20-42df-a3dd-a0878223729c",
      "metadata": {
        "id": "dc8e7874-0a20-42df-a3dd-a0878223729c"
      },
      "outputs": [],
      "source": [
        "# Tuple of # of rows, # of columns in DataFrame\n",
        "df.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b5496aa4-fd90-4cd9-aae4-8788977167f5",
      "metadata": {
        "id": "b5496aa4-fd90-4cd9-aae4-8788977167f5"
      },
      "outputs": [],
      "source": [
        "# # of distinct values in a column\n",
        "df[\"groups\"].n_unique()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b6ec0809-69b0-4165-a894-e1625ef85602",
      "metadata": {
        "id": "b6ec0809-69b0-4165-a894-e1625ef85602"
      },
      "outputs": [],
      "source": [
        "# Basic descriptive and statistics for each column\n",
        "df.describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "916e2e67-1914-4ffe-aea0-2e43599e0c53",
      "metadata": {
        "id": "916e2e67-1914-4ffe-aea0-2e43599e0c53"
      },
      "outputs": [],
      "source": [
        "# Aggregation functions\n",
        "df.select(\n",
        "    [\n",
        "        # Sum values\n",
        "        pl.sum(\"random\").alias(\"sum\"),\n",
        "        # Minimum value\n",
        "        pl.min(\"random\").alias(\"min\"),\n",
        "        # Maximum value\n",
        "        pl.max(\"random\").alias(\"max\"),\n",
        "        # or\n",
        "        pl.col(\"random\").max().alias(\"other_max\"),\n",
        "        # Standard deviation\n",
        "        pl.std(\"random\").alias(\"std_dev\"),\n",
        "        # Variance\n",
        "        pl.var(\"random\").alias(\"variance\"),\n",
        "        # Median\n",
        "        pl.median(\"random\").alias(\"median\"),\n",
        "        # Mean\n",
        "        pl.mean(\"random\").alias(\"mean\"),\n",
        "        # Quantile\n",
        "        pl.quantile(\"random\", 0.75).alias(\"quantile_0.75\"),\n",
        "        # or\n",
        "        pl.col(\"random\").quantile(0.75).alias(\"other_quantile_0.75\"),\n",
        "        # First value\n",
        "        pl.first(\"random\").alias(\"first\"),\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5a079cef-da46-4dbe-8a2f-e986892e520d",
      "metadata": {
        "id": "5a079cef-da46-4dbe-8a2f-e986892e520d"
      },
      "source": [
        "### Group Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "17b75d48-cb40-4d25-bdd7-1cc531a2a8e9",
      "metadata": {
        "id": "17b75d48-cb40-4d25-bdd7-1cc531a2a8e9"
      },
      "outputs": [],
      "source": [
        "# Group by values in column named \"col\", returning a GroupBy object\n",
        "df.groupby(\"groups\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bd41f117-d266-4968-bbed-30e8bbac6597",
      "metadata": {
        "id": "bd41f117-d266-4968-bbed-30e8bbac6597"
      },
      "outputs": [],
      "source": [
        "# All of the aggregation functions from above can be applied to a group as well\n",
        "df.groupby(by=\"groups\").agg(\n",
        "    [\n",
        "        # Sum values\n",
        "        pl.sum(\"random\").alias(\"sum\"),\n",
        "        # Minimum value\n",
        "        pl.min(\"random\").alias(\"min\"),\n",
        "        # Maximum value\n",
        "        pl.max(\"random\").alias(\"max\"),\n",
        "        # or\n",
        "        pl.col(\"random\").max().alias(\"other_max\"),\n",
        "        # Standard deviation\n",
        "        pl.std(\"random\").alias(\"std_dev\"),\n",
        "        # Variance\n",
        "        pl.var(\"random\").alias(\"variance\"),\n",
        "        # Median\n",
        "        pl.median(\"random\").alias(\"median\"),\n",
        "        # Mean\n",
        "        pl.mean(\"random\").alias(\"mean\"),\n",
        "        # Quantile\n",
        "        pl.quantile(\"random\", 0.75).alias(\"quantile_0.75\"),\n",
        "        # or\n",
        "        pl.col(\"random\").quantile(0.75).alias(\"other_quantile_0.75\"),\n",
        "        # First value\n",
        "        pl.first(\"random\").alias(\"first\"),\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f893fe3a-f0fb-4bc3-8e61-359d8cf621fb",
      "metadata": {
        "id": "f893fe3a-f0fb-4bc3-8e61-359d8cf621fb"
      },
      "outputs": [],
      "source": [
        "# Additional GroupBy functions\n",
        "df.groupby(by=\"groups\").agg(\n",
        "    [\n",
        "        # Count the number of values in each group\n",
        "        pl.count(\"random\").alias(\"size\"),\n",
        "        # Sample one element in each group\n",
        "        pl.col(\"names\").apply(lambda group_df: group_df.sample(1)),\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4e9db31f-1c82-43ed-b1a7-0dd2930065f3",
      "metadata": {
        "id": "4e9db31f-1c82-43ed-b1a7-0dd2930065f3"
      },
      "source": [
        "### Handling Missing Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fc1e75eb-1550-402b-b782-0046a44c8f09",
      "metadata": {
        "id": "fc1e75eb-1550-402b-b782-0046a44c8f09"
      },
      "outputs": [],
      "source": [
        "# Drop rows with any column having a null value\n",
        "df.drop_nulls()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7a7d23f2-830a-4acb-a044-a8ec41a43e92",
      "metadata": {
        "id": "7a7d23f2-830a-4acb-a044-a8ec41a43e92"
      },
      "outputs": [],
      "source": [
        "# Replace null values with given value\n",
        "df.fill_null(42)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "efd26792-c11e-4b74-9287-3992a814ebd1",
      "metadata": {
        "id": "efd26792-c11e-4b74-9287-3992a814ebd1"
      },
      "outputs": [],
      "source": [
        "# Replace null values using forward strategy\n",
        "df.fill_null(strategy=\"forward\")\n",
        "# Other fill strategies are \"backward\", \"min\", \"max\", \"mean\", \"zero\" and \"one\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "359e3c05-c44e-4d63-8bc4-d537fc33dff8",
      "metadata": {
        "id": "359e3c05-c44e-4d63-8bc4-d537fc33dff8"
      },
      "outputs": [],
      "source": [
        "# Replace floating point NaN values with given value\n",
        "df.fill_nan(42)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8de29e58-6806-4a16-abd1-46ceb8581d08",
      "metadata": {
        "id": "8de29e58-6806-4a16-abd1-46ceb8581d08"
      },
      "source": [
        "### Make New Columns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e1ae1b1e-ece1-4df8-8dcc-35b8e62064a4",
      "metadata": {
        "id": "e1ae1b1e-ece1-4df8-8dcc-35b8e62064a4"
      },
      "outputs": [],
      "source": [
        "# Add a new column to the DataFrame\n",
        "df.with_column((pl.col(\"random\") * pl.col(\"nrs\")).alias(\"product\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1fdb7344-8e71-4a79-bdf6-b78f85ac34b9",
      "metadata": {
        "id": "1fdb7344-8e71-4a79-bdf6-b78f85ac34b9"
      },
      "outputs": [],
      "source": [
        "# Add several new columns to the DataFrame\n",
        "df.with_columns(\n",
        "    [\n",
        "        (pl.col(\"random\") * pl.col(\"nrs\")).alias(\"product\"),\n",
        "        pl.col(\"names\").str.lengths().alias(\"names_lengths\"),\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "36b337f4-43c2-444a-9ec3-440acb62fdea",
      "metadata": {
        "id": "36b337f4-43c2-444a-9ec3-440acb62fdea"
      },
      "outputs": [],
      "source": [
        "# Add a column at index 0 that counts the rows\n",
        "df.with_row_count()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "553b2c8d-4048-4431-a859-e9816c2ac7ff",
      "metadata": {
        "id": "553b2c8d-4048-4431-a859-e9816c2ac7ff"
      },
      "source": [
        "### Rolling Functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8ff6233c-00cc-4a83-927d-a3d6ecd33f78",
      "metadata": {
        "tags": [],
        "id": "8ff6233c-00cc-4a83-927d-a3d6ecd33f78"
      },
      "outputs": [],
      "source": [
        "# The following rolling functions are available\n",
        "import numpy as np\n",
        "\n",
        "df.select(\n",
        "    [\n",
        "        pl.col(\"random\"),\n",
        "        # Rolling maximum value\n",
        "        pl.col(\"random\").rolling_max(window_size=2).alias(\"rolling_max\"),\n",
        "        # Rolling mean value\n",
        "        pl.col(\"random\").rolling_mean(window_size=2).alias(\"rolling_mean\"),\n",
        "        # Rolling median value\n",
        "        pl.col(\"random\")\n",
        "        .rolling_median(window_size=2, min_periods=2)\n",
        "        .alias(\"rolling_median\"),\n",
        "        # Rolling minimum value\n",
        "        pl.col(\"random\").rolling_min(window_size=2).alias(\"rolling_min\"),\n",
        "        # Rolling standard deviation\n",
        "        pl.col(\"random\").rolling_std(window_size=2).alias(\"rolling_std\"),\n",
        "        # Rolling sum values\n",
        "        pl.col(\"random\").rolling_sum(window_size=2).alias(\"rolling_sum\"),\n",
        "        # Rolling variance\n",
        "        pl.col(\"random\").rolling_var(window_size=2).alias(\"rolling_var\"),\n",
        "        # Rolling quantile\n",
        "        pl.col(\"random\")\n",
        "        .rolling_quantile(quantile=0.75, window_size=2, min_periods=2)\n",
        "        .alias(\"rolling_quantile\"),\n",
        "        # Rolling skew\n",
        "        pl.col(\"random\").rolling_skew(window_size=2).alias(\"rolling_skew\"),\n",
        "        # Rolling custom function\n",
        "        pl.col(\"random\")\n",
        "        .rolling_apply(function=np.nanstd, window_size=2)\n",
        "        .alias(\"rolling_apply\"),\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "60a278d1-ff9b-420e-b972-b90342d5051c",
      "metadata": {
        "id": "60a278d1-ff9b-420e-b972-b90342d5051c"
      },
      "source": [
        "### Window functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "62a52178-1a43-49de-8f75-5ea3469263f3",
      "metadata": {
        "id": "62a52178-1a43-49de-8f75-5ea3469263f3"
      },
      "outputs": [],
      "source": [
        "# Window functions allow to group by several columns simultaneously\n",
        "df.select(\n",
        "    [\n",
        "        \"names\",\n",
        "        \"groups\",\n",
        "        pl.col(\"random\").sum().over(\"names\").alias(\"sum_by_names\"),\n",
        "        pl.col(\"random\").sum().over(\"groups\").alias(\"sum_by_groups\"),\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f22bb998-15ca-4560-b42a-f330acbf809d",
      "metadata": {
        "id": "f22bb998-15ca-4560-b42a-f330acbf809d"
      },
      "source": [
        "### Combine Data Sets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3f3f6fa6-fbb6-45d2-a3b8-dfc8985be677",
      "metadata": {
        "tags": [],
        "id": "3f3f6fa6-fbb6-45d2-a3b8-dfc8985be677"
      },
      "outputs": [],
      "source": [
        "df4 = pl.DataFrame(\n",
        "    {\n",
        "        \"nrs\": [1, 2, 5, 6],\n",
        "        \"animals\": [\"cheetah\", \"lion\", \"leopard\", \"tiger\"],\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2ccddc52-7923-4276-952a-e49e709e8ee3",
      "metadata": {
        "id": "2ccddc52-7923-4276-952a-e49e709e8ee3"
      },
      "outputs": [],
      "source": [
        "# Inner join\n",
        "# Retains only rows with a match in the other set.\n",
        "df.join(df4, on=\"nrs\")\n",
        "# or\n",
        "df.join(df4, on=\"nrs\", how=\"inner\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "79b9aa6f-0e25-444f-91df-53586d4facc8",
      "metadata": {
        "id": "79b9aa6f-0e25-444f-91df-53586d4facc8"
      },
      "outputs": [],
      "source": [
        "# Left join\n",
        "# Retains each row from \"left\" set (df).\n",
        "df.join(df4, on=\"nrs\", how=\"left\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "204f1bdc-db7e-4351-9750-cc4c2044fcb3",
      "metadata": {
        "id": "204f1bdc-db7e-4351-9750-cc4c2044fcb3"
      },
      "outputs": [],
      "source": [
        "# Outer join\n",
        "# Retains each row, even if no other matching row exists.\n",
        "df.join(df4, on=\"nrs\", how=\"outer\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "19d90d0b-9aff-476d-be8b-23699b817842",
      "metadata": {
        "id": "19d90d0b-9aff-476d-be8b-23699b817842"
      },
      "outputs": [],
      "source": [
        "# Anti join\n",
        "# Contains all rows from df that do not have a match in df4.\n",
        "df.join(df4, on=\"nrs\", how=\"anti\")"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.9"
    },
    "vscode": {
      "interpreter": {
        "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
      }
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}