lottery_dataset_analyze.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "4fd5b34b-e653-4156-9e82-c10c9b3d802d",
      "metadata": {
        "tags": [],
        "id": "4fd5b34b-e653-4156-9e82-c10c9b3d802d"
      },
      "source": [
        "## Python Lottery Dataset Analyze\n",
        "\n",
        "This Jupyter notebook illustrates several methods for analyzing a dataset containing historical results from various lotteries. The example demonstrates how to analyze the linear relationships between individual fields by extending the dataset with astronomical data. Additionally, it shows how to visualize the distribution of numbers in specific positions. The `skyfield` package is used to calculate the distances between celestial bodies, and this data is subsequently appended to the dataset.\n",
        "\n",
        "(c) 2022 Marcin \"szczyglis\" Szczygliński\n",
        "\n",
        "GitHub page: https://github.com/szczyglis-dev/python-lottery-dataset-analyze\n",
        "\n",
        "Email: szczyglis@protonmail.com\n",
        "\n",
        "Version: 1.0.1\n",
        "\n",
        "This package is licensed under the MIT License.\n",
        "\n",
        "License text available at https://opensource.org/licenses/MIT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6a731618-3cd0-49df-8bf9-9abfd501857e",
      "metadata": {
        "id": "6a731618-3cd0-49df-8bf9-9abfd501857e"
      },
      "outputs": [],
      "source": [
        "!pip install pandas\n",
        "!pip install matplotlib\n",
        "!pip install seaborn\n",
        "!pip install scipy\n",
        "!pip install skyfield"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "838cc14f-a9bc-4c5f-9506-3a6a355aa698",
      "metadata": {
        "id": "838cc14f-a9bc-4c5f-9506-3a6a355aa698"
      },
      "source": [
        "**1.  Configuration, Initialization, and Module Import**\n",
        "\n",
        "We will use historical drawing results for several popular number lotteries in Poland as input data. The draw results will be downloaded to `CSV` files and saved in a local directory on the disk.\n",
        "\n",
        "The block below includes configurations for each of these lotteries, such as the names of the columns that will be used later in the `DataFrame` object created from the dataset, number ranges, and the format in which the individual drawing dates are saved. At the end of the block, you should specify the name of the lottery to be analyzed. This block will also load astronomical data for several celestial bodies, which will then be used to extend the dataset with the distances between these celestial bodies at the time of each draw. This data will be used to test the correlation between these events/variables."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7b969b87-b0d7-44ca-8033-341f854270ac",
      "metadata": {
        "id": "7b969b87-b0d7-44ca-8033-341f854270ac"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import math\n",
        "from datetime import datetime\n",
        "import requests\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from scipy.stats import linregress\n",
        "from skyfield.api import load\n",
        "\n",
        "# define URLs with lotteries historical results in CSV\n",
        "csv_urls = {\n",
        "    'lotto': 'https://www.wynikilotto.net.pl/download/lotto.csv',\n",
        "    'lotto_plus': 'https://www.wynikilotto.net.pl/download/lotto_plus.csv',\n",
        "    'eurojackpot': 'https://www.wynikilotto.net.pl/download/eurojackpot.csv',\n",
        "    'minilotto': 'https://www.wynikilotto.net.pl/download/mini_lotto.csv',\n",
        "    'multi': 'https://www.wynikilotto.net.pl/download/multi_multi.csv'\n",
        "}\n",
        "\n",
        "# [CSV config]\n",
        "# header - list with CSV column names\n",
        "#   idx - number of record\n",
        "#   date - date field\n",
        "#   time - time/hour field\n",
        "#   n(x) - primary number(x) field\n",
        "#   m(x) - secondary number(x) field\n",
        "# n_range - list with primary numbers range [from, to]\n",
        "# m_range - list with secondary numbers range [from, to]\n",
        "# n_count - number of primary numbers\n",
        "# m_count - number of secondary numbers\n",
        "# date_format - date field string format\n",
        "\n",
        "csv_config = {\n",
        "    'lotto': {\n",
        "        'header': ['idx', 'date', 'n1', 'n2', 'n3', 'n4', 'n5', 'n6'],\n",
        "        'n_range': [1, 49],\n",
        "        'm_range': [],\n",
        "        'n_count': 6,\n",
        "        'm_count': 0,\n",
        "        'date_format': '%d.%m.%Y'\n",
        "    },\n",
        "    'lotto_plus': {\n",
        "        'header': ['idx', 'date', 'n1', 'n2', 'n3', 'n4', 'n5', 'n6'],\n",
        "        'n_range': [1, 49],\n",
        "        'm_range': [],\n",
        "        'n_count': 6,\n",
        "        'm_count': 0,\n",
        "        'date_format': '%d.%m.%Y'\n",
        "    },\n",
        "    'eurojackpot': {\n",
        "        'header': ['idx', 'date', 'n1', 'n2', 'n3', 'n4', 'n5', 'm1', 'm2'],\n",
        "        'n_range': [1, 50],\n",
        "        'm_range': [1, 12],\n",
        "        'n_count': 5,\n",
        "        'm_count': 2,\n",
        "        'date_format': '%d.%m.%Y'\n",
        "    },\n",
        "    'minilotto': {\n",
        "        'header': ['idx', 'date', 'n1', 'n2', 'n3', 'n4', 'n5'],\n",
        "        'n_range': [1, 42],\n",
        "        'm_range': [],\n",
        "        'n_count': 5,\n",
        "        'm_count': 0,\n",
        "        'date_format': '%d.%m.%Y'\n",
        "    },\n",
        "    'multi': {\n",
        "        'header': ['idx', 'date', 'time', 'n1', 'n2', 'n3', 'n4', 'n5', 'n6', 'n7', 'n8', 'n9',\n",
        "                    'n10', 'n11', 'n12', 'n13', 'n14', 'n15', 'n16', 'n17', 'n18', 'n19', 'n20', 'm1'],\n",
        "        'n_range': [1, 80],\n",
        "        'm_range': [1, 80],\n",
        "        'n_count': 20,\n",
        "        'm_count': 1,\n",
        "        'date_format': '%d.%m.%Y'\n",
        "    }\n",
        "}\n",
        "\n",
        "# specify download dir for CSV files\n",
        "csv_dir = os.path.join(os.getcwd(), 'csv')\n",
        "\n",
        "# choose lottery\n",
        "lottery = 'lotto'\n",
        "\n",
        "# init astronomical data\n",
        "planets = load('de421.bsp')\n",
        "earth, moon, sun, mars = planets['earth'], planets['moon'], planets['sun'], planets['mars']"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dcde7bc2-1fe6-46da-8534-b4fcd6eb938e",
      "metadata": {
        "id": "dcde7bc2-1fe6-46da-8534-b4fcd6eb938e"
      },
      "source": [
        "**2. Function Definitions**\n",
        "\n",
        "The following cell defines the functions that will be used in subsequent blocks. These functions include downloading and saving the dataset to CSV files and extending the downloaded dataset with new values, which will then be used for further analysis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b19dd9be-1155-462f-ada7-a51dc269700d",
      "metadata": {
        "id": "b19dd9be-1155-462f-ada7-a51dc269700d"
      },
      "outputs": [],
      "source": [
        "# create directory for CSV download if not exists\n",
        "def csv_dir_create(csv_dir):\n",
        "    if not os.path.exists(csv_dir):\n",
        "        os.makedirs(csv_dir)\n",
        "\n",
        "\n",
        "# download and save CSV dataset\n",
        "def csv_update(csv_urls, csv_dir):\n",
        "    for k, url in csv_urls.items():\n",
        "        r = requests.get(url, allow_redirects=True)\n",
        "        name = k + '.csv'\n",
        "        fname = os.path.join(csv_dir, name)\n",
        "        open(fname, 'wb').write(r.content)\n",
        "        print('Downloaded: ' + fname)\n",
        "\n",
        "\n",
        "# load CSV dataset\n",
        "def csv_load(name, header, csv_dir):\n",
        "    file = os.path.join(csv_dir, name+'.csv')\n",
        "    return pd.read_csv(file, header=None, names=header)\n",
        "\n",
        "\n",
        "# save dataframe to CSV file\n",
        "def csv_save(df, name, csv_dir):\n",
        "    file = os.path.join(csv_dir, name+'.csv')\n",
        "    df.to_csv(file, index=False)\n",
        "\n",
        "\n",
        "# append date part to series\n",
        "def df_append_date_part(part, row, dt_format):\n",
        "    dt = datetime.strptime(row.date, dt_format)\n",
        "    return int(dt.strftime(part))\n",
        "\n",
        "\n",
        "# get date parts\n",
        "def df_get_date_parts(row, dt_format):\n",
        "    dt = datetime.strptime(row.date, dt_format)\n",
        "    y = int(dt.strftime('%Y'))\n",
        "    m = int(dt.strftime('%m'))\n",
        "    d = int(dt.strftime('%d'))\n",
        "    return y, m, d\n",
        "\n",
        "\n",
        "# append astro planets distance to series\n",
        "def df_append_astro_distance(obj1, obj2, row, dt_format):\n",
        "    ts = load.timescale()\n",
        "    y, m, d = df_get_date_parts(row, dt_format)\n",
        "    t = ts.utc(y, m, d, 9, 0)\n",
        "    return obj1.at(t).observe(obj2).apparent().distance().au\n",
        "\n",
        "\n",
        "# append numbers ranges to series\n",
        "def df_append_range(row, num_idx):\n",
        "    j = 10;\n",
        "    while j <= 100:\n",
        "        if row[num_idx] >= (j - 10) and row[num_idx] < j:\n",
        "            return int((j - 10)/10)\n",
        "        j+= 10"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "nUrj0mUBfEAA"
      },
      "id": "nUrj0mUBfEAA",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "63915abc-18dc-4c48-95d4-684a6df093fe",
      "metadata": {
        "id": "63915abc-18dc-4c48-95d4-684a6df093fe"
      },
      "source": [
        "**3. Download CSV Files with Datasets**\n",
        "\n",
        "The cell below will download CSV files containing the datasets. **Tip:** to avoid downloading new data and instead use the already downloaded files, this block should be commented out after the data has been downloaded."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fe88946b-14fd-4c09-9e65-989a8a85979c",
      "metadata": {
        "id": "fe88946b-14fd-4c09-9e65-989a8a85979c"
      },
      "outputs": [],
      "source": [
        "print('Downloading datasets....')\n",
        "\n",
        "csv_dir_create(csv_dir)\n",
        "csv_update(csv_urls, csv_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "52e7e0c9-bc3a-4ab5-b775-dcda7df840a0",
      "metadata": {
        "id": "52e7e0c9-bc3a-4ab5-b775-dcda7df840a0"
      },
      "source": [
        "**4. Extend the Dataset with Additional Fields**\n",
        "\n",
        "The following code expands the dataset with new fields. Numerical values for the saved draw dates will be added, such as year, month, day, day of the week, and day of the year. Additionally, the distances to individual celestial bodies (Earth-Moon, Earth-Sun, Earth-Mars) at the time of each draw will be calculated and appended to the dataset. The dataset will also include fields that define the range of numbers at a given position."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c78fb25b-5697-4785-9f11-1f905e10d751",
      "metadata": {
        "id": "c78fb25b-5697-4785-9f11-1f905e10d751"
      },
      "outputs": [],
      "source": [
        "# get CSV config for selected lottery\n",
        "cfg = csv_config[lottery]\n",
        "dt_format = cfg['date_format']\n",
        "header = cfg['header']\n",
        "\n",
        "# load CSV dataset and create Data Frame from it\n",
        "df = csv_load(lottery, header, csv_dir)\n",
        "\n",
        "# append date parts as integers\n",
        "df['year'] = df.apply(lambda row: df_append_date_part('%Y', row, dt_format), axis=1)\n",
        "df['month'] = df.apply(lambda row: df_append_date_part('%m', row, dt_format), axis=1)\n",
        "df['day'] = df.apply(lambda row: df_append_date_part('%d', row, dt_format), axis=1)\n",
        "df['day_of_week'] = df.apply(lambda row: df_append_date_part('%w', row, dt_format), axis=1)\n",
        "df['day_of_year'] = df.apply(lambda row: df_append_date_part('%j', row, dt_format), axis=1)\n",
        "\n",
        "# append distances from earth to moon, sun & mars\n",
        "df['dist_moon_au'] = df.apply(lambda row: df_append_astro_distance(earth, moon, row, dt_format), axis=1)\n",
        "df['dist_sun_au'] = df.apply(lambda row: df_append_astro_distance(earth, sun, row, dt_format), axis=1)\n",
        "df['dist_mars_au'] = df.apply(lambda row: df_append_astro_distance(earth, mars, row, dt_format), axis=1)\n",
        "\n",
        "# append decimal ranges of numbers to n(i)r fields that corresponds numbers at positions n1-n(i)\n",
        "limit = cfg['n_count']\n",
        "if limit > 0:\n",
        "    for i in range(1, limit+1):\n",
        "        range_field = 'n' + str(i) + 'r'\n",
        "        num_field = 'n' + str(i)\n",
        "        df[range_field] = df.apply(lambda row: df_append_range(row, num_field), axis=1)\n",
        "\n",
        "# append decimal ranges of numbers to m(i)r fields that corresponds numbers at positions m1-m(i)\n",
        "limit = cfg['m_count']\n",
        "if limit > 0:\n",
        "    for i in range(1, limit+1):\n",
        "        range_field = 'm' + str(i) + 'r'\n",
        "        num_field = 'm' + str(i)\n",
        "        df[range_field] = df.apply(lambda row: df_append_range(row, num_field), axis=1)\n",
        "\n",
        "# save extended dataset with appended extra data\n",
        "csv_save(df, lottery + '_extended', csv_dir)\n",
        "\n",
        "#df = df.iloc[4424:,:] # you can truncate dataset to period in time"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ddb8061d-3aca-4152-a0fc-c038eb555e0e",
      "metadata": {
        "id": "ddb8061d-3aca-4152-a0fc-c038eb555e0e"
      },
      "source": [
        "**5. Linear Regression Relationship Calculation**\n",
        "\n",
        "The code below calculates the correlation between various events, such as the impact of the distances between celestial bodies on the lottery numbers and the correlation between individual lottery numbers themselves. The correlation between the distances of celestial bodies and the numbers is expected to oscillate around 0, indicating that these events are not correlated. A slightly higher correlation may appear when attempting to correlate the numbers drawn in the same draw with each other."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7148a62c-1ac8-4eb1-a2dc-2ff2ecae2c1b",
      "metadata": {
        "id": "7148a62c-1ac8-4eb1-a2dc-2ff2ecae2c1b"
      },
      "outputs": [],
      "source": [
        "# define relationship pairs to check\n",
        "relations = {\n",
        "    'sun_n1': ['dist_sun_au', 'n1'],\n",
        "    'moon_n1': ['dist_moon_au', 'n1'],\n",
        "    'mars_n1': ['dist_mars_au', 'n1'],\n",
        "    'sun_n1r': ['dist_sun_au', 'n1r'],\n",
        "    'moon_n1r': ['dist_moon_au', 'n1r'],\n",
        "    'day_n1': ['day', 'n1'],\n",
        "    'month_n1': ['month', 'n1'],\n",
        "    'year_n1': ['year', 'n1'],\n",
        "    'dweek_n1': ['day_of_week', 'n1'],\n",
        "    'dweek_n1r': ['day_of_week', 'n1r'],\n",
        "    'dyear_n1': ['day_of_year', 'n1'],\n",
        "    'dyear_n1r': ['day_of_year', 'n1r'],\n",
        "    'n1_n2': ['n1', 'n2'],\n",
        "    'n2_n3': ['n2', 'n3'],\n",
        "    'n3_n4': ['n3', 'n4'],\n",
        "    'n4_n5': ['n4', 'n5'],\n",
        "    'n5_n6': ['n5', 'n6']\n",
        "}\n",
        "\n",
        "# calculate linear regression relationship between fields\n",
        "print(\"[R] - Linear Regression Relationship:\\n\")\n",
        "rX = []\n",
        "rY = []\n",
        "for name, item in relations.items():\n",
        "    x1 = item[0]\n",
        "    x2 = item[1]\n",
        "    if x1 in df and x2 in df:\n",
        "        slope, intercept, r, p, std_err = linregress(df[x1], df[x2])\n",
        "        rX.append(name)\n",
        "        rY.append(r)\n",
        "        print(x1+' > '+x2+': ' + str(r))\n",
        "\n",
        "rY, rX = zip(*sorted(zip(rY, rX)))\n",
        "\n",
        "# display relationships on plot\n",
        "fig = plt.figure()\n",
        "fig.set_figheight(5)\n",
        "fig.set_figwidth(15)\n",
        "ax = fig.add_subplot(1, 1, 1)\n",
        "ax.set_title('Relationship')\n",
        "ax.set_ylabel('Pair')\n",
        "ax.set_xlabel('R value')\n",
        "ax.barh(rX, rY)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7cd63db1-b9d3-4f14-9b28-e31160988e3c",
      "metadata": {
        "id": "7cd63db1-b9d3-4f14-9b28-e31160988e3c"
      },
      "source": [
        "**6. Distribution of the Frequency of Numbers in Specific Positions**\n",
        "\n",
        "The following cell displays the frequency distribution of the drawn numbers, categorized by their positions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f21dfc43-72b7-44f1-834b-12bef1eeaa6f",
      "metadata": {
        "id": "f21dfc43-72b7-44f1-834b-12bef1eeaa6f"
      },
      "outputs": [],
      "source": [
        "print('Distribution of the Frequency of Numbers in Specific Positions:')\n",
        "\n",
        "num_of_numbers = cfg['n_count']\n",
        "cols = 3\n",
        "rows = math.ceil(num_of_numbers/cols)\n",
        "row = 0\n",
        "col = 0\n",
        "f, ax = plt.subplots(rows, cols, figsize=(25, 15))\n",
        "for i in range(1, num_of_numbers+1):\n",
        "    idx = 'n' + str(i)\n",
        "    data = df[idx].to_numpy()\n",
        "    sns.histplot(data, kde=True, ax=ax[row][col])\n",
        "    ax[row][col].set_title('Position: '+idx)\n",
        "    ax[row][col].set_xlabel('Number')\n",
        "    ax[row][col].set_ylabel('Count')\n",
        "    ax[row][col].axvline(x=data.mean(), color='red')\n",
        "    if col >= (cols - 1):\n",
        "        col = 0\n",
        "        row+=1\n",
        "    else:\n",
        "        col+= 1\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a061f4e2-ef12-49eb-a5c0-702c3a9cadb5",
      "metadata": {
        "id": "a061f4e2-ef12-49eb-a5c0-702c3a9cadb5"
      },
      "source": [
        "**7. Display the Dataset**\n",
        "\n",
        "The cell below displays the extended dataset that was prepared in the previous steps."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "619632a7-f0aa-46e4-b078-da2867d607f5",
      "metadata": {
        "id": "619632a7-f0aa-46e4-b078-da2867d607f5"
      },
      "outputs": [],
      "source": [
        "# display dataset\n",
        "print(\"[DATASET]\\n\")\n",
        "print(df.to_string())"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}