job-scraper.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[],"dockerImageVersionId":30553,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# JOB SCRAPER\n\n* #### This notebook automates the extraction, cleaning, and transformation of job postings data from a specified website using Selenium WebDriver.\n* #### It then uploads the processed data to GitHub, providing a streamlined pipeline for data acquisition and storage.","metadata":{}},{"cell_type":"markdown","source":"### >>","metadata":{}},{"cell_type":"markdown","source":"## This notebook performs the following tasks:\n\n* ####  Installs necessary dependencies.\n* ####  Downloads and installs Chrome and Chromedriver.\n* ####  Installs necessary Python packages.\n* ####  Imports required libraries.\n* ####  Initializes Selenium WebDriver.\n* ####  Defines functions for configuring Selenium WebDriver, scraping job details, getting GitHub repository, uploading data to GitHub, sending email notification, extracting data from GitHub, and the main function.\n* ####  Calls the main function to execute the entire process.","metadata":{}},{"cell_type":"markdown","source":"### >>","metadata":{}},{"cell_type":"markdown","source":"## **Section 1** - Installation and Setup","metadata":{}},{"cell_type":"markdown","source":"#### This section covers the installation of necessary dependencies, downloading and installing Chrome and Chromedriver, installing required Python packages, and importing libraries.","metadata":{}},{"cell_type":"markdown","source":"## Update Linux Dependencies","metadata":{}},{"cell_type":"code","source":"!apt-get update -y\n!apt-get install -y \\\nlibglib2.0-0 \\\nlibnss3 \\\nlibdbus-glib-1-2 \\\nlibgconf-2-4 \\\nlibfontconfig1 \\\nlibvulkan1 \\\ngconf2-common \\\nlibwayland-server0 \\\nlibgbm1 \\\nudev \\\nlibu2f-udev \n!apt --fix-broken install -y  ","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:54:26.887716Z","iopub.execute_input":"2024-05-30T06:54:26.888187Z","iopub.status.idle":"2024-05-30T06:54:49.373393Z","shell.execute_reply.started":"2024-05-30T06:54:26.888140Z","shell.execute_reply":"2024-05-30T06:54:49.372263Z"},"trusted":true},"execution_count":2,"outputs":[{"name":"stdout","text":"Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease\nGet:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]        \nGet:4 http://packages.cloud.google.com/apt gcsfuse-focal InRelease [1225 B]    \nGet:5 https://packages.cloud.google.com/apt cloud-sdk InRelease [1616 B]       \nGet:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB]\nGet:7 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3616 kB]\nGet:8 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3766 kB]\nGet:9 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [29.8 kB]\nGet:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1208 kB]\nGet:11 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3674 kB]\nGet:12 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1504 kB]\nGet:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [4148 kB]\nGet:14 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [32.5 kB]\nGet:15 https://packages.cloud.google.com/apt cloud-sdk/main amd64 Packages [2999 kB]\nGet:16 https://packages.cloud.google.com/apt cloud-sdk/main all Packages [1449 kB]\nReading package lists... Done                              \nE: Repository 'http://packages.cloud.google.com/apt gcsfuse-focal InRelease' changed its 'Origin' value from 'gcsfuse-jessie' to 'gcsfuse-focal'\nE: Repository 'http://packages.cloud.google.com/apt gcsfuse-focal InRelease' changed its 'Label' value from 'gcsfuse-jessie' to 'gcsfuse-focal'\nN: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.\nReading package lists... Done\nBuilding dependency tree       \nReading state information... Done\nlibfontconfig1 is already the newest version (2.13.1-2ubuntu3).\nlibvulkan1 is already the newest version (1.2.131.2-1).\nlibvulkan1 set to manually installed.\nThe following additional packages will be installed:\n  gconf-service gconf-service-backend libglib2.0-bin libudev1\nRecommended packages:\n  xdg-user-dirs\nThe following NEW packages will be installed:\n  gconf-service gconf-service-backend gconf2-common libdbus-glib-1-2 libgbm1\n  libgconf-2-4 libu2f-udev libwayland-server0 udev\nThe following packages will be upgraded:\n  libglib2.0-0 libglib2.0-bin libnss3 libudev1\n4 upgraded, 9 newly installed, 0 to remove and 140 not upgraded.\nNeed to get 5180 kB of archives.\nAfter this operation, 18.4 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libudev1 amd64 245.4-4ubuntu3.23 [75.6 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-bin amd64 2.64.6-1~ubuntu20.04.7 [72.8 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libglib2.0-0 amd64 2.64.6-1~ubuntu20.04.7 [1289 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 udev amd64 245.4-4ubuntu3.23 [1366 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 gconf2-common all 3.2.6-6ubuntu1 [698 kB]\nGet:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libgconf-2-4 amd64 3.2.6-6ubuntu1 [84.8 kB]\nGet:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 gconf-service-backend amd64 3.2.6-6ubuntu1 [58.6 kB]\nGet:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 gconf-service amd64 3.2.6-6ubuntu1 [17.4 kB]\nGet:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libwayland-server0 amd64 1.18.0-1ubuntu0.1 [31.3 kB]\nGet:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libgbm1 amd64 21.2.6-0ubuntu0.1~20.04.2 [29.2 kB]\nGet:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnss3 amd64 2:3.98-0ubuntu0.20.04.2 [1391 kB]\nGet:13 http://archive.ubuntu.com/ubuntu focal/main amd64 libu2f-udev all 1.1.10-1 [6108 B]\nFetched 5180 kB in 2s (2339 kB/s) \n(Reading database ... 107763 files and directories currently installed.)\nPreparing to unpack .../libudev1_245.4-4ubuntu3.23_amd64.deb ...\nUnpacking libudev1:amd64 (245.4-4ubuntu3.23) over (245.4-4ubuntu3.22) ...\nSetting up libudev1:amd64 (245.4-4ubuntu3.23) ...\n(Reading database ... 107763 files and directories currently installed.)\nPreparing to unpack .../00-libglib2.0-bin_2.64.6-1~ubuntu20.04.7_amd64.deb ...\nUnpacking libglib2.0-bin (2.64.6-1~ubuntu20.04.7) over (2.64.6-1~ubuntu20.04.6) ...\nPreparing to unpack .../01-libglib2.0-0_2.64.6-1~ubuntu20.04.7_amd64.deb ...\nUnpacking libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.7) over (2.64.6-1~ubuntu20.04.6) ...\nSelecting previously unselected package udev.\nPreparing to unpack .../02-udev_245.4-4ubuntu3.23_amd64.deb ...\nUnpacking udev (245.4-4ubuntu3.23) ...\nSelecting previously unselected package libdbus-glib-1-2:amd64.\nPreparing to unpack .../03-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ...\nUnpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ...\nSelecting previously unselected package gconf2-common.\nPreparing to unpack .../04-gconf2-common_3.2.6-6ubuntu1_all.deb ...\nUnpacking gconf2-common (3.2.6-6ubuntu1) ...\nSelecting previously unselected package libgconf-2-4:amd64.\nPreparing to unpack .../05-libgconf-2-4_3.2.6-6ubuntu1_amd64.deb ...\nUnpacking libgconf-2-4:amd64 (3.2.6-6ubuntu1) ...\nSelecting previously unselected package gconf-service-backend.\nPreparing to unpack .../06-gconf-service-backend_3.2.6-6ubuntu1_amd64.deb ...\nUnpacking gconf-service-backend (3.2.6-6ubuntu1) ...\nSelecting previously unselected package gconf-service.\nPreparing to unpack .../07-gconf-service_3.2.6-6ubuntu1_amd64.deb ...\nUnpacking gconf-service (3.2.6-6ubuntu1) ...\nSelecting previously unselected package libwayland-server0:amd64.\nPreparing to unpack .../08-libwayland-server0_1.18.0-1ubuntu0.1_amd64.deb ...\nUnpacking libwayland-server0:amd64 (1.18.0-1ubuntu0.1) ...\nSelecting previously unselected package libgbm1:amd64.\nPreparing to unpack .../09-libgbm1_21.2.6-0ubuntu0.1~20.04.2_amd64.deb ...\nUnpacking libgbm1:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...\nPreparing to unpack .../10-libnss3_2%3a3.98-0ubuntu0.20.04.2_amd64.deb ...\nUnpacking libnss3:amd64 (2:3.98-0ubuntu0.20.04.2) over (2:3.49.1-1ubuntu1.9) ...\nSelecting previously unselected package libu2f-udev.\nPreparing to unpack .../11-libu2f-udev_1.1.10-1_all.deb ...\nUnpacking libu2f-udev (1.1.10-1) ...\nSetting up libwayland-server0:amd64 (1.18.0-1ubuntu0.1) ...\nSetting up libgbm1:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...\nSetting up libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.7) ...\nSetting up libglib2.0-bin (2.64.6-1~ubuntu20.04.7) ...\nSetting up libnss3:amd64 (2:3.98-0ubuntu0.20.04.2) ...\nSetting up gconf2-common (3.2.6-6ubuntu1) ...\n\nCreating config file /etc/gconf/2/path with new version\nSetting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ...\nSetting up udev (245.4-4ubuntu3.23) ...\ninvoke-rc.d: could not determine current runlevel\ninvoke-rc.d: policy-rc.d denied execution of start.\nSetting up libgconf-2-4:amd64 (3.2.6-6ubuntu1) ...\nSetting up libu2f-udev (1.1.10-1) ...\nFailed to send reload request: No such file or directory\nSetting up gconf-service (3.2.6-6ubuntu1) ...\nSetting up gconf-service-backend (3.2.6-6ubuntu1) ...\nProcessing triggers for libc-bin (2.31-0ubuntu9.9) ...\nProcessing triggers for systemd (245.4-4ubuntu3.22) ...\nProcessing triggers for man-db (2.9.1-1) ...\nReading package lists... Done\nBuilding dependency tree       \nReading state information... Done\n0 upgraded, 0 newly installed, 0 to remove and 140 not upgraded.\n","output_type":"stream"}]},{"cell_type":"markdown","source":"## Install Chrome","metadata":{}},{"cell_type":"code","source":"!wget -P /tmp https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/116.0.5845.96/linux64/chrome-linux64.zip\n!unzip /tmp/chrome-linux64.zip -d /usr/bin/","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:54:49.375065Z","iopub.execute_input":"2024-05-30T06:54:49.375430Z","iopub.status.idle":"2024-05-30T06:54:57.849722Z","shell.execute_reply.started":"2024-05-30T06:54:49.375398Z","shell.execute_reply":"2024-05-30T06:54:57.848392Z"},"trusted":true},"execution_count":3,"outputs":[{"name":"stdout","text":"--2024-05-30 06:54:50--  https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/116.0.5845.96/linux64/chrome-linux64.zip\nResolving edgedl.me.gvt1.com (edgedl.me.gvt1.com)... 34.104.35.123, 2600:1900:4110:86f::\nConnecting to edgedl.me.gvt1.com (edgedl.me.gvt1.com)|34.104.35.123|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 145898081 (139M) [application/octet-stream]\nSaving to: ‘/tmp/chrome-linux64.zip’\n\nchrome-linux64.zip  100%[===================>] 139.14M  61.7MB/s    in 2.3s    \n\n2024-05-30 06:54:52 (61.7 MB/s) - ‘/tmp/chrome-linux64.zip’ saved [145898081/145898081]\n\nArchive:  /tmp/chrome-linux64.zip\n  inflating: /usr/bin/chrome-linux64/MEIPreload/manifest.json  \n  inflating: /usr/bin/chrome-linux64/MEIPreload/preloaded_data.pb  \n  inflating: /usr/bin/chrome-linux64/chrome  \n  inflating: /usr/bin/chrome-linux64/chrome-wrapper  \n  inflating: /usr/bin/chrome-linux64/chrome_100_percent.pak  \n  inflating: /usr/bin/chrome-linux64/chrome_200_percent.pak  \n  inflating: /usr/bin/chrome-linux64/chrome_crashpad_handler  \n  inflating: /usr/bin/chrome-linux64/chrome_sandbox  \n  inflating: /usr/bin/chrome-linux64/icudtl.dat  \n  inflating: /usr/bin/chrome-linux64/libEGL.so  \n  inflating: /usr/bin/chrome-linux64/libGLESv2.so  \n  inflating: /usr/bin/chrome-linux64/libvk_swiftshader.so  \n  inflating: /usr/bin/chrome-linux64/libvulkan.so.1  \n  inflating: /usr/bin/chrome-linux64/nacl_helper  \n  inflating: /usr/bin/chrome-linux64/nacl_helper_bootstrap  \n  inflating: /usr/bin/chrome-linux64/nacl_irt_x86_64.nexe  \n extracting: /usr/bin/chrome-linux64/product_logo_48.png  \n  inflating: /usr/bin/chrome-linux64/resources.pak  \n  inflating: /usr/bin/chrome-linux64/v8_context_snapshot.bin  \n  inflating: /usr/bin/chrome-linux64/vk_swiftshader_icd.json  \n  inflating: /usr/bin/chrome-linux64/xdg-mime  \n  inflating: /usr/bin/chrome-linux64/xdg-settings  \n   creating: /usr/bin/chrome-linux64/locales/\n  inflating: /usr/bin/chrome-linux64/locales/fr.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ur.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/sw.pak  \n  inflating: /usr/bin/chrome-linux64/locales/ar.pak  \n  inflating: /usr/bin/chrome-linux64/locales/sk.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/hu.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ms.pak  \n  inflating: /usr/bin/chrome-linux64/locales/vi.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/es.pak  \n  inflating: /usr/bin/chrome-linux64/locales/he.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/hr.pak  \n  inflating: /usr/bin/chrome-linux64/locales/en-GB.pak  \n  inflating: /usr/bin/chrome-linux64/locales/zh-CN.pak  \n  inflating: /usr/bin/chrome-linux64/locales/pt-BR.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/en-US.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/de.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/af.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/fil.pak  \n  inflating: /usr/bin/chrome-linux64/locales/el.pak  \n  inflating: /usr/bin/chrome-linux64/locales/bg.pak  \n  inflating: /usr/bin/chrome-linux64/locales/fa.pak  \n  inflating: /usr/bin/chrome-linux64/locales/uk.pak  \n  inflating: /usr/bin/chrome-linux64/locales/kn.pak  \n  inflating: /usr/bin/chrome-linux64/locales/en-GB.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/sl.pak  \n  inflating: /usr/bin/chrome-linux64/locales/sv.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ko.pak  \n  inflating: /usr/bin/chrome-linux64/locales/nb.pak  \n  inflating: /usr/bin/chrome-linux64/locales/tr.pak  \n  inflating: /usr/bin/chrome-linux64/locales/th.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ar.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/pt-PT.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/pt-PT.pak  \n  inflating: /usr/bin/chrome-linux64/locales/vi.pak  \n  inflating: /usr/bin/chrome-linux64/locales/ml.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/es.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/mr.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/tr.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/hi.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/fa.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ja.pak  \n  inflating: /usr/bin/chrome-linux64/locales/bg.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/sr.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/fi.pak  \n  inflating: /usr/bin/chrome-linux64/locales/ca.pak  \n  inflating: /usr/bin/chrome-linux64/locales/et.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ko.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/gu.pak  \n  inflating: /usr/bin/chrome-linux64/locales/et.pak  \n  inflating: /usr/bin/chrome-linux64/locales/it.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/lv.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/sr.pak  \n  inflating: /usr/bin/chrome-linux64/locales/bn.pak  \n  inflating: /usr/bin/chrome-linux64/locales/pl.pak  \n  inflating: /usr/bin/chrome-linux64/locales/el.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/lv.pak  \n  inflating: /usr/bin/chrome-linux64/locales/ca.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/uk.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/nl.pak  \n  inflating: /usr/bin/chrome-linux64/locales/sw.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/de.pak  \n  inflating: /usr/bin/chrome-linux64/locales/af.pak  \n  inflating: /usr/bin/chrome-linux64/locales/hi.pak  \n  inflating: /usr/bin/chrome-linux64/locales/pl.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/he.pak  \n  inflating: /usr/bin/chrome-linux64/locales/sv.pak  \n  inflating: /usr/bin/chrome-linux64/locales/ms.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/sk.pak  \n  inflating: /usr/bin/chrome-linux64/locales/bn.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/cs.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/kn.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/zh-CN.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/id.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/es-419.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/lt.pak  \n  inflating: /usr/bin/chrome-linux64/locales/ur.pak  \n  inflating: /usr/bin/chrome-linux64/locales/ta.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/zh-TW.pak  \n  inflating: /usr/bin/chrome-linux64/locales/te.pak  \n  inflating: /usr/bin/chrome-linux64/locales/ja.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/en-US.pak  \n  inflating: /usr/bin/chrome-linux64/locales/nb.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/it.pak  \n  inflating: /usr/bin/chrome-linux64/locales/es-419.pak  \n  inflating: /usr/bin/chrome-linux64/locales/nl.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/cs.pak  \n  inflating: /usr/bin/chrome-linux64/locales/zh-TW.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ru.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/da.pak  \n  inflating: /usr/bin/chrome-linux64/locales/lt.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ml.pak  \n  inflating: /usr/bin/chrome-linux64/locales/mr.pak  \n  inflating: /usr/bin/chrome-linux64/locales/pt-BR.pak  \n  inflating: /usr/bin/chrome-linux64/locales/da.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/te.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/fil.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ta.pak  \n  inflating: /usr/bin/chrome-linux64/locales/id.pak  \n  inflating: /usr/bin/chrome-linux64/locales/am.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/am.pak  \n  inflating: /usr/bin/chrome-linux64/locales/hu.pak  \n  inflating: /usr/bin/chrome-linux64/locales/fi.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/sl.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ro.pak  \n  inflating: /usr/bin/chrome-linux64/locales/th.pak  \n  inflating: /usr/bin/chrome-linux64/locales/fr.pak  \n  inflating: /usr/bin/chrome-linux64/locales/gu.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/ru.pak  \n  inflating: /usr/bin/chrome-linux64/locales/ro.pak.info  \n  inflating: /usr/bin/chrome-linux64/locales/hr.pak.info  \n   creating: /usr/bin/chrome-linux64/resources/\n   creating: /usr/bin/chrome-linux64/resources/inspector_overlay/\n  inflating: /usr/bin/chrome-linux64/resources/inspector_overlay/inspector_overlay_resources.grd  \n  inflating: /usr/bin/chrome-linux64/resources/inspector_overlay/main.js  \n","output_type":"stream"}]},{"cell_type":"code","source":"!/usr/bin/chrome-linux64/chrome --version","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:54:57.853454Z","iopub.execute_input":"2024-05-30T06:54:57.853976Z","iopub.status.idle":"2024-05-30T06:54:59.009754Z","shell.execute_reply.started":"2024-05-30T06:54:57.853925Z","shell.execute_reply":"2024-05-30T06:54:59.008438Z"},"trusted":true},"execution_count":4,"outputs":[{"name":"stdout","text":"Google Chrome for Testing 116.0.5845.96 \n","output_type":"stream"}]},{"cell_type":"markdown","source":"## Install ChromeDriver","metadata":{}},{"cell_type":"code","source":"!wget -P /tmp https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/116.0.5845.96/linux64/chromedriver-linux64.zip\n!unzip /tmp/chromedriver-linux64.zip -d /usr/bin/","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:54:59.012981Z","iopub.execute_input":"2024-05-30T06:54:59.013469Z","iopub.status.idle":"2024-05-30T06:55:01.606858Z","shell.execute_reply.started":"2024-05-30T06:54:59.013424Z","shell.execute_reply":"2024-05-30T06:55:01.605385Z"},"trusted":true},"execution_count":5,"outputs":[{"name":"stdout","text":"--2024-05-30 06:54:59--  https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/116.0.5845.96/linux64/chromedriver-linux64.zip\nResolving edgedl.me.gvt1.com (edgedl.me.gvt1.com)... 34.104.35.123, 2600:1900:4110:86f::\nConnecting to edgedl.me.gvt1.com (edgedl.me.gvt1.com)|34.104.35.123|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 7271942 (6.9M) [application/octet-stream]\nSaving to: ‘/tmp/chromedriver-linux64.zip’\n\nchromedriver-linux6 100%[===================>]   6.93M  32.4MB/s    in 0.2s    \n\n2024-05-30 06:55:00 (32.4 MB/s) - ‘/tmp/chromedriver-linux64.zip’ saved [7271942/7271942]\n\nArchive:  /tmp/chromedriver-linux64.zip\n  inflating: /usr/bin/chromedriver-linux64/LICENSE.chromedriver  \n  inflating: /usr/bin/chromedriver-linux64/chromedriver  \n","output_type":"stream"}]},{"cell_type":"code","source":"!/usr/bin/chromedriver-linux64/chromedriver --version","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:55:01.608817Z","iopub.execute_input":"2024-05-30T06:55:01.609209Z","iopub.status.idle":"2024-05-30T06:55:02.625919Z","shell.execute_reply.started":"2024-05-30T06:55:01.609161Z","shell.execute_reply":"2024-05-30T06:55:02.624723Z"},"trusted":true},"execution_count":6,"outputs":[{"name":"stdout","text":"ChromeDriver 116.0.5845.96 (1a391816688002153ef791ffe60d9e899a71a037-refs/branch-heads/5845@{#1382})\n","output_type":"stream"}]},{"cell_type":"markdown","source":"## Install Selenium","metadata":{}},{"cell_type":"code","source":"!apt install -y python3-selenium\n!pip install selenium==3.141.0","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:55:02.627679Z","iopub.execute_input":"2024-05-30T06:55:02.628818Z","iopub.status.idle":"2024-05-30T06:55:30.004863Z","shell.execute_reply.started":"2024-05-30T06:55:02.628770Z","shell.execute_reply":"2024-05-30T06:55:30.003785Z"},"trusted":true},"execution_count":7,"outputs":[{"name":"stdout","text":"Reading package lists... Done\nBuilding dependency tree       \nReading state information... Done\nThe following additional packages will be installed:\n  apparmor chromium-browser chromium-chromedriver liblzo2-2 snapd\n  squashfs-tools\nSuggested packages:\n  apparmor-profiles-extra apparmor-utils zenity | kdialog\nThe following NEW packages will be installed:\n  apparmor chromium-browser chromium-chromedriver liblzo2-2 python3-selenium\n  snapd squashfs-tools\n0 upgraded, 7 newly installed, 0 to remove and 140 not upgraded.\nNeed to get 25.9 MB of archives.\nAfter this operation, 107 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apparmor amd64 2.13.3-7ubuntu5.3 [502 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 squashfs-tools amd64 1:4.4-1ubuntu0.3 [117 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 snapd amd64 2.62+20.04 [25.1 MB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 chromium-browser amd64 1:85.0.4183.83-0ubuntu0.20.04.3 [48.5 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 chromium-chromedriver amd64 1:85.0.4183.83-0ubuntu0.20.04.3 [2496 B]\nGet:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-selenium all 4.0.0~a1+dfsg1-1.1 [86.2 kB]\nFetched 25.9 MB in 2s (12.8 MB/s)           \u001b[0m\u001b[33m\nPreconfiguring packages ...\n\n\u001b7\u001b[0;23r\u001b8\u001b[1ASelecting previously unselected package apparmor.\n(Reading database ... 108057 files and directories currently installed.)\nPreparing to unpack .../apparmor_2.13.3-7ubuntu5.3_amd64.deb ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [  0%]\u001b[49m\u001b[39m [..........................................................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [  3%]\u001b[49m\u001b[39m [##........................................................] \u001b8Unpacking apparmor (2.13.3-7ubuntu5.3) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [  7%]\u001b[49m\u001b[39m [####......................................................] \u001b8Selecting previously unselected package liblzo2-2:amd64.\nPreparing to unpack .../liblzo2-2_2.10-2_amd64.deb ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 10%]\u001b[49m\u001b[39m [######....................................................] \u001b8Unpacking liblzo2-2:amd64 (2.10-2) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 14%]\u001b[49m\u001b[39m [########..................................................] \u001b8Selecting previously unselected package squashfs-tools.\nPreparing to unpack .../squashfs-tools_1%3a4.4-1ubuntu0.3_amd64.deb ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 17%]\u001b[49m\u001b[39m [#########.................................................] \u001b8Unpacking squashfs-tools (1:4.4-1ubuntu0.3) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 21%]\u001b[49m\u001b[39m [############..............................................] \u001b8Selecting previously unselected package snapd.\nPreparing to unpack .../snapd_2.62+20.04_amd64.deb ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 24%]\u001b[49m\u001b[39m [##############............................................] \u001b8Unpacking snapd (2.62+20.04) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 28%]\u001b[49m\u001b[39m [################..........................................] \u001b8Setting up apparmor (2.13.3-7ubuntu5.3) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 31%]\u001b[49m\u001b[39m [##################........................................] \u001b8Created symlink /etc/systemd/system/sysinit.target.wants/apparmor.service → /lib/systemd/system/apparmor.service.\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 34%]\u001b[49m\u001b[39m [###################.......................................] \u001b8Setting up liblzo2-2:amd64 (2.10-2) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 38%]\u001b[49m\u001b[39m [######################....................................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 41%]\u001b[49m\u001b[39m [########################..................................] \u001b8Setting up squashfs-tools (1:4.4-1ubuntu0.3) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 45%]\u001b[49m\u001b[39m [##########################................................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 48%]\u001b[49m\u001b[39m [############################..............................] \u001b8Setting up snapd (2.62+20.04) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 52%]\u001b[49m\u001b[39m [#############################.............................] \u001b8Created symlink /etc/systemd/system/multi-user.target.wants/snapd.apparmor.service → /lib/systemd/system/snapd.apparmor.service.\nCreated symlink /etc/systemd/system/multi-user.target.wants/snapd.autoimport.service → /lib/systemd/system/snapd.autoimport.service.\nCreated symlink /etc/systemd/system/multi-user.target.wants/snapd.core-fixup.service → /lib/systemd/system/snapd.core-fixup.service.\nCreated symlink /etc/systemd/system/multi-user.target.wants/snapd.recovery-chooser-trigger.service → /lib/systemd/system/snapd.recovery-chooser-trigger.service.\nCreated symlink /etc/systemd/system/multi-user.target.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.\nCreated symlink /etc/systemd/system/cloud-final.service.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.\nCreated symlink /etc/systemd/system/multi-user.target.wants/snapd.service → /lib/systemd/system/snapd.service.\nCreated symlink /etc/systemd/system/timers.target.wants/snapd.snap-repair.timer → /lib/systemd/system/snapd.snap-repair.timer.\nCreated symlink /etc/systemd/system/sockets.target.wants/snapd.socket → /lib/systemd/system/snapd.socket.\nCreated symlink /etc/systemd/system/final.target.wants/snapd.system-shutdown.service → /lib/systemd/system/snapd.system-shutdown.service.\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 55%]\u001b[49m\u001b[39m [################################..........................] \u001b8Selecting previously unselected package chromium-browser.\n(Reading database ... 108350 files and directories currently installed.)\nPreparing to unpack .../chromium-browser_1%3a85.0.4183.83-0ubuntu0.20.04.3_amd64.deb ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 59%]\u001b[49m\u001b[39m [##################################........................] \u001b8=> Installing the chromium snap\n==> Checking connectivity with the snap store\n===> System doesn't have a working snapd, skipping\nUnpacking chromium-browser (1:85.0.4183.83-0ubuntu0.20.04.3) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 62%]\u001b[49m\u001b[39m [####################################......................] \u001b8Selecting previously unselected package chromium-chromedriver.\nPreparing to unpack .../chromium-chromedriver_1%3a85.0.4183.83-0ubuntu0.20.04.3_amd64.deb ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 66%]\u001b[49m\u001b[39m [######################################....................] \u001b8Unpacking chromium-chromedriver (1:85.0.4183.83-0ubuntu0.20.04.3) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 69%]\u001b[49m\u001b[39m [#######################################...................] \u001b8Selecting previously unselected package python3-selenium.\nPreparing to unpack .../python3-selenium_4.0.0~a1+dfsg1-1.1_all.deb ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 72%]\u001b[49m\u001b[39m [##########################################................] \u001b8Unpacking python3-selenium (4.0.0~a1+dfsg1-1.1) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 76%]\u001b[49m\u001b[39m [############################################..............] \u001b8Setting up python3-selenium (4.0.0~a1+dfsg1-1.1) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 79%]\u001b[49m\u001b[39m [##############################################............] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 83%]\u001b[49m\u001b[39m [################################################..........] \u001b8Setting up chromium-browser (1:85.0.4183.83-0ubuntu0.20.04.3) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 86%]\u001b[49m\u001b[39m [##################################################........] \u001b8update-alternatives: using /usr/bin/chromium-browser to provide /usr/bin/x-www-browser (x-www-browser) in auto mode\nupdate-alternatives: using /usr/bin/chromium-browser to provide /usr/bin/gnome-www-browser (gnome-www-browser) in auto mode\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 90%]\u001b[49m\u001b[39m [####################################################......] \u001b8Setting up chromium-chromedriver (1:85.0.4183.83-0ubuntu0.20.04.3) ...\n\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 93%]\u001b[49m\u001b[39m [#####################################################.....] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 97%]\u001b[49m\u001b[39m [########################################################..] \u001b8Processing triggers for mime-support (3.64ubuntu1) ...\nProcessing triggers for hicolor-icon-theme (0.17-2) ...\nProcessing triggers for libc-bin (2.31-0ubuntu9.9) ...\nProcessing triggers for systemd (245.4-4ubuntu3.22) ...\nProcessing triggers for man-db (2.9.1-1) ...\nProcessing triggers for dbus (1.12.16-2ubuntu2.3) ...\n\n\u001b7\u001b[0;24r\u001b8\u001b[1A\u001b[JCollecting selenium==3.141.0\n  Downloading selenium-3.141.0-py2.py3-none-any.whl (904 kB)\n\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m904.6/904.6 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n\u001b[?25hRequirement already satisfied: urllib3 in /opt/conda/lib/python3.10/site-packages (from selenium==3.141.0) (1.26.15)\nInstalling collected packages: selenium\nSuccessfully installed selenium-3.141.0\n","output_type":"stream"}]},{"cell_type":"markdown","source":"## Install PyGithub","metadata":{}},{"cell_type":"code","source":"!pip install PyGithub","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:55:30.006605Z","iopub.execute_input":"2024-05-30T06:55:30.007024Z","iopub.status.idle":"2024-05-30T06:55:41.245561Z","shell.execute_reply.started":"2024-05-30T06:55:30.006990Z","shell.execute_reply":"2024-05-30T06:55:41.244161Z"},"trusted":true},"execution_count":8,"outputs":[{"name":"stdout","text":"Requirement already satisfied: PyGithub in /opt/conda/lib/python3.10/site-packages (2.3.0)\nRequirement already satisfied: pynacl>=1.4.0 in /opt/conda/lib/python3.10/site-packages (from PyGithub) (1.5.0)\nRequirement already satisfied: requests>=2.14.0 in /opt/conda/lib/python3.10/site-packages (from PyGithub) (2.31.0)\nRequirement already satisfied: pyjwt[crypto]>=2.4.0 in /opt/conda/lib/python3.10/site-packages (from PyGithub) (2.7.0)\nRequirement already satisfied: typing-extensions>=4.0.0 in /opt/conda/lib/python3.10/site-packages (from PyGithub) (4.6.3)\nRequirement already satisfied: urllib3>=1.26.0 in /opt/conda/lib/python3.10/site-packages (from PyGithub) (1.26.15)\nRequirement already satisfied: Deprecated in /opt/conda/lib/python3.10/site-packages (from PyGithub) (1.2.14)\nRequirement already satisfied: cryptography>=3.4.0 in /opt/conda/lib/python3.10/site-packages (from pyjwt[crypto]>=2.4.0->PyGithub) (38.0.4)\nRequirement already satisfied: cffi>=1.4.1 in /opt/conda/lib/python3.10/site-packages (from pynacl>=1.4.0->PyGithub) (1.15.1)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests>=2.14.0->PyGithub) (3.1.0)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests>=2.14.0->PyGithub) (3.4)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests>=2.14.0->PyGithub) (2023.7.22)\nRequirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.10/site-packages (from Deprecated->PyGithub) (1.14.1)\nRequirement already satisfied: pycparser in /opt/conda/lib/python3.10/site-packages (from cffi>=1.4.1->pynacl>=1.4.0->PyGithub) (2.21)\n","output_type":"stream"}]},{"cell_type":"markdown","source":"## Import Libraries","metadata":{}},{"cell_type":"code","source":"from selenium import webdriver\nfrom selenium.webdriver.chrome.options import Options\nfrom selenium.webdriver.common.by import By\nfrom github import Github\nimport datetime\nimport smtplib\nfrom email.message import EmailMessage\nimport pytz\nimport pandas as pd\nimport json\nimport github\nfrom kaggle_secrets import UserSecretsClient","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-05-30T06:55:42.018673Z","iopub.execute_input":"2024-05-30T06:55:42.018980Z","iopub.status.idle":"2024-05-30T06:55:42.027118Z","shell.execute_reply.started":"2024-05-30T06:55:42.018953Z","shell.execute_reply":"2024-05-30T06:55:42.026388Z"},"trusted":true},"execution_count":15,"outputs":[]},{"cell_type":"markdown","source":"### >>","metadata":{}},{"cell_type":"markdown","source":"## **Section 2** - Selenium WebDriver Initialization","metadata":{}},{"cell_type":"markdown","source":"#### Here, we initialize the Selenium WebDriver and define functions related to its configuration.","metadata":{}},{"cell_type":"code","source":"# Add configurable options\ndef add_driver_options(options):\n    chrome_options = Options()\n    for opt in options:\n        chrome_options.add_argument(opt)\n    return chrome_options\n\n# Function to get WebDriver object\ndef get_driver() -> webdriver.Chrome:\n    driver_config = {\n        \"options\": [\n            \"--headless\",\n            \"--no-sandbox\",\n            \"--start-fullscreen\",\n            \"--allow-insecure-localhost\",\n            \"--disable-dev-shm-usage\",\n            \"user-agent=Chrome/116.0.5845.96\"\n        ],\n    }\n    CHROME_BINARY_LOCATION = \"/usr/bin/chrome-linux64/chrome\"\n    CHROMEDRIVER_BINARY_LOCATION = \"/usr/bin/chromedriver-linux64/chromedriver\"\n    options = add_driver_options(driver_config[\"options\"])\n    options.binary_location = CHROME_BINARY_LOCATION\n    driver = webdriver.Chrome(\n        executable_path=CHROMEDRIVER_BINARY_LOCATION,\n        options=options)\n    return driver","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:55:42.056597Z","iopub.execute_input":"2024-05-30T06:55:42.057094Z","iopub.status.idle":"2024-05-30T06:55:42.067567Z","shell.execute_reply.started":"2024-05-30T06:55:42.057063Z","shell.execute_reply":"2024-05-30T06:55:42.066272Z"},"trusted":true},"execution_count":18,"outputs":[]},{"cell_type":"markdown","source":"### >>","metadata":{}},{"cell_type":"markdown","source":"## **Section 3** - Web Scraping","metadata":{}},{"cell_type":"markdown","source":"#### This section contains functions for scraping job details from job site and indian cities list from news articles","metadata":{}},{"cell_type":"code","source":"# Function to scrap job details and return data as a DataFrame\ndef get_source_data(job_role_lst: list, cnt: int) -> pd.DataFrame:\n    fnl_lst = []\n    key = 1\n\n    # Iterating through job roles\n    for job_role in job_role_lst:\n        lst = job_role.split(' ')\n        str_query = \"+\".join(lst)\n        page = f'https://www.foundit.in/srp/results?query=\"{str_query}\"'\n        url = page.encode('ascii', 'ignore').decode('unicode_escape')\n\n        driver = get_driver()\n        driver.get(url)\n        count = 0\n\n        # Scraping job details\n        while count <= cnt:\n            x_pth = \"/html\\\n                    /body\\\n                    /div[@id='srpThemeDefault']\\\n                    /div[@class='srpContainer']\\\n                    /div[@id='srpContent']\\\n                    /div[@class='srpCardContainer']\\\n                    /div[@class='srpResultCard']\\\n                    /div\"\n            elements = (driver.find_elements(By.XPATH, x_pth))\n\n            for element in elements:\n                try:\n                    job_dict = {}\n                    job_title = element.find_element(By.CLASS_NAME, \"jobTitle\").text\n                    company_name = element.find_element(By.CLASS_NAME, \"companyName\").text\n                    skills_str = ''\n\n                    # Extracting skills\n                    for i in element.find_elements(By.CLASS_NAME, \"skillTitle\"):\n                        skill = i.text\n                        if skill != '':\n                            skills_str += skill + ','\n\n                    sub_element = element.find_element(By.CLASS_NAME, \"cardBody\")\n                    job_type_str = sub_element.find_element(By.XPATH, \"div[1]/div[@class='details']\").text\n                    location_str = sub_element.find_element(By.XPATH, \"div[2]/div[@class='details']\").text\n                    experience_str = sub_element.find_element(By.XPATH, \"div[3]/div[@class='details']\").text\n\n                    # Storing job details in dictionary\n                    job_dict['key'] = key\n                    job_dict['job_role'] = job_role\n                    job_dict['job_title'] = job_title\n                    job_dict['company_name'] = company_name\n                    job_dict['skills'] = skills_str[:-1]\n                    job_dict['job_type'] = job_type_str\n                    job_dict['location'] = location_str\n                    job_dict['experience'] = experience_str\n                    fnl_lst.append(job_dict)\n                    count += 1\n                    key += 1\n\n                    if count == cnt:\n                        break\n                except:\n                    pass\n\n            try:\n                element.find_element(By.CLASS_NAME, \"mqfisrp-right-arrow\").click()\n            except:\n                break\n\n        driver.quit()\n\n    # Converting list of dictionaries to DataFrame\n    df = pd.DataFrame(fnl_lst)\n    return df","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:55:42.086156Z","iopub.execute_input":"2024-05-30T06:55:42.087083Z","iopub.status.idle":"2024-05-30T06:55:42.099664Z","shell.execute_reply.started":"2024-05-30T06:55:42.087049Z","shell.execute_reply":"2024-05-30T06:55:42.098661Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":20,"outputs":[]},{"cell_type":"code","source":"# Function to get cities in India as a DataFrame through web scraping\ndef get_cities_indian() -> pd.DataFrame:\n    cities = []\n    page = 'https://www.britannica.com/topic/list-of-cities-and-towns-in-India-2033033'\n    url = page.encode('ascii', 'ignore').decode('unicode_escape')\n    driver = get_driver()\n    driver.get(url)\n    div_element = driver.find_element(By.CLASS_NAME, 'reading-channel')\n    lists = div_element.find_elements(By.TAG_NAME, 'li')\n\n    # Extracting city names\n    for list in lists:\n        txt = list.find_element(By.TAG_NAME, 'a').text.lower()\n        cities.append(txt)\n\n    df = pd.DataFrame(cities, columns=[\"city\"])\n    return df","metadata":{"execution":{"iopub.status.busy":"2024-05-30T06:55:46.254721Z","iopub.execute_input":"2024-05-30T06:55:46.255122Z","iopub.status.idle":"2024-05-30T06:55:46.262434Z","shell.execute_reply.started":"2024-05-30T06:55:46.255089Z","shell.execute_reply":"2024-05-30T06:55:46.261218Z"},"trusted":true},"execution_count":22,"outputs":[]},{"cell_type":"markdown","source":"### >>","metadata":{}},{"cell_type":"markdown","source":"## **Section 4** - GitHub Interaction","metadata":{}},{"cell_type":"markdown","source":"#### Functions related to interacting with GitHub, such as getting the repository, uploading data, and extracting data, are defined here.","metadata":{}},{"cell_type":"code","source":"# Function to get GitHub repository object\ndef get_repository() -> github.Repository.Repository:\n    access_token = UserSecretsClient().get_secret('access_token')\n    repo_str = UserSecretsClient().get_secret('repo_str')\n    g = Github(access_token)\n    repo = g.get_repo(repo_str)\n    return repo","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:50.619233Z","iopub.execute_input":"2024-03-22T06:16:50.619791Z","iopub.status.idle":"2024-03-22T06:16:50.626256Z","shell.execute_reply.started":"2024-03-22T06:16:50.619748Z","shell.execute_reply":"2024-03-22T06:16:50.625175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to upload DataFrame as CSV file to GitHub\ndef upload_dataframe_to_github(df: pd.DataFrame, folder: str):\n    branch = 'master'\n    repo = get_repository()\n    csv_content = df.to_csv(index=False)\n\n    # Uploading CSV file to GitHub\n    if folder in ['source', 'consumption']:\n        IST = pytz.timezone('Asia/Kolkata')\n        dt_tm = str(datetime.datetime.now(IST))[:19].replace(' ', '-')\n        file_str = f'{dt_tm}.csv'\n        repo.create_file(folder+'/'+file_str, 'upload '+folder+' level data', csv_content, branch=branch)\n        print(folder+' layer file uploaded!')\n    elif folder == 'reference':\n        file_str = 'indian_cities.csv'\n        try:\n            repo.create_file(folder+'/'+file_str, 'create indian cities data', csv_content, branch=branch)\n            print(folder+' layer file uploaded!')\n        except:\n            file = repo.get_contents(folder+'/'+file_str)\n            repo.update_file(folder+'/'+file_str,'update indian cities data',csv_content,file.sha)\n            print(folder+' layer file uploaded!')\n    else:\n        raise Exception(\"folder is either 'source' or 'consumption' or 'reference'!\")","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:51.040261Z","iopub.execute_input":"2024-03-22T06:16:51.041544Z","iopub.status.idle":"2024-03-22T06:16:51.053676Z","shell.execute_reply.started":"2024-03-22T06:16:51.041497Z","shell.execute_reply":"2024-03-22T06:16:51.052343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to scrap Indian cities and update reference file in GitHub repo\ndef update_cities_data():\n    # Get list of indian cities as dataframe\n    df = get_cities_indian()\n    # Upload dataframe as csv file in GitHub repo\n    upload_dataframe_to_github(df=df, folder='reference')","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:51.464029Z","iopub.execute_input":"2024-03-22T06:16:51.465224Z","iopub.status.idle":"2024-03-22T06:16:51.471498Z","shell.execute_reply.started":"2024-03-22T06:16:51.465183Z","shell.execute_reply":"2024-03-22T06:16:51.469318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Iterate through the files and get the most recent one\ndef get_new_file(folder:str) -> str:\n    if folder not in ['source', 'consumption', 'target', 'reference']:\n        raise Exception(\"folder is either 'source' or 'consumption' or 'target' or 'reference'!\")\n\n    repo = get_repository()\n    contents = repo.get_contents(folder)\n    path = ''\n\n    # Finding the most recent file\n    for c in contents:\n        if path < c.path:\n            path = c.path\n\n    return path","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:51.895573Z","iopub.execute_input":"2024-03-22T06:16:51.896866Z","iopub.status.idle":"2024-03-22T06:16:51.904793Z","shell.execute_reply.started":"2024-03-22T06:16:51.896820Z","shell.execute_reply":"2024-03-22T06:16:51.903012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to get file path\ndef get_file_path(folder:str, path_type:str) -> str:\n    github_url = 'https://github.com'\n    repo_str = UserSecretsClient().get_secret('repo_str')\n    branch = 'master'\n    file_path = get_new_file(folder=folder)\n    file_url = github_url + '/' + repo_str + '/blob/' + branch + '/' + file_path\n\n    # Returning URL based on path type\n    if path_type == 'url':\n        return file_url\n    elif path_type == 'raw':\n        return file_url.replace('github', 'raw.githubusercontent').replace('blob/', '')\n    else:\n        raise Exception(\"path_type is either 'url' or 'raw'!\")","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:52.378339Z","iopub.execute_input":"2024-03-22T06:16:52.378845Z","iopub.status.idle":"2024-03-22T06:16:52.387884Z","shell.execute_reply.started":"2024-03-22T06:16:52.378798Z","shell.execute_reply":"2024-03-22T06:16:52.386744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to convert reference cities CSV file to list\ndef cities_lst() -> list:\n    ref_raw_path = get_file_path(folder='reference', path_type='raw')\n    df = pd.read_csv(ref_raw_path)\n    lst = df.city.tolist()\n    return lst","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:52.853014Z","iopub.execute_input":"2024-03-22T06:16:52.853725Z","iopub.status.idle":"2024-03-22T06:16:52.858946Z","shell.execute_reply.started":"2024-03-22T06:16:52.853688Z","shell.execute_reply":"2024-03-22T06:16:52.858103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to upload dictionary as JSON file to GitHub\ndef upload_dict_to_github(data_dict: dict, folder: str):\n    if folder == 'target':\n        branch = 'master'\n        repo = get_repository()\n        IST = pytz.timezone('Asia/Kolkata')\n        dt_tm = str(datetime.datetime.now(IST))[:19].replace(' ', '-')\n        file_str = f'{dt_tm}.json'\n        content=json.dumps(data_dict, indent=4)\n        repo.create_file(folder+'/'+file_str, 'upload '+folder+' level data', content, branch=branch)\n        print(folder+' layer file uploaded!')\n    else:\n        raise Exception(\"folder should be 'target'!\")","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:53.373183Z","iopub.execute_input":"2024-03-22T06:16:53.373677Z","iopub.status.idle":"2024-03-22T06:16:53.382365Z","shell.execute_reply.started":"2024-03-22T06:16:53.373639Z","shell.execute_reply":"2024-03-22T06:16:53.380847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### >>","metadata":{}},{"cell_type":"markdown","source":"## **Section 5** - Email Notification","metadata":{}},{"cell_type":"markdown","source":"#### Function for sending email notifications about the status of the process are defined in this section.","metadata":{}},{"cell_type":"code","source":"# Function to send status via email\ndef sendMail(reciever_id: str, exception=None):\n    # Configuration\n    sender_id = UserSecretsClient().get_secret('sender_id')\n    pass_word = UserSecretsClient().get_secret('pass_word')\n    git_link = UserSecretsClient().get_secret('git_link')\n    IST = pytz.timezone('Asia/Kolkata')\n    time_now = datetime.datetime.now(IST)\n\n    # Creating email message\n    message = \"STATUS:\\n\"\n    mail = EmailMessage()\n    mail['From'] = sender_id\n    mail['To'] = reciever_id\n\n    # Handling status based on exception\n    if exception is None:\n        mail['Subject'] = \"Extraction done \" + str(time_now.strftime(\"at %H:%M:%S, on %d/%m/%Y\"))\n        message += \"Data extraction, cleanup, and loading done successfully \" + str(time_now.strftime(\"at %H:%M:%S, on %d %B %Y (%A)\")) + \".\\n\"\n        message += \"Files uploaded at \" + git_link + \".\\n\"\n    else:\n        mail['Subject'] = \"Exception occured \" + str(time_now.strftime(\"at %H:%M:%S, on %d/%m/%Y\"))\n        message += \"During the process of data extraction an exception occured.\\n\"\n        message += \"Exception: \" + str(exception) + \".\\n\"\n        message += \"Visit https://www.kaggle.com/ to resolve the issue.\\n\"\n\n    message += \"\\nSent from kaggle notebook.\"\n    mail.set_content(message)\n\n    # Sending email\n    server = smtplib.SMTP(\"smtp.gmail.com\", 587)\n    server.starttls()\n    server.login(sender_id, pass_word)\n    server.send_message(mail)\n    server.close()","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:55.303983Z","iopub.execute_input":"2024-03-22T06:16:55.304919Z","iopub.status.idle":"2024-03-22T06:16:55.319805Z","shell.execute_reply.started":"2024-03-22T06:16:55.304865Z","shell.execute_reply":"2024-03-22T06:16:55.318345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### >>","metadata":{}},{"cell_type":"markdown","source":"## **Section 6** - Data Cleanup, Processing, and Transformation Functions","metadata":{}},{"cell_type":"markdown","source":"#### This section contains functions responsible for cleaning up raw data, processing it, and transforming it into a usable format.","metadata":{}},{"cell_type":"code","source":"# Function to extract cities from location string passed as parameter\ndef parse_location(location_str: str) -> str:\n    location = [i.strip().lower() for i in location_str.replace('/', ',').split(\",\")]\n    loc_lst = []\n\n    for loc in location:\n        if loc in cities:\n            loc_lst.append(loc)\n\n    lst = list(set(loc_lst)) # Remove duplicates\n    result_string = ','.join(lst)\n    return result_string","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:57.309273Z","iopub.execute_input":"2024-03-22T06:16:57.310087Z","iopub.status.idle":"2024-03-22T06:16:57.317577Z","shell.execute_reply.started":"2024-03-22T06:16:57.310045Z","shell.execute_reply":"2024-03-22T06:16:57.316478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to parse experience string passed and return required experience integer value\ndef parse_experience(experience_str: str) -> int:\n    experience_str = experience_str.strip().lower()\n\n    if '-' in experience_str:\n        lst = [int(i) for i in experience_str.split(' ')[0].split('-')]\n        res = sum(lst) // len(lst)\n    else:\n        if 'fresher' in experience_str:\n            res = 0\n        else:\n            try:\n                res = int(experience_str.split(' ')[0])\n            except:\n                res = -1\n\n    return res","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:57.774813Z","iopub.execute_input":"2024-03-22T06:16:57.775960Z","iopub.status.idle":"2024-03-22T06:16:57.784291Z","shell.execute_reply.started":"2024-03-22T06:16:57.775917Z","shell.execute_reply":"2024-03-22T06:16:57.782898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to process skills and return skills list\ndef process_skills(skill_str: str) -> str:\n    skill_lst = skill_str.lower().split(',')\n    job_roles = [\n        \"data engineer\",\n        \"data analyst\",\n        \"data architect\",\n        \"data scientist\",\n        \"machine learning engineer\"\n    ]\n\n    for i in job_roles:\n        for j in skill_lst:\n            if i in j:\n                skill_lst.remove(j)\n\n    result_string = ','.join(skill_lst)\n    return result_string","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:58.242944Z","iopub.execute_input":"2024-03-22T06:16:58.243408Z","iopub.status.idle":"2024-03-22T06:16:58.251920Z","shell.execute_reply.started":"2024-03-22T06:16:58.243373Z","shell.execute_reply":"2024-03-22T06:16:58.250149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to parse job type string passed and return a list of expected values\ndef parse_job_type(job_type_str: str) -> str:\n    job_type_str = job_type_str.lower()\n    type_lst = []\n\n    if 'full' in job_type_str:\n        type_lst.append('full time')\n    if 'home' in job_type_str:\n        type_lst.append('work from home')\n    if 'contract' in job_type_str:\n        type_lst.append('contract job')\n    if 'remote' in job_type_str:\n        type_lst.append('remote job')\n    if 'part' in job_type_str:\n        type_lst.append('part time')\n\n    result_string = ','.join(type_lst)\n    return result_string","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:58.746619Z","iopub.execute_input":"2024-03-22T06:16:58.747766Z","iopub.status.idle":"2024-03-22T06:16:58.756197Z","shell.execute_reply.started":"2024-03-22T06:16:58.747700Z","shell.execute_reply":"2024-03-22T06:16:58.754724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function that replaces NaN with values based on column and value pair passed\ndef replace_nan(df: pd.DataFrame, nan_values: dict) -> pd.DataFrame:\n\n    for col, val in nan_values.items():\n        df[col] = df[col].fillna(val)\n\n    return df","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:16:59.390487Z","iopub.execute_input":"2024-03-22T06:16:59.390948Z","iopub.status.idle":"2024-03-22T06:16:59.398858Z","shell.execute_reply.started":"2024-03-22T06:16:59.390908Z","shell.execute_reply":"2024-03-22T06:16:59.397151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to clean data\ndef clean_data(df: pd.DataFrame) -> pd.DataFrame:\n    nan_values = {\n        'job_role' : '',\n        'job_title' : '',\n        'company_name' : '',\n        'experience' : '',\n        'location' : '',\n        'skills' : '',\n        'job_type' : ''\n    }\n    df = replace_nan(df, nan_values=nan_values)\n    df['job_role'] = df['job_role'].str.lower()\n    df['job_title'] = df['job_title'].str.lower()\n    df['company_name'] = df['company_name'].str.lower()\n    df['experience'] = df.apply(lambda x : parse_experience(x['experience']), axis=1)\n    df['location'] = df.apply(lambda x : parse_location(x['location']), axis=1)\n    df['skills'] = df.apply(lambda x : process_skills(x['skills']), axis=1)\n    df['job_type'] = df.apply(lambda x : parse_job_type(x['job_type']), axis=1)\n    return df","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:17:00.303187Z","iopub.execute_input":"2024-03-22T06:17:00.303579Z","iopub.status.idle":"2024-03-22T06:17:00.313466Z","shell.execute_reply.started":"2024-03-22T06:17:00.303550Z","shell.execute_reply":"2024-03-22T06:17:00.311840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to sort dictionary based on the values in descending order and return first n key-value pairs\ndef sort_dict_by_value(dictionary: dict, descending=True, n=None) -> dict:\n    sorted_dict = dict(sorted(dictionary.items(), key=lambda item: item[1], reverse=descending))\n    if n is not None:\n        sorted_dict = dict(list(sorted_dict.items())[:n])\n    return sorted_dict","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:17:00.961009Z","iopub.execute_input":"2024-03-22T06:17:00.961711Z","iopub.status.idle":"2024-03-22T06:17:00.969272Z","shell.execute_reply.started":"2024-03-22T06:17:00.961675Z","shell.execute_reply":"2024-03-22T06:17:00.967806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function that aggregates consumption dataset and returns a dictionary\ndef roll_up_data(df_con: pd.DataFrame) -> dict:\n    job_roles = [\n        \"data engineer\",\n        \"data analyst\",\n        \"data architect\",\n        \"data scientist\",\n        \"machine learning engineer\"\n    ]\n    nan_values = {\n        'job_role' : '',\n        'job_title' : '',\n        'company_name' : '',\n        'experience' : 0,\n        'location' : '',\n        'skills' : '',\n        'job_type' : ''\n    }\n    \n    df_con = replace_nan(df_con, nan_values=nan_values)\n    final_dict = {}\n\n    for job_role in job_roles:\n        final_dict[job_role] = {\n            'company_dict' : {},\n            'skill_dict' : {},\n            'location_dict' : {},\n            'job_type_dict' : {},\n            'experience_dict' : {\n                'entry level' : 0,\n                'mid level': 0,\n                'senior level': 0\n            }\n        }\n\n    for index, row in df_con.iterrows():\n        job_role = None\n\n        for column, value in row.items():\n            if column == 'job_role':\n                job_role = value\n\n            if column == 'company_name' and value != '':\n                try:\n                    final_dict.get(job_role).get('company_dict')[value] += 1\n                except:\n                    final_dict.get(job_role).get('company_dict')[value] = 1\n            elif column == 'skills':\n                for s in value.split(','):\n                    if s != '':\n                        try:\n                            final_dict.get(job_role).get('skill_dict')[s] += 1\n                        except:\n                            final_dict.get(job_role).get('skill_dict')[s] = 1\n            elif column == 'location':\n                for l in value.split(','):\n                    if l != '':\n                        try:\n                            final_dict.get(job_role).get('location_dict')[l] += 1\n                        except:\n                            final_dict.get(job_role).get('location_dict')[l] = 1\n            elif column == 'job_type':\n                for j in value.split(','):\n                    if j != '':\n                        try:\n                            final_dict.get(job_role).get('job_type_dict')[j] += 1\n                        except:\n                            final_dict.get(job_role).get('job_type_dict')[j] = 1\n            elif column == 'experience':\n                if value < 3:\n                    final_dict.get(job_role).get('experience_dict')['entry level'] += 1\n                elif value >=3 and value <= 5:\n                    final_dict.get(job_role).get('experience_dict')['mid level'] += 1\n                else:\n                    final_dict.get(job_role).get('experience_dict')['senior level'] += 1\n\n    for k1, v1 in final_dict.items():\n        for k2 in v1.keys():\n            if k2 in ['company_dict', 'skill_dict', 'location_dict']:\n                final_dict[k1][k2] = sort_dict_by_value(final_dict[k1][k2], descending=True, n=10)\n\n    return final_dict","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:17:02.162241Z","iopub.execute_input":"2024-03-22T06:17:02.162763Z","iopub.status.idle":"2024-03-22T06:17:02.185195Z","shell.execute_reply.started":"2024-03-22T06:17:02.162724Z","shell.execute_reply":"2024-03-22T06:17:02.183600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function that checks if the dictionary passed is having enough data for target folder\ndef check_json(dictionary:dict) -> bool:\n  for key, value in dictionary.items():\n    for k, v in value.items():\n      if k in ['company_dict', 'skill_dict', 'location_dict'] and len(v) < 10:\n        return False\n      elif k == 'job_type_dict' and len(v) == 0:\n        print(k)\n        return False\n      elif k == 'experience_dict' and v == {'entry level': 0, 'mid level': 0, 'senior level': 0}:\n        print(k)\n        return False\n  return True","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:17:03.409513Z","iopub.execute_input":"2024-03-22T06:17:03.410150Z","iopub.status.idle":"2024-03-22T06:17:03.418753Z","shell.execute_reply.started":"2024-03-22T06:17:03.410105Z","shell.execute_reply":"2024-03-22T06:17:03.417427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### >>","metadata":{}},{"cell_type":"markdown","source":"## **Section 7** - Orchestration","metadata":{}},{"cell_type":"markdown","source":"#### This section contains functions responsible for orchestrating the entire process.","metadata":{}},{"cell_type":"code","source":"# Scraping and loading source data\ndef scrap_and_load():\n    # List of job roles\n    job_roles = [\n        \"Data Engineer\",\n        \"Data Analyst\",\n        \"Data Architect\",\n        \"Data Scientist\",\n        \"Machine Learning Engineer\"\n    ]\n    # Get source data (job postings)\n    df = get_source_data(job_role_lst=job_roles, cnt=500)\n    print('web scraping done!')\n    # Upload data to GitHub\n    upload_dataframe_to_github(df=df, folder='source')","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:17:05.961149Z","iopub.execute_input":"2024-03-22T06:17:05.962440Z","iopub.status.idle":"2024-03-22T06:17:05.968843Z","shell.execute_reply.started":"2024-03-22T06:17:05.962395Z","shell.execute_reply":"2024-03-22T06:17:05.967805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function that cleans source dataset and uploads consumption file\ndef upload_consumption_file():\n    # Read recent source file into DataFrame\n    raw_path = get_file_path(folder='source', path_type='raw')\n    df_raw = pd.read_csv(raw_path)\n    # Clean DataFrame\n    df_con = clean_data(df_raw)\n    # Upload file\n    upload_dataframe_to_github(df_con, folder='consumption')","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:17:06.851565Z","iopub.execute_input":"2024-03-22T06:17:06.852885Z","iopub.status.idle":"2024-03-22T06:17:06.858972Z","shell.execute_reply.started":"2024-03-22T06:17:06.852817Z","shell.execute_reply":"2024-03-22T06:17:06.857766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function that aggregates consumption dataset and uploads target file\ndef upload_target_file():\n    # Read recent consumption file into DataFrame\n    con_path = get_file_path(folder='consumption', path_type='raw')\n    df_con = pd.read_csv(con_path)\n    # Aggregate data\n    fnl_dict = roll_up_data(df_con)\n    if check_json(fnl_dict):\n        # Upload file\n        upload_dict_to_github(fnl_dict, folder='target')\n    else:\n        raise Exception(\"No enough data for target folder\")","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:17:08.663384Z","iopub.execute_input":"2024-03-22T06:17:08.664134Z","iopub.status.idle":"2024-03-22T06:17:08.671158Z","shell.execute_reply.started":"2024-03-22T06:17:08.664092Z","shell.execute_reply":"2024-03-22T06:17:08.669841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### >>","metadata":{}},{"cell_type":"markdown","source":"## **Section 8** - Execution","metadata":{}},{"cell_type":"markdown","source":"#### This section calls the main function to execute the entire process.","metadata":{}},{"cell_type":"code","source":"# Entry point of the script\nif __name__ == '__main__':\n    # My email id\n    reciever_id = UserSecretsClient().get_secret('reciever_id')\n    # Try-catch block to handle exceptions and send mail\n    try:\n        # Upload cities data in GitHub\n        update_cities_data()\n        # Indian cities list\n        cities = cities_lst()\n        # Scrap and load source data\n        scrap_and_load()\n        # Upload consumption file\n        upload_consumption_file()\n        # Upload target file\n        upload_target_file()   \n        # Send mail if process succeeded\n        sendMail(reciever_id)\n    except Exception as e:\n        print('exception occured!')\n        print(e)\n        # Send mail if exception occurred\n        sendMail(reciever_id, exception=e)","metadata":{"execution":{"iopub.status.busy":"2024-03-22T06:18:13.529064Z","iopub.execute_input":"2024-03-22T06:18:13.529526Z","iopub.status.idle":"2024-03-22T06:18:59.836883Z","shell.execute_reply.started":"2024-03-22T06:18:13.529492Z","shell.execute_reply":"2024-03-22T06:18:59.835467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### -------------------------","metadata":{}}]}