day29-31_wrap_up.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- [1] https://towardsdatascience.com/beyond-pandas-spark-dask-vaex-and-other-big-data-technologies-battling-head-to-head-a453a1f8cc13\n",
    "- [2] https://github.com/DataSystemsGroupUT/dataeng"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Week 1 - Beyond Pandas\n",
    "\n",
    "- I've got the inspiration about trying pandas alternative for data processing from this article [[1]](https://towardsdatascience.com/beyond-pandas-spark-dask-vaex-and-other-big-data-technologies-battling-head-to-head-a453a1f8cc13)\n",
    "- In terms of implementation, `koalas` became my personal favourite over the other libraries, as its syntax work in very similar way like `pandas`, so there is nothing much changes need in the code when we used `pandas` before.\n",
    "- My second favourite is `vaex`, as it has active github repositories and very good documentation\n",
    "- `Dask` is long-time state-of-the-art for multi-processing library in Python, but lack of developer support and missing a lot of important pandas features that made me hard to migrate from pandas \n",
    "- Overall, through this first week, I learned more about how multi-processing in Python can work better with open-source libraries, as an alternative to built-in `multiprocessing` library in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Week 2 - Data Streaming\n",
    "\n",
    "- This topic inspired by [Data Engineering](https://github.com/DataSystemsGroupUT/dataeng) course from the University of Tartu, Estonia (Autumn 2020)\n",
    "- Even I didn't dig up thoroughly for every new frameworks that I learned, this challenge helped me to getting started about the advantage and how it works from top layer\n",
    "- I learned Kafka and Airflow mostly from Data Engineering course, but still left me many task that I need to explore again in my free time\n",
    "- I learned Spark through my internship time in [Positium](positium.com) to improve their big data framework performance.\n",
    "- Flink and Storm are two new things that I started to learn through this 2nd week.\n",
    "- Meanwhile, Docker also become the most important part to setup many different tools in different OS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Week 3 - Data Visualization\n",
    "\n",
    "- DataViz came from my inner voice to learn something that not too much technical\n",
    "- It turned out that I really enjoying this 3rd week, as I getting to know to many Python and R dataviz libraries, in addition to state-of-the-art commercial dataviz such as Tableau\n",
    "- Apache Superset are the new dataviz library that works really different compare to others, it works through web server with enterprise-ready web app.\n",
    "- I've got Tableau student license which helped me to spend time learning through the official tutorial that later I've found out, it needs more time to explore as it's a stand-alone enterprise software, not only single library\n",
    "- Plotly become my personal favourite for open-source interactive dataviz, as it able to be expanded into dashboard throught its Dash dashboard library from the same developer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Week 4 - Dashboard\n",
    "\n",
    "- There are total 4 dashboard libraries that I learned through this 4th week: dash, django, bokeh, and R shiny.\n",
    "- Surprisingly I like how Django works as web app framework through Python, it's an anti-mainstream when you want try something new, beside using only Javascript.\n",
    "- I'm getting know to Shiny dashboard when I took Data Science 3-months course in Algoritma Data Science, Jakarta, Indonesia. On that time, Jan - Mar 2019, I focusedly learning Data Science using R, before moving on to Python.\n",
    "- Bokeh also interesting library, but I found out that Plotly Dash has more features compare to Bokeh, with very similar dataviz quality\n",
    "- Out there, probably open-source frameworks like Grafana and Kibana also worthful to learn in future"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's Next?\n",
    "\n",
    "- Currently,'m really interested to learn more about DevOps technology, especially in orchestration tool such as Kubernetes, Apache Zeppelin, or Apache Airflow\n",
    "- Due to my current study priorities to finish my Master's thesis in the field on NLP (Natural Language Processing) and Requirement Engineering, I decided to make next personal challenge in Feb 2021 as `#30DaysOfNLP`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}