VDA_Banking_L2.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"VDA_Banking_L2.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.5"}},"cells":[{"cell_type":"markdown","metadata":{"id":"vdaqf6vgTTg4"},"source":["<center>\n","    <img src=\"https://gitlab.com/ibm/skills-network/courses/placeholder101/-/raw/master/labs/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n","</center>\n"]},{"cell_type":"markdown","metadata":{"id":"y2VgPk2mTThF"},"source":["# **Visual data analysis in banking**\n"]},{"cell_type":"markdown","metadata":{"id":"jkRBtAnQTThK"},"source":["The purpose of this lab is to master visual data analysis in banking for machine learning models.\n","\n","After completing this lab you will be able to:\n","\n","1.  Visualize a banking dataset with MatplotLib, Seaborn and Plotly libraries.\n","2.  Visually analyze single features and feature interaction.\n","3.  Do a comprehensive visual data analysis for the source dataset.\n"]},{"cell_type":"markdown","metadata":{"id":"CHwOXFUpTThK"},"source":["## Outline\n"]},{"cell_type":"markdown","metadata":{"id":"bHSc4iksTThP"},"source":["*   Materials and Methods\n","*   General Part\n","    *   Import Libraries\n","    *   Load the Dataset\n","    *   Overview of Python libraries for visual data analysis\n","    *   Visual analysis of single features\n","    *   Visual analysis of feature interaction\n","    *   Comprehensive visual analysis of the source banking dataset\n","*   Tasks\n","*   Authors\n"]},{"cell_type":"markdown","metadata":{"id":"S4t5rpwITThQ"},"source":["***\n"]},{"cell_type":"markdown","metadata":{"id":"KoNAvKmglBm-"},"source":["## Materials and Methods\n"]},{"cell_type":"markdown","metadata":{"id":"ZOgyr5RSlI40"},"source":["The data that we are going to use for this is a subset of an open source Bank Marketing Data Set from the UCI ML repository: [https://archive.ics.uci.edu/ml/citation_policy.html](https://archive.ics.uci.edu/ml/citation_policy.html?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01).\n","\n","> This dataset is public available for research. The details are described in \\[Moro et al., 2014].\n","\n","Please include this citation if you plan to use this database:\n","\\[Moro et al., 2014] S. Moro, P. Cortez and P. Rita. A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014\n","\n","During the work, the task of a preliminary analysis of a positive response (term deposit) to direct calls from the bank is solved. In essence, the task is the matter of bank scoring, i.e. according to the characteristics of clients (potential clients), their behavior is predicted (loan default, a wish to open a deposit, etc.).\n","\n","In this lesson, we will try to give answers to a set of questions that may be relevant when analyzing banking data:\n","\n","1.  What are the most useful Python libraries for visual analysis?\n","2.  How to build interactive plots?\n","3.  How to visualize single features?\n","4.  How to do a visual analysis for the feature interaction?\n","5.  How to provide a comprehensive visual analysis for numerical and categorical features?\n","\n","In addition, we will make the conclusions for the obtained results of our visual analysis to plan marketing banking campaigns more effectively.\n"]},{"cell_type":"markdown","metadata":{"id":"t6D_qFGMqUaN"},"source":["[Matplotlib](https://matplotlib.org/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01) is a plotting library for the Python programming language and its numerical mathematics extension NumPy. Matplotlib uses an object oriented API to embed plots in Python applications.\n"]},{"cell_type":"markdown","metadata":{"id":"cL_X87MhfghK"},"source":["[Seaborn](https://seaborn.pydata.org/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01) is a Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics. Seaborn provides an API on top of Matplotlib that offers sane choices for plot style and color defaults, defines simple high-level functions for common statistical plot types, and integrates with the functionality provided by Pandas DataFrames.\n"]},{"cell_type":"markdown","metadata":{"id":"pqtJePMwg7gX"},"source":["[Plotly](https://plotly.com/python/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01) is an interactive, open-source plotting library that supports over 40 unique chart types covering a wide range of statistical, financial, geographic, scientific, and 3-dimensional use-cases. Built on top of the Plotly JavaScript library (plotly.js), plotly enables Python users to create beautiful interactive web-based visualizations that can be displayed in Jupyter notebooks and saved to standalone HTML files.\n"]},{"cell_type":"markdown","metadata":{"id":"xS8UOiQOTThR"},"source":["## Import Libraries\n"]},{"cell_type":"markdown","metadata":{"id":"H6YFQWnQHKui"},"source":["Download data using a URL.\n"]},{"cell_type":"code","metadata":{"id":"bN5NPujBTThV"},"source":["!wget https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-BnB2tiSZh4R"},"source":["Alternative URL for the dataset downloading.\n"]},{"cell_type":"code","metadata":{"id":"ORfD1EUiZuLK"},"source":["!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/VDA_Banking_L2/bank-additional.zip"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"AJAUUEruHYTj"},"source":["Unzipping to a folder. It is a good idea to apply the `-o` and `-q`  when unzipping to quiet the process and overwrite any existing folders.\n"]},{"cell_type":"code","metadata":{"id":"iZKbsroFTThY"},"source":["!unzip -o -q bank-additional.zip"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"eGH5DhdLHmOp"},"source":["Import the libraries necessary to use in this lab. We can add some aliases to make the libraries easier to use in our code and set a default figure size for further plots. Ignore the warnings.\n"]},{"cell_type":"code","metadata":{"id":"jdVcijxmTTha"},"source":["import pandas as pd\n","import matplotlib.pyplot as plt\n","import numpy as np\n","import seaborn as sns\n","%matplotlib inline\n","plt.rcParams[\"figure.figsize\"] = (8, 6)\n","\n","import warnings\n","warnings.filterwarnings('ignore')\n","\n","import plotly\n","import plotly.graph_objs as go\n","from plotly.offline import download_plotlyjs, init_notebook_mode, iplot, plot\n","init_notebook_mode(connected=True)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"evtbog-C13Z2"},"source":["Further specify the value of the `precision` parameter equal to 2 to display two decimal signs (instead of 6 as default).\n"]},{"cell_type":"code","metadata":{"id":"nwMoaJLV13Z3"},"source":["pd.set_option(\"precision\", 2)\n","pd.options.display.float_format = '{:.2f}'.format"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"SbmrZguMTThd"},"source":["## Load the Dataset\n"]},{"cell_type":"markdown","metadata":{"id":"EvMGO3m4TThd"},"source":["In this section you will load the source dataset.\n"]},{"cell_type":"code","metadata":{"id":"wamAf3HLm_FS"},"source":["df = pd.read_csv('bank-additional/bank-additional-full.csv', sep = ';')\n","df.head(5)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"aBKhvaq_paf5"},"source":["The target feature shows a positive behavior of a phone call during the marketing campaign. Mark the positive outcome as 1 and negative one as 0.\n"]},{"cell_type":"markdown","metadata":{"id":"r8srUBznuaRN"},"source":["Use the `map` function for **replacement the values ​​in column** by passing it as an argument dictionary in form of ` {old_value: new_value}  `. Replace ##YOUR CODE GOES HERE## with your Python code.\n"]},{"cell_type":"code","metadata":{"id":"eEJeG8GfuaRk"},"source":["d = {\"no\": 0, \"yes\": 1}\n","##YOUR CODE GOES HERE## "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"mFDBIaFrg--8"},"source":["Double-click **here** for the solution.\n","\n","<!-- \n","df[\"y\"] = df[\"y\"].map(d)\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"KVOmrfNH3pRs"},"source":["### Attribute Information\n"]},{"cell_type":"markdown","metadata":{"id":"TSTqzHlbKWoa"},"source":["Output the column (feature) names:\n"]},{"cell_type":"code","metadata":{"id":"3uuAnHe3nvCh"},"source":["df.columns"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"0SDpd1dxKeq4"},"source":["<details>\n","<summary><b>Click to see attribute information</b></summary>\n","Input features (column names):\n","\n","1.  `age` - client age in years (numeric)\n","2.  `job` - type of job (categorical: `admin.`, `blue-collar`, `entrepreneur`, `housemaid`, `management`, `retired`, `self-employed`, `services`, `student`, `technician`, `unemployed`, `unknown`)\n","3.  `marital` - marital status (categorical: `divorced`, `married`, `single`, `unknown`)\n","4.  `education` - client education (categorical: `basic.4y`, `basic.6y`, `basic.9y`, `high.school`, `illiterate`, `professional.course`, `university.degree`, `unknown`)\n","5.  `default` - has credit in default? (categorical: `no`, `yes`, `unknown`)\n","6.  `housing` - has housing loan? (categorical: `no`, `yes`, `unknown`)\n","7.  `loan` - has personal loan? (categorical: `no`, `yes`, `unknown`)\n","8.  `contact` - contact communication type (categorical: `cellular`, `telephone`)\n","9.  `month` - last contact month of the year (categorical: `jan`, `feb`, `mar`, ..., `nov`, `dec`)\n","10. `day_of_week` - last contact day of the week (categorical: `mon`, `tue`, `wed`, `thu`, `fri`)\n","11. `duration` - last contact duration, in seconds (numeric).\n","12. `campaign` - number of contacts performed for this client during this campaign (numeric, includes last contact)\n","13. `pdays` - number of days that have passed after the client was last contacted from the previous campaign (numeric; 999 means the client has not been previously contacted)\n","14. `previous` - number of contacts performed for this client before this campaign (numeric)\n","15. `poutcome` - outcome of the previous marketing campaign (categorical: `failure`, `nonexistent`, `success`)\n","16. `emp.var.rate` - employment variation rate, quarterly indicator (numeric)\n","17. `cons.price.idx` - consumer price index, monthly indicator (numeric)\n","18. `cons.conf.idx` - consumer confidence index, monthly indicator (numeric)\n","19. `euribor3m` - euribor 3 month rate, daily indicator (numeric)\n","20. `nr.employed` - number of employees, quarterly indicator (numeric)\n","\n","Output feature (desired target):\n","\n","21. `y` - has the client subscribed a term deposit? (binary: `yes`,`no`)\n","\n","</details>\n"]},{"cell_type":"markdown","metadata":{"id":"XB5od6PProOS"},"source":["Let's look at the dataset size.\n"]},{"cell_type":"code","metadata":{"id":"f49dL9CorTvr"},"source":["df.shape"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"g7DlvW_ArsaJ"},"source":["The dataset contains 41188 objects (rows), for each of which 21 features are set (columns), including 1 target feature (y).\n"]},{"cell_type":"markdown","metadata":{"id":"CBNS_29GnFmh"},"source":["## Overview of Python libraries for visual data analysis\n"]},{"cell_type":"markdown","metadata":{"id":"nHk4fQ0cpjLD"},"source":["### Matplotlib\n"]},{"cell_type":"markdown","metadata":{"id":"-C71ti6onFms"},"source":["Let's start our overview of Python libraries for visual data analysis with the simplest and fastest way to visualize data from Pandas DataFrame - to use the functions `plot` and`  hist `. The implementation of these functions in Pandas is based on the matplotlib library.\n"]},{"cell_type":"markdown","metadata":{"id":"A0Y8-x9msrNM"},"source":["For each feature, you can build a separate histogram with `hist` function:\n"]},{"cell_type":"code","metadata":{"id":"c_o0PxKfsrNW"},"source":["df[\"age\"].hist()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"7je24MhDsrNX"},"source":["The histogram shows that most of our clients are between the ages of 25 and 50, which corresponds to the actively working part of the population.\n"]},{"cell_type":"markdown","metadata":{"id":"n_q7TPhasvcU"},"source":["We will build a graph of the average client age depending on the marital status. To begin with, we only specify the columns we need, then calc the average values ​​and for the received DataFrame call the `plot` function without parameters.\n","\n","Replace ##YOUR CODE GOES HERE## with your Python code.\n"]},{"cell_type":"code","metadata":{"id":"H6osnIdpnFmt"},"source":["df[[\"age\", \"marital\"]].groupby(\n","    \"marital\"\n",")##YOUR CODE GOES HERE## "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ZEX8dxcf2G1g"},"source":["Double-click **here** for the solution.\n","\n","<!-- \n","df[[\"age\", \"marital\"]].groupby(\n","    \"marital\"\n",").mean().plot();\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"T4AvZrQNvaoa"},"source":["The plot shows that the average age of unmarried clients is significantly lower than that of the other clients.\n"]},{"cell_type":"markdown","metadata":{"id":"keZdzv8inFmt"},"source":["With the `kind` parameter you can change the plot type, for example, to a bar chart. MATPLOTLIB allows you to configure graphics very flexibly. You can change almost anything on the chart, but you will need to look up the necessary parameters in the [documentation](https://matplotlib.org/stable/contents.html?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01). For example, the `rot` parameter is responsible for the angle of tilt signatures to the x axis.\n"]},{"cell_type":"code","metadata":{"id":"J9Wk3yuPnFmu"},"source":["df[[\"age\", \"marital\"]].groupby(\n","    \"marital\"\n",").mean().plot(kind=\"bar\", rot=45);"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ZktFF1E4nFmv"},"source":["### Seaborn\n"]},{"cell_type":"markdown","metadata":{"id":"AwhSZvI13VmR"},"source":["Now let's go to the seaborn library. Seaborn is a higher-level API based on the matplotlib library. Seaborn contains more adequate default graphics settings. Also there are quite complex types of visualization in the library, which would require a large amount of code in matplotlib.\n","\n","We will get acquainted with the first \"complex\" type of pair plot graphics (Scatter Plot Matrix). This visualization will help us to look at one picture as at interconnection of various features.\n"]},{"cell_type":"code","metadata":{"id":"O-zVYPTynFmw"},"source":["sns.pairplot(\n","    df[[\"age\", \"duration\", \"campaign\"]]\n",");"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"doAEMmh_xQkP"},"source":["This visualization allows us to identify an interesting inverse relationship between a campaign and duration, which indicates a decrease in the duration of contact with the client with an increase in their contact quantity during the campaign.\n"]},{"cell_type":"markdown","metadata":{"id":"uUfKET7anFmw"},"source":["Also with the help of `seaborn` you can build a distribution, for example, look at the distribution of the client age. To do this, build `distplot`. By default, the graph shows a histogram and [Kernel Density Estimation](https://en.wikipedia.org/wiki/kernel_density_estimation?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01).\n"]},{"cell_type":"code","metadata":{"id":"DV4aX1spnFmx"},"source":["sns.distplot(df.age);"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vyZKe8kDnFmx"},"source":["In order to look more for the relationship between two numerical features, there is also `joint_plot` - this is a hybrid Scatter Plot and Histogram (there are also histograms of feature distributions). Let's look at the relationship between the number of contacts in a campaign and the last contact duration.\n"]},{"cell_type":"code","metadata":{"id":"6rdESC6BnFmy"},"source":["sns.jointplot(x=\"age\", y=\"duration\", data=df, kind=\"scatter\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ljz7TGqhnFmz"},"source":["Another useful seaborn plot type is [**Box Plot** (\"Box and whisker plot\")](https://en.wikipedia.org/wiki/box_plot?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01). Let's compare the age of customers for the top 5 of the most common employment forms.\n"]},{"cell_type":"code","metadata":{"id":"PFevJp6enFmz"},"source":["top_jobs = (\n","    df.job.value_counts().sort_values(ascending=False).head(5).index.values\n",")\n","sns.boxplot(\n","    y=\"job\", x=\"age\", data=df[df.job.isin(top_jobs)], orient=\"h\"\n",")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VTY-BtMR0tch"},"source":["The plot shows that among the top-5 client categories by the type of employment, the most senior customers represent the management, and the largest number of outliers is among the categories of admin. and technician.\n"]},{"cell_type":"markdown","metadata":{"id":"qIFKgSWInFm1"},"source":["And one more plot type (the last of those we consider in this chapter) is a `heat map`. A Heat Map allows you to look at the distribution of some numerical feature in two categories. We visualize the distribution of clients on family status and the type of employment.\n"]},{"cell_type":"code","metadata":{"id":"h7PiXTcjnFm2"},"source":["job_marital_y = (\n","    df.pivot_table(\n","        index=\"job\", columns=\"marital\", values=\"y\", aggfunc=sum\n","    )\n",")\n","sns.heatmap(job_marital_y, annot=True, fmt=\"d\", linewidths=0.5);"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"7VuAXN6T2ov1"},"source":["The plot shows that the largest number of attracted clients among administrative workers is married (652), and there is the smallest number of attracted clients among customers with an unknown family status.\n"]},{"cell_type":"markdown","metadata":{"id":"E65qpqFmnFm4"},"source":["### Plotly\n"]},{"cell_type":"markdown","metadata":{"id":"gMjIzZcC14WX"},"source":["We looked at the visualization based on the Library `Matplotlib` and `Seaborn`. However, this is not the only option to build charts with `Python`. We will also get acquainted with the library `plotly`. `Plotly` is an open-source library that allows you to build interactive graphics in a jupyter notebook without having to break into JavaScript code.\n","\n","The beauty of interactive graphs is that you can see the exact numerical value on mouse hover, hide the uninteresting rows in the visualization, zoom in a certain area of ​​graphics, etc.\n"]},{"cell_type":"markdown","metadata":{"id":"2tfDOIU-nFm6"},"source":["To begin with, we build **Line Plot** with the distribution of the total number and the number of attracted clients by age.\n"]},{"cell_type":"code","metadata":{"id":"yWD3rg2znFm6"},"source":["age_df = (\n","    df.groupby(\"age\")[[\"y\"]]\n","    .sum()\n","    .join(df.groupby(\"age\")[[\"y\"]].count(), rsuffix='_count')\n",")\n","age_df.columns = [\"Attracted\", \"Total Number\"]"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"cBrLH5x4nFm7"},"source":["In `Plotly`, we create the `Figure` object, which consists of data (list of lines that are called `traces`) and design/style, for which the object `Layout` was created. In simple cases, you can call the function `iplot` just for the `traces` list.\n"]},{"cell_type":"code","metadata":{"id":"5SPMxY_TnFm7"},"source":["trace0 = go.Scatter(x=age_df.index, y=age_df[\"Attracted\"], name=\"Attracted\")\n","trace1 = go.Scatter(x=age_df.index, y=age_df[\"Total Number\"], name=\"Total Number\")\n","\n","data = [trace0, trace1]\n","layout = {\"title\": \"Statistics by client age\"}\n","\n","fig = go.Figure(data=data, layout=layout)\n","\n","iplot(fig, show_link=False)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"gIzcpvXmnFm8"},"source":["Let us also see the distribution of customers by months, designed by the number of attracted clients and on the total number of clients. To do this, build **Bar Chart**.\n"]},{"cell_type":"code","metadata":{"id":"xPSmbzlLnFm8"},"source":["month_index = [\"jan\", \"feb\", \"mar\", \"apr\", \"may\", \"jun\", \"jul\", \"aug\", \"sep\", \"oct\", \"nov\", \"dec\"]\n","month_df = (\n","    df.groupby(\"month\")[[\"y\"]]\n","    .sum()\n","    .join(df.groupby(\"month\")[[\"y\"]].count(), rsuffix='_count')\n",").reindex(month_index)\n","month_df.columns = [\"Attracted\", \"Total Number\"]"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"Rn5IGabDnFm9"},"source":["trace0 = go.Bar(x=month_df.index, y=month_df[\"Attracted\"], name=\"Attracted\")\n","trace1 = go.Bar(x=month_df.index, y=month_df[\"Total Number\"], name=\"Total Number\")\n","\n","data = [trace0, trace1]\n","layout = {\"title\": \"Share of months\"}\n","\n","fig = go.Figure(data=data, layout=layout)\n","\n","iplot(fig, show_link=False)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"IMPRQDHRnFm-"},"source":["`plotly` can build the **Box plot**. Consider the differences in the client age depending on the family status.\n"]},{"cell_type":"code","metadata":{"id":"23-WrtdynFm_"},"source":["data = []\n","\n","for status in df.marital.unique():\n","    data.append(go.Box(y=df[df.marital == status].age, name=status))\n","iplot(data, show_link=False)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"A__r6gs-dlAY"},"source":["The plot clearly shows the distribution of clients by age, the presence of outliers for all categories of the family status, except for `unknown`. Moreover, the plot is interactive - hovering the mouse pointer to its elements allows you to obtain additional statistical characteristics of the series. Discover the characteristics.\n"]},{"cell_type":"markdown","metadata":{"id":"R_iXyoVrePOO"},"source":["## Visual analysis of single features\n"]},{"cell_type":"markdown","metadata":{"id":"LmUitDY8fj9e"},"source":["Let us give the most commonly used plot types to analyze single features of data sets.\n"]},{"cell_type":"markdown","metadata":{"id":"2Tc7h5qBfZxq"},"source":["#### Numerical features\n"]},{"cell_type":"markdown","metadata":{"id":"cNJ5jDD9f1o9"},"source":["For the analysis of numerical features, a histogram and a box plot are most often used.\n"]},{"cell_type":"code","metadata":{"id":"jPRHn93nePOP"},"source":["df[\"age\"].hist();"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Dxhq-8485iTH"},"source":["Build a box plot for the `cons.price.idx` feature with `sns.boxplot` function.\n","\n","Replace ##YOUR CODE GOES HERE## with your Python code.\n"]},{"cell_type":"code","metadata":{"id":"pnOKdgdMePOQ"},"source":["##YOUR CODE GOES HERE##"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"HsYHl4LS6CUr"},"source":["Double-click **here** for the solution.\n","\n","<!-- \n","sns.boxplot(df[\"cons.price.idx\"])\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"5MjC_c_bePOR"},"source":["### Categorical features\n"]},{"cell_type":"markdown","metadata":{"id":"gVZHWxFeg15I"},"source":["Use the `countplot` graphics for effective analysis of categorical features.\n","It's effective to use the graphics of the type `CountPlot` for analyzing categorical features.\n","\n","Calculate the client distribution of marital status.\n"]},{"cell_type":"code","metadata":{"id":"rtN2ao0nePOR"},"source":["df[\"marital\"].value_counts().head()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"6YunY_xihKzl"},"source":["Let's calculate the client distribution on the fact of their involvement for signing a deposit as well.\n"]},{"cell_type":"code","metadata":{"id":"XZ6TqF7hePOS"},"source":["df[\"y\"].value_counts()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"rhaeI_C0hfkd"},"source":["Present this information graphically.\n"]},{"cell_type":"code","metadata":{"id":"Bg1wiKelePOT"},"source":["sns.countplot(df[\"y\"]);"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"riCnSFNf-Ix4"},"source":["Build the count plot for the `marital` feature with `sns.countplot` function.\n","\n","Replace ##YOUR CODE GOES HERE## with your Python code.\n"]},{"cell_type":"code","metadata":{"id":"X5GIiK-p-Ix6"},"source":["##YOUR CODE GOES HERE##"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Ttt6-23M-Ix8"},"source":["Double-click **here** for the solution.\n","\n","<!-- \n","sns.countplot(df[\"marital\"])\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"TFLSunxoiAAa"},"source":["Plot the graphical client distribution by the 5 most common types of employment.\n"]},{"cell_type":"code","metadata":{"id":"BOWQyPnHePOT"},"source":["plot = sns.countplot(df[df[\"job\"].isin(df[\"job\"].value_counts().head(5).index)][\"job\"])\n","plt.setp(plot.get_xticklabels(), rotation=90)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"tZglBSkqePOU"},"source":["## Visual analysis of the feature interaction\n"]},{"cell_type":"markdown","metadata":{"id":"FlO9KrbliZBJ"},"source":["### Numerical features\n"]},{"cell_type":"markdown","metadata":{"id":"_O7wIKRxifkG"},"source":["To analyze the interaction of numerical features, use `hist` (histogram), `pairplot` and `heatmap` plot functions.\n"]},{"cell_type":"markdown","metadata":{"id":"lPridDoDi3hz"},"source":["We visualize the values ​​of the economy macro indicators from the dataset.\n"]},{"cell_type":"code","metadata":{"id":"BqfegTDTePOU"},"source":["feat = [\"cons.price.idx\", \"cons.conf.idx\", \"euribor3m\", \"nr.employed\"]\n","\n","df[feat].hist()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ODSMBE8tByVt"},"source":["Build a pair plot set for the `feat` list with `sns.pairplot` function.\n","\n","Replace ##YOUR CODE GOES HERE## with your Python code.\n"]},{"cell_type":"code","metadata":{"id":"XfVSmraFByVu"},"source":["##YOUR CODE GOES HERE##"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"u-pOK0x0ByVw"},"source":["Double-click **here** for the solution.\n","\n","<!-- \n","sns.pairplot(df[feat])\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"6NmhLZhFCN7o"},"source":["Build a Heat Map for the economy macro indicators [correlation matrix](https://en.wikipedia.org/wiki/Correlation_and_dependence?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01#Correlation_matrices).\n"]},{"cell_type":"code","metadata":{"id":"4fIgFIU5ePOW"},"source":["sns.heatmap(df[feat].corr());"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"nvjGuxZ-DIq2"},"source":["We see a strong interaction between the `euribor3m` and `nr.employed` features.\n"]},{"cell_type":"markdown","metadata":{"id":"sOPzXp7oePOX"},"source":["### Numerical and categorical features\n"]},{"cell_type":"markdown","metadata":{"id":"mVX9iA8QmhFJ"},"source":["`boxplot` and `violinplot` are used for visual analysis of the numerical and categorical features.\n","\n","Let's look at the `age` feature box plot by the target feature.\n"]},{"cell_type":"code","metadata":{"id":"xf1YC6ZbePOY"},"source":["sns.boxplot(x=\"y\", y=\"age\", data=df)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Tls_xJa-EvmE"},"source":["Build the box plot for the `marital` feature with `sns.boxplot` function.\n","\n","Replace ##YOUR CODE GOES HERE## with your Python code.\n"]},{"cell_type":"code","metadata":{"id":"wB-OF9DjEvme"},"source":["##YOUR CODE GOES HERE##"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"gfqmysWhEvmf"},"source":["Double-click **here** for the solution.\n","\n","<!-- \n","sns.boxplot(x=\"marital\", y=\"age\", data=df)\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"36BKNyV0Ofb1"},"source":["You can draw a combination of boxplot and kernel density estimate with a `violinplot` function. A [violin plot](https://en.wikipedia.org/wiki/Violin_plot?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01) plays a similar role as a box and whisker plot. It shows the distribution of quantitative data across several levels of one (or more) categorical variables such that those distributions can be compared.\n","\n","Plot the client age distribution across the target feature.\n"]},{"cell_type":"code","metadata":{"id":"Hp9ugQVaePOc"},"source":["sns.violinplot(x=\"y\", y=\"age\", data=df);"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"QNhDnqtiPR_-"},"source":["It is useful to combine grouping with a `boxplot`. Calculate the mean client for the grouping by the `housing` feature values.\n"]},{"cell_type":"code","metadata":{"id":"aIxRSOBZePOf"},"source":["df.groupby(\"housing\")[\"age\"].mean()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ZYcaSj0pQUKW"},"source":["Build a box plot for the `age` feature by the `housing` values with `sns.boxplot` function.\n","\n","Replace ##YOUR CODE GOES HERE## with your Python code.\n"]},{"cell_type":"code","metadata":{"id":"8pEykrY8QUK1"},"source":["##YOUR CODE GOES HERE##"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"v9XhOMjAQUK3"},"source":["Double-click **here** for the solution.\n","\n","<!-- \n","sns.boxplot(x=\"housing\", y=\"age\", data=df)\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"Cpes0GzhePOh"},"source":["### Categorical features\n"]},{"cell_type":"markdown","metadata":{"id":"FbgYuPVYo5JZ"},"source":["Use `countplot` for a visual interaction analysis between categorical features.\n","\n","Calculate and visualize the interaction between target and client marital status features.\n"]},{"cell_type":"code","metadata":{"id":"fuJ9p-OEePOx"},"source":["pd.crosstab(df[\"y\"], df[\"marital\"])"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"BQNGSaRCePOz"},"source":["sns.countplot(x=\"marital\", hue=\"y\", data=df)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"0z0H_pDhR9iy"},"source":["Build the count plot for the `month` feature by the `y` feature target values with `sns.countplot` function.\n","\n","Replace ##YOUR CODE GOES HERE## with your Python code.\n"]},{"cell_type":"code","metadata":{"id":"BeCQ6K81R9iz"},"source":["##YOUR CODE GOES HERE##"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"fam5ChN8R9iz"},"source":["Double-click **here** for the solution.\n","\n","<!-- \n","sns.countplot(x=\"month\", hue=\"y\", data=df)\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"C9j4xQI4qSjn"},"source":["## Comprehensive visual analysis of the source banking dataset\n"]},{"cell_type":"markdown","metadata":{"id":"0PY1o5GUpaf0"},"source":["Create the `categorical` and `numerical` lists for the correspondent dataset features.\n","\n","Let's look at the distribution of numerical features with `hist` function.\n"]},{"cell_type":"code","metadata":{"id":"-cEB9X9_paf1"},"source":["categorical = []\n","numerical = []\n","for feature in df.columns:\n","    if df[feature].dtype == object:\n","        categorical.append(feature)\n","    else:\n","        numerical.append(feature)\n","\n","df[numerical].hist(figsize=(20,12), bins=100, color='lightgreen')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"elOyPe3Epaf2"},"source":["From the histograms, we see that **for each numerical feature there is one or more dominant segments of values​**, that is why we got pronounced peaks.\n","\n","In addition, we see that the target feature is unbalanced. **The number of positive outcomes is significantly lower than negative**, which is quite natural for telephone marketing. As a result, the problem arises with the fact that many methods are sensitive to the imbalance of features. We will try to solve this problem later.\n","\n","Next, let's look at the categorical features.\n"]},{"cell_type":"code","metadata":{"id":"Isz2GxZPpaf2"},"source":["df.describe(include = ['object'])"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"YygpUEp5Ubo0"},"source":["Visualize the categorical features with bar plots.\n"]},{"cell_type":"code","metadata":{"id":"QlL5Uvpypaf2"},"source":["plt.rcParams['axes.labelsize'] = 20\n","plt.rcParams['axes.titlesize'] = 20\n","plt.rcParams['font.size'] = 20\n","\n","fig, axes = plt.subplots(ncols=4, nrows = 3, figsize=(24, 18))\n","plt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.4)\n","\n","for i in range(len(categorical)):\n","    df[categorical[i]].value_counts(normalize=True).plot(kind='bar', label=categorical[i], ax=axes[i//4, i%4], color='lightgreen')\n","    axes[i//4, i%4].set_title(categorical[i])\n","plt.tight_layout()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"aSiggj1tpaf3"},"source":["As we see, for many features, some of the groups stand out, for example, in the dataset more than half of the clients are married.\n"]},{"cell_type":"markdown","metadata":{"id":"RD6Cx4e9paf4"},"source":["Let's look at the correlation matrix (for the numerical features).\n"]},{"cell_type":"code","metadata":{"id":"q9SOOxyypaf4"},"source":["correlation_table = df.corr()\n","correlation_table"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"K94_sQAXpaf4"},"source":["We visualize the correlation matrix.\n"]},{"cell_type":"code","metadata":{"id":"qWA65imJpaf5"},"source":["sns.heatmap(correlation_table)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"jHlWT6Yjpaf6"},"source":["Let's look at the visualized dependences of numerical features from the target feature with scatter plots.\n"]},{"cell_type":"code","metadata":{"id":"BP2dEo3Jpaf6"},"source":["fig, axes = plt.subplots(ncols=4, nrows = 3, figsize=(24, 18))\n","plt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.4)\n","\n","for i in range(len(numerical)):\n","    df.plot(x=numerical[i], y = 'y', label=numerical[i], ax=axes[i//4, i%4], kind='scatter', color='green')\n","    axes[i//4, i%4].set_title(numerical[i])\n","plt.tight_layout()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"xUm7zu5kpaf7"},"source":["As you can see, there are points that can be interpreted as outliers, however, we will not hurry to delete them because they don't seem to be true outliers. These points are too strong so we will leave them. In addition, we will use some models that are resistant to outliers.\n"]},{"cell_type":"markdown","metadata":{"id":"-bDqLeOwpaf9"},"source":["We visualize **the distribution of positive target responses** by groups:\n"]},{"cell_type":"code","metadata":{"id":"bx5PGZS9paf9"},"source":["fig, axes = plt.subplots(ncols=4, nrows = 3, figsize=(24, 18))\n","plt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.4)\n","\n","for i in range(len(categorical)):\n","    df.groupby(categorical[i])['y'].mean().plot(kind='bar', ax=axes[i//4, i%4], color='green')\n","    axes[i//4, i%4].set_title(categorical[i])\n","\n","plt.tight_layout()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-sHNbOcNpaf-"},"source":["In such a form, plots are already more interesting. So we see, for many features, the chance of a positive response is significantly higher.\n","\n","We also see that `housing`, `loan` and `day_of_week` features will hardly help us, because judging by the plots, the share of positive target responses hardly depends on them.\n"]},{"cell_type":"markdown","metadata":{"id":"_vC4CsLTpaf_"},"source":["### Conclusions\n"]},{"cell_type":"markdown","metadata":{"id":"AUuYbj1bvujn"},"source":["There are neither any data missing, nor explicit outliers that should be cut. But we can omit `housing`, `loan` and `day_of_week` features in the next steps.\n","\n","The `euribor3m` and `nr.employed` features strongly correlate with `emp.var.rate`. Let me remind you that `emp.var.rate` - Employment Variation Rate is a quarterly indicator, `euribor3m` - euribor 3 month rate is a day indicator, and `nr.employed` - number of employees is a quarterly indicator. The correlation of the employment change with the number of employed issues itself is obvious, but its correlation with EURIBOR (Euro Interbank Offered Rate, the European interbank offer rate) is interesting. This indicator is based on the average interbank interest rates in Eurozone. It also has a positive effect since the higher the interest rate is, the more willingly customers will spend their money on financial tools.\n","\n","Therefore, if banks want to improve their lead generation, what they should do is to improve the quality of phone conversations and run their campaigns when interest rates are high and the macroeconomic environment is stable.\n"]},{"cell_type":"markdown","metadata":{"id":"VUMjKczqTThq"},"source":["## Tasks\n"]},{"cell_type":"markdown","metadata":{"id":"jLCT0HXkTThr"},"source":["In this section you will solve an additional task with the source bank dataset.\n"]},{"cell_type":"markdown","metadata":{"id":"_PNycuTMTThv"},"source":["Let's compare the age of customers for the top 3 of the most common levels of education using a box plot.\n"]},{"cell_type":"code","metadata":{"id":"o2Hxx2FzTThw"},"source":["# Code block for students' answer\n","# Replace ##YOUR CODE GOES HERE## with your Python code.\n","\n","top_3 = (\n","    ##YOUR CODE GOES HERE##\n",")\n","##YOUR CODE GOES HERE##(\n","    y=\"education\", x=\"age\", data=df[df.education.isin(top_3)], orient=\"h\"\n",")\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"1xAxHU1HotaC"},"source":["Double-click **here** for the solution.\n","\n","<!-- \n","top_3 = (\n","    df.education.value_counts().sort_values(ascending=False).head(3).index.values\n",")\n","sns.boxplot(\n","    y=\"education\", x=\"age\", data=df[df.education.isin(top_3)], orient=\"h\"\n",")\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"iEbNXiqmTTh0"},"source":["## Authors\n"]},{"cell_type":"markdown","metadata":{"id":"6_1ZOvXKTTh0"},"source":["[Roman Yatsenko](https://www.linkedin.com/in/yatsenkorn/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01)\n"]},{"cell_type":"markdown","metadata":{"id":"bHpM3GyTTTh2"},"source":["## Change Log\n"]},{"cell_type":"markdown","metadata":{"id":"mrgMUa_ETTh2"},"source":["| Date (YYYY-MM-DD) | Version | Changed By      | Change Description   |\n","| ----------------- | ------- | --------------- | -------------------- |\n","| 2021-06-12        | 0.2     | Yatsenko, Roman | Translate to english |\n","| 2021-06-04        | 0.1     | Yatsenko, Roman | Created Lab          |\n"]},{"cell_type":"markdown","metadata":{"id":"Ma5EMufCTTh2"},"source":["Copyright © 2021 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkQuickLabsVDA_Banking_L227636609-2021-01-01).\n"]}]}