RSNA-Exploratory Data Analysis Part 2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RSNA - Exploratory Data Analysis Part 2\n",
    "\n",
    "In this notebook, we are going to take a look at the DICOM dataset. We are going to extract the <b>dicom elements</b>, such as window center, pixel spacing, rows and columns, from each image. We will put then these information into a table.  \n",
    "\n",
    "By inspecting this table, we are going to gain insight on the technical specifications of these images, and hopefully answer the questions like \n",
    "\n",
    "\"Are all images of about the same size?\" \n",
    "\n",
    "\"Do these images have multiple windows?\" \n",
    "\n",
    "\"Do all images share similar technical specfications?\" \n",
    "\n",
    "\"Is there any oddity that may skew the predictions?\"\n",
    "\n",
    "\n",
    "Later on, we will use the window center, window width, rescale intercept and rescale slope to process each image. The processed images will be the input to our deep learning models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import pydicom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_dir = \"stage_1_train_images/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "674257\n"
     ]
    }
   ],
   "source": [
    "data_table = pd.read_pickle('rsna_data_table.pkl') \n",
    "print(len(data_table))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Take a look at a sample image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_id = data_table.index[0]\n",
    "file_name = os.path.join(img_dir,\"ID_\"+img_id+\".dcm\")\n",
    "dicom_elements = pydicom.dcmread(file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0008, 0018) SOP Instance UID                    UI: ID_000039fa0\n",
       "(0008, 0060) Modality                            CS: 'CT'\n",
       "(0010, 0020) Patient ID                          LO: 'ID_eeaf99e7'\n",
       "(0020, 000d) Study Instance UID                  UI: ID_134d398b61\n",
       "(0020, 000e) Series Instance UID                 UI: ID_5f8484c3e0\n",
       "(0020, 0010) Study ID                            SH: ''\n",
       "(0020, 0032) Image Position (Patient)            DS: ['-125.000000', '-141.318451', '62.720940']\n",
       "(0020, 0037) Image Orientation (Patient)         DS: ['1.000000', '0.000000', '0.000000', '0.000000', '0.968148', '-0.250380']\n",
       "(0028, 0002) Samples per Pixel                   US: 1\n",
       "(0028, 0004) Photometric Interpretation          CS: 'MONOCHROME2'\n",
       "(0028, 0010) Rows                                US: 512\n",
       "(0028, 0011) Columns                             US: 512\n",
       "(0028, 0030) Pixel Spacing                       DS: ['0.488281', '0.488281']\n",
       "(0028, 0100) Bits Allocated                      US: 16\n",
       "(0028, 0101) Bits Stored                         US: 16\n",
       "(0028, 0102) High Bit                            US: 15\n",
       "(0028, 0103) Pixel Representation                US: 1\n",
       "(0028, 1050) Window Center                       DS: \"30\"\n",
       "(0028, 1051) Window Width                        DS: \"80\"\n",
       "(0028, 1052) Rescale Intercept                   DS: \"-1024\"\n",
       "(0028, 1053) Rescale Slope                       DS: \"1\"\n",
       "(7fe0, 0010) Pixel Data                          OW: Array of 524288 elements"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#These are the dicom elements we are interested in\n",
    "dicom_dict = {'image pos':['0020','0032'],\n",
    "             'image orient':['0020','0037'],\n",
    "             'samples per pixel':['0028','0002'],\n",
    "             'photometric interp':['0028','0004'],\n",
    "             'rows':['0028','0010'],\n",
    "             'columns':['0028','0011'] ,\n",
    "             'pixel spacing':['0028','0030'],\n",
    "             'bits allocated':['0028','0100'],\n",
    "             'bits stored':['0028','0101'],\n",
    "             'high bits':['0028','0102'],\n",
    "             'pixel representation':['0028','0103'], \n",
    "             'window center':['0028','1050'], \n",
    "             'window width':['0028','1051'], \n",
    "             'rescale intercept':['0028','1052'], \n",
    "             'rescale slope':['0028','1053']}      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#A function to convert the element values into standard python types\n",
    "def parse_element(dicom_elements, element_code):\n",
    "    \n",
    "    element = dicom_elements.get(element_code)\n",
    "    \n",
    "    if element:\n",
    "        if isinstance(element.value, pydicom.multival.MultiValue):       \n",
    "            val = []\n",
    "            if isinstance(element.value[0], int):\n",
    "                for x in element.value:\n",
    "                    val.append(int(x))\n",
    "                \n",
    "            elif isinstance(element.value[0], str):\n",
    "                for x in element.value:\n",
    "                    val.append(str(x))\n",
    "                  \n",
    "            elif isinstance(element.value[0], pydicom.valuerep.DSfloat):    \n",
    "                for x in element.value:\n",
    "                    val.append(float(x))                \n",
    "            else:\n",
    "                val = 'warning: unknown list type'            \n",
    "        \n",
    "        elif isinstance(element.value, int):\n",
    "                val = element.value\n",
    "        elif isinstance(element.value, str):\n",
    "                val = element.value\n",
    "        elif isinstance(element.value, pydicom.valuerep.DSfloat):    \n",
    "                val = float(element.value)\n",
    "        else:\n",
    "                val = 'warning: unknown type'     \n",
    "    else:\n",
    "        val = 'warning: element not found'\n",
    "    \n",
    "    return val   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-125.0, -141.318451, 62.72094]\n",
      "[1.0, 0.0, 0.0, 0.0, 0.968148, -0.25038]\n",
      "1\n",
      "MONOCHROME2\n",
      "512\n",
      "512\n",
      "[0.488281, 0.488281]\n",
      "16\n",
      "16\n",
      "15\n",
      "1\n",
      "30.0\n",
      "80.0\n",
      "-1024.0\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "for x in dicom_dict:\n",
    "    print(parse_element(dicom_elements, dicom_dict[x]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Remark: we parsed the element in a DICOM file and converted them to Python types, now we can make a table of for all DICOM files in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Create a DICOM info table for the entire training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "table_columns = list(dicom_dict.keys())  \n",
    "table_columns.append('id')\n",
    "dicom_table = pd.DataFrame(columns = table_columns) \n",
    "\n",
    "#the dicom_table share the same id with the data_table, \n",
    "#for each image, we will retrieve the diagnosis from the data table,\n",
    "#and its dicom info from the DICOM info table.\n",
    "\n",
    "dicom_table['id'] = data_table.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#each row corresponds to the DICOM info of one image\n",
    "def read_img(table_row, img_dir, dicom_dict):\n",
    "    \n",
    "    img_id = table_row['id']   \n",
    "    img_file = os.path.join(img_dir,\"ID_\"+img_id+\".dcm\")\n",
    "    dicom_elements = pydicom.dcmread(img_file)\n",
    "    \n",
    "    for  element_name, element_code in dicom_dict.items():\n",
    "         table_row[element_name] = parse_element(dicom_elements, element_code)\n",
    "    return table_row       \n",
    "       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "dicom_table = dicom_table.apply(lambda x: read_img(x, img_dir, dicom_dict), axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dicom_table = dicom_table.set_index('id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Save the table to a file\n",
    "dicom_table.to_pickle('rsna_dicom_table.pkl') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Inspect the DICOM info table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image pos</th>\n",
       "      <th>image orient</th>\n",
       "      <th>samples per pixel</th>\n",
       "      <th>photometric interp</th>\n",
       "      <th>rows</th>\n",
       "      <th>columns</th>\n",
       "      <th>pixel spacing</th>\n",
       "      <th>bits allocated</th>\n",
       "      <th>bits stored</th>\n",
       "      <th>high bits</th>\n",
       "      <th>pixel representation</th>\n",
       "      <th>window center</th>\n",
       "      <th>window width</th>\n",
       "      <th>rescale intercept</th>\n",
       "      <th>rescale slope</th>\n",
       "      <th>multiple windows</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>000039fa0</td>\n",
       "      <td>[-125.0, -141.318451, 62.72094]</td>\n",
       "      <td>[1.0, 0.0, 0.0, 0.0, 0.968148, -0.25038]</td>\n",
       "      <td>1</td>\n",
       "      <td>MONOCHROME2</td>\n",
       "      <td>512</td>\n",
       "      <td>512</td>\n",
       "      <td>[0.488281, 0.488281]</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>80</td>\n",
       "      <td>-1024.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>00005679d</td>\n",
       "      <td>[-134.463, -110.785, -39.569]</td>\n",
       "      <td>[1.0, 0.0, 0.0, 0.0, 1.0, 0.0]</td>\n",
       "      <td>1</td>\n",
       "      <td>MONOCHROME2</td>\n",
       "      <td>512</td>\n",
       "      <td>512</td>\n",
       "      <td>[0.460938, 0.460938]</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>100</td>\n",
       "      <td>-1024.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>00008ce3c</td>\n",
       "      <td>[-125.0, -83.0468112, 175.995344]</td>\n",
       "      <td>[1.0, 0.0, 0.0, 0.0, 0.994521895, 0.104528463]</td>\n",
       "      <td>1</td>\n",
       "      <td>MONOCHROME2</td>\n",
       "      <td>512</td>\n",
       "      <td>512</td>\n",
       "      <td>[0.48828125, 0.48828125]</td>\n",
       "      <td>16</td>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>[40.0, 40.0]</td>\n",
       "      <td>[80.0, 80.0]</td>\n",
       "      <td>-1024.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0000950d7</td>\n",
       "      <td>[-126.437378, -126.437378, 157.5]</td>\n",
       "      <td>[1.0, 0.0, 0.0, 0.0, 1.0, 0.0]</td>\n",
       "      <td>1</td>\n",
       "      <td>MONOCHROME2</td>\n",
       "      <td>512</td>\n",
       "      <td>512</td>\n",
       "      <td>[0.494863, 0.494863]</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>135</td>\n",
       "      <td>-1024.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0000aee4b</td>\n",
       "      <td>[-108.5, 14.5, 94.0]</td>\n",
       "      <td>[1.0, 0.0, 0.0, 0.0, 1.0, 0.0]</td>\n",
       "      <td>1</td>\n",
       "      <td>MONOCHROME2</td>\n",
       "      <td>512</td>\n",
       "      <td>512</td>\n",
       "      <td>[0.423828125, 0.423828125]</td>\n",
       "      <td>16</td>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>[36.0, 36.0]</td>\n",
       "      <td>[80.0, 80.0]</td>\n",
       "      <td>-1024.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   image pos  \\\n",
       "id                                             \n",
       "000039fa0    [-125.0, -141.318451, 62.72094]   \n",
       "00005679d      [-134.463, -110.785, -39.569]   \n",
       "00008ce3c  [-125.0, -83.0468112, 175.995344]   \n",
       "0000950d7  [-126.437378, -126.437378, 157.5]   \n",
       "0000aee4b               [-108.5, 14.5, 94.0]   \n",
       "\n",
       "                                             image orient  samples per pixel  \\\n",
       "id                                                                             \n",
       "000039fa0        [1.0, 0.0, 0.0, 0.0, 0.968148, -0.25038]                  1   \n",
       "00005679d                  [1.0, 0.0, 0.0, 0.0, 1.0, 0.0]                  1   \n",
       "00008ce3c  [1.0, 0.0, 0.0, 0.0, 0.994521895, 0.104528463]                  1   \n",
       "0000950d7                  [1.0, 0.0, 0.0, 0.0, 1.0, 0.0]                  1   \n",
       "0000aee4b                  [1.0, 0.0, 0.0, 0.0, 1.0, 0.0]                  1   \n",
       "\n",
       "          photometric interp  rows  columns               pixel spacing  \\\n",
       "id                                                                        \n",
       "000039fa0        MONOCHROME2   512      512        [0.488281, 0.488281]   \n",
       "00005679d        MONOCHROME2   512      512        [0.460938, 0.460938]   \n",
       "00008ce3c        MONOCHROME2   512      512    [0.48828125, 0.48828125]   \n",
       "0000950d7        MONOCHROME2   512      512        [0.494863, 0.494863]   \n",
       "0000aee4b        MONOCHROME2   512      512  [0.423828125, 0.423828125]   \n",
       "\n",
       "           bits allocated  bits stored  high bits  pixel representation  \\\n",
       "id                                                                        \n",
       "000039fa0              16           16         15                     1   \n",
       "00005679d              16           16         15                     1   \n",
       "00008ce3c              16           12         11                     0   \n",
       "0000950d7              16           16         15                     1   \n",
       "0000aee4b              16           12         11                     0   \n",
       "\n",
       "          window center  window width  rescale intercept  rescale slope  \\\n",
       "id                                                                        \n",
       "000039fa0            30            80            -1024.0            1.0   \n",
       "00005679d            50           100            -1024.0            1.0   \n",
       "00008ce3c  [40.0, 40.0]  [80.0, 80.0]            -1024.0            1.0   \n",
       "0000950d7            35           135            -1024.0            1.0   \n",
       "0000aee4b  [36.0, 36.0]  [80.0, 80.0]            -1024.0            1.0   \n",
       "\n",
       "           multiple windows  \n",
       "id                           \n",
       "000039fa0             False  \n",
       "00005679d             False  \n",
       "00008ce3c             False  \n",
       "0000950d7             False  \n",
       "0000aee4b             False  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### observation: it seems some dicom elements, like window center  and window width, have variable length (1 or 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def item_length(x):\n",
    "    if isinstance(x, int) or isinstance(x,float) or isinstance(x,str):\n",
    "        return 1\n",
    "    return len(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3    674258\n",
      "Name: image pos, dtype: int64\n",
      "6    674258\n",
      "Name: image orient, dtype: int64\n",
      "1    674258\n",
      "Name: samples per pixel, dtype: int64\n",
      "1    674258\n",
      "Name: photometric interp, dtype: int64\n",
      "1    674258\n",
      "Name: rows, dtype: int64\n",
      "1    674258\n",
      "Name: columns, dtype: int64\n",
      "2    674258\n",
      "Name: pixel spacing, dtype: int64\n",
      "1    674258\n",
      "Name: bits allocated, dtype: int64\n",
      "1    674258\n",
      "Name: bits stored, dtype: int64\n",
      "1    674258\n",
      "Name: high bits, dtype: int64\n",
      "1    674258\n",
      "Name: pixel representation, dtype: int64\n",
      "1    341679\n",
      "2    332579\n",
      "Name: window center, dtype: int64\n",
      "1    341679\n",
      "2    332579\n",
      "Name: window width, dtype: int64\n",
      "1    674258\n",
      "Name: rescale intercept, dtype: int64\n",
      "1    674258\n",
      "Name: rescale slope, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#check if the dicom elements have variable length\n",
    "for  element_name, element_code in dicom_dict.items():\n",
    "     print(dicom_table[element_name].apply(item_length).value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### observation: only window center and window width have variable length. That means some DICOM images in the dataset have multiple windows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def all_values_equal(x):\n",
    "    if isinstance(x,float) or isinstance(x,int):\n",
    "        return True\n",
    "    y = np.asarray(x)\n",
    "    return (y == min(y)).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True     674224\n",
       "False        34\n",
       "Name: window center, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['window center'].apply(all_values_equal).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True     674224\n",
       "False        34\n",
       "Name: window width, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['window width'].apply(all_values_equal).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "34"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find the DICOM files that have multiple window\n",
    "multiple_width = ~dicom_table['window width'].apply(all_values_equal) \n",
    "multiple_center = ~dicom_table['window center'].apply(all_values_equal)\n",
    "dicom_table['multiple windows'] = multiple_width | multiple_center\n",
    "\n",
    "len(dicom_table[dicom_table['multiple windows']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### observation: 34 images have multiple windows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    674258\n",
       "Name: samples per pixel, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['samples per pixel'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MONOCHROME2    674258\n",
       "Name: photometric interp, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['photometric interp'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "512    673989\n",
       "638        49\n",
       "436        36\n",
       "408        33\n",
       "464        32\n",
       "462        32\n",
       "430        31\n",
       "666        29\n",
       "768        27\n",
       "Name: rows, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['rows'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "512    674018\n",
       "490        49\n",
       "436        36\n",
       "374        33\n",
       "464        32\n",
       "462        32\n",
       "404        31\n",
       "768        27\n",
       "Name: columns, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['columns'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### observation: image size varies quite a bit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16    674258\n",
       "Name: bits allocated, dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['bits allocated'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16    341679\n",
       "12    332579\n",
       "Name: bits stored, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['bits stored'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15    341679\n",
       "11    332579\n",
       "Name: high bits, dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['high bits'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    343931\n",
       "0    330327\n",
       "Name: pixel representation, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['pixel representation'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1024.0    662279\n",
       "-1000.0      6653\n",
       " 0.0         5276\n",
       " 1.0           50\n",
       "Name: rescale intercept, dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['rescale intercept'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### observation: about 5000 images used a very different linear function to convert from pixel to HU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0    674258\n",
       "Name: rescale slope, dtype: int64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['rescale slope'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    674258.000000\n",
       "mean          0.478767\n",
       "std           0.027809\n",
       "min           0.292969\n",
       "25%           0.488281\n",
       "50%           0.488281\n",
       "75%           0.488281\n",
       "max           0.976562\n",
       "Name: pixel spacing, dtype: float64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['pixel spacing'].apply(lambda x: x[0]).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    674258.000000\n",
       "mean          0.478767\n",
       "std           0.027809\n",
       "min           0.292969\n",
       "25%           0.488281\n",
       "50%           0.488281\n",
       "75%           0.488281\n",
       "max           0.976562\n",
       "Name: pixel spacing, dtype: float64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dicom_table['pixel spacing'].apply(lambda x: x[1]).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### observation: most images have similar pixel spacing, but there are extreme cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Save the table to a file\n",
    "dicom_table.to_pickle('rsna_dicom_table.pkl') "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}