NB - Data Curation 1 - Cleaning and Filtering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Data Curation Part 1 - Cleaning and Filtering\n",
    "\n",
    "This part filters and cleans the Swiss Dwellings (SD) dataset using the following steps (same as [paper](https://arxiv.org/abs/2407.10121); copy-pasted text):\n",
    "\n",
    "[$\\bullet$] **Feature removal.**\n",
    "All non-floorplan geometries are removed (\\eg \"bathtub\", \"stairs\"; see the full list in the suppl. mat.).\n",
    "\n",
    "[$\\bullet$] **Residential-only filtering.**\n",
    "We remove floor plans that include non-residential-like geometric details (\\eg areas categorized as \"waiting room\", \"dedicated medical room\"; see the full list in the suppl. mat.).\n",
    "This led to the removal of 2,305 (16.6\\%) floor plans.\n",
    "\n",
    "[$\\bullet$] **Near-duplicate removal.**\n",
    "Many floor plans that come from the same building stem from the same plan ID~\\cite{standfest_swiss_2022} (see suppl. mat. for details on ID nesting in SD).\n",
    "Floor plans with the same plan ID are based on the same layout, indicating that the spatial arrangements are nearly identical.\n",
    "Hence, we sample only one-floor plan per plan ID to drastically reduce the amount of near-duplicates.\n",
    "Specifically, we sample the floor plan with the lowest elevation.\n",
    "This led to the removal of 4,395 (31.6\\%) floor plans.\n",
    "\n",
    "[$\\bullet$] **Medium- to large-scale filtering.**\n",
    "Floor plans are removed that contain fewer than 15 areas. In addition, every floor plan should have at least two \"Zone 2\"-categorized areas.\n",
    "This led to the removal of 1,541 (11.1\\%) floor plans.\n",
    "\n",
    "Note that the computation needed for the filtering and cleaning is done in mere seconds."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Loading and acquiring information\n",
    "\n",
    "You need to download the Swiss Dwellings (SD) `geometries.csv` file here: [SD 3.0.0 (latest version)](https://zenodo.org/records/7788422).\n",
    "\n",
    "We extensively use `Pandas` and `GeoPandas` for filtering and cleaning of the dataset. In the following code block, we load the complete table as a `GeoDataFrame`:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:37:11.673394Z",
     "end_time": "2024-11-29T10:37:24.520719Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from geopandas import GeoDataFrame as gdf\n",
    "\n",
    "# Path to dataset\n",
    "path = r'C:\\Users\\caspervanengel\\OneDrive\\Documents\\PHD\\1_data\\swiss-dwellings\\3.0.0'\n",
    "\n",
    "# Load SD `geometries.csv` as geodataframe\n",
    "DF = gdf(pd.read_csv(os.path.join(path, 'geometries.csv')))\n",
    "DF = DF.rename(columns={'geometry': 'geom'})\n",
    "df = DF.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "For an in-depth understanding on the columns, please refer to the [documentation of SD](https://zenodo.org/records/7788422).\n",
    "\n",
    "We print some meta-level information of SD:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subtypes:\n",
      "\t['medical_room', 'entrance_door', 'loggia', 'shelter', 'ramp', 'elevator_facilities', 'wash_and_dry_room', 'salesroom', 'bike_storage', 'air', 'room', 'balcony', 'office_space', 'railing', 'community_room', 'washing_machine', 'patio', 'vehicle_traffic_area', 'open_plan_office', 'bathroom', 'corridors_and_halls', 'radation_therapy', 'kitchen_dining', 'pram_and_bike_storage_room', 'transport_shaft', 'cloakroom', 'shower', 'common_kitchen', 'carpark', 'cold_storage', 'basement_compartment', 'built_in_furniture', 'lightwell', 'water_supply', 'foyer', 'arcade', 'bedroom', 'dedicated_medical_room', 'break_room', 'not_defined', 'meeting_room', 'archive', 'heating', 'living_dining', 'gas', 'reception_room', 'corridor', 'dining', 'staircase', 'teaching_room', 'electrical_supply', 'outdoor_void', 'counter_room', 'lobby', 'office', 'sink', 'sanitary_rooms', 'bathtub', 'workshop', 'sports_rooms', 'void', 'kitchen', 'showroom', 'wintergarten', 'canteen', 'door', 'operations_facilities', 'column', 'pram', 'physio_and_rehabilitation', 'technical_area', 'factory_room', 'window', 'garage', 'shaft', 'storeroom', 'stairs', 'basement', 'office_tech_room', 'elevator', 'garden', 'warehouse', 'logistics', 'waiting_room', 'toilet', 'living_room', 'wall', 'terrace', 'studio', 'house_technics_facilities', 'oil_tank']\n",
      "\n",
      "Types:\n",
      "\t['feature', 'opening', 'area', 'separator']\n",
      "\n",
      "Number of apartments:\t45176\n",
      "Number of sites:\t\t1466\n",
      "Number of buildings:\t3184\n",
      "Number of floors:\t\t13905\n",
      "Number of plans:\t\t8941\n",
      "Number of units:\t\t47285\n",
      "Number of areas:\t\t315037\n",
      "Number of geometries:\t3255905\n"
     ]
    }
   ],
   "source": [
    "# Entity subtypes and types lists\n",
    "entity_subtypes = set(df.entity_subtype)\n",
    "entity_types = set(df.entity_type)\n",
    "\n",
    "# Print information; be aware that subtypes are capitalized in the real dataset\n",
    "print(f'Subtypes:\\n\\t{[cat.lower() for cat in entity_subtypes]}\\n')\n",
    "print(f'Types:\\n\\t{[cat.lower() for cat in entity_types]}\\n')\n",
    "\n",
    "# Print information on dimensions\n",
    "# e.g., Number of distinct apartments, sites, buildings, etc.\n",
    "print(f'Number of apartments:\\t{len(df.apartment_id.unique())}')\n",
    "print(f'Number of sites:\\t\\t{len(df.site_id.unique())}')\n",
    "print(f'Number of buildings:\\t{len(df.building_id.unique())}')\n",
    "print(f'Number of floors:\\t\\t{len(df.floor_id.unique())}')\n",
    "print(f'Number of plans:\\t\\t{len(df.plan_id.unique())}')\n",
    "print(f'Number of units:\\t\\t{len(df.unit_id.unique())}')\n",
    "print(f'Number of areas:\\t\\t{len(df.area_id.unique())}')\n",
    "print(f'Number of geometries:\\t{df.geom.count()}')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:37:24.512746Z",
     "end_time": "2024-11-29T10:37:25.667062Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We are most interested in the nesting / tree of (sub)types: I.e., which subtypes are children of which types (parents)."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: feature\n",
      "-- Subtypes:\n",
      "\t['sink', 'washing_machine', 'built_in_furniture', 'kitchen', 'shower', 'bathtub', 'stairs', 'ramp', 'toilet', 'elevator']\n",
      "\n",
      "Type: opening\n",
      "-- Subtypes:\n",
      "\t['door', 'entrance_door', 'window']\n",
      "\n",
      "Type: area\n",
      "-- Subtypes:\n",
      "\t['medical_room', 'loggia', 'shelter', 'elevator_facilities', 'wash_and_dry_room', 'oil_tank', 'salesroom', 'bike_storage', 'air', 'room', 'balcony', 'office_space', 'community_room', 'patio', 'vehicle_traffic_area', 'open_plan_office', 'bathroom', 'corridors_and_halls', 'radation_therapy', 'kitchen_dining', 'pram_and_bike_storage_room', 'transport_shaft', 'cloakroom', 'common_kitchen', 'carpark', 'cold_storage', 'basement_compartment', 'lightwell', 'water_supply', 'foyer', 'arcade', 'bedroom', 'dedicated_medical_room', 'break_room', 'not_defined', 'meeting_room', 'archive', 'heating', 'living_dining', 'gas', 'reception_room', 'corridor', 'dining', 'staircase', 'teaching_room', 'electrical_supply', 'outdoor_void', 'counter_room', 'lobby', 'office', 'sanitary_rooms', 'workshop', 'sports_rooms', 'void', 'kitchen', 'showroom', 'wintergarten', 'canteen', 'operations_facilities', 'pram', 'physio_and_rehabilitation', 'technical_area', 'factory_room', 'garage', 'shaft', 'storeroom', 'basement', 'office_tech_room', 'elevator', 'garden', 'warehouse', 'waiting_room', 'living_room', 'terrace', 'studio', 'house_technics_facilities', 'logistics']\n",
      "\n",
      "Type: separator\n",
      "-- Subtypes:\n",
      "\t['railing', 'column', 'wall']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for entity_type in entity_types:\n",
    "    print(f'Type: {entity_type}\\n'\n",
    "          f'-- Subtypes:\\n\\t{[cat.lower() for cat in set(df[df[\"entity_type\"]==entity_type].entity_subtype)]}\\n')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:37:25.881603Z",
     "end_time": "2024-11-29T10:37:27.195689Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Feature removal\n",
    "\n",
    "This step is one of the most delicate steps of the process. It comes from the fact that there are two equivalently named *subtypes* that have a different *type* category. For the 'feature' and 'area' types, subtypes can be both 'KITCHEN' and 'ELEVATOR'. We first remove these two categories from the 'feature' type."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [],
   "source": [
    "df = df[~(df['entity_type'] == 'feature')]\n",
    "df = df[~(df['entity_subtype'] == 'RAILING')]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:37:27.391967Z",
     "end_time": "2024-11-29T10:37:28.459064Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Residential-only filtering\n",
    "\n",
    "We remove floors that have at least one of the following subtypes:\n",
    "\n",
    "```python\n",
    "{'OFFICE', 'WASH_AND_DRY_ROOM', 'VEHICLE_TRAFFIC_AREA', 'BASEMENT', 'WAITING_ROOM',\n",
    "'MEDICAL_ROOM', 'CANTEEN', 'SALESROOM', 'COLD_STORAGE', 'WORKSHOP', 'OUTDOOR_VOID',\n",
    "'TECHNICAL_AREA', 'DEDICATED_MEDICAL_ROOM', 'WAREHOUSE', 'GARDEN', 'PRAM_AND_BIKE_STORAGE_ROOM',\n",
    "'OFFICE_SPACE', 'SHOWROOM', 'AIR', 'PRAM', 'SANITARY_ROOMS', 'LOBBY', 'WATER_SUPPLY', 'HEATING',\n",
    " 'LOGGIA', 'OIL_TANK', 'GAS', 'PATIO', 'FOYER', 'OPEN_PLAN_OFFICE', 'SHELTER', 'SPORTS_ROOMS',\n",
    " 'HOUSE_TECHNICS_FACILITIES', 'TEACHING_ROOM', 'OFFICE_TECH_ROOM', 'WINTERGARTEN', 'CLOAKROOM',\n",
    " 'RECEPTION_ROOM', 'CARPARK', 'STUDIO', 'NOT_DEFINED', 'COMMUNITY_ROOM',\n",
    " 'PHYSIO_AND_REHABILITATION', 'FACTORY_ROOM', 'ARCHIVE', 'OPERATIONS_FACILITIES', 'ARCADE',\n",
    " 'LOGISTICS', 'TRANSPORT_SHAFT', 'GARAGE', 'COUNTER_ROOM', 'BREAK_ROOM', 'ELECTRICAL_SUPPLY',\n",
    " 'ELEVATOR_FACILITIES', 'MEETING_ROOM', 'COMMON_KITCHEN', 'BASEMENT_COMPARTMENT', 'LIGHTWELL',\n",
    " 'BIKE_STORAGE', 'RADATION_THERAPY'}\n",
    "```"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of deleted floors:\t2305\n",
      "Number of remaining floors:\t11600\n"
     ]
    }
   ],
   "source": [
    "from constants import ZONING_MAPPING\n",
    "\n",
    "# Entity subtypes for which the floors should be completely removed\n",
    "subtypes_to_remove = set(df.entity_subtype) - set(list(ZONING_MAPPING.keys()))\n",
    "\n",
    "# Find floor ids of non- or mixed-residential\n",
    "mixed_floor_ids = set(df[df['entity_subtype'].isin(set(subtypes_to_remove))].floor_id)\n",
    "res_floor_ids = set(df.floor_id) - mixed_floor_ids  # The complement of the other\n",
    "\n",
    "# Filter on residential-only floor ids\n",
    "df = df[df['floor_id'].isin(res_floor_ids)].reset_index(drop=True)\n",
    "\n",
    "# If you run it again; there won't, obviously, be any floor plans to-be-removed !\n",
    "print(f'Number of deleted floors:\\t{len(mixed_floor_ids)}')\n",
    "print(f'Number of remaining floors:\\t{len(df.floor_id.unique())}')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:37:30.554946Z",
     "end_time": "2024-11-29T10:37:31.739175Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Near-duplicate removal\n",
    "\n",
    "We remove what we would like to call *near* duplicates.\n",
    "It is important to not have too similar instances in the machine learning dataset because two (or more) of the same signals and annotations will never help (and will only hinder) training and might even cause imbalanced model behavior (usually per epoch the whole dataset is seen once; hence if there are duplicates the model will better learn on those instances) and unfair evaluation."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of remaining floors: 7205\n"
     ]
    }
   ],
   "source": [
    "one_per_plan_floor_ids = df.sort_values('elevation').groupby('plan_id').head(1)['floor_id'].values\n",
    "df = df[df['floor_id'].isin(one_per_plan_floor_ids)]\n",
    "\n",
    "# after plan id selection\n",
    "print(f'Number of remaining floors: {df.floor_id.nunique()}')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:40:25.886981Z",
     "end_time": "2024-11-29T10:40:26.634797Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Medium- to large-scale filtering\n",
    "\n",
    "Floor plans are removed that contain fewer than 15 areas. In addition, every floor plan should have at least two \"Zone 2\"-categorized areas.\n",
    "This led to the removal of 1,541 (11.1\\%) floor plans.\n",
    "\n",
    "We first add two extra columns to the dataframe: One indicating the *zoning type* of the room; the other the *room type*.\n",
    "See `constants.py` for the zoning and room type categories and mappings between them and the (original) subtype categories."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "outputs": [],
   "source": [
    "from constants import ROOM_MAPPING\n",
    "\n",
    "df['zoning'] = df['entity_subtype'].map(ZONING_MAPPING)\n",
    "df['roomtype'] = df['entity_subtype'].map(ROOM_MAPPING)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:46:21.246990Z",
     "end_time": "2024-11-29T10:46:21.476207Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now, we find all the floor identities (IDs) that consist of a minimum of 15 rooms and have at least 2 rooms of type `Zone 2`:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "outputs": [
    {
     "data": {
      "text/plain": "  0%|          | 0/7205 [00:00<?, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "bddb7c49169c4db097cfcaa136c2fa5b"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tqdm.auto import tqdm\n",
    "\n",
    "floor_ids = df.floor_id.unique()\n",
    "\n",
    "min_nr_areas = 15\n",
    "min_nr_zone2 = 2\n",
    "\n",
    "# Initialize empty list\n",
    "floor_ids_at_scale = []\n",
    "\n",
    "# Takes a couple of seconds\n",
    "for floor_id in tqdm(floor_ids):\n",
    "\n",
    "    df_floor = df[df.floor_id == floor_id].reset_index(drop=True)\n",
    "    df_floor = df_floor[df_floor.zoning.isin(['Zone1', 'Zone2', 'Zone3', 'Zone4'])]\n",
    "    df_zone2 = df_floor[df_floor.zoning.isin(['Zone2'])]\n",
    "\n",
    "    # Compute number of rooms and those with type Zone 2\n",
    "    nr_areas = len(df_floor.zoning.values)\n",
    "    nr_zone2 = len(df_zone2.zoning.values)\n",
    "\n",
    "    # Filter on minimum number of areas and minimum number of Zone 2\n",
    "    if nr_areas >= min_nr_areas and nr_zone2 >= min_nr_zone2:\n",
    "        floor_ids_at_scale.append(floor_id)\n",
    "    else:\n",
    "        continue"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:49:28.567949Z",
     "end_time": "2024-11-29T10:49:47.526620Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "And filter the dataframe based on these IDs:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# floors: 5664\n"
     ]
    }
   ],
   "source": [
    "# remove small complexes\n",
    "df = df[df['floor_id'].isin(floor_ids_at_scale)].reset_index(drop=True)\n",
    "\n",
    "# after removal of selection\n",
    "print(f'Final number of floors: {df.floor_id.nunique()}')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:50:48.221186Z",
     "end_time": "2024-11-29T10:50:48.586012Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Saving the dataframe\n",
    "\n",
    "You can now save the cleaned and filtered dataframe anywhere that you like.\n",
    "In our GitHub, you can find it back under the folder `data`, saved as `f\"MSD 5.664k (V1).csv\"`"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "outputs": [],
   "source": [
    "df.to_csv(f'data/MSD {len(set(df.floor_id))/1000:.3f}k (V1).csv')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-11-29T10:56:08.663859Z",
     "end_time": "2024-11-29T10:56:24.906350Z"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}