notebooks/senzing-examples/python/senzing-G2Engine-reference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# G2Engine Reference\n",
    "\n",
    "More information:\n",
    "\n",
    "1. [GitHub repository](https://github.com/Senzing/docker-jupyter)\n",
    "1. [Senzing documentation](http://docs.senzing.com/?python#g2config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents\n",
    "\n",
    "1. [Prepare environment](#Prepare-environment)\n",
    "    1. [Initialize Senzing configuration](#Initialize-Senzing-configuration)\n",
    "    1. [Initialize python environment](#Initialize-python-environment)\n",
    "    1. [Helper class for JSON rendering](#Helper-class-for-JSON-rendering)\n",
    "    1. [System path](#System-path)\n",
    "    1. [Initialize variables](#Initialize-variables)\n",
    "1. [G2Engine](#G2Engine)\n",
    "    1. [G2Engine initialization](#G2Engine-initialization)\n",
    "    1. [initWithConfigIDV2](#initWithConfigIDV2)\n",
    "    1. [reinitV2](#reinitV2)\n",
    "    1. [primeEngine](#primeEngine)\n",
    "    1. [getActiveConfigID](#getActiveConfigID)\n",
    "    1. [exportConfig](#exportConfig)\n",
    "    1. [stats](#stats)\n",
    "    1. [getRepositoryLastModifiedTime](#getRepositoryLastModifiedTime)\n",
    "1. [Insert](#Insert)\n",
    "    1. [Insert parameters](#Insert-parameters)\n",
    "    1. [addRecord](#addRecord)\n",
    "    1. [addRecordWithReturnedRecordID](#addRecordWithReturnedRecordID)\n",
    "    1. [addRecordWithInfo](#addRecordWithInfo)\n",
    "1. [Search](#Search)\n",
    "    1. [Record search](#Record-search)\n",
    "        1. [getRecordV2](#getRecordV2)\n",
    "    1. [Entity Search](#Entity-Search)\n",
    "        1. [getEntityByRecordIDV2](#getEntityByRecordIDV2)\n",
    "        1. [getEntityByEntityIDV2](#getEntityByEntityIDV2)\n",
    "    1. [Search By Attributes](#Search-By-Attributes)\n",
    "        1. [searchByAttributes](#searchByAttributes)\n",
    "        1. [searchByAttributesV2](#searchByAttributesV2)\n",
    "    1. [Finding Paths](#Finding-Paths)\n",
    "        1. [findPathByEntityID](#findPathByEntityID)\n",
    "        1. [findPathByEntityIDV2](#findPathByEntityIDV2)\n",
    "        1. [findPathByRecordID](#findPathByRecordID)\n",
    "        1. [findPathByRecordIDV2](#findPathByRecordIDV2)\n",
    "    1. [Finding Paths with Exclusions](#Finding-Paths-with-Exclusions)\n",
    "        1. [findPathExcludingByEntityID](#findPathExcludingByEntityID)\n",
    "        1. [findPathExcludingByRecordID](#findPathExcludingByRecordID)\n",
    "    1. [Finding Paths with Required Sources](#Finding-Paths-with-Required-Sources)\n",
    "        1. [findPathIncludingSourceByEntityID](#findPathIncludingSourceByEntityID)\n",
    "        1. [findPathIncludingSourceByRecordID](#findPathIncludingSourceByRecordID)\n",
    "    1. [Finding Networks](#Finding-Networks)\n",
    "        1. [findNetworkByEntityID](#findNetworkByEntityID)\n",
    "        1. [findNetworkByEntityIDV2](#findNetworkByEntityIDV2)\n",
    "        1. [findNetworkByRecordID](#findNetworkByRecordID)\n",
    "        1. [findNetworkByRecordIDV2](#findNetworkByRecordIDV2)\n",
    "1. [Connection Details](#Connection-details)\n",
    "    1. [whyEntityByRecordID](#whyEntityByRecordID)\n",
    "    1. [whyEntityByRecordIDV2](#whyEntityByRecordIDV2)\n",
    "    1. [whyEntityByEntityID](#whyEntityByEntityID)\n",
    "    1. [whyEntityByEntityIDV2](#whyEntityByEntityIDV2)\n",
    "1. [Replace](#Replace)\n",
    "    1. [replaceRecord](#replaceRecord)\n",
    "    1. [replaceRecordWithInfo](#replaceRecordWithInfo)\n",
    "1. [Re-evaluate](#Re-evaluate)\n",
    "    1. [reevaluateRecord](#reevaluateRecord)\n",
    "    1. [reevaluateRecordWithInfo](#reevaluateRecordWithInfo)\n",
    "    1. [reevaluateEntity](#reevaluateEntity)\n",
    "    1. [reevaluateEntityWithInfo](#reevaluateEntityWithInfo)\n",
    "1. [Reporting](#Reporting)\n",
    "    1. [exportJSONEntityReport](#exportJSONEntityReport)\n",
    "    1. [fetchNext](#fetchNext)\n",
    "    1. [closeExport](#closeExport)\n",
    "    1. [exportCSVEntityReport](#exportCSVEntityReport)\n",
    "1. [Redo Processing](#Redo-Processing)\n",
    "    1. [countRedoRecords](#countRedoRecords)\n",
    "    1. [getRedoRecord](#getRedoRecord)\n",
    "    1. [process](#process)\n",
    "    1. [processWithInfo](#processWithInfo)\n",
    "    1. [processRedoRecord](#processRedoRecord)\n",
    "    1. [processRedoRecordWithInfo](#processRedoRecordWithInfo)\n",
    "1. [Delete](#Delete)\n",
    "    1. [deleteRecord](#deleteRecord)\n",
    "    1. [deleteRecordWithInfo](#deleteRecordWithInfo)\n",
    "1. [Cleanup](#Cleanup)\n",
    "    1. [purgeRepository](#purgeRepository)\n",
    "    1. [destroy](#destroy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Senzing configuration\n",
    "\n",
    "Run [senzing-G2ConfigMgr-reference.ipynb](senzing-G2ConfigMgr-reference.ipynb)\n",
    "to install a Senzing Engine configuration in the database."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize python environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import json\n",
    "\n",
    "# For RenderJSON\n",
    "\n",
    "import uuid\n",
    "from IPython.display import display_javascript, display_html, display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper class for JSON rendering\n",
    "\n",
    "A class for pretty-printing JSON.\n",
    "Not required by Senzing, \n",
    "but helps visualize JSON."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RenderJSON(object):\n",
    "    def __init__(self, json_data):\n",
    "        if isinstance(json_data, dict):\n",
    "            self.json_str = json.dumps(json_data)\n",
    "        elif isinstance(json_data, bytearray):\n",
    "            self.json_str = json_data.decode()\n",
    "        else:\n",
    "            self.json_str = json_data\n",
    "        self.uuid = str(uuid.uuid4())\n",
    "\n",
    "    def _ipython_display_(self):\n",
    "        display_html('<div id=\"{}\" style=\"height:100%; width:100%; background-color: LightCyan\"></div>'.format(self.uuid), raw=True)\n",
    "        display_javascript(\"\"\"\n",
    "        require([\"https://rawgit.com/caldwell/renderjson/master/renderjson.js\"], function() {\n",
    "        document.getElementById('%s').appendChild(renderjson(%s))\n",
    "        });\n",
    "        \"\"\" % (self.uuid, self.json_str), raw=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### System path\n",
    "\n",
    "Update system path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python_path = \"{0}/python\".format(\n",
    "    os.environ.get(\"SENZING_G2_DIR\", \"/opt/senzing/g2\"))\n",
    "sys.path.append(python_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize variables\n",
    "\n",
    "Create variables used for G2Engine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "module_name = 'pyG2EngineForAddRecord'\n",
    "\n",
    "config_path = os.environ.get(\"SENZING_ETC_DIR\", \"/etc/opt/senzing\")\n",
    "support_path = os.environ.get(\"SENZING_DATA_VERSION_DIR\", \"/opt/senzing/data\")\n",
    "\n",
    "resource_path = \"{0}/resources\".format(\n",
    "    os.environ.get(\"SENZING_G2_DIR\", \"/opt/senzing/g2\"))\n",
    "\n",
    "sql_connection = os.environ.get(\n",
    "    \"SENZING_SQL_CONNECTION\", \"sqlite3://na:na@/var/opt/senzing/sqlite/G2C.db\")\n",
    "\n",
    "verbose_logging = False\n",
    "\n",
    "senzing_config_dictionary = {\n",
    "    \"PIPELINE\": {\n",
    "        \"CONFIGPATH\": config_path,        \n",
    "        \"SUPPORTPATH\": support_path,\n",
    "        \"RESOURCEPATH\": resource_path\n",
    "    },\n",
    "    \"SQL\": {\n",
    "        \"CONNECTION\": sql_connection,\n",
    "    }\n",
    "}\n",
    "\n",
    "senzing_config_json = json.dumps(senzing_config_dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## G2Engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from G2Engine import G2Engine\n",
    "import G2Exception"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### G2Engine initialization\n",
    "\n",
    "To start using Senzing G2Engine, create and initialize an instance.\n",
    "This should be done once per process.\n",
    "The `initV2()` method accepts the following parameters:\n",
    "\n",
    "- **module_name:** A short name given to this instance of the G2Engine\n",
    "  object.\n",
    "- **senzing_config_json:** A JSON string containing configuration parameters.\n",
    "- **verbose_logging:** A boolean which enables diagnostic logging.\n",
    "- **config_id:** (optional) The identifier value for the engine configuration\n",
    "  can be returned here.\n",
    "\n",
    "Calling this function will return \"0\" upon success."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "g2_engine = G2Engine()\n",
    "\n",
    "return_code = g2_engine.initV2(\n",
    "    module_name,\n",
    "    senzing_config_json,\n",
    "    verbose_logging)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### initWithConfigIDV2\n",
    "\n",
    "Alternatively `initWithConfigIDV2()` can be used to specify a configuration.\n",
    "For more information, see\n",
    "[http://docs.senzing.com/?python](http://docs.senzing.com/?python#engine)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### reinitV2\n",
    "\n",
    "The `reinitV2()` function may be used to reinitialize the engine\n",
    "using a specified initConfigID. See\n",
    "[http://docs.senzing.com/?python](http://docs.senzing.com/?python#engine)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### primeEngine\n",
    "\n",
    "The `primeEngine()` method may optionally be called to pre-initialize\n",
    "some of the heavier weight internal resources of the G2 engine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "return_code = g2_engine.primeEngine()\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### getActiveConfigID\n",
    "\n",
    "Call `getActiveConfigID()` to return an identifier for the loaded\n",
    "Senzing engine configuration.\n",
    "The call will assign a long integer to a user-designated variable\n",
    "-- the function itself will return \"0\" upon success.\n",
    "The `getActiveConfigID()` method accepts one parameter as input:\n",
    "\n",
    "- **configuration_id:** The identifier value for the engine configuration.\n",
    "  The result of function call is returned here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "configuration_id = bytearray()\n",
    "return_code = g2_engine.getActiveConfigID(configuration_id)\n",
    "\n",
    "print(\"Return code: {0}\\nConfiguration id: {1}\".format(\n",
    "    return_code,\n",
    "    configuration_id.decode()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### exportConfig\n",
    "\n",
    "Call `exportConfig()` to retrieve your Senzing engine's configuration.\n",
    "The call will assign a JSON document to a user-designated buffer,\n",
    "containing all relevant configuration information\n",
    "-- the function itself will return \"0\" upon success.\n",
    "The exportConfig function accepts the following parameters as input:\n",
    "\n",
    "- **response_bytearray:** The memory buffer to retrieve the JSON\n",
    "  configuration document\n",
    "- **config_id_bytearray:** The identifier value for the engine configuration\n",
    "  can be returned here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "config_id_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.exportConfig(response_bytearray, config_id_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\\nConfiguration ID: {1}\".format(\n",
    "    return_code,\n",
    "    config_id_bytearray.decode()))\n",
    "\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### stats\n",
    "\n",
    "Call `stats()` to retrieve workload statistics for the current process.\n",
    "These statistics will automatically reset after retrieval.\n",
    "\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document.  If an error occurred, an error response is stored here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.stats(response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### getRepositoryLastModifiedTime\n",
    "\n",
    "Call `getRepositoryLastModifiedTime()` to obtain the last modified time of\n",
    "the Senzing repository,measured in the number of seconds between the last\n",
    "modified time and January 1, 1970 12:00am GMT (epoch time).\n",
    "The call will assign a long integer to a user-designated buffer\n",
    "-- the function itself will return \"0\" upon success.\n",
    "The getRepositoryLastModifiedTime() method accepts one parameter as input:\n",
    "\n",
    "- **last_modified_unixtime:** The last modified time. The result of function\n",
    "  call is returned here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_modified_timestamp = bytearray()\n",
    "\n",
    "return_code = g2_engine.getRepositoryLastModifiedTime(last_modified_timestamp)\n",
    "\n",
    "# Human readable output.\n",
    "\n",
    "from datetime import datetime\n",
    "last_modified_unixtime = int(int(last_modified_timestamp.decode()) / 1000)\n",
    "last_modified_datetime = datetime.fromtimestamp(last_modified_unixtime)\n",
    "\n",
    "print(\"Return Code: {0}\\nLast modified timestamp: {1}\\nLast modified time: {2}\"\n",
    "      .format(\n",
    "          return_code,\n",
    "          last_modified_timestamp.decode(),\n",
    "          last_modified_datetime))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Insert parameters\n",
    "\n",
    "The following variables are used as parameters to the Senzing API.\n",
    "Documentation for `g2_engine_flags` values is at\n",
    "[http://docs.senzing.com/?python](http://docs.senzing.com/?python#engine-control-flags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasource_code_1 = \"TEST\"\n",
    "record_id_1 = \"1\"\n",
    "datasource_code_2 = \"TEST\"\n",
    "record_id_2 = \"2\"\n",
    "datasource_code_3 = \"TEST\"\n",
    "record_id_3 = \"3\"\n",
    "datasource_code_4 = \"TEST\"\n",
    "record_id_4 = \"4\"\n",
    "datasource_code_5 = \"TEST\"\n",
    "record_id_5 = \"5\"\n",
    "datasource_code_6 = \"TEST\"\n",
    "record_id_6 = \"6\"\n",
    "datasource_code_7 = \"TEST\"\n",
    "record_id_7 = \"7\"\n",
    "\n",
    "load_id = None\n",
    "g2_engine_flags = G2Engine.G2_EXPORT_DEFAULT_FLAGS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initial data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"NAMES\": [{\n",
    "        \"NAME_TYPE\": \"PRIMARY\",\n",
    "        \"NAME_LAST\": \"Smith\",\n",
    "        \"NAME_FIRST\": \"John\",\n",
    "        \"NAME_MIDDLE\": \"M\"\n",
    "    }],\n",
    "    \"PASSPORT_NUMBER\": \"PP11111\",\n",
    "    \"PASSPORT_COUNTRY\": \"US\",\n",
    "    \"DRIVERS_LICENSE_NUMBER\": \"DL11111\",\n",
    "    \"SSN_NUMBER\": \"111-11-1111\"\n",
    "}\n",
    "data_as_json = json.dumps(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### addRecord\n",
    "\n",
    "Once the Senzing engine is initialized, use `addRecord()` to load a record\n",
    "into the Senzing repository\n",
    "-- `addRecord()` can be called as many times as desired and from multiple\n",
    "threads at the same time.\n",
    "The `addRecord()` function returns \"0\" upon success, and accepts four\n",
    "parameters as input:\n",
    "\n",
    "- **datasource_code:** The name of the data source the record\n",
    "  is associated with.\n",
    "  This value is configurable to the system\n",
    "- **record_id:** The record ID, used to identify distinct records\n",
    "- **data_as_json:** A JSON document with the attribute data for the record\n",
    "- **load_id:** The observation load ID for the record;\n",
    "  value can be null and will default to data_source\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "return_code = g2_engine.addRecord(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    data_as_json,\n",
    "    load_id)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### addRecordWithReturnedRecordID\n",
    "\n",
    "Alternatively `addRecordWithReturnedRecordID()` can be used to add a record.\n",
    "For more information, see\n",
    "[http://docs.senzing.com/?python](http://docs.senzing.com/?python#data-manipulation)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### addRecordWithInfo\n",
    "\n",
    "Use if you would like to know what resolved entities were modified when\n",
    "adding the new record.\n",
    "It behaves identically to addRecord(),\n",
    "but returns a json document containing the IDs of the affected entities.\n",
    "It accepts the following parameters:\n",
    "\n",
    "- **datasource_code:** The name of the data source the record is associated\n",
    "  with.  This value is configurable to the system.\n",
    "- **record_id:** The record ID, used to identify distinct records\n",
    "- **data_as_json:** A JSON document with the attribute data for the record\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document;  if an error occurred, an error response is stored here\n",
    "- **load_id:** The observation load ID for the record;\n",
    "  value can be null and will default to data_source\n",
    "- **g2_engine_flags:** Control flags for specifying what data about the\n",
    "  entity to retrieve\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.addRecordWithInfo(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    data_as_json,\n",
    "    response_bytearray,\n",
    "    load_id,\n",
    "    g2_engine_flags)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Record search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### getRecordV2\n",
    "\n",
    "Use `getRecordV2()` to retrieve a single record from the data repository;\n",
    "the record is assigned in JSON form to a user-designated buffer,\n",
    "and the function itself returns \"0\" upon success.\n",
    "Once the Senzing engine is initialized,\n",
    "`getRecordV2()` can be called as many times as desired and from multiple\n",
    "threads at the same time.\n",
    "The `getRecordV2()` function accepts the following parameters as input:\n",
    "\n",
    "- **datasource_code:** The name of the data source the record is associated\n",
    "  with.  This value is configurable to the system.\n",
    "- **record_id:** The record ID, used to identify the record for retrieval\n",
    "- **g2_engine_flags:** Control flags for specifying what data about the\n",
    "  record to retrieve.\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document; if an error occurred, an error response is stored here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getRecordV2(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `getRecordV2()` is an improved version of `getRecord()`\n",
    "that also allows you to use control flags.\n",
    "The `getRecord()` function has been deprecated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entity Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### getEntityByRecordIDV2\n",
    "\n",
    "Entity searching is a key component for interactive use of Entity Resolution\n",
    "intelligence.\n",
    "The core Senzing engine provides real-time search capabilities that are\n",
    "easily accessed via the Senzing API.\n",
    "Senzing offers methods for entity searching, all of which can be called\n",
    "as many times as desired and from multiple threads at the same time\n",
    "(and all of which return \"0\" upon success).\n",
    "\n",
    "Use `getEntityByRecordIDV2()` to retrieve entity data based on the ID of a\n",
    "resolved identity.\n",
    "This function accepts the following parameters as input:\n",
    "\n",
    "- **datasource_code:** The name of the data source the record is associated\n",
    "  with.  This value is configurable to the system.\n",
    "- **record_id:** The numeric ID of a resolved entity\n",
    "- **g2_engine_flags:** Control flags for specifying what data about the\n",
    "  entity to retrieve.\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document; if an error occurred, an error response is stored here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByRecordIDV2(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "response_dictionary = json.loads(response_bytearray)\n",
    "entity_id_1 = response_dictionary[\"RESOLVED_ENTITY\"][\"ENTITY_ID\"]\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### getEntityByEntityIDV2\n",
    "\n",
    "Entity searching is a key component for interactive use of Entity Resolution\n",
    "intelligence.\n",
    "The core Senzing engine provides real-time search capabilities that are\n",
    "easily accessed via the Senzing API.\n",
    "Senzing offers methods for entity searching, all of which can be called as\n",
    "many times\n",
    "as desired and from multiple threads at the same time (and all of which\n",
    "return \"0\" upon success).\n",
    "\n",
    "Use `getEntityByEntityIDV2()` to retrieve entity data based on the ID of a\n",
    "resolved identity.\n",
    "This function accepts the following parameters as input:\n",
    "\n",
    "- **entity_id:** The numeric ID of a resolved entity\n",
    "- **g2_engine_flags:** Control flags for specifying what data about the\n",
    "  entity to retrieve.\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document; if an error occurred, an error response is stored here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByEntityIDV2(\n",
    "    entity_id_1,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search By Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### searchByAttributes\n",
    "\n",
    "Entity searching is a key component for interactive use of Entity Resolution\n",
    "intelligence.\n",
    "The core Senzing engine provides real-time search capabilities that are\n",
    "easily accessed via the Senzing API.\n",
    "Senzing offers a method for entity searching by attributes,\n",
    "which can be called as many times as desired and from multiple threads at the\n",
    "same time\n",
    "(and all of which return \"0\" upon success).\n",
    "\n",
    "Use `searchByAttributes()` to retrieve entity data based on a user-specified\n",
    "set of entity attributes.\n",
    "This function accepts the following parameters as input:\n",
    "\n",
    "- **data_as_json:** A JSON document with the attribute data to search for.\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document; if an error occurred, an error response is stored here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "return_code = g2_engine.searchByAttributes(data_as_json, response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### searchByAttributesV2\n",
    "\n",
    "This function is similar but preferable to the searchByAttributes() function.\n",
    "This function has improved functionality and a better standardized output\n",
    "structure.\n",
    "\n",
    "Use `searchByAttributesV2()` to retrieve entity data based on\n",
    "a user-specified set of entity attributes.\n",
    "This function accepts the following parameters as input:\n",
    "\n",
    "- **data_as_json:** A JSON document with the attribute data to search for.\n",
    "- **g2_engine_flags:** Operational flags\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document; if an error occurred, an error response is stored here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.searchByAttributesV2(\n",
    "    data_as_json,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding Paths\n",
    "\n",
    "The `findPathByEntityID()` and `findPathByRecordID()` functions\n",
    "can be used to find single relationship paths between two entities.\n",
    "Paths are found using known relationships with other entities.\n",
    "\n",
    "Entities can be searched for by either Entity ID or by Record ID,\n",
    "depending on which function is chosen.\n",
    "\n",
    "These functions have the following parameters:\n",
    "\n",
    "- **entity_id_2:** The entity ID for the starting entity of the search path\n",
    "- **entity_id_3:** The entity ID for the ending entity of the search path\n",
    "- **datasource_code_2:** The data source for the starting entity of the\n",
    "  search path\n",
    "- **datasource_code_3:** The data source for the ending entity of the search\n",
    "  path\n",
    "- **record_id_2:** The record ID for the starting entity of the search path\n",
    "- **record_id_3:** The record ID for the ending entity of the search path\n",
    "- **max_degree:** The number of relationship degrees to search\n",
    "\n",
    "The functions return a JSON document that identifies the path between the\n",
    "entities,\n",
    "and the information on the entities in question.\n",
    "The document contains a section called \"ENTITY_PATHS\" which gives\n",
    "the path from one entity to the other.\n",
    "Example:\n",
    "\n",
    "```JSON\n",
    "{ \n",
    "  \"START_ENTITY_ID\": 10, \n",
    "  \"END_ENTITY_ID\": 13,\n",
    "  \"ENTITIES\": [10, 11, 12, 13]\n",
    "}\n",
    "```\n",
    "\n",
    "If no path was found, then the value of ENTITIES will be an empty list.\n",
    "\n",
    "The response document also contains a separate ENTITIES section,\n",
    "with the full information about the resolved entities along that path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First you will need to create some records so that you have some that you can\n",
    "compare.\n",
    "Can you see what is the same between this record and the previous one?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"NAMES\": [{\n",
    "        \"NAME_TYPE\": \"PRIMARY\",\n",
    "        \"NAME_LAST\": \"Miller\",\n",
    "        \"NAME_FIRST\": \"Max\",\n",
    "        \"NAME_MIDDLE\": \"W\"\n",
    "    }],\n",
    "    \"SSN_NUMBER\": \"111-11-1111\"\n",
    "}\n",
    "data_as_json = json.dumps(data)\n",
    "\n",
    "return_code = g2_engine.replaceRecord(\n",
    "    datasource_code_2,\n",
    "    record_id_2,\n",
    "    data_as_json,\n",
    "    None)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace values for Record #3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"NAMES\": [{\n",
    "        \"NAME_TYPE\": \"PRIMARY\",\n",
    "        \"NAME_LAST\": \"Miller\",\n",
    "        \"NAME_FIRST\": \"Mildred\"\n",
    "    }],\n",
    "    \"SSN_NUMBER\": \"111-11-1111\"\n",
    "}\n",
    "data_as_json = json.dumps(data)\n",
    "\n",
    "return_code = g2_engine.replaceRecord(\n",
    "    datasource_code_3,\n",
    "    record_id_3,\n",
    "    data_as_json,\n",
    "    None)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Locate \"entity identifier\" for Record #1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByRecordID(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    response_bytearray)\n",
    "\n",
    "response_dictionary = json.loads(response_bytearray)\n",
    "entity_id_1 = response_dictionary[\"RESOLVED_ENTITY\"][\"ENTITY_ID\"]\n",
    "\n",
    "print(\"Return Code: {0}\\nEntity ID: {1}\".format(return_code, entity_id_1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Locate \"entity identifier\" for Record #2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByRecordID(\n",
    "    datasource_code_2,\n",
    "    record_id_2,\n",
    "    response_bytearray)\n",
    "\n",
    "response_dictionary = json.loads(response_bytearray)\n",
    "entity_id_2 = response_dictionary[\"RESOLVED_ENTITY\"][\"ENTITY_ID\"]\n",
    "\n",
    "print(\"Return Code: {0}\\nEntity ID: {1}\".format(return_code, entity_id_2))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Locate \"entity identifier\" for Record #3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByRecordID(\n",
    "    datasource_code_3,\n",
    "    record_id_3,\n",
    "    response_bytearray)\n",
    "\n",
    "response_dictionary = json.loads(response_bytearray)\n",
    "entity_id_3 = response_dictionary[\"RESOLVED_ENTITY\"][\"ENTITY_ID\"]\n",
    "\n",
    "print(\"Return Code: {0}\\nEntity ID: {1}\".format(return_code, entity_id_3))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findPathByEntityID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "max_degree = 3\n",
    "\n",
    "# Find the path by entity ID.\n",
    "\n",
    "response_bytearray = bytearray([])\n",
    "\n",
    "return_code = g2_engine.findPathByEntityID(\n",
    "    entity_id_2,\n",
    "    entity_id_3,\n",
    "    max_degree,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findPathByEntityIDV2\n",
    "The function `findPathByEntityIDV2()` is an improved version of \n",
    "`findPathByEntityID()` that also allow you to use control flags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "max_degree = 3\n",
    "\n",
    "# Find the path by entity ID.\n",
    "\n",
    "response_bytearray = bytearray([])\n",
    "\n",
    "return_code = g2_engine.findPathByEntityIDV2(\n",
    "    entity_id_2,\n",
    "    entity_id_3,\n",
    "    max_degree,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findPathByRecordID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "max_degree = 3\n",
    "\n",
    "# Find the path by record ID.\n",
    "\n",
    "response_bytearray = bytearray([])\n",
    "\n",
    "return_code = g2_engine.findPathByRecordID(\n",
    "    datasource_code_2, record_id_2,\n",
    "    datasource_code_3, record_id_3,\n",
    "    max_degree,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findPathByRecordIDV2\n",
    "\n",
    "The function `findPathByRecordIDV2()` is an improved version of\n",
    "`findPathByRecordID()` that also allow you to use control flags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "max_degree = 3\n",
    "\n",
    "# Find the path by record ID.\n",
    "\n",
    "response_bytearray = bytearray([])\n",
    "\n",
    "return_code = g2_engine.findPathByRecordIDV2(\n",
    "    datasource_code_2, record_id_2,\n",
    "    datasource_code_3, record_id_3,\n",
    "    max_degree,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding Paths with Exclusions\n",
    "\n",
    "The `findPathExcludingByEntityID()` and `findPathExcludingByRecordID()`\n",
    "functions can be used to find single relationship paths between two\n",
    "entities.\n",
    "Paths are found using known relationships with other entities.\n",
    "In addition, it will find paths that exclude certain entities from being on\n",
    "the path.\n",
    "\n",
    "Entities can be searched for by either Entity ID or by Record ID,\n",
    "depending on which function is chosen.\n",
    "Additionally, entities to be excluded can also be specified by either Entity\n",
    "ID or by Record ID.\n",
    "\n",
    "When excluding entities, the user may choose to either (a) strictly exclude\n",
    "the entities,\n",
    "or (b) prefer to exclude the entities, but still include them if no other\n",
    "path is found.\n",
    "By default, entities will be strictly excluded.\n",
    "A \"preferred exclude\" may be done by specifying the\n",
    "`G2_FIND_PATH_PREFER_EXCLUDE` control flag.\n",
    "\n",
    "These functions have the following parameters:\n",
    "\n",
    "- **entity_id_2:** The entity ID for the starting entity of the search path\n",
    "- **entity_id_3:** The entity ID for the ending entity of the search path\n",
    "- **datasource_code_2:** The data source for the starting entity of the\n",
    "  search path\n",
    "- **datasource_code_3:** The data source for the ending entity of the search\n",
    "  path\n",
    "- **record_id_2:** The record ID for the starting entity of the search path\n",
    "- **record_id_3:** The record ID for the ending entity of the search path\n",
    "- **max_degree:** The number of relationship degrees to search\n",
    "- **excluded_entities_as_json:** Entities that should be avoided on the path\n",
    "  (JSON document)\n",
    "- **g2_engine_flags:** Operational flags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findPathExcludingByEntityID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "max_degree = 4\n",
    "excluded_entities = {\n",
    "    \"ENTITIES\": [{\n",
    "        \"ENTITY_ID\": entity_id_1\n",
    "    }]}\n",
    "excluded_entities_as_json = json.dumps(excluded_entities)\n",
    "\n",
    "# Find the path by entity ID.\n",
    "\n",
    "response_bytearray = bytearray([])\n",
    "\n",
    "return_code = g2_engine.findPathExcludingByEntityID(\n",
    "    entity_id_2,\n",
    "    entity_id_3,\n",
    "    max_degree,\n",
    "    excluded_entities_as_json,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findPathExcludingByRecordID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "excluded_records = {\n",
    "    \"RECORDS\": [{\n",
    "        \"RECORD_ID\": record_id_1,\n",
    "        \"DATA_SOURCE\": datasource_code_1\n",
    "    }]}\n",
    "excluded_records_as_json = json.dumps(excluded_records)\n",
    "\n",
    "# Find the path by record ID.\n",
    "\n",
    "response_bytearray = bytearray([])\n",
    "\n",
    "return_code = g2_engine.findPathExcludingByRecordID(\n",
    "    datasource_code_2, record_id_2,\n",
    "    datasource_code_3, record_id_3,\n",
    "    max_degree,\n",
    "    excluded_records_as_json,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding Paths with Required Sources\n",
    "\n",
    "The `findPathIncludingSourceByEntityID()` and \n",
    "`findPathIncludingSourceByRecordID()` functions\n",
    "can be used to find single relationship paths between two entities.\n",
    "In addition, one of the enties along the path must include a specified data\n",
    "source.\n",
    "\n",
    "Entities can be searched for by either Entity ID or by Record ID,\n",
    "depending on which function is chosen.\n",
    "The required data source or sources are specified by a json document list.\n",
    "\n",
    "Specific entities may also be excluded, using the same methodology as the\n",
    "`findPathExcludingByEntityID()` and `findPathExcludingByRecordID()`\n",
    "functions use.\n",
    "\n",
    "These functions have the following parameters:\n",
    "\n",
    "- **entity_id_2:** The entity ID for the starting entity of the search path\n",
    "- **entity_id_3:** The entity ID for the ending entity of the search path\n",
    "- **datasource_code_2:** The data source for the starting entity of the\n",
    "  search path\n",
    "- **datasource_code_3:** The data source for the ending entity of the search\n",
    "  path\n",
    "- **record_id_2:** The record ID for the starting entity of the search\n",
    "  path\n",
    "- **record_id_3:** The record ID for the ending entity of the search path\n",
    "- **max_degree:** The number of relationship degrees to search\n",
    "- **excluded_entities_as_json:** Entities that should be avoided on the path\n",
    "  (JSON document)\n",
    "- **required_dsrcs_as_json:** Entities that should be avoided on the path\n",
    "  (JSON document)\n",
    "- **g2_engine_flags:** Operational flags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findPathIncludingSourceByEntityID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "max_degree = 4\n",
    "excluded_entities = {\n",
    "    \"ENTITIES\": [{\n",
    "        \"ENTITY_ID\": entity_id_1\n",
    "    }]}\n",
    "excluded_entities_as_json = json.dumps(excluded_entities)\n",
    "required_dsrcs = {\n",
    "    \"DATA_SOURCES\": [\n",
    "        datasource_code_1\n",
    "    ]}\n",
    "required_dsrcs_as_json = json.dumps(excluded_entities)\n",
    "\n",
    "# Find the path by entity ID.\n",
    "\n",
    "response_bytearray = bytearray([])\n",
    "\n",
    "return_code = g2_engine.findPathIncludingSourceByEntityID(\n",
    "    entity_id_2,\n",
    "    entity_id_3,\n",
    "    max_degree,\n",
    "    excluded_entities_as_json,\n",
    "    required_dsrcs_as_json,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findPathIncludingSourceByRecordID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "excluded_records = {\n",
    "    \"RECORDS\": [{\n",
    "        \"RECORD_ID\": record_id_1,\n",
    "        \"DATA_SOURCE\": datasource_code_1\n",
    "    }]}\n",
    "excluded_records_as_json = json.dumps(excluded_records)\n",
    "\n",
    "# Find the path by record ID.\n",
    "\n",
    "response_bytearray = bytearray([])\n",
    "\n",
    "return_code = g2_engine.findPathIncludingSourceByRecordID(\n",
    "    datasource_code_2, record_id_2,\n",
    "    datasource_code_3, record_id_3,\n",
    "    max_degree, \n",
    "    excluded_records_as_json,\n",
    "    required_dsrcs_as_json,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding Networks\n",
    "\n",
    "The `findNetworkByEntityID()` and `findNetworkByRecordID()` functions\n",
    "can be used to find all entities surrounding a requested set of entities.\n",
    "This includes the requested entities, paths between them, and relations to\n",
    "other nearby entities.\n",
    "\n",
    "Entities can be searched for by either Entity ID or by Record ID,\n",
    "depending on which function is chosen.\n",
    "\n",
    "These functions have the following parameters:\n",
    "\n",
    "- **entity_list_as_json:** A list of entities, specified by Entity ID\n",
    "  (JSON document)\n",
    "- **record_list_as_json:** A list of entities, specified by Record ID\n",
    "  (JSON document)\n",
    "- **max_degree:** The maximum number of degrees in paths between search\n",
    "  entities\n",
    "- **buildout_degree:** The number of degrees of relationships to show around\n",
    "  each search entity\n",
    "- **max_entities:** The maximum number of entities to return in the\n",
    "  discovered network\n",
    "\n",
    "They also have various arguments used to return response documents\n",
    "\n",
    "The functions return a JSON document that identifies the path between the\n",
    "each set of search entities (if the path exists), and the information on the\n",
    "entities in question (search entities, path entities, and build-out entities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findNetworkByEntityID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "entity_list = {\n",
    "    \"ENTITIES\": [{\n",
    "        \"ENTITY_ID\": entity_id_1\n",
    "    }, {\n",
    "        \"ENTITY_ID\": entity_id_2\n",
    "    }, {\n",
    "        \"ENTITY_ID\": entity_id_3\n",
    "    }]}\n",
    "entity_list_as_json = json.dumps(entity_list)\n",
    "max_degree = 2\n",
    "buildout_degree = 1\n",
    "max_entities = 12\n",
    "\n",
    "# Find the network by entity ID.\n",
    "\n",
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.findNetworkByEntityID(\n",
    "    entity_list_as_json,\n",
    "    max_degree,\n",
    "    buildout_degree,\n",
    "    max_entities,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findNetworkByEntityIDV2\n",
    "\n",
    "The function `findNetworkByEntityIDV2()` is an improved version of\n",
    "`findNetworkByEntityID()` that also allow you to use control flags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "entity_list = {\n",
    "    \"ENTITIES\": [{\n",
    "        \"ENTITY_ID\": entity_id_1\n",
    "    }, {\n",
    "        \"ENTITY_ID\": entity_id_2\n",
    "    }, {\n",
    "        \"ENTITY_ID\": entity_id_3\n",
    "    }]}\n",
    "entity_list_as_json = json.dumps(entity_list)\n",
    "max_degree = 2\n",
    "buildout_degree = 1\n",
    "max_entities = 12\n",
    "\n",
    "# Find the network by entity ID.\n",
    "\n",
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.findNetworkByEntityIDV2(\n",
    "    entity_list_as_json,\n",
    "    max_degree, \n",
    "    buildout_degree,\n",
    "    max_entities,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findNetworkByRecordID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "record_list = {\n",
    "    \"RECORDS\": [{\n",
    "        \"RECORD_ID\": record_id_1,\n",
    "        \"DATA_SOURCE\": datasource_code_1\n",
    "    }, {\n",
    "        \"RECORD_ID\": record_id_2,\n",
    "        \"DATA_SOURCE\": datasource_code_2\n",
    "    }, {\n",
    "        \"RECORD_ID\": record_id_3,\n",
    "        \"DATA_SOURCE\": datasource_code_3\n",
    "    }]}\n",
    "record_list_as_json = json.dumps(record_list)\n",
    "\n",
    "\n",
    "# Find the network by record ID.\n",
    "\n",
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.findNetworkByRecordID(\n",
    "    record_list_as_json,\n",
    "    max_degree,\n",
    "    buildout_degree,\n",
    "    max_entities,\n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### findNetworkByRecordIDV2\n",
    "\n",
    "The function `findNetworkByRecordIDV2()` is an improved version of\n",
    "`findNetworkByRecordID()` that also allow you to use control flags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define search variables.\n",
    "\n",
    "record_list = {\n",
    "    \"RECORDS\": [{\n",
    "        \"RECORD_ID\": record_id_1,\n",
    "        \"DATA_SOURCE\": datasource_code_1\n",
    "    }, {\n",
    "        \"RECORD_ID\": record_id_2,\n",
    "        \"DATA_SOURCE\": datasource_code_2\n",
    "    }, {\n",
    "        \"RECORD_ID\": record_id_3,\n",
    "        \"DATA_SOURCE\": datasource_code_3\n",
    "    }]}\n",
    "record_list_as_json = json.dumps(record_list)\n",
    "\n",
    "# Find the network by record ID.\n",
    "\n",
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.findNetworkByRecordIDV2(\n",
    "    record_list_as_json,\n",
    "    max_degree,\n",
    "    buildout_degree,\n",
    "    max_entities,\n",
    "    g2_engine_flags,    \n",
    "    response_bytearray)\n",
    "\n",
    "# Print the results.\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connection Details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `whyEntityByEntityID()` and `whyEntityByRecordID()` functions can be used\n",
    "to determine why records belong to their resolved entities.\n",
    "These functions will compare the record data within an entity against the\n",
    "rest of the entity data, and show why they are connected.\n",
    "This is calculated based on the features that record data represents.\n",
    "\n",
    "Records can be chosen by either Record ID or by Entity ID,\n",
    "depending on which function is chosen.\n",
    "If a single record ID is used,\n",
    "then comparison results for that single record will be generated, as part of\n",
    "its entity.\n",
    "If an Entity ID is used,\n",
    "then comparison results will be generated for every record within that\n",
    "entity.\n",
    "\n",
    "These functions have the following parameters:\n",
    "\n",
    "- **entity_id:** The entity ID for the entity to be analyzed\n",
    "- **datasource_code:** The data source for the record to be analyzed\n",
    "- **record_id:** The record ID for the record to be analyzed\n",
    "- **g2_engine_flags:** Control flags for outputting entities\n",
    "\n",
    "They also have various arguments used to return response documents.\n",
    "\n",
    "The functions return a JSON document that gives the results of the record\n",
    "analysis.\n",
    "The document contains a section called \"WHY_RESULTS\",\n",
    "which shows how specific records relate to the rest of the entity.\n",
    "It has a \"WHY_KEY\", which is similar to a match key, in defining the relevant\n",
    "connected data.\n",
    "It shows candidate keys for features that initially cause the records\n",
    "to be analyzed for a relationship,\n",
    "plus a series of feature scores that show how similar the feature data was.\n",
    "\n",
    "The response document also contains a separate ENTITIES section,\n",
    "with the full information about the resolved entity.\n",
    "(Note: When working with this entity data,\n",
    "Senzing recommends using the flags `G2_ENTITY_SHOW_FEATURES_EXPRESSED`\n",
    "and `G2_ENTITY_SHOW_FEATURES_STATS`.\n",
    "This will provide detailed feature data that is not included by default,\n",
    "but is useful for understanding the WHY_RESULTS data.)\n",
    "\n",
    "The functions `whyEntityByEntityIDV2()` and `whyEntityByRecordV2()` are\n",
    "enhanced versions of `whyEntityByEntityID()` and `whyEntityByRecordID()`\n",
    "that also allow you to use control flags.\n",
    "The `whyEntityByEntityID()` and `whyEntityByRecordID()` functions work in the\n",
    "same way, but use the default flag value `G2_WHY_ENTITY_DEFAULT_FLAGS`.\n",
    "\n",
    "For more information, see\n",
    "[http://docs.senzing.com/?python](http://docs.senzing.com/?python#connection-details)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### whyEntityByRecordID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.whyEntityByRecordID(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### whyEntityByRecordIDV2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.whyEntityByRecordIDV2(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### whyEntityByEntityID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.whyEntityByEntityID(\n",
    "    entity_id_1,\n",
    "    response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### whyEntityByEntityIDV2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.whyEntityByEntityIDV2(\n",
    "    entity_id_1,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Replace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### replaceRecord\n",
    "\n",
    "Use the `replaceRecord()` function to update or replace a record in the data\n",
    "repository.\n",
    "If record doesn't exist, a new record is added to the data repository.\n",
    "Like the above functions, `replaceRecord()` returns \"0\" upon success,\n",
    "and it can be called as many times as desired and from multiple threads at\n",
    "the same time.\n",
    "The `replaceRecord()` function accepts four parameters as input:\n",
    "\n",
    "- **datasource_code:** The name of the data source the record is associated\n",
    "  with.  This value is configurable to the system\n",
    "- **record_id:** The record ID, used to identify distinct records\n",
    "- **data_as_json:** A JSON document with the attribute data for the record\n",
    "- **load_id:** The observation load ID for the record;\n",
    "  value can be null and will default to datasource_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"NAMES\": [{\n",
    "        \"NAME_TYPE\": \"PRIMARY\",\n",
    "        \"NAME_LAST\": \"Miller\",\n",
    "        \"NAME_FIRST\": \"John\",\n",
    "        \"NAME_MIDDLE\": \"M\"\n",
    "    }],\n",
    "    \"PASSPORT_NUMBER\": \"PP11111\",\n",
    "    \"PASSPORT_COUNTRY\": \"US\",\n",
    "    \"DRIVERS_LICENSE_NUMBER\": \"DL11111\",\n",
    "    \"SSN_NUMBER\": \"111-11-1111\"\n",
    "}\n",
    "data_as_json = json.dumps(data)\n",
    "\n",
    "return_code = g2_engine.replaceRecord(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    data_as_json,\n",
    "    load_id)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### replaceRecordWithInfo\n",
    "\n",
    "`replaceRecordWithInfo()` is available if you would like to know what\n",
    "resolved entities were modified when replacing a record.\n",
    "It behaves identically to `replaceRecord()`,\n",
    "but also returns a json document containing the IDs of the affected entities.\n",
    "It accepts the following parameters:\n",
    "\n",
    "- **datasource_code:** The name of the data source the record is associated\n",
    "  with.  This value is configurable to the system.\n",
    "- **record_id:** The record ID, used to identify distinct records\n",
    "- **data_as_json:** A JSON document with the attribute data for the record\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document; if an error occurred, an error response is stored here.\n",
    "- **load_id:** The observation load ID for the record;\n",
    "  value can be null and will default to datasource_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"NAMES\": [{\n",
    "        \"NAME_TYPE\": \"PRIMARY\",\n",
    "        \"NAME_LAST\": \"Jones\",\n",
    "        \"NAME_FIRST\": \"John\",\n",
    "        \"NAME_MIDDLE\": \"M\"\n",
    "    }],\n",
    "    \"PASSPORT_NUMBER\": \"PP11111\",\n",
    "    \"PASSPORT_COUNTRY\": \"US\",\n",
    "    \"DRIVERS_LICENSE_NUMBER\": \"DL11111\",\n",
    "    \"SSN_NUMBER\": \"111-11-1111\"\n",
    "}\n",
    "data_as_json = json.dumps(data)\n",
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.replaceRecordWithInfo(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    data_as_json,\n",
    "    response_bytearray,\n",
    "    load_id)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Re-evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### reevaluateRecord"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "return_code = g2_engine.reevaluateRecord(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    g2_engine_flags)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### reevaluateRecordWithInfo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.reevaluateRecordWithInfo(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    response_bytearray,\n",
    "    g2_engine_flags)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### reevaluateEntity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find an entity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByRecordIDV2(\n",
    "    datasource_code_1,\n",
    "    record_id_1,\n",
    "    g2_engine_flags,\n",
    "    response_bytearray)\n",
    "\n",
    "response_dictionary = json.loads(response_bytearray)\n",
    "entity_id_1 = response_dictionary[\"RESOLVED_ENTITY\"][\"ENTITY_ID\"]\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-evaluate the entity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "return_code = g2_engine.reevaluateEntity(entity_id_1, g2_engine_flags)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### reevaluateEntityWithInfo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.reevaluateEntityWithInfo(\n",
    "    entity_id_1,\n",
    "    response_bytearray,\n",
    "    g2_engine_flags)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reporting\n",
    "\n",
    "Exporting entity data from resolved entities is one of the core purposes of\n",
    "Senzing software.\n",
    "In just a few short steps, the Senzing engine allows users to export entity\n",
    "data in either JSON or CSV format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### exportJSONEntityReport\n",
    "\n",
    "There are three steps to exporting resolved entity data from the G2Engine\n",
    "object in JSON format.\n",
    "First, use the `exportJSONEntityReport()` method to generate a long integer,\n",
    "referred to here as an `export_handle`.\n",
    "The `exportJSONEntityReport()` method accepts one parameter as input:\n",
    "\n",
    "- **g2_engine_flags**: An integer specifying which entity details should be\n",
    "  included in the export. See the \"Entity Export Flags\" section for further\n",
    "  details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "export_handle = g2_engine.exportJSONEntityReport(g2_engine_flags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### fetchNext\n",
    "\n",
    "Second, use the `fetchNext()` method to read the exportHandle and export a\n",
    "row of JSON output containing the entity data for a single entity.\n",
    "Note that successive calls of `fetchNext()` will export successive rows of\n",
    "entity data.\n",
    "The `fetchNext()` method accepts the following parameters as input:\n",
    "\n",
    "- **export_handle:** A long integer from which resolved entity data may be\n",
    "  read and exported.\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document; if an error occurred, an error response is stored here.\n",
    "\n",
    "For more information, see\n",
    "[http://docs.senzing.com/?python](http://docs.senzing.com/?python#reporting)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "while True:\n",
    "    response_bytearray = bytearray()\n",
    "    g2_engine.fetchNext(export_handle, response_bytearray)\n",
    "    if not response_bytearray:\n",
    "        break\n",
    "    response_dictionary = json.loads(response_bytearray)        \n",
    "    response = json.dumps(response_dictionary, sort_keys=True, indent=4)        \n",
    "    print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### closeExport"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g2_engine.closeExport(export_handle)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### exportCSVEntityReport\n",
    "\n",
    "There are three steps to exporting resolved entity data from the G2Engine\n",
    "object in CSV format.\n",
    "First, use the `exportCSVEntityReportV2()` method to generate a long integer,\n",
    "referred to here as an 'export_handle'.\n",
    "\n",
    "The `exportCSVEntityReportV2()` method accepts these parameter as input:\n",
    "\n",
    "- **csv_column_list:** A comma-separated list of column names for the CSV\n",
    "  export. (These are listed a little further down.)\n",
    "- **g2_engine_flags:** An integer specifying which entity details should be\n",
    "  included in the export.\n",
    "  See the \"Entity Export Flags\" section for further details.\n",
    "\n",
    "Second, use the `fetchNext()` method to read the exportHandle and export a\n",
    "row of CSV output containing the entity data for a single entity.\n",
    "Note that the first call of `fetchNext()` will yield a header row,\n",
    "and that successive calls of `fetchNext()` will export successive rows of\n",
    "entity data.\n",
    "The `fetchNext()` method accepts the following parameters as input:\n",
    "\n",
    "- **export_handle:** A long integer from which resolved entity data may be\n",
    "  read and exported\n",
    "- **response_bytearray:** A memory buffer for returning the response\n",
    "  document; if an error occurred, an error response is stored here\n",
    "\n",
    "For more information, see\n",
    "[http://docs.senzing.com/?python](http://docs.senzing.com/?python#reporting)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "export_handle = g2_engine.exportCSVEntityReport(g2_engine_flags)\n",
    "\n",
    "while True:\n",
    "    response_bytearray = bytearray()\n",
    "    g2_engine.fetchNext(export_handle, response_bytearray)\n",
    "    if not response_bytearray:\n",
    "        break\n",
    "    print(response_bytearray.decode())\n",
    "    \n",
    "g2_engine.closeExport(export_handle)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Redo Processing\n",
    "\n",
    "Redo records are automatically created by Senzing when certain conditions\n",
    "occur where it believes more processing may be needed.\n",
    "Some examples:\n",
    "\n",
    "- A value becomes generic and previous decisions may need to be revisited\n",
    "- Clean up after some record deletes\n",
    "- Detected related entities were being changed at the same time\n",
    "- A table inconsistency exists, potentially after a non-graceful shutdown\n",
    "\n",
    "First we will need to have a total of 6 data sources so let's add 4 more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create Record and Entity #6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"NAMES\": [{\n",
    "        \"NAME_TYPE\": \"PRIMARY\",\n",
    "        \"NAME_LAST\": \"Owens\",\n",
    "        \"NAME_FIRST\": \"Lily\"\n",
    "    }],\n",
    "    \"SSN_NUMBER\": \"111-11-1111\"\n",
    "}\n",
    "data_as_json = json.dumps(data)\n",
    "\n",
    "return_code = g2_engine.replaceRecord(\n",
    "    datasource_code_4,\n",
    "    record_id_4,\n",
    "    data_as_json,\n",
    "    None)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByRecordID(\n",
    "    datasource_code_4,\n",
    "    record_id_4,\n",
    "    response_bytearray)\n",
    "\n",
    "response_dictionary = json.loads(response_bytearray)\n",
    "entity_id_6 = response_dictionary[\"RESOLVED_ENTITY\"][\"ENTITY_ID\"]\n",
    "\n",
    "print(\"Return Code: {0}\\nEntity ID: {1}\".format(return_code, entity_id_6))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create Record and Entity #7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"NAMES\": [{\n",
    "        \"NAME_TYPE\": \"PRIMARY\",\n",
    "        \"NAME_LAST\": \"Bauler\",\n",
    "        \"NAME_FIRST\": \"August\",\n",
    "        \"NAME_MIDDLE\": \"E\"\n",
    "    }],\n",
    "    \"SSN_NUMBER\": \"111-11-1111\"\n",
    "}\n",
    "data_as_json = json.dumps(data)\n",
    "\n",
    "return_code = g2_engine.replaceRecord(\n",
    "    datasource_code_5,\n",
    "    record_id_5,\n",
    "    data_as_json,\n",
    "    None)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByRecordID(\n",
    "    datasource_code_5,\n",
    "    record_id_5,\n",
    "    response_bytearray)\n",
    "\n",
    "response_dictionary = json.loads(response_bytearray)\n",
    "entity_id_7 = response_dictionary[\"RESOLVED_ENTITY\"][\"ENTITY_ID\"]\n",
    "\n",
    "print(\"Return Code: {0}\\nEntity ID: {1}\".format(return_code, entity_id_7))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create Record and Entity #8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"NAMES\": [{\n",
    "        \"NAME_TYPE\": \"PRIMARY\",\n",
    "        \"NAME_LAST\": \"Barcy\",\n",
    "        \"NAME_FIRST\": \"Brian\",\n",
    "        \"NAME_MIDDLE\": \"H\"\n",
    "    }],\n",
    "    \"SSN_NUMBER\": \"111-11-1111\"\n",
    "}\n",
    "data_as_json = json.dumps(data)\n",
    "\n",
    "return_code = g2_engine.replaceRecord(\n",
    "    datasource_code_6,\n",
    "    record_id_6,\n",
    "    data_as_json,\n",
    "    None)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByRecordID(\n",
    "    datasource_code_6,\n",
    "    record_id_6,\n",
    "    response_bytearray)\n",
    "\n",
    "response_dictionary = json.loads(response_bytearray)\n",
    "entity_id_8 = response_dictionary[\"RESOLVED_ENTITY\"][\"ENTITY_ID\"]\n",
    "\n",
    "print(\"Return Code: {0}\\nEntity ID: {1}\".format(return_code, entity_id_8))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create Record and Entity #9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    \"NAMES\": [{\n",
    "        \"NAME_TYPE\": \"PRIMARY\",\n",
    "        \"NAME_LAST\": \"Miller\",\n",
    "        \"NAME_FIRST\": \"Jack\",\n",
    "        \"NAME_MIDDLE\": \"H\"\n",
    "    }],\n",
    "    \"SSN_NUMBER\": \"111-11-1111\"\n",
    "}\n",
    "data_as_json = json.dumps(data)\n",
    "\n",
    "return_code = g2_engine.replaceRecord(\n",
    "    datasource_code_7,\n",
    "    record_id_7,\n",
    "    data_as_json,\n",
    "    None)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getEntityByRecordID(\n",
    "    datasource_code_7,\n",
    "    record_id_7,\n",
    "    response_bytearray)\n",
    "\n",
    "response_dictionary = json.loads(response_bytearray)\n",
    "entity_id_9 = response_dictionary[\"RESOLVED_ENTITY\"][\"ENTITY_ID\"]\n",
    "\n",
    "print(\"Return Code: {0}\\nEntity ID: {1}\".format(return_code, entity_id_9))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### countRedoRecords\n",
    "\n",
    "Once the Senzing engine is initialized, use `countRedoRecords()`\n",
    "to return the remaining internally queued maintenance records in the\n",
    "Senzing repository.\n",
    "`countRedoRecords()` takes no arguments and returns <0 for errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "return_code = g2_engine.countRedoRecords()\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### getRedoRecord\n",
    "\n",
    "Once the Senzing engine is initialized, \n",
    "use `getRedoRecord()` to retrieve the next internally queued maintenance\n",
    "record into the Senzing repository\n",
    "-- `getRedoRecord()` can be called as many times as desired and from multiple\n",
    "threads at the same time but all threads are required to be in the same\n",
    "process.\n",
    "`getRedoRecord()` should not be called from multiple processes.\n",
    "Unlike `processRedoRecord()`, `getRedoRecord()` does not actually process the\n",
    "record.\n",
    "To process the record, you would use the G2Engine `process()` function.\n",
    "The `getRedoRecord()` function returns \"0\" upon success and an empty response\n",
    "if there is nothing to do.\n",
    "\n",
    "- **response_bytearray:** A memory buffer for returning the maintenance\n",
    "  document (may be XML or JSON).\n",
    "  The format is internal to Senzing.\n",
    "  If empty it means there are no maintenance records to return."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.getRedoRecord(response_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### processWithInfo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (return_code == 0 and response_bytearray):\n",
    "    process_response_bytearray = bytearray()\n",
    "    \n",
    "    process_return_code = g2_engine.processWithInfo(\n",
    "        response_bytearray.decode(),\n",
    "        process_response_bytearray)\n",
    "\n",
    "    print(\"Return Code: {0}\".format(process_return_code))\n",
    "    RenderJSON(process_response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (return_code == 0 and response_bytearray):\n",
    "    g2_engine.process(response_bytearray.decode())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### processRedoRecord\n",
    "\n",
    "This processes the next redo record and returns it\n",
    "(If `processRedoRecord()` \"response\" returns 0\n",
    "and \"response_bytearray\" is blank then there are no more redo records to\n",
    "process and if you do `count.RedoRecords()` again it will return 0)\n",
    "Has potential to create more redo records in certian situations.\n",
    "\n",
    "- **response_bytearray:** A buffer that returns a JSON object that summaries\n",
    "  the changes cased by adding the record.\n",
    "  Also contains the recordID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "return_code = g2_engine.processRedoRecord(response_bytearray)\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "\n",
    "# Pretty-print XML.\n",
    "\n",
    "xml_string = response_bytearray.decode()\n",
    "if len(xml_string) > 0:\n",
    "    import xml.dom.minidom\n",
    "    xml = xml.dom.minidom.parseString(xml_string)\n",
    "    xml_pretty_string = xml.toprettyxml()\n",
    "    print(xml_pretty_string)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### processRedoRecordWithInfo\n",
    "\n",
    "`processRedoRecordWithInfo()` is available if you would like to know what\n",
    "resolved entities were modified when processing a redo record.\n",
    "It behaves identically to `processRedoRecord()`,\n",
    "but also returns a json document containing the IDs of the affected entities.\n",
    "It accepts the following parameters:\n",
    "\n",
    "- **response_bytearray:**  A buffer that returns a JSON object that summaries\n",
    "  the changes cased by adding the record. Also contains the recordID.\n",
    "- **response_bytearray:** A buffer that returns a JSON object that summaries\n",
    "  the changes cased by adding the record. Also contains the recordID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "info_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.processRedoRecordWithInfo(\n",
    "    response_bytearray,\n",
    "    info_bytearray)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "\n",
    "# Pretty-print XML.\n",
    "\n",
    "xml_string = response_bytearray.decode()\n",
    "if len(xml_string) > 0:\n",
    "    import xml.dom.minidom\n",
    "    xml = xml.dom.minidom.parseString(xml_string)\n",
    "    xml_pretty_string = xml.toprettyxml()\n",
    "    print(xml_pretty_string)\n",
    "    \n",
    "# Pretty-print JSON\n",
    "\n",
    "RenderJSON(info_bytearray)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### deleteRecord\n",
    "\n",
    "Use `deleteRecord()` to remove a record from the data repository\n",
    "(returns \"0\" upon success);\n",
    "`deleteRecord()` can be called as many times as desired and from multiple\n",
    "threads at the same time.\n",
    "The `deleteRecord()` function accepts three parameters as input:\n",
    "\n",
    "- **datasource_code:** The name of the data source the record is associated\n",
    "  with.  This value is configurable to the system.\n",
    "- **record_id:** The record ID, used to identify distinct records\n",
    "- **load_id:** The observation load ID for the record;\n",
    "  value can be null and will default to dataSourceCode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "return_code = g2_engine.deleteRecord(datasource_code_1, record_id_1, load_id)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### deleteRecordWithInfo\n",
    "\n",
    "`deleteRecordWithInfo()` behaves the same as `deleteRecord()`\n",
    "but also returns a json document containing the IDs of the affected entities.\n",
    "It accepts the following parameters:\n",
    "\n",
    "- **datasource_code:** The name of the data source the record is associated\n",
    "  with. This value is configurable to the system.\n",
    "- **record_id:** The record ID, used to identify distinct records.\n",
    "- **response_bytearray:** A buffer that returns a JSON object that summaries\n",
    "  the changes cased by adding the record. Also contains the recordID.\n",
    "- **load_id:** The observation load ID for the record;\n",
    "  value can be null and will default to dataSourceCode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_bytearray = bytearray()\n",
    "\n",
    "return_code = g2_engine.deleteRecordWithInfo(\n",
    "    datasource_code_2,\n",
    "    record_id_2,\n",
    "    response_bytearray,\n",
    "    load_id,\n",
    "    g2_engine_flags)\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))\n",
    "RenderJSON(response_bytearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attempt to get the record again.\n",
    "It should error and give an output similar to \"Unknown record\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    response_bytearray = bytearray()\n",
    "    \n",
    "    return_code = g2_engine.getRecord(\n",
    "        datasource_code_1,\n",
    "        record_id_1,\n",
    "        response_bytearray)\n",
    "    \n",
    "    response_dictionary = json.loads(response_bytearray)\n",
    "    response = json.dumps(response_dictionary, sort_keys=True, indent=4)\n",
    "    print(\"Return Code: {0}\\n{1}\".format(return_code, response))\n",
    "except G2Exception.G2ModuleGenericException as err:\n",
    "    print(\"Exception: {0}\".format(err))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "To purge the G2 repository, use the aptly named `purgeRepository()` method.\n",
    "This will remove every record in your current repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### purgeRepository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g2_engine.purgeRepository()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### destroy\n",
    "\n",
    "Once all searching is done in a given process,\n",
    "call `destroy()` to uninitialize Senzing and clean up resources.\n",
    "You should always do this once at the end of each process. See\n",
    "[http://docs.senzing.com/?python](http://docs.senzing.com/?python#engine)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "return_code = g2_engine.destroy()\n",
    "\n",
    "print(\"Return Code: {0}\".format(return_code))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}