Week5/notebooks/unixnetwork.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:575f713462dc334a7a8177bbaf8796d72a9fa3a918ae9c2e101c2221c4e44b20"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<DIV ALIGN=CENTER>\n",
      "\n",
      "# Unix Networking   \n",
      "## Professor Robert J. Brunner\n",
      "  \n",
      "</DIV>  \n",
      "-----\n",
      "-----"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Unix Networking Overview\n",
      "\n",
      "Unix-based operating systems form the backbone of the Internet. However, Unix is often used as a command line operating system. Thus there are a number of useful commands and tools that allow you to explore the networked world, to connect to remote computers, and to move data between systems. In this lesson, we will explore:\n",
      "\n",
      "- discovering and identifying network systems\n",
      "- remote connections to networked systems\n",
      "- data transport betwen networked systems\n",
      "\n",
      "-----"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Network Discovery\n",
      "\n",
      "When using a web browser, you are likely familiar with finding resources by entering a human readable name, like google.com or illinois.edu. Computers only work with numbers, however, so in reality networks operate by associated names to numbers. The formal specification for this is currently known as [IPv6](https://en.wikipedia.org/wiki/IPv6). THe former specification, [IPv4](https://en.wikipedia.org/wiki/IPv4) provided a four byte address for a computer on a network. Many of the commands we use still display the more easily understood IPV4 format. The mapping between human readble name (like google.com) and an IP address used by computers is handled by the domain name service (DNS).\n",
      "\n",
      "The following list summarizes some of the more useful Unix network commands:\n",
      "\n",
      "### `hostname`\n",
      "used to find out the name and IP address of the current host system\n",
      "\n",
      "    $ hostname\n",
      "    \n",
      "### `ping`\n",
      "\n",
      "Used to test a connection to a remote host. Can use the regular names or ip address.\n",
      "\n",
      "    $ ping google.com\n",
      "   \n",
      "### `traceroute`\n",
      "Used to find the route and time taken by communications sent from your host to reach the target host.\n",
      "\n",
      "    $ traceroute google.com\n",
      "\n",
      "### `dig`\n",
      "Used to find the ip address of a networked resource with a given name.\n",
      "\n",
      "    $ dig google.com\n",
      "\n",
      "### `ifconfig`\n",
      "Used to obtain detailed networking configuration information for the current host system.\n",
      "\n",
      "    $ ifconfig\n",
      "\n",
      "Note: these commands can be demonstrated by using a regular Unix shell, for example the boot2docker shell, but may not work properly in our Docker course container.\n",
      "\n",
      "-----"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Secure Shell\n",
      "\n",
      "With the growth of the Internet, standard mechanisms for network communication needed to be made more secure. This led to the development of the cryptographic network protocol for secure communications between two systems. Secure Shell (SSH) allows a user to securely connect to a remote system or to securely trasnfer files. The later functioantiiliy is generally done by using the Secure Shell cp `scp` or Secure Shell ftp `sftp` commands.\n",
      "\n",
      "SSH relies on public-private key pairs to enable ronbust compression. These keys are generated by the [`ssh-keygen`](https://en.wikipedia.org/wiki/Ssh-keygen) command.\n",
      "\n",
      "### Secure connection\n",
      "To connect to a remote computer by using SSH, we simply need to have our remote username and the remote system name:\n",
      "    $ ssh profbrunner@data.illinois.edu\n",
      "\n",
      "As part of the connection process, you will be required to enter your password on the remote system. When you connect for the first time (or if a machine name has changed its IP address) you may be presented with a warning message requesting that you double check the machine to which you are trying to connect. One example of where this can be important is to prevent a _man-in-the-middle_ attack where a hostile entity changes the IP-name mapping so that you are directed to connect to a system different than what you expect.\n",
      "\n",
      "### Secure file transport\n",
      "To transport files by using scp, a similar format is used, depending on whether files are being pulled or pushed.\n",
      "\n",
      "To pull a file from a remote host and save it locally:\n",
      "\n",
      "    $ scp profbrunner@data.illinois.edu:/home/profbrunner/myfile .\n",
      "\n",
      "To push a file to a remote host:  \n",
      "\n",
      "    $ scp myfile profbrunner@data.illinois.edu:/home/profbrunner/\n",
      "\n",
      "There is a recursive option `-r` to enable directory structures to be transported.\n",
      "\n",
      "-----"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Anonymous file download\n",
      "In many instances, we simply want to grab a file (or files) from an Internet resource, like a website. The simplest command to grab a file from a website is the `wget` command. When given a URL to the desired resource, `wget` will retrieve the file and save it locally. \n",
      "\n",
      "    /notebooks# wget http://w3c.org\n",
      "\n",
      "![WGET example](images/wget.png)\n",
      "\n",
      "`wget` is a powerful program with many options. For example, `wget` can recursively download an entire directory structure (which can be dangerous as you can easily fill up your available disk space!).\n",
      "     \n",
      "-----"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Connect to Running Contianer\n",
      "\n",
      "While `ssh` is a secure method to connect to a host machine, we can't use it to connect to a running docker container (at least not with our course docker image) since there is no ssh server running in the container. There are several options for connecting to a running container (i.e., `nsenter`), which can be very useful when a container is running in non-interactive mode (e.g., as a web server, or an ipython notebook server).\n",
      "\n",
      "To connect to a running Docker container, we can use the `docker exec` command. In this case, we will want to run this command in interactive mode and to start a bash shell. Given a container name, from `docker ps`, we can connect to a running container:\n",
      "\n",
      "    $ docker exec -it pensive_archimedes /bin/bash\n",
      "    \n",
      "\n",
      "![Docker Connect](images/docker-exec.png)\n",
      "\n",
      "As shown in the preceding figure, we have connected to the running Docker container and not a new container created form the original course image \n",
      "\n",
      "-----"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Breakout Session: Unix Networking\n",
      "\n",
      "During this session,  you should try out the different Unix networking commands we have discussed.\n",
      "\n",
      "### Use the ping command:\n",
      "This can be done in a Docker container or in the Unix shell.\n",
      "\n",
      "    /notebooks# ping www.johndeere.com\n",
      "    /notebooks# ping www.cat.com\n",
      "    /notebooks# ping google.com\n",
      "\n",
      "What are the differences? What do you learn from these commands? Try `ping` with a different hostname.\n",
      "\n",
      "### Use the traceroute command:\n",
      "At the Unix shell (not in a Docker container) try out the `traceroute` command.\n",
      "\n",
      "    $ traceroute google.com\n",
      "\n",
      "What does this tell you? Are the results the same when you repeat the command? What about a different host?\n",
      "\n",
      "### Grab a datafile\n",
      "We will want a dataset to use later in this week's lessons. A great test data set is the [US Airline data](http://stat-computing.org/dataexpo/2009/the-data.html), which is organized by year. Within your Docker container, use `wget` to grab the Airline data for the year 2001.\n",
      "\n",
      "    /notebooks# wget ????\n",
      "\n",
      "Replace the _????_ with the correct URL. The downloaded file has been compressed to save disk space. Using tools you learned in the previous lesson, unzip the file and take a look at the first few rows.\n",
      "\n",
      "-----"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Additional References\n",
      "\n",
      "1. A tutorial for [`ssh`](http://support.suso.com/supki/SSH_Tutorial_for_Linux)\n",
      "2. Wikipedia page for [`wget`](https://en.wikipedia.org/wiki/Wget)\n",
      "-----"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Return to the [Week 2 Index](index.ipynb).\n",
      "\n",
      "-----"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}