lesson1.md

Week 12 Lesson 1

Data Preparation

In this lesson, you will learn how to work with large data files in an IPython Notebook. This includes dealign with emory issues, both with the IPython Kernel and general Python memory usage. You will learn how to control the size and data type of data being read into a Python program. Finally, you will learn how to use Pandas to compactly read in a data subset and subsequently write this data in a compact HDF format.

###Objectives ### By the end of this lesson, you will be able to:

• Understand the importance of memory management.
• Understand how to use the free command to determine the amount of available memory and swap memory for a running application.
• Understand how to control the data type and subsetting of data being read into a Python program.
• Understand how to write data into a compact HDF format.

Time Estimate

Approximately 2 hours.

Readings

• Course IPython Notebook on Python data exploration

Optional Additional Readings####

• Pandas writing HDF

Assessment

When you have completed and worked through the above readings, please take the Week 12 Lesson 1 Assessment.