forked from swcarpentry/shell-novice
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy path03-pipefilter.html
296 lines (296 loc) · 21.8 KB
/
03-pipefilter.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="generator" content="pandoc">
<title>Software Carpentry: The Unix Shell</title>
<link rel="shortcut icon" type="image/x-icon" href="/favicon.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="stylesheet" type="text/css" href="css/bootstrap/bootstrap.css" />
<link rel="stylesheet" type="text/css" href="css/bootstrap/bootstrap-theme.css" />
<link rel="stylesheet" type="text/css" href="css/swc.css" />
<link rel="alternate" type="application/rss+xml" title="Software Carpentry Blog" href="http://software-carpentry.org/feed.xml"/>
<meta charset="UTF-8" />
<!-- HTML5 shim, for IE6-8 support of HTML5 elements -->
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
</head>
<body class="lesson">
<div class="container card">
<div class="banner">
<a href="http://software-carpentry.org" title="Software Carpentry">
<img alt="Software Carpentry banner" src="img/software-carpentry-banner.png" />
</a>
</div>
<article>
<div class="row">
<div class="col-md-10 col-md-offset-1">
<a href="index.html"><h1 class="title">The Unix Shell</h1></a>
<h2 class="subtitle">Pipes and Filters</h2>
<section class="objectives panel panel-warning">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-certificate"></span>Learning Objectives</h2>
</div>
<div class="panel-body">
<ul>
<li>Redirect a command’s output to a file.</li>
<li>Process a file instead of keyboard input using redirection.</li>
<li>Construct command pipelines with two or more stages.</li>
<li>Explain what usually happens if a program or pipeline isn’t given any input to process.</li>
<li>Explain Unix’s “small pieces, loosely joined” philosophy.</li>
</ul>
</div>
</section>
<p>Now that we know a few basic commands, we can finally look at the shell’s most powerful feature: the ease with which it lets us combine existing programs in new ways. We’ll start with a directory called <code>molecules</code> that contains six files describing some simple organic molecules. The <code>.pdb</code> extension indicates that these files are in Protein Data Bank format, a simple text format that specifies the type and position of each atom in the molecule.</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">ls</span> molecules</code></pre>
<pre class="output"><code>cubane.pdb ethane.pdb methane.pdb
octane.pdb pentane.pdb propane.pdb</code></pre>
<p>Let’s go into that directory with <code>cd</code> and run the command <code>wc *.pdb</code>. <code>wc</code> is the “word count” command: it counts the number of lines, words, and characters in files. The <code>*</code> in <code>*.pdb</code> matches zero or more characters, so the shell turns <code>*.pdb</code> into a complete list of <code>.pdb</code> files:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">cd</span> molecules
$ <span class="kw">wc</span> *.pdb</code></pre>
<pre class="output"><code> 20 156 1158 cubane.pdb
12 84 622 ethane.pdb
9 57 422 methane.pdb
30 246 1828 octane.pdb
21 165 1226 pentane.pdb
15 111 825 propane.pdb
107 819 6081 total</code></pre>
<aside class="callout panel panel-info">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-pushpin"></span>Wildcards</h2>
</div>
<div class="panel-body">
<p><code>*</code> is a <strong>wildcard</strong>. It matches zero or more characters, so <code>*.pdb</code> matches <code>ethane.pdb</code>, <code>propane.pdb</code>, and so on. On the other hand, <code>p*.pdb</code> only matches <code>pentane.pdb</code> and <code>propane.pdb</code>, because the ‘p’ at the front only matches itself.</p>
<p><code>?</code> is also a wildcard, but it only matches a single character. This means that <code>p?.pdb</code> matches <code>pi.pdb</code> or <code>p5.pdb</code>, but not <code>propane.pdb</code>. We can use any number of wildcards at a time: for example, <code>p*.p?*</code> matches anything that starts with a ‘p’ and ends with ‘.’, ‘p’, and at least one more character (since the ‘?’ has to match one character, and the final ‘*’ can match any number of characters). Thus, <code>p*.p?*</code> would match <code>preferred.practice</code>, and even <code>p.pi</code> (since the first ‘*’ can match no characters at all), but not <code>quality.practice</code> (doesn’t start with ‘p’) or <code>preferred.p</code> (there isn’t at least one character after the ‘.p’).</p>
<p>When the shell sees a wildcard, it expands the wildcard to create a list of matching filenames <em>before</em> running the command that was asked for. As an exception, if a wildcard expression does not match any file, Bash will pass the expression as a parameter to the command as it is. For example typing <code>ls *.pdf</code> in the molecules directory (which contains only files with names ending with <code>.pdb</code>) results in an error message that there is no file called <code>*.pdf</code>. However, generally commands like <code>wc</code> and <code>ls</code> see the lists of file names matching these expressions, but not the wildcards themselves. It is the shell, not the other programs, that deals with expanding wildcards, and this another example of orthogonal design.</p>
</div>
</aside>
<p>If we run <code>wc -l</code> instead of just <code>wc</code>, the output shows only the number of lines per file:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">wc</span> -l *.pdb</code></pre>
<pre class="output"><code> 20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total</code></pre>
<p>We can also use <code>-w</code> to get only the number of words, or <code>-c</code> to get only the number of characters.</p>
<p>Which of these files is shortest? It’s an easy question to answer when there are only six files, but what if there were 6000? Our first step toward a solution is to run the command:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">wc</span> -l *.pdb <span class="kw">></span> lengths.txt</code></pre>
<p>The greater than symbol, <code>></code>, tells the shell to <strong>redirect</strong> the command’s output to a file instead of printing it to the screen. The shell will create the file if it doesn’t exist, or overwrite the contents of that file if it does. (This is why there is no screen output: everything that <code>wc</code> would have printed has gone into the file <code>lengths.txt</code> instead.) <code>ls lengths.txt</code> confirms that the file exists:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">ls</span> lengths.txt</code></pre>
<pre class="output"><code>lengths.txt</code></pre>
<p>We can now send the content of <code>lengths.txt</code> to the screen using <code>cat lengths.txt</code>. <code>cat</code> stands for “concatenate”: it prints the contents of files one after another. There’s only one file in this case, so <code>cat</code> just shows us what it contains:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">cat</span> lengths.txt</code></pre>
<pre class="output"><code> 20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total</code></pre>
<p>Now let’s use the <code>sort</code> command to sort its contents. We will also use the -n flag to specify that the sort is numerical instead of alphabetical. This does <em>not</em> change the file; instead, it sends the sorted result to the screen:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">sort</span> -n lengths.txt</code></pre>
<pre class="output"><code> 9 methane.pdb
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
107 total</code></pre>
<p>We can put the sorted list of lines in another temporary file called <code>sorted-lengths.txt</code> by putting <code>> sorted-lengths.txt</code> after the command, just as we used <code>> lengths.txt</code> to put the output of <code>wc</code> into <code>lengths.txt</code>. Once we’ve done that, we can run another command called <code>head</code> to get the first few lines in <code>sorted-lengths.txt</code>:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">sort</span> -n lengths.txt <span class="kw">></span> sorted-lengths.txt
$ <span class="kw">head</span> -1 sorted-lengths.txt</code></pre>
<pre class="output"><code> 9 methane.pdb</code></pre>
<p>Using the parameter <code>-1</code> with <code>head</code> tells it that we only want the first line of the file; <code>-20</code> would get the first 20, and so on. Since <code>sorted-lengths.txt</code> contains the lengths of our files ordered from least to greatest, the output of <code>head</code> must be the file with the fewest lines.</p>
<p>If you think this is confusing, you’re in good company: even once you understand what <code>wc</code>, <code>sort</code>, and <code>head</code> do, all those intermediate files make it hard to follow what’s going on. We can make it easier to understand by running <code>sort</code> and <code>head</code> together:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">sort</span> -n lengths.txt <span class="kw">|</span> <span class="kw">head</span> -1</code></pre>
<pre class="output"><code> 9 methane.pdb</code></pre>
<p>The vertical bar between the two commands is called a <strong>pipe</strong>. It tells the shell that we want to use the output of the command on the left as the input to the command on the right. The computer might create a temporary file if it needs to, or copy data from one program to the other in memory, or something else entirely; we don’t have to know or care.</p>
<p>We can use another pipe to send the output of <code>wc</code> directly to <code>sort</code>, which then sends its output to <code>head</code>:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">wc</span> -l *.pdb <span class="kw">|</span> <span class="kw">sort</span> -n <span class="kw">|</span> <span class="kw">head</span> -1</code></pre>
<pre class="output"><code> 9 methane.pdb</code></pre>
<p>This is exactly like a mathematician nesting functions like <em>log(3x)</em> and saying “the log of three times <em>x</em>”. In our case, the calculation is “head of sort of line count of <code>*.pdb</code>”.</p>
<p>Here’s what actually happens behind the scenes when we create a pipe. When a computer runs a program — any program — it creates a <strong>process</strong> in memory to hold the program’s software and its current state. Every process has an input channel called <strong>standard input</strong>. (By this point, you may be surprised that the name is so memorable, but don’t worry: most Unix programmers call it “stdin”. Every process also has a default output channel called <strong>standard output</strong> (or “stdout”).</p>
<p>The shell is actually just another program. Under normal circumstances, whatever we type on the keyboard is sent to the shell on its standard input, and whatever it produces on standard output is displayed on our screen. When we tell the shell to run a program, it creates a new process and temporarily sends whatever we type on our keyboard to that process’s standard input, and whatever the process sends to standard output to the screen.</p>
<p>Here’s what happens when we run <code>wc -l *.pdb > lengths.txt</code>. The shell starts by telling the computer to create a new process to run the <code>wc</code> program. Since we’ve provided some filenames as parameters, <code>wc</code> reads from them instead of from standard input. And since we’ve used <code>></code> to redirect output to a file, the shell connects the process’s standard output to that file.</p>
<p>If we run <code>wc -l *.pdb | sort -n</code> instead, the shell creates two processes (one for each process in the pipe) so that <code>wc</code> and <code>sort</code> run simultaneously. The standard output of <code>wc</code> is fed directly to the standard input of <code>sort</code>; since there’s no redirection with <code>></code>, <code>sort</code>’s output goes to the screen. And if we run <code>wc -l *.pdb | sort -n | head -1</code>, we get three processes with data flowing from the files, through <code>wc</code> to <code>sort</code>, and from <code>sort</code> through <code>head</code> to the screen.</p>
<div class="figure">
<img src="fig/redirects-and-pipes.png" alt="Redirects and Pipes" /><p class="caption">Redirects and Pipes</p>
</div>
<p>This simple idea is why Unix has been so successful. Instead of creating enormous programs that try to do many different things, Unix programmers focus on creating lots of simple tools that each do one job well, and that work well with each other. This programming model is called “pipes and filters”. We’ve already seen pipes; a <strong>filter</strong> is a program like <code>wc</code> or <code>sort</code> that transforms a stream of input into a stream of output. Almost all of the standard Unix tools can work this way: unless told to do otherwise, they read from standard input, do something with what they’ve read, and write to standard output.</p>
<p>The key is that any program that reads lines of text from standard input and writes lines of text to standard output can be combined with every other program that behaves this way as well. You can <em>and should</em> write your programs this way so that you and other people can put those programs into pipes to multiply their power.</p>
<aside class="callout panel panel-info">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-pushpin"></span>Redirecting Input</h2>
</div>
<div class="panel-body">
<p>As well as using <code>></code> to redirect a program’s output, we can use <code><</code> to redirect its input, i.e., to read from a file instead of from standard input. For example, instead of writing <code>wc ammonia.pdb</code>, we could write <code>wc < ammonia.pdb</code>. In the first case, <code>wc</code> gets a command line parameter telling it what file to open. In the second, <code>wc</code> doesn’t have any command line parameters, so it reads from standard input, but we have told the shell to send the contents of <code>ammonia.pdb</code> to <code>wc</code>’s standard input.</p>
</div>
</aside>
<h3 id="nelles-pipeline-checking-files">Nelle’s Pipeline: Checking Files</h3>
<p>Nelle has run her samples through the assay machines and created 1520 files in the <code>north-pacific-gyre/2012-07-03</code> directory described earlier. As a quick sanity check, starting from her home directory, Nelle types:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">cd</span> north-pacific-gyre/2012-07-03
$ <span class="kw">wc</span> -l *.txt</code></pre>
<p>The output is 1520 lines that look like this:</p>
<pre class="output"><code>300 NENE01729A.txt
300 NENE01729B.txt
300 NENE01736A.txt
300 NENE01751A.txt
300 NENE01751B.txt
300 NENE01812A.txt
... ...</code></pre>
<p>Now she types this:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">wc</span> -l *.txt <span class="kw">|</span> <span class="kw">sort</span> -n <span class="kw">|</span> <span class="kw">head</span> -5</code></pre>
<pre class="output"><code> 240 NENE02018B.txt
300 NENE01729A.txt
300 NENE01729B.txt
300 NENE01736A.txt
300 NENE01751A.txt</code></pre>
<p>Whoops: one of the files is 60 lines shorter than the others. When she goes back and checks it, she sees that she did that assay at 8:00 on a Monday morning — someone was probably in using the machine on the weekend, and she forgot to reset it. Before re-running that sample, she checks to see if any files have too much data:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">wc</span> -l *.txt <span class="kw">|</span> <span class="kw">sort</span> -n <span class="kw">|</span> <span class="kw">tail</span> -5</code></pre>
<pre class="output"><code> 300 NENE02040A.txt
300 NENE02040B.txt
300 NENE02040Z.txt
300 NENE02043A.txt
300 NENE02043B.txt</code></pre>
<p>Those numbers look good — but what’s that ‘Z’ doing there in the third-to-last line? All of her samples should be marked ‘A’ or ‘B’; by convention, her lab uses ‘Z’ to indicate samples with missing information. To find others like it, she does this:</p>
<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">ls</span> *Z.txt</code></pre>
<pre class="output"><code>NENE01971Z.txt NENE02040Z.txt</code></pre>
<p>Sure enough, when she checks the log on her laptop, there’s no depth recorded for either of those samples. Since it’s too late to get the information any other way, she must exclude those two files from her analysis. She could just delete them using <code>rm</code>, but there are actually some analyses she might do later where depth doesn’t matter, so instead, she’ll just be careful later on to select files using the wildcard expression <code>*[AB].txt</code>. As always, the ‘*’ matches any number of characters; the expression <code>[AB]</code> matches either an ‘A’ or a ‘B’, so this matches all the valid data files she has.</p>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-pencil"></span>What does <code>sort -n</code> do?</h2>
</div>
<div class="panel-body">
<p>If we run <code>sort</code> on this file:</p>
<pre><code>10
2
19
22
6</code></pre>
<p>the output is:</p>
<pre><code>10
19
2
22
6</code></pre>
<p>If we run <code>sort -n</code> on the same input, we get this instead:</p>
<pre><code>2
6
10
19
22</code></pre>
<p>Explain why <code>-n</code> has this effect.</p>
</div>
</section>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-pencil"></span>What does <code><</code> mean?</h2>
</div>
<div class="panel-body">
<p>What is the difference between:</p>
<pre><code>wc -l < mydata.dat</code></pre>
<p>and:</p>
<pre><code>wc -l mydata.dat</code></pre>
</div>
</section>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-pencil"></span>What does <code>>></code> mean?</h2>
</div>
<div class="panel-body">
<p>What is the difference between:</p>
<pre><code>echo hello > testfile01.txt</code></pre>
<p>and:</p>
<pre><code>echo hello >> testfile02.txt</code></pre>
<p>Hint: Try executing each command twice in a row and then examining the output files.</p>
</div>
</section>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-pencil"></span>Piping commands together</h2>
</div>
<div class="panel-body">
<p>In our current directory, we want to find the 3 files which have the least number of lines. Which command listed below would work?</p>
<ol style="list-style-type: decimal">
<li><code>wc -l * > sort -n > head -3</code></li>
<li><code>wc -l * | sort -n | head 1-3</code></li>
<li><code>wc -l * | head -3 | sort -n</code></li>
<li><code>wc -l * | sort -n | head -3</code></li>
</ol>
</div>
</section>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-pencil"></span>Why does <code>uniq</code> only remove adjacent duplicates?</h2>
</div>
<div class="panel-body">
<p>The command <code>uniq</code> removes adjacent duplicated lines from its input. For example, if a file <code>salmon.txt</code> contains:</p>
<pre><code>coho
coho
steelhead
coho
steelhead
steelhead</code></pre>
<p>then <code>uniq salmon.txt</code> produces:</p>
<pre><code>coho
steelhead
coho
steelhead</code></pre>
<p>Why do you think <code>uniq</code> only removes <em>adjacent</em> duplicated lines? (Hint: think about very large data sets.) What other command could you combine with it in a pipe to remove all duplicated lines?</p>
</div>
</section>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-pencil"></span>Pipe reading comprehension</h2>
</div>
<div class="panel-body">
<p>A file called <code>animals.txt</code> contains the following data:</p>
<pre><code>2012-11-05,deer
2012-11-05,rabbit
2012-11-05,raccoon
2012-11-06,rabbit
2012-11-06,deer
2012-11-06,fox
2012-11-07,rabbit
2012-11-07,bear</code></pre>
<p>What text passes through each of the pipes and the final redirect in the pipeline below?</p>
<pre><code>cat animals.txt | head -5 | tail -3 | sort -r > final.txt</code></pre>
</div>
</section>
<section class="challenge panel panel-success">
<div class="panel-heading">
<h2><span class="glyphicon glyphicon-pencil"></span>Pipe construction</h2>
</div>
<div class="panel-body">
<p>The command:</p>
<pre><code>$ cut -d , -f 2 animals.txt</code></pre>
<p>produces the following output:</p>
<pre><code>deer
rabbit
raccoon
rabbit
deer
fox
rabbit
bear</code></pre>
<p>What other command(s) could be added to this in a pipeline to find out what animals the file contains (without any duplicates in their names)?</p>
</div>
</section>
</div>
</div>
</article>
<div class="footer">
<a class="label swc-blue-bg" href="http://software-carpentry.org">Software Carpentry</a>
<a class="label swc-blue-bg" href="https://github.com/swcarpentry/shell-novice">Source</a>
<a class="label swc-blue-bg" href="mailto:admin@software-carpentry.org">Contact</a>
<a class="label swc-blue-bg" href="LICENSE.html">License</a>
</div>
</div>
<!-- Javascript placed at the end of the document so the pages load faster -->
<script src="http://software-carpentry.org/v5/js/jquery-1.9.1.min.js"></script>
<script src="css/bootstrap/bootstrap-js/bootstrap.js"></script>
</body>
</html>