-
Notifications
You must be signed in to change notification settings - Fork 7
/
index10.html
462 lines (414 loc) · 64.2 KB
/
index10.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
<!DOCTYPE html>
<head>
<meta charset="utf-8">
<title>Store Halfword Byte-Reverse Indexed</title>
<meta name="author" content="OzLabs">
<link href="https://sthbrx.github.io/rss.xml" type="application/rss+xml" rel="alternate"
title="Store Halfword Byte-Reverse Indexed RSS Feed" />
<!-- http://t.co/dKP3o1e -->
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link href="https://sthbrx.github.io/favicon.png" rel="icon">
<link href="https://sthbrx.github.io/theme/css/main.css" media="screen, projection"
rel="stylesheet" type="text/css">
<link href="//fonts.googleapis.com/css?family=PT+Serif:regular,italic,bold,bolditalic"
rel="stylesheet" type="text/css">
<link href="//fonts.googleapis.com/css?family=PT+Sans:regular,italic,bold,bolditalic"
rel="stylesheet" type="text/css">
<script type="text/javascript">
document.addEventListener('DOMContentLoaded', function() {
var ts = document.createElement('span')
ts.className = 'toggle-sidebar'
ts = document.getElementById('content').appendChild(ts);
ts.addEventListener('click', function(e) {
e.preventDefault();
body = document.getElementsByTagName('body')[0];
bodyClasses = body.classList.toggle('collapse-sidebar');
});
var sections = document.querySelectorAll('aside.sidebar > section');
if (sections.length > 1) {
for (index = 0; index < sections.length; index++) {
section = sections[index];
if ((sections.length >= 3) && index % 3 === 0) {
section.classList.add("first");
}
var count = ((index +1) % 2) ? "odd" : "even";
section.classList.add(count);
}
}
if (sections.length >= 3) {
document.querySelector('aside.sidebar').classList.add('thirds');
}
});
</script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-91189608-1', 'auto');
ga('send', 'pageview');
</script>
</head>
<body>
<header role="banner"><hgroup>
<h1><a href="https://sthbrx.github.io/">Store Halfword Byte-Reverse Indexed</a></h1>
<h2>A Power Technical Blog</h2>
</hgroup></header>
<nav role="navigation"><ul class="subscription" data-subscription="rss">
<li><a href="https://sthbrx.github.io/rss.xml" rel="subscribe-rss">RSS</a></li>
</ul>
<ul class="main-navigation">
<li >
<a href="https://sthbrx.github.io/category/cryptography.html">Cryptography</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/development.html">Development</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/education.html">Education</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/openpower.html">OpenPOWER</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/performance.html">Performance</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/petitboot.html">Petitboot</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/snowpatch.html">snowpatch</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/virtualisation-and-emulation.html">Virtualisation and Emulation</a>
</li>
</ul></nav>
<div id="main">
<div id="content">
<div class="blog-index">
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/03/22/getting-logs-out-of-things/">Getting logs out of things</a>
</h1>
<p class="meta">
<time datetime="2016-03-22T18:00:00+11:00" pubdate>Tue 22 March 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/andrew-donnellan.html">Andrew Donnellan</a>
</span>
</span>
<time datetime="2016-03-22T18:00:00+11:00" pubdate>Tue 22 March 2016</time></div>
<div class="entry-content"><p>Here at OzLabs, we have an unfortunate habit of making our shiny Power computers very sad, which is a common problem in systems programming and kernel hacking. When this happens, we like having logs. In particular, we like to have the kernel log and the OPAL firmware log, which are, very surprisingly, rather helpful when debugging kernel and firmware issues.</p>
<p>Here's how to get them.</p>
<h2>From userspace</h2>
<p>You're lucky enough that your machine is still up, yay! As every Linux sysadmin knows, you can just grab the kernel log using <code>dmesg</code>.</p>
<p>As for the OPAL log: we can simply ask OPAL to tell us where its log is located in memory, copy it from there, and hand it over to userspace. In Linux, as per standard Unix conventions, we do this by exposing the log as a file, which can be found in <code>/sys/firmware/opal/msglog</code>.</p>
<p>Annoyingly, the <code>msglog</code> file reports itself as size 0 (I'm not sure exactly why, but I <em>think</em> it's due to limitations in sysfs), so if you try to copy the file with <code>cp</code>, you end up with just a blank file. However, you can read it with <code>cat</code> or <code>less</code>.</p>
<h2>From <code>xmon</code></h2>
<p><code>xmon</code> is a really handy in-kernel debugger for PowerPC that allows you to do basic debugging over the console without hooking up a second machine to use with <code>kgdb</code>. On our development systems, we often configure <code>xmon</code> to automatically begin debugging whenever we hit an oops or panic (using <code>xmon=on</code> on the kernel command line, or the <code>XMON_DEFAULT</code> Kconfig option). It can also be manually triggered:</p>
<div class="highlight"><pre><span></span><code><span class="n">root</span><span class="nv">@p86</span><span class="err">:</span><span class="o">~</span><span class="err">#</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">/</span><span class="k">proc</span><span class="o">/</span><span class="n">sysrq</span><span class="o">-</span><span class="k">trigger</span>
<span class="nl">sysrq</span><span class="p">:</span><span class="w"> </span><span class="n">SysRq</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">Entering</span><span class="w"> </span><span class="n">xmon</span>
<span class="n">cpu</span><span class="w"> </span><span class="mh">0x7</span><span class="err">:</span><span class="w"> </span><span class="nl">Vector</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="o">[</span><span class="n">c000000fcd717a80</span><span class="o">]</span>
<span class="nl">pc</span><span class="p">:</span><span class="w"> </span><span class="nl">c000000000085ad8</span><span class="p">:</span><span class="w"> </span><span class="n">sysrq_handle_xmon</span><span class="o">+</span><span class="mh">0x68</span><span class="o">/</span><span class="mh">0x80</span>
<span class="nl">lr</span><span class="p">:</span><span class="w"> </span><span class="nl">c000000000085ad8</span><span class="p">:</span><span class="w"> </span><span class="n">sysrq_handle_xmon</span><span class="o">+</span><span class="mh">0x68</span><span class="o">/</span><span class="mh">0x80</span>
<span class="nl">sp</span><span class="p">:</span><span class="w"> </span><span class="n">c000000fcd717be0</span>
<span class="nl">msr</span><span class="p">:</span><span class="w"> </span><span class="mi">9000000000009033</span>
<span class="k">current</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mh">0xc000000fcd689200</span>
<span class="n">paca</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mh">0xc00000000fe01c00</span><span class="w"> </span><span class="nl">softe</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nl">irq_happened</span><span class="p">:</span><span class="w"> </span><span class="mh">0x01</span>
<span class="n">pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">7127</span><span class="p">,</span><span class="w"> </span><span class="n">comm</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bash</span>
<span class="n">Linux</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="mf">4.5.0</span><span class="o">-</span><span class="n">ajd</span><span class="o">-</span><span class="mi">11118</span><span class="o">-</span><span class="n">g968f3e3</span><span class="w"> </span><span class="p">(</span><span class="n">ajd</span><span class="nv">@ka1</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">gcc</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="mf">5.2.1</span><span class="w"> </span><span class="mi">20150930</span><span class="w"> </span><span class="p">(</span><span class="n">GCC</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="n">#1</span><span class="w"> </span><span class="n">SMP</span><span class="w"> </span><span class="n">Tue</span><span class="w"> </span><span class="n">Mar</span><span class="w"> </span><span class="mi">22</span><span class="w"> </span><span class="mi">17</span><span class="err">:</span><span class="mi">01</span><span class="err">:</span><span class="mi">58</span><span class="w"> </span><span class="n">AEDT</span><span class="w"> </span><span class="mi">2016</span>
<span class="n">enter</span><span class="w"> </span><span class="vm">?</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">help</span>
<span class="mi">7</span><span class="err">:</span><span class="n">mon</span><span class="o">></span>
</code></pre></div>
<p>From <code>xmon</code>, simply type <code>dl</code> to dump out the kernel log. If you'd like to page through the log rather than dump the entire thing at once, use <code>#<n></code> to split it into groups of <code>n</code> lines.</p>
<p>Until recently, it wasn't as easy to extract the OPAL log without knowing magic offsets. A couple of months ago, I was debugging a nasty CAPI issue and got rather frustrated by this, so one day when I had a couple of hours free I <a href="http://patchwork.ozlabs.org/patch/581775/">refactored</a> the existing sysfs interface and <a href="http://patchwork.ozlabs.org/patch/581774/">added</a> the <code>do</code> command to <code>xmon</code>. These patches will be included from kernel 4.6-rc1 onwards.</p>
<p>When you're done, <code>x</code> will attempt to recover the machine and continue, <code>zr</code> will reboot, and <code>zh</code> will halt.</p>
<h2>From the FSP</h2>
<p>Sometimes, not even <code>xmon</code> will help you. In production environments, you're not generally going to start a debugger every time you have an incident. Additionally, a serious hardware error can cause a 'checkstop', which completely halts the system. (Thankfully, end users don't see this very often, but kernel developers, on the other hand...)</p>
<p>This is where the Flexible Service Processor, or FSP, comes in. The FSP is an IBM-developed baseboard management controller used on most IBM-branded Power Systems machines, and is responsible for a whole range of things, including monitoring system health. Among its many capabilities, the FSP can automatically take "system dumps" when fatal errors occur, capturing designated regions of memory for later debugging. System dumps can be configured and triggered via the FSP's web interface, which is beyond the scope of this post but is <a href="https://www.ibm.com/support/knowledgecenter/POWER8/p8ha5/mainstoragedump.htm?cp=POWER8%2F1-3-14-2">documented</a> in IBM Power Systems user manuals.</p>
<p>How does the FSP know what to capture? As it turns out, skiboot (the firmware which implements OPAL) maintains a <a href="https://github.com/open-power/skiboot/blob/master/hw/fsp/fsp-mdst-table.c">Memory Dump Source Table</a> which tells the FSP which memory regions to dump. MDST updates are recorded in the OPAL log:</p>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="mi">2690088026</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span><span class="w"> </span><span class="n">MDST</span><span class="o">:</span><span class="w"> </span><span class="n">Max</span><span class="w"> </span><span class="n">entries</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">MDST</span><span class="w"> </span><span class="n">table</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mi">256</span>
<span class="p">[</span><span class="mi">2690090666</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span><span class="w"> </span><span class="n">MDST</span><span class="o">:</span><span class="w"> </span><span class="n">Addr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mh">0x31000000</span><span class="w"> </span><span class="p">[</span><span class="n">size</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mh">0x100000</span><span class="w"> </span><span class="n">bytes</span><span class="p">]</span><span class="w"> </span><span class="n">added</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">MDST</span><span class="w"> </span><span class="n">table</span><span class="p">.</span>
<span class="p">[</span><span class="mi">2690093767</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span><span class="w"> </span><span class="n">MDST</span><span class="o">:</span><span class="w"> </span><span class="n">Addr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mh">0x31100000</span><span class="w"> </span><span class="p">[</span><span class="n">size</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mh">0x100000</span><span class="w"> </span><span class="n">bytes</span><span class="p">]</span><span class="w"> </span><span class="n">added</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">MDST</span><span class="w"> </span><span class="n">table</span><span class="p">.</span>
<span class="p">[</span><span class="mi">2750378890</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span><span class="w"> </span><span class="n">MDST</span><span class="o">:</span><span class="w"> </span><span class="n">Table</span><span class="w"> </span><span class="n">updated</span><span class="p">.</span>
<span class="p">[</span><span class="mi">11199672771</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span><span class="w"> </span><span class="n">MDST</span><span class="o">:</span><span class="w"> </span><span class="n">Addr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mh">0x1fff772780</span><span class="w"> </span><span class="p">[</span><span class="n">size</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mh">0x200000</span><span class="w"> </span><span class="n">bytes</span><span class="p">]</span><span class="w"> </span><span class="n">added</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">MDST</span><span class="w"> </span><span class="n">table</span><span class="p">.</span>
<span class="p">[</span><span class="mi">11215193760</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span><span class="w"> </span><span class="n">MDST</span><span class="o">:</span><span class="w"> </span><span class="n">Table</span><span class="w"> </span><span class="n">updated</span><span class="p">.</span>
<span class="p">[</span><span class="mi">28031311971</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span><span class="w"> </span><span class="n">MDST</span><span class="o">:</span><span class="w"> </span><span class="n">Table</span><span class="w"> </span><span class="n">updated</span><span class="p">.</span>
<span class="p">[</span><span class="mi">28411709421</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span><span class="w"> </span><span class="n">MDST</span><span class="o">:</span><span class="w"> </span><span class="n">Addr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mh">0x1fff830000</span><span class="w"> </span><span class="p">[</span><span class="n">size</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mh">0x100000</span><span class="w"> </span><span class="n">bytes</span><span class="p">]</span><span class="w"> </span><span class="n">added</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">MDST</span><span class="w"> </span><span class="n">table</span><span class="p">.</span>
<span class="p">[</span><span class="mi">28417251110</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span><span class="w"> </span><span class="n">MDST</span><span class="o">:</span><span class="w"> </span><span class="n">Table</span><span class="w"> </span><span class="n">updated</span><span class="p">.</span>
</code></pre></div>
<p>In the above log, we see four entries: the skiboot/OPAL log, the <a href="https://github.com/open-power/hostboot">hostboot</a> runtime log, the petitboot Linux kernel log (which doesn't make it into the final dump) and the real Linux kernel log. skiboot obviously adds the OPAL and hostboot logs to the MDST early in boot, but it also exposes the <a href="https://github.com/open-power/skiboot/blob/master/doc/opal-api/opal-register-dump-region-101.txt"><code>OPAL_REGISTER_DUMP_REGION</code></a> call which can be used by the operating system to register additional regions. Linux uses this to <a href="https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/powernv/opal.c#L608">register the kernel log buffer</a>. If you're a kernel developer, you could potentially use the OPAL call to register your own interesting bits of memory.</p>
<p>So, the MDST is all set up, we go about doing our business, and suddenly we checkstop. The FSP does its sysdump magic and a few minutes later it reboots the system. What now?</p>
<ul>
<li>
<p>After we come back up, the FSP notifies OPAL that a new dump is available. Linux exposes the dump to userspace under <code>/sys/firmware/opal/dump/</code>.</p>
</li>
<li>
<p><a href="https://sourceforge.net/projects/linux-diag/files/ppc64-diag/">ppc64-diag</a> is a suite of utilities that assist in manipulating FSP dumps, including the <code>opal_errd</code> daemon. <code>opal_errd</code> monitors new dumps and saves them in <code>/var/log/dump/</code> for later analysis.</p>
</li>
<li>
<p><code>opal-dump-parse</code> (also in the <code>ppc64-diag</code> suite) can be used to extract the sections we care about from the dump:</p>
<div class="highlight"><pre><span></span><code><span class="n">root</span><span class="err">@</span><span class="n">p86</span><span class="p">:</span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">dump</span><span class="c1"># opal-dump-parse -l SYSDUMP.842EA8A.00000001.20160322063051 </span>
<span class="o">|---------------------------------------------------------|</span>
<span class="o">|</span><span class="n">ID</span><span class="w"> </span><span class="n">SECTION</span><span class="w"> </span><span class="n">SIZE</span><span class="o">|</span>
<span class="o">|---------------------------------------------------------|</span>
<span class="o">|</span><span class="mi">1</span><span class="w"> </span><span class="n">Opal</span><span class="o">-</span><span class="nb">log</span><span class="w"> </span><span class="mi">1048576</span><span class="o">|</span>
<span class="o">|</span><span class="mi">2</span><span class="w"> </span><span class="n">HostBoot</span><span class="o">-</span><span class="n">Runtime</span><span class="o">-</span><span class="nb">log</span><span class="w"> </span><span class="mi">1048576</span><span class="o">|</span>
<span class="o">|</span><span class="mi">128</span><span class="w"> </span><span class="n">printk</span><span class="w"> </span><span class="mi">1048576</span><span class="o">|</span>
<span class="o">|---------------------------------------------------------|</span>
<span class="n">List</span><span class="w"> </span><span class="n">completed</span>
<span class="n">root</span><span class="err">@</span><span class="n">p86</span><span class="p">:</span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">dump</span><span class="c1"># opal-dump-parse -s 1 SYSDUMP.842EA8A.00000001.20160322063051 </span>
<span class="n">Captured</span><span class="w"> </span><span class="nb">log</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="n">Opal</span><span class="o">-</span><span class="nb">log</span><span class="o">.</span><span class="mi">842</span><span class="n">EA8A</span><span class="o">.</span><span class="mf">00000001.20160322063051</span>
<span class="n">root</span><span class="err">@</span><span class="n">p86</span><span class="p">:</span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">dump</span><span class="c1"># opal-dump-parse -s 2 SYSDUMP.842EA8A.00000001.20160322063051 </span>
<span class="n">Captured</span><span class="w"> </span><span class="nb">log</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="n">HostBoot</span><span class="o">-</span><span class="n">Runtime</span><span class="o">-</span><span class="nb">log</span><span class="o">.</span><span class="mi">842</span><span class="n">EA8A</span><span class="o">.</span><span class="mf">00000001.20160322063051</span>
<span class="n">root</span><span class="err">@</span><span class="n">p86</span><span class="p">:</span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">dump</span><span class="c1"># opal-dump-parse -s 128 SYSDUMP.842EA8A.00000001.20160322063051 </span>
<span class="n">Captured</span><span class="w"> </span><span class="nb">log</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="n">printk</span><span class="o">.</span><span class="mi">842</span><span class="n">EA8A</span><span class="o">.</span><span class="mf">00000001.20160322063051</span>
</code></pre></div>
</li>
</ul>
<p>There's various other types of dumps and logs that I won't get into here. I'm probably obliged to say that if you're having problems out in the wild, you should probably contact your friendly local IBM Service Representative...</p>
<h2>Acknowledgements</h2>
<p>Thanks to <a href="https://flamingspork.com">Stewart Smith</a> for pointing me in the right direction regarding FSP sysdumps and related tools.</p></div>
</article>
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/03/22/the-elegance-of-the-plaintext-patch/">The Elegance of the Plaintext Patch</a>
</h1>
<p class="meta">
<time datetime="2016-03-22T13:53:00+11:00" pubdate>Tue 22 March 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/russell-currey.html">Russell Currey</a>
</span>
</span>
<time datetime="2016-03-22T13:53:00+11:00" pubdate>Tue 22 March 2016</time></div>
<div class="entry-content"><p>I've only been working on the Linux kernel for a few months. Before that, I worked with proprietary source control at work and common tools like GitHub at home. The concept of the mailing list seemed obtuse to me. If I noticed a problem with some program, I'd be willing to open an issue on GitHub but not to send an email to a mailing list. Who still uses those, anyway?</p>
<p>Starting out with the kernel meant I had to figure this email thing out. <code>git format-patch</code> and <code>git send-email</code> take most of the pain out of formatting and submitting a patch, which is nice. The patch files generated by <code>format-patch</code> open nicely in Emacs by default, showing all whitespace and letting you pick up any irregularities. <code>send-email</code> means you can send it to yourself or a friend first, finding anything that looks stupid before being exposed to the public.</p>
<p>And then what? You've sent an email. It gets sent to hundreds or thousands of people. Nowhere near that many will read it. Some might miss it due to their mail server going down, or the list flagging your post as spam, or requiring moderation. Some recipients will be bots that archive mail on the list, or publish information about the patch. If you haven't formatted it correctly, someone will let you know quickly. If your patch is important or controversial, you'll have all sorts of responses. If your patch is small or niche, you might not ever hear anything back.</p>
<p>I remember when I sent my first patch. I was talking to a former colleague who didn't understand the patch/mailing list workflow at all. I sent him a link to my patch on a mail archive. I explained it like a pull request - here's my code, you can find the responses. What's missing from a GitHub-esque pull request? We don't know what tests it passed. We don't know if it's been merged yet, or if the maintainer has looked at it. It takes a bit of digging around to find out who's commented on it. If it's part of a series, that's awkward to find out as well. What about revisions of a series? That's another pain point.</p>
<p>Luckily, these problems do have solutions. <a href="http://jk.ozlabs.org/projects/patchwork/">Patchwork</a>, written by fellow OzLabs member <a href="http://jk.ozlabs.org">Jeremy Kerr</a>, changes the way we work with patches. Project maintainers rely on Pathwork instances, such as <a href="https://patchwork.ozlabs.org">https://patchwork.ozlabs.org</a>, for their day-to-day workflow: tagging reviewers, marking the status of patches, keeping track of tests, acks, reviews and comments in one place. Missing from this picture is support for series and revisions, which is a feature that's being developed by the <a href="https://www.freedesktop.org/wiki/">freedesktop</a> project. You can check out their changes in action <a href="https://patchwork.freedesktop.org">here</a>.</p>
<p>So, Patchwork helps patches and email catch up to what GitHub has in terms of ease of information. We're still missing testing and other hooks. What about review? What can we do with email, compared to GitHub and the like?</p>
<p>In my opinion, the biggest feature of email is the ease of review. Just reply inline and you're done. There's inline commenting on GitHub and GitLab, which works well but is a bit tacky, people commenting on the same thing overlap and conflict, each comment generates a notification (which can be an email until you turn that off). Plus, since it's email, it's really easy to bring in additional people to the conversation as necessary. If there's a super lengthy technical discussion in the kernel, it might just take Linus to resolve.</p>
<p>There are alternatives to just replying to email, too, such as <a href="https://www.gerritcodereview.com/">Gerrit</a>. Gerrit's pretty popular, and has a huge amount of features. I understand why people use it, though I'm not much of a fan. Reason being, it doesn't add to the email workflow, it replaces it. Plaintext email is supported on pretty much any device, with a bunch of different programs. From the goals of Patchwork: "patchwork should supplement mailing lists, not replace them".</p>
<p>Linus Torvalds famously explained why he prefers email over GitHub pull requests <a href="https://github.com/torvalds/linux/pull/17">here</a>, using <a href="https://groups.google.com/forum/#!topic/linux.kernel/w957vpu3PPU">this</a> pull request from Ben Herrenschmidt as an example of why git's own pull request format is superior to that of GitHub. Damien Lespiau, who is working on the freedesktop Patchwork fork, <a href="http://damien.lespiau.name/2016/02/augmenting-mailing-lists-with-patchwork.html">outlines on his blog</a> all the issues he has with mailing list workflows and why he thinks mailing lists are a relic of the past. His work on Patchwork has gone a long way to help fix those problems, however I don't think mailing lists are outdated and superceded, I think they are timeless. They are a technology-agnostic, simple and free system that will still be around if GitHub dies or alienates its community.</p>
<p>That said, there's still the case of the missing features. What about automated testing? What about developer feedback? What about making a maintainer's life easier? We've been working on improving these issues, and I'll outline how we're approaching them in a future post.</p></div>
</article>
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/03/21/no-network-for-you/">No Network For You</a>
</h1>
<p class="meta">
<time datetime="2016-03-21T15:23:00+11:00" pubdate>Mon 21 March 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/samuel-mendoza-jonas.html">Samuel Mendoza-Jonas</a>
</span>
</span>
<time datetime="2016-03-21T15:23:00+11:00" pubdate>Mon 21 March 2016</time></div>
<div class="entry-content"><p>In POWER land <a href="https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface">IPMI</a> is mostly known as the method to access the machine's console and start interacting with Petitboot. However it also has a plethora of other features, handily described in the 600ish page <a href="http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html">IPMI specification</a> (which you can go read yourself).</p>
<p>One especially relevant feature to Petitboot however is the 'chassis bootdev' command, which you can use to tell Petitboot to ignore any existing boot order, and only consider boot options of the type you specify (eg. 'network', 'disk', or 'setup' to not boot at all). Support for this has been in Petitboot for a while and should work on just about any machine you can get your hands on.</p>
<h2>Network Overrides</h2>
<p>Over in OpenPOWER<sup>1</sup> land however, someone took this idea and pushed it further - why not allow the network configuration to be overwritten too? This isn't in the IPMI spec, but if you cast your gaze down to page 398 where the spec lays out the entire format of the IPMI request, there is a certain field named "OEM Parameters". This is an optional amount of space set aside for whatever you like, which in this case is going to be data describing an override of the network config.</p>
<p>This allows a user to tell Petitboot over IPMI to either;</p>
<ul>
<li>Disable the network completely,</li>
<li>Set a particular interface to use DHCP, or</li>
<li>Set a particular interface to use a specific static configuration.</li>
</ul>
<p>Any of these options will cause any existing network configurations to be ignored.</p>
<h2>Building the Request</h2>
<p>Since this is an OEM-specific command, your average ipmitool package isn't going to have a nice way of making this request, such as 'chassis bootdev network'. Rather you need to do something like this:</p>
<div class="highlight"><pre><span></span><code><span class="n">ipmitool</span><span class="w"> </span><span class="o">-</span><span class="n">I</span><span class="w"> </span><span class="n">lanplus</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="o">$</span><span class="n">yourbmc</span><span class="w"> </span><span class="o">-</span><span class="n">U</span><span class="w"> </span><span class="o">$</span><span class="n">user</span><span class="w"> </span><span class="o">-</span><span class="n">P</span><span class="w"> </span><span class="o">$</span><span class="k">pass</span><span class="w"> </span><span class="n">raw</span><span class="w"> </span><span class="mh">0x00</span><span class="w"> </span><span class="mh">0x08</span><span class="w"> </span><span class="mh">0x61</span><span class="w"> </span><span class="mh">0x80</span><span class="w"> </span><span class="mh">0x21</span><span class="w"> </span><span class="mh">0x70</span><span class="w"> </span><span class="mh">0x62</span><span class="w"> </span><span class="mh">0x21</span><span class="w"> </span><span class="mh">0x00</span><span class="w"> </span><span class="mh">0x01</span><span class="w"> </span><span class="mh">0x06</span><span class="w"> </span><span class="mh">0x04</span><span class="w"> </span><span class="mh">0xf4</span><span class="w"> </span><span class="mh">0x52</span><span class="w"> </span><span class="mh">0x14</span><span class="w"> </span><span class="mh">0xf3</span><span class="w"> </span><span class="mh">0x01</span><span class="w"> </span><span class="mh">0xdf</span><span class="w"> </span><span class="mh">0x00</span><span class="w"> </span><span class="mh">0x01</span><span class="w"> </span><span class="mh">0x0a</span><span class="w"> </span><span class="mh">0x3d</span><span class="w"> </span><span class="mh">0xa1</span><span class="w"> </span><span class="mh">0x42</span><span class="w"> </span><span class="mh">0x10</span><span class="w"> </span><span class="mh">0x0a</span><span class="w"> </span><span class="mh">0x3d</span><span class="w"> </span><span class="mh">0x2</span><span class="w"> </span><span class="mh">0x1</span>
</code></pre></div>
<p>Horrific right? In the near future the Petitboot tree will include a helper program to format this request for you, but in the meantime (and for future reference), lets lay out how to put this together:</p>
<div class="highlight"><pre><span></span><code><span class="nx">Specify</span><span class="w"> </span><span class="nx">the</span><span class="w"> </span><span class="s">"chassis bootdev"</span><span class="w"> </span><span class="nx">command</span><span class="p">,</span><span class="w"> </span><span class="nx">field</span><span class="w"> </span><span class="mi">96</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="nx">field</span><span class="w"> </span><span class="mi">1</span><span class="p">:</span>
<span class="w"> </span><span class="mh">0x0</span><span class="mi">0</span><span class="w"> </span><span class="mh">0x0</span><span class="mi">8</span><span class="w"> </span><span class="mh">0x6</span><span class="mi">1</span><span class="w"> </span><span class="mh">0x8</span><span class="mi">0</span>
<span class="nx">Unique</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="nx">that</span><span class="w"> </span><span class="nx">Petitboot</span><span class="w"> </span><span class="nx">recognises</span><span class="p">:</span>
<span class="w"> </span><span class="mh">0x2</span><span class="mi">1</span><span class="w"> </span><span class="mh">0x7</span><span class="mi">0</span><span class="w"> </span><span class="mh">0x6</span><span class="mi">2</span><span class="w"> </span><span class="mh">0x2</span><span class="mi">1</span>
<span class="nx">Version</span><span class="w"> </span><span class="nx">field</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="w"> </span><span class="mh">0x0</span><span class="mi">0</span><span class="w"> </span><span class="mh">0x0</span><span class="mi">1</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="p">..</span>
<span class="nx">Size</span><span class="w"> </span><span class="nx">of</span><span class="w"> </span><span class="nx">the</span><span class="w"> </span><span class="nx">hardware</span><span class="w"> </span><span class="nx">address</span><span class="w"> </span><span class="p">(</span><span class="mi">6</span><span class="p">):</span>
<span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="mh">0x0</span><span class="mi">6</span><span class="w"> </span><span class="p">..</span>
<span class="nx">Size</span><span class="w"> </span><span class="nx">of</span><span class="w"> </span><span class="nx">the</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">address</span><span class="w"> </span><span class="p">(</span><span class="nx">IPv4</span><span class="o">/</span><span class="nx">IPv6</span><span class="p">):</span>
<span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="mh">0x0</span><span class="mi">4</span>
<span class="nx">Hardware</span><span class="w"> </span><span class="p">(</span><span class="nx">MAC</span><span class="p">)</span><span class="w"> </span><span class="nx">address</span><span class="p">:</span>
<span class="w"> </span><span class="mh">0xf</span><span class="mi">4</span><span class="w"> </span><span class="mh">0x5</span><span class="mi">2</span><span class="w"> </span><span class="mh">0x1</span><span class="mi">4</span><span class="w"> </span><span class="mh">0xf</span><span class="mi">3</span>
<span class="w"> </span><span class="mh">0x0</span><span class="mi">1</span><span class="w"> </span><span class="mh">0xd</span><span class="nx">f</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="p">..</span>
<span class="err">'</span><span class="nx">Ignore</span><span class="w"> </span><span class="nx">flag</span><span class="err">'</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="nx">DHCP</span><span class="o">/</span><span class="nx">Static</span><span class="w"> </span><span class="nx">flag</span><span class="w"> </span><span class="p">(</span><span class="nx">DHCP</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span>
<span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="mh">0x0</span><span class="mi">0</span><span class="w"> </span><span class="mh">0x0</span><span class="mi">1</span>
<span class="p">(</span><span class="nx">Below</span><span class="w"> </span><span class="nx">fields</span><span class="w"> </span><span class="nx">only</span><span class="w"> </span><span class="nx">required</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">setting</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="nx">static</span><span class="w"> </span><span class="nx">IP</span><span class="p">)</span>
<span class="nx">IP</span><span class="w"> </span><span class="nx">Address</span><span class="p">:</span>
<span class="w"> </span><span class="mh">0x0</span><span class="nx">a</span><span class="w"> </span><span class="mh">0x3</span><span class="nx">d</span><span class="w"> </span><span class="mh">0xa</span><span class="mi">1</span><span class="w"> </span><span class="mh">0x4</span><span class="mi">2</span>
<span class="nx">Subnet</span><span class="w"> </span><span class="nx">Mask</span><span class="w"> </span><span class="p">(</span><span class="nx">eg</span><span class="p">,</span><span class="w"> </span><span class="o">/</span><span class="mi">16</span><span class="p">):</span>
<span class="w"> </span><span class="mh">0x1</span><span class="mi">0</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="p">..</span>
<span class="nx">Gateway</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">Address</span><span class="p">:</span>
<span class="w"> </span><span class="p">..</span><span class="w"> </span><span class="mh">0x0</span><span class="nx">a</span><span class="w"> </span><span class="mh">0x3</span><span class="nx">d</span><span class="w"> </span><span class="mh">0x0</span><span class="mi">2</span>
<span class="w"> </span><span class="mh">0x0</span><span class="mi">1</span>
</code></pre></div>
<p>Clearing a network override is as simple as making a request empty aside from the header:</p>
<div class="highlight"><pre><span></span><code><span class="mf">0</span><span class="n">x00</span><span class="w"> </span><span class="mf">0</span><span class="n">x08</span><span class="w"> </span><span class="mf">0</span><span class="n">x61</span><span class="w"> </span><span class="mf">0</span><span class="n">x80</span><span class="w"> </span><span class="mf">0</span><span class="n">x21</span><span class="w"> </span><span class="mf">0</span><span class="n">x70</span><span class="w"> </span><span class="mf">0</span><span class="n">x62</span><span class="w"> </span><span class="mf">0</span><span class="n">x21</span><span class="w"> </span><span class="mf">0</span><span class="n">x00</span><span class="w"> </span><span class="mf">0</span><span class="n">x01</span><span class="w"> </span><span class="mf">0</span><span class="n">x00</span><span class="w"> </span><span class="mf">0</span><span class="n">x00</span>
</code></pre></div>
<p>You can also read back the request over IPMI with this request:</p>
<div class="highlight"><pre><span></span><code><span class="mf">0</span><span class="n">x00</span><span class="w"> </span><span class="mf">0</span><span class="n">x09</span><span class="w"> </span><span class="mf">0</span><span class="n">x61</span><span class="w"> </span><span class="mf">0</span><span class="n">x00</span><span class="w"> </span><span class="mf">0</span><span class="n">x00</span>
</code></pre></div>
<p>That's it! Ideally this is something you would be scripting rather than bashing out on the keyboard - the main use case at the moment is as a way to force a machine to netboot against a known good source, rather than whatever may be available on its other interfaces.</p>
<p>[1] The reason this is only available on OpenPOWER machines at the moment is that support for the IPMI command itself depends on the BMC firmware, and non-OpenPOWER machines use an FSP which is a different platform.</p></div>
</article>
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/03/15/and-now-for-something-completely-different-approximate-computing/">And now for something completely different: approximate computing</a>
</h1>
<p class="meta">
<time datetime="2016-03-15T11:30:00+11:00" pubdate>Tue 15 March 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/daniel-axtens.html">Daniel Axtens</a>
</span>
</span>
<time datetime="2016-03-15T11:30:00+11:00" pubdate>Tue 15 March 2016</time></div>
<div class="entry-content"><p>In early February I had the opportunity to go the the NICTA Systems Summer School, where Cyril and I were invited to represent IBM. There were a number of excellent talks across a huge range of systems related subjects, but the one that has stuck with me the most was a talk given by <a href="http://homes.cs.washington.edu/~luisceze/">Luis Ceze</a> on a topic called approximate computing. So here, in hopes that you too find it interesting, is a brief run-down on what I learned.</p>
<p>Approximate computing is fundamentally about trading off accuracy for something else - often speed or power consumption. Initially this sounded like a very weird proposition: computers do things like 'running your operating system' and 'reading from and writing to disks': things you need to always be absolutely correct if you want anything vaguely resembling reliability. It turns out that this is actually not as big a roadblock as I had assumed - you can work around it fairly easily.</p>
<p>The model proposed for approximate computing is as follows. You divide your computation up into two classes: 'precise', and 'approximate'. You use 'precise' computations when you need to get exact answers: so for example if you are constructing a JPEG file, you want the JPEG header to be exact. Then you have approximate computations: so for example the contents of your image can be approximate.</p>
<p>For correctness, you have to establish some boundaries: you say that precise data can be used in approximate calculations, but that approximate data isn't allowed to cross back over and pollute precise calculations. This, while intuitively correct, poses some problems in practise: when you want to write out your approximate JPEG data, you need an operation that allows you to 'bless' (or in their terms 'endorse') some approximate data so it can be used in the precise file system operations.</p>
<p>In the talk we were shown an implementation of this model in Java, called <a href="http://sampa.cs.washington.edu/research/approximation/enerj.html">EnerJ</a>. EnerJ allows you to label variables with either <code>@Precise</code> if you're dealing with precise data, or <code>@Approx</code> if you're dealing with approximate data. The compiler was modified so that it would do all sorts of weird things when it knew it was dealing with approximate data: for example, drop loop iterations entirely, do things in entirely non-determistic ways - all sorts of fun stuff. It turns out this works surprisingly well.</p>
<p>However, the approximate computing really shines when you can bring it all the way down to the hardware level. The first thing they tried was a CPU with both 'approximate' and precise execution engines, but this turned out not to have the power savings hoped for. What seemed to work really well was a model where some approximate calculations could be identified ahead of time, and then replaced with neural networks in hardware. These neural networks approximated the calculations, but did so at significantly lower power levels. This sounded like a really promising concept, and it will be interesting to see if this goes anywhere over the next few years.</p>
<p>There's a lot of work evaluating the quality of the approximate result, for cases where the set of inputs is known, and when the inputs is not known. This is largely beyond my understanding, so I'll simply refer you to some of the papers <a href="http://sampa.cs.washington.edu/research/approximation/enerj.html">listed on the website</a>.</p>
<p>The final thing covered in the talk was bringing approximate computing into current paradigms by just being willing to accept higher user-visible error rates. For example, they hacked up a network stack to accept packets with invalid checksums. This has had mixed results so far. A question I had (but didn't get around to asking!) would be whether the mathematical properties of checksums (i.e. that they can correct a certain number of bit errors) could be used to correct some of the errors, rather than just accepting/rejecting them blindly. Perhaps by first attempting to correct errors using the checksums, we will be able to fix the simpler errors, reducing the error rate visible to the user.</p>
<p>Overall, I found the NICTA Systems Summer School to be a really interesting experience (and I hope to blog more about it soon). If you're a university student in Australia, or an academic, see if you can make it in 2017!</p></div>
</article>
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/03/15/linuxconfau-2016-a-set-of-brief-thoughts/">linux.conf.au 2016: A set of brief thoughts</a>
</h1>
<p class="meta">
<time datetime="2016-03-15T11:30:00+11:00" pubdate>Tue 15 March 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/daniel-axtens.html">Daniel Axtens</a>
</span>
</span>
<time datetime="2016-03-15T11:30:00+11:00" pubdate>Tue 15 March 2016</time></div>
<div class="entry-content"><p>Recently most of us attended LCA2016. This is one set of reflections on what we heard and what we've thought since. (Hopefully not the only set of reflections that will be posted on this blog either!)</p>
<p>LCA was 2 days of miniconferences plus 3 days of talks. Here, I've picked some of the more interesting talks I attended, and I've written down some thoughts. If you find the thoughts interesting, you can click through and watch the whole talk video, because LCA is awesome like that.</p>
<h4>Life is better with Rust's community automation</h4>
<p><a href="https://www.youtube.com/watch?v=dIageYT0Vgg">This talk</a> is probably the one that's had the biggest impact on our team so far. We were really impressed by the community automation that Rust has: the way they can respond to pull requests from new community members in a way that lets them keep their code quality high and be nice to everyone at the same time.</p>
<p>The system that they've developed is fascinating (and seems fantastic). However, their system uses pull requests, while we use mailing lists. Pull requests are easy, because github has good hook support, but how do we link mailing lists to an automatic test system?</p>
<p>As it turns out, this is something we're working on: we already have <a href="http://patchwork.ozlabs.org/">Patchwork</a>, and <a href="https://openpower.xyz/">Jenkins</a>: how do we link them? We have something brewing, which we'll open source real soon now - stay tuned!</p>
<h4>Usable formal methods - are we there yet?</h4>
<p>I liked <a href="https://www.youtube.com/watch?v=RxHjhBVOCSU">this talk</a>, as I have a soft spot for formal methods (as I have a soft spot for maths). It covers applying a bunch of static analysis and some of the less intrusive formal methods (in particular <a href="http://www.cprover.org/cbmc/">cbmc</a>) to an operating system kernel. They were looking at eChronos rather than Linux, but it's still quite an interesting set of results.</p>
<p>We've also tried to increase our use of static analysis, which has already found a <a href="http://patchwork.ozlabs.org/patch/580629/">real bug</a>. We're hoping to scale this up, especially the use of sparse and cppcheck, but we're a bit short on developer cycles for it at the moment.</p>
<h4>Adventures in OpenPower Firmware</h4>
<p>Stewart Smith - another OzLabber - gave <a href="https://www.youtube.com/watch?v=a4XGvssR-ag">this talk</a> about, well, OpenPOWER firmware. This is a large part of our lives in OzLabs, so it's a great way to get a picture of what we do each day. It's also a really good explanation of the open source stack we have: a POWER8 CPU runs open-source from the first cycle.</p>
<h4>What Happens When 4096 Cores <code>All Do synchronize_rcu_expedited()</code>?</h4>
<p>Paul McKenney is a parallel programming genius - he literally <a href="https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html">'wrote the book'</a> (or at least, wrote <em>a</em> book!) on it. <a href="https://www.youtube.com/watch?v=1nfpjHTWaUc">His talk</a> is - as always - a brain-stretching look at parallel programming within the RCU subsystem of the Linux kernel. In particular, the tree structure for locking that he presents is really interesting and quite a clever way of scaling what at first seems to be a necessarily global lock.</p>
<p>I'd also really recommed <a href="https://www.youtube.com/watch?v=tFmajPt0_hI">RCU Mutation Testing</a>, from the kernel miniconf, also by Paul.</p>
<h4>What I've learned as the kernel docs maintainer</h4>
<p>As an extra bonus: I mention <a href="https://www.youtube.com/watch?v=gsJXf6oSbAE">this talk</a>, just to say "why on earth have we still not fixed the Linux kernel <a href="https://www.kernel.org/doc/linux/README">README</a>"?!!?</p></div>
</article>
<div class="pagination">
<a class="prev" href="https://sthbrx.github.io/index11.html">← Older</a>
<a class="next" href="https://sthbrx.github.io/index9.html">Newer →</a>
<br />
</div></div>
<aside class="sidebar">
<section>
<h1>Recent Posts</h1>
<ul id="recent_posts">
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/08/07/going-out-on-a-limb-efficient-elliptic-curve-arithmetic-in-openssl/">Going out on a Limb: Efficient Elliptic Curve Arithmetic in OpenSSL</a>
</li>
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/08/04/quirks-of-parsing-ssh-configs/">Quirks of parsing SSH configs</a>
</li>
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/04/05/detecting-rootless-docker/">Detecting rootless Docker</a>
</li>
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/04/04/dumb-bugs-the-pci-device-that-wasnt/">Dumb bugs: the PCI device that wasn't</a>
</li>
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/03/24/dumb-bugs-when-a-date-breaks-booting-the-kernel/">Dumb bugs: When a date breaks booting the kernel</a>
</li>
</ul>
</section>
<section>
<h1>Categories</h1>
<ul id="recent_posts">
<li><a href="https://sthbrx.github.io/category/cryptography.html">Cryptography</a></li>
<li><a href="https://sthbrx.github.io/category/development.html">Development</a></li>
<li><a href="https://sthbrx.github.io/category/education.html">Education</a></li>
<li><a href="https://sthbrx.github.io/category/openpower.html">OpenPOWER</a></li>
<li><a href="https://sthbrx.github.io/category/performance.html">Performance</a></li>
<li><a href="https://sthbrx.github.io/category/petitboot.html">Petitboot</a></li>
<li><a href="https://sthbrx.github.io/category/snowpatch.html">snowpatch</a></li>
<li><a href="https://sthbrx.github.io/category/virtualisation-and-emulation.html">Virtualisation and Emulation</a></li>
</ul>
</section>
<section>
<h1>Tags</h1>
<a href="https://sthbrx.github.io/tag/ssh.html">ssh</a>, <a href="https://sthbrx.github.io/tag/docker.html">Docker</a>, <a href="https://sthbrx.github.io/tag/syzkaller.html">syzkaller</a>, <a href="https://sthbrx.github.io/tag/linux.html">linux</a>, <a href="https://sthbrx.github.io/tag/power8.html">power8</a>, <a href="https://sthbrx.github.io/tag/distro.html">distro</a>, <a href="https://sthbrx.github.io/tag/kernel.html">kernel</a>, <a href="https://sthbrx.github.io/tag/hardening.html">hardening</a>, <a href="https://sthbrx.github.io/tag/testing.html">testing</a>, <a href="https://sthbrx.github.io/tag/conferences.html">conferences</a>, <a href="https://sthbrx.github.io/tag/instruction-set-architecture.html">Instruction Set Architecture</a>, <a href="https://sthbrx.github.io/tag/openpower.html">openpower</a>, <a href="https://sthbrx.github.io/tag/firmware.html">firmware</a>, <a href="https://sthbrx.github.io/tag/goodposts.html">goodposts</a>, <a href="https://sthbrx.github.io/tag/realcontent.html">realcontent</a>, <a href="https://sthbrx.github.io/tag/madposting.html">madposting</a>, <a href="https://sthbrx.github.io/tag/op-test.html">op-test</a>, <a href="https://sthbrx.github.io/tag/qemu.html">qemu</a>, <a href="https://sthbrx.github.io/tag/pci.html">pci</a>, <a href="https://sthbrx.github.io/tag/sparseposting.html">sparseposting</a>, <a href="https://sthbrx.github.io/tag/petitboot.html">petitboot</a>, <a href="https://sthbrx.github.io/tag/security.html">security</a>, <a href="https://sthbrx.github.io/tag/vscode.html">vscode</a>, <a href="https://sthbrx.github.io/tag/code.html">code</a>, <a href="https://sthbrx.github.io/tag/openbmc.html">openbmc</a>, <a href="https://sthbrx.github.io/tag/ipmi.html">ipmi</a>, <a href="https://sthbrx.github.io/tag/opencapi.html">opencapi</a>, <a href="https://sthbrx.github.io/tag/openpower-summit.html">openpower summit</a>, <a href="https://sthbrx.github.io/tag/easyposts.html">easyposts</a>, <a href="https://sthbrx.github.io/tag/linuxboot.html">linuxboot</a>, <a href="https://sthbrx.github.io/tag/google.html">google</a>, <a href="https://sthbrx.github.io/tag/intel.html">intel</a>, <a href="https://sthbrx.github.io/tag/osfc.html">osfc</a>, <a href="https://sthbrx.github.io/tag/shortposts.html">shortposts</a>, <a href="https://sthbrx.github.io/tag/facebook.html">facebook</a>, <a href="https://sthbrx.github.io/tag/performance.html">performance</a>, <a href="https://sthbrx.github.io/tag/phoronix.html">phoronix</a>, <a href="https://sthbrx.github.io/tag/benchmarks.html">benchmarks</a>, <a href="https://sthbrx.github.io/tag/stupid-ideas.html">stupid ideas</a>, <a href="https://sthbrx.github.io/tag/network.html">network</a>, <a href="https://sthbrx.github.io/tag/power.html">power</a>, <a href="https://sthbrx.github.io/tag/xdp.html">xdp</a>, <a href="https://sthbrx.github.io/tag/networking.html">networking</a>, <a href="https://sthbrx.github.io/tag/remoteposts.html">remoteposts</a>, <a href="https://sthbrx.github.io/tag/ceph.html">ceph</a>, <a href="https://sthbrx.github.io/tag/raid.html">raid</a>, <a href="https://sthbrx.github.io/tag/storage.html">storage</a>, <a href="https://sthbrx.github.io/tag/erasure.html">erasure</a>, <a href="https://sthbrx.github.io/tag/lustre.html">lustre</a>, <a href="https://sthbrx.github.io/tag/hpc.html">hpc</a>, <a href="https://sthbrx.github.io/tag/nvlink.html">nvlink</a>, <a href="https://sthbrx.github.io/tag/namd.html">namd</a>, <a href="https://sthbrx.github.io/tag/cuda.html">cuda</a>, <a href="https://sthbrx.github.io/tag/gpu.html">gpu</a>, <a href="https://sthbrx.github.io/tag/minsky.html">minsky</a>, <a href="https://sthbrx.github.io/tag/s822lc-for-hpc.html">S822LC for hpc</a>, <a href="https://sthbrx.github.io/tag/debug.html">debug</a>, <a href="https://sthbrx.github.io/tag/virtualisation.html">virtualisation</a>, <a href="https://sthbrx.github.io/tag/dmesg.html">dmesg</a>, <a href="https://sthbrx.github.io/tag/printk.html">printk</a>, <a href="https://sthbrx.github.io/tag/boot.html">boot</a>, <a href="https://sthbrx.github.io/tag/early.html">early</a>, <a href="https://sthbrx.github.io/tag/error.html">error</a>, <a href="https://sthbrx.github.io/tag/centos.html">centos</a>, <a href="https://sthbrx.github.io/tag/centos7.html">centos7</a>, <a href="https://sthbrx.github.io/tag/p8.html">p8</a>, <a href="https://sthbrx.github.io/tag/bmc.html">bmc</a>, <a href="https://sthbrx.github.io/tag/rhel.html">RHEL</a>, <a href="https://sthbrx.github.io/tag/skiroot.html">skiroot</a>, <a href="https://sthbrx.github.io/tag/devmapper.html">devmapper</a>, <a href="https://sthbrx.github.io/tag/lvm.html">lvm</a>, <a href="https://sthbrx.github.io/tag/cgroups.html">cgroups</a>, <a href="https://sthbrx.github.io/tag/numa.html">numa</a>, <a href="https://sthbrx.github.io/tag/development.html">Development</a>, <a href="https://sthbrx.github.io/tag/netboot.html">netboot</a>, <a href="https://sthbrx.github.io/tag/pxe.html">pxe</a>, <a href="https://sthbrx.github.io/tag/education.html">Education</a>, <a href="https://sthbrx.github.io/tag/work-experience.html">work experience</a>, <a href="https://sthbrx.github.io/tag/asm.html">asm</a>, <a href="https://sthbrx.github.io/tag/vdso.html">vdso</a>, <a href="https://sthbrx.github.io/tag/snowpatch.html">snowpatch</a>, <a href="https://sthbrx.github.io/tag/tools.html">tools</a>, <a href="https://sthbrx.github.io/tag/intern.html">intern</a>, <a href="https://sthbrx.github.io/tag/srop.html">SROP</a>, <a href="https://sthbrx.github.io/tag/mitigation.html">mitigation</a>, <a href="https://sthbrx.github.io/tag/double.html">double</a>, <a href="https://sthbrx.github.io/tag/float.html">float</a>, <a href="https://sthbrx.github.io/tag/hex.html">hex</a>, <a href="https://sthbrx.github.io/tag/debugging.html">debugging</a>, <a href="https://sthbrx.github.io/tag/skiboot.html">skiboot</a>, <a href="https://sthbrx.github.io/tag/opal.html">OPAL</a>, <a href="https://sthbrx.github.io/tag/fsp.html">FSP</a>, <a href="https://sthbrx.github.io/tag/patches.html">patches</a>, <a href="https://sthbrx.github.io/tag/based16.html">based16</a>, <a href="https://sthbrx.github.io/tag/linux-gods.html">Linux Gods</a>, <a href="https://sthbrx.github.io/tag/ozlabs.html">Ozlabs</a>, <a href="https://sthbrx.github.io/tag/offtopic.html">offtopic</a>, <a href="https://sthbrx.github.io/tag/autoboot.html">autoboot</a>, <a href="https://sthbrx.github.io/tag/kexec.html">kexec</a>, <a href="https://sthbrx.github.io/tag/aufs.html">aufs</a>, <a href="https://sthbrx.github.io/tag/overlay.html">overlay</a>, <a href="https://sthbrx.github.io/tag/php.html">php</a>, <a href="https://sthbrx.github.io/tag/capi.html">capi</a> </section>
<section>
<h1><a href="https://sthbrx.github.io/authors.html">Authors</a></h1>
<ul id="authors_list">
<li><a href="https://sthbrx.github.io/author/alastair-dsilva.html">Alastair D'Silva</a></li>
<li><a href="https://sthbrx.github.io/author/andrew-donnellan.html">Andrew Donnellan</a></li>
<li><a href="https://sthbrx.github.io/author/anton-blanchard.html">Anton Blanchard</a></li>
<li><a href="https://sthbrx.github.io/author/benjamin-gray.html">Benjamin Gray</a></li>
<li><a href="https://sthbrx.github.io/author/callum-scarvell.html">Callum Scarvell</a></li>
<li><a href="https://sthbrx.github.io/author/cyril-bur.html">Cyril Bur</a></li>
<li><a href="https://sthbrx.github.io/author/daniel-axtens.html">Daniel Axtens</a></li>
<li><a href="https://sthbrx.github.io/author/daniel-black.html">Daniel Black</a></li>
<li><a href="https://sthbrx.github.io/author/joel-stanley.html">Joel Stanley</a></li>
<li><a href="https://sthbrx.github.io/author/nick-piggin.html">Nick Piggin</a></li>
<li><a href="https://sthbrx.github.io/author/rashmica-gupta.html">Rashmica Gupta</a></li>
<li><a href="https://sthbrx.github.io/author/rohan-mclure.html">Rohan McLure</a></li>
<li><a href="https://sthbrx.github.io/author/russell-currey.html">Russell Currey</a></li>
<li><a href="https://sthbrx.github.io/author/samuel-mendoza-jonas.html">Samuel Mendoza-Jonas</a></li>
<li><a href="https://sthbrx.github.io/author/suraj-jitindar-singh.html">Suraj Jitindar Singh</a></li>
</ul>
</section>
<section>
<h1>Social</h1>
<ul>
<li><a href="https://sthbrx.github.io/rss.xml" type="application/rss+xml" rel="alternate">RSS</a></li>
<li><a href="https://github.com/sthbrx/" target="_blank">GitHub</a></li>
<li><a href="https://lists.ozlabs.org/listinfo/linuxppc-dev" target="_blank">linuxppc mailing list</a></li>
<li><a href="https://lists.ozlabs.org/listinfo/skiboot" target="_blank">Skiboot mailing list</a></li>
</ul>
</section>
<section>
<h1>Blogroll</h1>
<ul>
<li><a href="http://ozlabs.org" target="_blank">OzLabs</a></li>
</ul>
</section>
<section>
<h1>Disclaimer</h1>
<div>
This blog represents the views of the individual authors, and doesn't necessarily represent IBM's positions, strategies or opinions. </div>
</section>
</aside> </div>
</div>
<footer role="contentinfo"><p>
Copyright © 2015–2023 OzLabs —
<span class="credit">Powered by <a href="http://getpelican.com">Pelican</a></span>
</p></footer>
</body>
</html>