-
Notifications
You must be signed in to change notification settings - Fork 7
/
index8.html
479 lines (437 loc) · 39.2 KB
/
index8.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
<!DOCTYPE html>
<head>
<meta charset="utf-8">
<title>Store Halfword Byte-Reverse Indexed</title>
<meta name="author" content="OzLabs">
<link href="https://sthbrx.github.io/rss.xml" type="application/rss+xml" rel="alternate"
title="Store Halfword Byte-Reverse Indexed RSS Feed" />
<!-- http://t.co/dKP3o1e -->
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link href="https://sthbrx.github.io/favicon.png" rel="icon">
<link href="https://sthbrx.github.io/theme/css/main.css" media="screen, projection"
rel="stylesheet" type="text/css">
<link href="//fonts.googleapis.com/css?family=PT+Serif:regular,italic,bold,bolditalic"
rel="stylesheet" type="text/css">
<link href="//fonts.googleapis.com/css?family=PT+Sans:regular,italic,bold,bolditalic"
rel="stylesheet" type="text/css">
<script type="text/javascript">
document.addEventListener('DOMContentLoaded', function() {
var ts = document.createElement('span')
ts.className = 'toggle-sidebar'
ts = document.getElementById('content').appendChild(ts);
ts.addEventListener('click', function(e) {
e.preventDefault();
body = document.getElementsByTagName('body')[0];
bodyClasses = body.classList.toggle('collapse-sidebar');
});
var sections = document.querySelectorAll('aside.sidebar > section');
if (sections.length > 1) {
for (index = 0; index < sections.length; index++) {
section = sections[index];
if ((sections.length >= 3) && index % 3 === 0) {
section.classList.add("first");
}
var count = ((index +1) % 2) ? "odd" : "even";
section.classList.add(count);
}
}
if (sections.length >= 3) {
document.querySelector('aside.sidebar').classList.add('thirds');
}
});
</script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-91189608-1', 'auto');
ga('send', 'pageview');
</script>
</head>
<body>
<header role="banner"><hgroup>
<h1><a href="https://sthbrx.github.io/">Store Halfword Byte-Reverse Indexed</a></h1>
<h2>A Power Technical Blog</h2>
</hgroup></header>
<nav role="navigation"><ul class="subscription" data-subscription="rss">
<li><a href="https://sthbrx.github.io/rss.xml" rel="subscribe-rss">RSS</a></li>
</ul>
<ul class="main-navigation">
<li >
<a href="https://sthbrx.github.io/category/cryptography.html">Cryptography</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/development.html">Development</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/education.html">Education</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/openpower.html">OpenPOWER</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/performance.html">Performance</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/petitboot.html">Petitboot</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/snowpatch.html">snowpatch</a>
</li>
<li >
<a href="https://sthbrx.github.io/category/virtualisation-and-emulation.html">Virtualisation and Emulation</a>
</li>
</ul></nav>
<div id="main">
<div id="content">
<div class="blog-index">
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/07/06/where-to-get-a-power8-development-vm/">Where to Get a POWER8 Development VM</a>
</h1>
<p class="meta">
<time datetime="2016-07-06T16:00:00+10:00" pubdate>Wed 06 July 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/andrew-donnellan.html">Andrew Donnellan</a>
</span>
</span>
<time datetime="2016-07-06T16:00:00+10:00" pubdate>Wed 06 July 2016</time></div>
<div class="entry-content"><p><em>POWER8 sounds great, but where the heck can I get a Power VM so I can test my code?</em></p>
<p>This is a common question we get at OzLabs from other open source developers looking to port their software to the Power Architecture. Unfortunately, most developers don't have one of our amazing servers just sitting around under their desk.</p>
<p>Thankfully, there's a few IBM partners who offer free VMs for development use. If you're in need of a development VM, check out:</p>
<ul>
<li><a href="http://openpower.ic.unicamp.br/minicloud/">MiniCloud</a>, hosted by the State University of Campinas (Unicamp), Brazil</li>
<li><a href="http://osuosl.org/services/powerdev">OSU Open Source Lab</a>, hosted by the Oregon State University</li>
<li><a href="https://ptopenlab.com/cloudlabconsole">SuperVessel Cloud for Power/OpenPOWER</a>, hosted by IBM China</li>
</ul>
<p>So, next time you wonder how you can test your project on POWER8, request a VM and get to it!</p></div>
</article>
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/07/05/optical-action-at-a-distance/">Optical Action at a Distance</a>
</h1>
<p class="meta">
<time datetime="2016-07-05T15:23:00+10:00" pubdate>Tue 05 July 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/samuel-mendoza-jonas.html">Samuel Mendoza-Jonas</a>
</span>
</span>
<time datetime="2016-07-05T15:23:00+10:00" pubdate>Tue 05 July 2016</time></div>
<div class="entry-content"><p>Generally when someone wants to install a Linux distro they start with an ISO
file. Now we could burn that to a DVD, walk into the server room, and put it in
our machine, but that's a pain. Instead let's look at how to do this over the
network with Petitboot!</p>
<p>At the moment Petitboot won't be able to handle an ISO file unless it's
mounted in an expected place (eg. as a mounted DVD), so we need to unpack it
somewhere. Choose somewhere to host the result and unpack the ISO via whatever
method you prefer. (For example <code>bsdtar -xf /path/to/image.iso</code>).</p>
<p>You'll get a bunch of files but for our purposes we only care about a few; the
kernel, the initrd, and the bootloader configuration file. Using
the Ubuntu 16.04 ppc64el ISO as an example, these are:</p>
<div class="highlight"><pre><span></span><code>./install/vmlinux
./install/initrd.gz.
./boot/grub/grub.cfg
</code></pre></div>
<p>In grub.cfg we can see that the boot arguments are actually quite simple:</p>
<div class="highlight"><pre><span></span><code>set timeout=-1
menuentry "Install" {
linux /install/vmlinux tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false --- quiet
initrd /install/initrd.gz
}
menuentry "Rescue mode" {
linux /install/vmlinux rescue/enable=true --- quiet
initrd /install/initrd.gz
}
</code></pre></div>
<p>So all we need to do is create a PXE config file that points Petitboot towards
the correct files.</p>
<p>We're going to create a PXE config file which you could serve from your DHCP
server, but that does not mean we need to use PXE - if you just want a quick
install you only need make these files accessible to Petitboot, and then we can
use the 'Retrieve config from URL' option to download the files.</p>
<p>Create a petitboot.conf file somewhere accessible that contains (for Ubuntu):</p>
<div class="highlight"><pre><span></span><code>label Install Ubuntu 16.04 Xenial Xerus
kernel http://myaccesibleserver/path/to/vmlinux
initrd http://myaccesibleserver/path/to/initrd.gz
append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false --- quiet
</code></pre></div>
<p>Then in Petitboot, select 'Retrieve config from URL' and enter
<code>http://myaccesibleserver/path/to/petitboot.conf</code>. In the main menu your new
option should appear - select it and away you go!</p></div>
</article>
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/07/01/a-taste-of-ibm/">A Taste of IBM</a>
</h1>
<p class="meta">
<time datetime="2016-07-01T11:45:00+10:00" pubdate>Fri 01 July 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/rohan-mclure.html">Rohan McLure</a>
</span>
</span>
<time datetime="2016-07-01T11:45:00+10:00" pubdate>Fri 01 July 2016</time></div>
<div class="entry-content"><p>As a hobbyist programmer and Linux user, I was pretty stoked to be able to experience real work in the IT field that interests me most, Linux. With a mainly disconnected understanding of computer hardware and software, I braced myself to entirely relearn everything and anything I thought I knew. Furthermore, I worried that my usefulness in a world of maintainers, developers and testers would not be enough to provide any real contribution to the company. In actual fact however, the employees at OzLabs (IBM ADL) put a really great effort into making use of my existing skills, were attentive to my current knowledge and just filled in the gaps! The knowledge they've given me is practical, interlinked with hardware and provided me with the foot-up that I'd been itching for to establish my own portfolio as a programmer. I was both honoured and astonished by their dedication to helping me make a truly meaningful contribution!</p>
<p>On applying for the placement, I listed my skills and interests. Having a Mathematics, Science background, I listed among my greatest interests development of scientific simulation and graphics using libraries such as Python matplotlib and R. By the <em>first day</em> they got me to work, researching and implementing a routine in R that would qualitatively model the ability of a system to perform common tasks - a benchmark. A series of these microbenchmarks were made; I was in my element and actually able to contribute to a corporation much larger than I could imagine. The team at IBM reinforced my knowledge from the ground up, introducing the rigorous hardware and corporate elements at a level I was comfortable with.</p>
<p>I would say that my greatest single piece of take-home knowledge over the two weeks was knowledge of the Linux Kernel project, Git and GitHub. Having met the arch/powerpc and linux-next maintainers in person placed the Linux and Open Source development cycle in an entirely new perspective. I was introduced to the world of GitHub, and thanks to a few rigorous lessons of Git, I now have access to tools that empower me to safely and efficiently write code, and to build a public portfolio I can be proud of. Most members of the office donated their time to instruct me on all fronts, whether to do with career paths, programming expertise or conceptual knowledge, and the rest were all very good for a chat.</p>
<p>Approaching the tail-end of Year Twelve, I was blessed with some really good feedback and recommendations regarding further study. If during the two weeks I had any query regarding anything ranging from work-life to programming expertise even to which code editor I should use (a source of much contention) the people in the office were very happy to help me. Several employees donated their time to teach me really very intensive and long lessons regarding the software development concepts, including (but not limited to!) a thorough and helpful lesson on Git that was just on my level of understanding.</p>
<p>Working at IBM these past two weeks has not only bridged the gap between my hobby and my professional prospects, but more importantly established friendships with professionals in the field of Software Development. Without a doubt this really great experience of an environment that rewards my enthusiasm will fondly stay in my mind as I enter the next chapter of my life!</p></div>
</article>
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/06/24/kernel-interfaces-and-vdso-test/">Kernel interfaces and vDSO test</a>
</h1>
<p class="meta">
<time datetime="2016-06-24T16:30:00+10:00" pubdate>Fri 24 June 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/cyril-bur.html">Cyril Bur</a>
</span>
</span>
<time datetime="2016-06-24T16:30:00+10:00" pubdate>Fri 24 June 2016</time></div>
<div class="entry-content"><h3>Getting Suckered</h3>
<p>Last week a colleague of mine came up to me and showed me some of the
<abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> on PowerPC and asked why on earth does it fail
<a href="https://github.com/nlynch-mentor/vdsotest">vdsotest</a>. I should come
clean at this point and admit that I knew very little about the <abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr>
and hadn't heard of vdsotest. I had to admit to this colleague that I
had no idea everything looked super sane.</p>
<p>Unfortunately (for me) I got hooked, vdsotest was saying it was
getting '22' instead of '-1' and it was the case where the <abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> would
call into the kernel. It plagued me all night, 22 is so suspicious.
Right before I got to work the next morning I had an epiphany, "I bet
22 is EINVAL".</p>
<h3>Virtual Dynamically linked Shared Objects</h3>
<p>The <a href="https://en.wikipedia.org/wiki/VDSO"><abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr></a> is a mechanism to
expose some kernel functionality into userspace to avoid the cost of a
context switch into kernel mode. This is a great feat of engineering,
avoiding the context switch can have a dramatic speedup for userspace
code. Obviously not all kernel functionality can be placed into
userspace and even for the functionality which can,
there may be edge cases in which the <abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> needs to ask the kernel.</p>
<p>Who tests the <abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr>? For the portion that lies exclusively in userspace it
will escape all testing of the syscall interface which is really what
kernel developers are so focused on not breaking. Enter Nathan Lynch
with <a href="https://github.com/nlynch-mentor/vdsotest">vdsotest</a> who has
done some great work!</p>
<h3>The Kernel</h3>
<p>When the <abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> can't get the correct value without the kernel, it
simply calls into the kernel because the kernel is the definitive
reference for every syscall. On PowerPC something like this happens
(sorry, our <abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> is 100% asm):
<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup></p>
<div class="highlight"><pre><span></span><code><span class="cm">/*</span>
<span class="cm"> * Exact prototype of clock_gettime()</span>
<span class="cm"> *</span>
<span class="cm"> * int __kernel_clock_gettime(clockid_t clock_id, struct timespec *tp);</span>
<span class="cm"> *</span>
<span class="cm"> */</span>
<span class="nf">V_FUNCTION_BEGIN</span><span class="p">(</span><span class="no">__kernel_clock_gettime</span><span class="p">)</span>
<span class="w"> </span><span class="na">.cfi_startproc</span>
<span class="w"> </span><span class="cm">/* Check for supported clock IDs */</span>
<span class="w"> </span><span class="nf">cmpwi</span><span class="w"> </span><span class="no">cr0</span><span class="p">,</span><span class="no">r3</span><span class="p">,</span><span class="no">CLOCK_REALTIME</span>
<span class="w"> </span><span class="nf">cmpwi</span><span class="w"> </span><span class="no">cr1</span><span class="p">,</span><span class="no">r3</span><span class="p">,</span><span class="no">CLOCK_MONOTONIC</span>
<span class="w"> </span><span class="nf">cror</span><span class="w"> </span><span class="no">cr0</span><span class="p">*</span><span class="mi">4</span><span class="err">+</span><span class="no">eq</span><span class="p">,</span><span class="no">cr0</span><span class="p">*</span><span class="mi">4</span><span class="err">+</span><span class="no">eq</span><span class="p">,</span><span class="no">cr1</span><span class="p">*</span><span class="mi">4</span><span class="err">+</span><span class="no">eq</span>
<span class="w"> </span><span class="nf">bne</span><span class="w"> </span><span class="no">cr0</span><span class="p">,</span><span class="mi">99</span><span class="no">f</span>
<span class="w"> </span><span class="cm">/* [snip] */</span>
<span class="w"> </span><span class="cm">/*</span>
<span class="cm"> * syscall fallback</span>
<span class="cm"> */</span>
<span class="err">99:</span>
<span class="w"> </span><span class="nf">li</span><span class="w"> </span><span class="no">r0</span><span class="p">,</span><span class="no">__NR_clock_gettime</span>
<span class="w"> </span><span class="nf">sc</span>
<span class="w"> </span><span class="nf">blr</span>
</code></pre></div>
<p>For those not familiar, this couldn't be more simple. The start checks
to see if it is a clock id that the <abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> can handle and if not it jumps
to the 99 label. From here simply load the syscall number, jump to the
kernel and branch to link register aka 'return'. In this case the
'return' statement would return to the userspace code which called the
<abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> function.</p>
<p>Wait, having the <abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> calling into the kernel call gets us the wrong
result? Or course it should, vdsotest is assuming a C ABI with return
values and errno but the kernel doesn't do that, the kernel ABI is
different. How does this even work on x86? Ohhhhh vdsotest does <sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup></p>
<div class="highlight"><pre><span></span><code><span class="k">static</span><span class="w"> </span><span class="kr">inline</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">record_syscall_result</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">syscall_result</span><span class="w"> </span><span class="o">*</span><span class="n">res</span><span class="p">,</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">sr_ret</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">sr_errno</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="cm">/* Calling the vDSO directly instead of through libc can lead to:</span>
<span class="cm"> * - The vDSO code punts to the kernel (e.g. unrecognized clock id).</span>
<span class="cm"> * - The kernel returns an error (e.g. -22 (-EINVAL))</span>
<span class="cm"> * So we need to recognize this situation and fix things up.</span>
<span class="cm"> * Fortunately we're dealing only with syscalls that return -ve values</span>
<span class="cm"> * on error.</span>
<span class="cm"> */</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">sr_ret</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">sr_errno</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">sr_errno</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">-</span><span class="n">sr_ret</span><span class="p">;</span>
<span class="w"> </span><span class="n">sr_ret</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">-1</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="o">*</span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">syscall_result</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="p">.</span><span class="n">sr_ret</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sr_ret</span><span class="p">,</span>
<span class="w"> </span><span class="p">.</span><span class="n">sr_errno</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sr_errno</span><span class="p">,</span>
<span class="w"> </span><span class="p">};</span>
<span class="p">}</span>
</code></pre></div>
<p>That little hack isn't working on PowerPC and here's why:</p>
<p>The kernel puts the return value in the ABI specified return register
(r3) and uses a condition register bit (condition register field 0, SO
bit), so unlike x86 on error the return value isn't negative. To make
matters worse, the condition register is very difficult to access from
C. Depending on your definition of 'access from C' you might consider
it impossible, a fixup like that would be impossible.</p>
<h3>Lessons learnt</h3>
<ul>
<li><abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> supplied functions aren't quite the same as their libc
counterparts. Unless you have very good reason, and to be fair,
vdsotest does have a very good reason, always access the <abbr title="Virtual Dynamically linked Shared Objects">vDSO</abbr> through
libc</li>
<li>Kernel interfaces aren't C interfaces, yep, they're close but they
aren't the same</li>
<li>22 is in fact EINVAL</li>
<li>Different architectures are... Different!</li>
<li>Variety is the spice of life</li>
</ul>
<p>P.S I have a hacky patch waiting review</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>arch/powerpc/kernel/vdso64/gettimeofday.S <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>src/vdsotest.h <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div></div>
</article>
<article>
<header>
<h1 class="entry-title">
<a href="https://sthbrx.github.io/blog/2016/06/15/introducing-snowpatch-continuous-integration-for-patches/">Introducing snowpatch: continuous integration for patches</a>
</h1>
<p class="meta">
<time datetime="2016-06-15T15:33:00+10:00" pubdate>Wed 15 June 2016</time> </p>
</header>
<div class="byline_index">
<span class="byline author vcard">
Posted by <span class="fn">
<a href="https://sthbrx.github.io/author/russell-currey.html">Russell Currey</a>
</span>
</span>
<time datetime="2016-06-15T15:33:00+10:00" pubdate>Wed 15 June 2016</time></div>
<div class="entry-content"><p>Continuous integration has changed the way we develop software. The ability to make a code change and be notified quickly and automatically whether or not it works allows for faster iteration and higher quality. These processes and technologies allow products to quickly and consistently release new versions, driving continuous improvement to their users. For a web app, it's all pretty simple: write some tests, someone makes a pull request, you build it and run the tests. Tools like GitHub, Travis CI and Jenkins have made this process simple and efficient.</p>
<p>Let's throw some spanners in the works. What if instead of a desktop or web application, you're dealing with an operating system? What if your tests can only be run when booted on physical hardware? What if instead of something like a GitHub pull request, code changes were sent as plain-text emails to a mailing list? What if you didn't have control the development of this project, and you had to work with an existing, open community?</p>
<p>These are some of the problems faced by the Linux kernel, and many other open source projects. Mailing lists, along with tools like <code>git send-email</code>, have become core development infrastructure for many large open source projects. The idea of sending code via a plain-text email is simple and well-defined, not reliant on a proprietary service, and uses universal, well-defined technology. It does have shortcomings, though. How do you take a plain-text patch, which was sent as an email to a mailing list, and accomplish the continuous integration possibilities other tools have trivially?</p>
<p>Out of this problem birthed <a href="https://github.com/ruscur/snowpatch">snowpatch</a>, a continuous integration tool designed to enable these practices for projects that use mailing lists and plain-text patches. By taking patch metadata organised by <a href="https://patchwork.ozlabs.org">Patchwork</a>, performing a number of <code>git</code> operations and shipping them off to <a href="https://jenkins.io">Jenkins</a>, snowpatch can enable continuous integration for any mailing list-based project. At IBM <a href="https://ozlabs.org">OzLabs</a>, we're using snowpatch to automatically test new patches for <a href="https://github.com/linuxppc/linux/">Linux on POWER</a>, <a href="https://github.com/open-power/skiboot.git">skiboot</a>, snowpatch itself, and more.</p>
<p>snowpatch is written in <a href="https://rust-lang.org">Rust</a>, an exciting new systems programming language with a focus on speed and safety. Rust's amazing software ecosystem, enabled by its package manager <a href="https://crates.io">Cargo</a>, made development of snowpatch a breeze. Using Rust has been a lot of fun, along with the practical benefits of (in our experience) faster development, and confidence in the runtime stability of our code. It's still a young language, but it's quickly growing and has an amazing community that has always been happy to help.</p>
<p>We still have a lot of ideas for snowpatch that haven't been implemented yet. Once we've tested a patch and sent the results back to a patchwork instance, what if the project's maintainer (or a trusted contributor) could manually trigger some more intensive tests? How would we handle it if the traffic on the mailing list of a project is too fast for us to test? If we were running snowpatch on multiple machines on the same project, how would we avoid duplicating effort? These are unsolved problems, and if you'd like to help us with these or anything else you think would be good for snowpatch, we take contributions and ideas via our mailing list, which you can subscribe to <a href="https://lists.ozlabs.org/listinfo/snowpatch">here</a>. For more details, view our documentation on <a href="https://github.com/ruscur/snowpatch">GitHub</a>.</p>
<p>Thanks for taking your time to learn a bit about snowpatch. In future, we'll be talking about how we tie all these technologies together to build a continuous integration workflow for the Linux kernel and OpenPOWER firmware. Watch this space!</p>
<p><em>This article was originally posted on <a href="https://developer.ibm.com/open/">IBM developerWorks Open</a>. Check that out for more open source from IBM, and look out for more content in their <a href="https://developer.ibm.com/open/snowpatch">snowpatch</a> section.</em></p></div>
</article>
<div class="pagination">
<a class="prev" href="https://sthbrx.github.io/index9.html">← Older</a>
<a class="next" href="https://sthbrx.github.io/index7.html">Newer →</a>
<br />
</div></div>
<aside class="sidebar">
<section>
<h1>Recent Posts</h1>
<ul id="recent_posts">
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/08/07/going-out-on-a-limb-efficient-elliptic-curve-arithmetic-in-openssl/">Going out on a Limb: Efficient Elliptic Curve Arithmetic in OpenSSL</a>
</li>
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/08/04/quirks-of-parsing-ssh-configs/">Quirks of parsing SSH configs</a>
</li>
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/04/05/detecting-rootless-docker/">Detecting rootless Docker</a>
</li>
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/04/04/dumb-bugs-the-pci-device-that-wasnt/">Dumb bugs: the PCI device that wasn't</a>
</li>
<li class="post">
<a href="https://sthbrx.github.io/blog/2023/03/24/dumb-bugs-when-a-date-breaks-booting-the-kernel/">Dumb bugs: When a date breaks booting the kernel</a>
</li>
</ul>
</section>
<section>
<h1>Categories</h1>
<ul id="recent_posts">
<li><a href="https://sthbrx.github.io/category/cryptography.html">Cryptography</a></li>
<li><a href="https://sthbrx.github.io/category/development.html">Development</a></li>
<li><a href="https://sthbrx.github.io/category/education.html">Education</a></li>
<li><a href="https://sthbrx.github.io/category/openpower.html">OpenPOWER</a></li>
<li><a href="https://sthbrx.github.io/category/performance.html">Performance</a></li>
<li><a href="https://sthbrx.github.io/category/petitboot.html">Petitboot</a></li>
<li><a href="https://sthbrx.github.io/category/snowpatch.html">snowpatch</a></li>
<li><a href="https://sthbrx.github.io/category/virtualisation-and-emulation.html">Virtualisation and Emulation</a></li>
</ul>
</section>
<section>
<h1>Tags</h1>
<a href="https://sthbrx.github.io/tag/ssh.html">ssh</a>, <a href="https://sthbrx.github.io/tag/docker.html">Docker</a>, <a href="https://sthbrx.github.io/tag/syzkaller.html">syzkaller</a>, <a href="https://sthbrx.github.io/tag/linux.html">linux</a>, <a href="https://sthbrx.github.io/tag/power8.html">power8</a>, <a href="https://sthbrx.github.io/tag/distro.html">distro</a>, <a href="https://sthbrx.github.io/tag/kernel.html">kernel</a>, <a href="https://sthbrx.github.io/tag/hardening.html">hardening</a>, <a href="https://sthbrx.github.io/tag/testing.html">testing</a>, <a href="https://sthbrx.github.io/tag/conferences.html">conferences</a>, <a href="https://sthbrx.github.io/tag/instruction-set-architecture.html">Instruction Set Architecture</a>, <a href="https://sthbrx.github.io/tag/openpower.html">openpower</a>, <a href="https://sthbrx.github.io/tag/firmware.html">firmware</a>, <a href="https://sthbrx.github.io/tag/goodposts.html">goodposts</a>, <a href="https://sthbrx.github.io/tag/realcontent.html">realcontent</a>, <a href="https://sthbrx.github.io/tag/madposting.html">madposting</a>, <a href="https://sthbrx.github.io/tag/op-test.html">op-test</a>, <a href="https://sthbrx.github.io/tag/qemu.html">qemu</a>, <a href="https://sthbrx.github.io/tag/pci.html">pci</a>, <a href="https://sthbrx.github.io/tag/sparseposting.html">sparseposting</a>, <a href="https://sthbrx.github.io/tag/petitboot.html">petitboot</a>, <a href="https://sthbrx.github.io/tag/security.html">security</a>, <a href="https://sthbrx.github.io/tag/vscode.html">vscode</a>, <a href="https://sthbrx.github.io/tag/code.html">code</a>, <a href="https://sthbrx.github.io/tag/openbmc.html">openbmc</a>, <a href="https://sthbrx.github.io/tag/ipmi.html">ipmi</a>, <a href="https://sthbrx.github.io/tag/opencapi.html">opencapi</a>, <a href="https://sthbrx.github.io/tag/openpower-summit.html">openpower summit</a>, <a href="https://sthbrx.github.io/tag/easyposts.html">easyposts</a>, <a href="https://sthbrx.github.io/tag/linuxboot.html">linuxboot</a>, <a href="https://sthbrx.github.io/tag/google.html">google</a>, <a href="https://sthbrx.github.io/tag/intel.html">intel</a>, <a href="https://sthbrx.github.io/tag/osfc.html">osfc</a>, <a href="https://sthbrx.github.io/tag/shortposts.html">shortposts</a>, <a href="https://sthbrx.github.io/tag/facebook.html">facebook</a>, <a href="https://sthbrx.github.io/tag/performance.html">performance</a>, <a href="https://sthbrx.github.io/tag/phoronix.html">phoronix</a>, <a href="https://sthbrx.github.io/tag/benchmarks.html">benchmarks</a>, <a href="https://sthbrx.github.io/tag/stupid-ideas.html">stupid ideas</a>, <a href="https://sthbrx.github.io/tag/network.html">network</a>, <a href="https://sthbrx.github.io/tag/power.html">power</a>, <a href="https://sthbrx.github.io/tag/xdp.html">xdp</a>, <a href="https://sthbrx.github.io/tag/networking.html">networking</a>, <a href="https://sthbrx.github.io/tag/remoteposts.html">remoteposts</a>, <a href="https://sthbrx.github.io/tag/ceph.html">ceph</a>, <a href="https://sthbrx.github.io/tag/raid.html">raid</a>, <a href="https://sthbrx.github.io/tag/storage.html">storage</a>, <a href="https://sthbrx.github.io/tag/erasure.html">erasure</a>, <a href="https://sthbrx.github.io/tag/lustre.html">lustre</a>, <a href="https://sthbrx.github.io/tag/hpc.html">hpc</a>, <a href="https://sthbrx.github.io/tag/nvlink.html">nvlink</a>, <a href="https://sthbrx.github.io/tag/namd.html">namd</a>, <a href="https://sthbrx.github.io/tag/cuda.html">cuda</a>, <a href="https://sthbrx.github.io/tag/gpu.html">gpu</a>, <a href="https://sthbrx.github.io/tag/minsky.html">minsky</a>, <a href="https://sthbrx.github.io/tag/s822lc-for-hpc.html">S822LC for hpc</a>, <a href="https://sthbrx.github.io/tag/debug.html">debug</a>, <a href="https://sthbrx.github.io/tag/virtualisation.html">virtualisation</a>, <a href="https://sthbrx.github.io/tag/dmesg.html">dmesg</a>, <a href="https://sthbrx.github.io/tag/printk.html">printk</a>, <a href="https://sthbrx.github.io/tag/boot.html">boot</a>, <a href="https://sthbrx.github.io/tag/early.html">early</a>, <a href="https://sthbrx.github.io/tag/error.html">error</a>, <a href="https://sthbrx.github.io/tag/centos.html">centos</a>, <a href="https://sthbrx.github.io/tag/centos7.html">centos7</a>, <a href="https://sthbrx.github.io/tag/p8.html">p8</a>, <a href="https://sthbrx.github.io/tag/bmc.html">bmc</a>, <a href="https://sthbrx.github.io/tag/rhel.html">RHEL</a>, <a href="https://sthbrx.github.io/tag/skiroot.html">skiroot</a>, <a href="https://sthbrx.github.io/tag/devmapper.html">devmapper</a>, <a href="https://sthbrx.github.io/tag/lvm.html">lvm</a>, <a href="https://sthbrx.github.io/tag/cgroups.html">cgroups</a>, <a href="https://sthbrx.github.io/tag/numa.html">numa</a>, <a href="https://sthbrx.github.io/tag/development.html">Development</a>, <a href="https://sthbrx.github.io/tag/netboot.html">netboot</a>, <a href="https://sthbrx.github.io/tag/pxe.html">pxe</a>, <a href="https://sthbrx.github.io/tag/education.html">Education</a>, <a href="https://sthbrx.github.io/tag/work-experience.html">work experience</a>, <a href="https://sthbrx.github.io/tag/asm.html">asm</a>, <a href="https://sthbrx.github.io/tag/vdso.html">vdso</a>, <a href="https://sthbrx.github.io/tag/snowpatch.html">snowpatch</a>, <a href="https://sthbrx.github.io/tag/tools.html">tools</a>, <a href="https://sthbrx.github.io/tag/intern.html">intern</a>, <a href="https://sthbrx.github.io/tag/srop.html">SROP</a>, <a href="https://sthbrx.github.io/tag/mitigation.html">mitigation</a>, <a href="https://sthbrx.github.io/tag/double.html">double</a>, <a href="https://sthbrx.github.io/tag/float.html">float</a>, <a href="https://sthbrx.github.io/tag/hex.html">hex</a>, <a href="https://sthbrx.github.io/tag/debugging.html">debugging</a>, <a href="https://sthbrx.github.io/tag/skiboot.html">skiboot</a>, <a href="https://sthbrx.github.io/tag/opal.html">OPAL</a>, <a href="https://sthbrx.github.io/tag/fsp.html">FSP</a>, <a href="https://sthbrx.github.io/tag/patches.html">patches</a>, <a href="https://sthbrx.github.io/tag/based16.html">based16</a>, <a href="https://sthbrx.github.io/tag/linux-gods.html">Linux Gods</a>, <a href="https://sthbrx.github.io/tag/ozlabs.html">Ozlabs</a>, <a href="https://sthbrx.github.io/tag/offtopic.html">offtopic</a>, <a href="https://sthbrx.github.io/tag/autoboot.html">autoboot</a>, <a href="https://sthbrx.github.io/tag/kexec.html">kexec</a>, <a href="https://sthbrx.github.io/tag/aufs.html">aufs</a>, <a href="https://sthbrx.github.io/tag/overlay.html">overlay</a>, <a href="https://sthbrx.github.io/tag/php.html">php</a>, <a href="https://sthbrx.github.io/tag/capi.html">capi</a> </section>
<section>
<h1><a href="https://sthbrx.github.io/authors.html">Authors</a></h1>
<ul id="authors_list">
<li><a href="https://sthbrx.github.io/author/alastair-dsilva.html">Alastair D'Silva</a></li>
<li><a href="https://sthbrx.github.io/author/andrew-donnellan.html">Andrew Donnellan</a></li>
<li><a href="https://sthbrx.github.io/author/anton-blanchard.html">Anton Blanchard</a></li>
<li><a href="https://sthbrx.github.io/author/benjamin-gray.html">Benjamin Gray</a></li>
<li><a href="https://sthbrx.github.io/author/callum-scarvell.html">Callum Scarvell</a></li>
<li><a href="https://sthbrx.github.io/author/cyril-bur.html">Cyril Bur</a></li>
<li><a href="https://sthbrx.github.io/author/daniel-axtens.html">Daniel Axtens</a></li>
<li><a href="https://sthbrx.github.io/author/daniel-black.html">Daniel Black</a></li>
<li><a href="https://sthbrx.github.io/author/joel-stanley.html">Joel Stanley</a></li>
<li><a href="https://sthbrx.github.io/author/nick-piggin.html">Nick Piggin</a></li>
<li><a href="https://sthbrx.github.io/author/rashmica-gupta.html">Rashmica Gupta</a></li>
<li><a href="https://sthbrx.github.io/author/rohan-mclure.html">Rohan McLure</a></li>
<li><a href="https://sthbrx.github.io/author/russell-currey.html">Russell Currey</a></li>
<li><a href="https://sthbrx.github.io/author/samuel-mendoza-jonas.html">Samuel Mendoza-Jonas</a></li>
<li><a href="https://sthbrx.github.io/author/suraj-jitindar-singh.html">Suraj Jitindar Singh</a></li>
</ul>
</section>
<section>
<h1>Social</h1>
<ul>
<li><a href="https://sthbrx.github.io/rss.xml" type="application/rss+xml" rel="alternate">RSS</a></li>
<li><a href="https://github.com/sthbrx/" target="_blank">GitHub</a></li>
<li><a href="https://lists.ozlabs.org/listinfo/linuxppc-dev" target="_blank">linuxppc mailing list</a></li>
<li><a href="https://lists.ozlabs.org/listinfo/skiboot" target="_blank">Skiboot mailing list</a></li>
</ul>
</section>
<section>
<h1>Blogroll</h1>
<ul>
<li><a href="http://ozlabs.org" target="_blank">OzLabs</a></li>
</ul>
</section>
<section>
<h1>Disclaimer</h1>
<div>
This blog represents the views of the individual authors, and doesn't necessarily represent IBM's positions, strategies or opinions. </div>
</section>
</aside> </div>
</div>
<footer role="contentinfo"><p>
Copyright © 2015–2023 OzLabs —
<span class="credit">Powered by <a href="http://getpelican.com">Pelican</a></span>
</p></footer>
</body>
</html>