Add Nix expressions to build Snabb #876

domenkozar · 2016-04-18T13:04:26Z

Hydra is currently using these in release.nix and they build successfully at https://hydra.snabb.co/jobset/snabb/master

It's a good idea to put the upstream, because different Snabb versions may need different the Nix expressions. It's also handy to clone Snabb and just run nix-build release.nix and you should get the snabb, manual and tests (with logs) for that particular commit (if tests didn't fail).

There are two blockers:

I have no way to extract Snabb version from the repo. Running a command won't help in Nix. It needs to be statically in a file. Currently no version is added to the derivation name.
We depend on git for manual and also Snabb at build time. This is problematic since hash in Nix is content addressed (src = ./.; and .git mutates on each commit and thus changes regardless of the commit checkout. See my upstream comment at fetchgit with leaveDotGit = true is still not completely deterministic NixOS/nixpkgs#8567 (comment)

PS: There are lots of commits, maybe kbara-next needs to be rebase on master? Or should I just rebase on kbara-next?

This will be used by the lwAFTR.

Currently shm.map() will create the file and any containing directories if it does not exist. This is not so great, so I split it into separate open() and create() functions, adapt all callers, and remove map() from the public interface of the shm module.

This commit adds a histogram facility that can record the distribution of different sampled magnitudes. In particular, it is useful for recording distributions of times. Histograms can be mapped into /var/run/snabb, where they can be analyzed by a separate process. This commit also wires up the app.breathe() loop to record latencies for its breath cycles into a well-known file (/var/run/snabb/PID/engine/latency), and wires up "snabb top" to do some basic statistics on this data.

The idea is, there are lots of different kinds of analyses you might want to do on a histogram, and it's too early to canonicalize one of them. Better to build your own on top of iterate(). Also adds tests for iterate().

Imported from Igalia/lwaftr_starfruit, up to date with commit a87632f

kbara · 2016-04-18T13:20:21Z

@lukego is there anything it'd be helpful for me to do with kbara-next, like merge master into it? I'm still getting used to the upstream Snabb workflow... :-)

lukego · 2016-04-18T13:37:35Z

Hey, cool, there is actually no mess here and everybody is doing the right thing :) we are exercising a new edge in our collaboration graph here :-).

The reason we see many commits here is that downstream branch nix has merged a bunch of commits from the v2016.04 release on master but these commits are not yet in kbara-next. So Github is showing that the consequence of merging nix onto kbara-next would be to get all of these commits i.e. the v2016.04 release in addition to Domen's own change a357165.

The next steps could be either of these things:

kbara-next merges from master and then this PR should automatically update to show only the commits that are still missing i.e. a357165. Then this could be merged.
kbara-next merges from nix which in one fell swoop would bring in the v2016.04 release and also Domen's change i.e. all the commits we see listed above.

So @kbara it is really up to you which you prefer.

In a perfect world we might all merge master into our long-lived branches immediately when a new release is made, but this demand-driven merge may well be fine too. The main thing is to transfer all the commits between all the branches and git is good at keeping track of what is missing (provided that we don't rebase and change the commit IDs.)

lukego · 2016-04-18T13:46:56Z

Here is one git incantation to summarize the changes more succinctly:

$ git log --oneline --first-parent origin/kbara-next..origin/nix
a357165 Add Nix expressions to build Snabb project
03f5958 Merge PR #855 (v2016.04 release) onto master

Too bad there is no button on the Github UI to see --first-parent :)

kbara · 2016-04-18T14:14:55Z

Excellent. I've run git merge upstream/master on kbara-next and pushed that; I agree that doing that after each release sounds like a good workflow.

lukego · 2016-04-18T14:23:02Z

@kbara You will need to push your branch directly to the snabbco repo now e.g. with git push snabbco kbara-next if you have a remote called snabbco. The mirroring-from-our-forks hack is gone now - see #850.

kbara · 2016-04-18T14:59:22Z

Interesting. I've done that, it looks to me like it's updated, but the list of commits/etc above doesn't update.

lukego · 2016-04-18T15:20:13Z

Interesting, yep. I see that you have pushed:

$ git log -1 --oneline origin/kbara-next
d5d355a Merge remote-tracking branch 'upstream/master' into kbara-next
$ git log --oneline origin/kbara-next..origin/nix
a357165 Add Nix expressions to build Snabb project

but I suppose that the Github UI does not automagically refresh in this case. Oh well, probably not a big deal.

kbara · 2016-04-18T15:28:26Z

Yeah; I couldn't find a way to manually do that either. All the more reason to do this right after releases, eh?

kbara · 2016-04-18T15:31:00Z

There is a trailing whitespace after + tar xvzf ${test_env} -C ~/.test_env/; is this something people care about on this project?

kbara · 2016-04-18T15:57:02Z

+      export SNABB_PCI0="0000:01:00.0"
+      export SNABB_PCI_INTEL1="0000:01:00.1"
+      export SNABB_PCI_INTEL0="0000:01:00.0"

will clearly only work on some machines. Is the intent for this to just be running on the snabblab servers? If not, how are people with other PCI IDs expected to use it - local modifications to this file?

kbara · 2016-04-18T15:58:06Z

Aside from those two nits, it looks good to me, with the caveat that I haven't dug deeply into nix yet. :-)

domenkozar · 2016-04-19T08:58:14Z

You're right. I'll make sure the `SNABB_PCI* exports can be superseeded via bash env. And that whitespace should be gone.

domenkozar · 2016-04-19T10:08:33Z

Pushed the two fixes, but the diff is bigger now:

$ git log --oneline origin/kbara-next..origin/nix
6cce8f4 release.nix: whitespace
8982625 release.nix: respect bash env, provide fallback values
e1ff0a3 Swapped in lstopo.png, replacing lstopo.pdf
aa1aac9 Renamed performance tuning file, as per review request
afc0bfb added info on mlock=on
294fc3b braindump on performance parameters
dbe9a9a Initial stub on performance tuning
a357165 Add Nix expressions to build Snabb project

domenkozar · 2016-04-29T13:47:56Z

I've removed WIP, the two issues I mentioned can be addressed later.

Each migration is now only responsible for progressing to the next version which then will run the subsequent migration to progress further. This is done as a long chain until the configuration has been migrated to the latest configuration schema. This also fixes an issue with the 3.2.0 -> 2017.07.01 migration which no longer functioned correctly.

Fix snabbco#876 - Improve config migration by chaining together

wingo and others added 30 commits February 25, 2016 15:02

Add packet.shiftright

5306f25

This will be used by the lwAFTR.

Return from VirtioNet:pull if link is nil

e483d6b

Merge PR #781 (Add packet.shiftright) into max-next

f27adf3

tests require root access

d5d107e

Merge PR #791 (tests require root access) into max-next

27bce96

Merge PR #787 (Exit from VirtioNet:pull if link is nil) into max-next

2cc5ed6

shm.open does not truncate file

42e9286

Fix spelling of "hierarchical".

7c6255f

Fix permissions when opening writable shm files

b027b02

Express histogram:summarize, :report in terms of :iterate

9e79a1f

Move histogram:summarize to snabb top

f9abfd5

The idea is, there are lots of different kinds of analyses you might want to do on a histogram, and it's too early to canonicalize one of them. Better to build your own on top of iterate(). Also adds tests for iterate().

Updated histogram documentation.

1fe832c

Rename histogram.count to histogram.total.

89b0c5b

Adapt to feedback regarding iterate() return vals

05b87f7

More named constants, fewer magical numbers.

396a132

doc/style.css: Put ToC into a left column.

192fbf7

Merge branch 'master' into max-next

82df345

Merge PR #794 (core.shm: replace map with create/open) into max-next

ffee768

Merge PR #795 (measure latency of app.breathe) into max-next

88bc0e7

Print out error message in case PCAP could open file

da65030

branches.md: Added mellanox and multiproc branches

854af02

Import lightweight 4-over-6 AFTR

8364729

Imported from Igalia/lwaftr_starfruit, up to date with commit a87632f

Add configurable parameter 'overwrite'

b22e287

Add lwaftr branch.

5c337d0

Merge branch max-next (uvilla 1) into next

8bd3ae4

Merge PR #803 (lwaftr branch) into next

76dc489

Merge PR #800 (mellanox and multiproc branches) into next

616110d

branches.md: Fix URL for mellanox and multiproc

ebe29f5

kbara self-assigned this Apr 18, 2016

Katerina Barone-Adesi and others added 7 commits April 19, 2016 11:03

Initial stub on performance tuning

dbe9a9a

braindump on performance parameters

294fc3b

added info on mlock=on

afc0bfb

Renamed performance tuning file, as per review request

aa1aac9

Swapped in lstopo.png, replacing lstopo.pdf

e1ff0a3

release.nix: respect bash env, provide fallback values

8982625

release.nix: whitespace

6cce8f4

domenkozar changed the title ~~[WIP] Add Nix expressions to build Snabb~~ Add Nix expressions to build Snabb Apr 29, 2016

kbara added the merged label Apr 29, 2016

kbara pushed a commit to kbara/snabb that referenced this pull request Apr 29, 2016

Merge PR snabbco#876 (Add nix expressions) into kbara-next

463bfc5

kbara merged commit 6cce8f4 into kbara-next Apr 29, 2016

kbara mentioned this pull request Apr 29, 2016

Merge kbara-next (2016-april and 2016-march) into next #907

Merged

domenkozar mentioned this pull request May 2, 2016

src/nix/manual.nix: Nix expression to build the manual #831

Closed

takikawa pushed a commit to takikawa/snabb that referenced this pull request Aug 5, 2017

Merge pull request snabbco#882 from Igalia/876-chained-migrations

dc35b69

Fix snabbco#876 - Improve config migration by chaining together

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Nix expressions to build Snabb #876

Add Nix expressions to build Snabb #876

domenkozar commented Apr 18, 2016

kbara commented Apr 18, 2016

lukego commented Apr 18, 2016 •

edited

Loading

lukego commented Apr 18, 2016 •

edited

Loading

kbara commented Apr 18, 2016

lukego commented Apr 18, 2016

kbara commented Apr 18, 2016

lukego commented Apr 18, 2016

kbara commented Apr 18, 2016

kbara commented Apr 18, 2016

kbara commented Apr 18, 2016

kbara commented Apr 18, 2016

domenkozar commented Apr 19, 2016

domenkozar commented Apr 19, 2016

domenkozar commented Apr 29, 2016

Add Nix expressions to build Snabb #876

Add Nix expressions to build Snabb #876

Conversation

domenkozar commented Apr 18, 2016

kbara commented Apr 18, 2016

lukego commented Apr 18, 2016 • edited Loading

lukego commented Apr 18, 2016 • edited Loading

kbara commented Apr 18, 2016

lukego commented Apr 18, 2016

kbara commented Apr 18, 2016

lukego commented Apr 18, 2016

kbara commented Apr 18, 2016

kbara commented Apr 18, 2016

kbara commented Apr 18, 2016

kbara commented Apr 18, 2016

domenkozar commented Apr 19, 2016

domenkozar commented Apr 19, 2016

domenkozar commented Apr 29, 2016

lukego commented Apr 18, 2016 •

edited

Loading

lukego commented Apr 18, 2016 •

edited

Loading