CONFIG_HZ matters ? : BBR with LKL

[thehajime]

I'm playing with Google's new TCP cc flavor, BBR, with LKL and saw some odd behavior if CONFIG_HZ is 100.

• netperf TCP_STREAM test between an LKL-netperf and a pure Linux netserver.

cc	hz	tput
bbr	HZ=100	456.43 Mbps
cubic	HZ=100	9385.28 Mbps
bbr	HZ=1000	6965.05 Mbps
cubic	HZ=1000	9393.35 Mbps

(raw output: https://gist.github.com/thehajime/4efc5690bde1d1248087097a8635d31b)

though there is still more to investigate why this different number when HZ=1000, BBR seems to not be functioning if CONFIG_HZ=100.

I will separately check this BBR's number but was wondering if CONFIG_HZ=1000 makes sense to LKL or not.

tested with my tentative branch.
https://github.com/libos-nuse/lkl-linux/tree/bbr-linus-4.9-rc4

[thehajime]

adding the following function to arch/lkl/kernel/time.c alleviates this issue.

unsigned long long sched_clock(void)
{
	return lkl_ops->time();
}

for more detail in https://github.com/lkl/linux/blob/master/Documentation/timers/timekeeping.txt#L107

This will introduce more frequent calls of lkl_ops->time() - I'm really worried about the performance degradation with this. @liuyuan10 @tavip it would be great if you could test the above patch (adding sched_clock) with a simple benchmark. I will also investigate this before making a PR.

btw, printk also shows different timestamps started with non-zero value.

[21848.144705] Linux version 4.9.0-rc4+ (tazaki@zakzak1) (gcc version 5.3.1 20160406 (Red Hat 5.3.1-6) (GCC) ) #35 Tue Nov 29 22:27:15 JST 2016
[21848.144716] bootmem address range: 0x7f2dd0000000 - 0x7f2dd3fff000
[21848.145589] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 16159
[21848.145589] Kernel command line:  virtio_mmio.device=268@0x1000000:1
[21848.145602] PID hash table entries: 256 (order: -1, 2048 bytes)
[21848.145655] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes)

[liuyuan10]

I can't observe any performance issue of it. According to the doc you refer to, sched_clock should start from 0?

[thehajime]

I can't observe any performance issue of it.

thanks.

According to the doc you refer to, sched_clock should start from 0?

thanks too; it should be like the following where boot_time will be set during time_init().

unsigned long long sched_clock(void)
{
	if (!boot_time)
		return 0;

	return lkl_ops->time() - boot_time;
}

[thehajime]

Fixed by #276.