Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pids.max cgroup return "max", runc events broken on master #663

Closed
jessfraz opened this issue Mar 19, 2016 · 17 comments
Closed

pids.max cgroup return "max", runc events broken on master #663

jessfraz opened this issue Mar 19, 2016 · 17 comments

Comments

@jessfraz
Copy link
Contributor

If I run runc events on master it just hangs

I git bisected it to 2b1e086

but I cant tell why that broke it...

But on master my cgroups path in /run/runc/container_name/state.json is:

"cgroup_paths": {
    "blkio": "/sys/fs/cgroup/blkio/system.slice/system-runc.slice/chrome",
    "cpu": "/sys/fs/cgroup/cpu,cpuacct/system.slice/system-runc.slice/chrome",
    "cpuacct": "/sys/fs/cgroup/cpu,cpuacct/system.slice/system-runc.slice/chrome",
    "cpuset": "/sys/fs/cgroup/cpuset/system.slice/system-runc.slice/chrome",
    "devices": "/sys/fs/cgroup/devices/system.slice/system-runc.slice/chrome",
    "freezer": "/sys/fs/cgroup/freezer/system.slice/system-runc.slice/chrome",
    "hugetlb": "/sys/fs/cgroup/hugetlb/system.slice/system-runc.slice/chrome",
    "memory": "/sys/fs/cgroup/memory/system.slice/system-runc.slice/chrome",
    "name=systemd": "/sys/fs/cgroup/systemd/system.slice/system-runc.slice/chrome",
    "net_cls": "/sys/fs/cgroup/net_cls,net_prio/system.slice/system-runc.slice/chrome",
    "net_prio": "/sys/fs/cgroup/net_cls,net_prio/system.slice/system-runc.slice/chrome",
    "perf_event": "/sys/fs/cgroup/perf_event/system.slice/system-runc.slice/chrome",
    "pids": "/sys/fs/cgroup/pids/system.slice/system-runc.slice/chrome"
  },

whereas on v0.9 it is correctly:

 "cgroup_paths": {
    "blkio": "/sys/fs/cgroup/blkio/user.slice/htop",
    "cpu": "/sys/fs/cgroup/cpu,cpuacct/user.slice/htop",
    "cpuacct": "/sys/fs/cgroup/cpu,cpuacct/user.slice/htop",
    "cpuset": "/sys/fs/cgroup/cpuset/user.slice/htop",
    "devices": "/sys/fs/cgroup/devices/user.slice/htop",
    "freezer": "/sys/fs/cgroup/freezer/user.slice/htop",
    "hugetlb": "/sys/fs/cgroup/hugetlb/user.slice/htop",
    "memory": "/sys/fs/cgroup/memory/user.slice/htop",
    "name=systemd": "/sys/fs/cgroup/systemd/user.slice/htop",
    "net_cls": "/sys/fs/cgroup/net_cls,net_prio/user.slice/htop",
    "net_prio": "/sys/fs/cgroup/net_cls,net_prio/user.slice/htop",
    "perf_event": "/sys/fs/cgroup/perf_event/user.slice/htop",
    "pids": "/sys/fs/cgroup/pids/user.slice/htop"
  },
@jessfraz
Copy link
Contributor Author

systemd whyyyyyyyyy you troll me

@runcom
Copy link
Member

runcom commented Mar 19, 2016

I get correct results on master:

cat /run/runc/test/state.json  | jq '.cgroup_paths'
{
  "blkio": "/sys/fs/cgroup/blkio/user.slice/test",
  "cpu": "/sys/fs/cgroup/cpu,cpuacct/user.slice/test",
  "cpuacct": "/sys/fs/cgroup/cpu,cpuacct/user.slice/test",
  "cpuset": "/sys/fs/cgroup/cpuset/user.slice/test",
  "devices": "/sys/fs/cgroup/devices/user.slice/test",
  "freezer": "/sys/fs/cgroup/freezer/user.slice/test",
  "hugetlb": "/sys/fs/cgroup/hugetlb/user.slice/test",
  "memory": "/sys/fs/cgroup/memory/user.slice/test",
  "name=systemd": "/sys/fs/cgroup/systemd/user.slice/test",
  "net_cls": "/sys/fs/cgroup/net_cls,net_prio/user.slice/test",
  "net_prio": "/sys/fs/cgroup/net_cls,net_prio/user.slice/test",
  "perf_event": "/sys/fs/cgroup/perf_event/user.slice/test",
  "pids": "/sys/fs/cgroup/pids/user.slice/test"
}

@jessfraz
Copy link
Contributor Author

ugh what is your systemd version, i have

systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

@jessfraz
Copy link
Contributor Author

ok so its not the json that is the problem, like if i have a container w the systemd.slice json (aka created with master) and use the v0.9 binary to run events for that container it works, but runc events master just hangs

@jessfraz
Copy link
Contributor Author

this is the weirdest bug ever

@marcosnils
Copy link
Contributor

@jfrazelle I'm about to try this. Let you know in a couple of mins

@jessfraz
Copy link
Contributor Author

thanks I can't tell if im going insane

@jessfraz
Copy link
Contributor Author

also i get this when checking the max

$ sudo cat /sys/fs/cgroup/pids/user.slice/htop/pids.max
max

jessie at debian in ~/runc on master
$ sudo cat /sys/fs/cgroup/pids/user.slice/htop/pids.current 
1

@jessfraz
Copy link
Contributor Author

so if I dont set pids limit in my config I get max if its set i get what i set it to, so i think its a default problem or something

@marcosnils
Copy link
Contributor

@jfrazelle works for me:

marcos@XPS:~/Projects/containers/test$ sudo cat /run/runc/test/state.json  | jq '.cgroup_paths'                       
{
  "blkio": "/sys/fs/cgroup/blkio/user.slice/user-1000.slice/session-c2.scope/test",
  "cpu": "/sys/fs/cgroup/cpu,cpuacct/user.slice/user-1000.slice/session-c2.scope/test",
  "cpuacct": "/sys/fs/cgroup/cpu,cpuacct/user.slice/user-1000.slice/session-c2.scope/test",
  "cpuset": "/sys/fs/cgroup/cpuset/user.slice/user-1000.slice/session-c2.scope/test",
  "devices": "/sys/fs/cgroup/devices/user.slice/user-1000.slice/session-c2.scope/test",
  "freezer": "/sys/fs/cgroup/freezer/user.slice/user-1000.slice/session-c2.scope/test",
  "hugetlb": "/sys/fs/cgroup/hugetlb/user.slice/user-1000.slice/session-c2.scope/test",
  "memory": "/sys/fs/cgroup/memory/user.slice/user-1000.slice/session-c2.scope/test",
  "name=systemd": "/sys/fs/cgroup/systemd/user.slice/user-1000.slice/session-c2.scope/test",
  "net_cls": "/sys/fs/cgroup/net_cls,net_prio/user.slice/user-1000.slice/session-c2.scope/test",
  "net_prio": "/sys/fs/cgroup/net_cls,net_prio/user.slice/user-1000.slice/session-c2.scope/test",
  "perf_event": "/sys/fs/cgroup/perf_event/user.slice/user-1000.slice/session-c2.scope/test"
}

@jessfraz
Copy link
Contributor Author

you dont have pids cgroup, i figured out the problem its #663 (comment)

@jessfraz jessfraz changed the title runc events broken on master pid.max cgroup return "max", runc events broken on master Mar 19, 2016
@jessfraz jessfraz changed the title pid.max cgroup return "max", runc events broken on master pids.max cgroup return "max", runc events broken on master Mar 19, 2016
@marcosnils
Copy link
Contributor

@jfrazelle good catch. Using old kernels has it's advantages :trollface:

@cyphar
Copy link
Member

cyphar commented Mar 20, 2016

@jfrazelle Sorry, this was my fault. There's a fix in #644. Basically, it's because I forgot to account for the string "max" so parsing fails. The reason it wasn't merged sooner is because there was some discussion about what value we should return to represent "max".

@jessfraz
Copy link
Contributor Author

oh nice i had opened one too :) but your will hopefully get in first!

I closed mine :)

@mrunalp
Copy link
Contributor

mrunalp commented Mar 21, 2016

@jfrazelle Can we close this now that #644 is merged? :)

@jessfraz
Copy link
Contributor Author

thanks for reminding me sorry

@mrunalp
Copy link
Contributor

mrunalp commented Mar 21, 2016

Thanks! No worries :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants