check-disk.rb docker issue with permissions

[jcputter]

Hi,

I'm running the check-disk.rb plugin on a docker host. If i run /opt/sensu/embedded/bin/check-disk-usage.rb -x cgroup -p /var/lib/docker -c95 -w90 I have the expected output "OK"

However when its running via Sensu

{"timestamp":"2017-08-04T23:42:04.965151+0200","level":"info","message":"publishing check result","payload":{"client":"docker","check":{"command":"/opt/sensu/embedded/bin/check-disk-usage.rb -x cgroup,proc,shm,none,nsfs -p /run/lxcfs,/var/lib/docker -c95 -w90","subscribers":["docker"],"standalone":true,"handlers":["slack","pagerduty"],"interval":30,"occurrences":2,"refresh":1800,"name":"check_disk","issued":1501882924,"executed":1501882924,"duration":0.067,"output":"CheckDisk WARNING: /var/lib/docker/aufs/mnt/0c545df7098b189bf1b11c6efb0c2c0677f12db9636e8b19a726a5775d861f68 Unable to read., /var/lib/docker/containers/ccea5f270de5ac5274314e21b05e87077742ccccf05d6765cf0a10c54830996b/shm Unable to read., /var/lib/docker/aufs/mnt/b87f8b0b649c482c430285f73bfc1a4245bb967b3143c131ee9a0d8bc4aebd37 Unable to read., /var/lib/docker/containers/d130b3529c2e500ec607424c7c26ca6c4371beabd4154c0796e04c6d0d689127/shm Unable to read., /var/lib/docker/aufs/mnt/1115a010f5184a6163414428776e5fec80f90aadeb0c589907cdb8bdeb94185d Unable to read., /var/lib/docker/containers/822aa8148f35280d3463ff11c870c42ff89e312b80cb04362edfdb7c9d2c3204/shm Unable to read., /var/lib/docker/aufs/mnt/d7f1a80ae7293d680a5cf7dd7eae6812d0fc02138a32f85a5d5dadb5352b6c44 Unable to read., /var/lib/docker/containers/cfb46d35c2dfa815c9214b0795359f6c7c8350947cc79751d0cb94e21e06b069/shm Unable to read.\n","status":1}}}

OS: Ubuntu 16.04 LTS
Sensu: 1.0.2
Check-Disk: 2.4.0

[majormoses]

We can do a rescue and unknown but this is certainly a permission thing.

[majormoses]

Actually taking a look at the code that is exactly what it is doing: https://github.com/sensu-plugins/sensu-plugins-disk-checks/blob/2.4.0/bin/check-disk-usage.rb#L149

I have the expected output "OK"
It would but it can't read why would "OK" be expected?

Running the same command:

$ ./bin/check-disk-usage.rb -x cgroup -p /var/lib/docker -c95 -W90
CheckDisk WARNING: /sys/kernel/debug/tracing Unable to read., /run/docker/netns/39eec869e2b1 Unable to read., /run/docker/netns/70778fb53238 Unable to read., /run/docker/netns/abc810a2d2f4 Unable to read.

Verifying that that it is a permissions issue:

$ ls -l /sys/kernel/debug/tracing
ls: cannot access '/sys/kernel/debug/tracing': Permission denied

I think what we need is a flag to ignore not just mount points but also dirs inside them. Does that sound good?

[sysboss]

I have the same issue.

Solved by adding -d flag:
check-disk-usage.rb -x proc -x cgroup -x tmpfs -p sys

[snadorp]

To get around this problem we added these flags: -x tmpfs,overlay,nsfs

[majormoses]

It sounds like we should also write up something in the README with common gotchas and how to solve them.

[majormoses]

Anyone want to attempt a PR to ignore the dirs inside of them and not just the mount points?