Add in support for /etc/default/datadog-agent and set nicelevel from it

[directionless]

What does this PR do?

This adds support for /etc/default/datadog-agent into the debian and centos init scripts.
It then uses the NICELEVEL if set.

In debian, start-stop-daemon has a --nicelevel argument.

In centos, daemon does not, but seems to honor the NICELEVEL environmental variable

Motivation

In my environment, we run sshd at a high priority to aid in access during host issues. This higher priority is inherited by apt, and by the init script. Thus accidentally causing datadog-agent to run at high priority

We're finding that when datadog-agent runs at a high priority, gohai ends up causing some CPU locks. I referenced this in #3085 This is workaround.

Additional Notes

I don't know how to add this to the systemd configs. Though they won't have the same inheritance issue I'm running into.

[davexunit]

This patch would be extremely useful to me. Any chance of it getting merged soon?

[olivielpeau]

Thanks @directionless for your PR! And sorry for not responding earlier.

Overall I think this change makes sense. Pinging @truthbk to see what he thinks about it though :)

I'll put this on the Triage milestone until we have a final decision on whether we want to support this in the long term.

Systemd unit files support the EnvironmentFile= and Nice= options, that said I think the convention on systemd is to let users copy the default unit file in /lib/systemd/system/datadog-agent.service to /etc/systemd/system/datadog-agent and make modifications on the latter. Does that make sense to you?

Thanks again!

[olivielpeau]

on centos I think the convention is to source /etc/sysconfig/datadog-agent instead of /etc/default/datadog-agent

[directionless]

Okay! I don't know the centos conventions. I'll push an update

[directionless]

systemd does have more options. But it's easy for end users to adjust the service files.

One idiom I've seen is to set EnvironmentFile to that config file. Eg: https://github.com/facebook/osquery/blob/master/tools/deployment/osqueryd.service I'm not sure if that makes sense here.