Linux Tuning

Lab setup data from the Pluralsight course on:

Linux Performance Monitoring and Tuning


The Complete Obsolete Guide to Generative AI (from Manning) is a lighthearted look at programming with AI, as well as a rock-solid resource for getting the most out of these insanely powerful services. Let it be your guide to analyzing massive data sources, summarize pages and pages of text, and scour the live internet.

 


Here’s a relevant “public service announcement” from Brendan Gregg, senior performance architect at Netflix. In this short YouTube video, Brendan shows how CPU utilization percentages can sometimes provide little value for understanding what’s really going on your CPU. His blog post explains everything in greater detail.


Assessing Linux CPU and Memory Usage

less cpuinfo | grep processor
uptime 
top 
ps aux | grep <a PID>
sudo journalctl _PID=<a PID>

less /proc/meminfo 
free -h
vmstat 1 4

Controlling Process Access to Resources

yes > /dev/null &
ps
kill <PID>
killall yes

systemctl status mysql 
systemctl disable mysql 
Controlling Process Using NICE
you may get strange top results doing this in an LXC container...VirtualBox will probably work better]

nano stresser.sh

#!/bin/bash
# stress CPU
while true; do true; done

chmod +x stresser.sh
./stresser.sh &

nice -19 ./stresser1.sh &
Controlling Processes Using CGROUPS

yum install libcgroup libcgroup-tools
apt install cgroup-bin cgroup-lite cgroup-tools cgroupfs-mount libcgroup1

cgcreate -a ubuntu -g cpu:testgroup
/sys/fs/cgroup/cpu.cfs_period_us
/sys/fs/cgroup/cpu.cfs_quota_us
cgset -r cpu.cfs_quota_us=25000 testgroup

cgexec -g cpu:testgroup ./stresser.sh

cgcreate -a ubuntu -g memory:stopmarvin
cgset -r memory.limit_in_bytes=512m stopmarvin
cgexec -g memory:stopmarvin /opt/google/chrome/chrome [do not run]
ps aux | grep chrome

nano myscript.sh

#!/bin/bash
cgcreate -a ubuntu -g cpu:testgroup
cgset -r cpu.cfs_quota_us=25000 testgroup
cgcreate -a ubuntu -g memory:stopmarvin
cgset -r memory.limit_in_bytes=512m stopmarvin

sudo nano /etc/anacrontab

1 2 /home/ubuntu/myscript.sh

Assessing Storage Availability

sudo mount /dev/sdb1 /media/my-usb-mount

less /etc/logrotate.conf
ls /etc/logrotate.d/

ls /var/log/journal/
less /etc/systemd/journald.conf
Inodes

df -ih
find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
apt autoremove [Ubuntu]
package-cleanup [CentOS]

Controlling Access to Network Bandwidth

sudo iftop -i eth0
sudo nethogs eth0
Traffic Control (tc)

tc -s qdisc ls dev eth0
tc qdisc add dev eth0 root netem delay 100ms
tc -s qdisc ls dev eth0

Installing and Configuring a Standalone Collectd Server

apt install -y collectd python git build-essential librrds-perl libjson-perl libhtml-parser-perl apache2
a2enmod cgi cgid
cpan JSON
cpan CGI

nano /etc/apache2/sites-available/000-default.conf

<Directory /var/www/html/collectd-web/cgi-bin>
Options Indexes ExecCGI
AllowOverride All
AddHandler cgi-script .cgi
Require all granted
</Directory>

nano /etc/apache2/ports.conf

Listen 0.0.0.0:80

git clone https://github.com/httpdss/collectd-web.git
cd collectd-web
cd cgi-bin
chmod +x graphdefs.cgi
cd ..
nano runserver.py

change 127.0.0.1 to 0.0.0.0

./runserver.py &

Setting Up Client Monitoring with Collectd

You should launch one or two clients. Consider setting descriptive hostnames

hostname collectd-client1

[on clients]
nano /etc/collectd/collectd.conf 
uncomment open and close <plugin> lines
uncomment open and close <server> lines in client setup
add server IP to Server line
systemctl restart collectd
[on servers]
nano /etc/collectd/collectd.conf

uncomment open and close <plugin> lines
uncomment open and close <listen> lines in client setup
add 0.0.0.0 to Listen line

systemctl restart collectd

Notification Resources:

https://collectd.org/wiki/index.php/Notifications_and_thresholds
https://collectd.org/documentation/manpages/collectd.conf.5.shtml#threshold_configuration

Installing Nagios on Ubuntu

apt install -y autoconf gcc libc6 make wget unzip apache2 php libapache2-mod-php7.0 libgd2-xpm-dev
useradd nagios
usermod -a -G nagios www-data
wget -O nagioscore.tar.gz https://github.com/NagiosEnterprises/nagioscore/archive/nagios-4.3.4.tar.gz
tar xzf nagioscore.tar.gz
cd /nagioscore-nagios-4.3.4/
./configure --with-httpd-conf=/etc/apache2/sites-enabled

make all
make install
make install-init
update-rc.d nagios defaults
make install-commandmode
make install-config
make install-webconf
a2enmod rewrite
a2enmod cgi

htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
systemctl restart apache2
systemctl start nagios

Installing Nagios Plugins

apt install -y libmcrypt-dev make libssl-dev wget bc gawk dc build-essential snmp libnet-snmp-perl gettext
wget --no-check-certificate -O nagios-plugins.tar.gz https://github.com/nagios-plugins/nagios-plugins/archive/release-2.2.1.tar.gz
tar zxf nagios-plugins.tar.gz
cd nagios-plugins-release-2.2.1/
./tools/setup
./configure
make
make install
systemctl restart nagios
ls /usr/local/nagios/libexec

Nagios Notification Configuration

apt install postfix mailutils
nano /etc/postfix/main.cf 
inet_interfaces = localhost
systemctl restart postfix

nano /usr/local/nagios/etc/objects/contacts.cft
systemctl restart nagios

NRPE

[on server]
wget --no-check-certificate -O nrpe.tar.gz https://github.com/NagiosEnterprises/nrpe/archive/nrpe-3.2.1.tar.gz
tar zxf nrpe.tar.gz
cd nrpe-***
./configure --enable-command-args --with-ssl-lib=/usr/lib/x86_64-linux-gnu/
make all
make install
make install-config
sh -c "echo >> /etc/services"
sh -c "sudo echo '# Nagios services' >> /etc/services"
sh -c "sudo echo 'nrpe 5666/tcp' >> /etc/services"
make install-init
systemctl enable nrpe

cd /usr/local/nagios/etc
nano nagios.cfg

cfg_dir=/usr/local/nagios/etc/servers

mkdir /usr/local/nagios/etc/servers 
cd objects
nano commands.cfg
[add (to enable remote host checking):]

define command{
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

[on client]
useradd nagios
apt install build-essential libgd2-xpm-dev openssl libssl-dev autoconf gcc libc6 libmcrypt-dev make unzip wget
curl -L -O http://nagios-plugins.org/download/nagios-plugins-2.2.1.tar.gz
[http://nagios-plugins.org/downloads]
tar zxf nagios-plugins-*.tar.gz
cd nagios-plugins-*
./configure --with-nagios-user=nagios --with-nagios-group=nagios --with-openssl
make
make install

cd /home/ubuntu
curl -L -O https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-3.2.1/nrpe-3.2.1.tar.gz
tar zxf nrpe-*.tar.gz
cd nrpe-*
./configure --enable-command-args --with-nagios-user=nagios --with-nagios-group=nagios --with-ssl=/usr/bin/openssl --with-ssl-lib=/usr/lib/x86_64-linux-gnu
make all
make install
make install-config
make install-init

cd /usr/local/nagios/etc
nano nrpe.cfg

server_address=192.168.1.27
allowed_hosts=127.0.0.1,::1,192.168.1.23
command[check_sda2]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/sda2

systemctl restart nrpe

[on server]
/usr/local/nagios/libexec/check_nrpe -H <remote_host_ip>

nano /usr/local/nagios/etc/servers/nagios-client.cfg

define host {
use linux-server
host_name nagios-client
alias Client1
address 192.168.1.27
max_check_attempts 5
check_period 24x7
notification_interval 30
notification_period 24x7
}
define service {
use generic-service
host_name nagios-client
service_description CPU load
check_command check_nrpe!check_load
}
define service {
use generic-service
host_name nagios-client
service_description /dev/sda2 free space
check_command check_nrpe!check_sda2
}

systemctl restart nagios
systemctl enable nagios
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Munin

apt install apache2 munin
nano /etc/munin/munin.conf

## First our "normal" host.
[munin-node]
address 10.0.3.159

/etc/init.d/munin start 
systemctl restart apache2

[on node]
apt install munin-node
nano /etc/munin/munin-node.conf

allow ^10\.0\.3\.252$

systemctl restart munin-node

[on server]
less /etc/munin/apache24.conf
mv /etc/munin/apache24.conf /etc/munin/apache24.conf.backup
nano /etc/muninapache24.conf
Alias /munin /var/cache/munin/www
<Directory /var/cache/munin/www>
# Require local
Require all granted
Options FollowSymLinks SymLinksIfOwnerMatch
Options None
</Directory>

ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
<Location /munin-cgi/munin-cgi-graph>
# Require local
Require all granted
Options FollowSymLinks SymLinksIfOwnerMatch
<IfModule mod_fcgid.c>
SetHandler fcgid-script
</IfModule>
<IfModule !mod_fcgid.c>
SetHandler cgi-script
</IfModule>
</Location>

/etc/init.d/munin start 
systemctl restart apache2

NMON

[https://sourceforge.net/projects/nmon/files/]
wget http://sourceforge.net/projects/nmon/files/nmonchart32.tar 
sudo apt install ksh
sudo ./nmonchart <xxx>.nmon /var/www/html/datafile.html