Lab setup data from the Pluralsight course on:
Linux Performance Monitoring and Tuning
The Complete Obsolete Guide to Generative AI (from Manning) is a lighthearted look at programming with AI, as well as a rock-solid resource for getting the most out of these insanely powerful services. Let it be your guide to analyzing massive data sources, summarize pages and pages of text, and scour the live internet.
Here’s a relevant “public service announcement” from Brendan Gregg, senior performance architect at Netflix. In this short YouTube video, Brendan shows how CPU utilization percentages can sometimes provide little value for understanding what’s really going on your CPU. His blog post explains everything in greater detail.
Assessing Linux CPU and Memory Usage
less cpuinfo | grep processor uptime top ps aux | grep <a PID> sudo journalctl _PID=<a PID> less /proc/meminfo free -h vmstat 1 4
Controlling Process Access to Resources
yes > /dev/null & ps kill <PID> killall yes systemctl status mysql systemctl disable mysql Controlling Process Using NICE you may get strange top results doing this in an LXC container...VirtualBox will probably work better] nano stresser.sh #!/bin/bash # stress CPU while true; do true; done chmod +x stresser.sh ./stresser.sh & nice -19 ./stresser1.sh & Controlling Processes Using CGROUPS yum install libcgroup libcgroup-tools apt install cgroup-bin cgroup-lite cgroup-tools cgroupfs-mount libcgroup1 cgcreate -a ubuntu -g cpu:testgroup /sys/fs/cgroup/cpu.cfs_period_us /sys/fs/cgroup/cpu.cfs_quota_us cgset -r cpu.cfs_quota_us=25000 testgroup cgexec -g cpu:testgroup ./stresser.sh cgcreate -a ubuntu -g memory:stopmarvin cgset -r memory.limit_in_bytes=512m stopmarvin cgexec -g memory:stopmarvin /opt/google/chrome/chrome [do not run] ps aux | grep chrome nano myscript.sh #!/bin/bash cgcreate -a ubuntu -g cpu:testgroup cgset -r cpu.cfs_quota_us=25000 testgroup cgcreate -a ubuntu -g memory:stopmarvin cgset -r memory.limit_in_bytes=512m stopmarvin sudo nano /etc/anacrontab 1 2 /home/ubuntu/myscript.sh
Assessing Storage Availability
sudo mount /dev/sdb1 /media/my-usb-mount less /etc/logrotate.conf ls /etc/logrotate.d/ ls /var/log/journal/ less /etc/systemd/journald.conf Inodes df -ih find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n apt autoremove [Ubuntu] package-cleanup [CentOS]
Controlling Access to Network Bandwidth
sudo iftop -i eth0 sudo nethogs eth0 Traffic Control (tc) tc -s qdisc ls dev eth0 tc qdisc add dev eth0 root netem delay 100ms tc -s qdisc ls dev eth0
Installing and Configuring a Standalone Collectd Server
apt install -y collectd python git build-essential librrds-perl libjson-perl libhtml-parser-perl apache2 a2enmod cgi cgid cpan JSON cpan CGI nano /etc/apache2/sites-available/000-default.conf <Directory /var/www/html/collectd-web/cgi-bin> Options Indexes ExecCGI AllowOverride All AddHandler cgi-script .cgi Require all granted </Directory> nano /etc/apache2/ports.conf Listen 0.0.0.0:80 git clone https://github.com/httpdss/collectd-web.git cd collectd-web cd cgi-bin chmod +x graphdefs.cgi cd .. nano runserver.py change 127.0.0.1 to 0.0.0.0 ./runserver.py &
Setting Up Client Monitoring with Collectd
You should launch one or two clients. Consider setting descriptive hostnames
hostname collectd-client1 [on clients] nano /etc/collectd/collectd.conf uncomment open and close <plugin> lines uncomment open and close <server> lines in client setup add server IP to Server line systemctl restart collectd [on servers] nano /etc/collectd/collectd.conf
uncomment open and close <plugin> lines
uncomment open and close <listen> lines in client setup
add 0.0.0.0 to Listen line
systemctl restart collectd
Notification Resources:
https://collectd.org/wiki/index.php/Notifications_and_thresholds
https://collectd.org/documentation/manpages/collectd.conf.5.shtml#threshold_configuration
Installing Nagios on Ubuntu
apt install -y autoconf gcc libc6 make wget unzip apache2 php libapache2-mod-php7.0 libgd2-xpm-dev useradd nagios usermod -a -G nagios www-data wget -O nagioscore.tar.gz https://github.com/NagiosEnterprises/nagioscore/archive/nagios-4.3.4.tar.gz tar xzf nagioscore.tar.gz cd /nagioscore-nagios-4.3.4/ ./configure --with-httpd-conf=/etc/apache2/sites-enabled make all make install make install-init update-rc.d nagios defaults make install-commandmode make install-config make install-webconf a2enmod rewrite a2enmod cgi htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin systemctl restart apache2 systemctl start nagios
Installing Nagios Plugins
apt install -y libmcrypt-dev make libssl-dev wget bc gawk dc build-essential snmp libnet-snmp-perl gettext wget --no-check-certificate -O nagios-plugins.tar.gz https://github.com/nagios-plugins/nagios-plugins/archive/release-2.2.1.tar.gz tar zxf nagios-plugins.tar.gz cd nagios-plugins-release-2.2.1/ ./tools/setup ./configure make make install systemctl restart nagios ls /usr/local/nagios/libexec
Nagios Notification Configuration
apt install postfix mailutils nano /etc/postfix/main.cf inet_interfaces = localhost systemctl restart postfix nano /usr/local/nagios/etc/objects/contacts.cft systemctl restart nagios
NRPE
[on server]
wget --no-check-certificate -O nrpe.tar.gz https://github.com/NagiosEnterprises/nrpe/archive/nrpe-3.2.1.tar.gz
tar zxf nrpe.tar.gz
cd nrpe-***
./configure --enable-command-args --with-ssl-lib=/usr/lib/x86_64-linux-gnu/
make all
make install
make install-config
sh -c "echo >> /etc/services"
sh -c "sudo echo '# Nagios services' >> /etc/services"
sh -c "sudo echo 'nrpe 5666/tcp' >> /etc/services"
make install-init
systemctl enable nrpe
cd /usr/local/nagios/etc
nano nagios.cfg
cfg_dir=/usr/local/nagios/etc/servers
mkdir /usr/local/nagios/etc/servers
cd objects
nano commands.cfg
[add (to enable remote host checking):]
define command{
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}
[on client]
useradd nagios
apt install build-essential libgd2-xpm-dev openssl libssl-dev autoconf gcc libc6 libmcrypt-dev make unzip wget
curl -L -O http://nagios-plugins.org/download/nagios-plugins-2.2.1.tar.gz
[http://nagios-plugins.org/downloads]
tar zxf nagios-plugins-*.tar.gz
cd nagios-plugins-*
./configure --with-nagios-user=nagios --with-nagios-group=nagios --with-openssl
make
make install
cd /home/ubuntu
curl -L -O https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-3.2.1/nrpe-3.2.1.tar.gz
tar zxf nrpe-*.tar.gz
cd nrpe-*
./configure --enable-command-args --with-nagios-user=nagios --with-nagios-group=nagios --with-ssl=/usr/bin/openssl --with-ssl-lib=/usr/lib/x86_64-linux-gnu
make all
make install
make install-config
make install-init
cd /usr/local/nagios/etc
nano nrpe.cfg
server_address=192.168.1.27
allowed_hosts=127.0.0.1,::1,192.168.1.23
command[check_sda2]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/sda2
systemctl restart nrpe
[on server]
/usr/local/nagios/libexec/check_nrpe -H <remote_host_ip>
nano /usr/local/nagios/etc/servers/nagios-client.cfg
define host {
use linux-server
host_name nagios-client
alias Client1
address 192.168.1.27
max_check_attempts 5
check_period 24x7
notification_interval 30
notification_period 24x7
}
define service {
use generic-service
host_name nagios-client
service_description CPU load
check_command check_nrpe!check_load
}
define service {
use generic-service
host_name nagios-client
service_description /dev/sda2 free space
check_command check_nrpe!check_sda2
}
systemctl restart nagios
systemctl enable nagios
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
Munin
apt install apache2 munin nano /etc/munin/munin.conf ## First our "normal" host. [munin-node] address 10.0.3.159 /etc/init.d/munin start systemctl restart apache2 [on node] apt install munin-node nano /etc/munin/munin-node.conf allow ^10\.0\.3\.252$ systemctl restart munin-node [on server] less /etc/munin/apache24.conf mv /etc/munin/apache24.conf /etc/munin/apache24.conf.backup nano /etc/muninapache24.conf Alias /munin /var/cache/munin/www <Directory /var/cache/munin/www> # Require local Require all granted Options FollowSymLinks SymLinksIfOwnerMatch Options None </Directory> ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph <Location /munin-cgi/munin-cgi-graph> # Require local Require all granted Options FollowSymLinks SymLinksIfOwnerMatch <IfModule mod_fcgid.c> SetHandler fcgid-script </IfModule> <IfModule !mod_fcgid.c> SetHandler cgi-script </IfModule> </Location> /etc/init.d/munin start systemctl restart apache2
NMON
[https://sourceforge.net/projects/nmon/files/] wget http://sourceforge.net/projects/nmon/files/nmonchart32.tar sudo apt install ksh sudo ./nmonchart <xxx>.nmon /var/www/html/datafile.html