Lab setup data from the Pluralsight course on:
Linux Performance Monitoring and Tuning
The Complete Obsolete Guide to Generative AI (from Manning) is a lighthearted look at programming with AI, as well as a rock-solid resource for getting the most out of these insanely powerful services. Let it be your guide to analyzing massive data sources, summarize pages and pages of text, and scour the live internet.
Here’s a relevant “public service announcement” from Brendan Gregg, senior performance architect at Netflix. In this short YouTube video, Brendan shows how CPU utilization percentages can sometimes provide little value for understanding what’s really going on your CPU. His blog post explains everything in greater detail.
Assessing Linux CPU and Memory Usage
less cpuinfo | grep processor uptime top ps aux | grep <a PID> sudo journalctl _PID=<a PID> less /proc/meminfo free -h vmstat 1 4
Controlling Process Access to Resources
yes > /dev/null & ps kill <PID> killall yes systemctl status mysql systemctl disable mysql Controlling Process Using NICE you may get strange top results doing this in an LXC container...VirtualBox will probably work better] nano stresser.sh #!/bin/bash # stress CPU while true; do true; done chmod +x stresser.sh ./stresser.sh & nice -19 ./stresser1.sh & Controlling Processes Using CGROUPS yum install libcgroup libcgroup-tools apt install cgroup-bin cgroup-lite cgroup-tools cgroupfs-mount libcgroup1 cgcreate -a ubuntu -g cpu:testgroup /sys/fs/cgroup/cpu.cfs_period_us /sys/fs/cgroup/cpu.cfs_quota_us cgset -r cpu.cfs_quota_us=25000 testgroup cgexec -g cpu:testgroup ./stresser.sh cgcreate -a ubuntu -g memory:stopmarvin cgset -r memory.limit_in_bytes=512m stopmarvin cgexec -g memory:stopmarvin /opt/google/chrome/chrome [do not run] ps aux | grep chrome nano myscript.sh #!/bin/bash cgcreate -a ubuntu -g cpu:testgroup cgset -r cpu.cfs_quota_us=25000 testgroup cgcreate -a ubuntu -g memory:stopmarvin cgset -r memory.limit_in_bytes=512m stopmarvin sudo nano /etc/anacrontab 1 2 /home/ubuntu/myscript.sh
Assessing Storage Availability
sudo mount /dev/sdb1 /media/my-usb-mount less /etc/logrotate.conf ls /etc/logrotate.d/ ls /var/log/journal/ less /etc/systemd/journald.conf Inodes df -ih find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n apt autoremove [Ubuntu] package-cleanup [CentOS]
Controlling Access to Network Bandwidth
sudo iftop -i eth0 sudo nethogs eth0 Traffic Control (tc) tc -s qdisc ls dev eth0 tc qdisc add dev eth0 root netem delay 100ms tc -s qdisc ls dev eth0
Installing and Configuring a Standalone Collectd Server
apt install -y collectd python git build-essential librrds-perl libjson-perl libhtml-parser-perl apache2 a2enmod cgi cgid cpan JSON cpan CGI nano /etc/apache2/sites-available/000-default.conf <Directory /var/www/html/collectd-web/cgi-bin> Options Indexes ExecCGI AllowOverride All AddHandler cgi-script .cgi Require all granted </Directory> nano /etc/apache2/ports.conf Listen 0.0.0.0:80 git clone https://github.com/httpdss/collectd-web.git cd collectd-web cd cgi-bin chmod +x graphdefs.cgi cd .. nano runserver.py change 127.0.0.1 to 0.0.0.0 ./runserver.py &
Setting Up Client Monitoring with Collectd
You should launch one or two clients. Consider setting descriptive hostnames
hostname collectd-client1 [on clients] nano /etc/collectd/collectd.conf uncomment open and close <plugin> lines uncomment open and close <server> lines in client setup add server IP to Server line systemctl restart collectd [on servers] nano /etc/collectd/collectd.conf
uncomment open and close <plugin> lines
uncomment open and close <listen> lines in client setup
add 0.0.0.0 to Listen line
systemctl restart collectd
Notification Resources:
https://collectd.org/wiki/index.php/Notifications_and_thresholds
https://collectd.org/documentation/manpages/collectd.conf.5.shtml#threshold_configuration
Installing Nagios on Ubuntu
apt install -y autoconf gcc libc6 make wget unzip apache2 php libapache2-mod-php7.0 libgd2-xpm-dev useradd nagios usermod -a -G nagios www-data wget -O nagioscore.tar.gz https://github.com/NagiosEnterprises/nagioscore/archive/nagios-4.3.4.tar.gz tar xzf nagioscore.tar.gz cd /nagioscore-nagios-4.3.4/ ./configure --with-httpd-conf=/etc/apache2/sites-enabled make all make install make install-init update-rc.d nagios defaults make install-commandmode make install-config make install-webconf a2enmod rewrite a2enmod cgi htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin systemctl restart apache2 systemctl start nagios
Installing Nagios Plugins
apt install -y libmcrypt-dev make libssl-dev wget bc gawk dc build-essential snmp libnet-snmp-perl gettext wget --no-check-certificate -O nagios-plugins.tar.gz https://github.com/nagios-plugins/nagios-plugins/archive/release-2.2.1.tar.gz tar zxf nagios-plugins.tar.gz cd nagios-plugins-release-2.2.1/ ./tools/setup ./configure make make install systemctl restart nagios ls /usr/local/nagios/libexec
Nagios Notification Configuration
apt install postfix mailutils nano /etc/postfix/main.cf inet_interfaces = localhost systemctl restart postfix nano /usr/local/nagios/etc/objects/contacts.cft systemctl restart nagios
NRPE
[on server] wget --no-check-certificate -O nrpe.tar.gz https://github.com/NagiosEnterprises/nrpe/archive/nrpe-3.2.1.tar.gz tar zxf nrpe.tar.gz cd nrpe-*** ./configure --enable-command-args --with-ssl-lib=/usr/lib/x86_64-linux-gnu/ make all make install make install-config sh -c "echo >> /etc/services" sh -c "sudo echo '# Nagios services' >> /etc/services" sh -c "sudo echo 'nrpe 5666/tcp' >> /etc/services" make install-init systemctl enable nrpe cd /usr/local/nagios/etc nano nagios.cfg cfg_dir=/usr/local/nagios/etc/servers mkdir /usr/local/nagios/etc/servers cd objects nano commands.cfg [add (to enable remote host checking):] define command{ command_name check_nrpe command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ } [on client] useradd nagios apt install build-essential libgd2-xpm-dev openssl libssl-dev autoconf gcc libc6 libmcrypt-dev make unzip wget curl -L -O http://nagios-plugins.org/download/nagios-plugins-2.2.1.tar.gz [http://nagios-plugins.org/downloads] tar zxf nagios-plugins-*.tar.gz cd nagios-plugins-* ./configure --with-nagios-user=nagios --with-nagios-group=nagios --with-openssl make make install cd /home/ubuntu curl -L -O https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-3.2.1/nrpe-3.2.1.tar.gz tar zxf nrpe-*.tar.gz cd nrpe-* ./configure --enable-command-args --with-nagios-user=nagios --with-nagios-group=nagios --with-ssl=/usr/bin/openssl --with-ssl-lib=/usr/lib/x86_64-linux-gnu make all make install make install-config make install-init cd /usr/local/nagios/etc nano nrpe.cfg server_address=192.168.1.27 allowed_hosts=127.0.0.1,::1,192.168.1.23 command[check_sda2]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/sda2 systemctl restart nrpe [on server] /usr/local/nagios/libexec/check_nrpe -H <remote_host_ip> nano /usr/local/nagios/etc/servers/nagios-client.cfg define host { use linux-server host_name nagios-client alias Client1 address 192.168.1.27 max_check_attempts 5 check_period 24x7 notification_interval 30 notification_period 24x7 } define service { use generic-service host_name nagios-client service_description CPU load check_command check_nrpe!check_load } define service { use generic-service host_name nagios-client service_description /dev/sda2 free space check_command check_nrpe!check_sda2 } systemctl restart nagios systemctl enable nagios /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
Munin
apt install apache2 munin nano /etc/munin/munin.conf ## First our "normal" host. [munin-node] address 10.0.3.159 /etc/init.d/munin start systemctl restart apache2 [on node] apt install munin-node nano /etc/munin/munin-node.conf allow ^10\.0\.3\.252$ systemctl restart munin-node [on server] less /etc/munin/apache24.conf mv /etc/munin/apache24.conf /etc/munin/apache24.conf.backup nano /etc/muninapache24.conf Alias /munin /var/cache/munin/www <Directory /var/cache/munin/www> # Require local Require all granted Options FollowSymLinks SymLinksIfOwnerMatch Options None </Directory> ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph <Location /munin-cgi/munin-cgi-graph> # Require local Require all granted Options FollowSymLinks SymLinksIfOwnerMatch <IfModule mod_fcgid.c> SetHandler fcgid-script </IfModule> <IfModule !mod_fcgid.c> SetHandler cgi-script </IfModule> </Location> /etc/init.d/munin start systemctl restart apache2
NMON
[https://sourceforge.net/projects/nmon/files/] wget http://sourceforge.net/projects/nmon/files/nmonchart32.tar sudo apt install ksh sudo ./nmonchart <xxx>.nmon /var/www/html/datafile.html