Cluster filesystem utilization alerts
This is a quick and raw method to setup alerts when the filesystem fill above threshold.
Pre-requisites
- Monitored filesystems should be consistent, meaning available across all nodes
- passwordless ssh should be setup between the nodes.
Node where the alert script is running should be able to talk to other nodes over ssh. - Prepare the list of hostnames to be monitored.
This will be passed to pdsh using WCOLL variable.
Steps
- Create a bash script .sh file with below content
#!/bin/bash -x export PDSH_RCMD_TYPE='ssh' export WCOLL='/path/to/hosts-file' pdsh "df -Ph /tmp /var /home | grep /" | sed s/%//g | awk '$6 >= 85 {printf("%s%s\tis %d percent.\n", $1, $7, $6) }' | mail -s "Environment filesystem(s) almost full" -E -r [email protected] -c [email protected] [email protected]
- Change variables in above script, example
85 above, is threshold value
WCOLL param to the file containing list of hosts (one per line)
email addresses – add system admin and the group distribution list - Setup cron for the script file
HTH