AiPBX

Health checks

System → Health checks watches the server and e-mails the chosen people when something stops working — and again when it works again. Every morning it can send a daily summary for managers: yesterday’s calls and faxes and the state of every check.

Checks

Check	Default	Warning / failed when
Asterisk	5 min	Asterisk does not answer (failed); a security update (OpenSSL, the C library, SRTP …) waits for an Asterisk restart and the night restart window is off (warning). Updates of other libraries are not reported
Database	5 min	MariaDB does not answer
Web portal	5 min	the sign-in page does not answer 200
Chat service	5 min	aipbx-chat is not running
Trunks	5 min	an active trunk is unreachable (qualify) or its registration is rejected/unregistered
Disk space	1 h	warning from 85 %, failed from 95 % (/, /var, /var/spool/asterisk)
Memory	15 min	warning under 10 % available, failed under 5 %
E-mail sending	1 h	postfix not running (failed); messages queued for over an hour (warning)
Pending changes	1 h	changes not applied for over an hour
Outgoing faxes	1 h	faxes waiting to be sent for over an hour
Call records	15 min	Asterisk has handled calls since it started (15 min or more ago) but none reached the call history — the CDR database connection is broken; restart Asterisk
Trunk limits	5 min	a trunk refused calls over its channel or calls-per-minute limit in the last 15 minutes (see Trunks)
Failed registrations	15 min	failed SIP registrations or auth attempts in Asterisk’s security log exceed the threshold in 15 minutes (see PBX tuning)
Call quality	5 min	a trunk or extension is losing audio (from RTCP, see Call quality); not used while signaling capture is off
Other alerts	5 min	problems raised in the alert bell (plugins, failed scheduled tasks), so they are e-mailed too
DHCP / TFTP / FTP	15 min	switched on, but the service does not run
Server address	60 min	a stored address (server, phones, coturn, mDNS) is not this server’s any more: see Network settings
Local AI service	15 min	installed and enabled, but not running
TLS certificate	daily	warning 14 days before it ends, failed 3 days before
Daily backup	daily	the newest file in /var/backups/aipbx-daily is older than 26 h
Installation	daily	something install.sh sets up is missing or has the wrong owner/rights (helper, sudo rule, cron, TFTP folder, voicemail folders, masked vsftpd unit)
AiPBX version	daily	a newer release is out — information only, never an alarm

Each check’s interval can be changed on the page (off, 5 or 15 minutes, 1 or 6 hours, daily). A check that does not apply to the server (no trunks, FTP switched off) shows Not used.

Alarms

• E-mails go to the users ticked on the page (in their own language) and to other e-mail addresses.
• Checks that run every 15 minutes or more often report a problem only after two bad results in a row, so a short hiccup stays quiet. Less frequent checks report the first bad result.
• While a problem lasts, the alarm is repeated after the chosen number of hours; a check that gets worse (warning → failed) is reported at once.
• One problem, one e-mail per window. A check that fails, recovers and fails again inside that window is the same incident, so it does not mail again: the window starts at the last e-mail the check sent, and a different check is a different problem.
• Quiet hours (off by default) hold e-mails for warnings and for “back to normal” inside the chosen window (HH:MM-HH:MM, crossing midnight is fine). A failed check is never held: it is mailed at once. Nothing is lost by holding: the alarm is not recorded as reported, so the first run after the window ends mails whatever is still wrong, in one e-mail — and a warning that fixed itself overnight mails nothing, because there is nothing left to say. The daily summary is not held; it is a digest at the time you chose, not an alarm. Empty, malformed or zero-length hours mean no window, so a typo cannot silence alarms; the clock is the server’s, the same one the summary time uses.
• When it works again, a back to normal e-mail follows.
• All problems found in one run go into one e-mail.

The texts are e-mail templates (E-mail → Templates: Health alarm, Health back to normal, Daily summary). A server that is completely down cannot send an e-mail itself: watch it from outside for that (e.g. a monitoring service that opens the sign-in page).

Alert bell

Administrators have a bell in the top bar. Its number is how many problems are open now (red when one has failed); a dot marks changes since you last opened it. It lists:

• every check that is warning or failed now;
• problems plugins report (see Plugin development) and failed scheduled tasks, e-mailed through the check Other alerts;
• the alarm path itself: no recipient chosen (“Alarm e-mails go to nobody”), or no check has run for 30 minutes (“Health checks are not running”, i.e. the cron job stopped);
• below, the changes of the last 7 days (went wrong / back to normal).

It shows state, not a log: an item goes away by itself when its problem is fixed, nothing has to be cleared. Everything in it is also e-mailed to the recipients above when they are set; the bell works without e-mail, so a PBX whose mail is not set up or broken still shows its problems.

Daily summary

At the chosen time (default 08:00), to the same people:

• yesterday’s calls: total, incoming, outgoing, internal, answered, missed, busy, average and total talk time (counted per call, as on the call report), and faxes received / sent / failed;
• the state of every check now;
• the checks that raised an alarm yesterday.

Send the summary now on the page sends it at once (to check the recipients and the mail settings).

Command line

sudo aipbx-selftest

runs every check at once and prints the results; the exit code is 1 when a check fails. It sends no e-mail. install.sh runs it at the end of every installation and update, and CI runs it after a fresh install.

How it works

• /etc/cron.d/aipbx runs /usr/local/bin/aipbx_health.php every minute as root; it runs the checks whose interval has passed (most need root: Asterisk CLI, systemd, file owners), stores the results in health_checks, status changes in health_events (kept 30 days) and sends the e-mails. Check now on the page asks the next run to do every check.
• Checks only read; none of them changes anything on the server.

This site is open source. Improve this page.