NAME
zones_srv_monitor.sh - clear failed services inside OmniCube lipkg and LX zones
SYNOPSIS
/opt/omnicube/sbin/zones_srv_monitor.sh
zones_srv_monitor.sh
DESCRIPTION
zones_srv_monitor.sh looks inside every non-global zone of the host for services that have failed, and clears or restarts them. Both zone brands used by OmniCube are handled: lipkg zones through SMF, and lx zones through systemd. The script takes no options and no operands and is meant for unattended periodic execution from root's crontab, alongside the other periodic jobs whose sample entries are shipped in /opt/omnicube/share/README. It can be run by hand for an immediate sweep.
It runs with set -o pipefail and does not set its own PATH, so the invoking environment must resolve zoneadm, zlogin, svcprop, awk and pfexec; the XPG4 sed is called by absolute path as /usr/xpg4/bin/sed. It sources /opt/omnicube/lib/common/utils.sh for logging, locking, the run-level guard, the exclusion helper and ${PFEXEC}.
Locking
A single-instance lock directory /var/run/omnicube/<site>zones_srv_monitor.zones_srv_monitor.lock is created atomically with acquire_lock_or_exit(). If it already exists the script exits 0 without output, so an overlapping cron tick cannot start a second round of service restarts in the same zones. The lock is released by a trap on EXIT, SIGHUP, SIGINT and SIGTERM, so it is also released when the run-level guard aborts the run.
Run-level guard
Immediately after taking the lock the script calls abort_if_shutting_down and exits 0 silently when the host is at run level 0, 1, 5, 6, S or s, or when /etc/nologin exists. The guard is re-evaluated at the top of both zone loops, so a shutdown starting part way through stops the sweep before the next zone. OC_IGNORE_RUNLEVEL set to a non-empty value overrides it.
lipkg zones
Zones of brand lipkg are taken from zoneadm list -nv. For each one, svcs -xv is run inside the zone with ${PFEXEC} zlogin and the FMRIs of the services reported maintenance since are extracted. If any are found, the action is logged with warning() as Clearing <services> srv on LIPKG zone <zone> and the services are cleared in one call:
${PFEXEC} zlogin <zone> "svcadm clear <services>"
LX zones
Zones of brand lx are handled the same way, using systemctl --failed inside the zone and ignoring units whose state is masked. The action is logged as Clearing <services> srv on LX zone <zone> and remediated with:
${PFEXEC} zlogin <zone> "systemctl restart <services> && systemctl reset-failed"
The service list is passed space-separated in a single invocation, which is what allows the trailing reset-failed to run at all, and what stops the zone's shell from interpreting the second and later service names as separate commands.
Service name validation
The service names come from inside the zone and are interpolated into the command string that the zone's own root shell then evaluates. Each name is therefore checked against ^[a-zA-Z0-9][a-zA-Z0-9:/@._-]*$ before use, a charset that covers real FMRIs such as svc:/system/foo:default and systemd units such as nfs-server.service or foo@bar.service, while excluding whitespace and every shell metacharacter. A zone that reports anything else is logged as Suspicious service name '<name>' reported by zone <zone>, skipping zone with warning() and is skipped in its entirety for that run. The list is split into an array rather than left to an unquoted expansion, so a reported name of * cannot be replaced by the file names in the current directory.
Exclusions
Zones listed exactly in config/monitor_exclude_zones are skipped in both loops. The comparison is literal, character for character, so a zone name containing a dot is not treated as a wildcard. This is the supported way to opt out zones that orchestrate their own service restarts, zones under maintenance or debugging, and zones with deliberately failed services.
Alerting
Every remediation is announced with warning() from utils.sh: stdout plus syslog user.warn under the tag <site>zones_srv_monitor, which /etc/rsyslog.d/omnicube.conf routes to /var/log/misc.log, plus mail to oc-warn from $HOSTNAME@<config/mail_domain>. Mail is throttled per (severity, message) pair, so repeated clears of the same service in the same zone within config/mail_throttle_window seconds, default 3600, produce a single mail. The script does not call error(), so it never mails oc-alert. No timestamp is included in the messages; syslog and the mail header each carry their own.
OPERANDS
This command takes no options and no operands. Any argument given is ignored.
SMF PROPERTIES
Read from svc:/omnicube/sysadm:default unless OC_SMF_INSTANCE selects another instance.
EXAMPLES
Example 1: ten-minute cron entry
*/10 * * * * /opt/omnicube/sbin/zones_srv_monitor.sh >/dev/null
Example 2: opt a zone out of automatic clearing
svccfg -s svc:/omnicube/sysadm:default setprop \\
config/monitor_exclude_zones = astring: ( "zn06efit-ethereum" "other-zone" )
svcadm refresh svc:/omnicube/sysadm:default
Example 3: run one sweep by hand and see what was cleared
pfexec /opt/omnicube/sbin/zones_srv_monitor.sh grep zones_srv_monitor /var/log/misc.log
EXIT STATUS
No other status is produced deliberately. Because there is no final explicit exit, a non-zero status other than those above is the status of the last command executed, typically a failing zlogin remediation in the last zone processed.
FILES
ENVIRONMENT
SECURITY
Entering a zone requires zlogin, granted as uid 0 by the /usr/sbin/zlogin entry in /etc/security/exec_attr.d/omnicube under the root-equivalent OmniCube Administration profile. ${PFEXEC} is used unquoted and expands to nothing when the caller is already root, which avoids the pfexecd not running noise that a root pfexec zlogin into an LX zone produces once per zone.
The service-name charset check is the containment boundary for data coming out of a zone. Its blast radius would be limited to the already-compromised zone, but the check turns a would-be command injection into a logged rejection and skips the zone.
SEE ALSO
manage_zone.sh(8), pool_monitor.sh(8), sys_monitor(8), zone_monitor.sh(8), oc_lock(3), oc_log(3), oc_runlevel(3), oc_validate(3), omnicube_utils(3), omnicube(7).
NOTES
This monitor takes action inside zones without asking. A service that fails repeatedly will be cleared or restarted on every run, which can mask the underlying fault and, for a service that fails on start, produce a restart loop paced by the cron interval. Use config/monitor_exclude_zones while diagnosing such a zone.
A single suspicious service name skips the whole zone, not just that name, so one misbehaving service can stop the others in that zone from being cleared.
The script does not consult the isolation marker /var/run/omnicube/isolate_node.lock, so it keeps entering zones while the node is isolated. See isolate_node.sh(8).