NAME
sync_pool.sh - replicate asynchronous zone pools to the peer cluster nodes
SYNOPSIS
/opt/omnicube/sbin/sync_pool.sh
DESCRIPTION
sync_pool.sh keeps the zone datasets of asynchronous zones mirrored on the other nodes of the cluster, so a zone can be started on a peer without a full transfer. For each eligible pool it takes a fresh SYNC snapshot, sends it (incrementally when a previous snapshot exists) to every other node listed in the SMF property config/nodes, then rotates the pool's snapshots down to MAX_SNAP=16. It is meant to run from cron(8) in the global zone and takes no arguments. This is node-to-node replication inside the cluster, distinct from the off-site replication of autosync(8): the peers run the same package, so remote privileged commands are hard-coded pfexec, and no snapshot policy is involved.
Every pool from zpool list -H -o name is examined, and a pool is processed only when its name is neither rpool nor contains the short local hostname (hostname truncated at the first -, which excludes the node's own pools), a zone of the same name is installed here (zoneadm list -ni), and the SMF property config/type of svc:/service/omnicube/zone:pool is exactly async. The dataset replicated is pool/zones/pool. The run-level guard is re-evaluated before each pool, so a shutdown stops the job before the next send.
Send procedure
The run identifier is ID=$(date '+%Y%m%d%H%M%S'), shared by all pools of one invocation, and each pool's snapshot is ${DS}@SYNC${ID}. The incremental base is the newest existing snapshot of the dataset, chosen by the ZFS creation property rather than by name. For each node in config/nodes other than the local host, the name is checked with validate_name() and ssh_reachable() proves the whole ssh path before anything is sent; a ping style check is deliberately not used, since it cannot detect a down sshd, an unauthorised key or a host key missing from known_hosts under StrictHostKeyChecking=yes. The stream is piped through pv(1) for progress accounting:
pfexec zfs send [-i ${OLD_ID}] ${DS}@SYNC${ID} \\
| pv -tba | ssh ${host} 'pfexec zfs receive -u '${DS}
Both ends of the pipeline are checked through PIPESTATUS. A failure raises a warning, increments the error counter and disables snapshot rotation for this pool. If no node received the stream the local SYNC snapshot is destroyed again ("Rollback"); if at least one did, it is kept, because that node needs it as the incremental base of the next run.
Snapshot rotation and MAX_SNAP
Rotation runs only when every peer send for the pool succeeded. Snapshots are walked newest first, ignoring names matching @[HO]C-, and the first MAX_SNAP=16 are kept unconditionally. Each older snapshot is then either preserved, destroyed or skipped. It is preserved when its name, with trailing -digits groups stripped, matches ${DS}@hourly, daily, weekly, yearly or frequent: those belong to the autosnap(8) and autocleansnap(8) lifecycle and their retention is decided by the snapshot policy, not by MAX_SNAP. Otherwise it is destroyed locally and, after an ssh_reachable() check, on every peer node, provided ${PROPPREFIX}:remotebackup is not yes or ${PROPPREFIX}:insync is yes. Failing that it is kept and reported as "skipped (ongoing sync)", because autosync(8) still needs it.
Locking and error counting
The mutex is the directory /var/run/omnicube/sync_pool.lock created by acquire_lock_or_exit(); a concurrent invocation exits 0 silently. An EXIT HUP INT TERM trap removes it on every exit path, including the exit 1 sites inside the loop and a kill during a zfs send. This lock is unrelated to the similarly named lock of autosnap(8), which holds ${LOCK_BASE}/${logtag}.sync_pool.lock for historical reasons.
ERROR_COUNT counts unreachable peers and failed sends across the whole run and is tested at the top of each pool iteration: once it exceeds 3 the command logs "Too many errors, aborting..." and exits 1, leaving the remaining pools untouched.
OPTIONS
This command takes no options and no operands.
USAGE
The sample crontab in /opt/omnicube/share/README has no entry for sync_pool.sh; the jobs it does define are the ones whose snapshots this command must not rotate away:
6 * * * * /opt/omnicube/backup/autosnap >/dev/null 36 * * * * /opt/omnicube/backup/autocleansnap >/dev/null
Check which of this node's pools are eligible before scheduling it:
zoneadm list -ni svcprop -p config/type svc:/service/omnicube/zone:zethosmariadb01 svcprop -p config/nodes svc:/omnicube/sysadm:default
Run one pass by hand, then inspect the snapshots subject to rotation:
pfexec /opt/omnicube/sbin/sync_pool.sh
zfs list -H -d1 -t snapshot -o name,creation \\
zethosmariadb01/zones/zethosmariadb01
EXIT STATUS
FILES
ENVIRONMENT
SECURITY
Local ZFS operations use pfexec, and the receive and destroy commands on the peers are run as pfexec zfs over ssh, so the service account needs the OmniCube RBAC profile (/etc/security/prof_attr.d/omnicube, /etc/security/exec_attr.d/omnicube) on every node and a key authorised on each peer. Node names from config/nodes are validated before they reach an ssh command line.
SEE ALSO
autocleansnap(8), autosnap(8), autosync(8), isolate_node.sh(8), manage_zone.sh(8), pool_monitor.sh(8), default.conf(5), oc_lock(3), oc_runlevel(3), oc_ssh(3), omnicube_utils(3), omnicube(7).
NOTES
sync_pool.sh neither reads nor sets ${PROPPREFIX}:is_locked, so it can run concurrently with autosnap(8) and autosync(8) on the same pool; its lock only serialises it against itself.
MAX_SNAP is a constant in the script, not a tunable: retaining more SYNC snapshots means editing /opt/omnicube/sbin/sync_pool.sh.
Rotation propagates destroys to the peers, so a peer unreachable during the cleanup phase keeps snapshots the local node has dropped. Nothing is retried; the divergence has to be cleaned up by hand.
Only zones whose config/type is async are replicated. A sync type zone lives on shared storage and is handled by manage_zone.sh(8) instead.