NAME
oc_ssh, ssh_reachable, require_ssh, ssh_probe_reset - pre-flight check that a remote OmniCube host is usable over ssh
SYNOPSIS
amp;. /opt/omnicube/lib/common/utils.sh
ssh_reachable [user@]host [context]
require_ssh [user@]host [context]
ssh_probe_reset [ [user@]host ]
DESCRIPTION
Every workflow in the suite that talks to another host must confirm that the connection works before it starts changing local state. Without a pre-flight probe an unreachable or misconfigured peer is discovered halfway through the sequence, after the local side has already been torn down. The relocate path in manage_zone.sh(8) is the worst case: the zone is halted, its configuration deleted and its zpool exported, and only then does the first remote command fail. Recovery from that point is manual.
The probe deliberately tests the whole ssh path rather than reachability. A ping(8) only proves that the IP answers; it says nothing about sshd(8) being up, the key being authorized, or the host key being known — and with StrictHostKeyChecking=yes an unknown host key is a hard failure that ping cannot see. sync_pool.sh(8) used to gate on ping alone. Running true as the remote command means that a zero exit proves all of it at once: TCP reachable, sshd answering, authentication accepted without a prompt, host key already trusted, and a shell able to run a command.
Command and option variables
The library defines these at source time; see omnicube_utils(3).
BatchMode=yes is what makes these safe under cron and SMF: with no tty, an ssh that decides to ask for a password or a passphrase would hang forever instead of failing. StrictHostKeyChecking=yes is a refusal to auto-accept an unknown key, because zfs streams and zone configurations transit these connections between cluster nodes. It carries a deployment requirement: /etc/ssh/ssh_known_hosts (or the service account's own known_hosts) must be pre-populated with the host key of every cluster node and every backup host, or every remote operation fails closed.
The probe options differ from ${_SSH_OPTS} on two points, both intentional. ConnectionAttempts=1 because a gate must fail fast: ${_SSH_OPTS} retries five times with a five-second connect timeout, i.e. up to roughly 25 seconds per host, which is right for a real transfer and far too slow for a check that may run once per node inside a loop. -n redirects the probe's standard input from /dev/null, because several call sites sit inside a while read loop and an ssh that inherited the loop's stdin would consume the very input the loop is iterating over.
ssh_reachable
Probes [user@]host and returns 0 if a non-interactive ssh session to it works. context is an optional label naming the calling workflow and appears in the log line; it defaults to ssh.
Order of operations: reject an empty target; reject a target that does not match
^([a-zA-Z0-9][a-zA-Z0-9._-]*@)?[a-zA-Z0-9][a-zA-Z0-9._-]*$
which accepts host and user@host and rejects whitespace, anything that looks like an option, and every shell metacharacter before it can reach an ssh command line; consult the cache; otherwise run
ssh ${_SSH_PROBE_OPTS} "target" true
capturing both output streams.
On success the verdict 0 is cached and 0 returned. On failure the verdict 1 is cached, ssh's own diagnostic is collapsed to a single line (newlines turned into spaces, runs of spaces squeezed, trailing space trimmed) and reported through error() as
<context>: ssh to <target> is not usable, skipping remote work (ssh said: <diagnostic>)
Capturing stderr instead of discarding it is the point: the reason — unknown host key, permission denied, connection timed out — is the single most useful piece of information when this trips, and it is not recoverable afterwards.
require_ssh
ssh_reachable() plus a hint about the usual root causes, for the operator-facing entry points where the whole command is about to abort — the relocate and send paths of manage_zone.sh(8) and create_zone_pool.sh(8). On failure it emits a second error() line telling the operator to check that the target is up, that sshd is running, that this account's key is authorized there, and that the host key is in known_hosts.
Monitors and cron jobs call ssh_reachable() directly instead: they skip the unreachable peer and carry on rather than fail the whole run.
ssh_probe_reset
Drops the cached verdict for one target, or the entire cache when called with no argument. Intended for monitors that loop for minutes and want a fresh verdict rather than a decision made at start-up.
Probe caching
Verdicts are memoized for the lifetime of the process in the associative array _SSH_PROBE_CACHE, keyed by the exact "[user@]host" string given. Workflows probe inside per-node and per-dataset loops, so without the cache the same host would be re-probed on every iteration. Three consequences:
The key is the literal argument, so backup1 and jdoe@backup1 are two independent cache entries even though they name one host.
RETURN VALUES
Neither function calls exit; callers decide whether to continue, return or exit.
FILES
ENVIRONMENT
EXAMPLES
Example 1: a monitor skipping an unreachable node
for node in ${nodes[@]}; do
[[ ${node} == ${curr_host} ]] && continue
ssh_reachable "${node}" "pool_monitor lun ${lun}" || continue
${_SSH_CMD} "${node}" "pfexec /usr/sbin/stmfadm list-lu -v ${lun}"
done
Example 2: an operator command that must abort
require_ssh "${host}" "relocate ${zone}" || return 1
Example 3: a long-lived loop forcing a re-probe
while true; do
ssh_probe_reset "${BKPUSER}@${BKPHOST}"
ssh_reachable "${BKPUSER}@${BKPHOST}" "autosync ${DS}" && break
sleep 300
done
SECURITY
The target is pattern-checked before it reaches an ssh command line, so a node name from config/nodes or a BKPHOST from a policy file cannot smuggle an option or a shell metacharacter into the probe. BatchMode=yes guarantees no credential prompt in an unattended context, and StrictHostKeyChecking=yes makes an unknown or changed host key a hard failure rather than a silent trust-on-first-use.
SEE ALSO
omnicube_utils(3), oc_log(3), oc_lock(3), oc_runlevel(3), oc_validate(3), oc_policy(3), manage_zone.sh(8), sync_pool.sh(8), autosync(8), create_zone_pool.sh(8), omnicube(7).
NOTES
A successful probe proves the connection worked at probe time. It says nothing about privileges on the peer: the remote commands the suite then runs are prefixed with pfexec at the call site for cluster nodes, or with the policy's BKPPRIV for a backup host — see oc_policy(3).
The probe runs true on the peer, so an account whose shell is restricted to a command whitelist must permit it. See restrict-shell(8).
Neither function is used for the local host. Call sites skip ${curr_host} explicitly and do the work locally instead.