NAME
omnicube - OmniCube clustered zone and VM management suite
DESCRIPTION
OmniCube is a set of shell tools, SMF services and cron jobs that turn a small number of illumos hosts into a cluster on which zones and virtual machines are created, snapshotted, replicated and moved between nodes. Every component is a script under /opt/omnicube that can be read and run by hand, and all cluster state lives in SMF properties, ZFS user properties and ZFS snapshots.
A cluster is the set of nodes named in the SMF property config/nodes of svc:/omnicube/sysadm. All of them run the same package and must reach each other over ssh(1) non-interactively (BatchMode=yes and StrictHostKeyChecking=yes are enforced, so host keys have to be pre-populated). There is no quorum, no leader election and no distributed lock manager: coordination is a per-node lock directory under /var/run/omnicube, the site:is_locked ZFS property on the pool being operated on, and the isolation marker described below.
Storage is per zone. Every node holds one large raw disk image per zone in its own pool, at /node/raw/zone/zone-disk0.raw where node is the short host name, and the zone's zpool is built from those images by create_zone_pool.sh(8): mirrored into a single pool (SYNC) or one independent pool per node (ASYNC). A peer's image is reached over NFSv4, mounted at the same path as on the owning node, or as an iSCSI LUN created by create_iscsi_lun.sh(8) and exported with stmfadm(8); init_backup_zfs.sh(8) prepares the shares an administration zone backs up to. A zone's data is therefore always on at least two nodes, and its pool can be imported on whichever node is to run it.
Four kinds of zone run on that storage: native illumos zones (lipkg and relatives), Linux-branded lx zones, and virtual machines under bhyve or kvm. manage_zone.sh(8) is the single entry point for all of them and dispatches on the brand it reads back from zonecfg(8); VM zones are configured from the templates in vm_tpl.cfg(5).
SYNC and ASYNC zones
A zone is SYNC or ASYNC according to the config/type property of its own SMF instance, svc:/service/omnicube/zone:zone. The distinction is about how the zone's data reaches the other nodes, and it decides which command moves the zone:
Independently of that, any dataset can be snapshotted on a schedule and replicated to an off-cluster backup host by autosnap(8), autocleansnap(8) and autosync(8), driven by the retention policies documented in default.conf(5).
Monitoring and isolation
The monitors run from cron(8) on every node and report through info(), warning() and error() in oc_log(3), which log to syslogd(8) under the tag sitescript and send throttled mail to the aliases oc-warn and oc-alert. pool_monitor.sh(8) watches zpool health and iSCSI sessions, reconnecting targets and re-attaching disabled bridge links; sys_monitor(8) watches pool capacity, orphaned site:is_locked properties and clock drift; zone_monitor.sh(8) boots zones that should be running; zones_srv_monitor.sh(8) clears failed services inside the zones; and monitor_nvme.sh(8) reports NVMe health.
When a node loses its peer's storage, repairing pools locally is worse than doing nothing — the node would resilver mirrors whose other half is merely unreachable. So pool_monitor.sh(8) calls isolate_node.sh(8), which creates the marker directory /var/run/omnicube/isolate_node.lock and offlines every non-local pool and LUN. Every monitor aborts while that marker exists, so an isolated node makes no further storage decisions. The marker records a mode: auto isolation is cleared automatically by isolate_node.sh reconcile once the peer answers on the iSCSI port again, while manual isolation, and any marker whose mode cannot be read, is cleared only by an explicit isolate_node.sh release.
Two guards apply to nearly every command. The run-level guard makes a script exit 0 silently at run level 0, 1, 5, 6, S or s, or when /etc/nologin exists, so no job can race a shutdown; OC_IGNORE_RUNLEVEL bypasses it. The lock guard gives each job an atomically created single-instance lock directory under /var/run/omnicube and an EXIT trap that releases it. See oc_runlevel(3) and oc_lock(3).
Directory layout
SMF configuration
Cluster-wide configuration lives in the config property group of svc:/omnicube/sysadm, by default the default instance. Several instances may coexist, each with its own config/site prefix, so one host can serve more than one site or backup destination; OC_SMF_INSTANCE and autosync -i select one, and autocleansnap(8) deliberately iterates over all of them. Read a property with svcprop(1), set one with svccfg(8) followed by svcadm refresh.
Each managed zone additionally has its own instance, svc:/service/omnicube/zone:zone, with three properties: config/type (sync or async, see above), config/adm_status (started or stopped, the administrative intent that zone_monitor.sh(8) enforces and that manage_zone.sh(8) sets with -b and -s) and config/zpools (the list of zpools belonging to the zone, locked and handed over as a unit by a relocation). A zone with no SMF instance is an ASYNC zone managed entirely by hand; zone_monitor.sh(8) then falls back to the zone's autoboot property.
Command index
Zone and VM lifecycle:
manage_zone.sh(8)
— start, stop, relocate, send, resize and reconfigure any zone.
create_zone_pool.sh(8)
— create the zpool and datasets of a new SYNC or ASYNC zone.
create_zone_config.sh(8)
— create a zpool over an existing raw image and its zone datasets.
zone_save.sh(8)
— export every local zone's configuration next to its zonepath.
get_vm_disk_info.sh(8)
— report record size and virtual and actual size of a VM disk image.
activate_zone_vnc.sh(1)
— attach a VNC console to a bhyve or kvm zone.
generate_mac_address_vm(1)
— generate a MAC address inside the site's IEEE MA-M block.
Storage:
create_iscsi_lun.sh(8)
— create a zone's raw LUN, its slog LUN and the STMF views.
init_backup_zfs.sh(8)
— prepare and share the datasets an administration zone backs up to.
sync_pool.sh(8)
— snapshot and send ASYNC zone pools to the peer nodes.
Snapshots and off-cluster replication:
autosnap(8)
— create the scheduled snapshots a policy asks for.
autocleansnap(8)
— enforce retention locally and on the backup host.
autosync(8)
— replicate snapshots to the backup host.
Monitoring and isolation:
pool_monitor.sh(8)
— repair degraded pools, iSCSI sessions and bridge links.
sys_monitor(8)
— pool capacity, orphaned locks and time synchronisation.
zone_monitor.sh(8)
— boot zones that should be running.
zones_srv_monitor.sh(8)
— clear failed services inside the zones.
monitor_nvme.sh(8)
— report NVMe device health.
isolate_node.sh(8)
— enter, reconcile, release and report isolation mode.
logadm(8)
— rotate the suite's logs from cron.
Guest control and console access:
shutdown_windows_server(8),
restart_windows_server(8)
— drive a Windows guest's SAC console.
qemu-monitor-command(8)
— talk to a kvm guest's monitor and guest agent.
restrict-shell(8),
restrict-shell-cmd(8)
— give a user the console of the one zone named after their login.
sshpass(1)
— supply a password to
ssh
non-interactively.
Library and file formats:
omnicube_utils(3)
— the shared library as a whole, and
oc_log(3),
oc_lock(3),
oc_runlevel(3),
oc_validate(3),
oc_ssh(3),
oc_policy(3)
for its individual groups of functions.
vm_tpl.cfg(5)
— the kvm and bhyve zone templates.
default.conf(5)
— the snapshot retention policies.
ZFS properties
State that must travel with the data is kept in ZFS user properties prefixed with config/site. With config/site set to efit they are, on datasets, efit:snapshotting and efit:remotebackup (opt in to snapshotting and to replication), efit:snappolicy (the policy name) and efit:is_locked (pool busy, skip it), and on snapshots efit:insync and efit:bckcandel (replication progress, which is what stops autocleansnap(8) destroying a snapshot another job still needs). See autosnap(8), autosync(8), autocleansnap(8) and default.conf(5).
Viewing these manual pages
The pages are installed in /opt/omnicube/share/man, which is not on the default man(1) search path. Either name the tree explicitly:
man -M /opt/omnicube/share/man manage_zone.sh
or add it to MANPATH once, for instance in /etc/profile:
MANPATH=/opt/omnicube/share/man:$(manpath 2>/dev/null) export MANPATH man 5 default.conf
Page names are the command file names, including the .sh suffix where the script has one, so the page for /opt/omnicube/sbin/manage_zone.sh is manage_zone.sh(8); there is no manage_zone page.
ENVIRONMENT
FILES
SECURITY
Nothing in the suite is installed setuid. Privileged operations go through pfexec(1) and the two RBAC profiles in /etc/security/prof_attr.d/omnicube, assigned with usermod -P. Note that on illumos that option replaces the profiles= list in user_attr(5); there is no + modifier (that is a Solaris 11 extension), so to add a profile to an account that already has one, read the current list back from /etc/user_attr and pass all of them in a single comma-separated argument. Profiles handed out system-wide through PROFS_GRANTED in policy.conf(5) do not live in user_attr and are unaffected. Check the result with profiles -l.
The two profiles are:
pfexec matches exec_attr(5) entries on the resolved path of the binary, which is why the scripts export an explicit PATH and why relocating a tool means editing both the script and /etc/security/exec_attr.d/omnicube. In the scripts the prefix is ${PFEXEC}, which utils.sh sets to pfexec for an unprivileged caller and to the empty string when the effective uid is already 0; it must always be used unquoted.
Zone, node, user, host and dataset names are validated with validate_name and validate_dataset before they reach a command line; see oc_validate(3). Policy files and zone templates, by contrast, are sourced or applied as code, so their ownership and mode are the only boundary: both must be root-owned and not group or world writable.
SEE ALSO
The Command index above lists every page in the suite. The starting points are omnicube_utils(3), default.conf(5), vm_tpl.cfg(5), autosnap(8), autosync(8), autocleansnap(8), isolate_node.sh(8), manage_zone.sh(8), pool_monitor.sh(8), sys_monitor(8).
NOTES
The snapshot, replication and save jobs are driven from cron(8), not from SMF. The reference crontab shipped in /opt/omnicube/share/README is:
6 * * * * /opt/omnicube/backup/autosnap >/dev/null 26,46 * * * * /opt/omnicube/backup/autosync >/dev/null 20 3 * * * /opt/omnicube/sbin/zone_save.sh 36 * * * * /opt/omnicube/backup/autocleansnap >/dev/null
Registering a zone with the suite means creating its SMF instance; the README gives the sequence:
ZONE=myzone
svccfg -s svc:/service/omnicube/zone add ${ZONE}
SMF=svc:/service/omnicube/zone:${ZONE}
svccfg -s ${SMF} addpg config application
svccfg -s ${SMF} setprop config/type = astring: sync
svccfg -s ${SMF} setprop config/adm_status = astring: started
svcadm refresh ${SMF}
svcadm enable ${SMF}
Several operations are cluster-wide and have no automatic rollback: a relocation stops the zone, exports its pool and rebuilds it on the target, and a target that becomes unreachable half way through leaves the zone unconfigured, to be recovered by hand. This is why the entry points pre-flight ssh to every peer before changing anything, and why config/nodes must not name a decommissioned node.
Changing a zone's configuration on one node only is a latent failure: the peer copies still carry the old configuration and a relocation will silently revert the change. Follow every configuration change with manage_zone.sh -z zone -r.
A stuck site:is_locked property wedges every snapshot and replication job for the dataset it is set on, because those jobs skip a locked pool by design. The jobs release it from an EXIT trap, but a SIGKILL or a pool exported mid-relocation can leave it behind; sys_monitor(8) reports it after three consecutive observations, and it is cleared with zfs inherit -r site:is_locked pool.
The cluster nodes are illumos systems running the same package. Only the backup host of a replication policy may be Linux, which is why that one path avoids pfexec, ggrep and bash built-ins on the remote side.