Fencing (computing)

Fencing is the process of isolating a node of a computer cluster or protecting shared resources when a node appears to be malfunctioning.''Sun Cluster environment: Sun Cluster 2.2'' by Enrique Vargas, Joseph Bianco, David Deeths 2001 ISBN page 58

As the number of nodes in a cluster increases, so does the likelihood that one of them may fail at some point. The failed node may have control over shared resources that need to be reclaimed and if the node is acting erratically, the rest of the system needs to be protected. Fencing may thus either disable the node, or disallow shared storage access, thus ensuring data integrity.

Basic concepts

A node fence (or I/O fence) is a virtual "fence" that separates nodes which must not have access to a shared resource from that resource. It may separate an active node from its backup. If the backup crosses the fence and, for example, tries to control the same disk array as the primary, a data hazard may occur. Mechanisms such as STONITH are designed to prevent this condition.

Isolating a node means ensuring that I/O can no longer be done from it. Fencing is typically done automatically, by cluster infrastructure such as shared disk file systems, in order to protect processes from other active nodes modifying the resources during node failures. Mechanisms to support fencing, such as the reserve/release mechanism of SCSI, have existed since at least 1985.

Fencing is required because it is impossible to distinguish between a real failure and a temporary hang. If the malfunctioning node is really down, then it cannot do any damage, so theoretically no action would be required (it could simply be brought back into the cluster with the usual join process). However, because there is a possibility that a malfunctioning node could itself consider the rest of the cluster to be the one that is malfunctioning, a split brain condition could ensue, and cause data corruption. Instead, the system has to assume the worst scenario and always fence in case of problems.

Approaches to fencing

There are two classes of fencing methods, one which disables a node itself, the other disallows access to resources such as shared disks. In some cases, it is assumed that if a node does not respond after a given time-threshold it may be assumed as non-operational, although there are counterexamples, e.g. a long paging rampage.

The STONITH method stands for "Shoot The Other Node In The Head", meaning that the suspected node is disabled or powered off. For instance, ''power fencing'' uses a power controller to turn off an inoperable node. The node may then restart itself and join the cluster later. However, there are approaches in which an operator is informed of the need for a manual restart for the node.

The ''resources fencing'' approach disallows access to resources without powering off the node. This may include:

:* ''Persistent reservation fencing'' uses the SCSI3 persistent reservations to block access to shared storage.
:* Fibre Channel fencing disables the fibre channel port
:* Global network block device (GNBD) fencing which disables access to the GNBD server

When the cluster has only two nodes, the reserve/release method may be used as a two node STONITH whereby upon detecting that node B has 'failed', node A will issue the reserve and obtain all resources (e.g. shared disk) for itself. Node B will be disabled if it tries to do I/O (in case it was temporarily hung). On node B the I/O failure triggers some code to kill the node.

Persistent reservation is essentially a match on a key, so the node which has the right key can do I/O, otherwise its I/O fails. Therefore, it is sufficient to change the key on a failure to ensure the right behavior during failure. However, it may not always be possible to change the key on the failed node.

STONITH is an easier and simpler method to implement on multiple clusters, while the various approaches to resources fencing require specific implementation approaches for each cluster implementation.

STONITH

STONITH (shoot the other node in the head or shoot the offending node in the head), sometimes called STOMITH (shoot the other member/machine in the head), is a technique for fencing in computer clusters. Google's inclusive language developer documentation discourages usage of this term, and recommends it is replaced with the term fence failed nodes.

Fencing is the isolation of a failed node so that it does not cause disruption to a computer cluster. As its name suggests, STONITH fences failed nodes by resetting or powering down the failed node.

Multi-node error-prone contention in a cluster can have catastrophic results, such as if both nodes try writing to a shared storage resource. STONITH provides effective, if rather drastic, protection against these problems.

Single node systems use a comparable mechanism called a watchdog timer. A watchdog timer will reset the node if the node does not tell the watchdog circuit that it is operating well. A STONITH decision can be based on various decisions which can be customer specific plugins.

See also

* Fault tolerance
* Failover

References

External links

Red Hat GFS 6.0: Administrator's Guide - Using the Fencing System



{{Parallel Computing

Fault-tolerant computer systems