Using Red Hat Enterprise Linux HA with Caché
Caché can be configured as a resource controlled by Red Hat Enterprise Linux Server with High Availability Add-On (HA). This appendix highlights the key portions of the configuration of RHEL (Red Hat Enterprise Linux) HA, including how to incorporate the custom Caché resource. Refer to your Red Hat documentation and consult with Red Hat on all HA configurations.
When using Caché in a high availability environment controlled by RHEL HA:
-
Install the hardware and operating system according to your vendor recommendations for high availability, scalability and performance. Be sure to include the High Availability and Resilient Storage Add-Ons. For more information, see Hardware Configuration.
-
Configure RHEL HA to control a shared disk resource with a virtual IP address and test that common failures are detected and the cluster continues operating. See Red Hat Linux and Red Hat Linux HA Software for more information.
-
Install the Caché control script and Caché resource according to the information in Installing and Using the <cache /> Resource.
-
Install Caché according to the guidelines in this appendix and test connectivity to the database from your application; for more information, see Installing Caché in the Cluster.
-
Consider the application implications and test disk failures, network failures, and system crashes. Test and understand your application’s response to such failures; for more information, see Application and Other Considerations and Testing and Maintenance.
Hardware Configuration
Configure the hardware according to best practices for the application. In addition to adhering to the recommendations of Red Hat and your hardware vendor, consider the following:
Create LUNs/partitions, as required, for performance, scalability, availability and reliability. This includes using appropriate RAID levels, battery-backed and mirrored disk controller cache, multiple paths to the disk from each node of the cluster, and a partition on fast shared storage for your cluster quorum disk.
Use of a quorum disk and associated heuristics is recommended to prevent a split cluster.
Consider file system choice carefully. Of the journaling file systems supported by RHEL with the HA and Resilient Storage Add-Ons, ext4 file systems have the fastest recovery after failure.
With the Resilient Storage Add-On, RHEL systems provide a shared cluster logical volume manager (CLVM). For Caché-based applications, the CLVM daemon is what prevents Caché logical volumes from being concurrently mounted on multiple systems.
Use bonded multi-NIC connections through redundant switches/routers where possible to reduce single-points-of-failure. Separate the private network and the public network, and be sure the private network allows multicast for the heartbeat. Include all fully qualified public and private domain names in the hosts file on each node.
Fencing is the ability to cut off power or disk access to failed or unresponsive nodes in the cluster. Robust fencing ensures clean, fast restarts of an application after a failure. The safest and fastest fencing is external power fencing. For dual power supplies, be sure to configure fencing properly so all power is cut during a fencing event. Using an integrated power management device as a fencing option is also common.
In virtual machine environments, consider using a fencing device compatible with the hypervisor in use.
Recommendation: configure more than one type of fencing for availability.
Red Hat Linux and Red Hat Linux HA Software
Prior to installing the Caché-based application, follow the recommendations below when configuring Linux and the HA Add-On. These recommendations assume a two-node cluster with quorum disk and direct editing of the cluster.conf file.
Red Hat Linux
When configuring Linux on the nodes in the cluster, use the following guidelines:
-
All nodes in the cluster must have identical userids/groupids (that is, the name and ID number must be identical on all nodes); this is required for Caché.
-
Sufficient kernel resources (shmmax, and so on) must be configured on the nodes.
-
Only the minimum necessary Linux packages should be installed. Anything started past run level 3 (such as graphical consoles, and so on) is unnecessary for most Caché-based application and database servers. Each unnecessary package lengthens the time to boot and slows recovery after failures.
Red Hat Enterprise Linux HA
Install and test a standard shared disk cluster service using the following guidelines:
-
Configure the <clusternodes> such that each has one vote. Be sure to use fully qualified domain names as the name of each node.
-
Configure the <fence_daemon> with no post_fail_delay. This speeds the time to recovery.
-
For a two-node cluster with quorum disk, the <cman> configuration should include the following:
expected_votes="3" two_node="0"
-
Configure <totem> to have a token timeout equal to or greater than the <cman> quorum_dev_poll value.
-
Modify <cman> timeouts to be sure to account for speed of quorum disk.
-
Configure at least two fencing devices, with a direct power fencing option if possible for the fastest fencing. Be sure to use the passwd_script parameter rather than a plain text password in your cluster.conf file.
-
Simplify the cluster.conf to minimize errors due to complexity. Simple clusters consistently work best and are easiest to maintain.
-
The <ip> stanza is used to configure the virtual IP address resource. It is recommended that monitor_link="1". Be sure the application, all interfaces, any ECP clients, and so on use the virtual IP address as configured here to connect to the Caché resource.
-
Test loss of private and public network connectivity to be sure no split-brain occurs.
-
Add all the <lvm> and <fs> stanzas to the <service> stanza in the cluster.conf file required for Caché and the application. The following are recommended for the <fs/> stanza(s):
force_fsck = "1" force_unmount = "1" self_fence = "1"
Installing and Using the <cache /> Resource
A sample control script and a Caché resource are provided as part of a development Caché installation.
Note that a development installation is not required in the cluster; copy the sample control script (cache.sh) and the appropriate patch file (cluster.rng.patch or cluster.rng.patchRHEL6) from a development installation to the cluster nodes as needed. Alternatively, contact the InterSystems Worldwide Response Center (WRC)Opens in a new tab for information about receiving the cache.sh and appropriate cluster.rng.patch* files.
Installing the cache.sh Script
To install the Caché control script:
-
Locate the files in dev/cache/HAcluster/RedHat/ under the Caché installation directory.
-
Copy or place the cache.sh file in /usr/share/cluster and make sure the permissions, owner and group are identical to the other files in that directory.
Patching the cluster.rng Script
Patches are provided for the cluster.rng file as supplied with RHEL 5.5, 5.6, 5.7, or 5.8 or RHEL 6.2, 6.3, or 6.4. The patch simply adds the definition of the <cache /> resource to the list of cluster resources.
When using RHEL 6, patching is required since a cluster.conf file that includes the <cache /> resource cannot validate without the patched cluster.rng in place. If your system is running a version newer than RHEL 6.4, contact the InterSystems Worldwide Response Center (WRC)Opens in a new tab for information about options for patching your cluster.rng.
The cluster.rng file is typically found in either the /usr/share/cluster or the /var/lib/cluster directory. Be sure to patch the actual source file (and not simply a linked file).
-
Place the appropriate patch file (either cluster.rng.patch or cluster.rng.patchRHEL6) on the system in /tmp/.
-
Change to the appropriate directory (/var/lib/cluster or /usr/share/cluster) and use the following command to patch the existing cluster.rng file:
patch -b cluster.rng /tmp/cluster.rng.patchRHEL6
Note:cluster.rng.patch patches unaltered RHEL 5.5, 5.6, 5.7, or 5.8 cluster.rng files. cluster.rng.patchRHEL6 patches unaltered RHEL 6.2, 6.3, or 6.4 cluster.rng files.
Using the <cache /> Resource
The following is a sample resource set up to control an instance named cacheprod:
<cache __enforce_timeouts="1" cleanstop="1" name="cacheprod">
<action name="status" interval="30s"/>
<action name="start" timeout="10m"/>
<action name="stop" timeout="5m"/>
</cache>
In this sample, a service starts the cacheprod instance of Caché and waits up to 10 minutes for the resource start to return. The service is marked as failed if the start does not complete in 10 minutes. After successfully starting all the resources, status checks occur every 30 seconds.
If a clean service stop is not configured (cleanstop=”0”), the cacheprod instance is stopped with a ccontrol force cacheprod quietly. This results in a fast stop and protects against clean stop hangs due to unavailable disk or other Linux resources, ensuring a fast transition to the other node during faults that do not crash the system but result in a node transition. If cleanstop=”0” is used, system managers should manually stop Caché (ccontrol stop) before any manual service stop or service relocations. This ensures a clean stop during routine maintenance work.
In this example a clean service stop (cleanstop="1") is configured, the cacheprod instance is stopped with a ccontrol stop cacheprod quietly. If that fails or generates an error, a ccontrol force cacheprod quietly is attempted. Note though, that a hung ccontrol stop hangs forever.
In this example, however, with __enforce_timeouts=”1”, the stop must complete in five minutes. A stop that does not return within that timeout period causes the whole service to be marked as failed.
Note the following when using the <cache /> resource:
-
The <cache /> resource takes the following parameters:
-
name: The name of the Caché instance being controlled by the service.
-
__enforce_timeouts: A generic rgmanager setting; as a general rule, it is recommended that you set this = “1” to ensure that a start or stop of the service does not hang indefinitely.
-
cleanstop: If false (= “0”) a cluster service stop or relocate immediately forces Caché down rather than attempting a controlled stop of Caché; while the stop is faster, a force of Caché may lengthen startup time as transactions roll back, and so on
-
-
Action timeouts are honored only if __enforce_timeouts="1". Generic rgmanger start and stop timeouts should be configured based on application needs, as follows:
-
<action name="start" timeout="10m"/>: the start timeout should be long enough to allow the replay of the CACHE.WIJ and journal files, but not so long that severe cluster problem notifications are delayed while waiting for a start that will never complete.
-
<action name="stop" timeout="5m"/>: the stop timeout comes into play only if the service is stopped (as opposed to a fencing event that cuts power). This can happen at boot time or as part of cluster reorganization or during a manual service stop.
Note:If the start or stop timeout is reached the service is marked as FAILED and manual intervention is required to reenable it.
-
<action name="status" interval="30s"/>: As a general rule, set the status check interval to 30 seconds; more frequent status checks use CPU and may affect cluster responsiveness.
-
-
The status check uses ccontrol qlist [instance] to see if the instance is down or running. If it is in either of those states, the status check passes and no action is taken; this lets you stop Caché manually, if necessary, without stopping the service and without accidentally triggering a failover event.
-
The <cache /> resource is always started after the <lvm>, <fs> and <ip> resources, and stopped before those resources; this is without regard to placement within the <service> stanza of the cluster.conf file. For more complex cluster configurations, parent and child relationships may need to be configured, but a typical two-node cluster does not require the added complexity.
Installing Caché in the Cluster
After a service with disk and IP resources has been configured and tested, you must install Caché in the cluster. The procedure is outlined below.
It is assumed that the <cache/> resource definition is part of the cluster.rng file and that the cache.sh control script is in place in /usr/share/cluster. See Installing and Using the <cache /> Resource for more information.
There are different procedures depending on whether you are installing only one instance of Caché or multiple instances of Caché. Installing a single instance of Caché is common in clusters dedicated to the production instance. In development and test clusters, it is common to have multiple instances of Caché controlled by the cluster software, providing maximum configuration flexibility. If it is possible that you will install multiple instances of Caché in the future, follow the procedure for multiple instances.
For information about upgrading Caché in an existing failover cluster, see Upgrading a Cluster in the “Upgrading Caché” chapter of the Caché Installation Guide.
Installing a Single Instance of Caché
To install a single instance of Caché, use the following procedure.
If any Caché instance that is part of a failover cluster is to be added to a Caché mirror, you must use the procedure described in Installing Multiple Instances of Caché, rather than the procedure in this section.
-
Bring the service that contains all the disk and IP resources online. This mounts the entire set of disks required for Caché and the application to run (for installation, data files, journal files, and any other disk required for the application in use) on one node.
-
Check the file and directory ownerships and permissions on all mount points and subdirectories.
-
Prepare to install Caché by reviewing the “Installing Caché on UNIX® and Linux” chapter of the Caché Installation Guide.
-
-
Create a link from /usr/local/etc/cachesys to the shared disk. This forces the Caché registry and all supporting files to be stored on the shared disk resource you have configured as part of the service group.
A good choice is to use a ./usr/local/etc/cachesys/ subdirectory under your installation directory.
For example, assuming Caché is to be installed in /cacheprod/cachesys/, specify the following:
mkdir –p /cacheprod/cachesys/usr/local/etc/cachesys mkdir –p /usr/local/etc/ ln –s /cacheprod/cachesys/usr/local/etc/cachesys /usr/local/etc/cachesys
-
Run Caché cinstall on the node with the mounted disks. Be sure the users and groups (either default or custom) are available on all nodes in the cluster, and that they all have the same UIDs and GIDs.
-
Stop Caché and relocate the service to the other node. Note that there is no Caché resource configured, so the service does not control Caché yet.
-
On the second node in the cluster, create the link in /usr/local/etc/ and the links in /usr/bin for ccontrol and csession:
mkdir –p /usr/local/etc/ ln –s /cacheprod/cachesys/usr/local/etc/cachesys/ /usr/local/etc/cachesys/ ln –s /usr/local/etc/cachesys/ccontrol /usr/bin/ccontrol ln –s /usr/local/etc/cachesys/csession /usr/bin/csession
-
Add the Caché resource to the cluster.conf file and configure according to the needs of your application. At minimum, add the following line to the <service> stanza:
<cache name=”instance” __enforce_timeouts=”[1/0]” cleanstop=”[1/0]” />
where
-
instance is the name of the instance specified during Caché cinstall (step 3)
-
__enforce_timeouts can be specified as 1 or 0, with 1 recommended
-
cleanstop=”1” indicates that cache.sh stop should issue a ccontrol stop while cleanstop=”0” indicates that cache.sh stop should issue a ccontrol force
In addition, you can optionally specify any or all of the following stanzas within the <cache> stanza (see Installing and Using the <cache /> Resource).
<action start timeout=10m/> <action stop timeout=5m/> <action status interval=30s/>
-
-
If editing the cluster.conf file directly:
-
Update the version of the cluster.conf file
-
Validate cluster.conf (ccs_config_validate)
-
Distribute cluster.conf to all nodes (cman_tool version –r)
-
-
Restart the rgmanager daemon (service rgmanager restart) on all nodes and test.
Installing Multiple Instances of Caché
To install multiple instances of Caché, use the following procedure.
If any Caché instance that is part of a failover cluster is to be added to a Caché mirror, the ISCAgent (which is installed with Caché) must be properly configured; see Configuring the ISCAgent in the “Mirroring” chapter of this guide for more information.
-
Bring the service that contains all the disk and IP resources online. This mounts the entire set of disks required for Caché and the application to run (for installation, data files, journal files, and any other disk required for the application in use) on one node.
-
Check the file and directory ownerships and permissions on all mount points and subdirectories.
-
Prepare to install Caché by reviewing the “Installing Caché on UNIX® and Linux” chapter of the Caché Installation Guide.
-
-
Run Caché cinstall on the node with the mounted disks. Be sure the users and groups (either default or custom) are available on all nodes in the cluster, and that they all have the same UIDs and GIDs.
-
The /usr/local/etc/cachesys directory and all its files must be available to all nodes at all times. To enable this, after Caché is installed on the first node, copy /usr/local/etc/cachesys to each node in the cluster. The following method preserves symbolic links during the copy process:
cd /usr/local rsync -av -e ssh etc root@node2:/usr/local/
If rsync is not available, tar or other options may exist.
-
Verify that ownerships and permissions on the cachesys directory and its files are identical on all nodes
Note:In the future, keep the Caché registries on all nodes in sync using ccontrol create or ccontrol update or by copying the directory again; for example:
ccontrol create CSHAD directory=/myshadow/ versionid=2013.1.475
-
Stop Caché and relocate the service to the other node. Note that there is no Caché resource configured, so the service does not control Caché yet.
-
On the second node in the cluster, create the links in /usr/bin for ccontrol and csession:
ln –s /usr/local/etc/cachesys/ccontrol /usr/bin/ccontrol ln –s /usr/local/etc/cachesys/csession /usr/bin/csession
-
Stop Caché and disable the service.
-
Add the Caché resource to the cluster.conf file and configure according to the needs of your application. At minimum, add the following line to the <service> stanza:
<cache name=”instance” __enforce_timeouts=”[1/0]” cleanstop=”[1/0]” />
where
-
instance is the name of the instance specified during Caché cinstall (step 3)
-
__enforce_timeouts can be specified as 1 or 0, with 1 recommended
-
cleanstop=”1” indicates that cache.sh stop should issue a ccontrol stop while cleanstop=”0” indicates that cache.sh stop should issue a ccontrol force
In addition, you can optionally specify any or all of the following stanzas within the <cache> stanza (see Installing and Using the <cache /> Resource).
<action start timeout=10m/> <action stop timeout=5m/> <action status interval=30s/>
-
-
If editing the cluster.conf file directly:
-
Update the version of the cluster.conf file
-
Validate cluster.conf (ccs_config_validate)
-
Distribute cluster.conf to all nodes (cman_tool version –r).
-
-
Restart the rgmanager daemon (service rgmanager restart) on all nodes and test.
When you are ready to install a second instance of Caché within the same cluster, follow these additional steps:
-
Configure the cluster.conf file to add the <ip>, <fs>, and <lvm> resources associated with the second instance of Caché.
-
Bring the service online so the disks are mounted on one of the nodes.
-
Be sure the users and groups to be associated with this new instance are created and synchronized between nodes.
-
On the node with the mounted disk, run ccinstall following the procedures outlined in the "Installing Caché on UNIX® and Linux" chapter of the Caché Installation Guide.
-
Stop Caché.
-
Synchronize the Caché registry using the following steps:
-
On the install node run
ccontrol list
-
Record the instance name, version ID and installation directory of the instance you just installed.
-
On the other node, run the following command to create the registry entry, using the information you recorded from the recently installed instance:
ccontrol create inst_name versionid=vers_ID directory=inst_dir
-
-
Edit cluster.conf to add the <cache /> resource to the service controlling this instance.
-
Start the service and verify that Caché starts.
Application and Other Considerations
Consider the following for your applications:
-
Ensure that all disks required for the proper operation of your application are part of the cluster.conf file.
-
Ensure that all network ports required for interfaces, user connectivity and monitoring are open on all nodes in the cluster.
-
Connect all interfaces, web servers, ECP clients and users to the database using the virtual IP address over the public network as configured in the cluster.conf file.
-
Ensure that application daemons, Ensemble productions, and so on are set to autostart so the application is fully available to users after unscheduled failovers.
-
Consider carefully any code that is part of %ZSTART or otherwise occurs as part of the Caché startup. To minimize recovery time do not place heavy cleanup or query code in the startup.
-
Long open transactions that require rollback can slow shutdown or startup and affect recovery time. For example, during startup the cluster waits 10 minutes as configured in the above examples. If a system crash occurs when a transaction has been left open for hours or even days, during startup Caché must scan many journal files to roll back that transaction. If this takes longer than 10 minutes, the service is taken down and marked as failed.
-
Other applications or web servers, and so on can also be configured in the cluster, but these examples assume only Caché is installed under cluster control. Contact the InterSystems Worldwide Response Center (WRC)Opens in a new tab to consult about customizing your cluster.
Testing and Maintenance
Upon first setting up the cluster, be sure to test that failover works as planned. This also applies any time changes are made to the operating system, its installed packages, the disk, the network, Caché, or your application.
In addition to the topics described in this section, you should contact the InterSystems Worldwide Response Center (WRC)Opens in a new tab for assistance when planning and configuring RHEL HA service to control Caché. The WRC can check for any updates to the cache.sh and cluster.rng scripts, as well as discussing failover and HA strategies.
Failure Testing
Typical full scale testing must go beyond a controlled service relocation. While service relocation testing is necessary to validate that the cluster.conf file and the service resource scripts are all functioning properly, you should also test responses to simulated failures. Be sure to test failures such as:
-
Loss of public and private network connectivity to the active node
-
Loss of disk connectivity
-
Hard crash of active node
Testing with an application load builds confidence that the application will recover in the event of actual failure.
Try to test with a heavy disk write load. During heavy disk writes the database is at its most vulnerable.
Caché handles all recovery automatically using its CACHE.WIJ and journal files but testing a crash during an active disk write ensures that all file system and disk devices are properly failing over.
Software and Firmware Updates
Keep software patches and firmware revisions up to date. Avoid known problems by adhering to a patch and update schedule.
Monitor Logs
In RHEL 5, the /var/log/messages file is the default location for cluster-related messages. Keep an eye on this log for issues or cluster problems. The <cache /> resource also logs messages during start and stop events in the system log at log level 5.
In RHEL 6, the messages are also in various logs in the /var/log/clusters/ directory. Use the <log> stanza in the cluster.conf file to increase logging level during testing and debugging.
Use the Caché console log, the Caché Monitor and the Caché System Monitor to be alerted to problems with the database that may not be caught by the cluster software. (See the chapters “Monitoring Caché Using the Management Portal”, “Using the Caché Monitor” and “Using the Caché System Monitor” in the Caché Monitoring Guide for information about these tools.)