Using Veritas Cluster Server for Linux with Caché

Linux

When configuring Linux on the nodes in the cluster, use the following guidelines:

1. All nodes in the cluster must have identical userids/groupids (that is, the name and ID number must be identical on all nodes); this is required for Caché.

   These two users and two groups need to be added and synchronized between members:

   1. Users

      1. Owner(s) of the instance(s) of Caché

      2. Effective user(s) assigned to each instance’s Caché jobs

   2. Groups

      1. Effective group(s) to which each instance’s Caché processes belong.

      2. Group(s) allowed to start and stop the instance(s).

2. All volume groups required for Caché and the application are available to all nodes.

3. Include all fully qualified public and private domain names in the hosts file on each node.

Veritas Cluster Server

This document assumes Veritas Cluster Server (VCS) version 5.1 or newer. Other versions may work as well, but likely have different configuration options. Consult with Symmantec/Veritas and the InterSystems Worldwide Response Center (WRC)Opens in a new tab for guidance.

In general you will follow these steps:

1. Install and cable all hardware, disk and network.

2. Create a cluster service group that includes the network paths and volume groups of the shared disk.

Be sure to include the entire set of volume groups, logical volumes and mount points required for Caché and the application to run. These include those mount points required for the main Caché installation location, your data files, journal files, and any other disk required for the application in use.

Installing a Single Instance of Caché

Use the following procedure to install and configure a single instance of Caché in the VCS cluster.

1. Bring the service group online on one node. This should mount all required disks and allow for the proper installation of Caché.

   1. Check the file and directory ownerships and permissions on all mount points and subdirectories.

   2. Prepare to install Caché by reviewing the “Installing Caché on UNIX® and Linux” chapter of the Caché Installation Guide.

2. Create a link from /usr/local/etc/cachesys to the shared disk. This forces the Caché registry and all supporting files to be stored on the shared disk resource you have configured as part of the service group.

   A good choice is to use a ./usr/local/etc/cachesys/ subdirectory under your installation directory.

   For example, assuming Caché is to be installed in /cacheprod/cachesys/, specify the following:

   mkdir –p /cacheprod/cachesys/usr/local/etc/cachesys
   mkdir –p /usr/local/etc/
   ln –s /cacheprod/cachesys/usr/local/etc/cachesys /usr/local/etc/cachesys
   
   

   

3. Run Caché cinstall on the node with the mounted disks. Be sure the users and groups (either default or custom) have already been created on all nodes in the cluster, and that they all have the same UIDs and GIDs.

4. Stop Caché and relocate the service group to the other node. Note that the service group does not yet control Caché.

5. On the second node in the cluster, create the link in /usr/local/etc/ and the links in /usr/bin for ccontrol and csession:

   mkdir –p /usr/local/etc/
   ln –s /cacheprod/cachesys/usr/local/etc/cachesys/ /usr/local/etc/cachesys/
   ln –s /usr/local/etc/cachesys/ccontrol /usr/bin/ccontrol
   ln –s /usr/local/etc/cachesys/csession /usr/bin/csession
   
   

   

6. Manually start Caché using ccontrol start. Test connectivity to the cluster through the virtual IP address (VIP). Be sure the application, all interfaces, any ECP clients, and so on connect to Caché using the VIP as configured here.

7. Be certain Caché is stopped on all nodes. Shut down VCS to prepare to add reference to the Caché agent. Make sure the agent is installed in /opt/VRTSvcs/bin/Cache/. Make sure the CacheTypes.cf configuration file is in /etc/VRTSvcs/conf/config/. Make sure ownerships and permissions match VCS requirements. See Installing the VCS Caché Agent section of this appendix for more information about these requirements.

8. Add the Caché agent configured to control your new instance to your cluster service group, as follows. This example assumes the instance being controlled is named CACHEPROD. See the Understanding the VCS Caché Agent Options section of this appendix for information about the Inst and CleanStop options.

   Cache cacheprod (
         Inst = CACHEPROD
         CleanStop = 0
   )
   

   

9. The Caché resource must be configured to require the disk resource and optionally the IP resource.

10. Start VCS and verify that Caché starts on the primary node.

Installing Multiple Instances of Caché

To install multiple instances of Caché, use the following procedure.

1. Bring the service group online on one node. This should mount all required disks and allow for the proper installation of Caché.

   1. Check the file and directory ownerships and permissions on all mount points and subdirectories.

   2. Prepare to install Caché by reviewing the “Installing Caché on UNIX® and Linux” chapter of the Caché Installation Guide.

2. Run Caché cinstall on the node with the mounted disks. Be sure the users and groups (either default or custom) are already created on all nodes in the cluster, and that they all have the same UIDs and GIDs.

3. The /usr/local/etc/cachesys directory and all its files must be available to all nodes at all times. To enable this, copy /usr/local/etc/cachesys from the first node you install to each node in the cluster. The following method preserves symbolic links during the copy process:

   cd /usr/local/
   rsync -av -e ssh etc root@node2:/usr/local/
   

   

   Verify that ownerships and permissions on the cachesys directory and its files are identical on all nodes

   Note:

   In the future, keep the Caché registries on all nodes in sync using ccontrol create or ccontrol update or by copying the directory again; for example:

   ccontrol create CSHAD directory=/myshadow/ versionid=2013.1.475
   

   

4. Stop Caché and relocate the service to the other node. Note that the service group does not yet control Caché.

5. On the second node in the cluster, create the links in /usr/bin for ccontrol and csession, as follows

   ln –s /usr/local/etc/cachesys/ccontrol /usr/bin/ccontrol
   ln –s /usr/local/etc/cachesys/csession /usr/bin/csession
   
   

   

6. Manually start Caché using ccontrol start. Test connectivity to the cluster through the VIP. Be sure the application, all interfaces, any ECP clients, and so on connect to Caché using the VIP as configured here .

7. Be certain Caché is stopped on all nodes. Shut down VCS to prepare to add reference to the Caché agent. Make sure the agent is installed in /opt/VRTSvcs/bin/Cache/. Make sure the CacheTypes.cf configuration file is in /etc/VRTSvcs/conf/config/. Make sure ownerships and permissions match VCS requirements. See Installing the VCS Caché Agent section of this appendix for more information about these requirements.

8. Add the Caché agent configured to control your new instance to your cluster service group, as follows. This example assumes the instance being controlled is named CACHEPROD. See the Understanding the VCS Caché Agent Options section of this appendix for information about the Inst and CleanStop options.

   Cache cacheprod (
         Inst = CACHEPROD
         CleanStop = 0
   )
   

   

9. The Caché resource must be configured to require the disk resource and optionally the IP resource.

10. Start VCS and verify that Caché starts on the primary node.

When you are ready to install a second instance of Caché within the same cluster, follow these additional steps:

1. Configure VCS to add the disk and IP resources associated with the second instance of Caché.

2. Bring VCS online so the disks are mounted on one of the nodes.

3. Be sure the users and groups to be associated with the new instance are created and synchronized between nodes.

4. On the node with the mounted disk, run ccinstal following the procedures outlined in the "Installing Caché on UNIX® and Linux" chapter of the Caché Installation Guide.

5. Stop Caché.

6. Synchronize the Caché registry using the following steps:

   1. On the install node run

      ccontrol list
      

      

   2. Record the instance name, version ID and installation directory of the instance you just installed.

   3. On the other node, run the following command to create the registry entry, using the information you recorded from the recently installed instance:

      ccontrol create <instance_name> versionid=<version_ID> directory=<instance_directory>
      

      

7. Add the Caché agent controller for this instance to your main.cf script.

8. The Caché resource must be configured to require the disk resource and optionally the IP resource.

9. Start the cluster service group and verify that Caché starts.

Understanding the VCS Caché Agent Options

The VCS Caché agent has two options that can be configured as part of the resource:

CleanStop determines the behavior of Caché when VCS attempts to offline the resource. When CleanStop is set to 1, Caché first uses ccontrol stop. When CleanStop is set to 0, Caché immediately uses ccontrol force. Consider the following consequences when deciding about this option:

If a controlled failover is to occur, such as during routine maintenance, follow these steps:

1. Notify and remove the user community’s connections, stop batch and background jobs.

2. Stop Caché from the command line using ccontrol stop <instance_name>.

3. Fail over the cluster service.

Even if CleanStop is set to 0, the ccontrol force command issued during the stop of the cluster service has no effect since Caché is already cleanly stopped, with all transactions rolled back by the command line ccontrol stop before processes are halted .

Failure Testing

Typical full scale testing must go beyond a controlled service relocation. While service relocation testing is necessary to validate that the package configuration and the service scripts are all functioning properly, you should also test responses to simulated failures. Be sure to test failures such as:

• Loss of public and private network connectivity to the active node

• Loss of disk connectivity

• Hard crash of active node

Testing should include a simulated or real application load. Testing with an application load builds confidence that the application will recover in the event of an actual failure.

If possible, test with a heavy disk write load; during heavy disk writes the database is at its most vulnerable. Caché handles all recovery automatically using its CACHE.WIJ and journal files but testing a crash during an active disk write ensures that all file system and disk devices are properly failing over.

Software and Firmware Updates

Keep software patches and firmware revisions up to date. Avoid known problems by adhering to a patch and update schedule.

Monitor Logs

Keep an eye on the VCS logs in /var/VRTSvcs/log/. The Caché agent logs time-stamped information to the “engine” log during cluster events. To troubleshoot any problems, search for the Caché agent error code 60022.

Use the Caché console log, the Caché Monitor and the Caché System Monitor to be alerted to problems with the database that may not be caught by the cluster software. (See the chapters “Monitoring Caché Using the Management Portal”, “Using the Caché Monitor” and “Using the Caché System Monitor” in the Caché Monitoring Guide for information about these tools.)