Introduction to Data Integrity
The integrity of the Caché database is protected from system failure by the features described in this guide.
Fundamental Data Integrity Protection
In general, there are two different levels at which integrity can be viewed:
Structural database integrity, or physical integrity, refers to the contents of the database blocks on disk. To have structural integrity, the database blocks must be self-consistent and the globals traversable. Structural integrity during a system crash is maintained by Caché write image journal (WIJ) technology, as described in the chapter “Write Image Journaling and Recovery”, and Caché's internal algorithms.
Logical integrity refers to the data represented by the globals within the database, and encompasses the self-consistency of the data created by the application, its transactional integrity, and its being up-to-date with the real world. Logical integrity during a system crash is maintained by Caché journaling (see the “Journaling” chapter) and transaction processing. (Other aspects of logical integrity are under the control of application code, through proper use of interlocks, transactions, and other mechanisms specific to the programming paradigm that the application employs.)
Automatic WIJ and journal recovery are fundamental components of the InterSystems “bulletproof” database architecture that protects Caché databases from system failures.
Integrity Verification and Recovery Mechanisms
Although system crashes alone cannot lead to a loss of integrity, there is always a possibility that a storage device will fail catastrophically, sustain physical damage, or be tampered with. In that case, the integrity of the database, WIJ and journals can become compromised. To compensate for such disasters, Caché provides the following features:
Tools for checking the structural integrity of databases, described in Verifying Structural Integrity in this chapter.
Backup mechanisms, as described in the chapter “Backup and Restore”.
Journaling-based logical data replication for automatic failover and disaster recovery through mirroring (see the “Mirroring” chapter of the Caché High Availability Guide) and shadowing (see the “Shadowing” chapter of this guide).
DataCheck, a tool for checking the consistency of data between multiple systems when technologies such as mirroring and shadowing maintain a replicated copy of data (see the chapter “Data Consistency on Multiple Systems”).
Journaling on ECP-based shared-disk clustered systems in Caché (see the “Cluster Journaling” chapter).
Verifying Structural Integrity
An integrity check lets you verify the structural integrity (see Fundamental Data Integrity Protection) of a set of databases, or subset of globals within the databases.
The benefits of running an integrity check are as follows:
Integrity check can be integrated into your backup strategy to ensure that at the time of backup, the copy of the database was intact and that no errors were introduced during the backup itself, as discussed in External Backup in the “Backup and Restore” chapter.
Integrity check can detect corruption before users encounter it, giving time to make a plan before users are impacted.
Regular integrity checks provide a means by which the origin of any structural integrity problems that are found can be more accurately pinpointed in time, increasing the likelihood of identifying the root cause.
An integrity check lets you verify the integrity of all globals in selected databases, or of selected globals stored in a single specified database. You can run an integrity check from the Management Portal or using the ^Integrity utility in a Terminal window. This section covers the following topics:
Integrity Check False Positives
Running an integrity check on a volatile database may result in the false reporting of database integrity errors due to ongoing database updates.
When an integrity check is executed from the Management Portal or by the Task Manager, as described in Checking Database Integrity Using the Management Portal, it runs in the background, and automatically retests any globals in which errors are detected. Output from an integrity check that includes this automatic second pass reports on errors in the following manner:
If an error was detected in a global in the first pass but not in the second pass, the first error is assumed to be a false positive and no error is reported.
If the error detected in a global in the second pass differs from the error detected in the first pass, only the second-pass error is reported, with the text These errors in global <global_name> differ from the errors prior to the retry.
If the same error is detected in a global in both passes, the error is reported with the message When retried the errors in global <global_name> remained unchanged.
Integrity checks executed manually using the ^Integrity utility or one of the entry points described in Checking Database Integrity Using the ^Integrity Utility do not retest globals reporting errors on the first pass. If errors are returned, repeat the check for that particular database.
Generally, for an integrity check run on an active system, errors that are not repeated in a second pass are false positives, while errors that persist in a second pass represent actual integrity problems. The latter must be investigated, and the former may merit investigation as well, depending on the level of activity, the number of errors, and the extent to which false positives have previously occurred. The nature of your investigation will depend on your level of expertise and past experience of false positives. Steps you can take include:
Running the integrity check again, if possible during a period of lower system activity.
Running an integrity check on a restored copy of the most recent backup.
Examining the range of data in question for clues to the root problem.
Contacting the InterSystems Worldwide Response Center (WRC) for assistance.
The problem of false positives can be avoided by integrating integrity checks into your standard backup procedures, such as those described in External Backup in the “Backup” chapter of the Caché Data Integrity Guide, so that databases are checked immediately after taking a snapshot of the logical disk volume on which they reside.
Integrity Check Output
In addition to reporting any errors it encounters, the integrity check reports on the number of blocks in each global and the percentage of those blocks that is in use, breaking this information down by block level as well. For example, the following is a portion of the output of an integrity check on a SAMPLES database populated with 20,000 users:
File Name: c:\intersystems\20162555dec15a\mgr\integ.txt Cache Database Integrity Check - Report Created 01/25/2016 10:41:16 System: BBINSTOCK6440 Configuration: 20162555DEC15A No Errors were found. Full Listing of Databases Checked Directory: C:\InterSystems\20162555DEC15A\Mgr\samples\ 0 globals with errors found Global: Aviation.AircraftD 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (6% full) Data Level: # of blocks=64 512kb (87% full) Total: # of blocks=65 520kb (85% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:15 Global: Aviation.AircraftI 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (0% full) Data Level: # of blocks=4 32kb (83% full) Total: # of blocks=5 40kb (67% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:15 Global: Aviation.Countries 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (0% full) Data Level: # of blocks=1 8kb (52% full) Total: # of blocks=2 16kb (26% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:15 Global: Aviation.CrewI 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (1% full) Data Level: # of blocks=5 40kb (90% full) Total: # of blocks=6 48kb (75% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:15 Global: Aviation.EventD 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (41% full) Data Level: # of blocks=377 3,016kb (78% full) Big Strings: # of blocks=776 6,208kb (72% full) # = 479 Total: # of blocks=1,154 9,232kb (74% full) Elapsed Time = 0.1 seconds, Completed 01/25/2016 10:41:15 Global: Aviation.EventI 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (0% full) Data Level: # of blocks=3 24kb (77% full) Total: # of blocks=4 32kb (58% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:15 ... Global: ROUTINE 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (1% full) Data Level: # of blocks=6 48kb (78% full) Total: # of blocks=7 56kb (67% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:16 Global: SYS 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (0% full) Data Level: # of blocks=1 8kb (0% full) Total: # of blocks=2 16kb (0% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:16 Global: Sample.CompanyD 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (0% full) Data Level: # of blocks=1 8kb (35% full) Total: # of blocks=2 16kb (17% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:16 Global: Sample.CompanyI 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (0% full) Data Level: # of blocks=1 8kb (9% full) Total: # of blocks=2 16kb (4% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:16 Global: Sample.PersonD 0 errors found Top/Bottom Pnt Level: # of blocks=1 8kb (0% full) Data Level: # of blocks=5 40kb (81% full) Total: # of blocks=6 48kb (67% full) Elapsed Time = 0.0 seconds, Completed 01/25/2016 10:41:16 ...
When run from the Management Portal, the report begins with a listing of errors and warnings generated by the integrity check, if any. When run using the ^Integrity utility, the error summary is provided at the end of the output.
Checking Database Integrity Using the Management Portal
To check the integrity of selected databases, or of selected globals stored in a single database, navigate to the Databases page of the Management Portal (Home > System Operation > Databases) and use the following procedure:
Click Integrity Check to display a list of databases.
Select the appropriate check boxes for the databases you want to check.
If you want to check globals stored in a single database, select only the database that contains the globals you want to check, then click Select Globals to display a list of globals stored in the selected database. Select the globals you want to check, then click Save; if you do not select any globals from the list, all globals in the selected database are checked.
Specify the path of a log file in which to save the integrity check output (as described in Integrity Check Output). You can accept the default (install_dir/mgr/integ.txt), click Browse to choose an existing file, or enter your own file path.
If you want to stop checking the integrity of the database(s) upon encountering an error, select the Stop after any error check box.
Click OK to begin the integrity check. The integrity check process runs in the background.
Click Integrity Log to view the output from the most recent integrity check run using the portal. The path of this file and its contents are automatically displayed, but you can enter a new path, or browse for a different file, and click View File to view that file instead.
Integrity Check is also one of the default scheduled system background tasks in the Task Manager. You can schedule additional integrity checks if you wish, for example of different databases at different times. See Using the Task Manager in the “Managing Caché” chapter of the Caché System Administration Guide for more information about scheduling system tasks.
Checking Database Integrity Using the ^Integrity Utility
You can run a manual integrity check using the ^Integrity utility by opening a Terminal windows, switching to the %SYS namespace, and entering do ^Integrity. This is similar to running an integrity check from the Databases page of the Management Portal, except that, as noted in Integrity Check False Positives, the ^Integrity utility cannot recheck globals for which it finds errors in the first pass before completing and reporting its results, and it is therefore important to recheck globals for which errors are reported to eliminate false positives. (The Management Portal integrity check also distributes the integrity check across multiple jobs, instead of running a single job like the ^Integrity utility.)
This routine includes the following additional entry points:
Do CheckPointer^Integrity asks for a directory and a pointer block at which to start checking.
Do Silent^Integrity(logfilename,dirlist) starts a background process that does an integrity check on selected or all databases and puts the output in a file specified by the logfilename parameter. The optional dirlist parameter ($LIST format) identifies selected databases to check; if dirlist is not specified, all databases are checked.
This is the equivalent of doing Integrity Check from the Databases page (System Operation > Databases) of the Management Portal.
Do Query^Integrity(logfilename,outdevice) does not run an integrity check, but puts the contents of the file specified by the logfilename parameter, the results saved from a previous run, out on the current device or the device specified in the optional parameter outdevice.
If not specified, outdevice is the current terminal. Examples of outdevice are a printer, another display device, or another operating system file name. The latter makes a copy of logfilename.
Do SilentGlobalCheck^Integrity(logfilename,dir,gbllist) starts a background process that does an integrity check on selected globals in a selected database and puts the output in a file specified by the logfilename parameter. The dir parameter identifies the database that contains the globals you want to check. The required gbllist parameter ($LIST format) identifies one or more globals to check.
This is the equivalent of choosing Select Globals when doing Integrity Check from the Databases page (System Operation > Databases) of the Management Portal.
Do CheckList^Integrity(outputglobal,dirlist,stopafteranyerror,listofglolist,maxproc) stores the results of integrity check, including information and warnings, in the global specified by the optional outputglobal parameter; if it is not specified, the results are stored in ^CacheTempIntegrityOutput($JOB). The optional dirlist parameter specifies a $LIST of all the directories you want to check; if it is not specified, all directories are checked. If the stopafteranyerror parameter is specified, checking on a directory stops when an error is found. The optional listofglolist parameter specifies a $LIST of $LISTs of global name, one for each directory specified in the dirlist parameter which, for example, lets you check all oddDEF globals in all directories by specifying $LB($LB("oddDEF")); if there are fewer elements in the listofglolist parameter than in the list of directories, the last element is used for the rest of the directories. The optional maxproc parameter specifies the upper bound to the number of background jobs used: if it is <1, the number of cores (where the number of cores on the machine, as well as the number of directories to be checked, always act as an upper bound) on the machine are used; if it is 1, the integrity check is done directly in the foreground process.
Do Display^Integrity(integritout,flags,dirsum) displays the results of integrity check. The optional integritout parameter specifies the name of the global where the results if integrity check were placed (by CheckList^Integrity); if it is not specified, it defaults to ^CacheTempIntegrityOutput($JOB). The optional flags parameter specifies can be: 0 (to display all messages), 1 (to display only errors and warnings), or 2 (to display only errors); if it is not specified, the default is 0. If the optional dirsum parameter is specified and is not 0, the display will include a summary of blocks for each Directory scanned.
Do Exclude^Integrity asks for a list of databases to exclude from checking; entering ? displays a list of mounted databases.
While it is possible to use the Mount a database option of the ^DATABASE routine to mount any CACHE.DAT file accessible to the instance (see the “Using Character-based Security Management Routines” appendix of the Caché Security Administration Guide), if this is done with a database that was deleted from, or was never added to, the Management Portal database configuration (see Configuring Databases in the “Configuring Caché” chapter of this guide), the database is not added to the Management Portal configuration and is therefore unavailable for portal database operations and for some routines, including the ^Integrity utility described in this section.