Previous section   Next section

Creating an InterSystems Callout Library

An InterSystems Callout library is a shared library that contains your custom Callout functions and the enabling code that allows Caché to use them. This chapter describes how to create a Callout library and access it at runtime.
The following topics are discussed:
Note:
Shared Libraries and Callout Libraries
In this book, the term shared library refers to a dynamically linked file (a DLL file on Windows or an SO file on UNIX® and related operating systems). A Callout library is a shared library that includes hooks to the Callout Gateway, allowing it to be loaded and accessed at runtime by various $ZF functions.

Introduction to Callout Libraries

There are several different ways to access a Callout library from ObjectScript code, but the general principal is to specify the library name, the function name, and any required arguments (see “Invoking Callout Library Functions”). For example, the following code invokes a simple Callout library function:
Invoking function AddInt from Callout library simplecallout.dll
The following ObjectScript code is executed at the Terminal. It loads a Callout library named simplecallout.dll and invokes a library function named AddInt, which adds two integer arguments and returns the sum.
   USER> set sum = $ZF(-3,"simplecallout.dll","AddInt",2,2)
   USER> write "The sum is ",sum,!
   The sum is 4
This example uses $ZF(-3), which is the simplest way to invoke a single Callout library function. See “Invoking Callout Library Functions” for other options.
The simplecallout.dll Callout library is not much more complex than the code that calls it. It contains three elements required by all Callout libraries:
  1. Standard code provided when you include the cdzf.h Callout header file.
  2. One or more functions with correctly specified parameters.
  3. Macro code for a ZFEntry table, which generates the mechanism that Caché will use to locate your Callout functions when the library is loaded (see “Creating a ZFEntry Table” for details).
Here is the code that was compiled to produce the simplecallout.dll Callout library:
Callout code for simplecallout.dll
#define ZF_DLL  /* Required only for dynamically linked libraries. */
#include <cdzf.h>  /* Required for all Callout code. */

int AddTwoIntegers(int a, int b, int*outsum) {
   *outsum = a+b;   /* set value to be returned by the $ZF function call */
   return 0;   /* set the exit status code */
}
ZFBEGIN
   ZFENTRY("AddInt","iiP",AddTwoIntegers)
ZFEND

  • The first line must define
    ZF_DLL
    , which is the switch that indicates we are creating a dynamically linked Callout library rather than statically linking the code into Caché (as described later in “Statically Linked Callout Functions”).
  • The second line includes the cdzf.h file, required for both dynamic and static linked code.
  • The AddTwoIntegers() function is defined next. It has the following features:
    • Two input parameters, integers a and b, and one output parameter, integer pointer *outsum.
    • A statement assigning a value to output parameter *outsum. This will be the value returned by the call to $ZF(-3).
    • The
      return
      statement does not return the function output value. Instead, it specifies the exit status code that will be received by Caché if the $ZF call is successful. If the function fails, Caché will receive an exit status code generated by the system.
  • The last three lines are macro calls that generate the ZFEntry table used by Caché to locate your Callout library functions. This example has only a single entry, where:
    • "AddInt"
      is the string used to identify the function in a $ZF call.
    • "iiP"
      is a string that specifies datatypes for the two input values and the output value.
    • AddTwoIntegers
      is the entry point name of the C function.
The ZFEntry table is the mechanism that allows a shared library to be loaded and accessed by the Callout Gateway (see “Creating a ZFEntry Table”). A
ZFENTRY
declaration specifies the interface between the C function and the ObjectScript $ZF call. Here is how the interface works in this example:
  • The C function declaration specifies three parameters:
       int AddTwoIntegers(int a, int b, int*outsum)
    
    Parameters a and b are the inputs, and outsum will receive the output value. The return value of AddTwoIntegers is an exit status code, not the output value.
  • The
    ZFENTRY
    macro defines how the function will be identified in Caché, and how the parameters will be passed:
       ZFENTRY("AddInt","iiP",AddTwoIntegers)
    
    "AddInt"
    is the library function identifier used to specify C function AddTwoIntegers in a $ZF call. The linkage declaration (
    "iiP"
    ) declares parameters a and b as linkage type
    i
    (input-only integers), and outsum as linkage type
    P
    (an integer pointer that can be used for both input and output).
  • The $ZF(-3) function call specifies the library name, the library function identifier, and the input parameters, and returns the value of the output parameter:
       set sum = $ZF(-3,"simplecallout.dll","AddInt",2,2)
    
    Parameters a and b are specified by the last two arguments. No argument is required for the third parameter, outsum, because it is used only for output. The value of outsum is assigned to sum when the $ZF(-3) call returns .

Creating a ZFEntry Table

Every Callout library must define a ZFEntry table, which allows Caché to load and access your Callout functions (see the “Introduction to Callout Libraries” for a simple example). The ZFEntry table is generated by a block of macro code beginning with ZFBEGIN and ending with ZFEND. Between these two macros, a ZFENTRY macro must be called once for each function to be exposed.
Each ZFENTRY call takes three arguments:
   zfentry(zfname,linkage,entrypoint)
where zfname is the string used to specify the function in a $ZF call, linkage is a string that specifies how the arguments are to be passed, and entrypoint is the entry point name of the C function.
By default, the ZFEntry table will generate code that can only be used for statically linked functions (see “Statically Linked Callout Functions” for details). To create a Callout library, your code must contain a
#define ZF_DLL
directive, which is a switch that causes the macro code to generate a different mechanism for locating library functions. Instead of a static pointer table, it generates an internal GetZFTable function. When a Callout library is loaded, Caché calls this function to initialize the library for subsequent lookups of library function names.
Note:
ZFEntry Sequence Numbers
The position of an entry in the ZFEntry table can be significant. The $ZF(-5) and $ZF(-6) interfaces (described in the next chapter, “Invoking Callout Library Functions”) both invoke a library function by specifying its sequence number (starting with
1
) in the table. For example, $ZF(-6) would invoke the third function in a ZFEntry table with the following call:
   x = $ZF(-6,libID,3)
where libID is the library identifier and
3
is the sequence number of the third entry in the table.

ZFEntry Linkage Options

Each ZFENTRY statement (see “Creating a ZFEntry Table”) requires a string that determines how function arguments are passed. This section provides a detailed description of available linkage options.

Introduction to Linkages

Each ZFENTRY statement (see “Creating a ZFEntry Table”) requires a string that describes how the arguments are passed. For example,
"iP"
specifies two parameters: an integer, and a pointer to an integer. The second letter is capitalized to specify that the second argument may used for both input and output. Your code can have up to 32 actual and formal parameters.
If you specify an uppercase linkage type (permitted for all linkage types except
i
), the argument can be used for both input and output. If only one output argument is specified, its final value will be used as the return value of the function. If more than one output argument is specified, all output arguments will be returned as a comma-delimited string.
Output arguments do not have to be used as input arguments. If you specify output-only arguments after all input arguments, the function can be called without specifying any of the output arguments (see “Introduction to Callout Libraries” for an example).
From the prospective of the ObjectScript programmer, parameters are input only. The values of the actual parameters are evaluated by the $ZF call and linked to the formal parameters in the C routine declaration. Any changes to the C formal parameters are either lost or are available to be copied to the $ZF return value.
If the ZFENTRY macro does not specify a formal parameter to be used as the return value, the $ZF call will return an empty string (
""
). The linkage declaration can contain more than one output parameter. In this case, all the return values will be converted to a single comma-delimited string. There is no way to distinguish between the comma inserted between multiple return parameters, and a comma present in any one return value, so only the final return value should contain commas.
The following table describes the available options:
C Datatype Input In/Out Notes
int
i
none (use
P
)
The
i
linkage type is input only. To return an integer type, use
P
(int *). Input argument may be a numeric string (see note 1).
int *
p
P
Input argument may be a numeric string (see note 1).
double *
d
D
Input argument may be a numeric string (see note 1). Use
#D
to preserve a double * in radix 2 format (see note 2).
float *
f
F
Input argument may be a numeric string (see note 1). Use
#F
to preserve a float * in radix 2 format (see note 2).
char *
1c
or
c
1C
or
C
This is the common C NULL-terminated string (see note 3).
unsigned short *
2c
or
w
2C
or
W
This is a C style NULL-terminated UTF-16 string (see note 3).
wchar t *
4c
4C
This is a C style NULL-terminated string stored as a vector of wchar_t elements (see notes 3 and 4).
ZARRAYP
1b
or
b
1B
or
B
short 8-bit national strings (up to 32,767 characters).
ZWARRAYP
2b
or
s
2B
or
S
short 16-bit Unicode strings (up to 32,767 characters).
ZHARRAYP
4b
4B
short Unicode strings (up to 32,767 characters) stored in elements implemented by wchar_t (see note 4)
CACHE_EXSTR
1j
or
j
1J
or
J
Standard string (up to 3,641,144 characters) of 8-bit national characters
CACHE_EXSTR
2j
or
n
2J
or
N
Standard string (up to 3,641,144 characters) of 16-bit Unicode characters
CACHE_EXSTR
4j
4J
Standard string (up to 3,641,144 characters) of wchar_t characters (see note 4)
  1. i
    ,
    p
    ,
    d
    ,
    f
    — When numeric arguments are specified, Caché allows the input argument to be a string. See “Using Numeric Linkages” for details.
  2. #F
    ,
    #D
    — To preserve a number in radix 2 floating point format, use
    #F
    for float * or
    #D
    for double *. See “Using Numeric Linkages” for details.
  3. 1C
    ,
    2C
    ,
    4C
    — All strings passed with this linkage will be truncated at the first null character. See “Passing Null Terminated Strings with C Linkage Types” for details.
  4. 4B
    ,
    4C
    ,
    4J
    — Although wchar_t is typically 32 bits, Caché uses only 16 bits to store each Unicode character, so these linkages are useful only when interacting with existing code or system interfaces that specify use of wchar_t.
Structure and argument prototype definitions (including InterSystems internal definitions) can be seen in the include file cdzf.h.

Using Numeric Linkages

Numeric linkage types are provided for the following datatypes:
C Datatype Input In/Out Notes
int
i
none (use
P
)
The
i
linkage type is input only. To return an integer type, use
P
instead.
int *
p
P
Pointer to int.
double *
d
D
Use
#D
(output only) to return a double * in radix 2 format.
float *
f
F
Use
#F
(output only) to return a float * in radix 2 format.
When numeric arguments are specified, Caché allows the input argument to be a string. When a string is passed, a leading number will be parsed from string to derive a numeric value. If there is no leading number, the value
0
will be received. Thus
"2DOGS"
is received as
2.0
, while
"DOG"
is received as
0.0
. Integer arguments are truncated. For example,
"2.1DOGS"
is received as
2
. For a detailed discussion of this subject, see “String-to-Number Conversion” in Using ObjectScript.
Note:
Preserving Accuracy in Floating Point Numbers
When the output linkage is specified by
F
(float *) or
D
(double *), the number you return will be converted to an internal radix 10 number format. To preserve the number in radix 2 format, use
#F
for float * or
#D
for double *.
The
#
prefix is not permitted for input arguments. In order to avoid conversion, input values must be created with $DOUBLE in the ObjectScript code that calls the function, and the corresponding input linkages must be specified as lower case
f
or
d
.
By default, Caché stores fractional numbers in base 10 form. For example, the number
1.3
is stored as
13*.10
. In contrast, most high-level languages store fractional numbers in base 2 form (a binary number times some power of 2). If you describe an argument as
f
(float *) or
d
(double *), Caché converts its value from base 10 to base 2 on input and back to base 10 on output. This conversion may introduce slight inaccuracies.
Caché supports the $DOUBLE function for creating a standard IEEE format 64-bit floating point number. These numbers can be passed between external functions and Caché without any loss of precision (unless the external function uses the 32-bit float instead of the 64-bit double). For output, the use of the IEEE format is specified by adding the prefix character
#
to the
F
or
D
argument type. For example,
"i#D"
specifies an argument list with one integer input argument and one 64-bit floating point output argument.

Passing Null Terminated Strings with C Linkage Types

This linkage type should be used only when you know Caché will not send strings containing null (
$CHAR(0)
) characters. When using this datatype, your C function will truncate a string passed by Caché at the first null character, even if the string is actually longer. For example, the string
"ABC"_$CHAR(0)_"DEF"
would be truncated to
"ABC"
.
C Datatype Input In/Out Notes
char *
1c
or
c
1C
or
C
This is the common C NULL-terminated string.
unsigned short *
2c
or
w
2C
or
W
This is a C style NULL-terminated UTF-16 string.
wchar t *
4c
4C
This is a C style NULL-terminated string stored as a vector of wchar_t elements.
Here is a short Callout library that uses all three linkage types to return a numeric string:
Using C linkages to pass null-terminated strings
Each of the following three functions generates a random integer, transforms it into a numeric string containing up to 6 digits, and uses a C linkage to return the string .
#define ZF_DLL   // Required when creating a Callout library.
#include <cdzf.h>
#include <stdio.h>
#include <wchar.h>   // Required for 16-bit and 32-bit strings

int get_sample(char* retval) {  // 8-bit, null-terminated
   sprintf(retval,"%d",(rand()%1000000));
   return ZF_SUCCESS;
}

int get_sample_W(unsigned short* retval) {  // 16-bit, null-terminated
   swprintf(retval,6,L"%d",(rand()%1000000));
   return ZF_SUCCESS;
}

int get_sample_H(wchar_t* retval) {  // 32-bit, null-terminated
   swprintf(retval,6,L"%d",(rand()%1000000));
   return ZF_SUCCESS;
}

ZFBEGIN
ZFENTRY("GetSample","1C",get_sample)
ZFENTRY("GetSampleW","2C",get_sample_W)
ZFENTRY("GetSampleH","4C",get_sample_H)
ZFEND

Passing Short Counted Strings with B Linkage Types

The cdzf.h Callout header file defines counted string structures ZARRAY, ZWARRAY, and ZHARRAY, representing a short string (an InterSystems legacy string type). These structures contain an array of character elements (8-bit, 16-bit Unicode, or 32-bit wchar t, respectively) and a short integer (maximum value 32,768) specifying the number of elements in the array. For example:
typedef struct zarray {
    unsigned short len;
    unsigned char data[1]; /* 1 is a dummy value */
   } *ZARRAYP;
where
The
B
linkages specify pointer types ZARRAYP, ZWARRAYP, and ZHARRAYP, corresponding to the three array structures. The maximum size of the array returned is 32,767 characters.
C Datatype Input In/Out Notes
ZARRAYP
1b
or
b
1B
or
B
short national string containing up to 32,767 8-bit characters.
ZWARRAYP
2b
or
s
2B
or
S
short Unicode string containing up to 32,767 16-bit characters.
ZHARRAYP
4b
4B
short Unicode string containing up to 32,767 wchar_t characters.
The maximum total length of the arguments depends on the number of bytes per character (see “Configuring the $ZF Heap”).
Here is a Callout library that uses all three linkage types to return a numeric string:
Using B linkages to pass counted strings
Each of the following three functions generates a random integer, transforms it into a numeric string containing up to 6 digits, and uses a B linkage to return the string .
#define ZF_DLL   // Required when creating a Callout library.
#include <cdzf.h>
#include <stdio.h>
#include <wchar.h>   // Required for 16-bit and 16-bit characters

int get_sample_Z(ZARRAYP retval) {  // 8-bit, counted
   unsigned char numstr[6];
   sprintf(numstr,"%d",(rand()%1000000));
   retval->len = strlen(numstr);
   memcpy(retval->data,numstr,retval->len);
   return ZF_SUCCESS;
}

int get_sample_ZW(ZWARRAYP retval) {  // 16-bit, counted
   unsigned short numstr[6];
   swprintf(numstr,6,L"%d",(rand()%1000000));
   retval->len = wcslen(numstr);
   memcpy(retval->data,numstr,(retval->len*sizeof(unsigned short)));
   return ZF_SUCCESS;
}

int get_sample_ZH(ZHARRAYP retval) {  // 32-bit, counted
   wchar_t numstr[6];
   swprintf(numstr,6,L"%d",(rand()%1000000));
   retval->len = wcslen(numstr);
   memcpy(retval->data,numstr,(retval->len*sizeof(wchar_t)));
   return ZF_SUCCESS;
}


ZFBEGIN
ZFENTRY("GetSampleZ","1B",get_sample_Z)
ZFENTRY("GetSampleZW","2B",get_sample_ZW)
ZFENTRY("GetSampleZH","4B",get_sample_ZH)
ZFEND
Note:
Commas are used as separators in an output argument string that contains multiple values. Because commas can also be a part of counted string arrays, declare these arrays at the end of the argument list and use one array per call.

Passing Standard Counted Strings with J Linkage Types

The callin.h header file defines counted string structure CACHE_EXSTR, representing a standard InterSystems string. This structure contains an array of character elements (8-bit, 16-bit Unicode, or 32–bit wchar t) and an int value (maximum value 3,641,144) specifying the number of elements in the array:
typedef struct {
   unsigned int   len;         /* length of string */
   union {
      Callin_char_t  *ch;      /* text of the 8-bit string */
      unsigned short *wch;     /* text of the 16-bit string */
      wchar_t        *lch;     /* text of the 32-bit string */
/* OR unsigned short *lch   if 32-bit characters are not enabled */
   } str;
} CACHE_EXSTR, *CACHE_EXSTRP;

C Datatype Input In/Out Notes
CACHE_EXSTR
1j
or
j
1J
or
J
Standard string of 8-bit national characters
CACHE_EXSTR
2j
or
n
2J
or
N
Standard string of 16-bit Unicode characters
CACHE_EXSTR
4j
4J
Standard string of 32-bit characters wchar_t characters
Two InterSystems Callin functions (see the “Callin Function Reference” in Using the Callin API) are used to create or delete CACHE_EXSTR:
  • CacheExStrKill — Releases the storage associated with a CACHE_EXSTR string.
  • CacheExStrNew[W][H] — Allocates the requested amount of storage for a string, and fills in the CACHE_EXSTR structure with the length and a pointer to the value field of the structure.
See the following example for a demonstration of how these functions are used.
Here is a Callout library that uses all three linkage types to return a numeric string:
Using J linkages to pass strings
Each of the following three functions generates a random integer, transforms it into a numeric string containing up to 6 digits, and uses a J linkage to return the string .
#define ZF_DLL   // Required when creating a Callout library.
#include <cdzf.h>
#include <stdio.h>
#include <wchar.h>
#include <callin.h>

int get_sample_L(CACHE_EXSTRP retval) {  // 8-bit characters
   Callin_char_t numstr[6];
   size_t len = 0;
   sprintf(numstr,"%d",(rand()%1000000));
   len = strlen(numstr);
   CACHEEXSTRKILL(retval);
   if (!CACHEEXSTRNEW(retval,len)) {return ZF_FAILURE;}
   memcpy(retval->str.ch,numstr,len);   // copy to retval->str.ch
   return ZF_SUCCESS;
}

int get_sample_LW(CACHE_EXSTRP retval) {  // 16-bit characters
   unsigned short numstr[6];
   size_t len = 0;
   swprintf(numstr,6,L"%d",(rand()%1000000));
   len = wcslen(numstr);
   CACHEEXSTRKILL(retval);
   if (!CACHEEXSTRNEW(retval,len)) {return ZF_FAILURE;}
   memcpy(retval->str.wch,numstr,(len*sizeof(unsigned short)));   // copy to retval->str.wch
   return ZF_SUCCESS;
}

int get_sample_LH(CACHE_EXSTRP retval) {  // 32-bit characters
   wchar_t numstr[6];
   size_t len = 0;
   swprintf(numstr,6,L"%d",(rand()%1000000));
   len = wcslen(numstr);
   CACHEEXSTRKILL(retval);
   if (!CACHEEXSTRNEW(retval,len)) {return ZF_FAILURE;}
   memcpy(retval->str.lch,numstr,(len*sizeof(wchar_t)));   // copy to retval->str.lch
   return ZF_SUCCESS;
}

ZFBEGIN
ZFENTRY("GetSampleL","1J",get_sample_L)
ZFENTRY("GetSampleLW","2J",get_sample_LW)
ZFENTRY("GetSampleLH","4J",get_sample_LH)
ZFEND
Note:
Always kill CACHE_EXSTRP input arguments
In the previous example,
CACHEEXSTRKILL(retval)
is always called to remove the input argument from memory. This should always be done, even if the argument is not used for output. Failure to do so may result in memory leaks.

Configuring the $ZF Heap for Short Strings

Note:
This section applies only to legacy short strings (see “Passing Short Counted Strings with B Linkage Types”). Standard InterSystems strings (see “Passing Standard Counted Strings with J Linkage Types”) use their own stack.
The $ZF heap is the virtual memory space allocated for all $ZF short string input and output parameters. It is controlled by the following Caché system settings:
  • ZFString is the number of characters permitted for a single string parameter. The number of bytes this actually requires will vary depending on whether you are using 8-bit characters, 16-bit Unicode characters, or 32-bit characters on UNIX®. The permitted range for this setting is
    0
    to
    32767
    characters. The default is
    0
    , indicating that the maximum value should be used.
  • ZFSize is the total number of bytes Caché allocates for all $ZF input and output parameters. The permitted range for this setting is
    0
    to
    270336
    bytes, where
    0
    (the default setting) indicates that Caché should calculate an appropriate value based on the value of ZFString.
Calculate ZFSize (total number of bytes) based on ZFString (maximum number of characters per string) as follows:
      ZFSize = (<bytes per character> * ZFString) + 2050
For example, suppose ZFString has the default value of 32767 characters:
  • Using Unicode 16-bit characters, an appropriate value for ZFSize is
    (2 * 32767 + 2050) = 67584
    bytes.
  • Using UNIX® 32-bit characters, an appropriate value for ZFSize is
    (4 * 32767 + 2050) = 133118
    bytes.
These settings can be changed in either of the following places:

Callout Library Runup and Rundown Functions

An InterSystems Callout library can include custom internal functions that will be called when the shared object is loaded (runup) or unloaded (rundown). No arguments are passed in either case. The functions are used as follows:
  • ZFInit — is invoked when a Callout library is first loaded by $ZF(-3), $ZF(-4,1), or $ZF(-6). The return code from this function should be zero to indicate absence of error, or non-zero to indicate some problem. If the call was successful, the address of your ZFUnload rundown function is saved.
  • ZFUnload — is invoked when a Callout library is unloaded or replaced by a call to $ZF(-3), or is unloaded by $ZF(-4,2) or $ZF(-4,4). It is not invoked at process halt. If some error occurs during the rundown function, further calls to it will be disabled to allow unloading of the Callout library. The return value from ZFUnload is currently ignored.
When building the Callout library, you may need to explicitly export the symbols ZFInit and ZFUnload during the link procedure.

Troubleshooting

Just because you can call almost any routine with the Callout Gateway doesn’t mean you should. It is best used for math functions, and can be used effectively for interface to external devices not well handled with Caché I/O, or for some system services where a Caché interface does not otherwise exist.
The following actions can cause serious problems:
  • Accessing any memory that doesn’t belong to you
    Memory access violations will be handled by Caché, and will be treated as bugs in Caché.
  • Encountering any other errors handled by traps
    Errors handled by traps (such as divide by zero errors on most platforms) will also be treated as bugs in Caché.
  • Changing your processes priority
    Caché needs to interact with other processes running Caché. Lowering your priority can be just as bad as raising it. For example, imagine that your process acquires a spin-lock protected resource just before relinquishing the CPU. If your priority is too low, other processes with higher priority can fight for the resource, effectively preventing your process running so it can release the spin-lock.
  • Masking interrupts
    You might mask interrupts very briefly to implement your own interlock, but you should be very careful to not leave interrupts masked for any period of time.
  • Creating or opening any resource that you can’t clean-up
    It is fine to open files, and allocate memory with malloc, because those resources will be closed or freed upon termination of your process. If you create a second thread, you can’t guarantee the second thread will exit gracefully before the Caché process exits, so don’t create a second thread.
  • Returning non-opaque objects from your non-ObjectScript code
    Don’t malloc a block of memory in your code and expect to be able to use
    $VIEW(address,−3,size)
    to read it. Also, you should not pass a malloc block back to your non-ObjectScript code. Your code should return an opaque handle, and later when it receives an opaque handle it should verify that it is valid before using it.
  • Exiting your process
    You should never just call
    exit
    . Always return with either
    ZF_SUCCESS
    or
    ZF_FAILURE
    (and remember that the implementation of these values differs among Caché platforms).
  • Exiting by calling any variant of exec
    You can fork and then call exec in the child process, but be very sure that the parent will always return to Caché and the child process will never return to Caché.
  • Changing the error handling behavior for the process
    Unlike other operating systems, UNIX® systems only allow you to establish local error handling for the current and inner frames.
Previous section   Next section