Functions in shared libraries








Introduction

VFX Forth supports calling external API calls in dynamic link libraries (DLLs) for Windows and shared libraries in Linux and other Unix-derived operating systems. Various API libraries export functions in a variety of methods mostly transparent to programmers in languages such as C, Pascal and Fortran. Floating point data is supported for use with Lib\Ndp387.fth.

Before a library function can be used, the library itself must be declared, e.g.


LIBRARY: Kernel32.dll

Access to functions in a library is provided by the EXTERN: syntax which is similar to a C style function prototype, e.g.


EXTERN: int PASCAL SendMessage(
  HWND hwnd, DWORD mesg, WPARAM wparam, LPARAM lparam
);

This can be used to prototype the function SendMessage from the Microsoft Windows API, and produces a Forth word SendMessage.

  SendMessage  \ hwnd mesg wparam lparam -- int

For Linux and other Unices, the same notation is used. The default calling convention is nearly always applicable. The following example shows that definitions can occupy more than one line. It also indicates that some token separation may be necessary for pointers:


Library: libc.so.6

Extern: int execve(
  const char * path,
  char * const argv[],
  char * const envp[]
);

This produces a Forth word execve.

  execve       \ path argv envp -- int

The parser used to separate the tokens is not ideal. If you have problems with a definition, make sure that * tokens are white-space separated. Formal parameter names, e.g. argv above are ignored. Array indicators, [] above, are also ignored when part of the names.

The input types may be followed by a dummy name which is discarded. Everything on the source line after the closing ')' is discarded.

From VFX Forth v4.3 onwards, the use of PASCAL is not required as PASCAL is the default calling convention in the Windows version. The default for the Linux version is "C". The default is always used unless overridden in the declaration.




Format


EXTERN: <return> [ <callconv> ] <name> '(' <arglist> ')' ';'

<return>    := { <type> [ '*' ] |  void }
<arg>       := { <type> [ '*' ] [ <name> ] }
<args>      := { [ <arg>, ]* <arg> }
<arglist>   := { <args> | void }        Note: "void, void" etc. is illegal.
<callconv>  := { PASCAL | WINAPI | STDCALL | "PASCAL" | "C" }
<name>      := <any Forth acceptable namestring>
<type>      := ... (see below, "void" is a valid type)

Note that during searches <name> is passed to the operating system exactly as it is written, i.e. case sensitive. The Forth name is case-insensitive.

As a standard Forth's string length for dictionary names is only guaranteed up to 31 characters for portable source code, very long API names can cause problems. Therefore the word ALIASEDEXTERN: allows separate specification of API and Forth names (see below). ALIASEDEXTERN: also solves problems when API functions only differ in case or their names conflict with existing Forth word names.




Calling Conventions

In the discussion caller refers to the Forth system (below the application layer and callee refers to a a function in a DLL or shared library. The EXTERN: mechanism supports three calling conventions.

Unless otherwise specified, the Forth system's default convention is used. Under Windows this is WINAPI and under Linux and other Unices it is "C".




Promotion and Demotion

The system generates code to either promote or demote non-CELL sized arguments and return results which can be either signed or unsigned. Although Forth is an un-typed language it must deal with libraries which do have typed calling conventions. In general the use of non-CELL arguments should be avoided but return results should be declared in Forth with the same size as the C or PASCAL convention documented.




Argument Reversal

The default calling convention for the host operating system is used. The right-most argument/parameter in the C-style prototype is on the top the Forth data stack. When calling an external function the parameters are reordered if required by the operating system; this is to enable the argument list to read left to right in Forth source as well as in the C-style operating system documentation.

Under certain conditions, the order can be reversed. See the words "C" and "PASCAL" which define the order for the operating system. See L>R and R>L which define the Forth stack order with respect to the arguments in the prototype.




Controlling external references

1 value ExternWarnings? \ -- n
Set this true to get warning messages when an external reference is redefined.

0 value ExternRedefs?   \ -- n
If non-zero, redefinitions of existing imports are permitted. Zero is the default for VFX Forth so that redefinitions of existing imports are ignored.

1 value InExternals?    \ -- n
Set this true if following import definitions are to be in the EXTRERNALS vocabulary, false if they are to go into the wordlist specified in CURRENT. Non-Zero is the default for VFX Forth.

: InExternals   \ --
External imports are created in the EXTERNALS vocabulary.

: InCurrent     \ --
External imports are created in the wordlist specified by CURRENT.




Library Imports

In VFX Forth, libraries are held in the EXTERNALS vocabulary, which is part of the minimum search order. Other Forth systems may use the CURRENT wordlist.

For turnkey applications, initialisation, release and reload of required libraries is handled at start up.

variable lib-link
Anchors the chain of dynamic/shared libraries.

: init-libs     \ --
Release and reload the required libraries.

: find-libfunction      \ z-addr -- address|0
Given a zero terminated name, attempt to find the named function somewhere within the already active libraries.

: .Libs         \ --
Display the list of declared libraries.

: Library:      \ "<name>" -- ; -- loadaddr|0
Register a new library by name. Use in the form:

  LIBRARY: <name>

Executing <name> later will return its load address. This is useful when checking for libraries that may not be present. After definition, the library is the first one searched by import declarations.

: topLib        \ libfn --
Make the library structure the top/first in the library search order.

: firstLib      \ "<name>" --
Make the library top of the library search order. Use during interpretation in the form:

  FirstLib <name>

to make the library top of the search order. This is useful when you know that there may be several functions of the same name in different libraries.

: [firstLib]    \ "<name>" --
Make the library top of the library search order. Use during compilation in the form:

  [firstLib] <name>

to make the library top of the search order. This is useful when you know that there may be several functions of the same name in different libraries.




Function Imports

Function declarations in shared libraries are compiled into the EXTERNALS vocabulary. They form a single linked list. When a new function is declared, the list of previously declared libraries is scanned to find the function. If the function has already been declared, the new definition is ignored if ExternRedefs? is set to zero. Otherwise, the new definition overrides the old one as is usual in Forth.

In VFX Forth, ExternRedefs? is set by default to zero.

variable import-func-link       \ -- addr
Anchors the chain of imported functions in shared libraries.

: ExternLinked  \ c-addr u -- address|0
Given a string, attempt to find the named function in the already active libraries. Returns zero when the function is not found.

: init-imports  \ --
Initialise Import libraries. INIT-IMPORTS is called by the system cold chain.

defer preExtCall        \ --
Windows only. A hook provided for debugging and extending external calls without floating point parameters or return items. It is executed at the start of the external call before any parameter processing.

defer postExtCall       \ --
Windows only. A hook provided for debugging and extending external calls without floating point parameters or return items. It is executed at the end of the external call after return data processing.

defer preFPExtCall      \ --
Windows only. A hook provided for debugging and extending external calls with floating point parameters or return items. . It is executed at the start of the external call before any parameter processing.

defer postFPExtCall     \ --
Windows only. A hook provided for debugging and extending external calls with floating point parameters or return items. It is executed at the end of the external call after return data processing.

: InExternals   \ --
External imports are created in the EXTERNALS vocabulary.

: InCurrent     \ --
External imports are created in the wordlist specified by CURRENT.

: Extern:       \ "text" --
Declare an external API reference. See the syntax above. The Forth word has the same name as the function in the library, but the Forth word name is not case-sensitive. The length of the function's name may not be longer than a Forth word name.

: AliasedExtern:        \ "forthname" "text" --
Like EXTERN: but the declared external API reference is called by the explicitly specified forthname. The Forth word name follows and then the API name. Used to avoid name conflicts, e.g.

AliasedExtern: saccept int accept( HANDLE, void *, unsigned int *);

which refernces the Winsock accept function but gives it the Forth name SACCEPT. Note that here we use the fact that formal parameter names are optional.

: LocalExtern:  \ "forthname" "text" --
As AliasedExtern:, but the import is always built into the CURRENT wordlist.

: extern        \ "text" --
An alias for EXTERN:.

: .Externs      \ -- ; display EXTERNs
Display a list of the external API calls.

: .BadExterns   \ --
Display a list of any external API calls that have not been resolved.

: func-loaded?  \ xt -- addr|0
Given the XT of a word defined by EXTERN: or friends, returns the address of the DLL function in the DLL, or 0 if the function has not been loaded/imported yet.




Pre-Defined parameter types

The types known by the system are all found in the vocabulary TYPES. You can add new ones at will. Each TYPE definition modifies one or more of the following VALUEs. )

argSIZE

Size in bytes of data type.

argDEFSIGN

Default sign of data type if no override is supplied.

argREQSIGN

Sign OverRide. This and the previous use 0 = unsigned and 1 = signed.

argISPOINTER

1 if type is a pointer, 0 otherwise

Each TYPES definition can either set these flags directly or can be made up of existing types.

Note that you should explicitly specify a calling convention for every function defined.

Calling conventions

: "C"           \ --
Set Calling convention to "C" standard. Arguments are reversed, and the caller cleans up the stack.

: "PASCAL"      \ --
Set the calling convention to the "PASCAL" standard as used by Pascal compilers. Arguments are not reversed, and the called routine cleans up the stack. Note that this is not the same as the word PASCAL below.

: PASCAL        \ --
Set the calling convention to the Windows PASCAL standard. Arguments are reversed in C style, but the called routine cleans up the stack. This is the standard Win32 API calling convention. N.B. There are exceptions! This convention is also called "stdcall" and "winapi" by Microsoft, and is commonly used by Fortran programs.

: WinApi        \ --
A synonym for PASCAL.

: StdCall       \ --
A synonym for PASCAL.

: VC++          \ --
Defines the calling convention as being for a C++ member function which requires "this" in the ECX register. The function must be defined with an explicit this pointer (void * this). Because exported VC++ member functions can have either "C" or "PASCAL" styles, the this pointer must be positioned so that it is leftmost when reversed (C/WINAPI/StdCall style) or is rightmost when not reversed ("PASCAL" style). See also the later section on interfacing to C++ DLLs.

: R>L   \ --
By default, arguments are assumed to be on the Forth stack with the top item matching the rightmost argument in the declaration so that the Forth parameter order matches that in the C-style declaration. R>L reverses this.

: L>R   \ --
By default, arguments are assumed to be on the Forth stack with the top item matching the rightmost argument in the declaration so that the Forth parameter order matches that in the C-style declaration. L>R confirms this.

Basic Types

: unsigned      \ --
Request current parameter as being unsigned.

: signed        \ --
Request current parameter as being signed.

: int           \ --
Declare parameter as integer. This is a signed 32 bit quantity unless preceeded by unsigned.

: char          \ --
Declare parameter as character. This is a signed 8 bit quantity unless preceeded by unsigned.

: void          \ --
Declare parameter as void. A VOID parameter has no size. It is used to declare an empty parameter list, a null return type or is combined with * to indicate a generic pointer.

: *             \ --
Mark current parameter as a pointer.

: const ;       \ --
Marks next item as constant in C terminology. Ignored by VFX Forth.

: int32         \ --
A 32bit signed quantity.

: int16         \ --
A 16 bit signed quantity.

: int8          \ --
An 8 bit signed quantity.

: uint32        \ --
32bit unsigned quantity.

: uint16        \ --
16bit unsigned quantity.

: uint8         \ --
8bit unsigned quantity.

: LongLong      \ --
A 64 bit signed or unsigned integer. At run-time, the argument is taken from the Forth data stack as a normal Forth double with the top item on the top of the data stack.

: LONG          int  ;
A 32 bit signed quantity.

: SHORT         \ --
For most compilers a short is a 16 bit signed item, unless preceded by unsigned.

: BYTE          \ --
An 8 bit unsigned quantity.

: float         \ --
32 bit float.

: double        \ --
64 bit float.

Windows Types

The following parameter types are non "C" standard and are used by Windows in function declarations. They are all defined in terms of existing types.

: OSCALL        PASCAL  ;
Used for portable code to avoid two sets of declarations. For Windows, this is a synonym for PASCAL and under Linux and other Unices this is a synonym for "C".

: DWORD         unsigned int  ;
32 bit unsigned quantity.

: WORD          unsigned int  2 to argSIZE  ;
16 bit unsigned quantity.

: HANDLE        void *  ;
HANDLEs under Windows are effectively pointers.

: HMENU         handle  ;
A Menu HANDLE.

: HDWP          handle  ;
A DEFERWINDOWPOS structure Handle.

: HWND          handle  ;
A Window Handle.

: HDC           handle  ;
A Device Context Handle.

: HPEN          handle  ;
A Pen Handle.

: HINSTANCE     handle  ;
An Instance Handle.

: HBITMAP       handle  ;
A Bitmap Handle.

: HACCEL        handle  ;
An Accelerator Table Handle.

: HBRUSH        handle  ;
A Brush Handle.

: HMODULE       handle  ;
A module handle.

: HENHMETAFILE  handle  ;
A Meta File Handle.

: HFONT         handle  ;
A Font Handle.

: HRESULT       DWORD   ;
A 32bit Error/Warning code as returned by various COM/OLE calls.

: LPPOINT       void *  ;
Pointer to a POINT structure.

: LPACCEL       void *  ;
Pointer to an ACCEL structure.

: LPPAINTSTRUCT void *  ;
Pointer to a PAINTSTRUCT structure.

: LPSTR         void *  ;
Pointer to a zero terminated string buffer which may be modified.

: LPCTSTR       void *  ;
Pointer to a zero terminated string constant.

: LPCSTR        void *  ;
Another string pointer.

: LPTSTR        void *  ;
Another string pointer.

: LPDWORD       void *  ;
Pointer to a 32 bit DWORD.

: LPRECT        void *  ;
Pointer to a RECT structure.

: LPWNDPROC     void *  ;
Pointer to a WindowProc function.

: ATOM          word  ;
An identifier used to represent an atomic string in the OS table. See RegisterClass() in the Windows API for details.

: WPARAM        dword  ;
A parameter type which used to be 16 bit but under Win32 is an alias for DWORD.

: LPARAM        dword  ;
Used to mean LONG-PARAMETER (i.e. 32 bits, not 16 as under Win311) and is now effectively a DWORD.

: UINT          dword  ;
Windows type for unsigned INT.

: BOOL          int  ;
Windows Boolean type. 0 is false and non-zero is true.

: LRESULT       int  ;
Long-Result, under Win32 this is basically an integer.

: colorref      DWORD  ;
A packed encoding of a color made up of 8 bits RED, 8 bits GREEN, 8 bits BLUE and 8 bits ALPHA.

: SOCKET        dword  ;
Winsock socket reference.

Linux Types

: OSCALL        "C"  ;
Used for portable code to avoid two sets of declarations. For Windows, this is a synonym for PASCAL and under Linux this is a synonym for "C".

: FILE          uint32  ;
Always use as FILE * stream.

: size_t        uint32  ;
Linux type for unsigned INT.

: int32_t       int32  ;
Synonym for int32.

: int16_t       int16  ;
Synonym for int16.

: int8_t        int8  ;
Synonym for int8.

: uint32_t      uint32  ;
Synonym for uint32.

: uint16_t      uint16  ;
Synonym for uint16.

: uint8_t       uint8  ;
Synonym for uint8.

Mac OS X Types

: OSCALL        "C"  ;
Used for portable code to avoid two sets of declarations. For Windows, this is a synonym for PASCAL and under OS X this is a synonym for "C".

: FILE          uint32  ;
Always use as FILE * stream.

: size_t        uint32  ;
Unix type for unsigned INT.

: int32_t       int32  ;
Synonym for int32.

: int16_t       int16  ;
Synonym for int16.

: int8_t        int8  ;
Synonym for int8.

: uint32_t      uint32  ;
Synonym for uint32.

: uint16_t      uint16  ;
Synonym for uint16.

: uint8_t       uint8  ;
Synonym for uint8.




Using the Windows hooks

The hooks preExtCall and postExtCall are DEFERred words into which you can plug actions that will be run before and after any external call. They are principally used:

The hooks preFPExtCall and postFPExtCall are compiled into calls with floating point parameters or return values. They do not affect the NDP state.

The example below illustrates both actions.

defer preExtCall        \ --
Windows only. A hook provided for debugging and extending external calls. It is executed at the start of the external call before any parameter processing.

defer postExtCall       \ --
Windows only. A hook provided for debugging and extending external calls. It is executed at the end of the external call after return data processing.

defer preFPExtCall      \ --
Windows only. A hook provided for debugging and extending external calls with floating point parameters or return items. It is executed at the start of the external call before any parameter processing.

defer postFPExtCall     \ --
Windows only. A hook provided for debugging and extending external calls with floating point parameters or return items. It is executed at the end of the external call after return data processing.

variable XcallSaveNDP?  \ -- addr
Set true when imports must save and restore the NDP state. Windows only. From build 2069 onwards, the default behaviour for Windows includes saving and restoring the FPU state. This can be inhibited by clearing XcallSaveNDP? before execution.

variable abort-code     \ -- addr
Holds error code for higher level routines, especially RECOVERY below. Windows versions only.

defer recovery  \ -- ; handles errors in winprocs
Used by application code in the DEFERred words preExtCall and postExtCall above to install user-defined required actions.

code PreExtern  \ -- ; R: -- sys
\ Clears the abort code and saves the NDP state if XcallSaveNDP?
\ is set.
  mov   dword ptr abort-code , # 0      \ no previous abort code
  cmp     [] XcallSaveNDP? , # 0        \ Win: required
  nz, if,
    pop     eax
    lea     esp, -/fsave [esp]
    fsave   0 [esp]
    push    eax
  endif,
  ret
end-code
assign preExtern to-do preExtCall

code PostExtern \ -- ; R: sys --
\ Restore the NDP state if XcallSaveNDP? is set and test the
\ abort code.
  cmp   [] XcallSaveNDP? , # 0          \ required
  nz, if,
    pop     eax
    frstor  0 [esp]
    lea     esp, /fsave [esp]
    push eax
  endif,
  \ Detecting faults in nested callbacks.
  cmp   dword ptr abort-code , # 0      \ test previous aborting code
  nz, if,
    call  [] ^recovery                  \ execute RECOVERY if set
  endif,
  ret
end-code
assign postExtern to-do postExtCall

code PreFPExtern        \ -- ; R: -- sys ; SFP006
\ Clears the abort code.
  mov   dword ptr abort-code , # 0      \ no previous abort code
  ret
end-code
assign preFPExtern to-do preFPExtCall

code PostFPExtern       \ -- ; R: sys -- ; SFP006
\ Test the abort code.
  cmp   dword ptr abort-code , # 0      \ test previous aborting code
  nz, if,
    call  [] ^recovery                  \ execute RECOVERY if set
  endif,
  ret
end-code
assign postFPExtern to-do postFPExtCall



Interfacing to C++ DLLs

Caveats

These notes were written after testing on Visual C++ v6.0. Don't blame us if the rules change!

Example code

The example code may be found in the directory EXAMPLES\VC++. Because of the inordinate amount of time we spent wandering around inside debuggers to get this far, we recommend that you adopt a cooperative and investigative attitude when requesting technical support on this topic.

Accessing constructors and destructors

Example code for accessing the constructor of class is provided in TRYCPP.FTH which accesses the class DllTest in DLLTEST.CPP.

Since C++ is supposed to provide a higher level of abstraction, apparently simple operations may generate reams of code. So it is with the equivalent of

  pClass = new SomeClass;

The actual code generated may/will be a call to a function new to generate an object structure (not a single cell) followed by passing the return value from new to the class constructor.

The class constructor (in C++ CDllTest::CDllTest()) is not normally exported from C++ without some extra characters being added to the name. For example, the reference to it in the example code is:

  extern: PASCAL void * ??0CDLLTest@@QAE@XZ( void );

This function is not directly callable because it has to be passed the result of the new operator. To solve this problem DLLTest.dll contains a helper function CallNew which is passed the address of the constructor for the class. This is redefined as NEW for normal use.


\ C++ Helpers
extern: PASCAL void * CallNew( void * );
extern: PASCAL void CallDelete( void * this);

\ CDLLTest class specific
extern: PASCAL void * ??0CDLLTest@@QAE@XZ( void );

0 value CDLLTest        \ -- class|0

: InitCDLLTest  \ -- ; initialise the CPP interface
  ['] ??0CDLLTest@@QAE@XZ func-loaded? -> CDLLTest
;

: New   \ class -- *obj|0
  CallNew
;

: Delete        \ *obj --
  CallDelete
;

The word INITCDLLTEST gets the address of the constructor for the class, and NEW then runs the CallNew function which executes the C++ new operator and calls the constructor. Unfortunately, you will have to do this for each class that use in the DLL. What is returned by CallNew is an object pointer. This is not the object itself, but the address of another (undocumented) data structure. It can be used as the this pointer for all following member function calls.

Once you have finished with the onject, you must relase its resources using the delete method (the destructor). This is implemented in VC++ by passing the object pointer to the delete function. This is performed by the CallDelete function exported from the DLL. Again, the Forth word DELETE provides syntactic sugar by just calling CallDelete.

Accessing member functions

A Visual C++ member function exported from a DLL requires the "this" pointer in the ECX register. This can be achieved using the following form:


extern: VC++ PASCAL BOOL TestWindow1( void * this, char * ch, int n, int nbyte );

The function must be defined with an explicit this pointer (void * this). Because exported VC++ member functions can have either C or "PASCAL" styles, the this pointer must be positioned so that it is leftmost when reversed (C/PASCAL/WINAPI/StdCall/APIENTRY style) or is rightmost when not reversed ("PASCAL" style).


extern: PASCAL VC++ BOOL GetHello( void * this, char * buff, int len );
extern: PASCAL VC++ BOOL TestWindow1( void * this, char * ch, int n, int nbyte );
extern: PASCAL VC++ BOOL TestWindow2( void * this, void * pvoid, int ndword, int nlong );

0 value CDLLTest        \ -- constructor/class

: InitCDLLTest  \ --;  Initialise the CPP interface
  ['] ??0CDLLTest@@QAE@XZ func-loaded? to CDLLTest
;

create Magic#  $AAAA5555 ,      \ -- addr

#64 buffer: StringBuff  \ -- ; buffer for GetHello

: TestCDLLTest  \ -- ; test CDLLTest interface
  InitCDLLTest  CDLLTest if
    cr ." Initialisation succeeded"
    CDLLTest new ?dup if
      cr ." new succeeded"
      dup StringBuff #64 GetHello drop
      cr ." GetHello returns: " StringBuff .z$
      dup Magic# 4 5 TestWindow1 drop
      dup Magic# #20 #30 TestWindow2 drop
      delete
      cr ." delete done"
    else
      cr ." new failed"
    endif
  else
    cr ." Initialisation failed"
  endif
;

Please note that the actual code in TRYCPP.FTH may/will be different as we extend the facilities. See the source code itself!

Accessing third party C++ DLLs

Most third party C++ DLLs are provided with C header files which define the interfaces. Study of these will provide the information you need to determine how to access them.

For simple C++ classes, the DllTest.dll file can be used to provide constructor and destructor access. Note that classes with multiple constructors will export these as functions with the same basic name differentiated by the name mangling.

The DLL Fth2VC60.dll contains new and delete access for use with other DLLs. Note that the third party DLLs must be compatible with VC++ v6.0. The example file EXAMPLES\VC++\USECPP.FTH demonstrates using Fth2VC60.dll.


library: Fth2VC60.dll

extern: PASCAL void * FTH2CPPNew( void * constructor);
extern: PASCAL void FTH2CPPDelete( void * this);

: New           \ *class -- *obj|0
  FTH2CPPNew
;

: Delete        \ *obj --
  FTH2CPPDelete
;

If you are using an incompatible compiler or DLL, create a similar support DLL for that compiler. You can use the source code for Fth2VC60.dll as an example.




Changes at v4.3

The guts of the EXTERN: mechanism have been rewritten to provide more features and to support more operating systems.

Additional C types

The following C data types are now supported:

Floating point numbers are taken from the NDP floating point unit. This is directly compatible with the the Forth floating point pack in Lib\Ndp387.fth.

More Operating Systems

The requirements of newer operating systems, especially those for 64-bit operation, are more stringent for things like data alignment. Consequently the underlying mechanism has changed.

Miscellaneous

These notes are probably only relevant for code that has carnal knowledge of the VFX Forth internals.

The word NXCALL is provided for constructing your own import mechanisms, but it only deals with single-cell arguments and provides no type safety at all. It is used internally by VFX Forth in the first stage build of a console-mode kernel.

code NXCALL     \ i*x addr i -- res
Calls the operating system function at addr with i arguments, returning the result res. As far as the operating systems is concerned, i*x appear on the CPU return stack pointed to by ESP, and the return value is taken from EAX. After executing NXCALL the return value res is the contents of the EAX register.