The Pragma Token Syntax

1. Introduction

The token syntax is used to introduce references to program constructs such as types, expressions etc. that can be defined in other compilation modules. This can be thought of as a generalisation of function prototyping which is used to introduce references to functions defined elsewhere. The references introduced by the token syntax are called tokens because they are tokens for the program constructs that they reference. The token syntax is said to specify a token interface.

It is expected that the general user will have little direct contact with the token syntax, instead using the asbstract standard headers provided or using the tspec tool [Ref. 5] to generate their own token interface header files automatically. However, it may occasionally be necessary to use the raw power of the token syntax directly.

As an example of the power of the token syntax consider the program below:

#pragma token TYPE FILE#
#pragma token EXP rvalue:FILE *:stderr#

int fprintf(FILE *, const char *, ...);

void f(void) {
	fprintf(stderr, "hello world\n");
}

The first line of the program introduces a token, FILE, for a type. By using its identification, FILE, this token can be used wherever a type could have been used throughout the rest of the program. The compiler can then compile this program to TDF (the abstract TenDRA machine) even though it contains an undefined type. This is fundamental to the construction of portable software, where the developer cannot assume the definitions of various types as they may be different on different machines.

The second line of the example, which introduces a token for an expression, is somewhat more complicated. In order to make use of an expression, it is necessary to know its type and whether or not it is an lvalue (i.e. whether or not it can be assigned to). As can be seen from the example however, it is not necessary to know the exact type of the expression because a token can be used to represent its type.

The TenDRA compiler makes no assumptions about the possible definitions of tokens and will raise an error if a program requires information about an undefined token. In this way many errors resulting from inadvertent use of a definition present on the developer's system can be detected. For example, developers often assume that the type FILE will be implemented by a structure type when in fact the ISO C standard permits the implementation of FILE by any type. In the program above, any attempt to access members of stderr would cause the compiler to raise an error.

2. Program construction using TDF

Traditional program construction using the C language has two phases: compilation and linking.

In the compilation phase the source text written in the C language is mapped to an object code format. This object code is generally not complete in itself and must be linked with other program segments such as definitions from the system libraries.

When tokens are involved there is an extra stage in the construction process where undefined tokens in one program segment are linked with their definitions in another program segment. To summarise, program construction using TDF and the TenDRA tools has four basic operations:

Source file compilation to TDF

The TDF produced may be incomplete in the sense that it may contain undefined tokens;

TDF linking

The undefined tokens represented in TDF are linked to their definitions in other compilation modules or libraries. During linking, tokens with the same identifier are treated as the same token;

TDF translation

This is the conversion of TDF into standard object file format for a particular machine system. The program is still incomplete at this stage in the sense that it may contain undefined library functions;

Object file linking

This corresponds directly to standard object file linking.

3. The token syntax

The token syntax is an extension to the ISO C standard language to allow the use of tokens to represent program constructs. Tokens can be used either in place of, or as well as, the definitions required by a program. In the latter case, the tokens exist merely to enforce correct definitions and usage of the objects they reference. However it should be noted that the presence of a token introduction can alter the semantics of a program (examples are given in §5 Expression tokens). The semantics have been altered to force programs to respect token interfaces where they would otherwise fail to do so.

The token syntax takes the following basic form:

#pragma token token-introduction token-identification

It is introduced as a pragma to allow other compilers to ignore it, though if tokens are being used to replace the definitions needed by a program, ignoring these pragmas will generally cause the compilation to fail.

The token-introduction defines the kind of token being introduced along with any additional information associated with that kind of token. Currently there are five kinds of token that can be introduced, corresponding approximately to expressions, statements, type-names, member designators and function-like macros.

The token-identification provides the means of referring to the token, both internally within the program and externally for TDF linking purposes.

4. Token identification

The syntax for the token-identification is as follows:

token identification:
	name-space?identifier # external-identifier?

name-space:
	TAG

There is a default name space associated with each kind of token and internal identifiers for tokens generally reside in these default name spaces. The ISO C standard describes the five name spaces as being:

The label space

in which all label identifiers reside;

The tag space

in which structure, union and enumeration tags reside;

The member name space

in which structure and union member selectors reside;

The macro name space

in which all macro definitions reside. Token identifiers in the macro name space have no definition and so are not expanded. However, they behave as macros in all other respects;

The ordinary name space

in which all other identifiers reside.

The exception is compound type-token identifiers (see §7.4 Compound type tokens) which by default reside in the ordinary name space but can be forced to reside in the tag name space by setting the optional name-space to be TAG.

The first identifier of the token-identification provides the internal identification of the token. This is the name used to identify the token within the program. It must be followed by a #.

All further preprocessing tokens until the end of the line are treated as part of the external-identifier with non-empty white space sequences being replaced by a single space. The external-identifier specifies the external identification of the token which is used for TDF linking. External token identifications reside in their own name space which is distinct from the external name space for functions and objects. This means that it is possible to have both a function and a token with the same external identification. If the external-identifier is omitted it is assumed that the internal and external identifications are the same.

5. Expression tokens

There are various properties associated with expression tokens which are used to determine the operations that may be performed upon them.

• Designation is the classification of the value delivered by evaluating the expression. The three possible designations, implied by the ISO C standard, section 6.3, are:

  value

  expression describes the computation of a value;

  object

  expression designates a variable which may have an associated type qualifier giving the access conditions;

  function designation

  expression designates a function.

• Type specifies the type of the expression ignoring any type qualification;

• Constancy is the property of being a constant expression as specified in the ISO C Standard section 6.4.

The syntax for introducing expression tokens is:

exp-token:
	EXP exp-storage : type-name :
	NAT

exp-storage:
	rvalue
	lvalue
	const

Expression tokens can be introduced using either the EXP or NAT token introductions. Expression tokens introduced using NAT are constant value designations of type int i.e. they reference constant integer expressions. All other expression tokens are assumed to be non-constant and are introduced using EXP.

• The exp-storage is either lvalue or rvalue. If it is lvalue, then the token is an object designation without type qualification. If it is rvalue then the token is either a value or a function designation depending on whether or not its type is a function type.

• The type-name is the type of the expression to which the token refers.

All internal expression token identifiers must reside in the macro name space and this is consequently the default name space for such identifiers. Hence the optional name-space, TAG, should not be present in an EXP token introduction. Although the use of an expression token after it has been introduced is very similar to that of an ordinary identifier, as it resides in the macro name space, it has the additional properties listed below:

• expression tokens cannot be hidden by using an inner scope;

• with respect to #ifdef, expression tokens are defined;

• the scope of expression tokens can be terminated by #undef.

In order to make use of tokenised expressions, a new symbol, exp-token-name , has been introduced at translation phase seven of the syntax analysis as defined in the ISO C standard. When an expression token identifier is encountered by the preprocessor, an exp-token-name symbol is passed through to the syntax analyser. An exp-token-name provides information about an expression token in the same way that a typedef-name provides information about a type introduced using a typedef. This symbol can only occur as part of a primary-expression (ISO C standard section 6.3.1) and the expression resulting from the use of exp-token-name will have the type, designation and constancy specified in the token introduction. As an example, consider the pragma:

#pragma token EXP rvalue : int : x#

This introduces a token for an expression which is a value designation of type int with internal and external name x.

Expression tokens can either be defined using #define statements or by using externals. They can also be resolved as a result of applying the type-resolution or assignment-resolution operators (see §7.5). Expression token definitions are subject to the following constraints:

• if the exp-token-name refers to a constant expression (i.e. it was introduced using the NAT token introduction), then the defining expression must also be a constant expression as expressed in the ISO C standard, section 6.4;

• if the exp-token-name refers to an lvalue expression, then the defining expression must also designate an object and the type of the expression token must be resolvable to the type of the defining expression. All the type qualifiers of the defining expression must appear in the object designation of the token introduction;

• if the exp-token-name refers to an expression that has function designation, then the type of the expression token must be resolvable to the type of the defining expression.

The program below provides two examples of the violation of the second constraint.

#pragma token EXP lvalue : int : i#
extern short k;
#define i 6
#define i k

The expression token i is an object designation of type int. The first violation occurs because the expression, 6, does not designate an object. The second violation is because the type of the token expression, i, is int which cannot be resolved to the type short.

If the exp-token-name refers to an expression that designates a value, then the defining expression is converted, as if by assignment, to the type of the expression token using the assignment-resolution operator (see §7.5). With all other designations the defining expression is left unchanged. In both cases the resulting expression is used as the definition of the expression token. This can subtly alter the semantics of a program. Consider the program:

#pragma token EXP rvalue:long:li#
#define li 6
int f() {
	return sizeof(li);
}

The definition of the token li causes the expression, 6, to be converted to long (this is essential to separate the use of li from its definition). The function, f, then returns sizeof(long). If the token introduction was absent however f would return sizeof(int).

Although they look similar, expression token definitions using #defines are not quite the same as macro definitions. A macro can be defined by any preprocessing tokens which are then computed in phase 3 of translation as defined in the ISO C standard, whereas tokens are defined by assignment-expressions which are computed in phase 7. One of the consequences of this is illustrated by the program below:

#pragma token EXP rvalue:int :X#
#define X M + 3
#define M sizeof(int)
int f(int x)
{
	return (x + X);
}

If the token introduction of X is absent, the program above will compile as, at the time the definition of X is interpreted (when evaluating x + X), both M and X are in scope. When the token introduction is present the compilation will fail as the definition of X, being part of translation phase 7, is interpreted when it is encountered and at this stage M is not defined. This can be rectified by reversing the order of the definitions of X and M or by bracketing the definition of X. i.e.

#define X (M+3)

Conversely consider:

#pragma token EXP rvalue:int:X#
#define M sizeof(int)
#define X M + 3
#undef M
int M(int x) {
	return (x + X);
}

The definition of X is computed on line 3 when M is in scope, not on line 6 where it is used. Token definitions can be used in this way to relieve some of the pressures on name spaces by undefining macros that are only used in token definitions. This facility should be used with care as it may not be a straightforward matter to convert the program back to a conventional C program.

Expression tokens can also be defined by declaring the exp-token-name that references the token to be an object with external linkage e.g.

#pragma token EXP lvalue:int:x#
extern int x;

The semantics of this program are effectively the same as the semantics of:

#pragma token EXP lvalue:int:x#
extern int _x;
#define x _x

6. Statement tokens

The syntax for introducing a statement token is simply:

#pragma token STATEMENT init_globs#
int g(int);
int f(int x) {
	init_globs return g(x);
}

Internal statement token identifiers reside in the macro name space. The optional name space, TAG, should not appear in statement token introductions.

The use of statement tokens is analogous to the use of expression tokens (see §5). A new symbol, stat-token-name, has been introduced into the syntax analysis at phase 7 of translation as defined in the ISO C standard. This token is passed through to the syntax analyser whenever the preprocessor encounters an identifier referring to a statement token. A stat-token-name can only occur as part of the statement syntax (ISO C standard, section 6.6).

As with expression tokens, statement tokens are defined using #define statements. An example of this is shown below:

#pragma token STATEMENT i_globs#
#define i_globs {
	int i = x;
	x = 3;
}

The constraints on the definition of statement tokens are:

• the use of labels is forbidden unless the definition of the statement token occurs at the outer level (i.e outside of any compound statement forming a function definition);

• the use of return within the defining statement is not allowed.

The semantics of the defining statement are precisely the same as the semantics of a compound statement forming the definition of a function with no parameters and void result. The definition of statement tokens carries the same implications for phases of translation as the definition of expression tokens (see F.5 Expression tokens).

7. Type tokens

1. 7.1. General type tokens
2. 7.2. Integral type tokens
3. 7.3. Arithmetic type tokens
4. 7.4. Compound type tokens
5. 7.5. Type token compatibility, definitions etc.

Type tokens are used to introduce references to types. The ISO C standard, section 6.1.2.5, identifies the following classification of types:

• the type char;

• signed integral types;

• unsigned integral types;

• floating types;

• character types;

• enumeration types;

• array types;

• structure types;

• union types;

• function types;

• pointer types;

These types fall into the following broader type classifications:

• integral types - consisting of the signed integral types, the unsigned integral types and the type char;

• arithmetic types - consisting of the integral types and the floating types;

• scalar types - consisting of the arithmetic types and the pointer types;

• aggregate types - consisting of the structure and array types;

• derived types - consisting of array, structure, union, function and pointer types;

• derived declarator types - consisting of array, function and pointer types.

The classification of a type determines which operations are permitted on objects of that type. For example, the ! operator can only be applied to objects of scalar type. In order to reflect this, there are several type token introductions which can be used to classify the type to be referenced, so that the compiler can perform semantic checking on the application of operators. The possible type token introductions are:

type-token:
	TYPE
	VARIETY
	ARITHMETIC
	STRUCT
	UNION

#pragma token TYPE t_t#
#pragma token TYPE t_p#
#pragma token NAT n#
typedef t_t *ptr_type; /* introduces pointer type */
typedef t_t fn_type(t_p); /*introduces function type */
typedef t_t arr_type[n]; /*introduces array type */

#pragma token VARIETY i_t#
short f(void)
{
	i_t x_i = 5;
	return x_i;
}
	short g(void)
{
	long x_i = 5;
	return x_i;
}

#pragma token STRUCT n_t#
#pragma token STRUCT TAG s_t#
#pragma token UNION TAG u_t#

void f()
{
	n_t x1;
	struct n_t x2; /* Illegal, n_t not in the tag name space */
	s_t x3; /* Illegal, s_t not in the ordinary name space*/
	struct s_t x4;
	union u_t x5;
}

#pragma token TYPE t_t#
typedef t_t *ptr_t_t;
typedef int **ptr_t_t;

#pragma token EXP rvalue:int:N#
typedef int arr[N];
typedef int arr[4 * sizeof(int)];

8. Specifying conversions using the token syntax

1. 8.1. User-defined conversions
2. 8.2. Specifying integer promotions

The token syntax described in Annex F can be used to fine-tune the conversion analysis and integer range checks described in section 3.2.2 and chapter 4 respectively.

#pragma token TYPE IP#
#pragma no_def IP
#pragma token PROC{ TYPE t ,EXP rvalue : t* : e | EXP e}EXP rvalue:IP:p_to_ip#
#pragma TenDRA conversion p_to_ip allow
void f(void)
{
	IP i, *pip;
	i = pip;
}

#pragma TenDRA conversion conv_list allow

9. Selector tokens

The use of selector tokens is the primary method of adding member selectors to compound type tokens. (The only other method is to define the compound type token to be a particular structure or union type.) The introduction of new selector tokens can occur at any point in a program and they can thus be used to add new member selectors to existing compound types.

The syntax for introducing member selector tokens as follows:

selector-token:
	MEMBER selector-type-name : type-name :

selector-type-name:
	type-nametype-name % constant-expression

The selector-type-name specifies the type of the object selected by the selector token. If the selector-type-name is a plain type-name, the member selector token has that type. If the selector-type-name consists of a type-name and a constant-expression separated by a % sign, the member selector token refers to a bitfield of type type-name and width constant-expression. The second type-name gives the compound type to which the member selector belongs. For example:

#pragma token STRUCT TAG s_t#
#pragma token MEMBER char*: struct s_t:s_t_mem#

introduces a compound token type, s_t, which has a member selector, s_t_mem, which selects an object of type char *.

Internal identifiers of member selector tokens can only reside in the member name space of the compound type to which they belong. Clearly this is also the default name space for such identifiers.

When structure or union types are declared, according to the ISO C standard there is an implied ordering on the member selectors. In particular this means that:

• during initialisation with an initialiser-list the identified members of a structure are initialised in the order in which they were declared. The first identified member of a union is initialised;

• the addresses of structure members will increase in the order in which they were declared.

The member selectors introduced as selector tokens are not related to any other member selectors until they are defined. There is thus no ordering on the undefined tokenised member selectors of a compound type. If a compound type has only undefined token selectors, it cannot be initialised with an initialiser-list. There will be an ordering on the defined members of a compound type and in this case, the compound type can be initialised automatically.

The decision to allow unordered member selectors has been taken deliberately in order to separate the decision of which members belong to a structure from that of where such member components lie within the structure. This makes it possible to represent extensions to APIs which require extra member selectors to be added to existing compound types.

As an example of the use of token member selectors, consider the structure lconv specified in the ISO C Standard library (section 7.4.3.1). The standard does not specify all the members of struct lconv or the order in which they appear. This type cannot be represented naturally by existing C types, but can be described by the token syntax.

There are two methods for defining selector tokens, one explicit and one implicit. As selector token identifiers do not reside in the macro name space they cannot be defined using #define statements.

Suppose A is an undefined compound token type and mem is an undefined selector token for A. If A is later defined to be the compound type B and B has a member selector with identifier mem then A.mem is defined to be B.mem providing the type of A.mem can be resolved to the type of B.mem. This is known as implicit selector token definition.

In the program shown below the redefinition of the compound type s_t causes the token for the selector mem_x to be implicitly defined to be the second member of struct s_tag. The consequential type resolution leads to the token type t_t being defined to be int.

#pragma token TYPE t_t#
#pragma token STRUCT s_t#
#pragma token MEMBER t_t : s_t : mem_x#
#pragma token MEMBER t_t : s_t : mem_y#
struct s_tag { int a, mem_x, b; }
typedef struct s_tag s_t;

Explicit selector token definition takes place using the pragma:

#pragma DEFINE MEMBER type-nameidentifier : member-designator

member-designator:
	identifier
	identifier . member-designator

The type-name specifies the compound type to which the selector belongs.

The identifier provides the identification of the member selector within that compound type.

The member-designator provides the definition of the selector token. It must identify a selector of a compound type.

If the member-designator is an identifier, then the identifier must be a member of the compound type specified by the type-name. If the member-designator is an identifier, id say, followed by a further member-designator, M say, then:

• the identifier id must be a member identifying a selector of the compound type specified by type-name;

• the type of the selector identified by id must have compound type, C say;

• the member-designator M must identify a member selector of the compound type C.

As with implicit selector token definitions, the type of the selector token must be resolved to the type of the selector identified by the member-designator.

In the example shown below, the selector token mem is defined to be the second member of struct s which in turn is the second member of struct s_t.

#pragma token STRUCT s_t#
#pragma token MEMBER int : s_t : mem#
typedef struct {int x; struct {char y; int z;} s; } s_t;
#pragma DEFINE MEMBER s_t : mem s.z

10. Procedure tokens

1. 10.1. General procedure tokens
2. 10.2. Simple procedure tokens
3. 10.3. Function procedure tokens
4. 10.4. Defining procedure tokens

Consider the macro SWAP defined below:

#define SWAP(T, A, B) { \
	T x; \
	x=B; \
	B=A; \
	A=x; \
}

SWAP can be thought of as a statement that is parameterised by a type and two expressions.

Procedure tokens are based on this concept of parameterisation. Procedure tokens reference program constructs that are parameterised by other program constructs.

There are three methods of introducing procedure tokens. These are described in the sections below.

general procedure:
	PROC { bound-toks? | prog-pars? } token-introduction

simple procedure:
	PROC ( bound-toks? ) token-introduction

bound-toks:
	bound-token
	bound-token, bound-toks

bound-token:
	token-introduction name-space?identifier

prog-pars:
	program-parameterprogram-parameter, prog-pars

program parameter:
	EXP identifier
	STATEMENT identifier
	TYPE type-name-identifier
	MEMBER type-name-identifier : identifier

#pragma token PROC {TYPE t,EXP rvalue:t**:e|EXP e} EXP:rvalue:t:dderef#

/* General procedure introduction */
#pragma token PROC{TYPE t,EXP lvalue:t:e1,EXP lvalue:t:e2 | \
	TYPE t,EXP e1,EXP e2 } STATEMENT SWAP#
/* Simple procedure introduction */
#pragma token PROC(TYPE t,EXP lvalue:t:,EXP lvalue:t: ) STATEMENT SWAP#

function-procedure:
	FUNC type-name :

EXP rvalue:

#pragma token FUNC int(int): putchar#

#undef putchar

#define identifier ( id-list? ) assignment-expression

#define identifier ( id-list? ) statement

id-list:
	identifieridentifer, id-list

#pragma token PROC{TYPE t, EXP rvalue:t**:e|EXP e}EXP rvalue:t:dderef#
#define dderef (A) (**(A))

#pragma token PROC {TYPE t, EXP lvalue:t:,EXP lvalue:t:}STATEMENT SWAP#
#define SWAP(T, A, B) { \
	T x; \
	x = B; \
	B = A; \
	A = x; \
}

void f(int x, int y)
{
	SWAP(int, x, y)
}

#pragma token FUNC int(int, long) : func#
#define func(A, B) (func) (A, B)

11. Tokens and APIs

In Chapter 1 we mentioned that one of the main problems in writing portable software is the lack of separation between specification and implementation of APIs. The TenDRA technology uses the token syntax described in the previous sections to provide an abstract description of an API specification. Collections of tokens representing APIs are called interfaces. Tchk can compile programs with these interfaces in order to check applications against API specifications independently of any particular implementation that may be present on the developer's machine.

In order to produce executable code, definitions of the interface tokens must be provided on all target machines. This is done by compiling the interfaces with the system headers and libraries.

When developing applications, programmers must ensure that they do not accidentally define a token expressing an API. Implementers of APIs, however, do not want to inadvertently fail to define a token expressing that API. Token definition states have been introduced to enable programmers to instruct the compiler to check that tokens are defined when and only when they wish them to be. This is fundamental to the separation of programs into portable and unportable parts.

When tokens are first introduced, they are in the free state. This means that the token can be defined or left undefined and if the token is defined during compilation, its definition will be output as TDF.

Once a token has been given a valid definition, its definition state moves to defined. Tokens may only be defined once. Any attempt to define a token in the defined state is flagged as an error.

There are three more token definition states which may be set by the programmer. These are as follows:

Indefinable

The token is not defined and must not be defined. Any attempt to define the token will cause an error. When compiling applications, interface tokens should be in the indefinable state. It is not possible for a token to move from the state of defined to indefinable;

Committed

The token must be defined during the compilation. If no definition is found the compiler will raise an error. Interface tokens should be in the committed state when being compiled with the system headers and libraries to provide definitions;

Ignored

any definition of the token that is assigned during the compilation of the program will not be output as TDF;

These token definition states are set using the pragmas:

#pragma token-op token-id-list?

token-op:
	define
	no_def
	ignore
	interface

token-id-list:	
	TAG?identifierdot-list?token-id-list?

dot-list:
	. member-designator

The token-id-list is the list of tokens to which the definition state applies. The tokens in the token-id-list are identified by an identifier, optionally preceded by TAG. If TAG is present, the identifier refers to the tag name space, otherwise the macro and ordinary name spaces are searched for the identifier. If there is no dot-list present, the identifier must refer to a token. If the dot-listis present, the identifier must refer to a compound type and the member-designator must identify a member selector token of that compound type.

The token-op specifies the definition state to be associated with the tokens in the token-id-list. There are three literal operators and one context dependent operator, as follows:

• no_def causes the token state to move to indefinable.

• define causes the token state to move to committed;

• ignore causes the token state to move to ignored;

• interface is the context dependent operator and is used when describing extensions to existing APIs.

As an example of an extension API, consider the POSIX stdio.h. This is an extension of the ANSI stdio.h and uses the same tokens to represent the common part of the interface. When compiling applications, nothing can be assumed about the implementation of the ANSI tokens accessed via the POSIX API so they should be in the indefinable state. When the POSIX tokens are being implemented, however, the ANSI implementations can be assumed. The ANSI tokens are then in the ignored state. (Since the definitions of these tokens will have been output already during the definition of the ANSI interface, they should not be output again.)

The interface operator has a variable interpretation to allow the correct definition state to be associated with these `base-API tokens'. The compiler associates a compilation state with each file it processes. These compilation states determine the interpretation of the interface operator within that file.

The default compilation state is the standard state. In this state the interface operator is interpreted as the no_def operator. This is the standard state for compiling applications in the presence of APIs;

Files included using:

#include header

have the same compilation state as the file from which they were included.

The implementation compilation state is associated with files included using:

#pragma implement interface header

In this context the interface operator is interpreted as the define operator.

Including a file using:

#pragma extend interface header

causes the compilation state to be extension unless the file from which it was included was in the standard state, in which case the compilation state is the standard state. In the extension state the interface operator is interpreted as the ignore operator.

12. Token syntax

1. 12.1. Token specifications
   1. 12.1.1. Expression tokens
   2. 12.1.2. Statement tokens
   3. 12.1.3. Type tokens
   4. 12.1.4. Member tokens
   5. 12.1.5. Procedure tokens
2. 12.2. Token arguments
3. 12.3. Defining tokens

The C and C++ producers allow place-holders for various categories of syntactic classes to be expressed using directives of the form:

#pragma TenDRA token token-spec

or simply:

#pragma token token-spec

These place-holders are represented as TDF tokens and hence are called tokens. These tokens stand for a certain type, expression or whatever which is to be represented by a certain named TDF token in the producer output. This mechanism is used, for example, to allow C API specifications to be represented target independently. The types, functions and expressions comprising the API can be described using #pragma token directives and the target dependent definitions of these tokens, representing the implementation of the API on a particular machine, can be linked in later. This mechanism is described in detail elsewhere.

A summary of the grammar for the #pragma token directives accepted by the C++ producer is given in tdfc2pragma.

token-spec :
	token-introduction token-identification

token-introduction :
	exp-token
	statement-token
	type-token
	member-token
	procedure-token

token-identification :
	token-namespace? identifier # external-identifier?

token-namespace :
	TAG

external-identifier :
	-
	preproc-token-list

exp-token :
	EXP exp-storage? : type-id :
	NAT
	INTEGER

exp-storage :
	lvalue
	rvalue
	const

#pragma TenDRA++ rvalue token as const allow

statement-token :
	STATEMENT

type-token :
	TYPE
	VARIETY
	VARIETY signed
	VARIETY unsigned
	FLOAT
	ARITHMETIC
	SCALAR
	CLASS
	STRUCT
	UNION

member-token :
	MEMBER access-specifier? member-type-id : type-id :

member-type-id :
	type-id
	type-id % constant-expression

#pragma TenDRA++ incompatible member declaration allow

procedure-token :
	general-procedure
	simple-procedure
	function-procedure

general-procedure :
	PROC { bound-toks? | prog-pars? } token-introduction 

bound-toks :
	bound-token
	bound-token , bound-toks

bound-token :
	token-introduction token-namespace? identifier

prog-pars :
	program-parameter
	program-parameter , prog-pars

program-parameter :
	EXP identifier
	STATEMENT identifier
	TYPE type-id
	MEMBER type-id : identifier
	PROC identifier

simple-procedure :
	PROC ( simple-toks? ) token-introduction

simple-toks :
	simple-token
	simple-token , simple-toks

simple-token :
	token-introduction token-namespace? identifier?

function-procedure :
	FUNC type-id :

r ( p1, ...., pn )

PROC ( EXP rvalue : p1 :, ...., EXP rvalue : pn : ) EXP rvalue : r :

#pragma TenDRA incompatible interface declaration allow

member-offset :
	::? id-expression
	member-offset . ::? id-expression
	member-offset [ constant-expression ]

#pragma TenDRA compatible token allow

#pragma TenDRA no_def token-list
#pragma TenDRA define token-list
#pragma TenDRA reject token-list

#pragma no_def token-list
#pragma define token-list
#pragma ignore token-list

#define f( a1, ...., an )	( f ) ( a1, ...., an )

#pragma TenDRA++ implicit token definition allow
#pragma TenDRA++ no token definition allow

#pragma TenDRA interface token-list

#pragma interface token-list

#pragma TenDRA extend header-name
#pragma TenDRA implement header-name

#pragma extend interface header-name
#pragma implement interface header-name

#include header-name

no_def -> no_def define -> reject reject -> reject

no_def -> define define -> define reject -> reject

token-list :
token-id token-list?
# preproc-token-list

token-id :
token-namespace? identifier
type-id . identifier

#pragma TenDRA++ undef token token-list

#pragma TenDRA member definition type-id : identifier member-offset