MP_Lite: Using the MP_Lite syntax

MP_Lite has its own syntax, or can use the MPI syntax. The MP_Lite syntax is an experiment in trying to design an interface that can be more flexible and easier to use than MPI. There is also some additional experimental functions built in that are not in the MPI standard.

Both provide the same high level of performance and functionality. I'd recommend in general to program using the MPI standard to provide the greatest portability. However, MP_Lite applications can still be compiled to run on top of MPI, so there is no loss of portability by going with MP_Lite syntax. It is simply a matter of taste and programming style.

MP_Lite syntax does not abstract the operator and data types, so there is no need for the include files that define the operators and datatypes. Instead, multiple function bindings are used to more easily and clearly discriminate between the data types and operator types. For example, the double, float, and int data types (DOUBLE PRECISION/REAL*8, REAL*4, INTEGER) are represented by preceding the functions with d, f, or i. The default functions assume that you are specifying the number of bytes, allowing you to handle any other data types and structures. Global allreduce operations use the operator in the function name. For example, MP_dSum( darray, count ) does a global summation of count elements of the array darray of doubles. Asynchronous sends and receives use an A like in MP_ASend() instead of the i that MPI uses for immediate mode.

Below is a listing of the basic MP_Lite functions supported for C and Fortran. All the most common functions are supported, and operate in the manner typical of their MPI equivalents. For complex data types in Fortran, simply use the double precision/real*8 functions with the count doubled.

MP_Lite also has some unique features that are designed to make it easier to work in a multiprocessor environment. These include some I/O functions that provide several different approaches to handling file access. For example, global versions of the fscanf() and fprintf() functions have node 0 read/write the common data from the file and broadcast it to the other nodes in the case of the global fscanf(). This also works from Fortran, bringing C-style I/O functions to Fortran. Descriptions of the features unique to MP_Lite are at the end of this webpage.

MP_Lite Fortran functions
CALL MP_Init(0,0)
CALL MP_Myproc(myproc)
CALL MP_Nprocs(nprocs)
CALL MP_Finalize()
Send and Receive functions

CALL MP_Send(buf, nbytes, dest, tag)
MP_dSend(darray, count, dest, tag) MP_iSend(iarray, ...) MP_fSend(farray, ...)

CALL MP_Recv(buf, nbytes, source, tag)
MP_dRecv(darray, count, source, tag) MP_iRecv(iarray, ...) MP_fRecv(farray, ...)

CALL MP_ASend(buf, nbytes, dest, tag, msg_id)
MP_dASend(darray, count, dest, tag, msg_id) MP_iASend(iarray, ...) MP_fASend(farray, ...)

CALL MP_ARecv(buf, nbytes, source, tag, msg_id)
MP_dARecv(darray, count, source, tag, msg_id) MP_iARecv(iarray, ...) MP_fARecv(farray, ...)

CALL MP_SSend(buf, nbytes, dest, tag)
CALL MP_SRecv(buf, nbytes, source, tag)

CALL MP_Wait(msg_id)
CALL MP_Block(msg_id)
CALL MP_Test(msg_id, rtn_flag)
CALL MP_Testall(count, msg_ids, rtn_flag)

CALL MP_Get_Last_Tag( last_tag )
CALL MP_Get_Last_Source( last_source )
One-Sided Operations

CALL MP_Win_create( base, nbytes, disp_unit, window)
CALL MP_Put(buf, nbytes, dest, dest_disp, >window)
CALL MP_Get(buf, nbytes, source, source_disp, window)
CALL MP_APut(buf, nbytes, dest, dest_disp, window, msg_id)
CALL MP_AGet(buf, nbytes, source, source_disp, window, msg_id)
CALL MP_Win_free(base)
Typed Accumulate Functions (Sync and Async, d/i/f)

CALL MP_Accumulate(buf, nbytes, dest, dest_disp, op_type, window)
CALL MP_AAccumulate(buf, nbytes, dest, dest_disp, op_type, window, msg_id)
MP_dSumAccumulate(buf, nbytes, dest, dest_disp, window), MP_AdSumAccumulate, etc
MP_dMaxAccumulate, MP_iMaxAccumulate, MP_fMaxAccumulate
MP_AdMinAccumulate, MP_AiMinAccumulate, MP_AfMinAccumulate

Support functions for MPI 1-sided operations

MP_shMalloc, MP_Win_fence, MP_Win_lock, MP_Win_unlock, MP_Win_start, MP_Win_complete,
MP_Win_post, MP_Win_wait, MP_Win_test
Global operations
CALL MP_dSum( darray, count )
MP_iSum( iarray, count ) MP_fSum( farray, count )
CALL MP_dMin( darray, count )
MP_iMin( iarray, count ) MP_fMin( farray, count )
CALL MP_dMax( darray, count )
MP_iMax( iarray, count ) MP_fMax( farray, count )
CALL MP_Gather(buf, nbytes)
MP_dGather(darray, count) MP_iGather(iarray, ...) MP_fGather(farray, ...)

Note: Masked versions of these functions are available. See below for a description.

CALL MP_SShift( buf_out, dest, buf_in, source, nbytes)

CALL MP_Sync()
CALL MP_Bcast( buf, nbytes, root_node)
MP_dBcast( darray, count, root_node) MP_iBcast( iarray, ...) MP_fBcast( farray, ...)
CALL MP_Broadcast( buf, nbytes)
MP_dBroadcast( darray, count) MP_iBroadcast( iarray, ...) MP_fBroadcast( farray, ...)

Timing functions
t0 = MP_Time()

CALL MP_Enter('Function name')
CALL MP_Leave(0.0d0)
CALL MP_Time_Report('time.out')

CALL MP_Bedcheck()
I/O functions
CALL MP_Gfopen( unit)

CALL MP_Gfscanf( unit, format_string, arg1, ... )
CALL MP_Gfprintf( unit, format_string, arg1, ... )

CALL MP_NOfscanf( unit, format_string, arg1, ... )
CALL MP_NOfprintf( unit, format_string, arg1, ... )

CALL MP_dGRead( unit, darray, count)
MP_iGRead() MP_fGRead() MP_cGRead() MP_wGRead()
CALL MP_Echo( unit_in, unit_out)

CALL MP_Gfclose( unit)

CALL open( unit )
CALL MP_Get_Token( unit )
CALL MP_Pass_Token( unit )
CALL close( unit )

MP_Lite C functions
MP_Init(&argc, &argv);
MP_Myproc( &myproc );
MP_Nprocs( &nprocs );
Send and Receive functions

MP_Send(&buf, nbytes, dest, tag);
MP_dSend(&darray, count, dest, tag); MP_iSend(&iarray, ...); MP_fSend(&farray, ...);

MP_Recv(&buf, nbytes, source, tag);
MP_dRecv(&darray, count, source, tag); MP_iRecv(&iarray, ...); MP_fRecv(&farray, ...);

MP_ASend(&buf, nbytes, dest, tag, &msg_id);
MP_dASend(&darray, count, dest, tag, &msg_id); MP_iASend(&iarray, ...); MP_fASend(&farray, ...);

MP_ARecv(&buf, nbytes, source, tag, &msg_id);
MP_dARecv(&darray, count, source, tag, &msg_id); MP_iARecv(&iarray, ...); MP_fARecv(&farray, ...);

MP_SSend(&buf, nbytes, dest, tag);
MP_SRecv(&buf, nbytes, source, tag);

MP_Test(&msg_id, &rtn_flag);
MP_Testall(count, &msg_ids, &rtn_flag);

MP_Get_Last_Tag( &last_tag );
MP_Get_Last_Source( &last_source );
One-Sided Operations

MP_Win_create( &base, nbytes, disp_unit, &window);
MP_Put(&buf, nbytes, dest, dest_disp, &window);
MP_Get(&buf, nbytes, source, source_disp, &window);
MP_APut(&buf, nbytes, dest, dest_disp, &window, &msg_id);
MP_AGet(&buf, nbytes, source, source_disp, &window, &msg_id);
Typed Accumulate Functions (Sync and Async, d/i/f)

MP_Accumulate(&buf, nbytes, dest, dest_disp, op_type, &window);
MP_AAccumulate(&buf, nbytes, dest, dest_disp, op_type, &window, &msg_id);
MP_dSumAccumulate(&buf, count, dest, dest_disp, &window), MP_AdSumAccumulate, etc
MP_dMaxAccumulate, MP_iMaxAccumulate, MP_fMaxAccumulate
MP_AdMinAccumulate, MP_AiMinAccumulate, MP_AfMinAccumulate

Support functions for MPI 1-sided operations

MP_shMalloc, MP_Win_fence, MP_Win_lock, MP_Win_unlock, MP_Win_start, MP_Win_complete, MP_Win_post, MP_Win_wait, MP_Win_test
Global operations

MP_dSum( &darray, count ); MP_iSum(&iarray, count); MP_fSum(&farray, count);
MP_dMin( &darray, count ); MP_iMin(&iarray, count); MP_fMin(&farray, count);
MP_dMax( &darray, count ); MP_iMax(&iarray, count); MP_fMax(&farray, count);
MP_Gather(&buf, nbytes);
MP_dGather(&darray, count) MP_iGather(&iarray, ...) MP_fGather(&farray, ...)

Note: Masked versions of these functions are available. See below for a description.

MP_SShift( &buf_out, dest, &buf_in, source, nbytes);


MP_Bcast( &buf, nbytes, root_node);
MP_dBcast( &darray, count, root_node); MP_iBcast( &iarray, ...); MP_fBcast( &farray, ...);

MP_Broadcast( &buf, nbytes);
MP_dBroadcast( &darray, count); MP_iBroadcast( &iarray, ...); MP_fBroadcast( &farray, ...);

Timing functions
double t0, MP_Time();
t0 = MP_Time();

MP_Enter("Function name");

I/O functions

MP_Gfopen( &file_descriptor);

MP_Gfscanf( &file_descriptor, &format_string, &arg1, ... );
MP_Gfprintf( &file_descriptor, &format_string, arg1, ... );

MP_NOfscanf( &file_descriptor, &format_string, &arg1, ... );
MP_NOfprintf( &file_descriptor, &format_string, arg1, ... );

MP_dGRead( &file_descriptor, &darray, count);
MP_iGRead(); MP_fGRead(); MP_cGRead(); MP_wGRead();

MP_Gfclose( &file_descriptor);

fopen( &file_descriptor );
MP_Get_Token( &file_descriptor );
MP_Pass_Token( &file_descriptor );
fclose( &file_descriptor );

One-sided communications
The one-sided communications are currently only supported in the TCP module. They will soon be added to the SMP module.

MP_SSend() & MP_SRecv()
These are synchronous communications. The source node waits for the destination node to prepost the receive to guarantee that no buffering is done. This produces a higher latency, but insures that there will be no problems with the buffers when large messages are sent.

MP_Get_Last_Tag( &last_tag )
A message tag can be any positive integer. A negative message tag in a Recv() function is a wild card that will match any tag. MP_Get_Last_Tag() will return the tag for the message that matched the source and number of bytes.

MP_Get_Last_Source( &last_source )
A negative source in a Recv() function will match a message from any source. However, you should be aware that this often requires buffering of the messages which can slow communications down significantly. I highly recommend specifying the source node in any time-critical areas of your code. MP_Get_Last_Source() will return the source node that the last message came from.

MP_Block(&msg_id) will block for an MP_ASend() to complete without pushing the data to a send buffer.

MP_Broadcast() is the same as MP_Bcast(), but with the root as node 0.

MP_dSum_masked( &darray, count, &maskarray )
There are masked versions of the global Min(), Max(), and Sum() operations. The mask is simply an integer array from 0 to nprocs-1, where a 1 signifies that the node is taking part in the operation and a 0 signifies that it is not. You can therefore generate and use an array mask in the same way that you would generate a communicator in MPI.

MP_SShift(&buf_out, dest, &buf_in, source, tag)
Send nbytes of data from buf_out to dest and receive nbytes from source to buf_in at the same time.

MP_Lite uses the MP_Bedcheck() function to allow the user to put a job into idle mode when needed. mpstop from the MP_Lite/bin directory sends a SIGUSR1 signal to each process in a run. An MP_Bedcheck() call in a program checks to see whether the SIGUSR1 has been issued, and puts the code into an idle mode if it has. It then checks every 60 seconds for a second SIGUSR1 signal, which is sent using mpcont, which will wake the code back up. This allows the user to temporarily stop a job externally when the computing resources are needed for something more important. NOTE: The pThreads package uses SIGUSR1 and SIGUSR2, so this function will need to be disabled when a pThreads controller is written for MP_Lite.

I/O - Global
The MP_Gfopen() and MP_Gfclose() functions are similar to the C functions fopen() and fclose(), but only node 0 opens and closes the file. Node 0 is then responsible for reading and disseminating the data to the other nodes. The MP_Gfscanf() function is a global fscanf() where node 0 does a typical fscanf() and broadcasts the resulting data to all nodes. The MP_Gfprintf() simply has node 0 perform the fprintf(). All this works in f77 as well, bringing C-style I/O to Fortran codes.

I/O - Node Ordered
In the MP_NOfscanf() function, node 0 does an fscanf() from a common file for each node in order and sends the info to that node. Likewise, node 0 collects the information from each node in order in the MP_NOfprintf() function and writes it out to a common file.

I/O - Arrays
Global MP_xGRead() functions have node 0 read in a vector of doubles/REAL*8 (d), integers (i), floats/REAL*4 (f), characters (c), or words (w). Node 0 then broadcasts the vector to all nodes.

I/O - Token pass
Each node can do an fopen() command using the same filename and get a file pointer to the same file. MP_Pass_Token() and MP_Get_Token() can be used to pass this common file pointer or offset around between nodes. This allows each node to read or write whatever info it wants to or from a common file, then to pass the offset along to another node.

Cartesian coordinate functions

MPI has several functions to assist in mapping the nodes to a Cartesian coordinate system. These are useful, and easy to implement, so they are supported in MP_Lite.

MP_Lite Fortran cartesian coordinate functions
CALL MP_Cart_Create( ndims, dims )
CALL MP_Cart_Coords( rank, coords )
CALL MP_Cart_Rank( coords, rank )
CALL MP_Cart_Shift( direction, distance, source, dest )
MP_Lite C cartesian coordinate functions
MP_Cart_Create( ndims, dims );
MP_Cart_Coords( rank, coords );
MP_Cart_Rank( coords, &rank );
MP_Cart_Shift( direction, distance, &source, &dest );

The function MP_Cart_create() sets up the cartesian arrangement of nodes given the number of dimensions ndims and the number of nodes in each dimension as stored in the array dims. If dims[0] is set to 0, the function sets the dims array itself by trying to make all dimensions as close as possible.

MP_Cart_coords() returns the cartesian coordinates for a given node number rank. MP_Cart_rank() returns the rank given the cartesian coordinates.

MP_Cart_shift() returns the source and destination nodes for a global shift given a direction (which dimension) and distance (number of nodes).