The TCP module
The TCP module is designed for workstation/PC clusters running Unix
and for individual Unix machines. All machines must trust each other enough
to do an ssh/rsh without prompting for a password, and all must share
a common file system.
SIGIO interrupt-driven controller
This asynchronous or interrupt driven mode (make tcp) works
best if the TCP send and receive buffer sizes are increased as
described in the optimization page, but can
function well with the default buffer sizes.
All the message queues are local linked lists of message headers that
contain all the information about the current state of the message.
An MP_ASend() creates a message header and posts it to the end
of its send queue for the destination node send_q[dest].
If the queue was empty, the internal function do_send() will
activate that send queue, and a writev() copies the 12 byte header and as
much data as will fit into the TCP send buffer. If any message data does
not fit into the TCP send buffer, the controller sigio_handler() will take
care of it when a SIGIO interrupt is generated as the data drains from the
buffer. This routine services all active transfers by pushing more data into
the send buffers using writev() or pulling more data out of the active
receive buffers using readv(), then returns control to the main program.
If the send queue for the destination node was already active with
other messages, then the message header is simply posted at the end of the
send message queue where it will be handled in the order that it was posted.
If any data has not been sent when an MP_Wait() with the msg_id of
the MP_ASend() is encountered, it will malloc() and push the remaining
data to a send buffer to avoid any possibility of a lock-up condition.
The controller will then pull data from the malloc'ed send buffer instead of
its original place, and free the memory once the entire message has been sent.
A blocking MP_Send() is then just an MP_ASend() followed by an
An MP_ARecv() call from a known source creates a message header
and posts it to the end of the receive message queue for the source node
recv_q[source]. If the queue was empty, the internal function do_recv()
must first check for a matching message in the receive message buffer
msg_q[source] for the source node. If one exists, it is copied over to user
space and that receive buffer is freed.
If there was no matching message buffered in msg_q[source], do_recv() will
activate that receive queue, and a readv() will attempt to read a 12 byte
header and data from the TCP receive buffer for the source node.
If there is data and the header matches the number of bytes and message tag,
the data is copied to user space. If not all the data was there, or if no
header was found at all, control is returned to the main program and the
controller will continue to check that TCP receive buffer every time a
SIGIO interrupt is generated until the message has been completely received.
If a header and data are found, but the header does not match the current
message, then an out-of-order message condition exists that must be handled
by pushing the current message to the msg_q[source] buffer. A message header
is created, a malloc() call creates a buffer for the data, the data is copied
to this receive buffer, and the new message header is put at the start of the
receive queue so that it gets processed first. The MP_ARecv() then
must wait until the out-of-order receive has been completed before the
controller will again try to match it to the next header that arrives.
For an unknown source (-1), the msg_q[node] must be checked first for all
nodes. If no matching message is found, sigio_handler() must activate all
TCP receive buffers and keep messages flowing by buffering them to the
message queues if they don't match. This is much less efficient than when the
source is known, but can be a convenience when performance is not critical.
An MP_Wait() on the MP_ARecv() msg_id simply blocks until
sigio_handler() has completed the receive. MP_Recv() is then simply
an MP_ARecv() followed by an MP_Wait().
This module can use either POSIX or BSD signals. POSIX is the default,
since it works best for most flavors of Unix.
Messages sent out-of-order between two nodes will
reduce the performance since they get buffered on the destination node.
Receives posted without specifying the source node will also kill the
performance since all TCP receive buffers must be activated, and all
data encountered must be buffered if it does not match.
If the TCP buffers are larger than the messages being transferred,
preposting the receives really is not necessary since the data will flow
through the TCP buffers without any blocks. When the TCP buffer sizes
are smaller than typical message traffic, preposting the receives allows
the controller to move the data from the TCP buffer to user space as it
comes in, allowing more to flow in from the source node. If no receive
is preposted, data will fill the TCP receive buffer on the destination node,
then the TCP send buffer on the source node, and the source node may have
to block until data starts draining from on the destination node.
For Gigabit Ethernet networks, most MPI implementations can now
deliver the raw TCP capabilities to the application layer if the
system is tuned properly. This primarily means adjusting the
TCP socket buffer sizes. This may involve adjusting them both
at the OS level and setting the propper environmental variables
for the message-passing implementation.
MP_Lite already tries to increase these buffers to 1 MB, but most systems
have hard limits set. If you have control over the cluster you are
running on, you can increase these limits. For Linux,
the maximum TCP socket buffer sizes can be set in
the /etc/sysctl.conf file.
# Increase the maximum socket buffer sizes
net.core.rmem_max = 524288
net.core.wmem_max = 524288
All nodes open the common .mplite.config file generated by the
mprun script. This contains information such as the number of
nodes, the program name and arguments, and the host name or interface name for
each node (or multiple interface names if channel bonding is being used).
In order for the nodes to set up their socket connections, they need
to know the listen port numbers for each node. Each node tries to create a
listen socket using the same port number, but can rotate to a higher port
number if it fails for any reason. If a failure occurs for any reason,
file is created to signal node 0 that there was a problem and a new port is
Node 0 tries to contact the listen socket on the default port on each
host to make initial contact. This will succeed in most cases, but if
not node 0 looks for the .changeport file that will signal
it to try a different port. This is really only a backup mechanism
that makes the system a little more robust against hanging processes from
previous runs. However, it will eventually be used to allow multiple
MP_Lite jobs to be run concurrently on the same nodes. More work will
need to be done before this is possible though.
Once node 0 has all the port numbers accumulated, it sends them to each
node in turn. The nodes then start handshaking with each other, with each
node accepting connections from each of the lesser nodes and initiating
the connections to the greater nodes.
The MP_Sync() function uses a separate port to connect each node
to its neighbor below and the one above. This allows a barrier synchronization
to be done without being affected by data coming through the primary TCP
buffers. The barrier synchronization is done by sending a message from node
0 to node 1, which sends it to node 2, etc., until it is cycled back to
node 0. At this point, all nodes have reached the barrier and node 0 cycles
another go signal around to all nodes. This double cycle is not the
most efficient way of doing a synchronization, but it does provide an