The following graph illustrates the performance of the ethernet interfaces on the cluster. You can expect peak ethernet performance to be about 90Mbps for the fast-ethernet and ~800Mbps for the Gigabit Ethernet. Latency is similar for both at around 110-130 milliseconds. Note that you must use Jumbo Frames (mtu larger than 1500 bytes) to achieve throughput above 400Mbps. However, at 800Mbps (=100MB/sec) the performance of the network is substantially greater than disk IO.
Note that MPI does not fully realize the gain in performance offered by Jumbo Frames. Also there is a buffer problem around 4KB in the each of the MPI curves. Finally, note that the Alteon switch delivers almost the full back-back performance and is thus not an issue in the overall performance.