InfiniBand Research


InfiniBand Performance Review talk, given at Usenix 2004 in Boston, Ma.
(proceedings pdf) (proceedings html) (slides pdf)

We are currently attempting to bring up OpenIB on Macintosh G5 and IBM pSeries hardware running Linux.

Currently, we are running 2.6.11.4 linux kernels with the 'roland-uverbs' subversion branch on dual Opteron systems. NFS over IPoIB performance is not up to peak link bandwidth, but it is working. We are achieving 100MB/sec sustained throughput on reading a 10G file over nfs, and have run serveral computational chemistry jobs using scratch space over NFS. The file server machine is a dual Opteron machine with 4GB of memory, and (currently) a 6-drive raid-0 disk array.


With TCP:
10.0.0.10:/scratch/md1 on /nfs/da0/md1 type nfs
(rw,v3,rsize=32768,wsize=32768,hard,tcp,lock,addr=10.0.0.10)

opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 113.188032 seconds (94863547 bytes/sec)


With UDP:

opteron1:64bit:/nfs/da0$ mount  10.0.0.10:/scratch/md1 /nfs/da0/md1 -o
rsize=32768,wsize=32768,nfsvers=3
nfs warning: mount version older than kernel
opteron1:64bit:/nfs/da0$ cd md1/troy/
opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 96.758483 seconds (110971337 bytes/sec)

And one more time with tcp, in case there are buffer-cache effects.

opteron1:64bit:/nfs/da0$ mount  10.0.0.10:/scratch/md1 /nfs/da0/md1 -o
rsize=32768,wsize=32768,nfsvers=3,tcp
nfs warning: mount version older than kernel
opteron1:64bit:/nfs/da0$ cd md1/troy/
opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 129.839903 seconds (82697368 bytes/sec)

One last time with UDP:

opteron1:64bit:/nfs/da0$ mount  10.0.0.10:/scratch/md1 /nfs/da0/md1 -o
rsize=32768,wsize=32768,nfsvers=3
nfs warning: mount version older than kernel
opteron1:64bit:/nfs/da0$ cd troy/
opteron1:64bit:/nfs/da0/troy$ cd ../md1/troy/
opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 96.915829 seconds (110791172 bytes/sec)