PDA

View Full Version : NFS performance (VM guest is ~8x faster!)



af3556
8th November 2008, 07:43 PM
NFS file access on my 10.5.5 Mac Mini is dog slow - I can only ever get at most ~6 MB/s over a 100 Mb/s network, whereas Linux clients can max out at 10-11 MB/s (close enough to the theoretical network upper bound of 12 MB/s).

This seems a common complaint (http://www.google.com/search?q=%22os+x%22+nfs+slow), I'd be interested to see how other people fare. If you're using NFS on your OS X client, could you please try the following:

create a 100 MB file on the NFS server:

$ cd /path/to/your/exported/filesystem
$ dd if=/dev/urandom of=f1 bs=1k count=100k

on the client, read the file over the network:

$ mkdir m
$ mount servername:/your/exported/filesystem m
$ dd if=m/f1 of=/dev/null bs=1m
$ umount m

Repeat the 2nd through 4th commands (i.e. mount, dd, umount) a couple of times, results should be fairly consistent (within ~10%). Note that if you don't flush the cache via the umount/mount then the second and subsequent runs will be from the filesystem cache and invalid for the purposes of this test.

dd will report a transfer rate. Can you please report back here with the following:


NFS config (TCP/UDP, any tweaks)
NFS server type and specs (OS, CPU, RAM, actual network speed)
client system specs (OS, model, RAM, actual network speed)
how the two are connected together (e.g. 10/100 hub, gig switch, etc)
representative transfer rate


(Note I'm not really interested in minor variations, rather to see if my system's (lack of) performance is par for the OS X course, or if there's something to be had by looking further into it.)

If you have "modern" hardware and a 100 Mb/s LAN, the network should be the bottleneck at around 11-12 MB/s. At 1Gb/s, client and/or server CPU will probably be the limiting factor though I'd expect upward of 50-100 MB/s.

Here are my results: Linux server, OS X client = 1.5MB/s (slow!)

NFS config: export option async, otherwise all defaults
server: Linux 2.6.22, Intel Core2 Quad Q6600 @ 2.40GHz, 4 GB RAM, 100 Mb/s
client: Mac Mini 10.5.5, 1 GB, 1Gb/s
network: server -> 100 Mb/s switch -> Airport Extreme (Gb/s) -> Mini
104857600 bytes transferred in 66.302718 secs (1581498 bytes/sec) (= 1.5 MB/s)


and for comparison: Linux server, Linux client = 10 MB/s (fast!)

server: (same)
client: Linux 2.6.24, Intel Celeron 2.00GHz, 1.5 GB RAM, 100 Mb/s
network: server -> 100 Mb/s switch -> client
104857600 bytes (105 MB) copied, 10.2203 s, 10.3 MB/s


and most interestingly, a Linux guest running in VMWare Fusion on the above Mac Mini = 11 MB/s (fastest!!)

server: (same)
client: Linux 2.6.19 w/ 256 MB RAM, VM guest on Mac Mini 10.5.5, 1 GB, 1Gb/s
network: server -> 100 Mb/s switch -> Airport Extreme (Gb/s) -> Mini
104857600 bytes (105 MB) copied, 9.17409 seconds, 11.4 MB/s


Yes, the Linux guest is almost an order of magnitude *faster* than the OS X host it is running on.

If you fiddle with dd's blocksize (e.g. append "bs=1m" to the dd command) you can improve OS X's results up to ~6 MB/s, but that's still pretty mediocre. Similarly you can tune the NFS mount (http://nfs.sourceforge.net/nfs-howto/ar01s05.html), though as you can see from the numbers above, my server's easily capable of saturating the 100 Mb/s LAN as is. I've fiddled some with NFS options on the OS X box, to little benefit.

af3556
11th November 2008, 10:04 PM
No takers? C'mon - you know you want to show me how your Mac Pro is smokin' my little Mini!

themacuser
11th November 2008, 10:24 PM
I think the issue is that not many of us in here use NFS...

chrome
11th November 2008, 11:17 PM
Try mounting with larger rsize/wsize values. If your NFS server supports v4, I hear thats faster too, though I have no firsthand experience.

I don't use NFS with my mini, though I might try it. I currently just mount it's stuff with AFS.

There is probably some other tunables. macosxhints.com - Improve NFS client performance (http://www.macosxhints.com/article.php?story=20030504042711312) seems like a good start.

af3556
14th November 2008, 08:35 PM
There is probably some other tunables.

I hadn't come across the nfsiod thread suggestion before, so thanks for that. Turns out that on 10.5 nfsiod is deprecated. Evidently similar tweaks now belong in a file /etc/nfs.conf. The default nfsiod_thread_max is 16, which is much higher than the ca. 2003. I tried higher (and lower) values, didn't make any difference :-(


I currently just mount it's stuff with AFS.
(I assume you mean AFP (http://en.wikipedia.org/wiki/Apple_Filing_Protocol) rt. AFS (http://en.wikipedia.org/wiki/Andrew_File_System)?) What kind of speeds do you get?

(I have netatalk (AFP) and Samba running on my server: Samba is much slower again than NFS, and AFP a bit slower. I don't think OS X is built for high-speed networking...)

chrome
15th November 2008, 12:53 AM
I meant AFP yes. Works fine for me.


I don't think OS X is built for high-speed networking

When in doubt, always assume you're at fault. If I had a dollar every time someone said "X is shit at doing Y because I can't get it to work" I'd probably have $70 or so. That said, Apple probably made some choices (or they were made for them in the bsd userspace) that were bad, which may be a contributing factor.

Pushing 0's across the network with netcat (nc) I can push over 100MB/sec across my cheap 1GBit switch. The OS can easily push the packets, but for me to generate that much traffic I had to use two threads. Maybe there is more going on here?

af3556
16th November 2008, 09:35 PM
the OS can easily push the packets, but for me to generate that much traffic I had to use two threads. Maybe there is more going on here?

There's the rub - doing almost anything that hammers the network in OS X consume far more CPU than you'd expect cf. BSD and Linux boxes, and Windows for that matter. Even the dd transfers above seem CPU bound (close to 100 % of one core on my mini) for the default block sizes, it only drops down when you explicitly add the 'bs' option. This behaviour (larger blocksize = smaller CPU util.) is expected and the same across all os', but OS X is definitely the laggard. It's been this way "forever" - I still remember my Powerbook running 10.2 would be floored by a ping flood when connected at 100 Mb/s.

Witness the VMWare result: Linux running *inside* OS X is much faster than OS X at the same operation. OS X has (most/much of?) BSD's network stack, but a totally different kernel architecture. I'm guessing there's quite some optimisation that could be done...


I meant AFP yes. Works fine for me.

"Works fine" as in "I get 11 MB/s bulk transfer rates over 100Mb/s Ethernet"?

themacuser
18th November 2008, 01:18 AM
I'm guessing there's quite some optimisation that could be done...


This is probably why Apple's doing what they're doing with Snow Leopard.

TheCrusher
26th November 2008, 12:52 PM
Hi,

I've seen lots of reports, but we haven't had any problems (with performance that is).

I was just running some tests. My home directory is NFS mounted. I copied a 13 Meg file into my home directory in 1.2 seconds. And copied one out in similar time. My connection is 100 Mbit, so it doesn't get much better than that. My home directory is served by a NetApp.

I found a nice meaty 282 Meg file being served by a Linux box, and that copied in 26.6 seconds, which again, is the best you can hope for on 100 Mbit.

These are both automounted filesystems. The only change we've made in the NFS configuration on the mac side is the mount options used - we added resvport to make the Linux boxes happy.

The tests were done on a Mac Pro running 10.5.5. The connections seem to be TCP, based on tcpdump and netstat.

tom

SoVeReIgN
26th November 2008, 12:58 PM
How your mac mini on OS X go when you eliminate the airport as a possible problem?

af3556
7th December 2008, 11:22 PM
I found a nice meaty 282 Meg file being served by a Linux box, and that copied in 26.6 seconds, which again, is the best you can hope for on 100 Mbit.

Hmm. What Linux kernel are you using?. Agreed, at ~86 Mb/s, this is pretty much the upper limit on fastethernet. Sooo - why can't I get anywhere near that... Actually, I think I know - network I/O seems extremely resource intensive on OS X, and my machines are all pretty gutless (CPU and bus) compared to a Mac Pro. Any chance you could try your tests with a Mini or Macbook?


How your mac mini on OS X go when you eliminate the airport as a possible problem?

I've tested the Airport's throughput on the LAN ports - and whilst the ports aren't a simple switch (i.e. they aren't just three ports connected to a switch ASIC), they don't seem to be a bottle neck at 100 Mb/s - I can't measure any delay (e.g. using iperf (http://iperf.sourceforge.net/)). For kicks I patched the Macbook directly to the server (both at 1Gb/s), the default dd tests didn't change one iota, transfers were at ~1.5MB/s (yes, that's 1.5 Mb/s over a 1Gb/s link, or approx. 0.2% utilisation). CPU was at 100%. Using a much larger dd block size (bs=1M) brought xfer rates up to over 600 Mb/s (pretty close to the source disk bandwidth, unclear where the bottleneck is there) - so it's clearly an issue with the I/O rate rather than actual throughput.

(Incidentally, I also tested AFP transfers to a USB disk attached to the Airport - pretty damned ordinary results at around 5MB/s.)

TheCrusher
12th December 2008, 05:06 AM
Hmm. What Linux kernel are you using?. Agreed, at ~86 Mb/s, this is pretty much the upper limit on fastethernet. Sooo - why can't I get anywhere near that... Actually, I think I know - network I/O seems extremely resource intensive on OS X, and my machines are all pretty gutless (CPU and bus) compared to a Mac Pro. Any chance you could try your tests with a Mini or Macbook?


Fedora 8.

I just copied a 296136000 byte file in 27.93 seconds using a Mac Mini. It's a 2GHz Core 2
system with 2G of memory. Running 10.5.5 (9F33). Over NFS, connected via 100Mbit ethernet.

tom

af3556
13th December 2008, 10:27 PM
I just got through a series of tests using AFP between the Mini and Macbook, via an Airport Extreme (Gig ports all the way). I created 100, 200, 400 and 800 MB files on the Macbook and turned on file sharing. I tested reading each file from local disk, then over the network via AFP. Using dd to copy the files, I got pretty much disk-rate for each file (i.e. 16-22 MB/s). I then moved the Macbook to a 100 Mb/s port alongside the Linux server, and throughput dropped to about 9.7 MB/s which is pretty close to max for 100-meg ethernet. So all's good there.

I then repeated my tests with the share via NFS on the Macbook (enabled via NFS Manager (http://www.bresink.de/osx/NFSManager.html)). Throughput dropped to 1.6 MB/s with the default dd blocksize, CPU was quite high (50-80%). I used a 1MB blocksize ('bs=1m') and average transfer rates bumped back up to ~20 MBs/ on the Gig segment, and ~10 MB/s on the 100M, with CPU utilisation down to under 50%.

Note that these last tests gave better results than I saw with my initial post, so I repeated those (i.e. Mac Mini to Linux server, via NFS). The average rates with the default block size were still crap, ~1.6 MB/s but the 1MB block size jumped up to ~10 MB/s (cf. ~6 MB/s I saw previously).

I then used the Finder to copy the 100 and 200 MB files, Activity Monitor showed the network rate up around 10 MB/s.

As far as I know nothing has changed on the Mini and Linux server, other than the Mini having been rebooted a number of times (the server's been up for months), so I can't explain the difference between the previous tests and today's.

One thing is pretty clear: OS X's NFS throughput is quite variable with block size, which might explain why some apps perform better than others (e.g. iTunes often stumbles (SPOD) with my Library on the NFS server, which is what started me on this whole investigation in the first place...).

Thanks for the input, TheCrusher.

Ben

TheCrusher
16th December 2008, 05:54 AM
Hi,

I just checked (empirically, by watching reads in writes in nfsstat) and we're using 32K blocks which is the default for NFS3.

It seems likely you have some sort of network problem in the way. Are you sure you are negotiating full duplex? Half-duplex can really kill network performance. "/sbin/ifconfig en0" (or en1 or whatever) and look at the media line to see what you're getting.

tom