Performance factors (and how to debug low throughput)

[paulie-g]

I'm seeing ~8 MB/s throughput on a large file copy on 1 Gbps connection with a 6ms latency (with threads anywhere from 8 to 32), server is diod and client has it mounted via diodmount on v9fs (msize default all the way up to msize=1048576). scp is at ~35 MB/s, wget is 90 MB/s (all with no performance options). I've got what I think are reasonable tunables for tcp on either end. ss -tmi on client is showing a strange discrepancy between http and 9p via v9fs:

for wget 'http://box/test.dat':

ss -tmi

skmem:(r0,rb84751494,t4,tb46080,f0,w0,o0,bl0,d6) cubic wscale:14,14 rto:206.666 rtt:5.477/2.296 ato:40 mss:1388 pmtu:1500 rcvmss:1388 advmss:1448 cwnd:10 bytes_sent:127 bytes_acked:128 bytes_received:327466024 segs_out:39483 segs_in:236042 data_segs_out:1 data_segs_in:236040 send 20.3Mbps lastsnd:3940 pacing_rate 40.5Mbps delivery_rate 1.7Mbps delivered:2 app_limited busy:6ms rcv_rtt:7.672 rcv_space:3994532 rcv_ssthresh:60584075 minrtt:5.329 rcv_ooopack:7345 snd_wnd:32768 rcv_wnd:60588032

for cp /run/media/user/p9/test.dat /dev/null:

ss -tmi

skmem:(r0,rb881008,t0,tb46080,f3177,w919,o0,bl0,d535) cubic wscale:14,14 rto:206.666 rtt:6.35/0.219 ato:40 mss:1388 pmtu:1500 rcvmss:1388 advmss:1448 cwnd:10 bytes_sent:590149 bytes_retrans:11 bytes_acked:590116 bytes_received:1365512910 segs_out:168312 segs_in:1005326 data_segs_out:25687 data_segs_in:1004776 send 17.5Mbps pacing_rate 35Mbps delivery_rate 6.78Mbps delivered:25687 app_limited busy:168013ms unacked:1 retrans:0/1 dsack_dups:1 rcv_rtt:6.414 rcv_space:131153 rcv_ssthresh:615390 minrtt:5.494 snd_wnd:65536 rcv_wnd:622592

for diodcat with default msize:

ss -tmi

skmem:(r0,rb1253270,t0,tb46080,f3177,w919,o0,bl0,d0) cubic wscale:14,14 rto:206.666 rtt:6.524/0.633 ato:40 mss:1388 pmtu:1500 rcvmss:1388 advmss:1448 cwnd:10 bytes_sent:20978 bytes_acked:20956 bytes_received:58643268 segs_out:7206 segs_in:42968 data_segs_out:902 data_segs_in:42966 send 17Mbps lastsnd:4 lastrcv:4 lastack:4 pacing_rate 34Mbps delivery_rate 1.89Mbps delivered:902 app_limited busy:6079ms unacked:1 rcv_rtt:6.588 rcv_space:133822 rcv_ssthresh:875593 minrtt:5.867 snd_wnd:65536 rcv_wnd:884736

(9 MB/s)

for diodcat with -m 1048576

ss -tmi

skmem:(r0,rb805413,t0,tb46080,f3177,w919,o0,bl0,d0) cubic wscale:14,14 rto:206.666 rtt:6.504/0.275 ato:40 mss:1388 pmtu:1500 rcvmss:1388 advmss:1448 cwnd:10 bytes_sent:48417 bytes_acked:48395 bytes_received:136812207 segs_out:12623 segs_in:100233 data_segs_out:2095 data_segs_in:100230 send 17.1Mbps lastsnd:7 lastrcv:7 lastack:7 pacing_rate 34.1Mbps delivery_rate 2.31Mbps delivered:2095 app_limited busy:13455ms unacked:1 rcv_rtt:6.325 rcv_space:142150 rcv_ssthresh:562443 minrtt:4.815 snd_wnd:65536 rcv_wnd:573440

(just above 9 MB/s)

This throughput is well below even what was reported in #59 . I'm presuming this is atypical, do you have any pointers on where to look?
(Linux 6.12.8 PREEMPT_DYNAMIC on client, diod latest release converted from a debian .deb on server)

[paulie-g]

on server:

ss -tmi

skmem:(r0,rb4794816,t0,tb225792,f2877,w62659,o0,bl0,d0) cubic wscale:14,14 rto:210 rtt:7.092/0.688 ato:40 mss:1388 pmtu:1500 rcvmss:536 advmss:1448 cwnd:106 ssthresh:106 bytes_sent:248857398 bytes_retrans:861 bytes_acked:248796566 bytes_received:87724 segs_out:182317 segs_in:23224 data_segs_out:182313 data_segs_in:3804 send 166Mbps lastsnd:7 lastrcv:7 pacing_rate 199Mbps delivery_rate 78.4Mbps delivered:182270 app_limited busy:33963ms unacked:44 retrans:0/3 dsack_dups:3 reordering:48 rcv_rtt:26600.9 rcv_space:65184 rcv_ssthresh:65100 minrtt:6.29 snd_wnd:802816 rcv_wnd:65536

[garlick]

I haven't looked at this in so long that I don't really have any suggestions for you off the top of my head other tuning the msize and looking at tcp tunables. Of course with no caching, 9p doesn't perform well with general metadata focused workloads, but for bulk I/O where caching isn't expected, it shouldn't be too bad.

[paulie-g]

Of course with no caching, 9p doesn't perform well with general metadata focused workloads

I verified this with no caching and cache=fscache, as an fyi, although I would expect caching to matter for metadata vfs ops and seeks, not sequential reads if they're bottlenecked (otherwise readahead obviously). Maybe you or someone has a set of network stack tunables to share from a known-good setup? I've got to assume people have this running at much higher throughput by default or there'd be wailing and gnashing of teeth. There are a lot of tunables, so it would be quite tricky to bisect as the combinatorial space is pretty large.

[garlick]

Sorry, I've got nothing specific at this point. If I were seriously investigating I might start with a protocol trace to see what the banter between client and server looks like, I/O sizes, timings, etc.

I've got to assume people have this running at much higher throughput by default or there'd be wailing and gnashing of teeth.

Possibly people are joyful because they are using something else :-)

[paulie-g]

Sorry, I've got nothing specific at this point. If I were seriously investigating I might start with a protocol trace to see what the banter between client and server looks like, I/O sizes, timings, etc.

I'm assuming you'd want that on the diod server side? What's the best way to generate this, running with -d 1 or something else? I'm curious enough to possibly put a bit of time into it and I certainly don't expect someone else to debug it for me, but I'd prefer to short-circuit just cold reading the code.

I've got to assume people have this running at much higher throughput by default or there'd be wailing and gnashing of teeth.

Possibly people are joyful because they are using something else :-)

There's really not a lot of good alternatives, CIFS and NFS are far too chatty for WAN (and overkill for a lot of applications like exposing a media storage volume), sshfs performed abominably for me (p9/diod, despite throughput problems, is very responsive when seeking/scrubbing).

[paulie-g]

Incidentally, is v9fs via diodmount the only option on the client or is there a way to get a userspace client (from one of the plan9 projects) working?

[garlick]

Incidentally, is v9fs via diodmount the only option on the client or is there a way to get a userspace client (from one of the plan9 projects) working?

diod only speaks 9P2000.L, a dialect peculiar to the Linux kernel, so it's somewhat isolated in the ecosystem, unfortunately.

What's the best way to generate this, running with -d 1

Yes. Also in a pinch I think wireshark decodes 9P

Rereading the issue description, you mentioned diodcat but not a test run through the kernel client? diodcat is a standalone, single-threaded client that waits for Rread before sending the next Tread. The high latency (6ms) you mentioned would not help in that case.

I think you must've run a test through the kernel client but I'm not seeing results? I would expect the kernel client to lob concurrent Tread requests at the server and hide the network latency for a big transfer. Apologies if I'm misunderstanding what you wrote.

[paulie-g]

you mentioned diodcat but not a test run through the kernel client

The cp /run/media/user/p9/test.dat /dev/null case was through the kernelspace v9fs (mounted with diodmount). Note that this was just one example for the issue, I saw similar performance with pv for example, so it's typical for what happens when the access is through vfs ops via syscalls. All the ss -tmi output is from the client except as noted.

I think I'll try to get the master to build, try that and ping this issue with the results. Unless you think there are no possibly material changes in master since the least release.

[garlick]

Yeah, please do that if possible. It's been like 10 years since the last tag and there have been at least a few fixes to keep up with v9fs development.

p.s. If you need a deb to turn into your package, there is now a make deb target that I've been using for testing.

[paulie-g]

With a build from master and after swapoff/swapon on the server VM, sequential read throughput is now at ~35 MB/s which is in line with what other people have reported. This is good enough for me. In hindsight, I consider it more likely than not that this perf issue was an aberrant edge (edit: pathological was the word I was looking for) case, so not particularly interesting for other users either. Probably something to do with memory pressure. If you're agreed, I think this can be closed.

Edit: also, I passed in -omsize=1048576 to the diod server (not documented that this is possible), might have something to do with the improvement.

p.s: I saw that, but I think I've fixed the git master build, manual and diod-git from AUR pointed at my fork with the fix (and it wouldn't have worked anyway, I don't think - the value of converting an existing .deb was that it sidestepped the autogen issues).

[garlick]

Edit: also, I passed in -omsize=1048576 to the diod server (not documented that this is possible), might have something to do with the improvement.

I'm not sure that has any effect? Maybe a bug that the server didn't throw an error here.

Ah the server hard coded msize was raised to 1M in 2021 by 0196c0f which might explain why master did better than the ancient tag.

[paulie-g]

Ah the server hard coded msize was raised to 1M in 2021 by 0196c0f which might explain why master did better than the ancient tag.

I did see that PR discussion, didn't realise it was hard-coded. FYI, diodmount still has a DIOD_DEFAULT_MSIZE=65535 define. Is the msize a trade-off between throughput and interactive performance or is a larger one strictly better (modulo memory consumption)?

Half the problem I've had is just the lack of basic docs (or them being wildly incorrect). I'd volunteer to fix that, but it requires knowing things.