On Jun 08, 2010 at 08:43, Klaus Darilion <klaus.mailinglists(a)pernau.at> wrote: Not exactly, I plan to use it as a replacement for udp sockets (higher performance on send on linux) and in the future for handling icmp. The force_send_socket() stuff it's a good idea. Andrei

Am 08.06.2010 09:09, schrieb Andrei Pelinescu-Onciul: I just read about raw sockets (http://sock-raw.org/papers/sock_raw) and when sending on raw_sockets, there wont be any fragmentation. So that would need to be implemented in sr as well. Is the performance gain really worth the effort? I wonder why constructing of IP/UDP headers in sr is faster than having it done in the kernel. another interesting point is if the raw socket receives the packet before or after iptables filtering. regards klaus

On Thursday 10 June 2010, Andrei Pelinescu-Onciul wrote: Hello Andrei, the statistics looks promising, thanks. It would be indeed interesting to know how it performs with raw sockets then. I also looked a bit, aparently is also know for some other services like e.g. memcached: "We discovered that under load on Linux, UDP performance was downright horrible. This is caused by considerable lock contention on the UDP socket lock when transmitting through a single socket from multiple threads. Fixing the kernel by breaking up the lock is not easy. Instead, we used separate UDP sockets for transmitting replies (with one of these reply sockets per thread). With this change, we were able to deploy UDP without compromising performance on the backend." (http://www.facebook.com/note.php?note_id=39391378919) So it might be a good idea to evaluate how big is the actual improvement with raw sockets over the multiple sockets, if its make sense to from a maintenance POV to go with this solution (not sure how complicated the actual implementation will be..). Henning

Andrei Pelinescu-Onciul wrote: Hello Andrei, I might help with testing and providing some performance numbers. Unfortunately I only have access to some quad core Xeon servers, not 8 cores. If the weird UDP behavior appears also on 4 cores(less lock contention), I can run a test plan. My idea is to have several UDP workers (I plan to use 8- this will allow the kernel to distribute them evenly on all 4 CPUs - as the machine is idle). I plan to test using other 2-3 machines for sending SIP messages (using sipp or sipsak). Ser will just send some replies with some error code. I will measure the throughput over a 1 minute interval on the ser machine, with both current UDP sender and the new UDP over raw sock sender. If this is ok then in the afternoon I will bring some results. Any suggestion regarding the test plan is welcomed. Cheers Marius

Andrei Pelinescu-Onciul wrote: Hello, I want to test using some small sip replies(so I am not sure if fragmentation takes part). Of course the fragmentation code should also be tested for performance. Ser integration is another issue. Do you plan to use a global parameter to switch between normal sendto() function and raw sockets ? Marius

Andrei Pelinescu-Onciul wrote: Hello Andrei, Just performed a couple of tests (I was busy myself), but I think I have some interesting results. I have tested with 25 UAC/UAS's per test server, each pair generating 500 calls/s for a total of 12,500 calls/s . The test servers(running each 25 sipp as UAC and 25 sipp as UAS on different ports) where 2 quad core Xeon machines in the same LAN (Gigabit ethernet between them). Ser was doing a simple forward() based on the R-URI of the request, having 8 worker processes. 1. SER on a quad core Xeon, kernel 2.6.26. a. I have enable just one test server for a total of 12,500 calls/s. In this case the CPU usage was worse in case of UDP socks (udp_raw=0)(median value) "usr", "sys", "idl", "wai", "hiq", "siq" 13.584, 15.030, 50.713, 0.0, 2.950, 17.723 For RAW socks (udp_raw=1) these values showed up: "usr", "sys", "idl", "wai", "hiq", "siq" 10.396, 4.950, 76.238, 0.0, 2.970, 5.446 So the biggest difference is in software irq servicing time (last colum) and in sys. A little weird is the comparable usr CPU, I expected to be greater in raw sock mode. b. I enabled both testing machines for a total of 25,000 calls/s. In this case the CPU usage was almost identical, but mostly because the sipp instances couldn't send 500 reqs/s in the UDP mode .I limited sipp to send 20,000 calls per UAC/UAS pair. In the case of a raw sock it took an average of 55 s (closer to the 40s normal ideal value), but in udp mode it took almost 88s to send the 20,000 calls. The system load was the same (27% Idle). 2. SER on a Dual quad core Xeon, kernel 2.6.32 I have done only some basic runs but the results are not consistent with the ones on the other Ser machine. Siq time is the same, rate is steady at 500 calls/s but user CPU is greater in raw sock mode. I have dig around a bit and came over two interesting patches in 2.6.29 http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h… http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h… The release notes here: http://kernelnewbies.org/Linux_2_6_29#head-612c6b882f705935cc804d4af0b38316… As time allows me I will rerun some tests and provide some graphs if necessary. Marius

On Friday 11 June 2010, Andrei Pelinescu-Onciul wrote: Hi Andrei, thanks for the clarification. I completely forgot about the problem with the multiple source ports.. Just as a side note, maybe also interesting in this problem space are the new RFS/RPS implementations in upcoming 2.6.35: http://lwn.net/Articles/382428/ http://lwn.net/Articles/362339/ http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdi… http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdi… even if in the patches they mentioned mainly TCP based workloads, the gains looks promising. Regards, Henning