[SR-Users] Buffer size with xmlrpc module

Wed Apr 17 11:51:12 CEST 2013

Hello,

I think one solution can be to write the output in several tcp frames. 
Probably the xmlrpc module needs a bit of work.

Another option would be to expose a rpc function that returns chunks of 
location records.

Maybe others can jump in with other ideas, do some brainstorming and 
decide for a solution that can be applied to other cases, not only 
location records.

Cheers,
Daniel

On 4/16/13 11:27 AM, Jon Bergli Heier wrote:
> Hello,
>
> we use the xmlrpc module to fetch the list of users with the ul_dump MI command. When the number of users increased, this started to fail as the output size grew. Apparently this is because of too small write queue sizes (tcp_conn_wq_max). I discussed increasing the tcp_conn_wq_max setting with a coworker, who was reluctant to change it as it would affect SIP over TCP as well. We would probably also have to increase it if we hit the limit again at a later time. What we would like to know is if there's a way around this without having to fiddle with buffer sizes (or if this behavior is considered a bug).
>
> We originally used the mi_xmlrpc module, but changed to the xmlrpc module as the former made kamailio spawn zombie processes on xmlrpc requests (which is considered a bug I guess).
>
> FWIW, here's the log from a failed request, running kamailio 3.3.4:
>
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: DEBUG: <core> [ip_addr.c:247]: tcpconn_new: new tcp connection: [REMOVED]
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: DEBUG: <core> [tcp_main.c:1089]: tcpconn_new: on port 40261, type 2
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: DEBUG: <core> [tcp_main.c:1403]: tcpconn_add: hashes: 40:3802:373, 326
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: DEBUG: <core> [io_wait.h:390]: DBG: io_watch_add(0x828660, 94, 2, 0x7f2f87f31ca8), fd_no=85
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: DEBUG: <core> [io_wait.h:617]: DBG: io_watch_del (0x828660, 94, -1, 0x0) fd_no=86 called
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: DEBUG: <core> [tcp_main.c:4298]: tcp: DBG: sending to child, events 1
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: DEBUG: <core> [tcp_main.c:3969]: selected tcp worker 5 58(4960) for activity on [tcp:0.0.0.0:8080], 0x7f2f87f31ca8
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [tcp_read.c:1358]: received n=8 con=0x7f2f87f31ca8, fd=10
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [tcp_read.c:1168]: tcp_read_req: content-length= 153
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:624]: SIP Request:
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:626]:  method:  <POST>
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:628]:  uri:     </RPC2>
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:630]:  version: <HTTP/1.0>
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:202]: DEBUG: get_hdr_body : content_length=153
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:104]: found end of header
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [receive.c:149]: After parse_msg...
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: xmlrpc [xmlrpc.c:2273]: new fake xml msg created (355 bytes):#012<POST sip:127.0.0.1:9 HTTP/1.0#015#012Via: SIP/2.0/TCP [REMOVED]:40261#015#012Host: [REMOVED]:8080#015#012User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)#015#012Content-Type: text/xml#015#012Content-Length: 153#015#012#015#012<?xml version='1.0'?>#012<methodCall>#012<methodName>mi</methodName>#012<params>#012<param>#012<value><string>ul_dump</string></value>#012</param>#012</params>#012</methodCall>#012>
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:624]: SIP Request:
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:626]:  method:  <POST>
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:628]:  uri:     <sip:127.0.0.1:9>
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:630]:  version: <HTTP/1.0>
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/parse_via.c:2561]: end of header reached, state=5
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:511]: parse_headers: Via found, flags=2
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:513]: parse_headers: this is the first via
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:202]: DEBUG: get_hdr_body : content_length=153
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [parser/msg_parser.c:104]: found end of header
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [msg_translator.c:206]: check_via_address([REMOVED], [REMOVED], 0)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: sl [sl_funcs.c:499]: execute callback for event type 1
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: siptrace [siptrace.c:1343]: trace off...
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [tcp_main.c:2316]: tcp_send: send from reader (4960 (58)), reusing fd
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [tcp_main.c:2552]: tcp_send: sending...
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [tcp_main.c:2586]: tcp_send: after real write: c= 0x7f2f87f31ca8 n=15928 fd=10
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [tcp_main.c:2587]: tcp_send: buf=[REMOVED]
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: ERROR: <core> [tcp_main.c:708]: ERROR: wbufq_add(64634 bytes): write queue full or timeout  (0, total 0, last write 6648472 s ago)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: ERROR: sl [../../forward.h:171]: msg_send: ERROR: tcp_send failed
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: ERROR: xmlrpc [xmlrpc.c:804]: Error while sending reply
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: DEBUG: <core> [tcp_main.c:3620]: handle_ser_child: read response= 7f2f87f31ca8, -2, fd -1 from 58 (4960)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: ERROR: <core> [tcp_main.c:3634]: handle_ser_child: ERROR: received CON_ERROR for 0x7f2f87f31ca8 (id 7494), refcnt 3, flags 0x4018
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: xmlrpc [xmlrpc.c:2194]: xmlrpc: error while trying script
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [usr_avp.c:644]: DEBUG:destroy_avp_list: destroying list (nil)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [usr_avp.c:644]: DEBUG:destroy_avp_list: destroying list (nil)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [usr_avp.c:644]: DEBUG:destroy_avp_list: destroying list (nil)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [usr_avp.c:644]: DEBUG:destroy_avp_list: destroying list (nil)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [usr_avp.c:644]: DEBUG:destroy_avp_list: destroying list (nil)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [usr_avp.c:644]: DEBUG:destroy_avp_list: destroying list (nil)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [xavp.c:365]: destroying xavp list (nil)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [tcp_read.c:1285]: releasing con 0x7f2f87f31ca8, state -2, fd=10, id=7494
> Apr  1 23:38:24 server /usr/sbin/kamailio[4960]: DEBUG: <core> [tcp_read.c:1286]:  extra_data (nil)
> Apr  1 23:38:24 server /usr/sbin/kamailio[4982]: DEBUG: <core> [tcp_main.c:3381]: handle_tcp_child: reader response= 7f2f87f31ca8, -2 from 5
>
> Thanks,
> Jon
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, April 16-17, 2013, Berlin
  - http://conference.kamailio.com -