Re: [sr-dev] git:master:9c3ea838: dialog: re-entrant mutex for dialogs hash table slots

(gdb) bt #0 0x00007f7e98cfd8e9 in syscall () from /lib/libc.so.6 #1 0x00007f7e9359146a in ?? () from /usr/lib/kamailio/modules/dialog.so #2 0x00007f7e9359616d in ?? () from /usr/lib/kamailio/modules/dialog.so #3 0x00007f7e93596947 in get_dlg () from /usr/lib/kamailio/modules/dialog.so #4 0x00007f7e9358d8a2 in dlg_get_msg_dialog () from /usr/lib/kamailio/modules/dialog.so #5 0x00007f7e935ac77f in pv_get_dlg_variable () from /usr/lib/kamailio/modules/dialog.so #6 0x00000000004a046c in pv_get_spec_value (msg=0x7f7e98346208, sp=0x7f7e97fb30d0, value=0x7fffb0805630) at pvapi.c:1266 #7 0x00000000004d063a in rv_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rv=0x7f7e97fb30c8, cache=0x0) at rvalue.c:1859 #8 0x00000000004d06ed in int_rve_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb30c0) at rvalue.c:1895 #9 0x00000000004d14e5 in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb3fe0) at rvalue.c:2117 #10 0x00000000004d0adc in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805dd8, rve=0x7f7e97fb4e20) at rvalue.c:1960 #11 0x000000000041c912 in do_action (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1075 #12 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1599 #13 0x000000000041a9cc in do_action (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:715 #14 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1599 #15 0x00000000004261fb in run_actions_safe (h=0x7fffb0807520, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1664 #16 0x00000000004cd997 in rval_get_int (h=0x7fffb0807520, msg=0x7f7e98346208, i=0x7fffb0806978, rv=0x7f7e97d20500, cache=0x0) at rvalue.c:924 #17 0x00000000004d078b in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806978, rve=0x7f7e97d204f8) at rvalue.c:1918 #18 0x00000000004d09a2 in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806de8, rve=0x7f7e97d20c18) at rvalue.c:1926 #19 0x000000000041c912 in do_action (h=0x7fffb0807520, a=0x7f7e97d23890, msg=0x7f7e98346208) at action.c:1075 #20 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97d20198, msg=0x7f7e98346208) at action.c:1599 #21 0x000000000041cb82 in do_action (h=0x7fffb0807520, a=0x7f7e97dce4a8, msg=0x7f7e98346208) at action.c:1090 #22 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97cb6bb8, msg=0x7f7e98346208) at action.c:1599 #23 0x00000000004262c3 in run_top_route (a=0x7f7e97cb6bb8, msg=0x7f7e98346208, c=0x0) at action.c:1685 #24 0x00000000004ac5d5 in receive_msg ( buf=0x937360 "INVITE sip:49789654123@XXX.XXX.XXX.XXX:5060 SIP/2.0\r\nRecord-Route: <sip:XXX.XXX.XXX.XXX;lr=on>\r\nVia: SIP/2.0/UDP XXX.XXX.XXX.XXX;branch=z9hG4bK8081.1d9a97b13c3c42328b572ca0ddd297aa."..., len=1542, rcv_info=0x7fffb08077e0) at receive.c:212 #25 0x000000000054808d in udp_rcv_loop () at udp_server.c:536 #26 0x0000000000472f6d in main_loop () at main.c:1620 #27 0x000000000047692e in main (argc=13, argv=0x7fffb0807b28) at main.c:2566

(gdb) bt #0 0x00007f7e98cfd8e9 in syscall () from /lib/libc.so.6 #1 0x00007f7e9359146a in ?? () from /usr/lib/kamailio/modules/dialog.so #2 0x00007f7e9359616d in ?? () from /usr/lib/kamailio/modules/dialog.so #3 0x00007f7e93596947 in get_dlg () from /usr/lib/kamailio/modules/dialog.so #4 0x00007f7e9358d8a2 in dlg_get_msg_dialog () from /usr/lib/kamailio/modules/dialog.so #5 0x00007f7e935ac77f in pv_get_dlg_variable () from /usr/lib/kamailio/modules/dialog.so #6 0x00000000004a046c in pv_get_spec_value (msg=0x7f7e98346208, sp=0x7f7e97fb30d0, value=0x7fffb0805630) at pvapi.c:1266 #7 0x00000000004d063a in rv_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rv=0x7f7e97fb30c8, cache=0x0) at rvalue.c:1859 #8 0x00000000004d06ed in int_rve_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb30c0) at rvalue.c:1895 #9 0x00000000004d14e5 in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb3fe0) at rvalue.c:2117 #10 0x00000000004d0adc in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805dd8, rve=0x7f7e97fb4e20) at rvalue.c:1960 #11 0x000000000041c912 in do_action (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1075 #12 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1599 #13 0x000000000041a9cc in do_action (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:715 #14 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1599 #15 0x00000000004261fb in run_actions_safe (h=0x7fffb0807520, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1664 #16 0x00000000004cd997 in rval_get_int (h=0x7fffb0807520, msg=0x7f7e98346208, i=0x7fffb0806978, rv=0x7f7e97d20500, cache=0x0) at rvalue.c:924 #17 0x00000000004d078b in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806978, rve=0x7f7e97d204f8) at rvalue.c:1918 #18 0x00000000004d09a2 in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806de8, rve=0x7f7e97d20c18) at rvalue.c:1926 #19 0x000000000041c912 in do_action (h=0x7fffb0807520, a=0x7f7e97d23890, msg=0x7f7e98346208) at action.c:1075 #20 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97d20198, msg=0x7f7e98346208) at action.c:1599 #21 0x000000000041cb82 in do_action (h=0x7fffb0807520, a=0x7f7e97dce4a8, msg=0x7f7e98346208) at action.c:1090 #22 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97cb6bb8, msg=0x7f7e98346208) at action.c:1599 #23 0x00000000004262c3 in run_top_route (a=0x7f7e97cb6bb8, msg=0x7f7e98346208, c=0x0) at action.c:1685 #24 0x00000000004ac5d5 in receive_msg ( buf=0x937360 "INVITE sip:49321654987@XXX.XXX.XXX.XXX:5060 SIP/2.0\r\nRecord-Route: <sip:XXX.XXX.XXX.XXX;lr=on>\r\nVia: SIP/2.0/UDP XXX.XXX.XXX.XXX;branch=z9hG4bKa2d3.8d04ff9e19f9b1e4e9cbcb9bbae4cf41"..., len=1555, rcv_info=0x7fffb08077e0) at receive.c:212 #25 0x000000000054808d in udp_rcv_loop () at udp_server.c:536 #26 0x0000000000472f6d in main_loop () at main.c:1620 #27 0x000000000047692e in main (argc=13, argv=0x7fffb0807b28) at main.c:2566

(gdb) bt #0 0x00007f7e98cfd8e9 in syscall () from /lib/libc.so.6 #1 0x00007f7e9359146a in ?? () from /usr/lib/kamailio/modules/dialog.so #2 0x00007f7e9359616d in ?? () from /usr/lib/kamailio/modules/dialog.so #3 0x00007f7e93596767 in get_dlg () from /usr/lib/kamailio/modules/dialog.so #4 0x00007f7e9358d8a2 in dlg_get_msg_dialog () from /usr/lib/kamailio/modules/dialog.so #5 0x00007f7e935ac77f in pv_get_dlg_variable () from /usr/lib/kamailio/modules/dialog.so #6 0x00000000004a046c in pv_get_spec_value (msg=0x7f7e98346208, sp=0x7f7e97fb30d0, value=0x7fffb0805630) at pvapi.c:1266 #7 0x00000000004d063a in rv_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rv=0x7f7e97fb30c8, cache=0x0) at rvalue.c:1859 #8 0x00000000004d06ed in int_rve_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb30c0) at rvalue.c:1895 #9 0x00000000004d14e5 in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb3fe0) at rvalue.c:2117 #10 0x00000000004d0adc in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805dd8, rve=0x7f7e97fb4e20) at rvalue.c:1960 #11 0x000000000041c912 in do_action (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1075 #12 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1599 #13 0x000000000041a9cc in do_action (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:715 #14 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1599 #15 0x00000000004261fb in run_actions_safe (h=0x7fffb0807520, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1664 #16 0x00000000004cd997 in rval_get_int (h=0x7fffb0807520, msg=0x7f7e98346208, i=0x7fffb0806978, rv=0x7f7e97d20500, cache=0x0) at rvalue.c:924 #17 0x00000000004d078b in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806978, rve=0x7f7e97d204f8) at rvalue.c:1918 #18 0x00000000004d09a2 in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806de8, rve=0x7f7e97d20c18) at rvalue.c:1926 #19 0x000000000041c912 in do_action (h=0x7fffb0807520, a=0x7f7e97d23890, msg=0x7f7e98346208) at action.c:1075 #20 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97d20198, msg=0x7f7e98346208) at action.c:1599 #21 0x000000000041cb82 in do_action (h=0x7fffb0807520, a=0x7f7e97dce4a8, msg=0x7f7e98346208) at action.c:1090 #22 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97cb6bb8, msg=0x7f7e98346208) at action.c:1599 #23 0x00000000004262c3 in run_top_route (a=0x7f7e97cb6bb8, msg=0x7f7e98346208, c=0x0) at action.c:1685 #24 0x00000000004ac5d5 in receive_msg ( buf=0x937360 "INVITE sip:491234567@XXX.XXX.XXX.XXX:5060 SIP/2.0\r\nRecord-Route: <sip:XXX.XXX.XXX.XXX;lr=on>\r\nVia: SIP/2.0/UDP XXX.XXX.XXX.XXX;branch=z9hG4bK41dd.793c6bda464296fb439170efecc7c405."..., len=1542, rcv_info=0x7fffb08077e0) at receive.c:212 #25 0x000000000054808d in udp_rcv_loop () at udp_server.c:536 #26 0x0000000000472f6d in main_loop () at main.c:1620 #27 0x000000000047692e in main (argc=13, argv=0x7fffb0807b28) at main.c:2566

Hello, are you using cnxcc or other module on top of dialog? The reason was cnxcc, as it apparently does some lower level dialog ops that involve locking in case of a dialog being already locked. Have you got the backtrace in case of the lock you encountered to see what situations ended there? Cheers, Daniel On 07/08/15 13:14, Pawel Kuzak wrote: > Hi Daniel, > > We are testing Kamailio 4.1.8 and we are running into a deadlock in > the dialog module. I therefore searched the git commits if something > has been done to the locking in dialog module and I found this commit. > I wanted to ask you if there was a specific motivation for you to > change this. I mean in the previous locking (e.g. like it is still in > 4.1.8), do you see potential scenarios which could lead to a deadlock? > Obviously there is one, but at a first glance checking the code > acquiring and releasing locks looks fine. > Of course we will now give 4.2 a try. I am just curious, that's why I > am asking :-) . > > Regards, > Paul > > On 16.06.2015 15:18, Daniel-Constantin Mierla wrote: >> Module: kamailio >> Branch: master >> Commit: 9c3ea838b31039ac067e17d519df67b64b0dada1 >> URL: >> https://github.com/kamailio/kamailio/commit/9c3ea838b31039ac067e17d519df67b… >> >> Author: Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; >> Committer: Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; >> Date: 2015-06-16T15:18:27+02:00 >> >> dialog: re-entrant mutex for dialogs hash table slots >> >> - changed from a lock set usage to per slot lock field >> >> --- >> >> Modified: modules/dialog/dlg_hash.c >> Modified: modules/dialog/dlg_hash.h >> >> --- >> >> Diff: >> https://github.com/kamailio/kamailio/commit/9c3ea838b31039ac067e17d519df67b… >> Patch: >> https://github.com/kamailio/kamailio/commit/9c3ea838b31039ac067e17d519df67b… >> >> >> _______________________________________________ >> sr-dev mailing list >> sr-dev(a)lists.sip-router.org >> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev > > _______________________________________________ > sr-dev mailing list > sr-dev(a)lists.sip-router.org > http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev