Core crash in del_lump() "offset exceeds message size" when using subst() inside tm:local-request route.
28 Jul
2017
28 Jul
'17
5:53 p.m.
Hello.
Inside the tm:local-request route, I am trying to modify the body of outgoing NOTIFY
messages using the subst_body() function.
Things seem to work fine most of the time. The NOTIFY messages are correctly modified.
However, I will very rarely hit a crash.
I can force the crash to happen more quickly by stress testing with thousands of NOTIFY
messages being modified.
Is it safe to call subst() or subst_body() inside the tm:local-request route? If not, is
there another way I can modify the outgoing NOTIFY messages safely?
Note that I am also calling append_hf() inside the same tm:local-request route, but that
has been working for a very long time without causing any problems. Could there be some
kind of bad interaction between append_hf() and subst_body()?
Strangely, it seems like using subst() makes the crash more likely to happen. For some
reason, subst_body() seems to be more robust.
Here are the logs I see:
---------
Jul 28 10:57:40 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2476]: CRITICAL: <core>
[data_lump.c:292]: del_lump(): offset exceeds message size (1625 > 1194) aborting...
Jul 28 10:57:43 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2481]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 53
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:731]: handle_sigs(): child process 2476 exited by a signal 6
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:734]: handle_sigs(): core was generated
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: INFO: <core>
[main.c:756]: handle_sigs(): terminating due to SIGCHLD
---------
Here is a GDB back trace from the core dump:
---------
#0 0x00007fd91a567c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd91a56b028 in __GI_abort () at abort.c:89
#2 0x000000000044e981 in del_lump (msg=0x7fd917c5ed60 <lreq>, offset=1625, len=20,
type=HDR_OTHER_T) at data_lump.c:293
#3 0x00007fd91676297f in subst_f (msg=0x7fd917c5ed60 <lreq>, subst=0x7fd919e5ccc0
"\320\356\345\031\331\177", ignored=0x0) at textops.c:681
#4 0x000000000041e1dd in do_action (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1059
#5 0x000000000042a917 in run_actions (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#6 0x000000000042af94 in run_actions_safe (h=0x7ffd364ba5b0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1613
#7 0x0000000000549972 in rval_get_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60 <lreq>,
i=0x7ffd364b8b58, rv=0x7fd919e9ced8, cache=0x0) at rvalue.c:912
#8 0x000000000054dc7c in rval_expr_eval_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60
<lreq>, res=0x7ffd364b8b58, rve=0x7fd919e9ced0) at rvalue.c:1910
#9 0x000000000041dc52 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1029
#10 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#11 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9dad8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#12 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e973b8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#13 0x000000000041ab46 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95d70,
msg=0x7fd917c5ed60 <lreq>) at action.c:677
#14 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8f330,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#15 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95eb8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#16 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8c8e0,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#17 0x000000000042b07d in run_top_route (a=0x7fd919e8c8e0, msg=0x7fd917c5ed60
<lreq>, c=0x0) at action.c:1634
#18 0x00007fd917a0539a in t_uac_prepare (uac_r=0x7ffd364baf20, dst_req=0x7ffd364ba8f0,
dst_cell=0x7ffd364ba8f8) at uac.c:391
#19 0x00007fd917a07945 in t_uac_with_ids (uac_r=0x7ffd364baf20, ret_index=0x0,
ret_label=0x0) at uac.c:599
#20 0x00007fd917a07918 in t_uac (uac_r=0x7ffd364baf20) at uac.c:584
#21 0x00007fd917a09b9e in req_within (uac_r=0x7ffd364baf20) at uac.c:802
#22 0x00007fd914674052 in send_notify_request (subs=0x7ffd364bb200, watcher_subs=0x0,
n_body=0x0, force_null_body=0) at notify.c:1600
#23 0x00007fd9146754b3 in notify (subs=0x7ffd364bb200, watcher_subs=0x0, n_body=0x0,
force_null_body=0) at notify.c:1690
#24 0x00007fd9146cec6a in update_subscription (msg=0x7fd919efeb70, subs=0x7ffd364bb200,
to_tag_gen=1, sent_reply=0x7ffd364bb16c) at subscribe.c:697
#25 0x00007fd9146d45b7 in handle_subscribe (msg=0x7fd919efeb70, watcher_user=...,
watcher_domain=...) at subscribe.c:1057
#26 0x00007fd9146d08b0 in handle_subscribe0 (msg=0x7fd919efeb70) at subscribe.c:816
#27 0x000000000041e155 in do_action (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1053
#28 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1548
#29 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19650,
msg=0x7fd919efeb70) at action.c:1048
#30 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e02638,
msg=0x7fd919efeb70) at action.c:1548
#31 0x000000000041e0c4 in do_action (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1044
#32 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1548
#33 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19b08,
msg=0x7fd919efeb70) at action.c:1048
#34 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919df99d8,
msg=0x7fd919efeb70) at action.c:1548
#35 0x000000000041ab46 in do_action (h=0x7ffd364bd680, a=0x7fd919dbe960,
msg=0x7fd919efeb70) at action.c:677
#36 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919daedf8,
msg=0x7fd919efeb70) at action.c:1548
#37 0x000000000042b07d in run_top_route (a=0x7fd919daedf8, msg=0x7fd919efeb70, c=0x0) at
action.c:1634
#38 0x000000000050f678 in receive_msg (buf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept: application/dialog-info+xml,
application/pidf+xml, application/pidf-diff+xml,"..., len=1284,
rcv_info=0x7fd8f2c6cb50) at receive.c:196
#39 0x00000000005fb99e in receive_tcp_msg (tcpbuf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept: application/dialog-info+xml,
application/pidf+xml, application/pidf-diff+xml,"..., len=1284,
rcv_info=0x7fd8f2c6cb50, con=0x7fd8f2c6cb38) at tcp_read.c:1207
#40 0x00000000005fd48c in tcp_read_req (con=0x7fd8f2c6cb38, bytes_read=0x7ffd364bdb7c,
read_flags=0x7ffd364bdb84) at tcp_read.c:1411
#41 0x0000000000600aee in handle_io (fm=0x7fd919f07d80, events=1, idx=-1) at
tcp_read.c:1643
#42 0x00000000005f3ab8 in io_wait_loop_epoll (h=0xa3d3e0 <io_w>, t=2, repeat=0) at
io_wait.h:1061
#43 0x0000000000601cf7 in tcp_receive_loop (unix_sock=57) at tcp_read.c:1755
#44 0x00000000005ea235 in tcp_init_children () at tcp_main.c:4788
#45 0x00000000004ab048 in main_loop () at main.c:1675
#46 0x00000000004b10c2 in main (argc=13, argv=0x7ffd364be258) at main.c:2566
---------
Thanks very much.
-Cody
31 Jul
31 Jul
2:11 a.m.
Hello,
what's the version of Kamailio and the operating system you run on?
Cheers,
Daniel
On 28.07.17 23:53, Cody Herzog wrote:
Hello.
Inside the tm:local-request route, I am trying to modify the body of
outgoing NOTIFY messages using the subst_body() function.
Things seem to work fine most of the time. The NOTIFY messages are
correctly modified. However, I will very rarely hit a crash.
I can force the crash to happen more quickly by stress testing with
thousands of NOTIFY messages being modified.
Is it safe to call subst() or subst_body() inside the tm:local-request
route? If not, is there another way I can modify the outgoing NOTIFY
messages safely?
Note that I am also calling append_hf() inside the same
tm:local-request route, but that has been working for a very long time
without causing any problems. Could there be some kind of bad
interaction between append_hf() and subst_body()?
Strangely, it seems like using subst() makes the crash more likely to
happen. For some reason, subst_body() seems to be more robust.
Here are the logs I see:
---------
Jul 28 10:57:40 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2476]:
CRITICAL: <core> [data_lump.c:292]: del_lump(): offset exceeds message
size (1625 > 1194) aborting...
Jul 28 10:57:43 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2481]:
CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 53
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]:
ALERT: <core> [main.c:731]: handle_sigs(): child process 2476 exited
by a signal 6
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]:
ALERT: <core> [main.c:734]: handle_sigs(): core was generated
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]:
INFO: <core> [main.c:756]: handle_sigs(): terminating due to SIGCHLD
---------
Here is a GDB back trace from the core dump:
---------
#0 0x00007fd91a567c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd91a56b028 in __GI_abort () at abort.c:89
#2 0x000000000044e981 in del_lump (msg=0x7fd917c5ed60 <lreq>,
offset=1625, len=20, type=HDR_OTHER_T) at data_lump.c:293
#3 0x00007fd91676297f in subst_f (msg=0x7fd917c5ed60 <lreq>,
subst=0x7fd919e5ccc0 "\320\356\345\031\331\177", ignored=0x0) at
textops.c:681
#4 0x000000000041e1dd in do_action (h=0x7ffd364b87c0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1059
#5 0x000000000042a917 in run_actions (h=0x7ffd364b87c0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#6 0x000000000042af94 in run_actions_safe (h=0x7ffd364ba5b0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1613
#7 0x0000000000549972 in rval_get_int (h=0x7ffd364ba5b0,
msg=0x7fd917c5ed60 <lreq>, i=0x7ffd364b8b58, rv=0x7fd919e9ced8,
cache=0x0) at rvalue.c:912
#8 0x000000000054dc7c in rval_expr_eval_int (h=0x7ffd364ba5b0,
msg=0x7fd917c5ed60 <lreq>, res=0x7ffd364b8b58, rve=0x7fd919e9ced0) at
rvalue.c:1910
#9 0x000000000041dc52 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e9d990, msg=0x7fd917c5ed60 <lreq>) at action.c:1029
#10 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e9d990, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#11 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e9dad8, msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#12 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e973b8, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#13 0x000000000041ab46 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e95d70, msg=0x7fd917c5ed60 <lreq>) at action.c:677
#14 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e8f330, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#15 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e95eb8, msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#16 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e8c8e0, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#17 0x000000000042b07d in run_top_route (a=0x7fd919e8c8e0,
msg=0x7fd917c5ed60 <lreq>, c=0x0) at action.c:1634
#18 0x00007fd917a0539a in t_uac_prepare (uac_r=0x7ffd364baf20,
dst_req=0x7ffd364ba8f0, dst_cell=0x7ffd364ba8f8) at uac.c:391
#19 0x00007fd917a07945 in t_uac_with_ids (uac_r=0x7ffd364baf20,
ret_index=0x0, ret_label=0x0) at uac.c:599
#20 0x00007fd917a07918 in t_uac (uac_r=0x7ffd364baf20) at uac.c:584
#21 0x00007fd917a09b9e in req_within (uac_r=0x7ffd364baf20) at uac.c:802
#22 0x00007fd914674052 in send_notify_request (subs=0x7ffd364bb200,
watcher_subs=0x0, n_body=0x0, force_null_body=0) at notify.c:1600
#23 0x00007fd9146754b3 in notify (subs=0x7ffd364bb200,
watcher_subs=0x0, n_body=0x0, force_null_body=0) at notify.c:1690
#24 0x00007fd9146cec6a in update_subscription (msg=0x7fd919efeb70,
subs=0x7ffd364bb200, to_tag_gen=1, sent_reply=0x7ffd364bb16c) at
subscribe.c:697
#25 0x00007fd9146d45b7 in handle_subscribe (msg=0x7fd919efeb70,
watcher_user=..., watcher_domain=...) at subscribe.c:1057
#26 0x00007fd9146d08b0 in handle_subscribe0 (msg=0x7fd919efeb70) at
subscribe.c:816
#27 0x000000000041e155 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19508, msg=0x7fd919efeb70) at action.c:1053
#28 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e19508, msg=0x7fd919efeb70) at action.c:1548
#29 0x000000000041e112 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19650, msg=0x7fd919efeb70) at action.c:1048
#30 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e02638, msg=0x7fd919efeb70) at action.c:1548
#31 0x000000000041e0c4 in do_action (h=0x7ffd364bd680,
a=0x7fd919e199c0, msg=0x7fd919efeb70) at action.c:1044
#32 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e199c0, msg=0x7fd919efeb70) at action.c:1548
#33 0x000000000041e112 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19b08, msg=0x7fd919efeb70) at action.c:1048
#34 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919df99d8, msg=0x7fd919efeb70) at action.c:1548
#35 0x000000000041ab46 in do_action (h=0x7ffd364bd680,
a=0x7fd919dbe960, msg=0x7fd919efeb70) at action.c:677
#36 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919daedf8, msg=0x7fd919efeb70) at action.c:1548
#37 0x000000000042b07d in run_top_route (a=0x7fd919daedf8,
msg=0x7fd919efeb70, c=0x0) at action.c:1634
#38 0x000000000050f678 in receive_msg (buf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml,
application/pidf-diff+xml,"..., len=1284, rcv_info=0x7fd8f2c6cb50) at
receive.c:196
#39 0x00000000005fb99e in receive_tcp_msg (tcpbuf=0x7fd8f2c6ce28
"SUBSCRIBE sip:endpoint-1720@intouchstaging.com
SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml,
application/pidf-diff+xml,"..., len=1284, rcv_info=0x7fd8f2c6cb50,
con=0x7fd8f2c6cb38) at tcp_read.c:1207
#40 0x00000000005fd48c in tcp_read_req (con=0x7fd8f2c6cb38,
bytes_read=0x7ffd364bdb7c, read_flags=0x7ffd364bdb84) at tcp_read.c:1411
#41 0x0000000000600aee in handle_io (fm=0x7fd919f07d80, events=1,
idx=-1) at tcp_read.c:1643
#42 0x00000000005f3ab8 in io_wait_loop_epoll (h=0xa3d3e0 <io_w>, t=2,
repeat=0) at io_wait.h:1061
#43 0x0000000000601cf7 in tcp_receive_loop (unix_sock=57) at
tcp_read.c:1755
#44 0x00000000005ea235 in tcp_init_children () at tcp_main.c:4788
#45 0x00000000004ab048 in main_loop () at main.c:1675
#46 0x00000000004b10c2 in main (argc=13, argv=0x7ffd364be258) at
main.c:2566
---------
Thanks very much.
-Cody
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com
1:43 p.m.
Hello.
We are running Kamailio 4.3.5 on Ubuntu 14.04 LTS 64-bit.
Sorry, I should have included that information in my first email.
I have done more debugging, and I believe the problem is caused by the subst_run()
function being used on a message buffer which is not null terminated at msg->len.
The subst_f() and subst_body_f() functions both call subst_run(), and most of the time,
everything seems fine. However, in our use case, the msg->buf does not always seem to
be null terminated at msg->len.
In other words: msg->buf[msg->len] != '\0'
It seems there is sometimes some extra data in the buffer beyond msg->len, and
subst_run() may abort() if it ever moves beyond msg->len.
I found a few places in textops.c where a buffer is being temporarily null terminated in
order to perform a regexec () search, then restored later, as in:
body = hf->body;
c = body.s[body.len];
body.s[body.len] = '\0';
ret = regexec((regex_t*) re, body.s, 1, &pmatch, 0);
body.s[body.len] = c;
I've tried making the same type of change to subst_f() and subst_body_f(), and now I
am not seeing the crash.
If I guarantee that msg->buf[msg->len] == '\0' when subst_run() is called,
then everything seems to be OK.
What do you think? Is that a valid change to make?
Let me know if you'd like me to make a pull request on the main branch with those
changes.
Thanks very much.
-Cody
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Sunday, July 30, 2017 11:12 PM
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>rg>; Cody Herzog
<CHerzog(a)IntouchHealth.com>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
Hello,
what's the version of Kamailio and the operating system you run on?
Cheers,
Daniel
On 28.07.17 23:53, Cody Herzog wrote:
Hello.
Inside the tm:local-request route, I am trying to modify the body of outgoing NOTIFY
messages using the subst_body() function.
Things seem to work fine most of the time. The NOTIFY messages are correctly modified.
However, I will very rarely hit a crash.
I can force the crash to happen more quickly by stress testing with thousands of NOTIFY
messages being modified.
Is it safe to call subst() or subst_body() inside the tm:local-request route? If not, is
there another way I can modify the outgoing NOTIFY messages safely?
Note that I am also calling append_hf() inside the same tm:local-request route, but that
has been working for a very long time without causing any problems. Could there be some
kind of bad interaction between append_hf() and subst_body()?
Strangely, it seems like using subst() makes the crash more likely to happen. For some
reason, subst_body() seems to be more robust.
Here are the logs I see:
---------
Jul 28 10:57:40 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2476]: CRITICAL: <core>
[data_lump.c:292]: del_lump(): offset exceeds message size (1625 > 1194) aborting...
Jul 28 10:57:43 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2481]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 53
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:731]: handle_sigs(): child process 2476 exited by a signal 6
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:734]: handle_sigs(): core was generated
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: INFO: <core>
[main.c:756]: handle_sigs(): terminating due to SIGCHLD
---------
Here is a GDB back trace from the core dump:
---------
#0 0x00007fd91a567c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd91a56b028 in __GI_abort () at abort.c:89
#2 0x000000000044e981 in del_lump (msg=0x7fd917c5ed60 <lreq>, offset=1625, len=20,
type=HDR_OTHER_T) at data_lump.c:293
#3 0x00007fd91676297f in subst_f (msg=0x7fd917c5ed60 <lreq>, subst=0x7fd919e5ccc0
"\320\356\345\031\331\177", ignored=0x0) at textops.c:681
#4 0x000000000041e1dd in do_action (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1059
#5 0x000000000042a917 in run_actions (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#6 0x000000000042af94 in run_actions_safe (h=0x7ffd364ba5b0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1613
#7 0x0000000000549972 in rval_get_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60 <lreq>,
i=0x7ffd364b8b58, rv=0x7fd919e9ced8, cache=0x0) at rvalue.c:912
#8 0x000000000054dc7c in rval_expr_eval_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60
<lreq>, res=0x7ffd364b8b58, rve=0x7fd919e9ced0) at rvalue.c:1910
#9 0x000000000041dc52 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1029
#10 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#11 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9dad8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#12 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e973b8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#13 0x000000000041ab46 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95d70,
msg=0x7fd917c5ed60 <lreq>) at action.c:677
#14 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8f330,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#15 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95eb8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#16 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8c8e0,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#17 0x000000000042b07d in run_top_route (a=0x7fd919e8c8e0, msg=0x7fd917c5ed60
<lreq>, c=0x0) at action.c:1634
#18 0x00007fd917a0539a in t_uac_prepare (uac_r=0x7ffd364baf20, dst_req=0x7ffd364ba8f0,
dst_cell=0x7ffd364ba8f8) at uac.c:391
#19 0x00007fd917a07945 in t_uac_with_ids (uac_r=0x7ffd364baf20, ret_index=0x0,
ret_label=0x0) at uac.c:599
#20 0x00007fd917a07918 in t_uac (uac_r=0x7ffd364baf20) at uac.c:584
#21 0x00007fd917a09b9e in req_within (uac_r=0x7ffd364baf20) at uac.c:802
#22 0x00007fd914674052 in send_notify_request (subs=0x7ffd364bb200, watcher_subs=0x0,
n_body=0x0, force_null_body=0) at notify.c:1600
#23 0x00007fd9146754b3 in notify (subs=0x7ffd364bb200, watcher_subs=0x0, n_body=0x0,
force_null_body=0) at notify.c:1690
#24 0x00007fd9146cec6a in update_subscription (msg=0x7fd919efeb70, subs=0x7ffd364bb200,
to_tag_gen=1, sent_reply=0x7ffd364bb16c) at subscribe.c:697
#25 0x00007fd9146d45b7 in handle_subscribe (msg=0x7fd919efeb70, watcher_user=...,
watcher_domain=...) at subscribe.c:1057
#26 0x00007fd9146d08b0 in handle_subscribe0 (msg=0x7fd919efeb70) at subscribe.c:816
#27 0x000000000041e155 in do_action (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1053
#28 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1548
#29 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19650,
msg=0x7fd919efeb70) at action.c:1048
#30 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e02638,
msg=0x7fd919efeb70) at action.c:1548
#31 0x000000000041e0c4 in do_action (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1044
#32 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1548
#33 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19b08,
msg=0x7fd919efeb70) at action.c:1048
#34 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919df99d8,
msg=0x7fd919efeb70) at action.c:1548
#35 0x000000000041ab46 in do_action (h=0x7ffd364bd680, a=0x7fd919dbe960,
msg=0x7fd919efeb70) at action.c:677
#36 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919daedf8,
msg=0x7fd919efeb70) at action.c:1548
#37 0x000000000042b07d in run_top_route (a=0x7fd919daedf8, msg=0x7fd919efeb70, c=0x0) at
action.c:1634
#38 0x000000000050f678 in receive_msg (buf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on><sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml, application/pidf-diff+xml,"...,
len=1284, rcv_info=0x7fd8f2c6cb50) at receive.c:196
#39 0x00000000005fb99e in receive_tcp_msg (tcpbuf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on><sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml, application/pidf-diff+xml,"...,
len=1284, rcv_info=0x7fd8f2c6cb50, con=0x7fd8f2c6cb38) at tcp_read.c:1207
#40 0x00000000005fd48c in tcp_read_req (con=0x7fd8f2c6cb38, bytes_read=0x7ffd364bdb7c,
read_flags=0x7ffd364bdb84) at tcp_read.c:1411
#41 0x0000000000600aee in handle_io (fm=0x7fd919f07d80, events=1, idx=-1) at
tcp_read.c:1643
#42 0x00000000005f3ab8 in io_wait_loop_epoll (h=0xa3d3e0 <io_w>, t=2, repeat=0) at
io_wait.h:1061
#43 0x0000000000601cf7 in tcp_receive_loop (unix_sock=57) at tcp_read.c:1755
#44 0x00000000005ea235 in tcp_init_children () at tcp_main.c:4788
#45 0x00000000004ab048 in main_loop () at main.c:1675
#46 0x00000000004b10c2 in main (argc=13, argv=0x7ffd364be258) at main.c:2566
---------
Thanks very much.
-Cody
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
7 Aug
7 Aug
3:14 a.m.
Hello,
can you make a pull request with the patch you did to terminate the
buffer with '\0'? It is a valid fix if there are situations when the
buffer is not zero-terminated.
To figure out when it is not zero-terminated, if you don't know already,
in order to analyze if it has impact on other part of code, few questions:
- are you using msg_apply_changes()?
- do you receive traffic over tcp/tls?
Cheers,
Daniel
On 31.07.17 19:43, Cody Herzog wrote:
Hello.
We are running Kamailio 4.3.5 on Ubuntu 14.04 LTS 64-bit.
Sorry, I should have included that information in my first email.
I have done more debugging, and I believe the problem is caused by the
subst_run() function being used on a message buffer which is not null
terminated at msg->len.
The subst_f() and subst_body_f() functions both call subst_run(), and
most of the time, everything seems fine. However, in our use case, the
msg->buf does not always seem to be null terminated at msg->len.
In other words: msg->buf[msg->len] != '\0'
It seems there is sometimes some extra data in the buffer beyond
msg->len, and subst_run() may abort() if it ever moves beyond msg->len.
I found a few places in textops.c where a buffer is being temporarily
null terminated in order to perform a regexec () search, then restored
later, as in:
body = hf->body;
c = body.s[body.len];
body.s[body.len] = '\0';
ret = regexec((regex_t*) re, body.s, 1, &pmatch, 0);
body.s[body.len] = c;
I've tried making the same type of change to subst_f() and
subst_body_f(), and now I am not seeing the crash.
If I guarantee that msg->buf[msg->len] == '\0' when subst_run() is
called, then everything seems to be OK.
What do you think? Is that a valid change to make?
Let me know if you'd like me to make a pull request on the main branch
with those changes.
Thanks very much.
-Cody
*From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Sent:* Sunday, July 30, 2017 11:12 PM
*To:* Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>rg>; Cody Herzog <CHerzog(a)IntouchHealth.com>
*Subject:* Re: [SR-Users] Core crash in del_lump() "offset exceeds
message size" when using subst() inside tm:local-request route.
Hello,
what's the version of Kamailio and the operating system you run on?
Cheers,
Daniel
On 28.07.17 23:53, Cody Herzog wrote:
Hello.
Inside the tm:local-request route, I am trying to modify the body
of outgoing NOTIFY messages using the subst_body() function.
Things seem to work fine most of the time. The NOTIFY messages are
correctly modified. However, I will very rarely hit a crash.
I can force the crash to happen more quickly by stress testing
with thousands of NOTIFY messages being modified.
Is it safe to call subst() or subst_body() inside the
tm:local-request route? If not, is there another way I can modify
the outgoing NOTIFY messages safely?
Note that I am also calling append_hf() inside the same
tm:local-request route, but that has been working for a very long
time without causing any problems. Could there be some kind of bad
interaction between append_hf() and subst_body()?
Strangely, it seems like using subst() makes the crash more likely
to happen. For some reason, subst_body() seems to be more robust.
Here are the logs I see:
---------
Jul 28 10:57:40 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2476]:
CRITICAL: <core> [data_lump.c:292]: del_lump(): offset exceeds
message size (1625 > 1194) aborting...
Jul 28 10:57:43 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2481]:
CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 53
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]:
ALERT: <core> [main.c:731]: handle_sigs(): child process 2476
exited by a signal 6
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]:
ALERT: <core> [main.c:734]: handle_sigs(): core was generated
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]:
INFO: <core> [main.c:756]: handle_sigs(): terminating due to SIGCHLD
---------
Here is a GDB back trace from the core dump:
---------
#0 0x00007fd91a567c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd91a56b028 in __GI_abort () at abort.c:89
#2 0x000000000044e981 in del_lump (msg=0x7fd917c5ed60 <lreq>,
offset=1625, len=20, type=HDR_OTHER_T) at data_lump.c:293
#3 0x00007fd91676297f in subst_f (msg=0x7fd917c5ed60 <lreq>,
subst=0x7fd919e5ccc0 "\320\356\345\031\331\177", ignored=0x0) at
textops.c:681
#4 0x000000000041e1dd in do_action (h=0x7ffd364b87c0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1059
#5 0x000000000042a917 in run_actions (h=0x7ffd364b87c0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#6 0x000000000042af94 in run_actions_safe (h=0x7ffd364ba5b0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1613
#7 0x0000000000549972 in rval_get_int (h=0x7ffd364ba5b0,
msg=0x7fd917c5ed60 <lreq>, i=0x7ffd364b8b58, rv=0x7fd919e9ced8,
cache=0x0) at rvalue.c:912
#8 0x000000000054dc7c in rval_expr_eval_int (h=0x7ffd364ba5b0,
msg=0x7fd917c5ed60 <lreq>, res=0x7ffd364b8b58, rve=0x7fd919e9ced0)
at rvalue.c:1910
#9 0x000000000041dc52 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e9d990, msg=0x7fd917c5ed60 <lreq>) at action.c:1029
#10 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e9d990, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#11 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e9dad8, msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#12 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e973b8, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#13 0x000000000041ab46 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e95d70, msg=0x7fd917c5ed60 <lreq>) at action.c:677
#14 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e8f330, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#15 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e95eb8, msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#16 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e8c8e0, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#17 0x000000000042b07d in run_top_route (a=0x7fd919e8c8e0,
msg=0x7fd917c5ed60 <lreq>, c=0x0) at action.c:1634
#18 0x00007fd917a0539a in t_uac_prepare (uac_r=0x7ffd364baf20,
dst_req=0x7ffd364ba8f0, dst_cell=0x7ffd364ba8f8) at uac.c:391
#19 0x00007fd917a07945 in t_uac_with_ids (uac_r=0x7ffd364baf20,
ret_index=0x0, ret_label=0x0) at uac.c:599
#20 0x00007fd917a07918 in t_uac (uac_r=0x7ffd364baf20) at uac.c:584
#21 0x00007fd917a09b9e in req_within (uac_r=0x7ffd364baf20) at
uac.c:802
#22 0x00007fd914674052 in send_notify_request
(subs=0x7ffd364bb200, watcher_subs=0x0, n_body=0x0,
force_null_body=0) at notify.c:1600
#23 0x00007fd9146754b3 in notify (subs=0x7ffd364bb200,
watcher_subs=0x0, n_body=0x0, force_null_body=0) at notify.c:1690
#24 0x00007fd9146cec6a in update_subscription (msg=0x7fd919efeb70,
subs=0x7ffd364bb200, to_tag_gen=1, sent_reply=0x7ffd364bb16c) at
subscribe.c:697
#25 0x00007fd9146d45b7 in handle_subscribe (msg=0x7fd919efeb70,
watcher_user=..., watcher_domain=...) at subscribe.c:1057
#26 0x00007fd9146d08b0 in handle_subscribe0 (msg=0x7fd919efeb70)
at subscribe.c:816
#27 0x000000000041e155 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19508, msg=0x7fd919efeb70) at action.c:1053
#28 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e19508, msg=0x7fd919efeb70) at action.c:1548
#29 0x000000000041e112 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19650, msg=0x7fd919efeb70) at action.c:1048
#30 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e02638, msg=0x7fd919efeb70) at action.c:1548
#31 0x000000000041e0c4 in do_action (h=0x7ffd364bd680,
a=0x7fd919e199c0, msg=0x7fd919efeb70) at action.c:1044
#32 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e199c0, msg=0x7fd919efeb70) at action.c:1548
#33 0x000000000041e112 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19b08, msg=0x7fd919efeb70) at action.c:1048
#34 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919df99d8, msg=0x7fd919efeb70) at action.c:1548
#35 0x000000000041ab46 in do_action (h=0x7ffd364bd680,
a=0x7fd919dbe960, msg=0x7fd919efeb70) at action.c:677
#36 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919daedf8, msg=0x7fd919efeb70) at action.c:1548
#37 0x000000000042b07d in run_top_route (a=0x7fd919daedf8,
msg=0x7fd919efeb70, c=0x0) at action.c:1634
#38 0x000000000050f678 in receive_msg (buf=0x7fd8f2c6ce28
"SUBSCRIBE sip:endpoint-1720@intouchstaging.com
SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml,
application/pidf-diff+xml,"..., len=1284, rcv_info=0x7fd8f2c6cb50)
at receive.c:196
#39 0x00000000005fb99e in receive_tcp_msg (tcpbuf=0x7fd8f2c6ce28
"SUBSCRIBE sip:endpoint-1720@intouchstaging.com
SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml,
application/pidf-diff+xml,"..., len=1284, rcv_info=0x7fd8f2c6cb50,
con=0x7fd8f2c6cb38) at tcp_read.c:1207
#40 0x00000000005fd48c in tcp_read_req (con=0x7fd8f2c6cb38,
bytes_read=0x7ffd364bdb7c, read_flags=0x7ffd364bdb84) at
tcp_read.c:1411
#41 0x0000000000600aee in handle_io (fm=0x7fd919f07d80, events=1,
idx=-1) at tcp_read.c:1643
#42 0x00000000005f3ab8 in io_wait_loop_epoll (h=0xa3d3e0 <io_w>,
t=2, repeat=0) at io_wait.h:1061
#43 0x0000000000601cf7 in tcp_receive_loop (unix_sock=57) at
tcp_read.c:1755
#44 0x00000000005ea235 in tcp_init_children () at tcp_main.c:4788
#45 0x00000000004ab048 in main_loop () at main.c:1675
#46 0x00000000004b10c2 in main (argc=13, argv=0x7ffd364be258) at
main.c:2566
---------
Thanks very much.
-Cody
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com <http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com <http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com
1:47 p.m.
Thanks for the response.
I will make a pull request soon.
are you using msg_apply_changes()?
No. I tried to use it, but it does not seem to be allowed in the tm:local-request route.
do you receive traffic over tcp/tls?
Yes. We are using TLS.
Thanks.
-Cody
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Monday, August 7, 2017 12:15 AM
To: Cody Herzog <CHerzog(a)IntouchHealth.com>om>; Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
Hello,
can you make a pull request with the patch you did to terminate the buffer with
'\0'? It is a valid fix if there are situations when the buffer is not
zero-terminated.
To figure out when it is not zero-terminated, if you don't know already, in order to
analyze if it has impact on other part of code, few questions:
- are you using msg_apply_changes()?
- do you receive traffic over tcp/tls?
Cheers,
Daniel
On 31.07.17 19:43, Cody Herzog wrote:
Hello.
We are running Kamailio 4.3.5 on Ubuntu 14.04 LTS 64-bit.
Sorry, I should have included that information in my first email.
I have done more debugging, and I believe the problem is caused by the subst_run()
function being used on a message buffer which is not null terminated at msg->len.
The subst_f() and subst_body_f() functions both call subst_run(), and most of the time,
everything seems fine. However, in our use case, the msg->buf does not always seem to
be null terminated at msg->len.
In other words: msg->buf[msg->len] != '\0'
It seems there is sometimes some extra data in the buffer beyond msg->len, and
subst_run() may abort() if it ever moves beyond msg->len.
I found a few places in textops.c where a buffer is being temporarily null terminated in
order to perform a regexec () search, then restored later, as in:
body = hf->body;
c = body.s[body.len];
body.s[body.len] = '\0';
ret = regexec((regex_t*) re, body.s, 1, &pmatch, 0);
body.s[body.len] = c;
I've tried making the same type of change to subst_f() and subst_body_f(), and now I
am not seeing the crash.
If I guarantee that msg->buf[msg->len] == '\0' when subst_run() is called,
then everything seems to be OK.
What do you think? Is that a valid change to make?
Let me know if you'd like me to make a pull request on the main branch with those
changes.
Thanks very much.
-Cody
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Sunday, July 30, 2017 11:12 PM
To: Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>; Cody Herzog
<CHerzog@IntouchHealth.com><mailto:CHerzog@IntouchHealth.com>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
Hello,
what's the version of Kamailio and the operating system you run on?
Cheers,
Daniel
On 28.07.17 23:53, Cody Herzog wrote:
Hello.
Inside the tm:local-request route, I am trying to modify the body of outgoing NOTIFY
messages using the subst_body() function.
Things seem to work fine most of the time. The NOTIFY messages are correctly modified.
However, I will very rarely hit a crash.
I can force the crash to happen more quickly by stress testing with thousands of NOTIFY
messages being modified.
Is it safe to call subst() or subst_body() inside the tm:local-request route? If not, is
there another way I can modify the outgoing NOTIFY messages safely?
Note that I am also calling append_hf() inside the same tm:local-request route, but that
has been working for a very long time without causing any problems. Could there be some
kind of bad interaction between append_hf() and subst_body()?
Strangely, it seems like using subst() makes the crash more likely to happen. For some
reason, subst_body() seems to be more robust.
Here are the logs I see:
---------
Jul 28 10:57:40 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2476]: CRITICAL: <core>
[data_lump.c:292]: del_lump(): offset exceeds message size (1625 > 1194) aborting...
Jul 28 10:57:43 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2481]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 53
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:731]: handle_sigs(): child process 2476 exited by a signal 6
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:734]: handle_sigs(): core was generated
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: INFO: <core>
[main.c:756]: handle_sigs(): terminating due to SIGCHLD
---------
Here is a GDB back trace from the core dump:
---------
#0 0x00007fd91a567c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd91a56b028 in __GI_abort () at abort.c:89
#2 0x000000000044e981 in del_lump (msg=0x7fd917c5ed60 <lreq>, offset=1625, len=20,
type=HDR_OTHER_T) at data_lump.c:293
#3 0x00007fd91676297f in subst_f (msg=0x7fd917c5ed60 <lreq>, subst=0x7fd919e5ccc0
"\320\356\345\031\331\177", ignored=0x0) at textops.c:681
#4 0x000000000041e1dd in do_action (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1059
#5 0x000000000042a917 in run_actions (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#6 0x000000000042af94 in run_actions_safe (h=0x7ffd364ba5b0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1613
#7 0x0000000000549972 in rval_get_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60 <lreq>,
i=0x7ffd364b8b58, rv=0x7fd919e9ced8, cache=0x0) at rvalue.c:912
#8 0x000000000054dc7c in rval_expr_eval_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60
<lreq>, res=0x7ffd364b8b58, rve=0x7fd919e9ced0) at rvalue.c:1910
#9 0x000000000041dc52 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1029
#10 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#11 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9dad8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#12 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e973b8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#13 0x000000000041ab46 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95d70,
msg=0x7fd917c5ed60 <lreq>) at action.c:677
#14 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8f330,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#15 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95eb8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#16 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8c8e0,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#17 0x000000000042b07d in run_top_route (a=0x7fd919e8c8e0, msg=0x7fd917c5ed60
<lreq>, c=0x0) at action.c:1634
#18 0x00007fd917a0539a in t_uac_prepare (uac_r=0x7ffd364baf20, dst_req=0x7ffd364ba8f0,
dst_cell=0x7ffd364ba8f8) at uac.c:391
#19 0x00007fd917a07945 in t_uac_with_ids (uac_r=0x7ffd364baf20, ret_index=0x0,
ret_label=0x0) at uac.c:599
#20 0x00007fd917a07918 in t_uac (uac_r=0x7ffd364baf20) at uac.c:584
#21 0x00007fd917a09b9e in req_within (uac_r=0x7ffd364baf20) at uac.c:802
#22 0x00007fd914674052 in send_notify_request (subs=0x7ffd364bb200, watcher_subs=0x0,
n_body=0x0, force_null_body=0) at notify.c:1600
#23 0x00007fd9146754b3 in notify (subs=0x7ffd364bb200, watcher_subs=0x0, n_body=0x0,
force_null_body=0) at notify.c:1690
#24 0x00007fd9146cec6a in update_subscription (msg=0x7fd919efeb70, subs=0x7ffd364bb200,
to_tag_gen=1, sent_reply=0x7ffd364bb16c) at subscribe.c:697
#25 0x00007fd9146d45b7 in handle_subscribe (msg=0x7fd919efeb70, watcher_user=...,
watcher_domain=...) at subscribe.c:1057
#26 0x00007fd9146d08b0 in handle_subscribe0 (msg=0x7fd919efeb70) at subscribe.c:816
#27 0x000000000041e155 in do_action (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1053
#28 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1548
#29 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19650,
msg=0x7fd919efeb70) at action.c:1048
#30 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e02638,
msg=0x7fd919efeb70) at action.c:1548
#31 0x000000000041e0c4 in do_action (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1044
#32 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1548
#33 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19b08,
msg=0x7fd919efeb70) at action.c:1048
#34 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919df99d8,
msg=0x7fd919efeb70) at action.c:1548
#35 0x000000000041ab46 in do_action (h=0x7ffd364bd680, a=0x7fd919dbe960,
msg=0x7fd919efeb70) at action.c:677
#36 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919daedf8,
msg=0x7fd919efeb70) at action.c:1548
#37 0x000000000042b07d in run_top_route (a=0x7fd919daedf8, msg=0x7fd919efeb70, c=0x0) at
action.c:1634
#38 0x000000000050f678 in receive_msg (buf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on><sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml, application/pidf-diff+xml,"...,
len=1284, rcv_info=0x7fd8f2c6cb50) at receive.c:196
#39 0x00000000005fb99e in receive_tcp_msg (tcpbuf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on><sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml, application/pidf-diff+xml,"...,
len=1284, rcv_info=0x7fd8f2c6cb50, con=0x7fd8f2c6cb38) at tcp_read.c:1207
#40 0x00000000005fd48c in tcp_read_req (con=0x7fd8f2c6cb38, bytes_read=0x7ffd364bdb7c,
read_flags=0x7ffd364bdb84) at tcp_read.c:1411
#41 0x0000000000600aee in handle_io (fm=0x7fd919f07d80, events=1, idx=-1) at
tcp_read.c:1643
#42 0x00000000005f3ab8 in io_wait_loop_epoll (h=0xa3d3e0 <io_w>, t=2, repeat=0) at
io_wait.h:1061
#43 0x0000000000601cf7 in tcp_receive_loop (unix_sock=57) at tcp_read.c:1755
#44 0x00000000005ea235 in tcp_init_children () at tcp_main.c:4788
#45 0x00000000004ab048 in main_loop () at main.c:1675
#46 0x00000000004b10c2 in main (argc=13, argv=0x7ffd364be258) at main.c:2566
---------
Thanks very much.
-Cody
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
3:54 p.m.
Were you using the subst() in event_route[...]?
Cheers,
Daniel
On 07.08.17 19:47, Cody Herzog wrote:
Thanks for the response.
I will make a pull request soon.
--
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com
are you using msg_apply_changes()?
No. I tried to use it, but it does not seem to be allowed in the
tm:local-request route.
do you receive traffic over tcp/tls?
Yes. We are using TLS.
Thanks.
-Cody
*From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Sent:* Monday, August 7, 2017 12:15 AM
*To:* Cody Herzog <CHerzog(a)IntouchHealth.com>om>; Kamailio (SER) - Users
Mailing List <sr-users(a)lists.kamailio.org>
*Subject:* Re: [SR-Users] Core crash in del_lump() "offset exceeds
message size" when using subst() inside tm:local-request route.
Hello,
can you make a pull request with the patch you did to terminate the
buffer with '\0'? It is a valid fix if there are situations when the
buffer is not zero-terminated.
To figure out when it is not zero-terminated, if you don't know
already, in order to analyze if it has impact on other part of code,
few questions:
- are you using msg_apply_changes()?
- do you receive traffic over tcp/tls?
Cheers,
Daniel
On 31.07.17 19:43, Cody Herzog wrote:
Hello.
We are running Kamailio 4.3.5 on Ubuntu 14.04 LTS 64-bit.
Sorry, I should have included that information in my first email.
I have done more debugging, and I believe the problem is caused by
the subst_run() function being used on a message buffer which is
not null terminated at msg->len.
The subst_f() and subst_body_f() functions both call subst_run(),
and most of the time, everything seems fine. However, in our use
case, the msg->buf does not always seem to be null terminated at
msg->len.
In other words: msg->buf[msg->len] != '\0'
It seems there is sometimes some extra data in the buffer beyond
msg->len, and subst_run() may abort() if it ever moves beyond
msg->len.
I found a few places in textops.c where a buffer is being
temporarily null terminated in order to perform a regexec ()
search, then restored later, as in:
body = hf->body;
c = body.s[body.len];
body.s[body.len] = '\0';
ret = regexec((regex_t*) re, body.s, 1, &pmatch, 0);
body.s[body.len] = c;
I've tried making the same type of change to subst_f() and
subst_body_f(), and now I am not seeing the crash.
If I guarantee that msg->buf[msg->len] == '\0' when subst_run() is
called, then everything seems to be OK.
What do you think? Is that a valid change to make?
Let me know if you'd like me to make a pull request on the main
branch with those changes.
Thanks very much.
-Cody
*From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Sent:* Sunday, July 30, 2017 11:12 PM
*To:* Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>
<mailto:sr-users@lists.kamailio.org>; Cody Herzog
<CHerzog(a)IntouchHealth.com> <mailto:CHerzog@IntouchHealth.com>
*Subject:* Re: [SR-Users] Core crash in del_lump() "offset exceeds
message size" when using subst() inside tm:local-request route.
Hello,
what's the version of Kamailio and the operating system you run on?
Cheers,
Daniel
On 28.07.17 23:53, Cody Herzog wrote:
Hello.
Inside the tm:local-request route, I am trying to modify the
body of outgoing NOTIFY messages using the subst_body() function.
Things seem to work fine most of the time. The NOTIFY messages
are correctly modified. However, I will very rarely hit a crash.
I can force the crash to happen more quickly by stress testing
with thousands of NOTIFY messages being modified.
Is it safe to call subst() or subst_body() inside the
tm:local-request route? If not, is there another way I can
modify the outgoing NOTIFY messages safely?
Note that I am also calling append_hf() inside the same
tm:local-request route, but that has been working for a very
long time without causing any problems. Could there be some
kind of bad interaction between append_hf() and subst_body()?
Strangely, it seems like using subst() makes the crash more
likely to happen. For some reason, subst_body() seems to be
more robust.
Here are the logs I see:
---------
Jul 28 10:57:40 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2476]: CRITICAL: <core>
[data_lump.c:292]: del_lump(): offset exceeds message size
(1625 > 1194) aborting...
Jul 28 10:57:43 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2481]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 53
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2367]: ALERT: <core> [main.c:731]:
handle_sigs(): child process 2476 exited by a signal 6
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2367]: ALERT: <core> [main.c:734]:
handle_sigs(): core was generated
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2367]: INFO: <core> [main.c:756]:
handle_sigs(): terminating due to SIGCHLD
---------
Here is a GDB back trace from the core dump:
---------
#0 0x00007fd91a567c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd91a56b028 in __GI_abort () at abort.c:89
#2 0x000000000044e981 in del_lump (msg=0x7fd917c5ed60 <lreq>,
offset=1625, len=20, type=HDR_OTHER_T) at data_lump.c:293
#3 0x00007fd91676297f in subst_f (msg=0x7fd917c5ed60 <lreq>,
subst=0x7fd919e5ccc0 "\320\356\345\031\331\177", ignored=0x0)
at textops.c:681
#4 0x000000000041e1dd in do_action (h=0x7ffd364b87c0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1059
#5 0x000000000042a917 in run_actions (h=0x7ffd364b87c0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#6 0x000000000042af94 in run_actions_safe (h=0x7ffd364ba5b0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1613
#7 0x0000000000549972 in rval_get_int (h=0x7ffd364ba5b0,
msg=0x7fd917c5ed60 <lreq>, i=0x7ffd364b8b58,
rv=0x7fd919e9ced8, cache=0x0) at rvalue.c:912
#8 0x000000000054dc7c in rval_expr_eval_int
(h=0x7ffd364ba5b0, msg=0x7fd917c5ed60 <lreq>,
res=0x7ffd364b8b58, rve=0x7fd919e9ced0) at rvalue.c:1910
#9 0x000000000041dc52 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e9d990, msg=0x7fd917c5ed60 <lreq>) at action.c:1029
#10 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e9d990, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#11 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e9dad8, msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#12 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e973b8, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#13 0x000000000041ab46 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e95d70, msg=0x7fd917c5ed60 <lreq>) at action.c:677
#14 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e8f330, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#15 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e95eb8, msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#16 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e8c8e0, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#17 0x000000000042b07d in run_top_route (a=0x7fd919e8c8e0,
msg=0x7fd917c5ed60 <lreq>, c=0x0) at action.c:1634
#18 0x00007fd917a0539a in t_uac_prepare (uac_r=0x7ffd364baf20,
dst_req=0x7ffd364ba8f0, dst_cell=0x7ffd364ba8f8) at uac.c:391
#19 0x00007fd917a07945 in t_uac_with_ids
(uac_r=0x7ffd364baf20, ret_index=0x0, ret_label=0x0) at uac.c:599
#20 0x00007fd917a07918 in t_uac (uac_r=0x7ffd364baf20) at
uac.c:584
#21 0x00007fd917a09b9e in req_within (uac_r=0x7ffd364baf20) at
uac.c:802
#22 0x00007fd914674052 in send_notify_request
(subs=0x7ffd364bb200, watcher_subs=0x0, n_body=0x0,
force_null_body=0) at notify.c:1600
#23 0x00007fd9146754b3 in notify (subs=0x7ffd364bb200,
watcher_subs=0x0, n_body=0x0, force_null_body=0) at notify.c:1690
#24 0x00007fd9146cec6a in update_subscription
(msg=0x7fd919efeb70, subs=0x7ffd364bb200, to_tag_gen=1,
sent_reply=0x7ffd364bb16c) at subscribe.c:697
#25 0x00007fd9146d45b7 in handle_subscribe
(msg=0x7fd919efeb70, watcher_user=..., watcher_domain=...) at
subscribe.c:1057
#26 0x00007fd9146d08b0 in handle_subscribe0
(msg=0x7fd919efeb70) at subscribe.c:816
#27 0x000000000041e155 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19508, msg=0x7fd919efeb70) at action.c:1053
#28 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e19508, msg=0x7fd919efeb70) at action.c:1548
#29 0x000000000041e112 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19650, msg=0x7fd919efeb70) at action.c:1048
#30 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e02638, msg=0x7fd919efeb70) at action.c:1548
#31 0x000000000041e0c4 in do_action (h=0x7ffd364bd680,
a=0x7fd919e199c0, msg=0x7fd919efeb70) at action.c:1044
#32 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e199c0, msg=0x7fd919efeb70) at action.c:1548
#33 0x000000000041e112 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19b08, msg=0x7fd919efeb70) at action.c:1048
#34 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919df99d8, msg=0x7fd919efeb70) at action.c:1548
#35 0x000000000041ab46 in do_action (h=0x7ffd364bd680,
a=0x7fd919dbe960, msg=0x7fd919efeb70) at action.c:677
#36 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919daedf8, msg=0x7fd919efeb70) at action.c:1548
#37 0x000000000042b07d in run_top_route (a=0x7fd919daedf8,
msg=0x7fd919efeb70, c=0x0) at action.c:1634
#38 0x000000000050f678 in receive_msg (buf=0x7fd8f2c6ce28
"SUBSCRIBE sip:endpoint-1720@intouchstaging.com
SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml,
application/pidf-diff+xml,"..., len=1284,
rcv_info=0x7fd8f2c6cb50) at receive.c:196
#39 0x00000000005fb99e in receive_tcp_msg
(tcpbuf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml,
application/pidf-diff+xml,"..., len=1284,
rcv_info=0x7fd8f2c6cb50, con=0x7fd8f2c6cb38) at tcp_read.c:1207
#40 0x00000000005fd48c in tcp_read_req (con=0x7fd8f2c6cb38,
bytes_read=0x7ffd364bdb7c, read_flags=0x7ffd364bdb84) at
tcp_read.c:1411
#41 0x0000000000600aee in handle_io (fm=0x7fd919f07d80,
events=1, idx=-1) at tcp_read.c:1643
#42 0x00000000005f3ab8 in io_wait_loop_epoll (h=0xa3d3e0
<io_w>, t=2, repeat=0) at io_wait.h:1061
#43 0x0000000000601cf7 in tcp_receive_loop (unix_sock=57) at
tcp_read.c:1755
#44 0x00000000005ea235 in tcp_init_children () at tcp_main.c:4788
#45 0x00000000004ab048 in main_loop () at main.c:1675
#46 0x00000000004b10c2 in main (argc=13, argv=0x7ffd364be258)
at main.c:2566
---------
Thanks very much.
-Cody
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com <http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com
<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com <http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com <http://www.kamailioworld.com>
4:13 p.m.
Yes. The crash was happening when using subst() or subst_body() inside of
event_route[tm:local-request].
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Monday, August 7, 2017 12:55 PM
To: Cody Herzog <CHerzog(a)IntouchHealth.com>om>; Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
Were you using the subst() in event_route[...]?
Cheers,
Daniel
On 07.08.17 19:47, Cody Herzog wrote:
Thanks for the response.
I will make a pull request soon.
are you using msg_apply_changes()?
No. I tried to use it, but it does not seem to be allowed in the tm:local-request route.
do you receive traffic over tcp/tls?
Yes. We are using TLS.
Thanks.
-Cody
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Monday, August 7, 2017 12:15 AM
To: Cody Herzog <CHerzog@IntouchHealth.com><mailto:CHerzog@IntouchHealth.com>;
Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
Hello,
can you make a pull request with the patch you did to terminate the buffer with
'\0'? It is a valid fix if there are situations when the buffer is not
zero-terminated.
To figure out when it is not zero-terminated, if you don't know already, in order to
analyze if it has impact on other part of code, few questions:
- are you using msg_apply_changes()?
- do you receive traffic over tcp/tls?
Cheers,
Daniel
On 31.07.17 19:43, Cody Herzog wrote:
Hello.
We are running Kamailio 4.3.5 on Ubuntu 14.04 LTS 64-bit.
Sorry, I should have included that information in my first email.
I have done more debugging, and I believe the problem is caused by the subst_run()
function being used on a message buffer which is not null terminated at msg->len.
The subst_f() and subst_body_f() functions both call subst_run(), and most of the time,
everything seems fine. However, in our use case, the msg->buf does not always seem to
be null terminated at msg->len.
In other words: msg->buf[msg->len] != '\0'
It seems there is sometimes some extra data in the buffer beyond msg->len, and
subst_run() may abort() if it ever moves beyond msg->len.
I found a few places in textops.c where a buffer is being temporarily null terminated in
order to perform a regexec () search, then restored later, as in:
body = hf->body;
c = body.s[body.len];
body.s[body.len] = '\0';
ret = regexec((regex_t*) re, body.s, 1, &pmatch, 0);
body.s[body.len] = c;
I've tried making the same type of change to subst_f() and subst_body_f(), and now I
am not seeing the crash.
If I guarantee that msg->buf[msg->len] == '\0' when subst_run() is called,
then everything seems to be OK.
What do you think? Is that a valid change to make?
Let me know if you'd like me to make a pull request on the main branch with those
changes.
Thanks very much.
-Cody
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Sunday, July 30, 2017 11:12 PM
To: Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>; Cody Herzog
<CHerzog@IntouchHealth.com><mailto:CHerzog@IntouchHealth.com>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
Hello,
what's the version of Kamailio and the operating system you run on?
Cheers,
Daniel
On 28.07.17 23:53, Cody Herzog wrote:
Hello.
Inside the tm:local-request route, I am trying to modify the body of outgoing NOTIFY
messages using the subst_body() function.
Things seem to work fine most of the time. The NOTIFY messages are correctly modified.
However, I will very rarely hit a crash.
I can force the crash to happen more quickly by stress testing with thousands of NOTIFY
messages being modified.
Is it safe to call subst() or subst_body() inside the tm:local-request route? If not, is
there another way I can modify the outgoing NOTIFY messages safely?
Note that I am also calling append_hf() inside the same tm:local-request route, but that
has been working for a very long time without causing any problems. Could there be some
kind of bad interaction between append_hf() and subst_body()?
Strangely, it seems like using subst() makes the crash more likely to happen. For some
reason, subst_body() seems to be more robust.
Here are the logs I see:
---------
Jul 28 10:57:40 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2476]: CRITICAL: <core>
[data_lump.c:292]: del_lump(): offset exceeds message size (1625 > 1194) aborting...
Jul 28 10:57:43 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2481]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 53
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:731]: handle_sigs(): child process 2476 exited by a signal 6
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:734]: handle_sigs(): core was generated
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: INFO: <core>
[main.c:756]: handle_sigs(): terminating due to SIGCHLD
---------
Here is a GDB back trace from the core dump:
---------
#0 0x00007fd91a567c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd91a56b028 in __GI_abort () at abort.c:89
#2 0x000000000044e981 in del_lump (msg=0x7fd917c5ed60 <lreq>, offset=1625, len=20,
type=HDR_OTHER_T) at data_lump.c:293
#3 0x00007fd91676297f in subst_f (msg=0x7fd917c5ed60 <lreq>, subst=0x7fd919e5ccc0
"\320\356\345\031\331\177", ignored=0x0) at textops.c:681
#4 0x000000000041e1dd in do_action (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1059
#5 0x000000000042a917 in run_actions (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#6 0x000000000042af94 in run_actions_safe (h=0x7ffd364ba5b0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1613
#7 0x0000000000549972 in rval_get_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60 <lreq>,
i=0x7ffd364b8b58, rv=0x7fd919e9ced8, cache=0x0) at rvalue.c:912
#8 0x000000000054dc7c in rval_expr_eval_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60
<lreq>, res=0x7ffd364b8b58, rve=0x7fd919e9ced0) at rvalue.c:1910
#9 0x000000000041dc52 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1029
#10 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#11 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9dad8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#12 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e973b8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#13 0x000000000041ab46 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95d70,
msg=0x7fd917c5ed60 <lreq>) at action.c:677
#14 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8f330,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#15 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95eb8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#16 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8c8e0,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#17 0x000000000042b07d in run_top_route (a=0x7fd919e8c8e0, msg=0x7fd917c5ed60
<lreq>, c=0x0) at action.c:1634
#18 0x00007fd917a0539a in t_uac_prepare (uac_r=0x7ffd364baf20, dst_req=0x7ffd364ba8f0,
dst_cell=0x7ffd364ba8f8) at uac.c:391
#19 0x00007fd917a07945 in t_uac_with_ids (uac_r=0x7ffd364baf20, ret_index=0x0,
ret_label=0x0) at uac.c:599
#20 0x00007fd917a07918 in t_uac (uac_r=0x7ffd364baf20) at uac.c:584
#21 0x00007fd917a09b9e in req_within (uac_r=0x7ffd364baf20) at uac.c:802
#22 0x00007fd914674052 in send_notify_request (subs=0x7ffd364bb200, watcher_subs=0x0,
n_body=0x0, force_null_body=0) at notify.c:1600
#23 0x00007fd9146754b3 in notify (subs=0x7ffd364bb200, watcher_subs=0x0, n_body=0x0,
force_null_body=0) at notify.c:1690
#24 0x00007fd9146cec6a in update_subscription (msg=0x7fd919efeb70, subs=0x7ffd364bb200,
to_tag_gen=1, sent_reply=0x7ffd364bb16c) at subscribe.c:697
#25 0x00007fd9146d45b7 in handle_subscribe (msg=0x7fd919efeb70, watcher_user=...,
watcher_domain=...) at subscribe.c:1057
#26 0x00007fd9146d08b0 in handle_subscribe0 (msg=0x7fd919efeb70) at subscribe.c:816
#27 0x000000000041e155 in do_action (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1053
#28 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1548
#29 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19650,
msg=0x7fd919efeb70) at action.c:1048
#30 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e02638,
msg=0x7fd919efeb70) at action.c:1548
#31 0x000000000041e0c4 in do_action (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1044
#32 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1548
#33 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19b08,
msg=0x7fd919efeb70) at action.c:1048
#34 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919df99d8,
msg=0x7fd919efeb70) at action.c:1548
#35 0x000000000041ab46 in do_action (h=0x7ffd364bd680, a=0x7fd919dbe960,
msg=0x7fd919efeb70) at action.c:677
#36 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919daedf8,
msg=0x7fd919efeb70) at action.c:1548
#37 0x000000000042b07d in run_top_route (a=0x7fd919daedf8, msg=0x7fd919efeb70, c=0x0) at
action.c:1634
#38 0x000000000050f678 in receive_msg (buf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on><sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml, application/pidf-diff+xml,"...,
len=1284, rcv_info=0x7fd8f2c6cb50) at receive.c:196
#39 0x00000000005fb99e in receive_tcp_msg (tcpbuf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on><sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml, application/pidf-diff+xml,"...,
len=1284, rcv_info=0x7fd8f2c6cb50, con=0x7fd8f2c6cb38) at tcp_read.c:1207
#40 0x00000000005fd48c in tcp_read_req (con=0x7fd8f2c6cb38, bytes_read=0x7ffd364bdb7c,
read_flags=0x7ffd364bdb84) at tcp_read.c:1411
#41 0x0000000000600aee in handle_io (fm=0x7fd919f07d80, events=1, idx=-1) at
tcp_read.c:1643
#42 0x00000000005f3ab8 in io_wait_loop_epoll (h=0xa3d3e0 <io_w>, t=2, repeat=0) at
io_wait.h:1061
#43 0x0000000000601cf7 in tcp_receive_loop (unix_sock=57) at tcp_read.c:1755
#44 0x00000000005ea235 in tcp_init_children () at tcp_main.c:4788
#45 0x00000000004ab048 in main_loop () at main.c:1675
#46 0x00000000004b10c2 in main (argc=13, argv=0x7ffd364be258) at main.c:2566
---------
Thanks very much.
-Cody
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
8 Aug
8 Aug
4:16 a.m.
I pushed a patch to add 0 at the end of tm uac buffer -- somehow memory
was allocated to accommodate it but not set.
Thanks for troubleshooting this in detail ...
Cheers,
Daniel
On 07.08.17 22:13, Cody Herzog wrote:
Yes. The crash was happening when using subst() or subst_body() inside
of event_route[tm:local-request].
*From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Sent:* Monday, August 7, 2017 12:55 PM
*To:* Cody Herzog <CHerzog(a)IntouchHealth.com>om>; Kamailio (SER) - Users
Mailing List <sr-users(a)lists.kamailio.org>
*Subject:* Re: [SR-Users] Core crash in del_lump() "offset exceeds
message size" when using subst() inside tm:local-request route.
Were you using the subst() in event_route[...]?
Cheers,
Daniel
On 07.08.17 19:47, Cody Herzog wrote:
Thanks for the response.
I will make a pull request soon.
--
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com
are you using msg_apply_changes()?
No. I tried to use it, but it does not seem to be allowed in the
tm:local-request route.
do you receive traffic over tcp/tls?
Yes. We are using TLS.
Thanks.
-Cody
*From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Sent:* Monday, August 7, 2017 12:15 AM
*To:* Cody Herzog <CHerzog(a)IntouchHealth.com>
<mailto:CHerzog@IntouchHealth.com>; Kamailio (SER) - Users Mailing
List <sr-users(a)lists.kamailio.org>
<mailto:sr-users@lists.kamailio.org>
*Subject:* Re: [SR-Users] Core crash in del_lump() "offset exceeds
message size" when using subst() inside tm:local-request route.
Hello,
can you make a pull request with the patch you did to terminate
the buffer with '\0'? It is a valid fix if there are situations
when the buffer is not zero-terminated.
To figure out when it is not zero-terminated, if you don't know
already, in order to analyze if it has impact on other part of
code, few questions:
- are you using msg_apply_changes()?
- do you receive traffic over tcp/tls?
Cheers,
Daniel
On 31.07.17 19:43, Cody Herzog wrote:
Hello.
We are running Kamailio 4.3.5 on Ubuntu 14.04 LTS 64-bit.
Sorry, I should have included that information in my first email.
I have done more debugging, and I believe the problem is
caused by the subst_run() function being used on a message
buffer which is not null terminated at msg->len.
The subst_f() and subst_body_f() functions both call
subst_run(), and most of the time, everything seems fine.
However, in our use case, the msg->buf does not always seem to
be null terminated at msg->len.
In other words: msg->buf[msg->len] != '\0'
It seems there is sometimes some extra data in the buffer
beyond msg->len, and subst_run() may abort() if it ever moves
beyond msg->len.
I found a few places in textops.c where a buffer is being
temporarily null terminated in order to perform a regexec ()
search, then restored later, as in:
body = hf->body;
c = body.s[body.len];
body.s[body.len] = '\0';
ret = regexec((regex_t*) re, body.s, 1, &pmatch, 0);
body.s[body.len] = c;
I've tried making the same type of change to subst_f() and
subst_body_f(), and now I am not seeing the crash.
If I guarantee that msg->buf[msg->len] == '\0' when
subst_run() is called, then everything seems to be OK.
What do you think? Is that a valid change to make?
Let me know if you'd like me to make a pull request on the
main branch with those changes.
Thanks very much.
-Cody
*From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Sent:* Sunday, July 30, 2017 11:12 PM
*To:* Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>
<mailto:sr-users@lists.kamailio.org>; Cody Herzog
<CHerzog(a)IntouchHealth.com> <mailto:CHerzog@IntouchHealth.com>
*Subject:* Re: [SR-Users] Core crash in del_lump() "offset
exceeds message size" when using subst() inside
tm:local-request route.
Hello,
what's the version of Kamailio and the operating system you
run on?
Cheers,
Daniel
On 28.07.17 23:53, Cody Herzog wrote:
Hello.
Inside the tm:local-request route, I am trying to modify
the body of outgoing NOTIFY messages using the
subst_body() function.
Things seem to work fine most of the time. The NOTIFY
messages are correctly modified. However, I will very
rarely hit a crash.
I can force the crash to happen more quickly by stress
testing with thousands of NOTIFY messages being modified.
Is it safe to call subst() or subst_body() inside the
tm:local-request route? If not, is there another way I can
modify the outgoing NOTIFY messages safely?
Note that I am also calling append_hf() inside the same
tm:local-request route, but that has been working for a
very long time without causing any problems. Could there
be some kind of bad interaction between append_hf() and
subst_body()?
Strangely, it seems like using subst() makes the crash
more likely to happen. For some reason, subst_body() seems
to be more robust.
Here are the logs I see:
---------
Jul 28 10:57:40 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2476]: CRITICAL: <core>
[data_lump.c:292]: del_lump(): offset exceeds message size
(1625 > 1194) aborting...
Jul 28 10:57:43 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2481]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 53
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:731]: handle_sigs(): child process 2476 exited by
a signal 6
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:734]: handle_sigs(): core was generated
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST
/usr/local/sbin/kamailio[2367]: INFO: <core> [main.c:756]:
handle_sigs(): terminating due to SIGCHLD
---------
Here is a GDB back trace from the core dump:
---------
#0 0x00007fd91a567c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd91a56b028 in __GI_abort () at abort.c:89
#2 0x000000000044e981 in del_lump (msg=0x7fd917c5ed60
<lreq>, offset=1625, len=20, type=HDR_OTHER_T) at
data_lump.c:293
#3 0x00007fd91676297f in subst_f (msg=0x7fd917c5ed60
<lreq>, subst=0x7fd919e5ccc0 "\320\356\345\031\331\177",
ignored=0x0) at textops.c:681
#4 0x000000000041e1dd in do_action (h=0x7ffd364b87c0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1059
#5 0x000000000042a917 in run_actions (h=0x7ffd364b87c0,
a=0x7fd919e9b438, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#6 0x000000000042af94 in run_actions_safe
(h=0x7ffd364ba5b0, a=0x7fd919e9b438, msg=0x7fd917c5ed60
<lreq>) at action.c:1613
#7 0x0000000000549972 in rval_get_int (h=0x7ffd364ba5b0,
msg=0x7fd917c5ed60 <lreq>, i=0x7ffd364b8b58,
rv=0x7fd919e9ced8, cache=0x0) at rvalue.c:912
#8 0x000000000054dc7c in rval_expr_eval_int
(h=0x7ffd364ba5b0, msg=0x7fd917c5ed60 <lreq>,
res=0x7ffd364b8b58, rve=0x7fd919e9ced0) at rvalue.c:1910
#9 0x000000000041dc52 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e9d990, msg=0x7fd917c5ed60 <lreq>) at action.c:1029
#10 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e9d990, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#11 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e9dad8, msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#12 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e973b8, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#13 0x000000000041ab46 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e95d70, msg=0x7fd917c5ed60 <lreq>) at action.c:677
#14 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e8f330, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#15 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0,
a=0x7fd919e95eb8, msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#16 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0,
a=0x7fd919e8c8e0, msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#17 0x000000000042b07d in run_top_route (a=0x7fd919e8c8e0,
msg=0x7fd917c5ed60 <lreq>, c=0x0) at action.c:1634
#18 0x00007fd917a0539a in t_uac_prepare
(uac_r=0x7ffd364baf20, dst_req=0x7ffd364ba8f0,
dst_cell=0x7ffd364ba8f8) at uac.c:391
#19 0x00007fd917a07945 in t_uac_with_ids
(uac_r=0x7ffd364baf20, ret_index=0x0, ret_label=0x0) at
uac.c:599
#20 0x00007fd917a07918 in t_uac (uac_r=0x7ffd364baf20) at
uac.c:584
#21 0x00007fd917a09b9e in req_within
(uac_r=0x7ffd364baf20) at uac.c:802
#22 0x00007fd914674052 in send_notify_request
(subs=0x7ffd364bb200, watcher_subs=0x0, n_body=0x0,
force_null_body=0) at notify.c:1600
#23 0x00007fd9146754b3 in notify (subs=0x7ffd364bb200,
watcher_subs=0x0, n_body=0x0, force_null_body=0) at
notify.c:1690
#24 0x00007fd9146cec6a in update_subscription
(msg=0x7fd919efeb70, subs=0x7ffd364bb200, to_tag_gen=1,
sent_reply=0x7ffd364bb16c) at subscribe.c:697
#25 0x00007fd9146d45b7 in handle_subscribe
(msg=0x7fd919efeb70, watcher_user=..., watcher_domain=...)
at subscribe.c:1057
#26 0x00007fd9146d08b0 in handle_subscribe0
(msg=0x7fd919efeb70) at subscribe.c:816
#27 0x000000000041e155 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19508, msg=0x7fd919efeb70) at action.c:1053
#28 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e19508, msg=0x7fd919efeb70) at action.c:1548
#29 0x000000000041e112 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19650, msg=0x7fd919efeb70) at action.c:1048
#30 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e02638, msg=0x7fd919efeb70) at action.c:1548
#31 0x000000000041e0c4 in do_action (h=0x7ffd364bd680,
a=0x7fd919e199c0, msg=0x7fd919efeb70) at action.c:1044
#32 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919e199c0, msg=0x7fd919efeb70) at action.c:1548
#33 0x000000000041e112 in do_action (h=0x7ffd364bd680,
a=0x7fd919e19b08, msg=0x7fd919efeb70) at action.c:1048
#34 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919df99d8, msg=0x7fd919efeb70) at action.c:1548
#35 0x000000000041ab46 in do_action (h=0x7ffd364bd680,
a=0x7fd919dbe960, msg=0x7fd919efeb70) at action.c:677
#36 0x000000000042a917 in run_actions (h=0x7ffd364bd680,
a=0x7fd919daedf8, msg=0x7fd919efeb70) at action.c:1548
#37 0x000000000042b07d in run_top_route (a=0x7fd919daedf8,
msg=0x7fd919efeb70, c=0x0) at action.c:1634
#38 0x000000000050f678 in receive_msg (buf=0x7fd8f2c6ce28
"SUBSCRIBE sip:endpoint-1720@intouchstaging.com
SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml,
application/pidf-diff+xml,"..., len=1284,
rcv_info=0x7fd8f2c6cb50) at receive.c:196
#39 0x00000000005fb99e in receive_tcp_msg
(tcpbuf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com
SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on>
<sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml,
application/pidf-diff+xml,"..., len=1284,
rcv_info=0x7fd8f2c6cb50, con=0x7fd8f2c6cb38) at
tcp_read.c:1207
#40 0x00000000005fd48c in tcp_read_req
(con=0x7fd8f2c6cb38, bytes_read=0x7ffd364bdb7c,
read_flags=0x7ffd364bdb84) at tcp_read.c:1411
#41 0x0000000000600aee in handle_io (fm=0x7fd919f07d80,
events=1, idx=-1) at tcp_read.c:1643
#42 0x00000000005f3ab8 in io_wait_loop_epoll (h=0xa3d3e0
<io_w>, t=2, repeat=0) at io_wait.h:1061
#43 0x0000000000601cf7 in tcp_receive_loop (unix_sock=57)
at tcp_read.c:1755
#44 0x00000000005ea235 in tcp_init_children () at
tcp_main.c:4788
#45 0x00000000004ab048 in main_loop () at main.c:1675
#46 0x00000000004b10c2 in main (argc=13,
argv=0x7ffd364be258) at main.c:2566
---------
Thanks very much.
-Cody
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
<mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com <http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com
<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com <http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com
<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com <http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com <http://www.kamailioworld.com>
1:47 p.m.
My pleasure. Thanks very much.
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Tuesday, August 8, 2017 1:17 AM
To: Cody Herzog <CHerzog(a)IntouchHealth.com>om>; Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
I pushed a patch to add 0 at the end of tm uac buffer -- somehow memory was allocated to
accommodate it but not set.
Thanks for troubleshooting this in detail ...
Cheers,
Daniel
On 07.08.17 22:13, Cody Herzog wrote:
Yes. The crash was happening when using subst() or subst_body() inside of
event_route[tm:local-request].
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Monday, August 7, 2017 12:55 PM
To: Cody Herzog <CHerzog@IntouchHealth.com><mailto:CHerzog@IntouchHealth.com>;
Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
Were you using the subst() in event_route[...]?
Cheers,
Daniel
On 07.08.17 19:47, Cody Herzog wrote:
Thanks for the response.
I will make a pull request soon.
are you using msg_apply_changes()?
No. I tried to use it, but it does not seem to be allowed in the tm:local-request route.
do you receive traffic over tcp/tls?
Yes. We are using TLS.
Thanks.
-Cody
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Monday, August 7, 2017 12:15 AM
To: Cody Herzog <CHerzog@IntouchHealth.com><mailto:CHerzog@IntouchHealth.com>;
Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
Hello,
can you make a pull request with the patch you did to terminate the buffer with
'\0'? It is a valid fix if there are situations when the buffer is not
zero-terminated.
To figure out when it is not zero-terminated, if you don't know already, in order to
analyze if it has impact on other part of code, few questions:
- are you using msg_apply_changes()?
- do you receive traffic over tcp/tls?
Cheers,
Daniel
On 31.07.17 19:43, Cody Herzog wrote:
Hello.
We are running Kamailio 4.3.5 on Ubuntu 14.04 LTS 64-bit.
Sorry, I should have included that information in my first email.
I have done more debugging, and I believe the problem is caused by the subst_run()
function being used on a message buffer which is not null terminated at msg->len.
The subst_f() and subst_body_f() functions both call subst_run(), and most of the time,
everything seems fine. However, in our use case, the msg->buf does not always seem to
be null terminated at msg->len.
In other words: msg->buf[msg->len] != '\0'
It seems there is sometimes some extra data in the buffer beyond msg->len, and
subst_run() may abort() if it ever moves beyond msg->len.
I found a few places in textops.c where a buffer is being temporarily null terminated in
order to perform a regexec () search, then restored later, as in:
body = hf->body;
c = body.s[body.len];
body.s[body.len] = '\0';
ret = regexec((regex_t*) re, body.s, 1, &pmatch, 0);
body.s[body.len] = c;
I've tried making the same type of change to subst_f() and subst_body_f(), and now I
am not seeing the crash.
If I guarantee that msg->buf[msg->len] == '\0' when subst_run() is called,
then everything seems to be OK.
What do you think? Is that a valid change to make?
Let me know if you'd like me to make a pull request on the main branch with those
changes.
Thanks very much.
-Cody
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Sunday, July 30, 2017 11:12 PM
To: Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>; Cody Herzog
<CHerzog@IntouchHealth.com><mailto:CHerzog@IntouchHealth.com>
Subject: Re: [SR-Users] Core crash in del_lump() "offset exceeds message size"
when using subst() inside tm:local-request route.
Hello,
what's the version of Kamailio and the operating system you run on?
Cheers,
Daniel
On 28.07.17 23:53, Cody Herzog wrote:
Hello.
Inside the tm:local-request route, I am trying to modify the body of outgoing NOTIFY
messages using the subst_body() function.
Things seem to work fine most of the time. The NOTIFY messages are correctly modified.
However, I will very rarely hit a crash.
I can force the crash to happen more quickly by stress testing with thousands of NOTIFY
messages being modified.
Is it safe to call subst() or subst_body() inside the tm:local-request route? If not, is
there another way I can modify the outgoing NOTIFY messages safely?
Note that I am also calling append_hf() inside the same tm:local-request route, but that
has been working for a very long time without causing any problems. Could there be some
kind of bad interaction between append_hf() and subst_body()?
Strangely, it seems like using subst() makes the crash more likely to happen. For some
reason, subst_body() seems to be more robust.
Here are the logs I see:
---------
Jul 28 10:57:40 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2476]: CRITICAL: <core>
[data_lump.c:292]: del_lump(): offset exceeds message size (1625 > 1194) aborting...
Jul 28 10:57:43 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2481]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 53
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:731]: handle_sigs(): child process 2476 exited by a signal 6
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: ALERT: <core>
[main.c:734]: handle_sigs(): core was generated
Jul 28 10:57:44 SIPCOMM-VEGAS-TEST /usr/local/sbin/kamailio[2367]: INFO: <core>
[main.c:756]: handle_sigs(): terminating due to SIGCHLD
---------
Here is a GDB back trace from the core dump:
---------
#0 0x00007fd91a567c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd91a56b028 in __GI_abort () at abort.c:89
#2 0x000000000044e981 in del_lump (msg=0x7fd917c5ed60 <lreq>, offset=1625, len=20,
type=HDR_OTHER_T) at data_lump.c:293
#3 0x00007fd91676297f in subst_f (msg=0x7fd917c5ed60 <lreq>, subst=0x7fd919e5ccc0
"\320\356\345\031\331\177", ignored=0x0) at textops.c:681
#4 0x000000000041e1dd in do_action (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1059
#5 0x000000000042a917 in run_actions (h=0x7ffd364b87c0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#6 0x000000000042af94 in run_actions_safe (h=0x7ffd364ba5b0, a=0x7fd919e9b438,
msg=0x7fd917c5ed60 <lreq>) at action.c:1613
#7 0x0000000000549972 in rval_get_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60 <lreq>,
i=0x7ffd364b8b58, rv=0x7fd919e9ced8, cache=0x0) at rvalue.c:912
#8 0x000000000054dc7c in rval_expr_eval_int (h=0x7ffd364ba5b0, msg=0x7fd917c5ed60
<lreq>, res=0x7ffd364b8b58, rve=0x7fd919e9ced0) at rvalue.c:1910
#9 0x000000000041dc52 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1029
#10 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e9d990,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#11 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e9dad8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#12 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e973b8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#13 0x000000000041ab46 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95d70,
msg=0x7fd917c5ed60 <lreq>) at action.c:677
#14 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8f330,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#15 0x000000000041e0c4 in do_action (h=0x7ffd364ba5b0, a=0x7fd919e95eb8,
msg=0x7fd917c5ed60 <lreq>) at action.c:1044
#16 0x000000000042a917 in run_actions (h=0x7ffd364ba5b0, a=0x7fd919e8c8e0,
msg=0x7fd917c5ed60 <lreq>) at action.c:1548
#17 0x000000000042b07d in run_top_route (a=0x7fd919e8c8e0, msg=0x7fd917c5ed60
<lreq>, c=0x0) at action.c:1634
#18 0x00007fd917a0539a in t_uac_prepare (uac_r=0x7ffd364baf20, dst_req=0x7ffd364ba8f0,
dst_cell=0x7ffd364ba8f8) at uac.c:391
#19 0x00007fd917a07945 in t_uac_with_ids (uac_r=0x7ffd364baf20, ret_index=0x0,
ret_label=0x0) at uac.c:599
#20 0x00007fd917a07918 in t_uac (uac_r=0x7ffd364baf20) at uac.c:584
#21 0x00007fd917a09b9e in req_within (uac_r=0x7ffd364baf20) at uac.c:802
#22 0x00007fd914674052 in send_notify_request (subs=0x7ffd364bb200, watcher_subs=0x0,
n_body=0x0, force_null_body=0) at notify.c:1600
#23 0x00007fd9146754b3 in notify (subs=0x7ffd364bb200, watcher_subs=0x0, n_body=0x0,
force_null_body=0) at notify.c:1690
#24 0x00007fd9146cec6a in update_subscription (msg=0x7fd919efeb70, subs=0x7ffd364bb200,
to_tag_gen=1, sent_reply=0x7ffd364bb16c) at subscribe.c:697
#25 0x00007fd9146d45b7 in handle_subscribe (msg=0x7fd919efeb70, watcher_user=...,
watcher_domain=...) at subscribe.c:1057
#26 0x00007fd9146d08b0 in handle_subscribe0 (msg=0x7fd919efeb70) at subscribe.c:816
#27 0x000000000041e155 in do_action (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1053
#28 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e19508,
msg=0x7fd919efeb70) at action.c:1548
#29 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19650,
msg=0x7fd919efeb70) at action.c:1048
#30 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e02638,
msg=0x7fd919efeb70) at action.c:1548
#31 0x000000000041e0c4 in do_action (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1044
#32 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919e199c0,
msg=0x7fd919efeb70) at action.c:1548
#33 0x000000000041e112 in do_action (h=0x7ffd364bd680, a=0x7fd919e19b08,
msg=0x7fd919efeb70) at action.c:1048
#34 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919df99d8,
msg=0x7fd919efeb70) at action.c:1548
#35 0x000000000041ab46 in do_action (h=0x7ffd364bd680, a=0x7fd919dbe960,
msg=0x7fd919efeb70) at action.c:677
#36 0x000000000042a917 in run_actions (h=0x7ffd364bd680, a=0x7fd919daedf8,
msg=0x7fd919efeb70) at action.c:1548
#37 0x000000000042b07d in run_top_route (a=0x7fd919daedf8, msg=0x7fd919efeb70, c=0x0) at
action.c:1634
#38 0x000000000050f678 in receive_msg (buf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on><sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml, application/pidf-diff+xml,"...,
len=1284, rcv_info=0x7fd8f2c6cb50) at receive.c:196
#39 0x00000000005fb99e in receive_tcp_msg (tcpbuf=0x7fd8f2c6ce28 "SUBSCRIBE
sip:endpoint-1720@intouchstaging.com SIP/2.0\r\nRecord-Route:
<sip:64.64.203.109:443;transport=tls;lr=on><sip:64.64.203.109:443;transport=tls;lr=on>\r\nAccept:
application/dialog-info+xml, application/pidf+xml, application/pidf-diff+xml,"...,
len=1284, rcv_info=0x7fd8f2c6cb50, con=0x7fd8f2c6cb38) at tcp_read.c:1207
#40 0x00000000005fd48c in tcp_read_req (con=0x7fd8f2c6cb38, bytes_read=0x7ffd364bdb7c,
read_flags=0x7ffd364bdb84) at tcp_read.c:1411
#41 0x0000000000600aee in handle_io (fm=0x7fd919f07d80, events=1, idx=-1) at
tcp_read.c:1643
#42 0x00000000005f3ab8 in io_wait_loop_epoll (h=0xa3d3e0 <io_w>, t=2, repeat=0) at
io_wait.h:1061
#43 0x0000000000601cf7 in tcp_receive_loop (unix_sock=57) at tcp_read.c:1755
#44 0x00000000005ea235 in tcp_init_children () at tcp_main.c:4788
#45 0x00000000004ab048 in main_loop () at main.c:1675
#46 0x00000000004b10c2 in main (argc=13, argv=0x7ffd364be258) at main.c:2566
---------
Thanks very much.
-Cody
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - www.asipto.com<http://www.asipto.com>
Kamailio World Conference - www.kamailioworld.com<http://www.kamailioworld.com>
2701
days inactive
2712
days old
8 comments
2 participants
participants (2)
-
Cody Herzog -
Daniel-Constantin Mierla