Here it is: ``` (gdb) frame 0 #0 0x00007f3b4fd779da in update_dialog_vars_dbinfo (cell=0x7f3b3c3aa958, var=0x3433) at dlg_db_handler.c:651 651 SET_STR_VALUE(values+2, var->key); (gdb) p *cell $1 = {ref = 1, next = 0x0, prev = 0x0, h_id = 10067, h_entry = 3273, state = 1, lifetime = 10800, init_ts = 1440488615, start_ts = 0, dflags = 0, iflags = 0, sflags = 0, toroute = 0, toroute_name = { s = 0x0, len = 0}, from_rr_nb = 0, tl = {next = 0x0, prev = 0x0, timeout = 0}, callid = { s = 0x7f3b3c3aaab8 "256700-13146@192.168.0.186sip:user1@test1.comsip:09011112222@192.168.2.185:5060sip:09011112222@192.168.2.186:5030168.2.186:5030r\370\001", len = 26}, from_uri = { s = 0x7f3b3c3aaad2 "sip:user1@test1.comsip:09011112222@192.168.2.185:5060sip:09011112222@192.168.2.186:5030168.2.186:5030r\370\001", len = 19}, to_uri = { s = 0x7f3b3c3aaae5 "sip:09011112222@192.168.2.185:5060sip:09011112222@192.168.2.186:5030168.2.186:5030r\370\001", len = 34}, req_uri = { s = 0x7f3b3c3aab07 "sip:09011112222@192.168.2.186:5030168.2.186:5030r\370\001", len = 34}, tag = {{s = 0x7f3b3c8a1ea0 "13146SIPpTag00256700sip:user1@192.168.2.50:5010", len = 20}, {s = 0x0, len = 0}}, cseq = {{s = 0x7f3b3c1e85c0 "802r1", len = 3}, {s = 0x0, len = 0}}, route_set = {{s = 0x0, len = 0}, {s = 0x0, len = 0}}, contact = {{s = 0x7f3b3c8a1eb4 "sip:user1@192.168.2.50:5010", len = 27}, {s = 0x0, len = 0}}, bind_addr = {0x7f3b52e24730, 0x0}, cbs = {first = 0x7f3b3c536060, types = 41172}, profile_links = 0x7f3b3c43ded8, vars = 0x7f3b3c1f7590} (gdb) ``` --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-134562206

``` (gdb) p *cell->vars $5 = {key = {s = 0x7f3b3c43e7c8 "answer_time", len = 11}, value = { s = 0x7f3b3c553ea0 "1440488615.120", len = 14}, vflags = 0, next = 0x7f3b3c35c678} (gdb) set $p=cell->vars->next (gdb) while ($p != 0) $6 = {key = {s = 0x7f3b3c4064f0 "calling_number", len = 14}, value = { s = 0x7f3b3c5950e0 "0312341234mber", len = 10}, vflags = 0, next = 0x7f3b3c8e09e8} $7 = {key = {s = 0x7f3b3c7f4be8 "destination", len = 11}, value = { s = 0x7f3b3c2d0710 "09011112222086", len = 11}, vflags = 0, next = 0x7f3b3c22fc80} $8 = {key = {s = 0x7f3b3c407cc0 "caller_username", len = 15}, value = { s = 0x7f3b3c2e5f68 "user1\002", len = 5}, vflags = 0, next = 0x7f3b3c1fd168} $9 = {key = {s = 0x7f3b3c1be160 "caller_domain", len = 13}, value = { s = 0x7f3b3c1baf28 "test1.comon", len = 9}, vflags = 0, next = 0x7f3b3c4984f8} $10 = {key = {s = 0x7f3b3c4ddd70 "start_time", len = 10}, value = { s = 0x7f3b3c331790 "1440488616", len = 10}, vflags = 0, next = 0x0} ``` --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-134565668

There are errors in the log, but not for the callid mentioned in the bt. Here is the full content of kamailio.log: ``` [root@lab002185-flip-server ~]# cat /var/log/kamailio/kamailio.log 2015-08-25T09:27:02.220867+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[911]: WARNING: dispatcher [dispatch.c:792]: ds_load_db(): no dispatching data in the db -- empty destination set 2015-08-25T09:33:02.458093+09:00 lab002185-flip-server kamailio: WARNING: <core> [socket_info.c:1371]: fix_hostname(): could not rev. resolve 192.168.2.185 2015-08-25T09:33:02.460171+09:00 lab002185-flip-server kamailio: WARNING: <core> [socket_info.c:1371]: fix_hostname(): could not rev. resolve 192.168.2.185 2015-08-25T09:33:02.461934+09:00 lab002185-flip-server kamailio: WARNING: <core> [socket_info.c:1371]: fix_hostname(): could not rev. resolve 192.168.2.185 2015-08-25T09:33:02.463584+09:00 lab002185-flip-server kamailio: WARNING: <core> [socket_info.c:1371]: fix_hostname(): could not rev. resolve 192.168.2.185 2015-08-25T09:33:02.540595+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9236]: WARNING: tls [tls_init.c:651]: init_tls_h(): tls: openssl bug #1491 (crash/mem leaks on low memory) workaround enabled (on low memory tls operations will fail preemptively) with free memory thresholds 8912896 and 4456448 bytes 2015-08-25T09:33:02.545072+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9236]: ERROR: dispatcher [dispatcher.c:768]: ds_warn_fixup(): failover functions used, but required AVP parameters are NULL -- feature disabled 2015-08-25T09:33:02.545790+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9236]: ERROR: dispatcher [dispatcher.c:768]: ds_warn_fixup(): failover functions used, but required AVP parameters are NULL -- feature disabled 2015-08-25T09:33:02.545918+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9236]: ERROR: dispatcher [dispatcher.c:768]: ds_warn_fixup(): failover functions used, but required AVP parameters are NULL -- feature disabled 2015-08-25T09:33:02.545988+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9236]: ERROR: dispatcher [dispatcher.c:768]: ds_warn_fixup(): failover functions used, but required AVP parameters are NULL -- feature disabled 2015-08-25T09:33:02.546037+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9236]: ERROR: dispatcher [dispatcher.c:768]: ds_warn_fixup(): failover functions used, but required AVP parameters are NULL -- feature disabled 2015-08-25T14:42:46.801070+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9253]: CRITICAL: dialog [dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg tl=0x7f3b3c3d75e0 tl->next=(nil) tl->prev=(nil) 2015-08-25T14:42:46.803130+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9253]: ERROR: dialog [dlg_handlers.c:1308]: dlg_onroute(): failed to update dialog lifetime 2015-08-25T14:42:56.792419+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9253]: WARNING: dialog [dlg_handlers.c:1279]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f3b3c3d7588 [1631:8150] with clid &#39;185161-13161(a)192.168.2.186&#39; and tags '13161SIPpTag00185161' '13130SIPpTag01370270' 2015-08-25T16:12:35.277204+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9252]: CRITICAL: dialog [dlg_hash.c:861]: log_next_state_dlg(): bogus event 2 in state 3 for dlg 0x7f3b3c297170 [1393:11970] with clid &#39;238421-13146(a)192.168.0.186&#39; and tags '13146SIPpTag00238421' '13130SIPpTag01476964' 2015-08-25T16:43:40.709136+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9277]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 12 2015-08-25T16:43:40.709177+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9236]: ALERT: <core> [main.c:728]: handle_sigs(): child process 9251 exited by a signal 11 2015-08-25T16:43:40.709187+09:00 lab002185-flip-server /usr/local/src/git/kamailio-4.3/kamailio[9236]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated ``` --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-134574244

Yes. Only one core file. I was running a load test with sipp. When kamailio stopped replying, i found the core file. So I don't think it got overwritten unless it is possible for two processes to segfault at the same time and dump to the same file name. ``` (gdb) p process_no $1 = 1 (gdb) p pt[process_no] $2 = {pid = 9251, unix_sock = 12, idx = -1, desc = "udp receiver child=0 sock=192.168.2.185:5060", '\000' <repeats 83 times>} ``` I was testing on a low RAM virtual machine (512MB) with kamailio started with 256 MB of shared mem. I will start another VM with more RAM and load test kamailio there to try to recreate this: maybe lack of memory caused trouble in the generation of the core file. --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-134729443

OK, I am testing master too. I have started the load tests. I should have results within 24 hours. However, latest commit 8d8e1b957d9461d80b15b8e52464322a4e2b5fb1 of master is logging this: 2015-08-27T14:36:07.034018+09:00 lab002187-flip-server /usr/local/src/git/kamailio-devel/kamailio[1958]: ERROR: db_mysql [km_dbase.c:121]: db_mysql_submit_query(): driver error on query: Column 'caller_contact' cannot be null 2015-08-27T14:36:07.034032+09:00 lab002187-flip-server /usr/local/src/git/kamailio-devel/kamailio[1958]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query 2015-08-27T14:36:07.034039+09:00 lab002187-flip-server /usr/local/src/git/kamailio-devel/kamailio[1958]: ERROR: dialog [dlg_db_handler.c:803]: update_dialog_dbinfo_unsafe(): could not add another dialog to db This doesn't cause the test calls to fail but of course, it means the dialog data will not be serialized to DB so it will not survive a restart/switchover. Here is the kamailio version info: [root@lab002187-flip-server ~]# kamctl fifo version Server:: kamailio (4.4.0-dev2 (x86_64/linux)) Build:: mi_core.c compiled on 08:08:00 Aug 27 2015 with gcc 4.4.7 Flags:: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES GIT:: 8d8e1b --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-135302557

Here's what I got using 3 VMs: 1) CentOS6.6, 1CPU, 1GB, kamailio 4.3 commit 48d78ce87e2f42bdc77e688a0f45ff725ea97de5: Crashed in update_dialog_vars_dbinfo 2) CentOS6.6, 1CPU, 2GB, kamailio 4.3 commit 48d78ce87e2f42bdc77e688a0f45ff725ea97de5 Crashed in update_dialog_vars_dbinfo generating 2 core files 3) CentOS6.7, 1CPU, 512GB, kamailio 4.4 commit 8d8e1b957d9461d80b15b8e52464322a4e2b5fb1 Crashed in update_dialog_vars_dbinfo In all crashes, var is always set to 0x3433. Here are full results applying the commands you passed before: ``` 1) CentOS6.6, 1CPU, 1GB, kamailio 4.3 commit 48d78ce87e2f42bdc77e688a0f45ff725ea97de5 Core was generated by `/usr/local/src/git/kamailio-4.3/kamailio -m 512 -M 16 -P /var/run/kamailio.pid'. Program terminated with signal 11, Segmentation fault. #0 0x00007f125df5b9da in update_dialog_vars_dbinfo (cell=0x7f123a855bf8, var=0x3433) at dlg_db_handler.c:651 651 SET_STR_VALUE(values+2, var->key); Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 db4-4.7.25-19.el6_6.x86_64 elfutils-libelf-0.161-3.el6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libacl-2.2.49-6.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcap-2.16-5.5.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 lm_sensors-libs-3.1.1-17.el6.x86_64 lua-5.1.4-4.1.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 net-snmp-libs-5.5-54.el6_7.1.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.19.1-3.el6_6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 nss-util-3.19.1-1.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 pcre-7.8-7.el6.x86_64 perl-libs-5.10.1-141.el6.x86_64 popt-1.13-7.el6.x86_64 rpm-libs-4.8.0-47.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 xz-libs-4.999.9-0.5.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f125df5b9da in update_dialog_vars_dbinfo (cell=0x7f123a855bf8, var=0x3433) at dlg_db_handler.c:651 #1 0x00007f125df5c511 in update_dialog_dbinfo_unsafe (cell=0x7f123a855bf8) at dlg_db_handler.c:718 #2 0x00007f125df5dafc in update_dialog_dbinfo (cell=0x7f123a855bf8) at dlg_db_handler.c:861 #3 0x00007f125df9d3dc in set_dlg_variable (dlg=0x7f123a855bf8, key=0x7f125ea9c760, val=0x7fffbab27030) at dlg_var.c:305 #4 0x00007f125e87aa9c in set_start_time (dialog=0x7f123a855bf8) at acc_cdr.c:523 #5 0x00007f125e880932 in cdr_on_create (dialog=0x7f123a855bf8, type=2, params=0x7f125e1be0a0) at acc_cdr.c:790 #6 0x00007f125df4c398 in run_create_callbacks (dlg=0x7f123a855bf8, msg=0x7f1261235dd8) at dlg_cb.c:230 #7 0x00007f125df66590 in dlg_new_dialog (req=0x7f1261235dd8, t=0x7f123ab55958, run_initial_cbs=1) at dlg_handlers.c:865 #8 0x00007f125df65006 in dlg_onreq (t=0x7f123ab55958, type=1, param=0x7f1260fd6760) at dlg_handlers.c:717 #9 0x00007f1260d1ac3a in run_reqin_callbacks_internal (hl=0x7f12399bb480, trans=0x7f123ab55958, params=0x7f1260fd6760) at t_hooks.c:360 #10 0x00007f1260d1ad48 in run_reqin_callbacks (trans=0x7f123ab55958, req=0x7f1261235dd8, code=1) at t_hooks.c:385 #11 0x00007f1260cdbb58 in build_cell (p_msg=0x7f1261235dd8) at h_table.c:356 #12 0x00007f1260d2c283 in new_t (p_msg=0x7f1261235dd8) at t_lookup.c:1269 #13 0x00007f1260d2d4b3 in t_newtran (p_msg=0x7f1261235dd8) at t_lookup.c:1409 #14 0x00007f1260d01e9c in t_relay_to (p_msg=0x7f1261235dd8, proxy=0x0, proto=0, replicate=0) at t_funcs.c:236 #15 0x00007f1260d40bbb in _w_t_relay_to (p_msg=0x7f1261235dd8, proxy=0x0, force_proto=0) at tm.c:1476 #16 0x00007f1260d41d20 in w_t_relay (p_msg=0x7f1261235dd8, _foo=0x0, _bar=0x0) at tm.c:1677 #17 0x000000000041ddad in do_action (h=0x7fffbab27e50, a=0x7f12610e5f60, msg=0x7f1261235dd8) at action.c:1053 #18 0x000000000042a4c3 in run_actions (h=0x7fffbab27e50, a=0x7f12610e5f60, msg=0x7f1261235dd8) at action.c:1548 #19 0x000000000042ab28 in run_actions_safe (h=0x7fffbab28a70, a=0x7f12610e5f60, msg=0x7f1261235dd8) at action.c:1613 #20 0x0000000000543c94 in rval_get_int (h=0x7fffbab28a70, msg=0x7f1261235dd8, i=0x7fffbab28328, rv=0x7f12610e65b0, cache=0x0) at rvalue.c:912 #21 0x0000000000547ecc in rval_expr_eval_int (h=0x7fffbab28a70, msg=0x7f1261235dd8, res=0x7fffbab28328, rve=0x7f12610e65a8) at rvalue.c:1906 #22 0x00000000005482c2 in rval_expr_eval_int (h=0x7fffbab28a70, msg=0x7f1261235dd8, res=0x7fffbab287b0, rve=0x7f12610e6ca8) at rvalue.c:1914 #23 0x000000000041d897 in do_action (h=0x7fffbab28a70, a=0x7f12610e6258, msg=0x7f1261235dd8) at action.c:1029 #24 0x000000000042a4c3 in run_actions (h=0x7fffbab28a70, a=0x7f1261029308, msg=0x7f1261235dd8) at action.c:1548 #25 0x000000000042abf0 in run_top_route (a=0x7f1261029308, msg=0x7f1261235dd8, c=0x0) at action.c:1634 #26 0x000000000050a938 in receive_msg ( buf=0xa6fee0 "INVITE sip:09011112222@192.168.2.183:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.0.184:5010;rport;branch=z9hG4bK-11917-63211-4\r\nFrom: \"user1\" <sip:user1@test1.com>;tag=11917SIPpTag0063211\r\nTo: <sip:0901111"..., len=944, rcv_info=0x7fffbab28d60) at receive.c:196 #27 0x000000000060a3ea in udp_rcv_loop () at udp_server.c:495 #28 0x00000000004a7ef7 in main_loop () at main.c:1573 #29 0x00000000004ae2d2 in main (argc=7, argv=0x7fffbab29198) at main.c:2533 (gdb) frame 0 #0 0x00007f125df5b9da in update_dialog_vars_dbinfo (cell=0x7f123a855bf8, var=0x3433) at dlg_db_handler.c:651 651 SET_STR_VALUE(values+2, var->key); (gdb) p *cell $1 = {ref = 1, next = 0x0, prev = 0x0, h_id = 6815, h_entry = 2610, state = 1, lifetime = 10800, init_ts = 1440663325, start_ts = 0, dflags = 0, iflags = 0, sflags = 0, toroute = 0, toroute_name = { s = 0x0, len = 0}, from_rr_nb = 0, tl = {next = 0x0, prev = 0x0, timeout = 0}, callid = { s = 0x7f123a855d58 "63211-11917@192.168.0.184sip:user1@test1.comsip:09011112222@192.168.2.183:5060sip:09011112222@192.168.2.184:5030\b\002", len = 25}, from_uri = { s = 0x7f123a855d71 "sip:user1@test1.comsip:09011112222@192.168.2.183:5060sip:09011112222@192.168.2.184:5030\b\002", len = 19}, to_uri = { s = 0x7f123a855d84 "sip:09011112222@192.168.2.183:5060sip:09011112222@192.168.2.184:5030\b\002", len = 34}, req_uri = {s = 0x7f123a855da6 "sip:09011112222@192.168.2.184:5030\b\002", len = 34}, tag = { {s = 0x7f123ac9d580 "11917SIPpTag0063211sip:user1@192.168.2.50:5010", len = 19}, {s = 0x0, len = 0}}, cseq = {{s = 0x7f123a4c7360 "802r1\n", len = 3}, {s = 0x0, len = 0}}, route_set = {{s = 0x0, len = 0}, {s = 0x0, len = 0}}, contact = {{s = 0x7f123ac9d593 "sip:user1@192.168.2.50:5010", len = 27}, {s = 0x0, len = 0}}, bind_addr = {0x7f1261008730, 0x0}, cbs = {first = 0x7f123a69fa08, types = 41172}, profile_links = 0x7f123a78f7d8, vars = 0x7f123a600c78} (gdb) p *cell->vars $2 = {key = {s = 0x7f123a4d1d98 "answer_time", len = 11}, value = {s = 0x7f123aa2bbd0 "1440663325.614", len = 14}, vflags = 0, next = 0x7f123a4ca800} (gdb) set $p=cell->vars->next (gdb) while ($p != 0) $3 = {key = {s = 0x7f123a5c7760 "calling_number", len = 14}, value = {s = 0x7f123a4a9cb8 "0312341234mber", len = 10}, vflags = 0, next = 0x7f123a565978} $4 = {key = {s = 0x7f123a3dd708 "destination", len = 11}, value = {s = 0x7f123a55aef8 "09011112222in", len = 11}, vflags = 0, next = 0x7f123a385dc8} $5 = {key = {s = 0x7f123ac919e0 "caller_username", len = 15}, value = {s = 0x7f123a7758b0 "user1", len = 5}, vflags = 0, next = 0x7f123a828b98} $6 = {key = {s = 0x7f123a59aa40 "caller_domain", len = 13}, value = {s = 0x7f123a48ca80 "test1.comon", len = 9}, vflags = 0, next = 0x7f123a585f38} $7 = {key = {s = 0x7f123a3c35f0 "start_time", len = 10}, value = {s = 0x7f123a874098 "2015-08-27 17:15:27", len = 19}, vflags = 0, next = 0x0} (gdb) p process_no $8 = 1 (gdb) p pt[process_no] $9 = {pid = 1947, unix_sock = 12, idx = -1, desc = "udp receiver child=0 sock=192.168.2.183:5060", '\000' <repeats 83 times>} ========================================================================== 2) CentOS6.6, 1CPU, 2GB, kamailio 4.3 commit 48d78ce87e2f42bdc77e688a0f45ff725ea97de5 Two core files generated: [root@lab002185-flip-server kamailio-4.3]# ls -l /core.534* -rw------- 1 root root 1095712768 8月 27 16:39 2015 /core.5341 -rw------- 1 root root 1095712768 8月 27 16:39 2015 /core.5342 [root@lab002185-flip-server kamailio-4.3]# diff /core.534* Binary files /core.5341 and /core.5342 differ /core.5341: Core was generated by `/usr/local/src/git/kamailio-4.3/kamailio -m 1024 -M 16 -P /var/run/kamailio.pid'. Program terminated with signal 11, Segmentation fault. #0 0x00007fa7cdfa59da in update_dialog_vars_dbinfo (cell=0x7fa78abf06d0, var=0x3433) at dlg_db_handler.c:651 651 SET_STR_VALUE(values+2, var->key); Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 db4-4.7.25-19.el6_6.x86_64 elfutils-libelf-0.161-3.el6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libacl-2.2.49-6.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcap-2.16-5.5.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 lm_sensors-libs-3.1.1-17.el6.x86_64 lua-5.1.4-4.1.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 net-snmp-libs-5.5-54.el6_7.1.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.19.1-3.el6_6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 nss-util-3.19.1-1.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 pcre-7.8-7.el6.x86_64 perl-libs-5.10.1-141.el6.x86_64 popt-1.13-7.el6.x86_64 rpm-libs-4.8.0-47.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 xz-libs-4.999.9-0.5.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fa7cdfa59da in update_dialog_vars_dbinfo (cell=0x7fa78abf06d0, var=0x3433) at dlg_db_handler.c:651 #1 0x00007fa7cdfa6511 in update_dialog_dbinfo_unsafe (cell=0x7fa78abf06d0) at dlg_db_handler.c:718 #2 0x00007fa7cdfa7afc in update_dialog_dbinfo (cell=0x7fa78abf06d0) at dlg_db_handler.c:861 #3 0x00007fa7cdfe73dc in set_dlg_variable (dlg=0x7fa78abf06d0, key=0x7fa7ceae6760, val=0x7ffc91e90910) at dlg_var.c:305 #4 0x00007fa7ce8c4a9c in set_start_time (dialog=0x7fa78abf06d0) at acc_cdr.c:523 #5 0x00007fa7ce8ca932 in cdr_on_create (dialog=0x7fa78abf06d0, type=2, params=0x7fa7ce2080a0) at acc_cdr.c:790 #6 0x00007fa7cdf96398 in run_create_callbacks (dlg=0x7fa78abf06d0, msg=0x7fa7d1280850) at dlg_cb.c:230 #7 0x00007fa7cdfb0590 in dlg_new_dialog (req=0x7fa7d1280850, t=0x7fa78a5e81e8, run_initial_cbs=1) at dlg_handlers.c:865 #8 0x00007fa7cdfaf006 in dlg_onreq (t=0x7fa78a5e81e8, type=1, param=0x7fa7d1020760) at dlg_handlers.c:717 #9 0x00007fa7d0d64c3a in run_reqin_callbacks_internal (hl=0x7fa789a19480, trans=0x7fa78a5e81e8, params=0x7fa7d1020760) at t_hooks.c:360 #10 0x00007fa7d0d64d48 in run_reqin_callbacks (trans=0x7fa78a5e81e8, req=0x7fa7d1280850, code=1) at t_hooks.c:385 #11 0x00007fa7d0d25b58 in build_cell (p_msg=0x7fa7d1280850) at h_table.c:356 #12 0x00007fa7d0d76283 in new_t (p_msg=0x7fa7d1280850) at t_lookup.c:1269 #13 0x00007fa7d0d774b3 in t_newtran (p_msg=0x7fa7d1280850) at t_lookup.c:1409 #14 0x00007fa7d0d4be9c in t_relay_to (p_msg=0x7fa7d1280850, proxy=0x0, proto=0, replicate=0) at t_funcs.c:236 #15 0x00007fa7d0d8abbb in _w_t_relay_to (p_msg=0x7fa7d1280850, proxy=0x0, force_proto=0) at tm.c:1476 #16 0x00007fa7d0d8bd20 in w_t_relay (p_msg=0x7fa7d1280850, _foo=0x0, _bar=0x0) at tm.c:1677 #17 0x000000000041ddad in do_action (h=0x7ffc91e91730, a=0x7fa7d112a9b8, msg=0x7fa7d1280850) at action.c:1053 #18 0x000000000042a4c3 in run_actions (h=0x7ffc91e91730, a=0x7fa7d112a9b8, msg=0x7fa7d1280850) at action.c:1548 #19 0x000000000042ab28 in run_actions_safe (h=0x7ffc91e92350, a=0x7fa7d112a9b8, msg=0x7fa7d1280850) at action.c:1613 #20 0x0000000000543c94 in rval_get_int (h=0x7ffc91e92350, msg=0x7fa7d1280850, i=0x7ffc91e91c08, rv=0x7fa7d112b008, cache=0x0) at rvalue.c:912 #21 0x0000000000547ecc in rval_expr_eval_int (h=0x7ffc91e92350, msg=0x7fa7d1280850, res=0x7ffc91e91c08, rve=0x7fa7d112b000) at rvalue.c:1906 #22 0x00000000005482c2 in rval_expr_eval_int (h=0x7ffc91e92350, msg=0x7fa7d1280850, res=0x7ffc91e92090, rve=0x7fa7d112b700) at rvalue.c:1914 #23 0x000000000041d897 in do_action (h=0x7ffc91e92350, a=0x7fa7d112acb0, msg=0x7fa7d1280850) at action.c:1029 #24 0x000000000042a4c3 in run_actions (h=0x7ffc91e92350, a=0x7fa7d1073308, msg=0x7fa7d1280850) at action.c:1548 #25 0x000000000042abf0 in run_top_route (a=0x7fa7d1073308, msg=0x7fa7d1280850, c=0x0) at action.c:1634 #26 0x000000000050a938 in receive_msg ( buf=0xa6fee0 "INVITE sip:09011112222@192.168.2.185:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.0.186:5010;rport;branch=z9hG4bK-16716-21895-4\r\nFrom: \"user1\" <sip:user1@test1.com>;tag=16716SIPpTag0021895\r\nTo: <sip:0901111"..., len=944, rcv_info=0x7ffc91e92640) at receive.c:196 #27 0x000000000060a3ea in udp_rcv_loop () at udp_server.c:495 #28 0x00000000004a7ef7 in main_loop () at main.c:1573 #29 0x00000000004ae2d2 in main (argc=7, argv=0x7ffc91e92a78) at main.c:2533 (gdb) frame 0 #0 0x00007fa7cdfa59da in update_dialog_vars_dbinfo (cell=0x7fa78abf06d0, var=0x3433) at dlg_db_handler.c:651 651 SET_STR_VALUE(values+2, var->key); (gdb) p *cell $1 = {ref = 1, next = 0x0, prev = 0x0, h_id = 2258, h_entry = 2559, state = 1, lifetime = 10800, init_ts = 1440661168, start_ts = 0, dflags = 0, iflags = 0, sflags = 0, toroute = 0, toroute_name = { s = 0x0, len = 0}, from_rr_nb = 0, tl = {next = 0x0, prev = 0x0, timeout = 0}, callid = { s = 0x7fa78abf0830 "21895-16716@192.168.0.186sip:user1@test1.comsip:09011112222@192.168.2.185:5060sip:09011112222@192.168.2.186:50300", len = 25}, from_uri = { s = 0x7fa78abf0849 "sip:user1@test1.comsip:09011112222@192.168.2.185:5060sip:09011112222@192.168.2.186:50300", len = 19}, to_uri = { s = 0x7fa78abf085c "sip:09011112222@192.168.2.185:5060sip:09011112222@192.168.2.186:50300", len = 34}, req_uri = {s = 0x7fa78abf087e "sip:09011112222@192.168.2.186:50300", len = 34}, tag = {{ s = 0x7fa78a5e0970 "16716SIPpTag0021895sip:user1@192.168.2.50:5010", len = 19}, {s = 0x0, len = 0}}, cseq = {{s = 0x7fa78a3fe4e0 "80294\017", len = 3}, {s = 0x0, len = 0}}, route_set = {{s = 0x0, len = 0}, {s = 0x0, len = 0}}, contact = {{s = 0x7fa78a5e0983 "sip:user1@192.168.2.50:5010", len = 27}, {s = 0x0, len = 0}}, bind_addr = {0x7fa7d1052730, 0x0}, cbs = {first = 0x7fa78a606488, types = 41172}, profile_links = 0x7fa78a9df160, vars = 0x7fa78a58ce28} (gdb) p *cell->vars $2 = {key = {s = 0x7fa78a7cd658 "answer_time", len = 11}, value = {s = 0x7fa78a6ecb38 "1440661168.396e", len = 14}, vflags = 0, next = 0x7fa78a8b65b8} (gdb) set $p=cell->vars->next (gdb) while ($p != 0) $3 = {key = {s = 0x7fa78a5a6280 "calling_number", len = 14}, value = {s = 0x7fa78a8fed10 "0312341234mber", len = 10}, vflags = 0, next = 0x7fa78a6e7dc8} $4 = {key = {s = 0x7fa78a96ec10 "destination", len = 11}, value = {s = 0x7fa78a48a060 "09011112222in", len = 11}, vflags = 0, next = 0x7fa78a4898f8} $5 = {key = {s = 0x7fa78a7c2690 "caller_username", len = 15}, value = {s = 0x7fa78a600420 "user1\r", len = 5}, vflags = 0, next = 0x7fa78a6b9840} $6 = {key = {s = 0x7fa78aa682b0 "caller_domain", len = 13}, value = {s = 0x7fa78abdfe48 "test1.comon", len = 9}, vflags = 0, next = 0x7fa78a8c6608} $7 = {key = {s = 0x7fa78a3abaa0 "start_time", len = 10}, value = {s = 0x7fa78a8e4a78 "1440661168ain", len = 10}, vflags = 0, next = 0x0} (gdb) p process_no $8 = 3 (gdb) p pt[process_no] $9 = {pid = 5341, unix_sock = 14, idx = -1, desc = "udp receiver child=2 sock=192.168.2.185:5060", '\000' <repeats 83 times>} /core.5342: Core was generated by `/usr/local/src/git/kamailio-4.3/kamailio -m 1024 -M 16 -P /var/run/kamailio.pid'. Program terminated with signal 11, Segmentation fault. #0 0x00007fa7cdfa59da in update_dialog_vars_dbinfo (cell=0x7fa78a7ec0a8, var=0x3433) at dlg_db_handler.c:651 651 SET_STR_VALUE(values+2, var->key); Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 db4-4.7.25-19.el6_6.x86_64 elfutils-libelf-0.161-3.el6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libacl-2.2.49-6.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcap-2.16-5.5.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 lm_sensors-libs-3.1.1-17.el6.x86_64 lua-5.1.4-4.1.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 net-snmp-libs-5.5-54.el6_7.1.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.19.1-3.el6_6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 nss-util-3.19.1-1.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 pcre-7.8-7.el6.x86_64 perl-libs-5.10.1-141.el6.x86_64 popt-1.13-7.el6.x86_64 rpm-libs-4.8.0-47.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 xz-libs-4.999.9-0.5.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fa7cdfa59da in update_dialog_vars_dbinfo (cell=0x7fa78a7ec0a8, var=0x3433) at dlg_db_handler.c:651 #1 0x00007fa7cdfa6511 in update_dialog_dbinfo_unsafe (cell=0x7fa78a7ec0a8) at dlg_db_handler.c:718 #2 0x00007fa7cdfa7afc in update_dialog_dbinfo (cell=0x7fa78a7ec0a8) at dlg_db_handler.c:861 #3 0x00007fa7cdfe73dc in set_dlg_variable (dlg=0x7fa78a7ec0a8, key=0x7fa7ceae6760, val=0x7ffc91e90910) at dlg_var.c:305 #4 0x00007fa7ce8c4a9c in set_start_time (dialog=0x7fa78a7ec0a8) at acc_cdr.c:523 #5 0x00007fa7ce8ca932 in cdr_on_create (dialog=0x7fa78a7ec0a8, type=2, params=0x7fa7ce2080a0) at acc_cdr.c:790 #6 0x00007fa7cdf96398 in run_create_callbacks (dlg=0x7fa78a7ec0a8, msg=0x7fa7d1280850) at dlg_cb.c:230 #7 0x00007fa7cdfb0590 in dlg_new_dialog (req=0x7fa7d1280850, t=0x7fa78a740c60, run_initial_cbs=1) at dlg_handlers.c:865 #8 0x00007fa7cdfaf006 in dlg_onreq (t=0x7fa78a740c60, type=1, param=0x7fa7d1020760) at dlg_handlers.c:717 #9 0x00007fa7d0d64c3a in run_reqin_callbacks_internal (hl=0x7fa789a19480, trans=0x7fa78a740c60, params=0x7fa7d1020760) at t_hooks.c:360 #10 0x00007fa7d0d64d48 in run_reqin_callbacks (trans=0x7fa78a740c60, req=0x7fa7d1280850, code=1) at t_hooks.c:385 #11 0x00007fa7d0d25b58 in build_cell (p_msg=0x7fa7d1280850) at h_table.c:356 #12 0x00007fa7d0d76283 in new_t (p_msg=0x7fa7d1280850) at t_lookup.c:1269 #13 0x00007fa7d0d774b3 in t_newtran (p_msg=0x7fa7d1280850) at t_lookup.c:1409 #14 0x00007fa7d0d4be9c in t_relay_to (p_msg=0x7fa7d1280850, proxy=0x0, proto=0, replicate=0) at t_funcs.c:236 #15 0x00007fa7d0d8abbb in _w_t_relay_to (p_msg=0x7fa7d1280850, proxy=0x0, force_proto=0) at tm.c:1476 #16 0x00007fa7d0d8bd20 in w_t_relay (p_msg=0x7fa7d1280850, _foo=0x0, _bar=0x0) at tm.c:1677 #17 0x000000000041ddad in do_action (h=0x7ffc91e91730, a=0x7fa7d112a9b8, msg=0x7fa7d1280850) at action.c:1053 #18 0x000000000042a4c3 in run_actions (h=0x7ffc91e91730, a=0x7fa7d112a9b8, msg=0x7fa7d1280850) at action.c:1548 #19 0x000000000042ab28 in run_actions_safe (h=0x7ffc91e92350, a=0x7fa7d112a9b8, msg=0x7fa7d1280850) at action.c:1613 #20 0x0000000000543c94 in rval_get_int (h=0x7ffc91e92350, msg=0x7fa7d1280850, i=0x7ffc91e91c08, rv=0x7fa7d112b008, cache=0x0) at rvalue.c:912 #21 0x0000000000547ecc in rval_expr_eval_int (h=0x7ffc91e92350, msg=0x7fa7d1280850, res=0x7ffc91e91c08, rve=0x7fa7d112b000) at rvalue.c:1906 #22 0x00000000005482c2 in rval_expr_eval_int (h=0x7ffc91e92350, msg=0x7fa7d1280850, res=0x7ffc91e92090, rve=0x7fa7d112b700) at rvalue.c:1914 #23 0x000000000041d897 in do_action (h=0x7ffc91e92350, a=0x7fa7d112acb0, msg=0x7fa7d1280850) at action.c:1029 #24 0x000000000042a4c3 in run_actions (h=0x7ffc91e92350, a=0x7fa7d1073308, msg=0x7fa7d1280850) at action.c:1548 #25 0x000000000042abf0 in run_top_route (a=0x7fa7d1073308, msg=0x7fa7d1280850, c=0x0) at action.c:1634 #26 0x000000000050a938 in receive_msg ( buf=0xa6fee0 "INVITE sip:09011112222@192.168.2.185:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.0.186:5010;rport;branch=z9hG4bK-16716-21896-4\r\nFrom: \"user1\" <sip:user1@test1.com>;tag=16716SIPpTag0021896\r\nTo: <sip:0901111"..., len=944, rcv_info=0x7ffc91e92640) at receive.c:196 #27 0x000000000060a3ea in udp_rcv_loop () at udp_server.c:495 #28 0x00000000004a7ef7 in main_loop () at main.c:1573 #29 0x00000000004ae2d2 in main (argc=7, argv=0x7ffc91e92a78) at main.c:2533 (gdb) frame 0 #0 0x00007fa7cdfa59da in update_dialog_vars_dbinfo (cell=0x7fa78a7ec0a8, var=0x3433) at dlg_db_handler.c:651 651 SET_STR_VALUE(values+2, var->key); (gdb) p *cell $1 = {ref = 1, next = 0x0, prev = 0x0, h_id = 3538, h_entry = 250, state = 1, lifetime = 10800, init_ts = 1440661168, start_ts = 0, dflags = 0, iflags = 0, sflags = 0, toroute = 0, toroute_name = { s = 0x0, len = 0}, from_rr_nb = 0, tl = {next = 0x0, prev = 0x0, timeout = 0}, callid = { s = 0x7fa78a7ec208 "21896-16716@192.168.0.186sip:user1@test1.comsip:09011112222@192.168.2.185:5060sip:09011112222@192.168.2.186:50300", len = 25}, from_uri = { s = 0x7fa78a7ec221 "sip:user1@test1.comsip:09011112222@192.168.2.185:5060sip:09011112222@192.168.2.186:50300", len = 19}, to_uri = { s = 0x7fa78a7ec234 "sip:09011112222@192.168.2.185:5060sip:09011112222@192.168.2.186:50300", len = 34}, req_uri = {s = 0x7fa78a7ec256 "sip:09011112222@192.168.2.186:50300", len = 34}, tag = {{ s = 0x7fa78a842160 "16716SIPpTag0021896sip:user1@192.168.2.50:5010", len = 19}, {s = 0x0, len = 0}}, cseq = {{s = 0x7fa78a8de008 "802r1\002", len = 3}, {s = 0x0, len = 0}}, route_set = {{s = 0x0, len = 0}, {s = 0x0, len = 0}}, contact = {{s = 0x7fa78a842173 "sip:user1@192.168.2.50:5010", len = 27}, {s = 0x0, len = 0}}, bind_addr = {0x7fa7d1052730, 0x0}, cbs = {first = 0x7fa78a6d6278, types = 41172}, profile_links = 0x7fa78a863710, vars = 0x7fa78a9c9c30} (gdb) p *cell->vars $2 = {key = {s = 0x7fa78a3f4fa8 "answer_time", len = 11}, value = {s = 0x7fa78a555708 "1440661168.396e", len = 14}, vflags = 0, next = 0x7fa78abe6620} (gdb) set $p=cell->vars->next (gdb) while ($p != 0) $3 = {key = {s = 0x7fa78abe7020 "calling_number", len = 14}, value = {s = 0x7fa78a825460 "0312341234mber", len = 10}, vflags = 0, next = 0x7fa78aa4cd70} $4 = {key = {s = 0x7fa78a533cb0 "destination", len = 11}, value = {s = 0x7fa78a574110 "09011112222in", len = 11}, vflags = 0, next = 0x7fa78a5163d0} $5 = {key = {s = 0x7fa78a4ecaf0 "caller_username", len = 15}, value = {s = 0x7fa78a465758 "user1\002", len = 5}, vflags = 0, next = 0x7fa78a6bf620} $6 = {key = {s = 0x7fa78aa68cb8 "caller_domain", len = 13}, value = {s = 0x7fa78a4e3d70 "test1.comon", len = 9}, vflags = 0, next = 0x7fa78a543e90} $7 = {key = {s = 0x7fa78a4d20f8 "start_time", len = 10}, value = {s = 0x7fa78a905530 "1440661168.649", len = 10}, vflags = 0, next = 0x0} (gdb) p process_no $8 = 4 (gdb) p pt[process_no] $9 = {pid = 5342, unix_sock = 15, idx = -1, desc = "udp receiver child=3 sock=192.168.2.185:5060", '\000' <repeats 83 times>} ========================================================================== 3) CentOS6.7, 1CPU, 512GB, kamailio 4.4 commit 8d8e1b957d9461d80b15b8e52464322a4e2b5fb1 Core was generated by `/usr/local/src/git/kamailio-devel/kamailio -m 256 -M 16 -P /var/run/kamailio.pi'. Program terminated with signal 11, Segmentation fault. #0 0x00007f711633b5ac in update_dialog_vars_dbinfo (cell=0x7f7100188c40, var=0x3433) at dlg_db_handler.c:652 652 SET_STR_VALUE(values+2, var->key); Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 db4-4.7.25-19.el6_6.x86_64 elfutils-libelf-0.161-3.el6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libacl-2.2.49-6.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcap-2.16-5.5.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 lm_sensors-libs-3.1.1-17.el6.x86_64 lua-5.1.4-4.1.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 net-snmp-libs-5.5-54.el6_7.1.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.19.1-3.el6_6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 nss-util-3.19.1-1.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 pcre-7.8-7.el6.x86_64 perl-libs-5.10.1-141.el6.x86_64 popt-1.13-7.el6.x86_64 rpm-libs-4.8.0-47.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 xz-libs-4.999.9-0.5.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f711633b5ac in update_dialog_vars_dbinfo (cell=0x7f7100188c40, var=0x3433) at dlg_db_handler.c:652 #1 0x00007f711633c105 in update_dialog_dbinfo_unsafe (cell=0x7f7100188c40) at dlg_db_handler.c:719 #2 0x00007f711633d7ec in update_dialog_dbinfo (cell=0x7f7100188c40) at dlg_db_handler.c:866 #3 0x00007f71163808e3 in set_dlg_variable (dlg=0x7f7100188c40, key=0x7f7116e83900, val=0x7ffe8e783d00) at dlg_var.c:307 #4 0x00007f7116c61eb0 in set_start_time (dialog=0x7f7100188c40) at acc_cdr.c:523 #5 0x00007f7116c67ec6 in cdr_on_create (dialog=0x7f7100188c40, type=2, params=0x7f71165a20c0) at acc_cdr.c:790 #6 0x00007f711632b9f6 in run_create_callbacks (dlg=0x7f7100188c40, msg=0x7f711bc8be60) at dlg_cb.c:230 #7 0x00007f7116346574 in dlg_new_dialog (req=0x7f711bc8be60, t=0x7f71002c3cd0, run_initial_cbs=1) at dlg_handlers.c:865 #8 0x00007f7116344f8d in dlg_onreq (t=0x7f71002c3cd0, type=1, param=0x7f711ba265c0) at dlg_handlers.c:717 #9 0x00007f711b7682ee in run_reqin_callbacks_internal (hl=0x7f70ff731be8, trans=0x7f71002c3cd0, params=0x7f711ba265c0) at t_hooks.c:360 #10 0x00007f711b7683fe in run_reqin_callbacks (trans=0x7f71002c3cd0, req=0x7f711bc8be60, code=1) at t_hooks.c:385 #11 0x00007f711b727ef1 in build_cell (p_msg=0x7f711bc8be60) at h_table.c:356 #12 0x00007f711b779e92 in new_t (p_msg=0x7f711bc8be60) at t_lookup.c:1269 #13 0x00007f711b77b10f in t_newtran (p_msg=0x7f711bc8be60) at t_lookup.c:1409 #14 0x00007f711b74ed9b in t_relay_to (p_msg=0x7f711bc8be60, proxy=0x0, proto=0, replicate=0) at t_funcs.c:236 #15 0x00007f711b78ed9f in _w_t_relay_to (p_msg=0x7f711bc8be60, proxy=0x0, force_proto=0) at tm.c:1476 #16 0x00007f711b78ff56 in w_t_relay (p_msg=0x7f711bc8be60, _foo=0x0, _bar=0x0) at tm.c:1677 #17 0x000000000041e1ca in do_action (h=0x7ffe8e784c80, a=0x7f711bb33600, msg=0x7f711bc8be60) at action.c:1053 #18 0x000000000042ab90 in run_actions (h=0x7ffe8e784c80, a=0x7f711bb33600, msg=0x7f711bc8be60) at action.c:1548 #19 0x000000000042b216 in run_actions_safe (h=0x7ffe8e785a30, a=0x7f711bb33600, msg=0x7f711bc8be60) at action.c:1613 #20 0x000000000054bc3a in rval_get_int (h=0x7ffe8e785a30, msg=0x7f711bc8be60, i=0x7ffe8e785228, rv=0x7f711bb33c50, cache=0x0) at rvalue.c:912 #21 0x000000000054fff1 in rval_expr_eval_int (h=0x7ffe8e785a30, msg=0x7f711bc8be60, res=0x7ffe8e785228, rve=0x7f711bb33c48) at rvalue.c:1906 #22 0x000000000055041f in rval_expr_eval_int (h=0x7ffe8e785a30, msg=0x7f711bc8be60, res=0x7ffe8e785760, rve=0x7f711bb34348) at rvalue.c:1914 #23 0x000000000041dc7c in do_action (h=0x7ffe8e785a30, a=0x7f711bb338f8, msg=0x7f711bc8be60) at action.c:1029 #24 0x000000000042ab90 in run_actions (h=0x7ffe8e785a30, a=0x7f711ba769a8, msg=0x7f711bc8be60) at action.c:1548 #25 0x000000000042b2de in run_top_route (a=0x7f711ba769a8, msg=0x7f711bc8be60, c=0x0) at action.c:1634 #26 0x000000000051139c in receive_msg ( buf=0xa84320 "INVITE sip:09011112222@192.168.2.187:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.0.188:5010;rport;branch=z9hG4bK-20004-110686-4\r\nFrom: \"user1\" <sip:user1@test1.com>;tag=20004SIPpTag00110686\r\nTo: <sip:09011"..., len=947, rcv_info=0x7ffe8e785d50) at receive.c:196 #27 0x000000000061809c in udp_rcv_loop () at udp_server.c:495 #28 0x00000000004ac5ba in main_loop () at main.c:1573 #29 0x00000000004b2bc5 in main (argc=7, argv=0x7ffe8e7861e8) at main.c:2533 (gdb) frame 0 #0 0x00007f711633b5ac in update_dialog_vars_dbinfo (cell=0x7f7100188c40, var=0x3433) at dlg_db_handler.c:652 652 SET_STR_VALUE(values+2, var->key); (gdb) p *cell $1 = {ref = 1, next = 0x0, prev = 0x0, h_id = 6034, h_entry = 2375, state = 1, lifetime = 10800, init_ts = 1440666688, start_ts = 0, dflags = 0, iflags = 0, sflags = 0, toroute = 0, toroute_name = { s = 0x0, len = 0}, from_rr_nb = 0, tl = {next = 0x0, prev = 0x0, timeout = 0}, callid = { s = 0x7f7100188da0 "110686-20004@192.168.0.188sip:user1@test1.comsip:09011112222@192.168.2.187:5060sip:09011112222@192.168.2.188:5030168.2.188:5030gent: VS4000/SIP Ver1.0.0\r\nContent-Length: 0\r\n\r\n", len = 26}, from_uri = {s = 0x7f7100188dba "sip:user1@test1.comsip:09011112222@192.168.2.187:5060sip:09011112222@192.168.2.188:5030168.2.188:5030gent: VS4000/SIP Ver1.0.0\r\nContent-Length: 0\r\n\r\n", len = 19}, to_uri = {s = 0x7f7100188dcd "sip:09011112222@192.168.2.187:5060sip:09011112222@192.168.2.188:5030168.2.188:5030gent: VS4000/SIP Ver1.0.0\r\nContent-Length: 0\r\n\r\n", len = 34}, req_uri = {s = 0x7f7100188def "sip:09011112222@192.168.2.188:5030168.2.188:5030gent: VS4000/SIP Ver1.0.0\r\nContent-Length: 0\r\n\r\n", len = 34}, tag = {{s = 0x7f710026ae28 "20004SIPpTag00110686", len = 20}, {s = 0x0, len = 0}}, cseq = {{s = 0x7f710008fc88 "802r1", len = 3}, {s = 0x0, len = 0}}, route_set = {{s = 0x0, len = 0}, {s = 0x0, len = 0}}, contact = {{ s = 0x7f7100015be0 "sip:user1@192.168.2.50:5010", len = 27}, {s = 0x0, len = 0}}, bind_addr = {0x7f711ba58730, 0x0}, cbs = {first = 0x7f71002a9170, types = 41172}, profile_links = 0x7f710006b1d8, vars = 0x7f710013daf0} (gdb) p *cell->vars $2 = {key = {s = 0x7f71001240f8 "answer_time", len = 11}, value = {s = 0x7f7100196358 "1440666688.375", len = 14}, vflags = 0, next = 0x7f710017f108} (gdb) set $p=cell->vars->next (gdb) while ($p != 0) $3 = {key = {s = 0x7f7100184890 "calling_number", len = 14}, value = {s = 0x7f7100436aa0 "0312341234", len = 10}, vflags = 0, next = 0x7f7100443180} $4 = {key = {s = 0x7f71003ecb78 "destination", len = 11}, value = {s = 0x7f710006d080 "09011112222", len = 11}, vflags = 0, next = 0x7f71001496f0} $5 = {key = {s = 0x7f71000bab68 "caller_username", len = 15}, value = {s = 0x7f71001a8170 "user1", len = 5}, vflags = 0, next = 0x7f7100032020} $6 = {key = {s = 0x7f710013d498 "caller_domain", len = 13}, value = {s = 0x7f710013c3f0 "test1.com", len = 9}, vflags = 0, next = 0x7f7100226ba0} $7 = {key = {s = 0x7f7100294bb0 "start_time", len = 10}, value = {s = 0x7f71001a2f58 "2015-08-27 18:11:28", len = 19}, vflags = 0, next = 0x0} (gdb) p process_no $8 = 2 (gdb) p pt[process_no] $9 = {pid = 14673, unix_sock = -1, idx = -1, desc = "udp receiver child=1 sock=192.168.2.187:5060", '\000' <repeats 83 times>} (gdb) ``` I am also load testing latest commit 9e03ac7410c879dc9e91dd21c04c0597ce1ccff3 of version 4.1 and so far the problem didn't happen with it (4.1 is what I am running in production. Since i need to update my kamailio.cfg with new functionality, I am trying to upgrade kamailio version to not lag too much behind). --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-135577581

Interesting that it is same invalid value var=0x3433, but dlg->vars list is ok. Is v4.1 saving the dlg vars in db? Can you describe a bit when the crash is happening? Like: if you run a stress test, after how many iterations does it happen; what kind of calls are there, between two users or call is forwarded outside to a gateway... --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-135665945

Yes. It is exactly the same cfg file (I had only to adjust DB tables when moving to to 4.3). I am load testing simulating transcoding. There are actually 2 concurrent tests: ``` sipp simulating user --> kamailio --> sipp simulating gateway refuses with "406 Not Acceptable". This triggers branch to transcoder server sipp simulating user --> kamailio --> sipp simulating transcoder (200 OK) ``` ``` sipp simulating gateway calling user --> kamailio forwarding call to gateway --> sipp simulating gateway refuses with "406 Not Acceptable". This triggers branch to transcoder server sipp simulating gateway calling user --> kamailio --> sipp simulating transcoder (200 OK) ``` Here are the last number of calls based on acc_cdrs after crash: ``` mysql> select count(*) from acc_cdrs; +----------+ | count(*) | +----------+ | 168623 | +----------+ mysql> select count(*) from acc_cdrs; +----------+ | count(*) | +----------+ | 221434 | +----------+ mysql> select count(*) from acc_cdrs; +----------+ | count(*) | +----------+ | 244369 | +----------+ mysql> select count(*) from acc_cdrs; +----------+ | count(*) | +----------+ | 150870 | +----------+ ``` If you need it, I can send you the kamailio.cfg privately (I cannot make it public because it belongs to the company I work for). --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-135742969

Strange, nothing looks broken. I pushed a patch to see how many of the dialog variables are stored - commit 2962852bf706692bdbc9b51419dbdc2012f34e1b . Can you test with it? Then send the output for: ``` frame 0 p *cell p *cell->vars set $p=cell->vars->next while ($p != 0) p *$p set $p=$p->next end frame 1 info locals ``` Can you check the syslog file and see what error messages are printed just before the crash? Also, given it is a test system, would it be possible to grant remote access to investigate the core file? --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-137694399

I pushed two commits to the dialog module, one to handle a corner situation that could appear if you do stress testing and the other one to avoid saving incomplete dialog structures. The first one might have been the one causing the crashes for you. The commits are 7f9194c416e171c2b723430eb98a55a52b7b5e38 and cecdd0817052046e634152d6e512165451b5f6d4. I plan to backport them to 4.3 branch to be part of 4.3.2 release. Testing and reporting the results will be appreciated. --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-139193896

Let's wait to see if there is another crash after the last patches. Analyzing the old core might be waste of time now. --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-139506787

Have you tested only with commit cecdd08? Or with both 7f9194c and cecdd08? The two have to be used together. You can eventually just test v4.3.2 now. --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-139925103

OK -- I thought you could do a cherry-pick to get a commit by id from master to branch 4.3. But apparently you were testing the master branch, which is ok as well, just need to know what source tree to investigate. --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-139926456

OK. I have restarted tests (6 kamailio VMs on kvm, vmware and DigitalOcean) with master branch latest commit 171553ebcac2f00a015191f36f2f228c0d3e483b. --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-140050255

One of the VM hosts in our lab got a hardware failure so i could not keep the tests running. I could not reproduce the problem with kamailio stopping processing messages. But i got the same crash again in one of the remaining VMs in the lab: Core was generated by `/usr/local/src/git/kamailio-4.4/kamailio -x qm -m 256 -M 16 -P /var/run/kamaili'. Program terminated with signal 11, Segmentation fault. #0 0x00007f889edb82d6 in update_dialog_vars_dbinfo (cell=0x7f888b80f5d0, var=0x3433) at dlg_db_handler.c:652 652 SET_STR_VALUE(values+2, var->key); Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 db4-4.7.25-19.el6_6.x86_64 elfutils-libelf-0.161-3.el6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libacl-2.2.49-6.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcap-2.16-5.5.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 lm_sensors-libs-3.1.1-17.el6.x86_64 lua-5.1.4-4.1.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 net-snmp-libs-5.5-54.el6_7.1.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.19.1-3.el6_6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 nss-util-3.19.1-1.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 pcre-7.8-7.el6.x86_64 perl-libs-5.10.1-141.el6.x86_64 popt-1.13-7.el6.x86_64 rpm-libs-4.8.0-47.el6.x86_64 tcp_wrappers-libs-7.6-57.el6.x86_64 xz-libs-4.999.9-0.5.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f889edb82d6 in update_dialog_vars_dbinfo (cell=0x7f888b80f5d0, var=0x3433) at dlg_db_handler.c:652 #1 0x00007f889edb9123 in update_dialog_dbinfo_unsafe (cell=0x7f888b80f5d0) at dlg_db_handler.c:725 #2 0x00007f889edbab8d in update_dialog_dbinfo (cell=0x7f888b80f5d0) at dlg_db_handler.c:874 #3 0x00007f889edfe93a in pv_set_dlg_variable (msg=0x7f88a2102998, param=0x7f88a1f37120, op=254, val=0x7ffe0d5c19d0) at dlg_var.c:428 #4 0x00000000004a092c in lval_pvar_assign (h=0x7ffe0d5c2c30, msg=0x7f88a2102998, lv=0x7f88a20c8d30, rv=0x7f88a20c9008) at lvalue.c:351 #5 0x00000000004a1331 in lval_assign (h=0x7ffe0d5c2c30, msg=0x7f88a2102998, lv=0x7f88a20c8d30, rve=0x7f88a20c9000) at lvalue.c:399 #6 0x0000000000429a7f in do_action (h=0x7ffe0d5c2c30, a=0x7f88a20c9728, msg=0x7f88a2102998) at action.c:1429 #7 0x000000000042b429 in run_actions (h=0x7ffe0d5c2c30, a=0x7f88a20c9728, msg=0x7f88a2102998) at action.c:1548 #8 0x000000000041e93b in do_action (h=0x7ffe0d5c2c30, a=0x7f88a20d3d10, msg=0x7f88a2102998) at action.c:1044 #9 0x000000000042b429 in run_actions (h=0x7ffe0d5c2c30, a=0x7f88a20d3d10, msg=0x7f88a2102998) at action.c:1548 #10 0x000000000042bb77 in run_top_route (a=0x7f88a20d3d10, msg=0x7f88a2102998, c=0x7ffe0d5c2c30) at action.c:1634 #11 0x00007f88a1bdbe80 in reply_received (p_msg=0x7f88a2102998) at t_reply.c:2317 #12 0x0000000000491c1b in do_forward_reply (msg=0x7f88a2102998, mode=0) at forward.c:747 #13 0x0000000000493308 in forward_reply (msg=0x7f88a2102998) at forward.c:849 #14 0x00000000005119de in receive_msg ( buf=0xa92340 "SIP/2.0 200 OK\r\nTo: <sip:09011112222@192.168.2.107:5060>;tag=4754SIPpTag0114640\r\nFrom: 0312341234 <sip:0312341234@test1.com>;tag=4729SIPpTag0049083\r\nCall-ID: 49083-4729(a)192.168.0.108\r\nCSeq: 802 INVITE"..., len=811, rcv_info=0x7ffe0d5c3200) at receive.c:255 #15 0x0000000000617928 in udp_rcv_loop () at udp_server.c:495 #16 0x00000000004ac857 in main_loop () at main.c:1579 #17 0x00000000004b2f11 in main (argc=9, argv=0x7ffe0d5c3698) at main.c:2566 Meanwhile, I am running load tests against two DigitalOcean VMs so that when a crash happens there I can give access to them to you (but there, for some unknown reason, it takes more time to reproduce the problem (actually, it happened only once, after one week of load)). --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-141332714

The problem of having kamailio stopping to process messages happened in one of the DigitalOcean VMs. I can give you access to it if you want. Just pass me the email to where i should send the login info. --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#issuecomment-141348170

Closed #301. --- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/301#event-425942865