Hi Daniel, Please find output attached. Waiting your feedback. Thanks, F.Chahrour From: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Sent: Thursday, March 01, 2012 11:34 AM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Cc: Fatima Chahrour~Vanrise Support Subject: Re: [SR-Users] DRouting cause Kamailio 3.2 to stop Hello, that's the core file, do: gdb /usr/local/sbin/kamailio /core The send out the output from the gdb commands: bt bt full Cheers, Daniel On 3/1/12 10:03 AM, Fatima Chahrour~Vanrise Support wrote: Dear Daniel, Thanks for your reply.. I'd like to ask you too if my script in my first post that am using in the routing area is correct? I found file named core in the '/' location as you mentioned. When I open it using #cat core, the opened file go crazy and writes strange characters and it ending with the below: /usr/local/sbin/kamailio-P/var/run/kamailio.pid-f/usr/local/etc/kamailio//ka mailio.cfgSSH_AGENT_PID=3234GPG_AGENT_INFO=/tmp/seahorse-jKaZEU/S.gpg-agent: 3246:1TERM=xtermDESKTOP_STARTUP_ID=SHELL=/bin/bashGTK_RC_FILES=/etc/gtk/gtkr c:/home/vanrise/.gtkrc-1.2-gnome2WINDOWID=33580848USER=rootGNOME_KEYRING_SOC KET=/tmp/keyring-ZKxFck/socketSSH_AUTH_SOCK=/tmp/keyring-ZKxFck/sshUSERNAME= vanriseSESSION_MANAGER=local/debian:/tmp/.ICE-unix/3188MAIL=/var/mail/rootPA TH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/DESKTOP_SES SION=defaultGDM_XSERVER_LOCATION=localPWD=/usr/local/etc/kamailioLANG=en_US. UTF-8GDM_LANG=en_US.UTF-8GDMSESSION=defaultSHLVL=4HOME=/rootGNOME_DESKTOP_SE SSION_ID=DefaultLOGNAME=rootDBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus -sH8atcWdX4,guid=f0dcba443f8fd1d5267161364f4dd99dXDG_DATA_DIRS=/usr/local/sh are/:/usr/share/:/usrdebian:/# 62;9;c62;9 Sorry, I've just realized an info that I should mention and would be helpful, am running this Kamailio on a debian Lenny VMware as a trial before migrating it to its server. Thank you, F.Chahrour From: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Sent: Thursday, March 01, 2012 10:15 AM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Cc: Fatima Chahrour~Vanrise Support; sr-users-bounces(a)lists.sip-router.org Subject: Re: [SR-Users] DRouting cause Kamailio 3.2 to stop Hello, have you found the core file in /var/log/? It is a strange location unless you started with -w /var/log. The core file is named usually 'core' or core.pid (replace pid with same number, like, 'core.1234'). The default location in in '/' unless there is a different location set via -w or enforced by operating system (e.g., mac os x puts it in '/corefiles'). Hope it helps to locate the core file and send the backtrace. Cheers, Daniel On 3/1/12 9:05 AM, Fatima Chahrour~Vanrise Support wrote: I also tried : #gdb /usr/local/sbin/kamailio /var/log/core.file (gdb) bt And got No Stack.. BR, F.Chahrour _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users(a)lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda

Hello, Full version output is: debian:/usr/local/etc/kamailio# /usr/local/sbin/kamailio -V version: kamailio 3.2.2 (i386/linux) 5fc59c flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 5fc59c compiled on 06:46:26 Feb 28 2012 with gcc 4.3.2 Thanks in advance. F.Chahrour From: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Sent: Thursday, March 01, 2012 3:44 PM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Cc: Fatima Chahrour~Vanrise Support Subject: Re: [SR-Users] DRouting cause Kamailio 3.2 to stop Hello, can you send also the full version - -the output of '/usr/local/sbin/kamailio -V' Based on the backtrace, I could not match a valid line in the source code that could cause a crash. Also, you haven't sent the output of 'bt full' -- you sent only from 'bt', first part just limited number of printed line. Do 'bt full' in gdb and sent the output. Cheers, Daniel On 3/1/12 11:02 AM, Fatima Chahrour~Vanrise Support wrote: Hi Daniel, Please find output attached. Waiting your feedback. Thanks, F.Chahrour From: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Sent: Thursday, March 01, 2012 11:34 AM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Cc: Fatima Chahrour~Vanrise Support Subject: Re: [SR-Users] DRouting cause Kamailio 3.2 to stop Hello, that's the core file, do: gdb /usr/local/sbin/kamailio /core The send out the output from the gdb commands: bt bt full Cheers, Daniel On 3/1/12 10:03 AM, Fatima Chahrour~Vanrise Support wrote: Dear Daniel, Thanks for your reply.. I'd like to ask you too if my script in my first post that am using in the routing area is correct? I found file named core in the '/' location as you mentioned. When I open it using #cat core, the opened file go crazy and writes strange characters and it ending with the below: /usr/local/sbin/kamailio-P/var/run/kamailio.pid-f/usr/local/etc/kamailio//ka mailio.cfgSSH_AGENT_PID=3234GPG_AGENT_INFO=/tmp/seahorse-jKaZEU/S.gpg-agent: 3246:1TERM=xtermDESKTOP_STARTUP_ID=SHELL=/bin/bashGTK_RC_FILES=/etc/gtk/gtkr c:/home/vanrise/.gtkrc-1.2-gnome2WINDOWID=33580848USER=rootGNOME_KEYRING_SOC KET=/tmp/keyring-ZKxFck/socketSSH_AUTH_SOCK=/tmp/keyring-ZKxFck/sshUSERNAME= vanriseSESSION_MANAGER=local/debian:/tmp/.ICE-unix/3188MAIL=/var/mail/rootPA TH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/DESKTOP_SES SION=defaultGDM_XSERVER_LOCATION=localPWD=/usr/local/etc/kamailioLANG=en_US. UTF-8GDM_LANG=en_US.UTF-8GDMSESSION=defaultSHLVL=4HOME=/rootGNOME_DESKTOP_SE SSION_ID=DefaultLOGNAME=rootDBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus -sH8atcWdX4,guid=f0dcba443f8fd1d5267161364f4dd99dXDG_DATA_DIRS=/usr/local/sh are/:/usr/share/:/usrdebian:/# 62;9;c62;9 Sorry, I've just realized an info that I should mention and would be helpful, am running this Kamailio on a debian Lenny VMware as a trial before migrating it to its server. Thank you, F.Chahrour From: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Sent: Thursday, March 01, 2012 10:15 AM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Cc: Fatima Chahrour~Vanrise Support; sr-users-bounces(a)lists.sip-router.org Subject: Re: [SR-Users] DRouting cause Kamailio 3.2 to stop Hello, have you found the core file in /var/log/? It is a strange location unless you started with -w /var/log. The core file is named usually 'core' or core.pid (replace pid with same number, like, 'core.1234'). The default location in in '/' unless there is a different location set via -w or enforced by operating system (e.g., mac os x puts it in '/corefiles'). Hope it helps to locate the core file and send the backtrace. Cheers, Daniel On 3/1/12 9:05 AM, Fatima Chahrour~Vanrise Support wrote: I also tried : #gdb /usr/local/sbin/kamailio /var/log/core.file (gdb) bt And got No Stack.. BR, F.Chahrour _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users(a)lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users(a)lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda

By the way, in the config you sent in the very first email, you do do_routing("0"), but I could not see rules with id 0. Can you add such rules or use the id from the rules? Just to see if this is related or not, it should not crash even if the id is missing. Daniel On 3/1/12 3:52 PM, Fatima Chahrour~Vanrise Support wrote: -- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda

Hi, Thanks for the reply, it works with no crash now and I can now understand the reason behind the crash; Note: After removing 4 from dr_rule table no more crash occurs, yet the call is still ending with Request Timeout. Shall I add this issue in a new post or can I continue here? Because I still need help to know why is my call not successful. Thanks in advance, F.Chahrour From: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Sent: Friday, March 16, 2012 7:12 PM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Cc: Fatima Chahrour~Vanrise Support Subject: Re: [SR-Users] DRouting cause Kamailio 3.2 to stop Hello, finally I had the time to look at the documents you sent. The problem is that you have recursive execution of do_routing("0") from do_routing("0"). This happens because you have route[4] like: route[4] { if (!do_routing("0")) { send_reply("503", "No Rules matching the URI"); exit; } } route[4] is executed from request_route. But then, in database dr_routes table, for the prefix 961 you are using for testing, the routeid value is 4, meaning that, when do_routing("0") is matching will execute the route[4] from the config. Which again is calling do_routing("0") that matches again and executes again route[4] and so on ... At some point the OS kills kamailio, resulting in a crash reported at random lines of code -- these were misleading so far, because at some of them was an impossible code fault crash condition. So, remove 4 from routeid column in the database dr_rules table. Cheers, Daniel On 3/2/12 9:57 AM, Daniel-Constantin Mierla wrote: By the way, in the config you sent in the very first email, you do do_routing("0"), but I could not see rules with id 0. Can you add such rules or use the id from the rules? Just to see if this is related or not, it should not crash even if the id is missing. Daniel On 3/1/12 3:52 PM, Fatima Chahrour~Vanrise Support wrote: Hello, Full version output is: debian:/usr/local/etc/kamailio# /usr/local/sbin/kamailio -V version: kamailio 3.2.2 (i386/linux) 5fc59c flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 5fc59c compiled on 06:46:26 Feb 28 2012 with gcc 4.3.2 Thanks in advance. F.Chahrour From: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Sent: Thursday, March 01, 2012 3:44 PM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Cc: Fatima Chahrour~Vanrise Support Subject: Re: [SR-Users] DRouting cause Kamailio 3.2 to stop Hello, can you send also the full version - -the output of '/usr/local/sbin/kamailio -V' Based on the backtrace, I could not match a valid line in the source code that could cause a crash. Also, you haven't sent the output of 'bt full' -- you sent only from 'bt', first part just limited number of printed line. Do 'bt full' in gdb and sent the output. Cheers, Daniel On 3/1/12 11:02 AM, Fatima Chahrour~Vanrise Support wrote: Hi Daniel, Please find output attached. Waiting your feedback. Thanks, F.Chahrour From: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Sent: Thursday, March 01, 2012 11:34 AM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Cc: Fatima Chahrour~Vanrise Support Subject: Re: [SR-Users] DRouting cause Kamailio 3.2 to stop Hello, that's the core file, do: gdb /usr/local/sbin/kamailio /core The send out the output from the gdb commands: bt bt full Cheers, Daniel On 3/1/12 10:03 AM, Fatima Chahrour~Vanrise Support wrote: Dear Daniel, Thanks for your reply.. I'd like to ask you too if my script in my first post that am using in the routing area is correct? I found file named core in the '/' location as you mentioned. When I open it using #cat core, the opened file go crazy and writes strange characters and it ending with the below: /usr/local/sbin/kamailio-P/var/run/kamailio.pid-f/usr/local/etc/kamailio//ka mailio.cfgSSH_AGENT_PID=3234GPG_AGENT_INFO=/tmp/seahorse-jKaZEU/S.gpg-agent: 3246:1TERM=xtermDESKTOP_STARTUP_ID=SHELL=/bin/bashGTK_RC_FILES=/etc/gtk/gtkr c:/home/vanrise/.gtkrc-1.2-gnome2WINDOWID=33580848USER=rootGNOME_KEYRING_SOC KET=/tmp/keyring-ZKxFck/socketSSH_AUTH_SOCK=/tmp/keyring-ZKxFck/sshUSERNAME= vanriseSESSION_MANAGER=local/debian:/tmp/.ICE-unix/3188MAIL=/var/mail/rootPA TH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/DESKTOP_SES SION=defaultGDM_XSERVER_LOCATION=localPWD=/usr/local/etc/kamailioLANG=en_US. UTF-8GDM_LANG=en_US.UTF-8GDMSESSION=defaultSHLVL=4HOME=/rootGNOME_DESKTOP_SE SSION_ID=DefaultLOGNAME=rootDBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus -sH8atcWdX4,guid=f0dcba443f8fd1d5267161364f4dd99dXDG_DATA_DIRS=/usr/local/sh are/:/usr/share/:/usrdebian:/# 62;9;c62;9 Sorry, I've just realized an info that I should mention and would be helpful, am running this Kamailio on a debian Lenny VMware as a trial before migrating it to its server. Thank you, F.Chahrour From: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Sent: Thursday, March 01, 2012 10:15 AM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Cc: Fatima Chahrour~Vanrise Support; sr-users-bounces(a)lists.sip-router.org Subject: Re: [SR-Users] DRouting cause Kamailio 3.2 to stop Hello, have you found the core file in /var/log/? It is a strange location unless you started with -w /var/log. The core file is named usually 'core' or core.pid (replace pid with same number, like, 'core.1234'). The default location in in '/' unless there is a different location set via -w or enforced by operating system (e.g., mac os x puts it in '/corefiles'). Hope it helps to locate the core file and send the backtrace. Cheers, Daniel On 3/1/12 9:05 AM, Fatima Chahrour~Vanrise Support wrote: I also tried : #gdb /usr/local/sbin/kamailio /var/log/core.file (gdb) bt And got No Stack.. BR, F.Chahrour -- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda -- Daniel-Constantin Mierla Kamailio Advanced Training, April 23-26, 2012, Berlin, Germany http://www.asipto.com/index.php/kamailio-advanced-training/