Hi,
seems like CDP module is giving us hard time in last few days. For some reason s-cscf and i-cscf is loosing connection to HSS and is not even trying to close socket nor trying to reconnect back. Here is what I see on the kamailio node:
Name : kamailio Relocations: (not relocatable) Version : 4.2.0 Vendor: kamailio.org Release : dev0.0.el6 Build Date: Wed Dec 18 12:46:11 2013
[root@kamailio-4-1-staging kamailio]# netstat -apn | grep 3868 tcp 0 0 10.96.173.17:34915 10.96.173.18:3868 CLOSE_WAIT 10087/kamailio tcp 0 0 10.96.173.17:35361 10.96.173.18:3868 CLOSE_WAIT 19308/kamailio [root@kamailio-4-1-staging kamailio]#
and HSS:
[root@hss-config-staging phss-sop]# netstat -apn | grep 3868 tcp 0 0 10.96.173.18:3868 0.0.0.0:* LISTEN 10549/ugc_server [root@hss-config-staging phss-sop]#
I have core files (created by gcore) if this helps.
thanks Dan
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
Hi,
can you send us a pcap as well as a log file? I can't debug much with what you have given so far.
Cheers Jason
On Tue, Jan 28, 2014 at 8:42 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:
Hi,
seems like CDP module is giving us hard time in last few days. For some reason s-cscf and i-cscf is loosing connection to HSS and is not even trying to close socket nor trying to reconnect back. Here is what I see on the kamailio node:
Name : kamailio Relocations: (not relocatable) Version : 4.2.0 Vendor: kamailio.org Release : dev0.0.el6 Build Date: Wed Dec 18 12:46:11 2013
[root@kamailio-4-1-staging kamailio]# netstat -apn | grep 3868 tcp 0 0 10.96.173.17:34915 10.96.173.18:3868 CLOSE_WAIT 10087/kamailio tcp 0 0 10.96.173.17:35361 10.96.173.18:3868 CLOSE_WAIT 19308/kamailio [root@kamailio-4-1-staging kamailio]#
and HSS:
[root@hss-config-staging phss-sop]# netstat -apn | grep 3868 tcp 0 0 10.96.173.18:3868 0.0.0.0:* LISTEN 10549/ugc_server [root@hss-config-staging phss-sop]#
I have core files (created by gcore) if this helps.
thanks Dan
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
Unfortunately the only thing I have is core file and bt. But I can assure you that there was absolutely no traffic between diameter interfaces. Looks like I will need to reproduce the problem to get right input.
thanks Dan
On 01/29/2014 02:39 AM, Jason Penton wrote: Hi,
can you send us a pcap as well as a log file? I can't debug much with what you have given so far.
Cheers Jason
On Tue, Jan 28, 2014 at 8:42 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: Hi,
seems like CDP module is giving us hard time in last few days. For some reason s-cscf and i-cscf is loosing connection to HSS and is not even trying to close socket nor trying to reconnect back. Here is what I see on the kamailio node:
Name : kamailio Relocations: (not relocatable) Version : 4.2.0 Vendor: kamailio.orghttp://kamailio.org Release : dev0.0.el6 Build Date: Wed Dec 18 12:46:11 2013
[root@kamailio-4-1-staging kamailio]# netstat -apn | grep 3868 tcp 0 0 10.96.173.17:34915http://10.96.173.17:34915 10.96.173.18:3868http://10.96.173.18:3868 CLOSE_WAIT 10087/kamailio tcp 0 0 10.96.173.17:35361http://10.96.173.17:35361 10.96.173.18:3868http://10.96.173.18:3868 CLOSE_WAIT 19308/kamailio [root@kamailio-4-1-staging kamailio]#
and HSS:
[root@hss-config-staging phss-sop]# netstat -apn | grep 3868 tcp 0 0 10.96.173.18:3868http://10.96.173.18:3868 0.0.0.0:* LISTEN 10549/ugc_server [root@hss-config-staging phss-sop]#
I have core files (created by gcore) if this helps.
thanks Dan
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
_______________________________________________ sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
(gdb) bt #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 #13 0x000000000046d57c in main_loop () at main.c:1694 #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 (gdb) bt full #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 now = 1390934103 i = 0x0 cb = 0 ptr = 0x0 interval = 1 __FUNCTION__ = "timer_loop" #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 __FUNCTION__ = "timer_process" #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 pid = 0 k = -2 p = 0x0 __FUNCTION__ = "diameter_peer_start" #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 __FUNCTION__ = "cdp_child_init" #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 __FUNCTION__ = "init_mod_child" #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 No locals. #13 0x000000000046d57c in main_loop () at main.c:1694 i = 8 pid = 19298 si = 0x0 si_desc = "udp receiver child=7 sock=10.96.173.17:6060\000[\177\000\000\001\000\000\000z\000\000\000~}^\000\000\000\000\000`o^\000\000\000\000\000\254\305\006m\000\000\000\000\300LA\000\000\000\000\000 \021\342\363\377\177", '\000' <repeats 18 times>, "`\017\342\363\377\177\000\000.\241K\000\000\000\000" nrprocs = 8 __FUNCTION__ = "main_loop" #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 cfg_stream = 0x1a7f010 c = -1 r = 0 tmp = 0x7ffff3e22f40 "" tmp_len = 0 port = 0 proto = 0 options = 0x5e02c0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 2714878883 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 ---Type <return> to continue, or q <return> to quit--- n_lst = 0x7f5b5645bb88 p = 0x5ca3e0 "H\211l$\330L\211d$\340H\215-\017\234*" __FUNCTION__ = "main" (gdb) On 01/29/2014 08:45 AM, Daniel Ciprus wrote: Unfortunately the only thing I have is core file and bt. But I can assure you that there was absolutely no traffic between diameter interfaces. Looks like I will need to reproduce the problem to get right input.
thanks Dan
On 01/29/2014 02:39 AM, Jason Penton wrote: Hi,
can you send us a pcap as well as a log file? I can't debug much with what you have given so far.
Cheers Jason
On Tue, Jan 28, 2014 at 8:42 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: Hi,
seems like CDP module is giving us hard time in last few days. For some reason s-cscf and i-cscf is loosing connection to HSS and is not even trying to close socket nor trying to reconnect back. Here is what I see on the kamailio node:
Name : kamailio Relocations: (not relocatable) Version : 4.2.0 Vendor: kamailio.orghttp://kamailio.org Release : dev0.0.el6 Build Date: Wed Dec 18 12:46:11 2013
[root@kamailio-4-1-staging kamailio]# netstat -apn | grep 3868 tcp 0 0 10.96.173.17:34915http://10.96.173.17:34915 10.96.173.18:3868http://10.96.173.18:3868 CLOSE_WAIT 10087/kamailio tcp 0 0 10.96.173.17:35361http://10.96.173.17:35361 10.96.173.18:3868http://10.96.173.18:3868 CLOSE_WAIT 19308/kamailio [root@kamailio-4-1-staging kamailio]#
and HSS:
[root@hss-config-staging phss-sop]# netstat -apn | grep 3868 tcp 0 0 10.96.173.18:3868http://10.96.173.18:3868 0.0.0.0:* LISTEN 10549/ugc_server [root@hss-config-staging phss-sop]#
I have core files (created by gcore) if this helps.
thanks Dan
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
_______________________________________________ sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
Hi Daniel,
This looks like a bootup of kamailio, where the peer process is trying to start? Is your HSS up. Can you telnet to your HSS:3868?
Also, your first message says you are having trouble with CDP "lately". What does this mean? There have been no changes to CDP code for some time now...
Cheers jason
On Wed, Jan 29, 2014 at 4:30 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:
(gdb) bt #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 #13 0x000000000046d57c in main_loop () at main.c:1694 #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 (gdb) bt full #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 now = 1390934103 i = 0x0 cb = 0 ptr = 0x0 interval = 1 __FUNCTION__ = "timer_loop" #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 __FUNCTION__ = "timer_process" #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 pid = 0 k = -2 p = 0x0 __FUNCTION__ = "diameter_peer_start" #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 __FUNCTION__ = "cdp_child_init" #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 __FUNCTION__ = "init_mod_child" #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 No locals. #13 0x000000000046d57c in main_loop () at main.c:1694 i = 8 pid = 19298 si = 0x0 si_desc = "udp receiver child=7 sock=10.96.173.17:6060\000[\177\000\000\001\000\000\000z\000\000\000~}^\000\000\000\000\000`o^\000\000\000\000\000\254\305\006m\000\000\000\000\300LA\000\000\000\000\000 \021\342\363\377\177", '\000' <repeats 18 times>, "`\017\342\363\377\177\000\000.\241K\000\000\000\000" nrprocs = 8 __FUNCTION__ = "main_loop" #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 cfg_stream = 0x1a7f010 c = -1 r = 0 tmp = 0x7ffff3e22f40 "" tmp_len = 0 port = 0 proto = 0 options = 0x5e02c0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 2714878883 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 ---Type <return> to continue, or q <return> to quit--- n_lst = 0x7f5b5645bb88 p = 0x5ca3e0 "H\211l$\330L\211d$\340H\215-\017\234*" __FUNCTION__ = "main" (gdb) On 01/29/2014 08:45 AM, Daniel Ciprus wrote:
Unfortunately the only thing I have is core file and bt. But I can assure you that there was absolutely no traffic between diameter interfaces. Looks like I will need to reproduce the problem to get right input.
thanks Dan
On 01/29/2014 02:39 AM, Jason Penton wrote:
Hi,
can you send us a pcap as well as a log file? I can't debug much with what you have given so far.
Cheers Jason
On Tue, Jan 28, 2014 at 8:42 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:
Hi,
seems like CDP module is giving us hard time in last few days. For some reason s-cscf and i-cscf is loosing connection to HSS and is not even trying to close socket nor trying to reconnect back. Here is what I see on the kamailio node:
Name : kamailio Relocations: (not relocatable) Version : 4.2.0 Vendor: kamailio.org Release : dev0.0.el6 Build Date: Wed Dec 18 12:46:11 2013
[root@kamailio-4-1-staging kamailio]# netstat -apn | grep 3868 tcp 0 0 10.96.173.17:34915 10.96.173.18:3868 CLOSE_WAIT 10087/kamailio tcp 0 0 10.96.173.17:35361 10.96.173.18:3868 CLOSE_WAIT 19308/kamailio [root@kamailio-4-1-staging kamailio]#
and HSS:
[root@hss-config-staging phss-sop]# netstat -apn | grep 3868 tcp 0 0 10.96.173.18:3868 0.0.0.0:* LISTEN 10549/ugc_server [root@hss-config-staging phss-sop]#
I have core files (created by gcore) if this helps.
thanks Dan
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
Jason,
this is bt generated from running process .. I think my scscf is in this stage now. Obviously hss has listener up and running and I can telnet to the port. "Lately" means that nobody paid attention to this issue before :-), it was discovered just recently.
As I said: nothing's in capture :
[root@kamailio-4-1-test kamailio]# date ; tcpdump -i any -s0 -w /tmp/crap.pcap -v port 3868 -n ; date Wed Jan 29 16:45:15 EST 2014 tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes ^C0 packets captured 0 packets received by filter 0 packets dropped by kernel Wed Jan 29 17:09:08 EST 2014 [root@kamailio-4-1-test kamailio]#
Once I restart scscf/icscf everything will be OK ... so there is likely no way to get logging out of the running scscf, is it ?
On 01/29/2014 09:45 AM, Jason Penton wrote: Hi Daniel,
This looks like a bootup of kamailio, where the peer process is trying to start? Is your HSS up. Can you telnet to your HSS:3868?
Also, your first message says you are having trouble with CDP "lately". What does this mean? There have been no changes to CDP code for some time now...
Cheers jason
On Wed, Jan 29, 2014 at 4:30 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: (gdb) bt #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 #13 0x000000000046d57c in main_loop () at main.c:1694 #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 (gdb) bt full #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 now = 1390934103 i = 0x0 cb = 0 ptr = 0x0 interval = 1 __FUNCTION__ = "timer_loop" #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 __FUNCTION__ = "timer_process" #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 pid = 0 k = -2 p = 0x0 __FUNCTION__ = "diameter_peer_start" #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 __FUNCTION__ = "cdp_child_init" #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 __FUNCTION__ = "init_mod_child" #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 No locals. #13 0x000000000046d57c in main_loop () at main.c:1694 i = 8 pid = 19298 si = 0x0 si_desc = "udp receiver child=7 sock=10.96.173.17:6060http://10.96.173.17:6060\000[\177\000\000\001\000\000\000z\000\000\000~}^\000\000\000\000\000`o^\000\000\000\000\000\254\305\006m\000\000\000\000\300LA\000\000\000\000\000 \021\342\363\377\177", '\000' <repeats 18 times>, "`\017\342\363\377\177\000\000.\241K\000\000\000\000" nrprocs = 8 __FUNCTION__ = "main_loop" #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 cfg_stream = 0x1a7f010 c = -1 r = 0 tmp = 0x7ffff3e22f40 "" tmp_len = 0 port = 0 proto = 0 options = 0x5e02c0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 2714878883 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 ---Type <return> to continue, or q <return> to quit--- n_lst = 0x7f5b5645bb88 p = 0x5ca3e0 "H\211l$\330L\211d$\340H\215-\017\234*" __FUNCTION__ = "main" (gdb) On 01/29/2014 08:45 AM, Daniel Ciprus wrote: Unfortunately the only thing I have is core file and bt. But I can assure you that there was absolutely no traffic between diameter interfaces. Looks like I will need to reproduce the problem to get right input.
thanks Dan
On 01/29/2014 02:39 AM, Jason Penton wrote: Hi,
can you send us a pcap as well as a log file? I can't debug much with what you have given so far.
Cheers Jason
On Tue, Jan 28, 2014 at 8:42 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: Hi,
seems like CDP module is giving us hard time in last few days. For some reason s-cscf and i-cscf is loosing connection to HSS and is not even trying to close socket nor trying to reconnect back. Here is what I see on the kamailio node:
Name : kamailio Relocations: (not relocatable) Version : 4.2.0 Vendor: kamailio.orghttp://kamailio.org Release : dev0.0.el6 Build Date: Wed Dec 18 12:46:11 2013
[root@kamailio-4-1-staging kamailio]# netstat -apn | grep 3868 tcp 0 0 10.96.173.17:34915http://10.96.173.17:34915 10.96.173.18:3868http://10.96.173.18:3868 CLOSE_WAIT 10087/kamailio tcp 0 0 10.96.173.17:35361http://10.96.173.17:35361 10.96.173.18:3868http://10.96.173.18:3868 CLOSE_WAIT 19308/kamailio [root@kamailio-4-1-staging kamailio]#
and HSS:
[root@hss-config-staging phss-sop]# netstat -apn | grep 3868 tcp 0 0 10.96.173.18:3868http://10.96.173.18:3868 0.0.0.0:* LISTEN 10549/ugc_server [root@hss-config-staging phss-sop]#
I have core files (created by gcore) if this helps.
thanks Dan
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
_______________________________________________ sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
Hi Daniel,
That bt you sent is normal. That is the CDP timer process so I don't see any issues there. What you can do is send me bt's on the "CDP worker childs", the "cdp_receiver_peer" for your HSS connection, and the "cdp acceptor".
to get the processes, use sercmd ps. For example, mine looks like this:
8633 cdp worker child=0 8634 cdp worker child=1 8635 cdp worker child=2 8636 cdp worker child=3 8637 cdp worker child=4 8638 cdp worker child=5 8639 cdp worker child=6 8640 cdp worker child=7 8641 cdp worker child=8 8642 cdp worker child=9 8643 cdp receiver peer unknown 8644 cdp_receiver_peer=ZAUbuntu005.it.za.smilecoms.com 8645 cdp_acceptor 8646 cdp_timer
re. logs, you can turn on debugging using sercmd/kamcmd.
cheers Jason
Cheers Jason
On Thu, Jan 30, 2014 at 12:11 AM, Daniel Ciprus daniel.ciprus@acision.comwrote:
Jason,
this is bt generated from running process .. I think my scscf is in this stage now. Obviously hss has listener up and running and I can telnet to the port. "Lately" means that nobody paid attention to this issue before :-), it was discovered just recently.
As I said: nothing's in capture :
[root@kamailio-4-1-test kamailio]# date ; tcpdump -i any -s0 -w /tmp/crap.pcap -v port 3868 -n ; date Wed Jan 29 16:45:15 EST 2014 tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes ^C0 packets captured 0 packets received by filter 0 packets dropped by kernel Wed Jan 29 17:09:08 EST 2014 [root@kamailio-4-1-test kamailio]#
Once I restart scscf/icscf everything will be OK ... so there is likely no way to get logging out of the running scscf, is it ?
On 01/29/2014 09:45 AM, Jason Penton wrote:
Hi Daniel,
This looks like a bootup of kamailio, where the peer process is trying to start? Is your HSS up. Can you telnet to your HSS:3868?
Also, your first message says you are having trouble with CDP "lately". What does this mean? There have been no changes to CDP code for some time now...
Cheers jason
On Wed, Jan 29, 2014 at 4:30 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:
(gdb) bt #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 #13 0x000000000046d57c in main_loop () at main.c:1694 #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 (gdb) bt full #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 now = 1390934103 i = 0x0 cb = 0 ptr = 0x0 interval = 1 __FUNCTION__ = "timer_loop" #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 __FUNCTION__ = "timer_process" #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 pid = 0 k = -2 p = 0x0 __FUNCTION__ = "diameter_peer_start" #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 __FUNCTION__ = "cdp_child_init" #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 __FUNCTION__ = "init_mod_child" #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 No locals. #13 0x000000000046d57c in main_loop () at main.c:1694 i = 8 pid = 19298 si = 0x0 si_desc = "udp receiver child=7 sock=10.96.173.17:6060\000[\177\000\000\001\000\000\000z\000\000\000~}^\000\000\000\000\000`o^\000\000\000\000\000\254\305\006m\000\000\000\000\300LA\000\000\000\000\000 \021\342\363\377\177", '\000' <repeats 18 times>, "`\017\342\363\377\177\000\000.\241K\000\000\000\000" nrprocs = 8 __FUNCTION__ = "main_loop" #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 cfg_stream = 0x1a7f010 c = -1 r = 0 tmp = 0x7ffff3e22f40 "" tmp_len = 0 port = 0 proto = 0 options = 0x5e02c0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 2714878883 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 ---Type <return> to continue, or q <return> to quit--- n_lst = 0x7f5b5645bb88 p = 0x5ca3e0 "H\211l$\330L\211d$\340H\215-\017\234*" __FUNCTION__ = "main" (gdb) On 01/29/2014 08:45 AM, Daniel Ciprus wrote:
Unfortunately the only thing I have is core file and bt. But I can assure you that there was absolutely no traffic between diameter interfaces. Looks like I will need to reproduce the problem to get right input.
thanks Dan
On 01/29/2014 02:39 AM, Jason Penton wrote:
Hi,
can you send us a pcap as well as a log file? I can't debug much with what you have given so far.
Cheers Jason
On Tue, Jan 28, 2014 at 8:42 PM, Daniel Ciprus <daniel.ciprus@acision.com
wrote:
Hi,
seems like CDP module is giving us hard time in last few days. For some reason s-cscf and i-cscf is loosing connection to HSS and is not even trying to close socket nor trying to reconnect back. Here is what I see on the kamailio node:
Name : kamailio Relocations: (not relocatable) Version : 4.2.0 Vendor: kamailio.org Release : dev0.0.el6 Build Date: Wed Dec 18 12:46:11 2013
[root@kamailio-4-1-staging kamailio]# netstat -apn | grep 3868 tcp 0 0 10.96.173.17:34915 10.96.173.18:3868 CLOSE_WAIT 10087/kamailio tcp 0 0 10.96.173.17:35361 10.96.173.18:3868 CLOSE_WAIT 19308/kamailio [root@kamailio-4-1-staging kamailio]#
and HSS:
[root@hss-config-staging phss-sop]# netstat -apn | grep 3868 tcp 0 0 10.96.173.18:3868 0.0.0.0:* LISTEN 10549/ugc_server [root@hss-config-staging phss-sop]#
I have core files (created by gcore) if this helps.
thanks Dan
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
Thanks Jason,
as soon as I notice this behavior again I will grab data requested data ...
Dan
On 01/30/2014 02:29 AM, Jason Penton wrote: Hi Daniel,
That bt you sent is normal. That is the CDP timer process so I don't see any issues there. What you can do is send me bt's on the "CDP worker childs", the "cdp_receiver_peer" for your HSS connection, and the "cdp acceptor".
to get the processes, use sercmd ps. For example, mine looks like this:
8633 cdp worker child=0 8634 cdp worker child=1 8635 cdp worker child=2 8636 cdp worker child=3 8637 cdp worker child=4 8638 cdp worker child=5 8639 cdp worker child=6 8640 cdp worker child=7 8641 cdp worker child=8 8642 cdp worker child=9 8643 cdp receiver peer unknown 8644 cdp_receiver_peer=ZAUbuntu005.it.za.smilecoms.comhttp://ZAUbuntu005.it.za.smilecoms.com 8645 cdp_acceptor 8646 cdp_timer
re. logs, you can turn on debugging using sercmd/kamcmd.
cheers Jason
Cheers Jason
On Thu, Jan 30, 2014 at 12:11 AM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: Jason,
this is bt generated from running process .. I think my scscf is in this stage now. Obviously hss has listener up and running and I can telnet to the port. "Lately" means that nobody paid attention to this issue before :-), it was discovered just recently.
As I said: nothing's in capture :
[root@kamailio-4-1-test kamailio]# date ; tcpdump -i any -s0 -w /tmp/crap.pcap -v port 3868 -n ; date Wed Jan 29 16:45:15 EST 2014 tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes ^C0 packets captured 0 packets received by filter 0 packets dropped by kernel Wed Jan 29 17:09:08 EST 2014 [root@kamailio-4-1-test kamailio]#
Once I restart scscf/icscf everything will be OK ... so there is likely no way to get logging out of the running scscf, is it ?
On 01/29/2014 09:45 AM, Jason Penton wrote: Hi Daniel,
This looks like a bootup of kamailio, where the peer process is trying to start? Is your HSS up. Can you telnet to your HSS:3868?
Also, your first message says you are having trouble with CDP "lately". What does this mean? There have been no changes to CDP code for some time now...
Cheers jason
On Wed, Jan 29, 2014 at 4:30 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: (gdb) bt #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 #13 0x000000000046d57c in main_loop () at main.c:1694 #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 (gdb) bt full #0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5b5095cb71 in timer_loop () at timer.c:120 now = 1390934103 i = 0x0 cb = 0 ptr = 0x0 interval = 1 __FUNCTION__ = "timer_loop" #3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207 __FUNCTION__ = "timer_process" #4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396 pid = 0 k = -2 p = 0x0 __FUNCTION__ = "diameter_peer_start" #5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237 __FUNCTION__ = "cdp_child_init" #6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924 __FUNCTION__ = "init_mod_child" #7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948 No locals. #13 0x000000000046d57c in main_loop () at main.c:1694 i = 8 pid = 19298 si = 0x0 si_desc = "udp receiver child=7 sock=10.96.173.17:6060http://10.96.173.17:6060\000[\177\000\000\001\000\000\000z\000\000\000~}^\000\000\000\000\000`o^\000\000\000\000\000\254\305\006m\000\000\000\000\300LA\000\000\000\000\000 \021\342\363\377\177", '\000' <repeats 18 times>, "`\017\342\363\377\177\000\000.\241K\000\000\000\000" nrprocs = 8 __FUNCTION__ = "main_loop" #14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533 cfg_stream = 0x1a7f010 c = -1 r = 0 tmp = 0x7ffff3e22f40 "" tmp_len = 0 port = 0 proto = 0 options = 0x5e02c0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 2714878883 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 ---Type <return> to continue, or q <return> to quit--- n_lst = 0x7f5b5645bb88 p = 0x5ca3e0 "H\211l$\330L\211d$\340H\215-\017\234*" __FUNCTION__ = "main" (gdb) On 01/29/2014 08:45 AM, Daniel Ciprus wrote: Unfortunately the only thing I have is core file and bt. But I can assure you that there was absolutely no traffic between diameter interfaces. Looks like I will need to reproduce the problem to get right input.
thanks Dan
On 01/29/2014 02:39 AM, Jason Penton wrote: Hi,
can you send us a pcap as well as a log file? I can't debug much with what you have given so far.
Cheers Jason
On Tue, Jan 28, 2014 at 8:42 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: Hi,
seems like CDP module is giving us hard time in last few days. For some reason s-cscf and i-cscf is loosing connection to HSS and is not even trying to close socket nor trying to reconnect back. Here is what I see on the kamailio node:
Name : kamailio Relocations: (not relocatable) Version : 4.2.0 Vendor: kamailio.orghttp://kamailio.org Release : dev0.0.el6 Build Date: Wed Dec 18 12:46:11 2013
[root@kamailio-4-1-staging kamailio]# netstat -apn | grep 3868 tcp 0 0 10.96.173.17:34915http://10.96.173.17:34915 10.96.173.18:3868http://10.96.173.18:3868 CLOSE_WAIT 10087/kamailio tcp 0 0 10.96.173.17:35361http://10.96.173.17:35361 10.96.173.18:3868http://10.96.173.18:3868 CLOSE_WAIT 19308/kamailio [root@kamailio-4-1-staging kamailio]#
and HSS:
[root@hss-config-staging phss-sop]# netstat -apn | grep 3868 tcp 0 0 10.96.173.18:3868http://10.96.173.18:3868 0.0.0.0:* LISTEN 10549/ugc_server [root@hss-config-staging phss-sop]#
I have core files (created by gcore) if this helps.
thanks Dan
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
_______________________________________________ sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-
Daniel Ciprus -
Jason Penton