On Thu, Jan 30, 2014 at 12:11 AM, Daniel Ciprus <daniel.ciprus@acision.com> wrote:

Jason,

this is bt generated from running process .. I think my scscf is in this stage now. Obviously hss has listener up and running and I can telnet to the port. "Lately" means that nobody paid attention to this issue before :-), it was discovered just recently.

As I said: nothing's in capture :

[root@kamailio-4-1-test kamailio]# date ; tcpdump -i any -s0 -w /tmp/crap.pcap -v port 3868 -n ; date
Wed Jan 29 16:45:15 EST 2014
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
^C0 packets captured
0 packets received by filter
0 packets dropped by kernel
Wed Jan 29 17:09:08 EST 2014
[root@kamailio-4-1-test kamailio]#

Once I restart scscf/icscf everything will be OK ... so there is likely no way to get logging out of the running scscf, is it ?

On 01/29/2014 09:45 AM, Jason Penton wrote:

Hi Daniel,

This looks like a bootup of kamailio, where the peer process is trying to start? Is your HSS up. Can you telnet to your HSS:3868?

Also, your first message says you are having trouble with CDP "lately". What does this mean? There have been no changes to CDP code for some time now...

Cheers

jason

On Wed, Jan 29, 2014 at 4:30 PM, Daniel Ciprus <daniel.ciprus@acision.com> wrote:

(gdb) bt
#0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6
#1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6
#2 0x00007f5b5095cb71 in timer_loop () at timer.c:120
#3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207
#4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396
#5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237
#6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924
#7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921
#8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921
#9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921
#10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921
#11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921
#12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948
#13 0x000000000046d57c in main_loop () at main.c:1694
#14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533
(gdb) bt full
#0 0x00007f5b564f7900 in __nanosleep_nocancel () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f5b564f7790 in sleep () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007f5b5095cb71 in timer_loop () at timer.c:120
        now = 1390934103
        i = 0x0
        cb = 0
        ptr = 0x0
        interval = 1
        __FUNCTION__ = "timer_loop"
#3 0x00007f5b5095d5a4 in timer_process (returns=0) at timer.c:207
        __FUNCTION__ = "timer_process"
#4 0x00007f5b5093f891 in diameter_peer_start (blocking=0) at diameter_peer.c:396
        pid = 0
        k = -2
        p = 0x0
        __FUNCTION__ = "diameter_peer_start"
#5 0x00007f5b5094102b in cdp_child_init (rank=0) at mod.c:237
        __FUNCTION__ = "cdp_child_init"
#6 0x00000000004f7ec2 in init_mod_child (m=0x7f5b5609a670, rank=0) at sr_module.c:924
        __FUNCTION__ = "init_mod_child"
#7 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b218, rank=0) at sr_module.c:921
        __FUNCTION__ = "init_mod_child"
#8 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b5c0, rank=0) at sr_module.c:921
        __FUNCTION__ = "init_mod_child"
#9 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609b970, rank=0) at sr_module.c:921
        __FUNCTION__ = "init_mod_child"
#10 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c000, rank=0) at sr_module.c:921
        __FUNCTION__ = "init_mod_child"
#11 0x00000000004f7d65 in init_mod_child (m=0x7f5b5609c478, rank=0) at sr_module.c:921
        __FUNCTION__ = "init_mod_child"
#12 0x00000000004f8048 in init_child (rank=0) at sr_module.c:948
No locals.
#13 0x000000000046d57c in main_loop () at main.c:1694
        i = 8
        pid = 19298
        si = 0x0
        si_desc = "udp receiver child=7 sock=10.96.173.17:6060\000[\177\000\000\001\000\000\000z\000\000\000~}^\000\000\000\000\000`o^\000\000\000\000\000\254\305\006m\000\000\000\000\300LA\000\000\000\000\000 \021\342\363\377\177", '\000' <repeats 18 times>, "`\017\342\363\377\177\000\000.\241K\000\000\000\000"
        nrprocs = 8
        __FUNCTION__ = "main_loop"
#14 0x000000000047030b in main (argc=13, argv=0x7ffff3e21128) at main.c:2533
        cfg_stream = 0x1a7f010
        c = -1
        r = 0
        tmp = 0x7ffff3e22f40 ""
        tmp_len = 0
        port = 0
        proto = 0
        options = 0x5e02c0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
        ret = -1
        seed = 2714878883
        rfd = 4
        debug_save = 0
        debug_flag = 0
        dont_fork_cnt = 0
---Type <return> to continue, or q <return> to quit---
        n_lst = 0x7f5b5645bb88
        p = 0x5ca3e0 "H\211l$\330L\211d$\340H\215-\017\234*"
        __FUNCTION__ = "main"
(gdb)

On 01/29/2014 08:45 AM, Daniel Ciprus wrote:

Unfortunately the only thing I have is core file and bt. But I can assure you that there was absolutely no traffic between diameter interfaces. Looks like I will need to reproduce the problem to get right input.

thanks
Dan

On 01/29/2014 02:39 AM, Jason Penton wrote:

Hi,

can you send us a pcap as well as a log file? I can't debug much with what you have given so far.

Cheers

Jason

On Tue, Jan 28, 2014 at 8:42 PM, Daniel Ciprus <daniel.ciprus@acision.com> wrote:

Hi,

seems like CDP module is giving us hard time in last few days. For some
reason s-cscf and i-cscf is loosing connection to HSS and is not even
trying to close socket nor trying to reconnect back. Here is what I see
on the kamailio node:

Name : kamailio Relocations: (not relocatable)
Version : 4.2.0 Vendor: kamailio.org
Release : dev0.0.el6 Build Date: Wed Dec 18
12:46:11 2013

[root@kamailio-4-1-staging kamailio]# netstat -apn | grep 3868
tcp 0 0 10.96.173.17:34915 10.96.173.18:3868
CLOSE_WAIT 10087/kamailio
tcp 0 0 10.96.173.17:35361 10.96.173.18:3868
CLOSE_WAIT 19308/kamailio
[root@kamailio-4-1-staging kamailio]#

and HSS:

[root@hss-config-staging phss-sop]# netstat -apn | grep 3868
tcp 0 0 10.96.173.18:3868 0.0.0.0:*
LISTEN 10549/ugc_server
[root@hss-config-staging phss-sop]#

I have core files (created by gcore) if this helps.

thanks
Dan

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

_______________________________________________
sr-dev mailing list
sr-dev@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

--
Daniel Ciprus
Integration engineer
http://www.acision.com

9954 Mayland Dr
Suite 3100
Richmond, VA 23233
USA
T: +1 804 762 5601
E: daniel.ciprus@acision.com

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

--
Daniel Ciprus
Integration engineer
http://www.acision.com

9954 Mayland Dr
Suite 3100
Richmond, VA 23233
USA
T: +1 804 762 5601
E: daniel.ciprus@acision.com

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

--
Daniel Ciprus
Integration engineer
http://www.acision.com

9954 Mayland Dr
Suite 3100
Richmond, VA 23233
USA
T: +1 804 762 5601
E: daniel.ciprus@acision.com

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.