[SR-Users] kamailio does not responde if an rtpengine is unreachable

Mon Dec 24 13:22:40 CET 2018

Daniel-Constantin Mierla writes:

> Can you see how many kamailio processes are running (w.g., with ps)? Are
> there expected number of there?

Same number of processes when I start K with proper rtpengine set and
with one that has an rtpengine that does not respond.

> If yes, take the PID of few of them and attach with gdb, then grab the
> back trace in order to see what they do.

There is 41 processes. Below is a few different ones.

-- Juha

---------------
#0  0x00007f9958d04210 in __pause_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055ce69f98509 in main_loop () at main.c:1755
#2  0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645

#0  0x00007f9958d359c3 in __recvfrom_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055ce6a029443 in udp_rcv_loop () at core/udp_server.c:460
#2  0x000055ce69f969cf in main_loop () at main.c:1621
#3  0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645

#0  0x00007f9958c7fe89 in __GI___sigwaitinfo (set=<optimized out>, info=0x0)
    at ../sysdeps/unix/sysv/linux/sigwaitinfo.c:56
#1  0x000055ce6a04c36a in slow_timer_main () at core/timer.c:1093
#2  0x000055ce69f971ab in main_loop () at main.c:1677
#3  0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645

#0  0x00007f9958d359c3 in __recvfrom_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055ce6a153168 in async_task_run (idx=1) at core/async_task.c:269
#2  0x000055ce6a151f9c in async_task_child_init (rank=0) at core/async_task.c:185
#3  0x000055ce6a0d5e3a in init_child (rank=0) at core/sr_module.c:867
#4  0x000055ce69f977cc in main_loop () at main.c:1703
#5  0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645

#0  0x00007f9958d350a3 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f99554117e7 in io_wait_loop_epoll (h=0x7f9955643340 <io_h>, t=10, repeat=0) at ../../core/io_wait.h:1034
#2  0x00007f9955415d1a in io_listen_loop (fd_no=1, cs_lst=0x55ce6c6b6680) at io_listener.c:281
#3  0x00007f9955431c4c in mod_child (rank=0) at ctl.c:338
#4  0x000055ce6a0d5a84 in init_mod_child (m=0x7f99580b0ea0, rank=0) at core/sr_module.c:843
#5  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b12d0, rank=0) at core/sr_module.c:839
#6  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b1600, rank=0) at core/sr_module.c:839
#7  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b1ac0, rank=0) at core/sr_module.c:839
#8  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b1ee0, rank=0) at core/sr_module.c:839
#9  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b2720, rank=0) at core/sr_module.c:839
#10 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b2a30, rank=0) at core/sr_module.c:839
#11 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b3210, rank=0) at core/sr_module.c:839
#12 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b34a0, rank=0) at core/sr_module.c:839
#13 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b36f0, rank=0) at core/sr_module.c:839
#14 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b3a20, rank=0) at core/sr_module.c:839
#15 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b4270, rank=0) at core/sr_module.c:839
#16 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b44c0, rank=0) at core/sr_module.c:839
#17 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b47f0, rank=0) at core/sr_module.c:839
#18 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b4c80, rank=0) at core/sr_module.c:839
#19 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b5250, rank=0) at core/sr_module.c:839
#20 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b6e20, rank=0) at core/sr_module.c:839
#21 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b7490, rank=0) at core/sr_module.c:839
#22 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b77a0, rank=0) at core/sr_module.c:839
#23 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b7b20, rank=0) at core/sr_module.c:839
#24 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b7d70, rank=0) at core/sr_module.c:839
#25 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b87d0, rank=0) at core/sr_module.c:839
#26 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b8f50, rank=0) at core/sr_module.c:839
#27 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b95d0, rank=0) at core/sr_module.c:839
#28 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580be830, rank=0) at core/sr_module.c:839
#29 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580beeb0, rank=0) at core/sr_module.c:839
#30 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bf220, rank=0) at core/sr_module.c:839
#31 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bf5c0, rank=0) at core/sr_module.c:839
#32 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bf810, rank=0) at core/sr_module.c:839
#33 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bfea0, rank=0) at core/sr_module.c:839
#34 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c01a0, rank=0) at core/sr_module.c:839
#35 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c0580, rank=0) at core/sr_module.c:839
#36 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c0f90, rank=0) at core/sr_module.c:839
#37 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c14c0, rank=0) at core/sr_module.c:839
#38 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c17d0, rank=0) at core/sr_module.c:839
#39 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c1d90, rank=0) at core/sr_module.c:839
#40 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c1fe0, rank=0) at core/sr_module.c:839
#41 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c2310, rank=0) at core/sr_module.c:839
#42 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c2cf0, rank=0) at core/sr_module.c:839
#43 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c3340, rank=0) at core/sr_module.c:839
#44 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c3700, rank=0) at core/sr_module.c:839
#45 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c4590, rank=0) at core/sr_module.c:839
#46 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c47b0, rank=0) at core/sr_module.c:839
#47 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c49d0, rank=0) at core/sr_module.c:839
#48 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c4c90, rank=0) at core/sr_module.c:839
#49 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c5230, rank=0) at core/sr_module.c:839
#50 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c5530, rank=0) at core/sr_module.c:839
#51 0x000055ce6a0d5e59 in init_child (rank=0) at core/sr_module.c:870
#52 0x000055ce69f977cc in main_loop () at main.c:1703
#53 0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645

#0  0x00007f9958d2d3a3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055ce6a185727 in sleep_us (nusecs=100) at core/ut.h:513
#2  0x000055ce6a185898 in fork_basic_utimer (child_id=-1, desc=0x7f994f19434a "SIPDUMP WRITE TIMER", make_sock=1, 
    f=0x7f994f192e3c <sipdump_timer_exec>, param=0x0, uinterval=100) at core/timer_proc.c:116
#3  0x00007f994f18e782 in child_init (rank=0) at sipdump_mod.c:126
#4  0x000055ce6a0d5a84 in init_mod_child (m=0x7f99580b4270, rank=0) at core/sr_module.c:843
#5  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b44c0, rank=0) at core/sr_module.c:839
#6  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b47f0, rank=0) at core/sr_module.c:839
#7  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b4c80, rank=0) at core/sr_module.c:839
#8  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b5250, rank=0) at core/sr_module.c:839
#9  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b6e20, rank=0) at core/sr_module.c:839
#10 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b7490, rank=0) at core/sr_module.c:839
#11 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b77a0, rank=0) at core/sr_module.c:839
#12 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b7b20, rank=0) at core/sr_module.c:839
#13 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b7d70, rank=0) at core/sr_module.c:839
#14 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b87d0, rank=0) at core/sr_module.c:839
#15 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b8f50, rank=0) at core/sr_module.c:839
#16 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580b95d0, rank=0) at core/sr_module.c:839
#17 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580be830, rank=0) at core/sr_module.c:839
#18 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580beeb0, rank=0) at core/sr_module.c:839
#19 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bf220, rank=0) at core/sr_module.c:839
#20 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bf5c0, rank=0) at core/sr_module.c:839
#21 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bf810, rank=0) at core/sr_module.c:839
#22 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bfea0, rank=0) at core/sr_module.c:839
#23 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c01a0, rank=0) at core/sr_module.c:839
#24 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c0580, rank=0) at core/sr_module.c:839
#25 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c0f90, rank=0) at core/sr_module.c:839
#26 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c14c0, rank=0) at core/sr_module.c:839
#27 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c17d0, rank=0) at core/sr_module.c:839
#28 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c1d90, rank=0) at core/sr_module.c:839
#29 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c1fe0, rank=0) at core/sr_module.c:839
#30 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c2310, rank=0) at core/sr_module.c:839
#31 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c2cf0, rank=0) at core/sr_module.c:839
#32 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c3340, rank=0) at core/sr_module.c:839
#33 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c3700, rank=0) at core/sr_module.c:839
#34 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c4590, rank=0) at core/sr_module.c:839
#35 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c47b0, rank=0) at core/sr_module.c:839
#36 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c49d0, rank=0) at core/sr_module.c:839
#37 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c4c90, rank=0) at core/sr_module.c:839
#38 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c5230, rank=0) at core/sr_module.c:839
#39 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c5530, rank=0) at core/sr_module.c:839
#40 0x000055ce6a0d5e59 in init_child (rank=0) at core/sr_module.c:870
#41 0x000055ce69f977cc in main_loop () at main.c:1703
#42 0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645

#0  0x00007f9958d2d3a3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055ce6a185727 in sleep_us (nusecs=60000000) at core/ut.h:513
#2  0x000055ce6a185a8b in fork_sync_timer (child_id=-1, desc=0x7f994466af8a "USRLOC Timer", make_sock=1, 
    f=0x7f9944646573 <ul_local_timer>, param=0x0, interval=60000) at core/timer_proc.c:221
#3  0x00007f994464538a in child_init (_rank=0) at usrloc_mod.c:398
#4  0x000055ce6a0d5a84 in init_mod_child (m=0x7f99580bf5c0, rank=0) at core/sr_module.c:843
#5  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bf810, rank=0) at core/sr_module.c:839
#6  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580bfea0, rank=0) at core/sr_module.c:839
#7  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c01a0, rank=0) at core/sr_module.c:839
#8  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c0580, rank=0) at core/sr_module.c:839
#9  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c0f90, rank=0) at core/sr_module.c:839
#10 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c14c0, rank=0) at core/sr_module.c:839
#11 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c17d0, rank=0) at core/sr_module.c:839
#12 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c1d90, rank=0) at core/sr_module.c:839
#13 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c1fe0, rank=0) at core/sr_module.c:839
#14 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c2310, rank=0) at core/sr_module.c:839
#15 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c2cf0, rank=0) at core/sr_module.c:839
#16 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c3340, rank=0) at core/sr_module.c:839
#17 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c3700, rank=0) at core/sr_module.c:839
#18 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c4590, rank=0) at core/sr_module.c:839
#19 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c47b0, rank=0) at core/sr_module.c:839
#20 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c49d0, rank=0) at core/sr_module.c:839
#21 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c4c90, rank=0) at core/sr_module.c:839
#22 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c5230, rank=0) at core/sr_module.c:839
#23 0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c5530, rank=0) at core/sr_module.c:839
#24 0x000055ce6a0d5e59 in init_child (rank=0) at core/sr_module.c:870
#25 0x000055ce69f977cc in main_loop () at main.c:1703
#26 0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645

#0  0x00007f9958d04270 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f9958d041da in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55
#2  0x000055ce6a1857d1 in fork_basic_timer (child_id=-1, desc=0x7ffd8be61540 "RTIMER EXEC child=0 timer=minute_timer", 
    make_sock=1, f=0x7f994275d35a <stm_timer_exec>, param=0x7f99580d0190, interval=60) at core/timer_proc.c:77
#3  0x00007f994275cfd6 in child_init (rank=0) at rtimer_mod.c:166
#4  0x000055ce6a0d5a84 in init_mod_child (m=0x7f99580c4590, rank=0) at core/sr_module.c:843
#5  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c47b0, rank=0) at core/sr_module.c:839
#6  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c49d0, rank=0) at core/sr_module.c:839
#7  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c4c90, rank=0) at core/sr_module.c:839
#8  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c5230, rank=0) at core/sr_module.c:839
#9  0x000055ce6a0d5707 in init_mod_child (m=0x7f99580c5530, rank=0) at core/sr_module.c:839
#10 0x000055ce6a0d5e59 in init_child (rank=0) at core/sr_module.c:870
#11 0x000055ce69f977cc in main_loop () at main.c:1703
#12 0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645

#0  0x00007f9958d350a3 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055ce6a1d2713 in io_wait_loop_epoll (h=0x55ce6a653fc0 <io_w>, t=2, repeat=0) at core/io_wait.h:1034
#2  0x000055ce6a1e77bb in tcp_receive_loop (unix_sock=45) at core/tcp_read.c:1974
#3  0x000055ce6a09ae90 in tcp_init_children () at core/tcp_main.c:4853
#4  0x000055ce69f97ab2 in main_loop () at main.c:1711
#5  0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645

#0  0x00007f9958d350a3 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055ce6a05f47a in io_wait_loop_epoll (h=0x55ce6a60adc0 <io_h>, t=5, repeat=0) at core/io_wait.h:1034
#2  0x000055ce6a0971d1 in tcp_main_loop () at core/tcp_main.c:4547
#3  0x000055ce69f97e0c in main_loop () at main.c:1719
#4  0x000055ce69f9eb39 in main (argc=17, argv=0x7ffd8be61d18) at main.c:2645