From misak@misak.dk Thu Apr  7 21:02:34 2011
From: Morten Isaksen <misak@misak.dk>
To: sr-users@lists.kamailio.org
Subject: [SR-Users] Kamailio stops responding after 10 days or so
Date: Thu, 07 Apr 2011 21:02:28 +0200
Message-ID: <BANLkTi=o=DWLWF9c9KOtdsO=TEZMhdUp4Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0504317668=="

--===============0504317668==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Hi!

Kamailio 3.0.3.

I have a strange problem with one of our Kamailio servers. This one is
used for routing (with carrierroute) and to send presence information
(with pua module)

Once every 10 day or so I get this error and then Kamailio stops
responding to any SIP packets.

Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: WARNING:
<script>: Failure route - M=INVITE RURI=sip:8615XXXX(a)178.XX.XX.XX
F=sip:861XXXXX(a)188.120.93.114:1025 T=sip:86155XXXX(a)sip1.uni-tel.dk
IP=178.XX.XX.XX ID=6de881ec07f9c6494ee589cf208da358(a)10.11.87.206
Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
carrierroute [cr_func.c:95]: cannot find AVP 'carrier'
Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
carrierroute [cr_func.c:805]: invalid carrier id -1
Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
<script>: cr_next_domain failed


Shared memory size is 128M and over halv is free just before the
error. The server is in production and does handle debug>1 well, so I
do not have much information in the log files. Private memory is the
default size.

Any ideas what it could be, or how to investegate further?

I think my next steps would be to increase the private memory og to
increase children=4 to children=8

-- 
Morten Isaksen


--===============0504317668==--


From miconda@gmail.com Thu Apr  7 22:52:46 2011
From: Daniel-Constantin Mierla <miconda@gmail.com>
To: sr-users@lists.kamailio.org
Subject: Re: [SR-Users] Kamailio stops responding after 10 days or so
Date: Thu, 07 Apr 2011 22:52:36 +0200
Message-ID: <4D9E2414.1020007@gmail.com>
In-Reply-To: <BANLkTi=o=DWLWF9c9KOtdsO=TEZMhdUp4Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1558586619=="

--===============1558586619==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Hello,

do you get high CPU usage by kamailio?

What you can do is to attach with gdb to kamailio processes and see what 
they are doing:

gdb   /path/to/kamailio   pid_of_a_kamailio_process
bt

You should attach to the sip worker processes - you can find the type of 
processes with 'kamctl ps'.

Cheers,
Daniel

On 4/7/11 9:02 PM, Morten Isaksen wrote:
> Hi!
>
> Kamailio 3.0.3.
>
> I have a strange problem with one of our Kamailio servers. This one is
> used for routing (with carrierroute) and to send presence information
> (with pua module)
>
> Once every 10 day or so I get this error and then Kamailio stops
> responding to any SIP packets.
>
> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: WARNING:
> <script>: Failure route - M=INVITE RURI=sip:8615XXXX(a)178.XX.XX.XX
> F=sip:861XXXXX(a)188.120.93.114:1025 T=sip:86155XXXX(a)sip1.uni-tel.dk
> IP=178.XX.XX.XX ID=6de881ec07f9c6494ee589cf208da358(a)10.11.87.206
> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
> carrierroute [cr_func.c:95]: cannot find AVP 'carrier'
> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
> carrierroute [cr_func.c:805]: invalid carrier id -1
> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
> <script>: cr_next_domain failed
>
>
> Shared memory size is 128M and over halv is free just before the
> error. The server is in production and does handle debug>1 well, so I
> do not have much information in the log files. Private memory is the
> default size.
>
> Any ideas what it could be, or how to investegate further?
>
> I think my next steps would be to increase the private memory og to
> increase children=4 to children=8
>

-- 
Daniel-Constantin Mierla
http://www.asipto.com



--===============1558586619==--


From misak@misak.dk Thu Apr  7 23:00:24 2011
From: Morten Isaksen <misak@misak.dk>
To: sr-users@lists.kamailio.org
Subject: Re: [SR-Users] Kamailio stops responding after 10 days or so
Date: Thu, 07 Apr 2011 23:00:18 +0200
Message-ID: <BANLkTik+praiOxU9Wd3W8kHW==+yuOJg-w@mail.gmail.com>
In-Reply-To: <4D9E2414.1020007@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1481794296=="

--===============1481794296==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit

Hi,

No, normal CPU usage.

Thanks for the sugestion - I will try that next time.

/Morten

On Thu, Apr 7, 2011 at 10:52 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
> Hello,
>
> do you get high CPU usage by kamailio?
>
> What you can do is to attach with gdb to kamailio processes and see what
> they are doing:
>
> gdb   /path/to/kamailio   pid_of_a_kamailio_process
> bt
>
> You should attach to the sip worker processes - you can find the type of
> processes with 'kamctl ps'.
>
> Cheers,
> Daniel
>
> On 4/7/11 9:02 PM, Morten Isaksen wrote:
>>
>> Hi!
>>
>> Kamailio 3.0.3.
>>
>> I have a strange problem with one of our Kamailio servers. This one is
>> used for routing (with carrierroute) and to send presence information
>> (with pua module)
>>
>> Once every 10 day or so I get this error and then Kamailio stops
>> responding to any SIP packets.
>>
>> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: WARNING:
>> <script>: Failure route - M=INVITE RURI=sip:8615XXXX(a)178.XX.XX.XX
>> F=sip:861XXXXX(a)188.120.93.114:1025 T=sip:86155XXXX(a)sip1.uni-tel.dk
>> IP=178.XX.XX.XX ID=6de881ec07f9c6494ee589cf208da358(a)10.11.87.206
>> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>> carrierroute [cr_func.c:95]: cannot find AVP 'carrier'
>> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>> carrierroute [cr_func.c:805]: invalid carrier id -1
>> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>> <script>: cr_next_domain failed
>>
>>
>> Shared memory size is 128M and over halv is free just before the
>> error. The server is in production and does handle debug>1 well, so I
>> do not have much information in the log files. Private memory is the
>> default size.
>>
>> Any ideas what it could be, or how to investegate further?
>>
>> I think my next steps would be to increase the private memory og to
>> increase children=4 to children=8
>>
>
> --
> Daniel-Constantin Mierla
> http://www.asipto.com
>
>



-- 
Morten Isaksen


--===============1481794296==--


From marius.zbihlei@1and1.ro Fri Apr  8 11:11:05 2011
From: marius zbihlei <marius.zbihlei@1and1.ro>
To: sr-users@lists.kamailio.org
Subject: Re: [SR-Users] Kamailio stops responding after 10 days or so
Date: Fri, 08 Apr 2011 12:08:41 +0300
Message-ID: <4D9ED099.6020008@1and1.ro>
In-Reply-To: <BANLkTi=o=DWLWF9c9KOtdsO=TEZMhdUp4Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1738336648=="

--===============1738336648==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On 04/07/2011 10:02 PM, Morten Isaksen wrote:
> Hi!
>
> Kamailio 3.0.3.
>
> I have a strange problem with one of our Kamailio servers. This one is
> used for routing (with carrierroute) and to send presence information
> (with pua module)
>
> Once every 10 day or so I get this error and then Kamailio stops
> responding to any SIP packets.
>
> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: WARNING:
> <script>: Failure route - M=INVITE RURI=sip:8615XXXX(a)178.XX.XX.XX
> F=sip:861XXXXX(a)188.120.93.114:1025 T=sip:86155XXXX(a)sip1.uni-tel.dk
> IP=178.XX.XX.XX ID=6de881ec07f9c6494ee589cf208da358(a)10.11.87.206
> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
> carrierroute [cr_func.c:95]: cannot find AVP 'carrier'
> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
> carrierroute [cr_func.c:805]: invalid carrier id -1
> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
> <script>: cr_next_domain failed
>
>    
Hello Morten,

Do you have the carrier avp set in the failure route (it is ok if it's 
set  in normal route it should be visible from there)? Also can you 
paste the lines from failure_route that are around the cr_next_domain . 
My suspicion is that you have some calls that reach failure route, but 
because of some particular cfg lines, they are not correctly routed and 
the daemon just sleeps in some lines of code on each worker.

For replicating the bug and retrieve the line of code where the workers 
block(as Daniel suggested), it would be better if you decrease the 
number of worker and keep the same shm memory config value.

Marius

> Shared memory size is 128M and over halv is free just before the
> error. The server is in production and does handle debug>1 well, so I
> do not have much information in the log files. Private memory is the
> default size.
>
> Any ideas what it could be, or how to investegate further?
>
> I think my next steps would be to increase the private memory og to
> increase children=4 to children=8
>
>    



--===============1738336648==--


From misak@misak.dk Sat Apr  9 10:09:57 2011
From: Morten Isaksen <misak@misak.dk>
To: sr-users@lists.kamailio.org
Subject: Re: [SR-Users] Kamailio stops responding after 10 days or so
Date: Sat, 09 Apr 2011 10:09:50 +0200
Message-ID: <BANLkTinE-nGy14KFRdjOZLDA+FpC=wdgQQ@mail.gmail.com>
In-Reply-To: <4D9ED099.6020008@1and1.ro>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0539674901=="

--===============0539674901==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi Marius,

The carrier avp is always set in route[0].

My failure_route looks like this.

        xlog("L_WARN", "Failure route - M=3D$rm RURI=3D$ru F=3D$fu T=3D$tu
IP=3D$si ID=3D$ci\n");
        if (t_check_status("408|404|5[0-9][0-9]|6[0-9][0-9]") &&
!t_check_status("503")) {
                revert_uri();
                if (!cr_next_domain("$avp(s:carrier)", "$avp(s:domain)", "$rU=
",
                                        "$avp(s:host)",
"$T_reply_code", "$avp(s:domain)")) {
                        xlog("L_ERR", "cr_next_domain failed\n");
                        exit;
                }
                if (!cr_route("$avp(s:carrier)", "$avp(s:domain)", "$rU", "$r=
U",
                                        "call_id")) {
                        xlog("L_ERR", "cr_route failed\n");
                        exit;
                }
                $avp(s:host)=3D $rd;
                t_on_failure("COREROUTE");
                append_branch();
                xlog("L_WARN", "Outgoing M=3D$rm RURI=3D$ru F=3D$fu T=3D$tu
IP=3D$si ID=3D$ci\n");
                xlog("$si -> $rd");
                if (!t_relay()) {
                        xlog("L_ERR", "t_relay failed\n");
                        exit;
                };
        }



On Fri, Apr 8, 2011 at 11:08 AM, marius zbihlei <marius.zbihlei(a)1and1.ro> w=
rote:
> On 04/07/2011 10:02 PM, Morten Isaksen wrote:
>>
>> Hi!
>>
>> Kamailio 3.0.3.
>>
>> I have a strange problem with one of our Kamailio servers. This one is
>> used for routing (with carrierroute) and to send presence information
>> (with pua module)
>>
>> Once every 10 day or so I get this error and then Kamailio stops
>> responding to any SIP packets.
>>
>> Apr =C2=A06 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: WARNING:
>> <script>: Failure route - M=3DINVITE RURI=3Dsip:8615XXXX(a)178.XX.XX.XX
>> F=3Dsip:861XXXXX(a)188.120.93.114:1025 T=3Dsip:86155XXXX(a)sip1.uni-tel.dk
>> IP=3D178.XX.XX.XX ID=3D6de881ec07f9c6494ee589cf208da358(a)10.11.87.206
>> Apr =C2=A06 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>> carrierroute [cr_func.c:95]: cannot find AVP 'carrier'
>> Apr =C2=A06 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>> carrierroute [cr_func.c:805]: invalid carrier id -1
>> Apr =C2=A06 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>> <script>: cr_next_domain failed
>>
>>
>
> Hello Morten,
>
> Do you have the carrier avp set in the failure route (it is ok if it's set
> =C2=A0in normal route it should be visible from there)? Also can you paste =
the
> lines from failure_route that are around the cr_next_domain . My suspicion
> is that you have some calls that reach failure route, but because of some
> particular cfg lines, they are not correctly routed and the daemon just
> sleeps in some lines of code on each worker.
>
> For replicating the bug and retrieve the line of code where the workers
> block(as Daniel suggested), it would be better if you decrease the number of
> worker and keep the same shm memory config value.
>
> Marius
>
>> Shared memory size is 128M and over halv is free just before the
>> error. The server is in production and does handle debug>1 well, so I
>> do not have much information in the log files. Private memory is the
>> default size.
>>
>> Any ideas what it could be, or how to investegate further?
>>
>> I think my next steps would be to increase the private memory og to
>> increase children=3D4 to children=3D8
>>
>>
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users(a)lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>



--=20
Morten Isaksen


--===============0539674901==--


From marius.zbihlei@1and1.ro Mon Apr 11 10:37:27 2011
From: marius zbihlei <marius.zbihlei@1and1.ro>
To: sr-users@lists.kamailio.org
Subject: Re: [SR-Users] Kamailio stops responding after 10 days or so
Date: Mon, 11 Apr 2011 11:35:02 +0300
Message-ID: <4DA2BD36.1080203@1and1.ro>
In-Reply-To: <BANLkTinE-nGy14KFRdjOZLDA+FpC=wdgQQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1670023395=="

--===============1670023395==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On 04/09/2011 11:09 AM, Morten Isaksen wrote:
> Hi Marius,
>
>   =20

Hello,

Thanks for the update, but unfortunately I can't spot anything wrong. As=20
Daniel noted, please send a trace of the worker process (you can also=20
select the worker's pid from the log file)Attack with GDB for several=20
times and issue a bt (or bt full) command. If the traces are different,=20
this will help. Also do a ngrep to see that indeed you have traffic and=20
K is not responding at all.

Marius
> The carrier avp is always set in route[0].
>
> My failure_route looks like this.
>
>          xlog("L_WARN", "Failure route - M=3D$rm RURI=3D$ru F=3D$fu T=3D$tu
> IP=3D$si ID=3D$ci\n");
>          if (t_check_status("408|404|5[0-9][0-9]|6[0-9][0-9]")&&
> !t_check_status("503")) {
>                  revert_uri();
>                  if (!cr_next_domain("$avp(s:carrier)", "$avp(s:domain)", "=
$rU",
>                                          "$avp(s:host)",
> "$T_reply_code", "$avp(s:domain)")) {
>                          xlog("L_ERR", "cr_next_domain failed\n");
>                          exit;
>                  }
>                  if (!cr_route("$avp(s:carrier)", "$avp(s:domain)", "$rU", =
"$rU",
>                                          "call_id")) {
>                          xlog("L_ERR", "cr_route failed\n");
>                          exit;
>                  }
>                  $avp(s:host)=3D $rd;
>                  t_on_failure("COREROUTE");
>                  append_branch();
>                  xlog("L_WARN", "Outgoing M=3D$rm RURI=3D$ru F=3D$fu T=3D$tu
> IP=3D$si ID=3D$ci\n");
>                  xlog("$si ->  $rd");
>                  if (!t_relay()) {
>                          xlog("L_ERR", "t_relay failed\n");
>                          exit;
>                  };
>          }
>   =20



--===============1670023395==--


From misak@misak.dk Tue Apr 19 22:17:30 2011
From: Morten Isaksen <misak@misak.dk>
To: sr-users@lists.kamailio.org
Subject: Re: [SR-Users] Kamailio stops responding after 10 days or so
Date: Tue, 19 Apr 2011 22:17:23 +0200
Message-ID: <BANLkTikC9gPwV-_kZWKQfaiK7Opgafr44w@mail.gmail.com>
In-Reply-To: <4D9E2414.1020007@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1075978791=="

--===============1075978791==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi,

It happend again.

I got the result from gdb. Here is the output from a bt indside gdb
for each worker and one bt full for one of the workers.

They are all stuck in futexlock.h

My C debuging skills are a bit rusty so I hope one of you have some ideas.

/Morten

(gdb) bt
#0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
#1  0x00002acaf17b2e0d in futex_get (t=3D<value optimized out>,
type=3D<value optimized out>, ps=3D0x7fffafb7d7a0)
    at ../../mem/../futexlock.h:113
#2  publ_cback_func (t=3D<value optimized out>, type=3D<value optimized
out>, ps=3D0x7fffafb7d7a0) at send_publish.c:272
#3  0x00002acaeeb97815 in run_trans_callbacks_internal
(cb_lst=3D0x2acaf3cfd260, type=3D256, trans=3D0x2acaf3cfd1f0,
params=3D0x7fffafb7d7a0)
    at t_hooks.c:290
#4  0x00002acaeeb97a6e in run_trans_callbacks (type=3D0, trans=3D0x2,
req=3D0x0, rpl=3D0x246, code=3D0) at t_hooks.c:317
#5  0x00002acaeebbfcb3 in local_reply (t=3D0x2acaf3cfd1f0, p_msg=3D<value
optimized out>, branch=3D0, msg_status=3D<value optimized out>,
    cancel_bitmap=3D<value optimized out>) at t_reply.c:1847
#6  0x00002acaeebc29e8 in reply_received (p_msg=3D0x8d36c8) at t_reply.c:2133
#7  0x000000000044780e in forward_reply (msg=3D0x8d36c8) at forward.c:689
#8  0x000000000047ff02 in receive_msg (
    buf=3D0x874a40 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
178.21.248.8;branch=3Dz9hG4bKcd3a.d99d6403.0\r\nTo:
sip:+380432571592(a)sip.uni-tel.dk;user=3Dphone;tag=3D4e546d1f8c10c6f99af9c51=
d895c9c87-c9ca\r\nFrom:
sip:+380432571592(a)sip.uni-"..., len=3D<value optimized out>,
    rcv_info=3D0x7fffafb7dd20) at receive.c:257
#9  0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
#10 0x0000000000455cdf in main_loop () at main.c:1447
#11 0x0000000000456de2 in main (argc=3D<value optimized out>,
argv=3D0x7fffafb7dfe8) at main.c:2251


(gdb) bt
#0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
#1  0x00002acaeeb7ea32 in futex_get (i=3D<value optimized out>) at
../../mem/../futexlock.h:100
#2  _lock (i=3D<value optimized out>) at lock.h:98
#3  lock_hash (i=3D<value optimized out>) at h_table.c:98
#4  0x00002acaeeba1b99 in t_lookup_request (p_msg=3D0x8d36c8,
leave_new_locked=3D0, cancel=3D0x7fffafb7d5e0) at t_lookup.c:548
#5  0x00002acaeeba3532 in t_check_msg (p_msg=3D0x8d36c8,
param_branch=3D<value optimized out>) at t_lookup.c:1104
#6  0x00002acaeebab7b1 in t_check_trans (msg=3D0x2acaf1f4b2e8,
foo=3D<value optimized out>, bar=3D0x2 <Address 0x2 out of bounds>) at
tm.c:1881
#7  0x000000000041345c in do_action (h=3D0x7fffafb7dab0, a=3D0x8a6f98,
msg=3D0x8d36c8) at action.c:860
#8  0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x89f8d8,
msg=3D0x8d36c8) at action.c:1293
#9  0x0000000000416084 in run_top_route (a=3D0x89f8d8, msg=3D0x8d36c8,
c=3D<value optimized out>) at action.c:1341
#10 0x000000000047ff4c in receive_msg (
    buf=3D0x874a40 "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
SIP/2.0\r\nRecord-Route:
<sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
81.27."..., len=3D<value optimized out>,
    rcv_info=3D0x7fffafb7dd20) at receive.c:196
#11 0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
#12 0x0000000000455cdf in main_loop () at main.c:1447
#13 0x0000000000456de2 in main (argc=3D<value optimized out>,
argv=3D0x7fffafb7dfe8) at main.c:2251

#0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
#1  0x00002acaeeb7ea32 in futex_get (i=3D<value optimized out>) at
../../mem/../futexlock.h:100
#2  _lock (i=3D<value optimized out>) at lock.h:98
#3  lock_hash (i=3D<value optimized out>) at h_table.c:98
#4  0x00002acaeeba1b99 in t_lookup_request (p_msg=3D0x8d36c8,
leave_new_locked=3D0, cancel=3D0x7fffafb7d5e0) at t_lookup.c:548
#5  0x00002acaeeba3532 in t_check_msg (p_msg=3D0x8d36c8,
param_branch=3D<value optimized out>) at t_lookup.c:1104
#6  0x00002acaeebab7b1 in t_check_trans (msg=3D0x2acaf1f4b2e8,
foo=3D<value optimized out>, bar=3D0x2 <Address 0x2 out of bounds>) at
tm.c:1881
#7  0x000000000041345c in do_action (h=3D0x7fffafb7dab0, a=3D0x8a6f98,
msg=3D0x8d36c8) at action.c:860
#8  0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x89f8d8,
msg=3D0x8d36c8) at action.c:1293
#9  0x0000000000416084 in run_top_route (a=3D0x89f8d8, msg=3D0x8d36c8,
c=3D<value optimized out>) at action.c:1341
#10 0x000000000047ff4c in receive_msg (
    buf=3D0x874a40 "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
SIP/2.0\r\nRecord-Route:
<sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
81.27."..., len=3D<value optimized out>,
    rcv_info=3D0x7fffafb7dd20) at receive.c:196
#11 0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
#12 0x0000000000455cdf in main_loop () at main.c:1447
#13 0x0000000000456de2 in main (argc=3D<value optimized out>,
argv=3D0x7fffafb7dfe8) at main.c:2251

(gdb) bt
#0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
#1  0x00002acaeeb7e9fa in futex_get (i=3D<value optimized out>) at
../../mem/../futexlock.h:113
#2  _lock (i=3D<value optimized out>) at lock.h:98
#3  lock_hash (i=3D<value optimized out>) at h_table.c:98
#4  0x00002acaeebca0e3 in t_uac_prepare (uac_r=3D0x7fffafb7c270,
dst_req=3D0x7fffafb7c060, dst_cell=3D0x7fffafb7c058) at uac.c:319
#5  0x00002acaeebcb4ae in t_uac_with_ids (uac_r=3D0x2acaf1f4b2e8,
ret_index=3D0x0, ret_label=3D0x0) at uac.c:531
#6  0x00002acaeebcc9c7 in request (uac_r=3D0x7fffafb7c270, ruri=3D0x11,
to=3D0x1b, from=3D0x8e0fe0, next_hop=3D0x2acaf19c01c0) at uac.c:778
#7  0x00002acaf17af3ff in send_publish (publ=3D<value optimized out>) at
send_publish.c:556
#8  0x00002acaf19c4f67 in dialog_publish (state=3D<value optimized out>,
entity=3D<value optimized out>, peer=3D<value optimized out>,
    callid=3D<value optimized out>, initiator=3D<value optimized out>,
lifetime=3D43200, localtag=3D0x0, remotetag=3D0x0, localtarget=3D0x0,
    remotetarget=3D0x0) at dialog_publish.c:349
#9  0x00002acaf19c6880 in __dialog_created (dlg=3D0x2acaf575eb38,
type=3D<value optimized out>, _params=3D<value optimized out>)
    at pua_dialoginfo.c:351
#10 0x00002acaf134ef0f in run_create_callbacks (dlg=3D0x2acaf575eb38,
msg=3D<value optimized out>) at dlg_cb.c:230
#11 0x00002acaf13563c7 in dlg_new_dialog (msg=3D0x8d36c8,
t=3D0x2acaf157d940) at dlg_handlers.c:643
#12 0x00002acaeeb9751a in run_reqin_callbacks_internal
(hl=3D0x2acaf1e55018, trans=3D0x2acaf6fa1630, params=3D0x2acaeede2920) at
t_hooks.c:376
#13 0x00002acaeeb7ec5a in build_cell (p_msg=3D0x8d36c8) at h_table.c:346
#14 0x00002acaeeba240e in new_t (p_msg=3D0x8d36c8) at t_lookup.c:1332
#15 t_newtran (p_msg=3D0x8d36c8) at t_lookup.c:1472
#16 0x00002acaeeb8d92b in t_relay_to (p_msg=3D0x2acaf1f4b2e8, proxy=3D0x0,
proto=3D0, replicate=3D0) at t_funcs.c:272
#17 0x000000000041345c in do_action (h=3D0x7fffafb7dab0, a=3D0x8c8918,
msg=3D0x8d36c8) at action.c:860
#18 0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x8c8918,
msg=3D0x8d36c8) at action.c:1293
#19 0x0000000000496815 in rval_get_int (h=3D0x2acaf1f4b2e8, msg=3D0x0,
i=3D0x7fffafb7d1ac, rv=3D0xffffffffffffffff, cache=3D0x0) at rvalue.c:897
#20 0x000000000049b68c in rval_expr_eval_int (h=3D0x7fffafb7dab0,
msg=3D0x8d36c8, res=3D0x7fffafb7d1ac, rve=3D0x8c8a08) at rvalue.c:1837
#21 0x000000000049b73d in rval_expr_eval_int (h=3D0x7fffafb7dab0,
msg=3D0x8d36c8, res=3D0x7fffafb7d5b8, rve=3D0x8c90d8) at rvalue.c:1844
#22 0x0000000000411d16 in do_action (h=3D0x7fffafb7dab0, a=3D0x8c9928,
msg=3D0x8d36c8) at action.c:834
#23 0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x8bda00,
msg=3D0x8d36c8) at action.c:1293
#24 0x0000000000413054 in do_action (h=3D0x7fffafb7dab0, a=3D0x8bd7f0,
msg=3D0x8d36c8) at action.c:479
#25 0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x89f8d8,
msg=3D0x8d36c8) at action.c:1293
#26 0x0000000000416084 in run_top_route (a=3D0x89f8d8, msg=3D0x8d36c8,
c=3D<value optimized out>) at action.c:1341
#27 0x000000000047ff4c in receive_msg (
    buf=3D0x874a40 "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
SIP/2.0\r\nRecord-Route: <sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia
            : SIP/2.0/UDP
178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
81.27."..., len=3D<value optimized out>,
    rcv_info=3D0x7fffafb7dd20) at receive.c:196
#28 0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
#29 0x0000000000455cdf in main_loop () at main.c:1447
#30 0x0000000000456de2 in main (argc=3D<value optimized out>,
argv=3D0x7fffafb7dfe8) at main.c:2251

(gdb) bt full
#0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
No symbol table info available.
#1  0x00002acaeeb7ea32 in futex_get (i=3D<value optimized out>) at
../../mem/../futexlock.h:100
        v =3D <value optimized out>
#2  _lock (i=3D<value optimized out>) at lock.h:98
No locals.
#3  lock_hash (i=3D<value optimized out>) at h_table.c:98
No locals.
#4  0x00002acaeeba1b99 in t_lookup_request (p_msg=3D0x8d36c8,
leave_new_locked=3D0, cancel=3D0x7fffafb7d5e0) at t_lookup.c:548
        p_cell =3D <value optimized out>
        isACK =3D 0
        t_msg =3D <value optimized out>
        branch =3D 0xfffffffffffffffc
        match_status =3D <value optimized out>
        hash_bucket =3D <value optimized out>
#5  0x00002acaeeba3532 in t_check_msg (p_msg=3D0x8d36c8,
param_branch=3D<value optimized out>) at t_lookup.c:1104
        local_branch =3D 0
        canceled =3D 9042808
        ret =3D <value optimized out>
#6  0x00002acaeebab7b1 in t_check_trans (msg=3D0x2acaf1f4b2e8,
foo=3D<value optimized out>, bar=3D0x2 <Address 0x2 out of bounds>) at
tm.c:1881
        t =3D <value optimized out>
#7  0x000000000041345c in do_action (h=3D0x7fffafb7dab0, a=3D0x8a6f98,
msg=3D0x8d36c8) at action.c:860
        ret =3D -5
        v =3D <value optimized out>
        dst =3D {send_sock =3D 0x2acaedccc000, to =3D {s =3D {sa_family =3D
37544, sa_data =3D "\220\000\000\000\000\000`\253\220\000\000\000\000"},
sin =3D {
              sin_family =3D 37544, sin_port =3D 144, sin_addr =3D {s_addr =3D
0}, sin_zero =3D "`\253\220\000\000\000\000"}, sin6 =3D {sin6_family =3D
37544,
              sin6_port =3D 144, sin6_flowinfo =3D 0, sin6_addr =3D {in6_u =3D
{u6_addr8 =3D "`\253\220\000\000\000\000\000LK\207\000\000\000\000",
u6_addr16 =3D {
                    43872, 144, 0, 0, 19276, 135, 0, 0}, u6_addr32 =3D
{9481056, 0, 8866636, 0}}}, sin6_scope_id =3D 9480096}}, id =3D 0, proto =3D
-46 '\322',
          send_flags =3D 74 'J'}
        tmp =3D <value optimized out>
        new_uri =3D <value optimized out>
        end =3D <value optimized out>
        crt =3D <value optimized out>
        f =3D 0xfffffffffffffffc
        len =3D <value optimized out>
        user =3D <value optimized out>
        uri =3D {user =3D {s =3D 0x7fffafb7da60 "\310\066\215", len =3D 0},
passwd =3D {s =3D 0x7fffafb7da40 "\001", len =3D 5767096}, host =3D {
            s =3D 0x1be1460 "8*\225\233\062", len =3D -1684723232}, port =3D
{s =3D 0x100000000 <Address 0x100000000 out of bounds>, len =3D 134},
params =3D {
            s =3D 0x0, len =3D 29234864}, sip_params =3D {s =3D 0x100000000
<Address 0x100000000 out of bounds>, len =3D -1687727470}, headers =3D {
            s =3D 0x36b7 <Address 0x36b7 out of bounds>, len =3D 8208},
port_no =3D 24939, proto =3D 24941, type =3D 1869180009, flags =3D 21,
transport =3D {
---Type <return> to continue, or q <return> to quit---
            s =3D 0x1be1460 "8*\225\233\062", len =3D 0}, ttl =3D {s =3D 0x0,
len =3D 21}, user_param =3D {s =3D 0x1be1460 "8*\225\233\062", len =3D 21},
maddr =3D {
            s =3D 0x0, len =3D -1346905536}, method =3D {s =3D 0x57ffb8 "%s:
%sERROR: forward_reply: no 2nd via found in reply\n", len =3D
-1687738517}, lr =3D {
            s =3D 0x18 <Address 0x18 out of bounds>, len =3D 21}, r2 =3D {s
=3D 0x15 <Address 0x15 out of bounds>, len =3D 29234272}, transport_val =3D
{
            s =3D 0x15 <Address 0x15 out of bounds>, len =3D 21}, ttl_val
=3D {s =3D 0x1be1460 "8*\225\233\062", len =3D -1687354197}, user_param_val
=3D {
            s =3D 0x2a00000033 <Address 0x2a00000033 out of bounds>, len
=3D 5458017}, maddr_val =3D {s =3D 0x6f00000003 <Address 0x6f00000003 out of
bounds>,
            len =3D 2}, method_val =3D {s =3D 0x1 <Address 0x1 out of
bounds>, len =3D 7200}, lr_val =3D {s =3D 0x1bacb40 "CEST", len =3D 5267113},
r2_val =3D {
            s =3D 0x90a606 "", len =3D -1346905500}}
        next_hop =3D {user =3D {s =3D 0x10 <Address 0x10 out of bounds>, len
=3D -1687029130}, passwd =3D {s =3D 0x1ff4 <Address 0x1ff4 out of bounds>,
            len =3D -1687023672}, host =3D {s =3D 0x8 <Address 0x8 out of
bounds>, len =3D 29234892}, port =3D {
            s =3D 0x1be16cc "16:42:57 /usr/local/sbin/kamailio[14007]:
WARNING: <script>: ua=3DCEC-PBX", len =3D -1684723232}, params =3D {s =3D
0x329b952a00 "",
            len =3D -1684723200}, sip_params =3D {s =3D 0x2acaf3e05470 "",
len =3D -1684723144}, headers =3D {s =3D 0x329b94fae0 "@=C3=ADen =3D -1346905=
792},
          port_no =3D 16, proto =3D 0, type =3D 48, flags =3D 2948061504,
transport =3D {s =3D 0x7fffafb7d880 "", len =3D 29235008}, ttl =3D {
            s =3D 0x1e8c0 <Address 0x1e8c0 out of bounds>, len =3D
29243072}, user_param =3D {s =3D 0x329b722698 "double free or corruption
(out)",
            len =3D 117056}, maddr =3D {s =3D 0x329b9529e0 "", len =3D
-1687739828}, method =3D {s =3D 0x329b9535a0 "", len =3D 5767096}, lr =3D {
            s =3D 0x1 <Address 0x1 out of bounds>, len =3D -1687631697},
r2 =3D {s =3D 0x7fffafb7dad8 "\330", <incomplete sequence \370\211>, len =3D
8231792},
          transport_val =3D {s =3D 0x7fffafb7dd8c "\020", len =3D
-289787385}, ttl_val =3D {s =3D 0x8d36c8 "\033x\r", len =3D 412},
user_param_val =3D {
            s =3D 0x313500000000 <Address 0x313500000000 out of bounds>,
len =3D 29243072}, maddr_val =3D {s =3D 0x2010 <Address 0x2010 out of
bounds>,
            len =3D 134}, method_val =3D {s =3D 0x2010 <Address 0x2010 out
of bounds>, len =3D 29234864}, lr_val =3D {s =3D 0x90 <Address 0x90 out of
bounds>,
            len =3D -1687731159}, r2_val =3D {s =3D 0x3000000010 <Address
0x3000000010 out of bounds>, len =3D -1684723232}}
        u =3D <value optimized out>
        port =3D <value optimized out>
        i =3D <value optimized out>
        flags =3D <value optimized out>
        sct =3D <value optimized out>
        sjt =3D <value optimized out>
        rve =3D <value optimized out>
        mct =3D <value optimized out>
        rv =3D <value optimized out>
        rv1 =3D <value optimized out>
        c1 =3D {cache_type =3D 8866741, val_type =3D RV_NONE, c =3D {avp_val =
=3D
{n =3D 133, s =3D {s =3D 0x85 <Address 0x85 out of bounds>, len =3D 29234880},
              re =3D 0x85}, pval =3D {rs =3D {s =3D 0x85 <Address 0x85 out of
bounds>, len =3D 29234880}, ri =3D 1303224171, flags =3D 0}}}
#8  0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x89f8d8,
msg=3D0x8d36c8) at action.c:1293
        t =3D 0x8a6f98
        ret =3D 1
        mod =3D <value optimized out>
#9  0x0000000000416084 in run_top_route (a=3D0x89f8d8, msg=3D0x8d36c8,
c=3D<value optimized out>) at action.c:1341
        ctx =3D {rec_lev =3D 1, run_flags =3D 0, last_retcode =3D -1, jmp_env
=3D {{__jmpbuf =3D {9254600, -7918255915006962580, 0, 9042136, 9482320,
                140736141450636, -7918229962976915956,
-7918255915011241414}, __mask_was_saved =3D 0, __saved_mask =3D {__val =3D
{9254600, 2, 5317260, 928,
                  8866420, 0, 8866368, 9254608, 1, 140736141450528,
9254600, 8866429, 5321012, 9479616, 47051076858778, 9295328}}}}}
        p =3D 0x7fffafb7dab0
---Type <return> to continue, or q <return> to quit---
        ret =3D <value optimized out>
        sfbk =3D 0
#10 0x000000000047ff4c in receive_msg (
    buf=3D0x874a40 "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
SIP/2.0\r\nRecord-Route:
<sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
81.27."..., len=3D<value optimized out>, rcv_info=3D0x7fffafb7dd20) at
receive.c:196
        msg =3D 0x0
        ctx =3D {rec_lev =3D 9034719, run_flags =3D 0, last_retcode =3D
9295720, jmp_env =3D {{__jmpbuf =3D {47051129731136, 8, 47051127553476, 2,
217355168616,
                47051061455248, 47051127553501, 4294967295},
__mask_was_saved =3D -236996544, __saved_mask =3D {__val =3D {8230504,
9007856, 16, 9482320,
                  140736141450636, 0, 217350982997, 1, 0, 8963168,
47051127486569, 217355168616, 2, 150476210, 217351006434, 0}}}}}
        ret =3D <value optimized out>
        inb =3D {
          s =3D 0x874a40 "INVITE
sip:20322595(a)178.21.248.8:5060;transport=3Dudp SIP/2.0\r\nRecord-Route:
<sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
81.27."..., len =3D 928}
#11 0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
        len =3D 928
        tmp =3D <value optimized out>
        from =3D <value optimized out>
        fromlen =3D 16
        ri =3D {src_ip =3D {af =3D 2, len =3D 4, u =3D {addrl =3D {351802802,
420}, addr32 =3D {351802802, 0, 420, 0}, addr16 =3D {5554, 5368, 0, 0,
420, 0, 0, 0},
              addr =3D
"\262\025\370\024\000\000\000\000\244\001\000\000\000\000\000"}},
dst_ip =3D {af =3D 2, len =3D 4, u =3D {addrl =3D {150476210, 0},
              addr32 =3D {150476210, 0, 0, 0}, addr16 =3D {5554, 2296, 0,
0, 0, 0, 0, 0}, addr =3D "\262\025\370\b", '\000' <repeats 11 times>}},
          src_port =3D 5060, dst_port =3D 5060, proto_reserved1 =3D 0,
proto_reserved2 =3D 0, src_su =3D {s =3D {sa_family =3D 2,
              sa_data =3D
"\023=C3=84=C2=B2\025\370\024\000\000\000\000\000\000\000"}, sin =3D {sin_fam=
ily =3D
2, sin_port =3D 50195, sin_addr =3D {s_addr =3D 351802802},
              sin_zero =3D "\000\000\000\000\000\000\000"}, sin6 =3D
{sin6_family =3D 2, sin6_port =3D 50195, sin6_flowinfo =3D 351802802,
sin6_addr =3D {in6_u =3D {
                  u6_addr8 =3D '\000' <repeats 15 times>, u6_addr16 =3D
{0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 =3D {0, 0, 0, 0}}}, sin6_scope_id =3D
0}},
          bind_address =3D 0x8972f0, proto =3D 1 '\001'}
        buf =3D "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
SIP/2.0\r\nRecord-Route:
<sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
81.27."...
#12 0x0000000000455cdf in main_loop () at main.c:1447
        i =3D 2
        pid =3D <value optimized out>
        si =3D 0x8972f0
        si_desc =3D "udp receiver child=3D2
sock=3D178.21.248.8:5060\000\000\000\000\000\004", '\000' <repeats 11
times>, "\001", '\000' <repeats 11 times>,
"\b\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\004",
'\000' <repeats 23 times>"\350,
\337\267\257\377\177\000\000\252\267H\000\000\000\000"
#13 0x0000000000456de2 in main (argc=3D<value optimized out>,
argv=3D0x7fffafb7dfe8) at main.c:2251
        cfg_stream =3D 0x1b95010
        c =3D <value optimized out>
        r =3D <value optimized out>
        tmp =3D 0x7fffafb7ef76 ""
---Type <return> to continue, or q <return> to quit---
        tmp_len =3D 16777216
        port =3D 0
        proto =3D 0
        ret =3D <value optimized out>
        seed =3D 524492455
        rfd =3D <value optimized out>
        debug_save =3D 0
        debug_flag =3D 0
        dont_fork_cnt =3D 0
        n_lst =3D 0x0





On Thu, Apr 7, 2011 at 10:52 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
> Hello,
>
> do you get high CPU usage by kamailio?
>
> What you can do is to attach with gdb to kamailio processes and see what
> they are doing:
>
> gdb =C2=A0 /path/to/kamailio =C2=A0 pid_of_a_kamailio_process
> bt
>
> You should attach to the sip worker processes - you can find the type of
> processes with 'kamctl ps'.
>
> Cheers,
> Daniel
>
> On 4/7/11 9:02 PM, Morten Isaksen wrote:
>>
>> Hi!
>>
>> Kamailio 3.0.3.
>>
>> I have a strange problem with one of our Kamailio servers. This one is
>> used for routing (with carrierroute) and to send presence information
>> (with pua module)
>>
>> Once every 10 day or so I get this error and then Kamailio stops
>> responding to any SIP packets.
>>
>> Apr =C2=A06 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: WARNING:
>> <script>: Failure route - M=3DINVITE RURI=3Dsip:8615XXXX(a)178.XX.XX.XX
>> F=3Dsip:861XXXXX(a)188.120.93.114:1025 T=3Dsip:86155XXXX(a)sip1.uni-tel.dk
>> IP=3D178.XX.XX.XX ID=3D6de881ec07f9c6494ee589cf208da358(a)10.11.87.206
>> Apr =C2=A06 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>> carrierroute [cr_func.c:95]: cannot find AVP 'carrier'
>> Apr =C2=A06 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>> carrierroute [cr_func.c:805]: invalid carrier id -1
>> Apr =C2=A06 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>> <script>: cr_next_domain failed
>>
>>
>> Shared memory size is 128M and over halv is free just before the
>> error. The server is in production and does handle debug>1 well, so I
>> do not have much information in the log files. Private memory is the
>> default size.
>>
>> Any ideas what it could be, or how to investegate further?
>>
>> I think my next steps would be to increase the private memory og to
>> increase children=3D4 to children=3D8
>>
>
> --
> Daniel-Constantin Mierla
> http://www.asipto.com
>
>



--=20
Morten Isaksen


--===============1075978791==--


From miconda@gmail.com Tue Apr 26 10:50:34 2011
From: Daniel-Constantin Mierla <miconda@gmail.com>
To: sr-users@lists.kamailio.org
Subject: Re: [SR-Users] Kamailio stops responding after 10 days or so
Date: Tue, 26 Apr 2011 10:50:24 +0200
Message-ID: <4DB68750.608@gmail.com>
In-Reply-To: <BANLkTikC9gPwV-_kZWKQfaiK7Opgafr44w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0801942135=="

--===============0801942135==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hello,

thanks for feedback. Going to work on it asap.

Cheers,
Daniel

On 4/19/11 10:17 PM, Morten Isaksen wrote:
> Hi,
>
> It happend again.
>
> I got the result from gdb. Here is the output from a bt indside gdb
> for each worker and one bt full for one of the workers.
>
> They are all stuck in futexlock.h
>
> My C debuging skills are a bit rusty so I hope one of you have some ideas.
>
> /Morten
>
> (gdb) bt
> #0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
> #1  0x00002acaf17b2e0d in futex_get (t=3D<value optimized out>,
> type=3D<value optimized out>, ps=3D0x7fffafb7d7a0)
>      at ../../mem/../futexlock.h:113
> #2  publ_cback_func (t=3D<value optimized out>, type=3D<value optimized
> out>, ps=3D0x7fffafb7d7a0) at send_publish.c:272
> #3  0x00002acaeeb97815 in run_trans_callbacks_internal
> (cb_lst=3D0x2acaf3cfd260, type=3D256, trans=3D0x2acaf3cfd1f0,
> params=3D0x7fffafb7d7a0)
>      at t_hooks.c:290
> #4  0x00002acaeeb97a6e in run_trans_callbacks (type=3D0, trans=3D0x2,
> req=3D0x0, rpl=3D0x246, code=3D0) at t_hooks.c:317
> #5  0x00002acaeebbfcb3 in local_reply (t=3D0x2acaf3cfd1f0, p_msg=3D<value
> optimized out>, branch=3D0, msg_status=3D<value optimized out>,
>      cancel_bitmap=3D<value optimized out>) at t_reply.c:1847
> #6  0x00002acaeebc29e8 in reply_received (p_msg=3D0x8d36c8) at t_reply.c:21=
33
> #7  0x000000000044780e in forward_reply (msg=3D0x8d36c8) at forward.c:689
> #8  0x000000000047ff02 in receive_msg (
>      buf=3D0x874a40 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
> 178.21.248.8;branch=3Dz9hG4bKcd3a.d99d6403.0\r\nTo:
> sip:+380432571592(a)sip.uni-tel.dk;user=3Dphone;tag=3D4e546d1f8c10c6f99af9c=
51d895c9c87-c9ca\r\nFrom:
> sip:+380432571592(a)sip.uni-"..., len=3D<value optimized out>,
>      rcv_info=3D0x7fffafb7dd20) at receive.c:257
> #9  0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
> #10 0x0000000000455cdf in main_loop () at main.c:1447
> #11 0x0000000000456de2 in main (argc=3D<value optimized out>,
> argv=3D0x7fffafb7dfe8) at main.c:2251
>
>
> (gdb) bt
> #0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
> #1  0x00002acaeeb7ea32 in futex_get (i=3D<value optimized out>) at
> ../../mem/../futexlock.h:100
> #2  _lock (i=3D<value optimized out>) at lock.h:98
> #3  lock_hash (i=3D<value optimized out>) at h_table.c:98
> #4  0x00002acaeeba1b99 in t_lookup_request (p_msg=3D0x8d36c8,
> leave_new_locked=3D0, cancel=3D0x7fffafb7d5e0) at t_lookup.c:548
> #5  0x00002acaeeba3532 in t_check_msg (p_msg=3D0x8d36c8,
> param_branch=3D<value optimized out>) at t_lookup.c:1104
> #6  0x00002acaeebab7b1 in t_check_trans (msg=3D0x2acaf1f4b2e8,
> foo=3D<value optimized out>, bar=3D0x2<Address 0x2 out of bounds>) at
> tm.c:1881
> #7  0x000000000041345c in do_action (h=3D0x7fffafb7dab0, a=3D0x8a6f98,
> msg=3D0x8d36c8) at action.c:860
> #8  0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x89f8d8,
> msg=3D0x8d36c8) at action.c:1293
> #9  0x0000000000416084 in run_top_route (a=3D0x89f8d8, msg=3D0x8d36c8,
> c=3D<value optimized out>) at action.c:1341
> #10 0x000000000047ff4c in receive_msg (
>      buf=3D0x874a40 "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
> SIP/2.0\r\nRecord-Route:
> <sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
> 178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
> 81.27."..., len=3D<value optimized out>,
>      rcv_info=3D0x7fffafb7dd20) at receive.c:196
> #11 0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
> #12 0x0000000000455cdf in main_loop () at main.c:1447
> #13 0x0000000000456de2 in main (argc=3D<value optimized out>,
> argv=3D0x7fffafb7dfe8) at main.c:2251
>
> #0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
> #1  0x00002acaeeb7ea32 in futex_get (i=3D<value optimized out>) at
> ../../mem/../futexlock.h:100
> #2  _lock (i=3D<value optimized out>) at lock.h:98
> #3  lock_hash (i=3D<value optimized out>) at h_table.c:98
> #4  0x00002acaeeba1b99 in t_lookup_request (p_msg=3D0x8d36c8,
> leave_new_locked=3D0, cancel=3D0x7fffafb7d5e0) at t_lookup.c:548
> #5  0x00002acaeeba3532 in t_check_msg (p_msg=3D0x8d36c8,
> param_branch=3D<value optimized out>) at t_lookup.c:1104
> #6  0x00002acaeebab7b1 in t_check_trans (msg=3D0x2acaf1f4b2e8,
> foo=3D<value optimized out>, bar=3D0x2<Address 0x2 out of bounds>) at
> tm.c:1881
> #7  0x000000000041345c in do_action (h=3D0x7fffafb7dab0, a=3D0x8a6f98,
> msg=3D0x8d36c8) at action.c:860
> #8  0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x89f8d8,
> msg=3D0x8d36c8) at action.c:1293
> #9  0x0000000000416084 in run_top_route (a=3D0x89f8d8, msg=3D0x8d36c8,
> c=3D<value optimized out>) at action.c:1341
> #10 0x000000000047ff4c in receive_msg (
>      buf=3D0x874a40 "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
> SIP/2.0\r\nRecord-Route:
> <sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
> 178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
> 81.27."..., len=3D<value optimized out>,
>      rcv_info=3D0x7fffafb7dd20) at receive.c:196
> #11 0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
> #12 0x0000000000455cdf in main_loop () at main.c:1447
> #13 0x0000000000456de2 in main (argc=3D<value optimized out>,
> argv=3D0x7fffafb7dfe8) at main.c:2251
>
> (gdb) bt
> #0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
> #1  0x00002acaeeb7e9fa in futex_get (i=3D<value optimized out>) at
> ../../mem/../futexlock.h:113
> #2  _lock (i=3D<value optimized out>) at lock.h:98
> #3  lock_hash (i=3D<value optimized out>) at h_table.c:98
> #4  0x00002acaeebca0e3 in t_uac_prepare (uac_r=3D0x7fffafb7c270,
> dst_req=3D0x7fffafb7c060, dst_cell=3D0x7fffafb7c058) at uac.c:319
> #5  0x00002acaeebcb4ae in t_uac_with_ids (uac_r=3D0x2acaf1f4b2e8,
> ret_index=3D0x0, ret_label=3D0x0) at uac.c:531
> #6  0x00002acaeebcc9c7 in request (uac_r=3D0x7fffafb7c270, ruri=3D0x11,
> to=3D0x1b, from=3D0x8e0fe0, next_hop=3D0x2acaf19c01c0) at uac.c:778
> #7  0x00002acaf17af3ff in send_publish (publ=3D<value optimized out>) at
> send_publish.c:556
> #8  0x00002acaf19c4f67 in dialog_publish (state=3D<value optimized out>,
> entity=3D<value optimized out>, peer=3D<value optimized out>,
>      callid=3D<value optimized out>, initiator=3D<value optimized out>,
> lifetime=3D43200, localtag=3D0x0, remotetag=3D0x0, localtarget=3D0x0,
>      remotetarget=3D0x0) at dialog_publish.c:349
> #9  0x00002acaf19c6880 in __dialog_created (dlg=3D0x2acaf575eb38,
> type=3D<value optimized out>, _params=3D<value optimized out>)
>      at pua_dialoginfo.c:351
> #10 0x00002acaf134ef0f in run_create_callbacks (dlg=3D0x2acaf575eb38,
> msg=3D<value optimized out>) at dlg_cb.c:230
> #11 0x00002acaf13563c7 in dlg_new_dialog (msg=3D0x8d36c8,
> t=3D0x2acaf157d940) at dlg_handlers.c:643
> #12 0x00002acaeeb9751a in run_reqin_callbacks_internal
> (hl=3D0x2acaf1e55018, trans=3D0x2acaf6fa1630, params=3D0x2acaeede2920) at
> t_hooks.c:376
> #13 0x00002acaeeb7ec5a in build_cell (p_msg=3D0x8d36c8) at h_table.c:346
> #14 0x00002acaeeba240e in new_t (p_msg=3D0x8d36c8) at t_lookup.c:1332
> #15 t_newtran (p_msg=3D0x8d36c8) at t_lookup.c:1472
> #16 0x00002acaeeb8d92b in t_relay_to (p_msg=3D0x2acaf1f4b2e8, proxy=3D0x0,
> proto=3D0, replicate=3D0) at t_funcs.c:272
> #17 0x000000000041345c in do_action (h=3D0x7fffafb7dab0, a=3D0x8c8918,
> msg=3D0x8d36c8) at action.c:860
> #18 0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x8c8918,
> msg=3D0x8d36c8) at action.c:1293
> #19 0x0000000000496815 in rval_get_int (h=3D0x2acaf1f4b2e8, msg=3D0x0,
> i=3D0x7fffafb7d1ac, rv=3D0xffffffffffffffff, cache=3D0x0) at rvalue.c:897
> #20 0x000000000049b68c in rval_expr_eval_int (h=3D0x7fffafb7dab0,
> msg=3D0x8d36c8, res=3D0x7fffafb7d1ac, rve=3D0x8c8a08) at rvalue.c:1837
> #21 0x000000000049b73d in rval_expr_eval_int (h=3D0x7fffafb7dab0,
> msg=3D0x8d36c8, res=3D0x7fffafb7d5b8, rve=3D0x8c90d8) at rvalue.c:1844
> #22 0x0000000000411d16 in do_action (h=3D0x7fffafb7dab0, a=3D0x8c9928,
> msg=3D0x8d36c8) at action.c:834
> #23 0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x8bda00,
> msg=3D0x8d36c8) at action.c:1293
> #24 0x0000000000413054 in do_action (h=3D0x7fffafb7dab0, a=3D0x8bd7f0,
> msg=3D0x8d36c8) at action.c:479
> #25 0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x89f8d8,
> msg=3D0x8d36c8) at action.c:1293
> #26 0x0000000000416084 in run_top_route (a=3D0x89f8d8, msg=3D0x8d36c8,
> c=3D<value optimized out>) at action.c:1341
> #27 0x000000000047ff4c in receive_msg (
>      buf=3D0x874a40 "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
> SIP/2.0\r\nRecord-Route:<sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia
>              : SIP/2.0/UDP
> 178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
> 81.27."..., len=3D<value optimized out>,
>      rcv_info=3D0x7fffafb7dd20) at receive.c:196
> #28 0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
> #29 0x0000000000455cdf in main_loop () at main.c:1447
> #30 0x0000000000456de2 in main (argc=3D<value optimized out>,
> argv=3D0x7fffafb7dfe8) at main.c:2251
>
> (gdb) bt full
> #0  0x000000329b6d0a19 in syscall () from /lib64/libc.so.6
> No symbol table info available.
> #1  0x00002acaeeb7ea32 in futex_get (i=3D<value optimized out>) at
> ../../mem/../futexlock.h:100
>          v =3D<value optimized out>
> #2  _lock (i=3D<value optimized out>) at lock.h:98
> No locals.
> #3  lock_hash (i=3D<value optimized out>) at h_table.c:98
> No locals.
> #4  0x00002acaeeba1b99 in t_lookup_request (p_msg=3D0x8d36c8,
> leave_new_locked=3D0, cancel=3D0x7fffafb7d5e0) at t_lookup.c:548
>          p_cell =3D<value optimized out>
>          isACK =3D 0
>          t_msg =3D<value optimized out>
>          branch =3D 0xfffffffffffffffc
>          match_status =3D<value optimized out>
>          hash_bucket =3D<value optimized out>
> #5  0x00002acaeeba3532 in t_check_msg (p_msg=3D0x8d36c8,
> param_branch=3D<value optimized out>) at t_lookup.c:1104
>          local_branch =3D 0
>          canceled =3D 9042808
>          ret =3D<value optimized out>
> #6  0x00002acaeebab7b1 in t_check_trans (msg=3D0x2acaf1f4b2e8,
> foo=3D<value optimized out>, bar=3D0x2<Address 0x2 out of bounds>) at
> tm.c:1881
>          t =3D<value optimized out>
> #7  0x000000000041345c in do_action (h=3D0x7fffafb7dab0, a=3D0x8a6f98,
> msg=3D0x8d36c8) at action.c:860
>          ret =3D -5
>          v =3D<value optimized out>
>          dst =3D {send_sock =3D 0x2acaedccc000, to =3D {s =3D {sa_family =3D
> 37544, sa_data =3D "\220\000\000\000\000\000`\253\220\000\000\000\000"},
> sin =3D {
>                sin_family =3D 37544, sin_port =3D 144, sin_addr =3D {s_addr=
 =3D
> 0}, sin_zero =3D "`\253\220\000\000\000\000"}, sin6 =3D {sin6_family =3D
> 37544,
>                sin6_port =3D 144, sin6_flowinfo =3D 0, sin6_addr =3D {in6_u=
 =3D
> {u6_addr8 =3D "`\253\220\000\000\000\000\000LK\207\000\000\000\000",
> u6_addr16 =3D {
>                      43872, 144, 0, 0, 19276, 135, 0, 0}, u6_addr32 =3D
> {9481056, 0, 8866636, 0}}}, sin6_scope_id =3D 9480096}}, id =3D 0, proto =3D
> -46 '\322',
>            send_flags =3D 74 'J'}
>          tmp =3D<value optimized out>
>          new_uri =3D<value optimized out>
>          end =3D<value optimized out>
>          crt =3D<value optimized out>
>          f =3D 0xfffffffffffffffc
>          len =3D<value optimized out>
>          user =3D<value optimized out>
>          uri =3D {user =3D {s =3D 0x7fffafb7da60 "\310\066\215", len =3D 0},
> passwd =3D {s =3D 0x7fffafb7da40 "\001", len =3D 5767096}, host =3D {
>              s =3D 0x1be1460 "8*\225\233\062", len =3D -1684723232}, port =
=3D
> {s =3D 0x100000000<Address 0x100000000 out of bounds>, len =3D 134},
> params =3D {
>              s =3D 0x0, len =3D 29234864}, sip_params =3D {s =3D 0x100000000
> <Address 0x100000000 out of bounds>, len =3D -1687727470}, headers =3D {
>              s =3D 0x36b7<Address 0x36b7 out of bounds>, len =3D 8208},
> port_no =3D 24939, proto =3D 24941, type =3D 1869180009, flags =3D 21,
> transport =3D {
> ---Type<return>  to continue, or q<return>  to quit---
>              s =3D 0x1be1460 "8*\225\233\062", len =3D 0}, ttl =3D {s =3D 0=
x0,
> len =3D 21}, user_param =3D {s =3D 0x1be1460 "8*\225\233\062", len =3D 21},
> maddr =3D {
>              s =3D 0x0, len =3D -1346905536}, method =3D {s =3D 0x57ffb8 "%=
s:
> %sERROR: forward_reply: no 2nd via found in reply\n", len =3D
> -1687738517}, lr =3D {
>              s =3D 0x18<Address 0x18 out of bounds>, len =3D 21}, r2 =3D {s
> =3D 0x15<Address 0x15 out of bounds>, len =3D 29234272}, transport_val =3D
> {
>              s =3D 0x15<Address 0x15 out of bounds>, len =3D 21}, ttl_val
> =3D {s =3D 0x1be1460 "8*\225\233\062", len =3D -1687354197}, user_param_val
> =3D {
>              s =3D 0x2a00000033<Address 0x2a00000033 out of bounds>, len
> =3D 5458017}, maddr_val =3D {s =3D 0x6f00000003<Address 0x6f00000003 out of
> bounds>,
>              len =3D 2}, method_val =3D {s =3D 0x1<Address 0x1 out of
> bounds>, len =3D 7200}, lr_val =3D {s =3D 0x1bacb40 "CEST", len =3D 5267113=
},
> r2_val =3D {
>              s =3D 0x90a606 "", len =3D -1346905500}}
>          next_hop =3D {user =3D {s =3D 0x10<Address 0x10 out of bounds>, len
> =3D -1687029130}, passwd =3D {s =3D 0x1ff4<Address 0x1ff4 out of bounds>,
>              len =3D -1687023672}, host =3D {s =3D 0x8<Address 0x8 out of
> bounds>, len =3D 29234892}, port =3D {
>              s =3D 0x1be16cc "16:42:57 /usr/local/sbin/kamailio[14007]:
> WARNING:<script>: ua=3DCEC-PBX", len =3D -1684723232}, params =3D {s =3D
> 0x329b952a00 "",
>              len =3D -1684723200}, sip_params =3D {s =3D 0x2acaf3e05470 "",
> len =3D -1684723144}, headers =3D {s =3D 0x329b94fae0 "@=C3=ADen =3D -13469=
05792},
>            port_no =3D 16, proto =3D 0, type =3D 48, flags =3D 2948061504,
> transport =3D {s =3D 0x7fffafb7d880 "", len =3D 29235008}, ttl =3D {
>              s =3D 0x1e8c0<Address 0x1e8c0 out of bounds>, len =3D
> 29243072}, user_param =3D {s =3D 0x329b722698 "double free or corruption
> (out)",
>              len =3D 117056}, maddr =3D {s =3D 0x329b9529e0 "", len =3D
> -1687739828}, method =3D {s =3D 0x329b9535a0 "", len =3D 5767096}, lr =3D {
>              s =3D 0x1<Address 0x1 out of bounds>, len =3D -1687631697},
> r2 =3D {s =3D 0x7fffafb7dad8 "\330",<incomplete sequence \370\211>, len =3D
> 8231792},
>            transport_val =3D {s =3D 0x7fffafb7dd8c "\020", len =3D
> -289787385}, ttl_val =3D {s =3D 0x8d36c8 "\033x\r", len =3D 412},
> user_param_val =3D {
>              s =3D 0x313500000000<Address 0x313500000000 out of bounds>,
> len =3D 29243072}, maddr_val =3D {s =3D 0x2010<Address 0x2010 out of
> bounds>,
>              len =3D 134}, method_val =3D {s =3D 0x2010<Address 0x2010 out
> of bounds>, len =3D 29234864}, lr_val =3D {s =3D 0x90<Address 0x90 out of
> bounds>,
>              len =3D -1687731159}, r2_val =3D {s =3D 0x3000000010<Address
> 0x3000000010 out of bounds>, len =3D -1684723232}}
>          u =3D<value optimized out>
>          port =3D<value optimized out>
>          i =3D<value optimized out>
>          flags =3D<value optimized out>
>          sct =3D<value optimized out>
>          sjt =3D<value optimized out>
>          rve =3D<value optimized out>
>          mct =3D<value optimized out>
>          rv =3D<value optimized out>
>          rv1 =3D<value optimized out>
>          c1 =3D {cache_type =3D 8866741, val_type =3D RV_NONE, c =3D {avp_v=
al =3D
> {n =3D 133, s =3D {s =3D 0x85<Address 0x85 out of bounds>, len =3D 29234880=
},
>                re =3D 0x85}, pval =3D {rs =3D {s =3D 0x85<Address 0x85 out =
of
> bounds>, len =3D 29234880}, ri =3D 1303224171, flags =3D 0}}}
> #8  0x0000000000415d2b in run_actions (h=3D0x7fffafb7dab0, a=3D0x89f8d8,
> msg=3D0x8d36c8) at action.c:1293
>          t =3D 0x8a6f98
>          ret =3D 1
>          mod =3D<value optimized out>
> #9  0x0000000000416084 in run_top_route (a=3D0x89f8d8, msg=3D0x8d36c8,
> c=3D<value optimized out>) at action.c:1341
>          ctx =3D {rec_lev =3D 1, run_flags =3D 0, last_retcode =3D -1, jmp_=
env
> =3D {{__jmpbuf =3D {9254600, -7918255915006962580, 0, 9042136, 9482320,
>                  140736141450636, -7918229962976915956,
> -7918255915011241414}, __mask_was_saved =3D 0, __saved_mask =3D {__val =3D
> {9254600, 2, 5317260, 928,
>                    8866420, 0, 8866368, 9254608, 1, 140736141450528,
> 9254600, 8866429, 5321012, 9479616, 47051076858778, 9295328}}}}}
>          p =3D 0x7fffafb7dab0
> ---Type<return>  to continue, or q<return>  to quit---
>          ret =3D<value optimized out>
>          sfbk =3D 0
> #10 0x000000000047ff4c in receive_msg (
>      buf=3D0x874a40 "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
> SIP/2.0\r\nRecord-Route:
> <sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
> 178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
> 81.27."..., len=3D<value optimized out>, rcv_info=3D0x7fffafb7dd20) at
> receive.c:196
>          msg =3D 0x0
>          ctx =3D {rec_lev =3D 9034719, run_flags =3D 0, last_retcode =3D
> 9295720, jmp_env =3D {{__jmpbuf =3D {47051129731136, 8, 47051127553476, 2,
> 217355168616,
>                  47051061455248, 47051127553501, 4294967295},
> __mask_was_saved =3D -236996544, __saved_mask =3D {__val =3D {8230504,
> 9007856, 16, 9482320,
>                    140736141450636, 0, 217350982997, 1, 0, 8963168,
> 47051127486569, 217355168616, 2, 150476210, 217351006434, 0}}}}}
>          ret =3D<value optimized out>
>          inb =3D {
>            s =3D 0x874a40 "INVITE
> sip:20322595(a)178.21.248.8:5060;transport=3Dudp SIP/2.0\r\nRecord-Route:
> <sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
> 178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
> 81.27."..., len =3D 928}
> #11 0x00000000005067ab in udp_rcv_loop () at udp_server.c:520
>          len =3D 928
>          tmp =3D<value optimized out>
>          from =3D<value optimized out>
>          fromlen =3D 16
>          ri =3D {src_ip =3D {af =3D 2, len =3D 4, u =3D {addrl =3D {3518028=
02,
> 420}, addr32 =3D {351802802, 0, 420, 0}, addr16 =3D {5554, 5368, 0, 0,
> 420, 0, 0, 0},
>                addr =3D
> "\262\025\370\024\000\000\000\000\244\001\000\000\000\000\000"}},
> dst_ip =3D {af =3D 2, len =3D 4, u =3D {addrl =3D {150476210, 0},
>                addr32 =3D {150476210, 0, 0, 0}, addr16 =3D {5554, 2296, 0,
> 0, 0, 0, 0, 0}, addr =3D "\262\025\370\b", '\000'<repeats 11 times>}},
>            src_port =3D 5060, dst_port =3D 5060, proto_reserved1 =3D 0,
> proto_reserved2 =3D 0, src_su =3D {s =3D {sa_family =3D 2,
>                sa_data =3D
> "\023=C3=84=C2=B2\025\370\024\000\000\000\000\000\000\000"}, sin =3D {sin_f=
amily =3D
> 2, sin_port =3D 50195, sin_addr =3D {s_addr =3D 351802802},
>                sin_zero =3D "\000\000\000\000\000\000\000"}, sin6 =3D
> {sin6_family =3D 2, sin6_port =3D 50195, sin6_flowinfo =3D 351802802,
> sin6_addr =3D {in6_u =3D {
>                    u6_addr8 =3D '\000'<repeats 15 times>, u6_addr16 =3D
> {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 =3D {0, 0, 0, 0}}}, sin6_scope_id =3D
> 0}},
>            bind_address =3D 0x8972f0, proto =3D 1 '\001'}
>          buf =3D "INVITE sip:20322595(a)178.21.248.8:5060;transport=3Dudp
> SIP/2.0\r\nRecord-Route:
> <sip:178.21.248.20;lr;ftag=3Das3d313976>\r\nVia: SIP/2.0/UDP
> 178.21.248.20;branch=3Dz9hG4bKdd3a.3d0516e1.0\r\nVia: SIP/2.0/UDP
> 81.27."...
> #12 0x0000000000455cdf in main_loop () at main.c:1447
>          i =3D 2
>          pid =3D<value optimized out>
>          si =3D 0x8972f0
>          si_desc =3D "udp receiver child=3D2
> sock=3D178.21.248.8:5060\000\000\000\000\000\004", '\000'<repeats 11
> times>, "\001", '\000'<repeats 11 times>,
> "\b\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\004",
> '\000'<repeats 23 times>"\350,
> \337\267\257\377\177\000\000\252\267H\000\000\000\000"
> #13 0x0000000000456de2 in main (argc=3D<value optimized out>,
> argv=3D0x7fffafb7dfe8) at main.c:2251
>          cfg_stream =3D 0x1b95010
>          c =3D<value optimized out>
>          r =3D<value optimized out>
>          tmp =3D 0x7fffafb7ef76 ""
> ---Type<return>  to continue, or q<return>  to quit---
>          tmp_len =3D 16777216
>          port =3D 0
>          proto =3D 0
>          ret =3D<value optimized out>
>          seed =3D 524492455
>          rfd =3D<value optimized out>
>          debug_save =3D 0
>          debug_flag =3D 0
>          dont_fork_cnt =3D 0
>          n_lst =3D 0x0
>
>
>
>
>
> On Thu, Apr 7, 2011 at 10:52 PM, Daniel-Constantin Mierla
> <miconda(a)gmail.com>  wrote:
>> Hello,
>>
>> do you get high CPU usage by kamailio?
>>
>> What you can do is to attach with gdb to kamailio processes and see what
>> they are doing:
>>
>> gdb   /path/to/kamailio   pid_of_a_kamailio_process
>> bt
>>
>> You should attach to the sip worker processes - you can find the type of
>> processes with 'kamctl ps'.
>>
>> Cheers,
>> Daniel
>>
>> On 4/7/11 9:02 PM, Morten Isaksen wrote:
>>> Hi!
>>>
>>> Kamailio 3.0.3.
>>>
>>> I have a strange problem with one of our Kamailio servers. This one is
>>> used for routing (with carrierroute) and to send presence information
>>> (with pua module)
>>>
>>> Once every 10 day or so I get this error and then Kamailio stops
>>> responding to any SIP packets.
>>>
>>> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: WARNING:
>>> <script>: Failure route - M=3DINVITE RURI=3Dsip:8615XXXX(a)178.XX.XX.XX
>>> F=3Dsip:861XXXXX(a)188.120.93.114:1025 T=3Dsip:86155XXXX(a)sip1.uni-tel.dk
>>> IP=3D178.XX.XX.XX ID=3D6de881ec07f9c6494ee589cf208da358(a)10.11.87.206
>>> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>>> carrierroute [cr_func.c:95]: cannot find AVP 'carrier'
>>> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>>> carrierroute [cr_func.c:805]: invalid carrier id -1
>>> Apr  6 08:05:48 sip-core-1 /usr/local/sbin/kamailio[9186]: ERROR:
>>> <script>: cr_next_domain failed
>>>
>>>
>>> Shared memory size is 128M and over halv is free just before the
>>> error. The server is in production and does handle debug>1 well, so I
>>> do not have much information in the log files. Private memory is the
>>> default size.
>>>
>>> Any ideas what it could be, or how to investegate further?
>>>
>>> I think my next steps would be to increase the private memory og to
>>> increase children=3D4 to children=3D8
>>>
>> --
>> Daniel-Constantin Mierla
>> http://www.asipto.com
>>
>>
>
>

--=20
Daniel-Constantin Mierla
http://www.asipto.com



--===============0801942135==--