[SR-Users] too many recursive routing table lookups
Camila Troncoso
ctroncoso at redvoiss.net
Tue Mar 11 15:05:23 CET 2014
Daniel,
I have maximum 40 gateways to try. That's why this error is strange.
Indeed the error is presenting in calls having no more than 12 retries (
because of channel exceed or failure messages) before they could be relay
(see the attached file). So I don't understang why is the error is
appearing if I don't reach 100 recursive executions.
Regards,
Camila
*De:* Daniel-Constantin Mierla [mailto:miconda at gmail.com]
*Enviado el:* martes, 11 de marzo de 2014 10:45
*Para:* Camila Troncoso
*CC:* Kamailio (SER) - Users Mailing List
*Asunto:* Re: [SR-Users] too many recursive routing table lookups
How many gateways you could possible try? More than 100?
Perhaps you can use a while loop instead of recursive subroute executions.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com
On 11 Mar 2014, at 14:01, Camila Troncoso <ctroncoso at redvoiss.net> wrote:
Daniel,
Attached will you find the graph.
I don't really understand much of it.
My route CHANNELS is where I check if the gateway is full, if that is the
case , a go back to RELAY route , load next_gw() and then look for the
channels capacity from the new gateway .
If I received a failure response , I load next_gw and check for channels.
Regards,
Camila
*De:* sr-users-bounces at lists.sip-router.org [mailto:
sr-users-bounces at lists.sip-router.org] *En nombre de *Daniel-Constantin
Mierla
*Enviado el:* lunes, 10 de marzo de 2014 17:11
*Para:* Kamailio (SER) - Users Mailing List
*Asunto:* Re: [SR-Users] too many recursive routing table lookups
This is when you have a recursive loop or too many chained subroutes
executed. Like:
request_route {
route(x);
}
route[x] {
route(y);
}
route[y] {
route(x);
}
if you have sources of kamailio, then you can check with:
./utils/route_graph/route_graph.py /path/to/kamailio.cfg
See if you can spot the loop.
Cheers,
Daniel
On 10/03/14 20:36, Camila Troncoso wrote:
I have another ERROR that is presenting:
"WARNING: too many recursive routing table lookups (101) giving up!"
I have a routing diagram that includes max number of channels per gateway.
The dialog module keeps track of the number of channels use by active calls
( set_dlg_profile, get_profile_size) and if a gateway has reached his
maximum , I search for next_gw().
This recursive mechanism can be made several times if many gateways of one
route are full.
But I analyzed the error and I have maximum 24 repeats of this process per
call, never reach 100 repeats as shown by the ERROR.
Can the maximum value of recursive lookups be the sum of lookups counting
all the early dialogs?
Regards,
Camila
*De:* Daniel-Constantin Mierla [mailto:miconda at gmail.com]
*Enviado el:* lunes, 10 de marzo de 2014 14:21
*Para:* Camila Troncoso; Kamailio (SER) - Users Mailing List
*Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory
issue
Indeed, "kamctl fifo get_statistics all" is the same as "kamctl stats" in
newer versions.
You run with default 32MB which is rather low, imo. You have less than 1MB
free at this moment.
Start kamailio with larger memory, like -m 512. It is not going to be taken
from the system unless kamailio is really using it.
Cheers,
Daniel
On 10/03/14 17:01, Camila Troncoso wrote:
Daniel,
With the command " kamctl fifo get_statistics all" i get this:
shmem:fragments = 2565
shmem:free_size = 1028552
shmem:max_used_size = 33554432
shmem:real_used_size = 32525880
shmem:total_size = 33554432
shmem:used_size = 26417552
Regards,
Camila
*De:* Camila Troncoso [mailto:ctroncoso at redvoiss.net]
*Enviado el:* lunes, 10 de marzo de 2014 12:43
*Para:* 'miconda at gmail.com'; 'Kamailio (SER) - Users Mailing List'
*Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory
issue
Daniel,
I haven't use the -m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
*De:* sr-users-bounces at lists.sip-router.org [
mailto:sr-users-bounces at lists.sip-router.org<sr-users-bounces at lists.sip-router.org>]
*En nombre de *Daniel-Constantin Mierla
*Enviado el:* lunes, 10 de marzo de 2014 11:52
*Para:* Kamailio (SER) - Users Mailing List
*Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory
issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have
you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers,
Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi,
Last Friday I presented a increase of load in kamailio by a client. The
load was almost twice as usual. By that time I encountered with many issues
in kamailio. The calls were not relay to gateways, and a lot of messages
about memory were appearing in log file. Kamailio didn't crash. When I
remove the extra traffic all went back to normal. Is there a issue about
memory leak or about to few memory resources to manage this amount of
traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req:
out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build
ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build
ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req:
out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request:/p>
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req:
out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate
memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry,
server error occurred (1/SL)
Regards,
Camila
*<image001.gif>*
*Camila Troncoso Solar **|* Ingeniero de Desarollo
+56 2 22408535 *·* +56 9 97327220 | ctroncoso at redvoiss.net
Badajoz 130, piso 16, Las Condes | Santiago - CHILE | *www.redvoiss.com
<http://www.redvoiss.com>*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users at lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference - April 2-4, 2014, Berlin, Germany
http://www.kamailioworld.com
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference - April 2-4, 2014, Berlin, Germany
http://www.kamailioworld.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users at lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference - April 2-4, 2014, Berlin, Germany
http://www.kamailioworld.com
D"http://www.kamailioworld.com">http://www.kamailioworld.com
<graph.txt>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140311/ab894844/attachment-0001.html>
-------------- next part --------------
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7929]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [Ruta LCR] :: Cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.131:5060;transport=udp
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7929]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [RELAY SECTION] :: INVITE t_relay a sip:02670056226850700 at 64.76.155.131:5060;transport=udp
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [FAIL SECTION] :: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.12:5060;transport=udp en failure_route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.12 channel limit exceeded [20/20]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.139:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.139 channel limit exceeded [19/19]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.11:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.11 channel limit exceeded [20/20]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.130:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.130 channel limit exceeded [23/23]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.25:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.25 channel limit exceeded [9/9]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.135:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.135 channel limit exceeded [4/4]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.5:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.5 channel limit exceeded [7/7]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.141:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.141 channel limit exceeded [10/10]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.138:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.138 channel limit exceeded [29/29]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.8:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.8 channel limit exceeded [5/5]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: INVITE cambiando sip:02670056226850700 at siplcr.redvoiss.net:5060 por sip:02670056226850700 at 64.76.155.14:5060;transport=udp en channels route
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: INFO: <script>: [61a5bc7ea92511e3a6d00019b9eacc40 at 66.165.170.214-b2b_1] [CHANNELS]:: Call crtl: gateway 64.76.155.14 channel limit exceeded [19/19]
Mar 11 10:59:36 vlcr /usr/local/sbin/kamailio[7920]: ERROR: <core> [action.c:1586]: WARNING: too many recursive routing table lookups (101) giving up!
More information about the sr-users
mailing list