Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn't crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
*Camila Troncoso Solar **|* Ingeniero de Desarollo
+56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net
Badajoz 130, piso 16, Las Condes | Santiago - CHILE | *www.redvoiss.com http://www.redvoiss.com*
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn’t crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
Camila Troncoso Solar **|*Ingeniero de Desarollo**
+56 2 22408535 *·*+56 9 97327220 | ctroncoso@redvoiss.net mailto:ctroncoso@redvoiss.net__
Badajoz 130, piso 16, Las Condes | Santiago - CHILE| /_www.redvoiss.com http://www.redvoiss.com_/**
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Daniel,
I haven't use the -m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [mailto: sr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn't crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
*Camila Troncoso Solar **|* Ingeniero de Desarollo
+56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net
Badajoz 130, piso 16, Las Condes | Santiago - CHILE | *www.redvoiss.com http://www.redvoiss.com*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Daniel,
With the command " kamctl fifo get_statistics all" i get this:
shmem:fragments = 2565
shmem:free_size = 1028552
shmem:max_used_size = 33554432
shmem:real_used_size = 32525880
shmem:total_size = 33554432
shmem:used_size = 26417552
Regards,
Camila
*De:* Camila Troncoso [mailto:ctroncoso@redvoiss.net] *Enviado el:* lunes, 10 de marzo de 2014 12:43 *Para:* 'miconda@gmail.com'; 'Kamailio (SER) - Users Mailing List' *Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Daniel,
I haven't use the -m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [ mailto:sr-users-bounces@lists.sip-router.orgsr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn't crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
*Camila Troncoso Solar **|* Ingeniero de Desarollo
+56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net
Badajoz 130, piso 16, Las Condes | Santiago - CHILE | *www.redvoiss.com http://www.redvoiss.com*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Indeed, “kamctl fifo get_statistics all” is the same as "kamctl stats" in newer versions.
You run with default 32MB which is rather low, imo. You have less than 1MB free at this moment.
Start kamailio with larger memory, like -m 512. It is not going to be taken from the system unless kamailio is really using it.
Cheers, Daniel
On 10/03/14 17:01, Camila Troncoso wrote:
Daniel,
With the command “kamctl fifo get_statistics all” i get this:
shmem:fragments = 2565
shmem:free_size = 1028552
shmem:max_used_size = 33554432
shmem:real_used_size = 32525880
shmem:total_size = 33554432
shmem:used_size = 26417552
Regards,
Camila
*De:*Camila Troncoso [mailto:ctroncoso@redvoiss.net mailto:ctroncoso@redvoiss.net] *Enviado el:* lunes, 10 de marzo de 2014 12:43 *Para:* 'miconda@gmail.com mailto:miconda@gmail.com'; 'Kamailio (SER) - Users Mailing List' *Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Daniel,
I haven’t use the –m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
*De:*sr-users-bounces@lists.sip-router.org mailto:sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi, Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn’t crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic? ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL) ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL) ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL) Regards, Camila ** * Camila Troncoso Solar **|* Ingeniero de Desarollo +56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net <mailto:ctroncoso@redvoiss.net>__ Badajoz 130, piso 16, Las Condes | Santiago - CHILE| /_www.redvoiss.com <http://www.redvoiss.com>_/ _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla -http://www.asipto.com http://twitter.com/#!/miconda http://twitter.com/#%21/miconda -http://www.linkedin.com/in/miconda Kamailio World Conference - April 2-4, 2014, Berlin, Germany http://www.kamailioworld.com
I have another ERROR that is presenting:
"WARNING: too many recursive routing table lookups (101) giving up!"
I have a routing diagram that includes max number of channels per gateway. The dialog module keeps track of the number of channels use by active calls ( set_dlg_profile, get_profile_size) and if a gateway has reached his maximum , I search for next_gw().
This recursive mechanism can be made several times if many gateways of one route are full.
But I analyzed the error and I have maximum 24 repeats of this process per call, never reach 100 repeats as shown by the ERROR.
Can the maximum value of recursive lookups be the sum of lookups counting all the early dialogs?
Regards,
Camila
*De:* Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Enviado el:* lunes, 10 de marzo de 2014 14:21 *Para:* Camila Troncoso; Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Indeed, "kamctl fifo get_statistics all" is the same as "kamctl stats" in newer versions.
You run with default 32MB which is rather low, imo. You have less than 1MB free at this moment.
Start kamailio with larger memory, like -m 512. It is not going to be taken from the system unless kamailio is really using it.
Cheers, Daniel
On 10/03/14 17:01, Camila Troncoso wrote:
Daniel,
With the command " kamctl fifo get_statistics all" i get this:
shmem:fragments = 2565
shmem:free_size = 1028552
shmem:max_used_size = 33554432
shmem:real_used_size = 32525880
shmem:total_size = 33554432
shmem:used_size = 26417552
Regards,
Camila
*De:* Camila Troncoso [mailto:ctroncoso@redvoiss.net] *Enviado el:* lunes, 10 de marzo de 2014 12:43 *Para:* 'miconda@gmail.com'; 'Kamailio (SER) - Users Mailing List' *Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Daniel,
I haven't use the -m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [ mailto:sr-users-bounces@lists.sip-router.orgsr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn't crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
*Camila Troncoso Solar **|* Ingeniero de Desarollo
+56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net
Badajoz 130, piso 16, Las Condes | Santiago - CHILE | *www.redvoiss.com http://www.redvoiss.com*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
This is when you have a recursive loop or too many chained subroutes executed. Like:
request_route { route(x); }
route[x] { route(y); }
route[y] { route(x); }
if you have sources of kamailio, then you can check with:
./utils/route_graph/route_graph.py /path/to/kamailio.cfg
See if you can spot the loop.
Cheers, Daniel
On 10/03/14 20:36, Camila Troncoso wrote:
I have another ERROR that is presenting:
“WARNING: too many recursive routing table lookups (101) giving up!”
I have a routing diagram that includes max number of channels per gateway. The dialog module keeps track of the number of channels use by active calls ( set_dlg_profile, get_profile_size) and if a gateway has reached his maximum , I search for next_gw().
This recursive mechanism can be made several times if many gateways of one route are full.
But I analyzed the error and I have maximum 24 repeats of this process per call, never reach 100 repeats as shown by the ERROR.
Can the maximum value of recursive lookups be the sum of lookups counting all the early dialogs?
Regards,
Camila
*De:*Daniel-Constantin Mierla [mailto:miconda@gmail.com mailto:miconda@gmail.com] *Enviado el:* lunes, 10 de marzo de 2014 14:21 *Para:* Camila Troncoso; Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Indeed, “kamctl fifo get_statistics all” is the same as "kamctl stats" in newer versions.
You run with default 32MB which is rather low, imo. You have less than 1MB free at this moment.
Start kamailio with larger memory, like -m 512. It is not going to be taken from the system unless kamailio is really using it.
Cheers, Daniel
On 10/03/14 17:01, Camila Troncoso wrote:
Daniel, With the command “kamctl fifo get_statistics all” i get this: shmem:fragments = 2565 shmem:free_size = 1028552 shmem:max_used_size = 33554432 shmem:real_used_size = 32525880 shmem:total_size = 33554432 shmem:used_size = 26417552 Regards, Camila *De:*Camila Troncoso [mailto:ctroncoso@redvoiss.net <mailto:ctroncoso@redvoiss.net>] *Enviado el:* lunes, 10 de marzo de 2014 12:43 *Para:* 'miconda@gmail.com <mailto:miconda@gmail.com>'; 'Kamailio (SER) - Users Mailing List' *Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue Daniel, I haven’t use the –m command. I´m running kamailio 3.2.4 version. The command kamctl stats is not available. Regards, Camila *De:*sr-users-bounces@lists.sip-router.org <mailto:sr-users-bounces@lists.sip-router.org> [mailto:sr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue Hello, how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter? You can see it also via: kamctl stats Look for shmem fields. Cheers, Daniel On 10/03/14 14:37, Camila Troncoso wrote: Hi, Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn’t crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic? ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL) ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL) ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL) Regards, Camila ** * Camila Troncoso Solar **|* Ingeniero de Desarollo +56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net <mailto:ctroncoso@redvoiss.net>__ Badajoz 130, piso 16, Las Condes | Santiago - CHILE| /_www.redvoiss.com <http://www.redvoiss.com>_/ _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -http://www.asipto.com http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Kamailio World Conference - April 2-4, 2014, Berlin, Germany http://www.kamailioworld.com-- Daniel-Constantin Mierla -http://www.asipto.com http://twitter.com/#!/miconda http://twitter.com/#%21/miconda -http://www.linkedin.com/in/miconda Kamailio World Conference - April 2-4, 2014, Berlin, Germany http://www.kamailioworld.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Daniel,
The output is :
Main
|
- REQINIT
|
- TRUSTEDIP
|
- WITHINDLG
| |
| - RELAY
| | |
| | - CHANNELS
| | | |
| | | - RELAY
| | | | |
| | | | - CHANNELS
| | | | | |
| | | | | - RELAY
| | | | | | |
| | | | | | - CHANNELS
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | |
| | | | | - RELAY
| | | | | | |
| | | | | | - CHANNELS
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | |
| | | - RELAY
| | | | |
| | | | - CHANNELS
| | | | | |
| | | | | - RELAY
| | | | | | |
| | | | | | - CHANNELS
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | |
| | | | | - RELAY
| | | | | | |
| | | | | | - CHANNELS
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
|
- CHEQUEO
| |
| - ALERTA
| | |
| | - RELAY
| | | |
| | | - CHANNELS
| | | | |
| | | | - RELAY
| | | | | |
| | | | | - CHANNELS
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | |
| | | | - RELAY
| | | | | |
| | | | | - CHANNELS
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
|
- REGISTRAR
|
- LCR
| |
| - RELAY
| | |
| | - CHANNELS
| | | |
| | | - RELAY
| | | | |
| | | | - CHANNELS
| | | | | |
| | | | | - RELAY
| | | | | | |
| | | | | | - CHANNELS
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | |
| | | | | - RELAY
| | | | | | |
| | | | | | - CHANNELS
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | |
| | | - RELAY
| | | | |
| | | | - CHANNELS
| | | | | |
| | | | | - RELAY
| | | | | | |
| | | | | | - CHANNELS
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | |
| | | | | - RELAY
| | | | | | |
| | | | | | - CHANNELS
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | |
| | | | | | | - RELAY
| | | | | | | | |
| | | | | | | | - CHANNELS
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
| | | | | | | | | |
| | | | | | | | | - RELAY
warning: max_depth reached
Failure routes
--------------
FAIL_ONE
|
- RELAY
| |
| - CHANNELS
| | |
| | - RELAY
| | | |
| | | - CHANNELS
| | | | |
| | | | - RELAY
| | | | | |
| | | | | - CHANNELS
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | |
| | | | - RELAY
| | | | | |
| | | | | - CHANNELS
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | |
| | - RELAY
| | | |
| | | - CHANNELS
| | | | |
| | | | - RELAY
| | | | | |
| | | | | - CHANNELS
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | |
| | | | - RELAY
| | | | | |
| | | | | - CHANNELS
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | |
| | | | | | - RELAY
| | | | | | | |
| | | | | | | - CHANNELS
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
| | | | | | | | |
| | | | | | | | - RELAY
| | | | | | | | | |
| | | | | | | | | - CHANNELS
warning: max_depth reached
warning: max_depth reached
Event routes
--------------
tm:local-request
I don't really understand this graph.
My route CHANNELS is where I check if the gateway is full, if that is the case , a go back to RELAY route , load next_gw() and then look for the channels capacity from the new gateway .
If I received a failure response , I load next_gw and check for channels.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [mailto: sr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 17:11 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] too many recursive routing table lookups
This is when you have a recursive loop or too many chained subroutes executed. Like:
request_route { route(x); }
route[x] { route(y); }
route[y] { route(x); }
if you have sources of kamailio, then you can check with:
./utils/route_graph/route_graph.py /path/to/kamailio.cfg
See if you can spot the loop.
Cheers, Daniel
On 10/03/14 20:36, Camila Troncoso wrote:
I have another ERROR that is presenting:
"WARNING: too many recursive routing table lookups (101) giving up!"
I have a routing diagram that includes max number of channels per gateway. The dialog module keeps track of the number of channels use by active calls ( set_dlg_profile, get_profile_size) and if a gateway has reached his maximum , I search for next_gw().
This recursive mechanism can be made several times if many gateways of one route are full.
But I analyzed the error and I have maximum 24 repeats of this process per call, never reach 100 repeats as shown by the ERROR.
Can the maximum value of recursive lookups be the sum of lookups counting all the early dialogs?
Regards,
Camila
*De:* Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Enviado el:* lunes, 10 de marzo de 2014 14:21 *Para:* Camila Troncoso; Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Indeed, "kamctl fifo get_statistics all" is the same as "kamctl stats" in newer versions.
You run with default 32MB which is rather low, imo. You have less than 1MB free at this moment.
Start kamailio with larger memory, like -m 512. It is not going to be taken from the system unless kamailio is really using it.
Cheers, Daniel
On 10/03/14 17:01, Camila Troncoso wrote:
Daniel,
With the command " kamctl fifo get_statistics all" i get this:
shmem:fragments = 2565
shmem:free_size = 1028552
shmem:max_used_size = 33554432
shmem:real_used_size = 32525880
shmem:total_size = 33554432
shmem:used_size = 26417552
Regards,
Camila
*De:* Camila Troncoso [mailto:ctroncoso@redvoiss.net] *Enviado el:* lunes, 10 de marzo de 2014 12:43 *Para:* 'miconda@gmail.com'; 'Kamailio (SER) - Users Mailing List' *Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Daniel,
I haven't use the -m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [ mailto:sr-users-bounces@lists.sip-router.orgsr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn't crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
*Camila Troncoso Solar **|* Ingeniero de Desarollo
+56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net
Badajoz 130, piso 16, Las Condes | Santiago - CHILE | *www.redvoiss.com http://www.redvoiss.com*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Daniel,
Attached will you find the graph.
I don't really understand much of it.
My route CHANNELS is where I check if the gateway is full, if that is the case , a go back to RELAY route , load next_gw() and then look for the channels capacity from the new gateway .
If I received a failure response , I load next_gw and check for channels.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [mailto: sr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 17:11 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] too many recursive routing table lookups
This is when you have a recursive loop or too many chained subroutes executed. Like:
request_route { route(x); }
route[x] { route(y); }
route[y] { route(x); }
if you have sources of kamailio, then you can check with:
./utils/route_graph/route_graph.py /path/to/kamailio.cfg
See if you can spot the loop.
Cheers, Daniel
On 10/03/14 20:36, Camila Troncoso wrote:
I have another ERROR that is presenting:
"WARNING: too many recursive routing table lookups (101) giving up!"
I have a routing diagram that includes max number of channels per gateway. The dialog module keeps track of the number of channels use by active calls ( set_dlg_profile, get_profile_size) and if a gateway has reached his maximum , I search for next_gw().
This recursive mechanism can be made several times if many gateways of one route are full.
But I analyzed the error and I have maximum 24 repeats of this process per call, never reach 100 repeats as shown by the ERROR.
Can the maximum value of recursive lookups be the sum of lookups counting all the early dialogs?
Regards,
Camila
*De:* Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Enviado el:* lunes, 10 de marzo de 2014 14:21 *Para:* Camila Troncoso; Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Indeed, "kamctl fifo get_statistics all" is the same as "kamctl stats" in newer versions.
You run with default 32MB which is rather low, imo. You have less than 1MB free at this moment.
Start kamailio with larger memory, like -m 512. It is not going to be taken from the system unless kamailio is really using it.
Cheers, Daniel
On 10/03/14 17:01, Camila Troncoso wrote:
Daniel,
With the command " kamctl fifo get_statistics all" i get this:
shmem:fragments = 2565
shmem:free_size = 1028552
shmem:max_used_size = 33554432
shmem:real_used_size = 32525880
shmem:total_size = 33554432
shmem:used_size = 26417552
Regards,
Camila
*De:* Camila Troncoso [mailto:ctroncoso@redvoiss.net] *Enviado el:* lunes, 10 de marzo de 2014 12:43 *Para:* 'miconda@gmail.com'; 'Kamailio (SER) - Users Mailing List' *Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Daniel,
I haven't use the -m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [ mailto:sr-users-bounces@lists.sip-router.orgsr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn't crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
*Camila Troncoso Solar **|* Ingeniero de Desarollo
+56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net
Badajoz 130, piso 16, Las Condes | Santiago - CHILE | *www.redvoiss.com http://www.redvoiss.com*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
How many gateways you could possible try? More than 100?
Perhaps you can use a while loop instead of recursive subroute executions.
Cheers, Daniel
-- Daniel-Constantin Mierla http://www.asipto.com
On 11 Mar 2014, at 14:01, Camila Troncoso ctroncoso@redvoiss.net wrote:
Daniel,
Attached will you find the graph.
I don’t really understand much of it.
My route CHANNELS is where I check if the gateway is full, if that is the case , a go back to RELAY route , load next_gw() and then look for the channels capacity from the new gateway . If I received a failure response , I load next_gw and check for channels.
Regards,
Camila
De: sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] En nombre de Daniel-Constantin Mierla Enviado el: lunes, 10 de marzo de 2014 17:11 Para: Kamailio (SER) - Users Mailing List Asunto: Re: [SR-Users] too many recursive routing table lookups
This is when you have a recursive loop or too many chained subroutes executed. Like:
request_route { route(x); }
route[x] { route(y); }
route[y] { route(x); }
if you have sources of kamailio, then you can check with:
./utils/route_graph/route_graph.py /path/to/kamailio.cfg
See if you can spot the loop.
Cheers, Daniel
On 10/03/14 20:36, Camila Troncoso wrote: I have another ERROR that is presenting:
“WARNING: too many recursive routing table lookups (101) giving up!”
I have a routing diagram that includes max number of channels per gateway. The dialog module keeps track of the number of channels use by active calls ( set_dlg_profile, get_profile_size) and if a gateway has reached his maximum , I search for next_gw(). This recursive mechanism can be made several times if many gateways of one route are full. But I analyzed the error and I have maximum 24 repeats of this process per call, never reach 100 repeats as shown by the ERROR.
Can the maximum value of recursive lookups be the sum of lookups counting all the early dialogs?
Regards,
Camila
De: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Enviado el: lunes, 10 de marzo de 2014 14:21 Para: Camila Troncoso; Kamailio (SER) - Users Mailing List Asunto: Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Indeed, “kamctl fifo get_statistics all” is the same as "kamctl stats" in newer versions.
You run with default 32MB which is rather low, imo. You have less than 1MB free at this moment.
Start kamailio with larger memory, like -m 512. It is not going to be taken from the system unless kamailio is really using it.
Cheers, Daniel
On 10/03/14 17:01, Camila Troncoso wrote: Daniel,
With the command “ kamctl fifo get_statistics all” i get this:
shmem:fragments = 2565 shmem:free_size = 1028552 shmem:max_used_size = 33554432 shmem:real_used_size = 32525880 shmem:total_size = 33554432 shmem:used_size = 26417552
Regards,
Camila
De: Camila Troncoso [mailto:ctroncoso@redvoiss.net] Enviado el: lunes, 10 de marzo de 2014 12:43 Para: 'miconda@gmail.com'; 'Kamailio (SER) - Users Mailing List' Asunto: RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Daniel,
I haven’t use the –m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
De: sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] En nombre de Daniel-Constantin Mierla Enviado el: lunes, 10 de marzo de 2014 11:52 Para: Kamailio (SER) - Users Mailing List Asunto: Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote: Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn’t crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request:/p>
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL) ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
<image001.gif>
Camila Troncoso Solar | Ingeniero de Desarollo +56 2 22408535 · +56 9 97327220 | ctroncoso@redvoiss.net Badajoz 130, piso 16, Las Condes | Santiago - CHILE | www.redvoiss.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference - April 2-4, 2014, Berlin, Germany http://www.kamailioworld.com
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference - April 2-4, 2014, Berlin, Germany http://www.kamailioworld.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference - April 2-4, 2014, Berlin, Germany http://www.kamailioworld.com D"http://www.kamailioworld.com%22%3Ehttp://www.kamailioworld.com <graph.txt>
Daniel,
I have maximum 40 gateways to try. That's why this error is strange. Indeed the error is presenting in calls having no more than 12 retries ( because of channel exceed or failure messages) before they could be relay (see the attached file). So I don't understang why is the error is appearing if I don't reach 100 recursive executions.
Regards,
Camila
*De:* Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Enviado el:* martes, 11 de marzo de 2014 10:45 *Para:* Camila Troncoso *CC:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] too many recursive routing table lookups
How many gateways you could possible try? More than 100?
Perhaps you can use a while loop instead of recursive subroute executions.
Cheers,
Daniel
-- Daniel-Constantin Mierla
On 11 Mar 2014, at 14:01, Camila Troncoso ctroncoso@redvoiss.net wrote:
Daniel,
Attached will you find the graph.
I don't really understand much of it.
My route CHANNELS is where I check if the gateway is full, if that is the case , a go back to RELAY route , load next_gw() and then look for the channels capacity from the new gateway .
If I received a failure response , I load next_gw and check for channels.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [mailto: sr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 17:11 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] too many recursive routing table lookups
This is when you have a recursive loop or too many chained subroutes executed. Like:
request_route { route(x); }
route[x] { route(y); }
route[y] { route(x); }
if you have sources of kamailio, then you can check with:
./utils/route_graph/route_graph.py /path/to/kamailio.cfg
See if you can spot the loop.
Cheers, Daniel
On 10/03/14 20:36, Camila Troncoso wrote:
I have another ERROR that is presenting:
"WARNING: too many recursive routing table lookups (101) giving up!"
I have a routing diagram that includes max number of channels per gateway. The dialog module keeps track of the number of channels use by active calls ( set_dlg_profile, get_profile_size) and if a gateway has reached his maximum , I search for next_gw().
This recursive mechanism can be made several times if many gateways of one route are full.
But I analyzed the error and I have maximum 24 repeats of this process per call, never reach 100 repeats as shown by the ERROR.
Can the maximum value of recursive lookups be the sum of lookups counting all the early dialogs?
Regards,
Camila
*De:* Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Enviado el:* lunes, 10 de marzo de 2014 14:21 *Para:* Camila Troncoso; Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Indeed, "kamctl fifo get_statistics all" is the same as "kamctl stats" in newer versions.
You run with default 32MB which is rather low, imo. You have less than 1MB free at this moment.
Start kamailio with larger memory, like -m 512. It is not going to be taken from the system unless kamailio is really using it.
Cheers, Daniel
On 10/03/14 17:01, Camila Troncoso wrote:
Daniel,
With the command " kamctl fifo get_statistics all" i get this:
shmem:fragments = 2565
shmem:free_size = 1028552
shmem:max_used_size = 33554432
shmem:real_used_size = 32525880
shmem:total_size = 33554432
shmem:used_size = 26417552
Regards,
Camila
*De:* Camila Troncoso [mailto:ctroncoso@redvoiss.net] *Enviado el:* lunes, 10 de marzo de 2014 12:43 *Para:* 'miconda@gmail.com'; 'Kamailio (SER) - Users Mailing List' *Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Daniel,
I haven't use the -m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [ mailto:sr-users-bounces@lists.sip-router.orgsr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn't crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request:/p>
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
*<image001.gif>*
*Camila Troncoso Solar **|* Ingeniero de Desarollo
+56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net
Badajoz 130, piso 16, Las Condes | Santiago - CHILE | *www.redvoiss.com http://www.redvoiss.com*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Daniel,
Is there a limit of times that one can invoke next_gw() function without relaying the call? Can these be the problem?
Regards,
Camila
*De:* Camila Troncoso [mailto:ctroncoso@redvoiss.net] *Enviado el:* martes, 11 de marzo de 2014 11:05 *Para:* Daniel-Constantin Mierla *CC:* Kamailio (SER) - Users Mailing List *Asunto:* RE: [SR-Users] too many recursive routing table lookups
Daniel,
I have maximum 40 gateways to try. That's why this error is strange. Indeed the error is presenting in calls having no more than 12 retries ( because of channel exceed or failure messages) before they could be relay (see the attached file). So I don't understang why is the error is appearing if I don't reach 100 recursive executions.
Regards,
Camila
*De:* Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Enviado el:* martes, 11 de marzo de 2014 10:45 *Para:* Camila Troncoso *CC:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] too many recursive routing table lookups
How many gateways you could possible try? More than 100?
Perhaps you can use a while loop instead of recursive subroute executions.
Cheers,
Daniel
-- Daniel-Constantin Mierla
On 11 Mar 2014, at 14:01, Camila Troncoso ctroncoso@redvoiss.net wrote:
Daniel,
Attached will you find the graph.
I don't really understand much of it.
My route CHANNELS is where I check if the gateway is full, if that is the case , a go back to RELAY route , load next_gw() and then look for the channels capacity from the new gateway .
If I received a failure response , I load next_gw and check for channels.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [mailto: sr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 17:11 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] too many recursive routing table lookups
This is when you have a recursive loop or too many chained subroutes executed. Like:
request_route { route(x); }
route[x] { route(y); }
route[y] { route(x); }
if you have sources of kamailio, then you can check with:
./utils/route_graph/route_graph.py /path/to/kamailio.cfg
See if you can spot the loop.
Cheers, Daniel
On 10/03/14 20:36, Camila Troncoso wrote:
I have another ERROR that is presenting:
"WARNING: too many recursive routing table lookups (101) giving up!"
I have a routing diagram that includes max number of channels per gateway. The dialog module keeps track of the number of channels use by active calls ( set_dlg_profile, get_profile_size) and if a gateway has reached his maximum , I search for next_gw().
This recursive mechanism can be made several times if many gateways of one route are full.
But I analyzed the error and I have maximum 24 repeats of this process per call, never reach 100 repeats as shown by the ERROR.
Can the maximum value of recursive lookups be the sum of lookups counting all the early dialogs?
Regards,
Camila
*De:* Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Enviado el:* lunes, 10 de marzo de 2014 14:21 *Para:* Camila Troncoso; Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Indeed, "kamctl fifo get_statistics all" is the same as "kamctl stats" in newer versions.
You run with default 32MB which is rather low, imo. You have less than 1MB free at this moment.
Start kamailio with larger memory, like -m 512. It is not going to be taken from the system unless kamailio is really using it.
Cheers, Daniel
On 10/03/14 17:01, Camila Troncoso wrote:
Daniel,
With the command " kamctl fifo get_statistics all" i get this:
shmem:fragments = 2565
shmem:free_size = 1028552
shmem:max_used_size = 33554432
shmem:real_used_size = 32525880
shmem:total_size = 33554432
shmem:used_size = 26417552
Regards,
Camila
*De:* Camila Troncoso [mailto:ctroncoso@redvoiss.net] *Enviado el:* lunes, 10 de marzo de 2014 12:43 *Para:* 'miconda@gmail.com'; 'Kamailio (SER) - Users Mailing List' *Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Daniel,
I haven't use the -m command.
I´m running kamailio 3.2.4 version.
The command kamctl stats is not available.
Regards,
Camila
*De:* sr-users-bounces@lists.sip-router.org [ mailto:sr-users-bounces@lists.sip-router.orgsr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue
Hello,
how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter?
You can see it also via:
kamctl stats
Look for shmem fields.
Cheers, Daniel
On 10/03/14 14:37, Camila Troncoso wrote:
Hi,
Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn't crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic?
ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer
ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory
ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request:/p>
ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory
ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem
ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches
ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL)
ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory
ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL)
Regards,
Camila
*<image001.gif>*
*Camila Troncoso Solar **|* Ingeniero de Desarollo
+56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net
Badajoz 130, piso 16, Las Condes | Santiago - CHILE | *www.redvoiss.com http://www.redvoiss.com*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 11/03/14 15:05, Camila Troncoso wrote:
Daniel,
I have maximum 40 gateways to try. That’s why this error is strange. Indeed the error is presenting in calls having no more than 12 retries ( because of channel exceed or failure messages) before they could be relay (see the attached file). So I don’t understang why is the error is appearing if I don’t reach 100 recursive executions.
You are looping using two routes, thus for each gateway is counting twice or more, depending on how exactly the conditions are there, plus the other routes executed from main route to those looping.
Cheers, Daniel
Regards,
Camila
*De:*Daniel-Constantin Mierla [mailto:miconda@gmail.com mailto:miconda@gmail.com] *Enviado el:* martes, 11 de marzo de 2014 10:45 *Para:* Camila Troncoso *CC:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] too many recursive routing table lookups
How many gateways you could possible try? More than 100?
Perhaps you can use a while loop instead of recursive subroute executions.
Cheers,
Daniel
-- Daniel-Constantin Mierla
On 11 Mar 2014, at 14:01, Camila Troncoso <ctroncoso@redvoiss.net mailto:ctroncoso@redvoiss.net> wrote:
Daniel, Attached will you find the graph. I don’t really understand much of it. My route CHANNELS is where I check if the gateway is full, if that is the case , a go back to RELAY route , load next_gw() and then look for the channels capacity from the new gateway . If I received a failure response , I load next_gw and check for channels. Regards, Camila *De:*sr-users-bounces@lists.sip-router.org <mailto:sr-users-bounces@lists.sip-router.org> [mailto:sr-users-bounces@lists.sip-router.org <mailto:sr-users-bounces@lists.sip-router.org>] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 17:11 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] too many recursive routing table lookups This is when you have a recursive loop or too many chained subroutes executed. Like: request_route { route(x); } route[x] { route(y); } route[y] { route(x); } if you have sources of kamailio, then you can check with: ./utils/route_graph/route_graph.py /path/to/kamailio.cfg See if you can spot the loop. Cheers, Daniel On 10/03/14 20:36, Camila Troncoso wrote: I have another ERROR that is presenting: “WARNING: too many recursive routing table lookups (101) giving up!” I have a routing diagram that includes max number of channels per gateway. The dialog module keeps track of the number of channels use by active calls ( set_dlg_profile, get_profile_size) and if a gateway has reached his maximum , I search for next_gw(). This recursive mechanism can be made several times if many gateways of one route are full. But I analyzed the error and I have maximum 24 repeats of this process per call, never reach 100 repeats as shown by the ERROR. Can the maximum value of recursive lookups be the sum of lookups counting all the early dialogs? Regards, Camila *De:*Daniel-Constantin Mierla [mailto:miconda@gmail.com <mailto:miconda@gmail.com>] *Enviado el:* lunes, 10 de marzo de 2014 14:21 *Para:* Camila Troncoso; Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue Indeed, “kamctl fifo get_statistics all” is the same as "kamctl stats" in newer versions. You run with default 32MB which is rather low, imo. You have less than 1MB free at this moment. Start kamailio with larger memory, like -m 512. It is not going to be taken from the system unless kamailio is really using it. Cheers, Daniel On 10/03/14 17:01, Camila Troncoso wrote: Daniel, With the command “kamctl fifo get_statistics all” i get this: shmem:fragments = 2565 shmem:free_size = 1028552 shmem:max_used_size = 33554432 shmem:real_used_size = 32525880 shmem:total_size = 33554432 shmem:used_size = 26417552 Regards, Camila *De:*Camila Troncoso [mailto:ctroncoso@redvoiss.net <mailto:ctroncoso@redvoiss.net>] *Enviado el:* lunes, 10 de marzo de 2014 12:43 *Para:* 'miconda@gmail.com <mailto:miconda@gmail.com>'; 'Kamailio (SER) - Users Mailing List' *Asunto:* RE: [SR-Users] Kamailio LCR unable to relay new calls, memory issue Daniel, I haven’t use the –m command. I´m running kamailio 3.2.4 version. The command kamctl stats is not available. Regards, Camila *De:*sr-users-bounces@lists.sip-router.org <mailto:sr-users-bounces@lists.sip-router.org> [mailto:sr-users-bounces@lists.sip-router.org] *En nombre de *Daniel-Constantin Mierla *Enviado el:* lunes, 10 de marzo de 2014 11:52 *Para:* Kamailio (SER) - Users Mailing List *Asunto:* Re: [SR-Users] Kamailio LCR unable to relay new calls, memory issue Hello, how much shared memory have you allocated to kamailio? In other words, have you given any value via -m command line parameter? You can see it also via: kamctl stats Look for shmem fields. Cheers, Daniel On 10/03/14 14:37, Camila Troncoso wrote: Hi, Last Friday I presented a increase of load in kamailio by a client. The load was almost twice as usual. By that time I encountered with many issues in kamailio. The calls were not relay to gateways, and a lot of messages about memory were appearing in log file. Kamailio didn’t crash. When I remove the extra traffic all went back to normal. Is there a issue about memory leak or about to few memory resources to manage this amount of traffic? ERROR: tm [t_reply.c:1799]: ERROR: relay_reply: cannot alloc reply shmem ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed ERROR: tm [t_reply.c:598]: ERROR: _reply_light: cannot allocate shmem buffer ERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request MERROR: tm [t_msgbuilder.c:371]: ERROR: cannot allocate shared memory ERROR: tm [t_msgbuilder.c:521]: ERROR: build_local_reparse: cannot build ACK request ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL) ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request:/p> ERROR: <core> [msg_translator.c:1782]: ERROR: build_req_buf_from_sip_req: out of memory ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem ERROR: tm [t_fwd.c:1534]: ERROR: t_forward_nonack: failure to add branches ERROR: tm [tm.c:1369]: ERROR: w_t_relay_to: t_relay_to failed ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: No error (2/SL) ERROR: <core> [sip_msg_clone.c:506]: ERROR: sip_msg_cloner: cannot allocate memory ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL) Regards, Camila *<image001.gif>* * Camila Troncoso Solar **|* Ingeniero de Desarollo +56 2 22408535 *·* +56 9 97327220 | ctroncoso@redvoiss.net <mailto:ctroncoso@redvoiss.net>__ Badajoz 130, piso 16, Las Condes | Santiago - CHILE| /_www.redvoiss.com <http://www.redvoiss.com>_/ _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -http://www.asipto.com http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Kamailio World Conference - April 2-4, 2014, Berlin, Germany http://www.kamailioworld.com -- Daniel-Constantin Mierla -http://www.asipto.com http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Kamailio World Conference - April 2-4, 2014, Berlin, Germany http://www.kamailioworld.com _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -http://www.asipto.com http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Kamailio World Conference - April 2-4, 2014, Berlin, Germany http://www.kamailioworld.com D"http://www.kamailioworld.com">http://www.kamailioworld.com <graph.txt>
-
Camila Troncoso -
Daniel-Constantin Mierla