Hi all
we are running kamailio 3.1.1. Basically for every call it has to query a mtree that is reloaded every 15 minutes. The mtree table is about 4M records and is stored in a Pgostgres database. For the reload we use the command mi mt_reload embedded in a xmlrpc request. The code for this is: route[XMLRPC]{ if (method == "POST" || method == "GET") { if (search("^User-Agent:")) { xlog("xmlrpc request\n"); set_reply_close(); set_reply_no_connect(); # optional dispatch_rpc(); } exit; } } This server has been running with no issued for some weeks now, but at some point during the weekend the mi command started to fail. This is what the logs show:
ERROR: <core> [db_res.c:181]: no private memory left ERROR: db_postgres [km_res.c:225]: could not allocate rows ERROR: db_postgres [km_dbase.c:280]: failed to convert rows ERROR: mtree [mtree_mod.c:627]: Error while fetching result ERROR: db_postgres [km_dbase.c:318]: invalid parameter value ERROR: mtree [mtree_mod.c:738]: cannot re-load info from database
I've restarted kamailio and now it's working fine. Should we increase the private memory? On the other hand, we'd like to detect this failure condition on the script and act accordingly (for instance sending an email with an alarm message), is there anyway to do this?
Thanks in advance
Regards
Javier
On Monday 21 February 2011, Javier Gallart wrote:
we are running kamailio 3.1.1. Basically for every call it has to query a mtree that is reloaded every 15 minutes. The mtree table is about 4M records and is stored in a Pgostgres database. For the reload we use the command mi mt_reload embedded in a xmlrpc request. The code for this is: route[XMLRPC]{ if (method == "POST" || method == "GET") { if (search("^User-Agent:")) { xlog("xmlrpc request\n"); set_reply_close(); set_reply_no_connect(); # optional dispatch_rpc(); } exit; } } This server has been running with no issued for some weeks now, but at some point during the weekend the mi command started to fail. This is what the logs show:
ERROR: <core> [db_res.c:181]: no private memory left ERROR: db_postgres [km_res.c:225]: could not allocate rows ERROR: db_postgres [km_dbase.c:280]: failed to convert rows ERROR: mtree [mtree_mod.c:627]: Error while fetching result ERROR: db_postgres [km_dbase.c:318]: invalid parameter value ERROR: mtree [mtree_mod.c:738]: cannot re-load info from database
I've restarted kamailio and now it's working fine. Should we increase the private memory? On the other hand, we'd like to detect this failure condition on the script and act accordingly (for instance sending an email with an alarm message), is there anyway to do this?
Hi Javier,
this probably caused of the fragmentation of the internal private memory pool from the frequent dumps and other load. You could increase the pool, or better - switch the MI to the system memory pool by uncommenting this line in kmi/Makefile
#DEFS+= -DMI_SYSTEM_MALLOC
and recompile the server.
Cheers,
Henning
Thanks Henning,
I'll go with to recompile option and post the results...
Regards
Javier
2011/2/21 Henning Westerholt henning.westerholt@1und1.de
On Monday 21 February 2011, Javier Gallart wrote:
we are running kamailio 3.1.1. Basically for every call it has to query a mtree that is reloaded every 15 minutes. The mtree table is about 4M records and is stored in a Pgostgres database. For the reload we use the command mi mt_reload embedded in a xmlrpc request. The code for this is: route[XMLRPC]{ if (method == "POST" || method == "GET") { if (search("^User-Agent:")) { xlog("xmlrpc request\n"); set_reply_close(); set_reply_no_connect(); # optional dispatch_rpc(); } exit; } } This server has been running with no issued for some weeks now, but at
some
point during the weekend the mi command started to fail. This is what the logs show:
ERROR: <core> [db_res.c:181]: no private memory left ERROR: db_postgres [km_res.c:225]: could not allocate rows ERROR: db_postgres [km_dbase.c:280]: failed to convert rows ERROR: mtree [mtree_mod.c:627]: Error while fetching result ERROR: db_postgres [km_dbase.c:318]: invalid parameter value ERROR: mtree [mtree_mod.c:738]: cannot re-load info from database
I've restarted kamailio and now it's working fine. Should we increase the private memory? On the other hand, we'd like to detect this failure condition on the script and act accordingly (for instance sending an
with an alarm message), is there anyway to do this?
Hi Javier,
this probably caused of the fragmentation of the internal private memory pool from the frequent dumps and other load. You could increase the pool, or better
- switch the MI to the system memory pool by uncommenting this line in
kmi/Makefile
#DEFS+= -DMI_SYSTEM_MALLOC
and recompile the server.
Cheers,
Henning
Hello
I've upgraded to 3.1.2 and compiled with the DEFS+= -DMI_SYSTEM_MALLOC option. Looking good so far. Any thoughts about my second question, that is, how to be aware from the scripts if the any problems has arisen in the mi command?
Thanks
Javi
On Mon, Feb 21, 2011 at 5:21 PM, Javier Gallart jgallartm@gmail.com wrote:
Thanks Henning,
I'll go with to recompile option and post the results...
Regards
Javier
2011/2/21 Henning Westerholt henning.westerholt@1und1.de
On Monday 21 February 2011, Javier Gallart wrote:
we are running kamailio 3.1.1. Basically for every call it has to query
a
mtree that is reloaded every 15 minutes. The mtree table is about 4M records and is stored in a Pgostgres database. For the reload we use
the
command mi mt_reload embedded in a xmlrpc request. The code for this is: route[XMLRPC]{ if (method == "POST" || method == "GET") { if (search("^User-Agent:")) { xlog("xmlrpc request\n"); set_reply_close(); set_reply_no_connect(); # optional dispatch_rpc(); } exit; } } This server has been running with no issued for some weeks now, but at
some
point during the weekend the mi command started to fail. This is what
the
logs show:
ERROR: <core> [db_res.c:181]: no private memory left ERROR: db_postgres [km_res.c:225]: could not allocate rows ERROR: db_postgres [km_dbase.c:280]: failed to convert rows ERROR: mtree [mtree_mod.c:627]: Error while fetching result ERROR: db_postgres [km_dbase.c:318]: invalid parameter value ERROR: mtree [mtree_mod.c:738]: cannot re-load info from database
I've restarted kamailio and now it's working fine. Should we increase
the
private memory? On the other hand, we'd like to detect this failure condition on the script and act accordingly (for instance sending an
with an alarm message), is there anyway to do this?
Hi Javier,
this probably caused of the fragmentation of the internal private memory pool from the frequent dumps and other load. You could increase the pool, or better
- switch the MI to the system memory pool by uncommenting this line in
kmi/Makefile
#DEFS+= -DMI_SYSTEM_MALLOC
and recompile the server.
Cheers,
Henning
On Wednesday 23 February 2011, Javier Gallart wrote:
I've upgraded to 3.1.2 and compiled with the DEFS+= -DMI_SYSTEM_MALLOC option. Looking good so far. Any thoughts about my second question, that is, how to be aware from the scripts if the any problems has arisen in the mi command?
Hi Javier,
there are several ways how to monitor this condition:
* create a check that grep in the log file for the error msg * check the output of "kamctl fifo get_statistics all" for the memory informations * do a OPTIONS ping to the server that is replied statefully (thus allocates memory) and check this * and probably the best, just check the return value of the mtree reload FIFO cmd?
Cheers,
Henning
Thanks Henning,
we are currently doing it looking for error messages in the logs, and it´s ok, but since the request is processed by the kamailio routing script via the xmlrpc module with the dispatch_rpc() command , I was wondering if there could be a way to check the command output inside the same script.
Regards
Javi
2011/2/24 Henning Westerholt henning.westerholt@1und1.de
On Wednesday 23 February 2011, Javier Gallart wrote:
I've upgraded to 3.1.2 and compiled with the DEFS+= -DMI_SYSTEM_MALLOC option. Looking good so far. Any thoughts about my second question, that is, how to be aware from the scripts if the any problems has arisen in
the
mi command?
Hi Javier,
there are several ways how to monitor this condition:
- create a check that grep in the log file for the error msg
- check the output of "kamctl fifo get_statistics all" for the memory informations
- do a OPTIONS ping to the server that is replied statefully (thus
allocates memory) and check this
- and probably the best, just check the return value of the mtree reload
FIFO cmd?
Cheers,
Henning
On Friday 25 February 2011, Javier Gallart wrote:
we are currently doing it looking for error messages in the logs, and it´s ok, but since the request is processed by the kamailio routing script via the xmlrpc module with the dispatch_rpc() command , I was wondering if there could be a way to check the command output inside the same script.
Hi Javier,
this xmlrpc transport does not return a result from the command execution on the server?
Cheers,
Henning
Hi Henning
yes, it does. We want to build the logic for acting upon a tree reload failure -like sending an alarm or something of the kind-. We were deciding whether to do it at the external java application that actually sends the xmlrpc request and that will receive the response, or at the kamailo itself, but I've found no way to access to the command output directly from the kamailio configuration script. But that's fine, we will just do it outside.
Thanks for everything
Regards
Javi
On Mon, Feb 28, 2011 at 1:38 PM, Henning Westerholt < henning.westerholt@1und1.de> wrote:
On Friday 25 February 2011, Javier Gallart wrote:
we are currently doing it looking for error messages in the logs, and
it´s
ok, but since the request is processed by the kamailio routing script via the xmlrpc module with the dispatch_rpc() command , I was wondering if there could be a way to check the command output inside the same script.
Hi Javier,
this xmlrpc transport does not return a result from the command execution on the server?
Cheers,
Henning
-
Henning Westerholt -
Javier Gallart