18 May
2009
18 May
'09
3:24 p.m.
Hi all,
we're using kamailio 1.5.1 with modules htable and perl to generate
CDRs. Using htable we save some variables which are dumped to file using
the perl module at the end of the call.
We did a small stress test with 5 calls in parallel (each call about 20
seconds) and we got after about 6 hours the following error:
ERROR:core:pv_sprintf: pv_sprintf: Memory exhausted!
No CDRs were written anymore, the perl method was not called. Everything
else seems to run fine.
After stopping the stress test doing one simple call with a softphone
worked and a CDR had been written (without restarting kamailio!).
Perhaps the perl script is buggy?
Any ideas how to fix this are greatly appreciated. We don't think
extending the PKG_MEM_POOL_SIZE will solve the problem. This is
currently 4MB and should be enough for 5 concurrent calls!?!
Thanks,
Christian Koch
The perl script:
use OpenSER;
use OpenSER::Constants;
use IO::Socket;
sub write_cdr {
my $m = shift;
#my $v = $m->getVersion();
#my $t = $m->getType();
my $index = $m->pseudoVar("\$ci");
my $endTimeStamp = $m->pseudoVar("\$Ts");
my $startTimeStamp = $endTimeStamp -
$m->pseudoVar("\$DLG_lifetime");
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($startTimeStamp);
my $startTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday,
$mon+1, $year+1900, $hour, $min, $sec);
my $fileDate = sprintf("%02d%02d%02d", $mday, $mon+1, $year%100);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($endTimeStamp);
my $endTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday,
$mon+1, $year+1900, $hour, $min, $sec);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($m->pseudoVar("\$sht(a=>".$index."::invtrytime)"));
my $InvTryingTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d",
$mday, $mon+1, $year+1900, $hour, $min, $sec);
$ReplyReason=$m->pseudoVar("\$rr");
$ReplyReason=~ s/;/_/g;
$FromDisplayName =
$m->pseudoVar("\$sht(a=>".$index."::fromdplname)");
$FromDisplayName=~ s/;/_/g;
$ToUri = $m->pseudoVar("\$sht(a=>".$index."::touri)");
$ToUri=~s/;/_/g;
$FromUri =
$m->pseudoVar("\$sht(a=>".$index."::fromuri)");
$FromUri=~s/;/_/g;
$UserAgent =
$m->pseudoVar("\$sht(a=>".$index."::useragent)");
$UserAgent=~s/;/_/g;
my $CDR = $startTime.";"; # 1 + 2
$CDR .= $endTime.";"; # 3 + 4
$CDR .= $m->pseudoVar("\$DLG_lifetime").";"; # 5
$CDR .= $index.";"; #6 CallID
$CDR .=
$m->pseudoVar("\$sht(a=>".$index."::srcip)").";"; #
source-IP of INVITE
$CDR .= $m->pseudoVar("\$si").";"; # source-IP of
BYE/CANCEL
$CDR .= $FromDisplayName.";"; # from display-name INVITE
$CDR .= $InvTryingTime.";"; # Alerting time
$CDR .= ($startTimeStamp -
$m->pseudoVar("\$sht(a=>".$index."::invtrytime)")).";";
#Alerting period
$CDR .= $m->pseudoVar("\$DLG_count").";"; # current amount
of calls
$CDR .= $ToUri.";"; # ToUri from Invite
$CDR .= $FromUri.";"; # FromUri from Invite
$CDR .= $UserAgent.";"; # UserAgent from Invite
$CDR .= $m->pseudoVar("\$rm").";"; # Method
(INVITE,BYE,CANCEL)
$CDR .= $m->pseudoVar("\$rs").";"; # SIP reply status
$CDR .= $ReplyReason.";"; #SIP Reply reason
open(MYOUTFILE,
">>/home/sys/log/kamailio".$fileDate.".cdr.".$m->pseudoVar("\$pp"));
print MYOUTFILE $CDR."\n";
close(MYOUTFILE);
#delete(%sourceIp->{$index});
#delete(%fromDisplayName->{$index});
#delete(%InviteTryingTimestamp->{$index});
#delete(%ToUri->{$index});
#delete(%FromUri->{$index});
#delete(%UserAgent->{$index});
return 1;
}
1;
18 May
18 May
3:39 p.m.
Hello,
On 05/18/2009 04:24 PM, Christian Koch wrote:
Hi all,
we're using kamailio 1.5.1 with modules htable and perl to generate
CDRs. Using htable we save some variables which are dumped to file using
the perl module at the end of the call.
We did a small stress test with 5 calls in parallel (each call about 20
seconds) and we got after about 6 hours the following error:
ERROR:core:pv_sprintf: pv_sprintf: Memory exhausted!
this is system memory error. Has nothing to do with pkg memory used by
Kamailio. Seems that perl module allocates some memory and forgets to
free it. Do you see other errors in your syslog?
One memleak I could spot is that the allocated buffer to print the
output in pv_sprintf is not freed if the formatted string is bad. Then I
see that the output buffer is duplicated with strdup and it is freed. So
seems there are some system memory leaks in perl module, indeed.
No CDRs were written anymore, the perl method was not
called. Everything
else seems to run fine.
Yes, because the problem was with system memory, while kamailio uses
internally pkg.
After stopping the stress test doing one simple call
with a softphone
worked and a CDR had been written (without restarting kamailio!).
Perhaps the perl script is buggy?
I have no much experience with perl to state that.
Any ideas how to fix this are greatly appreciated. We
don't think
extending the PKG_MEM_POOL_SIZE will solve the problem. This is
currently 4MB and should be enough for 5 concurrent calls!?!
pkg is more than enough.
Cheers,
Daniel
Thanks,
Christian Koch
The perl script:
use OpenSER;
use OpenSER::Constants;
use IO::Socket;
sub write_cdr {
my $m = shift;
#my $v = $m->getVersion();
#my $t = $m->getType();
my $index = $m->pseudoVar("\$ci");
my $endTimeStamp = $m->pseudoVar("\$Ts");
my $startTimeStamp = $endTimeStamp -
$m->pseudoVar("\$DLG_lifetime");
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($startTimeStamp);
my $startTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday,
$mon+1, $year+1900, $hour, $min, $sec);
my $fileDate = sprintf("%02d%02d%02d", $mday, $mon+1, $year%100);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($endTimeStamp);
my $endTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday,
$mon+1, $year+1900, $hour, $min, $sec);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($m->pseudoVar("\$sht(a=>".$index."::invtrytime)"));
my $InvTryingTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d",
$mday, $mon+1, $year+1900, $hour, $min, $sec);
$ReplyReason=$m->pseudoVar("\$rr");
$ReplyReason=~ s/;/_/g;
$FromDisplayName =
$m->pseudoVar("\$sht(a=>".$index."::fromdplname)");
$FromDisplayName=~ s/;/_/g;
$ToUri = $m->pseudoVar("\$sht(a=>".$index."::touri)");
$ToUri=~s/;/_/g;
$FromUri =
$m->pseudoVar("\$sht(a=>".$index."::fromuri)");
$FromUri=~s/;/_/g;
$UserAgent =
$m->pseudoVar("\$sht(a=>".$index."::useragent)");
$UserAgent=~s/;/_/g;
my $CDR = $startTime.";"; # 1 + 2
$CDR .= $endTime.";"; # 3 + 4
$CDR .= $m->pseudoVar("\$DLG_lifetime").";"; # 5
$CDR .= $index.";"; #6 CallID
$CDR .=
$m->pseudoVar("\$sht(a=>".$index."::srcip)").";"; #
source-IP of INVITE
$CDR .= $m->pseudoVar("\$si").";"; # source-IP of
BYE/CANCEL
$CDR .= $FromDisplayName.";"; # from display-name INVITE
$CDR .= $InvTryingTime.";"; # Alerting time
$CDR .= ($startTimeStamp -
$m->pseudoVar("\$sht(a=>".$index."::invtrytime)")).";";
#Alerting period
$CDR .= $m->pseudoVar("\$DLG_count").";"; # current
amount of calls
$CDR .= $ToUri.";"; # ToUri from Invite
$CDR .= $FromUri.";"; # FromUri from Invite
$CDR .= $UserAgent.";"; # UserAgent from Invite
$CDR .= $m->pseudoVar("\$rm").";"; # Method
(INVITE,BYE,CANCEL)
$CDR .= $m->pseudoVar("\$rs").";"; # SIP reply status
$CDR .= $ReplyReason.";"; #SIP Reply reason
open(MYOUTFILE,
">>/home/sys/log/kamailio".$fileDate.".cdr.".$m->pseudoVar("\$pp"));
print MYOUTFILE $CDR."\n";
close(MYOUTFILE);
#delete(%sourceIp->{$index});
#delete(%fromDisplayName->{$index});
#delete(%InviteTryingTimestamp->{$index});
#delete(%ToUri->{$index});
#delete(%FromUri->{$index});
#delete(%UserAgent->{$index});
return 1;
}
1;
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
--
Daniel-Constantin Mierla
http://www.asipto.com/
19 May
19 May
3:25 p.m.
Hello Daniel,
Daniel-Constantin Mierla schrieb:
Are you sure this is system memory? We changed PKG_MEM_POOL_SIZE from 4
to 8MB and a stress test didn't show any problems for more then 12
hours. Then we changed it to 0.5MB and the problem occured after about 3
hours again wiht the same error message.
The error message in pv_sprintf() occurs when pkg_malloc() returns NULL,
so this should be a PKG issue?!?
Currently we're running a test with 0.5MB of pkg memory, but we've fixed
the error in pv_sprintf where the format is wrong and the buffer is not
freed. In the test this morning (with 0.5MB) we had the message "wrong
format" only once, and in all the other tests the message didn't show
up. So we don't think this fixes the problem.
Do you think only one missing pkg_free() (like in the last test) can
corrupt the memory so this error occurs?
Thanks for your help!
Christian
we're
using kamailio 1.5.1 with modules htable and perl to generate
CDRs. Using htable we save some variables which are dumped to file
using the perl module at the end of the call.
We did a small stress test with 5 calls in parallel (each call about
20 seconds) and we got after about 6 hours the following error:
ERROR:core:pv_sprintf: pv_sprintf: Memory exhausted!
this is system memory error. Has nothing to do with pkg memory used by
Kamailio. Seems that perl module allocates some memory and forgets to
free it. Do you see other errors in your syslog?
One memleak I could spot is that the allocated buffer to print the
output in pv_sprintf is not freed if the formatted string is bad. Then
I see that the output buffer is duplicated with strdup and it is
freed. So seems there are some system memory leaks in perl module,
indeed.
3:50 p.m.
Hello,
On 05/19/2009 04:25 PM, Christian Koch wrote:
Hello Daniel,
Daniel-Constantin Mierla schrieb:
Are you sure this is system memory?
you are right, somehow I was blind
when I checked the pv_sprintf, it is
clear pkg_malloc().
we're
using kamailio 1.5.1 with modules htable and perl to generate
CDRs. Using htable we save some variables which are dumped to file
using the perl module at the end of the call.
We did a small stress test with 5 calls in parallel (each call about
20 seconds) and we got after about 6 hours the following error:
ERROR:core:pv_sprintf: pv_sprintf: Memory exhausted!
this is system memory error. Has nothing to do with pkg memory used by
Kamailio. Seems that perl module allocates some memory and forgets to
free it. Do you see other errors in your syslog?
One memleak I could spot is that the allocated buffer to print the
output in pv_sprintf is not freed if the formatted string is bad. Then
I see that the output buffer is duplicated with strdup and it is
freed. So seems there are some system memory leaks in perl module,
indeed.
We changed PKG_MEM_POOL_SIZE from 4
to 8MB and a stress test didn't show any problems for more then 12
hours. Then we changed it to 0.5MB and the problem occured after about 3
hours again wiht the same error message.
The error message in pv_sprintf() occurs when pkg_malloc() returns NULL,
so this should be a PKG issue?!?
Currently we're running a test with 0.5MB of pkg memory, but we've fixed
the error in pv_sprintf where the format is wrong and the buffer is not
freed. In the test this morning (with 0.5MB) we had the message "wrong
format" only once, and in all the other tests the message didn't show
up. So we don't think this fixes the problem.
Do you think only one missing pkg_free() (like in the last test) can
corrupt the memory so this error occurs?
Now that the issue is pkg, you can compile and run in mem debug mode so
we can detect easier where the leak is, check this:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com/
20 May
20 May
11:17 a.m.
Hi Daniel,
Daniel-Constantin Mierla schrieb:
We've compiled kamailio with mem debug, but now kamailio is very, very
slow, so that it's really not possible to make a stress test. We were
able to place one single call (The INVITE took aboute 1:30 min until it
reached the destination).
The memory status at shutdown is available at:
http://rcpt.yousendit.com/690135386/7f3f82ffe0daf468b783ecf6e9e61bc8
Hopefully this will be helpful.
Thanks for your help,
Christian
We changed
PKG_MEM_POOL_SIZE from 4 to 8MB and a stress test didn't
show any problems for more then 12 hours. Then we changed it to 0.5MB
and the problem occured after about 3 hours again wiht the same error
message.
The error message in pv_sprintf() occurs when pkg_malloc() returns
NULL, so this should be a PKG issue?!?
Currently we're running a test with 0.5MB of pkg memory, but we've
fixed the error in pv_sprintf where the format is wrong and the
buffer is not freed. In the test this morning (with 0.5MB) we had the
message "wrong format" only once, and in all the other tests the
message didn't show up. So we don't think this fixes the problem.
Do you think only one missing pkg_free() (like in the last test) can
corrupt the memory so this error occurs?
Now that the issue is pkg, you can compile and run in mem debug mode
so we can detect easier where the leak is, check this:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
11:25 a.m.
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
Hi Christian,
you say that processing one INVITE needs 1:30 min? This is way to slow even
with memory debugging. I regularly use this for debugging, normally you should
not notice a big difference in operation. Do you get any "unusual" messages
during the processing of the INVITE with mem debug?
Henning
[..]
Now that the issue is pkg, you can compile and run in mem debug mode
so we can detect easier where the leak is, check this:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
We've compiled kamailio with mem debug, but now kamailio is very, very
slow, so that it's really not possible to make a stress test. We were
able to place one single call (The INVITE took aboute 1:30 min until it
reached the destination).
11:34 a.m.
Hello,
On 05/20/2009 12:25 PM, Henning Westerholt wrote:
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
Hi Christian,
you say that processing one INVITE needs 1:30 min? This is way to slow
even with memory debugging. I regularly use this for debugging,
normally you should not notice a big difference in operation. Do you
get any "unusual" messages during the processing of the INVITE with
mem debug?
is the syslog configured in asynchronous mode?
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com/
[..]
Now that the issue is pkg, you can compile and run in mem debug mode
so we can detect easier where the leak is, check this:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
We've compiled kamailio with mem debug, but now kamailio is very, very
slow, so that it's really not possible to make a stress test. We were
able to place one single call (The INVITE took aboute 1:30 min until it
reached the destination).
11:34 a.m.
Hi Henning,
Henning Westerholt schrieb:
you say that processing one INVITE needs 1:30 min?
This is way to slow
even with memory debugging. I regularly use this for debugging,
normally you should not notice a big difference in operation. Do you
get any "unusual" messages during the processing of the INVITE with
mem debug?
I can't see any unusal messages or errors (beside the fact, that I'm not
sure what is unusal ;-) ), but even startup is very slow. It needs
several minutes before we even see all kamailio processes in ps aux.
Here is one fragment of the startup logfile, where you can see, that
within one second not really much happens:
May 20 09:42:20 AmbriaSip1 kamailio: params (0x81677e0, 96), called from
route_struct.c: mk_action(106)
May 20 09:42:20 AmbriaSip1 kamailio: params (0x81677e0, 96), returns
address 0x81b3d9c frag. 0x81b3d84 (size=96) on 1 -th hit
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 96), called from
route_struct.c: mk_action(106)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 96), returns
address 0x81b3e2c frag. 0x81b3e14 (size=96) on 1 -th hit
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 96), called from
route_struct.c: mk_action(106)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 96), returns
address 0x81b3ebc frag. 0x81b3ea4 (size=96) on 1 -th hit
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 3200), called
from socket_info.c: add_interfaces(397)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 3200), returns
address 0x81b3f4c frag. 0x81b3f34 (size=3200) on 1 -th hit
May 20 09:42:21 AmbriaSip1 kamailio: params(0x81677e0, 0x81b3f4c),
called from socket_info.c: add_interfaces(411)
May 20 09:42:21 AmbriaSip1 kamailio: freeing frag. 0x81b3f34 alloc'ed
from socket_info.c: add_interfaces(397)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 6400), called
from socket_info.c: add_interfaces(397)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 6400), returns
address 0x81b4bfc frag. 0x81b4be4 (size=6400) on 1 -th hit
May 20 09:42:21 AmbriaSip1 kamailio: params(0x81677e0, 0x81b4bfc),
called from socket_info.c: add_interfaces(479)
May 20 09:42:21 AmbriaSip1 kamailio: freeing frag. 0x81b4be4 alloc'ed
from socket_info.c: add_interfaces(397)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 3200), called
from socket_info.c: add_interfaces(397)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 3200), returns
address 0x81b3f4c frag. 0x81b3f34 (size=3200) on 1 -th hit
May 20 09:42:21 AmbriaSip1 kamailio: params(0x81677e0, 0x81b3f4c),
called from socket_info.c: add_interfaces(411)
May 20 09:42:21 AmbriaSip1 kamailio: freeing frag. 0x81b3f34 alloc'ed
from socket_info.c: add_interfaces(397)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 6400), called
from socket_info.c: add_interfaces(397)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 6400), returns
address 0x81b4bfc frag. 0x81b4be4 (size=6400) on 1 -th hit
May 20 09:42:21 AmbriaSip1 kamailio: params(0x81677e0, 0x81b4bfc),
called from socket_info.c: add_interfaces(479)
May 20 09:42:21 AmbriaSip1 kamailio: freeing frag. 0x81b4be4 alloc'ed
from socket_info.c: add_interfaces(397)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 5), called from
socket_info.c: fix_socket_list(532)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 8), returns
address 0x81b3f4c frag. 0x81b3f34 (size=8) on 1 -th hit
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 15), called from
socket_info.c: fix_socket_list(583)
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 16), returns
address 0x81b3f84 frag. 0x81b3f6c (size=16) on 1 -th hit
May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 16), called from
name_alias.h: add_alias(99)
May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 16), returns
address 0x81b3fc4 frag. 0x81b3fac (size=16) on 1 -th hit
May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 11), called from
name_alias.h: add_alias(101)
May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 12), returns
address 0x81b4004 frag. 0x81b3fec (size=12) on 1 -th hit
The logfile is full with this kind of messages.
The load of the machine (Xeon dualcore @3GHz) goes up to 2, but using
top we can't see that kamailio is using much CPU time (sometimes about
20%, but not more).
Regards,
Christian
11:42 a.m.
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
> you say that processing one INVITE needs 1:30
min? This is way to slow
> even with memory debugging. I regularly use this for debugging,
> normally you should not notice a big difference in operation. Do you
> get any "unusual" messages during the processing of the INVITE with
> mem debug?
Hi Christian,
I can't see any unusal messages or errors (beside
the fact, that I'm not
sure what is unusal ;-) ), but even startup is very slow. It needs
several minutes before we even see all kamailio processes in ps aux.
Here is one fragment of the startup logfile, where you can see, that
within one second not really much happens:
This long startup is also a bit unusual. Sure, it needs to allocates some
memory, and this informations needs to be written to syslog, but several
minutes is too slow. Do you read some database content in memory on startup,
like a huge location table, or a big carrierroute route tree with several
hundreds of MB?
May 20 09:42:20 AmbriaSip1 kamailio: params
(0x81677e0, 96), called from
route_struct.c: mk_action(106)
[..]
May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 12), returns
address 0x81b4004 frag. 0x81b3fec (size=12) on 1 -th hit
The logfile is full with this kind of messages.
The load of the machine (Xeon dualcore @3GHz) goes up to 2, but using
top we can't see that kamailio is using much CPU time (sometimes about
20%, but not more).
So in which processes is the rest of the CPU time spend?
Cheers,
Henning
11:50 a.m.
Hi Henning,
Henning Westerholt schrieb:
We're connecting to a database, but it
includes only a few (about 20)
subscribers, nothing else. We didn't stress test the database
(-connection). Without memdebug kamailio runs fast.
Most of the time the CPU
seems to be idle. Sometimes syslog shows up in
top, but without much CPU time.
Do you think this could be a dual-CPU issue? Perhaps syncing the memory
access between the CPUs is slow?
If you like to see the complete logfile we can post this too.
Regards,
Christian
I can't
see any unusal messages or errors (beside the fact, that I'm not
sure what is unusal ;-) ), but even startup is very slow. It needs
several minutes before we even see all kamailio processes in ps aux.
Here is one fragment of the startup logfile, where you can see, that
within one second not really much happens:
This long startup is also a bit unusual. Sure, it needs to allocates
some memory, and this informations needs to be written to syslog, but
several minutes is too slow. Do you read some database content in
memory on startup, like a huge location table, or a big carrierroute
route tree with several hundreds of MB? May 20 09:42:20 AmbriaSip1 kamailio: params
(0x81677e0, 96), called from
route_struct.c: mk_action(106)
[..]
May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 12), returns
address 0x81b4004 frag. 0x81b3fec (size=12) on 1 -th hit
The logfile is full with this kind of messages.
The load of the machine (Xeon dualcore @3GHz) goes up to 2, but using
top we can't see that kamailio is using much CPU time (sometimes about
20%, but not more).
So in which processes is the rest of the CPU time spend?
12:22 p.m.
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
This is strange. Do you log to a remote host, for example with TCP? Or do
you've another big IO load on the machine, that could block the writing of
logs to the disk somehow?
Most of the time the CPU seems to be idle. Sometimes syslog shows up in
top, but without much CPU time.
Do you think this could be a dual-CPU issue? Perhaps syncing the memory
access between the CPUs is slow?
If you like to see the complete logfile we can post this too.
I'd guess most people run kamailio on a multi-CPU setup nowadays, so i don't
think this is the reason. So what causes the load of 2 during startup, if not
kamailio or syslog?
Henning
This long
startup is also a bit unusual. Sure, it needs to allocates
some memory, and this informations needs to be written to syslog, but
several minutes is too slow. Do you read some database content in
memory on startup, like a huge location table, or a big carrierroute
route tree with several hundreds of MB?
We're connecting to a database, but it includes only a few (about 20)
subscribers, nothing else. We didn't stress test the database
(-connection). Without memdebug kamailio runs fast. May 20 09:42:20 AmbriaSip1 kamailio: params
(0x81677e0, 96), called
from route_struct.c: mk_action(106)
[..]
May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 12), returns
address 0x81b4004 frag. 0x81b3fec (size=12) on 1 -th hit
The logfile is full with this kind of messages.
The load of the machine (Xeon dualcore @3GHz) goes up to 2, but using
top we can't see that kamailio is using much CPU time (sometimes about
20%, but not more).
So in which processes is the rest of the CPU time spend?
2:08 p.m.
Henning Westerholt schrieb:
We solved the problem with the bad performance in mem debuge mode.
Kamailio not only logs to /var/log/kamailio but also to
/var/log/messages (?). /var/log/messages was not configured asynchrouns
in syslog.conf. We fixed this and will retry the stress test with
enabled mem debug.
Thanks a lot!
Christian
We're
connecting to a database, but it includes only a few (about 20)
subscribers, nothing else. We didn't stress test the database
(-connection). Without memdebug kamailio runs fast.
This is strange. Do you log to a remote host, for example with TCP? Or
do you've another big IO load on the machine, that could block the
writing of logs to the disk somehow?
5:52 p.m.
Hello Daniel, hello Henning,
Daniel-Constantin Mierla schrieb:
After solving all the performance issues with syslog we now made a
stress test with memdebug enabled. Now kamailio terminates itself
(perhaps because memory is corrupted?).
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]:
CRITICAL:core:qm_free: bad pointer (nil) (out of memory block!) - aborting
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27697]:
CRITICAL:core:receive_fd: EOF on 12
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]:
INFO:core:handle_sigs: child process 27684 exited by a signal 6
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]:
INFO:core:handle_sigs: core was not generated
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]:
INFO:core:handle_sigs: terminating due to SIGCHLD
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]:
INFO:core:sig_usr: signal 15 received
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: Memory
status (pkg):
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: qm_status
(0x81677e0):
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: heap size=
1048576
The complete output of the memory status is available here:
https://rcpt.yousendit.com/690295962/7b39d332264f086b1bf0f134c026fad3
It would be really great if you could have a look at the logfile!
Perhaps you have any ideas what could cause this problem, so we can fix it.
If needed, the complete logfile (from startup to termination) is also
available:
https://rcpt.yousendit.com/690307202/14929008dc7a452d69abb3cbc0a9b58a
Thanks a lot for your support,
Christian
The error
message in pv_sprintf() occurs when pkg_malloc() returns
NULL, so this should be a PKG issue?!?
Currently we're running a test with 0.5MB of pkg memory, but we've
fixed the error in pv_sprintf where the format is wrong and the
buffer is not freed. In the test this morning (with 0.5MB) we had the
message "wrong format" only once, and in all the other tests the
message didn't show up. So we don't think this fixes the problem.
Do you think only one missing pkg_free() (like in the last test) can
corrupt the memory so this error occurs?
We changed PKG_MEM_POOL_SIZE from 4 to 8MB and a stress test didn't
show any problems for more then 12 hours. Then we changed it to 0.5MB
and the problem occured after about 3 hours again wiht the same error
message.
Now that the issue is pkg, you can compile and run in mem debug mode
so we can detect easier where the leak is, check this:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
25 May
25 May
11:24 a.m.
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
[..]
After solving all the performance issues with syslog we now made a
stress test with memdebug enabled. Now kamailio terminates itself
(perhaps because memory is corrupted?).
Hi Christian,
yes, this looks like the memory is corrupt, as it aparently crashes in a core
function (added content from the log file), which normally should be pretty
stable:
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0,
15), called from proxy.c: hostent_cpy(148)
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0,
16), returns address 0x8265b20 frag. 0x8265b08 (size=32) on 1 -th hit
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0,
4), called from proxy.c: hostent_cpy(159)
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0,
4), returns address 0x81bcbe8 frag. 0x81bcbd0 (size=4) on 1 -th hit
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0,
8), called from proxy.c: hostent_cpy(182)
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params(0x81677e0,
0x8265b20), called from proxy.c: hostent_cpy(185)
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: freeing frag.
0x8265b08 alloc'ed from proxy.c: hostent_cpy(148)
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params(0x81677e0,
(nil)), called from proxy.c: hostent_cpy(187)
May 20 15:31:55 AmbriaSip1
/usr/local/sbin/kamailio[27684]:
CRITICAL:core:qm_free: bad pointer (nil) (out of memory block!) - aborting
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27697]:
CRITICAL:core:receive_fd: EOF on 12
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]:
INFO:core:handle_sigs: child process 27684 exited by a signal 6
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]:
INFO:core:handle_sigs: core was not generated
Can you configure the kamailio server that it generates a core file? Then take
a look to the backtrace where the invalid memory access was done, to verify if
its really crashed in the core function, or perhaps some other parts has a
problem here. Further informations:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
May 20 15:31:55 AmbriaSip1
/usr/local/sbin/kamailio[27679]:
INFO:core:handle_sigs: terminating due to SIGCHLD
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]:
INFO:core:sig_usr: signal 15 received
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: Memory
status (pkg):
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: qm_status
(0x81677e0):
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: heap size=
1048576
The complete output of the memory status is available here:
https://rcpt.yousendit.com/690295962/7b39d332264f086b1bf0f134c026fad3
From the logs it seems that indeed a log of memory was allocated from the pv
core. One of the main callers is pv_parse_ht_name, which is from the htable
module. Not sure if this is a valid condition that it allocates that much
pkg_mem, Daniel, can you perhaps take a look?
Cheers,
Henning
12:06 p.m.
Hello Henning, hello Daniel,
Henning Westerholt schrieb:
For one child
process we checked, how many allocated fragments had not
been freed:
[...]
173 -> mk_action(106)
237 -> addstr(787)
3277 -> pv_parse_ht_name(121)
3361 -> pv_parse_format(727)
So we had a look at pv_parse_ht_name() and we wonder, wheter the
variable hpv is freed. Before "return 0" it is used but not freed.
Additionaly the return from pv_parse_format() (&hpv->pve) seems not to
be freed.
Could this be the problem?
Regards,
Christian
Can you configure the kamailio server that it
generates a core file?
Then take a look to the backtrace where the invalid memory access was
done, to verify if its really crashed in the core function, or perhaps
some other parts has a problem here. Further informations:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We will try
this and inform you about the results.
May 20 15:31:55 AmbriaSip1
/usr/local/sbin/kamailio[27679]:
INFO:core:handle_sigs: terminating due to SIGCHLD
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]:
INFO:core:sig_usr: signal 15 received
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: Memory
status (pkg):
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: qm_status
(0x81677e0):
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: heap size=
1048576
The complete output of the memory status is available here:
https://rcpt.yousendit.com/690295962/7b39d332264f086b1bf0f134c026fad3
From the logs it seems that indeed a log of memory was allocated from
the pv core. One of the main callers is pv_parse_ht_name, which is
from the htable module. Not sure if this is a valid condition that it
allocates that much pkg_mem, Daniel, can you perhaps take a look?
26 May
26 May
10:46 a.m.
Hi Henning,
Henning Westerholt schrieb:
Can you configure the kamailio server that it
generates a core file?
Then take a look to the backtrace where the invalid memory access was
done, to verify if its really crashed in the core function, or perhaps
some other parts has a problem here. Further informations:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt
#0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00b64825 in raise () from /lib/tls/libc.so.6
#2 0x00b66289 in abort () from /lib/tls/libc.so.6
#3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e
"proxy.c", func=0x811724e "hostent_cpy", line=187) at
mem/q_malloc.c:444
#4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at proxy.c:187
#5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0, is_sips=0)
at proxy.c:305
#6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218,
uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at ut.h:67
#7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218,
proxy=0x0) at t_fwd.c:630
#8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0, flags=Variable
"flags" is not available.
) at t_funcs.c:264
#9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0) at
tm.c:1002
#10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at action.c:874
#11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at
action.c:145
#12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at
route.c:1171
#13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at
route.c:1488
#14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at
route.c:1493
#15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at action.c:729
#16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at
action.c:145
#17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at action.c:120
#18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at
action.c:145
#19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at action.c:746
#20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at
action.c:145
#21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at action.c:752
#22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at
action.c:145
#23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at action.c:120
#24 0x08087994 in receive_msg (
buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187
SIP/2.0\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175
#25 0x080be606 in udp_rcv_loop () at udp_server.c:449
#26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg
memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1));
if (dst->h_addr_list==0){
ser_error=ret=E_OUT_OF_MEM;
pkg_free(dst->h_name);
for(r=0; dst->h_aliases[r]; r++)
pkg_free(dst->h_aliases[r]);
pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187,
WHERE THE CORE FILE IS GENERATED
pkg_free(dst->h_aliases);
goto error;
}
So, there may be an error in proxy.c, but perhaps the reason for our
problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where the
variable hpv is not freed?
Any comments on pv_parse_ht_name() are greatly appreciated. We could try
to fix this, but we're not sure about the sideeffects.
Regards,
Christian
11:20 a.m.
On Dienstag, 26. Mai 2009, Christian Koch wrote:
> Can you configure the kamailio server that it
generates a core file?
> Then take a look to the backtrace where the invalid memory access was
> done, to verify if its really crashed in the core function, or perhaps
> some other parts has a problem here. Further informations:
> http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
Hi Christian,
We generated a core file, the output of the backtrace
is:
(gdb) bt
#0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00b64825 in raise () from /lib/tls/libc.so.6
#2 0x00b66289 in abort () from /lib/tls/libc.so.6
#3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e
"proxy.c", func=0x811724e "hostent_cpy", line=187) at
mem/q_malloc.c:444
#4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at proxy.c:187
[..]
Thanks for the backtrace, you could try the attached patch (for 1.5.x) if it
solves the crash for you. The additional free after the loop looks indeed a
bit suspicious.
So, there may be an error in proxy.c, but perhaps the
reason for our
problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where the
variable hpv is not freed?
Any comments on pv_parse_ht_name() are greatly appreciated. We could try
to fix this, but we're not sure about the sideeffects.
I also suspect the reason for the crash lays in this area. I did not wrote
that module, thus i asked Daniel.
Regards,
Henning
3 Jun
3 Jun
2:01 p.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hi Henning, hi Daniel,
we did some more investigation and now we think, the problem may occur
when accessing the "inner name" of pv's from htable.
We included a call to backtrace() in pv_parse_ht_name() (where most of
the memory was allocated) to find out, which calling function would be
responsible for freeing the allocated memory. First we thought it could
be xlog, but more tests showed, the error also occurs when called from
other modules.
pv_parse_ht_name() is set as the innername in the htable module:
static pv_export_t mod_pvs[] = {
{ {"sht", sizeof("sht")-1}, PVT_OTHER, pv_get_ht_cell,
pv_set_ht_cell,
pv_parse_ht_name, 0, 0, 0 },
pv_parse_ht_name() allocates some memory (line 121, ht_var.c), saves it
in sp->pvp.pvn.u.dname (line 165) and sets the type to PV_NAME_PVAR. As
"u" is a union we thought freeing the memory should depend on the type.
We can not find any place, where type is checked for PV_NAME_PVAR and
any memory is freed.
Could you give us any hints where this should happen? Maybe this is the
only thing we have to add and the memory leak is gone.
By the way, when searching for PV_NAME_INTSTR we found a potential bug:
In pvapi.c there is the follwoing piece of code (line 773 to 785):
if(ip->pvn.type==PV_NAME_INTSTR)
{
[...]
} else if(ip->pvn.type==PV_NAME_INTSTR) {
/* pvar */
The second if also checks for PV_NAME_INTSTR so it will never be
executed. Shouldn't this be PV_NAME_PVAR?
Regards,
Christian
Christian Koch schrieb:
Hi Henning,
Henning Westerholt schrieb:
Can you configure the kamailio server that it
generates a core file?
Then take a look to the backtrace where the invalid memory access was
done, to verify if its really crashed in the core function, or
perhaps some other parts has a problem here. Further informations:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt
#0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00b64825 in raise () from /lib/tls/libc.so.6
#2 0x00b66289 in abort () from /lib/tls/libc.so.6
#3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e
"proxy.c", func=0x811724e "hostent_cpy", line=187) at
mem/q_malloc.c:444
#4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at
proxy.c:187
#5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0,
is_sips=0) at proxy.c:305
#6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218,
uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at
ut.h:67
#7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218,
proxy=0x0) at t_fwd.c:630
#8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0,
flags=Variable "flags" is not available.
) at t_funcs.c:264
#9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0) at
tm.c:1002
#10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at action.c:874
#11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at
action.c:145
#12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at
route.c:1171
#13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at
route.c:1488
#14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at
route.c:1493
#15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at action.c:729
#16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at
action.c:145
#17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at action.c:120
#18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at
action.c:145
#19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at action.c:746
#20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at
action.c:145
#21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at action.c:752
#22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at
action.c:145
#23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at
action.c:120
#24 0x08087994 in receive_msg (
buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187
SIP/2.0\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175
#25 0x080be606 in udp_rcv_loop () at udp_server.c:449
#26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg
memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1));
if (dst->h_addr_list==0){
ser_error=ret=E_OUT_OF_MEM;
pkg_free(dst->h_name);
for(r=0; dst->h_aliases[r]; r++)
pkg_free(dst->h_aliases[r]);
pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187,
WHERE THE CORE FILE IS GENERATED
pkg_free(dst->h_aliases);
goto error;
}
So, there may be an error in proxy.c, but perhaps the reason for our
problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where the
variable hpv is not freed?
Any comments on pv_parse_ht_name() are greatly appreciated. We could
try to fix this, but we're not sure about the sideeffects.
Regards,
Christian
9 Jun
9 Jun
9:38 a.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello again,
as no one answered our question yet, we would like to ask again if
someone is maybe able to take a deeper look in this?
We are really stuck here and will also post this issue again on the
devel list, maybe someone there is also able to take a look into this.
Any help is really appreciated.
Thanks and regards,
Christian Koch
Christian Koch schrieb:
Hi Henning, hi Daniel,
we did some more investigation and now we think, the problem may occur
when accessing the "inner name" of pv's from htable.
We included a call to backtrace() in pv_parse_ht_name() (where most of
the memory was allocated) to find out, which calling function would be
responsible for freeing the allocated memory. First we thought it
could be xlog, but more tests showed, the error also occurs when
called from other modules.
pv_parse_ht_name() is set as the innername in the htable module:
static pv_export_t mod_pvs[] = {
{ {"sht", sizeof("sht")-1}, PVT_OTHER, pv_get_ht_cell,
pv_set_ht_cell,
pv_parse_ht_name, 0, 0, 0 },
pv_parse_ht_name() allocates some memory (line 121, ht_var.c), saves
it in sp->pvp.pvn.u.dname (line 165) and sets the type to
PV_NAME_PVAR. As "u" is a union we thought freeing the memory should
depend on the type. We can not find any place, where type is checked
for PV_NAME_PVAR and any memory is freed.
Could you give us any hints where this should happen? Maybe this is
the only thing we have to add and the memory leak is gone.
By the way, when searching for PV_NAME_INTSTR we found a potential
bug: In pvapi.c there is the follwoing piece of code (line 773 to 785):
if(ip->pvn.type==PV_NAME_INTSTR)
{
[...]
} else if(ip->pvn.type==PV_NAME_INTSTR) {
/* pvar */
The second if also checks for PV_NAME_INTSTR so it will never be
executed. Shouldn't this be PV_NAME_PVAR?
Regards,
Christian
Christian Koch schrieb:
Hi Henning,
Henning Westerholt schrieb:
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users Can you configure the kamailio server that it
generates a core file?
Then take a look to the backtrace where the invalid memory access
was done, to verify if its really crashed in the core function, or
perhaps some other parts has a problem here. Further informations:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt
#0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00b64825 in raise () from /lib/tls/libc.so.6
#2 0x00b66289 in abort () from /lib/tls/libc.so.6
#3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e
"proxy.c", func=0x811724e "hostent_cpy", line=187) at
mem/q_malloc.c:444
#4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at
proxy.c:187
#5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0,
is_sips=0) at proxy.c:305
#6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218,
uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at
ut.h:67
#7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218,
proxy=0x0) at t_fwd.c:630
#8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0,
flags=Variable "flags" is not available.
) at t_funcs.c:264
#9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0)
at tm.c:1002
#10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at action.c:874
#11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at
action.c:145
#12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at
route.c:1171
#13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at
route.c:1488
#14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at
route.c:1493
#15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at action.c:729
#16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at
action.c:145
#17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at action.c:120
#18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at
action.c:145
#19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at action.c:746
#20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at
action.c:145
#21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at action.c:752
#22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at
action.c:145
#23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at
action.c:120
#24 0x08087994 in receive_msg (
buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187
SIP/2.0\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175
#25 0x080be606 in udp_rcv_loop () at udp_server.c:449
#26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg
memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1));
if (dst->h_addr_list==0){
ser_error=ret=E_OUT_OF_MEM;
pkg_free(dst->h_name);
for(r=0; dst->h_aliases[r]; r++)
pkg_free(dst->h_aliases[r]);
pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187,
WHERE THE CORE FILE IS GENERATED
pkg_free(dst->h_aliases);
goto error;
}
So, there may be an error in proxy.c, but perhaps the reason for our
problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where
the variable hpv is not freed?
Any comments on pv_parse_ht_name() are greatly appreciated. We could
try to fix this, but we're not sure about the sideeffects.
Regards,
Christian
9:41 a.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello Christian,
I got your email but I was out in a short vacation. I will check it.
Thanks,
Daniel
On 06/09/2009 10:38 AM, Christian Koch wrote:
Hello again,
as no one answered our question yet, we would like to ask again if
someone is maybe able to take a deeper look in this?
We are really stuck here and will also post this issue again on the
devel list, maybe someone there is also able to take a look into this.
Any help is really appreciated.
Thanks and regards,
Christian Koch
Christian Koch schrieb:
--
Daniel-Constantin Mierla
http://www.asipto.com/
Hi Henning, hi Daniel,
we did some more investigation and now we think, the problem may
occur when accessing the "inner name" of pv's from htable.
We included a call to backtrace() in pv_parse_ht_name() (where most
of the memory was allocated) to find out, which calling function
would be responsible for freeing the allocated memory. First we
thought it could be xlog, but more tests showed, the error also
occurs when called from other modules.
pv_parse_ht_name() is set as the innername in the htable module:
static pv_export_t mod_pvs[] = {
{ {"sht", sizeof("sht")-1}, PVT_OTHER, pv_get_ht_cell,
pv_set_ht_cell,
pv_parse_ht_name, 0, 0, 0 },
pv_parse_ht_name() allocates some memory (line 121, ht_var.c), saves
it in sp->pvp.pvn.u.dname (line 165) and sets the type to
PV_NAME_PVAR. As "u" is a union we thought freeing the memory should
depend on the type. We can not find any place, where type is checked
for PV_NAME_PVAR and any memory is freed.
Could you give us any hints where this should happen? Maybe this is
the only thing we have to add and the memory leak is gone.
By the way, when searching for PV_NAME_INTSTR we found a potential
bug: In pvapi.c there is the follwoing piece of code (line 773 to 785):
if(ip->pvn.type==PV_NAME_INTSTR)
{
[...]
} else if(ip->pvn.type==PV_NAME_INTSTR) {
/* pvar */
The second if also checks for PV_NAME_INTSTR so it will never be
executed. Shouldn't this be PV_NAME_PVAR?
Regards,
Christian
Christian Koch schrieb:
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hi Henning,
Henning Westerholt schrieb:
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users Can you configure the kamailio server that it
generates a core
file? Then take a look to the backtrace where the invalid memory
access was done, to verify if its really crashed in the core
function, or perhaps some other parts has a problem here. Further
informations:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt
#0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00b64825 in raise () from /lib/tls/libc.so.6
#2 0x00b66289 in abort () from /lib/tls/libc.so.6
#3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e
"proxy.c", func=0x811724e "hostent_cpy", line=187) at
mem/q_malloc.c:444
#4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at
proxy.c:187
#5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0,
is_sips=0) at proxy.c:305
#6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218,
uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at
ut.h:67
#7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218,
proxy=0x0) at t_fwd.c:630
#8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0,
flags=Variable "flags" is not available.
) at t_funcs.c:264
#9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0)
at tm.c:1002
#10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at
action.c:874
#11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at
action.c:145
#12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at
route.c:1171
#13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at
route.c:1488
#14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at
route.c:1493
#15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at
action.c:729
#16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at
action.c:145
#17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at
action.c:120
#18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at
action.c:145
#19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at
action.c:746
#20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at
action.c:145
#21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at
action.c:752
#22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at
action.c:145
#23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at
action.c:120
#24 0x08087994 in receive_msg (
buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187
SIP/2.0\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175
#25 0x080be606 in udp_rcv_loop () at udp_server.c:449
#26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg
memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1));
if (dst->h_addr_list==0){
ser_error=ret=E_OUT_OF_MEM;
pkg_free(dst->h_name);
for(r=0; dst->h_aliases[r]; r++)
pkg_free(dst->h_aliases[r]);
pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187,
WHERE THE CORE FILE IS GENERATED
pkg_free(dst->h_aliases);
goto error;
}
So, there may be an error in proxy.c, but perhaps the reason for our
problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where
the variable hpv is not freed?
Any comments on pv_parse_ht_name() are greatly appreciated. We could
try to fix this, but we're not sure about the sideeffects.
Regards,
Christian
11:13 a.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello,
On 06/03/2009 03:01 PM, Christian Koch wrote:
Hi Henning, hi Daniel,
we did some more investigation and now we think, the problem may occur
when accessing the "inner name" of pv's from htable.
We included a call to backtrace() in pv_parse_ht_name() (where most of
the memory was allocated) to find out, which calling function would be
responsible for freeing the allocated memory. First we thought it
could be xlog, but more tests showed, the error also occurs when
called from other modules.
pv_parse_ht_name() is set as the innername in the htable module:
static pv_export_t mod_pvs[] = {
{ {"sht", sizeof("sht")-1}, PVT_OTHER, pv_get_ht_cell,
pv_set_ht_cell,
pv_parse_ht_name, 0, 0, 0 },
pv_parse_ht_name() allocates some memory (line 121, ht_var.c), saves
it in sp->pvp.pvn.u.dname (line 165) and sets the type to
PV_NAME_PVAR. As "u" is a union we thought freeing the memory should
depend on the type. We can not find any place, where type is checked
for PV_NAME_PVAR and any memory is freed.
Could you give us any hints where this should happen? Maybe this is
the only thing we have to add and the memory leak is gone.
Do you call
pv_parse_ht_name() from perl? If not, that function is
executed only at startup and it is not a cause for leaking. A clean
shutdown should free it, but there is not complete support for it (some
fixup functions have a free function, others done, as well as the script
actions tree).
That structure is not freed because it used at runtime. During startup,
the name of $sht is parsed and kept as custom structure to be used
latter, for speed reasons.
I tried to access the files you uploaded, with the log, but they
expired. If you have the one with memory debug, please send it to me.
By the way, when searching for PV_NAME_INTSTR we found a potential
bug: In pvapi.c there is the follwoing piece of code (line 773 to 785):
if(ip->pvn.type==PV_NAME_INTSTR)
{
[...]
} else if(ip->pvn.type==PV_NAME_INTSTR) {
/* pvar */
The second if also checks for PV_NAME_INTSTR so it will never be
executed. Shouldn't this be PV_NAME_PVAR?
Yes, should have been so and it is
now fixed. The issue was pretty
harmless, as that function was called mainly for avps and hdrs PV with
int or str name.
Thanks,
Daniel
Regards,
Christian
Christian Koch schrieb:
--
Daniel-Constantin Mierla
http://www.asipto.com/
Hi Henning,
Henning Westerholt schrieb:
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Can you configure the kamailio server that it
generates a core file?
Then take a look to the backtrace where the invalid memory access
was done, to verify if its really crashed in the core function, or
perhaps some other parts has a problem here. Further informations:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt
#0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00b64825 in raise () from /lib/tls/libc.so.6
#2 0x00b66289 in abort () from /lib/tls/libc.so.6
#3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e
"proxy.c", func=0x811724e "hostent_cpy", line=187) at
mem/q_malloc.c:444
#4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at
proxy.c:187
#5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0,
is_sips=0) at proxy.c:305
#6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218,
uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at
ut.h:67
#7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218,
proxy=0x0) at t_fwd.c:630
#8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0,
flags=Variable "flags" is not available.
) at t_funcs.c:264
#9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0)
at tm.c:1002
#10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at action.c:874
#11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at
action.c:145
#12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at
route.c:1171
#13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at
route.c:1488
#14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at
route.c:1493
#15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at action.c:729
#16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at
action.c:145
#17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at action.c:120
#18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at
action.c:145
#19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at action.c:746
#20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at
action.c:145
#21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at action.c:752
#22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at
action.c:145
#23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at
action.c:120
#24 0x08087994 in receive_msg (
buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187
SIP/2.0\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc>\r\nRecord-Route:
<sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175
#25 0x080be606 in udp_rcv_loop () at udp_server.c:449
#26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg
memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1));
if (dst->h_addr_list==0){
ser_error=ret=E_OUT_OF_MEM;
pkg_free(dst->h_name);
for(r=0; dst->h_aliases[r]; r++)
pkg_free(dst->h_aliases[r]);
pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187,
WHERE THE CORE FILE IS GENERATED
pkg_free(dst->h_aliases);
goto error;
}
So, there may be an error in proxy.c, but perhaps the reason for our
problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where
the variable hpv is not freed?
Any comments on pv_parse_ht_name() are greatly appreciated. We could
try to fix this, but we're not sure about the sideeffects.
Regards,
Christian
12:14 p.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello Daniel,
Daniel-Constantin Mierla schrieb:
Do you call pv_parse_ht_name() from perl? If not, that
function is
executed only at startup and it is not a cause for leaking. A clean
shutdown should free it, but there is not complete support for it
(some fixup functions have a free function, others done, as well as
the script actions tree).
We're not calling this directly from perl, but it
seems it is called
automatically every time when accessing the pv $sht. We've modified
ht_var.c from module htable for getting a backtrace every time memory is
allocated in pv_parse_ht_name:
void printBacktrace()
{
void *array[100];
size_t size;
char **strings;
size_t i;
size = backtrace (array, 100);
strings = backtrace_symbols (array, size);
LM_DBG("Obtained %zd stack frames.\n", size);
for (i = 0; i < size; i++)
LM_DBG("%s\n", strings[i]);
free (strings);
}
int pv_parse_ht_name(pv_spec_p sp, str *in)
{
ht_pv_t *hpv=NULL;
char *p;
str pvs;
if(in->s==NULL || in->len<=0)
return -1;
hpv = (ht_pv_t*)pkg_malloc(sizeof(ht_pv_t));
if(hpv==NULL)
return -1;
LM_DBG("Allocated new memory %u/0x%x in pv_parse_ht_name()",
hpv, hpv);
printBacktrace();
[...]
When making a call we get the following outputs:
xlog:
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:pv_parse_ht_name: Allocated new memory 136020048 in
pv_parse_ht_name()
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: Obtained 10 stack frames.
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/htable.so(printBacktrace+0x2d)
[0x1dd1c5]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/htable.so(pv_parse_ht_name+0x8f)
[0x1dd8fe]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(pv_parse_spec+0x58f) [0x8082c35]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(pv_parse_format+0x1eb) [0x808409e]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/xlog.so
[0x1482c6]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/xlog.so
[0x148620]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio [0x808e68e]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio(fix_rls+0x12b)
[0x8096892]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio(main+0xcd)
[0x8068215]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/lib/tls/libc.so.6(__libc_start_main+0xd3) [0xb51df3]
and perl:
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:pv_parse_ht_name: Allocated new memory
136017028/0x81b7484 in pv_parse_ht_name()
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace: Obtained 27 stack frames.
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/htable.so(printBacktrace+0x33)
[0xf631cb]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/htable.so(pv_parse_ht_name+0x8f)
[0xf63a42]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(pv_parse_spec+0x58f) [0x8082c35]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(pv_parse_format+0x1eb) [0x808409e]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/perl.so(pv_sprintf+0x93) [0x2a40c8]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/perl.so(XS_OpenSER__Message_pseudoVar+0x17f)
[0x2a92f3]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_pp_entersub+0x2ea)
[0x813c
b2]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0xba)
[0x7f71
6d]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so [0x7a1b26]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x6d7)
[0x7a7ef1]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_call_pv+0x40)
[0x7a81f9]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/perl.so(perl_exec2+0x198) [0x2af1c4]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/perl.so(perl_exec1+0x2c) [0x2af69d]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(do_action+0x1401) [0x80531ed]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(do_action+0x20ed) [0x8053ed9]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(do_action+0x3199) [0x8054f85]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(do_action+0x3199) [0x8054f85]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_top_route+0x6d) [0x805570e]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(receive_msg+0x572) [0x8087934]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(udp_rcv_loop+0xd26) [0x80be5a6]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio(main+0x3219)
[0x806b361]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/lib/tls/libc.so.6(__libc_start_main+0xd3) [0xb51df3]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio [0x8051991]
So because of this we don't think it is just called once but the
allocated memory should be freed after processing a message.
That structure is not freed because it used at runtime. During
startup, the name of $sht is parsed and kept as custom structure to be
used latter, for speed reasons.
I tried to access the files you uploaded, with the log, but they
expired. If you have the one with memory debug, please send it to me.
We still have
it, but its 43MB (already zipped) and it's without the
above "printBacktrace". If you need it anyway or need a complete logfile
including the backtrace we could send it to you.
Thanks for your help,
Christian
1:03 p.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello Christian,
On 06/09/2009 01:14 PM, Christian Koch wrote:
Hello Daniel,
Daniel-Constantin Mierla schrieb:
unfortunately you cannot use
$sht in this way from perl module. The same
limitation that prevent calling any function from modules apply here --
the fixup system used at startup, when string parameters are
pre-compiled in different forms to speed up runtime execution.
It can be added as a feature request for future. For module functions I
added the core support for free fixup functions, but nobody committed to
continue the work in perl module.
If you want to work on it, I can assist and guide you.
Cheers,
Daniel
Do you call pv_parse_ht_name() from perl? If not,
that function is
executed only at startup and it is not a cause for leaking. A clean
shutdown should free it, but there is not complete support for it
(some fixup functions have a free function, others done, as well as
the script actions tree).
We're not calling this directly from perl, but it
seems it is called
automatically every time when accessing the pv $sht. We've modified ht_var.c from module htable for
getting a backtrace
every time memory is allocated in pv_parse_ht_name:
void printBacktrace()
{
void *array[100];
size_t size;
char **strings;
size_t i;
size = backtrace (array, 100);
strings = backtrace_symbols (array, size);
LM_DBG("Obtained %zd stack frames.\n", size);
for (i = 0; i < size; i++)
LM_DBG("%s\n", strings[i]);
free (strings);
}
int pv_parse_ht_name(pv_spec_p sp, str *in)
{
ht_pv_t *hpv=NULL;
char *p;
str pvs;
if(in->s==NULL || in->len<=0)
return -1;
hpv = (ht_pv_t*)pkg_malloc(sizeof(ht_pv_t));
if(hpv==NULL)
return -1;
LM_DBG("Allocated new memory %u/0x%x in
pv_parse_ht_name()", hpv, hpv);
printBacktrace();
[...]
When making a call we get the following outputs:
xlog:
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:pv_parse_ht_name: Allocated new memory 136020048 in
pv_parse_ht_name()
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: Obtained 10 stack frames.
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/htable.so(printBacktrace+0x2d)
[0x1dd1c5]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/htable.so(pv_parse_ht_name+0x8f)
[0x1dd8fe]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(pv_parse_spec+0x58f) [0x8082c35]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(pv_parse_format+0x1eb) [0x808409e]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/xlog.so
[0x1482c6]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/xlog.so
[0x148620]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio [0x808e68e]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio(fix_rls+0x12b)
[0x8096892]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio(main+0xcd)
[0x8068215]
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]:
DBG:htable:printBacktrace:
/lib/tls/libc.so.6(__libc_start_main+0xd3) [0xb51df3]
and perl:
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:pv_parse_ht_name: Allocated new memory
136017028/0x81b7484 in pv_parse_ht_name()
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace: Obtained 27 stack frames.
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/htable.so(printBacktrace+0x33)
[0xf631cb]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/htable.so(pv_parse_ht_name+0x8f)
[0xf63a42]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(pv_parse_spec+0x58f) [0x8082c35]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(pv_parse_format+0x1eb) [0x808409e]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/perl.so(pv_sprintf+0x93) [0x2a40c8]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/perl.so(XS_OpenSER__Message_pseudoVar+0x17f)
[0x2a92f3]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_pp_entersub+0x2ea)
[0x813c
b2]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0xba)
[0x7f71
6d]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so
[0x7a1b26]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x6d7)
[0x7a7ef1]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_call_pv+0x40)
[0x7a81f9]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/perl.so(perl_exec2+0x198) [0x2af1c4]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/lib/kamailio/modules/perl.so(perl_exec1+0x2c) [0x2af69d]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(do_action+0x1401) [0x80531ed]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(do_action+0x20ed) [0x8053ed9]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(do_action+0x3199) [0x8054f85]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(do_action+0x3199) [0x8054f85]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(run_top_route+0x6d) [0x805570e]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(receive_msg+0x572) [0x8087934]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/usr/local/sbin/kamailio(udp_rcv_loop+0xd26) [0x80be5a6]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio(main+0x3219)
[0x806b361]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace:
/lib/tls/libc.so.6(__libc_start_main+0xd3) [0xb51df3]
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]:
DBG:htable:printBacktrace: /usr/local/sbin/kamailio [0x8051991]
So because of this we don't think it is just called once but the
allocated memory should be freed after processing a message.
--
Daniel-Constantin Mierla
http://www.asipto.com/
That structure is not freed because it used at runtime. During
startup, the name of $sht is parsed and kept as custom structure to
be used latter, for speed reasons.
I tried to access the files you uploaded, with the log, but they
expired. If you have the one with memory debug, please send it to me.
We still
have it, but its 43MB (already zipped) and it's without the
above "printBacktrace". If you need it anyway or need a complete
logfile including the backtrace we could send it to you.
Thanks for your help,
Christian
------------------------------------------------------------------------
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
3:56 p.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello Daniel,
Daniel-Constantin Mierla schrieb:
unfortunately you cannot use $sht in this way from
perl module. The
same limitation that prevent calling any function from modules apply
here -- the fixup system used at startup, when string parameters are
pre-compiled in different forms to speed up runtime execution.
So there is no way
to access $sht from other moduls (like perl or xlog)?
It seems to work, but obviously the memory isn't freed :-)
What we are trying to do is the following:
During processing of "INVITE" we need to save some call details (source
IP, URI, ...) and want to write them to file when the call is finished
(including duration, ...), while processing BYE or CANCEL, to have CDRs.
As there are multiple childs of kamailio we tried to save the calls
details from the INVITE in shared memory using htable (e.g.
"$sht(a=>$ci::srcip) = $si"), so we can use them while processing the
BYE (possibly in another child). There we wanted to use perl to be able
to write the saved details to a file.
We used the shared memory module as we don´t want to save the infos to
DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way. Do
you have any idea what else we can do to generate CDRs as mentioned?
By using other modules for example or a complete other way you may know?
It can be added as a feature request for future. For module functions
I added the core support for free fixup functions, but nobody
committed to continue the work in perl module.
If you want to work on it, I can assist and guide you.
Well, honestly we are
looking for a fast way to implement it.
How long will it take to implement it with your guidance?
Thanks and regards,
Christian
6 p.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Christian,
Not sure if this will help you, but it's a good starting point if you want
to use Perl. I grab a good deal of information using the headers and AVP's
in order to write data to a MSSQL database and do LCR. Here's essentially
how I grab the values:
#Begin
use strict;
use OpenSER qw ( log );
use OpenSER::Constants;
use OpenSER::Message;
Sub your_function {
# get headers
my $m = shift;
my ($lrn, $inrpid);
# Headers
my $ruri = $m->getHeader("To");
# Pseudo Variables
my $source_ip = $m->pseudoVar("\$si");
my $in_ANI = $m->pseudoVar("\$fU");
my $insrc = $m->pseudoVar("\$fd");
my $daytim = $m->pseudoVar("\$Tf");
my $callid = $m->pseudoVar("\$ci");
# Check if PV's are set
if( $m->pseudoVar("\$re") != undef ){
$inrpid = $m->pseudoVar("\$re");
}
# AVP's
if( OpenSER::AVP::get(60) != undef ){
$lrn = OpenSER::AVP::get(60);
}
# Write some data to the log
log(L_INFO, sprintf("callid: %s\n", $callid));
# DO WHATEVER ELSE
}
# END
You can set AVP's in your script before calling the function:
$avp(my_AVP) = $sht(a=>$tu-translate);
Mik
-----Original Message-----
From: users-bounces(a)lists.kamailio.org
[mailto:users-bounces@lists.kamailio.org] On Behalf Of Christian Koch
Sent: Tuesday, June 09, 2009 6:56 AM
To: Daniel-Constantin Mierla
Cc: users(a)lists.kamailio.org
Subject: Re: [Kamailio-Users] freeing innername from htable Re: perl module,
"pv_sprintf: Memory exhausted!"
Hello Daniel,
Daniel-Constantin Mierla schrieb:
unfortunately you cannot use $sht in this way from
perl module. The
same limitation that prevent calling any function from modules apply
here -- the fixup system used at startup, when string parameters are
pre-compiled in different forms to speed up runtime execution.
So there is no way
to access $sht from other moduls (like perl or xlog)?
It seems to work, but obviously the memory isn't freed :-)
What we are trying to do is the following:
During processing of "INVITE" we need to save some call details (source
IP, URI, ...) and want to write them to file when the call is finished
(including duration, ...), while processing BYE or CANCEL, to have CDRs.
As there are multiple childs of kamailio we tried to save the calls
details from the INVITE in shared memory using htable (e.g.
"$sht(a=>$ci::srcip) = $si"), so we can use them while processing the
BYE (possibly in another child). There we wanted to use perl to be able
to write the saved details to a file.
We used the shared memory module as we don´t want to save the infos to
DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way. Do
you have any idea what else we can do to generate CDRs as mentioned?
By using other modules for example or a complete other way you may know?
It can be added as a feature request for future. For module functions
I added the core support for free fixup functions, but nobody
committed to continue the work in perl module.
If you want to work on it, I can assist and guide you.
Well, honestly we are
looking for a fast way to implement it.
How long will it take to implement it with your guidance?
Thanks and regards,
Christian
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
10 Jun
10 Jun
9:09 a.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hi Mik,
thanks for your hint!!!
We're now trying to save the variables from $sht in AVPs and accessing
only the AVPs from perl to avoid the memory leak. Currently we're
running our stress test. We'll inform you about the result.
Christian
Mik Cheez schrieb:
Christian,
Not sure if this will help you, but it's a good starting point if you want
to use Perl. I grab a good deal of information using the headers and AVP's
in order to write data to a MSSQL database and do LCR. Here's essentially
how I grab the values:
#Begin
use strict;
use OpenSER qw ( log );
use OpenSER::Constants;
use OpenSER::Message;
Sub your_function {
# get headers
my $m = shift;
my ($lrn, $inrpid);
# Headers
my $ruri = $m->getHeader("To");
# Pseudo Variables
my $source_ip = $m->pseudoVar("\$si");
my $in_ANI = $m->pseudoVar("\$fU");
my $insrc = $m->pseudoVar("\$fd");
my $daytim = $m->pseudoVar("\$Tf");
my $callid = $m->pseudoVar("\$ci");
# Check if PV's are set
if( $m->pseudoVar("\$re") != undef ){
$inrpid = $m->pseudoVar("\$re");
}
# AVP's
if( OpenSER::AVP::get(60) != undef ){
$lrn = OpenSER::AVP::get(60);
}
# Write some data to the log
log(L_INFO, sprintf("callid: %s\n", $callid));
# DO WHATEVER ELSE
}
# END
You can set AVP's in your script before calling the function:
$avp(my_AVP) = $sht(a=>$tu-translate);
Mik
-----Original Message-----
From: users-bounces(a)lists.kamailio.org
[mailto:users-bounces@lists.kamailio.org] On Behalf Of Christian Koch
Sent: Tuesday, June 09, 2009 6:56 AM
To: Daniel-Constantin Mierla
Cc: users(a)lists.kamailio.org
Subject: Re: [Kamailio-Users] freeing innername from htable Re: perl module,
"pv_sprintf: Memory exhausted!"
Hello Daniel,
Daniel-Constantin Mierla schrieb:
unfortunately you cannot use $sht in this way
from perl module. The
same limitation that prevent calling any function from modules apply
here -- the fixup system used at startup, when string parameters are
pre-compiled in different forms to speed up runtime execution.
So there is no way to access $sht from other moduls (like perl or xlog)?
It seems to work, but obviously the memory isn't freed :-)
What we are trying to do is the following:
During processing of "INVITE" we need to save some call details (source
IP, URI, ...) and want to write them to file when the call is finished
(including duration, ...), while processing BYE or CANCEL, to have CDRs.
As there are multiple childs of kamailio we tried to save the calls
details from the INVITE in shared memory using htable (e.g.
"$sht(a=>$ci::srcip) = $si"), so we can use them while processing the
BYE (possibly in another child). There we wanted to use perl to be able
to write the saved details to a file.
We used the shared memory module as we don´t want to save the infos to
DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way. Do
you have any idea what else we can do to generate CDRs as mentioned?
By using other modules for example or a complete other way you may know?
It can be added as a feature request for future.
For module functions
I added the core support for free fixup functions, but nobody
committed to continue the work in perl module.
If you want to work on it, I can assist and guide you.
Well, honestly we are looking for a fast way to implement it.
How long will it take to implement it with your guidance?
Thanks and regards,
Christian
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
9:22 a.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello Christian,
On 06/09/2009 04:56 PM, Christian Koch wrote:
--
Daniel-Constantin Mierla
http://www.asipto.com/
Hello Daniel,
Daniel-Constantin Mierla schrieb:
you could
access it by adding some sht extensions to the Perl API, same
way as for AVP, but this requires development as well.
From other modules is easier, as one can evaluate the name and then do
search in the hashtable.
unfortunately you cannot use $sht in this way
from perl module. The
same limitation that prevent calling any function from modules apply
here -- the fixup system used at startup, when string parameters are
pre-compiled in different forms to speed up runtime execution.
So there is no way
to access $sht from other moduls (like perl or
xlog)? It seems to work, but obviously the memory isn't freed :-)
What we are trying to do is the following:
During processing of "INVITE" we need to save some call details
(source IP, URI, ...) and want to write them to file when the call is
finished (including duration, ...), while processing BYE or CANCEL, to
have CDRs. As there are multiple childs of kamailio we tried to save
the calls details from the INVITE in shared memory using htable (e.g.
"$sht(a=>$ci::srcip) = $si"), so we can use them while processing the
BYE (possibly in another child). There we wanted to use perl to be
able to write the saved details to a file.
We used the shared memory module as we don´t want to save the infos to
DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way. Do
you have any idea what else we can do to generate CDRs as mentioned?
By using other modules for example or a complete other way you may know?
You can use shared memory from perl, see persistence.pl in
modules/perl/doc/samples/
Cheers,
Daniel
It can be added as a feature request for future. For module functions
I added the core support for free fixup functions, but nobody
committed to continue the work in perl module.
If you want to work on it, I can assist and guide you.
Well, honestly we are
looking for a fast way to implement it.
How long will it take to implement it with your guidance?
Thanks and regards,
Christian
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
11 Jun
11 Jun
9:28 a.m.
New subject: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello all,
Daniel-Constantin Mierla schrieb:
Well, we've already tried that. But it occurred several times, that
after about two weeks the shared memory got corrupted and we had to
reboot the machine. It seems "IPC::Shareable" is not ready for
production use. We also tried "IPC::ShareLite" instead of
"IPC::Shareable". "IPC::ShareLite" seems to be newer, better
maintained
and supported. But we had no success fetching saved values from shared
memory - probably we didn't use it the right way.
Regards,
Christian
Hello Christian,
On 06/09/2009 04:56 PM, Christian Koch wrote:
We tried the solution suggested by Mik (copy $sht-values
to AVP in
config file and accessing only AVPs from perl) and made about 20.000
calls without any problems. Seems that solved our problem. Thanks a lot!!!
Hello Daniel,
Daniel-Constantin Mierla schrieb:
you could
access it by adding some sht extensions to the Perl API,
same way as for AVP, but this requires development as well.
From other modules is easier, as one can evaluate the name and then do
search in the hashtable. unfortunately you cannot use $sht in this way
from perl module. The
same limitation that prevent calling any function from modules apply
here -- the fixup system used at startup, when string parameters are
pre-compiled in different forms to speed up runtime execution.
So there is no way
to access $sht from other moduls (like perl or
xlog)? It seems to work, but obviously the memory isn't freed :-)
What we are trying to do is the following:
During processing of "INVITE" we need to save some call details
(source IP, URI, ...) and want to write them to file when the call is
finished (including duration, ...), while processing BYE or CANCEL,
to have CDRs. As there are multiple childs of kamailio we tried to
save the calls details from the INVITE in shared memory using htable
(e.g. "$sht(a=>$ci::srcip) = $si"), so we can use them while
processing the BYE (possibly in another child). There we wanted to
use perl to be able to write the saved details to a file.
We used the shared memory module as we don´t want to save the infos
to DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way.
Do you have any idea what else we can do to generate CDRs as mentioned?
By using other modules for example or a complete other way you may know?
You can use shared memory from perl, see persistence.pl in
modules/perl/doc/samples/
5441
days inactive
5465
days old
28 comments
4 participants
participants (4)
-
Christian Koch -
Daniel-Constantin Mierla -
Henning Westerholt -
Mik Cheez