Hi all,
we're using kamailio 1.5.1 with modules htable and perl to generate CDRs. Using htable we save some variables which are dumped to file using the perl module at the end of the call. We did a small stress test with 5 calls in parallel (each call about 20 seconds) and we got after about 6 hours the following error:
ERROR:core:pv_sprintf: pv_sprintf: Memory exhausted!
No CDRs were written anymore, the perl method was not called. Everything else seems to run fine. After stopping the stress test doing one simple call with a softphone worked and a CDR had been written (without restarting kamailio!). Perhaps the perl script is buggy?
Any ideas how to fix this are greatly appreciated. We don't think extending the PKG_MEM_POOL_SIZE will solve the problem. This is currently 4MB and should be enough for 5 concurrent calls!?!
Thanks, Christian Koch
The perl script:
use OpenSER; use OpenSER::Constants; use IO::Socket;
sub write_cdr { my $m = shift;
#my $v = $m->getVersion(); #my $t = $m->getType();
my $index = $m->pseudoVar("$ci");
my $endTimeStamp = $m->pseudoVar("$Ts"); my $startTimeStamp = $endTimeStamp - $m->pseudoVar("$DLG_lifetime");
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($startTimeStamp); my $startTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday, $mon+1, $year+1900, $hour, $min, $sec); my $fileDate = sprintf("%02d%02d%02d", $mday, $mon+1, $year%100);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($endTimeStamp); my $endTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday, $mon+1, $year+1900, $hour, $min, $sec);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($m->pseudoVar("$sht(a=>".$index."::invtrytime)")); my $InvTryingTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday, $mon+1, $year+1900, $hour, $min, $sec);
$ReplyReason=$m->pseudoVar("$rr"); $ReplyReason=~ s/;/_/g;
$FromDisplayName = $m->pseudoVar("$sht(a=>".$index."::fromdplname)"); $FromDisplayName=~ s/;/_/g;
$ToUri = $m->pseudoVar("$sht(a=>".$index."::touri)"); $ToUri=~s/;/_/g;
$FromUri = $m->pseudoVar("$sht(a=>".$index."::fromuri)"); $FromUri=~s/;/_/g;
$UserAgent = $m->pseudoVar("$sht(a=>".$index."::useragent)"); $UserAgent=~s/;/_/g;
my $CDR = $startTime.";"; # 1 + 2 $CDR .= $endTime.";"; # 3 + 4 $CDR .= $m->pseudoVar("$DLG_lifetime").";"; # 5 $CDR .= $index.";"; #6 CallID $CDR .= $m->pseudoVar("$sht(a=>".$index."::srcip)").";"; # source-IP of INVITE $CDR .= $m->pseudoVar("$si").";"; # source-IP of BYE/CANCEL $CDR .= $FromDisplayName.";"; # from display-name INVITE $CDR .= $InvTryingTime.";"; # Alerting time $CDR .= ($startTimeStamp - $m->pseudoVar("$sht(a=>".$index."::invtrytime)")).";"; #Alerting period $CDR .= $m->pseudoVar("$DLG_count").";"; # current amount of calls $CDR .= $ToUri.";"; # ToUri from Invite $CDR .= $FromUri.";"; # FromUri from Invite $CDR .= $UserAgent.";"; # UserAgent from Invite $CDR .= $m->pseudoVar("$rm").";"; # Method (INVITE,BYE,CANCEL) $CDR .= $m->pseudoVar("$rs").";"; # SIP reply status $CDR .= $ReplyReason.";"; #SIP Reply reason
open(MYOUTFILE, ">>/home/sys/log/kamailio".$fileDate.".cdr.".$m->pseudoVar("$pp")); print MYOUTFILE $CDR."\n"; close(MYOUTFILE);
#delete(%sourceIp->{$index}); #delete(%fromDisplayName->{$index}); #delete(%InviteTryingTimestamp->{$index}); #delete(%ToUri->{$index}); #delete(%FromUri->{$index}); #delete(%UserAgent->{$index});
return 1; }
1;
Hello,
On 05/18/2009 04:24 PM, Christian Koch wrote:
Hi all,
we're using kamailio 1.5.1 with modules htable and perl to generate CDRs. Using htable we save some variables which are dumped to file using the perl module at the end of the call. We did a small stress test with 5 calls in parallel (each call about 20 seconds) and we got after about 6 hours the following error:
ERROR:core:pv_sprintf: pv_sprintf: Memory exhausted!
this is system memory error. Has nothing to do with pkg memory used by Kamailio. Seems that perl module allocates some memory and forgets to free it. Do you see other errors in your syslog?
One memleak I could spot is that the allocated buffer to print the output in pv_sprintf is not freed if the formatted string is bad. Then I see that the output buffer is duplicated with strdup and it is freed. So seems there are some system memory leaks in perl module, indeed.
No CDRs were written anymore, the perl method was not called. Everything else seems to run fine.
Yes, because the problem was with system memory, while kamailio uses internally pkg.
After stopping the stress test doing one simple call with a softphone worked and a CDR had been written (without restarting kamailio!). Perhaps the perl script is buggy?
I have no much experience with perl to state that.
Any ideas how to fix this are greatly appreciated. We don't think extending the PKG_MEM_POOL_SIZE will solve the problem. This is currently 4MB and should be enough for 5 concurrent calls!?!
pkg is more than enough.
Cheers, Daniel
Thanks, Christian Koch
The perl script:
use OpenSER; use OpenSER::Constants; use IO::Socket;
sub write_cdr { my $m = shift;
#my $v = $m->getVersion(); #my $t = $m->getType(); my $index = $m->pseudoVar("\$ci"); my $endTimeStamp = $m->pseudoVar("\$Ts"); my $startTimeStamp = $endTimeStamp -$m->pseudoVar("$DLG_lifetime");
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($startTimeStamp); my $startTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday, $mon+1, $year+1900, $hour, $min, $sec); my $fileDate = sprintf("%02d%02d%02d", $mday, $mon+1, $year%100);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($endTimeStamp); my $endTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday, $mon+1, $year+1900, $hour, $min, $sec);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime($m->pseudoVar("$sht(a=>".$index."::invtrytime)")); my $InvTryingTime = sprintf("%02d.%02d.%4d;%02d:%02d:%02d", $mday, $mon+1, $year+1900, $hour, $min, $sec);
$ReplyReason=$m->pseudoVar("\$rr"); $ReplyReason=~ s/;/_/g; $FromDisplayName =$m->pseudoVar("$sht(a=>".$index."::fromdplname)"); $FromDisplayName=~ s/;/_/g;
$ToUri = $m->pseudoVar("\$sht(a=>".$index."::touri)"); $ToUri=~s/;/_/g; $FromUri = $m->pseudoVar("\$sht(a=>".$index."::fromuri)"); $FromUri=~s/;/_/g; $UserAgent = $m->pseudoVar("\$sht(a=>".$index."::useragent)"); $UserAgent=~s/;/_/g; my $CDR = $startTime.";"; # 1 + 2 $CDR .= $endTime.";"; # 3 + 4 $CDR .= $m->pseudoVar("\$DLG_lifetime").";"; # 5 $CDR .= $index.";"; #6 CallID $CDR .= $m->pseudoVar("\$sht(a=>".$index."::srcip)").";"; #source-IP of INVITE $CDR .= $m->pseudoVar("$si").";"; # source-IP of BYE/CANCEL $CDR .= $FromDisplayName.";"; # from display-name INVITE $CDR .= $InvTryingTime.";"; # Alerting time $CDR .= ($startTimeStamp - $m->pseudoVar("$sht(a=>".$index."::invtrytime)")).";"; #Alerting period $CDR .= $m->pseudoVar("$DLG_count").";"; # current amount of calls $CDR .= $ToUri.";"; # ToUri from Invite $CDR .= $FromUri.";"; # FromUri from Invite $CDR .= $UserAgent.";"; # UserAgent from Invite $CDR .= $m->pseudoVar("$rm").";"; # Method (INVITE,BYE,CANCEL) $CDR .= $m->pseudoVar("$rs").";"; # SIP reply status $CDR .= $ReplyReason.";"; #SIP Reply reason
open(MYOUTFILE,">>/home/sys/log/kamailio".$fileDate.".cdr.".$m->pseudoVar("$pp")); print MYOUTFILE $CDR."\n"; close(MYOUTFILE);
#delete(%sourceIp->{$index}); #delete(%fromDisplayName->{$index}); #delete(%InviteTryingTimestamp->{$index}); #delete(%ToUri->{$index}); #delete(%FromUri->{$index}); #delete(%UserAgent->{$index}); return 1;}
1;
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello Daniel,
Daniel-Constantin Mierla schrieb:
we're using kamailio 1.5.1 with modules htable and perl to generate CDRs. Using htable we save some variables which are dumped to file using the perl module at the end of the call. We did a small stress test with 5 calls in parallel (each call about 20 seconds) and we got after about 6 hours the following error:
ERROR:core:pv_sprintf: pv_sprintf: Memory exhausted!this is system memory error. Has nothing to do with pkg memory used by Kamailio. Seems that perl module allocates some memory and forgets to free it. Do you see other errors in your syslog?
One memleak I could spot is that the allocated buffer to print the output in pv_sprintf is not freed if the formatted string is bad. Then I see that the output buffer is duplicated with strdup and it is freed. So seems there are some system memory leaks in perl module, indeed.
Are you sure this is system memory? We changed PKG_MEM_POOL_SIZE from 4 to 8MB and a stress test didn't show any problems for more then 12 hours. Then we changed it to 0.5MB and the problem occured after about 3 hours again wiht the same error message. The error message in pv_sprintf() occurs when pkg_malloc() returns NULL, so this should be a PKG issue?!? Currently we're running a test with 0.5MB of pkg memory, but we've fixed the error in pv_sprintf where the format is wrong and the buffer is not freed. In the test this morning (with 0.5MB) we had the message "wrong format" only once, and in all the other tests the message didn't show up. So we don't think this fixes the problem. Do you think only one missing pkg_free() (like in the last test) can corrupt the memory so this error occurs?
Thanks for your help!
Christian
Hello,
On 05/19/2009 04:25 PM, Christian Koch wrote:
Hello Daniel,
Daniel-Constantin Mierla schrieb:
we're using kamailio 1.5.1 with modules htable and perl to generate CDRs. Using htable we save some variables which are dumped to file using the perl module at the end of the call. We did a small stress test with 5 calls in parallel (each call about 20 seconds) and we got after about 6 hours the following error:
ERROR:core:pv_sprintf: pv_sprintf: Memory exhausted!this is system memory error. Has nothing to do with pkg memory used by Kamailio. Seems that perl module allocates some memory and forgets to free it. Do you see other errors in your syslog?
One memleak I could spot is that the allocated buffer to print the output in pv_sprintf is not freed if the formatted string is bad. Then I see that the output buffer is duplicated with strdup and it is freed. So seems there are some system memory leaks in perl module, indeed.
Are you sure this is system memory?
you are right, somehow I was blind when I checked the pv_sprintf, it is clear pkg_malloc().
We changed PKG_MEM_POOL_SIZE from 4 to 8MB and a stress test didn't show any problems for more then 12 hours. Then we changed it to 0.5MB and the problem occured after about 3 hours again wiht the same error message. The error message in pv_sprintf() occurs when pkg_malloc() returns NULL, so this should be a PKG issue?!? Currently we're running a test with 0.5MB of pkg memory, but we've fixed the error in pv_sprintf where the format is wrong and the buffer is not freed. In the test this morning (with 0.5MB) we had the message "wrong format" only once, and in all the other tests the message didn't show up. So we don't think this fixes the problem. Do you think only one missing pkg_free() (like in the last test) can corrupt the memory so this error occurs?
Now that the issue is pkg, you can compile and run in mem debug mode so we can detect easier where the leak is, check this: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
Cheers, Daniel
Hi Daniel,
Daniel-Constantin Mierla schrieb:
We changed PKG_MEM_POOL_SIZE from 4 to 8MB and a stress test didn't show any problems for more then 12 hours. Then we changed it to 0.5MB and the problem occured after about 3 hours again wiht the same error message. The error message in pv_sprintf() occurs when pkg_malloc() returns NULL, so this should be a PKG issue?!? Currently we're running a test with 0.5MB of pkg memory, but we've fixed the error in pv_sprintf where the format is wrong and the buffer is not freed. In the test this morning (with 0.5MB) we had the message "wrong format" only once, and in all the other tests the message didn't show up. So we don't think this fixes the problem. Do you think only one missing pkg_free() (like in the last test) can corrupt the memory so this error occurs?
Now that the issue is pkg, you can compile and run in mem debug mode so we can detect easier where the leak is, check this: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
We've compiled kamailio with mem debug, but now kamailio is very, very slow, so that it's really not possible to make a stress test. We were able to place one single call (The INVITE took aboute 1:30 min until it reached the destination). The memory status at shutdown is available at: http://rcpt.yousendit.com/690135386/7f3f82ffe0daf468b783ecf6e9e61bc8 Hopefully this will be helpful.
Thanks for your help, Christian
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
[..] Now that the issue is pkg, you can compile and run in mem debug mode so we can detect easier where the leak is, check this: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
We've compiled kamailio with mem debug, but now kamailio is very, very slow, so that it's really not possible to make a stress test. We were able to place one single call (The INVITE took aboute 1:30 min until it reached the destination).
Hi Christian,
you say that processing one INVITE needs 1:30 min? This is way to slow even with memory debugging. I regularly use this for debugging, normally you should not notice a big difference in operation. Do you get any "unusual" messages during the processing of the INVITE with mem debug?
Henning
Hello,
On 05/20/2009 12:25 PM, Henning Westerholt wrote:
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
[..] Now that the issue is pkg, you can compile and run in mem debug mode so we can detect easier where the leak is, check this: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
We've compiled kamailio with mem debug, but now kamailio is very, very slow, so that it's really not possible to make a stress test. We were able to place one single call (The INVITE took aboute 1:30 min until it reached the destination).
Hi Christian,
you say that processing one INVITE needs 1:30 min? This is way to slow even with memory debugging. I regularly use this for debugging, normally you should not notice a big difference in operation. Do you get any "unusual" messages during the processing of the INVITE with mem debug?
is the syslog configured in asynchronous mode?
Cheers, Daniel
Hi Henning,
Henning Westerholt schrieb:
you say that processing one INVITE needs 1:30 min? This is way to slow even with memory debugging. I regularly use this for debugging, normally you should not notice a big difference in operation. Do you get any "unusual" messages during the processing of the INVITE with mem debug?
I can't see any unusal messages or errors (beside the fact, that I'm not sure what is unusal ;-) ), but even startup is very slow. It needs several minutes before we even see all kamailio processes in ps aux. Here is one fragment of the startup logfile, where you can see, that within one second not really much happens:
May 20 09:42:20 AmbriaSip1 kamailio: params (0x81677e0, 96), called from route_struct.c: mk_action(106) May 20 09:42:20 AmbriaSip1 kamailio: params (0x81677e0, 96), returns address 0x81b3d9c frag. 0x81b3d84 (size=96) on 1 -th hit May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 96), called from route_struct.c: mk_action(106) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 96), returns address 0x81b3e2c frag. 0x81b3e14 (size=96) on 1 -th hit May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 96), called from route_struct.c: mk_action(106) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 96), returns address 0x81b3ebc frag. 0x81b3ea4 (size=96) on 1 -th hit May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 3200), called from socket_info.c: add_interfaces(397) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 3200), returns address 0x81b3f4c frag. 0x81b3f34 (size=3200) on 1 -th hit May 20 09:42:21 AmbriaSip1 kamailio: params(0x81677e0, 0x81b3f4c), called from socket_info.c: add_interfaces(411) May 20 09:42:21 AmbriaSip1 kamailio: freeing frag. 0x81b3f34 alloc'ed from socket_info.c: add_interfaces(397) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 6400), called from socket_info.c: add_interfaces(397) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 6400), returns address 0x81b4bfc frag. 0x81b4be4 (size=6400) on 1 -th hit May 20 09:42:21 AmbriaSip1 kamailio: params(0x81677e0, 0x81b4bfc), called from socket_info.c: add_interfaces(479) May 20 09:42:21 AmbriaSip1 kamailio: freeing frag. 0x81b4be4 alloc'ed from socket_info.c: add_interfaces(397) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 3200), called from socket_info.c: add_interfaces(397) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 3200), returns address 0x81b3f4c frag. 0x81b3f34 (size=3200) on 1 -th hit May 20 09:42:21 AmbriaSip1 kamailio: params(0x81677e0, 0x81b3f4c), called from socket_info.c: add_interfaces(411) May 20 09:42:21 AmbriaSip1 kamailio: freeing frag. 0x81b3f34 alloc'ed from socket_info.c: add_interfaces(397) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 6400), called from socket_info.c: add_interfaces(397) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 6400), returns address 0x81b4bfc frag. 0x81b4be4 (size=6400) on 1 -th hit May 20 09:42:21 AmbriaSip1 kamailio: params(0x81677e0, 0x81b4bfc), called from socket_info.c: add_interfaces(479) May 20 09:42:21 AmbriaSip1 kamailio: freeing frag. 0x81b4be4 alloc'ed from socket_info.c: add_interfaces(397) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 5), called from socket_info.c: fix_socket_list(532) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 8), returns address 0x81b3f4c frag. 0x81b3f34 (size=8) on 1 -th hit May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 15), called from socket_info.c: fix_socket_list(583) May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 16), returns address 0x81b3f84 frag. 0x81b3f6c (size=16) on 1 -th hit May 20 09:42:21 AmbriaSip1 kamailio: params (0x81677e0, 16), called from name_alias.h: add_alias(99) May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 16), returns address 0x81b3fc4 frag. 0x81b3fac (size=16) on 1 -th hit May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 11), called from name_alias.h: add_alias(101) May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 12), returns address 0x81b4004 frag. 0x81b3fec (size=12) on 1 -th hit
The logfile is full with this kind of messages. The load of the machine (Xeon dualcore @3GHz) goes up to 2, but using top we can't see that kamailio is using much CPU time (sometimes about 20%, but not more).
Regards, Christian
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
you say that processing one INVITE needs 1:30 min? This is way to slow even with memory debugging. I regularly use this for debugging, normally you should not notice a big difference in operation. Do you get any "unusual" messages during the processing of the INVITE with mem debug?
Hi Christian,
I can't see any unusal messages or errors (beside the fact, that I'm not sure what is unusal ;-) ), but even startup is very slow. It needs several minutes before we even see all kamailio processes in ps aux. Here is one fragment of the startup logfile, where you can see, that within one second not really much happens:
This long startup is also a bit unusual. Sure, it needs to allocates some memory, and this informations needs to be written to syslog, but several minutes is too slow. Do you read some database content in memory on startup, like a huge location table, or a big carrierroute route tree with several hundreds of MB?
May 20 09:42:20 AmbriaSip1 kamailio: params (0x81677e0, 96), called from route_struct.c: mk_action(106) [..] May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 12), returns address 0x81b4004 frag. 0x81b3fec (size=12) on 1 -th hit
The logfile is full with this kind of messages. The load of the machine (Xeon dualcore @3GHz) goes up to 2, but using top we can't see that kamailio is using much CPU time (sometimes about 20%, but not more).
So in which processes is the rest of the CPU time spend?
Cheers,
Henning
Hi Henning,
Henning Westerholt schrieb:
I can't see any unusal messages or errors (beside the fact, that I'm not sure what is unusal ;-) ), but even startup is very slow. It needs several minutes before we even see all kamailio processes in ps aux. Here is one fragment of the startup logfile, where you can see, that within one second not really much happens:
This long startup is also a bit unusual. Sure, it needs to allocates some memory, and this informations needs to be written to syslog, but several minutes is too slow. Do you read some database content in memory on startup, like a huge location table, or a big carrierroute route tree with several hundreds of MB?
We're connecting to a database, but it includes only a few (about 20) subscribers, nothing else. We didn't stress test the database (-connection). Without memdebug kamailio runs fast.
May 20 09:42:20 AmbriaSip1 kamailio: params (0x81677e0, 96), called from route_struct.c: mk_action(106) [..] May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 12), returns address 0x81b4004 frag. 0x81b3fec (size=12) on 1 -th hit
The logfile is full with this kind of messages. The load of the machine (Xeon dualcore @3GHz) goes up to 2, but using top we can't see that kamailio is using much CPU time (sometimes about 20%, but not more).
So in which processes is the rest of the CPU time spend?
Most of the time the CPU seems to be idle. Sometimes syslog shows up in top, but without much CPU time. Do you think this could be a dual-CPU issue? Perhaps syncing the memory access between the CPUs is slow? If you like to see the complete logfile we can post this too.
Regards, Christian
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
This long startup is also a bit unusual. Sure, it needs to allocates some memory, and this informations needs to be written to syslog, but several minutes is too slow. Do you read some database content in memory on startup, like a huge location table, or a big carrierroute route tree with several hundreds of MB?
We're connecting to a database, but it includes only a few (about 20) subscribers, nothing else. We didn't stress test the database (-connection). Without memdebug kamailio runs fast.
This is strange. Do you log to a remote host, for example with TCP? Or do you've another big IO load on the machine, that could block the writing of logs to the disk somehow?
May 20 09:42:20 AmbriaSip1 kamailio: params (0x81677e0, 96), called from route_struct.c: mk_action(106) [..] May 20 09:42:22 AmbriaSip1 kamailio: params (0x81677e0, 12), returns address 0x81b4004 frag. 0x81b3fec (size=12) on 1 -th hit
The logfile is full with this kind of messages. The load of the machine (Xeon dualcore @3GHz) goes up to 2, but using top we can't see that kamailio is using much CPU time (sometimes about 20%, but not more).
So in which processes is the rest of the CPU time spend?
Most of the time the CPU seems to be idle. Sometimes syslog shows up in top, but without much CPU time. Do you think this could be a dual-CPU issue? Perhaps syncing the memory access between the CPUs is slow? If you like to see the complete logfile we can post this too.
I'd guess most people run kamailio on a multi-CPU setup nowadays, so i don't think this is the reason. So what causes the load of 2 during startup, if not kamailio or syslog?
Henning
Henning Westerholt schrieb:
We're connecting to a database, but it includes only a few (about 20) subscribers, nothing else. We didn't stress test the database (-connection). Without memdebug kamailio runs fast.
This is strange. Do you log to a remote host, for example with TCP? Or do you've another big IO load on the machine, that could block the writing of logs to the disk somehow?
We solved the problem with the bad performance in mem debuge mode. Kamailio not only logs to /var/log/kamailio but also to /var/log/messages (?). /var/log/messages was not configured asynchrouns in syslog.conf. We fixed this and will retry the stress test with enabled mem debug.
Thanks a lot!
Christian
Hello Daniel, hello Henning,
Daniel-Constantin Mierla schrieb:
The error message in pv_sprintf() occurs when pkg_malloc() returns NULL, so this should be a PKG issue?!? Currently we're running a test with 0.5MB of pkg memory, but we've fixed the error in pv_sprintf where the format is wrong and the buffer is not freed. In the test this morning (with 0.5MB) we had the message "wrong format" only once, and in all the other tests the message didn't show up. So we don't think this fixes the problem. Do you think only one missing pkg_free() (like in the last test) can corrupt the memory so this error occurs?
We changed PKG_MEM_POOL_SIZE from 4 to 8MB and a stress test didn't show any problems for more then 12 hours. Then we changed it to 0.5MB and the problem occured after about 3 hours again wiht the same error message. Now that the issue is pkg, you can compile and run in mem debug mode so we can detect easier where the leak is, check this: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
After solving all the performance issues with syslog we now made a stress test with memdebug enabled. Now kamailio terminates itself (perhaps because memory is corrupted?).
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: CRITICAL:core:qm_free: bad pointer (nil) (out of memory block!) - aborting May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27697]: CRITICAL:core:receive_fd: EOF on 12 May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: child process 27684 exited by a signal 6 May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: core was not generated May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: terminating due to SIGCHLD May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: INFO:core:sig_usr: signal 15 received May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: Memory status (pkg): May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: qm_status (0x81677e0): May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: heap size= 1048576
The complete output of the memory status is available here: https://rcpt.yousendit.com/690295962/7b39d332264f086b1bf0f134c026fad3
It would be really great if you could have a look at the logfile! Perhaps you have any ideas what could cause this problem, so we can fix it. If needed, the complete logfile (from startup to termination) is also available: https://rcpt.yousendit.com/690307202/14929008dc7a452d69abb3cbc0a9b58a
Thanks a lot for your support, Christian
On Mittwoch, 20. Mai 2009, Christian Koch wrote:
[..] After solving all the performance issues with syslog we now made a stress test with memdebug enabled. Now kamailio terminates itself (perhaps because memory is corrupted?).
Hi Christian,
yes, this looks like the memory is corrupt, as it aparently crashes in a core function (added content from the log file), which normally should be pretty stable:
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 15), called from proxy.c: hostent_cpy(148) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 16), returns address 0x8265b20 frag. 0x8265b08 (size=32) on 1 -th hit May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 4), called from proxy.c: hostent_cpy(159) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 4), returns address 0x81bcbe8 frag. 0x81bcbd0 (size=4) on 1 -th hit May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 8), called from proxy.c: hostent_cpy(182) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params(0x81677e0, 0x8265b20), called from proxy.c: hostent_cpy(185) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: freeing frag. 0x8265b08 alloc'ed from proxy.c: hostent_cpy(148) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params(0x81677e0, (nil)), called from proxy.c: hostent_cpy(187)
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: CRITICAL:core:qm_free: bad pointer (nil) (out of memory block!) - aborting May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27697]: CRITICAL:core:receive_fd: EOF on 12 May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: child process 27684 exited by a signal 6 May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: core was not generated
Can you configure the kamailio server that it generates a core file? Then take a look to the backtrace where the invalid memory access was done, to verify if its really crashed in the core function, or perhaps some other parts has a problem here. Further informations: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: terminating due to SIGCHLD May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: INFO:core:sig_usr: signal 15 received May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: Memory status (pkg): May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: qm_status (0x81677e0): May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: heap size= 1048576
The complete output of the memory status is available here: https://rcpt.yousendit.com/690295962/7b39d332264f086b1bf0f134c026fad3
From the logs it seems that indeed a log of memory was allocated from the pv core. One of the main callers is pv_parse_ht_name, which is from the htable module. Not sure if this is a valid condition that it allocates that much pkg_mem, Daniel, can you perhaps take a look?
Cheers, Henning
Hello Henning, hello Daniel,
Henning Westerholt schrieb:
Can you configure the kamailio server that it generates a core file? Then take a look to the backtrace where the invalid memory access was done, to verify if its really crashed in the core function, or perhaps some other parts has a problem here. Further informations: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We will try this and inform you about the results.
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: terminating due to SIGCHLD May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: INFO:core:sig_usr: signal 15 received May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: Memory status (pkg): May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: qm_status (0x81677e0): May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: heap size= 1048576
The complete output of the memory status is available here: https://rcpt.yousendit.com/690295962/7b39d332264f086b1bf0f134c026fad3
From the logs it seems that indeed a log of memory was allocated from the pv core. One of the main callers is pv_parse_ht_name, which is from the htable module. Not sure if this is a valid condition that it allocates that much pkg_mem, Daniel, can you perhaps take a look?
For one child process we checked, how many allocated fragments had not been freed:
[...] 173 -> mk_action(106) 237 -> addstr(787) 3277 -> pv_parse_ht_name(121) 3361 -> pv_parse_format(727)
So we had a look at pv_parse_ht_name() and we wonder, wheter the variable hpv is freed. Before "return 0" it is used but not freed. Additionaly the return from pv_parse_format() (&hpv->pve) seems not to be freed. Could this be the problem?
Regards, Christian
Hi Henning,
Henning Westerholt schrieb:
Can you configure the kamailio server that it generates a core file? Then take a look to the backtrace where the invalid memory access was done, to verify if its really crashed in the core function, or perhaps some other parts has a problem here. Further informations: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt #0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00b64825 in raise () from /lib/tls/libc.so.6 #2 0x00b66289 in abort () from /lib/tls/libc.so.6 #3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e "proxy.c", func=0x811724e "hostent_cpy", line=187) at mem/q_malloc.c:444 #4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at proxy.c:187 #5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0, is_sips=0) at proxy.c:305 #6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218, uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at ut.h:67 #7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218, proxy=0x0) at t_fwd.c:630 #8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0, flags=Variable "flags" is not available. ) at t_funcs.c:264 #9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0) at tm.c:1002 #10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at action.c:874 #11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at action.c:145 #12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at route.c:1171 #13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at route.c:1488 #14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at route.c:1493 #15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at action.c:729 #16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at action.c:145 #17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at action.c:120 #18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at action.c:145 #19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at action.c:746 #20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at action.c:145 #21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at action.c:752 #22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at action.c:145 #23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at action.c:120 #24 0x08087994 in receive_msg ( buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187 SIP/2.0\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: <sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175 #25 0x080be606 in udp_rcv_loop () at udp_server.c:449 #26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1)); if (dst->h_addr_list==0){ ser_error=ret=E_OUT_OF_MEM; pkg_free(dst->h_name); for(r=0; dst->h_aliases[r]; r++) pkg_free(dst->h_aliases[r]); pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187, WHERE THE CORE FILE IS GENERATED pkg_free(dst->h_aliases); goto error; }
So, there may be an error in proxy.c, but perhaps the reason for our problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where the variable hpv is not freed? Any comments on pv_parse_ht_name() are greatly appreciated. We could try to fix this, but we're not sure about the sideeffects.
Regards, Christian
On Dienstag, 26. Mai 2009, Christian Koch wrote:
Can you configure the kamailio server that it generates a core file? Then take a look to the backtrace where the invalid memory access was done, to verify if its really crashed in the core function, or perhaps some other parts has a problem here. Further informations: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
Hi Christian,
We generated a core file, the output of the backtrace is:
(gdb) bt #0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00b64825 in raise () from /lib/tls/libc.so.6 #2 0x00b66289 in abort () from /lib/tls/libc.so.6 #3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e "proxy.c", func=0x811724e "hostent_cpy", line=187) at mem/q_malloc.c:444 #4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at proxy.c:187 [..]
Thanks for the backtrace, you could try the attached patch (for 1.5.x) if it solves the crash for you. The additional free after the loop looks indeed a bit suspicious.
So, there may be an error in proxy.c, but perhaps the reason for our problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where the variable hpv is not freed? Any comments on pv_parse_ht_name() are greatly appreciated. We could try to fix this, but we're not sure about the sideeffects.
I also suspect the reason for the crash lays in this area. I did not wrote that module, thus i asked Daniel.
Regards, Henning
Hi Henning, hi Daniel,
we did some more investigation and now we think, the problem may occur when accessing the "inner name" of pv's from htable.
We included a call to backtrace() in pv_parse_ht_name() (where most of the memory was allocated) to find out, which calling function would be responsible for freeing the allocated memory. First we thought it could be xlog, but more tests showed, the error also occurs when called from other modules. pv_parse_ht_name() is set as the innername in the htable module:
static pv_export_t mod_pvs[] = { { {"sht", sizeof("sht")-1}, PVT_OTHER, pv_get_ht_cell, pv_set_ht_cell, pv_parse_ht_name, 0, 0, 0 },
pv_parse_ht_name() allocates some memory (line 121, ht_var.c), saves it in sp->pvp.pvn.u.dname (line 165) and sets the type to PV_NAME_PVAR. As "u" is a union we thought freeing the memory should depend on the type. We can not find any place, where type is checked for PV_NAME_PVAR and any memory is freed.
Could you give us any hints where this should happen? Maybe this is the only thing we have to add and the memory leak is gone.
By the way, when searching for PV_NAME_INTSTR we found a potential bug: In pvapi.c there is the follwoing piece of code (line 773 to 785):
if(ip->pvn.type==PV_NAME_INTSTR) { [...] } else if(ip->pvn.type==PV_NAME_INTSTR) { /* pvar */
The second if also checks for PV_NAME_INTSTR so it will never be executed. Shouldn't this be PV_NAME_PVAR?
Regards, Christian
Christian Koch schrieb:
Hi Henning,
Henning Westerholt schrieb:
Can you configure the kamailio server that it generates a core file? Then take a look to the backtrace where the invalid memory access was done, to verify if its really crashed in the core function, or perhaps some other parts has a problem here. Further informations: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt #0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00b64825 in raise () from /lib/tls/libc.so.6 #2 0x00b66289 in abort () from /lib/tls/libc.so.6 #3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e "proxy.c", func=0x811724e "hostent_cpy", line=187) at mem/q_malloc.c:444 #4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at proxy.c:187 #5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0, is_sips=0) at proxy.c:305 #6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218, uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at ut.h:67 #7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218, proxy=0x0) at t_fwd.c:630 #8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0, flags=Variable "flags" is not available. ) at t_funcs.c:264 #9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0) at tm.c:1002 #10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at action.c:874 #11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at action.c:145 #12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at route.c:1171 #13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at route.c:1488 #14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at route.c:1493 #15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at action.c:729 #16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at action.c:145 #17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at action.c:120 #18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at action.c:145 #19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at action.c:746 #20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at action.c:145 #21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at action.c:752 #22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at action.c:145 #23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at action.c:120 #24 0x08087994 in receive_msg ( buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187 SIP/2.0\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: <sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175 #25 0x080be606 in udp_rcv_loop () at udp_server.c:449 #26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1)); if (dst->h_addr_list==0){ ser_error=ret=E_OUT_OF_MEM; pkg_free(dst->h_name); for(r=0; dst->h_aliases[r]; r++)pkg_free(dst->h_aliases[r]); pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187, WHERE THE CORE FILE IS GENERATED pkg_free(dst->h_aliases); goto error; }
So, there may be an error in proxy.c, but perhaps the reason for our problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where the variable hpv is not freed? Any comments on pv_parse_ht_name() are greatly appreciated. We could try to fix this, but we're not sure about the sideeffects.
Regards, Christian
Hello again,
as no one answered our question yet, we would like to ask again if someone is maybe able to take a deeper look in this? We are really stuck here and will also post this issue again on the devel list, maybe someone there is also able to take a look into this. Any help is really appreciated.
Thanks and regards, Christian Koch
Christian Koch schrieb:
Hi Henning, hi Daniel,
we did some more investigation and now we think, the problem may occur when accessing the "inner name" of pv's from htable.
We included a call to backtrace() in pv_parse_ht_name() (where most of the memory was allocated) to find out, which calling function would be responsible for freeing the allocated memory. First we thought it could be xlog, but more tests showed, the error also occurs when called from other modules. pv_parse_ht_name() is set as the innername in the htable module:
static pv_export_t mod_pvs[] = { { {"sht", sizeof("sht")-1}, PVT_OTHER, pv_get_ht_cell, pv_set_ht_cell, pv_parse_ht_name, 0, 0, 0 },
pv_parse_ht_name() allocates some memory (line 121, ht_var.c), saves it in sp->pvp.pvn.u.dname (line 165) and sets the type to PV_NAME_PVAR. As "u" is a union we thought freeing the memory should depend on the type. We can not find any place, where type is checked for PV_NAME_PVAR and any memory is freed.
Could you give us any hints where this should happen? Maybe this is the only thing we have to add and the memory leak is gone.
By the way, when searching for PV_NAME_INTSTR we found a potential bug: In pvapi.c there is the follwoing piece of code (line 773 to 785):
if(ip->pvn.type==PV_NAME_INTSTR) { [...] } else if(ip->pvn.type==PV_NAME_INTSTR) { /* pvar */
The second if also checks for PV_NAME_INTSTR so it will never be executed. Shouldn't this be PV_NAME_PVAR?
Regards, Christian
Christian Koch schrieb:
Hi Henning,
Henning Westerholt schrieb:
Can you configure the kamailio server that it generates a core file? Then take a look to the backtrace where the invalid memory access was done, to verify if its really crashed in the core function, or perhaps some other parts has a problem here. Further informations: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt #0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00b64825 in raise () from /lib/tls/libc.so.6 #2 0x00b66289 in abort () from /lib/tls/libc.so.6 #3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e "proxy.c", func=0x811724e "hostent_cpy", line=187) at mem/q_malloc.c:444 #4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at proxy.c:187 #5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0, is_sips=0) at proxy.c:305 #6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218, uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at ut.h:67 #7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218, proxy=0x0) at t_fwd.c:630 #8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0, flags=Variable "flags" is not available. ) at t_funcs.c:264 #9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0) at tm.c:1002 #10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at action.c:874 #11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at action.c:145 #12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at route.c:1171 #13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at route.c:1488 #14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at route.c:1493 #15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at action.c:729 #16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at action.c:145 #17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at action.c:120 #18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at action.c:145 #19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at action.c:746 #20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at action.c:145 #21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at action.c:752 #22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at action.c:145 #23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at action.c:120 #24 0x08087994 in receive_msg ( buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187 SIP/2.0\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: <sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175 #25 0x080be606 in udp_rcv_loop () at udp_server.c:449 #26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1)); if (dst->h_addr_list==0){ ser_error=ret=E_OUT_OF_MEM; pkg_free(dst->h_name); for(r=0; dst->h_aliases[r]; r++)pkg_free(dst->h_aliases[r]); pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187, WHERE THE CORE FILE IS GENERATED pkg_free(dst->h_aliases); goto error; }
So, there may be an error in proxy.c, but perhaps the reason for our problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where the variable hpv is not freed? Any comments on pv_parse_ht_name() are greatly appreciated. We could try to fix this, but we're not sure about the sideeffects.
Regards, Christian
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello Christian,
I got your email but I was out in a short vacation. I will check it.
Thanks, Daniel
On 06/09/2009 10:38 AM, Christian Koch wrote:
Hello again,
as no one answered our question yet, we would like to ask again if someone is maybe able to take a deeper look in this? We are really stuck here and will also post this issue again on the devel list, maybe someone there is also able to take a look into this. Any help is really appreciated.
Thanks and regards, Christian Koch
Christian Koch schrieb:
Hi Henning, hi Daniel,
we did some more investigation and now we think, the problem may occur when accessing the "inner name" of pv's from htable.
We included a call to backtrace() in pv_parse_ht_name() (where most of the memory was allocated) to find out, which calling function would be responsible for freeing the allocated memory. First we thought it could be xlog, but more tests showed, the error also occurs when called from other modules. pv_parse_ht_name() is set as the innername in the htable module:
static pv_export_t mod_pvs[] = { { {"sht", sizeof("sht")-1}, PVT_OTHER, pv_get_ht_cell, pv_set_ht_cell, pv_parse_ht_name, 0, 0, 0 },
pv_parse_ht_name() allocates some memory (line 121, ht_var.c), saves it in sp->pvp.pvn.u.dname (line 165) and sets the type to PV_NAME_PVAR. As "u" is a union we thought freeing the memory should depend on the type. We can not find any place, where type is checked for PV_NAME_PVAR and any memory is freed.
Could you give us any hints where this should happen? Maybe this is the only thing we have to add and the memory leak is gone.
By the way, when searching for PV_NAME_INTSTR we found a potential bug: In pvapi.c there is the follwoing piece of code (line 773 to 785):
if(ip->pvn.type==PV_NAME_INTSTR) { [...] } else if(ip->pvn.type==PV_NAME_INTSTR) { /* pvar */
The second if also checks for PV_NAME_INTSTR so it will never be executed. Shouldn't this be PV_NAME_PVAR?
Regards, Christian
Christian Koch schrieb:
Hi Henning,
Henning Westerholt schrieb:
Can you configure the kamailio server that it generates a core file? Then take a look to the backtrace where the invalid memory access was done, to verify if its really crashed in the core function, or perhaps some other parts has a problem here. Further informations: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt #0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00b64825 in raise () from /lib/tls/libc.so.6 #2 0x00b66289 in abort () from /lib/tls/libc.so.6 #3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e "proxy.c", func=0x811724e "hostent_cpy", line=187) at mem/q_malloc.c:444 #4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at proxy.c:187 #5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0, is_sips=0) at proxy.c:305 #6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218, uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at ut.h:67 #7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218, proxy=0x0) at t_fwd.c:630 #8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0, flags=Variable "flags" is not available. ) at t_funcs.c:264 #9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0) at tm.c:1002 #10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at action.c:874 #11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at action.c:145 #12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at route.c:1171 #13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at route.c:1488 #14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at route.c:1493 #15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at action.c:729 #16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at action.c:145 #17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at action.c:120 #18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at action.c:145 #19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at action.c:746 #20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at action.c:145 #21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at action.c:752 #22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at action.c:145 #23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at action.c:120 #24 0x08087994 in receive_msg ( buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187 SIP/2.0\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: <sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175 #25 0x080be606 in udp_rcv_loop () at udp_server.c:449 #26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1)); if (dst->h_addr_list==0){ ser_error=ret=E_OUT_OF_MEM; pkg_free(dst->h_name); for(r=0; dst->h_aliases[r]; r++)pkg_free(dst->h_aliases[r]); pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187, WHERE THE CORE FILE IS GENERATED pkg_free(dst->h_aliases); goto error; }
So, there may be an error in proxy.c, but perhaps the reason for our problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where the variable hpv is not freed? Any comments on pv_parse_ht_name() are greatly appreciated. We could try to fix this, but we're not sure about the sideeffects.
Regards, Christian
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello,
On 06/03/2009 03:01 PM, Christian Koch wrote:
Hi Henning, hi Daniel,
we did some more investigation and now we think, the problem may occur when accessing the "inner name" of pv's from htable.
We included a call to backtrace() in pv_parse_ht_name() (where most of the memory was allocated) to find out, which calling function would be responsible for freeing the allocated memory. First we thought it could be xlog, but more tests showed, the error also occurs when called from other modules. pv_parse_ht_name() is set as the innername in the htable module:
static pv_export_t mod_pvs[] = { { {"sht", sizeof("sht")-1}, PVT_OTHER, pv_get_ht_cell, pv_set_ht_cell, pv_parse_ht_name, 0, 0, 0 },
pv_parse_ht_name() allocates some memory (line 121, ht_var.c), saves it in sp->pvp.pvn.u.dname (line 165) and sets the type to PV_NAME_PVAR. As "u" is a union we thought freeing the memory should depend on the type. We can not find any place, where type is checked for PV_NAME_PVAR and any memory is freed.
Could you give us any hints where this should happen? Maybe this is the only thing we have to add and the memory leak is gone.
Do you call pv_parse_ht_name() from perl? If not, that function is executed only at startup and it is not a cause for leaking. A clean shutdown should free it, but there is not complete support for it (some fixup functions have a free function, others done, as well as the script actions tree).
That structure is not freed because it used at runtime. During startup, the name of $sht is parsed and kept as custom structure to be used latter, for speed reasons.
I tried to access the files you uploaded, with the log, but they expired. If you have the one with memory debug, please send it to me.
By the way, when searching for PV_NAME_INTSTR we found a potential bug: In pvapi.c there is the follwoing piece of code (line 773 to 785):
if(ip->pvn.type==PV_NAME_INTSTR) { [...] } else if(ip->pvn.type==PV_NAME_INTSTR) { /* pvar */
The second if also checks for PV_NAME_INTSTR so it will never be executed. Shouldn't this be PV_NAME_PVAR?
Yes, should have been so and it is now fixed. The issue was pretty harmless, as that function was called mainly for avps and hdrs PV with int or str name.
Thanks, Daniel
Regards, Christian
Christian Koch schrieb:
Hi Henning,
Henning Westerholt schrieb:
Can you configure the kamailio server that it generates a core file? Then take a look to the backtrace where the invalid memory access was done, to verify if its really crashed in the core function, or perhaps some other parts has a problem here. Further informations: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
We generated a core file, the output of the backtrace is:
(gdb) bt #0 0x00b237a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00b64825 in raise () from /lib/tls/libc.so.6 #2 0x00b66289 in abort () from /lib/tls/libc.so.6 #3 0x080c4436 in qm_free (qm=0x81677e0, p=0x0, file=0x811728e "proxy.c", func=0x811724e "hostent_cpy", line=187) at mem/q_malloc.c:444 #4 0x0807e329 in hostent_cpy (dst=0x8266e28, src=0x8146134) at proxy.c:187 #5 0x0807ea16 in mk_proxy (name=0xbfe8f380, port=0, proto=0, is_sips=0) at proxy.c:305 #6 0x0023cf81 in add_uac (t=0xb61d6128, request=0x81bb218, uri=0xbfe8f538, next_hop=0xbfe8f540, path=0x81bb538, proxy=0x0) at ut.h:67 #7 0x0023e74c in t_forward_nonack (t=0xb61d6128, p_msg=0x81bb218, proxy=0x0) at t_fwd.c:630 #8 0x0023a501 in t_relay_to (p_msg=0x81bb218, proxy=0x0, flags=Variable "flags" is not available. ) at t_funcs.c:264 #9 0x0024eb9f in w_t_relay (p_msg=0x81bb218, proxy=0x0, flags=0x0) at tm.c:1002 #10 0x080531ed in do_action (a=0x81a966c, msg=0x81bb218) at action.c:874 #11 0x08055406 in run_action_list (a=0x81a966c, msg=0x81bb218) at action.c:145 #12 0x08094fb0 in eval_expr (e=0x81a96fc, msg=0x81bb218, val=0x0) at route.c:1171 #13 0x08095214 in eval_expr (e=0x81a974c, msg=0x81bb218, val=0x0) at route.c:1488 #14 0x08094b6f in eval_expr (e=0x81a979c, msg=0x81bb218, val=0x0) at route.c:1493 #15 0x080526d8 in do_action (a=0x81a9d7c, msg=0x81bb218) at action.c:729 #16 0x08055406 in run_action_list (a=0x81a952c, msg=0x81bb218) at action.c:145 #17 0x08053ed9 in do_action (a=0x81a7efc, msg=0x81bb218) at action.c:120 #18 0x08055406 in run_action_list (a=0x81a7efc, msg=0x81bb218) at action.c:145 #19 0x08054f85 in do_action (a=0x81a86ac, msg=0x81bb218) at action.c:746 #20 0x08055406 in run_action_list (a=0x81a86ac, msg=0x81bb218) at action.c:145 #21 0x08054fbc in do_action (a=0x81a873c, msg=0x81bb218) at action.c:752 #22 0x08055406 in run_action_list (a=0x81a246c, msg=0x81bb218) at action.c:145 #23 0x0805570e in run_top_route (a=0x81a246c, msg=0x81bb218) at action.c:120 #24 0x08087994 in receive_msg ( buf=0x8146ba0 "CANCEL sip:1001@212.59.42.187 SIP/2.0\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: sip:212.59.42.187;lr=on;ftag=ACU-6afbe8bb-f11cd9dc\r\nRecord-Route: <sip:212.59"..., len=1499, rcv_info=0xbfe91240) at receive.c:175 #25 0x080be606 in udp_rcv_loop () at udp_server.c:449 #26 0x0806b361 in main (argc=3, argv=0xbfe914b4) at main.c:774
This core file obviuosly only occurs, because there is no more pkg memory. proxy.c:
dst->h_addr_list=(char**)pkg_malloc(sizeof(char*)*(len+1)); if (dst->h_addr_list==0){ ser_error=ret=E_OUT_OF_MEM; pkg_free(dst->h_name); for(r=0; dst->h_aliases[r]; r++)pkg_free(dst->h_aliases[r]); pkg_free(dst->h_aliases[r]); <-- THIS IS LINE 187, WHERE THE CORE FILE IS GENERATED pkg_free(dst->h_aliases); goto error; }
So, there may be an error in proxy.c, but perhaps the reason for our problem is in modules/htable/ht_var.c in pv_parse_ht_name(), where the variable hpv is not freed? Any comments on pv_parse_ht_name() are greatly appreciated. We could try to fix this, but we're not sure about the sideeffects.
Regards, Christian
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello Daniel,
Daniel-Constantin Mierla schrieb:
Do you call pv_parse_ht_name() from perl? If not, that function is executed only at startup and it is not a cause for leaking. A clean shutdown should free it, but there is not complete support for it (some fixup functions have a free function, others done, as well as the script actions tree).
We're not calling this directly from perl, but it seems it is called automatically every time when accessing the pv $sht. We've modified ht_var.c from module htable for getting a backtrace every time memory is allocated in pv_parse_ht_name:
void printBacktrace() { void *array[100]; size_t size; char **strings; size_t i;
size = backtrace (array, 100); strings = backtrace_symbols (array, size);
LM_DBG("Obtained %zd stack frames.\n", size);
for (i = 0; i < size; i++) LM_DBG("%s\n", strings[i]);
free (strings); }
int pv_parse_ht_name(pv_spec_p sp, str *in) { ht_pv_t *hpv=NULL; char *p; str pvs;
if(in->s==NULL || in->len<=0) return -1;
hpv = (ht_pv_t*)pkg_malloc(sizeof(ht_pv_t)); if(hpv==NULL) return -1;
LM_DBG("Allocated new memory %u/0x%x in pv_parse_ht_name()", hpv, hpv); printBacktrace(); [...]
When making a call we get the following outputs:
xlog:
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:pv_parse_ht_name: Allocated new memory 136020048 in pv_parse_ht_name() May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: Obtained 10 stack frames. May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/htable.so(printBacktrace+0x2d) [0x1dd1c5] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/htable.so(pv_parse_ht_name+0x8f) [0x1dd8fe] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(pv_parse_spec+0x58f) [0x8082c35] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(pv_parse_format+0x1eb) [0x808409e] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/xlog.so [0x1482c6] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/xlog.so [0x148620] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio [0x808e68e] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(fix_rls+0x12b) [0x8096892] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(main+0xcd) [0x8068215] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /lib/tls/libc.so.6(__libc_start_main+0xd3) [0xb51df3]
and perl:
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:pv_parse_ht_name: Allocated new memory 136017028/0x81b7484 in pv_parse_ht_name() Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: Obtained 27 stack frames. Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/htable.so(printBacktrace+0x33) [0xf631cb] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/htable.so(pv_parse_ht_name+0x8f) [0xf63a42] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(pv_parse_spec+0x58f) [0x8082c35] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(pv_parse_format+0x1eb) [0x808409e] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/perl.so(pv_sprintf+0x93) [0x2a40c8] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/perl.so(XS_OpenSER__Message_pseudoVar+0x17f) [0x2a92f3] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_pp_entersub+0x2ea) [0x813c b2] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0xba) [0x7f71 6d] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so [0x7a1b26] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x6d7) [0x7a7ef1] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_call_pv+0x40) [0x7a81f9] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/perl.so(perl_exec2+0x198) [0x2af1c4] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/perl.so(perl_exec1+0x2c) [0x2af69d] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(do_action+0x1401) [0x80531ed] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(do_action+0x20ed) [0x8053ed9] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(do_action+0x3199) [0x8054f85] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(do_action+0x3199) [0x8054f85] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_top_route+0x6d) [0x805570e] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(receive_msg+0x572) [0x8087934] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(udp_rcv_loop+0xd26) [0x80be5a6] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(main+0x3219) [0x806b361] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /lib/tls/libc.so.6(__libc_start_main+0xd3) [0xb51df3] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio [0x8051991]
So because of this we don't think it is just called once but the allocated memory should be freed after processing a message.
That structure is not freed because it used at runtime. During startup, the name of $sht is parsed and kept as custom structure to be used latter, for speed reasons.
I tried to access the files you uploaded, with the log, but they expired. If you have the one with memory debug, please send it to me.
We still have it, but its 43MB (already zipped) and it's without the above "printBacktrace". If you need it anyway or need a complete logfile including the backtrace we could send it to you.
Thanks for your help, Christian
Hello Christian,
On 06/09/2009 01:14 PM, Christian Koch wrote:
Hello Daniel,
Daniel-Constantin Mierla schrieb:
Do you call pv_parse_ht_name() from perl? If not, that function is executed only at startup and it is not a cause for leaking. A clean shutdown should free it, but there is not complete support for it (some fixup functions have a free function, others done, as well as the script actions tree).
We're not calling this directly from perl, but it seems it is called automatically every time when accessing the pv $sht.
unfortunately you cannot use $sht in this way from perl module. The same limitation that prevent calling any function from modules apply here -- the fixup system used at startup, when string parameters are pre-compiled in different forms to speed up runtime execution.
It can be added as a feature request for future. For module functions I added the core support for free fixup functions, but nobody committed to continue the work in perl module.
If you want to work on it, I can assist and guide you.
Cheers, Daniel
We've modified ht_var.c from module htable for getting a backtrace every time memory is allocated in pv_parse_ht_name:
void printBacktrace() { void *array[100]; size_t size; char **strings; size_t i; size = backtrace (array, 100); strings = backtrace_symbols (array, size); LM_DBG("Obtained %zd stack frames.\n", size); for (i = 0; i < size; i++) LM_DBG("%s\n", strings[i]); free (strings); } int pv_parse_ht_name(pv_spec_p sp, str *in) { ht_pv_t *hpv=NULL; char *p; str pvs; if(in->s==NULL || in->len<=0) return -1; hpv = (ht_pv_t*)pkg_malloc(sizeof(ht_pv_t)); if(hpv==NULL) return -1; LM_DBG("Allocated new memory %u/0x%x in pv_parse_ht_name()", hpv, hpv); printBacktrace(); [...]When making a call we get the following outputs:
xlog:
May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:pv_parse_ht_name: Allocated new memory 136020048 in pv_parse_ht_name() May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: Obtained 10 stack frames. May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/htable.so(printBacktrace+0x2d) [0x1dd1c5] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/htable.so(pv_parse_ht_name+0x8f) [0x1dd8fe] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(pv_parse_spec+0x58f) [0x8082c35] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(pv_parse_format+0x1eb) [0x808409e] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/xlog.so [0x1482c6] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/xlog.so [0x148620] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio [0x808e68e] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(fix_rls+0x12b) [0x8096892] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(main+0xcd) [0x8068215] May 29 16:56:01 AmbriaSip1 /usr/local/sbin/kamailio[27187]: DBG:htable:printBacktrace: /lib/tls/libc.so.6(__libc_start_main+0xd3) [0xb51df3]and perl:
Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:pv_parse_ht_name: Allocated new memory 136017028/0x81b7484 in pv_parse_ht_name() Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: Obtained 27 stack frames. Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/htable.so(printBacktrace+0x33) [0xf631cb] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/htable.so(pv_parse_ht_name+0x8f) [0xf63a42] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(pv_parse_spec+0x58f) [0x8082c35] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(pv_parse_format+0x1eb) [0x808409e] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/perl.so(pv_sprintf+0x93) [0x2a40c8] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/perl.so(XS_OpenSER__Message_pseudoVar+0x17f) [0x2a92f3] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_pp_entersub+0x2ea) [0x813c b2] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_runops_debug+0xba) [0x7f71 6d] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so [0x7a1b26] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x6d7) [0x7a7ef1] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so(Perl_call_pv+0x40) [0x7a81f9] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/perl.so(perl_exec2+0x198) [0x2af1c4] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/lib/kamailio/modules/perl.so(perl_exec1+0x2c) [0x2af69d] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(do_action+0x1401) [0x80531ed] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(do_action+0x20ed) [0x8053ed9] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(do_action+0x3199) [0x8054f85] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(do_action+0x3199) [0x8054f85] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_action_list+0x28) [0x8055406] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(run_top_route+0x6d) [0x805570e] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(receive_msg+0x572) [0x8087934] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(udp_rcv_loop+0xd26) [0x80be5a6] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio(main+0x3219) [0x806b361] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /lib/tls/libc.so.6(__libc_start_main+0xd3) [0xb51df3] Jun 2 10:53:28 AmbriaSip1 /usr/local/sbin/kamailio[19777]: DBG:htable:printBacktrace: /usr/local/sbin/kamailio [0x8051991]So because of this we don't think it is just called once but the allocated memory should be freed after processing a message.
That structure is not freed because it used at runtime. During startup, the name of $sht is parsed and kept as custom structure to be used latter, for speed reasons.
I tried to access the files you uploaded, with the log, but they expired. If you have the one with memory debug, please send it to me.
We still have it, but its 43MB (already zipped) and it's without the above "printBacktrace". If you need it anyway or need a complete logfile including the backtrace we could send it to you.
Thanks for your help, Christian
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello Daniel,
Daniel-Constantin Mierla schrieb:
unfortunately you cannot use $sht in this way from perl module. The same limitation that prevent calling any function from modules apply here -- the fixup system used at startup, when string parameters are pre-compiled in different forms to speed up runtime execution.
So there is no way to access $sht from other moduls (like perl or xlog)? It seems to work, but obviously the memory isn't freed :-)
What we are trying to do is the following: During processing of "INVITE" we need to save some call details (source IP, URI, ...) and want to write them to file when the call is finished (including duration, ...), while processing BYE or CANCEL, to have CDRs. As there are multiple childs of kamailio we tried to save the calls details from the INVITE in shared memory using htable (e.g. "$sht(a=>$ci::srcip) = $si"), so we can use them while processing the BYE (possibly in another child). There we wanted to use perl to be able to write the saved details to a file. We used the shared memory module as we don´t want to save the infos to DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way. Do you have any idea what else we can do to generate CDRs as mentioned? By using other modules for example or a complete other way you may know?
It can be added as a feature request for future. For module functions I added the core support for free fixup functions, but nobody committed to continue the work in perl module.
If you want to work on it, I can assist and guide you.
Well, honestly we are looking for a fast way to implement it. How long will it take to implement it with your guidance?
Thanks and regards, Christian
Christian,
Not sure if this will help you, but it's a good starting point if you want to use Perl. I grab a good deal of information using the headers and AVP's in order to write data to a MSSQL database and do LCR. Here's essentially how I grab the values:
#Begin use strict; use OpenSER qw ( log ); use OpenSER::Constants; use OpenSER::Message;
Sub your_function { # get headers my $m = shift; my ($lrn, $inrpid);
# Headers my $ruri = $m->getHeader("To");
# Pseudo Variables my $source_ip = $m->pseudoVar("$si"); my $in_ANI = $m->pseudoVar("$fU"); my $insrc = $m->pseudoVar("$fd"); my $daytim = $m->pseudoVar("$Tf"); my $callid = $m->pseudoVar("$ci");
# Check if PV's are set if( $m->pseudoVar("$re") != undef ){ $inrpid = $m->pseudoVar("$re"); }
# AVP's if( OpenSER::AVP::get(60) != undef ){ $lrn = OpenSER::AVP::get(60); }
# Write some data to the log log(L_INFO, sprintf("callid: %s\n", $callid));
# DO WHATEVER ELSE
} # END
You can set AVP's in your script before calling the function: $avp(my_AVP) = $sht(a=>$tu-translate);
Mik
-----Original Message----- From: users-bounces@lists.kamailio.org [mailto:users-bounces@lists.kamailio.org] On Behalf Of Christian Koch Sent: Tuesday, June 09, 2009 6:56 AM To: Daniel-Constantin Mierla Cc: users@lists.kamailio.org Subject: Re: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello Daniel,
Daniel-Constantin Mierla schrieb:
unfortunately you cannot use $sht in this way from perl module. The same limitation that prevent calling any function from modules apply here -- the fixup system used at startup, when string parameters are pre-compiled in different forms to speed up runtime execution.
So there is no way to access $sht from other moduls (like perl or xlog)? It seems to work, but obviously the memory isn't freed :-)
What we are trying to do is the following: During processing of "INVITE" we need to save some call details (source IP, URI, ...) and want to write them to file when the call is finished (including duration, ...), while processing BYE or CANCEL, to have CDRs. As there are multiple childs of kamailio we tried to save the calls details from the INVITE in shared memory using htable (e.g. "$sht(a=>$ci::srcip) = $si"), so we can use them while processing the BYE (possibly in another child). There we wanted to use perl to be able to write the saved details to a file. We used the shared memory module as we don´t want to save the infos to DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way. Do you have any idea what else we can do to generate CDRs as mentioned? By using other modules for example or a complete other way you may know?
It can be added as a feature request for future. For module functions I added the core support for free fixup functions, but nobody committed to continue the work in perl module.
If you want to work on it, I can assist and guide you.
Well, honestly we are looking for a fast way to implement it. How long will it take to implement it with your guidance?
Thanks and regards, Christian
_______________________________________________ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hi Mik,
thanks for your hint!!! We're now trying to save the variables from $sht in AVPs and accessing only the AVPs from perl to avoid the memory leak. Currently we're running our stress test. We'll inform you about the result.
Christian
Mik Cheez schrieb:
Christian,
Not sure if this will help you, but it's a good starting point if you want to use Perl. I grab a good deal of information using the headers and AVP's in order to write data to a MSSQL database and do LCR. Here's essentially how I grab the values:
#Begin use strict; use OpenSER qw ( log ); use OpenSER::Constants; use OpenSER::Message;
Sub your_function { # get headers my $m = shift; my ($lrn, $inrpid);
# Headers my $ruri = $m->getHeader("To");
# Pseudo Variables my $source_ip = $m->pseudoVar("$si"); my $in_ANI = $m->pseudoVar("$fU"); my $insrc = $m->pseudoVar("$fd"); my $daytim = $m->pseudoVar("$Tf"); my $callid = $m->pseudoVar("$ci");
# Check if PV's are set if( $m->pseudoVar("$re") != undef ){ $inrpid = $m->pseudoVar("$re"); }
# AVP's if( OpenSER::AVP::get(60) != undef ){ $lrn = OpenSER::AVP::get(60); }
# Write some data to the log log(L_INFO, sprintf("callid: %s\n", $callid));
# DO WHATEVER ELSE
} # END
You can set AVP's in your script before calling the function: $avp(my_AVP) = $sht(a=>$tu-translate);
Mik
-----Original Message----- From: users-bounces@lists.kamailio.org [mailto:users-bounces@lists.kamailio.org] On Behalf Of Christian Koch Sent: Tuesday, June 09, 2009 6:56 AM To: Daniel-Constantin Mierla Cc: users@lists.kamailio.org Subject: Re: [Kamailio-Users] freeing innername from htable Re: perl module, "pv_sprintf: Memory exhausted!"
Hello Daniel,
Daniel-Constantin Mierla schrieb:
unfortunately you cannot use $sht in this way from perl module. The same limitation that prevent calling any function from modules apply here -- the fixup system used at startup, when string parameters are pre-compiled in different forms to speed up runtime execution.
So there is no way to access $sht from other moduls (like perl or xlog)? It seems to work, but obviously the memory isn't freed :-)
What we are trying to do is the following: During processing of "INVITE" we need to save some call details (source IP, URI, ...) and want to write them to file when the call is finished (including duration, ...), while processing BYE or CANCEL, to have CDRs. As there are multiple childs of kamailio we tried to save the calls details from the INVITE in shared memory using htable (e.g. "$sht(a=>$ci::srcip) = $si"), so we can use them while processing the BYE (possibly in another child). There we wanted to use perl to be able to write the saved details to a file. We used the shared memory module as we don´t want to save the infos to DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way. Do you have any idea what else we can do to generate CDRs as mentioned? By using other modules for example or a complete other way you may know?
It can be added as a feature request for future. For module functions I added the core support for free fixup functions, but nobody committed to continue the work in perl module.
If you want to work on it, I can assist and guide you.
Well, honestly we are looking for a fast way to implement it. How long will it take to implement it with your guidance?
Thanks and regards, Christian
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello Christian,
On 06/09/2009 04:56 PM, Christian Koch wrote:
Hello Daniel,
Daniel-Constantin Mierla schrieb:
unfortunately you cannot use $sht in this way from perl module. The same limitation that prevent calling any function from modules apply here -- the fixup system used at startup, when string parameters are pre-compiled in different forms to speed up runtime execution.
So there is no way to access $sht from other moduls (like perl or xlog)? It seems to work, but obviously the memory isn't freed :-)
you could access it by adding some sht extensions to the Perl API, same way as for AVP, but this requires development as well.
From other modules is easier, as one can evaluate the name and then do search in the hashtable.
What we are trying to do is the following: During processing of "INVITE" we need to save some call details (source IP, URI, ...) and want to write them to file when the call is finished (including duration, ...), while processing BYE or CANCEL, to have CDRs. As there are multiple childs of kamailio we tried to save the calls details from the INVITE in shared memory using htable (e.g. "$sht(a=>$ci::srcip) = $si"), so we can use them while processing the BYE (possibly in another child). There we wanted to use perl to be able to write the saved details to a file. We used the shared memory module as we don´t want to save the infos to DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way. Do you have any idea what else we can do to generate CDRs as mentioned? By using other modules for example or a complete other way you may know?
You can use shared memory from perl, see persistence.pl in modules/perl/doc/samples/
Cheers, Daniel
It can be added as a feature request for future. For module functions I added the core support for free fixup functions, but nobody committed to continue the work in perl module.
If you want to work on it, I can assist and guide you.
Well, honestly we are looking for a fast way to implement it. How long will it take to implement it with your guidance?
Thanks and regards, Christian
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello all,
Daniel-Constantin Mierla schrieb:
Hello Christian,
On 06/09/2009 04:56 PM, Christian Koch wrote:
Hello Daniel,
Daniel-Constantin Mierla schrieb:
unfortunately you cannot use $sht in this way from perl module. The same limitation that prevent calling any function from modules apply here -- the fixup system used at startup, when string parameters are pre-compiled in different forms to speed up runtime execution.
So there is no way to access $sht from other moduls (like perl or xlog)? It seems to work, but obviously the memory isn't freed :-)
you could access it by adding some sht extensions to the Perl API, same way as for AVP, but this requires development as well.
From other modules is easier, as one can evaluate the name and then do search in the hashtable.
We tried the solution suggested by Mik (copy $sht-values to AVP in config file and accessing only AVPs from perl) and made about 20.000 calls without any problems. Seems that solved our problem. Thanks a lot!!!
What we are trying to do is the following: During processing of "INVITE" we need to save some call details (source IP, URI, ...) and want to write them to file when the call is finished (including duration, ...), while processing BYE or CANCEL, to have CDRs. As there are multiple childs of kamailio we tried to save the calls details from the INVITE in shared memory using htable (e.g. "$sht(a=>$ci::srcip) = $si"), so we can use them while processing the BYE (possibly in another child). There we wanted to use perl to be able to write the saved details to a file. We used the shared memory module as we don´t want to save the infos to DB because of performance issues we suspect when using DBs.
But as you mentioned $sht is not supposed to be used in such a way. Do you have any idea what else we can do to generate CDRs as mentioned? By using other modules for example or a complete other way you may know?
You can use shared memory from perl, see persistence.pl in modules/perl/doc/samples/
Well, we've already tried that. But it occurred several times, that after about two weeks the shared memory got corrupted and we had to reboot the machine. It seems "IPC::Shareable" is not ready for production use. We also tried "IPC::ShareLite" instead of "IPC::Shareable". "IPC::ShareLite" seems to be newer, better maintained and supported. But we had no success fetching saved values from shared memory - probably we didn't use it the right way.
Regards, Christian
-
Christian Koch -
Daniel-Constantin Mierla -
Henning Westerholt -
Mik Cheez