HelloOur kamailio proxy (version 1.5.2 ) crashed today due to a segmentation fault:Feb 22 16:52:10 AW-PROXY-01 kernel: kamailio[3667]: segfault at 000000000000000b rip 00002ab2b47bf93f rsp 00007ffff62dd270 error 6Apparently due to lack of private memory:[root@AW-PROXY-01 log]# cat proxy.log | grep ERROR | grep dbFeb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_convert_rows: no private memory left Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_convert_result: converting rows failed Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_store_result: failed to convert result Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:core:db_do_query: error while storing result Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_fetch_result: no private memory left Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:htable:ht_db_load_table: Error while fetching result
I was hoping to see a core file somewhere but I haven't found one yet. I did not have the line "disable_core_dump=no" set on the config file because supposedly is set to no by default:
From the online guide for my version:
//////////////////// Disable_core_dumpCan be 'yes' or 'no'. By default core dump limits are set to unlimited or a high enough value. Set this config variable to 'yes' to disable core dump-ing (will set core limits to 0).Default value is 'no'. ///////////////////How can I make sure that a core file is created after a situation like this? Where will it be created? How can I check the status of the private memory? I use the "kamctl fifo get_statistics all" but I don't know which one relates to "private memory": Example:shmem:total_size = 2147483648 shmem:used_size = 16860928 shmem:real_used_size = 17033968 shmem:max_used_size = 19174528 shmem:free_size = 2130449680 shmem:fragments = 1850I know that with so little information is hard to diagnose, but if anybody has an idea as to what else I can do I greatly appreciate it. Thank you in advancefborot
I am reporting the message because it got truncated: HelloOur kamailio proxy (version 1.5.2 ) crashed today due to a segmentation fault:Feb 22 16:52:10 AW-PROXY-01 kernel: kamailio[3667]: segfault at 000000000000000b rip 00002ab2b47bf93f rsp 00007ffff62dd270 error 6 Apparently due to lack of private memory: [root@AW-PROXY-01 log]# cat proxy.log | grep ERROR | grep db Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_convert_rows: no private memory left Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_convert_result: converting rows failed Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_store_result: failed to convert result Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:core:db_do_query: error while storing result Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_fetch_result: no private memory left Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:htable:ht_db_load_table: Error while fetching result
I was hoping to see a core file somewhere but I haven't found one yet. I did not have the line "disable_core_dump=no" set on the config file because supposedly is set to no by default:
From the online guide for my version:
Disable_core_dump: Can be 'yes' or 'no'. By default core dump limits are set to unlimited or a high enough value. Set this config variable to 'yes' to disable core dump-ing(will set core limits to 0).Default value is 'no'. How can I make sure that a core file is created after a situation like this? Where will it be created? How can I check the status of the private memory? I use the "kamctl fifo get_statistics all" but I don't know which one relates to "private memory": Example:shmem:total_size = 2147483648 shmem:used_size = 16860928 shmem:real_used_size = 17033968 shmem:max_used_size = 19174528 shmem:free_size = 2130449680 shmem:fragments = 1850I know that with so little information is hard to diagnose, but if anybody has an idea as to what else I can do I greatly appreciate it. Thank you in advancefborot
I am reporting the message because it got truncated (again, now trying as plain text):
Hello Our kamailio proxy (version 1.5.2 ) crashed today due to a segmentation fault: Feb 22 16:52:10 AW-PROXY-01 kernel: kamailio[3667]: segfault at 000000000000000b rip 00002ab2b47bf93f rsp 00007ffff62dd270 error 6
Apparently due to lack of private memory: [root@AW-PROXY-01 log]# cat proxy.log | grep ERROR | grep db Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_convert_rows: no private memory left Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_convert_result: converting rows failed Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_store_result: failed to convert result Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:core:db_do_query: error while storing result Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:db_unixodbc:db_unixodbc_fetch_result: no private memory left Feb 22 16:22:10 AW-PROXY-01 /usr/local/kamailio-install-dir/sbin/kamailio[3667]: ERROR:htable:ht_db_load_table: Error while fetching result I was hoping to see a core file somewhere but I haven't found one yet. I did not have the line "disable_core_dump=no" set on the config file because supposedly is set to no by default:
From the online guide for my version: Disable_core_dump: Can be 'yes' or 'no'. By default core dump limits are set to unlimited or
a high enough value. Set this config variable to 'yes' to disable core dump-ing(will set core limits to 0).Default value is 'no'.
How can I make sure that a core file is created after a situation like this? Where will it be created? How can I check the status of the private memory? I use the "kamctl fifo get_statistics all" but I don't know which one relates to "private memory": Example:shmem:total_size = 2147483648 shmem:used_size = 16860928 shmem:real_used_size = 17033968 shmem:max_used_size = 19174528 shmem:free_size = 2130449680 shmem:fragments = 1850 I know that with so little information is hard to diagnose, but if anybody has an idea as to what else I can do I greatly appreciate it. Thank you in advance fborot
Last part of the post that was not posted, sorry for the mess:
From the online guide for my version:
Disable_core_dump: Can be 'yes' or 'no'. By default core dump limits are set to unlimited or a high enough value. Set this config variable to 'yes' to disable core dump-ing (will set core limits to 0).
Default value is 'no'.
How can I make sure that a core file is created after a situation like this? Where will it be created? How can I check the status of the private memory? I use the "kamctl fifo get_statistics all" but I don't know which one relates to "private memory": Example:
shmem:total_size = 2147483648 shmem:used_size = 16860928 shmem:real_used_size = 17033968 shmem:max_used_size = 19174528 shmem:free_size = 2130449680 shmem:fragments = 1850
I know that with so little information is hard to diagnose, but if anybody has an idea as to what else I can do I greatly appreciate it. Thank you in advance
fborot
How can I make sure that a core file is created after a situation like this? Where will it be created? How can I check the status of the private memory? I use the "kamctl fifo get_statistics all" but I don't know which one relates to "private memory": Example:
shmem:total_size = 2147483648 shmem:used_size = 16860928 shmem:real_used_size = 17033968 shmem:max_used_size = 19174528 shmem:free_size = 2130449680 shmem:fragments = 1850
I know that with so little information is hard to diagnose, but if anybody has an idea as to what else I can do I greatly appreciate it. Thank you in advance
Usually (on Debian) the core file will be activated in /etc/default/kamailio and the location of the core file is specified in /etc/init.d/kamailio.
regards klaus
On 22.02.2012 23:45, Fabian Borot wrote:
How can I make sure that a core file is created after a situation like this? Where will it be created? How can I check the status of the private memory? I use the "kamctl fifo get_statistics all" but I don't know which one relates to "private memory": Example:
shmem:total_size = 2147483648 shmem:used_size = 16860928 shmem:real_used_size = 17033968 shmem:max_used_size = 19174528 shmem:free_size = 2130449680 shmem:fragments = 1850
I know that with so little information is hard to diagnose, but if anybody has an idea as to what else I can do I greatly appreciate it. Thank you in advance
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Also, with some kernels, it is disabled for non-root users, so might be better to run as root when trying to reproduce.
Private memory stats are available starting with 3.2, via 'sercmd pkg.stats'.
For older versions, you would have to walk the private memory with gdb: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory#walking_thr...
and you can see where are the chunks allocated from.
Cheers, Daniel
On 2/23/12 10:40 AM, Klaus Darilion wrote:
Usually (on Debian) the core file will be activated in /etc/default/kamailio and the location of the core file is specified in /etc/init.d/kamailio.
regards klaus
On 22.02.2012 23:45, Fabian Borot wrote:
How can I make sure that a core file is created after a situation like this? Where will it be created? How can I check the status of the private memory? I use the "kamctl fifo get_statistics all" but I don't know which one relates to "private memory": Example:
shmem:total_size = 2147483648 shmem:used_size = 16860928 shmem:real_used_size = 17033968 shmem:max_used_size = 19174528 shmem:free_size = 2130449680 shmem:fragments = 1850
I know that with so little information is hard to diagnose, but if anybody has an idea as to what else I can do I greatly appreciate it. Thank you in advance
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
thank you guys, for that setup we are using 1.5.2, and we are running it as root, on CentOS 5.2. what can I do to make sure that the core is created should this happen again?
----------------------------------------
From: fborot@hotmail.com
To: users@lists.kamailio.org
Subject: RE: Kamailio segmentation fault
Date: Wed, 22 Feb 2012 17:45:38 -0500
How can I make sure that a core file is created after a situation like this? Where will it be created?
How can I check the status of the private memory? I use the "kamctl fifo get_statistics all" but I don't know which one relates to "private memory":
Example:
shmem:total_size = 2147483648
shmem:used_size = 16860928
shmem:real_used_size = 17033968
shmem:max_used_size = 19174528
shmem:free_size = 2130449680
shmem:fragments = 1850
I know that with so little information is hard to diagnose, but if anybody has an idea as to what else I can do I greatly appreciate it.
Thank you in advance
-
Daniel-Constantin Mierla -
Fabian Borot -
Klaus Darilion