Hi,
Has anybody seen this :
Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): out of mem Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer
followed by:
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): out of mem Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2168]: build_res_buf_from_sip_req(): ERROR: build_res_buf_from_sip_req: out of memory ; needs 386
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/contact.c:194]: parse_contacts(): parse_contacts(): No memory left Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:59]: contact_parser(): contact_parser(): Error while parsing contacts Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:88]: parse_contact(): parse_contact(): Error while parsing
However, seems like we still had enough mem:
[root@kamailio kamailio]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 9526348 230308 13282472 0 0 0 4 0 1 6 0 94 0 0 1 0 0 9526216 230308 13282524 0 0 0 0 1313 84424 17 8 75 0 0 1 0 0 9526092 230308 13282616 0 0 0 0 1482 86882 16 9 75 0 0
[root@kamailio monit.d]# kamailio -v version: kamailio 4.2.0-dev0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 17:42:42 Dec 18 2013 with gcc 4.4.7
Unfortunately I had to reload p-cscf to get service working again.
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
Hi Daniel,
no we have not seen that... Can you provide details around the scenario - registration, calling, mixture, etc? Load? We have done extensive testing especially w.r.t. memory usage so I am very interested in this. Could you do a memory dump and send us the log?
Also, will you share your cfg file? you can send the cfg file directly to me if you prefer...
Cheers Jason
On Mon, Mar 3, 2014 at 10:58 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:
Hi,
Has anybody seen this :
Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): *out of mem* Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer
followed by:
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): *out of mem* Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2168]: build_res_buf_from_sip_req(): *ERROR: build_res_buf_from_sip_req: out of memory ; needs 386*
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/contact.c:194]: parse_contacts(): parse_contacts(): *No memory left* Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:59]: contact_parser(): contact_parser(): Error while parsing contacts Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:88]: parse_contact(): parse_contact(): Error while parsing
However, seems like we still had enough mem:
[root@kamailio kamailio]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 9526348 230308 13282472 0 0 0 4 0 1 6 0 94 0 0 1 0 0 9526216 230308 13282524 0 0 0 0 1313 84424 17 8 75 0 0 1 0 0 9526092 230308 13282616 0 0 0 0 1482 86882 16 9 75 0 0
[root@kamailio monit.d]# kamailio -v version: kamailio 4.2.0-dev0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 17:42:42 Dec 18 2013 with gcc 4.4.7
Unfortunately I had to reload p-cscf to get service working again.
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Jason,
HW configuration : this is a virtual machine running 4 cores with 24GB ram (VM hosting).
processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz stepping : 4 cpu MHz : 2900.000 cache size : 15360 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida arat epb pln pts dts bogomips : 5800.00 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
As far as scenario: it's a mixture of REGISTER/MESSAGE/SUBSCRIBE/NOTIFY. I will send logs (configs + /var/log/messages) privately .. but generally speaking config is almost identical with configs from samples directory for p-cscf.
Dan
On 03/03/2014 04:02 PM, Jason Penton wrote: Hi Daniel,
no we have not seen that... Can you provide details around the scenario - registration, calling, mixture, etc? Load? We have done extensive testing especially w.r.t. memory usage so I am very interested in this. Could you do a memory dump and send us the log?
Also, will you share your cfg file? you can send the cfg file directly to me if you prefer...
Cheers Jason
On Mon, Mar 3, 2014 at 10:58 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: Hi,
Has anybody seen this :
Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): out of mem Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer
followed by:
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): out of mem Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2168]: build_res_buf_from_sip_req(): ERROR: build_res_buf_from_sip_req: out of memory ; needs 386
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/contact.c:194]: parse_contacts(): parse_contacts(): No memory left Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:59]: contact_parser(): contact_parser(): Error while parsing contacts Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:88]: parse_contact(): parse_contact(): Error while parsing
However, seems like we still had enough mem:
[root@kamailio kamailio]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 9526348 230308 13282472 0 0 0 4 0 1 6 0 94 0 0 1 0 0 9526216 230308 13282524 0 0 0 0 1313 84424 17 8 75 0 0 1 0 0 9526092 230308 13282616 0 0 0 0 1482 86882 16 9 75 0 0
[root@kamailio monit.d]# kamailio -v version: kamailio 4.2.0-dev0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 17:42:42 Dec 18 2013 with gcc 4.4.7
Unfortunately I had to reload p-cscf to get service working again.
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.orgmailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Jason Penton Senior Manager: Applications and Services Smile Communications Pty (Ltd) Mobile: +27 (0) 83 283 7000 Skype: jason.barry.penton
jason.penton@smilecoms.commailto:name.surname@smilecoms.com www.smilecoms.comhttp://www.smilecoms.com/
This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
Thanks Daniel,
How much pkg and shm memory is assigned to your Kamailio instance? Also, how many users are you registering concurrently? Please also recreate and dump shm memory when you get a chance. If you are not sure how to dump shm memory - see here - http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
Cheers Jason
On Mon, Mar 3, 2014 at 11:07 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:
Jason,
HW configuration : this is a virtual machine running 4 cores with 24GB ram (VM hosting).
processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz stepping : 4 cpu MHz : 2900.000 cache size : 15360 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida arat epb pln pts dts bogomips : 5800.00 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
As far as scenario: it's a mixture of REGISTER/MESSAGE/SUBSCRIBE/NOTIFY. I will send logs (configs + /var/log/messages) privately .. but generally speaking config is almost identical with configs from samples directory for p-cscf.
Dan
On 03/03/2014 04:02 PM, Jason Penton wrote:
Hi Daniel,
no we have not seen that... Can you provide details around the scenario
- registration, calling, mixture, etc? Load? We have done extensive testing
especially w.r.t. memory usage so I am very interested in this. Could you do a memory dump and send us the log?
Also, will you share your cfg file? you can send the cfg file directly to me if you prefer...
Cheers Jason
On Mon, Mar 3, 2014 at 10:58 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:
Hi,
Has anybody seen this :
Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): *out of mem* Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer
followed by:
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): *out of mem* Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2168]: build_res_buf_from_sip_req(): *ERROR: build_res_buf_from_sip_req: out of memory ; needs 386*
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/contact.c:194]: parse_contacts(): parse_contacts(): *No memory left* Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:59]: contact_parser(): contact_parser(): Error while parsing contacts Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:88]: parse_contact(): parse_contact(): Error while parsing
However, seems like we still had enough mem:
[root@kamailio kamailio]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 9526348 230308 13282472 0 0 0 4 0 1 6 0 94 0 0 1 0 0 9526216 230308 13282524 0 0 0 0 1313 84424 17 8 75 0 0 1 0 0 9526092 230308 13282616 0 0 0 0 1482 86882 16 9 75 0 0
[root@kamailio monit.d]# kamailio -v version: kamailio 4.2.0-dev0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 17:42:42 Dec 18 2013 with gcc 4.4.7
Unfortunately I had to reload p-cscf to get service working again.
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
*Jason Penton* *Senior Manager: Applications and Services* *SmileCommunications Pty (Ltd)* *Mobile:* +27 (0) 83 283 7000 *Skype:* jason.barry.penton jason.penton@smilecoms.comname.surname@smilecoms.com www.smilecoms.com
This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
Hey Daniel,
Please check this commit. I think you are running before this was added?
commit 1d6b1d8765d7a6d21c291d2076a3a80a1710c11c Author: Jason Penton jason.penton@gmail.com Date: Fri Feb 14 09:57:45 2014 +0200
lib/ims:modules/ims_charging,ims_icscf,ims_qos,ims_registrar_scscf: fixed pkg_mem leak - fixed leak when parsing msg headers on a shm msg - instead now pass in flag to say if parsing is on shm_msg or not (is_shm) - assumes all common headers are already parsed before pkg_mem msg is cloned
Cheers Jason
On Mon, Mar 3, 2014 at 11:22 PM, Jason Penton jason.penton@smilecoms.comwrote:
Thanks Daniel,
How much pkg and shm memory is assigned to your Kamailio instance? Also, how many users are you registering concurrently? Please also recreate and dump shm memory when you get a chance. If you are not sure how to dump shm memory - see here - http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
Cheers Jason
On Mon, Mar 3, 2014 at 11:07 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:
Jason,
HW configuration : this is a virtual machine running 4 cores with 24GB ram (VM hosting).
processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz stepping : 4 cpu MHz : 2900.000 cache size : 15360 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida arat epb pln pts dts bogomips : 5800.00 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
As far as scenario: it's a mixture of REGISTER/MESSAGE/SUBSCRIBE/NOTIFY. I will send logs (configs + /var/log/messages) privately .. but generally speaking config is almost identical with configs from samples directory for p-cscf.
Dan
On 03/03/2014 04:02 PM, Jason Penton wrote:
Hi Daniel,
no we have not seen that... Can you provide details around the scenario
- registration, calling, mixture, etc? Load? We have done extensive testing
especially w.r.t. memory usage so I am very interested in this. Could you do a memory dump and send us the log?
Also, will you share your cfg file? you can send the cfg file directly to me if you prefer...
Cheers Jason
On Mon, Mar 3, 2014 at 10:58 PM, Daniel Ciprus <daniel.ciprus@acision.com
wrote:
Hi,
Has anybody seen this :
Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): *out of mem* Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer
followed by:
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): *out of mem* Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2168]: build_res_buf_from_sip_req(): *ERROR: build_res_buf_from_sip_req: out of memory ; needs 386*
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/contact.c:194]: parse_contacts(): parse_contacts(): *No memory left* Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:59]: contact_parser(): contact_parser(): Error while parsing contacts Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:88]: parse_contact(): parse_contact(): Error while parsing
However, seems like we still had enough mem:
[root@kamailio kamailio]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 9526348 230308 13282472 0 0 0 4 0 1 6 0 94 0 0 1 0 0 9526216 230308 13282524 0 0 0 0 1313 84424 17 8 75 0 0 1 0 0 9526092 230308 13282616 0 0 0 0 1482 86882 16 9 75 0 0
[root@kamailio monit.d]# kamailio -v version: kamailio 4.2.0-dev0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 17:42:42 Dec 18 2013 with gcc 4.4.7
Unfortunately I had to reload p-cscf to get service working again.
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
*Jason Penton* *Senior Manager: Applications and Services* *SmileCommunications Pty (Ltd)* *Mobile:* +27 (0) 83 283 7000 *Skype:* jason.barry.penton jason.penton@smilecoms.comname.surname@smilecoms.com www.smilecoms.com
This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
--
*Jason Penton**Senior Manager: Applications and Services* *Smile Communications Pty (Ltd)* *Mobile:* +27 (0) 83 283 7000*Skype:*jason.barry.penton jason.penton@smilecoms.com name.surname@smilecoms.com www.smilecoms.com
Binaries which I'm running are older than this commit, that's for sure. Thanks, I will upgrade as soon as I can.
On 03/07/2014 10:21 AM, Jason Penton wrote: Hey Daniel,
Please check this commit. I think you are running before this was added?
commit 1d6b1d8765d7a6d21c291d2076a3a80a1710c11c Author: Jason Penton <jason.penton@gmail.commailto:jason.penton@gmail.com> Date: Fri Feb 14 09:57:45 2014 +0200
lib/ims:modules/ims_charging,ims_icscf,ims_qos,ims_registrar_scscf: fixed pkg_mem leak - fixed leak when parsing msg headers on a shm msg - instead now pass in flag to say if parsing is on shm_msg or not (is_shm) - assumes all common headers are already parsed before pkg_mem msg is cloned
Cheers Jason
On Mon, Mar 3, 2014 at 11:22 PM, Jason Penton <jason.penton@smilecoms.commailto:jason.penton@smilecoms.com> wrote: Thanks Daniel,
How much pkg and shm memory is assigned to your Kamailio instance? Also, how many users are you registering concurrently? Please also recreate and dump shm memory when you get a chance. If you are not sure how to dump shm memory - see here - http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
Cheers Jason
On Mon, Mar 3, 2014 at 11:07 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: Jason,
HW configuration : this is a virtual machine running 4 cores with 24GB ram (VM hosting).
processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz stepping : 4 cpu MHz : 2900.000 cache size : 15360 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida arat epb pln pts dts bogomips : 5800.00 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
As far as scenario: it's a mixture of REGISTER/MESSAGE/SUBSCRIBE/NOTIFY. I will send logs (configs + /var/log/messages) privately .. but generally speaking config is almost identical with configs from samples directory for p-cscf.
Dan
On 03/03/2014 04:02 PM, Jason Penton wrote: Hi Daniel,
no we have not seen that... Can you provide details around the scenario - registration, calling, mixture, etc? Load? We have done extensive testing especially w.r.t. memory usage so I am very interested in this. Could you do a memory dump and send us the log?
Also, will you share your cfg file? you can send the cfg file directly to me if you prefer...
Cheers Jason
On Mon, Mar 3, 2014 at 10:58 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: Hi,
Has anybody seen this :
Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): out of mem Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer
followed by:
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): out of mem Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2168]: build_res_buf_from_sip_req(): ERROR: build_res_buf_from_sip_req: out of memory ; needs 386
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/contact.c:194]: parse_contacts(): parse_contacts(): No memory left Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:59]: contact_parser(): contact_parser(): Error while parsing contacts Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:88]: parse_contact(): parse_contact(): Error while parsing
However, seems like we still had enough mem:
[root@kamailio kamailio]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 9526348 230308 13282472 0 0 0 4 0 1 6 0 94 0 0 1 0 0 9526216 230308 13282524 0 0 0 0 1313 84424 17 8 75 0 0 1 0 0 9526092 230308 13282616 0 0 0 0 1482 86882 16 9 75 0 0
[root@kamailio monit.d]# kamailio -v version: kamailio 4.2.0-dev0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 17:42:42 Dec 18 2013 with gcc 4.4.7
Unfortunately I had to reload p-cscf to get service working again.
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.orgmailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Jason Penton Senior Manager: Applications and Services Smile Communications Pty (Ltd) Mobile: +27 (0) 83 283 7000tel:%2B27%20%280%29%2083%20283%207000 Skype: jason.barry.penton
jason.penton@smilecoms.commailto:name.surname@smilecoms.com www.smilecoms.comhttp://www.smilecoms.com/
This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
--
Jason Penton Senior Manager: Applications and Services Smile Communications Pty (Ltd) Mobile: +27 (0) 83 283 7000tel:%2B27%20%280%29%2083%20283%207000 Skype: jason.barry.penton
jason.penton@smilecoms.commailto:name.surname@smilecoms.com www.smilecoms.comhttp://www.smilecoms.com/ [http://196.33.227.129/%7Esmlcoms/sigs/pty/images/smile_signature_07_09.jpg]
--
Jason Penton Senior Manager: Applications and Services Smile Communications Pty (Ltd) Mobile: +27 (0) 83 283 7000 Skype: jason.barry.penton
jason.penton@smilecoms.commailto:name.surname@smilecoms.com www.smilecoms.comhttp://www.smilecoms.com/ [http://196.33.227.129/%7Esmlcoms/sigs/pty/images/smile_signature_07_09.jpg]
This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/
-- Daniel Ciprus Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com
________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
Great. Keep us posted
-- Sent using my phone and may be brief, to the point or contain typos -- On Mar 7, 2014 6:49 PM, "Daniel Ciprus" daniel.ciprus@acision.com wrote:
Binaries which I'm running are older than this commit, that's for sure. Thanks, I will upgrade as soon as I can.
On 03/07/2014 10:21 AM, Jason Penton wrote:
Hey Daniel,
Please check this commit. I think you are running before this was added?
commit 1d6b1d8765d7a6d21c291d2076a3a80a1710c11c Author: Jason Penton jason.penton@gmail.com Date: Fri Feb 14 09:57:45 2014 +0200
lib/ims:modules/ims_charging,ims_icscf,ims_qos,ims_registrar_scscf:fixed pkg_mem leak - fixed leak when parsing msg headers on a shm msg - instead now pass in flag to say if parsing is on shm_msg or not (is_shm) - assumes all common headers are already parsed before pkg_mem msg is cloned
Cheers Jason
On Mon, Mar 3, 2014 at 11:22 PM, Jason Penton jason.penton@smilecoms.comwrote:
Thanks Daniel,
How much pkg and shm memory is assigned to your Kamailio instance? Also, how many users are you registering concurrently? Please also recreate and dump shm memory when you get a chance. If you are not sure how to dump shm memory - see here - http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
Cheers Jason
On Mon, Mar 3, 2014 at 11:07 PM, Daniel Ciprus <daniel.ciprus@acision.com
wrote:
Jason,
HW configuration : this is a virtual machine running 4 cores with 24GB ram (VM hosting).
processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz stepping : 4 cpu MHz : 2900.000 cache size : 15360 KB fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida arat epb pln pts dts bogomips : 5800.00 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
As far as scenario: it's a mixture of REGISTER/MESSAGE/SUBSCRIBE/NOTIFY. I will send logs (configs + /var/log/messages) privately .. but generally speaking config is almost identical with configs from samples directory for p-cscf.
Dan
On 03/03/2014 04:02 PM, Jason Penton wrote:
Hi Daniel,
no we have not seen that... Can you provide details around the scenario - registration, calling, mixture, etc? Load? We have done extensive testing especially w.r.t. memory usage so I am very interested in this. Could you do a memory dump and send us the log?
Also, will you share your cfg file? you can send the cfg file directly to me if you prefer...
Cheers Jason
On Mon, Mar 3, 2014 at 10:58 PM, Daniel Ciprus < daniel.ciprus@acision.com> wrote:
Hi,
Has anybody seen this :
Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): *out of mem* Mar 2 17:25:16 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer
followed by:
Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): *out of mem* Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer Mar 2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2168]: build_res_buf_from_sip_req(): *ERROR: build_res_buf_from_sip_req: out of memory ; needs 386*
Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/contact.c:194]: parse_contacts(): parse_contacts(): *No memory left* Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:59]: contact_parser(): contact_parser(): Error while parsing contacts Mar 2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:88]: parse_contact(): parse_contact(): Error while parsing
However, seems like we still had enough mem:
[root@kamailio kamailio]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 9526348 230308 13282472 0 0 0 4 0 1 6 0 94 0 0 1 0 0 9526216 230308 13282524 0 0 0 0 1313 84424 17 8 75 0 0 1 0 0 9526092 230308 13282616 0 0 0 0 1482 86882 16 9 75 0 0
[root@kamailio monit.d]# kamailio -v version: kamailio 4.2.0-dev0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 17:42:42 Dec 18 2013 with gcc 4.4.7
Unfortunately I had to reload p-cscf to get service working again.
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
*Jason Penton* *Senior Manager: Applications and Services* *SmileCommunications Pty (Ltd)* *Mobile:* +27 (0) 83 283 7000 *Skype:* jason.barry.penton jason.penton@smilecoms.comname.surname@smilecoms.com www.smilecoms.com
This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
--
*Jason Penton* *Senior Manager: Applications and Services* *SmileCommunications Pty (Ltd)* *Mobile:* +27 (0) 83 283 7000 *Skype:* jason.barry.penton jason.penton@smilecoms.comname.surname@smilecoms.com www.smilecoms.com
--
*Jason Penton* *Senior Manager: Applications and Services* *SmileCommunications Pty (Ltd)* *Mobile:* +27 (0) 83 283 7000 *Skype:* jason.barry.penton jason.penton@smilecoms.comname.surname@smilecoms.com www.smilecoms.com
This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/
-- *Daniel Ciprus* Integration engineer http://www.acision.com
9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.
-
Daniel Ciprus -
Jason Penton