[SR-Users] p-cscf - out of mem

Fri Mar 7 17:49:29 CET 2014

Binaries which I'm running are older than this commit, that's for sure. Thanks, I will upgrade as soon as I can.

On 03/07/2014 10:21 AM, Jason Penton wrote:
Hey Daniel,

Please check this commit. I think you are running before this was added?

commit 1d6b1d8765d7a6d21c291d2076a3a80a1710c11c
Author: Jason Penton <jason.penton at gmail.com<mailto:jason.penton at gmail.com>>
Date:   Fri Feb 14 09:57:45 2014 +0200

    lib/ims:modules/ims_charging,ims_icscf,ims_qos,ims_registrar_scscf: fixed pkg_mem leak
        - fixed leak when parsing msg headers on a shm msg
        - instead now pass in flag to say if parsing is on shm_msg or not (is_shm)
        - assumes all common headers are already parsed before pkg_mem msg is cloned

Cheers
Jason

On Mon, Mar 3, 2014 at 11:22 PM, Jason Penton <jason.penton at smilecoms.com<mailto:jason.penton at smilecoms.com>> wrote:
Thanks Daniel,

How much pkg and shm memory is assigned to your Kamailio instance? Also, how many users are you registering concurrently? Please also recreate and dump shm memory when you get a chance. If you are not sure how to dump shm memory - see here - http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory

Cheers
Jason

On Mon, Mar 3, 2014 at 11:07 PM, Daniel Ciprus <daniel.ciprus at acision.com<mailto:daniel.ciprus at acision.com>> wrote:
Jason,

HW configuration : this is a virtual machine running 4 cores with 24GB ram (VM hosting).

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz
stepping        : 4
cpu MHz         : 2900.000
cache size      : 15360 KB
fpu     : yes
fpu_exception   : yes
cpuid level     : 11
wp      : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm ida arat epb pln pts dts
bogomips        : 5800.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

As far as scenario: it's a mixture of REGISTER/MESSAGE/SUBSCRIBE/NOTIFY. I will send logs (configs + /var/log/messages) privately .. but generally speaking config is almost identical with configs from samples directory for p-cscf.

Dan

On 03/03/2014 04:02 PM, Jason Penton wrote:
Hi Daniel,

no we have not seen that... Can you provide details around the scenario - registration, calling, mixture, etc? Load? We have done extensive testing especially w.r.t. memory usage so I am very interested in this. Could you do a memory dump and send us the log?

Also, will you share your cfg file? you can send the cfg file directly to me if you prefer...

Cheers
Jason

On Mon, Mar 3, 2014 at 10:58 PM, Daniel Ciprus <daniel.ciprus at acision.com<mailto:daniel.ciprus at acision.com>> wrote:
Hi,

Has anybody seen this :

Mar  2 17:25:16 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): out of mem
Mar  2 17:25:16 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer

followed by:

Mar  2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2011]: generate_res_buf_from_sip_res(): out of mem
Mar  2 17:27:03 kamailio kam-pcscf[20741]: ERROR: tm [t_reply.c:1943]: relay_reply(): ERROR: relay_reply: no mem for outbound reply buffer
Mar  2 17:27:03 kamailio kam-pcscf[20741]: ERROR: <core> [msg_translator.c:2168]: build_res_buf_from_sip_req(): ERROR: build_res_buf_from_sip_req: out of memory  ; needs 386

Mar  2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/contact.c:194]: parse_contacts(): parse_contacts(): No memory left
Mar  2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:59]: contact_parser(): contact_parser(): Error while parsing contacts
Mar  2 17:27:53 kamailio kam-pcscf[20741]: ERROR: <core> [parser/contact/parse_contact.c:88]: parse_contact(): parse_contact(): Error while parsing

However, seems like we still had enough mem:

[root at kamailio kamailio]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 9526348 230308 13282472    0    0     0     4    0    1  6  0 94  0  0
 1  0      0 9526216 230308 13282524    0    0     0     0 1313 84424 17  8 75  0  0
 1  0      0 9526092 230308 13282616    0    0     0     0 1482 86882 16  9 75  0  0

[root at kamailio monit.d]# kamailio -v
version: kamailio 4.2.0-dev0 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 17:42:42 Dec 18 2013 with gcc 4.4.7

Unfortunately I had to reload p-cscf to get service working again.

--
Daniel Ciprus
Integration engineer
http://www.acision.com

9954 Mayland Dr
Suite 3100
Richmond, VA 23233
USA
T: +1 804 762 5601<tel:%2B1%20804%20762%205601>
E: daniel.ciprus at acision.com<mailto:daniel.ciprus at acision.com>

________________________________
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users at lists.sip-router.org<mailto:sr-users at lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

--

Jason Penton
Senior Manager: Applications and Services
Smile Communications Pty (Ltd)
Mobile: +27 (0) 83 283 7000<tel:%2B27%20%280%29%2083%20283%207000>
Skype:  jason.barry.penton

jason.penton at smilecoms.com<mailto:name.surname at smilecoms.com>
www.smilecoms.com<http://www.smilecoms.com/>

This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/

--
Daniel Ciprus
Integration engineer
http://www.acision.com

9954 Mayland Dr
Suite 3100
Richmond, VA 23233
USA
T: +1 804 762 5601<tel:%2B1%20804%20762%205601>
E: daniel.ciprus at acision.com<mailto:daniel.ciprus at acision.com>

________________________________
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

--

Jason Penton
Senior Manager: Applications and Services
Smile Communications Pty (Ltd)
Mobile: +27 (0) 83 283 7000<tel:%2B27%20%280%29%2083%20283%207000>
Skype:  jason.barry.penton

jason.penton at smilecoms.com<mailto:name.surname at smilecoms.com>
www.smilecoms.com<http://www.smilecoms.com/>
[http://196.33.227.129/%7Esmlcoms/sigs/pty/images/smile_signature_07_09.jpg]

--

Jason Penton
Senior Manager: Applications and Services
Smile Communications Pty (Ltd)
Mobile: +27 (0) 83 283 7000
Skype:  jason.barry.penton

jason.penton at smilecoms.com<mailto:name.surname at smilecoms.com>
www.smilecoms.com<http://www.smilecoms.com/>
[http://196.33.227.129/%7Esmlcoms/sigs/pty/images/smile_signature_07_09.jpg]

This email is subject to the disclaimer of Smile Communications at http://www.smilecoms.com/home/email-disclaimer/

--
Daniel Ciprus
Integration engineer
http://www.acision.com

9954 Mayland Dr
Suite 3100
Richmond, VA 23233
USA
T: +1 804 762 5601
E: daniel.ciprus at acision.com<mailto:daniel.ciprus at acision.com>

________________________________
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140307/833cbd18/attachment-0001.html>