Hello,
 
We have exactly the same problem. Openser after a few hours starts loging "out of memory" errors and soon after that it cannot process any messages. I am using 1.2 branch from SVN and the last export that I got was at the begining of June, the revision number was 2333. Today I was able to catch the problem while it is happening and I did a memory dump on the process that run out of memory (process 4197). ps shows that this process is a receiver with id=1 and child=0. I have not restarted the server yet, but pretty soon I will have to restart because more processes will run out of memory. If there is some additional information (statistics, ngrep etc) that I could gather while the server is still running, please let me know. I don't have quick access to a public ftp site, so if someone can take a look at the log file and the memory dump, please let me know and I will email it (1MB).
 
thank you for any help.
 
George
 
 
 
 
Date: Mon, 11 Jun 2007 17:52:08 +0200
From: Klaus Darilion <klaus.mailinglists@pernau.at>
Subject: Re: [Users] Memory issues
To: Christian <mailing-lists@switzernet.com>
Cc: users@openser.org
Message-ID: <466D6FA8.9010201@pernau.at>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 
 
Christian wrote:
> Thank you Bogdan,
>
> For the moment I am not very sure to switch to SVN branch (this is a
> production server, so every minor bug can be dramatic)
SVN != unstable
SVN is the code repository. In the repository there are different branches. E.g. there is
1.1: openser 1.1 branch
1.2: openser 1.2 branch
trunk: openser development branch
Thus, you should update to the 1.2 branch from SVN. This is the currents table release with all bugfixes:
> But I will at least upgrade to 1.2.1 asap, and see if the issue gets
> solved.
1.2.1 is a snapshot taken from 1.2 branch
regards
klaus
> The main issue here is that the memory leak doesn't crash openser, but
> puts it in a state where messages are accepted but not relayed.
> Shouldn't it rather send a SIP error reply (500 I guess), so that
> handover on other servers gets properly handled?
>
> Regards,
> Christian
>
>
>
> Bogdan-Andrei Iancu a écrit :
>> it looks like PKG memory problem.
>> Christian, try to update to the latest 1.2 version from SVN and see
>> if the problem disappears - there were some pkg mem leak fixed
>> since the 1.2 release.
>>
>> regards,
>> bogdan
>>
>>
>> Klaus Darilion wrote:
>>> http://www.openser.org/dokuwiki/doku.php/troubleshooting:memory
>>>
>>> Make sure to use newest 1.2 (make a SVN checkout of /branches/1.2/ )
>>>
>>> regards
>>> klaus
>>>
>>> Christian wrote:
>>>> Hi All,
>>>>
>>>> I have a recurrent memory problem with openser 1.2.0. In this case
>>>> it only acts as a SIP router. No registration, accounting etc. and
>>>> only approx. 20 simultaneous calls.
>>>>
>>>> Here is the problem: everything works fine (calls from/to PSTN
>>>> through different SIP servers), but after a few days, "out of
>>>> memory" errors start to occur. After this, calls don't work
>>>> anymore, even if openser continues to run.
>>>>
>>>> Here is a part of the log, showing the first errors:
>>>>
>>>> Jun 5 17:25:43 localhost SER[27772]: ERROR:
>>>> build_res_buf_from_sip_res: out of mem Jun 5 17:25:43 localhost
>>>> SER[27772]: ERROR:tm:relay_reply: no mem for outbound reply buffer
>>>> Jun 5 17:25:43 localhost SER[27772]: ERROR: via_builder: out of
>>>> memory Jun 5 17:25:43 localhost SER[27772]: ERROR: build_local: no
>>>> via header got from builder Jun 5 17:25:43 localhost SER[27772]:
>>>> ERROR: attempt to build a CANCEL failed Jun 5 17:25:50 localhost
>>>> SER[27772]: ERROR:
>>>> build_res_buf_from_sip_res: out of mem Jun 5 17:25:50 localhost
>>>> SER[27772]: ERROR:tm:relay_reply: no mem for outbound reply buffer
>>>> Jun 5 17:25:50 localhost SER[27772]: ERROR:
>>>> build_res_buf_from_sip_req: out of memory ; needs 410 Jun 5
>>>> 17:26:11 localhost SER[27772]: ERROR:
>>>> build_res_buf_from_sip_res: out of mem Jun 5 17:26:11 localhost
>>>> SER[27772]: ERROR:tm:relay_reply: no mem for outbound reply buffer
>>>> Jun 5 17:26:11 localhost SER[27772]: ERROR:
>>>> build_res_buf_from_sip_req: out of memory ; needs 404 Jun 5
>>>> 17:26:25 localhost SER[27772]: eval_elem: no more memory Jun 5
>>>> 17:26:25 localhost SER[27772]: do_assign: no value in right
>>>> expression Jun 5 17:26:25 localhost SER[27772]: ERROR:
>>>> build_res_buf_from_sip_res: out of mem Jun 5 17:26:25 localhost
>>>> SER[27772]: ERROR:tm:relay_reply: no mem for outbound reply buffer
>>>> Jun 5 17:26:25 localhost SER[27772]: ERROR:
>>>> build_res_buf_from_sip_req: out of memory ; needs 420 Jun 5
>>>> 17:26:25 localhost SER[27771]: ERROR:forward_reply: no 2nd via
>>>> found in reply Jun 5 17:26:31 localhost SER[27772]: ERROR:
>>>> build_res_buf_from_sip_res: out of mem Jun 5 17:26:31 localhost
>>>> SER[27772]: ERROR:forward_reply: building rpl from req failed Jun
>>>> 5 17:26:33 localhost SER[27772]: ERROR:
>>>> build_res_buf_from_sip_res: out of mem Jun 5 17:26:33 localhost
>>>> SER[27772]: ERROR:forward_reply: building rpl from req failed Jun
>>>> 5 17:26:38 localhost SER[27772]: ERROR: insert_new_lump_after:
>>>> out of memory
>>>> Jun 5 17:26:38 localhost SER[27772]: ERROR: insert_new_lump_before:
>>>> out of memory
>>>> Jun 5 17:26:38 localhost SER[27772]: insert_RR(): Error while
>>>> inserting conditional lump Jun 5 17:26:38 localhost SER[27772]:
>>>> ERROR: subst_str: mem.
>>>> allocation error (res->s)
>>>> Jun 5 17:26:38 localhost SER[27772]: ERROR: do_action: memory
>>>> allocation failure Jun 5 17:26:38 localhost SER[27772]: ERROR:
>>>> build_res_buf_from_sip_req: out of memory ; needs 375 Jun 5
>>>> 17:26:38 localhost SER[27772]: ERROR: mk_proxy: memory allocation
>>>> failure Jun 5 17:26:38 localhost SER[27772]:
>>>> ERROR:tm:t_forward_nonack:
>>>> failure to add branches
>>>>
>>>> At this point, similar errors continue to appear and the only way
>>>> to recover is to restart openser.
>>>> Nothing unusual seems to cause the initial error. The output of
>>>> "top" appears to be fine after the "crash", and there is still
>>>> plenty free memory on the server.
>>>>
>>>>
>>>> Could anybody give me some hints on how to solve the problem? I
>>>> suspect a portion of the config file where the record-route headers
>>>> are split on multiple lines (for compatibility issues with another
>>>> SIP proxy). It's the only place where variables are used. You can
>>>> find it here:
>>>>
>>>> http://switzernet.com/people/christian-lathion/files/070605-openser
>>>> -config-part.txt
>>>>
>>>>
>>>> Otherwise the config file is quite usual, with basic processing and
>>>> routing of SIP packets.
>>>>
>>>> Regards,
>>>> Christian
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users@openser.org
>>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users@openser.org
>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>>
>>
>
> _______________________________________________
> Users mailing list
> Users@openser.org
> http://openser.org/cgi-bin/mailman/listinfo/users
 

Disclaimer

The information in this e-mail and any attachments is confidential. It is intended solely for the attention and use of the named addressee(s). If you are not the intended recipient, or person responsible for delivering this information to the intended recipient, please notify the sender immediately. Unless you are the intended recipient or his/her representative you are not authorized to, and must not, read, copy, distribute, use or retain this message or any part of it. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.