21 Jun
2004
21 Jun
'04
8:47 p.m.
Some more update.. got the same problem today.. and when it happend, ser
blocked. and I got ALOT of these error messages. Anybody know why ?
-Atle
VOIP /usr/local/sbin/ser[1677]: ERROR: new_t: out
of mem:
VOIP /usr/local/sbin/ser[1677]: ERROR: t_newtran:
new_t failed
VOIP /usr/local/sbin/ser[1677]: ERROR:
sl_reply_error used: I'm terribly sorry, server error occured (2/SL)
* Atle Samuelsen <clona(a)camaro.no> [040621 07:35]:
Hi Guys and Girls.
I ran into a problem a couple of days ago, when trying to restart ser.
The scenario was that ser was sending out ALOT of 500 "I'm sorry... ". I
found it then best to restart ser, to see if it stoped. When I then was
going to start ser again (via ser's rc scripts) It did'nt want to start.
Here's the messages in /var/log/messages. (First it looked like ser
started normaly)
VOIP /usr/local/sbin/ser[1302]: mod_init(): Database connection opened successfuly
VOIP ser: ser startup succeeded
VOIP /usr/local/sbin/ser[1302]: convert_row(): No memory left
VOIP /usr/local/sbin/ser[1302]: convert_rows(): Error while converting row #6256
VOIP /usr/local/sbin/ser[1302]: convert_result(): Error while converting rows
VOIP /usr/local/sbin/ser[1302]: get_result(): Error while converting result
VOIP /usr/local/sbin/ser[1302]: preload_udomain(): Error while doing db_query
VOIP /usr/local/sbin/ser[1302]: register_udomain(): Error while preloading domain
'location'
VOIP /usr/local/sbin/ser[1302]: domain_fixup(): Error while registering domain
VOIP /usr/local/sbin/ser[1302]: ERROR: fix_expr : fix_actions error
To get ser upandRunning again, I had to delete all the entries in
mysql::ser::location.
Have you guys ever seen anything like this before?
- Atle
_______________________________________________
Serusers mailing list
serusers(a)lists.iptel.org
http://lists.iptel.org/mailman/listinfo/serusers
22 Jun
22 Jun
1:06 p.m.
New subject: [Serusers] Weird problem when, restarting ser.
Hi,
Atle Samuelsen wrote:
Some more update.. got the same problem today.. and
when it happend, ser
blocked. and I got ALOT of these error messages. Anybody know why ?
Are there any subscribers with a lot of location entries? I had this
problem some months ago with Ser 0.8.10 IIRC, and the reason was a user
accidently registering very often (~500 times or so) with different IPs
because of a DHCP issue. After deleting his userlocs it started up again...
Andy
PS: I currently watch the number of locations per user with an external
script, but in general a user should only be allowed to register one
location here, so maybe I should think about some sort of check when
registering a client. Are there any out-of-the-box solutions or is this
another task for an external script or extra module?
1:58 p.m.
New subject: [Serusers] Weird problem when, restarting ser.
On Jun 21, 2004 at 19:47, Atle Samuelsen <clona(a)cyberhouse.no> wrote:
Some more update.. got the same problem today.. and
when it happend, ser
blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local
(per process memory) and copied and inserted in the usrloc shared memory
list. (probably you have a lot of entries)
By default ser uses maximum 1Mb of local memory/process. Try to
increases it to a bigger value.
Change PKG_MEM_POOL_SIZE in config.h and recompile.
Andrei
24 Jun
24 Jun
1:54 p.m.
New subject: [Serusers] Weird problem when, restarting ser.
Andrei Pelinescu-Onciul wrote:
On Jun 21, 2004 at 19:47, Atle Samuelsen
<clona(a)cyberhouse.no> wrote:
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR:
build_req_buf_from_sip_req: out of memory
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR:
print_uac_request: no pkg_mem
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR:
t_forward_nonack: failure to add branches
I thought that maybe a restart could help, but SER didn't come up again,
shouting:
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_row(): No
memory left
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_rows():
Error while converting row #6346
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_result():
Error while converting rows
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: get_result(): Error
while converting result
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: preload_udomain():
Error while doing db_query
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: register_udomain():
Error while preloading domain 'location'
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: domain_fixup():
Error while registering domain
After deleting 100 entries from location it came up again. Hope that the
increase of PKG_MEM_POOL_SIZE to 2MB helps.
So it seems that after about 6300 entries or so in location Ser stops
working with the default configuration!? I'd be interested in how many
users do other Ser-Servers out there serve? Are there similiar problems?
Regards,
Andy
Some more update.. got the same problem today..
and when it happend, ser
blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local
(per process memory) and copied and inserted in the usrloc shared memory
list. (probably you have a lot of entries)
By default ser uses maximum 1Mb of local memory/process. Try to
increases it to a bigger value.
Change PKG_MEM_POOL_SIZE in config.h and recompile.
3:14 p.m.
New subject: [Serusers] Weird problem when, restarting ser.
On Jun 24, 2004 at 12:54, Andreas Granig <a.granig(a)inode.at> wrote:
Andrei Pelinescu-Onciul wrote:
[...]
On Jun 21, 2004 at 19:47, Atle Samuelsen
<clona(a)cyberhouse.no> wrote:
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
Some more update.. got the same problem today..
and when it happend, ser
blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local
(per process memory) and copied and inserted in the usrloc shared memory
list. (probably you have a lot of entries)
By default ser uses maximum 1Mb of local memory/process. Try to
increases it to a bigger value.
Change PKG_MEM_POOL_SIZE in config.h and recompile.
After deleting 100 entries from location it came up again. Hope that the
increase of PKG_MEM_POOL_SIZE to 2MB helps.
You'll be ok until you reach twice the location table size you have
right now.
So it seems that after about 6300 entries or so in location Ser stops
working with the default configuration!? I'd be interested in how many
users do other Ser-Servers out there serve? Are there similiar problems?
Yes there are and you are right they usually start at ~6000 entries.
We will either increase the default temp. memory size or we will switch
to shared memory for loading the locations tables on startup. The ideea
behind the local memory is to have some kind of private per process,
very fast to allocate memory space, used mainly for temporary stuff.
However in the usrloc case since the tables are loaded on startup we
don't care so much about the performance and we could switch to shared
memory (which is by default 32Mb and can be changed with the command
line switch -m).
Andrei
25 Jun
25 Jun
12:26 a.m.
New subject: [Serusers] Weird problem when, restarting ser.
Thanks Andrei :-)
- Atle
* Andrei Pelinescu-Onciul <pelinescu-onciul(a)fokus.fraunhofer.de> [040624 14:14]:
On Jun 24, 2004 at 12:54, Andreas Granig
<a.granig(a)inode.at> wrote:
Andrei Pelinescu-Onciul wrote:
[...]
On Jun 21, 2004 at 19:47, Atle Samuelsen
<clona(a)cyberhouse.no> wrote:
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
Some more update.. got the same problem today..
and when it happend, ser
blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local
(per process memory) and copied and inserted in the usrloc shared memory
list. (probably you have a lot of entries)
By default ser uses maximum 1Mb of local memory/process. Try to
increases it to a bigger value.
Change PKG_MEM_POOL_SIZE in config.h and recompile.
After deleting 100 entries from location it came up again. Hope that the
increase of PKG_MEM_POOL_SIZE to 2MB helps.
You'll be ok until you reach twice the location table size you have
right now.
So it seems that after about 6300 entries or so in location Ser stops
working with the default configuration!? I'd be interested in how many
users do other Ser-Servers out there serve? Are there similiar problems?
Yes there are and you are right they usually start at ~6000 entries.
We will either increase the default temp. memory size or we will switch
to shared memory for loading the locations tables on startup. The ideea
behind the local memory is to have some kind of private per process,
very fast to allocate memory space, used mainly for temporary stuff.
However in the usrloc case since the tables are loaded on startup we
don't care so much about the performance and we could switch to shared
memory (which is by default 32Mb and can be changed with the command
line switch -m).
Andrei
1:30 a.m.
New subject: [Serusers] Weird problem when, restarting ser.
Under normal operation, SER can handle significantly more registrations, by
orders of magnitude. To me, it sounds much more like a result of a memory leak.
Are you using stock SER or some modified SER? In any case, the way to catch such
leaks is to set a talkative memory log level (memlog=2 in ser.cfg), recompile ser
with memory debugging informaiton (uncomment DBG_QM_MALLOC, comment F_MALLOC
in Makefile.defs and do recompile completely) and watch log output on SER shutdown
for excessive occurent of unallocated memory fragement. I would be very thankful
if you could report your observations back to us.
-jiri
At 12:54 PM 6/24/2004, Andreas Granig wrote:
Andrei Pelinescu-Onciul wrote:
--
Jiri Kuthan http://iptel.org/~jiri/
On Jun 21, 2004 at 19:47, Atle Samuelsen
<clona(a)cyberhouse.no> wrote:
D'oh, same problem again here now (Ser 0.8.12)!
It started with log entries like this:
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: build_req_buf_from_sip_req:
out of memory
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no pkg_mem
Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure to
add branches
I thought that maybe a restart could help, but SER didn't come up again, shouting:
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_row(): No memory left
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_rows(): Error while
converting row #6346
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_result(): Error while
converting rows
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: get_result(): Error while converting
result
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: preload_udomain(): Error while doing
db_query
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: register_udomain(): Error while
preloading domain 'location'
Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: domain_fixup(): Error while
registering domain
After deleting 100 entries from location it came up again. Hope that the increase of
PKG_MEM_POOL_SIZE to 2MB helps.
So it seems that after about 6300 entries or so in location Ser stops working with the
default configuration!? I'd be interested in how many users do other Ser-Servers out
there serve? Are there similiar problems?
Regards,
Andy
_______________________________________________
Serusers mailing list
serusers(a)lists.iptel.org
http://lists.iptel.org/mailman/listinfo/serusers Some more update.. got the same problem today..
and when it happend, ser
blocked. and I got ALOT of these error messages. Anybody know why ?
Ser runs out of "local" memory. The localtions are first loaded in local
(per process memory) and copied and inserted in the usrloc shared memory
list. (probably you have a lot of entries)
By default ser uses maximum 1Mb of local memory/process. Try to
increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and recompile.
1:43 a.m.
New subject: [Serusers] Weird problem when, restarting ser.
On Jun 25, 2004 at 00:30, Jiri Kuthan <jiri(a)iptel.org> wrote:
Under normal operation, SER can handle significantly
more registrations, by
orders of magnitude. To me, it sounds much more like a result of a memory leak.
Jiri, note that this happens on startup. In fact it runs out of mem.
when it tries to alloc. a chunk of memory big enough to hold all the
entries in location. This is not a memory leak and it really happens if
PKG_MEM_POOL_SIZE is 1Mb and you have >6000 entries in location (it
depends of course on your average entry size, but the critical point is
in general between 6000-7000).
Are you using stock SER or some modified SER? In any
case, the way to catch such
leaks is to set a talkative memory log level (memlog=2 in ser.cfg), recompile ser
with memory debugging informaiton (uncomment DBG_QM_MALLOC, comment F_MALLOC
in Makefile.defs and do recompile completely) and watch log output on SER shutdown
for excessive occurent of unallocated memory fragement. I would be very thankful
if you could report your observations back to us.
-jiri
At 12:54 PM 6/24/2004, Andreas Granig wrote:
>Andrei Pelinescu-Onciul wrote:
>>On Jun 21, 2004 at 19:47, Atle Samuelsen <clona(a)cyberhouse.no> wrote:
>>
>>>Some more update.. got the same problem today.. and when it happend, ser
>>>blocked. and I got ALOT of these error messages. Anybody know why ?
>>
>>Ser runs out of "local" memory. The localtions are first loaded in
local
>>(per process memory) and copied and inserted in the usrloc shared memory
>>list. (probably you have a lot of entries)
>>By default ser uses maximum 1Mb of local memory/process. Try to
>>increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and
recompile.
>
>D'oh, same problem again here now (Ser 0.8.12)!
>
>It started with log entries like this:
>
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR:
build_req_buf_from_sip_req: out of memory
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no
pkg_mem
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure
to add branches
This could be a mem. leak (but this is unrelated to the startup
problem).
>
>I thought that maybe a restart could help, but SER didn't come up again,
shouting:
>
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_row(): No memory left
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_rows(): Error while
converting row #6346
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_result(): Error while
converting rows
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: get_result(): Error while
converting result
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: preload_udomain(): Error while
doing db_query
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: register_udomain(): Error while
preloading domain 'location'
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: domain_fixup(): Error while
registering domain
>
>After deleting 100 entries from location it came up again. Hope that the increase of
PKG_MEM_POOL_SIZE to 2MB helps.
>
>So it seems that after about 6300 entries or so in location Ser stops working with
the default configuration!? I'd be interested in how many users do other Ser-Servers
out there serve? Are there similiar problems?
>
>Regards,
>Andy
>
Andrei
1:45 a.m.
New subject: [Serusers] Weird problem when, restarting ser.
At 12:43 AM 6/25/2004, Andrei Pelinescu-Onciul wrote:
On Jun 25, 2004 at 00:30, Jiri Kuthan
<jiri(a)iptel.org> wrote:
oh thanks -- I misread. -jiri
Under normal operation, SER can handle
significantly more registrations, by
orders of magnitude. To me, it sounds much more like a result of a memory leak.
Jiri, note that this happens on startup. In fact it runs out of mem. when it tries to alloc. a chunk of memory big enough to
hold all the
entries in location. This is not a memory leak and it really happens if
PKG_MEM_POOL_SIZE is 1Mb and you have >6000 entries in location (it
depends of course on your average entry size, but the critical point is
in general between 6000-7000).
--
Jiri Kuthan http://iptel.org/~jiri/
Are you using stock SER or some modified SER? In
any case, the way to catch such
leaks is to set a talkative memory log level (memlog=2 in ser.cfg), recompile ser
with memory debugging informaiton (uncomment DBG_QM_MALLOC, comment F_MALLOC
in Makefile.defs and do recompile completely) and watch log output on SER shutdown
for excessive occurent of unallocated memory fragement. I would be very thankful
if you could report your observations back to us.
-jiri
At 12:54 PM 6/24/2004, Andreas Granig wrote:
>Andrei Pelinescu-Onciul wrote:
>>On Jun 21, 2004 at 19:47, Atle Samuelsen <clona(a)cyberhouse.no> wrote:
>>
>>>Some more update.. got the same problem today.. and when it happend, ser
>>>blocked. and I got ALOT of these error messages. Anybody know why ?
>>
>>Ser runs out of "local" memory. The localtions are first loaded in
local
>>(per process memory) and copied and inserted in the usrloc shared memory
>>list. (probably you have a lot of entries)
>>By default ser uses maximum 1Mb of local memory/process. Try to
>>increases it to a bigger value. Change PKG_MEM_POOL_SIZE in config.h and
recompile.
>
>D'oh, same problem again here now (Ser 0.8.12)!
>
>It started with log entries like this:
>
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR:
build_req_buf_from_sip_req: out of memory
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no
pkg_mem
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure
to add branches
This could be a mem. leak (but this is unrelated to the startup
problem).
>
>I thought that maybe a restart could help, but SER didn't come up again,
shouting:
>
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_row(): No memory left
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_rows(): Error while
converting row #6346
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: convert_result(): Error while
converting rows
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: get_result(): Error while
converting result
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: preload_udomain(): Error while
doing db_query
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: register_udomain(): Error while
preloading domain 'location'
>Jun 24 12:01:02 voip /usr/local/ser/sbin/ser[19117]: domain_fixup(): Error while
registering domain
>
>After deleting 100 entries from location it came up again. Hope that the increase of
PKG_MEM_POOL_SIZE to 2MB helps.
>
>So it seems that after about 6300 entries or so in location Ser stops working with
the default configuration!? I'd be interested in how many users do other Ser-Servers
out there serve? Are there similiar problems?
>
>Regards,
>Andy
>
Andrei
3:14 a.m.
New subject: [Serusers] Weird problem when, restarting ser.
Andrei Pelinescu-Onciul wrote:
I use a stock SER (0.8.12) without modifications. I just execute some
external C code by exec_dset and exec_msg, but this shouldn't matter.
Since this problem occured on our public server, I just wanted to get
the service up and running again and thus didn't think about dumping the
database for later analysis on the test system. However, I'll try to
reproduce this effect and send some logs...
Regards,
Andy
>It started
with log entries like this:
>
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR:
build_req_buf_from_sip_req: out of memory
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: print_uac_request: no
pkg_mem
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR: t_forward_nonack: failure
to add branches
This could be a mem. leak (but this is unrelated to the startup
problem).
3:28 a.m.
New subject: [Serusers] Weird problem when, restarting ser.
Hi Andy
* Andreas Granig <a.granig(a)inode.at> [040625 02:19]:
Andrei Pelinescu-Onciul wrote:
I use a stock SER (0.8.12) without modifications. I just execute some
external C code by exec_dset and exec_msg, but this shouldn't matter.
Since this problem occured on our public server, I just wanted to get
the service up and running again and thus didn't think about dumping the
database for later analysis on the test system. However, I'll try to
reproduce this effect and send some logs...
I would check with Andrei and see what he says.
-Atle
>It
started with log entries like this:
>
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR:
>build_req_buf_from_sip_req: out of memory
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR:
>print_uac_request: no pkg_mem
>Jun 24 11:55:54 voip /usr/local/ser/sbin/ser[16801]: ERROR:
>t_forward_nonack: failure to add branches
This could be a mem. leak (but this is unrelated to the startup
problem).
7457
days inactive
7461
days old
10 comments
5 participants
participants (5)
-
Andreas Granig -
Andrei Pelinescu-Onciul -
Atle Samuelsen -
Atle Samuelsen -
Jiri Kuthan