Hi Cesc!

Where do you have t_newtran() - in ser or openser?

I have it here in openser just before handle_publish(). It works but now openser has run out of mem during my sipp performance tests :-(

...more to come....

Cesc wrote:

I think I can answer that one ... stateless?

I have a presence openser proxied by a SER, both on same machine ... For easier configuration purposes, openser and ser communicate via 127.0.0.1 ...

The result is that OK from the PUAs do not get to openser (i guess SER gets confussed by the vias of openser being 0.0.0.0:5060 and the 127.0.0.1 stuff) ... So, openser gets no OK, but it does not matter ... BTW, i have a t_newtran() in there ...

Oh, for sure my configuration is messy and improvable ... but for now it works steady :) But the OKs not going through is worrying me a bit ... and also openser not "timing out" or giving an error.

Cesc

On 4/25/07, Klaus Darilion klaus.mailinglists@pernau.at wrote:

...

Hi!

Does presence module handles PUBLISH/SUBSCRIBE stateful or stateless?

Thus, how should retransmission be handled. Will they be detected by presence module or is it possible to wrap the handle_publish inside t_newtran()?

regards klaus

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

Hi!

Openser 1.2 runs out of memory during presence load tests with sipp.

Config snippet:

if (is_method("PUBLISH")) { xlog("L_INFO","$ci PUBLISH detected ... outside t_newtran\n"); if (t_newtran()) { xlog("L_INFO","$ci PUBLISH detected ... inside t_newtran\n"); handle_publish(); xlog("L_INFO","$ci PUBLISH detected, handle_publish ... done\n"); exit; } else { sl_reply_error(); exit; } }

Memory log after openser stop (after 20 minutes of inactivity): http://pernau.at/kd/openser/debug-stop.txt

regards klaus

Klaus Darilion wrote:

Hi Cesc!

Where do you have t_newtran() - in ser or openser?

I have it here in openser just before handle_publish(). It works but now openser has run out of mem during my sipp performance tests :-(

...more to come....

Cesc wrote:

...

I think I can answer that one ... stateless?

I have a presence openser proxied by a SER, both on same machine ... For easier configuration purposes, openser and ser communicate via 127.0.0.1 ...

The result is that OK from the PUAs do not get to openser (i guess SER gets confussed by the vias of openser being 0.0.0.0:5060 and the 127.0.0.1 stuff) ... So, openser gets no OK, but it does not matter ... BTW, i have a t_newtran() in there ...

Oh, for sure my configuration is messy and improvable ... but for now it works steady :) But the OKs not going through is worrying me a bit ... and also openser not "timing out" or giving an error.

Cesc

On 4/25/07, Klaus Darilion klaus.mailinglists@pernau.at wrote:

...

Hi!

Does presence module handles PUBLISH/SUBSCRIBE stateful or stateless?

Thus, how should retransmission be handled. Will they be detected by presence module or is it possible to wrap the handle_publish inside t_newtran()?

regards klaus

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

Hi Anca!

There is still a leak: http://pernau.at/kd/openser/debug-stop2.txt

regards klaus

Anca-Maria Vamanu wrote:

Hello,

I have made a fix. Please test again with sipp and say if it works now.

regards,

Anca

Klaus Darilion wrote:

...

Hi!

Openser 1.2 runs out of memory during presence load tests with sipp.

Config snippet:

if (is_method("PUBLISH")) { xlog("L_INFO","$ci PUBLISH detected ... outside t_newtran\n"); if (t_newtran()) { xlog("L_INFO","$ci PUBLISH detected ... inside t_newtran\n"); handle_publish(); xlog("L_INFO","$ci PUBLISH detected, handle_publish ... done\n"); exit; } else { sl_reply_error(); exit; } }

Memory log after openser stop (after 20 minutes of inactivity): http://pernau.at/kd/openser/debug-stop.txt

regards klaus

Klaus Darilion wrote:

...

Hi Cesc!

Where do you have t_newtran() - in ser or openser?

I have it here in openser just before handle_publish(). It works but now openser has run out of mem during my sipp performance tests :-(

...more to come....

Cesc wrote:

...

I think I can answer that one ... stateless?

I have a presence openser proxied by a SER, both on same machine ... For easier configuration purposes, openser and ser communicate via 127.0.0.1 ...

The result is that OK from the PUAs do not get to openser (i guess SER gets confussed by the vias of openser being 0.0.0.0:5060 and the 127.0.0.1 stuff) ... So, openser gets no OK, but it does not matter ... BTW, i have a t_newtran() in there ...

Oh, for sure my configuration is messy and improvable ... but for now it works steady :) But the OKs not going through is worrying me a bit ... and also openser not "timing out" or giving an error.

Cesc

On 4/25/07, Klaus Darilion klaus.mailinglists@pernau.at wrote:

...

Hi!

Does presence module handles PUBLISH/SUBSCRIBE stateful or stateless?

Thus, how should retransmission be handled. Will they be detected by presence module or is it possible to wrap the handle_publish inside t_newtran()?

regards klaus

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

Hello Klaus,

if that is the full log, seems no mem leak in shm. What is the shm memory size you use?

Once you get out of memory can you issue" openserctl fifo get_statistics" and send output?

I don't think is fragmentation, but you can try to compile with -DQM_JOIN_FREE

Do you do the tests over TLS?

Cheers, Daniel

On 04/26/07 10:17, Klaus Darilion wrote:

btw: there is no "PRESENCE:notify: ERROR in function tmb.t_request_within" in my log file, thus I guess the problem is somewhere else.

regards klaus

Klaus Darilion wrote:

...

Hi Anca!

There is still a leak: http://pernau.at/kd/openser/debug-stop2.txt

regards klaus

Anca-Maria Vamanu wrote:

...

Hello,

I have made a fix. Please test again with sipp and say if it works now.

regards,

Anca

Klaus Darilion wrote:

...

Hi!

Openser 1.2 runs out of memory during presence load tests with sipp.

Config snippet:

if (is_method("PUBLISH")) { xlog("L_INFO","$ci PUBLISH detected ... outside t_newtran\n"); if (t_newtran()) { xlog("L_INFO","$ci PUBLISH detected ... inside t_newtran\n"); handle_publish(); xlog("L_INFO","$ci PUBLISH detected, handle_publish ... done\n"); exit; } else { sl_reply_error(); exit; } }

Memory log after openser stop (after 20 minutes of inactivity): http://pernau.at/kd/openser/debug-stop.txt

regards klaus Klaus Darilion wrote:

...

Hi Cesc!

Where do you have t_newtran() - in ser or openser?

I have it here in openser just before handle_publish(). It works but now openser has run out of mem during my sipp performance tests :-(

...more to come....

Cesc wrote:

...

I think I can answer that one ... stateless?

I have a presence openser proxied by a SER, both on same machine ... For easier configuration purposes, openser and ser communicate via 127.0.0.1 ...

The result is that OK from the PUAs do not get to openser (i guess SER gets confussed by the vias of openser being 0.0.0.0:5060 and the 127.0.0.1 stuff) ... So, openser gets no OK, but it does not matter ... BTW, i have a t_newtran() in there ...

Oh, for sure my configuration is messy and improvable ... but for now it works steady :) But the OKs not going through is worrying me a bit ... and also openser not "timing out" or giving an error.

Cesc

On 4/25/07, Klaus Darilion klaus.mailinglists@pernau.at wrote:

> Hi! > > Does presence module handles PUBLISH/SUBSCRIBE stateful or > stateless? > > Thus, how should retransmission be handled. Will they be > detected by > presence module or is it possible to wrap the handle_publish inside > t_newtran()? > > regards > klaus > > _______________________________________________ > Users mailing list > Users@openser.org > http://openser.org/cgi-bin/mailman/listinfo/users >

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

# openserctl fifo get_statistics 400 Too few or too many arguments

:-(

I'm using 1.2

regards klaus

Klaus Darilion wrote:

Daniel-Constantin Mierla wrote:

...

Hello Klaus,

if that is the full log, seems no mem leak in shm. What is the shm memory size you use?

I don't know - I use the default values. But the size should be irrelevant as the "no memory left" is still there after 20 minutes of inactivity.

...

Once you get out of memory can you issue" openserctl fifo get_statistics" and send output?

ok

...

I don't think is fragmentation, but you can try to compile with -DQM_JOIN_FREE

ok

...

Do you do the tests over TLS?

ok

...

Cheers, Daniel

On 04/26/07 10:17, Klaus Darilion wrote:

...

btw: there is no "PRESENCE:notify: ERROR in function tmb.t_request_within" in my log file, thus I guess the problem is somewhere else.

regards klaus

Klaus Darilion wrote:

...

Hi Anca!

There is still a leak: http://pernau.at/kd/openser/debug-stop2.txt

regards klaus

Anca-Maria Vamanu wrote:

...

Hello,

I have made a fix. Please test again with sipp and say if it works now.

regards,

Anca

Klaus Darilion wrote:

...

Hi!

Openser 1.2 runs out of memory during presence load tests with sipp.

Config snippet:

if (is_method("PUBLISH")) { xlog("L_INFO","$ci PUBLISH detected ... outside t_newtran\n"); if (t_newtran()) { xlog("L_INFO","$ci PUBLISH detected ... inside t_newtran\n"); handle_publish(); xlog("L_INFO","$ci PUBLISH detected, handle_publish ... done\n"); exit; } else { sl_reply_error(); exit; } }

Memory log after openser stop (after 20 minutes of inactivity): http://pernau.at/kd/openser/debug-stop.txt

regards klaus Klaus Darilion wrote:

> Hi Cesc! > > Where do you have t_newtran() - in ser or openser? > > I have it here in openser just before handle_publish(). It works > but now openser has run out of mem during my sipp performance > tests :-( > > ...more to come.... > > Cesc wrote: > >> I think I can answer that one ... stateless? >> >> I have a presence openser proxied by a SER, both on same machine >> ... >> For easier configuration purposes, openser and ser communicate via >> 127.0.0.1 ... >> >> The result is that OK from the PUAs do not get to openser (i >> guess SER >> gets confussed by the vias of openser being 0.0.0.0:5060 and the >> 127.0.0.1 stuff) ... So, openser gets no OK, but it does not matter >> ... >> BTW, i have a t_newtran() in there ... >> >> Oh, for sure my configuration is messy and improvable ... but >> for now >> it works steady :) But the OKs not going through is worrying me >> a bit >> ... and also openser not "timing out" or giving an error. >> >> Cesc >> >> On 4/25/07, Klaus Darilion klaus.mailinglists@pernau.at wrote: >> >>> Hi! >>> >>> Does presence module handles PUBLISH/SUBSCRIBE stateful or >>> stateless? >>> >>> Thus, how should retransmission be handled. Will they be >>> detected by >>> presence module or is it possible to wrap the handle_publish >>> inside >>> t_newtran()? >>> >>> regards >>> klaus >>> >>> _______________________________________________ >>> Users mailing list >>> Users@openser.org >>> http://openser.org/cgi-bin/mailman/listinfo/users >>> > > _______________________________________________ > Users mailing list > Users@openser.org > http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

Openser runs out of mem: $ sudo openserctl fifo get_statistics all core:rcv_requests = 13893 core:rcv_replies = 18007 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65351684 shmem:real_used_size = 66591880 shmem:max_used_size = 66953980 shmem:free_size = 516984 shmem:fragments = 1052 sl:1xx_replies = 0 sl:2xx_replies = 8878 sl:3xx_replies = 0 sl:4xx_replies = 2322 sl:5xx_replies = 2308 sl:6xx_replies = 0 sl:sent_replies = 13508 sl:sent_err_replies = 2308 sl:received_ACKs = 0 tm:received_replies = 18007 tm:relayed_replies = 17652 tm:local_replies = 1 tm:UAS_transactions = 8861 tm:UAC_transactions = 17653 tm:2xx_transactions = 17652 tm:3xx_transactions = 0 tm:4xx_transactions = 1 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8861 usrloc:registered_users = 2 usrloc:location-users = 2 usrloc:location-contacts = 2 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 12 registrar:rejected_regs = 0

after 1h of inactivity: $ sudo openserctl fifo get_statistics all Password: core:rcv_requests = 14285 core:rcv_replies = 18007 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65351684 shmem:real_used_size = 66591880 shmem:max_used_size = 66953980 shmem:free_size = 516984 shmem:fragments = 1052 sl:1xx_replies = 0 sl:2xx_replies = 8922 sl:3xx_replies = 0 sl:4xx_replies = 2481 sl:5xx_replies = 2497 sl:6xx_replies = 0 sl:sent_replies = 13900 sl:sent_err_replies = 2497 sl:received_ACKs = 0 tm:received_replies = 18007 tm:relayed_replies = 17652 tm:local_replies = 1 tm:UAS_transactions = 8861 tm:UAC_transactions = 17653 tm:2xx_transactions = 17652 tm:3xx_transactions = 0 tm:4xx_transactions = 1 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8861 usrloc:registered_users = 2 usrloc:location-users = 2 usrloc:location-contacts = 2 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 51 registrar:rejected_regs = 0

regards klaus

Daniel-Constantin Mierla wrote:

my mistake:

# openserctl fifo get_statistics all

Cheers, Daniel

On 04/26/07 12:36, Klaus Darilion wrote:

...

# openserctl fifo get_statistics 400 Too few or too many arguments

:-(

I'm using 1.2

regards klaus

Klaus Darilion wrote:

...

Daniel-Constantin Mierla wrote:

...

Hello Klaus,

if that is the full log, seems no mem leak in shm. What is the shm memory size you use?

I don't know - I use the default values. But the size should be irrelevant as the "no memory left" is still there after 20 minutes of inactivity.

...

Once you get out of memory can you issue" openserctl fifo get_statistics" and send output?

ok

...

I don't think is fragmentation, but you can try to compile with -DQM_JOIN_FREE

ok

...

Do you do the tests over TLS?

ok

...

Cheers, Daniel

On 04/26/07 10:17, Klaus Darilion wrote:

...

btw: there is no "PRESENCE:notify: ERROR in function tmb.t_request_within" in my log file, thus I guess the problem is somewhere else.

regards klaus

Klaus Darilion wrote:

...

Hi Anca!

There is still a leak: http://pernau.at/kd/openser/debug-stop2.txt

regards klaus

Anca-Maria Vamanu wrote: > Hello, > > I have made a fix. Please test again with sipp and say if it > works now. > > regards, > > Anca > > Klaus Darilion wrote: > >> Hi! >> >> Openser 1.2 runs out of memory during presence load tests with >> sipp. >> >> Config snippet: >> >> if (is_method("PUBLISH")) { >> xlog("L_INFO","$ci PUBLISH detected ... outside t_newtran\n"); >> if (t_newtran()) { >> xlog("L_INFO","$ci PUBLISH detected ... inside t_newtran\n"); >> handle_publish(); >> xlog("L_INFO","$ci PUBLISH detected, handle_publish ... >> done\n"); >> exit; >> } else { >> sl_reply_error(); >> exit; >> } >> } >> >> Memory log after openser stop (after 20 minutes of inactivity): >> http://pernau.at/kd/openser/debug-stop.txt >> >> >> regards >> klaus Klaus Darilion wrote: >> >>> Hi Cesc! >>> >>> Where do you have t_newtran() - in ser or openser? >>> >>> I have it here in openser just before handle_publish(). It >>> works but now openser has run out of mem during my sipp >>> performance tests :-( >>> >>> ...more to come.... >>> >>> Cesc wrote: >>> >>>> I think I can answer that one ... stateless? >>>> >>>> I have a presence openser proxied by a SER, both on same >>>> machine ... >>>> For easier configuration purposes, openser and ser communicate >>>> via >>>> 127.0.0.1 ... >>>> >>>> The result is that OK from the PUAs do not get to openser (i >>>> guess SER >>>> gets confussed by the vias of openser being 0.0.0.0:5060 and the >>>> 127.0.0.1 stuff) ... So, openser gets no OK, but it does not >>>> matter >>>> ... >>>> BTW, i have a t_newtran() in there ... >>>> >>>> Oh, for sure my configuration is messy and improvable ... but >>>> for now >>>> it works steady :) But the OKs not going through is worrying >>>> me a bit >>>> ... and also openser not "timing out" or giving an error. >>>> >>>> Cesc >>>> >>>> On 4/25/07, Klaus Darilion klaus.mailinglists@pernau.at wrote: >>>> >>>>> Hi! >>>>> >>>>> Does presence module handles PUBLISH/SUBSCRIBE stateful or >>>>> stateless? >>>>> >>>>> Thus, how should retransmission be handled. Will they be >>>>> detected by >>>>> presence module or is it possible to wrap the handle_publish >>>>> inside >>>>> t_newtran()? >>>>> >>>>> regards >>>>> klaus >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@openser.org >>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@openser.org >>> http://openser.org/cgi-bin/mailman/listinfo/users >> >> >> _______________________________________________ >> Users mailing list >> Users@openser.org >> http://openser.org/cgi-bin/mailman/listinfo/users >> >

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

Hi!

I also tried now with QM_JOIN_FREE but the same result: The shared mem (64M) is full after ~8000 PUBLISH (=8000 NOTIFY) and does not get freed:

darilion@ds3000:~/software/openser-svn-1.2/openser$ sudo openserctl fifo get_statistics all core:rcv_requests = 8743 core:rcv_replies = 8198 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65865712 shmem:real_used_size = 67004532 shmem:max_used_size = 67108376 shmem:free_size = 104332 shmem:fragments = 239 sl:1xx_replies = 0 sl:2xx_replies = 8221 sl:3xx_replies = 0 sl:4xx_replies = 196 sl:5xx_replies = 186 sl:6xx_replies = 0 sl:sent_replies = 8603 sl:sent_err_replies = 186 sl:received_ACKs = 0 tm:received_replies = 8198 tm:relayed_replies = 8198 tm:local_replies = 0 tm:UAS_transactions = 8218 tm:UAC_transactions = 8200 tm:2xx_transactions = 8198 tm:3xx_transactions = 0 tm:4xx_transactions = 0 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8217 usrloc:registered_users = 0 usrloc:location-users = 0 usrloc:location-contacts = 0 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 1 registrar:rejected_regs = 0 darilion@ds3000:~/software/openser-svn-1.2/openser$

regards klaus

Klaus Darilion wrote:

memory logs:

kill -SIGUSR1: http://pernau.at/kd/openser/debug-kill-usr1.txt

openser stop: http://pernau.at/kd/openser/debug-stop3.txt

regards klaus

Klaus Darilion wrote:

...

Openser runs out of mem: $ sudo openserctl fifo get_statistics all core:rcv_requests = 13893 core:rcv_replies = 18007 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65351684 shmem:real_used_size = 66591880 shmem:max_used_size = 66953980 shmem:free_size = 516984 shmem:fragments = 1052 sl:1xx_replies = 0 sl:2xx_replies = 8878 sl:3xx_replies = 0 sl:4xx_replies = 2322 sl:5xx_replies = 2308 sl:6xx_replies = 0 sl:sent_replies = 13508 sl:sent_err_replies = 2308 sl:received_ACKs = 0 tm:received_replies = 18007 tm:relayed_replies = 17652 tm:local_replies = 1 tm:UAS_transactions = 8861 tm:UAC_transactions = 17653 tm:2xx_transactions = 17652 tm:3xx_transactions = 0 tm:4xx_transactions = 1 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8861 usrloc:registered_users = 2 usrloc:location-users = 2 usrloc:location-contacts = 2 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 12 registrar:rejected_regs = 0

after 1h of inactivity: $ sudo openserctl fifo get_statistics all Password: core:rcv_requests = 14285 core:rcv_replies = 18007 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65351684 shmem:real_used_size = 66591880 shmem:max_used_size = 66953980 shmem:free_size = 516984 shmem:fragments = 1052 sl:1xx_replies = 0 sl:2xx_replies = 8922 sl:3xx_replies = 0 sl:4xx_replies = 2481 sl:5xx_replies = 2497 sl:6xx_replies = 0 sl:sent_replies = 13900 sl:sent_err_replies = 2497 sl:received_ACKs = 0 tm:received_replies = 18007 tm:relayed_replies = 17652 tm:local_replies = 1 tm:UAS_transactions = 8861 tm:UAC_transactions = 17653 tm:2xx_transactions = 17652 tm:3xx_transactions = 0 tm:4xx_transactions = 1 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8861 usrloc:registered_users = 2 usrloc:location-users = 2 usrloc:location-contacts = 2 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 51 registrar:rejected_regs = 0

regards klaus

Daniel-Constantin Mierla wrote:

...

my mistake:

# openserctl fifo get_statistics all

Cheers, Daniel

On 04/26/07 12:36, Klaus Darilion wrote:

...

# openserctl fifo get_statistics 400 Too few or too many arguments

:-(

I'm using 1.2

regards klaus

Klaus Darilion wrote:

...

Daniel-Constantin Mierla wrote:

...

Hello Klaus,

if that is the full log, seems no mem leak in shm. What is the shm memory size you use?

I don't know - I use the default values. But the size should be irrelevant as the "no memory left" is still there after 20 minutes of inactivity.

...

Once you get out of memory can you issue" openserctl fifo get_statistics" and send output?

ok

...

I don't think is fragmentation, but you can try to compile with -DQM_JOIN_FREE

ok

...

Do you do the tests over TLS?

ok

...

Cheers, Daniel

On 04/26/07 10:17, Klaus Darilion wrote: > btw: there is no > "PRESENCE:notify: ERROR in function tmb.t_request_within" > in my log file, thus I guess the problem is somewhere else. > > regards > klaus > > Klaus Darilion wrote: >> Hi Anca! >> >> There is still a leak: >> http://pernau.at/kd/openser/debug-stop2.txt >> >> regards >> klaus >> >> Anca-Maria Vamanu wrote: >>> Hello, >>> >>> I have made a fix. Please test again with sipp and say if it >>> works now. >>> >>> regards, >>> >>> Anca >>> >>> Klaus Darilion wrote: >>> >>>> Hi! >>>> >>>> Openser 1.2 runs out of memory during presence load tests with >>>> sipp. >>>> >>>> Config snippet: >>>> >>>> if (is_method("PUBLISH")) { >>>> xlog("L_INFO","$ci PUBLISH detected ... outside t_newtran\n"); >>>> if (t_newtran()) { >>>> xlog("L_INFO","$ci PUBLISH detected ... inside t_newtran\n"); >>>> handle_publish(); >>>> xlog("L_INFO","$ci PUBLISH detected, handle_publish ... >>>> done\n"); >>>> exit; >>>> } else { >>>> sl_reply_error(); >>>> exit; >>>> } >>>> } >>>> >>>> Memory log after openser stop (after 20 minutes of inactivity): >>>> http://pernau.at/kd/openser/debug-stop.txt >>>> >>>> >>>> regards >>>> klaus Klaus Darilion wrote: >>>> >>>>> Hi Cesc! >>>>> >>>>> Where do you have t_newtran() - in ser or openser? >>>>> >>>>> I have it here in openser just before handle_publish(). It >>>>> works but now openser has run out of mem during my sipp >>>>> performance tests :-( >>>>> >>>>> ...more to come.... >>>>> >>>>> Cesc wrote: >>>>> >>>>>> I think I can answer that one ... stateless? >>>>>> >>>>>> I have a presence openser proxied by a SER, both on same >>>>>> machine ... >>>>>> For easier configuration purposes, openser and ser >>>>>> communicate via >>>>>> 127.0.0.1 ... >>>>>> >>>>>> The result is that OK from the PUAs do not get to openser (i >>>>>> guess SER >>>>>> gets confussed by the vias of openser being 0.0.0.0:5060 and >>>>>> the >>>>>> 127.0.0.1 stuff) ... So, openser gets no OK, but it does not >>>>>> matter >>>>>> ... >>>>>> BTW, i have a t_newtran() in there ... >>>>>> >>>>>> Oh, for sure my configuration is messy and improvable ... >>>>>> but for now >>>>>> it works steady :) But the OKs not going through is worrying >>>>>> me a bit >>>>>> ... and also openser not "timing out" or giving an error. >>>>>> >>>>>> Cesc >>>>>> >>>>>> On 4/25/07, Klaus Darilion klaus.mailinglists@pernau.at >>>>>> wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> Does presence module handles PUBLISH/SUBSCRIBE stateful or >>>>>>> stateless? >>>>>>> >>>>>>> Thus, how should retransmission be handled. Will they be >>>>>>> detected by >>>>>>> presence module or is it possible to wrap the >>>>>>> handle_publish inside >>>>>>> t_newtran()? >>>>>>> >>>>>>> regards >>>>>>> klaus >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@openser.org >>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@openser.org >>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users@openser.org >>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>> >>> >> >> _______________________________________________ >> Users mailing list >> Users@openser.org >> http://openser.org/cgi-bin/mailman/listinfo/users > > _______________________________________________ > Users mailing list > Users@openser.org > http://openser.org/cgi-bin/mailman/listinfo/users >

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

Hi Daniel!

One more comment:

This leads to memory leak:

if (is_method("PUBLISH")) { if (t_newtran()) { handle_publish(); exit; } else { sl_reply_error(); exit; } }

This works fine:

if (is_method("PUBLISH")) { handle_publish(); exit; }

Thus, it is related to the generated transaction of t_newtran.

regards klaus

Klaus Darilion wrote:

Hi!

I also tried now with QM_JOIN_FREE but the same result: The shared mem (64M) is full after ~8000 PUBLISH (=8000 NOTIFY) and does not get freed:

darilion@ds3000:~/software/openser-svn-1.2/openser$ sudo openserctl fifo get_statistics all core:rcv_requests = 8743 core:rcv_replies = 8198 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65865712 shmem:real_used_size = 67004532 shmem:max_used_size = 67108376 shmem:free_size = 104332 shmem:fragments = 239 sl:1xx_replies = 0 sl:2xx_replies = 8221 sl:3xx_replies = 0 sl:4xx_replies = 196 sl:5xx_replies = 186 sl:6xx_replies = 0 sl:sent_replies = 8603 sl:sent_err_replies = 186 sl:received_ACKs = 0 tm:received_replies = 8198 tm:relayed_replies = 8198 tm:local_replies = 0 tm:UAS_transactions = 8218 tm:UAC_transactions = 8200 tm:2xx_transactions = 8198 tm:3xx_transactions = 0 tm:4xx_transactions = 0 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8217 usrloc:registered_users = 0 usrloc:location-users = 0 usrloc:location-contacts = 0 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 1 registrar:rejected_regs = 0 darilion@ds3000:~/software/openser-svn-1.2/openser$

regards klaus

Klaus Darilion wrote:

...

memory logs:

kill -SIGUSR1: http://pernau.at/kd/openser/debug-kill-usr1.txt

openser stop: http://pernau.at/kd/openser/debug-stop3.txt

regards klaus

Klaus Darilion wrote:

...

Openser runs out of mem: $ sudo openserctl fifo get_statistics all core:rcv_requests = 13893 core:rcv_replies = 18007 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65351684 shmem:real_used_size = 66591880 shmem:max_used_size = 66953980 shmem:free_size = 516984 shmem:fragments = 1052 sl:1xx_replies = 0 sl:2xx_replies = 8878 sl:3xx_replies = 0 sl:4xx_replies = 2322 sl:5xx_replies = 2308 sl:6xx_replies = 0 sl:sent_replies = 13508 sl:sent_err_replies = 2308 sl:received_ACKs = 0 tm:received_replies = 18007 tm:relayed_replies = 17652 tm:local_replies = 1 tm:UAS_transactions = 8861 tm:UAC_transactions = 17653 tm:2xx_transactions = 17652 tm:3xx_transactions = 0 tm:4xx_transactions = 1 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8861 usrloc:registered_users = 2 usrloc:location-users = 2 usrloc:location-contacts = 2 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 12 registrar:rejected_regs = 0

after 1h of inactivity: $ sudo openserctl fifo get_statistics all Password: core:rcv_requests = 14285 core:rcv_replies = 18007 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65351684 shmem:real_used_size = 66591880 shmem:max_used_size = 66953980 shmem:free_size = 516984 shmem:fragments = 1052 sl:1xx_replies = 0 sl:2xx_replies = 8922 sl:3xx_replies = 0 sl:4xx_replies = 2481 sl:5xx_replies = 2497 sl:6xx_replies = 0 sl:sent_replies = 13900 sl:sent_err_replies = 2497 sl:received_ACKs = 0 tm:received_replies = 18007 tm:relayed_replies = 17652 tm:local_replies = 1 tm:UAS_transactions = 8861 tm:UAC_transactions = 17653 tm:2xx_transactions = 17652 tm:3xx_transactions = 0 tm:4xx_transactions = 1 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8861 usrloc:registered_users = 2 usrloc:location-users = 2 usrloc:location-contacts = 2 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 51 registrar:rejected_regs = 0

regards klaus

Daniel-Constantin Mierla wrote:

...

my mistake:

# openserctl fifo get_statistics all

Cheers, Daniel

On 04/26/07 12:36, Klaus Darilion wrote:

...

# openserctl fifo get_statistics 400 Too few or too many arguments

:-(

I'm using 1.2

regards klaus

Klaus Darilion wrote:

...

Daniel-Constantin Mierla wrote: > Hello Klaus, > > if that is the full log, seems no mem leak in shm. What is the > shm memory size you use?

I don't know - I use the default values. But the size should be irrelevant as the "no memory left" is still there after 20 minutes of inactivity.

> Once you get out of memory can you issue" openserctl fifo > get_statistics" and send output? ok > > I don't think is fragmentation, but you can try to compile with > -DQM_JOIN_FREE > ok > Do you do the tests over TLS? ok > > Cheers, > Daniel > > > On 04/26/07 10:17, Klaus Darilion wrote: >> btw: there is no >> "PRESENCE:notify: ERROR in function tmb.t_request_within" >> in my log file, thus I guess the problem is somewhere else. >> >> regards >> klaus >> >> Klaus Darilion wrote: >>> Hi Anca! >>> >>> There is still a leak: >>> http://pernau.at/kd/openser/debug-stop2.txt >>> >>> regards >>> klaus >>> >>> Anca-Maria Vamanu wrote: >>>> Hello, >>>> >>>> I have made a fix. Please test again with sipp and say if it >>>> works now. >>>> >>>> regards, >>>> >>>> Anca >>>> >>>> Klaus Darilion wrote: >>>> >>>>> Hi! >>>>> >>>>> Openser 1.2 runs out of memory during presence load tests >>>>> with sipp. >>>>> >>>>> Config snippet: >>>>> >>>>> if (is_method("PUBLISH")) { >>>>> xlog("L_INFO","$ci PUBLISH detected ... outside t_newtran\n"); >>>>> if (t_newtran()) { >>>>> xlog("L_INFO","$ci PUBLISH detected ... inside >>>>> t_newtran\n"); >>>>> handle_publish(); >>>>> xlog("L_INFO","$ci PUBLISH detected, handle_publish ... >>>>> done\n"); >>>>> exit; >>>>> } else { >>>>> sl_reply_error(); >>>>> exit; >>>>> } >>>>> } >>>>> >>>>> Memory log after openser stop (after 20 minutes of inactivity): >>>>> http://pernau.at/kd/openser/debug-stop.txt >>>>> >>>>> >>>>> regards >>>>> klaus Klaus Darilion wrote: >>>>> >>>>>> Hi Cesc! >>>>>> >>>>>> Where do you have t_newtran() - in ser or openser? >>>>>> >>>>>> I have it here in openser just before handle_publish(). It >>>>>> works but now openser has run out of mem during my sipp >>>>>> performance tests :-( >>>>>> >>>>>> ...more to come.... >>>>>> >>>>>> Cesc wrote: >>>>>> >>>>>>> I think I can answer that one ... stateless? >>>>>>> >>>>>>> I have a presence openser proxied by a SER, both on same >>>>>>> machine ... >>>>>>> For easier configuration purposes, openser and ser >>>>>>> communicate via >>>>>>> 127.0.0.1 ... >>>>>>> >>>>>>> The result is that OK from the PUAs do not get to openser >>>>>>> (i guess SER >>>>>>> gets confussed by the vias of openser being 0.0.0.0:5060 >>>>>>> and the >>>>>>> 127.0.0.1 stuff) ... So, openser gets no OK, but it does >>>>>>> not matter >>>>>>> ... >>>>>>> BTW, i have a t_newtran() in there ... >>>>>>> >>>>>>> Oh, for sure my configuration is messy and improvable ... >>>>>>> but for now >>>>>>> it works steady :) But the OKs not going through is >>>>>>> worrying me a bit >>>>>>> ... and also openser not "timing out" or giving an error. >>>>>>> >>>>>>> Cesc >>>>>>> >>>>>>> On 4/25/07, Klaus Darilion klaus.mailinglists@pernau.at >>>>>>> wrote: >>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> Does presence module handles PUBLISH/SUBSCRIBE stateful or >>>>>>>> stateless? >>>>>>>> >>>>>>>> Thus, how should retransmission be handled. Will they be >>>>>>>> detected by >>>>>>>> presence module or is it possible to wrap the >>>>>>>> handle_publish inside >>>>>>>> t_newtran()? >>>>>>>> >>>>>>>> regards >>>>>>>> klaus >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@openser.org >>>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@openser.org >>>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@openser.org >>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>>> >>>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@openser.org >>> http://openser.org/cgi-bin/mailman/listinfo/users >> >> _______________________________________________ >> Users mailing list >> Users@openser.org >> http://openser.org/cgi-bin/mailman/listinfo/users >>

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

Daniel-Constantin Mierla wrote:

Could you try :

t_release();

Will this release the transcation immediately or is it kept some seconds to absorb retransmissions?

regards klaus

after handle_publish();

Cheers, Daniel

On 04/26/07 17:10, Klaus Darilion wrote:

...

Hi Daniel!

One more comment:

This leads to memory leak:

if (is_method("PUBLISH")) { if (t_newtran()) { handle_publish(); exit; } else { sl_reply_error(); exit; } }

This works fine:

if (is_method("PUBLISH")) { handle_publish(); exit; }

Thus, it is related to the generated transaction of t_newtran.

regards klaus

Klaus Darilion wrote:

...

Hi!

I also tried now with QM_JOIN_FREE but the same result: The shared mem (64M) is full after ~8000 PUBLISH (=8000 NOTIFY) and does not get freed:

darilion@ds3000:~/software/openser-svn-1.2/openser$ sudo openserctl fifo get_statistics all core:rcv_requests = 8743 core:rcv_replies = 8198 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65865712 shmem:real_used_size = 67004532 shmem:max_used_size = 67108376 shmem:free_size = 104332 shmem:fragments = 239 sl:1xx_replies = 0 sl:2xx_replies = 8221 sl:3xx_replies = 0 sl:4xx_replies = 196 sl:5xx_replies = 186 sl:6xx_replies = 0 sl:sent_replies = 8603 sl:sent_err_replies = 186 sl:received_ACKs = 0 tm:received_replies = 8198 tm:relayed_replies = 8198 tm:local_replies = 0 tm:UAS_transactions = 8218 tm:UAC_transactions = 8200 tm:2xx_transactions = 8198 tm:3xx_transactions = 0 tm:4xx_transactions = 0 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8217 usrloc:registered_users = 0 usrloc:location-users = 0 usrloc:location-contacts = 0 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 1 registrar:rejected_regs = 0 darilion@ds3000:~/software/openser-svn-1.2/openser$

regards klaus

Klaus Darilion wrote:

...

memory logs:

kill -SIGUSR1: http://pernau.at/kd/openser/debug-kill-usr1.txt

openser stop: http://pernau.at/kd/openser/debug-stop3.txt

regards klaus

Klaus Darilion wrote:

...

Openser runs out of mem: $ sudo openserctl fifo get_statistics all core:rcv_requests = 13893 core:rcv_replies = 18007 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65351684 shmem:real_used_size = 66591880 shmem:max_used_size = 66953980 shmem:free_size = 516984 shmem:fragments = 1052 sl:1xx_replies = 0 sl:2xx_replies = 8878 sl:3xx_replies = 0 sl:4xx_replies = 2322 sl:5xx_replies = 2308 sl:6xx_replies = 0 sl:sent_replies = 13508 sl:sent_err_replies = 2308 sl:received_ACKs = 0 tm:received_replies = 18007 tm:relayed_replies = 17652 tm:local_replies = 1 tm:UAS_transactions = 8861 tm:UAC_transactions = 17653 tm:2xx_transactions = 17652 tm:3xx_transactions = 0 tm:4xx_transactions = 1 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8861 usrloc:registered_users = 2 usrloc:location-users = 2 usrloc:location-contacts = 2 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 12 registrar:rejected_regs = 0

after 1h of inactivity: $ sudo openserctl fifo get_statistics all Password: core:rcv_requests = 14285 core:rcv_replies = 18007 core:fwd_requests = 0 core:fwd_replies = 0 core:drop_requests = 0 core:drop_replies = 0 core:err_requests = 0 core:err_replies = 0 core:bad_URIs_rcvd = 0 core:unsupported_methods = 0 core:bad_msg_hdr = 0 shmem:total_size = 67108864 shmem:used_size = 65351684 shmem:real_used_size = 66591880 shmem:max_used_size = 66953980 shmem:free_size = 516984 shmem:fragments = 1052 sl:1xx_replies = 0 sl:2xx_replies = 8922 sl:3xx_replies = 0 sl:4xx_replies = 2481 sl:5xx_replies = 2497 sl:6xx_replies = 0 sl:sent_replies = 13900 sl:sent_err_replies = 2497 sl:received_ACKs = 0 tm:received_replies = 18007 tm:relayed_replies = 17652 tm:local_replies = 1 tm:UAS_transactions = 8861 tm:UAC_transactions = 17653 tm:2xx_transactions = 17652 tm:3xx_transactions = 0 tm:4xx_transactions = 1 tm:5xx_transactions = 0 tm:6xx_transactions = 0 tm:inuse_transactions = 8861 usrloc:registered_users = 2 usrloc:location-users = 2 usrloc:location-contacts = 2 usrloc:location-expires = 0 registrar:max_expires = 0 registrar:max_contacts = 0 registrar:default_expire = 3600 registrar:accepted_regs = 51 registrar:rejected_regs = 0

regards klaus

Daniel-Constantin Mierla wrote:

...

my mistake:

# openserctl fifo get_statistics all

Cheers, Daniel

On 04/26/07 12:36, Klaus Darilion wrote: > # openserctl fifo get_statistics > 400 Too few or too many arguments > > :-( > > I'm using 1.2 > > > regards > klaus > > > > Klaus Darilion wrote: >> >> >> Daniel-Constantin Mierla wrote: >>> Hello Klaus, >>> >>> if that is the full log, seems no mem leak in shm. What is the >>> shm memory size you use? >> >> I don't know - I use the default values. But the size should be >> irrelevant as the "no memory left" is still there after 20 >> minutes of inactivity. >> >>> Once you get out of memory can you issue" openserctl fifo >>> get_statistics" and send output? >> ok >>> >>> I don't think is fragmentation, but you can try to compile with >>> -DQM_JOIN_FREE >>> >> ok >>> Do you do the tests over TLS? >> ok >>> >>> Cheers, >>> Daniel >>> >>> >>> On 04/26/07 10:17, Klaus Darilion wrote: >>>> btw: there is no >>>> "PRESENCE:notify: ERROR in function tmb.t_request_within" >>>> in my log file, thus I guess the problem is somewhere else. >>>> >>>> regards >>>> klaus >>>> >>>> Klaus Darilion wrote: >>>>> Hi Anca! >>>>> >>>>> There is still a leak: >>>>> http://pernau.at/kd/openser/debug-stop2.txt >>>>> >>>>> regards >>>>> klaus >>>>> >>>>> Anca-Maria Vamanu wrote: >>>>>> Hello, >>>>>> >>>>>> I have made a fix. Please test again with sipp and say if it >>>>>> works now. >>>>>> >>>>>> regards, >>>>>> >>>>>> Anca >>>>>> >>>>>> Klaus Darilion wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> Openser 1.2 runs out of memory during presence load tests >>>>>>> with sipp. >>>>>>> >>>>>>> Config snippet: >>>>>>> >>>>>>> if (is_method("PUBLISH")) { >>>>>>> xlog("L_INFO","$ci PUBLISH detected ... outside >>>>>>> t_newtran\n"); >>>>>>> if (t_newtran()) { >>>>>>> xlog("L_INFO","$ci PUBLISH detected ... inside >>>>>>> t_newtran\n"); >>>>>>> handle_publish(); >>>>>>> xlog("L_INFO","$ci PUBLISH detected, handle_publish ... >>>>>>> done\n"); >>>>>>> exit; >>>>>>> } else { >>>>>>> sl_reply_error(); >>>>>>> exit; >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> Memory log after openser stop (after 20 minutes of >>>>>>> inactivity): >>>>>>> http://pernau.at/kd/openser/debug-stop.txt >>>>>>> >>>>>>> >>>>>>> regards >>>>>>> klaus Klaus Darilion wrote: >>>>>>> >>>>>>>> Hi Cesc! >>>>>>>> >>>>>>>> Where do you have t_newtran() - in ser or openser? >>>>>>>> >>>>>>>> I have it here in openser just before handle_publish(). It >>>>>>>> works but now openser has run out of mem during my sipp >>>>>>>> performance tests :-( >>>>>>>> >>>>>>>> ...more to come.... >>>>>>>> >>>>>>>> Cesc wrote: >>>>>>>> >>>>>>>>> I think I can answer that one ... stateless? >>>>>>>>> >>>>>>>>> I have a presence openser proxied by a SER, both on same >>>>>>>>> machine ... >>>>>>>>> For easier configuration purposes, openser and ser >>>>>>>>> communicate via >>>>>>>>> 127.0.0.1 ... >>>>>>>>> >>>>>>>>> The result is that OK from the PUAs do not get to openser >>>>>>>>> (i guess SER >>>>>>>>> gets confussed by the vias of openser being 0.0.0.0:5060 >>>>>>>>> and the >>>>>>>>> 127.0.0.1 stuff) ... So, openser gets no OK, but it does >>>>>>>>> not matter >>>>>>>>> ... >>>>>>>>> BTW, i have a t_newtran() in there ... >>>>>>>>> >>>>>>>>> Oh, for sure my configuration is messy and improvable ... >>>>>>>>> but for now >>>>>>>>> it works steady :) But the OKs not going through is >>>>>>>>> worrying me a bit >>>>>>>>> ... and also openser not "timing out" or giving an error. >>>>>>>>> >>>>>>>>> Cesc >>>>>>>>> >>>>>>>>> On 4/25/07, Klaus Darilion klaus.mailinglists@pernau.at >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi! >>>>>>>>>> >>>>>>>>>> Does presence module handles PUBLISH/SUBSCRIBE stateful >>>>>>>>>> or stateless? >>>>>>>>>> >>>>>>>>>> Thus, how should retransmission be handled. Will they be >>>>>>>>>> detected by >>>>>>>>>> presence module or is it possible to wrap the >>>>>>>>>> handle_publish inside >>>>>>>>>> t_newtran()? >>>>>>>>>> >>>>>>>>>> regards >>>>>>>>>> klaus >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users@openser.org >>>>>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@openser.org >>>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@openser.org >>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@openser.org >>>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users@openser.org >>>> http://openser.org/cgi-bin/mailman/listinfo/users >>>> >> >> _______________________________________________ >> Users mailing list >> Users@openser.org >> http://openser.org/cgi-bin/mailman/listinfo/users >

---

Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users

On Friday 27 April 2007 08:27, Klaus Darilion wrote:

There was no core generated (although DUMP_CORE=yes), maybe signal 9 doesn't cause a core to be generated?

Hello Klaus,

on my system i must add a path to "core pattern" in the proc fs, to get the core dump in daemon mode.

You could try to add --------- COREDIR=/home/corefiles [ -d $COREDIR ] || mkdir $COREDIR chmod 777 $COREDIR echo "$COREDIR/core.%e.sig%s.%p" > /proc/sys/kernel/core_pattern --------- to your init file. Should i add code to handle this to the init file in svn?

Cheers,

Henning

On Monday 30 April 2007 12:59, Klaus Darilion wrote:

Hi Henning!

This solved my issues with the corefile (tested with signal 11). Maybe things changed from Debian 3.1 to 4.0 as I never had to do this before.

Hi Klaus,

i added a short note regarding this problem to the openser debian default config file, and a example core directory configuration to the init file. As this problem happens only on few systems, its commented out for now.

Regards,

Henning

Daniel-Constantin Mierla wrote:

Hello Klaus,

On 04/27/07 09:27, Klaus Darilion wrote:

...

Hi Daniel!

I've tried with t_release and it was running fine several hours without leaking. But then, unfortunately openser terminated with signal 9. I've never seen this before.

signal 9 is KILL, it is very strange if it was not issued by a user or other process.

We discovered the issue (tm/uac.c, line 224), ther eis flag that is kept to see if there was some operation with the transaction, but in case of handle_publish() that flag is set by TM api when sending NOTIFY. The patch is trivial, removing a line, but we have to investigate if there are some other effects -- so it may take a while. t_release() should solve meanwhile.

Should solve the memory-leak - but the signal 9?

regards klaus

Hi!

I tried again and it happened again:

Apr 30 15:00:54 ds3000 /usr/sbin/openser[7648]: 32b24f15e52d603ba890a9729723c4b0.0167///45-6782@83.136.32.132 PUBLISH detected, handle_publish ... outside t_newtran Apr 30 15:00:54 ds3000 /usr/sbin/openser[7655]: 32b24f15e52d603ba890a9729723c4b0.7e11///14-6782@83.136.32.132 PUBLISH detected, handle_publish ... outside t_newtran Apr 30 15:00:54 ds3000 /usr/sbin/openser[7648]: 32b24f15e52d603ba890a9729723c4b0.0167///45-6782@83.136.32.132 PUBLISH detected, handle_publish ... inside t_newtran Apr 30 15:00:54 ds3000 /usr/sbin/openser[7655]: 32b24f15e52d603ba890a9729723c4b0.7e11///14-6782@83.136.32.132 PUBLISH detected, handle_publish ... inside t_newtran Apr 30 15:01:03 ds3000 /usr/sbin/openser[7644]: child process 7648 exited by a signal 9 Apr 30 15:01:08 ds3000 /usr/sbin/openser[7644]: core was not generated Apr 30 15:01:08 ds3000 /usr/sbin/openser[7644]: INFO: terminating due to SIGCHLD Apr 30 15:01:14 ds3000 /usr/sbin/openser[7657]: INFO: signal 15 received Apr 30 15:01:14 ds3000 /usr/sbin/openser[7657]: Memory status (pkg): Apr 30 15:01:14 ds3000 /usr/sbin/openser[7657]: qm_status (0x8145960): Apr 30 15:01:14 ds3000 /usr/sbin/openser[7657]: heap size= 1048576 Apr 30 15:01:14 ds3000 /usr/sbin/openser[7659]: INFO: signal 15 received Apr 30 15:01:14 ds3000 /usr/sbin/openser[7653]: INFO: signal 15 received Apr 30 15:01:14 ds3000 /usr/sbin/openser[7653]: Memory status (pkg): Apr 30 15:01:14 ds3000 /usr/sbin/openser[7650]: INFO: signal 15 received

Any hints how to debug this?

regards klaus

Daniel-Constantin Mierla wrote:

Hello Klaus,

On 04/30/07 13:55, Klaus Darilion wrote:

...

Daniel-Constantin Mierla wrote:

...

Hello Klaus,

On 04/27/07 09:27, Klaus Darilion wrote:

...

Hi Daniel!

I've tried with t_release and it was running fine several hours without leaking. But then, unfortunately openser terminated with signal 9. I've never seen this before.

signal 9 is KILL, it is very strange if it was not issued by a user or other process.

We discovered the issue (tm/uac.c, line 224), ther eis flag that is kept to see if there was some operation with the transaction, but in case of handle_publish() that flag is set by TM api when sending NOTIFY. The patch is trivial, removing a line, but we have to investigate if there are some other effects -- so it may take a while. t_release() should solve meanwhile.

Should solve the memory-leak - but the signal 9?

It might be that it took so long to write the mem long at shut down and openser killed itself. If it was not due to a openser stop, then I am not aware of any case when signal 9 is sent unless issued by user.

Cheers, Daniel

...

regards klaus

Hi Daniel!

Summary: - Without t_release() (no modifications to source code) openser leaks memory. - with t_release() openser does not leak. But after some time there is strange behaviour, e.g.: -: openser stops reacting for some minutes and afterwards gets terminated with signal 9. When openser stops working the load increase to > 40. This happend 3 times now. -: openser stops reacting for some minutes and the linux PC where openser is running gets unresponsive. No login. Open SSH sessions are unresponsive. I had to reboot the PC. Happend 1 time.

Maybe this is not pure openser related, but a problem with openser and Linux (as I had to reboot the server one time).

Any hints how to debug this?

regards klaus

Daniel-Constantin Mierla wrote:

Hello Klaus,

I will try to find some ways to investigate the signal 9. As said, except while waiting for writing the mem log, there is no signal 9 to be issued.

Regarding the tm stuff, I am not sure whether your last answer about the mem leak as being solved is due to t_release() or you tried the second option as well (removing in tm/uac.c the line 224). Can you give a short summary?

Cheers, Daniel

On 04/30/07 16:34, Klaus Darilion wrote:

...

Hi!

I tried again and it happened again:

Apr 30 15:00:54 ds3000 /usr/sbin/openser[7648]: 32b24f15e52d603ba890a9729723c4b0.0167///45-6782@83.136.32.132 PUBLISH detected, handle_publish ... outside t_newtran Apr 30 15:00:54 ds3000 /usr/sbin/openser[7655]: 32b24f15e52d603ba890a9729723c4b0.7e11///14-6782@83.136.32.132 PUBLISH detected, handle_publish ... outside t_newtran Apr 30 15:00:54 ds3000 /usr/sbin/openser[7648]: 32b24f15e52d603ba890a9729723c4b0.0167///45-6782@83.136.32.132 PUBLISH detected, handle_publish ... inside t_newtran Apr 30 15:00:54 ds3000 /usr/sbin/openser[7655]: 32b24f15e52d603ba890a9729723c4b0.7e11///14-6782@83.136.32.132 PUBLISH detected, handle_publish ... inside t_newtran Apr 30 15:01:03 ds3000 /usr/sbin/openser[7644]: child process 7648 exited by a signal 9 Apr 30 15:01:08 ds3000 /usr/sbin/openser[7644]: core was not generated Apr 30 15:01:08 ds3000 /usr/sbin/openser[7644]: INFO: terminating due to SIGCHLD Apr 30 15:01:14 ds3000 /usr/sbin/openser[7657]: INFO: signal 15 received Apr 30 15:01:14 ds3000 /usr/sbin/openser[7657]: Memory status (pkg): Apr 30 15:01:14 ds3000 /usr/sbin/openser[7657]: qm_status (0x8145960): Apr 30 15:01:14 ds3000 /usr/sbin/openser[7657]: heap size= 1048576 Apr 30 15:01:14 ds3000 /usr/sbin/openser[7659]: INFO: signal 15 received Apr 30 15:01:14 ds3000 /usr/sbin/openser[7653]: INFO: signal 15 received Apr 30 15:01:14 ds3000 /usr/sbin/openser[7653]: Memory status (pkg): Apr 30 15:01:14 ds3000 /usr/sbin/openser[7650]: INFO: signal 15 received

Any hints how to debug this?

regards klaus

Daniel-Constantin Mierla wrote:

...

Hello Klaus,

On 04/30/07 13:55, Klaus Darilion wrote:

...

Daniel-Constantin Mierla wrote:

...

Hello Klaus,

On 04/27/07 09:27, Klaus Darilion wrote:

...

Hi Daniel!

I've tried with t_release and it was running fine several hours without leaking. But then, unfortunately openser terminated with signal 9. I've never seen this before.

signal 9 is KILL, it is very strange if it was not issued by a user or other process.

We discovered the issue (tm/uac.c, line 224), ther eis flag that is kept to see if there was some operation with the transaction, but in case of handle_publish() that flag is set by TM api when sending NOTIFY. The patch is trivial, removing a line, but we have to investigate if there are some other effects -- so it may take a while. t_release() should solve meanwhile.

Should solve the memory-leak - but the signal 9?

It might be that it took so long to write the mem long at shut down and openser killed itself. If it was not due to a openser stop, then I am not aware of any case when signal 9 is sent unless issued by user.

Cheers, Daniel

...

regards klaus

Hi Bogdan!

I've attached with strace to all openser threads and waited for the crash. Here is the strace log of the "attendant" process (ID=0):

Process 2340 attached - interrupt to quit pause() = ? ERESTARTNOHAND (To be restarted) --- SIGCHLD (Child exited) @ 0 (0) --- sigreturn() = ? (mask now []) waitpid(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], WNOHANG) = 2344 time([1178548261]) = 1178548261 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=801, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=801, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=801, ...}) = 0 send(3, "<134>May 7 16:31:01 /usr/sbin/o"..., 86, MSG_NOSIGNAL) = 86 time([1178548262]) = 1178548262 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=801, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=801, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=801, ...}) = 0 send(3, "<134>May 7 16:31:02 /usr/sbin/o"..., 69, MSG_NOSIGNAL) = 69 waitpid(-1, 0xbfd58ecc, WNOHANG) = 0 time([1178548262]) = 1178548262 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=801, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=801, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=801, ...}) = 0 send(3, "<134>May 7 16:31:02 /usr/sbin/o"..., 79, MSG_NOSIGNAL) = 79 kill(0, SIGTERM) = 0 --- SIGTERM (Terminated) @ 0 (0) --- sigreturn() = ? (mask now []) rt_sigaction(SIGALRM, {0x8067830, [ALRM], SA_RESTART}, {SIG_DFL}, 8) = 0 alarm(60) = 0 wait4(-1, NULL, 0, NULL) = 2350 --- SIGCHLD (Child exited) @ 0 (0) --- sigreturn() = ? (mask now []) wait4(-1, NULL, 0, NULL) = 2345 --- SIGCHLD (Child exited) @ 0 (0) --- sigreturn() = ? (mask now []) wait4(-1, NULL, 0, NULL) = 2349 --- SIGCHLD (Child exited) @ 0 (0) --- sigreturn() = ? (mask now []) wait4(-1, NULL, 0, NULL) = 2341 --- SIGCHLD (Child exited) @ 0 (0) --- sigreturn() = ? (mask now []) wait4(-1, NULL, 0, NULL) = 2347 --- SIGCHLD (Child exited) @ 0 (0) --- sigreturn() = ? (mask now []) wait4(-1, NULL, 0, NULL) = 2346 --- SIGCHLD (Child exited) @ 0 (0) --- sigreturn() = ? (mask now []) wait4(-1, NULL, 0, NULL) = 2348 --- SIGCHLD (Child exited) @ 0 (0) --- sigreturn() = ? (mask now []) wait4(-1, NULL, 0, NULL) = 2342 --- SIGCHLD (Child exited) @ 0 (0) --- sigreturn() = ? (mask now []) wait4(-1, NULL, 0, NULL) = ? ERESTARTSYS (To be restarted) --- SIGALRM (Alarm clock) @ 0 (0) --- kill(0, SIGKILL) = 0 +++ killed by SIGKILL +++ Process 2340 detached

If I read it correct, the SIGKILL is sent by this process, after sending SIGTERM to all its childs. The SIGTERM is sent, because a child exited. But which child? And why?

The openser log says: May 7 16:31:02 debian /usr/sbin/openser[2340]: child process 2344 exited by a signal 9 May 7 16:31:02 debian /usr/sbin/openser[2340]: core was not generated May 7 16:31:02 debian /usr/sbin/openser[2340]: INFO: terminating due to SIGCHLD

To me this looks like 2344 (a UDP thread) exited with signal 9. Thus, the main thread receives SIGCHLD and then sends SIGTERM and afterwards SIGKILL to all other threads and itself.

But why received the thread 2344 a SIGKILL and who sent the SIGKILL?

I need some more debugging tips. Bogdan, you mentioned gdb - how can I debug this with gdb?

regards klaus

Bogdan-Andrei Iancu wrote:

Hi Klaus,

I applied on SVN the fix for the TM memory leak - it should not happen anymore now, even if you do not use t_release()...

regarding the openser stop reacting - can you attach with gdb to see what the process are done?

regards, bogdan

Klaus Darilion wrote:

...

Hi Daniel!

Summary:

• Without t_release() (no modifications to source code) openser leaks

memory.

• with t_release() openser does not leak. But after some time there is

strange behaviour, e.g.: -: openser stops reacting for some minutes and afterwards gets terminated with signal 9. When openser stops working the load increase to > 40. This happend 3 times now. -: openser stops reacting for some minutes and the linux PC where openser is running gets unresponsive. No login. Open SSH sessions are unresponsive. I had to reboot the PC. Happend 1 time.

Maybe this is not pure openser related, but a problem with openser and Linux (as I had to reboot the server one time).

Any hints how to debug this?

regards klaus

On Dienstag, 8. Mai 2007, Klaus Darilion wrote:

Hi again!

This time I've started openser with only 1 child and attached with gdb to the UDP thread in hope that gdb will show more info about the signal 9. But without luck:

# gdb /usr/sbin/openser 2855 ... Failed to read a valid object file image from memory. [..] I have run out of ideas to debug this - thus, please send me your ideas.

Hello Klaus,

probably a stupid question, but is the machine otherwise stable? I have seen mysterious core dumps in the past due bad memory or temperature problems on several machines.

Memcheck and cpuburn could help debug such a problem.

Just guessing,

Henning

Klaus Darilion wrote:

I've tried with t_release and it was running fine several hours without leaking. But then, unfortunately openser terminated with signal 9. I've never seen this before. Do you know what this means?

FYI: http://openser.org/dokuwiki/doku.php/troubleshooting:signal9

regards klaus