[Users] BUG: qm_free ?
Papadopoulos Georgios
geop at altectelecoms.gr
Wed May 30 15:41:35 CEST 2007
No, I don't use t_newtran at all. Now I am trying to get this to happen
again but no luck... Anyway, is it something to worry about or will the
leaked transactions be cleaned up?
Regards,
George
ps. I cannot connect to the SVN, is it down?
> -----Original Message-----
> From: Bogdan-Andrei Iancu [mailto:bogdan at voice-system.ro]
> Sent: Wednesday, May 30, 2007 4:33 PM
> To: Papadopoulos Georgios
> Cc: Klaus Darilion; users at openser.org
> Subject: Re: [Users] BUG: qm_free ?
>
> Perfect - thanks for testing - I applied the fix also on 1.2.
>
> regarding the Warning message: there were in 1.2 some fixes
> related to detection of the transactions leak due scripting
> issues. Probably 1.1 was not able to detect and report all cases.
>
> Do you use in your script t_newtran() function? typically,
> when you use this function without terminating the
> transaction (relay or reply in a stateful way), you will get
> the warning. can you check this on your script?
>
> regards,
> bogdan
>
> Papadopoulos Georgios wrote:
> > Solved! Thank you!
> > However, before applying the patch, I increased private
> memory to 2MB
> > and it did not crash. Then I set it back to 1MB, applied
> the patch and
> > it worked.
> >
> > Now I have a ton of these messages:
> > WARNING: script writer didn't release transaction What does
> it mean?
> > This was not present in 1.1 and my config script has not changed
> > (except for the necessary changes for the migration)
> >
> > Regards
> >
> > George
> >
> >
> >
> >
> >
> >> -----Original Message-----
> >> From: Bogdan-Andrei Iancu [mailto:bogdan at voice-system.ro]
> >> Sent: Wednesday, May 30, 2007 2:18 PM
> >> To: Papadopoulos Georgios
> >> Cc: Klaus Darilion; users at openser.org
> >> Subject: Re: [Users] BUG: qm_free ?
> >>
> >> Hi George,
> >>
> >> I managed to find the bug without the backtrace. I applied
> a fix for
> >> development version (SVN trunk) and find attached a patch
> for 1.2 -
> >> could you please give it a try and see if it solves the bug?
> >>
> >> thanks and regards,
> >> bogdan
> >>
> >> Papadopoulos Georgios wrote:
> >>
> >>> I cannot find a core file anywhere. Where should it be? I can
> >>> reproduce the problem very easily, so please give me more
> >>>
> >> instructions
> >>
> >>> on how to get the coredump+backtrace that you need.
> >>>
> >>> thank you
> >>>
> >>> George
> >>>
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Klaus Darilion [mailto:klaus.mailinglists at pernau.at]
> >>>> Sent: Tuesday, May 29, 2007 8:34 PM
> >>>> To: Papadopoulos Georgios
> >>>> Cc: users at openser.org
> >>>> Subject: Re: [Users] BUG: qm_free ?
> >>>>
> >>>> Maybe a bug?
> >>>>
> >>>> Can you provide a coredump+backtrace? I'm sure Bogdan or
> >>>>
> >> Daniel will
> >>
> >>>> take a look at it.
> >>>>
> >>>> regards
> >>>> klaus
> >>>>
> >>>> Papadopoulos Georgios wrote:
> >>>>
> >>>>
> >>>>> Hi Klaus,
> >>>>>
> >>>>> It crashes immediately, I have no time to do anything. I
> >>>>>
> >>>>>
> >>>> tried again
> >>>>
> >>>>
> >>>>> and I increased the log level. So now it is process 692.
> >>>>>
> >>>>>
> >>>> Here is part
> >>>>
> >>>>
> >>>>> of the output. I can send you the whole output if that helps.
> >>>>>
> >>>>> thank you
> >>>>>
> >>>>> George
> >>>>>
> >>>>>
> >>>>> 16(692) DEBUG: timer routine:5,tl=0xb61abdc4 next=0xb61adfd8,
> >>>>> timeout=3000000
> >>>>> 16(692) DEBUG: retransmission_handler : request resending
> >>>>> (t=0xb61abc78, REGISTER ... )
> >>>>> 16(692) DEBUG:tm:set_timer: relative timeout is 3000000
> >>>>> 16(692) DEBUG: add_to_tail_of_timer[6]: 0xb61abdc4 (6000000)
> >>>>> 16(692) DEBUG: retransmission_handler : done
> >>>>> 16(692) DEBUG: timer routine:5,tl=0xb61adfd8 next=(nil),
> >>>>> timeout=3000000
> >>>>> 16(692) DEBUG: retransmission_handler : request resending
> >>>>> (t=0xb61ade8c, REGISTER ... )
> >>>>> 16(692) DEBUG:tm:set_timer: relative timeout is 3000000
> >>>>> 16(692) DEBUG: add_to_tail_of_timer[6]: 0xb61adfd8 (6000000)
> >>>>> 16(692) DEBUG: retransmission_handler : done
> >>>>> 16(692) DEBUG: timer routine:0,tl=0xb6182810 next=0xb61845ac,
> >>>>> timeout=3
> >>>>> 16(692) DEBUG: final_response_handler:stop retr. and send CANCEL
> >>>>> (0xb61826a8)
> >>>>> 16(692) DEBUG:tm:t_should_relay_response: T_code=0, new_code=408
> >>>>> 16(692) DEBUG:tm:t_pick_branch: picked branch 0, code 408
> >>>>> 16(692) DEBUG:tm:t_should_relay_response: dns-failover
> >>>>>
> >>>>>
> >>>> test: branch=0,
> >>>>
> >>>>
> >>>>> last_recv=408, flags=0
> >>>>> 16(692) DEBUG:tm:t_should_relay_response: trying
> >>>>>
> >> DNS-based failover
> >>
> >>>>> 16(692) qm_malloc(0xb5f99000, 72) called from blacklists.c:
> >>>>> add_rule_to_list(377)
> >>>>> 16(692) qm_malloc(0xb5f99000, 72) returns address
> 0xb6278238 frag.
> >>>>> 0xb6278220 (size=72) on 1 -th hit
> >>>>> 16(692) DBG:add_list_to_head: adding to bl dns
> >>>>>
> >> 0xb6278238,0xb6278238
> >>
> >>>>> 16(692) DEBUG:tm:local_reply: branch=0, save=0, winner=0
> >>>>> 16(692) DEBUG:tm:local_reply: local transaction completed
> >>>>> 16(692) DEBUG: add_to_tail_of_timer[2]: 0xb61826f0 (8)
> >>>>> 16(692) DEBUG: final_response_handler : done
> >>>>> 16(692) DEBUG: timer routine:0,tl=0xb61845ac next=0xb6186c6c,
> >>>>> timeout=3
> >>>>> 16(692) DEBUG: timer routine:0,tl=0xb6186c6c next=0xb6188d98,
> >>>>> timeout=3
> >>>>> 16(692) DEBUG: final_response_handler:stop retr. and send CANCEL
> >>>>> (0xb6186b04)
> >>>>> 16(692) DEBUG:tm:t_should_relay_response: T_code=0, new_code=408
> >>>>> 16(692) DEBUG:tm:t_pick_branch: picked branch 0, code 408
> >>>>> 16(692) DEBUG:tm:t_should_relay_response: dns-failover
> >>>>>
> >>>>>
> >>>> test: branch=0,
> >>>>
> >>>>
> >>>>> last_recv=408, flags=0
> >>>>> 16(692) DEBUG:tm:t_should_relay_response: trying
> >>>>>
> >> DNS-based failover
> >>
> >>>>> 16(692) qm_malloc(0xb5f99000, 72) called from blacklists.c:
> >>>>> add_rule_to_list(377)
> >>>>> 16(692) qm_malloc(0xb5f99000, 72) returns address
> 0xb62782b0 frag.
> >>>>> 0xb6278298 (size=72) on 1 -th hit
> >>>>> 16(692) DBG:add_list_to_head: adding to bl dns
> >>>>>
> >> 0xb62782b0,0xb62782b0
> >>
> >>>>> 16(692) qm_free(0x8151c40, 0xb62782b0), called from
> blacklists.c:
> >>>>> rm_dups(440)
> >>>>> 16(692) BUG: qm_free: bad pointer 0xb62782b0 (out of memory
> >>>>>
> >>>>>
> >>>> block!) -
> >>>>
> >>>>
> >>>>> aborting
> >>>>>
> >>>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Klaus Darilion [mailto:klaus.mailinglists at pernau.at]
> >>>>>> Sent: Tuesday, May 29, 2007 5:48 PM
> >>>>>> To: Papadopoulos Georgios
> >>>>>> Cc: users at openser.org
> >>>>>> Subject: Re: [Users] BUG: qm_free ?
> >>>>>>
> >>>>>> The bug was in process 32133
> >>>>>>
> >>>>>> There are no log statements from this process. What what this
> >>>>>> process?
> >>>>>> (use "openserctl ps" to see the PIDs of the processes)
> >>>>>>
> >>>>>> regards
> >>>>>> klaus
> >>>>>>
> >>>>>> Papadopoulos Georgios wrote:
> >>>>>>
> >>>>>>
> >>>>>>> Hello all,
> >>>>>>>
> >>>>>>> I am having some trouble geting Openser 1.2 to work. It
> >>>>>>>
> >>>>>>>
> >>>>>> crashes pretty
> >>>>>>
> >>>>>>
> >>>>>>> much immediately. I have tried it successfully on a test
> >>>>>>>
> >>>>>>>
> >>>>>> system but as
> >>>>>>
> >>>>>>
> >>>>>>> soon as I start it in the production environment, it
> >>>>>>>
> >>>>>>>
> >>>>>> crashes. Any ideas?
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> Thank you for any help.
> >>>>>>>
> >>>>>>> George
> >>>>>>>
> >>>>>>>
> >>>>>>> 0(0) INFO: statistics manager successfully initialized
> >>>>>>> 0(0) StateLess module - initializing
> >>>>>>> 0(0) TM - initializing...
> >>>>>>> 0(0) Maxfwd module- initializing
> >>>>>>> 0(0) INFO:ul_init_locks: locks array size 512
> >>>>>>> 0(0) TextOPS - initializing
> >>>>>>> 0(0) AVPops - initializing
> >>>>>>> 0(0) permissions - initializing
> >>>>>>> 0(0) WARNING: File not found:
> >>>>>>>
> >>>>>>>
> >>>>>> /root/openser/config/permissions.allow
> >>>>>>
> >>>>>>
> >>>>>>> 0(0) Default allow file
> >>>>>>>
> >>>>>>>
> >>>>>> (/root/openser/config/permissions.allow) not
> >>>>>>
> >>>>>>
> >>>>>>> found => empty rule set
> >>>>>>> 0(0) WARNING: File not found:
> >>>>>>>
> >>>>>>>
> >>>> /root/openser/config/permissions.deny
> >>>>
> >>>>
> >>>>>>> 0(0) Default deny file
> >>>>>>>
> >>>>>>>
> >>>> (/root/openser/config/permissions.deny) not
> >>>>
> >>>>
> >>>>>>> found => empty rule set
> >>>>>>> 0(0) AUTH module - initializing
> >>>>>>> 0(0) AUTH_DB module - initializing
> >>>>>>> 0(0) ACC - initializing
> >>>>>>> 0(0) INFO:xl_parse_name: using hdr type (4) instead of <From>
> >>>>>>> 0(0) INFO:xl_parse_name: using hdr type (3) instead of <To>
> >>>>>>> 0(0) INFO:textops:hname_fixup: using hdr type name <Nat-flag>
> >>>>>>> 0(0) INFO:textops:hname_fixup: using hdr type (31)
> instead of
> >>>>>>> <Remote-Party-ID>
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is initially 110592
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is finally 262142
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is initially 110592
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is finally 262142
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is initially 110592
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is finally 262142
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is initially 110592
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is finally 262142
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is initially 110592
> >>>>>>> 0(0) INFO: udp_init: SO_RCVBUF is finally 262142
> >>>>>>> 1(32087) INFO:mi_fifo:mi_child_init(1): extra fifo
> >>>>>>>
> >>>>>>>
> >>>>>> listener processes
> >>>>>>
> >>>>>>
> >>>>>>> created
> >>>>>>> 2(32093) pre_auth(): Credentials received are not
> >>>>>>>
> >> filled properly
> >>
> >>>>>>> 1(32087) ERROR: parse_to_param: spitting out [<] in status 22
> >>>>>>> 1(32087) ERROR: parse_to: invalid To - unexpected end of
> >>>>>>>
> >>>>>>>
> >>>>>> header in
> >>>>>>
> >>>>>>
> >>>>>>> state 22
> >>>>>>> 1(32087) ERROR: get_hdr_field: bad to header
> >>>>>>> 1(32087) ERROR: bad header field
> >>>>>>> 1(32087) ERROR:maxfwd:is_maxfwd_present : parsing
> >>>>>>>
> >>>>>>>
> >>>>>> MAX_FORWARD header
> >>>>>>
> >>>>>>
> >>>>>>> failed!
> >>>>>>> 1(32087) ERROR: parse_to_param: spitting out [<] in status 22
> >>>>>>> 1(32087) ERROR: parse_to: invalid To - unexpected end of
> >>>>>>>
> >>>>>>>
> >>>>>> header in
> >>>>>>
> >>>>>>
> >>>>>>> state 22
> >>>>>>> 1(32087) ERROR: get_hdr_field: bad to header
> >>>>>>> 1(32087) ERROR: bad header field
> >>>>>>> 1(32087) find_first_route: Error while parsing headers
> >>>>>>> 1(32087) ERROR: parse_to_param: spitting out [<] in status 22
> >>>>>>> 1(32087) ERROR: parse_to: invalid To - unexpected end of
> >>>>>>>
> >>>>>>>
> >>>>>> header in
> >>>>>>
> >>>>>>
> >>>>>>> state 22
> >>>>>>> 1(32087) ERROR: get_hdr_field: bad to header
> >>>>>>> 1(32087) ERROR: bad header field
> >>>>>>> 1(32087) find_credentials(): Error while parsing headers
> >>>>>>> 1(32087) pre_auth(): Error while looking for credentials
> >>>>>>> 1(32087) ERROR: parse_to_param: spitting out [<] in status 22
> >>>>>>> 1(32087) ERROR: parse_to: invalid To - unexpected end of
> >>>>>>>
> >>>>>>>
> >>>>>> header in
> >>>>>>
> >>>>>>
> >>>>>>> state 22
> >>>>>>> 1(32087) ERROR: get_hdr_field: bad to header
> >>>>>>> 1(32087) ERROR: bad header field
> >>>>>>> 1(32087) ERROR: parse_to_param: spitting out [<] in status 22
> >>>>>>> 1(32087) ERROR: parse_to: invalid To - unexpected end of
> >>>>>>>
> >>>>>>>
> >>>>>> header in
> >>>>>>
> >>>>>>
> >>>>>>> state 22
> >>>>>>> 1(32087) ERROR: get_hdr_field: bad to header
> >>>>>>> 1(32087) ERROR: bad header field
> >>>>>>> 1(32087) ERROR: build_res_buf_from_sip_req: alas,
> >>>>>>>
> >> parse_headers
> >>
> >>>>>>> failed
> >>>>>>> 1(32087) pre_auth(): Error while sending 400 reply
> >>>>>>> INFO: started (pid: 32073)
> >>>>>>> [root at ser1 sbin]# 1(32087) pre_auth(): Credentials
> >>>>>>>
> >>>>>>>
> >>>>>> received are not
> >>>>>>
> >>>>>>
> >>>>>>> filled properly
> >>>>>>> 16(32133) BUG: qm_free: bad pointer 0xb648f38c (out of
> >>>>>>>
> >>>>>>>
> >>>>>> memory block!)
> >>>>>>
> >>>>>>
> >>>>>>> - aborting
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 2(32093) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 2(32093) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 2(32093) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 2(32093) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 2(32093) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 2(32093) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) ERROR: parse_via: invalid char <=> in state 201
> >>>>>>> 3(32096) error: parse_via_param
> >>>>>>> 3(32096) ERROR: parse_via on: <SIP/2.0/UDP
> >>>>>>> 87.202.97.60:59945;branc==z9hG4bK2e302aa9;rport
> >>>>>>> From: <sip:gdt at sip.i-call.gr>;tag=as52e610af
> >>>>>>> To: <sip:gdt at sip.i-call.gr>
> >>>>>>> Call-ID: 1b7dabcf26487db47f520b5c7195958e at gdt.ath.cx
> >>>>>>> <mailto:1b7dabcf26487db47f520b5c7195958e at gdt.ath.cx>
> >>>>>>> CSeq: 4471 REGISTER
> >>>>>>> User-Agent: Asterisk PBX
> >>>>>>> Max-Forwards: 70
> >>>>>>> Expires: 120
> >>>>>>> Contact: <sip:s at 87.202.97.60:59945>
> >>>>>>> Event: registration
> >>>>>>> Content-Length: 0
> >>>>>>>
> >>>>>>> 3(32096) ERROR: parse_via parse error, parsed so
> >>>>>>>
> >>>>>>>
> >>>> far:<SIP/2.0/UDP
> >>>>
> >>>>
> >>>>>>> 87.202.97.60:59945;branc=>
> >>>>>>> 3(32096) ERROR: get_hdr_field: bad via
> >>>>>>> 3(32096) ERROR: bad header field
> >>>>>>> 3(32096) ERROR: parse_msg: message=<REGISTER
> sip:sip.i-call.gr
> >>>>>>> SIP/2.0
> >>>>>>> Via: SIP/2.0/UDP
> 87.202.97.60:59945;branc==z9hG4bK2e302aa9;rport
> >>>>>>> From: <sip:gdt at sip.i-call.gr>;tag=as52e610af
> >>>>>>> To: <sip:gdt at sip.i-call.gr>
> >>>>>>> Call-ID: 1b7dabcf26487db47f520b5c7195958e at gdt.ath.cx
> >>>>>>> CSeq: 4471 REGISTER
> >>>>>>> User-Agent: Asterisk PBX
> >>>>>>> Max-Forwards: 70
> >>>>>>> Expires: 120
> >>>>>>> Contact: <sip:s at 87.202.97.60:59945>
> >>>>>>> Event: registration
> >>>>>>> Content-Length: 0
> >>>>>>>
> >>>>>>> 3(32096) ERROR: receive_msg: parse_msg failed
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 3(32096) WARNING: script writer didn't release transaction
> >>>>>>> 2(32093) WARNING: script writer didn't release transaction
> >>>>>>> 1(32087) WARNING: script writer didn't release transaction
> >>>>>>> 2(32093) WARNING: script writer didn't release transaction
> >>>>>>> 2(32093) WARNING: script writer didn't release transaction
> >>>>>>> 0(32073) child process 32133 exited by a signal 6
> >>>>>>> 0(32073) core was generated
> >>>>>>> 0(32073) INFO: terminating due to SIGCHLD
> >>>>>>> 1(32087) INFO: signal 15 received
> >>>>>>> 2(32093) INFO: signal 15 received
> >>>>>>> 3(32096) INFO: signal 15 received
> >>>>>>> 5(32102) INFO: signal 15 received
> >>>>>>> 6(32103) INFO: signal 15 received
> >>>>>>> 13(32123) INFO: signal 15 received
> >>>>>>> 14(32127) INFO: signal 15 received
> >>>>>>> 15(32130) INFO: signal 15 received
> >>>>>>> 1(32087) 4(32099) 7(32104) INFO: signal 15 received
> >>>>>>> 8(32109) INFO: signal 15 received
> >>>>>>> 9(32112) INFO: signal 15 received
> >>>>>>> 10(32115) INFO: signal 15 received
> >>>>>>> 11(32118) INFO: signal 15 received
> >>>>>>> 12(32121) INFO: signal 15 received
> >>>>>>> INFO: signal 15 received
> >>>>>>> INFO: signal 15 received
> >>>>>>> 0(32073) INFO:mi_fifo:mi_destroy: seems that fifo child
> >>>>>>>
> >>>>>>>
> >>>> is already
> >>>>
> >>>>
> >>>>>>> dead!
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Disclaimer
> >>>>>>> The information in this e-mail and any attachments is
> >>>>>>>
> >>>>>>>
> >>>>>> confidential. It is intended solely for the attention and
> >>>>>>
> >>>>>>
> >>>> use of the
> >>>>
> >>>>
> >>>>>> named addressee(s). If you are not the intended recipient,
> >>>>>>
> >>>>>>
> >>>> or person
> >>>>
> >>>>
> >>>>>> responsible for delivering this information to the intended
> >>>>>> recipient, please notify the sender immediately. Unless
> >>>>>>
> >>>>>>
> >>>> you are the
> >>>>
> >>>>
> >>>>>> intended recipient or his/her representative you are not
> >>>>>>
> >>>>>>
> >>>> authorized
> >>>>
> >>>>
> >>>>>> to, and must not, read, copy, distribute, use or retain
> >>>>>>
> >>>>>>
> >>>> this message
> >>>>
> >>>>
> >>>>>> or any part of it. E-mail transmission cannot be
> >>>>>>
> >> guaranteed to be
> >>
> >>>>>> secure or error-free as information could be intercepted,
> >>>>>>
> >>>>>>
> >>>> corrupted,
> >>>>
> >>>>
> >>>>>> lost, destroyed, arrive late or incomplete, or contain viruses.
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>
> ---------------------------------------------------------------------
> >>
> >>>>
> >>>>
> >>>>>> -
> >>>>>>
> >>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> Users mailing list
> >>>>>>> Users at openser.org
> >>>>>>> http://openser.org/cgi-bin/mailman/listinfo/users
> >>>>>>>
> >>>>>>>
> >>> _______________________________________________
> >>> Users mailing list
> >>> Users at openser.org
> >>> http://openser.org/cgi-bin/mailman/listinfo/users
> >>>
> >>>
> >>>
> >>
> >
> >
>
>
More information about the Users
mailing list