Re: [SR-Dev] [Serdev] CVS:commitlog: sip_router/modules/tm t_cancel.c t_fwd.c t_reply.c t_reply.h t_suspend.c t_suspend.h tm.c tm_load.c tm_load.h
12 Nov
2008
12 Nov
'08
9:32 a.m.
[crossposting since this is of general interest]
On Nov 12, 2008 at 00:30, Klaus Darilion <klaus.mailinglists(a)pernau.at> wrote:
Hi Miklos!
This sounds great. Can you make a sample function which uses the new
feature?
We are thinking of doing a dns_prefetch module using this.
The sip-router.cfg will look like:
route{
...
dns_prefetch("uri", ROUTE_DNS_OK);
# end of script
}
route[ROUTE_DNS_OK]{
if (dns_error){
...
}
# continue normal processing
# the uri was resolved and is in the dns cache
...
t_relay()
...
}
However don't expect something quickly as everybody is quite busy and we
would still need to write a dns resolver process pool (time consuming).
Note that this is not the "final" async. solution since it's not
transparent for the script writer (scripts would have to be written in a
special way), works only with tm and it's also a little inefficient for
dns usage.
In the long term we will have completely transparent async. support for dns
and maybe db lookups (though it's controversial if we really need it for
normal db lookups). This will take much longer and will require among
other things reviewing all the global variable use (most of the global
variables will either need to be registered or allocated in some known
special place).
If Miklos doesn't beat me to it, I'll probably start working on it only
after the core merging is complete.
Can this also be used for DNS lookups and TCP/TLS connection setup?
Yes for DNS (see above) and in general for any route-level async processing
involving tm (e.g. lookup some part of the message in a slow DB and
execute automatically another route when the DB responds).
It cannot be used for TCP/TLS, but it's not needed anyway. In ser TCP
connection setup and TCP send is already asynchronous, at least if
you have tcp_buf_write=yes in ser.cfg (I agree it's not a very well
chosen name :-)). TLS does not support it yet, so if you use TLS
you should not set this option (that's why it's not yet the default).
TLS support for it requires non-trivial changes in the tcp core and it
will probably come after the merge (it's started already but far from
finished).
(BTW: most of the core new options documentation in ser can be found
in NEWS)
Andrei
Further, also for Kamailio cooperation better documentation would be
needed. Thus, instead of writing long CVS commit messages I think it
would be better to write the text into the module documentation, and
then copy/paste it into the CVS commit.
regards
Klaus
Miklos Tirpak wrote:
tirpi 2008/11/10 13:47:03 CET
SER CVS Repository
Modified files:
modules/tm t_cancel.c t_fwd.c t_reply.c t_reply.h
tm.c tm_load.c tm_load.h
Added files:
modules/tm t_suspend.c t_suspend.h
Log:
t_suspend() and t_continue() functions are introduced.
These fuctions can be used by other modules to implement
asynchronous actions: t_suspend() saves the transaction, returns its
identifiers, and t_continue() continues the SIP request processing.
(The transaction processing does not continue from the same point
in the script, a separate route block defined by the parameter of
t_continue() is executed instead. The reply lock is held during the
route block execution.) FR timer is ticking while the
transaction is suspended, and the transaction's failure route is
executed if t_continue() is not called in time.
Missing: msg lumps are saved by t_suspend() and are not updated by
the subsequent t_relay(). This means that the modifications made
between them are lost.
Revision Changes Path
1.32 +7 -2 sip_router/modules/tm/t_cancel.c
http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/sip_router/modules/tm/t_cance…
1.107 +8 -2 sip_router/modules/tm/t_fwd.c
http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/sip_router/modules/tm/t_fwd.c…
1.162 +4 -4 sip_router/modules/tm/t_reply.c
http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/sip_router/modules/tm/t_reply…
1.31 +7 -1 sip_router/modules/tm/t_reply.h
http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/sip_router/modules/tm/t_reply…
1.1 +154 -0 sip_router/modules/tm/t_suspend.c (new)
http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/sip_router/modules/tm/t_suspe…
1.1 +42 -0 sip_router/modules/tm/t_suspend.h (new)
http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/sip_router/modules/tm/t_suspe…
1.177 +3 -1 sip_router/modules/tm/tm.c
http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/sip_router/modules/tm/tm.c.di…
1.31 +11 -1 sip_router/modules/tm/tm_load.c
http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/sip_router/modules/tm/tm_load…
1.31 +4 -1 sip_router/modules/tm/tm_load.h
http://cvs.berlios.de/cgi-bin/viewcvs.cgi/ser/sip_router/modules/tm/tm_load…
_______________________________________________
Serdev mailing list
Serdev(a)lists.iptel.org
http://lists.iptel.org/mailman/listinfo/serdev
_______________________________________________
Serdev mailing list
Serdev(a)lists.iptel.org
http://lists.iptel.org/mailman/listinfo/serdev 
12 Nov
12 Nov
10:33 a.m.
New subject: [SR-Dev] [Serdev] CVS:commitlog: sip_router/modules/tm t_cancel.c t_fwd.c t_reply.c t_reply.h t_suspend.c t_suspend.h tm.c tm_load.c tm_load.h
Andrei Pelinescu-Onciul schrieb:
Interesting. Is it really completely asynchronous with any blocking, or
is it handed over to a dedicated "TCP send" process which is then blocked?
[crossposting since this is of general interest]
On Nov 12, 2008 at 00:30, Klaus Darilion <klaus.mailinglists(a)pernau.at> wrote:
Ok. I see.
Hi Miklos!
This sounds great. Can you make a sample function which uses the new
feature?
We are thinking of doing a dns_prefetch module using this.
The sip-router.cfg will look like:
route{
...
dns_prefetch("uri", ROUTE_DNS_OK);
# end of script
}
route[ROUTE_DNS_OK]{
if (dns_error){
...
}
# continue normal processing
# the uri was resolved and is in the dns cache
...
t_relay()
...
} However don't expect something quickly as
everybody is quite busy and we
would still need to write a dns resolver process pool (time consuming).
I still wonder if the DNS resolver in ser is really a good idea. For
example if you take a look at bind, it is really mature and still they
fix several issues with each release - and doing all the tricks to avoid
cache poisoning and handling DNSSEC correctly is not easy. Thus, maybe
having a DNS cache in ser can make sense, but having a full resolver IMO
not.
Can this also
be used for DNS lookups and TCP/TLS connection setup?
Yes for DNS (see above) and in general for any route-level async processing
involving tm (e.g. lookup some part of the message in a slow DB and
execute automatically another route when the DB responds).
It cannot be used for TCP/TLS, but it's not needed anyway. In ser TCP
connection setup and TCP send is already asynchronous, at least if
you have tcp_buf_write=yes in ser.cfg (I agree it's not a very well (BTW: most of the core new options documentation in
ser can be found
in NEWS)
Probably this should be merged too.
thanks
klaus
11:16 a.m.
New subject: [SR-Dev] [Serdev] CVS:commitlog: sip_router/modules/tm t_cancel.c t_fwd.c t_reply.c t_reply.h t_suspend.c t_suspend.h tm.c tm_load.c tm_load.h
On Nov 12, 2008 at 10:33, Klaus Darilion <klaus.mailinglists(a)pernau.at> wrote:
Interesting. Is it really completely asynchronous with any blocking, or
is it handed over to a dedicated "TCP send" process which is then blocked?
Yes, it's completely asynchronous, no process is blocked.
A brief description on how it's working for send on an already opened
connection:
- check if the connection has data buffered. If it does it means that
either is still connecting or a previous send failed to be executed
in-place and queued the data => add to the queued data, return.
- if no data buffered try non-blocking send. If it fails queue data
to the ser internal tcp connection structure and enable waiting for
POLLOUT events (socket send buffer has enough space). return.
For connect is similar: connect() non blocking, queue data, return.
When a socket can be written, the main tcp process will write the queued
data (or as much as possible from it).
-
Yes.
Andrei
Andrei Pelinescu-Onciul schrieb:
Sorry for the name, I was not talking about a full resolver.
The DNS cache uses in fact a minimalistic resolver. We would just need to
replace res_search with a res_mkquery and then "manually" send the
request and wait for an answer in a poll() loop (we won't actually use
poll, but the io_wait.h optimized stuff that uses epoll(), kqueue() a.s.o
depending on the system).
We don't need a recursive resolver, we only need something minimal.
One should still use a close DNS server since the internal resolver does
not support recursive queries, DNSSEC a.s.o (we could add them, but so
far there would be little benefit).
[crossposting since this is of general interest]
On Nov 12, 2008 at 00:30, Klaus Darilion <klaus.mailinglists(a)pernau.at>
wrote:
Ok. I see.
Hi Miklos!
This sounds great. Can you make a sample function which uses the new
feature?
We are thinking of doing a dns_prefetch module using this.
The sip-router.cfg will look like:
route{
...
dns_prefetch("uri", ROUTE_DNS_OK);
# end of script
}
route[ROUTE_DNS_OK]{
if (dns_error){
...
}
# continue normal processing
# the uri was resolved and is in the dns cache
...
t_relay()
...
} However don't expect something quickly as
everybody is quite busy and we
would still need to write a dns resolver process pool (time consuming).
I still wonder if the DNS resolver in ser is really a good idea. For
example if you take a look at bind, it is really mature and still they
fix several issues with each release - and doing all the tricks to avoid
cache poisoning and handling DNSSEC correctly is not easy. Thus, maybe
having a DNS cache in ser can make sense, but having a full resolver IMO
not. Can this
also be used for DNS lookups and TCP/TLS connection setup?
Yes for DNS (see above) and in general for any route-level async
processing
involving tm (e.g. lookup some part of the message in a slow DB and
execute automatically another route when the DB responds).
It cannot be used for TCP/TLS, but it's not needed anyway. In ser TCP
connection setup and TCP send is already asynchronous, at least if
you have tcp_buf_write=yes in ser.cfg (I agree it's not a very well (BTW: most of the core new options documentation
in ser can be found
in NEWS)
Probably this should be merged too. 
2:47 p.m.
New subject: [SR-Dev] [Serdev] CVS:commitlog: sip_router/modules/tm t_cancel.c t_fwd.c t_reply.c t_reply.h t_suspend.c t_suspend.h tm.c tm_load.c tm_load.h
Hi Klaus,
regarding your original question, you are right, I have not documented
the functions yet. I will add the doc to the devel section of the readme
as soon as I will have some time.
I also have a small module that I was testing with, I can also commit
this module if you wish as a sample. There is no real functionality in
the module, it only suspends the transaction for some seconds, and when
the timer fires, the transaction continues. It is not even good for
performance testing, because there is a single timer process that calls
t_continue() for all the transactions. As Andrei wrote, writing a real
async function would take more time, this was only a baby step.
Regards,
Miklos
On 11/12/2008 11:16 AM, Andrei Pelinescu-Onciul wrote:
On Nov 12, 2008 at 10:33, Klaus Darilion
<klaus.mailinglists(a)pernau.at> wrote:
Interesting. Is it really completely asynchronous with any blocking, or
is it handed over to a dedicated "TCP send" process which is then blocked?
Yes, it's completely asynchronous, no process is blocked.
A brief description on how it's working for send on an already opened
connection:
- check if the connection has data buffered. If it does it means that
either is still connecting or a previous send failed to be executed
in-place and queued the data => add to the queued data, return.
- if no data buffered try non-blocking send. If it fails queue data
to the ser internal tcp connection structure and enable waiting for
POLLOUT events (socket send buffer has enough space). return.
For connect is similar: connect() non blocking, queue data, return.
When a socket can be written, the main tcp process will write the queued
data (or as much as possible from it).
-
Yes.
Andrei
Andrei Pelinescu-Onciul schrieb:
Sorry for the name, I was not talking about a full resolver.
The DNS cache uses in fact a minimalistic resolver. We would just need to
replace res_search with a res_mkquery and then "manually" send the
request and wait for an answer in a poll() loop (we won't actually use
poll, but the io_wait.h optimized stuff that uses epoll(), kqueue() a.s.o
depending on the system).
We don't need a recursive resolver, we only need something minimal.
One should still use a close DNS server since the internal resolver does
not support recursive queries, DNSSEC a.s.o (we could add them, but so
far there would be little benefit).
[crossposting since this is of general interest]
On Nov 12, 2008 at 00:30, Klaus Darilion <klaus.mailinglists(a)pernau.at>
wrote:
Ok. I see.
Hi Miklos!
This sounds great. Can you make a sample function which uses the new
feature?
We are thinking of doing a dns_prefetch module using this.
The sip-router.cfg will look like:
route{
...
dns_prefetch("uri", ROUTE_DNS_OK);
# end of script
}
route[ROUTE_DNS_OK]{
if (dns_error){
...
}
# continue normal processing
# the uri was resolved and is in the dns cache
...
t_relay()
...
} However don't expect something quickly as
everybody is quite busy and we
would still need to write a dns resolver process pool (time consuming).
I still
wonder if the DNS resolver in ser is really a good idea. For
example if you take a look at bind, it is really mature and still they
fix several issues with each release - and doing all the tricks to avoid
cache poisoning and handling DNSSEC correctly is not easy. Thus, maybe
having a DNS cache in ser can make sense, but having a full resolver IMO
not. Can this also be used for DNS lookups and TCP/TLS
connection setup?
Yes for DNS (see above) and in general for any route-level async
processing
involving tm (e.g. lookup some part of the message in a slow DB and
execute automatically another route when the DB responds).
It cannot be used for TCP/TLS, but it's not needed anyway. In ser TCP
connection setup and TCP send is already asynchronous, at least if
you have tcp_buf_write=yes in ser.cfg (I agree it's not a very well (BTW:
most of the core new options documentation in ser can be found
in NEWS)
Probably this should be merged too.
9 Dec
9 Dec
12:07 p.m.
New subject: [SR-Dev] [Serdev] CVS:commitlog: sip_router/modules/tm t_cancel.c t_fwd.c t_reply.c t_reply.h t_suspend.c t_suspend.h tm.c tm_load.c tm_load.h
Andrei Pelinescu-Onciul schrieb:
[crossposting since this is of general interest]
On Nov 12, 2008 at 00:30, Klaus Darilion <klaus.mailinglists(a)pernau.at> wrote:
Hi Andrei!
I wonder what happen if the asynchronous sending fails (e.g. TCP
connection is lost). Will it jump to failure route (for requests)? What
happens if the sending of the response (e.g. 200 ok) fails? Will acc
module account the failed 200 ok?
regards
klaus
Can this also be used for DNS lookups and TCP/TLS
connection setup?
Yes for DNS (see above) and in general for any route-level async processing
involving tm (e.g. lookup some part of the message in a slow DB and
execute automatically another route when the DB responds).
It cannot be used for TCP/TLS, but it's not needed anyway. In ser TCP
connection setup and TCP send is already asynchronous, at least if
you have tcp_buf_write=yes in ser.cfg (I agree it's not a very well
chosen name :-)). TLS does not support it yet, so if you use TLS
11 Dec
11 Dec
10:34 p.m.
New subject: [SR-Dev] [Serdev] CVS:commitlog: sip_router/modules/tm t_cancel.c t_fwd.c t_reply.c t_reply.h t_suspend.c t_suspend.h tm.c tm_load.c tm_load.h
On Dec 09, 2008 at 12:07, Klaus Darilion <klaus.mailinglists(a)pernau.at> wrote:
Andrei Pelinescu-Onciul schrieb:
It will time out and then jump into the failure route (it doesn't jump
directly on async send failure, it will wait for the transaction timeout).
[crossposting since this is of general interest]
On Nov 12, 2008 at 00:30, Klaus Darilion <klaus.mailinglists(a)pernau.at>
wrote:
Hi Andrei!
I wonder what happen if the asynchronous sending fails (e.g. TCP
connection is lost). Will it jump to failure route (for requests)? Can this also be used for DNS lookups and TCP/TLS
connection setup?
Yes for DNS (see above) and in general for any route-level async
processing
involving tm (e.g. lookup some part of the message in a slow DB and
execute automatically another route when the DB responds).
It cannot be used for TCP/TLS, but it's not needed anyway. In ser TCP
connection setup and TCP send is already asynchronous, at least if
you have tcp_buf_write=yes in ser.cfg (I agree it's not a very well
chosen name :-)). TLS does not support it yet, so if you use TLS What happens if the sending of the response (e.g. 200
ok) fails? Will acc
module account the failed 200 ok?
No, not directly (it has to look for the ACK). On tcp in general is not
possible to say what packet made it to the other side when a connection
is suddenly closed, while still having data in the socket send buffers.
With the async send we could make it a little better and detect _some_
packets that we know for sure we did not send (and callback tm and
fallback or execute the failure route for them), but we cannot tell
anything about all the stuff in the socket send buffers (64k+).
I think this would help only for the async. connect part (where we don't
have any kernel buffering), but for already opened connection that close
due to an error it won't help much.
Andrei
12 Dec
12 Dec
7:28 a.m.
New subject: [SR-Dev] [Serdev] CVS:commitlog: sip_router/modules/tm t_cancel.c t_fwd.c t_reply.c t_reply.h t_suspend.c t_suspend.h tm.c tm_load.c tm_load.h
Andrei Pelinescu-Onciul writes:
if this is the same as what happens when udp destination is not
responding, then it is fine.
-- juha
I wonder what
happen if the asynchronous sending fails (e.g. TCP
connection is lost). Will it jump to failure route (for requests)?
It will time out and then jump into the failure route (it doesn't jump
directly on async send failure, it will wait for the transaction
timeout).
9:38 a.m.
New subject: [SR-Dev] [Serdev] CVS:commitlog: sip_router/modules/tm t_cancel.c t_fwd.c t_reply.c t_reply.h t_suspend.c t_suspend.h tm.c tm_load.c tm_load.h
On Dec 12, 2008 at 08:28, Juha Heinanen <jh(a)tutpro.com> wrote:
Andrei Pelinescu-Onciul writes:
if this is the same as what happens when udp destination is not
responding, then it is fine.
Yes, it's the same.
Andrei
I wonder
what happen if the asynchronous sending fails (e.g. TCP
connection is lost). Will it jump to failure route (for requests)?
It will time out and then jump into the failure route (it doesn't jump
directly on async send failure, it will wait for the transaction
timeout).
5613
days inactive
5643
days old
7 comments
4 participants
participants (4)
-
Andrei Pelinescu-Onciul -
jh＠tutpro.com -
Klaus Darilion -
Miklos Tirpak