when i try my test program with k mi_xmlrpc module, it works fine and ngrep shows:

######## T 2009/07/06 18:12:34.028112 192.98.101.10:41014 -> 192.98.101.10:6060 [AP] POST /RPC2 HTTP/1.0. Host: 192.98.101.10:6060. User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com). Content-Type: text/xml. Content-Length: 104. .

## T 2009/07/06 18:12:34.028289 192.98.101.10:41014 -> 192.98.101.10:6060 [AP] <?xml version='1.0'?> <methodCall> <methodName>lcr_reload</methodName> <params> </params> </methodCall>

## T 2009/07/06 18:12:34.030189 192.98.101.10:6060 -> 192.98.101.10:41014 [AP] HTTP/1.1 200 OK.

# T 2009/07/06 18:12:34.030289 192.98.101.10:6060 -> 192.98.101.10:41014 [AP] Content-type: text/xml; charset="utf-8".

# T 2009/07/06 18:12:34.032315 192.98.101.10:6060 -> 192.98.101.10:41014 [AFP] Content-length: 147. Connection: close. Date: Mon, 06 Jul 2009 16:12:34 GMT. Server: XMLRPC_ABYSS/1.06. . <?xml version="1.0" encoding="UTF-8"?>. <methodResponse>. <params>. <param><value><string></string></value></param>. </params>. </methodResponse>.

####

when i try domain.reload with xmlrpc module, it hangs and ngrep reports:

T 2009/07/06 18:18:25.496760 192.98.101.10:37656 -> 192.98.101.10:6060 [AP] POST /RPC2 HTTP/1.0. Host: 192.98.101.10:6060. User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com). Content-Type: text/xml. Content-Length: 107. .

## T 2009/07/06 18:18:25.496953 192.98.101.10:37656 -> 192.98.101.10:6060 [AP] <?xml version='1.0'?> <methodCall> <methodName>domain.reload</methodName> <params> </params> </methodCall>

## T 2009/07/06 18:18:25.498713 192.98.101.10:6060 -> 192.98.101.10:37656 [AP] HTTP/1.0 200 OK. Via: SIP/2.0/TCP 192.98.101.10:37656. Server: OpenXg SIP Router (2.99.0-dev01-serk (i386/linux)). Content-Length: 108. . <?xml version="1.0"?> <methodResponse> <params> <param> <value></value> </param> </params> </methodResponse> #

one difference that could explain the hangup is that xmlrpc module does not close the connection like mi_xmlrpc server does.

-- juha

Andrei Pelinescu-Onciul writes:

We could also close the connection in sr, but that would require additional code and it's the xmlrpc lib that's broken (parsing the answer only on close() is really braindamaged). However if there are lots of applications relying on the broken behaviour, I might be convinced to add the close option in sr xmlrpc.

well, python and php both are very popular for writing management software and a basic xmlrpc client written in either does not work with sr xmlrpc module. they do work with k mi_xmlrpc module which makes migration from mi_xmlrpc to xmlrpc module painful.

i would suggest to do whatever is necessary make this simple call work:

server_path = XMLRPC_PROTOCOL + '://' + XMLRPC_SERVER + \ ':' + str(XMLRPC_PORT) + XMLRPC_DIR c = xmlrpclib.ServerProxy(server_path)

if you look at any examples on the web on python xmlrpc clients, the above is what you will find. if sr requires something more complicated, it is not a good idea.

-- juha

Andrei Pelinescu-Onciul writes:

So whoever is interested in xmlrpc close connection after reply mode, should vote for it (but please vote only if you really have "legacy" hard to change applications that would be affected by it).

i would be interested, but perhaps i'm the only one who as xmlrpc client apps written in python and php.

it is simpler for me to add mi xmlrpc support to domain module that try to figure out what is wrong with python and php libs. i can then keep on using mi_xmlrpc module an forget xmlrpc module.

user should need only one of those two modules, not both.

-- juha

On Jul 06, 2009 at 23:41, Juha Heinanen jh@tutpro.com wrote:

i may be wrong, but it is hard for me to believe that python xmlrpclib would be badly broken, because it is very widely used.

The parser is not broken, it works. The problem is how it uses the transport by default. It looks like it waits for the remote close before it starts parsing the reply, which is wrong.

If one changes the transport (like in the example I've sent or in ser_ctl), then it works perfectly. One has only to create a new Transport class and then import it in all his code and instead of:

c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" + str(XMLRPC_PORT))

use

transport=Transport() c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" + str(XMLRPC_PORT), transport)

(this will removes the wait for the remote close() and it will work also with mi_xmlrpc)

Anyway IMHO for more complex applications the default transport should be changed anyway if there's any need for some sort of security (and for that a look at ser_ctl will really pay off: you can authenticate the HTML requests using the transport class defined there and all you need to do on ser side is to add a www_authorize/challenge in the xmlrpc route).

if i look more closely at body of sr response:

[...]

i notice that each body line is terminated only by one char (cr or lf) and the last line is not terminated by any char. content-length 108 seems to verify this.

i don't know if it is ok that lines end with only one char and that the last line does not end with anything.

The separator doesn't matter as long as it's some kind of whitespace. We can even get rid of all of them (everything on one line) and it should still work.

for comparison, mi xmlrpc module terminates each body line (including the last) with both cr and lf:

Try the latest sr version and add at the end of the xmlrpc route a return -1 or drop -1 (e.g.: route[XMLRPC]{dispatch_rpc(); drop -1; }) and it should close the connection after sending a reply. (I've come to the conclusion that is faster to implement it then answering to emails :-)). You could even do the return -1 only if the user agent is xmlrpclib.py (so that non-broken clients would still be able to keep 1 open connection and send xmlrpc requests on it).

Andrei

On 06-07 22:46, Raúl Alexis Betancor Santana wrote:

On Monday 06 July 2009 22:13:27 Andrei Pelinescu-Onciul wrote:

...

On Jul 06, 2009 at 23:41, Juha Heinanen jh@tutpro.com wrote:

...

i may be wrong, but it is hard for me to believe that python xmlrpclib would be badly broken, because it is very widely used.

The parser is not broken, it works. The problem is how it uses the transport by default. It looks like it waits for the remote close before it starts parsing the reply, which is wrong.

If one changes the transport (like in the example I've sent or in ser_ctl), then it works perfectly. One has only to create a new Transport class and then import it in all his code and instead of:

c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" + str(XMLRPC_PORT))

use

transport=Transport() c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" + str(XMLRPC_PORT), transport)

(this will removes the wait for the remote close() and it will work also with mi_xmlrpc)

Umm, re-checking your example and the responses that Juha pasted ... you are wrong, I explain myself ...

xmlrpc responses are sended as HTTP/1.0 transport, SO AS PER HTTP/1.0 RFC, only ONE request/response per conection are allowed so xmlrpclib is waiting for the remote end to close the conection because it see HTTP/1.0 on the headers.

I tend to disagree. The RFC you refer to says that if the response carries Content-Length header field then the value of the header field determines the end of the reply. If there is no Content-Length header field in the reply then the reply is complete when the server closes the connection. Even for HTTP/1.0 requests and responses.

This really is a bug in the xmlrpc client library, because it ignores the Content-Length header field. Unfortunately it is not triggered by most other servers because they close the connection immediately.

Could someone try to check what happens if transport is HTTP/1.1 instead of HTTP/1.0 ?

Nothing, in fact the xmlrpc module returns in reply whatever the client sent in the request.

Jan.

On Jul 06, 2009 at 22:37, Ra?l Alexis Betancor Santana rabs@dimension-virtual.com wrote:

On Monday 06 July 2009 21:41:40 Juha Heinanen wrote:

...

i may be wrong, but it is hard for me to believe that python xmlrpclib would be badly broken, because it is very widely used.

I also doub that xmlrpclib it's the problem

Being standard HTTP, the xmlrpc client should close the connection after receiving the answer (if it doesn't want a connection persistent mode) and not the reverse.

...

i don't know if it is ok that lines end with only one char and that the last line does not end with anything.

It's not ok, as per HTTP/1.1 RFC, headers lines must end with <CR><LF> then an empy <CR><LF> line must be used as boundary between headers and body and finaly HTTP request or response must end with <CR><LF> on its own line. So taking into account that xmlrpc it's XML over HTTP, we must follow the RFC on that.

Juha meant only the xml body. All the HTTP header lines are CRLF terminated.

...

for comparison, mi xmlrpc module terminates each body line (including the last) with both cr and lf:

I have not read HTTP/1.1 RFC line by line .. so I could not tell that it's true for all body lines, but we must follow the specs about header/body boundary and all other specs.

It's true only for the headers. How the body looks depends on what does it carry. For the case of xmlrpc it's xml which is quite free-form (you can have the whole xml part on one line if you want to).

Andrei

On Jul 06, 2009 at 23:16, Ra?l Alexis Betancor Santana rabs@dimension-virtual.com wrote:

On Monday 06 July 2009 22:50:38 Andrei Pelinescu-Onciul wrote:

...

On Jul 06, 2009 at 22:37, Ra?l Alexis Betancor Santana

rabs@dimension-virtual.com wrote:

...

...

On Monday 06 July 2009 21:41:40 Juha Heinanen wrote:

...

i may be wrong, but it is hard for me to believe that python xmlrpclib would be badly broken, because it is very widely used.

I also doub that xmlrpclib it's the problem

Being standard HTTP, the xmlrpc client should close the connection after receiving the answer (if it doesn't want a connection persistent mode) and not the reverse.

But server is sending a response not ended by CR+LF, so client doesn't know that it must close the connection and also as serving HTTP/1.0 resquest server MUST close the connection when it send its response.

AFAIK the request does not have to end in anything specific. The headers end with a double CRLF and then there are Content-Lenght bytes of the body.

What I see is that server is sending out HTTP/1.0 responses so you could NEVER have a client that could wait for persistent connections because if it does, it will be breaking HTTP/1.0 RFC

The server server sends exactly the same HTTP version in the reponse that it has received in the request (if the request is HTTP/1.1 the reply is HTTP/1.1).

...

Juha meant only the xml body. All the HTTP header lines are CRLF terminated.

And boundary between headers and body ?

It's ok (or else the body would not be at all parse-able and no xmlrpc client would work).

...

It's true only for the headers. How the body looks depends on what does it carry. For the case of xmlrpc it's xml which is quite free-form (you can have the whole xml part on one line if you want to).

Yes, but I was not telling that body must follow cr+lf spec .. I told that response should be:

HTTP/1.0 header: value<cr><lf> <cr><lf> {xml response(no matter if one line or more)}<cr><lf> <cr><lf>

It should be easy to check, I don't use xmlrpc on s-r so I could not check it

It is. The problem is the python xmlrpc client waits for the server to close the connection. It might be related to the fact that it uses http/1.0 by default (but I don't know how can that be easily changed). However the perl client works, the python client works with a transport class and the latest sr version has support for closing the connection after sending a reply if one really wants to and prefer this rather then updating his python/php program.

Andrei

On 06-07 23:41, Juha Heinanen wrote:

i may be wrong, but it is hard for me to believe that python xmlrpclib would be badly broken, because it is very widely used.

I do not recall all the details, but I remember that I encountered the same problem long time ago. I even reported it to the maintainer of the xmlrpc library in python. He confirmed that this was a "shortcomming" in the library, but refused to fix it because "all other xmlrpc servers close the connection".

In ser_ctl (the tools written in python) we patched the library, as far as I can remember.

Jan.

if i look more closely at body of sr response:

## T 2009/07/06 18:18:25.498713 192.98.101.10:6060 -> 192.98.101.10:37656 [AP] HTTP/1.0 200 OK. Via: SIP/2.0/TCP 192.98.101.10:37656. Server: OpenXg SIP Router (2.99.0-dev01-serk (i386/linux)). Content-Length: 108. .

<?xml version="1.0"?>

<methodResponse> <params> <param> <value></value> </param> </params> </methodResponse> #

i notice that each body line is terminated only by one char (cr or lf) and the last line is not terminated by any char. content-length 108 seems to verify this.

i don't know if it is ok that lines end with only one char and that the last line does not end with anything.

for comparison, mi xmlrpc module terminates each body line (including the last) with both cr and lf:

# T 2009/07/06 18:12:34.032315 192.98.101.10:6060 -> 192.98.101.10:41014 [AFP] Content-length: 147. Connection: close. Date: Mon, 06 Jul 2009 16:12:34 GMT. Server: XMLRPC_ABYSS/1.06. .

<?xml version="1.0" encoding="UTF-8"?>.

<methodResponse>. <params>.

<param><value><string></string></value></param>. </params>. </methodResponse>.

####

-- juha

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev