On 07/06/2009 02:35 PM, Andrei Pelinescu-Onciul wrote:

On Jul 04, 2009 at 12:14, Juha Heinanen jh@tutpro.com wrote:

...

in situation where both k and s modules are in use, is there a need for two different xmlrpc clients or can s or k xmlrpc interface be used to issue all xmlrpc commands?

i have noticed that serctl has mi argument after which a k ctl command can be given via ser ctl transport. does the same apply also to ser xmlrpc transport?

Theoretically yes, but in practice there might be some change needed. For example the mi_rpc module seems to include the reason (e.g. 200 Ok) among the parameters and then it prints some extra info for the value, e.g.:

sercmd> mi pwd 200 OK

• WD:: /home/andrei/sr.git

this has to be worked out more.

The output is pre-formatted to be nicer displayed in sercmd, but definitely cannot be used for xmlrpc transport. In K, mi tree can have nodes with values and attributes. The attributes can have also values. To be able to distinguish between nodes and attributes, I added +/-. Also, some node names are missing, e.g., in 'which' command. A parameter to control pretty formating along with mi command or a new command like 'mi-pf' are options. Moreover, asynchronous mi command support is not implemented.

Personally I don't think is time to remove any of the modules, because we may get trapped in some limitations of the other one. We should follow the usual path, decide which is better, if can handle everything the other does, mark the other obsolete for one release and remove after another release cycle.

I am more cautious to the changes done to things that interfaces with the configuration file and mi/rpc transports, as these will break installations and custom tools done by users, making migration nightmare.

Cheers, Daniel

The same info will appear in xmlrpc and I assume that this is not what is expected (it should be only "/home/andrei/sr.git", at least that's the output of core.pwd). Another possible problem is the ARRAY type for xmlrpc (the xmlrpc specifications doesn't support multiple return values in the same reply, but it does support returning an array). So far in ser an rpc function returning an array must be marked explicitly (the RET_ARRAY flag). That would be a problem for mi commands that return multiple values (either the mi_rpc modules should know them somehow and add the RET_ARRAY flag automatically for them, or we should change the xmlrpc value to automatically detect if an array is needed). Note that this problem appears only for the xmlrpc transport, for the binrpc/ctl transport it doesn't matter since it supports multiple values in replies.

What we would need are some example mi_xmlrpc replies from kamailio and the same replies from sr xmlrpc via mi_rpc.

Andrei

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

On 07/07/2009 12:40 AM, Andrei Pelinescu-Onciul wrote:

On Jul 06, 2009 at 23:42, Daniel-Constantin Mierla miconda@gmail.com wrote: [...]

...

this has to be worked out more.

The output is pre-formatted to be nicer displayed in sercmd, but definitely cannot be used for xmlrpc transport. In K, mi tree can have nodes with values and attributes. The attributes can have also values. To be able to distinguish between nodes and attributes, I added +/-. Also, some node names are missing, e.g., in 'which' command. A parameter to control pretty formating along with mi command or a new command like 'mi-pf' are options. Moreover, asynchronous mi command support is not implemented.

What are attributes used for? I don't see them mapping to anything in terms of xmlrpc or standard rpc. Is anybody using the attributes?

there are some mi commands that set attributes in reply nodes, for example ul_dump -- I discovered while implementing the mi_rpc.

As I am not a xmlrpc heavy user, I do not know if and how they map.

In mi, a node can have value and a set of attributes. Just for example (does not mean it is so right now in the code):

<process pid="1234" childid="10"> UDP listener </process>

What's that asynchronous command support? How can one use it using xmlrpc (AFAIK xmlrpc does not support async. mode)? Is anybody using it?

This is used by some tm commands, for example: t_uac -- request is sent in one process and reply could come back in another one. There are some structures cloned in shm, filled on callback.

...

Personally I don't think is time to remove any of the modules, because we may get trapped in some limitations of the other one. We should follow the usual path, decide which is better, if can handle everything the other does, mark the other obsolete for one release and remove after another release cycle.

We shouldn't remove anything now, but after the next release I don't see any reason for keeping mi (other then maybe some legacy application that uses some obscure feature).

do you mean dropping MI the interface/library or the mi_xmlrpc? MI might take longer, but mi_xmlrpc can be done in the next release.

I agree rpc in ser is easier to handle from developer point of view, the concern I have in regard to xmlrpc(s) relates to the fact that is a limited implementation, for what ser needed. Seems to be more that what K needs in mi_xmlrpc, according to your review.

On the other hand, mi_xmlrpc uses a library from distro, which as aside being developed for this purpose and probably follows more the specs, proved to be buggy sometime.

So, I have no personal preference, mi_xmlrpc can be migrated anytime to use the xmlrpc code from ser as long as it maintain the request/reply format that has it in k 1.5.x, for the sake of migration easiness.

Cheers, Daniel

Correct me if I'm wrong (I did only a quick code read):

mi_xmlrpc:

• supports only strings as parameters

• responses are either one big string (reply_option=0) or one array of string values (reply_option=1).

  For the first option (one big string) it kinds of defeats the purpose of XMLRPC. XMLRPC is an incredibly bloated and slow way of doing remote procedure calls, but at least is easy to use. You get the answer pre-parsed. However mi_rpc in one big string mode would require additional parsing of the big string to get individual values for any mi command returning a list (e.g. ps).

  For the second mode it always returns an array of string values, but this values look like: name:: value [attr_name=attr_value ...]. So again you'll have to do additional string parsing.

• cannot return integers

• cannot returns structures

xmlrpc(s) doesn't have any of the above limitations. For example take core.tcp_info, an rpc that returns a structure with informations related to tcp:

sercmd> core.tcp_info { readers: 8 max_connections: 2048 opened_connections: 0 write_queued_bytes: 0 }

Using xmlrpc from perl I can access the opened_connections with the following piece of perl code:

use XMLRPC::Lite; my $res=XMLRPC::Lite ->proxy("http://127.0.0.1:5060%22)-%3Ecall(%22core.tcp_info") ->result; my %tcp_info=%{$res}; print "opened connections: $tcp_info{opened_connections}\n";

No extra parsing needed (try doing the same with a mi command).

In general to me MI looks too complex. It seems is more complex and more annoying to write then ser RPCs and does less... RPC are easier to write, map perfectly on xmlrpc, support more types then just plain strings and have integrated docs (try sercmd help <rpc_command> or sending system.methodHelp <rpc_command> over xmlrpc).

Example: The core.version rpc: (1 line of effective code) static void core_version(rpc_t* rpc, void* c) { rpc->add(c, "s", SERVER_HDR " " REPO_VER ); }

versus mi_version (modules_k/kex line 121): (28 lines of code excluding whitelines and defines, could be reduced to 20) static struct mi_root *mi_version(struct mi_root *cmd, void *param) { struct mi_root *rpl_tree; struct mi_node *rpl; struct mi_node *node;

rpl_tree = init_mi_tree( 200, MI_SSTR(MI_OK));
if (rpl_tree==0)
    return 0;
rpl = &rpl_tree->node;

node = add_mi_node_child( rpl, 0, MI_SSTR("Server"), SERVER_HDR+8,
    SERVER_HDR_LEN-8);
if (node==0) 
    goto error;

node = add_mi_node_child( rpl, 0, MI_SSTR("Build"), BUILD_STR,
        BUILD_STR_LEN);
if (node==0) 
    goto error;

#ifndef KAMAILIO_COMPILE_FLAGS #define KAMAILIO_COMPILE_FLAGS SER_COMPILE_FLAGS #endif

node = add_mi_node_child( rpl, 0, MI_SSTR("Flags"), KAMAILIO_COMPILE_FLAGS,
        sizeof(KAMAILIO_COMPILE_FLAGS)-1);
if (node==0) 
    goto error; 

node = add_mi_node_child( rpl, 0, MI_SSTR("SVN"), SVNREVISION,
        sizeof(SVNREVISION)-1);
if (node==0) 
    goto error;



return rpl_tree;

error: LM_ERR("failed to add node\n"); free_mi_tree(rpl_tree); return 0;

}

...

I am more cautious to the changes done to things that interfaces with the configuration file and mi/rpc transports, as these will break installations and custom tools done by users, making migration nightmare.

I guess we have to find out how many custom tools there are out there that also process the responses of the mi commands (more then checking if the command succeeded or not).

Andrei

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

On 07/07/2009 01:24 PM, Andrei Pelinescu-Onciul wrote:

[...]

...

there are some mi commands that set attributes in reply nodes, for example ul_dump -- I discovered while implementing the mi_rpc.

As I am not a xmlrpc heavy user, I do not know if and how they map.

They cannot map since xmlrpc has no support for attributes. I'm not sure what was the design goal of mi. If it was to enable full xmlrpc use it has failed (it uses xmlrpc in an extremely limited way and has some extra features that are not part of xmlrpc or any rpc I know).

it started to map on xml-like format. Practically is a lightweight representation of a xml document.

...

In mi, a node can have value and a set of attributes. Just for example (does not mean it is so right now in the code):

<process pid="1234" childid="10"> UDP listener </process>

There are 2 ways of doing this using standard rpcs: either just list all the values ( 1234,10,"udp listener" ....) and rely on having a fixed number of records per process (that's what core.ps does), or use an array of structures, eg.: { pid = 1234 childid = 10 type = "UDP listener" }

The advantage would be that this is a standard approach directly supported by xmlrpc (and other rpcs), so you don't need to deal with proprietary encodings (e.g. mi encoding over xmlrpc) and even the client code is much simpler.

...

...

What's that asynchronous command support? How can one use it using xmlrpc (AFAIK xmlrpc does not support async. mode)? Is anybody using it?

This is used by some tm commands, for example: t_uac -- request is sent in one process and reply could come back in another one. There are some structures cloned in shm, filled on callback.

Does it mean that the xmlrpc reply is not sent immediately, but at a later point?

Yes, not sure for k xmlrpc since I haven't used it, but with fifo/udp/unixsock you can do it.

Is it something used by seas?

No, seas has its own protocol to communicate with the sip proxy.

...

...

...

Personally I don't think is time to remove any of the modules, because we may get trapped in some limitations of the other one. We should follow the usual path, decide which is better, if can handle everything the other does, mark the other obsolete for one release and remove after another release cycle.

We shouldn't remove anything now, but after the next release I don't see any reason for keeping mi (other then maybe some legacy application that uses some obscure feature).

do you mean dropping MI the interface/library or the mi_xmlrpc? MI might take longer, but mi_xmlrpc can be done in the next release.

I mean making mi obsolete for any new stuff. The existing mi functions should/can be replaced but that is not a priority.

If people using mi_xmlrpc can tell us what they expect (some packet dump will be nice), we can modify mi_rpc to produce exactly the expected format and so we could make mi_xmlrpc obsolete immediately.

ok making the module obsolete if we get same interfacing with mi_rpc, but let's keep it at least one release.

BTW: is it ok with you if I add more functions to mi_rpc?

Feel free to add as much as you want and even change current reply format for mi command. It is formatted for printing to terminal, but not convenient for working directly with other applications. It is better to have another command for pretty formating.

I want to have a version more compatible with ser RPCs (no attributes, only the value). Unfortunately it will still be limited to string only and no structs (since mi doesn't support them, it would be very hard to automatically guess the best type).

Actually you can have structs - nodes whose value are a list of nodes. If you look at the way is printed now, there is some indentation for these cases.

...

I agree rpc in ser is easier to handle from developer point of view, the concern I have in regard to xmlrpc(s) relates to the fact that is a limited implementation, for what ser needed. Seems to be more that what K needs in mi_xmlrpc, according to your review.

Looking at the complex tree structures from mi, I thought mi_xmlrpc would do some very complex stuff. It turns out that is just a way of passing strings over xmlrpc (kind of the old fifo packed in xml, no xmlrpc feature is used).

...

On the other hand, mi_xmlrpc uses a library from distro, which as aside being developed for this purpose and probably follows more the specs, proved to be buggy sometime.

xmlrpc(s) uses also a library for doing the parsing.

parsing the xml content, via libxml2, but not a dedicated smlrpc library. Or I am missing something after a quick look and, iirc, it was mentioned in the docs somewhere.

The difference is, it does not use a separate library for HTTP, it just reuses ser SIP-TCP transport for HTTP (that's why you can call a sr route for each XMLRPC request and do extra stuff like authentication, certificate verification a.s.o.).

This I see as a nice feature, since we can do IP auth and other checks that were not possible with k mi_xmlrpc library.

Cheers, Daniel

...

So, I have no personal preference, mi_xmlrpc can be migrated anytime to use the xmlrpc code from ser as long as it maintain the request/reply format that has it in k 1.5.x, for the sake of migration easiness.

Andrei