[SR-Users] Understanding the new interface (kemi) for embedded interpreters

[Daniel-Constantin Mierla]

Hello,

(cross-posting because the details are useful for both devels and users)

Yesterday during the IRC meeting about Kamailio, one group of topics was
about Kamailio 5.0, and the new feature of using embedded interpreters
for external programming languages was highly debated. So I am trying to
clarify some of its aspects.


The new framework is referenced as kemi, chosen by me as an abbreviation
for Kamaialio EMbedded Interface.

We already had embedded interpreters for many years, respectively:
  - Lua via app_lua module (probably most developed and most used)
  - Perl via app_perl module (iirc, the oldest embedded interpreter)
  - Python via app_python (I discovered during the last days that it has
quite a lot of features, just not documented -- never used it before)
  - Java via app_java (this one is a bit of a blackhole for me, never
looked at it, don't know what it offers/how it is supposed to be used)
  - .NET group of languages via app_mono (e.g., C#, but also some
variants of JavaScript, Python and maybe even Java)

To clarify, an embedded interpreter means that Kamailio is linking to
itself the interpreter of that language and kamailio becomes the
interpreter of the language. It is not launching the standard
interpreter for that language, therefore it is very fast at runtime.
Moreover, kamailio is extending the language with new modules, giving
access to Kamailio C functions. In other words, for example with Python,
Kamailio becomes equivalent to the "python" application plus two
extensions (modules) named Router (developed in the past inside
app_python) and KSR (developed for kemi).

At this moment, the kemi was implemented in app_lua and app_python modules.

To understand what is all about, I am going to give an example with a
function from maxfwd module. In default kamailio.cfg next statement is
present:

      if (!mf_process_maxfwd_header("10")) {
          sl_send_reply("483","Too Many Hops");
          exit;
      }

mf_process_maxfwd_header("10") is the function exported by maxfwd:

  -
https://www.kamailio.org/docs/modules/stable/modules/maxfwd.html#maxfwd.f.mf_process_maxfwd_header

The parameter can be as a static number (like above) or a number
provided via a variable, like next:

$var(p) = 10;
mf_process_maxfwd_header("$var(p)")

The corresponding C code for this config function is:

static int w_process_maxfwd_header(struct sip_msg* msg, char* str1,
char* str2)
{
    int mfval;
    if (get_int_fparam(&mfval, msg, (fparam_t*) str1) < 0) {
        LM_ERR("could not get param value\n");
        return -1;
    }
    return process_maxfwd_header(msg, mfval);
}

The function get_int_fparam() is practically returning the int value of
the parmeter, no matter it was a static value or a variable, then
process_maxfwd_header(...) is executed with the int parameter. The msg
is the SIP message structure present mostly everywhere in the C code
related to the classic config interpreter.

To be used inside the config file, the C function has to be exported via
a cmd_export_t structure added to "struct module_exports exports",
respectively:

static cmd_export_t cmds[]={
   ...
    {"mf_process_maxfwd_header", (cmd_function)w_process_maxfwd_header, 1,
        fixup_var_int_1, 0, REQUEST_ROUTE},
   ...

To get the same function in a kemi interpreter (Lua, Python at this
moment), following C code was added:

static sr_kemi_t sr_kemi_maxfwd_exports[] = {
    { str_init("maxfwd"), str_init("process_maxfwd"),
        SR_KEMIP_INT, process_maxfwd_header,
        { SR_KEMIP_INT, SR_KEMIP_NONE, SR_KEMIP_NONE,
            SR_KEMIP_NONE, SR_KEMIP_NONE, SR_KEMIP_NONE }
    },

    { {0, 0}, {0, 0}, 0, NULL, { 0, 0, 0, 0, 0, 0 } }
};

int mod_register(char *path, int *dlflags, void *p1, void *p2)
{
    sr_kemi_modules_add(sr_kemi_maxfwd_exports);
    return 0;
}

The structure sr_kemi_t sr_kemi_maxfwd_exports exports directly the
existing function process_maxfwd_header(), which is also executed by the
function from kamailio.cfg after resolving the parameter to an int
value. In the case of kemi, the Lua or Python will give directly an
integer value, so no need for resolving that with a fixup wrapper.

The mod_register() function is something already existing, providing a
way to execute code when a module was loaded (e.g., is used for many
years by modules such as tls, mysql, etc.). In this case it was added,
because maxfwd didn't need anything at load time before.

Practically, no C code extension was written to maxfwd module from point
of view of SIP routing. Just an existing function was exported.

Now, what's actually the role of this kemi framework ...

The app_lua already exported the kamailio.cfg function
mf_process_maxfwd_header() as sr.process_maxfwd() since Kamailio v3.1,
see its usage in the tutorial:

  - https://kb.asipto.com/kamailio:usage:k32-lua-routing#lua_script

But a Lua specific wrapper had to be written:

  -
https://github.com/kamailio/kamailio/blob/master/modules/app_lua/app_lua_exp.c#L1327

All the other embedded interpreters would have needed to get a wrapper
as well to export this function. That's obviously costing time and
maintenance resources.

So kemi is a common framework to export C function to any of the
embedded interpreters, meaning that once a function is exported from a
module, it becomes available to all embedded interpreters, no longer
being necessary to write a specif wrapper per interpreter. The app_*
module needs a bit of coding to use the kemi exported functions (already
done for Lua and Python), but then nothing needs to be done for new
exported functions by modules.

As a matter of fact, lots of functions can be just exported, as you
could see in the example with maxfwd. In some cases, maybe some
functions need to be split, so the fixup handling is done separately and
then calls a common C function.

So, what we had so far:

[native kamailio cfg script] <===> [fixup framework for variables] <===>
[C implementation of function]

The above stays in place, no change to it. What we got extra:

[embedded language script] <===> [kemi framework] <===> [C
implementation of function]

In addition, kemi allows to write the routing blocks all in embedded
language. Core parameters, loading modules and modules' parameters stay
like so far, but there is no need to write request_route{},
failure_route, etc.. blocks. Some functions with a specific names have
to be written in the embedded language script, see more at:

  - https://www.kamailio.org/wiki/devel/config-engines

In several days I expect to be possible to have routing blocks of
kamailio-basic.cfg written via kemi interpreters. That will allow
testing the performance impact.

Right now Lua and Python use different mechanism of exporting Kamailio C
functions, the one from Python should be faster, but relies on defining
some C wrappers inside the module. Anyhow Lua might be faster as
interpreter itself -- and at the end it can just get the same mechanism
as Python.

app_lua already supports reloading the routing script without restarting
kamailio, one of the features asked from time to time.

Besides that, of course, a big benefit is the access to a very large set
of extensions already available as Lua/Python libraries. Also important,
many peoples are already familiar with these languages and there is
plenty of good documentation about those languages.

The classic/native kamailio.cfg format (and the interpreter for it)
stays there -- same common C functions are shared with kemi, so any
addition in the future will be visible to everywhere. The native
interpreter will remain the option for extreme optimizations for those
that deal with enormous amount of SIP traffic (e.g., heavy load
balancers), but I expect that at least Lua will be in pair of
performances with a registrar deployment.

I hope I could shed some light on various aspects of the new framework
for embedded interpreters. Expect very soon news about ability to build
full routing logic in Lua/Python, hopefully many of you will join to
test the results and compare with native alternatives.

Cheers,
Daniel

-- 
Daniel-Constantin Mierla
http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, Berlin, May 18-20, 2016 - http://www.kamailioworld.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20160422/c2d54358/attachment.html>