[sr-dev] Understanding the new interface (kemi) for embedded interpreters

Hello, (cross-posting because the details are useful for both devels and users) Yesterday during the IRC meeting about Kamailio, one group of topics was about Kamailio 5.0, and the new feature of using embedded interpreters for external programming languages was highly debated. So I am trying to clarify some of its aspects. The new framework is referenced as kemi, chosen by me as an abbreviation for Kamaialio EMbedded Interface. We already had embedded interpreters for many years, respectively: - Lua via app_lua module (probably most developed and most used) - Perl via app_perl module (iirc, the oldest embedded interpreter) - Python via app_python (I discovered during the last days that it has quite a lot of features, just not documented -- never used it before) - Java via app_java (this one is a bit of a blackhole for me, never looked at it, don't know what it offers/how it is supposed to be used) - .NET group of languages via app_mono (e.g., C#, but also some variants of JavaScript, Python and maybe even Java) To clarify, an embedded interpreter means that Kamailio is linking to itself the interpreter of that language and kamailio becomes the interpreter of the language. It is not launching the standard interpreter for that language, therefore it is very fast at runtime. Moreover, kamailio is extending the language with new modules, giving access to Kamailio C functions. In other words, for example with Python, Kamailio becomes equivalent to the "python" application plus two extensions (modules) named Router (developed in the past inside app_python) and KSR (developed for kemi). At this moment, the kemi was implemented in app_lua and app_python modules. To understand what is all about, I am going to give an example with a function from maxfwd module. In default kamailio.cfg next statement is present: if (!mf_process_maxfwd_header("10")) { sl_send_reply("483","Too Many Hops"); exit; } mf_process_maxfwd_header("10") is the function exported by maxfwd: - https://www.kamailio.org/docs/modules/stable/modules/maxfwd.html#maxfwd.f.m… The parameter can be as a static number (like above) or a number provided via a variable, like next: $var(p) = 10; mf_process_maxfwd_header("$var(p)") The corresponding C code for this config function is: static int w_process_maxfwd_header(struct sip_msg* msg, char* str1, char* str2) { int mfval; if (get_int_fparam(&mfval, msg, (fparam_t*) str1) < 0) { LM_ERR("could not get param value\n"); return -1; } return process_maxfwd_header(msg, mfval); } The function get_int_fparam() is practically returning the int value of the parmeter, no matter it was a static value or a variable, then process_maxfwd_header(...) is executed with the int parameter. The msg is the SIP message structure present mostly everywhere in the C code related to the classic config interpreter. To be used inside the config file, the C function has to be exported via a cmd_export_t structure added to "struct module_exports exports", respectively: static cmd_export_t cmds[]={ ... {"mf_process_maxfwd_header", (cmd_function)w_process_maxfwd_header, 1, fixup_var_int_1, 0, REQUEST_ROUTE}, ... To get the same function in a kemi interpreter (Lua, Python at this moment), following C code was added: static sr_kemi_t sr_kemi_maxfwd_exports[] = { { str_init("maxfwd"), str_init("process_maxfwd"), SR_KEMIP_INT, process_maxfwd_header, { SR_KEMIP_INT, SR_KEMIP_NONE, SR_KEMIP_NONE, SR_KEMIP_NONE, SR_KEMIP_NONE, SR_KEMIP_NONE } }, { {0, 0}, {0, 0}, 0, NULL, { 0, 0, 0, 0, 0, 0 } } }; int mod_register(char *path, int *dlflags, void *p1, void *p2) { sr_kemi_modules_add(sr_kemi_maxfwd_exports); return 0; } The structure sr_kemi_t sr_kemi_maxfwd_exports exports directly the existing function process_maxfwd_header(), which is also executed by the function from kamailio.cfg after resolving the parameter to an int value. In the case of kemi, the Lua or Python will give directly an integer value, so no need for resolving that with a fixup wrapper. The mod_register() function is something already existing, providing a way to execute code when a module was loaded (e.g., is used for many years by modules such as tls, mysql, etc.). In this case it was added, because maxfwd didn't need anything at load time before. Practically, no C code extension was written to maxfwd module from point of view of SIP routing. Just an existing function was exported. Now, what's actually the role of this kemi framework ... The app_lua already exported the kamailio.cfg function mf_process_maxfwd_header() as sr.process_maxfwd() since Kamailio v3.1, see its usage in the tutorial: - https://kb.asipto.com/kamailio:usage:k32-lua-routing#lua_script But a Lua specific wrapper had to be written: - https://github.com/kamailio/kamailio/blob/master/modules/app_lua/app_lua_ex… All the other embedded interpreters would have needed to get a wrapper as well to export this function. That's obviously costing time and maintenance resources. So kemi is a common framework to export C function to any of the embedded interpreters, meaning that once a function is exported from a module, it becomes available to all embedded interpreters, no longer being necessary to write a specif wrapper per interpreter. The app_* module needs a bit of coding to use the kemi exported functions (already done for Lua and Python), but then nothing needs to be done for new exported functions by modules. As a matter of fact, lots of functions can be just exported, as you could see in the example with maxfwd. In some cases, maybe some functions need to be split, so the fixup handling is done separately and then calls a common C function. So, what we had so far: [native kamailio cfg script] <===> [fixup framework for variables] <===> [C implementation of function] The above stays in place, no change to it. What we got extra: [embedded language script] <===> [kemi framework] <===> [C implementation of function] In addition, kemi allows to write the routing blocks all in embedded language. Core parameters, loading modules and modules' parameters stay like so far, but there is no need to write request_route{}, failure_route, etc.. blocks. Some functions with a specific names have to be written in the embedded language script, see more at: - https://www.kamailio.org/wiki/devel/config-engines In several days I expect to be possible to have routing blocks of kamailio-basic.cfg written via kemi interpreters. That will allow testing the performance impact. Right now Lua and Python use different mechanism of exporting Kamailio C functions, the one from Python should be faster, but relies on defining some C wrappers inside the module. Anyhow Lua might be faster as interpreter itself -- and at the end it can just get the same mechanism as Python. app_lua already supports reloading the routing script without restarting kamailio, one of the features asked from time to time. Besides that, of course, a big benefit is the access to a very large set of extensions already available as Lua/Python libraries. Also important, many peoples are already familiar with these languages and there is plenty of good documentation about those languages. The classic/native kamailio.cfg format (and the interpreter for it) stays there -- same common C functions are shared with kemi, so any addition in the future will be visible to everywhere. The native interpreter will remain the option for extreme optimizations for those that deal with enormous amount of SIP traffic (e.g., heavy load balancers), but I expect that at least Lua will be in pair of performances with a registrar deployment. I hope I could shed some light on various aspects of the new framework for embedded interpreters. Expect very soon news about ability to build full routing logic in Lua/Python, hopefully many of you will join to test the results and compare with native alternatives. Cheers, Daniel -- Daniel-Constantin Mierla http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, Berlin, May 18-20, 2016 - http://www.kamailioworld.com