Experimenting with KEMI for the first time, I ran into a few issues and hoping someone has feedback.
kamailio 5.8.3
In the following python example, 'if ksr.is_method("ACK"):' should never be reached. Below is debug.
if not ksr.siputils.has_totag():
ksr.info("has to tag")
sys.exit()
if ksr.is_method("ACK"):
if ksr.tm.t_check_trans():
ksr.relay()
sys.exit)
else:
sys.exit()
ksr.sl.send_reply(404, "Not here")
sys,exit()
.....
DEBUG: app_python3s [apy3s_kemi.c:365]: sr_apy_kemi_exec_func_ex(): execution of method: siputils.has_totag
DEBUG: siputils [checks.c:122]: has_totag(): no totag
DEBUG: app_python3s [apy3s_kemi.c:368]: sr_apy_kemi_exec_func_ex(): execution of method: is_method
DEBUG: app_python3s [apy3s_kemi.c:389]: sr_apy_kemi_exec_func_ex(): number of arguments: 1
DEBUG: app_python3s [apy3s_kemi.c:365]: sr_apy_kemi_exec_func_ex(): execution of method: sl.send_reply
DEBUG: app_python3s [apy3s_kemi.c:389]: sr_apy_kemi_exec_func_ex(): number of arguments: 2
DEBUG: sl [sl.c:306]: send_reply(): reply in stateless mode (sl)
.....
Second, here are a couple errors I have seen a few times but havent been able to identify the underlying issue, mainly hoping for more context on the meaning of 'execution of route type 1 with no name returned -1'.
ERROR: app_python3s [apy3s_kemi.c:146]: apy3s_exec_func(): error exception occurred
DEBUG: app_python3s [apy3s_kemi.c:230]: sr_kemi_config_engine_python(): execution of route type 1 with no name returned -1
Any input is appreciated.
*
dan
Thank you Henning!
Here are the configured limits for Kamailio
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 31543 31543 processes
Max open files 16384 16384 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 31543 31543 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
This is what I know so far:
There seems to be a problem with siptrace module (going via TCP to a heplify-server on another host), but I’m not sure if this is a coincidence or the cause of the problems.
I see quite a bit of these before the actual problem arises:
Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322822]: ERROR: <core> [core/tcp_main.c:630]: _wbufq_add(): (591 bytes): write queue full or timeout (32498, total 43196, last write 0 s ago)
Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322822]: ERROR: siptrace [../../core/forward.h:261]: msg_send_buffer(): tcp_send failed
Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322822]: ERROR: siptrace [siptrace_hep.c:229]: trace_send_hep3_duplicate(): cannot send hep duplicate message
Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: ERROR: <core> [core/tcp_main.c:4023]: handle_ser_child(): received CON_ERROR for 0x7fe054f8ae10 (id 727321), refcnt 2, flags 0x3096
At some point then all things go south and the log is flooded with these messages
Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 2244, not in [0, 482)
Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: ERROR: <core> [core/tcp_main.c:4677]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fe054f8ae10, fd 2244
And then ultimately the above combined with the initially mentioned
Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: CRITICAL: <core> [core/tcp_main.c:4264]: handle_ser_child(): failed to add new socket to the fd list
And occasionally also something like this
Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: ERROR: <core> [core/io_wait.h:373]: io_watch_add(): trying to overwrite entry 2247 watched for 5 in the hash 0x5621dc404ee0 (fd:-601539712, type:22049, data:0x5621dc23c004) with (2247, 2, 0x7fe04f85c2a0)
In total these add up to roughly 3600 lines per second (!!) in the log, so it is quickly flooded with these.
Florian FLOIMAIR
Software Development - Symphony Cloud Services (1568)
Von: Henning Westerholt <hw(a)gilawa.com>
Datum: Montag, 11. November 2024 um 19:52
An: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Cc: Floimair Florian <f.floimair(a)commend.com>
Betreff: [External] RE: Question regarding error message "failed to add new socket to the fd list"
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hello Florian,
Any other error messages before that error happens, e.g. something about the memory.
Otherwise, it could be indeed a ulimit issue. First step would be to check the actual configured limits for the kamailio user.
Cheers,
Henning
From: Floimair Florian via sr-users <sr-users(a)lists.kamailio.org>
Sent: Montag, 11. November 2024 17:45
To: sr-users(a)lists.kamailio.org
Cc: Floimair Florian <f.floimair(a)commend.com>
Subject: [SR-Users] Question regarding error message "failed to add new socket to the fd list"
Hi!
We have recently had issues with one of our Production Kamailios.
When those happened, the log was filled with the following message:
CRITICAL: <core> [core/tcp_main.c:4528]: handle_new_connect(): failed to add new socket to the fd list
Now I wonder what the best approach is to prevent this.
We are using TCP/TLS only and I think this might be related to the file ulimit, but I am not sure about that.
Shared memory is set to 512MB
Can you give me a hint on what to look for?
Thank you very much!
P.S.: Sorry, I accidentally replied to a previous post of a different topic before which is totally unrelated (I think I should stop working for today 😉)
FLORIAN FLOIMAIR
Software Development - Symphony Cloud Services
Commend International GmbH
Saalachstrasse 51
5020 Salzburg, Austria
[signature_2072127332]
commend.com
LG Salzburg / FN 178618z
Hi Gang
I'm having another go with a dedicated kamailio running topos to replace
a B2BUA SBC.
I came across this situation:
Server Topology:
Kamaiolio SBC (topos) <=> Kamailio Registrar <=> Kamailio Router (DLG +
CDRS)
Core IP Address 10.0.0.1
Registrar IP 10.0.0.2
Two different CPE are registered on Registrar via SBC
One CPE is calling the other CPE
So the path the call is taking is:
CPE A => SBC => Registrar => Core => Registrar => SBC => CPE B
Invite A => B goes through, no issue.
180 Ringing, required 100rel B => A requires the route to be recorded
therefore contains 3 record route statements:
Record Route: <sip:10.0.0.2;lr>
Record Route: <sip:10.0.0.1;lr>
Record Route: <sip:10.0.0.2;lr>
To ensure answer will get route back to the registrar => core
=> registrar and then the contact pointing to the SBC with topos to
restore the last hop.
Ringing B => A works as expected and is answered by A with PRACK.
topos would now need set Route: in this PRACK message, according to the
received Record Route Header.
Unfortunately I fear Topos 'removes' the routes with duplicate host ip.
I only get:
Route: <sip:10.0.0.2>
Route: <sip:10.0.0.1>
Causing the Core not to be able to route the PRACK back to the
registrar.
To test, which line is being dropped, I added an mark to each one and
determined the 3rd Record Router is being dropped by topos, probably
considered a duplicate.
I found no setting of the 'do not perform record route header dupe
detection'.
Interestingly, when I look at the database field topos_d.b_rr all
three entries are present in the correct order.
Has anyone come over the same issue and found a solution?
Maybe related to:
https://github.com/kamailio/kamailio/issues/3778 ?
Mit freundlichen Grüssen
-Benoît Panizzon-
--
I m p r o W a r e A G - Leiter Commerce Kunden
______________________________________________________
Zurlindenstrasse 29 Tel +41 61 826 93 00
CH-4133 Pratteln Fax +41 61 826 93 01
Schweiz Web http://www.imp.ch
______________________________________________________
Hello,
Kamailio SIP Server project is organizing another meeting of its
developers and community members during November 19-20, 2024 (Tue-Wed),
hosted again by sipgate.de in Dusseldorf, Germany.
The event is intended to facilitate the interaction between Kamailio
developers and contributors in order to offer a convenient environment
for working together on several topics of high interest for the project,
including writing code for Kamailio and its tools, improving
documentation, or discuss about future development.
Everyone from the community is welcome to join, developer or user
interested in helping the project. Please note we have a limited
capacity of seats in the meeting room, the main policy for accepting
participants being first come first server. Also, very important to be
aware that this is not an event to learn how to use Kamailio.
More details about the event, the venue, how to register, are available at:
* https://www.kamailio.org/w/developers-meeting/
Looking forward to those two intensive hacking Kamailio days in Dusseldorf!
Cheers,
Daniel
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy, Training and Development Services -- asipto.com
Hello,
Kamailio SIP Server v5.8.4 stable release is out.
This is a maintenance release of the latest stable branch, 5.8, that
includes fixes since the release of v5.8.3. There is no change to
database schema or configuration language structure that you have to do
on previous installations of v5.8.x. Deployments running previous v5.8.x
versions are strongly recommended to be upgraded to v5.8.4.
For more details about version 5.8.4 (including links and guidelines to
download the tarball or from GIT repository), visit:
* https://www.kamailio.org/w/2024/11/kamailio-v5-8-4-released/
RPM, Debian/Ubuntu packages will be available soon as well.
Many thanks to all contributing and using Kamailio!
Cheers,
Daniel
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy, Training and Development Services -- asipto.com
Hello,
I am considering to release Kamailio v5.8.4 (out of branch 5.8) sometime
next week, most likely on Tuesday, Nov 12, 2024. If anyone is aware of
issues not yet on the bug tracker, report them there asap in order to
have a better chance to be fixed.
Cheers,
Daniel
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy, Training and Development Services -- asipto.com
When Kamailio message parser produces this kind of syslog message:
024-11-11T21:16:40.757364+02:00 box /usr/bin/sip-proxy[181398]: ERROR: <core> [core/parser/msg_parser.c:791]: parse_msg(): ERROR: parse_msg: message=<#026#003#001#006\340#001>
fail2ban complains:
Consider setting logencoding to appropriate encoding for this jail. Continuing to process line ignoring invalid characters: b'2024-11-11T21:06:15.347318+02:00 box /usr/bin/sip-proxy[181398]: ERROR: <core> [core/parser/msg_parser.c:791]: parse_msg(): ERROR: parse_msg: message=<#026#003#001#006\xc0#001>\n': 1 Time(s)
Error decoding line from '/var/log/syslog' with 'UTF-8'.: 1 Time(s)
Unable to retrieve Watch object associated to <_RawEvent cookie=0 mask=0x2 name='' wd=13 >: 3 Time(s)
Would it be possible to somehow ensure that Kamailio syslog messages
only contain valid UTF-8 characters?
-- Juha
Hi all
I noticed, in my config, that if the last line of the reply_route is a
function that does not succeed, then the SIP response is not relayed
and kamailio throws an error. I've searched documentation to indicate
why this might be, but I can't find anything, so this looks to me like
a bug.
I use 5.8.1 for testing.
Here's a sample config that shows what goes wrong.
####################
#!KAMAILIO
listen=udp:203.0.113.57:5060
debug=1
loadmodule "textops"
request_route{exit;}
reply_route{remove_hf_re("^X-Custom");}
####################
Here's a sample SIP message that works. When I send this to
udp:203.0.113.57:5060, kamailio relays it.
####################
SIP/2.0 200 Testing
Via: SIP/2.0/UDP 203.0.113.57:5060;branch=z9hG4bK-d8754z
Via: SIP/2.0/UDP 198.18.0.157:5060;branch=z9hG4bK-d8754z
T: <sip:c@s>;tag=676fbf45
F: <sip:c@s>;tag=676fbf45
X-Custom: lkjdfl
i: testing-XXX
CSeq: 1 INFO
l: 0
####################
If I then remove the X-Custom header field from the SIP reply and try
again, the message does not get relayed and I see this in the log.
5(1356) WARNING: <core> [core/receive.c:605]: receive_msg(): error
while trying onreply script
If I put a log command after the remove_hf_re(), then the message gets
relayed. It looks like the reply_route always fails if the last
function is not successful.
Is this normal or is it a bug?
James
Hi!
We have recently had issues with one of our Production Kamailios.
When those happened, the log was filled with the following message:
CRITICAL: <core> [core/tcp_main.c:4528]: handle_new_connect(): failed to add new socket to the fd list
Now I wonder what the best approach is to prevent this.
We are using TCP/TLS only and I think this might be related to the file ulimit, but I am not sure about that.
Shared memory is set to 512MB
Can you give me a hint on what to look for?
Thank you very much!
P.S.: Sorry, I accidentally replied to a previous post of a different topic before which is totally unrelated (I think I should stop working for today 😉)
FLORIAN FLOIMAIR
Software Development - Symphony Cloud Services
Commend International GmbH
Saalachstrasse 51
5020 Salzburg, Austria
[signature_2072127332]
commend.com
LG Salzburg / FN 178618z
When I look at the debian repositories maintained by the project, there are instructions to import the public key of the repository for apt using this command:
wget -O http://deb.kamailio.org/kamailiodebkey.gpg | sudo apt-key add -
which results in:
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
gpg: no valid OpenPGP data found.
Is there an alternate method that can be posted in the repository that will get better results?
Thank you,
TL