Hello,
We're running three instances of Kamailo v5.14 as registrars handling
registrations from ~2000 SIP clients, with one instance being primary and
the other two as backups.
The three of them are using the dmq and dmq_usrloc modules to synchronize
user locations, however after a couple of days of operation the two
failover instances show memory leak behaviors, with mem usage assigned to
the core taking all available resources.
When this happens we've noticed that:
- The shared memory used by the function "sip_msg_shm_clone" spikes (from
1kb to 1.5GB).
- The shared memory used by the function "dmq:worker.c:job_queue_push"
also increases, but not as much (from 1kb to 1MB)
- DMQ request are not being answered (with a 200 OK) by the affected
instance during this memory leak, which make us think that DMQ module
becomes unresponsive.
A few more notes:
- The failover instances are doing nothing except receiving replicated
contacts.
- The shared memory grows at the same rate on both instances, but the
critical behavior never happens at the same time.
- We are allocating 1GB memory on startup to each instance.
- We store the location DB in a psql DB and we load it at startup.
- We didn't find any errors in syslog, even at debug level.
Has anyone experienced a similar issue who can suggest a possible solution?
Thanks,
Rogelio Perez
Telnyx
Dear list,
on my kamailio 4.4.7 i use interactions with MYSQL.
For example .. i use it for some black list control and other like this..
sql_query("cb", "select id from blacklist where cli='$var(g2)'
and (ippeer = '$si' or ippeer = 'ALL')", "rabl");
if($dbr(rabl=>rows)>0){
sql_result_free("rabl");
sl_send_reply("403","Forbidden blacklist");
exit;
}
sql_result_free("rabl");
I was wonder what was the best method to prevent the query going on if
mysql don't respond for high load or other situation..
If Mysql is not able to reply to the query tight now the call continue
over the configuration... what is the best way to prevent this and for
examples drop it with a standard SIP 503 error.
I have search on docs.. but i haven't found any valid way.
Thanks
Laura
Hi,
I run Kamailio 5.0.1 with MariaDB database.
There is no issue at the initialization, meaning the db is opened by
kamailio and the hash tables loaded from the db tables.
Example:
# kamcmd htable.dump propertiesHash
{
entry: 1
size: 1
slot: {
item: {
name: cfg.env
But the htable.reload command fails:
error: 500 - Failed to open htable db connection
The Kamailio log reports that it can't connect the db :
DEBUG: ctl [../../core/io_wait.h:380]: io_watch_add(): DBG:
io_watch_add(0x7f68c74ab300, 10, 3, 0xdc9410), fd_no=1
DEBUG: ctl [io_listener.c:442]: handle_new_connect(): handle_stream read:
new connection (1) on /var/run/kamailio//kamailio_ctl
DEBUG: <core> [core/sr_module.c:694]: find_mod_export_record():
find_export_record: found <db_bind_api> in module db_mysql
[/usr/lib64/kamailio/modules/db_mysql.so]
DEBUG: <core> [db.c:209]: db_bind_mod(): using db bind api for db_mysql
DEBUG: <core> [db.c:314]: db_do_init2(): connection 0x7f68cc354c10 not found
in pool
DEBUG: db_mysql [km_my_con.c:98]: db_mysql_new_connection(): opening
connection: mysql://xxxx:xxxx@/siprouter
ERROR: db_mysql [km_my_con.c:124]: db_mysql_new_connection(): driver error:
Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (2 "No such file or directory")
ERROR: <core> [db.c:318]: db_do_init2(): could not add connection to the
pool
ERROR: htable [ht_db.c:92]: ht_db_open_con(): failed to connect to the
database
My local db is located at /myqdata/mysiprdb/data where is the mysiprdb.sock
file. But the log shows that Kamailio is looking for
/var/lib/mysql/mysql.sock.
Does this error make sense to you ?
Is it possible to fix this problem by configuration in Kamailio ?
My kamailio.cfg script contains :
# ----- htable
modparam("htable", "db_url", "mysql://[kamailio]/siprouter")
modparam("htable", "fetch_rows", 1000)
modparam("htable", "htable", "mbxrangeHash=>size=4;dbtable=mbxrange;")
modparam("htable", "htable", "abpqHash=>size=12;dbtable=abpq;")
modparam("htable", "htable", "zabpqHash=>size=12;dbtable=zabpq;")
modparam("htable", "htable", "propertiesHash=>size=4;dbtable=properties;")
In the /myqdata/mysiprdb/data/my.cnf, I added the following entries for
Kamailio:
[kamailio]
socket = /myqdata/mysiprdb/data/mysiprdb.sock
user = siprouter
password = ##########
default-character-set = utf8
Thanks all for help.
Cordialement.
Patrick GINHOUX
One my server with perl external script time to time is stopping processing
SIP messages. Perl script make calls to kamailio database on mysql.
I have found this in logs
DEBUG: <core> [core/io_wait.h:380]: io_watch_add(): DBG:
io_watch_add(0x56427eb40c40, 495, 2, 0x7f1a9f9ba408), fd_no=33
DEBUG: <core> [core/io_wait.h:602]: io_watch_del(): DBG: io_watch_del
(0x56427eb40c40, 495, -1, 0x0) fd_no=34 called
DEBUG: <core> [core/tcp_main.c:4196]: handle_tcpconn_ev(): sending to
child, events 1
DEBUG: <core> [core/tcp_main.c:3871]: send2child(): WARNING: no free tcp
receiver, connection passed to the least busy one (56)
DEBUG: <core> [core/tcp_main.c:3875]: send2child(): selected tcp worker 4
20(24) for activity on [tcp:0.0.0.0:81], 0x7f1a9f9ba408
And no next messages.
How get info what is load on "tcp receiver". This host in dev mode and
handles not more 5 devices.
Sergey
Hey all,
I was wondering if the kamctl stats output could be updated to provide a bit more parseable JSON? For instance:
# kamctl stats shmem | jq .
{
"jsonrpc": "2.0",
"result": [
"shmem:fragments = 18",
"shmem:free_size = 467187808",
"shmem:max_used_size = 69694104",
"shmem:real_used_size = 69683104",
"shmem:total_size = 536870912",
"shmem:used_size = 41048984"
],
"id": 4300
}
I would hope would be more like this:
{
"jsonrpc":"2.0",
"result":[
{
"shmem":[
{
"fragments":18,
"free_size":467187808,
"max_used_size":69694104,
"real_used_size":69683104,
"total_size":536870912,
"used_size":41048984
}
]
}
],
"id":4300
}
I apologize in advanced because I don’t have the skillset to contribute such a feature.
Thanks!
~Noah
Hi,
I've set up a dmq server for other kamailio nodes with this parameters:
```
modparam("dmq", "server_address","sip:10.0.0.101:5060")
modparam("dmq", "notification_address", "sip:127.0.0.1:5060")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
modparam("dmq", "ping_interval", 15)
```
On the kamailio nodes the config is the following:
```
modparam("dmq", "server_address", "sip:10.0.0.x:5060")
modparam("dmq", "notification_address", "sip:10.0.0.101:5060")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
modparam("dmq", "ping_interval", 15)
```
On the kamailio dmq server I've got this in the logs:
```
25(41) ERROR: dmq [notification_peer.c:596]:
notification_resp_callback_f(): deleting server sip:127.0.0.1:5060
because of failed request
25(41) ERROR: dmq [notification_peer.c:599]:
notification_resp_callback_f(): not deleting notification_peer
```
On the kamailio nodes the same thing with the ip of the dmq server:
```
25(112) ERROR: dmq [notification_peer.c:596]:
notification_resp_callback_f(): deleting server sip:10.0.0.101:5060
because of failed request
25(112) ERROR: dmq [notification_peer.c:599]:
notification_resp_callback_f(): not deleting notification_peer
```
The request route on the dmq server is simple as this:
```
request_route {
if(is_method("KDMQ")){
dmq_handle_message();
}
}#end request_route
```
I've tried with some xlog-s inside the if(is_method("KDMQ")) but they
don't show up so the dmq_handle_message() is not even executed.
I've also tried to bind the dmq server to a different port but with no
success on making it work.
The dmq.list_nodes shows only the dmq server IP and the current node
IP, the IPs of other nodes are not shown:
```
root@proxy-94b6ccf46-t8fx2:/# kamcmd dmq.list_nodes
{
host: 10.0.0.101
port: 5060
resolved_ip: 10.0.0.101
status: active
last_notification: 0
local: 0
}
{
host: 10.0.0.45
port: 5060
resolved_ip: 10.0.0.45
status: active
last_notification: 0
local: 1
}
```
And it's the same on all other kamailio nodes: dmq server ip and local ip.
The kamailio version on all the nodes is:
5.2.0~dev6+0~20180616010152.1138+xenial
Thanks,
--
Aleksandar Sosic
mail: alex.sosic(a)evosip.cloud
Hello,
I want to highlight that the last stable versions (for the two maintained
series: 5.0 and 5.1) include fixes for an security issues that can crash a
running instance of Kamailio, therefore it is strongly recommended to upgrade.
Details:
There exists a security vulnerability in the Kamailio SIP server related to
"To" header processing. A specially crafted SIP message with double "To"
header and an empty "To" tag causes a segmentation fault and crashes Kamailio.
The reason is missing input validation in the "build_res_buf_from_sip_req"
core function.
If an attacker sends many of this messages this would lead to a Denial of
Service of the attacked infrastructure. This is especially critical as no
authentication for the remote source is needed.
You find all the details including a proof of concept code in the published
security announcement in my blog:
https://skalatan.de/blog/advisory-hw-2018-05
A CVE report for this issue is going to be created as well in the near future.
The issues were found some months ago and were fixed quickly. The code related
to the reported issues is rather old and there are no known incidents of
exploiting these issues so far. However, once the CVE report becomes public,
there could be a higher risk of exploitation.
Please address any detailed technical questions related to this to the
developer list at sr-dev(a)lists.kamailio.org .
In case of confidential remarks related to this or other security issues,
please address them to the Kamailio Management.
Best regards,
Henning Westerholt
--
Henning Westerholt
https://skalatan.de/blog/
Hi All,
I've come across and oddity whereby my is_method("KDMQ") checks are not
matching.
I'm testing with kamailio master:
version: kamailio 5.2.0-dev6 (x86_64/linux) a16272
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: a16272
compiled on 15:27:21 Jul 31 2018 with gcc 7.3.1
Sample routing logic:
request_route {
route("InitialMessageChecks");
exit;
}
route[InitialMessageChecks] {
xlog("L_INFO", "Received message type $rm from $si:$sp on $Ri:$Rp");
if ( is_method("KDMQ") ) { xlog("L_INFO", "KDMQ check passed."); }
if ( is_method("OPTIONS") ) { xlog("L_INFO", "OPTIONS check
passed."); }
if ( is_method("SUBSCRIBE") ) { xlog("L_INFO", "SUBSCRIBE check
passed."); }
}
In the logs I am only receiving the 1st log line in the initial message
checks route for message types of KDMQ, eg:
INFO: <script>: Received message KDMQ from 1.1.1.1:5060 on 1.1.1.2:5060
INFO: <script>: Received message KDMQ from 1.1.1.3:5060 on 1.1.1.2:5060
I never seem to hit the is_method check log line when the method type is
KDMQ.
I do see the options/subscribe logs though:
INFO: <script>: OPTIONS check passed.
INFO: <script>: SUBSCRIBE check passed.
nothing for KDMQ.
Reason for asking is: this check never matches and breaks the dmq bus:
if ( is_method("KDMQ") ) {
if ( dmq_is_from_node() ) {
dmq_handle_message();
exit;
} else {
exit;
}
}
Any thoughts, or am I going crazy :)
Thanks
Hi All,
I'm unable to register an extension to the Upstream Provider by using uac
module.
Getting the following error:
*Jul 31 15:49:39 k1-core1 k1[13146]: ERROR: <core> [core/forward.c:181]:
get_out_socket(): no socket found*
*Jul 31 15:49:39 k1-core1 k1[13146]: ERROR: <core> [core/forward.c:183]:
get_out_socket(): no corresponding socket found for(udp:8.8.8.8:5060
<http://8.8.8.8:5060>)*
*Jul 31 15:49:39 k1-core1 k1[13146]: ERROR: tm [ut.h:317]: uri2dst2(): no
corresponding socket found for "svr04.nuacom.net <http://svr04.nuacom.net>"
af 2 (udp:8.8.8.8:5060 <http://8.8.8.8:5060>)*
*Jul 31 15:49:39 k1-core1 k1[13146]: ERROR: tm [uac.c:449]:
t_uac_prepare(): no socket found*
*Jul 31 15:49:39 k1-core1 k1[13146]: ERROR: uac [uac_reg.c:1161]:
uac_reg_update(): failed to send request for [1050]*
Kamailio is listening on 2 interfaces:
[root@kamailio user]# /usr/local/sbin/kamcmd core.sockets_list
{
socket: {
proto: udp
address: 10.1.1.11
ipaddress: 10.1.1.11
port: 5060
mcast: no
mhomed: no
}
socket: {
proto: udp
address: 2.2.2.2
ipaddress: 2.2.2.2
port: 5060
mcast: no
mhomed: no
}
mhomed is enabled in the configuration file: *mhomed=1*
I'm not using ' force_send_socket' anywhere in the configuration file.
This is the uac module configuration:
modparam("uac", "reg_keep_callid", 1)
modparam("uac", "reg_contact_addr", "2.2.2.2")
modparam("uac", "reg_db_url","cluster://kamailio")
If i'm disabling mhomed from configuration file - i don't receive any
errors - but then kamailio is trying to send the message to the external
server (8.8.8.8) through internal interface (10.1.1.1).
Thank You.
