[SR-Users] Unable to start kamailio_pcscf.service

Henning Westerholt hw at skalatan.de
Thu Aug 22 13:57:32 CEST 2019 

Hello,

Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: ERROR: <script>: event_route[htable:mod-init] {

Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: ERROR: db_mysql [km_my_con.c:139]: db_mysql_new_connection(): driver error: MySQL client ran out of memory

this error looks like you try to load a big table into memory on startup. Investigate this error, try with a smaller data set first.

You need to investigate this errors and find out why they happen in your setup. I don't know all the details of your setup.

Cheers,

Henning

Am 22.08.19 um 13:18 schrieb Jack R:
Thanks Henning,

I did that and restarted the service same error.

As per this article: https://github.com/open5gs/nextepc/commit/9d8203eb685db75955b6bcfc27db0396e9958ffd#diff-4e5dd35f5e9f04228b7fe66c6d24fdaf

All three 3 services have been split. (pscsf, iscsf, scscf)
So iscsf show active (running) rest other two active (exited)

Also, I happen to check the syslog whenever this command is issued> systemctl restart kamailio_pcscf (same even shows previously

root at ubuntu18-04:/etc/default# less /var/log/syslog
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: INFO: jsonrpcs [jsonrpcs_sock.c:197]: jsonrpc_dgram_mod_init(): the socket /var/run/kamailio/kamailio_rpc.sock already exists, trying to delete it...
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: INFO: <core> [main.c:2669]: main(): processes (at least): 66 - shm size: 268435456 - pkg size: 16777216
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: WARNING: tm [tm.c:511]: fixup_routes(): t_on_failure("NATMANAGE"): empty/non existing route
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: INFO: <core> [core/udp_server.c:153]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: INFO: <core> [core/udp_server.c:205]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: INFO: <core> [core/udp_server.c:153]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: INFO: <core> [core/udp_server.c:205]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: INFO: <core> [core/udp_server.c:153]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: INFO: <core> [core/udp_server.c:205]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: ERROR: <script>: event_route[htable:mod-init] {
Aug 22 11:04:05 localhost rtpengine[6116]: INFO: Received command 'ping' from 127.0.0.1:48276<http://127.0.0.1:48276>
Aug 22 11:04:05 localhost rtpengine[6116]: INFO: Replying to 'ping' from 127.0.0.1:48276<http://127.0.0.1:48276> (elapsed time 0.000001 sec)
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20777]: INFO: rtpengine [rtpengine.c:2623]: rtpp_test(): rtp proxy <udp:localhost:12222> found, support for it enabled
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20775]: ERROR: db_mysql [km_my_con.c:139]: db_mysql_new_connection(): driver error: MySQL client ran out of memory
Aug 22 11:04:05 localhost t of memory [20775]
Aug 22 11:04:05 localhost t message repeated 3 times: [ of memory [20775]]
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20847]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20871]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20866]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20862]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20865]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20855]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20853]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20836]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20845]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20851]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20829]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20831]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20826]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20817]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20842]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20833]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20821]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20858]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20840]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20803]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20824]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20787]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20819]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20849]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20813]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost /usr/sbin/kamailio[20869]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 11:04:05 localhost t of memory [20775]
Aug 22 11:04:05 localhost t message repeated 9 times: [ of memory [20775]]
Aug 22 11:04:05 localhost kamailio_pcscf[20737]:  *  already running
Aug 22 11:04:05 localhost kamailio_pcscf[20737]:    ...done.
Aug 22 11:04:05 localhost systemd[1]: Started LSB: Start the Kamailio SIP proxy server.


Please advise.

Regards.

On Thu, Aug 22, 2019 at 4:14 PM Henning Westerholt <hw at skalatan.de<mailto:hw at skalatan.de>> wrote:

Hello Jack,

you need to remove the '#' in front of the respective lines, this is just a comment.

You could use 256 MB of SHM_MEM and 16 MB of PKG, kamailio will just use the memory if necessary.

Try if this works. The "signal 15 recieved" messages are probably just from the restart.

Check e.g. with "kamcmd core.uptime" or "ps aux | grep kamailio" if Kamailio is running.

Cheers,

Henning

Am 22.08.19 um 12:17 schrieb Jack R:
Thanks Henning,

1. I tried changing the numbers keeping '#' by default number was 64 and 8 changed to 96 and 16
# Amount of shared and private memory to allocate
# for the running Kamailio server (in Mb)
#SHM_MEMORY=96
#PKG_MEMORY=16

2. Then got a different message when restarted the service in status;
root at ubuntu18-04:/etc/default# systemctl start kamailio_pcscf
root at ubuntu18-04:/etc/default# systemctl status kamailio_pcscf
● kamailio_pcscf.service - LSB: Start the Kamailio SIP proxy server
   Loaded: loaded (/etc/init.d/kamailio_pcscf; generated)
   Active: active (exited) since Thu 2019-08-22 08:33:39 UTC; 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 14702 ExecStart=/etc/init.d/kamailio_pcscf start (code=exited, status=0/SUCCESS)

Aug 22 08:32:39 ubuntu18-04 /usr/sbin/kamailio[14825]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 08:32:39 ubuntu18-04 /usr/sbin/kamailio[14809]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 08:32:39 ubuntu18-04 /usr/sbin/kamailio[14811]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 08:32:39 ubuntu18-04 /usr/sbin/kamailio[14826]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 08:32:39 ubuntu18-04 /usr/sbin/kamailio[14830]: INFO: ctl [io_listener.c:210]: io_listen_loop(): io_listen_loop: using epoll_lt as the io watch method (auto detecte
Aug 22 08:32:39 ubuntu18-04 /usr/sbin/kamailio[14737]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
Aug 22 08:33:39 ubuntu18-04 kamailio[14733]: ERROR: <core> [core/daemonize.c:303]: daemonize(): Main process exited before writing to pipe
Aug 22 08:33:39 ubuntu18-04 kamailio_pcscf[14702]:  *  already running
Aug 22 08:33:39 ubuntu18-04 kamailio_pcscf[14702]:    ...done.
Aug 22 08:33:39 ubuntu18-04 systemd[1]: Started LSB: Start the Kamailio SIP proxy server.

3. Restarted the service again now same error was back of 'out of memory'. So now changed SHM_MEMORY PKG_MEMORY number with and without '#':
I noticed that whenever starting service takes little bit time it shows the different message, the above one, not the 'out of memory'  and this happened only twice, generally the service starts quickly and now only showing whenever I am restarting the service:

root at ubuntu18-04:/home/adminipit# systemctl stop kamailio_pcscf
root at ubuntu18-04:/home/adminipit# systemctl start kamailio_pcscf
root at ubuntu18-04:/home/adminipit# systemctl status kamailio_pcscf
● kamailio_pcscf.service - LSB: Start the Kamailio SIP proxy server
   Loaded: loaded (/etc/init.d/kamailio_pcscf; generated)
   Active: active (exited) since Thu 2019-08-22 10:12:32 UTC; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 19992 ExecStart=/etc/init.d/kamailio_pcscf start (code=exited, status=0/SUCCESS)

Aug 22 10:12:32 ubuntu18-04 kamailio[20024]: out of memory [20024]
Aug 22 10:12:32 ubuntu18-04 kamailio[20024]: out of memory [20024]
Aug 22 10:12:32 ubuntu18-04 kamailio[20024]: out of memory [20024]
Aug 22 10:12:32 ubuntu18-04 kamailio[20024]: out of memory [20024]
Aug 22 10:12:32 ubuntu18-04 kamailio[20024]: out of memory [20024]
Aug 22 10:12:32 ubuntu18-04 kamailio[20024]: out of memory [20024]
Aug 22 10:12:32 ubuntu18-04 kamailio[20024]: out of memory [20024]
Aug 22 10:12:32 ubuntu18-04 kamailio_pcscf[19992]:  *  already running
Aug 22 10:12:32 ubuntu18-04 kamailio_pcscf[19992]:    ...done.
Aug 22 10:12:32 ubuntu18-04 systemd[1]: Started LSB: Start the Kamailio SIP proxy server.

Please advise.

Regards.

On Thu, Aug 22, 2019 at 1:26 PM Henning Westerholt <hw at skalatan.de<mailto:hw at skalatan.de>> wrote:

Hello Jack,

the memory errors indeed don't look good. Check your SHM and PKG memory size and try to increase it. At Debian/Ubuntu this should be configured at /etc/default/kamailio

Cheers,

Henning

Am 22.08.19 um 09:28 schrieb Jack R:
Hi Henning,

Still no luck, Now service Active: active (exited)

root at ubuntu18-04:/etc/init.d#<mailto:root at ubuntu18-04:/etc/init.d#> ./kamailio_pcscf status
● kamailio_pcscf.service - LSB: Start the Kamailio SIP proxy server
   Loaded: loaded (/etc/init.d/kamailio_pcscf; generated)
   Active: active (exited) since Thu 2019-08-22 07:19:21 UTC; 7min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13495 ExecStop=/etc/init.d/kamailio_pcscf stop (code=exited, status=0/SUCCESS)
  Process: 13517 ExecStart=/etc/init.d/kamailio_pcscf start (code=exited, status=0/SUCCESS)

Aug 22 07:19:21 ubuntu18-04 kamailio[13562]: out of memory [13562]
Aug 22 07:19:21 ubuntu18-04 kamailio[13562]: out of memory [13562]
Aug 22 07:19:21 ubuntu18-04 kamailio[13562]: out of memory [13562]
Aug 22 07:19:21 ubuntu18-04 kamailio[13562]: out of memory [13562]
Aug 22 07:19:21 ubuntu18-04 kamailio[13562]: out of memory [13562]
Aug 22 07:19:21 ubuntu18-04 kamailio[13562]: out of memory [13562]
Aug 22 07:19:21 ubuntu18-04 kamailio[13562]: out of memory [13562]
Aug 22 07:19:21 ubuntu18-04 kamailio_pcscf[13517]:  *  already running
Aug 22 07:19:21 ubuntu18-04 kamailio_pcscf[13517]:    ...done.
Aug 22 07:19:21 ubuntu18-04 systemd[1]: Started LSB: Start the Kamailio SIP proxy server.

(as below log show) Had restarted the service at 07:19>./kamailio_pcscf restart

root at ubuntu18-04:/etc/init.d#<mailto:root at ubuntu18-04:/etc/init.d#> less /var/log/syslog | grep pcscf
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13300]: INFO: ims_usrloc_pcscf [hslot.c:62]: ul_init_locks(): locks array size 512
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13397]: WARNING: ims_usrloc_pcscf [usrloc_db.c:61]: connect_db(): DB connection already open... continuing
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13399]: WARNING: ims_usrloc_pcscf [usrloc_db.c:61]: connect_db(): DB connection already open... continuing
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13400]: WARNING: ims_usrloc_pcscf [usrloc_db.c:61]: connect_db(): DB connection already open... continuing
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13401]: WARNING: ims_usrloc_pcscf [usrloc_db.c:61]: connect_db(): DB connection already open... continuing
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13403]: WARNING: ims_usrloc_pcscf [usrloc_db.c:61]: connect_db(): DB connection already open... continuing
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13404]: WARNING: ims_usrloc_pcscf [usrloc_db.c:61]: connect_db(): DB connection already open... continuing
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13405]: WARNING: ims_usrloc_pcscf [usrloc_db.c:61]: connect_db(): DB connection already open... continuing
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13406]: WARNING: ims_usrloc_pcscf [usrloc_db.c:61]: connect_db(): DB connection already open... continuing
Aug 22 07:14:19 localhost /usr/sbin/kamailio[13413]: WARNING: ims_usrloc_pcscf [usrloc_db.c:61]: connect_db(): DB connection already open... continuing
Aug 22 07:19:20 localhost kamailio_pcscf[13495]:  * Stopping Kamailio SIP Server: kamailio_pcscf
Aug 22 07:19:20 localhost kamailio_pcscf[13495]:    ...done.
Aug 22 07:19:21 localhost kamailio_pcscf[13517]:  * Starting Kamailio SIP Server: kamailio_pcscf
Aug 22 07:19:21 localhost kamailio_pcscf[13517]: loading modules under config path: /usr/lib64/kamailio/modules_k/:/usr/lib64/kamailio/modules/:/usr/lib/kamailio/modules_k/:/usr/lib/kamailio/modules/:/usr/local/lib64/kamailio/modules/
Aug 22 07:19:21 localhost kamailio_pcscf[13517]: Listening on
Aug 22 07:19:21 localhost kamailio_pcscf[13517]:              udp: 10.4.98.3 [10.4.98.3]:5060
Aug 22 07:19:21 localhost kamailio_pcscf[13517]: Aliases:
Aug 22 07:19:21 localhost kamailio_pcscf[13517]:              *: pcscf.mnc096.mcc262.3gppnetwork.org:*
Aug 22 07:19:21 localhost /usr/sbin/kamailio[13562]: INFO: ims_usrloc_pcscf [hslot.c:62]: ul_init_locks(): locks array size 512
Aug 22 07:19:21 localhost kamailio_pcscf[13517]:  *  already running
Aug 22 07:19:21 localhost kamailio_pcscf[13517]:    ...done.

On Tue, Aug 20, 2019 at 12:18 AM Henning Westerholt <hw at skalatan.de<mailto:hw at skalatan.de>> wrote:

Hello Jack,

as you can see in the logs there is an error in the cfg file. Have a look to the kamailio logs (e.g. in /var/log/messages or /var/log/syslog) to get more hints about the cause.

Cheers,

Henning

Am 19.08.19 um 12:23 schrieb Jack R:
1. Followed steps from Step # 3 to Step # 16 : https://github.com/open5gs/nextepc/blob/master/docs/_docs/guide/04-setting-up-kamailio-IMS.md

2. When I run last command >systemctl start kamailio_icscf kamailio_pcscf kamailio_scscfthe

root at ubuntu18-04:/etc/init.d#<mailto:root at ubuntu18-04:/etc/init.d#> systemctl start kamailio_icscf kamailio_pcscf kamailio_scscf
Job for kamailio_pcscf.service failed because the control process exited with error code.
See "systemctl status kamailio_pcscf.service" and "journalctl -xe" for details.

3. All services start but not kamailio_pcscf:
root at ubuntu18-04:/etc/init.d#<mailto:root at ubuntu18-04:/etc/init.d#> systemctl status kamailio_pcscf.service
● kamailio_pcscf.service - LSB: Start the Kamailio SIP proxy server
   Loaded: loaded (/etc/init.d/kamailio_pcscf; generated)
   Active: failed (Result: exit-code) since Mon 2019-08-19 09:05:30 UTC; 45min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 52271 ExecStart=/etc/init.d/kamailio_pcscf start (code=exited, status=1/FAILURE)

Aug 19 09:05:30 ubuntu18-04 kamailio_pcscf[52271]:  0(52294) INFO: pv [pv_shv.c:60]: shvar_init_locks(): locks array size 16
Aug 19 09:05:30 ubuntu18-04 kamailio_pcscf[52271]: ERROR: bad config file (1 errors)
Aug 19 09:05:30 ubuntu18-04 kamailio_pcscf[52271]:  0(52294) WARNING: <core> [core/mem/q_malloc.c:480]: qm_free(): WARNING: free(0) called from cdp_avp: cdp_avp_mod.c: cdp
Aug 19 09:05:30 ubuntu18-04 kamailio_pcscf[52271]:  0(52294) INFO: cdp [cdp_mod.c:255]: cdp_exit(): CDiameterPeer child stopping ...
Aug 19 09:05:30 ubuntu18-04 kamailio_pcscf[52271]:  0(52294) INFO: cdp [cdp_mod.c:257]: cdp_exit(): ... CDiameterPeer child stopped
Aug 19 09:05:30 ubuntu18-04 kamailio_pcscf[52271]:  0(52294) INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized
Aug 19 09:05:30 ubuntu18-04 kamailio_pcscf[52271]:  *
Aug 19 09:05:30 ubuntu18-04 systemd[1]: kamailio_pcscf.service: Control process exited, code=exited status=1
Aug 19 09:05:30 ubuntu18-04 systemd[1]: kamailio_pcscf.service: Failed with result 'exit-code'.
Aug 19 09:05:30 ubuntu18-04 systemd[1]: Failed to start LSB: Start the Kamailio SIP proxy server.
lines 1-16/16 (END)

Please advise.

Thanks and Regards.



_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users


--
Henning Westerholt - https://skalatan.de/blog/
Kamailio services - https://skalatan.de/services

--
Henning Westerholt - https://skalatan.de/blog/
Kamailio services - https://skalatan.de/services

--
Henning Westerholt - https://skalatan.de/blog/
Kamailio services - https://skalatan.de/services

--
Henning Westerholt - https://skalatan.de/blog/
Kamailio services - https://skalatan.de/services
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190822/05feafb2/attachment.html>

More information about the sr-users mailing list