[SR-Users] Intermittent start up issue
Daniel-Constantin Mierla
miconda at gmail.com
Tue Jan 3 11:44:22 CET 2012
Hello,
just another idea to try, not sure if it is convenient -- can you
install and use different version of mysql server and client library?
This should reveal whether is an issue specific to mysql version.
Cheers,
Daniel
On 1/3/12 11:35 AM, Daniel-Constantin Mierla wrote:
> Hello,
>
> all it is fine with the output. So it looks like the limit is
> somewhere above 25 connections.
>
> As a try of making a workaround in kamailio side, could be a patch to
> add a global parameter to introduce some latency before forking a
> worker, so these processes won't be created all at a time, but one
> after the other.
>
> However, would be good to find the real reason for such behavior. If
> the tcp connection gets to mysql server, maybe the question should be
> addressed to mysql forums/mailing lists.
>
> Cheers,
> Daniel
>
>
>
> On 1/3/12 10:48 AM, Ali Jawad wrote:
>> Hi
>> I dont think a few hundred per second is enough I need to find out
>> what is wrong and bring up no of children to 8. The MySQL server is at
>> another host on the same LAN, as for the command kamctl ps see below
>>
>> Process:: ID=0 PID=13949 Type=attendant
>> Process:: ID=1 PID=13950 Type=udp receiver child=0
>> sock=xx.xx.xx.xx:5060
>> Process:: ID=2 PID=13951 Type=udp receiver child=1
>> sock=xx.xx.xx.xx:5060
>> Process:: ID=3 PID=13952 Type=udp receiver child=2
>> sock=xx.xx.xx.xx:5060
>> Process:: ID=4 PID=13953 Type=udp receiver child=0
>> sock=xx.xx.xx.xx:5061
>> Process:: ID=5 PID=13954 Type=udp receiver child=1
>> sock=xx.xx.xx.xx:5061
>> Process:: ID=6 PID=13955 Type=udp receiver child=2
>> sock=xx.xx.xx.xx:5061
>> Process:: ID=7 PID=13956 Type=udp receiver child=0 sock=xx.xx.xx.xx:443
>> Process:: ID=8 PID=13957 Type=udp receiver child=1 sock=xx.xx.xx.xx:443
>> Process:: ID=9 PID=13958 Type=udp receiver child=2 sock=xx.xx.xx.xx:443
>> Process:: ID=10 PID=13959 Type=udp receiver child=0
>> sock=xx.xx.xx.xx:8080
>> Process:: ID=11 PID=13960 Type=udp receiver child=1
>> sock=xx.xx.xx.xx:8080
>> Process:: ID=12 PID=13961 Type=udp receiver child=2
>> sock=xx.xx.xx.xx:8080
>> Process:: ID=13 PID=13962 Type=udp receiver child=0
>> sock=xx.xx.xx.xx:123
>> Process:: ID=14 PID=13963 Type=udp receiver child=1
>> sock=xx.xx.xx.xx:123
>> Process:: ID=15 PID=13964 Type=udp receiver child=2
>> sock=xx.xx.xx.xx:123
>> Process:: ID=16 PID=13965 Type=slow timer
>> Process:: ID=17 PID=13966 Type=timer
>> Process:: ID=18 PID=13967 Type=MI FIFO
>> Process:: ID=19 PID=13968 Type=ctl handler
>> Process:: ID=20 PID=13969 Type=TIMER NH
>> Process:: ID=21 PID=13970 Type=tcp receiver child=0
>> Process:: ID=22 PID=13971 Type=tcp receiver child=1
>> Process:: ID=23 PID=13972 Type=tcp receiver child=2
>> Process:: ID=24 PID=13973 Type=tcp main process
>>
>> I use a few different ports as you can see.
>>
>>
>> All fine with that output at 3 ?
>>
>> Regards
>>
>> On Tue, Jan 3, 2012 at 11:29 AM, Daniel-Constantin Mierla
>> <miconda at gmail.com> wrote:
>>> Hello,
>>>
>>>
>>> On 1/3/12 9:14 AM, Ali Jawad wrote:
>>>> Hi Daniel
>>>>
>>>> SeLinux is disabled, as I have played around with Children and got a
>>>> stable startup at CHILDREN=3, at 4 and up I get the problem described
>>>> earlier right away, so it seems that MySQL is throttling Kamailio ?
>>>>
>>>> 1- What is the effect of running 3 children only ?
>>> the capacity of handling sip traffic is lowered. You should still be
>>> able to
>>> handle couple of hundred messages per second if you don't have slow
>>> sql/dns
>>> operations. For production, it is recommended to have at least 8 to
>>> be in
>>> safe side.
>>>
>>>
>>>> 2- I know this is not a Kamailio thing, but any idea about what needs
>>>> to be changed on MySQL server to accommodate more children ? No of
>>>> connections is 500 conns out of which max 120 are utilized
>>> So mysql is on a different server, right? Are you sure there is no
>>> other
>>> switch/router doing such throttling? Can you watch on both kamailio and
>>> mysql server to see if the connections are initiated from kamailio and
>>> reaching mysql? Maybe you can discover if mysql server get the tcp
>>> connection but refuses it -- not sure if mysql server can run in
>>> debug/verbose mode to get some log messages with hints about what is
>>> the
>>> problem.
>>>
>>> If the tcp connection does not get to mysql server, then it is a
>>> throttling
>>> in mysql client library or on a device in between the servers.
>>>
>>> If you do 'kamctl ps' when having children=3, how many kamailio
>>> processes
>>> are there (extra should be timers, ...)? Each should have one mysql
>>> connection, so it helps to see at which number the limit starts.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>> Regards
>>>>
>>>> On Mon, Jan 2, 2012 at 1:31 PM, Daniel-Constantin Mierla
>>>> <miconda at gmail.com> wrote:
>>>>> Hello,
>>>>>
>>>>>
>>>>> On 1/2/12 12:24 PM, Ali Jawad wrote:
>>>>>> Hi Daniel
>>>>>> Running on a LAN with no firewalls enabled,
>>>>>
>>>>> limits can be from kernel, selinix, mysql server a.s.o. not only
>>>>> from
>>>>> firewalls.
>>>>>
>>>>>
>>>>>> will try you suggestion
>>>>>> though first thing tomorrow morning, what are the implications of
>>>>>> running children=1 ?
>>>>>
>>>>> This will create only one worker process (not advisable for
>>>>> production in
>>>>> long term, just for testing) -- in this way there should be very few
>>>>> mysql
>>>>> connections opened, revealing if there is a problem or not with the
>>>>> number
>>>>> of connections and the short period to create them.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Mon, Jan 2, 2012 at 2:03 PM, Daniel-Constantin Mierla
>>>>>> <miconda at gmail.com> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> check also for limits in the system/firewall, maybe there are
>>>>>>> limits on
>>>>>>> new
>>>>>>> connections per second. Try with children=1 and see if it happens
>>>>>>> again,
>>>>>>> if
>>>>>>> not, then it is something that blocks many new connections in short
>>>>>>> time.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>>
>>>>>>> On 1/2/12 5:31 AM, Sammy Govind wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> Its all upto you then, that's an intermittent issue and
>>>>>>> information is
>>>>>>> not
>>>>>>> complete to trace the issue. Did you see the same Errors every
>>>>>>> time it
>>>>>>> fails
>>>>>>> !?
>>>>>>>
>>>>>>> --
>>>>>>> Regards
>>>>>>> Sammy
>>>>>>>
>>>>>>> On Sun, Jan 1, 2012 at 10:06 PM, Ali Jawad<ali.jawad at splendor.net>
>>>>>>> wrote:
>>>>>>>> Hi
>>>>>>>> Thanks for your input, but as said this works at times yes and
>>>>>>>> times
>>>>>>>> now so I am not sure it is a definite DB issue, the servers are
>>>>>>>> in the
>>>>>>>> same LAN, and there are other modules that use the DB string
>>>>>>>> that work
>>>>>>>> when launched before this one while this one does not work.
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> On Sun, Jan 1, 2012 at 4:13 PM, Sammy Govind<govoiper at gmail.com>
>>>>>>>> wrote:
>>>>>>>>> Hi,
>>>>>>>>> forget the whole list of errors and just resolve the DB
>>>>>>>>> connectivity
>>>>>>>>> between
>>>>>>>>> the Kamailio server and DB server first .
>>>>>>>>>
>>>>>>>>>> 27(24762) ERROR: db_mysql [km_my_con.c:109]: driver error: Can't
>>>>>>>>>> connect to MySQL server on 'xxxxxxxxx' (4)
>>>>>>>>>
>>>>>>>>> try connecting to the remote DB server 'xxxxxxxxx' from linux
>>>>>>>>> shell
>>>>>>>>> i.e
>>>>>>>>>
>>>>>>>>> # mysql -u<user> -p<passwd> -h 'xxxxxxxxx'
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Sammy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Dec 30, 2011 at 3:33 PM, Ali
>>>>>>>>> Jawad<ali.jawad at splendor.net>
>>>>>>>>> wrote:
>>>>>>>>>> Hi All
>>>>>>>>>> I have tried the below with Kamailio 3.1.0 and 3.2.1. the
>>>>>>>>>> problem is
>>>>>>>>>> that Kamailio at times fails to start up for no obvious reasons,
>>>>>>>>>> I.e.
>>>>>>>>>> in 10 restarts Kamailio might fail to start 3,0 or 5 times.
>>>>>>>>>> Te debug
>>>>>>>>>> log "level 2 " shows the following :
>>>>>>>>>>
>>>>>>>>>> 27(24762) ERROR: db_mysql [km_my_con.c:109]: driver error: Can't
>>>>>>>>>> connect to MySQL server on 'xxxxxxxxx' (4)
>>>>>>>>>> 27(24762) ERROR:<core> [db.c:303]: could not add
>>>>>>>>>> connection to
>>>>>>>>>> the
>>>>>>>>>> pool27(24762) ERROR: group [group.c:211]: unable to connect
>>>>>>>>>> to the
>>>>>>>>>> database
>>>>>>>>>> 27(24762) ERROR:<core> [sr_module.c:889]:
>>>>>>>>>> init_mod_child(): Error
>>>>>>>>>> while initializing module group
>>>>>>>>>> (/usr/local/kamailio/lib/kamailio/modules_k/group.so)
>>>>>>>>>> 27(24762) ERROR:<core> [pt.c:481]: ERROR:
>>>>>>>>>> fork_tcp_process():
>>>>>>>>>> init_child failed for process 27, pid 24762, "tcp receiver
>>>>>>>>>> child=1"
>>>>>>>>>>
>>>>>>>>>> and then
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 30(24765) :<core> [pass_fd.c:293]: ERROR: receive_fd:
>>>>>>>>>> EOF on 45
>>>>>>>>>> 0(24735) ALERT:<core> [main.c:748]: child process
>>>>>>>>>> 24762 exited
>>>>>>>>>> normally, status=255
>>>>>>>>>> 0(24735) INFO:<core> [main.c:766]: INFO: terminating
>>>>>>>>>> due to
>>>>>>>>>> SIGCHLD
>>>>>>>>>>
>>>>>>>>>> The problem happens on 4 different servers, I did try to compile
>>>>>>>>>> more
>>>>>>>>>> than once, the servers are 32 Bit CentOS 5.6. The connection
>>>>>>>>>> to the
>>>>>>>>>> database is monitored and there are no issues with connecting
>>>>>>>>>> to the
>>>>>>>>>> DB "it works at times and times not for Kamailio so a
>>>>>>>>>> user/pwd issue
>>>>>>>>>> is not likely to cause this, manual cnnect worked", config file
>>>>>>>>>> errors
>>>>>>>>>> are not found.
>>>>>>>>>>
>>>>>>>>>> I did also check max no of connections on MySQL server which
>>>>>>>>>> is set
>>>>>>>>>> to
>>>>>>>>>> 500 on the server and only 100 cons are used. The group
>>>>>>>>>> module uses
>>>>>>>>>> the same connection string as other modules which is predefined
>>>>>>>>>> and
>>>>>>>>>> there are other modules that access the db before the group
>>>>>>>>>> module
>>>>>>>>>> and
>>>>>>>>>> those do not have a problem.
>>>>>>>>>>
>>>>>>>>>> Please advice.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>>>>>>>>>> mailing
>>>>>>>>>> list
>>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>>>>>>>>> mailing
>>>>>>>>> list
>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Ali Jawad
>>>>>>>> Information Systems Manager
>>>>>>>> Splendor Telecom (www.splendor.net)
>>>>>>>> Beirut, Lebanon
>>>>>>>> Phone: +9611373725/ext 116
>>>>>>>> FAX: +9611375554
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>> list
>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>>>>>>> mailing list
>>>>>>> sr-users at lists.sip-router.org
>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>>>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>>>>>>> mailing list
>>>>>>> sr-users at lists.sip-router.org
>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>>
>>>>
>>> --
>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>
>>
>>
>
--
Daniel-Constantin Mierla -- http://www.asipto.com
http://linkedin.com/in/miconda -- http://twitter.com/miconda
More information about the sr-users
mailing list