[SR-Users] Intermittent start up issue
Daniel-Constantin Mierla
miconda at gmail.com
Tue Jan 3 16:10:35 CET 2012
Hello,
On 1/3/12 11:58 AM, Ali Jawad wrote:
> Same client library different server version working fine !
ok, good to know. What was the version of mysql sever giving troubles?
> Any chance
> the patch can be released anytime soon ?
That was an idea, but it is not sure it will fix, it has to be tried. I
will see when I have time for it and notify you to try it once is committed.
Cheers,
Daniel
> I will migrate to the newer
> MySQL server in the next few mos though, I can wait if needed.
> Regards
>
> On Tue, Jan 3, 2012 at 12:44 PM, Daniel-Constantin Mierla
> <miconda at gmail.com> wrote:
>> Hello,
>>
>> just another idea to try, not sure if it is convenient -- can you install
>> and use different version of mysql server and client library? This should
>> reveal whether is an issue specific to mysql version.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 1/3/12 11:35 AM, Daniel-Constantin Mierla wrote:
>>> Hello,
>>>
>>> all it is fine with the output. So it looks like the limit is somewhere
>>> above 25 connections.
>>>
>>> As a try of making a workaround in kamailio side, could be a patch to add
>>> a global parameter to introduce some latency before forking a worker, so
>>> these processes won't be created all at a time, but one after the other.
>>>
>>> However, would be good to find the real reason for such behavior. If the
>>> tcp connection gets to mysql server, maybe the question should be addressed
>>> to mysql forums/mailing lists.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>>
>>> On 1/3/12 10:48 AM, Ali Jawad wrote:
>>>> Hi
>>>> I dont think a few hundred per second is enough I need to find out
>>>> what is wrong and bring up no of children to 8. The MySQL server is at
>>>> another host on the same LAN, as for the command kamctl ps see below
>>>>
>>>> Process:: ID=0 PID=13949 Type=attendant
>>>> Process:: ID=1 PID=13950 Type=udp receiver child=0 sock=xx.xx.xx.xx:5060
>>>> Process:: ID=2 PID=13951 Type=udp receiver child=1 sock=xx.xx.xx.xx:5060
>>>> Process:: ID=3 PID=13952 Type=udp receiver child=2 sock=xx.xx.xx.xx:5060
>>>> Process:: ID=4 PID=13953 Type=udp receiver child=0 sock=xx.xx.xx.xx:5061
>>>> Process:: ID=5 PID=13954 Type=udp receiver child=1 sock=xx.xx.xx.xx:5061
>>>> Process:: ID=6 PID=13955 Type=udp receiver child=2 sock=xx.xx.xx.xx:5061
>>>> Process:: ID=7 PID=13956 Type=udp receiver child=0 sock=xx.xx.xx.xx:443
>>>> Process:: ID=8 PID=13957 Type=udp receiver child=1 sock=xx.xx.xx.xx:443
>>>> Process:: ID=9 PID=13958 Type=udp receiver child=2 sock=xx.xx.xx.xx:443
>>>> Process:: ID=10 PID=13959 Type=udp receiver child=0
>>>> sock=xx.xx.xx.xx:8080
>>>> Process:: ID=11 PID=13960 Type=udp receiver child=1
>>>> sock=xx.xx.xx.xx:8080
>>>> Process:: ID=12 PID=13961 Type=udp receiver child=2
>>>> sock=xx.xx.xx.xx:8080
>>>> Process:: ID=13 PID=13962 Type=udp receiver child=0 sock=xx.xx.xx.xx:123
>>>> Process:: ID=14 PID=13963 Type=udp receiver child=1 sock=xx.xx.xx.xx:123
>>>> Process:: ID=15 PID=13964 Type=udp receiver child=2 sock=xx.xx.xx.xx:123
>>>> Process:: ID=16 PID=13965 Type=slow timer
>>>> Process:: ID=17 PID=13966 Type=timer
>>>> Process:: ID=18 PID=13967 Type=MI FIFO
>>>> Process:: ID=19 PID=13968 Type=ctl handler
>>>> Process:: ID=20 PID=13969 Type=TIMER NH
>>>> Process:: ID=21 PID=13970 Type=tcp receiver child=0
>>>> Process:: ID=22 PID=13971 Type=tcp receiver child=1
>>>> Process:: ID=23 PID=13972 Type=tcp receiver child=2
>>>> Process:: ID=24 PID=13973 Type=tcp main process
>>>>
>>>> I use a few different ports as you can see.
>>>>
>>>>
>>>> All fine with that output at 3 ?
>>>>
>>>> Regards
>>>>
>>>> On Tue, Jan 3, 2012 at 11:29 AM, Daniel-Constantin Mierla
>>>> <miconda at gmail.com> wrote:
>>>>> Hello,
>>>>>
>>>>>
>>>>> On 1/3/12 9:14 AM, Ali Jawad wrote:
>>>>>> Hi Daniel
>>>>>>
>>>>>> SeLinux is disabled, as I have played around with Children and got a
>>>>>> stable startup at CHILDREN=3, at 4 and up I get the problem described
>>>>>> earlier right away, so it seems that MySQL is throttling Kamailio ?
>>>>>>
>>>>>> 1- What is the effect of running 3 children only ?
>>>>> the capacity of handling sip traffic is lowered. You should still be
>>>>> able to
>>>>> handle couple of hundred messages per second if you don't have slow
>>>>> sql/dns
>>>>> operations. For production, it is recommended to have at least 8 to be
>>>>> in
>>>>> safe side.
>>>>>
>>>>>
>>>>>> 2- I know this is not a Kamailio thing, but any idea about what needs
>>>>>> to be changed on MySQL server to accommodate more children ? No of
>>>>>> connections is 500 conns out of which max 120 are utilized
>>>>> So mysql is on a different server, right? Are you sure there is no other
>>>>> switch/router doing such throttling? Can you watch on both kamailio and
>>>>> mysql server to see if the connections are initiated from kamailio and
>>>>> reaching mysql? Maybe you can discover if mysql server get the tcp
>>>>> connection but refuses it -- not sure if mysql server can run in
>>>>> debug/verbose mode to get some log messages with hints about what is the
>>>>> problem.
>>>>>
>>>>> If the tcp connection does not get to mysql server, then it is a
>>>>> throttling
>>>>> in mysql client library or on a device in between the servers.
>>>>>
>>>>> If you do 'kamctl ps' when having children=3, how many kamailio
>>>>> processes
>>>>> are there (extra should be timers, ...)? Each should have one mysql
>>>>> connection, so it helps to see at which number the limit starts.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>> Regards
>>>>>>
>>>>>> On Mon, Jan 2, 2012 at 1:31 PM, Daniel-Constantin Mierla
>>>>>> <miconda at gmail.com> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>>
>>>>>>> On 1/2/12 12:24 PM, Ali Jawad wrote:
>>>>>>>> Hi Daniel
>>>>>>>> Running on a LAN with no firewalls enabled,
>>>>>>>
>>>>>>> limits can be from kernel, selinix, mysql server a.s.o. not only from
>>>>>>> firewalls.
>>>>>>>
>>>>>>>
>>>>>>>> will try you suggestion
>>>>>>>> though first thing tomorrow morning, what are the implications of
>>>>>>>> running children=1 ?
>>>>>>>
>>>>>>> This will create only one worker process (not advisable for production
>>>>>>> in
>>>>>>> long term, just for testing) -- in this way there should be very few
>>>>>>> mysql
>>>>>>> connections opened, revealing if there is a problem or not with the
>>>>>>> number
>>>>>>> of connections and the short period to create them.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Mon, Jan 2, 2012 at 2:03 PM, Daniel-Constantin Mierla
>>>>>>>> <miconda at gmail.com> wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> check also for limits in the system/firewall, maybe there are limits
>>>>>>>>> on
>>>>>>>>> new
>>>>>>>>> connections per second. Try with children=1 and see if it happens
>>>>>>>>> again,
>>>>>>>>> if
>>>>>>>>> not, then it is something that blocks many new connections in short
>>>>>>>>> time.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Daniel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 1/2/12 5:31 AM, Sammy Govind wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Its all upto you then, that's an intermittent issue and information
>>>>>>>>> is
>>>>>>>>> not
>>>>>>>>> complete to trace the issue. Did you see the same Errors every time
>>>>>>>>> it
>>>>>>>>> fails
>>>>>>>>> !?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards
>>>>>>>>> Sammy
>>>>>>>>>
>>>>>>>>> On Sun, Jan 1, 2012 at 10:06 PM, Ali Jawad<ali.jawad at splendor.net>
>>>>>>>>> wrote:
>>>>>>>>>> Hi
>>>>>>>>>> Thanks for your input, but as said this works at times yes and
>>>>>>>>>> times
>>>>>>>>>> now so I am not sure it is a definite DB issue, the servers are in
>>>>>>>>>> the
>>>>>>>>>> same LAN, and there are other modules that use the DB string that
>>>>>>>>>> work
>>>>>>>>>> when launched before this one while this one does not work.
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> On Sun, Jan 1, 2012 at 4:13 PM, Sammy Govind<govoiper at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> forget the whole list of errors and just resolve the DB
>>>>>>>>>>> connectivity
>>>>>>>>>>> between
>>>>>>>>>>> the Kamailio server and DB server first .
>>>>>>>>>>>
>>>>>>>>>>>> 27(24762) ERROR: db_mysql [km_my_con.c:109]: driver error: Can't
>>>>>>>>>>>> connect to MySQL server on 'xxxxxxxxx' (4)
>>>>>>>>>>>
>>>>>>>>>>> try connecting to the remote DB server 'xxxxxxxxx' from linux
>>>>>>>>>>> shell
>>>>>>>>>>> i.e
>>>>>>>>>>>
>>>>>>>>>>> # mysql -u<user> -p<passwd> -h 'xxxxxxxxx'
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Sammy
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Dec 30, 2011 at 3:33 PM, Ali Jawad<ali.jawad at splendor.net>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi All
>>>>>>>>>>>> I have tried the below with Kamailio 3.1.0 and 3.2.1. the problem
>>>>>>>>>>>> is
>>>>>>>>>>>> that Kamailio at times fails to start up for no obvious reasons,
>>>>>>>>>>>> I.e.
>>>>>>>>>>>> in 10 restarts Kamailio might fail to start 3,0 or 5 times. Te
>>>>>>>>>>>> debug
>>>>>>>>>>>> log "level 2 " shows the following :
>>>>>>>>>>>>
>>>>>>>>>>>> 27(24762) ERROR: db_mysql [km_my_con.c:109]: driver error: Can't
>>>>>>>>>>>> connect to MySQL server on 'xxxxxxxxx' (4)
>>>>>>>>>>>> 27(24762) ERROR:<core> [db.c:303]: could not add connection
>>>>>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>> pool27(24762) ERROR: group [group.c:211]: unable to connect to
>>>>>>>>>>>> the
>>>>>>>>>>>> database
>>>>>>>>>>>> 27(24762) ERROR:<core> [sr_module.c:889]: init_mod_child():
>>>>>>>>>>>> Error
>>>>>>>>>>>> while initializing module group
>>>>>>>>>>>> (/usr/local/kamailio/lib/kamailio/modules_k/group.so)
>>>>>>>>>>>> 27(24762) ERROR:<core> [pt.c:481]: ERROR:
>>>>>>>>>>>> fork_tcp_process():
>>>>>>>>>>>> init_child failed for process 27, pid 24762, "tcp receiver
>>>>>>>>>>>> child=1"
>>>>>>>>>>>>
>>>>>>>>>>>> and then
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 30(24765) :<core> [pass_fd.c:293]: ERROR: receive_fd: EOF on
>>>>>>>>>>>> 45
>>>>>>>>>>>> 0(24735) ALERT:<core> [main.c:748]: child process 24762
>>>>>>>>>>>> exited
>>>>>>>>>>>> normally, status=255
>>>>>>>>>>>> 0(24735) INFO:<core> [main.c:766]: INFO: terminating due to
>>>>>>>>>>>> SIGCHLD
>>>>>>>>>>>>
>>>>>>>>>>>> The problem happens on 4 different servers, I did try to compile
>>>>>>>>>>>> more
>>>>>>>>>>>> than once, the servers are 32 Bit CentOS 5.6. The connection to
>>>>>>>>>>>> the
>>>>>>>>>>>> database is monitored and there are no issues with connecting to
>>>>>>>>>>>> the
>>>>>>>>>>>> DB "it works at times and times not for Kamailio so a user/pwd
>>>>>>>>>>>> issue
>>>>>>>>>>>> is not likely to cause this, manual cnnect worked", config file
>>>>>>>>>>>> errors
>>>>>>>>>>>> are not found.
>>>>>>>>>>>>
>>>>>>>>>>>> I did also check max no of connections on MySQL server which is
>>>>>>>>>>>> set
>>>>>>>>>>>> to
>>>>>>>>>>>> 500 on the server and only 100 cons are used. The group module
>>>>>>>>>>>> uses
>>>>>>>>>>>> the same connection string as other modules which is predefined
>>>>>>>>>>>> and
>>>>>>>>>>>> there are other modules that access the db before the group
>>>>>>>>>>>> module
>>>>>>>>>>>> and
>>>>>>>>>>>> those do not have a problem.
>>>>>>>>>>>>
>>>>>>>>>>>> Please advice.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>>>>>>>>>>>> mailing
>>>>>>>>>>>> list
>>>>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>>>>> list
>>>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ali Jawad
>>>>>>>>>> Information Systems Manager
>>>>>>>>>> Splendor Telecom (www.splendor.net)
>>>>>>>>>> Beirut, Lebanon
>>>>>>>>>> Phone: +9611373725/ext 116
>>>>>>>>>> FAX: +9611375554
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>>>> list
>>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>>> list
>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>>>>>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>>> list
>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>
>>>>>>> --
>>>>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>>>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>>
>>>>
>> --
>> Daniel-Constantin Mierla -- http://www.asipto.com
>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>
>
>
--
Daniel-Constantin Mierla -- http://www.asipto.com
http://linkedin.com/in/miconda -- http://twitter.com/miconda
More information about the sr-users
mailing list