[SR-Users] Intermittent start up issue

Ali Jawad ali.jawad at splendor.net
Tue Jan 3 11:58:09 CET 2012 

Same client library different server version working fine ! Any chance
the patch can be released anytime soon ? I will migrate to the newer
MySQL server in the next few mos though, I can wait if needed.
Regards

On Tue, Jan 3, 2012 at 12:44 PM, Daniel-Constantin Mierla
<miconda at gmail.com> wrote:
> Hello,
>
> just another idea to try, not sure if it is convenient -- can you install
> and use different version of mysql server and client library? This should
> reveal whether is an issue specific to mysql version.
>
> Cheers,
> Daniel
>
>
> On 1/3/12 11:35 AM, Daniel-Constantin Mierla wrote:
>>
>> Hello,
>>
>> all it is fine with the output. So it looks like the limit is somewhere
>> above 25 connections.
>>
>> As a try of making a workaround in kamailio side, could be a patch to add
>> a global parameter to introduce some latency before forking a worker, so
>> these processes won't be created all at a time, but one after the other.
>>
>> However, would be good to find the real reason for such behavior. If the
>> tcp connection gets to mysql server, maybe the question should be addressed
>> to mysql forums/mailing lists.
>>
>> Cheers,
>> Daniel
>>
>>
>>
>> On 1/3/12 10:48 AM, Ali Jawad wrote:
>>>
>>> Hi
>>> I dont think a few hundred per second is enough I need to find out
>>> what is wrong and bring up no of children to 8. The MySQL server is at
>>> another host on the same LAN, as for the command kamctl ps see below
>>>
>>> Process::  ID=0 PID=13949 Type=attendant
>>> Process::  ID=1 PID=13950 Type=udp receiver child=0 sock=xx.xx.xx.xx:5060
>>> Process::  ID=2 PID=13951 Type=udp receiver child=1 sock=xx.xx.xx.xx:5060
>>> Process::  ID=3 PID=13952 Type=udp receiver child=2 sock=xx.xx.xx.xx:5060
>>> Process::  ID=4 PID=13953 Type=udp receiver child=0 sock=xx.xx.xx.xx:5061
>>> Process::  ID=5 PID=13954 Type=udp receiver child=1 sock=xx.xx.xx.xx:5061
>>> Process::  ID=6 PID=13955 Type=udp receiver child=2 sock=xx.xx.xx.xx:5061
>>> Process::  ID=7 PID=13956 Type=udp receiver child=0 sock=xx.xx.xx.xx:443
>>> Process::  ID=8 PID=13957 Type=udp receiver child=1 sock=xx.xx.xx.xx:443
>>> Process::  ID=9 PID=13958 Type=udp receiver child=2 sock=xx.xx.xx.xx:443
>>> Process::  ID=10 PID=13959 Type=udp receiver child=0
>>> sock=xx.xx.xx.xx:8080
>>> Process::  ID=11 PID=13960 Type=udp receiver child=1
>>> sock=xx.xx.xx.xx:8080
>>> Process::  ID=12 PID=13961 Type=udp receiver child=2
>>> sock=xx.xx.xx.xx:8080
>>> Process::  ID=13 PID=13962 Type=udp receiver child=0 sock=xx.xx.xx.xx:123
>>> Process::  ID=14 PID=13963 Type=udp receiver child=1 sock=xx.xx.xx.xx:123
>>> Process::  ID=15 PID=13964 Type=udp receiver child=2 sock=xx.xx.xx.xx:123
>>> Process::  ID=16 PID=13965 Type=slow timer
>>> Process::  ID=17 PID=13966 Type=timer
>>> Process::  ID=18 PID=13967 Type=MI FIFO
>>> Process::  ID=19 PID=13968 Type=ctl handler
>>> Process::  ID=20 PID=13969 Type=TIMER NH
>>> Process::  ID=21 PID=13970 Type=tcp receiver child=0
>>> Process::  ID=22 PID=13971 Type=tcp receiver child=1
>>> Process::  ID=23 PID=13972 Type=tcp receiver child=2
>>> Process::  ID=24 PID=13973 Type=tcp main process
>>>
>>> I use a few different ports as you can see.
>>>
>>>
>>> All fine with that output at 3 ?
>>>
>>> Regards
>>>
>>> On Tue, Jan 3, 2012 at 11:29 AM, Daniel-Constantin Mierla
>>> <miconda at gmail.com>  wrote:
>>>>
>>>> Hello,
>>>>
>>>>
>>>> On 1/3/12 9:14 AM, Ali Jawad wrote:
>>>>>
>>>>> Hi Daniel
>>>>>
>>>>> SeLinux is disabled, as I have played around with Children and got a
>>>>> stable startup at CHILDREN=3, at 4 and up I get the problem described
>>>>> earlier right away, so it seems that MySQL is throttling Kamailio ?
>>>>>
>>>>> 1- What is the effect of running 3 children only ?
>>>>
>>>> the capacity of handling sip traffic is lowered. You should still be
>>>> able to
>>>> handle couple of hundred messages per second if you don't have slow
>>>> sql/dns
>>>> operations. For production, it is recommended to have at least 8 to be
>>>> in
>>>> safe side.
>>>>
>>>>
>>>>> 2- I know this is not a Kamailio thing, but any idea about what needs
>>>>> to be changed on MySQL server to accommodate more children ? No of
>>>>> connections is 500 conns out of which max 120 are utilized
>>>>
>>>> So mysql is on a different server, right? Are you sure there is no other
>>>> switch/router doing such throttling? Can you watch on both kamailio and
>>>> mysql server to see if the connections are initiated from kamailio and
>>>> reaching mysql? Maybe you can discover if mysql server get the tcp
>>>> connection but refuses it -- not sure if mysql server can run in
>>>> debug/verbose mode to get some log messages with hints about what is the
>>>> problem.
>>>>
>>>> If the tcp connection does not get to mysql server, then it is a
>>>> throttling
>>>> in mysql client library or on a device in between the servers.
>>>>
>>>> If you do 'kamctl ps' when having children=3, how many kamailio
>>>> processes
>>>> are there (extra should be timers, ...)? Each should have one mysql
>>>> connection, so it helps to see at which number the limit starts.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>> Regards
>>>>>
>>>>> On Mon, Jan 2, 2012 at 1:31 PM, Daniel-Constantin Mierla
>>>>> <miconda at gmail.com>    wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>
>>>>>> On 1/2/12 12:24 PM, Ali Jawad wrote:
>>>>>>>
>>>>>>> Hi Daniel
>>>>>>> Running on a LAN with no firewalls enabled,
>>>>>>
>>>>>>
>>>>>> limits can be from kernel, selinix, mysql server  a.s.o. not only from
>>>>>> firewalls.
>>>>>>
>>>>>>
>>>>>>>  will try you suggestion
>>>>>>> though first thing tomorrow morning, what are the implications of
>>>>>>> running children=1 ?
>>>>>>
>>>>>>
>>>>>> This will create only one worker process (not advisable for production
>>>>>> in
>>>>>> long term, just for testing) -- in this way there should be very few
>>>>>> mysql
>>>>>> connections opened, revealing if there is a problem or not with the
>>>>>> number
>>>>>> of connections and the short period to create them.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Mon, Jan 2, 2012 at 2:03 PM, Daniel-Constantin Mierla
>>>>>>> <miconda at gmail.com>      wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> check also for limits in the system/firewall, maybe there are limits
>>>>>>>> on
>>>>>>>> new
>>>>>>>> connections per second. Try with children=1 and see if it happens
>>>>>>>> again,
>>>>>>>> if
>>>>>>>> not, then it is something that blocks many new connections in short
>>>>>>>> time.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>>
>>>>>>>> On 1/2/12 5:31 AM, Sammy Govind wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> Its all upto you then, that's an intermittent issue and information
>>>>>>>> is
>>>>>>>> not
>>>>>>>> complete to trace the issue. Did you see the same Errors every time
>>>>>>>> it
>>>>>>>> fails
>>>>>>>> !?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards
>>>>>>>> Sammy
>>>>>>>>
>>>>>>>> On Sun, Jan 1, 2012 at 10:06 PM, Ali Jawad<ali.jawad at splendor.net>
>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>> Thanks for your input, but as said this works at times yes and
>>>>>>>>> times
>>>>>>>>> now so I am not sure it is a definite DB issue, the servers are in
>>>>>>>>> the
>>>>>>>>> same LAN, and there are other modules that use the DB string that
>>>>>>>>> work
>>>>>>>>> when launched before this one while this one does not work.
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> On Sun, Jan 1, 2012 at 4:13 PM, Sammy Govind<govoiper at gmail.com>
>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> forget the whole list of errors and just resolve the DB
>>>>>>>>>> connectivity
>>>>>>>>>> between
>>>>>>>>>> the Kamailio server and DB server first .
>>>>>>>>>>
>>>>>>>>>>> 27(24762) ERROR: db_mysql [km_my_con.c:109]: driver error: Can't
>>>>>>>>>>> connect to MySQL server on 'xxxxxxxxx' (4)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> try connecting to the remote DB server 'xxxxxxxxx' from linux
>>>>>>>>>> shell
>>>>>>>>>> i.e
>>>>>>>>>>
>>>>>>>>>> # mysql -u<user>      -p<passwd>      -h 'xxxxxxxxx'
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Sammy
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 30, 2011 at 3:33 PM, Ali Jawad<ali.jawad at splendor.net>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All
>>>>>>>>>>> I have tried the below with Kamailio 3.1.0 and 3.2.1. the problem
>>>>>>>>>>> is
>>>>>>>>>>> that Kamailio at times fails to start up for no obvious reasons,
>>>>>>>>>>> I.e.
>>>>>>>>>>> in 10 restarts Kamailio might fail to start 3,0 or 5 times. Te
>>>>>>>>>>> debug
>>>>>>>>>>> log "level 2 " shows the following :
>>>>>>>>>>>
>>>>>>>>>>> 27(24762) ERROR: db_mysql [km_my_con.c:109]: driver error: Can't
>>>>>>>>>>> connect to MySQL server on 'xxxxxxxxx' (4)
>>>>>>>>>>> 27(24762) ERROR:<core>      [db.c:303]: could not add connection
>>>>>>>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>> pool27(24762) ERROR: group [group.c:211]: unable to connect to
>>>>>>>>>>> the
>>>>>>>>>>> database
>>>>>>>>>>> 27(24762) ERROR:<core>      [sr_module.c:889]: init_mod_child():
>>>>>>>>>>> Error
>>>>>>>>>>> while initializing module group
>>>>>>>>>>> (/usr/local/kamailio/lib/kamailio/modules_k/group.so)
>>>>>>>>>>> 27(24762) ERROR:<core>      [pt.c:481]: ERROR:
>>>>>>>>>>> fork_tcp_process():
>>>>>>>>>>> init_child failed for process 27, pid 24762, "tcp receiver
>>>>>>>>>>> child=1"
>>>>>>>>>>>
>>>>>>>>>>> and then
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 30(24765) :<core>      [pass_fd.c:293]: ERROR: receive_fd: EOF on
>>>>>>>>>>> 45
>>>>>>>>>>>  0(24735) ALERT:<core>      [main.c:748]: child process 24762
>>>>>>>>>>> exited
>>>>>>>>>>> normally, status=255
>>>>>>>>>>>  0(24735) INFO:<core>      [main.c:766]: INFO: terminating due to
>>>>>>>>>>> SIGCHLD
>>>>>>>>>>>
>>>>>>>>>>> The problem happens on 4 different servers, I did try to compile
>>>>>>>>>>> more
>>>>>>>>>>> than once, the servers are 32 Bit CentOS 5.6. The connection to
>>>>>>>>>>> the
>>>>>>>>>>> database is monitored and there are no issues with connecting to
>>>>>>>>>>> the
>>>>>>>>>>> DB "it works at times and times not for Kamailio so a user/pwd
>>>>>>>>>>> issue
>>>>>>>>>>> is not likely to cause this, manual cnnect worked", config file
>>>>>>>>>>> errors
>>>>>>>>>>> are not found.
>>>>>>>>>>>
>>>>>>>>>>> I did also check max no of connections on MySQL server which is
>>>>>>>>>>> set
>>>>>>>>>>> to
>>>>>>>>>>> 500 on the server and only 100 cons are used. The group module
>>>>>>>>>>> uses
>>>>>>>>>>> the same connection string as other modules  which is predefined
>>>>>>>>>>>  and
>>>>>>>>>>> there are other modules that access the db before the group
>>>>>>>>>>> module
>>>>>>>>>>> and
>>>>>>>>>>> those do not have a problem.
>>>>>>>>>>>
>>>>>>>>>>> Please advice.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>>>>>>>>>>> mailing
>>>>>>>>>>> list
>>>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>>>> list
>>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ali Jawad
>>>>>>>>> Information Systems Manager
>>>>>>>>> Splendor Telecom (www.splendor.net)
>>>>>>>>> Beirut, Lebanon
>>>>>>>>> Phone: +9611373725/ext 116
>>>>>>>>> FAX: +9611375554
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>>> list
>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>> list
>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>>>>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>> list
>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>>>
>>>>>
>>>> --
>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>
>>>
>>>
>>
>
> --
> Daniel-Constantin Mierla -- http://www.asipto.com
> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>



-- 
Ali Jawad
Information Systems Manager
Splendor Telecom (www.splendor.net)
Beirut, Lebanon
Phone: +9611373725/ext 116
FAX: +9611375554

More information about the sr-users mailing list