Thanks Daniel.  We did perform a restart of the AWS instance to apply a kernel patch and when the process started up during bootup, that's when we saw the messages.  However, manually starting it seems to work.

We will work on upgrading to 5.3.x as you suggested.

On Tue, Sep 1, 2020 at 10:43 AM Daniel-Constantin Mierla <miconda@gmail.com> wrote:

Hello,

first it is really recommended you run at least latest 5.1.x version, the 5.1.2 is old in 5.1.x release series and many bugs may have been fixed. Even more, 5.1.x is no longer maintained, so I would strongly recommend to upgrade to latest in series 5.3.x or 5.4.x.

Then, searching on google on Bad RIP value returns mainly discussions about Kernel panic. According to this article:

  * https://www.opensourceforu.com/2011/01/understanding-a-kernel-oops/

RIP is the CPU register containing the address of the instruction that is getting executed. Without a clear corefile with backtrace pointing to kamailio, it is not easy to say if Kamailio has anything to do with or there is some linux kernel issue on your system. Try searching on google, there should be some hints to try to troubleshoot by ip and sp values, maybe you can get further details.

Cheers,
Daniel

On 01.09.20 15:12, Andrew Chen wrote:
Hey guys,

Our monitoring system caught kamailio coring.  I did a dmesg and gave me this:

[Tue Sep  1 09:12:34 2020] kamailio[1375]: segfault at 1 ip 0000000000000001 sp 00007fff4203f098 error 14 in kamailio[563969b5a000+4a4000]
[Tue Sep  1 09:12:34 2020] Code: Bad RIP value.
[Tue Sep  1 09:12:34 2020] kamailio[1363]: segfault at 0 ip 0000000000000000 sp 00007fff4203f368 error 14 in kamailio[563969b5a000+4a4000]
[Tue Sep  1 09:12:34 2020] Code: Bad RIP value.
[Tue Sep  1 09:12:34 2020] kamailio[1377]: segfault at 1 ip 0000000000000001 sp 00007fff4203e308 error 14 in kamailio[563969b5a000+4a4000]
[Tue Sep  1 09:12:34 2020] Code: Bad RIP value.

After doing gdb bt, didn't tell me much except it can't access some parts of the memory:

warning: Unexpected size of section `.reg-xstate/1624' in core file.
Cannot access memory at address 0x7fb23433e148
Cannot access memory at address 0x7fb23433e140
Failed to read a valid object file image from memory.
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section `.reg-xstate/1624' in core file.
#0  0x0000000000000001 in ?? ()
(gdb) bt full
#0  0x0000000000000001 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fff4203e1f8
(gdb) quit

Any idea on how to troubleshoot this further?

Thanks.

--
Andy Chen
Sr. Telephony Lead Engineer



*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla


--
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)


*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*