The core is generated by the shutdown procedure, so not a runtime event, as side effect of running out of memory when building the transaction -- maybe some of the fields might be inconsistent. It needs to be fixed, but it is a bit hard to track it without proper core file and debugging symbols. What are the values for shared memory and private memory (the -m and -M command line parameters)? Did you killed kamailio or it was stopping itself? If the later, can you be sure that each process generates a core file, because shutdown procedure can overwrite the core file that is generated at startup. You should have at least two core files in such case. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285327005

I don't know about *BSD, but in Linux there is an option to enable corefiles per process (per pid). It is something like: ``` echo "1" > /proc/sys/kernel/core_uses_pid ``` Can you see if there is something similar for *BSD, enable it and test again? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285333557

I also know little of BSD, but this is what was required on Linux to get a dump into /tmp: ```phil@ua-proxy-01:/etc/sysctl.d$ cat 61-core-pattern.conf fs.suid_dumpable=2 kernel.core_uses_pid=1 kernel.core_pattern=/tmp/core.%e.%p.%h.%t``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285341692

What is the gdb backtrace for the new core? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285342788

Proper gdb backtrace with debugging symbols is needed here. Do you compile from sources directly, or do you do it via ports/something else? Can you try with gcc instead if clang? Debugging symbols should be compiled by default, at least with gcc, as I haven't contributed the clang part, I can't say it for sure for it. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285352644

I've built kamailio with gcc: Now I can't reproduce core dump. I'll try to dig in it and post results later. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285384805

The crash happens due to `signlal 10` -- can you get the output of `kill -l` to see what is that on your system? It might be specific to the OS, like SIGUSR1 or SIGBUS... -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285430033

In the bt file, is `list` output taken for the same corefile in the frame 0? So 10 is `SIGBUS`, which typically appears due to unaligned memory access, but can be other reasons: * https://en.wikipedia.org/wiki/Bus_error Can you check on all core files that you got after enabling it per pid to see if they crash at the same point in code (is fame 0 line the same)? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285482576

Yes, it is. But now it seems that they crash at the same point in code. Here is bt (only 4 core, not 5): http://tmp.lehis.ru/tmp/kam_bt_27333.txt http://tmp.lehis.ru/tmp/kam_bt_27334.txt http://tmp.lehis.ru/tmp/kam_bt_27336.txt http://tmp.lehis.ru/tmp/kam_bt_27337.txt -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285681015

Done: ``` ps ax | grep kama 27526 - S 0:00,05 /usr/local/sbin/kamailio -x qm 27527 - S 0:00,19 /usr/local/sbin/kamailio -x qm ... ``` And now I've got five core files without load... -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285695878

Sorry, I've updated my previous post. I can create virtualbox image for you, Daniel, and provide ssh access to it, if it can helps you. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-285702378

Ok, I'll write when it will be ready. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-286130837

I connected to the box and started kamailio with `-x qm` -- it works, you said that it crashes immediately: ``` [VM-TEST@sip7:/home/daniel] ps auxw | grep kamai root 19951 0.0 1.8 222420 37196 - S 4:24PM 0:00.05 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19952 0.0 1.8 222420 37200 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19953 0.0 1.8 222420 37188 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19954 0.0 1.8 222420 37188 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19955 0.0 1.8 222420 37188 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19956 0.0 1.8 222420 37188 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19957 0.0 1.8 222420 37188 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19958 0.0 1.8 222420 37188 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19959 0.0 1.8 222420 37188 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19960 0.0 1.8 222420 37236 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19961 0.0 1.8 222420 37208 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19962 0.0 1.8 222420 37204 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd root 19963 0.0 1.8 222420 37204 - S 4:24PM 0:00.00 /usr/local/sbin/kamailio -x qm -a no -E -e -ddd ``` Is it something different in the way you start it? Also, can you reproduce the crash with load testing so I can get some core files? Let me know where they are written so I can troubleshoot. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-286425365

I've got core dump: it locates at /mnt/hdd3/coredumps/kamailio.20041.core -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-286429354

I've compiled kamailio using gmake, without any special options. I'll try to detect where was included using of PTHREAD_MUTEX -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-286667596

I can confirm that kamailio works perfectly without PTHREAD_MUTEX: ``` Version: kamailio 4.4.5 (x86_64/freebsd) f98162 Default config: /usr/local/etc/kamailio/kamailio.cfg Default paths to modules: /usr/local/lib/kamailio/modules Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES MAX_RECV_BUFFER_SIZE=262144 MAX_LISTEN=16 MAX_URI_SIZE=1024 BUF_SIZE=65535 DEFAULT PKG_SIZE=8MB DEFAULT SHM_SIZE=64MB ADAPTIVE_WAIT_LOOPS=1024 TCP poll methods: poll, select, kqueue Source code revision ID: f98162 Compiled with: clang 3.4 Compiled on: 12:45:18 Mar 15 2017 ``` I've found diff between build from src and using ports framework - it's target arch: when it builds from src **make cfg** shows: `target architecture <x86_64>, host architecture <x86_64>` when it builds from ports it shows: `target architecture <amd64>, host architecture <x86_64>` In last case Makefile.defs did know nothing about arch amd64 and that's why use_fast_lock hasn't set to yes and that's why PTHREAD_MUTEX was used. But it is another story, where I and **opsec** will try to do something this it. Daniel, thanks for your help and patience! -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-286705117

I see that PTHREAD_MUTEX are set as default instead of POSIX, if fast locks are not used. Maybe someone itried to see when they work very long time ago and eventually it was left like this (I tried to track the change quickly, but couldn't find it easy, too many changes in the file). Pthread mutex doesn't work for processes in most of the cases. I am closing this one and open a new item to review the default locking option. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1021#issuecomment-286710430