core dump error when running under docker

List overview All Threads
Download

newer

older

Kamailio behind HAProxy

What is the lifecycle of Branch...

Carlos Vicente

28 Nov 2020 28 Nov '20

3:34 p.m.

Greetings,

I'm having a "fun" time trying to run Kamailio inside a Docker container. It keeps failing with:

0(238) DEBUG: <core> [core/mem/pkg.c:108]: pkg_print_manager(): pkg - using memory manager: q_malloc 0(238) DEBUG: <core> [core/mem/shm.c:300]: shm_print_manager(): shm - using memory manager: q_malloc Illegal instruction (core dumped)

Unfortunately the core dump is nowhere to be found. I have tried to follow the instructions at:

https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile

but that hasn't helped.

The environment is as follows:

* Host hardware: Raspberry Pi 4 (4G memory) * Host OS: Ubuntu 20.04.1 LTS * Docker version 19.03.13 * Docker base image: arm32v7/ubuntu

root@2ef8a21534fa:/# kamailio -v version: kamailio 5.3.2 (arm/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 9.2.1

The config file passes the syntax check:

kamailio -c -f /etc/kamailio/kamailio.cfg

and, the most interesting part, if I run Kamailio on the host machine, _with the exact same config file_, it runs fine.

I'd appreciate any clues you may have. Thanks.

Attachments:

attachment.html (text/html — 2.1 KB)

Show replies by date

Sergey Safarov

28 Nov 28 Nov

4:36 p.m.

You can start kamailio under GDB debugger. When issue reproduced, then you can get stack trace and variables values

сб, 28 нояб. 2020 г., 18:35 Carlos Vicente cvicente.lists@gmail.com:

...

Greetings,

I'm having a "fun" time trying to run Kamailio inside a Docker container. It keeps failing with:

0(238) DEBUG: <core> [core/mem/pkg.c:108]: pkg_print_manager(): pkg - using memory manager: q_malloc 0(238) DEBUG: <core> [core/mem/shm.c:300]: shm_print_manager(): shm - using memory manager: q_malloc Illegal instruction (core dumped)

Unfortunately the core dump is nowhere to be found. I have tried to follow the instructions at:
https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile
but that hasn't helped.

The environment is as follows:

Host hardware: Raspberry Pi 4 (4G memory)

Host OS: Ubuntu 20.04.1 LTS

Docker version 19.03.13

Docker base image: arm32v7/ubuntu

root@2ef8a21534fa:/# kamailio -v version: kamailio 5.3.2 (arm/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 9.2.1

The config file passes the syntax check:
kamailio -c -f /etc/kamailio/kamailio.cfg
and, the most interesting part, if I run Kamailio on the host machine, _with the exact same config file_, it runs fine.

I'd appreciate any clues you may have. Thanks.

cv

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Carlos Vicente

29 Nov 29 Nov

1:13 p.m.

Thank you for your response.

I'm trying to run it under gdb but it's not doing anything:

``` (gdb) exec-file /usr/sbin/kamailio (gdb) set args -f /etc/kamailio/kamailio.cfg -D -d -E (gdb) run Starting program: /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -D -d -E warning: Error disabling address space randomization: Success ```

It just stops there. Am I missing something?

On Sat, Nov 28, 2020 at 11:37 AM Sergey Safarov s.safarov@gmail.com wrote:

...

You can start kamailio under GDB debugger. When issue reproduced, then you can get stack trace and variables values

сб, 28 нояб. 2020 г., 18:35 Carlos Vicente cvicente.lists@gmail.com:

...
Greetings,

I'm having a "fun" time trying to run Kamailio inside a Docker container. It keeps failing with:

0(238) DEBUG: <core> [core/mem/pkg.c:108]: pkg_print_manager(): pkg - using memory manager: q_malloc 0(238) DEBUG: <core> [core/mem/shm.c:300]: shm_print_manager(): shm - using memory manager: q_malloc Illegal instruction (core dumped)

Unfortunately the core dump is nowhere to be found. I have tried to follow the instructions at:
https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile
but that hasn't helped.

The environment is as follows:

Host hardware: Raspberry Pi 4 (4G memory)

Host OS: Ubuntu 20.04.1 LTS

Docker version 19.03.13

Docker base image: arm32v7/ubuntu

root@2ef8a21534fa:/# kamailio -v version: kamailio 5.3.2 (arm/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 9.2.1

The config file passes the syntax check:
kamailio -c -f /etc/kamailio/kamailio.cfg
and, the most interesting part, if I run Kamailio on the host machine, _with the exact same config file_, it runs fine.

I'd appreciate any clues you may have. Thanks.

cv

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Sergey Safarov

1:23 p.m.

you can change set args -f /etc/kamailio/kamailio.cfg -D -d -E

To set args -f /etc/kamailio/kamailio.cfg -DD -d -E

On Sun, Nov 29, 2020 at 4:14 PM Carlos Vicente cvicente.lists@gmail.com wrote:

...

Thank you for your response.

I'm trying to run it under gdb but it's not doing anything:
(gdb) exec-file /usr/sbin/kamailio
(gdb) set args -f /etc/kamailio/kamailio.cfg -D -d -E
(gdb) run
Starting program: /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -D -d -E
warning: Error disabling address space randomization: Success
It just stops there. Am I missing something?

cv

On Sat, Nov 28, 2020 at 11:37 AM Sergey Safarov s.safarov@gmail.com wrote:

...
You can start kamailio under GDB debugger. When issue reproduced, then you can get stack trace and variables values

сб, 28 нояб. 2020 г., 18:35 Carlos Vicente cvicente.lists@gmail.com:

...
Greetings,

I'm having a "fun" time trying to run Kamailio inside a Docker container. It keeps failing with:

0(238) DEBUG: <core> [core/mem/pkg.c:108]: pkg_print_manager(): pkg - using memory manager: q_malloc 0(238) DEBUG: <core> [core/mem/shm.c:300]: shm_print_manager(): shm - using memory manager: q_malloc Illegal instruction (core dumped)

Unfortunately the core dump is nowhere to be found. I have tried to follow the instructions at:
https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile
but that hasn't helped.

The environment is as follows:

Host hardware: Raspberry Pi 4 (4G memory)

Host OS: Ubuntu 20.04.1 LTS

Docker version 19.03.13

Docker base image: arm32v7/ubuntu

root@2ef8a21534fa:/# kamailio -v version: kamailio 5.3.2 (arm/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 9.2.1

The config file passes the syntax check:
kamailio -c -f /etc/kamailio/kamailio.cfg
and, the most interesting part, if I run Kamailio on the host machine, _with the exact same config file_, it runs fine.

I'd appreciate any clues you may have. Thanks.

cv

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Daniel-Constantin Mierla

30 Nov 30 Nov

9:27 a.m.

Maybe the hints on stackoverflow are also useful:

* https://stackoverflow.com/questions/28335614/how-to-generate-core-file-in-do...

Cheers, Daniel

On 29.11.20 14:23, Sergey Safarov wrote:

...

you can change set args -f /etc/kamailio/kamailio.cfg -D -d -E

To set args -f /etc/kamailio/kamailio.cfg -DD -d -E

On Sun, Nov 29, 2020 at 4:14 PM Carlos Vicente <cvicente.lists@gmail.com mailto:cvicente.lists@gmail.com> wrote:

Thank you for your response.

I'm trying to run it under gdb but it's not doing anything:

```
(gdb) exec-file /usr/sbin/kamailio
(gdb) set args -f /etc/kamailio/kamailio.cfg -D -d -E
(gdb) run
Starting program: /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-D -d -E
warning: Error disabling address space randomization: Success
```

It just stops there. Am I missing something?

cv

On Sat, Nov 28, 2020 at 11:37 AM Sergey Safarov
<s.safarov@gmail.com <mailto:s.safarov@gmail.com>> wrote:

    You can start kamailio under GDB debugger.
    When issue reproduced, then you can get stack trace and
    variables values


    сб, 28 нояб. 2020 г., 18:35 Carlos Vicente
    <cvicente.lists@gmail.com <mailto:cvicente.lists@gmail.com>>:

        Greetings,

        I'm having a "fun" time trying to run Kamailio inside a
        Docker container. It keeps failing with:

         0(238) DEBUG: <core> [core/mem/pkg.c:108]:
        pkg_print_manager(): pkg - using memory manager: q_malloc
         0(238) DEBUG: <core> [core/mem/shm.c:300]:
        shm_print_manager(): shm - using memory manager: q_malloc
        Illegal instruction (core dumped)

        Unfortunately the core dump is nowhere to be found. I have
        tried to follow the instructions at:

           
        https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile
        <https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile>

        but that hasn't helped.

        The environment is as follows:

        * Host hardware: Raspberry Pi 4 (4G memory)
        * Host OS: Ubuntu 20.04.1 LTS
        * Docker version 19.03.13
        * Docker base image: arm32v7/ubuntu

        root@2ef8a21534fa:/# kamailio -v
        version: kamailio 5.3.2 (arm/linux)
        flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
        USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
        SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
        DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP,
        USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
        USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
        ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144,
        MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
        poll method support: poll, epoll_lt, epoll_et, sigio_rt,
        select.
        id: unknown
        compiled with gcc 9.2.1

        The config file passes the syntax check:

            kamailio -c -f /etc/kamailio/kamailio.cfg

        and, the most interesting part, if I run Kamailio on the
        host machine, _with the exact same config file_, it runs fine.

        I'd appreciate any clues you may have. Thanks.

        cv


        _______________________________________________
        Kamailio (SER) - Users Mailing List
        sr-users@lists.kamailio.org
        <mailto:sr-users@lists.kamailio.org>
        https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
        <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

    _______________________________________________
    Kamailio (SER) - Users Mailing List
    sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
    https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
    <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
<https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Carlos Vicente

2:22 p.m.

Daniel,

Thank you. That finally helped getting the core file written. Here is the gdb output:

...

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-linux-gnueabihf". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio... (No debugging symbols found in /usr/sbin/kamailio) [New LWP 207] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -DD -E -ddd'. Program terminated with signal SIGILL, Illegal instruction. #0 0x007d9cb8 in qm_shm_malloc () (gdb) bt #0 0x007d9cb8 in qm_shm_malloc () #1 0x0061a58c in init_atomic_ops () #2 0x00562508 in main () ```

What can be done about that?

On Mon, Nov 30, 2020 at 4:29 AM Daniel-Constantin Mierla miconda@gmail.com wrote:

...

Maybe the hints on stackoverflow are also useful:

https://stackoverflow.com/questions/28335614/how-to-generate-core-file-in-do...

Cheers, Daniel On 29.11.20 14:23, Sergey Safarov wrote:

you can change set args -f /etc/kamailio/kamailio.cfg -D -d -E

To set args -f /etc/kamailio/kamailio.cfg -DD -d -E

On Sun, Nov 29, 2020 at 4:14 PM Carlos Vicente cvicente.lists@gmail.com wrote:

...
Thank you for your response.

I'm trying to run it under gdb but it's not doing anything:
(gdb) exec-file /usr/sbin/kamailio
(gdb) set args -f /etc/kamailio/kamailio.cfg -D -d -E
(gdb) run
Starting program: /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -D -d
-E
warning: Error disabling address space randomization: Success
It just stops there. Am I missing something?

cv

On Sat, Nov 28, 2020 at 11:37 AM Sergey Safarov s.safarov@gmail.com wrote:

...
You can start kamailio under GDB debugger. When issue reproduced, then you can get stack trace and variables values

сб, 28 нояб. 2020 г., 18:35 Carlos Vicente cvicente.lists@gmail.com:

...
Greetings,

I'm having a "fun" time trying to run Kamailio inside a Docker container. It keeps failing with:

0(238) DEBUG: <core> [core/mem/pkg.c:108]: pkg_print_manager(): pkg - using memory manager: q_malloc 0(238) DEBUG: <core> [core/mem/shm.c:300]: shm_print_manager(): shm - using memory manager: q_malloc Illegal instruction (core dumped)

Unfortunately the core dump is nowhere to be found. I have tried to follow the instructions at:

https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile

but that hasn't helped.

The environment is as follows:

Host hardware: Raspberry Pi 4 (4G memory)

Host OS: Ubuntu 20.04.1 LTS

Docker version 19.03.13

Docker base image: arm32v7/ubuntu

root@2ef8a21534fa:/# kamailio -v version: kamailio 5.3.2 (arm/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 9.2.1

The config file passes the syntax check:
kamailio -c -f /etc/kamailio/kamailio.cfg
and, the most interesting part, if I run Kamailio on the host machine, _with the exact same config file_, it runs fine.

I'd appreciate any clues you may have. Thanks.

cv

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Sergey Safarov

2:32 p.m.

Are you compiled Kamailio from sources?

If compiled from sources or installed "debug" files then pelase show output of "bt full" command.

On Mon, Nov 30, 2020 at 5:23 PM Carlos Vicente cvicente.lists@gmail.com wrote:

...

Daniel,

Thank you. That finally helped getting the core file written. Here is the gdb output:
root@6e5c06e979cf:/# gdb /usr/sbin/kamailio /cores/core.kamailio.207
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kamailio...
(No debugging symbols found in /usr/sbin/kamailio)
[New LWP 207]
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-DD -E -ddd'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x007d9cb8 in qm_shm_malloc ()
(gdb) bt
#0  0x007d9cb8 in qm_shm_malloc ()
#1  0x0061a58c in init_atomic_ops ()
#2  0x00562508 in main ()
What can be done about that?

cv

On Mon, Nov 30, 2020 at 4:29 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Maybe the hints on stackoverflow are also useful:

https://stackoverflow.com/questions/28335614/how-to-generate-core-file-in-do...

Cheers, Daniel On 29.11.20 14:23, Sergey Safarov wrote:

you can change set args -f /etc/kamailio/kamailio.cfg -D -d -E

To set args -f /etc/kamailio/kamailio.cfg -DD -d -E

On Sun, Nov 29, 2020 at 4:14 PM Carlos Vicente cvicente.lists@gmail.com wrote:

...
Thank you for your response.

I'm trying to run it under gdb but it's not doing anything:
(gdb) exec-file /usr/sbin/kamailio
(gdb) set args -f /etc/kamailio/kamailio.cfg -D -d -E
(gdb) run
Starting program: /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -D -d
-E
warning: Error disabling address space randomization: Success
It just stops there. Am I missing something?

cv

On Sat, Nov 28, 2020 at 11:37 AM Sergey Safarov s.safarov@gmail.com wrote:

...
You can start kamailio under GDB debugger. When issue reproduced, then you can get stack trace and variables values

сб, 28 нояб. 2020 г., 18:35 Carlos Vicente cvicente.lists@gmail.com:

...
Greetings,

I'm having a "fun" time trying to run Kamailio inside a Docker container. It keeps failing with:

0(238) DEBUG: <core> [core/mem/pkg.c:108]: pkg_print_manager(): pkg - using memory manager: q_malloc 0(238) DEBUG: <core> [core/mem/shm.c:300]: shm_print_manager(): shm - using memory manager: q_malloc Illegal instruction (core dumped)

Unfortunately the core dump is nowhere to be found. I have tried to follow the instructions at:

https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile

but that hasn't helped.

The environment is as follows:

Host hardware: Raspberry Pi 4 (4G memory)

Host OS: Ubuntu 20.04.1 LTS

Docker version 19.03.13

Docker base image: arm32v7/ubuntu

root@2ef8a21534fa:/# kamailio -v version: kamailio 5.3.2 (arm/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 9.2.1

The config file passes the syntax check:
kamailio -c -f /etc/kamailio/kamailio.cfg
and, the most interesting part, if I run Kamailio on the host machine, _with the exact same config file_, it runs fine.

I'd appreciate any clues you may have. Thanks.

cv

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Carlos Vicente

3:38 p.m.

I couldn't find any "-debug" packages, so I compiled the latest version:

``` root@348cbcfcb96c:~# /usr/local/sbin/kamailio -v version: kamailio 5.5.0-dev3 (arm/linux) d43163 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: d43163 compiled on 15:17:51 Nov 30 2020 with gcc 9.3.0 ```

But still not symbols:

...

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/kamailio... (No debugging symbols found in /usr/local/sbin/kamailio) [New LWP 19] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1". Core was generated by `/usr/local/sbin/kamailio -f /etc/kamailio/kamailio.cfg -DD -E -d'. Program terminated with signal SIGILL, Illegal instruction. #0 0x00b06320 in tsl () (gdb) bt full #0 0x00b06320 in tsl () No symbol table info available. #1 0x00b06380 in get_lock () No symbol table info available. #2 0x00b1979c in qm_shm_malloc () No symbol table info available. #3 0x009504dc in ksr_shutdown_phase_init () No symbol table info available. #4 0x0078e3f4 in main () No symbol table info available. (gdb) quit ```

TIA

On Mon, Nov 30, 2020 at 9:33 AM Sergey Safarov s.safarov@gmail.com wrote:

...

Are you compiled Kamailio from sources?

If compiled from sources or installed "debug" files then pelase show output of "bt full" command.

On Mon, Nov 30, 2020 at 5:23 PM Carlos Vicente cvicente.lists@gmail.com wrote:

...
Daniel,

Thank you. That finally helped getting the core file written. Here is the gdb output:
root@6e5c06e979cf:/# gdb /usr/sbin/kamailio /cores/core.kamailio.207
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kamailio...
(No debugging symbols found in /usr/sbin/kamailio)
[New LWP 207]
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-DD -E -ddd'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x007d9cb8 in qm_shm_malloc ()
(gdb) bt
#0  0x007d9cb8 in qm_shm_malloc ()
#1  0x0061a58c in init_atomic_ops ()
#2  0x00562508 in main ()
What can be done about that?

cv

On Mon, Nov 30, 2020 at 4:29 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Maybe the hints on stackoverflow are also useful:

https://stackoverflow.com/questions/28335614/how-to-generate-core-file-in-do...

Cheers, Daniel On 29.11.20 14:23, Sergey Safarov wrote:

you can change set args -f /etc/kamailio/kamailio.cfg -D -d -E

To set args -f /etc/kamailio/kamailio.cfg -DD -d -E

On Sun, Nov 29, 2020 at 4:14 PM Carlos Vicente cvicente.lists@gmail.com wrote:

...
Thank you for your response.

I'm trying to run it under gdb but it's not doing anything:
(gdb) exec-file /usr/sbin/kamailio
(gdb) set args -f /etc/kamailio/kamailio.cfg -D -d -E
(gdb) run
Starting program: /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -D
-d -E
warning: Error disabling address space randomization: Success
It just stops there. Am I missing something?

cv

On Sat, Nov 28, 2020 at 11:37 AM Sergey Safarov s.safarov@gmail.com wrote:

...
You can start kamailio under GDB debugger. When issue reproduced, then you can get stack trace and variables values

сб, 28 нояб. 2020 г., 18:35 Carlos Vicente cvicente.lists@gmail.com:

...
Greetings,

I'm having a "fun" time trying to run Kamailio inside a Docker container. It keeps failing with:

0(238) DEBUG: <core> [core/mem/pkg.c:108]: pkg_print_manager(): pkg

using memory manager: q_malloc

0(238) DEBUG: <core> [core/mem/shm.c:300]: shm_print_manager(): shm

using memory manager: q_malloc

Illegal instruction (core dumped)

Unfortunately the core dump is nowhere to be found. I have tried to follow the instructions at:

https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile

but that hasn't helped.

The environment is as follows:

Host hardware: Raspberry Pi 4 (4G memory)

Host OS: Ubuntu 20.04.1 LTS

Docker version 19.03.13

Docker base image: arm32v7/ubuntu

root@2ef8a21534fa:/# kamailio -v version: kamailio 5.3.2 (arm/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 9.2.1

The config file passes the syntax check:
kamailio -c -f /etc/kamailio/kamailio.cfg
and, the most interesting part, if I run Kamailio on the host machine, _with the exact same config file_, it runs fine.

I'd appreciate any clues you may have. Thanks.

cv

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Daniel-Constantin Mierla

8:16 p.m.

How did you compile kamailio? The binaries do not contain the debugging symbols...

Cheers, Daniel

On 30.11.20 16:38, Carlos Vicente wrote:

...

I couldn't find any "-debug" packages, so I compiled the latest version:

root@348cbcfcb96c:~# /usr/local/sbin/kamailio -v
version: kamailio 5.5.0-dev3 (arm/linux) d43163
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE
1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: d43163
compiled on 15:17:51 Nov 30 2020 with gcc 9.3.0

But still not symbols:

root@348cbcfcb96c:~# gdb /usr/local/sbin/kamailio /cores/core.kamailio.19
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html <http://gnu.org/licenses/gpl.html>>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/
<http://www.gnu.org/software/gdb/bugs/>>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/
<http://www.gnu.org/software/gdb/documentation/>>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/kamailio...
(No debugging symbols found in /usr/local/sbin/kamailio)
[New LWP 19]
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `/usr/local/sbin/kamailio -f
/etc/kamailio/kamailio.cfg -DD -E -d'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x00b06320 in tsl ()
(gdb) bt full
#0  0x00b06320 in tsl ()
No symbol table info available.
#1  0x00b06380 in get_lock ()
No symbol table info available.
#2  0x00b1979c in qm_shm_malloc ()
No symbol table info available.
#3  0x009504dc in ksr_shutdown_phase_init ()
No symbol table info available.
#4  0x0078e3f4 in main ()
No symbol table info available.
(gdb) quit

TIA

On Mon, Nov 30, 2020 at 9:33 AM Sergey Safarov <s.safarov@gmail.com mailto:s.safarov@gmail.com> wrote:

Are you compiled Kamailio from sources?

If compiled from sources or installed "debug" files then pelase
show output of "bt full" command.

On Mon, Nov 30, 2020 at 5:23 PM Carlos Vicente
<cvicente.lists@gmail.com <mailto:cvicente.lists@gmail.com>> wrote:

    Daniel,

    Thank you. That finally helped getting the core file written.
    Here is the gdb output:

    ```
    root@6e5c06e979cf:/# gdb /usr/sbin/kamailio
    /cores/core.kamailio.207
    GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
    Copyright (C) 2020 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later
    <http://gnu.org/licenses/gpl.html
    <http://gnu.org/licenses/gpl.html>>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    Type "show copying" and "show warranty" for details.
    This GDB was configured as "arm-linux-gnueabihf".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/
    <http://www.gnu.org/software/gdb/bugs/>>.
    Find the GDB manual and other documentation resources online at:
        <http://www.gnu.org/software/gdb/documentation/
    <http://www.gnu.org/software/gdb/documentation/>>.

    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from /usr/sbin/kamailio...
    (No debugging symbols found in /usr/sbin/kamailio)
    [New LWP 207]
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library
    "/lib/arm-linux-gnueabihf/libthread_db.so.1".
    Core was generated by `/usr/sbin/kamailio -f
    /etc/kamailio/kamailio.cfg -DD -E -ddd'.
    Program terminated with signal SIGILL, Illegal instruction.
    #0  0x007d9cb8 in qm_shm_malloc ()
    (gdb) bt
    #0  0x007d9cb8 in qm_shm_malloc ()
    #1  0x0061a58c in init_atomic_ops ()
    #2  0x00562508 in main ()
    ```

    What can be done about that?

    cv

    On Mon, Nov 30, 2020 at 4:29 AM Daniel-Constantin Mierla
    <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

        Maybe the hints on stackoverflow are also useful:

          *
        https://stackoverflow.com/questions/28335614/how-to-generate-core-file-in-docker-container
        <https://stackoverflow.com/questions/28335614/how-to-generate-core-file-in-docker-container>

        Cheers,
        Daniel

        On 29.11.20 14:23, Sergey Safarov wrote:

...

        you can change
        set args -f /etc/kamailio/kamailio.cfg -D -d -E

        To
        set args -f /etc/kamailio/kamailio.cfg -DD -d -E

        On Sun, Nov 29, 2020 at 4:14 PM Carlos Vicente
        <cvicente.lists@gmail.com
        <mailto:cvicente.lists@gmail.com>> wrote:

            Thank you for your response.

            I'm trying to run it under gdb but it's not doing
            anything:

            ```
            (gdb) exec-file /usr/sbin/kamailio
            (gdb) set args -f /etc/kamailio/kamailio.cfg -D -d -E
            (gdb) run
            Starting program: /usr/sbin/kamailio -f
            /etc/kamailio/kamailio.cfg -D -d -E
            warning: Error disabling address space randomization:
            Success
            ```

            It just stops there. Am I missing something?

            cv

            On Sat, Nov 28, 2020 at 11:37 AM Sergey Safarov
            <s.safarov@gmail.com <mailto:s.safarov@gmail.com>> wrote:

                You can start kamailio under GDB debugger.
                When issue reproduced, then you can get stack
                trace and variables values


                сб, 28 нояб. 2020 г., 18:35 Carlos Vicente
                <cvicente.lists@gmail.com
                <mailto:cvicente.lists@gmail.com>>:

                    Greetings,

                    I'm having a "fun" time trying to run
                    Kamailio inside a Docker container. It keeps
                    failing with:

                     0(238) DEBUG: <core> [core/mem/pkg.c:108]:
                    pkg_print_manager(): pkg - using memory
                    manager: q_malloc
                     0(238) DEBUG: <core> [core/mem/shm.c:300]:
                    shm_print_manager(): shm - using memory
                    manager: q_malloc
                    Illegal instruction (core dumped)

                    Unfortunately the core dump is nowhere to be
                    found. I have tried to follow the
                    instructions at:

                       
                    https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile
                    <https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile>

                    but that hasn't helped.

                    The environment is as follows:

                    * Host hardware: Raspberry Pi 4 (4G memory)
                    * Host OS: Ubuntu 20.04.1 LTS
                    * Docker version 19.03.13
                    * Docker base image: arm32v7/ubuntu

                    root@2ef8a21534fa:/# kamailio -v
                    version: kamailio 5.3.2 (arm/linux)
                    flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
                    USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
                    DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
                    F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
                    USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP,
                    USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
                    USE_DST_BLACKLIST, HAVE_RESOLV_RES,
                    TLS_PTHREAD_MUTEX_SHARED
                    ADAPTIVE_WAIT_LOOPS 1024,
                    MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE
                    1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
                    poll method support: poll, epoll_lt,
                    epoll_et, sigio_rt, select.
                    id: unknown
                    compiled with gcc 9.2.1

                    The config file passes the syntax check:

                        kamailio -c -f /etc/kamailio/kamailio.cfg

                    and, the most interesting part, if I run
                    Kamailio on the host machine, _with the exact
                    same config file_, it runs fine.

                    I'd appreciate any clues you may have. Thanks.

                    cv


                    _______________________________________________
                    Kamailio (SER) - Users Mailing List
                    sr-users@lists.kamailio.org
                    <mailto:sr-users@lists.kamailio.org>
                    https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
                    <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

                _______________________________________________
                Kamailio (SER) - Users Mailing List
                sr-users@lists.kamailio.org
                <mailto:sr-users@lists.kamailio.org>
                https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
                <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

            _______________________________________________
            Kamailio (SER) - Users Mailing List
            sr-users@lists.kamailio.org
            <mailto:sr-users@lists.kamailio.org>
            https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
            <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>


        _______________________________________________
        Kamailio (SER) - Users Mailing List
        sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
        https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

        -- 
        Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
        www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
        Funding: https://www.paypal.me/dcmierla <https://www.paypal.me/dcmierla>

        _______________________________________________
        Kamailio (SER) - Users Mailing List
        sr-users@lists.kamailio.org
        <mailto:sr-users@lists.kamailio.org>
        https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
        <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

    _______________________________________________
    Kamailio (SER) - Users Mailing List
    sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
    https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
    <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
<https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Carlos Vicente

8:31 p.m.

Hi Daniel,

I followed this guide: https://kamailio.org/docs/tutorials/5.5.x/kamailio-install-guide-git/

On Mon, Nov 30, 2020 at 3:16 PM Daniel-Constantin Mierla miconda@gmail.com wrote:

...

How did you compile kamailio? The binaries do not contain the debugging symbols...

Cheers, Daniel On 30.11.20 16:38, Carlos Vicente wrote:

I couldn't find any "-debug" packages, so I compiled the latest version:
root@348cbcfcb96c:~# /usr/local/sbin/kamailio -v
version: kamailio 5.5.0-dev3 (arm/linux) d43163
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: d43163
compiled on 15:17:51 Nov 30 2020 with gcc 9.3.0
But still not symbols:
root@348cbcfcb96c:~# gdb /usr/local/sbin/kamailio /cores/core.kamailio.19
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/kamailio...
(No debugging symbols found in /usr/local/sbin/kamailio)
[New LWP 19]
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `/usr/local/sbin/kamailio -f
/etc/kamailio/kamailio.cfg -DD -E -d'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x00b06320 in tsl ()
(gdb) bt full
#0  0x00b06320 in tsl ()
No symbol table info available.
#1  0x00b06380 in get_lock ()
No symbol table info available.
#2  0x00b1979c in qm_shm_malloc ()
No symbol table info available.
#3  0x009504dc in ksr_shutdown_phase_init ()
No symbol table info available.
#4  0x0078e3f4 in main ()
No symbol table info available.
(gdb) quit
TIA

cv

On Mon, Nov 30, 2020 at 9:33 AM Sergey Safarov s.safarov@gmail.com wrote:

...
Are you compiled Kamailio from sources?

If compiled from sources or installed "debug" files then pelase show output of "bt full" command.

On Mon, Nov 30, 2020 at 5:23 PM Carlos Vicente cvicente.lists@gmail.com wrote:

...
Daniel,

Thank you. That finally helped getting the core file written. Here is the gdb output:
root@6e5c06e979cf:/# gdb /usr/sbin/kamailio /cores/core.kamailio.207
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kamailio...
(No debugging symbols found in /usr/sbin/kamailio)
[New LWP 207]
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-DD -E -ddd'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x007d9cb8 in qm_shm_malloc ()
(gdb) bt
#0  0x007d9cb8 in qm_shm_malloc ()
#1  0x0061a58c in init_atomic_ops ()
#2  0x00562508 in main ()
What can be done about that?

cv

On Mon, Nov 30, 2020 at 4:29 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Maybe the hints on stackoverflow are also useful:

https://stackoverflow.com/questions/28335614/how-to-generate-core-file-in-do...

Cheers, Daniel On 29.11.20 14:23, Sergey Safarov wrote:

you can change set args -f /etc/kamailio/kamailio.cfg -D -d -E

To set args -f /etc/kamailio/kamailio.cfg -DD -d -E

On Sun, Nov 29, 2020 at 4:14 PM Carlos Vicente < cvicente.lists@gmail.com> wrote:

...
Thank you for your response.

I'm trying to run it under gdb but it's not doing anything:
(gdb) exec-file /usr/sbin/kamailio
(gdb) set args -f /etc/kamailio/kamailio.cfg -D -d -E
(gdb) run
Starting program: /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -D
-d -E
warning: Error disabling address space randomization: Success
It just stops there. Am I missing something?

cv

On Sat, Nov 28, 2020 at 11:37 AM Sergey Safarov s.safarov@gmail.com wrote:

...
You can start kamailio under GDB debugger. When issue reproduced, then you can get stack trace and variables values

сб, 28 нояб. 2020 г., 18:35 Carlos Vicente <cvicente.lists@gmail.com >:

> Greetings, > > I'm having a "fun" time trying to run Kamailio inside a Docker > container. It keeps failing with: > > 0(238) DEBUG: <core> [core/mem/pkg.c:108]: pkg_print_manager(): pkg > - using memory manager: q_malloc > 0(238) DEBUG: <core> [core/mem/shm.c:300]: shm_print_manager(): shm > - using memory manager: q_malloc > Illegal instruction (core dumped) > > Unfortunately the core dump is nowhere to be found. I have tried to > follow the instructions at: > > > https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile > > but that hasn't helped. > > The environment is as follows: > > * Host hardware: Raspberry Pi 4 (4G memory) > * Host OS: Ubuntu 20.04.1 LTS > * Docker version 19.03.13 > * Docker base image: arm32v7/ubuntu > > root@2ef8a21534fa:/# kamailio -v > version: kamailio 5.3.2 (arm/linux) > flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, > DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, > F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, > FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, > USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED > ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE > 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > id: unknown > compiled with gcc 9.2.1 > > The config file passes the syntax check: > > kamailio -c -f /etc/kamailio/kamailio.cfg > > and, the most interesting part, if I run Kamailio on the host > machine, _with the exact same config file_, it runs fine. > > I'd appreciate any clues you may have. Thanks. > > cv > > > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Henning Westerholt

8:39 p.m.

Hello,

it might be something related to the ARM compilation, that maybe the build system does not generate proper debug symbols here.

I need to check on my Rasberries as well, but do not remember problems last time I compiled it there.

Cheers,

Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com https://gilawa.com/

From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Carlos Vicente Sent: Monday, November 30, 2020 9:32 PM To: miconda@gmail.com Cc: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] core dump error when running under docker

Hi Daniel,

I followed this guide: https://kamailio.org/docs/tutorials/5.5.x/kamailio-install-guide-git/

On Mon, Nov 30, 2020 at 3:16 PM Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:

How did you compile kamailio? The binaries do not contain the debugging symbols...

Cheers, Daniel On 30.11.20 16:38, Carlos Vicente wrote: I couldn't find any "-debug" packages, so I compiled the latest version:

But still not symbols:

``` root@348cbcfcb96c:~# gdb /usr/local/sbin/kamailio /cores/core.kamailio.19 GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-linux-gnueabihf". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

TIA

On Mon, Nov 30, 2020 at 9:33 AM Sergey Safarov <s.safarov@gmail.commailto:s.safarov@gmail.com> wrote: Are you compiled Kamailio from sources?

If compiled from sources or installed "debug" files then pelase show output of "bt full" command.

On Mon, Nov 30, 2020 at 5:23 PM Carlos Vicente <cvicente.lists@gmail.commailto:cvicente.lists@gmail.com> wrote: Daniel,

Thank you. That finally helped getting the core file written. Here is the gdb output:

``` root@6e5c06e979cf:/# gdb /usr/sbin/kamailio /cores/core.kamailio.207 GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-linux-gnueabihf". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

What can be done about that?

On Mon, Nov 30, 2020 at 4:29 AM Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:

Maybe the hints on stackoverflow are also useful:

* https://stackoverflow.com/questions/28335614/how-to-generate-core-file-in-do...

Cheers, Daniel On 29.11.20 14:23, Sergey Safarov wrote: you can change set args -f /etc/kamailio/kamailio.cfg -D -d -E

To set args -f /etc/kamailio/kamailio.cfg -DD -d -E

On Sun, Nov 29, 2020 at 4:14 PM Carlos Vicente <cvicente.lists@gmail.commailto:cvicente.lists@gmail.com> wrote: Thank you for your response.

I'm trying to run it under gdb but it's not doing anything:

It just stops there. Am I missing something?

On Sat, Nov 28, 2020 at 11:37 AM Sergey Safarov <s.safarov@gmail.commailto:s.safarov@gmail.com> wrote: You can start kamailio under GDB debugger. When issue reproduced, then you can get stack trace and variables values

сб, 28 нояб. 2020 г., 18:35 Carlos Vicente <cvicente.lists@gmail.commailto:cvicente.lists@gmail.com>: Greetings,

I'm having a "fun" time trying to run Kamailio inside a Docker container. It keeps failing with:

Unfortunately the core dump is nowhere to be found. I have tried to follow the instructions at:

https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile

but that hasn't helped.

The environment is as follows:

* Host hardware: Raspberry Pi 4 (4G memory) * Host OS: Ubuntu 20.04.1 LTS * Docker version 19.03.13 * Docker base image: arm32v7/ubuntu

The config file passes the syntax check:

kamailio -c -f /etc/kamailio/kamailio.cfg

and, the most interesting part, if I run Kamailio on the host machine, _with the exact same config file_, it runs fine.

I'd appreciate any clues you may have. Thanks.

_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

_______________________________________________

Kamailio (SER) - Users Mailing List

sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org

https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com

www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda

Funding: https://www.paypal.me/dcmierla _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

_______________________________________________

Kamailio (SER) - Users Mailing List

sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org

https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com

www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda

Funding: https://www.paypal.me/dcmierla

1543

Age (days ago)

1545

Last active (days ago)

sr-users@lists.kamailio.org

10 comments

4 participants

tags (0)

participants (4)

Carlos Vicente
Daniel-Constantin Mierla
Henning Westerholt
Sergey Safarov