Hi Fred,

We'll, problem solved...

I was able to duplicate the problem with the IP configured on em3 using # nc -lvu  10.102.88.81 5060. No traffic.

As stated before, the IP works on interface em1(10g), not em3(1g). Centos 7, Dell R630. Firewalld correct.

I started digging into the Dell Broadcom BCM57800 card. Numerous reports of issues with the ports not working. Which comes as quite a surprise, considering we have 8 x R630's running the cards since January!

I've never seen a network interface not pass the packet to the stack - kind of the whole point. Ordering replacement Dell/Intel cards now.

Thanks to all for your help!!! Anxious to put Kamailio in production to handle STIR/SHAKEN (and maybe some cool routing).

Thanks,
Dan

Fred Posner wrote on 10/13/21 9:39 AM:
I think at this point, it would be best to rule out your
system/environment before debugging code.

Have you see if another application layer program can see the SIP
traffic (such as sipp)?

If a netstat shows kamailio listening on the IP/port, and kamailio
didn't indicate any errors binding to the IP provided, we must further
rule out something on the system.

Fred Posner | palner.com
Matrix: @fred:matrix.lod.com
o: +1 (212) 937-7844

On 10/13/21 7:58 AM, dem@htsweb.com wrote:
Hi Sergey,

Only testing with the SIP INVITE from one of our production SIP trunk
providers. The 'length's are listed below for the tcpdump. Same packet
size/contents for all tests. Only change was moving IP from interface
em3 to em1. Doesn't look like packet size is the issue.

Looking at some of the socket code. Could be related to socket structs,
maybe an array enumeration issue. We have 4 interfaces: em1 ((10g up),
em2(10g down), em3 (1g up), em4 (1g up). Just switched IP from 3rd
interface (2nd up) to 1st interface (1st up) - it worked.

If somebody would like to provide a debug message/location, I could
recompile to test the array bucket theory. Quick way to check.

Note, the config below is the entire test config. No changes. Only
purpose was to receive SIP INVITE to test socket.

Thanks,
Dan

Sergey Safarov wrote on 10/13/21 4:08 AM:
you also need to check UDP packets are fragmented or not.
you must receive all fragments before Kamailio is able to parse the
message.

On Wed, Oct 13, 2021 at 1:28 AM David Villasmil
<david.villasmil.work@gmail.com
<mailto:david.villasmil.work@gmail.com>> wrote:

    use the

    listen=udp:YOUR-LOCAL-IP:5060

    then do a netstat to see if it's listening on 5060


    Regards,

    David Villasmil
    email: david.villasmil.work@gmail.com
    <mailto:david.villasmil.work@gmail.com>
    phone: +34669448337


    On Tue, Oct 12, 2021 at 11:17 PM <dem@htsweb.com
    <mailto:dem@htsweb.com>> wrote:

        Okay, think I discovered something....

        I'm attempting to run the kamailio on interface em3 - no joy. 
        I moved the 10.102.88.81 ip to the em1 interface. kamailio is
        showing the UDP packets as received (via my horribly simple
        config below).

        This test also answers the firewall question, UDP port is allowed.

        So, that prompts the question... how do you use kamailio with
        UDP on a specific interface ?  I'm thinking it might not do it
        currently ?

        Also, forgive me if I messed up the prior reply to the list.

        Thanks,
        Dan

        dem@htsweb.com <mailto:dem@htsweb.com> wrote on 10/12/21 5:28 PM:
        Hello,

        The tcpdump (below) shows packet reaching em3 interface. The
        firewalld command (below) shows UDP port 5060 open.

        netcat doesn't do specific interfaces. Can't do ALL
        interfaces, since em4 is a pcap of hundreds of thousands of
        live UDP 5060.

        I'll have to write a standalone to prove that the firewall
        allows UDP 5060 through the em3 interface, post firewall...
        (betting it does). Should have test prog tomorrow.

        Thanks,
        Dan

        Daniel-Constantin Mierla wrote on 10/12/21 4:27 PM:
        Hello,

        tcpdump and other network sniffers (e.g., ngrep, sngrep)
        capture the sip traffic before firewall, so even if you see
        packets with them, the application can still not get them
        due to firewall.

        To test, you can replace kamailio with netcat listening on
        UDP port 5060 and see if it receives the sip traffic.

        Cheers,
        Daniel

        On 12.10.21 19:08, dem@htsweb.com <mailto:dem@htsweb.com> wrote:
        After 4 (long) days of trying to receive any UDP SIP, I'm
        hoping someone can point out what I'm doing wrong. I've
        looked at the kamailio code. I'm beginning to think it's
        the 'polling' code, but it looks fine. It could be Centos
        or a Dell network driver, but not seeing any errors anywhere.

        - stock build/compile...
          Version: kamailio 5.5.2 (x86_64/linux) 0d53d9
          Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
        USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
        SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
        DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
        USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
        USE_DST_BLOCKLIST, HAVE_RESOLV_RES
          MAX_RECV_BUFFER_SIZE=262144
          MAX_URI_SIZE=1024
          BUF_SIZE=65535
          DEFAULT PKG_SIZE=8MB
          DEFAULT SHM_SIZE=64MB
          ADAPTIVE_WAIT_LOOPS=1024
          TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select
          Source code revision ID: 0d53d9
          Compiled with: gcc 4.8.5
          Compiled architecture: x86_64
          Compiled on: 08:44:19 Oct 11 2021

        Running Centos 7 all updated. Dell R630.
        # sestatus
        SELinux status:                 disabled
        # firewall-cmd --list-all
          ports: 5666/tcp 873/tcp 873/udp 80/tcp 443/tcp 5060/tcp
        5060/udp 5061/tcp 5061/udp

        - Just trying to send SIP INVITE from trunk provider to
        R630 kamailio...
        (SIP trunk) UDP 172.200.200.202:5060
        <http://172.200.200.202:5060> -> (firewall) UDP
        172.250.250.162:5060 <http://172.250.250.162:5060> -> 1:1
        NAT -> (R630) UDP 10.102.88.81:5060 <http://10.102.88.81:5060>

        - I can 'see' the packet hit the em3 network...
        #  tcpdump -i em3 -e -vv -n port 5060
        11:53:33.260892 99:99:ef:3e:69:c1 > 99:99:1c:48:4e:e3,
        ethertype IPv4 (0x0800), length 1366: (tos 0x0, ttl 54, id
        20430, offset 0, flags [none], proto UDP (17), length 1352)
            172.200.200.202.sip > 10.102.88.81.sip: [udp sum ok]
        SIP, length: 1324
            INVITE sip:+18005551234@172.250.250.162:5060
        <mailto:sip:+18005551234@172.250.250.162:5060> SIP/2.0
            ...

        At this point, I just need to see a SIP INVITE received.
        Then I can go back to working on a real config.

        I've tried with TCP on/off, listening on all networks,
        alias on/off, strace and everything else I could think
        of.Truly appreciate any guidance.

        Tried every sample config possible, but I just need to test
        receiving the SIP UDP packet, so this 'should' be minimum
        config...

        #!KAMAILIO
        debug=9
        log_stderror=no
        log_facility=LOG_LOCAL0
        log_prefix="{$mt $hdr(CSeq) $ci} "
        children=2
        disable_tcp=yes
        udp4_raw=on
        auto_aliases=no
        alias="sbc01.mysite.com <http://sbc01.mysite.com>"
        listen=udp:10.102.88.81:5060

        loadmodule "debugger.so"
        loadmodule "jsonrpcs.so"
        loadmodule "kex.so"
        loadmodule "corex.so"
        loadmodule "sl.so"
        loadmodule "rr.so"
        loadmodule "pv.so"
        loadmodule "textops.so"
        loadmodule "siputils.so"
        loadmodule "xlog.so"
        loadmodule "sanity.so"
        loadmodule "ctl.so"
        loadmodule "cfg_rpc.so"

        modparam("debugger", "cfgtrace", 1)

        request_route {
          xlog("SIP received xxxxxxxxxxxxxxxxxxx\n");
        }

        # kamcmd ps
        11578    main process - attendant
        11579    udp receiver child=0 sock=10.102.88.81:5060
        <http://10.102.88.81:5060>
        11580    udp receiver child=1 sock=10.102.88.81:5060
        <http://10.102.88.81:5060>
        11581    slow timer
        11583    timer
        11584    secondary timer
        11585    JSONRPCS FIFO
        11586    JSONRPCS DATAGRAM
        11587    ctl handler

        - I get plenty of logging, but the logging just waits here...
        Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]:
        INFO: ctl [io_listener.c:210]: io_listen_loop():
        io_listen_loop: using epoll_lt as the io watch method (auto
        detected)
        Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]:
        DEBUG: <core> [core/mem/q_malloc.c:374]: qm_malloc():
        qm_malloc(0x7fd2a8bae010, 4800) called from core:
        core/io_wait.c: init_io_wait(469)
        Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10059]:
        DEBUG: <core> [core/mem/q_malloc.c:419]: qm_malloc():
        qm_malloc(0x7fd2a8bae010, 65456) returns address
        0x7fd2a8c652d8 frag. 0x7fd2a8c652a0 (size=65456) on 1 -th hit
        Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]:
        DEBUG: <core> [core/mem/q_malloc.c:419]: qm_malloc():
        qm_malloc(0x7fd2a8bae010, 4800) returns address
        0x7fd2a8c652d8 frag. 0x7fd2a8c652a0 (size=4800) on 1 -th hit
        Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]:
        DEBUG: <core> [core/mem/q_malloc.c:374]: qm_malloc():
        qm_malloc(0x7fd2a8bae010, 2400) called from core:
        core/io_wait.c: init_io_wait(516)
        Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]:
        DEBUG: <core> [core/mem/q_malloc.c:419]: qm_malloc():
        qm_malloc(0x7fd2a8bae010, 2400) returns address
        0x7fd2a8c66600 frag. 0x7fd2a8c665c8 (size=2400) on 1 -th hit
        Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]:
        DEBUG: ctl [io_listener.c:246]: io_listen_loop():
        io_listen_loop: adding socket 8, type 2, transport 3
        (/var/run/kamailio//kamailio_ctl)
        Oct 12 11:38:04 sbc01 /usr/local/sbin/kamailio[10060]:
        DEBUG: ctl [../../core/io_wait.h:375]: io_watch_add(): DBG:
        io_watch_add(0x7fd2a70a5360, 8, 2, 0x2439960), fd_no=0

        - The only other thing that I see is transport 3. Isn't
        that a UNIXS_SOCK?



        __________________________________________________________
        Kamailio - Users Mailing List - Non Commercial Discussions
          * sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
        Important: keep the mailing list in the recipients, do not reply only to the sender!
        Edit mailing list options or unsubscribe:
          * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
        -- 
        Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
        www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
        Kamailio Advanced Training - Online
        Nov 08-11, 2021 (Europe Timezone) - Nov 22-25, 2021 (America Timezone)
          * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>

          
        __________________________________________________________
        Kamailio - Users Mailing List - Non Commercial Discussions
          * sr-users@lists.kamailio.org
        <mailto:sr-users@lists.kamailio.org>
        Important: keep the mailing list in the recipients, do not
        reply only to the sender!
        Edit mailing list options or unsubscribe:
          *
        https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
        <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

    __________________________________________________________
    Kamailio - Users Mailing List - Non Commercial Discussions
      * sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
    Important: keep the mailing list in the recipients, do not reply
    only to the sender!
    Edit mailing list options or unsubscribe:
      * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
    <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>


__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users