If you don't have the mask ip to your host ip (kamailio listening ip) like Alex pointed, do you have a core dump from where you can send the backtrace?

Cheers, Daniel

On 11/24/10 2:58 AM, Alex Balashov wrote:

You don't happen to be setting your mask IP to your host's actual IP, do you? There is a periodic crash bug that occurs when that happens.

-- Alex Balashov - Principal Evariste Systems LLC 1170 Peachtree Street 12th Floor, Suite 1200 Atlanta, GA 30309 Tel: +1-678-954-0670 Fax: +1-404-961-1892 Web: http://www.evaristesys.com/

On Nov 23, 2010, at 8:33 PM, dotnetdub dotnetdub@gmail.com wrote:

...

Hi All,

Kamailio had just started to get SIP attacked and it looked like they were sending a bad header and Kamailio died..

Nov 20 19:48:51 sipproxy sip[3383]: INFO: <script>: [R-MAIN =>] Request method OPTIONS is to our domain (212.124.123.170) Nov 20 19:48:51 sipproxy sip[3378]: INFO: <script>: [ROUTE-1] Received registration from sip:1368250498@sipproxy (212.124.123.170) Nov 20 19:48:51 sipproxy sip[3378]: INFO: <script>: [ROUTE-1 !] Credentials invalid; issuing challenge Nov 20 19:48:51 sipproxy sip[3381]: INFO: <script>: [ROUTE-1] Received registration from sip:100@sipproxy (212.124.123.170) Nov 20 19:48:51 sipproxy sip[3381]: INFO: <script>: [ROUTE-1 !] Credentials invalid; issuing challenge Nov 20 19:48:51 sipproxy sip[3374]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.401446] kamailio[3374]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:51 sipproxy sip[3373]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.636774] kamailio[3373]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:51 sipproxy sip[3388]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.680305] kamailio[3388]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:51 sipproxy sip[3390]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.710444] kamailio[3390]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:51 sipproxy sip[3383]: INFO: <script>: [ROUTE-1] Received registration from sip:2478223614@sipproxy (212.124.123.170) Nov 20 19:48:51 sipproxy sip[3383]: INFO: <script>: [ROUTE-1 !] Credentials invalid; issuing challenge Nov 20 19:48:51 sipproxy sip[3378]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.753334] kamailio[3378]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:51 sipproxy sip[3379]: INFO: <script>: [ROUTE-1] Received registration from sip:noauth@sipproxy (212.124.123.170) Nov 20 19:48:51 sipproxy sip[3379]: INFO: <script>: [ROUTE-1 !] Credentials invalid; issuing challenge Nov 20 19:48:51 sipproxy sip[3375]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.768137] kamailio[3375]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:51 sipproxy sip[3383]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.775628] kamailio[3383]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:51 sipproxy sip[3386]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.790925] kamailio[3386]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:51 sipproxy sip[3379]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.798926] kamailio[3379]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:51 sipproxy sip[3381]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:51 sipproxy kernel: [1033573.815493] kamailio[3381]: segfault at 18 ip b70070d9 sp bf916118 error 4 in topoh.so[b7004000+c000] Nov 20 19:48:54 sipproxy sip[3344]: INFO: <core> [main.c:737]: INFO: terminating due to SIGCHLD Nov 20 19:48:54 sipproxy sip[3399]: INFO: <core> [main.c:788]: INFO: signal 15 received Nov 20 19:48:54 sipproxy sip[3396]: INFO: <core> [main.c:788]: INFO: signal 15 received Nov 20 19:48:54 sipproxy sip[3400]: INFO: <core> [main.c:788]: INFO: signal 15 received Nov 20 19:48:54 sipproxy sip[3392]: INFO: <core> [main.c:788]: INFO: signal 15 received Nov 20 19:48:54 sipproxy sip[3401]: INFO: <core> [main.c:788]: INFO: signal 15 received Nov 20 19:48:54 sipproxy sip[3394]: INFO: <core> [main.c:788]: INFO: signal 15 received Nov 20 19:48:54 sipproxy sip[3402]: INFO: <core> [main.c:788]: INFO: signal 15 received

Thanks Brian

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

On 11/25/2010 12:00 PM, dotnetdub wrote:

On 24 November 2010 07:10, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

If you don't have the mask ip to your host ip (kamailio listening
ip) like Alex pointed, do you have a core dump from where you can
send the backtrace?

Cheers,
Daniel

Hi Alex/Daniel,

so THe IP of this SIP proxy is x.x.x.11 and I am using x.x.x.10 for the Mask IP.

I see the coredump in the root of the server..

ls /core -lah -rw------- 1 root root 134M 2010-11-20 19:48 /core

which is the date it crashed.

gdb /sbin/kamailio /core

This GDB was configured as "i486-linux-gnu"... Cannot access memory at address 0xb75f3fb0

I am running gdb from debian repo on 5.04 maybe I need to compile gdb?

Hello,

No, the problem is with the coredump AFAIK. Try looking thru /var/log/messages and dmesg for any indication of a coredump and the module that generated it. Please email those lines as well.

Marius

Thanks Brian

On 25 November 2010 11:00, marius zbihlei marius.zbihlei@1and1.ro wrote:

On 11/25/2010 12:07 PM, marius zbihlei wrote:

On 11/25/2010 12:00 PM, dotnetdub wrote:

On 24 November 2010 07:10, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

If you don't have the mask ip to your host ip (kamailio listening ip) like Alex pointed, do you have a core dump from where you can send the backtrace?

Cheers, Daniel

Hi Alex/Daniel,

so THe IP of this SIP proxy is x.x.x.11 and I am using x.x.x.10 for the Mask IP.

I see the coredump in the root of the server..

ls /core -lah -rw------- 1 root root 134M 2010-11-20 19:48 /core

which is the date it crashed.

gdb /sbin/kamailio /core

This GDB was configured as "i486-linux-gnu"... Cannot access memory at address 0xb75f3fb0

I am running gdb from debian repo on 5.04 maybe I need to compile gdb?

Hello,

No, the problem is with the coredump AFAIK. Try looking thru /var/log/messages and dmesg for any indication of a coredump and the module that generated it. Please email those lines as well.

Hello I see that the interesting line is already in the first mail . Please try this

readelf -S /path/to/topoh.so

There are 36 section headers, starting at offset 0x17b34:

Section Headers: [Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 0] NULL 00000000 000000 000000 00 0 0 0 [ 1] .hash HASH 000000b4 0000b4 000498 04 A 3 0 4 [ 2] .gnu.hash GNU_HASH 0000054c 00054c 0002d8 04 A 3 0 4 [ 3] .dynsym DYNSYM 00000824 000824 000650 10 A 4 1 4 [ 4] .dynstr STRTAB 00000e74 000e74 000581 00 A 0 0 1 [ 5] .gnu.version VERSYM 000013f6 0013f6 0000ca 02 A 3 0 2 [ 6] .gnu.version_r VERNEED 000014c0 0014c0 000030 00 A 4 1 4 [ 7] .rel.dyn REL 000014f0 0014f0 0001d0 08 A 3 0 4 [ 8] .rel.plt REL 000016c0 0016c0 000210 08 A 3 10 4 [ 9] .init PROGBITS 000018d0 0018d0 000030 00 AX 0 0 4 [10] .plt PROGBITS 00001900 001900 000430 04 AX 0 0 4 [11] .text PROGBITS 00001d30 001d30 0080d8 00 AX 0 0 16 [12] .fini PROGBITS 00009e08 009e08 00001c 00 AX 0 0 4 [13] .rodata PROGBITS 00009e24 009e24 001618 01 AMS 0 0 4 [14] .eh_frame PROGBITS 0000b43c 00b43c 000004 00 A 0 0 4 [15] .ctors PROGBITS 0000c440 00b440 000008 00 WA 0 0 4 [16] .dtors PROGBITS 0000c448 00b448 000008 00 WA 0 0 4 [17] .jcr PROGBITS 0000c450 00b450 000004 00 WA 0 0 4 [18] .dynamic DYNAMIC 0000c454 00b454 0000c8 08 WA 4 0 4 [19] .got PROGBITS 0000c51c 00b51c 00006c 04 WA 0 0 4 [20] .got.plt PROGBITS 0000c588 00b588 000114 04 WA 0 0 4 [21] .data PROGBITS 0000c6a0 00b6a0 00012c 00 WA 0 0 32 [22] .bss NOBITS 0000c7e0 00b7cc 0004a0 00 WA 0 0 32 [23] .comment PROGBITS 00000000 00b7cc 0000d9 00 0 0 1 [24] .debug_aranges PROGBITS 00000000 00b8a8 0000b0 00 0 0 8 [25] .debug_pubnames PROGBITS 00000000 00b958 000448 00 0 0 1 [26] .debug_info PROGBITS 00000000 00bda0 005dd1 00 0 0 1 [27] .debug_abbrev PROGBITS 00000000 011b71 0007a8 00 0 0 1 [28] .debug_line PROGBITS 00000000 012319 0014b9 00 0 0 1 [29] .debug_frame PROGBITS 00000000 0137d4 0004b0 00 0 0 4 [30] .debug_str PROGBITS 00000000 013c84 001d76 01 MS 0 0 1 [31] .debug_loc PROGBITS 00000000 0159fa 001ebe 00 0 0 1 [32] .debug_ranges PROGBITS 00000000 0178b8 000140 00 0 0 8 [33] .shstrtab STRTAB 00000000 0179f8 00013b 00 0 0 1 [34] .symtab SYMTAB 00000000 0180d4 0009e0 10 35 58 4 [35] .strtab STRTAB 00000000 018ab4 0006ec 00 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings) I (info), L (link order), G (group), x (unknown) O (extra OS processing required) o (OS specific), p (processor specific)

You should have somethink like this

[Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 0] NULL 00000000 000000 000000 00 0 0 0 ..... [10] .init PROGBITS 00001914 001914 000030 00 AX 0 0 4 [11] .plt PROGBITS 00001944 001944 000430 04 AX 0 0 4 [12] .text PROGBITS 00001d80 001d80 008008 00 AX 0 0 16

Note the address of the .text segment (in my case is 0x00001d80)

Then

gdb /path/to/kamailio core.dump

All ok

in the gdb console load the topoh.so

add-symbol-file /path/to/topoh.so 0xb7004000 + 0x<offset of .text segment found as above>

Ok.

then check the value of the Instruction pointer IP (0xb70070d9) Still in the gdb console

x/s 0xb70070d9

x/s 0xb70070d9 0xb70070d9: <Address 0xb70070d9 out of bounds>

(gdb) x/s 0x00001d30 0x1d30 <__do_global_dtors_aux>: <Address 0x1d30 out of bounds> (gdb) x/s 0xb700400 0xb700400 <__do_global_dtors_aux>: <Address 0xb700400 out of bounds>

Brian

You will get an offset relative to a function where the crashed happened. This would be very helpful so please share the finding. Also do a info registers (in the gdb console) and paste the content of the registers as well.

Marius

Marius

Thanks Brian

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

On 25 November 2010 12:33, marius zbihlei marius.zbihlei@1and1.ro wrote:

On 11/25/2010 01:31 PM, dotnetdub wrote:

There are 36 section headers, starting at offset 0x17b34:

Section Headers: [Nr] Name Type Addr Off Size ES Flg Lk Inf Al [ 0] NULL 00000000 000000 000000 00 0 0 0 [ 1] .hash HASH 000000b4 0000b4 000498 04 A 3 0 4 [ 2] .gnu.hash GNU_HASH 0000054c 00054c 0002d8 04 A 3 0 4 [ 3] .dynsym DYNSYM 00000824 000824 000650 10 A 4 1 4 [ 4] .dynstr STRTAB 00000e74 000e74 000581 00 A 0 0 1 [ 5] .gnu.version VERSYM 000013f6 0013f6 0000ca 02 A 3 0 2 [ 6] .gnu.version_r VERNEED 000014c0 0014c0 000030 00 A 4 1 4 [ 7] .rel.dyn REL 000014f0 0014f0 0001d0 08 A 3 0 4 [ 8] .rel.plt REL 000016c0 0016c0 000210 08 A 3 10 4 [ 9] .init PROGBITS 000018d0 0018d0 000030 00 AX 0 0 4 [10] .plt PROGBITS 00001900 001900 000430 04 AX 0 0 4 [11] .text PROGBITS 00001d30 001d30 0080d8 00 AX 0 0 16 [12] .fini PROGBITS 00009e08 009e08 00001c 00 AX 0 0 4 [13] .rodata PROGBITS 00009e24 009e24 001618 01 AMS 0 0 4 [14] .eh_frame PROGBITS 0000b43c 00b43c 000004 00 A 0 0 4 [15] .ctors PROGBITS 0000c440 00b440 000008 00 WA 0 0 4 [16] .dtors PROGBITS 0000c448 00b448 000008 00 WA 0 0 4 [17] .jcr PROGBITS 0000c450 00b450 000004 00 WA 0 0 4 [18] .dynamic DYNAMIC 0000c454 00b454 0000c8 08 WA 4 0 4 [19] .got PROGBITS 0000c51c 00b51c 00006c 04 WA 0 0 4 [20] .got.plt PROGBITS 0000c588 00b588 000114 04 WA 0 0 4 [21] .data PROGBITS 0000c6a0 00b6a0 00012c 00 WA 0 0 32 [22] .bss NOBITS 0000c7e0 00b7cc 0004a0 00 WA 0 0 32 [23] .comment PROGBITS 00000000 00b7cc 0000d9 00 0 0 1 [24] .debug_aranges PROGBITS 00000000 00b8a8 0000b0 00 0 0 8 [25] .debug_pubnames PROGBITS 00000000 00b958 000448 00 0 0 1 [26] .debug_info PROGBITS 00000000 00bda0 005dd1 00 0 0 1 [27] .debug_abbrev PROGBITS 00000000 011b71 0007a8 00 0 0 1 [28] .debug_line PROGBITS 00000000 012319 0014b9 00 0 0 1 [29] .debug_frame PROGBITS 00000000 0137d4 0004b0 00 0 0 4 [30] .debug_str PROGBITS 00000000 013c84 001d76 01 MS 0 0 1 [31] .debug_loc PROGBITS 00000000 0159fa 001ebe 00 0 0 1 [32] .debug_ranges PROGBITS 00000000 0178b8 000140 00 0 0 8 [33] .shstrtab STRTAB 00000000 0179f8 00013b 00 0 0 1 [34] .symtab SYMTAB 00000000 0180d4 0009e0 10 35 58 4 [35] .strtab STRTAB 00000000 018ab4 0006ec 00 0 0 1 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings) I (info), L (link order), G (group), x (unknown) O (extra OS processing required) o (OS specific), p (processor specific)

...

in the gdb console load the topoh.so

add-symbol-file /path/to/topoh.so 0xb7004000 + 0x<offset of .text segment found as above>

Ok.

Hello

Did you give gdb the command add-symbol-file /path/to/topoh.so 0xb7004000+0x00001d30

Gdb should inquire you if you want to lead topoh.so at offset 0xb7005d30 (press 'y'). Then the IP should not be out of bounds ... it should be mapped to the topoh.so text segment. Ah, I have made a mistake, there should be no spaces between the address and the offset (as seen above)

Anyway you should get something like this (on my machine - on your use the offsets from above) (gdb) add-symbol-file modules/topoh/topoh.so 0xb7004000+0x00001d80 add symbol table from file "modules/topoh/topoh.so" at .text_addr = 0xb7005d80 (y or n) y Reading symbols from /home/marius/dev/sip-router/modules/topoh/topoh.so...done.

Then (gdb) x/s 0xb70070d9 !!!!!!!!!!!!!!!!!!0xb70070d9 <th_get_hdr_cookie+25>: <Address 0xb70070d9 out of bounds>!!!!!!!!!!!!!!!!!

The line between all the !!!!!! is the line I care about

Also don't forget (gdb)info registers

Thank you Marius

Hi Marius,

I hope this is what your after!

(gdb) add-symbol-file /lib/kamailio/modules/topoh.so 0xb7004000+0x00001d30 add symbol table from file "/lib/kamailio/modules/topoh.so" at .text_addr = 0xb7005d30 (y or n) y Reading symbols from /lib/kamailio/modules/topoh.so...done. (gdb) x/s 0xb70070d9 0xb70070d9 <th_skip_msg+9>: <Address 0xb70070d9 out of bounds> (gdb) info registers eax 0xbf916160 -1080991392 ecx 0x0 0 edx 0x2 2 ebx 0xb7010588 -1224669816 esp 0xbf916118 0xbf916118 ebp 0xbf916118 0xbf916118 esi 0xbf916160 -1080991392 edi 0x82500c0 136642752 eip 0xb70070d9 0xb70070d9 <th_skip_msg+9> eflags 0x10246 [ PF ZF IF RF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0x7b 123 fs 0x0 0 gs 0x33 51

Thanks Brian

Hi Marius,

...

I hope this is what your after!

(gdb) add-symbol-file /lib/kamailio/modules/topoh.so 0xb7004000+0x00001d30 add symbol table from file "/lib/kamailio/modules/topoh.so" at .text_addr = 0xb7005d30 (y or n) y Reading symbols from /lib/kamailio/modules/topoh.so...done. (gdb) x/s 0xb70070d9 0xb70070d9 <th_skip_msg+9>: <Address 0xb70070d9 out of bounds> (gdb) info registers

Yes I think it is

Looking at the debug messages I see the CSeq is wrong.

But :

int th_skip_msg(sip_msg_t *msg) { if((get_cseq(msg)->method_id)&(METHOD_REGISTER|METHOD_PUBLISH)) return 1;

    return 0;

}

As the cseq is wrong the get_cseq macro probably returns a NULL Pointer who gets dereferenced (BANG the crash). Any other Ideas ?!

The patch is trivial ( if(!get_cseq(msg))) parse_cseq(....) ) something in this line. Daniel, What do you think ?

Marius

Thanks Marius.

Glad that we were able to find the issue.

Regards Brian

On 11/25/10 6:38 PM, marius zbihlei wrote:

On 11/25/2010 07:32 PM, dotnetdub wrote:

...

...

    Hi Marius,


I hope this is what your after!

(gdb) add-symbol-file /lib/kamailio/modules/topoh.so
0xb7004000+0x00001d30
add symbol table from file "/lib/kamailio/modules/topoh.so" at
.text_addr = 0xb7005d30
(y or n) y
Reading symbols from /lib/kamailio/modules/topoh.so...done.
(gdb) x/s 0xb70070d9
0xb70070d9 <th_skip_msg+9>:<Address 0xb70070d9 out of bounds>
(gdb) info registers

Yes I think it is

Looking at the debug messages I see the CSeq is wrong.

But :

int th_skip_msg(sip_msg_t *msg)
{

if((get_cseq(msg)->method_id)&(METHOD_REGISTER|METHOD_PUBLISH))
                return 1;

        return 0;
}

As the cseq is wrong the get_cseq macro probably returns a NULL
Pointer who gets dereferenced (BANG the crash). Any other Ideas ?!

The patch is trivial ( if(!get_cseq(msg))) parse_cseq(....) )
something in this line. Daniel, What do you think ?

Marius

Thanks Marius.

Glad that we were able to find the issue.

Are you able to test a patch if a provide one to you? I wanted to wait for Daniel's opinion as I have no way of testing it. If you have a dump of the attack traffic or you can generate more with bad CSEQ (as from the message log you provided) you can test the patch against your cfg and see if it still crashes(hope not). In my opinion the crash should be deterministic. You will find the trivial patch attached. If you can test it and it works I will push it to upstream (also to 3.0 branch). Keep in mind that other probles might appear as well during the processing of the SIP messages. If a core does appear please retry the steps in the previous mail with the new core and .so offset.

Apply the patch with the patch utility (copy to the modules/topoh and run patch < patch) . I await some feedback :)

The patch is ok, please go ahead and commit it.

Even if the cause for this case would be in some other place, the patch is still good and harmless for proper formatted sip messages.

Thanks, Daniel

On 11/26/2010 12:38 AM, dotne

Hi Marius,

Will apply tomorrow and recompile.. I don't have a dump of the attack traffic but I'm sure it won't take long . . .

Thanks for your assistance.

Regards Brian

Hello,

Please check commit 29990057d8b17cf0ded395438465c6cb2c38207f on the 3.0 branch.(patch cherry-picked to 3.1 and master branch)

Marius

On 01/13/2011 02:51 AM, dotnetdub wrote:

On 29 November 2010 09:33, marius zbihlei <marius.zbihlei@1and1.ro mailto:marius.zbihlei@1and1.ro> wrote:

On 11/26/2010 12:38 AM, dotne


    Hi Marius,

    Will apply tomorrow and recompile.. I don't have a dump of the
    attack traffic but I'm sure it won't take long . . .

    Thanks for your assistance.

    Regards
    Brian

Hello,

Please check commit 29990057d8b17cf0ded395438465c6cb2c38207f on
the 3.0 branch.(patch cherry-picked to 3.1 and master branch)

Marius

Hello,

Did this fix make it to the main 3.1 - I did install 3.1 last week and just checked the source..

int th_skip_msg(sip_msg_t *msg) { if (!get_cseq(msg)) { LM_WARN("Invalid/Unparsed CSeq in message. Skipping."); return 1; }

    if((get_cseq(msg)->method_id)&(METHOD_REGISTER|METHOD_PUBLISH))
            return 1;

    return 0;

Regards, Brian

Hello Brian,

Module: sip-router Branch: 3.1 Commit: a72e59d23d4b104af6d7f30d1dc02a5fe175f3af

Also master and 3.0 are checked. the patch is this simple check for messages that don't have a correct CSeq header.

Marius

On 25 November 2010 17:38, marius zbihlei marius.zbihlei@1and1.ro wrote:

On 11/25/2010 07:32 PM, dotnetdub wrote:

...

...

Are you able to test a patch if a provide one to you? I wanted to wait for Daniel's opinion as I have no way of testing it. If you have a dump of the attack traffic or you can generate more with bad CSEQ (as from the message log you provided) you can test the patch against your cfg and see if it still crashes(hope not). In my opinion the crash should be deterministic. You will find the trivial patch attached. If you can test it and it works I will push it to upstream (also to 3.0 branch). Keep in mind that other probles might appear as well during the processing of the SIP messages. If a core does appear please retry the steps in the previous mail with the new core and .so offset.

Apply the patch with the patch utility (copy to the modules/topoh and run patch < patch) . I await some feedback :)

Marius

Hi Marius,

I did apply this patch and recompile. I checked the lib folder and date of topoh changed to compile date.

Another SIP attack and core dump again.

This looks like different memory addresses though.

proxy:/var/log# dmesg [1853341.778338] kamailio[20503]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000] [1853341.921334] kamailio[20507]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000] [1853341.991430] kamailio[20498]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000] [1853342.057429] kamailio[20506]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000] [1853342.139751] kamailio[20505]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000] [1853342.149429] kamailio[20499]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000] [1853342.156097] kamailio[20502]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000] [1853342.160097] kamailio[20501]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000] [1853342.163561] kamailio[20500]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000] [1853342.168357] kamailio[20504]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20503]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 kernel: [1853341.778338] kamailio[20503]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20503]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20503]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20503]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 sip[20500]: INFO: <script>: [ROUTE-1] Received registration from sip:3564815798@195.191.29.11sip%3A3564815798@195.191.29.11(180.148.1.3)

Feb 4 16:19:09 proxy1 sip[20500]: INFO: <script>: [ROUTE-1 !] Credentials invalid; issuing challenge

Feb 4 16:19:09 proxy1 sip[20507]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 sip[20507]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20507]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20507]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 kernel: [1853341.921334] kamailio[20507]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20498]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 sip[20498]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20498]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20498]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 kernel: [1853341.991430] kamailio[20498]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20506]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 sip[20506]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20506]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20506]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 kernel: [1853342.057429] kamailio[20506]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20505]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 sip[20505]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20505]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20505]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 kernel: [1853342.139751] kamailio[20505]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20499]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 sip[20499]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20499]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20499]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 kernel: [1853342.149429] kamailio[20499]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20502]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 sip[20502]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20502]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20502]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 kernel: [1853342.156097] kamailio[20502]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20501]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 sip[20501]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20501]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20501]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 kernel: [1853342.160097] kamailio[20501]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20500]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 sip[20500]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20500]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20500]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 kernel: [1853342.163561] kamailio[20500]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:09 proxy1 sip[20504]: ERROR: <core> [parser/parse_cseq.c:97]: ERROR: CSeq EoL expected

Feb 4 16:19:09 proxy1 sip[20504]: ERROR: <core> [parser/parse_cseq.c:100]: ERROR: parse_cseq: bad cseq

Feb 4 16:19:09 proxy1 sip[20504]: ERROR: <core> [parser/msg_parser.c:158]: ERROR: get_hdr_field: bad cseq

Feb 4 16:19:09 proxy1 sip[20504]: INFO: <core> [parser/msg_parser.c:353]: ERROR: bad header field [CSeq: 1 REGISTER ACK]

Feb 4 16:19:09 proxy1 kernel: [1853342.168357] kamailio[20504]: segfault at 18 ip b7064220 sp bf9c3370 error 4 in topoh.so[b7061000+d000]

Feb 4 16:19:13 proxy1 sip[20497]: ALERT: <core> [main.c:741]: child process 20507 exited by a signal 11

Regards, Brian

Regards Brian

On 25 November 2010 10:07, marius zbihlei marius.zbihlei@1and1.ro wrote:

On 11/25/2010 12:00 PM, dotnetdub wrote:

On 24 November 2010 07:10, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

If you don't have the mask ip to your host ip (kamailio listening ip) like Alex pointed, do you have a core dump from where you can send the backtrace?

Cheers, Daniel

Hi Alex/Daniel,

so THe IP of this SIP proxy is x.x.x.11 and I am using x.x.x.10 for the Mask IP.

I see the coredump in the root of the server..

ls /core -lah -rw------- 1 root root 134M 2010-11-20 19:48 /core

which is the date it crashed.

gdb /sbin/kamailio /core

This GDB was configured as "i486-linux-gnu"... Cannot access memory at address 0xb75f3fb0

I am running gdb from debian repo on 5.04 maybe I need to compile gdb?

Hello,

No, the problem is with the coredump AFAIK. Try looking thru /var/log/messages and dmesg for any indication of a coredump and the module that generated it. Please email those lines as well.

Marius

Thanks Brian

Marius,

Only indications of core I have in any logs:

Nov 20 19:48:51 proxy sip[3381]: INFO: <core> [parser/msg_parser.c:339]: ERROR: bad header field [CSeq: 1 REGISTER ACK] Nov 20 19:48:54 proxy sip[3344]: ALERT: <core> [main.c:722]: child process 3390 exited by a signal 11 Nov 20 19:48:54 proxy sip[3344]: ALERT: <core> [main.c:725]: core was generated Nov 20 19:48:54 proxy sip[3344]: INFO: <core> [main.c:737]: INFO: terminating due to SIGCHLD

Will respond to other mail now..

Brian

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users