kamailio 4.0.3 crash

List overview All Threads
Download

newer

older

ndb_cassandra - New module proposal

[tracker] Task opened:...

Dragos Oancea

21 Nov 2013 21 Nov '13

5:36 p.m.

Hello

We had this crash today with kamailio 4.0.3 . It ran stable for few weeks until this crash.

GDB here:

http://pastebin.com/rACV31z8

Regards, Dragos

Attachments:

attachment.html (text/html — 2.2 KB)

Show replies by date

Daniel-Constantin Mierla

21 Nov 21 Nov

5:43 p.m.

Hello,

can you provde the output in gdb for:

frame 0

p *frag

p *((char*)frag + sizeof(struct qm_frag))

There were similar reports, so apparently there is a buffer overflow somewhere.

You should update to latest git branch 4.0, because there were some other fixes from 4.0.3. With this occasion, you should set MEMDBG=1 in Makefile.defs before recompiling the new version to be able to catch easier the overwrites of memory.

Cheers, Daniel

On 11/21/13 6:36 PM, Dragos Oancea wrote:

...

Hello

We had this crash today with kamailio 4.0.3 . It ran stable for few weeks until this crash.

GDB here:

http://pastebin.com/rACV31z8

Regards, Dragos

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28 - more details about Kamailio trainings at http://www.asipto.com -

Dragos Oancea

6:39 p.m.

Here is the output:

gdb) (gdb) frame 0 #0 qm_detach_free (qm=0x7fd96175e010, size=112) at mem/q_malloc.c:266 266in mem/q_malloc.c (gdb) #0 qm_detach_free (qm=0x7fd96175e010, size=112) at mem/q_malloc.c:266 266in mem/q_malloc.c (gdb) p *frag $1 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) $2 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) $3 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) p *((char*)frag + sizeof(struct qm_frag)) $4 = 99 'c'

Is it okay to run with MEMDBG=1 in production ? Wouldn't it make it a little slow ?

Just let me know if u want so see something else with gdb. Unfortunally I do not have SIP traces, but I have the core file and some logs.

Regards, Dragos

On Thursday, November 21, 2013 6:50 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

can you provde the output in gdb for:

frame 0 p *frag p *((char*)frag + sizeof(struct qm_frag)) There were similar reports, so apparently there is a buffer overflow somewhere.

Cheers, Daniel

On 11/21/13 6:36 PM, Dragos Oancea wrote:

Hello

...

We had this crash today with kamailio 4.0.3 . It ran stable for few weeks until this crash.

GDB here:

http://pastebin.com/rACV31z8

Regards, Dragos

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

Dragos Oancea

6:48 p.m.

And this is the gdb output for another crash from today as well (very likely the same bug): http://pastebin.com/0XU2BRgS

gdb :

(gdb) frame 0 #0 qm_detach_free (qm=0x7f563a226010, size=64) at mem/q_malloc.c:269 269in mem/q_malloc.c (gdb) #0 qm_detach_free (qm=0x7f563a226010, size=64) at mem/q_malloc.c:269 269in mem/q_malloc.c (gdb) p *frag $5 = {size = 64, u = {nxt_free = 0x7f563a3ee550, is_free = 140008321115472}} (gdb) $6 = {size = 64, u = {nxt_free = 0x7f563a3ee550, is_free = 140008321115472}} (gdb) $7 = {size = 64, u = {nxt_free = 0x7f563a3ee550, is_free = 140008321115472}} (gdb) p *((char*)frag + sizeof(struct qm_frag)) $8 = 1 '\001'

Regards, Dragos

On , Dragos Oancea droancea@yahoo.com wrote:

Here is the output:

Is it okay to run with MEMDBG=1 in production ? Wouldn't it make it a little slow ?

Just let me know if u want so see something else with gdb. Unfortunally I do not have SIP traces, but I have the core file and some logs.

Regards, Dragos

On Thursday, November 21, 2013 6:50 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

can you provde the output in gdb for:

frame 0 p *frag p *((char*)frag + sizeof(struct qm_frag)) There were similar reports, so apparently there is a buffer overflow somewhere.

Cheers, Daniel

On 11/21/13 6:36 PM, Dragos Oancea wrote:

Hello

...

We had this crash today with kamailio 4.0.3 . It ran stable for few weeks until this crash.

GDB here:

http://pastebin.com/rACV31z8

Regards, Dragos

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

Daniel-Constantin Mierla

6:58 p.m.

Hello,

the values are invalid, probably the previous fragment was writing more.

For now, can you give:

p *(struct qm_frag_end*)((char*)frag-sizeof(struct qm_frag_end))

I expect to be some invalid values as well.

Later I will try to come up with a gdb script to spot the previous fragment.

MEMDBG=1 will make it slightly slower and increases a bit the overhead. But I guess you don't run at the limits of CPU. In the past we used to have it for couple of releases on and nobody complained about performances.

Cheers, Daniel

On 11/21/13 7:39 PM, Dragos Oancea wrote:

...

Hi

Here is the output:

gdb) (gdb) frame 0 #0 qm_detach_free (qm=0x7fd96175e010, size=112) at mem/q_malloc.c:266 266in mem/q_malloc.c (gdb) #0 qm_detach_free (qm=0x7fd96175e010, size=112) at mem/q_malloc.c:266 266in mem/q_malloc.c (gdb) p *frag $1 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) $2 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) $3 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) p *((char*)frag + sizeof(struct qm_frag)) $4 = 99 'c'

Is it okay to run with MEMDBG=1 in production ? Wouldn't it make it a little slow ?

Just let me know if u want so see something else with gdb. Unfortunally I do not have SIP traces, but I have the core file and some logs.

Regards, Dragos

On Thursday, November 21, 2013 6:50 PM, Daniel-Constantin Mierla miconda@gmail.com wrote: Hello,

can you provde the output in gdb for:

frame 0

p *frag

p *((char*)frag + sizeof(struct qm_frag))

There were similar reports, so apparently there is a buffer overflow somewhere.

You should update to latest git branch 4.0, because there were some other fixes from 4.0.3. With this occasion, you should set MEMDBG=1 in Makefile.defs before recompiling the new version to be able to catch easier the overwrites of memory.

Cheers, Daniel

On 11/21/13 6:36 PM, Dragos Oancea wrote:

...
Hello

We had this crash today with kamailio 4.0.3 . It ran stable for few weeks until this crash.

GDB here:

http://pastebin.com/rACV31z8

Regards, Dragos

Dragos Oancea

22 Nov 22 Nov

10:26 a.m.

Here is the output.

1st core file: (gdb) p *(struct qm_frag_end*)((char*)frag-sizeof(struct qm_frag_end)) $1 = {size = 64, prev_free = 0x7fd961885e90}

2nd core file: (gdb) p *(struct qm_frag_end*)((char*)frag-sizeof(struct qm_frag_end)) $1 = {size = 64, prev_free = 0x7f563a34e0d0}

Regards, Dragos

On Thursday, November 21, 2013 7:58 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

the values are invalid, probably the previous fragment was writing more.

For now, can you give:

p *(struct qm_frag_end*)((char*)frag-sizeof(struct qm_frag_end))

I expect to be some invalid values as well.

Later I will try to come up with a gdb script to spot the previous fragment.

Cheers, Daniel

On 11/21/13 7:39 PM, Dragos Oancea wrote:

...

Here is the output:

gdb) (gdb) frame 0 #0 qm_detach_free (qm=0x7fd96175e010, size=112) at mem/q_malloc.c:266 266in mem/q_malloc.c (gdb) #0 qm_detach_free (qm=0x7fd96175e010, size=112) at mem/q_malloc.c:266 266in mem/q_malloc.c (gdb) p *frag $1 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) $2 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) $3 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) p *((char*)frag + sizeof(struct qm_frag)) $4 = 99 'c'

Is it okay to run with MEMDBG=1 in production ? Wouldn't it make it a little slow ?

Just let me know if u want so see something else with gdb. Unfortunally I do not have SIP traces, but I have the core file and some logs.

Regards, Dragos

On Thursday, November 21, 2013 6:50 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

can you provde the output in gdb for:

frame 0 p *frag p *((char*)frag + sizeof(struct qm_frag))

There were similar reports, so apparently there is a buffer overflow somewhere.

...

You should update to latest git branch 4.0, because

there were some other fixes from 4.0.3. With this occasion, you should set MEMDBG=1 in Makefile.defs before recompiling the new version to be able to catch easier the overwrites of memory.

...

Cheers, Daniel

On 11/21/13 6:36 PM, Dragos Oancea wrote:

Hello

...
We had this crash today with kamailio 4.0.3 . It ran stable for few weeks until this crash.

GDB here:

http://pastebin.com/rACV31z8

Regards, Dragos

Daniel-Constantin Mierla

24 Nov 24 Nov

1:48 p.m.

Hello,

for later references in the archive -- based on private follow up with more details from the core and troubleshooting, the issue should be fixed between 4.0.3 and 4.0.4.

Cheers, Daniel

On 11/22/13 11:26 AM, Dragos Oancea wrote:

...

Hi

Here is the output.

1st core file: (gdb) p *(struct qm_frag_end*)((char*)frag-sizeof(struct qm_frag_end)) $1 = {size = 64, prev_free = 0x7fd961885e90}

2nd core file: (gdb) p *(struct qm_frag_end*)((char*)frag-sizeof(struct qm_frag_end)) $1 = {size = 64, prev_free = 0x7f563a34e0d0}

Regards, Dragos

On Thursday, November 21, 2013 7:58 PM, Daniel-Constantin Mierla miconda@gmail.com wrote: Hello,

the values are invalid, probably the previous fragment was writing more.

For now, can you give:

p *(struct qm_frag_end*)((char*)frag-sizeof(struct qm_frag_end))

I expect to be some invalid values as well.

Later I will try to come up with a gdb script to spot the previous fragment.

MEMDBG=1 will make it slightly slower and increases a bit the overhead. But I guess you don't run at the limits of CPU. In the past we used to have it for couple of releases on and nobody complained about performances.

Cheers, Daniel

On 11/21/13 7:39 PM, Dragos Oancea wrote:

...
Hi

Here is the output:

gdb) (gdb) frame 0 #0 qm_detach_free (qm=0x7fd96175e010, size=112) at mem/q_malloc.c:266 266in mem/q_malloc.c (gdb) #0 qm_detach_free (qm=0x7fd96175e010, size=112) at mem/q_malloc.c:266 266in mem/q_malloc.c (gdb) p *frag $1 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) $2 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) $3 = {size = 7599108840079127868, u = {nxt_free = 0x3965663931343a64, is_free = 4135824228634344036}} (gdb) p *((char*)frag + sizeof(struct qm_frag)) $4 = 99 'c'

Is it okay to run with MEMDBG=1 in production ? Wouldn't it make it a little slow ?

Just let me know if u want so see something else with gdb. Unfortunally I do not have SIP traces, but I have the core file and some logs.

Regards, Dragos

On Thursday, November 21, 2013 6:50 PM, Daniel-Constantin Mierla miconda@gmail.com mailto:miconda@gmail.com wrote: Hello,

can you provde the output in gdb for:

frame 0

p *frag

p *((char*)frag + sizeof(struct qm_frag))

There were similar reports, so apparently there is a buffer overflow somewhere.

You should update to latest git branch 4.0, because there were some other fixes from 4.0.3. With this occasion, you should set MEMDBG=1 in Makefile.defs before recompiling the new version to be able to catch easier the overwrites of memory.

Cheers, Daniel

On 11/21/13 6:36 PM, Dragos Oancea wrote:

...
Hello

We had this crash today with kamailio 4.0.3 . It ran stable for few weeks until this crash.

GDB here:

http://pastebin.com/rACV31z8

Regards, Dragos

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

4174

Age (days ago)

4177

Last active (days ago)

sr-dev@lists.kamailio.org

6 comments

2 participants

tags (0)

participants (2)

Daniel-Constantin Mierla
Dragos Oancea