reload to memory is much slower and has problms after upgrade to 3.3.2 and using mem_join=1

List overview All Threads
Download

newer

older

Accounting to mysql resulted in...

Re: [SR-Users] Logging and...

Uri Shacked

18 Nov 2012 18 Nov '12

4:52 a.m.

Hi,

I am using MTREE and DIALPLAN modules to load lots of info to kamailio. (6 million rows).

When kamailio was running with 3.2.1 (no mem_join=1 option), the used size was increasing but the process of loading the data was fast eanough.

I upgraded to 3.3.2 and set mem_join=1. Now the loading process take about 10 time longer and sometimes stops kamailio from responding to traffic.

Any ideas?

Thanks,

Uri

Attachments:

attachment.html (text/html — 1.9 KB)

Show replies by date

Uri Shacked

18 Nov 18 Nov

10:09 a.m.

After some testing I notice the following: First reload of 5 million records after kamailio started took about 9 sec. Second reload (4 minutes after the first one) took 60 sec. The third one (again about 4 minutes after the secind) got kamailio to use 100% cpu and after 13 minutes! i killed it.....

I can understand that the memory manger works harder, still, any ideas on how to use mem_join and keep on reloading data. (in real life our data loads 5 million records once a day when almost no traffic. still after a few days it stops...)

Thanks, Uri

On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked ushacked@gmail.com wrote:

...

Hi,

I am using MTREE and DIALPLAN modules to load lots of info to kamailio. (6 million rows).

When kamailio was running with 3.2.1 (no mem_join=1 option), the used size was increasing but the process of loading the data was fast eanough.

I upgraded to 3.3.2 and set mem_join=1. Now the loading process take about 10 time longer and sometimes stops kamailio from responding to traffic.

Any ideas?

Thanks,

Uri

Daniel-Constantin Mierla

19 Nov 19 Nov

3:50 a.m.

Hello,

if you set memjoin to 0, do you see any difference?

Can you try again (with memjoin 1 as well as 0) and send the output of:

kamctl mi get_statistics shmem:

before executing the reload commands?

When it gets to 100%, can you see which process is using the cpu and attach to it with:

gdb /path/to/kamailio PID

then do:

bt full

and send output here?

Cheers, Daniel

On 11/18/12 4:09 PM, Uri Shacked wrote:

...

After some testing I notice the following: First reload of 5 million records after kamailio started took about 9 sec. Second reload (4 minutes after the first one) took 60 sec. The third one (again about 4 minutes after the secind) got kamailio to use 100% cpu and after 13 minutes! i killed it..... I can understand that the memory manger works harder, still, any ideas on how to use mem_join and keep on reloading data. (in real life our data loads 5 million records once a day when almost no traffic. still after a few days it stops...) Thanks, Uri

On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked <ushacked@gmail.com mailto:ushacked@gmail.com> wrote:
Hi,

I am using MTREE and DIALPLAN modules to load lots of info to
kamailio. (6 million rows).

When kamailio was running with 3.2.1 (no mem_join=1 option), the
used size was increasing but the process of loading the data was
fast eanough.

I upgraded to 3.3.2 and set mem_join=1. Now the loading process
take about 10 time longer and sometimes stops kamailio from
responding to traffic.

Any ideas?

Thanks,

Uri
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

20 Nov 20 Nov

8:57 a.m.

Daniel hi,

I attached 2 txt files. One with mem_join=1, the other with mem_join=0, and the info you asked for. Let me know if it is OK.

Thanks, Uri

On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...

Hello,

if you set memjoin to 0, do you see any difference?

Can you try again (with memjoin 1 as well as 0) and send the output of:

kamctl mi get_statistics shmem:

before executing the reload commands?

When it gets to 100%, can you see which process is using the cpu and attach to it with:

gdb /path/to/kamailio PID

then do:

bt full

and send output here?

Cheers, Daniel

On 11/18/12 4:09 PM, Uri Shacked wrote:

After some testing I notice the following: First reload of 5 million records after kamailio started took about 9 sec. Second reload (4 minutes after the first one) took 60 sec. The third one (again about 4 minutes after the secind) got kamailio to use 100% cpu and after 13 minutes! i killed it.....

I can understand that the memory manger works harder, still, any ideas on how to use mem_join and keep on reloading data. (in real life our data loads 5 million records once a day when almost no traffic. still after a few days it stops...)

Thanks, Uri

On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked ushacked@gmail.com wrote:

...
Hi,

I am using MTREE and DIALPLAN modules to load lots of info to kamailio. (6 million rows).

When kamailio was running with 3.2.1 (no mem_join=1 option), the used size was increasing but the process of loading the data was fast eanough.

I upgraded to 3.3.2 and set mem_join=1. Now the loading process take about 10 time longer and sometimes stops kamailio from responding to traffic.

Any ideas?

Thanks,

Uri

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

11:26 a.m.

Hello,

ok, I will look over it. At this moment the f_malloc (which is enabled for 3.3) has a pretty inefficient mem join implementation, can you try with q_malloc? Edit Makefile.defs and set:

MEMDBG=1

Then compile and install.

The join operation should be faster, let's see if you get blocking issues with this one.

Cheers, Daniel

On 11/20/12 2:57 PM, Uri Shacked wrote:

...

Daniel hi, I attached 2 txt files. One with mem_join=1, the other with mem_join=0, and the info you asked for. Let me know if it is OK. Thanks, Uri

On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,

if you set memjoin to 0, do you see any difference?

Can you try again (with memjoin 1 as well as 0) and send the
output of:

kamctl mi get_statistics shmem:

before executing the reload commands?

When it gets to 100%, can you see which process is using the cpu
and attach to it with:

gdb /path/to/kamailio PID

then do:

bt full

and send output here?

Cheers,
Daniel


On 11/18/12 4:09 PM, Uri Shacked wrote:

...

After some testing I notice the following:
First reload of 5 million records after kamailio started took
about 9 sec.
Second reload (4 minutes after the first one) took 60 sec.
The third one (again about 4 minutes after the secind) got
kamailio to use 100% cpu and after 13 minutes! i killed it.....
I can understand that the memory manger works harder, still, any
ideas on how to use mem_join and keep on reloading data.
(in real life our data loads 5 million records once a day when
almost no traffic. still after a few days it stops...)
Thanks,
Uri


On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked <ushacked@gmail.com
<mailto:ushacked@gmail.com>> wrote:

    Hi,

    I am using MTREE and DIALPLAN modules to load lots of info to
    kamailio. (6 million rows).

    When kamailio was running with 3.2.1 (no mem_join=1 option),
    the used size was increasing but the process of loading the
    data was fast eanough.

    I upgraded to 3.3.2 and set mem_join=1. Now the loading
    process take about 10 time longer and sometimes stops
    kamailio from responding to traffic.

    Any ideas?

    Thanks,

    Uri




_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

1:34 p.m.

Hi,

can you be a litle more specific of the steps of the install and where do i make the changes? some words of what is the diff between f_malloc and q_malloc will be great :-).

thanks, Uri

On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla <miconda@gmail.com

...

wrote:

...

Hello,

ok, I will look over it. At this moment the f_malloc (which is enabled for 3.3) has a pretty inefficient mem join implementation, can you try with q_malloc? Edit Makefile.defs and set:

MEMDBG=1

Then compile and install.

The join operation should be faster, let's see if you get blocking issues with this one.

Cheers, Daniel

On 11/20/12 2:57 PM, Uri Shacked wrote:

Daniel hi,

I attached 2 txt files. One with mem_join=1, the other with mem_join=0, and the info you asked for. Let me know if it is OK.

Thanks, Uri

On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

if you set memjoin to 0, do you see any difference?

Can you try again (with memjoin 1 as well as 0) and send the output of:

kamctl mi get_statistics shmem:

before executing the reload commands?

When it gets to 100%, can you see which process is using the cpu and attach to it with:

gdb /path/to/kamailio PID

then do:

bt full

and send output here?

Cheers, Daniel

On 11/18/12 4:09 PM, Uri Shacked wrote:

After some testing I notice the following: First reload of 5 million records after kamailio started took about 9 sec. Second reload (4 minutes after the first one) took 60 sec. The third one (again about 4 minutes after the secind) got kamailio to use 100% cpu and after 13 minutes! i killed it.....

I can understand that the memory manger works harder, still, any ideas on how to use mem_join and keep on reloading data. (in real life our data loads 5 million records once a day when almost no traffic. still after a few days it stops...)

Thanks, Uri

On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked ushacked@gmail.com wrote:

...
Hi,

I am using MTREE and DIALPLAN modules to load lots of info to kamailio. (6 million rows).

When kamailio was running with 3.2.1 (no mem_join=1 option), the used size was increasing but the process of loading the data was fast eanough.

I upgraded to 3.3.2 and set mem_join=1. Now the loading process take about 10 time longer and sometimes stops kamailio from responding to traffic.

Any ideas?

Thanks,

Uri

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

2:45 p.m.

Hello,

On 11/20/12 7:34 PM, Uri Shacked wrote:

...

Hi, can you be a litle more specific of the steps of the install and where do i make the changes?

in the source tree, edit the file Makefile.defs and set:

MEMDBG=1

then run:

make all make install

...

some words of what is the diff between f_malloc and q_malloc will be great :-).

q_malloc is more debugging purposes, keeping more information for each chunk, therefore the overhead is a bit higher than with f_malloc, but because keeps more details, it is faster to find the fragments that can be joined.

Cheers, Daniel

...

thanks, Uri

On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,

ok, I will look over it. At this moment the f_malloc (which is
enabled for 3.3) has a pretty inefficient mem join implementation,
can you try with q_malloc? Edit Makefile.defs and set:

MEMDBG=1

Then compile and install.

The join operation should be faster, let's see if you get blocking
issues with this one.

Cheers,
Daniel


On 11/20/12 2:57 PM, Uri Shacked wrote:

...

Daniel hi,
I attached 2 txt files.
One with mem_join=1, the other with mem_join=0, and the info you
asked for.
Let me know if it is OK.
Thanks,
Uri

On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,

    if you set memjoin to 0, do you see any difference?

    Can you try again (with memjoin 1 as well as 0) and send the
    output of:

    kamctl mi get_statistics shmem:

    before executing the reload commands?

    When it gets to 100%, can you see which process is using the
    cpu and attach to it with:

    gdb /path/to/kamailio PID

    then do:

    bt full

    and send output here?

    Cheers,
    Daniel


    On 11/18/12 4:09 PM, Uri Shacked wrote:

...

    After some testing I notice the following:
    First reload of 5 million records after kamailio started
    took about 9 sec.
    Second reload (4 minutes after the first one) took 60 sec.
    The third one (again about 4 minutes after the secind) got
    kamailio to use 100% cpu and after 13 minutes! i killed it.....
    I can understand that the memory manger works harder, still,
    any ideas on how to use mem_join and keep on reloading data.
    (in real life our data loads 5 million records once a day
    when almost no traffic. still after a few days it stops...)
    Thanks,
    Uri


    On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked
    <ushacked@gmail.com <mailto:ushacked@gmail.com>> wrote:

        Hi,

        I am using MTREE and DIALPLAN modules to load lots of
        info to kamailio. (6 million rows).

        When kamailio was running with 3.2.1 (no mem_join=1
        option), the used size was increasing but the process of
        loading the data was fast eanough.

        I upgraded to 3.3.2 and set mem_join=1. Now the loading
        process take about 10 time longer and sometimes stops
        kamailio from responding to traffic.

        Any ideas?

        Thanks,

        Uri




    _______________________________________________
    SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
    sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
    http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

21 Nov 21 Nov

7:33 a.m.

Hi,

I recompiled with MEMDBG=1 and installed. here are the results for reloading 5 million rows with MTREE:

mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around 1.2Gb. mem_join=0 --> takes 10 seconds and the real_used_size of shmem is around 2Gb does it seems normal? 56 seconds is a lot of time......

by the way, when the f_malloc was used, the size of the real_used shmem was twice smaller.

Thanks. On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla <miconda@gmail.com

...

wrote:

...

Hello,

On 11/20/12 7:34 PM, Uri Shacked wrote:

Hi,

can you be a litle more specific of the steps of the install and where do i make the changes?

in the source tree, edit the file Makefile.defs and set:

MEMDBG=1

then run:

make all make install

some words of what is the diff between f_malloc and q_malloc will be great :-).

q_malloc is more debugging purposes, keeping more information for each chunk, therefore the overhead is a bit higher than with f_malloc, but because keeps more details, it is faster to find the fragments that can be joined.

Cheers, Daniel

thanks, Uri

On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

ok, I will look over it. At this moment the f_malloc (which is enabled for 3.3) has a pretty inefficient mem join implementation, can you try with q_malloc? Edit Makefile.defs and set:

MEMDBG=1

Then compile and install.

The join operation should be faster, let's see if you get blocking issues with this one.

Cheers, Daniel

On 11/20/12 2:57 PM, Uri Shacked wrote:

Daniel hi,

I attached 2 txt files. One with mem_join=1, the other with mem_join=0, and the info you asked for. Let me know if it is OK.

Thanks, Uri

On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

if you set memjoin to 0, do you see any difference?

Can you try again (with memjoin 1 as well as 0) and send the output of:

kamctl mi get_statistics shmem:

before executing the reload commands?

When it gets to 100%, can you see which process is using the cpu and attach to it with:

gdb /path/to/kamailio PID

then do:

bt full

and send output here?

Cheers, Daniel

On 11/18/12 4:09 PM, Uri Shacked wrote:

After some testing I notice the following: First reload of 5 million records after kamailio started took about 9 sec. Second reload (4 minutes after the first one) took 60 sec. The third one (again about 4 minutes after the secind) got kamailio to use 100% cpu and after 13 minutes! i killed it.....

I can understand that the memory manger works harder, still, any ideas on how to use mem_join and keep on reloading data. (in real life our data loads 5 million records once a day when almost no traffic. still after a few days it stops...)

Thanks, Uri

On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked ushacked@gmail.comwrote:

...
Hi,

I am using MTREE and DIALPLAN modules to load lots of info to kamailio. (6 million rows).

When kamailio was running with 3.2.1 (no mem_join=1 option), the used size was increasing but the process of loading the data was fast eanough.

I upgraded to 3.3.2 and set mem_join=1. Now the loading process take about 10 time longer and sometimes stops kamailio from responding to traffic.

Any ideas?

Thanks,

Uri

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

1:55 p.m.

Hello,

On 11/21/12 1:33 PM, Uri Shacked wrote:

...

Hi, I recompiled with MEMDBG=1 and installed. here are the results for reloading 5 million rows with MTREE: mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around 1.2Gb. mem_join=0 --> takes 10 seconds and the real_used_size of shmem is around 2Gb does it seems normal? 56 seconds is a lot of time......

the join is done for the free operation, meaning that most of the time is spent when freeing the old tree from memory. The new values will be used after loading the database records, then the old tree is destroyed (this involves the join operation). Also, the sip routing is not affected, loading the new records and destroying old memory tree is done in the MI/RPC process.

In other words, while the MI/RPC process takes care of loading new data and destroying the old one, the SIP routing is not affected at all.

Even when the reload command is executed, the old tree is used until all the records are loaded in a new tree. At that moment, the pointer to the active tree is changed from the old tree to the new tree (a very simple sequence of assignments, very fast). Routing will use the new tree and the Mi/RPC process starts destroying the old tree.

...

by the way, when the f_malloc was used, the size of the real_used shmem was twice smaller.

The overhead when storing small values is significant for q_malloc, each fragment keeping references (pointers) to file name and line where it was allocated and freed. In addition it keeps information to get to previews and next fragment, resulting in faster join.

It is some space to improve, in order to make less overhead (like a compile time option not to keep info about file and line of malloc/free). I will think about what can be done for the next release.

Btw, is the reload time constant now? Even if you run couple of times?

What are the values for 'kamctl mi get_statistics shmem:'?

Cheers, Daniel

...

Thanks. On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,


On 11/20/12 7:34 PM, Uri Shacked wrote:

...

Hi,
can you be a litle more specific of the steps of the install and
where do i make the changes?

in the source tree, edit the file Makefile.defs and set:

MEMDBG=1

then run:

make all
make install

...

some words of what is the diff between f_malloc and q_malloc will
be great :-).

q_malloc is more debugging purposes, keeping more information for
each chunk, therefore the overhead is a bit higher than with
f_malloc, but because keeps more details, it is faster to find the
fragments that can be joined.

Cheers,
Daniel

...

thanks,
Uri

On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,

    ok, I will look over it. At this moment the f_malloc (which
    is enabled for 3.3) has a pretty inefficient mem join
    implementation, can you try with q_malloc? Edit Makefile.defs
    and set:

    MEMDBG=1

    Then compile and install.

    The join operation should be faster, let's see if you get
    blocking issues with this one.

    Cheers,
    Daniel


    On 11/20/12 2:57 PM, Uri Shacked wrote:

...

    Daniel hi,
    I attached 2 txt files.
    One with mem_join=1, the other with mem_join=0, and the info
    you asked for.
    Let me know if it is OK.
    Thanks,
    Uri

    On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla
    <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

        Hello,

        if you set memjoin to 0, do you see any difference?

        Can you try again (with memjoin 1 as well as 0) and send
        the output of:

        kamctl mi get_statistics shmem:

        before executing the reload commands?

        When it gets to 100%, can you see which process is using
        the cpu and attach to it with:

        gdb /path/to/kamailio PID

        then do:

        bt full

        and send output here?

        Cheers,
        Daniel


        On 11/18/12 4:09 PM, Uri Shacked wrote:

...

        After some testing I notice the following:
        First reload of 5 million records after kamailio
        started took about 9 sec.
        Second reload (4 minutes after the first one) took 60 sec.
        The third one (again about 4 minutes after the
        secind) got kamailio to use 100% cpu and after 13
        minutes! i killed it.....
        I can understand that the memory manger works harder,
        still, any ideas on how to use mem_join and keep on
        reloading data.
        (in real life our data loads 5 million records once a
        day when almost no traffic. still after a few days it
        stops...)
        Thanks,
        Uri


        On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked
        <ushacked@gmail.com <mailto:ushacked@gmail.com>> wrote:

            Hi,

            I am using MTREE and DIALPLAN modules to load lots
            of info to kamailio. (6 million rows).

            When kamailio was running with 3.2.1 (no mem_join=1
            option), the used size was increasing but the
            process of loading the data was fast eanough.

            I upgraded to 3.3.2 and set mem_join=1. Now the
            loading process take about 10 time longer and
            sometimes stops kamailio from responding to traffic.

            Any ideas?

            Thanks,

            Uri




        _______________________________________________
        SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
        sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
        http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

        -- 
        Daniel-Constantin Mierla -http://www.asipto.com
        http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

22 Nov 22 Nov

2:51 a.m.

Hi,

...

...
Btw, is the reload time constant now? Even if you run couple of times?

yes, the reload time is constant - 56 sec. i tested for 100 reloads in an hour and it was OK.

...

...
What are the values for 'kamctl mi get_statistics shmem:'?

i configured kamailio to start with 4Gb and after reload the shmem (real_used) take around 30% of it. but, after 20 reloads it grows in 1%. so, after the 100 reloads the real_used take around 34%-35% of shmem. i made the choise to compile again with f_malloc and not use mem_join. the reloads are faster, it uses less real_size (12% and not 30%) and the increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i will keep track on it and update.

thanks a lot.

Btw, what do you think? would you use f_malloc with no mem_join or q_malloc with join?

BR, Uri

On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla <miconda@gmail.com

...

wrote:

...

Hello,

On 11/21/12 1:33 PM, Uri Shacked wrote:

Hi,

I recompiled with MEMDBG=1 and installed. here are the results for reloading 5 million rows with MTREE:

mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around 1.2Gb. mem_join=0 --> takes 10 seconds and the real_used_size of shmem is around 2Gb does it seems normal? 56 seconds is a lot of time......

the join is done for the free operation, meaning that most of the time is spent when freeing the old tree from memory. The new values will be used after loading the database records, then the old tree is destroyed (this involves the join operation). Also, the sip routing is not affected, loading the new records and destroying old memory tree is done in the MI/RPC process.

In other words, while the MI/RPC process takes care of loading new data and destroying the old one, the SIP routing is not affected at all.

Even when the reload command is executed, the old tree is used until all the records are loaded in a new tree. At that moment, the pointer to the active tree is changed from the old tree to the new tree (a very simple sequence of assignments, very fast). Routing will use the new tree and the Mi/RPC process starts destroying the old tree.

by the way, when the f_malloc was used, the size of the real_used shmem was twice smaller.

The overhead when storing small values is significant for q_malloc, each fragment keeping references (pointers) to file name and line where it was allocated and freed. In addition it keeps information to get to previews and next fragment, resulting in faster join.

It is some space to improve, in order to make less overhead (like a compile time option not to keep info about file and line of malloc/free). I will think about what can be done for the next release.

Btw, is the reload time constant now? Even if you run couple of times?

What are the values for 'kamctl mi get_statistics shmem:'?

Cheers, Daniel

Thanks. On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

On 11/20/12 7:34 PM, Uri Shacked wrote:

Hi,

can you be a litle more specific of the steps of the install and where do i make the changes?

in the source tree, edit the file Makefile.defs and set:

MEMDBG=1

then run:

make all make install

some words of what is the diff between f_malloc and q_malloc will be great :-).

q_malloc is more debugging purposes, keeping more information for each chunk, therefore the overhead is a bit higher than with f_malloc, but because keeps more details, it is faster to find the fragments that can be joined.

Cheers, Daniel

thanks, Uri

On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

ok, I will look over it. At this moment the f_malloc (which is enabled for 3.3) has a pretty inefficient mem join implementation, can you try with q_malloc? Edit Makefile.defs and set:

MEMDBG=1

Then compile and install.

The join operation should be faster, let's see if you get blocking issues with this one.

Cheers, Daniel

On 11/20/12 2:57 PM, Uri Shacked wrote:

Daniel hi,

I attached 2 txt files. One with mem_join=1, the other with mem_join=0, and the info you asked for. Let me know if it is OK.

Thanks, Uri

On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

if you set memjoin to 0, do you see any difference?

Can you try again (with memjoin 1 as well as 0) and send the output of:

kamctl mi get_statistics shmem:

before executing the reload commands?

When it gets to 100%, can you see which process is using the cpu and attach to it with:

gdb /path/to/kamailio PID

then do:

bt full

and send output here?

Cheers, Daniel

On 11/18/12 4:09 PM, Uri Shacked wrote:

After some testing I notice the following: First reload of 5 million records after kamailio started took about 9 sec. Second reload (4 minutes after the first one) took 60 sec. The third one (again about 4 minutes after the secind) got kamailio to use 100% cpu and after 13 minutes! i killed it.....

I can understand that the memory manger works harder, still, any ideas on how to use mem_join and keep on reloading data. (in real life our data loads 5 million records once a day when almost no traffic. still after a few days it stops...)

Thanks, Uri

On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked ushacked@gmail.comwrote:

...
Hi,

I am using MTREE and DIALPLAN modules to load lots of info to kamailio. (6 million rows).

When kamailio was running with 3.2.1 (no mem_join=1 option), the used size was increasing but the process of loading the data was fast eanough.

I upgraded to 3.3.2 and set mem_join=1. Now the loading process take about 10 time longer and sometimes stops kamailio from responding to traffic.

Any ideas?

Thanks,

Uri

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

26 Nov 26 Nov

10:43 a.m.

Hello,

On 11/22/12 8:51 AM, Uri Shacked wrote:

...

Hi,

...
...
Btw, is the reload time constant now? Even if you run couple of times?

yes, the reload time is constant - 56 sec. i tested for 100 reloads in an hour and it was OK.

ok - it's 56sec or 36sec?

...

...
...
What are the values for 'kamctl mi get_statistics shmem:'?

i configured kamailio to start with 4Gb and after reload the shmem (real_used) take around 30% of it. but, after 20 reloads it grows in 1%. so, after the 100 reloads the real_used take around 34%-35% of shmem. i made the choise to compile again with f_malloc and not use mem_join. the reloads are faster, it uses less real_size (12% and not 30%) and the increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i will keep track on it and update. thanks a lot. Btw, what do you think? would you use f_malloc with no mem_join or q_malloc with join?

Can you try with master branch? I switched to q_malloc with no debug info. That means the overhead should be lower and the joining faster.

You can do it also with 3.3, you have to edit Makefile.defs: - be sure MEMDBG=0 - replace the next block:

ifeq ($(MEMDBG), 1) C_DEFS+= -DDBG_QM_MALLOC C_DEFS+= -DMEM_JOIN_FREE else C_DEFS+= -DF_MALLOC C_DEFS+= -DMEM_JOIN_FREE endif

with:

ifeq ($(MEMDBG), 1) C_DEFS+= -DDBG_QM_MALLOC C_DEFS+= -DMEM_JOIN_FREE else C_DEFS+= -DMEM_JOIN_FREE endif

Practically is removal of line C_DEFS+= -DF_MALLOC

Then recompile and reinstall..

Cheers, Daniel

...

BR, Uri

On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,


On 11/21/12 1:33 PM, Uri Shacked wrote:

...

Hi,
I recompiled with MEMDBG=1 and installed.
here are the results for reloading 5 million rows with MTREE:
mem_join=1 -->takes 56 seconds and the real_used_size of shmem is
around 1.2Gb.
mem_join=0 --> takes 10 seconds and the real_used_size of shmem
is around 2Gb
does it seems normal?
56 seconds is a lot of time......

the join is done for the free operation, meaning that most of the
time is spent when freeing the old tree from memory. The new
values will be used after loading the database records, then the
old tree is destroyed (this involves the join operation). Also,
the sip routing is not affected, loading the new records and
destroying old memory tree is done in the MI/RPC process.

In other words, while the MI/RPC process takes care of loading new
data and destroying the old one, the SIP routing is not affected
at all.

Even when the reload command is executed, the old tree is used
until all the records are loaded in a new tree. At that moment,
the pointer to the active tree is changed from the old tree to the
new tree (a very simple sequence of assignments, very fast).
Routing will use the new tree and the Mi/RPC process starts
destroying the old tree.

...

by the way, when the f_malloc was used, the size of the real_used
shmem was twice smaller.

The overhead when storing small values is significant for
q_malloc, each fragment keeping references (pointers) to file name
and line where it was allocated and freed. In addition it keeps
information to get to previews and next fragment, resulting in
faster join.

It is some space to improve, in order to make less overhead (like
a compile time option not to keep info about file and line of
malloc/free). I will think about what can be done for the next
release.

Btw, is the reload time constant now? Even if you run couple of times?

What are the values for 'kamctl mi get_statistics shmem:'?

Cheers,
Daniel

...

Thanks.
On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,


    On 11/20/12 7:34 PM, Uri Shacked wrote:

...

    Hi,
    can you be a litle more specific of the steps of the install
    and where do i make the changes?

    in the source tree, edit the file Makefile.defs and set:

    MEMDBG=1

    then run:

    make all
    make install

...

    some words of what is the diff between f_malloc and q_malloc
    will be great :-).

    q_malloc is more debugging purposes, keeping more information
    for each chunk, therefore the overhead is a bit higher than
    with f_malloc, but because keeps more details, it is faster
    to find the fragments that can be joined.

    Cheers,
    Daniel

...

    thanks,
    Uri

    On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla
    <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

        Hello,

        ok, I will look over it. At this moment the f_malloc
        (which is enabled for 3.3) has a pretty inefficient mem
        join implementation, can you try with q_malloc? Edit
        Makefile.defs and set:

        MEMDBG=1

        Then compile and install.

        The join operation should be faster, let's see if you
        get blocking issues with this one.

        Cheers,
        Daniel


        On 11/20/12 2:57 PM, Uri Shacked wrote:

...

        Daniel hi,
        I attached 2 txt files.
        One with mem_join=1, the other with mem_join=0, and the
        info you asked for.
        Let me know if it is OK.
        Thanks,
        Uri

        On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin
        Mierla <miconda@gmail.com <mailto:miconda@gmail.com>>
        wrote:

            Hello,

            if you set memjoin to 0, do you see any difference?

            Can you try again (with memjoin 1 as well as 0) and
            send the output of:

            kamctl mi get_statistics shmem:

            before executing the reload commands?

            When it gets to 100%, can you see which process is
            using the cpu and attach to it with:

            gdb /path/to/kamailio PID

            then do:

            bt full

            and send output here?

            Cheers,
            Daniel


            On 11/18/12 4:09 PM, Uri Shacked wrote:

...

            After some testing I notice the following:
            First reload of 5 million records after kamailio
            started took about 9 sec.
            Second reload (4 minutes after the first one) took
            60 sec.
            The third one (again about 4 minutes after the
            secind) got kamailio to use 100% cpu and after 13
            minutes! i killed it.....
            I can understand that the memory manger works
            harder, still, any ideas on how to use mem_join
            and keep on reloading data.
            (in real life our data loads 5 million
            records once a day when almost no traffic. still
            after a few days it stops...)
            Thanks,
            Uri


            On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked
            <ushacked@gmail.com <mailto:ushacked@gmail.com>>
            wrote:

                Hi,

                I am using MTREE and DIALPLAN modules to load
                lots of info to kamailio. (6 million rows).

                When kamailio was running with 3.2.1 (no
                mem_join=1 option), the used size was
                increasing but the process of loading the data
                was fast eanough.

                I upgraded to 3.3.2 and set mem_join=1. Now
                the loading process take about 10 time longer
                and sometimes stops kamailio from responding
                to traffic.

                Any ideas?

                Thanks,

                Uri




            _______________________________________________
            SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
            sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
            http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

            -- 
            Daniel-Constantin Mierla -http://www.asipto.com
            http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

        -- 
        Daniel-Constantin Mierla -http://www.asipto.com
        http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

27 Nov 27 Nov

5:37 a.m.

...

...
ok - it's 56sec or 36sec?

It was 56 sec.

Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the shmem real used is 25% of 4Gb. When mem_join=, the reload take 30 sec and the shmem real used is 17%.

Looks good with mem_join=1.

Thanks, Uri

On Mon, Nov 26, 2012 at 5:43 PM, Daniel-Constantin Mierla <miconda@gmail.com

...

wrote:

...

Hello,

On 11/22/12 8:51 AM, Uri Shacked wrote:

Hi,

...
...
Btw, is the reload time constant now? Even if you run couple of times?

yes, the reload time is constant - 56 sec. i tested for 100 reloads in an hour and it was OK.

ok - it's 56sec or 36sec?

...
...
What are the values for 'kamctl mi get_statistics shmem:'?

i configured kamailio to start with 4Gb and after reload the shmem (real_used) take around 30% of it. but, after 20 reloads it grows in 1%. so, after the 100 reloads the real_used take around 34%-35% of shmem. i made the choise to compile again with f_malloc and not use mem_join. the reloads are faster, it uses less real_size (12% and not 30%) and the increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i will keep track on it and update.

thanks a lot.

Btw, what do you think? would you use f_malloc with no mem_join or q_malloc with join?

Can you try with master branch? I switched to q_malloc with no debug info. That means the overhead should be lower and the joining faster.

You can do it also with 3.3, you have to edit Makefile.defs:

be sure MEMDBG=0

replace the next block:

ifeq ($(MEMDBG), 1) C_DEFS+= -DDBG_QM_MALLOC C_DEFS+= -DMEM_JOIN_FREE else C_DEFS+= -DF_MALLOC C_DEFS+= -DMEM_JOIN_FREE endif

with:

ifeq ($(MEMDBG), 1) C_DEFS+= -DDBG_QM_MALLOC C_DEFS+= -DMEM_JOIN_FREE else C_DEFS+= -DMEM_JOIN_FREE endif

Practically is removal of line C_DEFS+= -DF_MALLOC

Then recompile and reinstall..

Cheers, Daniel

BR, Uri

On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

On 11/21/12 1:33 PM, Uri Shacked wrote:

Hi,

I recompiled with MEMDBG=1 and installed. here are the results for reloading 5 million rows with MTREE:

mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around 1.2Gb. mem_join=0 --> takes 10 seconds and the real_used_size of shmem is around 2Gb does it seems normal? 56 seconds is a lot of time......

the join is done for the free operation, meaning that most of the time is spent when freeing the old tree from memory. The new values will be used after loading the database records, then the old tree is destroyed (this involves the join operation). Also, the sip routing is not affected, loading the new records and destroying old memory tree is done in the MI/RPC process.

In other words, while the MI/RPC process takes care of loading new data and destroying the old one, the SIP routing is not affected at all.

Even when the reload command is executed, the old tree is used until all the records are loaded in a new tree. At that moment, the pointer to the active tree is changed from the old tree to the new tree (a very simple sequence of assignments, very fast). Routing will use the new tree and the Mi/RPC process starts destroying the old tree.

by the way, when the f_malloc was used, the size of the real_used shmem was twice smaller.

The overhead when storing small values is significant for q_malloc, each fragment keeping references (pointers) to file name and line where it was allocated and freed. In addition it keeps information to get to previews and next fragment, resulting in faster join.

It is some space to improve, in order to make less overhead (like a compile time option not to keep info about file and line of malloc/free). I will think about what can be done for the next release.

Btw, is the reload time constant now? Even if you run couple of times?

What are the values for 'kamctl mi get_statistics shmem:'?

Cheers, Daniel

Thanks. On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

On 11/20/12 7:34 PM, Uri Shacked wrote:

Hi,

can you be a litle more specific of the steps of the install and where do i make the changes?

in the source tree, edit the file Makefile.defs and set:

MEMDBG=1

then run:

make all make install

some words of what is the diff between f_malloc and q_malloc will be great :-).

q_malloc is more debugging purposes, keeping more information for each chunk, therefore the overhead is a bit higher than with f_malloc, but because keeps more details, it is faster to find the fragments that can be joined.

Cheers, Daniel

thanks, Uri

On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

ok, I will look over it. At this moment the f_malloc (which is enabled for 3.3) has a pretty inefficient mem join implementation, can you try with q_malloc? Edit Makefile.defs and set:

MEMDBG=1

Then compile and install.

The join operation should be faster, let's see if you get blocking issues with this one.

Cheers, Daniel

On 11/20/12 2:57 PM, Uri Shacked wrote:

Daniel hi,

I attached 2 txt files. One with mem_join=1, the other with mem_join=0, and the info you asked for. Let me know if it is OK.

Thanks, Uri

On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

if you set memjoin to 0, do you see any difference?

Can you try again (with memjoin 1 as well as 0) and send the output of:

kamctl mi get_statistics shmem:

before executing the reload commands?

When it gets to 100%, can you see which process is using the cpu and attach to it with:

gdb /path/to/kamailio PID

then do:

bt full

and send output here?

Cheers, Daniel

On 11/18/12 4:09 PM, Uri Shacked wrote:

After some testing I notice the following: First reload of 5 million records after kamailio started took about 9 sec. Second reload (4 minutes after the first one) took 60 sec. The third one (again about 4 minutes after the secind) got kamailio to use 100% cpu and after 13 minutes! i killed it.....

I can understand that the memory manger works harder, still, any ideas on how to use mem_join and keep on reloading data. (in real life our data loads 5 million records once a day when almost no traffic. still after a few days it stops...)

Thanks, Uri

On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked ushacked@gmail.comwrote:

...
Hi,

I am using MTREE and DIALPLAN modules to load lots of info to kamailio. (6 million rows).

When kamailio was running with 3.2.1 (no mem_join=1 option), the used size was increasing but the process of loading the data was fast eanough.

I upgraded to 3.3.2 and set mem_join=1. Now the loading process take about 10 time longer and sometimes stops kamailio from responding to traffic.

Any ideas?

Thanks,

Uri

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

3 Mar 3 Mar

12:36 p.m.

Hi,

I have this memory issue again.

The status now is that I compiled kamailio as followed here and use mem_join=1.

I reload my data from DB every minute! I know i reload a lot, will take care of that (will update you all on April :-)).

When kamailio starts the shmem is about 17% of 4Gb and after 7 days (600 reloads per day) the shmem is about 24%.

The traffic is not the issue.

Still, can defrag of memory improve? Ideas for defrag?

Thanks,

Uri

On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked ushacked@gmail.com wrote:

...

OK

...
...
ok - it's 56sec or 36sec?

It was 56 sec.

Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the shmem real used is 25% of 4Gb. When mem_join=, the reload take 30 sec and the shmem real used is 17%.

Looks good with mem_join=1.

Thanks, Uri

On Mon, Nov 26, 2012 at 5:43 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

On 11/22/12 8:51 AM, Uri Shacked wrote:

Hi,

...
...
Btw, is the reload time constant now? Even if you run couple of times?

yes, the reload time is constant - 56 sec. i tested for 100 reloads in an hour and it was OK.

ok - it's 56sec or 36sec?

...
...
What are the values for 'kamctl mi get_statistics shmem:'?

i configured kamailio to start with 4Gb and after reload the shmem (real_used) take around 30% of it. but, after 20 reloads it grows in 1%. so, after the 100 reloads the real_used take around 34%-35% of shmem. i made the choise to compile again with f_malloc and not use mem_join. the reloads are faster, it uses less real_size (12% and not 30%) and the increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i will keep track on it and update.

thanks a lot.

Btw, what do you think? would you use f_malloc with no mem_join or q_malloc with join?

Can you try with master branch? I switched to q_malloc with no debug info. That means the overhead should be lower and the joining faster.

You can do it also with 3.3, you have to edit Makefile.defs:

be sure MEMDBG=0

replace the next block:

ifeq ($(MEMDBG), 1) C_DEFS+= -DDBG_QM_MALLOC C_DEFS+= -DMEM_JOIN_FREE else C_DEFS+= -DF_MALLOC C_DEFS+= -DMEM_JOIN_FREE endif

with:

ifeq ($(MEMDBG), 1) C_DEFS+= -DDBG_QM_MALLOC C_DEFS+= -DMEM_JOIN_FREE else C_DEFS+= -DMEM_JOIN_FREE endif

Practically is removal of line C_DEFS+= -DF_MALLOC

Then recompile and reinstall..

Cheers, Daniel

BR, Uri

On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

On 11/21/12 1:33 PM, Uri Shacked wrote:

Hi,

I recompiled with MEMDBG=1 and installed. here are the results for reloading 5 million rows with MTREE:

mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around 1.2Gb. mem_join=0 --> takes 10 seconds and the real_used_size of shmem is around 2Gb does it seems normal? 56 seconds is a lot of time......

the join is done for the free operation, meaning that most of the time is spent when freeing the old tree from memory. The new values will be used after loading the database records, then the old tree is destroyed (this involves the join operation). Also, the sip routing is not affected, loading the new records and destroying old memory tree is done in the MI/RPC process.

In other words, while the MI/RPC process takes care of loading new data and destroying the old one, the SIP routing is not affected at all.

Even when the reload command is executed, the old tree is used until all the records are loaded in a new tree. At that moment, the pointer to the active tree is changed from the old tree to the new tree (a very simple sequence of assignments, very fast). Routing will use the new tree and the Mi/RPC process starts destroying the old tree.

by the way, when the f_malloc was used, the size of the real_used shmem was twice smaller.

The overhead when storing small values is significant for q_malloc, each fragment keeping references (pointers) to file name and line where it was allocated and freed. In addition it keeps information to get to previews and next fragment, resulting in faster join.

It is some space to improve, in order to make less overhead (like a compile time option not to keep info about file and line of malloc/free). I will think about what can be done for the next release.

Btw, is the reload time constant now? Even if you run couple of times?

What are the values for 'kamctl mi get_statistics shmem:'?

Cheers, Daniel

Thanks. On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

On 11/20/12 7:34 PM, Uri Shacked wrote:

Hi,

can you be a litle more specific of the steps of the install and where do i make the changes?

in the source tree, edit the file Makefile.defs and set:

MEMDBG=1

then run:

make all make install

some words of what is the diff between f_malloc and q_malloc will be great :-).

q_malloc is more debugging purposes, keeping more information for each chunk, therefore the overhead is a bit higher than with f_malloc, but because keeps more details, it is faster to find the fragments that can be joined.

Cheers, Daniel

thanks, Uri

On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

ok, I will look over it. At this moment the f_malloc (which is enabled for 3.3) has a pretty inefficient mem join implementation, can you try with q_malloc? Edit Makefile.defs and set:

MEMDBG=1

Then compile and install.

The join operation should be faster, let's see if you get blocking issues with this one.

Cheers, Daniel

On 11/20/12 2:57 PM, Uri Shacked wrote:

Daniel hi,

I attached 2 txt files. One with mem_join=1, the other with mem_join=0, and the info you asked for. Let me know if it is OK.

Thanks, Uri

On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

if you set memjoin to 0, do you see any difference?

Can you try again (with memjoin 1 as well as 0) and send the output of:

kamctl mi get_statistics shmem:

before executing the reload commands?

When it gets to 100%, can you see which process is using the cpu and attach to it with:

gdb /path/to/kamailio PID

then do:

bt full

and send output here?

Cheers, Daniel

On 11/18/12 4:09 PM, Uri Shacked wrote:

After some testing I notice the following: First reload of 5 million records after kamailio started took about 9 sec. Second reload (4 minutes after the first one) took 60 sec. The third one (again about 4 minutes after the secind) got kamailio to use 100% cpu and after 13 minutes! i killed it.....

I can understand that the memory manger works harder, still, any ideas on how to use mem_join and keep on reloading data. (in real life our data loads 5 million records once a day when almost no traffic. still after a few days it stops...)

Thanks, Uri

On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked ushacked@gmail.comwrote:

> Hi, > > I am using MTREE and DIALPLAN modules to load lots of info to > kamailio. (6 million rows). > > When kamailio was running with 3.2.1 (no mem_join=1 option), the > used size was increasing but the process of loading the data was fast > eanough. > > I upgraded to 3.3.2 and set mem_join=1. Now the loading process take > about 10 time longer and sometimes stops kamailio from responding to > traffic. > > Any ideas? > > Thanks, > > Uri >

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

24 Nov 24 Nov

4:01 a.m.

Hi,

It has been a while.

Still, I have some problems with reloading data into shared memory.

I can obviously notice that there is always an increase of shmem after each reload.

When my system starts, it has 4G of memory reserved and approximately 0.8G is populated at start.

After 30 days, when reloading 15-25 times per day only part of the 0.8G, the shmem size is approximately 3.5G.

As I mentioned before, the traffic does not seems to be the issue. How do I, or can I, check the size of memory occupied by traffic only?

Any ideas how to handle the reloads?

I am using kamailio 3.3.2 with modules like ACC, DIALOG, CARRIERROUTE and more….

Thanks,

BR,

Uri

On Sun, Mar 3, 2013 at 7:36 PM, Uri Shacked ushacked@gmail.com wrote:

...

Hi,

I have this memory issue again.

The status now is that I compiled kamailio as followed here and use mem_join=1.

I reload my data from DB every minute! I know i reload a lot, will take care of that (will update you all on April :-)).

When kamailio starts the shmem is about 17% of 4Gb and after 7 days (600 reloads per day) the shmem is about 24%.

The traffic is not the issue.

Still, can defrag of memory improve? Ideas for defrag?

Thanks,

Uri

On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked ushacked@gmail.com wrote:

...
OK

...
...
ok - it's 56sec or 36sec?

It was 56 sec.

Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the shmem real used is 25% of 4Gb. When mem_join=, the reload take 30 sec and the shmem real used is 17%.

Looks good with mem_join=1.

Thanks, Uri

On Mon, Nov 26, 2012 at 5:43 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

On 11/22/12 8:51 AM, Uri Shacked wrote:

Hi,

...
...
Btw, is the reload time constant now? Even if you run couple of times?

yes, the reload time is constant - 56 sec. i tested for 100 reloads in an hour and it was OK.

ok - it's 56sec or 36sec?

...
...
What are the values for 'kamctl mi get_statistics shmem:'?

i configured kamailio to start with 4Gb and after reload the shmem (real_used) take around 30% of it. but, after 20 reloads it grows in 1%. so, after the 100 reloads the real_used take around 34%-35% of shmem. i made the choise to compile again with f_malloc and not use mem_join. the reloads are faster, it uses less real_size (12% and not 30%) and the increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i will keep track on it and update.

thanks a lot.

Btw, what do you think? would you use f_malloc with no mem_join or q_malloc with join?

Can you try with master branch? I switched to q_malloc with no debug info. That means the overhead should be lower and the joining faster.

You can do it also with 3.3, you have to edit Makefile.defs:

be sure MEMDBG=0

replace the next block:

ifeq ($(MEMDBG), 1) C_DEFS+= -DDBG_QM_MALLOC C_DEFS+= -DMEM_JOIN_FREE else C_DEFS+= -DF_MALLOC C_DEFS+= -DMEM_JOIN_FREE endif

with:

ifeq ($(MEMDBG), 1) C_DEFS+= -DDBG_QM_MALLOC C_DEFS+= -DMEM_JOIN_FREE else C_DEFS+= -DMEM_JOIN_FREE endif

Practically is removal of line C_DEFS+= -DF_MALLOC

Then recompile and reinstall..

Cheers, Daniel

BR, Uri

On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

On 11/21/12 1:33 PM, Uri Shacked wrote:

Hi,

I recompiled with MEMDBG=1 and installed. here are the results for reloading 5 million rows with MTREE:

mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around 1.2Gb. mem_join=0 --> takes 10 seconds and the real_used_size of shmem is around 2Gb does it seems normal? 56 seconds is a lot of time......

the join is done for the free operation, meaning that most of the time is spent when freeing the old tree from memory. The new values will be used after loading the database records, then the old tree is destroyed (this involves the join operation). Also, the sip routing is not affected, loading the new records and destroying old memory tree is done in the MI/RPC process.

In other words, while the MI/RPC process takes care of loading new data and destroying the old one, the SIP routing is not affected at all.

Even when the reload command is executed, the old tree is used until all the records are loaded in a new tree. At that moment, the pointer to the active tree is changed from the old tree to the new tree (a very simple sequence of assignments, very fast). Routing will use the new tree and the Mi/RPC process starts destroying the old tree.

by the way, when the f_malloc was used, the size of the real_used shmem was twice smaller.

The overhead when storing small values is significant for q_malloc, each fragment keeping references (pointers) to file name and line where it was allocated and freed. In addition it keeps information to get to previews and next fragment, resulting in faster join.

It is some space to improve, in order to make less overhead (like a compile time option not to keep info about file and line of malloc/free). I will think about what can be done for the next release.

Btw, is the reload time constant now? Even if you run couple of times?

What are the values for 'kamctl mi get_statistics shmem:'?

Cheers, Daniel

Thanks. On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

On 11/20/12 7:34 PM, Uri Shacked wrote:

Hi,

can you be a litle more specific of the steps of the install and where do i make the changes?

in the source tree, edit the file Makefile.defs and set:

MEMDBG=1

then run:

make all make install

some words of what is the diff between f_malloc and q_malloc will be great :-).

q_malloc is more debugging purposes, keeping more information for each chunk, therefore the overhead is a bit higher than with f_malloc, but because keeps more details, it is faster to find the fragments that can be joined.

Cheers, Daniel

thanks, Uri

On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

ok, I will look over it. At this moment the f_malloc (which is enabled for 3.3) has a pretty inefficient mem join implementation, can you try with q_malloc? Edit Makefile.defs and set:

MEMDBG=1

Then compile and install.

The join operation should be faster, let's see if you get blocking issues with this one.

Cheers, Daniel

On 11/20/12 2:57 PM, Uri Shacked wrote:

Daniel hi,

I attached 2 txt files. One with mem_join=1, the other with mem_join=0, and the info you asked for. Let me know if it is OK.

Thanks, Uri

On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

> Hello, > > if you set memjoin to 0, do you see any difference? > > Can you try again (with memjoin 1 as well as 0) and send the output > of: > > kamctl mi get_statistics shmem: > > before executing the reload commands? > > When it gets to 100%, can you see which process is using the cpu and > attach to it with: > > gdb /path/to/kamailio PID > > then do: > > bt full > > and send output here? > > Cheers, > Daniel > > > On 11/18/12 4:09 PM, Uri Shacked wrote: > > After some testing I notice the following: > First reload of 5 million records after kamailio started took about > 9 sec. > Second reload (4 minutes after the first one) took 60 sec. > The third one (again about 4 minutes after the secind) got kamailio > to use 100% cpu and after 13 minutes! i killed it..... > > I can understand that the memory manger works harder, still, any > ideas on how to use mem_join and keep on reloading data. > (in real life our data loads 5 million records once a day when > almost no traffic. still after a few days it stops...) > > Thanks, > Uri > > > > On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked ushacked@gmail.comwrote: > >> Hi, >> >> I am using MTREE and DIALPLAN modules to load lots of info to >> kamailio. (6 million rows). >> >> When kamailio was running with 3.2.1 (no mem_join=1 option), the >> used size was increasing but the process of loading the data was fast >> eanough. >> >> I upgraded to 3.3.2 and set mem_join=1. Now the loading process >> take about 10 time longer and sometimes stops kamailio from responding to >> traffic. >> >> Any ideas? >> >> Thanks, >> >> Uri >> > > > > _______________________________________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users > > > -- > Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > >

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

25 Nov 25 Nov

3:40 p.m.

Hello,

are you still using mem_join=1?

Cheers, Daniel

On 11/24/13 10:01 AM, Uri Shacked wrote:

...

Hi,

It has been a while.

Still, I have some problems with reloading data into shared memory.

I can obviously notice that there is always an increase of shmem after each reload.

When my system starts, it has 4G of memory reserved and approximately 0.8G is populated at start.

After 30 days, when reloading 15-25 times per day only part of the 0.8G, the shmem size is approximately 3.5G.

As I mentioned before, the traffic does not seems to be the issue. How do I, or can I, check the size of memory occupied by traffic only?

Any ideas how to handle the reloads?

I am using kamailio 3.3.2 with modules like ACC, DIALOG, CARRIERROUTE and more….

Thanks,

BR,

Uri

On Sun, Mar 3, 2013 at 7:36 PM, Uri Shacked <ushacked@gmail.com mailto:ushacked@gmail.com> wrote:
Hi,

I have this memory issue again.

The status now is that I compiled kamailio as followed here and
use mem_join=1.

I reload my data from DB every minute! I know i reload a lot, will
take care of that (will update you all on April :-)).

When kamailio starts the shmem is about 17% of 4Gb and after 7
days (600 reloads per day) the shmem is about 24%.

The traffic is not the issue.

Still, can defrag of memory improve? Ideas for defrag?

Thanks,

Uri


On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked <ushacked@gmail.com
<mailto:ushacked@gmail.com>> wrote:


    OK
    >>ok - it's 56sec or 36sec?
    It was 56 sec.
    Now, after recompile, when cfg mem_join=0 the reload takes 8
    sec and the shmem real used is 25% of 4Gb.
    When mem_join=, the reload take 30 sec and the shmem real used
    is 17%.
    Looks good with mem_join=1.
    Thanks,
    Uri

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

26 Nov 26 Nov

2:38 a.m.

mem_join=1 on cfg. MEMDBG=0 on makefile.def

On Mon, Nov 25, 2013 at 10:40 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...

Hello,

are you still using mem_join=1?

Cheers, Daniel

On 11/24/13 10:01 AM, Uri Shacked wrote:

Hi,

It has been a while.

Still, I have some problems with reloading data into shared memory.

I can obviously notice that there is always an increase of shmem after each reload.

When my system starts, it has 4G of memory reserved and approximately 0.8G is populated at start.

After 30 days, when reloading 15-25 times per day only part of the 0.8G, the shmem size is approximately 3.5G.

As I mentioned before, the traffic does not seems to be the issue. How do I, or can I, check the size of memory occupied by traffic only?

Any ideas how to handle the reloads?

I am using kamailio 3.3.2 with modules like ACC, DIALOG, CARRIERROUTE and more….

Thanks,

BR,

Uri

On Sun, Mar 3, 2013 at 7:36 PM, Uri Shacked ushacked@gmail.com wrote:

...
Hi,

I have this memory issue again.

The status now is that I compiled kamailio as followed here and use mem_join=1.

I reload my data from DB every minute! I know i reload a lot, will take care of that (will update you all on April :-)).

When kamailio starts the shmem is about 17% of 4Gb and after 7 days (600 reloads per day) the shmem is about 24%.

The traffic is not the issue.

Still, can defrag of memory improve? Ideas for defrag?

Thanks,

Uri

On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked ushacked@gmail.comwrote:

...
OK

...
...
ok - it's 56sec or 36sec?

It was 56 sec.

Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the shmem real used is 25% of 4Gb. When mem_join=, the reload take 30 sec and the shmem real used is 17%.

Looks good with mem_join=1.

Thanks, Uri

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

4 Dec 4 Dec

2:29 p.m.

Any ideas?

On Tue, Nov 26, 2013 at 9:38 AM, Uri Shacked ushacked@gmail.com wrote:

...

mem_join=1 on cfg. MEMDBG=0 on makefile.def

On Mon, Nov 25, 2013 at 10:40 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

are you still using mem_join=1?

Cheers, Daniel

On 11/24/13 10:01 AM, Uri Shacked wrote:

Hi,

It has been a while.

Still, I have some problems with reloading data into shared memory.

I can obviously notice that there is always an increase of shmem after each reload.

When my system starts, it has 4G of memory reserved and approximately 0.8G is populated at start.

After 30 days, when reloading 15-25 times per day only part of the 0.8G, the shmem size is approximately 3.5G.

As I mentioned before, the traffic does not seems to be the issue. How do I, or can I, check the size of memory occupied by traffic only?

Any ideas how to handle the reloads?

I am using kamailio 3.3.2 with modules like ACC, DIALOG, CARRIERROUTE and more….

Thanks,

BR,

Uri

On Sun, Mar 3, 2013 at 7:36 PM, Uri Shacked ushacked@gmail.com wrote:

...
Hi,

I have this memory issue again.

The status now is that I compiled kamailio as followed here and use mem_join=1.

I reload my data from DB every minute! I know i reload a lot, will take care of that (will update you all on April :-)).

When kamailio starts the shmem is about 17% of 4Gb and after 7 days (600 reloads per day) the shmem is about 24%.

The traffic is not the issue.

Still, can defrag of memory improve? Ideas for defrag?

Thanks,

Uri

On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked ushacked@gmail.comwrote:

...
OK

...
...
ok - it's 56sec or 36sec?

It was 56 sec.

Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the shmem real used is 25% of 4Gb. When mem_join=, the reload take 30 sec and the shmem real used is 17%.

Looks good with mem_join=1.

Thanks, Uri

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

3:04 p.m.

To refresh, what is the version you are using now? Looking at old messages in this thread is about 3.3.2, but memjoin was not there, iirc.

Can you give the output for next command?

kamctl mi get_statistics shmem:

You can walk the shared memory chunks with gdb, but it may take quite a while and not sure how useful will be.

Cheers, Daniel

On 12/4/13 8:29 PM, Uri Shacked wrote:

...

Any ideas?

On Tue, Nov 26, 2013 at 9:38 AM, Uri Shacked <ushacked@gmail.com mailto:ushacked@gmail.com> wrote:

mem_join=1 on cfg.
MEMDBG=0 on makefile.def


On Mon, Nov 25, 2013 at 10:40 PM, Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,

    are you still using mem_join=1?

    Cheers,
    Daniel


    On 11/24/13 10:01 AM, Uri Shacked wrote:

...

    Hi,

    It has been a while.

    Still, I have some problems with reloading data into shared
    memory.

    I can obviously notice that there is always an increase of
    shmem after each reload.

    When my system starts, it has 4G of memory reserved and
    approximately 0.8G is populated at start.

    After 30 days, when reloading 15-25 times per day only part
    of the 0.8G, the shmem size is approximately 3.5G.

    As I mentioned before, the traffic does not seems to be the
    issue. How do I, or can I, check the size of memory occupied
    by traffic only?

    Any ideas how to handle the reloads?

    I am using kamailio 3.3.2 with modules like ACC, DIALOG,
    CARRIERROUTE and more….

    Thanks,

    BR,

    Uri



    On Sun, Mar 3, 2013 at 7:36 PM, Uri Shacked
    <ushacked@gmail.com <mailto:ushacked@gmail.com>> wrote:


        Hi,

        I have this memory issue again.

        The status now is that I compiled kamailio as followed
        here and use mem_join=1.

        I reload my data from DB every minute! I know i reload a
        lot, will take care of that (will update you all on April
        :-)).

        When kamailio starts the shmem is about 17% of 4Gb and
        after 7 days (600 reloads per day) the shmem is about 24%.

        The traffic is not the issue.

        Still, can defrag of memory improve? Ideas for defrag?

        Thanks,

        Uri


        On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked
        <ushacked@gmail.com <mailto:ushacked@gmail.com>> wrote:


            OK
            >>ok - it's 56sec or 36sec?
            It was 56 sec.
            Now, after recompile, when cfg mem_join=0 the reload
            takes 8 sec and the shmem real used is 25% of 4Gb.
            When mem_join=, the reload take 30 sec and the shmem
            real used is 17%.
            Looks good with mem_join=1.
            Thanks,
            Uri

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Uri Shacked

5 Dec 5 Dec

2:29 a.m.

The version is 3.3.2 a91b2b-dirty

The cpmmand output is:

shmem:fragments = 433 shmem:free_size = 2595411760 shmem:max_used_size = 2358563728 shmem:real_used_size = 1699555536 shmem:total_size = 4294967296 shmem:used_size = 1359869056

Thanks

On Wed, Dec 4, 2013 at 10:04 PM, Daniel-Constantin Mierla <miconda@gmail.com

...

wrote:

...

To refresh, what is the version you are using now? Looking at old messages in this thread is about 3.3.2, but memjoin was not there, iirc.

Can you give the output for next command?

kamctl mi get_statistics shmem:

You can walk the shared memory chunks with gdb, but it may take quite a while and not sure how useful will be.

Cheers, Daniel

On 12/4/13 8:29 PM, Uri Shacked wrote:

Any ideas?

On Tue, Nov 26, 2013 at 9:38 AM, Uri Shacked ushacked@gmail.com wrote:

...
mem_join=1 on cfg. MEMDBG=0 on makefile.def

On Mon, Nov 25, 2013 at 10:40 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...
Hello,

are you still using mem_join=1?

Cheers, Daniel

On 11/24/13 10:01 AM, Uri Shacked wrote:

Hi,

It has been a while.

Still, I have some problems with reloading data into shared memory.

I can obviously notice that there is always an increase of shmem after each reload.

When my system starts, it has 4G of memory reserved and approximately 0.8G is populated at start.

After 30 days, when reloading 15-25 times per day only part of the 0.8G, the shmem size is approximately 3.5G.

As I mentioned before, the traffic does not seems to be the issue. How do I, or can I, check the size of memory occupied by traffic only?

Any ideas how to handle the reloads?

I am using kamailio 3.3.2 with modules like ACC, DIALOG, CARRIERROUTE and more….

Thanks,

BR,

Uri

On Sun, Mar 3, 2013 at 7:36 PM, Uri Shacked ushacked@gmail.com wrote:

...
Hi,

I have this memory issue again.

The status now is that I compiled kamailio as followed here and use mem_join=1.

I reload my data from DB every minute! I know i reload a lot, will take care of that (will update you all on April :-)).

When kamailio starts the shmem is about 17% of 4Gb and after 7 days (600 reloads per day) the shmem is about 24%.

The traffic is not the issue.

Still, can defrag of memory improve? Ideas for defrag?

Thanks,

Uri

On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked ushacked@gmail.comwrote:

...
OK

...
>ok - it's 56sec or 36sec?

It was 56 sec.

Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the shmem real used is 25% of 4Gb. When mem_join=, the reload take 30 sec and the shmem real used is 17%.

Looks good with mem_join=1.

Thanks, Uri

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

ziad habchi

4:32 a.m.

New subject: reload to memory is much slower and has problms after upgrade to 3.3.2 and using mem_join=1

I am having the same issue on version 4.0.4

From: sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Uri Shacked Sent: Thursday, December 5, 2013 9:30 AM To: Daniel-Constantin Mierla Cc: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Subject: Re: [SR-Users] reload to memory is much slower and has problms after upgrade to 3.3.2 and using mem_join=1

The version is 3.3.2 a91b2b-dirty

The cpmmand output is:

shmem:fragments = 433

shmem:free_size = 2595411760

shmem:max_used_size = 2358563728

shmem:real_used_size = 1699555536

shmem:total_size = 4294967296

shmem:used_size = 1359869056

Thanks

On Wed, Dec 4, 2013 at 10:04 PM, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com > wrote:

To refresh, what is the version you are using now? Looking at old messages in this thread is about 3.3.2, but memjoin was not there, iirc.

Can you give the output for next command?

kamctl mi get_statistics shmem:

You can walk the shared memory chunks with gdb, but it may take quite a while and not sure how useful will be.

Cheers, Daniel

On 12/4/13 8:29 PM, Uri Shacked wrote:

Any ideas?

On Tue, Nov 26, 2013 at 9:38 AM, Uri Shacked <ushacked@gmail.com mailto:ushacked@gmail.com > wrote:

mem_join=1 on cfg.

MEMDBG=0 on makefile.def

On Mon, Nov 25, 2013 at 10:40 PM, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com > wrote:

Hello,

are you still using mem_join=1?

Cheers, Daniel

On 11/24/13 10:01 AM, Uri Shacked wrote:

Hi,

It has been a while.

Still, I have some problems with reloading data into shared memory.

I can obviously notice that there is always an increase of shmem after each reload.

When my system starts, it has 4G of memory reserved and approximately 0.8G is populated at start.

After 30 days, when reloading 15-25 times per day only part of the 0.8G, the shmem size is approximately 3.5G.

As I mentioned before, the traffic does not seems to be the issue. How do I, or can I, check the size of memory occupied by traffic only?

Any ideas how to handle the reloads?

I am using kamailio 3.3.2 with modules like ACC, DIALOG, CARRIERROUTE and more..

Thanks,

BR,

Uri

On Sun, Mar 3, 2013 at 7:36 PM, Uri Shacked <ushacked@gmail.com mailto:ushacked@gmail.com > wrote:

Hi,

I have this memory issue again.

The status now is that I compiled kamailio as followed here and use mem_join=1.

I reload my data from DB every minute! I know i reload a lot, will take care of that (will update you all on April :-)).

When kamailio starts the shmem is about 17% of 4Gb and after 7 days (600 reloads per day) the shmem is about 24%.

The traffic is not the issue.

Still, can defrag of memory improve? Ideas for defrag?

Thanks,

Uri

On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked <ushacked@gmail.com mailto:ushacked@gmail.com > wrote:

...

...
ok - it's 56sec or 36sec?

It was 56 sec.

Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the shmem real used is 25% of 4Gb.

When mem_join=, the reload take 30 sec and the shmem real used is 17%.

Looks good with mem_join=1.

Thanks,

Uri

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda http://twitter.com/#%21/miconda - http://www.linkedin.com/in/miconda -- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

4104

Age (days ago)

4486

Last active (days ago)

sr-users@lists.kamailio.org

19 comments

3 participants

tags (0)

participants (3)

Daniel-Constantin Mierla
Uri Shacked
ziad habchi