[SR-Users] PDT ran out of pkg memory using pdt_list

[JR Richardson]

On Tue, Jul 20, 2010 at 12:04 PM, Elena-Ramona Modroiu
<ramona at asipto.com> wrote:
> On 07/20/2010 05:06 PM, JR Richardson wrote:
>>>>>
>>>>> [..]
>>>>> When I added 180K records in the database, I got the "no more pkg mem"
>>>>> error again.
>>>>> I increased again:
>>>>> #define PKG_MEM_POOL_SIZE 32*1024*1024
>>>>> This allowed me to execute pdt_list with 180K records loaded.
>>>>>
>>>>> I increased database record count to 240K and got the "no more pkg
>>>>> mem" error again.
>>>>>
>>>>> So I don't think it is prudent to just keep increasing
>>>>> PKG_MEM_POOL_SIZE.  Is this an architectural limitation with fifo
>>>>> pkg_mem, shouldn't this be a dynamic allocation if it's not within
>>>>> shmem and has no affect on core sip-router function?  While the
>>>>> pdt_reload and pdt_list is going on (takes a few seconds to load and
>>>>> list), I don't see any problems with the sip-router executions.  I
>>>>> guess I can just use the old fashion database query to look up routes
>>>>> instead of fifo pdt_list.
>>>>>
>>>>>
>>>>
>>>> Hi Jr,
>>>>
>>>> if you don't use the PKG_MALLOC MM in your system at least the kamailio
>>>> 1.5
>>>> will fallback to the system malloc, which don't have this size
>>>> restrictions.
>>>> For 3.x/ sr there is an option to build the MI interface with system
>>>> malloc as
>>>> well, it this commit:
>>>>
>>>> commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec
>>>> Author: Daniel-Constantin Mierla<miconda at gmail.com>
>>>> Date:   Mon Jun 29 12:34:08 2009 +0200
>>>>
>>>>     lib/kmi: option to build it with system malloc
>>>>
>>>>     - MI lib can be built independently of core with system malloc for
>>>> pkg
>>>>       memory (see Makefile)
>>>>
>>>>
>>>
>>> this is the option added to overcome the MI design limitations - while
>>> most
>>> of dumped structures are in shared memory, MI builds the output in PKG
>>> memory, resulting in insufficient space since shm is much bigger than pkg
>>> -
>>> also this sometime ended as well in lot of mem fragmentation for mi
>>> process
>>> in the past.
>>>
>>> It needs to edit lib/kmi/Makefile and enable system malloc, recompile and
>>> reinstall.
>>>
>>> Regards,
>>> Ramona
>>>
>>>
>>
>> Hi Ramona,
>>
>> Thanks for your suggestions.  I started over with default mem
>> parameters in config.h and compiled with:
>> ## uncomment next line for using system malloc with MI
>> DEFS+= -DMI_SYSTEM_MALLOC
>>
>> I loaded the pdt database table with 360K records and also started
>> kamailio with '-m 512' but the table still has a max record
>> limitation, will not load propperly and I get these errors:
>>  0(22456) ERROR: pdt [pdtree.c:283]: bad parameters
>>  0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
>>
>> Of course wihout a propper table load, pdt_list does not produce any
>> results (but i don;t get the pkg_mem error any more).
>>
>> sip-router2:~# kamctl fifo get_statistics all
>> shmem:total_size = 536870912
>> shmem:used_size = 1229472
>> shmem:real_used_size = 1247560
>> shmem:max_used_size = 1255376
>> shmem:free_size = 535623352
>>
>> There seems to still be a bottleneck with shmem, even through I have
>> 512M+ available, I only seem to be using ~1.2M.  Is there an internal
>> shmem hard limit that can be adjusted?
>>
>
> Hi JR,
>
> There is no limit hard-coded, I just loaded 1mio records in pdt (sdomain=*,
> prefix=1mio..2mio, domain=test.org) and statistics show:
>
> shmem:total_size = 268435456
> shmem:used_size = 26594256
> shmem:real_used_size = 80148548
> shmem:max_used_size = 80148548
> shmem:free_size = 188286908
>
> So the records are reflected in shared memory. What is the output of
> "kamailio -V"? What are the parameters for pdt?
>
> Regards,
> Ramona

sip-router2:~# kamailio -V
version: kamailio 3.0.2 (i386/linux) a6141a
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
@(#) $Id$
main.c compiled on 09:25:15 Jul 20 2010 with gcc 4.3.2
sip-router2:~#

# -----  pdt params -----
modparam("pdt", "db_url", "mysql://openserro:openserro@localhost/openser")
modparam("pdt", "char_list", "0123456789")
modparam("pdt", "check_domain", 0)
modparam("pdt", "fetch_rows", 5000)

Here is an interesting observation:
When I have 300K+ records in the pdt table, it seems like it will not
load properly and the shmem is very low, as expected without a large
memory pool needed to hold the records.  Statistics show:
shmem:total_size = 33554432
shmem:used_size = 1229472
shmem:real_used_size = 1247512
shmem:max_used_size = 1247512
shmem:free_size = 32306920
shmem:fragments = 11

When I decrease the table recrods to 240K, the complete table loads ok
and shmem reflect properly:
shmem:total_size = 33554432
shmem:used_size = 6349784
shmem:real_used_size = 8501144
shmem:max_used_size = 8501144
shmem:free_size = 25053288
shmem:fragments = 1

So maybe I'm not having an issue with shmem at all but a loading
problem with the pdt tabe after a certain record count.

I changed the "modparam("pdt", "fetch_rows", 5000)" to see if that
would help.  When the table loads properly is is quicker than with the
default 1000 rows, but that is the only difference I experience.  When
the pdt table has more than 300K records, it just does not load into
memory.

Could there be a pdt module timeout that is being met when loading?
For instance, if it takes more than N seconds to load, consider a
fault and not load?

sip-router2:/etc/kamailio# mysql -V
mysql  Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2
sip-router2:/etc/kamailio#

Thanks.

JR
-- 
JR Richardson
Engineering for the Masses