[SR-Users] PDT ran out of pkg memory using pdt_list

[Elena-Ramona Modroiu]

On 07/20/2010 05:06 PM, JR Richardson wrote:
>>>> [..]
>>>> When I added 180K records in the database, I got the "no more pkg mem"
>>>> error again.
>>>> I increased again:
>>>> #define PKG_MEM_POOL_SIZE 32*1024*1024
>>>> This allowed me to execute pdt_list with 180K records loaded.
>>>>
>>>> I increased database record count to 240K and got the "no more pkg
>>>> mem" error again.
>>>>
>>>> So I don't think it is prudent to just keep increasing
>>>> PKG_MEM_POOL_SIZE.  Is this an architectural limitation with fifo
>>>> pkg_mem, shouldn't this be a dynamic allocation if it's not within
>>>> shmem and has no affect on core sip-router function?  While the
>>>> pdt_reload and pdt_list is going on (takes a few seconds to load and
>>>> list), I don't see any problems with the sip-router executions.  I
>>>> guess I can just use the old fashion database query to look up routes
>>>> instead of fifo pdt_list.
>>>>
>>>>          
>>> Hi Jr,
>>>
>>> if you don't use the PKG_MALLOC MM in your system at least the kamailio
>>> 1.5
>>> will fallback to the system malloc, which don't have this size
>>> restrictions.
>>> For 3.x/ sr there is an option to build the MI interface with system
>>> malloc as
>>> well, it this commit:
>>>
>>> commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec
>>> Author: Daniel-Constantin Mierla<miconda at gmail.com>
>>> Date:   Mon Jun 29 12:34:08 2009 +0200
>>>
>>>      lib/kmi: option to build it with system malloc
>>>
>>>      - MI lib can be built independently of core with system malloc for pkg
>>>        memory (see Makefile)
>>>
>>>        
>> this is the option added to overcome the MI design limitations - while most
>> of dumped structures are in shared memory, MI builds the output in PKG
>> memory, resulting in insufficient space since shm is much bigger than pkg -
>> also this sometime ended as well in lot of mem fragmentation for mi process
>> in the past.
>>
>> It needs to edit lib/kmi/Makefile and enable system malloc, recompile and
>> reinstall.
>>
>> Regards,
>> Ramona
>>
>>      
> Hi Ramona,
>
> Thanks for your suggestions.  I started over with default mem
> parameters in config.h and compiled with:
> ## uncomment next line for using system malloc with MI
> DEFS+= -DMI_SYSTEM_MALLOC
>
> I loaded the pdt database table with 360K records and also started
> kamailio with '-m 512' but the table still has a max record
> limitation, will not load propperly and I get these errors:
>   0(22456) ERROR: pdt [pdtree.c:283]: bad parameters
>   0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
>
> Of course wihout a propper table load, pdt_list does not produce any
> results (but i don;t get the pkg_mem error any more).
>
> sip-router2:~# kamctl fifo get_statistics all
> shmem:total_size = 536870912
> shmem:used_size = 1229472
> shmem:real_used_size = 1247560
> shmem:max_used_size = 1255376
> shmem:free_size = 535623352
>
> There seems to still be a bottleneck with shmem, even through I have
> 512M+ available, I only seem to be using ~1.2M.  Is there an internal
> shmem hard limit that can be adjusted?
>    
Hi JR,

There is no limit hard-coded, I just loaded 1mio records in pdt 
(sdomain=*, prefix=1mio..2mio, domain=test.org) and statistics show:

shmem:total_size = 268435456
shmem:used_size = 26594256
shmem:real_used_size = 80148548
shmem:max_used_size = 80148548
shmem:free_size = 188286908

So the records are reflected in shared memory. What is the output of 
"kamailio -V"? What are the parameters for pdt?

Regards,
Ramona