[SR-Users] PDT ran out of pkg memory using pdt_list

Wed Jul 21 10:56:48 CEST 2010

On 07/20/2010 07:45 PM, JR Richardson wrote:
> On Tue, Jul 20, 2010 at 12:04 PM, Elena-Ramona Modroiu
> <ramona at asipto.com>  wrote:
>    
>> On 07/20/2010 05:06 PM, JR Richardson wrote:
>>      
>>>>>> [..]
>>>>>> When I added 180K records in the database, I got the "no more pkg mem"
>>>>>> error again.
>>>>>> I increased again:
>>>>>> #define PKG_MEM_POOL_SIZE 32*1024*1024
>>>>>> This allowed me to execute pdt_list with 180K records loaded.
>>>>>>
>>>>>> I increased database record count to 240K and got the "no more pkg
>>>>>> mem" error again.
>>>>>>
>>>>>> So I don't think it is prudent to just keep increasing
>>>>>> PKG_MEM_POOL_SIZE.  Is this an architectural limitation with fifo
>>>>>> pkg_mem, shouldn't this be a dynamic allocation if it's not within
>>>>>> shmem and has no affect on core sip-router function?  While the
>>>>>> pdt_reload and pdt_list is going on (takes a few seconds to load and
>>>>>> list), I don't see any problems with the sip-router executions.  I
>>>>>> guess I can just use the old fashion database query to look up routes
>>>>>> instead of fifo pdt_list.
>>>>>>
>>>>>>
>>>>>>              
>>>>> Hi Jr,
>>>>>
>>>>> if you don't use the PKG_MALLOC MM in your system at least the kamailio
>>>>> 1.5
>>>>> will fallback to the system malloc, which don't have this size
>>>>> restrictions.
>>>>> For 3.x/ sr there is an option to build the MI interface with system
>>>>> malloc as
>>>>> well, it this commit:
>>>>>
>>>>> commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec
>>>>> Author: Daniel-Constantin Mierla<miconda at gmail.com>
>>>>> Date:   Mon Jun 29 12:34:08 2009 +0200
>>>>>
>>>>>      lib/kmi: option to build it with system malloc
>>>>>
>>>>>      - MI lib can be built independently of core with system malloc for
>>>>> pkg
>>>>>        memory (see Makefile)
>>>>>
>>>>>
>>>>>            
>>>> this is the option added to overcome the MI design limitations - while
>>>> most
>>>> of dumped structures are in shared memory, MI builds the output in PKG
>>>> memory, resulting in insufficient space since shm is much bigger than pkg
>>>> -
>>>> also this sometime ended as well in lot of mem fragmentation for mi
>>>> process
>>>> in the past.
>>>>
>>>> It needs to edit lib/kmi/Makefile and enable system malloc, recompile and
>>>> reinstall.
>>>>
>>>> Regards,
>>>> Ramona
>>>>
>>>>
>>>>          
>>> Hi Ramona,
>>>
>>> Thanks for your suggestions.  I started over with default mem
>>> parameters in config.h and compiled with:
>>> ## uncomment next line for using system malloc with MI
>>> DEFS+= -DMI_SYSTEM_MALLOC
>>>
>>> I loaded the pdt database table with 360K records and also started
>>> kamailio with '-m 512' but the table still has a max record
>>> limitation, will not load propperly and I get these errors:
>>>   0(22456) ERROR: pdt [pdtree.c:283]: bad parameters
>>>   0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
>>>
>>> Of course wihout a propper table load, pdt_list does not produce any
>>> results (but i don;t get the pkg_mem error any more).
>>>
>>> sip-router2:~# kamctl fifo get_statistics all
>>> shmem:total_size = 536870912
>>> shmem:used_size = 1229472
>>> shmem:real_used_size = 1247560
>>> shmem:max_used_size = 1255376
>>> shmem:free_size = 535623352
>>>
>>> There seems to still be a bottleneck with shmem, even through I have
>>> 512M+ available, I only seem to be using ~1.2M.  Is there an internal
>>> shmem hard limit that can be adjusted?
>>>
>>>        
>> Hi JR,
>>
>> There is no limit hard-coded, I just loaded 1mio records in pdt (sdomain=*,
>> prefix=1mio..2mio, domain=test.org) and statistics show:
>>
>> shmem:total_size = 268435456
>> shmem:used_size = 26594256
>> shmem:real_used_size = 80148548
>> shmem:max_used_size = 80148548
>> shmem:free_size = 188286908
>>
>> So the records are reflected in shared memory. What is the output of
>> "kamailio -V"? What are the parameters for pdt?
>>
>> Regards,
>> Ramona
>>      
> sip-router2:~# kamailio -V
> version: kamailio 3.0.2 (i386/linux) a6141a
> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
> F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> @(#) $Id$
> main.c compiled on 09:25:15 Jul 20 2010 with gcc 4.3.2
> sip-router2:~#
>
> # -----  pdt params -----
> modparam("pdt", "db_url", "mysql://openserro:openserro@localhost/openser")
> modparam("pdt", "char_list", "0123456789")
> modparam("pdt", "check_domain", 0)
> modparam("pdt", "fetch_rows", 5000)
>
> Here is an interesting observation:
> When I have 300K+ records in the pdt table, it seems like it will not
> load properly and the shmem is very low, as expected without a large
> memory pool needed to hold the records.  Statistics show:
> shmem:total_size = 33554432
> shmem:used_size = 1229472
> shmem:real_used_size = 1247512
> shmem:max_used_size = 1247512
> shmem:free_size = 32306920
> shmem:fragments = 11
>
> When I decrease the table recrods to 240K, the complete table loads ok
> and shmem reflect properly:
> shmem:total_size = 33554432
> shmem:used_size = 6349784
> shmem:real_used_size = 8501144
> shmem:max_used_size = 8501144
> shmem:free_size = 25053288
> shmem:fragments = 1
>
> So maybe I'm not having an issue with shmem at all but a loading
> problem with the pdt tabe after a certain record count.
>
> I changed the "modparam("pdt", "fetch_rows", 5000)" to see if that
> would help.  When the table loads properly is is quicker than with the
> default 1000 rows, but that is the only difference I experience.  When
> the pdt table has more than 300K records, it just does not load into
> memory.
>
> Could there be a pdt module timeout that is being met when loading?
> For instance, if it takes more than N seconds to load, consider a
> fault and not load?
>
> sip-router2:/etc/kamailio# mysql -V
> mysql  Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2
> sip-router2:/etc/kamailio#
>    

try to play with mysql module timeout parameter in case the records are 
not loaded fast enough:

http://kamailio.org/docs/modules/stable/modules/db_mysql.html#id2928400

Regards,
Ramona