[SR-Users] PDT ran out of pkg memory using pdt_list

[JR Richardson]

>> sip-router2:~# kamailio -V
>> version: kamailio 3.0.2 (i386/linux) a6141a
>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>> F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>> @(#) $Id$
>> main.c compiled on 09:25:15 Jul 20 2010 with gcc 4.3.2
>> sip-router2:~#
>>
>> # -----  pdt params -----
>> modparam("pdt", "db_url", "mysql://openserro:openserro@localhost/openser")
>> modparam("pdt", "char_list", "0123456789")
>> modparam("pdt", "check_domain", 0)
>> modparam("pdt", "fetch_rows", 5000)
>>
>> Here is an interesting observation:
>> When I have 300K+ records in the pdt table, it seems like it will not
>> load properly and the shmem is very low, as expected without a large
>> memory pool needed to hold the records.  Statistics show:
>> shmem:total_size = 33554432
>> shmem:used_size = 1229472
>> shmem:real_used_size = 1247512
>> shmem:max_used_size = 1247512
>> shmem:free_size = 32306920
>> shmem:fragments = 11
>>
>> When I decrease the table recrods to 240K, the complete table loads ok
>> and shmem reflect properly:
>> shmem:total_size = 33554432
>> shmem:used_size = 6349784
>> shmem:real_used_size = 8501144
>> shmem:max_used_size = 8501144
>> shmem:free_size = 25053288
>> shmem:fragments = 1
>>
>> So maybe I'm not having an issue with shmem at all but a loading
>> problem with the pdt tabe after a certain record count.
>>
>> I changed the "modparam("pdt", "fetch_rows", 5000)" to see if that
>> would help.  When the table loads properly is is quicker than with the
>> default 1000 rows, but that is the only difference I experience.  When
>> the pdt table has more than 300K records, it just does not load into
>> memory.
>>
>> Could there be a pdt module timeout that is being met when loading?
>> For instance, if it takes more than N seconds to load, consider a
>> fault and not load?
>>
>> sip-router2:/etc/kamailio# mysql -V
>> mysql  Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using
>> readline 5.2
>> sip-router2:/etc/kamailio#
>>
>
> try to play with mysql module timeout parameter in case the records are not
> loaded fast enough:
>
> http://kamailio.org/docs/modules/stable/modules/db_mysql.html#id2928400
>
> Regards,
> Ramona

Ok, I figured out that sr was taking about 9-10 seconds to load 300K
records so I increased:
 # ----- db_mysql params -----
modparam("db_mysql", "timeout_interval", 20)

And found that the table would load, so I started increasing the
record count by 60K, then the fault condition occurred again at around
400K records, then went away by itself.  I got all the way to 760K
records without a problem then hit another wall at 800K records.

I think there is something flaky going on, I can not reproduce it
reliable, sometimes the table loads just fine and other times it does
not load at all.

I started monitoring mysqld.  When I restart sr or issue pdt_reload
and msqld has a load, the table seems to load fine, regardless of the
table size or "timeout_interval" setting.  When the problem is
occurring, myslqd has zero usage.  When the issue was occurring, I
restarted the mysql process and the table loaded into sr.  I'm now
suspecting I have a buggy mysqld so I'm going to try to update that
and see if I can reproduce the problem.  So for now I can't isolate
the issue, but I'm still working on it.

I do see a difference with the "timeout_interval" during sr restarts,
the longer the time the better it does.  Also there is a time
difference when loading the table, sometimes it takes 30 seconds,
sometimes 90 seconds.  I think maybe there is a combination of issues
going on here.  I need to rebuild a fresh install and test a bit more
methodically to find a root cause.  Also I want to grab a newer
version of mysql-server.

I'll update with my findings.

Thanks.

JR
-- 
JR Richardson
Engineering for the Masses