[SR-Users] Memory leak in 3.3.4

David Cunningham dcunningham at voisonics.com
Tue Oct 1 12:03:05 CEST 2013 

Hello,

Following up on this after a long break. It seems we are still having a
memory leak. We've added the memory debugging options as per the
documentation with the following output.
I'm not sure how to read it though - can anyone help? Thank you.

Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core>
[main.c:754]: Memory status (pkg):
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
(0x4019b008):
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: heap
size= 4194304
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: used=
191100, used+overhead=462800, free=3731504
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max
used (+overhead)= 480224
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: dumping
all alloc'ed. fragments:
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      0.
N  address=0x401cf414 frag=0x401cf3fc size=512 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      1.
N  address=0x401cf644 frag=0x401cf62c size=128 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      2.
N  address=0x401cf6f4 frag=0x401cf6dc size=256 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: counters.c: cnt_hash_add(331)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      3.
N  address=0x401cf824 frag=0x401cf80c size=64 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: counters.c: init_counters(121)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      4.
N  address=0x401cf894 frag=0x401cf87c size=8 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: route.c: init_rlist(180)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      5.
N  address=0x401cf8cc frag=0x401cf8b4 size=64 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      6.
N  address=0x401cf93c frag=0x401cf924 size=24 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: route.c: route_add(159)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      7.
N  address=0x401cf984 frag=0x401cf96c size=8 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: route.c: init_rlist(180)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      8.
N  address=0x401cf9bc frag=0x401cf9a4 size=64 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:      9.
N  address=0x401cfa2c frag=0x401cfa14 size=24 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: route.c: route_add(159)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:     10.
N  address=0x401cfa74 frag=0x401cfa5c size=8 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: db_id.c: dupl_string(50)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:     11.
N  address=0x401cfaac frag=0x401cfa94 size=64 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:     12.
N  address=0x401cfb1c frag=0x401cfb04 size=24 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: route.c: route_add(159)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:     13.
N  address=0x401cfb64 frag=0x401cfb4c size=8 used=1
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:           alloc'd from <core>: route.c: init_rlist(180)
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:     14.
N  address=0x401cfb9c frag=0x401cfb84 size=64 used=1




On 2 August 2013 11:34, David Cunningham <dcunningham at voisonics.com> wrote:

> Hi Daniel,
>
> Thank you, we will try that.
>
>
> On 31 July 2013 21:54, Daniel-Constantin Mierla <miconda at gmail.com> wrote:
>
>>  Hello,
>>
>> I pushed the commit to branch 3.3 now -- forgot about it.
>>
>> The code you pasted below becomes (after the two patches, thus is the
>> final version that should be used):
>>
>>
>>     ENTER;                /* everything created after here */
>>     SAVETMPS;            /* ...is a temporary variable.   */
>>     PUSHMARK(SP);        /* remember the stack pointer    */
>>
>>     m = sv_newmortal();
>>     sv_setref_pv(m, "Kamailio::Message", (void *)_msg);
>>     SvREADONLY_on(SvRV(m));
>>
>>     XPUSHs(m);            /* Our reference to the stack... */
>>
>>     if (mystr)
>>         XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
>>
>>
>> Cheers,
>> Daniel
>>
>>
>> On 7/31/13 1:48 PM, David Cunningham wrote:
>>
>> Hi Daniel,
>>
>> In 3.3.x the code is arranged a little differently and I'd like to make
>> sure we get it right. Could you please advise?
>>
>>
>>         m = sv_newmortal();
>>         sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
>>         SvREADONLY_on(SvRV(m));
>>
>>
>>         ENTER;                          /* everything created after here
>> */
>>         SAVETMPS;                       /* ...is a temporary variable.
>> */
>>         PUSHMARK(SP);                   /* remember the stack pointer
>> */
>>         XPUSHs(m);                      /* Our reference to the stack...
>> */
>>
>>         if (mystr)
>>                 XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
>>                                         /* Our string to the stack... */
>>
>>
>>
>> On 31 July 2013 21:44, David Cunningham <dcunningham at voisonics.com>wrote:
>>
>>> Hi Daniel,
>>>
>>> Thank you, we will try that.
>>>
>>>
>>>
>>> On 31 July 2013 20:54, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>
>>>>  Hello,
>>>>
>>>> revising the patch I noticed I was moving the initialization of the
>>>> variable after pushing it to perl environment (from the perl docs, the
>>>> variable should have been initialized after initializing the environment --
>>>> what I tried to do with previous patch).
>>>>
>>>> There is a new smaller patch to be added:
>>>>
>>>>
>>>> http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plain;h=3935fedf23f3bf2b6675182193cef6af3bbd903a
>>>>
>>>> Practically, the line XPUSHs(m); has to be moved after the line with
>>>> SvREADONLY_on(SvRV(m));
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 7/31/13 8:26 AM, David Cunningham wrote:
>>>>
>>>> Hi Daniel,
>>>>
>>>> We tried that patch, but Kamailio logged lots of errors like the
>>>> following. The undefined value in question is $m, which should be the SIP
>>>> message. Would you have any advice? Thanks.
>>>>
>>>> Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
>>>> [openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an
>>>> undefined value at Foo.pm line 247.#012
>>>>
>>>>
>>>> On 25 July 2013 17:11, David Cunningham <dcunningham at voisonics.com>wrote:
>>>>
>>>>> Hi Daniel,
>>>>>
>>>>> I'll suggest that to the customer. Thank you!
>>>>>
>>>>>
>>>>>
>>>>> On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>>>
>>>>>>  Hello,
>>>>>>
>>>>>> can you try the attached patch? It's the same patch, just for two
>>>>>> versions, one is for 3.3.x and the other for devel version
>>>>>>
>>>>>> It initializes the SIP message variable that is passed to perl after
>>>>>> creating the temporary environment, so it is actually destroyed by the perl
>>>>>> embedded interpreter.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>> On 7/25/13 1:29 AM, David Cunningham wrote:
>>>>>>
>>>>>> Hi Daniel,
>>>>>>
>>>>>> The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
>>>>>> release 5.4. If I remember right programs running under Valgrind can have
>>>>>> issues, so I'm not sure if the customer will want to do that. Ideally we'd
>>>>>> do it on a test system, but I'm not sure if we have any RHEL available.
>>>>>> I'll see what we can do. Thanks again.
>>>>>>
>>>>>>
>>>>>> On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>>>>
>>>>>>>  Hello,
>>>>>>>
>>>>>>> I would say that perl_exec() is the one with the highest chances to
>>>>>>> be the reason for the leak. Next is line would be db_mysql module, if liked
>>>>>>> with some custom mysql client library, although even in this case will be
>>>>>>> unlikely.
>>>>>>>
>>>>>>> Back to perl, the module itself does not call any malloc, so it
>>>>>>> might be the embedding Perl API that is not used properly in the module.
>>>>>>>
>>>>>>> Can you use some testbed, set children=1 and run kamailio under
>>>>>>> valgrind, then do some calls and see if it detects the source of the leak?
>>>>>>>
>>>>>>> I'm not using the perl module, I will try to check it whenever I get
>>>>>>> a chance in the next days. What version of perl do you have installed?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>>
>>>>>>> On 7/24/13 10:31 AM, David Cunningham wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> We don't do any kamctl commands at all. We do have various modules
>>>>>>> loaded, as follows. The primary functions we use Kamailio for are phone
>>>>>>> registrations through usrloc, and routing calls to Asterisk through logic
>>>>>>> contained in Perl via perl_exec().
>>>>>>> Thanks for all your advice so far!
>>>>>>>
>>>>>>> loadmodule "tm.so"
>>>>>>> loadmodule "tmx.so"
>>>>>>> loadmodule "usrloc.so"
>>>>>>> loadmodule "auth.so"
>>>>>>> loadmodule "auth_db.so"
>>>>>>> loadmodule "ctl.so"
>>>>>>> loadmodule "db_mysql.so"
>>>>>>> loadmodule "kex.so"
>>>>>>> loadmodule "maxfwd.so"
>>>>>>> loadmodule "mi_fifo.so"
>>>>>>> loadmodule "mi_rpc.so"
>>>>>>> loadmodule "nathelper.so"
>>>>>>> loadmodule "perl.so"
>>>>>>> loadmodule "pv.so"
>>>>>>> loadmodule "registrar.so"
>>>>>>> loadmodule "rr.so"
>>>>>>> loadmodule "sanity.so"
>>>>>>> loadmodule "siputils.so"
>>>>>>> loadmodule "sl.so"
>>>>>>> loadmodule "textops.so"
>>>>>>> loadmodule "xlog.so"
>>>>>>>
>>>>>>>
>>>>>>> On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>>>>>
>>>>>>>>  Hello,
>>>>>>>>
>>>>>>>>
>>>>>>>> On 7/24/13 4:24 AM, David Cunningham wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>  Thank you very much for the email. In reply:
>>>>>>>>
>>>>>>>> 1. The system ran out of memory. Linux's oom-killer killed Kamailio.
>>>>>>>>
>>>>>>>> then all the instructions I gave are useless, they are for
>>>>>>>> debugging kamailio's internal memory manager, which handles pkg and shm
>>>>>>>> mallocs.
>>>>>>>>
>>>>>>>> The chances to be from kamailio itself are very low now. Do you do
>>>>>>>> lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but
>>>>>>>> the rest of code should use internal memory manager which does not go
>>>>>>>> beyond the limits set with -m and -M, thus not causing an OS memory
>>>>>>>> exhaustion.
>>>>>>>>
>>>>>>>> Can you list what modules are you loading? At some point it was a
>>>>>>>> leak in libssl, in case you use tls a lot. But could be another external
>>>>>>>> library...
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2. You're right, DEBUG_MEMORY is a local configuration setting. If
>>>>>>>> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
>>>>>>>>
>>>>>>>> 3. We'll try setting mem_summary=12, thanks.
>>>>>>>>
>>>>>>>> 4. We'll try setting asynchronous syslog, thanks.
>>>>>>>>
>>>>>>>> 5.  Our configuration totals 338 lines, or approx 8.5kb. Is that a
>>>>>>>> lot?
>>>>>>>>
>>>>>>>> 6. We'll try setting mem_join=1, thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> first, to clarify, is the system memory or kamailio's pkg/shm
>>>>>>>>> memory running out? If the operating system runs out of memory, then should
>>>>>>>>> be a leak in a library, because kamailio modules uses only from a
>>>>>>>>> pre-allocated chunk, not going over it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 7/23/13 7:33 AM, David Cunningham wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly
>>>>>>>>>> using more and more memory. Over a couple of weeks it will run out of
>>>>>>>>>> system memory.
>>>>>>>>>>
>>>>>>>>>> We tried to enable memory debugging doing the following, but it
>>>>>>>>>> resulted in Kamailio not responding to any SIP packets. Would anyone have
>>>>>>>>>> advice please on how to debug the situation?
>>>>>>>>>>
>>>>>>>>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>>>>>>>>>> 2. In kamailio.cfg add the line:
>>>>>>>>>> #!define DEBUG_MEMORY 1
>>>>>>>>>>
>>>>>>>>>  do you set something special in config when DEBUG_MEMORY is 1? It
>>>>>>>>> is not by default there, so I assume you added some rules based on this
>>>>>>>>> pre-processor directive.
>>>>>>>>>
>>>>>>>>> For memory troubleshooting, set memlog to a value lower than debug
>>>>>>>>> parameter in config file and try with mem_summary=12 for a more compact
>>>>>>>>> output. See more about these parameters in the wiki:
>>>>>>>>>
>>>>>>>>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>>>>>>>>
>>>>>>>>> Run kamailio for a while in normal conditions, then restart it to
>>>>>>>>> get the memory usage summaries. There should be indication if there is some
>>>>>>>>> leak, by seeing memory chunks allocated many times from a function used at
>>>>>>>>> runtime. You can send the memory summary for a process here, we can look at
>>>>>>>>> it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> While this was running and Kamailio didn't respond to packets, it
>>>>>>>>>> logged lots of lines like this:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Do you have syslog to be configured in asynchronous mode? See the
>>>>>>>>> notes from:
>>>>>>>>>
>>>>>>>>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>>>>>>>>
>>>>>>>>> The memdbg is less than debug value, that means printing few log
>>>>>>>>> messages for each memory operation. You can make memdbg higher and rely on
>>>>>>>>> memlog for memory summaries, otherwise will be lot of log messages related
>>>>>>>>> to memory.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>>>>>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>>>>>>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>>>>>>>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
>>>>>>>>>> (size=128) on 1 -th hit
>>>>>>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>>>>>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>>>>>>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>>>>>>>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
>>>>>>>>>> (size=128) on 1 -th hit
>>>>>>>>>>
>>>>>>>>>  addstr() is a function used only for parsing configuration file,
>>>>>>>>> as long as you can still see them, the configuration file parsing was not
>>>>>>>>> finish. addstr() is not a source of leaks because it is not used at runtime.
>>>>>>>>>
>>>>>>>>> If you have large config file, then you can get close to the
>>>>>>>>> limits of the private memory, which is set to 4MB. You can increase its
>>>>>>>>> value using -M parameter (e.g., start kamailio with -M 8 to set it to use
>>>>>>>>> 8MB of memory).
>>>>>>>>>
>>>>>>>>> Over the time, the private memory can get used due to
>>>>>>>>> fragmentation, you can set the mem_join parameter in config file to avoid
>>>>>>>>> it (works when compiled with MEMDBG=1).
>>>>>>>>>
>>>>>>>>> To monitor usage of internal pkg memory, then you can use sercmd
>>>>>>>>> with pkg.stats command:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>>>>>>>>
>>>>>>>>> Shared memory stats are printed by 'kamctl fifo get_statistics
>>>>>>>>> shmem:'
>>>>>>>>>
>>>>>>>>> When you see significant increase of the memory usage, then you
>>>>>>>>> can restart to get the summaries.
>>>>>>>>>
>>>>>>>>> You should run these commands after start, just to see the initial
>>>>>>>>> usage of memory.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Daniel
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Daniel-Constantin Mierla - http://www.asipto.com
>>>>>>>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>>> list
>>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> David Cunningham, Voisonics
>>>>>>>> http://voisonics.com/
>>>>>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>>>>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>>>>>>> Australia: +61 (0) 2 8063 9019
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> David Cunningham, Voisonics
>>>>>>> http://voisonics.com/
>>>>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>>>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>>>>>> Australia: +61 (0) 2 8063 9019
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> David Cunningham, Voisonics
>>>>>> http://voisonics.com/
>>>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>>>>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> David Cunningham, Voisonics
>>>>> http://voisonics.com/
>>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>>> UK: +44 (0) 20 3298 1642
>>>>> Australia: +61 (0) 2 8063 9019
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> David Cunningham, Voisonics
>>>> http://voisonics.com/
>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>
>>>
>>> --
>>> David Cunningham, Voisonics
>>> http://voisonics.com/
>>> USA: +1 213 221 1092
>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>> Australia: +61 (0) 2 8063 9019
>>>
>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092
>> UK: +44 (0) 20 3298 1642
>> Australia: +61 (0) 2 8063 9019
>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>
>>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> UK: +44 (0) 20 3298 1642
> Australia: +61 (0) 2 8063 9019
>



-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131001/301eaf92/attachment-0001.html>

More information about the sr-users mailing list