<div dir="ltr">Hello!<div><br></div><div><b>Sep 5 08:07:59 kamailio[1603]: 12(1603) WARNING: <core> [core/timer.c:936]: timer_handler(): slow timer too slow: overflow (0 - 64673 = -64673)</b><br></div><div><br></div><div>I caught the problem again, here is the output of the commands that you recommended to check:<br></div><div><br></div><div><div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre"><div>root@kamailio:/home/azureuser# kamctl ps</div><div>{</div><div> <span style="color:rgb(156,220,254)">"jsonrpc"</span>: <span style="color:rgb(206,145,120)">"2.0"</span>,</div><div> <span style="color:rgb(156,220,254)">"result"</span>: [</div><div> {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">0</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1574</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"main process - attendant"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">1</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1592</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=0 sock=<a href="http://192.168.20.20:5060">192.168.20.20:5060</a> (<a href="http://20.20.20.20:5060">20.20.20.20:5060</a>)"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">2</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1593</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=1 sock=<a href="http://192.168.20.20:5060">192.168.20.20:5060</a> (<a href="http://20.20.20.20:5060">20.20.20.20:5060</a>)"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">3</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1594</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=0 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">4</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1595</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=1 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">5</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1596</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=2 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">6</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1597</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=3 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">7</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1598</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=4 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">8</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1599</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=5 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">9</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1600</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=6 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">10</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1601</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=7 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">11</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1602</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"slow timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">12</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1603</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">13</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1604</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"secondary timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">14</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1605</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"JSONRPCS FIFO"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">15</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1606</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"JSONRPCS DATAGRAM"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">16</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1607</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"USRLOC Timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">17</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1608</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"ctl handler"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">18</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1609</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"DMQ WORKER"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">19</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1610</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"DMQ WORKER"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">20</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1611</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"DMQ WORKER"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">21</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1612</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"DMQ WORKER"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">22</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1613</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"TIMER NH"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">23</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1614</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"Dialog Clean Timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">24</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1615</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"TIMER UAC REG"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">25</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1616</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"RTIMER EXEC child=0 timer=uac"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">26</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1617</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"RTIMER EXEC child=0 timer=cluster"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1618</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"RTIMER EXEC child=0 timer=apiban"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">28</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1619</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=0"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">29</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1620</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=1"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">30</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1621</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=2"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">31</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1622</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=3"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">32</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1623</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=4"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">33</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1624</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=5"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">34</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1625</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=6"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">35</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1626</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=7"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"IDX"</span>: <span style="color:rgb(181,206,168)">36</span>,</div><div> <span style="color:rgb(156,220,254)">"PID"</span>: <span style="color:rgb(181,206,168)">1627</span>,</div><div> <span style="color:rgb(156,220,254)">"DSC"</span>: <span style="color:rgb(206,145,120)">"tcp main process"</span></div><div> }</div><div> ],</div><div> <span style="color:rgb(156,220,254)">"id"</span>: <span style="color:rgb(181,206,168)">33990</span></div><div>}</div><br><div>root@kamailio:/home/azureuser# kamctl rpc pkg.stats</div><div>{</div><div> <span style="color:rgb(156,220,254)">"jsonrpc"</span>: <span style="color:rgb(206,145,120)">"2.0"</span>,</div><div> <span style="color:rgb(156,220,254)">"result"</span>: [</div><div> {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">0</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1574</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">0</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1797960</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249896</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527320</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"main process - attendant"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">1</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1592</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">14690568</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">49800</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">16727416</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">7</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=0 sock=<a href="http://192.168.20.20:5060">192.168.20.20:5060</a> (<a href="http://20.20.20.20:5060">20.20.20.20:5060</a>)"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">2</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1593</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">2</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">14734792</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">1208</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">16776008</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">8</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=1 sock=<a href="http://192.168.20.20:5060">192.168.20.20:5060</a> (<a href="http://20.20.20.20:5060">20.20.20.20:5060</a>)"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">3</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1594</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">3</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798184</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249256</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527960</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=0 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">4</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1595</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">4</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798184</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249256</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527960</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=1 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">5</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1596</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">5</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798184</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249256</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527960</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=2 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">6</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1597</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">6</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798184</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249256</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527960</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=3 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">7</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1598</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">7</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798184</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249256</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527960</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=4 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">8</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1599</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">8</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798184</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249256</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527960</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=5 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">9</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1600</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">9</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798184</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249256</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527960</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=6 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">10</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1601</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">10</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798184</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249256</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527960</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"udp receiver child=7 sock=<a href="http://192.168.20.20:5080">192.168.20.20:5080</a>"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">11</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1602</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1799296</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14248040</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2529176</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">26</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"slow timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">12</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1603</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798160</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249280</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527936</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">13</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1604</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798160</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249280</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527936</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"secondary timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">14</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1605</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-2</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798056</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249488</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527728</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"JSONRPCS FIFO"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">15</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1606</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-2</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1863512</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14183928</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2593288</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"JSONRPCS DATAGRAM"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">16</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1607</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798160</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249280</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527936</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"USRLOC Timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">17</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1608</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-2</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1808104</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14239232</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2537984</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"ctl handler"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">18</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1609</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-2</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798056</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249488</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527728</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"DMQ WORKER"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">19</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1610</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-2</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798056</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249488</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527728</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"DMQ WORKER"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">20</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1611</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-2</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798056</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249488</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527728</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"DMQ WORKER"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">21</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1612</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-2</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798056</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249488</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527728</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"DMQ WORKER"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">22</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1613</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798160</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249280</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527936</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"TIMER NH"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">23</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1614</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798160</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14249280</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2527936</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"Dialog Clean Timer"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">24</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1615</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798744</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14246720</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2530496</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">26</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"TIMER UAC REG"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">25</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1616</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1798272</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14248960</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2528256</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">28</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"RTIMER EXEC child=0 timer=uac"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">26</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1617</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1799344</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14247888</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2529328</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"RTIMER EXEC child=0 timer=cluster"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1618</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-1</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1794504</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14252728</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2524488</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">28</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"RTIMER EXEC child=0 timer=apiban"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">28</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1619</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">11</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1878904</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14168328</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2608888</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=0"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">29</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1620</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">12</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1878904</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14168328</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2608888</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=1"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">30</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1621</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">13</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1878904</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14168328</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2608888</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=2"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">31</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1622</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">14</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1878904</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14168328</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2608888</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=3"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">32</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1623</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">15</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1878904</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14168328</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2608888</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=4"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">33</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1624</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">16</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1878904</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14168328</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2608888</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=5"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">34</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1625</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">17</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1878904</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14168328</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2608888</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">27</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=6"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">35</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1626</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">18</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1878992</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14168032</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2609184</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">28</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"tcp receiver (generic) child=7"</span></div><div> }, {</div><div> <span style="color:rgb(156,220,254)">"entry"</span>: <span style="color:rgb(181,206,168)">36</span>,</div><div> <span style="color:rgb(156,220,254)">"pid"</span>: <span style="color:rgb(181,206,168)">1627</span>,</div><div> <span style="color:rgb(156,220,254)">"rank"</span>: <span style="color:rgb(181,206,168)">-4</span>,</div><div> <span style="color:rgb(156,220,254)">"used"</span>: <span style="color:rgb(181,206,168)">1880080</span>,</div><div> <span style="color:rgb(156,220,254)">"free"</span>: <span style="color:rgb(181,206,168)">14167776</span>,</div><div> <span style="color:rgb(156,220,254)">"real_used"</span>: <span style="color:rgb(181,206,168)">2609440</span>,</div><div> <span style="color:rgb(156,220,254)">"total_size"</span>: <span style="color:rgb(181,206,168)">16777216</span>,</div><div> <span style="color:rgb(156,220,254)">"total_frags"</span>: <span style="color:rgb(181,206,168)">31</span>,</div><div> <span style="color:rgb(156,220,254)">"desc"</span>: <span style="color:rgb(206,145,120)">"tcp main process"</span></div><div> }</div><div> ],</div><div> <span style="color:rgb(156,220,254)">"id"</span>: <span style="color:rgb(181,206,168)">34083</span></div><div>}</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">пн, 29 авг. 2022 г. в 17:43, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>I would also worry about the out of memory logs. For
troubleshooting in such case, you need to get:</p>
<p>kamctl ps</p>
<p>kamctl rpc pkg.stats</p>
<p>Maybe also you set latency limits via core parameter to see if
some operations are slow in the config.</p>
<p>If there are slow C functions run internally on timer, then maybe
you can troubleshoot with debug=3.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 29.08.22 16:30, Denys Pozniak wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello!</div>
<div><br>
After some time, Kamailio stops processing incoming messages.<br>
<span style="background-color:rgb(30,30,30);color:rgb(212,212,212);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap">slow timer too slow: overflow</span> messages
appear in the log (I suspect that the problem is related to
the rtimer module)</div>
<div><br>
</div>
<div>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre-wrap"><div>Aug 29 14:19:17 kamailio[<span style="color:rgb(206,145,120)">11320</span>]: 2(11320) ERROR: {1 10 OPTIONS <a href="mailto:2b45547f33f5203f-4091@10.10.10.10" target="_blank">2b45547f33f5203f-4091@10.10.10.10</a>} <span style="color:rgb(128,128,128)"><</span><span style="color:rgb(86,156,214)">core</span><span style="color:rgb(128,128,128)">></span> [<span style="color:rgb(206,145,120)">core/mem/q_malloc.c:430</span>]: qm_malloc(): qm_malloc(0x7f163b9f9010, 960) called from core: core/rvalue.c: rval_new_empty(246), module: core; Free fragment></div><div>Aug 29 14:19:17 kamailio[<span style="color:rgb(206,145,120)">11320</span>]: 2(11320) ERROR: {1 10 OPTIONS <a href="mailto:2b45547f33f5203f-4091@10.10.10.10" target="_blank">2b45547f33f5203f-4091@10.10.10.10</a>} <span style="color:rgb(128,128,128)"><</span><span style="color:rgb(86,156,214)">core</span><span style="color:rgb(128,128,128)">></span> [<span style="color:rgb(206,145,120)">core/rvalue.c:253</span>]: rval_new_empty(): could not allocate private memory from pkg pool</div><div>Aug 29 14:19:17 kamailio[<span style="color:rgb(206,145,120)">11330</span>]: 12(11330) WARNING: <span style="color:rgb(128,128,128)"><</span><span style="color:rgb(86,156,214)">core</span><span style="color:rgb(128,128,128)">></span> [<span style="color:rgb(206,145,120)">core/timer.c:936</span>]: timer_handler(): slow timer too slow: overflow (62671 - 61648 = 1023)</div><div>Aug 29 14:19:17 kamailio[<span style="color:rgb(206,145,120)">11320</span>]: 2(11320) ERROR: {1 10 OPTIONS <a href="mailto:34618af41c3866bc-16408@10.10.10.20" target="_blank">34618af41c3866bc-16408@10.10.10.20</a>} <span style="color:rgb(128,128,128)"><</span><span style="color:rgb(86,156,214)">core</span><span style="color:rgb(128,128,128)">></span> [<span style="color:rgb(206,145,120)">core/mem/q_malloc.c:297</span>]: qm_find_free(): qm_find_free(0x7f163b9f9010, 960); Free fragment not found!</div><div>Aug 29 14:19:17 kamailio[<span style="color:rgb(206,145,120)">11320</span>]: 2(11320) ERROR: {1 10 OPTIONS <a href="mailto:34618af41c3866bc-16408@10.10.10.20" target="_blank">34618af41c3866bc-16408@10.10.10.20</a>} <span style="color:rgb(128,128,128)"><</span><span style="color:rgb(86,156,214)">core</span><span style="color:rgb(128,128,128)">></span> [<span style="color:rgb(206,145,120)">core/mem/q_malloc.c:430</span>]: qm_malloc(): qm_malloc(0x7f163b9f9010, 960) called from core: core/rvalue.c: rval_new_empty(246), module: core; Free fragment no></div><div>Aug 29 14:19:17 kamailio[<span style="color:rgb(206,145,120)">11320</span>]: 2(11320) ERROR: {1 10 OPTIONS <a href="mailto:34618af41c3866bc-16408@10.10.10.20" target="_blank">34618af41c3866bc-16408@10.10.10.20</a>} <span style="color:rgb(128,128,128)"><</span><span style="color:rgb(86,156,214)">core</span><span style="color:rgb(128,128,128)">></span> [<span style="color:rgb(206,145,120)">core/rvalue.c:253</span>]: rval_new_empty(): could not allocate private memory from pkg pool</div><div>Aug 29 14:19:17 kamailio[<span style="color:rgb(206,145,120)">11330</span>]: 12(11330) WARNING: <span style="color:rgb(128,128,128)"><</span><span style="color:rgb(86,156,214)">core</span><span style="color:rgb(128,128,128)">></span> [<span style="color:rgb(206,145,120)">core/timer.c:936</span>]: timer_handler(): slow timer too slow: overflow (62671 - 61648 = 1023)</div><div>Aug 29 14:19:18 kamailio[<span style="color:rgb(206,145,120)">11320</span>]: 2(11320) ERROR: {1 10 KDMQ <a href="mailto:1c18959d07113951-203309@10.10.10.30" target="_blank">1c18959d07113951-203309@10.10.10.30</a>} <span style="color:rgb(128,128,128)"><</span><span style="color:rgb(86,156,214)">core</span><span style="color:rgb(128,128,128)">></span> [<span style="color:rgb(206,145,120)">core/mem/q_malloc.c:297</span>]: qm_find_free(): qm_find_free(0x7f163b9f9010, 960); Free fragment not found!</div><div>Aug 29 14:19:18 kamailio[<span style="color:rgb(206,145,120)">11320</span>]: 2(11320) ERROR: {1 10 KDMQ <a href="mailto:1c18959d07113951-203309@10.10.10.30" target="_blank">1c18959d07113951-203309@10.10.10.30</a>} <span style="color:rgb(128,128,128)"><</span><span style="color:rgb(86,156,214)">core</span><span style="color:rgb(128,128,128)">></span> [<span style="color:rgb(206,145,120)">core/mem/q_malloc.c:430</span>]: qm_malloc(): qm_malloc(0x7f163b9f9010, 960) called from core: core/rvalue.c: rval_new_empty(246), module: core; Free fragment no></div></div>
</div>
<div><br>
</div>
<div>
<div>Please point me out where the source of the problem could
be.<br>
</div>
<div><br>
</div>
<div>version: kamailio 5.6.1 (x86_64/linux)<br>
</div>
</div>
<div>-- <br>
<div dir="ltr">
<div dir="ltr"><br>
<div>
<div>BR,</div>
<div>Denys Pozniak<br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><br><div><div>BR,</div><div>Denys Pozniak<br></div><div><br></div><div><br></div></div></div></div>