Greetings,
I'm trying to use SQlite as my engine in order to improve performance. I would like to have my Database stored in memory for faster access.
I see that Kamailio can open the SQlite database as Read-Only, but i can't find a way to open it in memory. Is this possible using Kamailio? What's the best way to do this?
Best Regards,
Duarte Rocha
What sort of insufficient read performance are you seeing from SQLite on modern storage which would lead you to this unnecessary ideation? Read performance on SQLite is insanely fast...
— Sent from mobile, with due apologies for brevity and errors.
On Feb 27, 2019, at 12:23 PM, Duarte Rocha duarterocha91@gmail.com wrote:
Greetings,
I'm trying to use SQlite as my engine in order to improve performance. I would like to have my Database stored in memory for faster access.
I see that Kamailio can open the SQlite database as Read-Only, but i can't find a way to open it in memory. Is this possible using Kamailio? What's the best way to do this?
Best Regards,
Duarte Rocha _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Greetings Alex,
At the moment i'm trying to debug some issues with the server and try some solutions to see the impact. With little load i'm getting some percentage of "503 Server Unavaliable" errors from Kamailio. I made some testing without using database for anything but Dispatcher and Accounting and the performance got way better.
Right now, on this server, under as little as 20CPS Kamailio is returning "503" on 1-2% of all calls. However, my CPU and Memory look like this:
%Cpu(s): 20.3 us, 6.1 sy, 0.0 ni, 73.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.2 st
Memory :
total used free shared buff/cache available Mem: 1830 1172 382 109 274 375 Swap: 819 49 770
I have around 100 processes for Kamailio.
Cheers
Hi, you can use the WAL mode, so that readers are never delayed.
Just make sure all the connections to the DB are using the WAL mode.
The downside is that the writers can be delayed sometimes, similar to Postgres read only replica
On Wed, Feb 27, 2019 at 9:57 AM Duarte Rocha duarterocha91@gmail.com wrote:
Greetings Alex,
At the moment i'm trying to debug some issues with the server and try some solutions to see the impact. With little load i'm getting some percentage of "503 Server Unavaliable" errors from Kamailio. I made some testing without using database for anything but Dispatcher and Accounting and the performance got way better.
Right now, on this server, under as little as 20CPS Kamailio is returning "503" on 1-2% of all calls. However, my CPU and Memory look like this:
%Cpu(s): 20.3 us, 6.1 sy, 0.0 ni, 73.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.2 st
Memory :
total used free shared buff/cache availableMem: 1830 1172 382 109 274 375 Swap: 819 49 770
I have around 100 processes for Kamailio.
Cheers _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
*https://www.kamailio.org/docs/modules/5.2.x/modules/db_sqlite.html#db_sqlite... https://www.kamailio.org/docs/modules/5.2.x/modules/db_sqlite.html#db_sqlite.p.db_set_journal_mode*
On Wed, Feb 27, 2019 at 10:27 AM Julien Chavanton jchavanton@gmail.com wrote:
Hi, you can use the WAL mode, so that readers are never delayed.
Just make sure all the connections to the DB are using the WAL mode.
The downside is that the writers can be delayed sometimes, similar to Postgres read only replica
On Wed, Feb 27, 2019 at 9:57 AM Duarte Rocha duarterocha91@gmail.com wrote:
Greetings Alex,
At the moment i'm trying to debug some issues with the server and try some solutions to see the impact. With little load i'm getting some percentage of "503 Server Unavaliable" errors from Kamailio. I made some testing without using database for anything but Dispatcher and Accounting and the performance got way better.
Right now, on this server, under as little as 20CPS Kamailio is returning "503" on 1-2% of all calls. However, my CPU and Memory look like this:
%Cpu(s): 20.3 us, 6.1 sy, 0.0 ni, 73.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.2 st
Memory :
total used free shared buff/cache availableMem: 1830 1172 382 109 274 375 Swap: 819 49 770
I have around 100 processes for Kamailio.
Cheers _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Okay, but at a paltry 20 CPS -- or even several multiples of that -- none of this should be an issue. There's something else going on.
On Wed, Feb 27, 2019 at 10:28:54AM -0800, Julien Chavanton wrote:
*https://www.kamailio.org/docs/modules/5.2.x/modules/db_sqlite.html#db_sqlite... https://www.kamailio.org/docs/modules/5.2.x/modules/db_sqlite.html#db_sqlite.p.db_set_journal_mode*
On Wed, Feb 27, 2019 at 10:27 AM Julien Chavanton jchavanton@gmail.com wrote:
Hi, you can use the WAL mode, so that readers are never delayed.
Just make sure all the connections to the DB are using the WAL mode.
The downside is that the writers can be delayed sometimes, similar to Postgres read only replica
On Wed, Feb 27, 2019 at 9:57 AM Duarte Rocha duarterocha91@gmail.com wrote:
Greetings Alex,
At the moment i'm trying to debug some issues with the server and try some solutions to see the impact. With little load i'm getting some percentage of "503 Server Unavaliable" errors from Kamailio. I made some testing without using database for anything but Dispatcher and Accounting and the performance got way better.
Right now, on this server, under as little as 20CPS Kamailio is returning "503" on 1-2% of all calls. However, my CPU and Memory look like this:
%Cpu(s): 20.3 us, 6.1 sy, 0.0 ni, 73.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.2 st
Memory :
total used free shared buff/cache availableMem: 1830 1172 382 109 274 375 Swap: 819 49 770
I have around 100 processes for Kamailio.
Cheers _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Ok, i reduced the number of processes and tried a few different values, but the performance hasn't changed a lot.
How can i debug this and find the bottleneck in my system?
A quarta, 27/02/2019, 17:56, Duarte Rocha duarterocha91@gmail.com escreveu:
Greetings Alex,
At the moment i'm trying to debug some issues with the server and try some solutions to see the impact. With little load i'm getting some percentage of "503 Server Unavaliable" errors from Kamailio. I made some testing without using database for anything but Dispatcher and Accounting and the performance got way better.
Right now, on this server, under as little as 20CPS Kamailio is returning "503" on 1-2% of all calls. However, my CPU and Memory look like this:
%Cpu(s): 20.3 us, 6.1 sy, 0.0 ni, 73.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.2 st
Memory :
total used free shared buff/cache availableMem: 1830 1172 382 109 274 375 Swap: 819 49 770
I have around 100 processes for Kamailio.
Cheers
I would say the first step is to gauge the total velocity of any given initial request through your config.
One way to do that is:
https://kamailio.org/docs/modules/5.2.x/modules/benchmark.html
But I prefer this approach:
route { $avp(proc_start) = $TV(Sn);
...
if(is_method("INVITE")) { processing_etc();
$var(cur_time) = $TV(Sn);
$var(proc_diff) = ( ((( $(var(cur_time){s.select,0,.}{s.int}) - $(avp(proc_start){s.select,0,.}{s.int}) ) * 1000000) + ( $(var(cur_time){s.select,1,.}{s.int}) - $(avp(proc_start){s.select,1,.}{s.int}) ) / 1000) mod 1000 );
xlog("L_INFO", "[$ci] Request processing delay: $var(proc_diff) ms\n");
if(!t_relay()) sl_reply_error();
exit; } }
I don't vouch for the precision/granularity of this timing scheme, but it's good enough that it should tell you if your total request processing volume is insane. Then you can move these breakpoints around until you find which operation is actually so slow.
That said, I am not aware of any aspect by which Kamailio intrinsically sends a 503 if it is overwhelmed with requests. It just drops them/doesn't process.
-- Alex
On Wed, Feb 27, 2019 at 06:35:46PM +0000, Duarte Rocha wrote:
Ok, i reduced the number of processes and tried a few different values, but the performance hasn't changed a lot.
How can i debug this and find the bottleneck in my system?
A quarta, 27/02/2019, 17:56, Duarte Rocha duarterocha91@gmail.com escreveu:
Greetings Alex,
At the moment i'm trying to debug some issues with the server and try some solutions to see the impact. With little load i'm getting some percentage of "503 Server Unavaliable" errors from Kamailio. I made some testing without using database for anything but Dispatcher and Accounting and the performance got way better.
Right now, on this server, under as little as 20CPS Kamailio is returning "503" on 1-2% of all calls. However, my CPU and Memory look like this:
%Cpu(s): 20.3 us, 6.1 sy, 0.0 ni, 73.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.2 st
Memory :
total used free shared buff/cache availableMem: 1830 1172 382 109 274 375 Swap: 819 49 770
I have around 100 processes for Kamailio.
Cheers
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Do you use a lot of logging? Have you enabled asynchronous logging?
If you are using the DB only for dispatcher, then you have other issues then the db. The dispatcher module is caching the data on memory.
You can do hundreds of CPS with only two workers and the CPU will be at 3% utilization on very old hardware. There's something else in your config creating issues.
Regards, Ovidiu Sas
On Wed, Feb 27, 2019 at 1:36 PM Duarte Rocha duarterocha91@gmail.com wrote:
Ok, i reduced the number of processes and tried a few different values, but the performance hasn't changed a lot.
How can i debug this and find the bottleneck in my system?
A quarta, 27/02/2019, 17:56, Duarte Rocha duarterocha91@gmail.com escreveu:
Greetings Alex,
At the moment i'm trying to debug some issues with the server and try some solutions to see the impact. With little load i'm getting some percentage of "503 Server Unavaliable" errors from Kamailio. I made some testing without using database for anything but Dispatcher and Accounting and the performance got way better.
Right now, on this server, under as little as 20CPS Kamailio is returning "503" on 1-2% of all calls. However, my CPU and Memory look like this:
%Cpu(s): 20.3 us, 6.1 sy, 0.0 ni, 73.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.2 st
Memory :
total used free shared buff/cache availableMem: 1830 1172 382 109 274 375 Swap: 819 49 770
I have around 100 processes for Kamailio.
Cheers
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On Wed, Feb 27, 2019 at 01:43:02PM -0500, Ovidiu Sas wrote:
You can do hundreds of CPS with only two workers and the CPU will be at 3% utilization on very old hardware. There's something else in your config creating issues.
+1
I use database for lots of configs. On any given first request i do around 10 queries.
A while ago i measured the processing time (using bm_timer) when i was still using MySql and got times of around 6ms per first Invite and around 2ms for other in-dialog requests.
While i think those are big processing times, i don't think they should be the problem at a rate as low as 20CPS.
Thank you for your help :)
A quarta, 27/02/2019, 18:35, Duarte Rocha duarterocha91@gmail.com escreveu:
Ok, i reduced the number of processes and tried a few different values, but the performance hasn't changed a lot.
How can i debug this and find the bottleneck in my system?
A quarta, 27/02/2019, 17:56, Duarte Rocha duarterocha91@gmail.com escreveu:
Greetings Alex,
At the moment i'm trying to debug some issues with the server and try some solutions to see the impact. With little load i'm getting some percentage of "503 Server Unavaliable" errors from Kamailio. I made some testing without using database for anything but Dispatcher and Accounting and the performance got way better.
Right now, on this server, under as little as 20CPS Kamailio is returning "503" on 1-2% of all calls. However, my CPU and Memory look like this:
%Cpu(s): 20.3 us, 6.1 sy, 0.0 ni, 73.3 id, 0.0 wa, 0.0 hi, 0.2 si, 0.2 st
Memory :
total used free shared buff/cache availableMem: 1830 1172 382 109 274 375 Swap: 819 49 770
I have around 100 processes for Kamailio.
Cheers
On Wed, Feb 27, 2019 at 06:53:54PM +0000, Duarte Rocha wrote:
I use database for lots of configs. On any given first request i do around 10 queries.
A while ago i measured the processing time (using bm_timer) when i was still using MySql and got times of around 6ms per first Invite and around 2ms for other in-dialog requests.
While i think those are big processing times, i don't think they should be the problem at a rate as low as 20CPS.
No, those aren't big processing times. Any cumulative round trips to the database, even a local one, will amount to a few ms at least.
It doesn't sound like that's where the issue lies...
-
Alex Balashov -
Duarte Rocha -
Julien Chavanton -
Ovidiu Sas