[SR-Users] memcache problem - reporting protocol error

[Charles Chance]

Although it's difficult to know the exact cause of your "protocol" error,
it does appear from your tests that the graceful handling of errors and
subsequent reconnects is better in libmemcached than in libmemcache. So if
you do encounter the same conditions again, I would imagine that the
library would handle it and attempt to reconnect transparently.

If not, however, then at least you are more likely to find support as the
library is still under active development :)

Cheers,

Charles
 On 6 Jun 2013 18:12, "Dragos Oancea" <droancea at yahoo.com> wrote:

Hi Charles,

I tried first to trigger again the bug using some iptables rules (to
simulate a network problem) , so Kamailio could not communicate with the
memcache anymore:

 WARNING: memcached [memcached.c:189]: WARNING: memcached:
mcm_server_readable():2582: timeout: select(2) call timed out for
read(2)able fds
WARNING: memcached [memcached.c:189]: WARNING: memcached:
mcm_server_writable():3173: timeout: write select(2) call timed out
WARNING: memcached [memcached.c:189]: WARNING: memcached:
mcm_server_connect():2290: select(2) failed: select(2) timed out on
establishing conn
ection
NOTICE: memcached [memcached.c:194]: NOTICE: memcached:
mcm_server_connect():2297: connect(2) failed
NOTICE: memcached [memcached.c:194]: NOTICE: memcached:
mcm_server_connect_next_avail():2323: unable to find a server to connect to

This looks very much like what i've seen in the logs just before it started
to report the "protocol error" problem.

But in the simulated case , after I remove the iptables rules that block
the access to memcache, Kamailo does not recover from the situation, it
just keeps on printing messages like the above, when it should
connect to memcache instead. It looks like another libmemcache-related bug
to me.

Then I tried the same thing with  the latest memcached module from git:
With the memcached firewalled after starting Kamailio it prints this to the
logs:

ERROR: memcached [mcd_var.c:154]: pv_get_mcd_value_helper(): could not get
result for
key 29515_HA1 - error was 'SERVER HAS FAILED AND IS DISABLED UNTIL TIMED
RETRY'

When I flush the firewall rules, it reconnects to memcache, which makes it
better than the old one.
But how can I be sure that the new one will not have the same "protocol
error" issue, since I could not reproduce exactly the problem so far ?


Regards,
Dragos



  ------------------------------
 *From:* Charles Chance <charles.chance at sipcentric.com>
*To:* Dragos Oancea <droancea at yahoo.com>; Kamailio (SER) - Users Mailing
List <sr-users at lists.sip-router.org>
*Sent:* Thursday, June 6, 2013 6:27 PM
*Subject:* Re: [SR-Users] memcache problem - reporting protocol error

Hi Dragos,

The memcached module has indeed been updated in the master to use
libmemcached, as the old libmemcache library is no longer under active
development. Have you tried using the latest version of the module from
git? Do you still get the same errors?

Cheers,

Charles


On 6 June 2013 16:18, Dragos Oancea <droancea at yahoo.com> wrote:

Hello


We are having issues with the memcached module (Kamailio 4.0.0) .


All of a sudden we are getting this kind of messages in the logs, and the
values for the requested keys are not retrieved.


ALERT: memcached [memcached.c:189]: ALERT: memcached: mcm_fetch_cmd():1305:
memcache(4) protocol error: STORED#015#012VALUE 29515_HA1 0
32#015#012b52299247f16c6adfa4c8cc53b34e74a#015#012END#015#012STORED#015#012
[...]
ALERT: memcached [memcached.c:189]: ALERT: memcached: mcm_fetch_cmd():1176:
memcache(4) protocol error: server sent data for key not in request


I suspect that at some point there was a network problem between the
memcached server and Kamailio that triggered this abnormal behavior and I
also suspect that it is
connected with the (old) libmemcache library (libmemcache-1.4.0.rc2) .


This is fixed only with a Kamailio restart.


Has anyone experience this ?  Is there a fix ?
I noticed that the Kamailio from the trunk compiles the memcached module
with the more new libmemcached library from here:
https://launchpad.net/libmemcached/+download .
Maybe this is one of the reasons libmemcache is going to be dropped ?

Thank you very much.

Regards,
Dragos




_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users at lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users




www.sipcentric.com

Follow us on twitter @sipcentric <http://twitter.com/sipcentric>

Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered
office: Unit 10 iBIC, Birmingham Science Park, Holt Court South, Birmingham
B7 4EJ.

-- 
www.sipcentric.com

Follow us on twitter @sipcentric <http://twitter.com/sipcentric>

Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered 
office: Unit 10 iBIC, Birmingham Science Park, Holt Court South, Birmingham 
B7 4EJ.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20130606/15ec4dc0/attachment.html>