[sr-dev] qpid-proton and rabbitmq (and Kamailio kazoo module)

[Daniel Pocock]

On 30/03/17 13:25, Gordon Sim wrote:
> On 30/03/17 09:52, Daniel Pocock wrote:
>> Sending to RabbitMQ with qpid-proton python
>> -------------------------------------------
>>
>> I started with the simple_send.py[3] example and trimmed it down to send
>> a single string and then stop:
>>
>>     def on_sendable(self, event):
>>         if self.msg_body is not None:
>>             print("on_sendable !")
>>             print ("sending : %s" % (self.msg_body,))
>>             msg = Message(body=self.msg_body)
>>             event.sender.send(msg)
>>             self.msg_body = None
>>
>> I found that the connection to RabbitMQ would be dropped and an error
>> would appear in the RabbitMQ log.  I tried changing the body type to a
>> dict containing a string, e.g. {'my_body':'foo'} and then it would
>> successfully pass the message to RabbitMQ but then the receiver would
>> complain that it was a map and not a string.
>>
>> I found that the problem could be fixed by adding "inferred=True" to the
>> Message constructor:
>>
>>             msg = Message(body=self.msg_body, inferred=True)
> 
> 
> The python proton library will encode a binary message body as a Data
> section if inferred is not set to true (otherwise as a binary value in a
> Value section). RabbitMQ does not support Data sections yet I believe.
> 


Is there a bug or feature request for Data sections in RabbitMQ?  I
couldn't find their bug tracker.  I've opened a bug report[1] against
the Debian package.


> As Robbie mentioned, in python 2.x string literals are not always
> explicitly utf8 and are then treated as binary. You can either do u'my
> utf8' or import unicode_literals from __future__. If you do that then
> you should get a Value section regardless of the inferred property.
> 

In my case the strings are coming from the command line, so now I change
the string to unicode like this:

   _s = unicode(s, "utf-8")

and then:

a) I can send without inferred=True and the broker accepts the message

b) in the C++ receiver, the message is now received as a string and I
can use:

    _json = proton::get<std::string>(m.body());


Could this be made more obvious in the examples perhaps, e.g.
helloworld.py could change from:

    event.sender.send(Message(body="Hello World!"))

to

    event.sender.send(Message(body=unicode("Hello World!", "utf-8")))

I realize that is a bit verbose and redundant when you already have
unicode_literals, but for people who are cutting and pasting from the
examples they will get up and running more quickly.


>> Below is a complete example of the error from the RabbitMQ log when
>> inferred=True is not present, the body in this case was the string "foo"
>>
>>
>> Receiving from RabbitMQ with qpid-proton C++
>> --------------------------------------------
>>
>> I looked at the receive example simple_recv.cpp[4] and other examples
>> and noticed code like this being used to access the message body as
>> std::string:
>>
>> proton::get<std::string>(m.body())
>>
>> However, that didn't work for me, it threw a conversion_error exception
>> "unexpected type, want: string got: binary" complaining that the body
>> was binary.  I had to use something like this to convert my message body
>> to a std::string:
>>
>>    proton::binary __b = proton::get<proton::binary>(m.body());
>>    std::string _s = (std::string)__b;
>>
>> If I send a string from Python, should it appear as binary in the
>> receiver?
> 
> As above it sounds like the message *is* actually being sent as a binary.
> 
>> What is the suggested way to write a receiver that can handle
>> any arbitrary message that started as a string and may arrive in some
>> other format, especially if the message broker or client is changed at
>> some arbitrary time in the future?
>>
>> I also observed similar issues receiving messages that had been sent
>> into RabbitMQ by Kamailio's kazoo module[5], it is linked with the
>> librabbitmq client library.
> 
> I'm assuming that is sending over AMQP 0-10. If so, how that is
> converted into AMQP 1.0 is dependent on the RabbitMQ implementation. It
> may well be always sent as a binary value. However hopefully it would
> have a content-type to indicate how to interpret the data (though
> structly speaking the AMQP 1.0 spec disallows that).
> 


The kazoo module source code is on Github[2], I had a quick look at it
and it isn't immediately obvious to me which AMQP version it uses or
which data type it is using for the message body.  Is anybody more
familiar with AMQP C client programming able to spot any keywords in
that code that suggest what it is doing?  I've added the kazoo module
developers on CC in case they can comment on this, it would be useful to
answer this in the module's README file.

Regards,

Daniel

1. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859399
2. https://github.com/sipwise/kamailio/tree/master/modules/kazoo