[sr-dev] rtpengine transcoding with libs other than ffmpeg

[Richard Fuchs]

On 2018-02-20 04:45 PM, Jeff Brower wrote:
> It's great to see that rtpengine now supports transcoding.  I always watched rtpproxy (for so many years) to see when it might happen.
>
> I have some questions.  I could not find on ffmpeg.org specific mention of:
>
>    -advanced audio jitter buffer capability

For the time being, rtpengine does not try to play nice on jitter. Audio 
data is decoded as soon as it arrives, passed to the encoder, and 
encoded data is sent out as soon as it's available. Depending on which 
codecs and packet sizes are involved, this may turn out transparent 
jitter-wise, or it may introduce considerable additional jitter. Dealing 
with jitter is thus left up to the endpoints.

>    -advanced RTP related RFCs, eg. RFC 8108

This would have to be handled in rtpengine itself and isn't directly 
related to transcoding.

>    -advanced codec capability, eg. G711 Appendix II (DTX and CNG)

This would also have to be handled in rtpengine as ffmpeg doesn't really 
support G.711 per se, but rather supports PCM a-Law und µ-Law as raw 
audio codecs. RFC 4733 will be handled in the near future.

>
> In particular, it's not clear to me where jitter buffer and RTP RFCs are handled -- inside rtpgengine or ffmpeg.  I don't see any mention of jitter
> buffer on the rtpengine web and github pages.  In general ffmpeg's focus is on content delivery and not bidirectional real-time communication.

While I do believe that ffmpeg supports RTP in some way, this is not 
used by rtpengine -- ffmpeg is used only for raw codec support (and 
resampling).

>
> 1) Is there an rtpengine-ffmpeg software architecture or data flow diagram available ?

No, but you're welcome to submit one. :)

>
> 2) Is it possible to connect libraries besides ffmpeg to the "other side" of rtpengine ?  For example, using the rtpengine interface, send and
> receive a UDP/RTP packet stream to/from a third-party library and let it handle jitter buffer, encoding/decoding, ptime mismatch (transrating), and
> more ?

Well, it's just software, so anything is possible, right? :)

While this wasn't something we had on the radar when implementing this, 
I don't see why such a tie-in shouldn't be possible. At least non-ffmpeg 
codecs can be tied in relatively easily, which is what we did with bcg729.

>
> 3) If architecturally that's a do-able thing, is there a spec on how rtpengine is currently interfacing to ffmpeg ?  Which APIs are being used ?  (I
> assume the command line interface is not being used).  Re. ffmpeg APIs I found this:
>
>    http://dranger.com/ffmpeg/
>
> but maybe there is something more recent or rtpengine source code we can look at.

The most relevant API is the avcodec API, described at 
https://ffmpeg.org/doxygen/trunk/avcodec_8h.html together with the 
related structures (AVFrame and AVPacket).

Also used is avresample/swresample from 
https://ffmpeg.org/doxygen/trunk/avresample_8h.html and the audio FIFO 
API https://ffmpeg.org/doxygen/trunk/audio__fifo_8h.html

The guts of how rtpengine ties into avcodec are 
https://github.com/sipwise/rtpengine/blob/master/lib/codeclib.c and most 
of the interfaces to the rest of rtpengine (SDP and control stuff) are 
at https://github.com/sipwise/rtpengine/blob/master/daemon/codec.c

HTH
Cheers