[sr-dev] Handling large responses with utils:http_query()

[Alex Balashov]

Hi,

I have a ~21 kB HTTP response that I need to handle with http_query() 
from the utils module.

There are two separate obstacles to this. The first is insufficient size 
of the default PV text buffer, which is solvable by increasing it using 
the 'pv_buffer_size' core parameter.

The second is the CURL wrapper in modules/utils/functions.c itself. It 
uses the write_function() callback to pump the data into a buffer:

     char *stream;
     ...
     stream = NULL;
     curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_function);
     curl_easy_setopt(curl, CURLOPT_WRITEDATA, &stream);

As far as I can tell from how write_function() is constructed, it 
expects to be called once:

size_t write_function( void *ptr, size_t size, size_t nmemb, void *stream)
{
     /* Allocate memory and copy */
     char* data;

     data = (char*)pkg_malloc((size* nmemb) + 1);
     if (data == NULL) {
         LM_ERR("cannot allocate memory for stream\n");
         return CURLE_WRITE_ERROR;
     }

     memcpy(data, (char*)ptr, size* nmemb);
     data[nmemb] = '\0';

     *((char**) stream) = data;

     return size* nmemb;
  }

That's not the case for a 21 kB reply:

size_t write_function( void *ptr, size_t size, size_t nmemb, void *stream)
{
     /* Allocate memory and copy */
     char* data;

     LM_INFO("Write function called with additional buffer: %u\n",
         (unsigned int) size * nmemb);


  0(9696) INFO: utils [functions.c:56]: write_function(): Write function 
called with additional buffer: 1193
  0(9696) INFO: utils [functions.c:56]: write_function(): Write function 
called with additional buffer: 1448
  0(9696) INFO: utils [functions.c:56]: write_function(): Write function 
called with additional buffer: 10136
  0(9696) INFO: utils [functions.c:56]: write_function(): Write function 
called with additional buffer: 1448
  0(9696) INFO: utils [functions.c:56]: write_function(): Write function 
called with additional buffer: 1448
  0(9696) INFO: utils [functions.c:56]: write_function(): Write function 
called with additional buffer: 4344
  0(9696) INFO: utils [functions.c:56]: write_function(): Write function 
called with additional buffer: 1448
  0(9696) INFO: utils [functions.c:56]: write_function(): Write function 
called with additional buffer: 327

So, I'm not sure what the easiest way to go here is.

1) Does this function need to be rewritten to realloc() and concatenate?

2) Is there an option in CURL that can use internal buffering and only 
invoke the callback once all the response chunks are received?

3) Would increasing the curl.h constant CURL_MAX_WRITE_SIZE help?

Input is appreciated!

-- Alex

-- 
Alex Balashov - Principal
Evariste Systems LLC
235 E Ponce de Leon Ave
Suite 106
Decatur, GA 30030
United States

Tel: +1-678-954-0670
Web: http://www.evaristesys.com/, http://www.alexbalashov.com/