[sr-dev] Handling large responses with utils:http_query()

Hi, I have a ~21 kB HTTP response that I need to handle with http_query() from the utils module. There are two separate obstacles to this. The first is insufficient size of the default PV text buffer, which is solvable by increasing it using the 'pv_buffer_size' core parameter. The second is the CURL wrapper in modules/utils/functions.c itself. It uses the write_function() callback to pump the data into a buffer: char *stream; ... stream = NULL; curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_function); curl_easy_setopt(curl, CURLOPT_WRITEDATA, &stream); As far as I can tell from how write_function() is constructed, it expects to be called once: size_t write_function( void *ptr, size_t size, size_t nmemb, void *stream) { /* Allocate memory and copy */ char* data; data = (char*)pkg_malloc((size* nmemb) + 1); if (data == NULL) { LM_ERR("cannot allocate memory for stream\n"); return CURLE_WRITE_ERROR; } memcpy(data, (char*)ptr, size* nmemb); data[nmemb] = '\0'; *((char**) stream) = data; return size* nmemb; } That's not the case for a 21 kB reply: size_t write_function( void *ptr, size_t size, size_t nmemb, void *stream) { /* Allocate memory and copy */ char* data; LM_INFO("Write function called with additional buffer: %u\n", (unsigned int) size * nmemb); 0(9696) INFO: utils [functions.c:56]: write_function(): Write function called with additional buffer: 1193 0(9696) INFO: utils [functions.c:56]: write_function(): Write function called with additional buffer: 1448 0(9696) INFO: utils [functions.c:56]: write_function(): Write function called with additional buffer: 10136 0(9696) INFO: utils [functions.c:56]: write_function(): Write function called with additional buffer: 1448 0(9696) INFO: utils [functions.c:56]: write_function(): Write function called with additional buffer: 1448 0(9696) INFO: utils [functions.c:56]: write_function(): Write function called with additional buffer: 4344 0(9696) INFO: utils [functions.c:56]: write_function(): Write function called with additional buffer: 1448 0(9696) INFO: utils [functions.c:56]: write_function(): Write function called with additional buffer: 327 So, I'm not sure what the easiest way to go here is. 1) Does this function need to be rewritten to realloc() and concatenate? 2) Is there an option in CURL that can use internal buffering and only invoke the callback once all the response chunks are received? 3) Would increasing the curl.h constant CURL_MAX_WRITE_SIZE help? Input is appreciated! -- Alex -- Alex Balashov - Principal Evariste Systems LLC 235 E Ponce de Leon Ave Suite 106 Decatur, GA 30030 United States Tel: +1-678-954-0670 Web: http://www.evaristesys.com/, http://www.alexbalashov.com/