[sr-dev] Fwd: Handling large responses with utils:http_query()

[Daniel Stenberg]

Hey

Responding to a forwarded message and I'm not subscribed. Excuse the 
formatting. I'm the lead developer of libcurl and was asked to comment on 
this.

>> From: Alex Balashov <abalashov at evaristesys.com>

>>    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_function);
>>
>> As far as I can tell from how write_function() is constructed, it expects 
>> to be called once:

>> 1) Does this function need to be rewritten to realloc() and concatenate?

Yes. libcurl will deliver data to the callback as soon as it has read some off 
the network, it can thus deliver one byte at a time during slow conditions or 
a whole bunch. It will keep calling the callback as long as data is coming.

This is also documented in the man page:
http://curl.haxx.se/libcurl/c/CURLOPT_WRITEFUNCTION.html

>> 2) Is there an option in CURL that can use internal buffering and only 
>> invoke the callback once all the response chunks are received?

Nope.

>> 3) Would increasing the curl.h constant CURL_MAX_WRITE_SIZE help?

No. That would just make libcurl able to read a larger chunk from the socket 
into its own buffer, but it wouldn't prevent it from calling the callback one 
or many times.

-- 

  / daniel.haxx.se