Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: HTTP POST of large file consumes a lot of RAM
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Daniel Stenberg via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 19 Oct 2026 10:52:12 +0200 (CEST)
On Mon, 19 Oct 2026, Marek Knápek via curl-users wrote:
> To me, this behavior seems like bad design. Or, of course, good-enough
> design back then when curl was created and there was never enough free time
> for developers to improve the behavior in presence of large files, possibly
> larger than the RAM of the computer.
From then to now, the concept of time is the same and seemingly there is never
enough.
> 1) Read in larger chunks. 4 kB is ... good for 1998. Today, I
> would suggest at least 64 kB at minimum. Of course you need to
> support large number of operating systems, for some 64 kB might be
> too much. So maybe 64 kB for 64bit operating systems only?
Did you try changing the buffer sizes and time the difference? I doubt you'll
see a difference that is greater than milliseconds on a modern machine.
But we always welcome improvements so feel free to submit your pull requests.
> 3) Memory-map the file instead of reading it into private committed
> memory. Because private memory eats my RAM and eats my SWAP, memory-mapped
> file does not.
I would urge you to take a look at the libcurl API which is what the curl tool
uses to send the data. It adds some restrictions as to what the tool side can
do and not. Or we would need to add stuff to the libcurl API, but I'm
personally not convinced we want that, at least not for this purpose.
> 4) The best option, only read a few (naturally sized, based on
> underlying FS) chunks (lets say 8) of the file into memory at a time.
> Upload them into the web server. After the oldest one is successfully
> uploaded, read another file chunk, and so on.
I think you're focusing on the wrong thing. The chunk size is not important.
If the tool would be tweaked to instead use a different libcurl API when the
data is provided as "--data-binary _at_filename" it could stream it without
having to load the entire thing. It would take some attention to detail but I
think it can be done, and then just use a kilobytes of memory instead of the
entire file's size.
Totally doable I think. Someone just needs to step up and do it.
Date: Mon, 19 Oct 2026 10:52:12 +0200 (CEST)
On Mon, 19 Oct 2026, Marek Knápek via curl-users wrote:
> To me, this behavior seems like bad design. Or, of course, good-enough
> design back then when curl was created and there was never enough free time
> for developers to improve the behavior in presence of large files, possibly
> larger than the RAM of the computer.
From then to now, the concept of time is the same and seemingly there is never
enough.
> 1) Read in larger chunks. 4 kB is ... good for 1998. Today, I
> would suggest at least 64 kB at minimum. Of course you need to
> support large number of operating systems, for some 64 kB might be
> too much. So maybe 64 kB for 64bit operating systems only?
Did you try changing the buffer sizes and time the difference? I doubt you'll
see a difference that is greater than milliseconds on a modern machine.
But we always welcome improvements so feel free to submit your pull requests.
> 3) Memory-map the file instead of reading it into private committed
> memory. Because private memory eats my RAM and eats my SWAP, memory-mapped
> file does not.
I would urge you to take a look at the libcurl API which is what the curl tool
uses to send the data. It adds some restrictions as to what the tool side can
do and not. Or we would need to add stuff to the libcurl API, but I'm
personally not convinced we want that, at least not for this purpose.
> 4) The best option, only read a few (naturally sized, based on
> underlying FS) chunks (lets say 8) of the file into memory at a time.
> Upload them into the web server. After the oldest one is successfully
> uploaded, read another file chunk, and so on.
I think you're focusing on the wrong thing. The chunk size is not important.
If the tool would be tweaked to instead use a different libcurl API when the
data is provided as "--data-binary _at_filename" it could stream it without
having to load the entire thing. It would take some attention to detail but I
think it can be done, and then just use a kilobytes of memory instead of the
entire file's size.
Totally doable I think. Someone just needs to step up and do it.
-- / daniel.haxx.se || https://rock-solid.curl.dev
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2026-10-19