Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
HTTP POST of large file consumes a lot of RAM
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Marek Knápek via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 19 Oct 2026 10:13:02 +0200
Hi curl maintainers,
I noticed a large RAM usage when running the curl program.
I used the curl command-line tool to upload a rather large file to my
web server using the POST method.
There is a problem with my web server, it is behind a proxy server,
and it is not consuming the incoming data. Thus the curl process
hangs. So far so good, all expected behavior. But the curl process is
quite large. In my case about 233 MB when uploading a 231 MB file.
Later, I repeated the experiment with a 2 GB file and, to my
surprise, the curl process consumed 4 GB of RAM for a short moment,
then settled to 2 GB, and hang. Again, the hanging is not a problem,
it is caused by my hand-written web server. The entire file in memory
at once is the problem.
I used the API Monitor tool (similar to strace) to see what is
happening. And really, the curl reads the entire file to its memory.
It is doing this by 4 kB chunks using the fread C standard library
function.
To me, this behavior seems like bad design. Or, of course,
good-enough design back then when curl was created and there was
never enough free time for developers to improve the behavior in
presence of large files, possibly larger than the RAM of the
computer.
I have a few suggestions to improve this situation.
1) Read in larger chunks. 4 kB is ... good for 1998. Today, I
would suggest at least 64 kB at minimum. Of course you need to
support large number of operating systems, for some 64 kB might be
too much. So maybe 64 kB for 64bit operating systems only?
2) Query the file, or its underlying file-system, for a natural
cluster size. And use that size to read the file in chunks. For
Windows OS GetDiskFreeSpaceW, GetVolumePathNameW,
GetFileInformationByHandleEx functions could be used for this
purpose.
3) Memory-map the file instead of reading it into private
committed memory. Because private memory eats my RAM and eats my
SWAP, memory-mapped file does not.
4) The best option, only read a few (naturally sized, based on
underlying FS) chunks (lets say 8) of the file into memory at a time.
Upload them into the web server. After the oldest one is successfully
uploaded, read another file chunk, and so on.
Of course everything is easier said than done. I have no code to
offer to you.
My command line is:
curl -v -i -X POST -x http://127.0.0.1:4444 http://example.com/ --data-binary _at_big-file.dat
My operating system is:
Windows Server 2022 Standard [Version 21H2 10.0.20348.5761] 64-bit
My curl version is (distributed together with Windows):
curl 8.21.0 (Windows) libcurl/8.21.0 Schannel zlib/1.3.2 WinIDN WinLDAP
Release-Date: 2026-06-24
My other curl version is (downloaded from curl.se):
curl 8.23.0 (x86_64-w64-mingw32) libcurl/8.23.0 LibreSSL/4.3.3 zlib/1.3.1.zlib-ng brotli/1.2.0 zstd/1.5.7 WinIDN libpsl/0.23.3 libssh2/1.11.1 nghttp2/1.70.0 ngtcp2/1.25.0 nghttp3/1.18.0 WinLDAP
Release-Date: 2026-10-14
Marek
Date: Mon, 19 Oct 2026 10:13:02 +0200
Hi curl maintainers,
I noticed a large RAM usage when running the curl program.
I used the curl command-line tool to upload a rather large file to my
web server using the POST method.
There is a problem with my web server, it is behind a proxy server,
and it is not consuming the incoming data. Thus the curl process
hangs. So far so good, all expected behavior. But the curl process is
quite large. In my case about 233 MB when uploading a 231 MB file.
Later, I repeated the experiment with a 2 GB file and, to my
surprise, the curl process consumed 4 GB of RAM for a short moment,
then settled to 2 GB, and hang. Again, the hanging is not a problem,
it is caused by my hand-written web server. The entire file in memory
at once is the problem.
I used the API Monitor tool (similar to strace) to see what is
happening. And really, the curl reads the entire file to its memory.
It is doing this by 4 kB chunks using the fread C standard library
function.
To me, this behavior seems like bad design. Or, of course,
good-enough design back then when curl was created and there was
never enough free time for developers to improve the behavior in
presence of large files, possibly larger than the RAM of the
computer.
I have a few suggestions to improve this situation.
1) Read in larger chunks. 4 kB is ... good for 1998. Today, I
would suggest at least 64 kB at minimum. Of course you need to
support large number of operating systems, for some 64 kB might be
too much. So maybe 64 kB for 64bit operating systems only?
2) Query the file, or its underlying file-system, for a natural
cluster size. And use that size to read the file in chunks. For
Windows OS GetDiskFreeSpaceW, GetVolumePathNameW,
GetFileInformationByHandleEx functions could be used for this
purpose.
3) Memory-map the file instead of reading it into private
committed memory. Because private memory eats my RAM and eats my
SWAP, memory-mapped file does not.
4) The best option, only read a few (naturally sized, based on
underlying FS) chunks (lets say 8) of the file into memory at a time.
Upload them into the web server. After the oldest one is successfully
uploaded, read another file chunk, and so on.
Of course everything is easier said than done. I have no code to
offer to you.
My command line is:
curl -v -i -X POST -x http://127.0.0.1:4444 http://example.com/ --data-binary _at_big-file.dat
My operating system is:
Windows Server 2022 Standard [Version 21H2 10.0.20348.5761] 64-bit
My curl version is (distributed together with Windows):
curl 8.21.0 (Windows) libcurl/8.21.0 Schannel zlib/1.3.2 WinIDN WinLDAP
Release-Date: 2026-06-24
My other curl version is (downloaded from curl.se):
curl 8.23.0 (x86_64-w64-mingw32) libcurl/8.23.0 LibreSSL/4.3.3 zlib/1.3.1.zlib-ng brotli/1.2.0 zstd/1.5.7 WinIDN libpsl/0.23.3 libssh2/1.11.1 nghttp2/1.70.0 ngtcp2/1.25.0 nghttp3/1.18.0 WinLDAP
Release-Date: 2026-10-14
Marek
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2026-10-19