This is technically a duplicate of #1279, but that one was closed with a user-space workaround. However, I think there is specific incorrect library behavior that could be improved.
Related to #1863 too.
When using the Stream API with the default options and piping directly to a Node.js http.ServerResponse
, and the upstream server returns a compressed non-chunked response, Got decompresses it but then incorrectly copies the original Content-Length
header. From the client's perspective, this results in a partial response with the rest of the upstream data missing.
If the response is large enough and the client terminates the stream once it receives the expected amount of bytes, this also results in the error:
error: Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:371:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:140:30)
at ServerResponse.emit (node:events:402:35)
at Socket.onServerResponseClose (node:_http_server:235:23)
at Socket.emit (node:events:402:35)
at TCP.<anonymous> (node:net:687:12)
The workaround in #1279 (comment) does work, but has the downside that it turns off compression entirely.
A workaround which preserves upstream compression, but technically results in an invalid HTTP response, is to delete the Content-Length
header from the upstream response:
const stream = got.stream(...);
source.on("response", () => {
delete source.response.headers["content-length"];
});
// then pipe the stream as usual
await pipeline(stream, res);
(Incidentally, the workaround posted first in #1279 (comment) didn't work.)
Got should set the correct Content-Length
header so that the http.ServerResponse
contains all of the data. I can't think of a use case where the current behavior is desirable.
import http from 'http';
import {pipeline} from 'stream/promises';
import got from 'got';
http.createServer(async (res, res) => {
const stream = got.stream.get(/ * some url with gzip compression and no chunked encoding */);
await pipeline(stream, res);
}).listen(8080);
CodeSandbox with various test cases: https://codesandbox.io/s/weathered-bird-gz9b94
The sandbox uses an internal upstream URL for testing purposes, but the bug can be replicated with real-world URLs.
Pay now to fund the work behind this issue.
Get updates on progress being made.
Maintainer is rewarded once the issue is completed.
You're funding impactful open source efforts
You want to contribute to this effort
You want to get funding like this too