First of all, thank you for Got & all the other amazing projects! I really appreciate your work! 😁

What would you like to discuss?

I’m using Got & Express to create an image proxy to prevent mixed content errors (see references at the end for more information on the topic). I want to allowlist the headers received from upstream before I forward them. I started with the following:

app.get("/", async (req, res) => {
  await stream.pipeline(got.stream(req.query.url), res);
});

As per the documentation this forwards all headers from upstream:

When piping to ServerResponse, the headers will be automatically copied.

But the documentation also says what to do about it:

In order to prevent this behavior you need to override the request headers in a beforeRequest hook.

As far as I understand, that means the following:

app.get("/", async (req, res) => {
  await stream.pipeline(
    got.stream(req.query.url, {
      hooks: {
        beforeRequest: [
          (options) => {
            options.headers = {};
          },
        ],
      },
    }),
    res
  );
});

But that’s a bit weird: Why would overwriting the request headers change the behavior of forwarding the response headers?

Unsurprisingly, this didn’t work: All the upstream headers were still being forwarded.

I then came with the following:

app.get("/", async (req, res) => {
  await stream.pipeline(
    got.stream(req.query.url).on("response", (response) => {
      for (const header of Object.keys(response.headers))
        if (!["content-type", "content-length"].includes(header.toLowerCase()))
          delete response.headers[header];
    }),
    res
  );
});

This works: The upstream headers are being allowlisted.

But it leads me to two questions:

1. Am I misunderstanding something about Got’s documentation? Or should we perhaps patch the documentation?

2. Are there performance drawbacks from using an .on("response") this way? Am I now buffering the whole response on the Node.js server instead of just piping data as the stream comes in?

Note: The code above is a simplified version. The full version checks arguments, deals with errors, and so forth. I’m saying this for the benefit of people finding this in the future and trying to copy-and-paste. For reference, this is the current full implementation (if you find any issues in this, please let me know!):

app.get<{}, any, {}, { url?: string }, {}>(
  "/content/image-proxy",
  asyncHandler(async (req, res) => {
    if (
      typeof req.query.url !== "string" ||
      !["http://", "https://"].some((urlPrefix) =>
        req.query.url!.toLowerCase().startsWith(urlPrefix)
      )
    )
      return res.status(422).end();
    await stream.pipeline(
      got
        .stream(req.query.url, {
          throwHttpErrors: false,
          retry: { limit: 0 },
          timeout: { request: 10000 },
        })
        .on("response", (response) => {
          for (const header of Object.keys(response.headers))
            if (
              !["content-type", "content-length"].includes(
                header.toLowerCase()
              )
            )
              delete response.headers[header];
        }),
      res
    );
  })
);

Checklist

• I have read the documentation.

References

• https://github.com/atmos/camo
• https://github.com/imgproxy/imgproxy
• https://github.com/willnorris/imageproxy
• https://github.com/http-party/node-http-proxy
• https://github.com/chimurai/http-proxy-middleware
• https://github.com/cookpad/ecamo
• https://github.com/weserv/images
• https://github.com/jpmckinney/image-proxy
• https://github.com/sdepold/node-imageable
• https://github.com/marcjacobs1021/node-image-proxy

Edit

Oh, and one more thing: When I enabled HTTP2 (http2: true), I noticed that the HTTP status from upstream was no longer forwarded. Instead, it always came up as 200, which made me give up on using the option. Is that expected?