When I send many get requests, timeout
and response.timings
are not working as expected.
I send around 500 get requests to different urls concurrently by using promise, timings.dns
of each request is increasing.
timings.dns
should show actual time.
const got = require("got");
console.time('total');
const urlList = [
"https://devblogs.microsoft.com/landingpage/",
"https://it.impress.co.jp/list/feed/rss",
"https://uxmilk.jp/feed",
... more 500 urls
];
const pList = urlList.map(function(url) {
return got(url, {timeout: {request: 60000, }, retry: {limit: 0}}).then((result) => {
const phases = result.timings.phases;
const paheseResult = [url, phases.wait, phases.dns, phases.tcp, phases.tls, phases.request, phases.firstByte, phases.download, phases.total];
console.log(paheseResult.join(","));
}).catch((e) => {
return null;
});
});
Promise.all(pList).then((result) => {
console.timeEnd('total');
}).catch((e) => {
console.log('error: ' + e);
});
This is the graph which I made from each url's response.timings.
dns
is increasing, so if I set timeout to 30s, most of requests fails.
Sorry if this is my misunderstanding because I'm new to node, thanks in advance!
Pay now to fund the work behind this issue.
Get updates on progress being made.
Maintainer is rewarded once the issue is completed.
You're funding impactful open source efforts
You want to contribute to this effort
You want to get funding like this too