When sending a GET request to certain websites with caching enabled on got
,
I noticed that this is not reproducible for all websites, but found that two websites, danbooru and tatsu, using Cloudflare that have their cf-cache-status
set to DYNAMIC
appear to consistently result in freezes after attempting subsequent requests.
Other websites that don't use CloudFlare, like GitHub on Varnish, or have cf-cache-status
of HIT
or REVALIDATED
, like jsonplaceholder and oneskyapp, appear to be perfectly fine, with the response sent from cache.
docker run --name some-redis -d -p 6379:6379 redis
See: https://github.com/Aericio/got-cache-issue
You will notice that, on the first run of the code:
requesting
is the final message, no still alive
which should print after request has been completed.{
url: res.url,
cache: res.isFromCache,
"cf-cache-status": res.headers["cf-cache-status"], // cloudflare cache
"x-cache": res.headers["x-cache"] // GitHub varnish cache
}
requesting
{
url: 'https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json',
cache: true,
'cf-cache-status': undefined,
'x-cache': 'HIT'
}
still alive
requesting
{
url: 'https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json',
cache: true,
'cf-cache-status': undefined,
'x-cache': 'HIT'
}
still alive
requesting
{
url: 'https://jsonplaceholder.typicode.com/todos/1',
cache: true,
'cf-cache-status': 'HIT',
'x-cache': undefined
}
still alive
requesting
{
url: 'https://jsonplaceholder.typicode.com/todos/1',
cache: true,
'cf-cache-status': 'HIT',
'x-cache': undefined
}
still alive
requesting
{
url: 'https://support.oneskyapp.com/hc/en-us/article_attachments/202761727/example_2.json',
cache: true,
'cf-cache-status': 'REVALIDATED',
'x-cache': undefined
}
still alive
requesting
{
url: 'https://support.oneskyapp.com/hc/en-us/article_attachments/202761727/example_2.json',
cache: true,
'cf-cache-status': 'REVALIDATED',
'x-cache': undefined
}
still alive
requesting
{
url: 'https://danbooru.donmai.us/posts.json',
cache: false,
'cf-cache-status': 'DYNAMIC',
'x-cache': undefined
}
still alive
requesting
```</details>
<details>
<summary>redis keys</summary>
<br>
127.0.0.1:6379> flushall
OK
127.0.0.1:6379> keys *
- A potential cause could be that `cf-cache-status` is `DYNAMIC` here, compared to `HIT` or `REVALIDATED`.
- On the fifth test, I found another website with `cf-cache-status: DYNAMIC` header.
- After commenting out the fourth test so that I could get to the fifth test, I was met with the same behavior.
- First request goes through when cache doesn't exist, but hangs after the request has been added to the cache.
<details>
<summary>request log</summary>
<br>
```json5
... (omitted)
requesting
{
url: 'https://support.oneskyapp.com/hc/en-us/article_attachments/202761727/example_2.json',
cache: true,
'cf-cache-status': 'HIT',
'x-cache': undefined
}
still alive
requesting
{
url: 'https://tatsu.gg/lotties/tatsugotchi.json',
cache: false,
'cf-cache-status': 'DYNAMIC',
'x-cache': undefined
}
still alive
requesting
127.0.0.1:6379> flushall
OK
127.0.0.1:6379> keys *
1) "namespace:cacheable-request"
2) "cacheable-request:GET:https://tatsu.gg/lotties/tatsugotchi.json"
3) "cacheable-request:GET:https://support.oneskyapp.com/hc/en-us/article_attachments/202761727/example_2.json"
4) "cacheable-request:GET:https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json"
5) "cacheable-request:GET:https://jsonplaceholder.typicode.com/todos/1"
new Map()
as the cache.requesting
{
url: 'https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json',
cache: false,
'cf-cache-status': undefined,
'x-cache': 'HIT'
}
still alive
[Map Iterator] {
'cacheable-request:GET:https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json'
}
requesting
{
url: 'https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json',
cache: true,
'cf-cache-status': undefined,
'x-cache': 'HIT'
}
still alive
[Map Iterator] {
'cacheable-request:GET:https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json'
}
requesting
{
url: 'https://jsonplaceholder.typicode.com/todos/1',
cache: false,
'cf-cache-status': 'HIT',
'x-cache': undefined
}
still alive
[Map Iterator] {
'cacheable-request:GET:https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json',
'cacheable-request:GET:https://jsonplaceholder.typicode.com/todos/1'
}
requesting
{
url: 'https://jsonplaceholder.typicode.com/todos/1',
cache: true,
'cf-cache-status': 'HIT',
'x-cache': undefined
}
still alive
[Map Iterator] {
'cacheable-request:GET:https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json',
'cacheable-request:GET:https://jsonplaceholder.typicode.com/todos/1'
}
requesting
{
url: 'https://support.oneskyapp.com/hc/en-us/article_attachments/202761727/example_2.json',
cache: false,
'cf-cache-status': 'HIT',
'x-cache': undefined
}
still alive
[Map Iterator] {
'cacheable-request:GET:https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json',
'cacheable-request:GET:https://jsonplaceholder.typicode.com/todos/1',
'cacheable-request:GET:https://support.oneskyapp.com/hc/en-us/article_attachments/202761727/example_2.json'
}
requesting
{
url: 'https://support.oneskyapp.com/hc/en-us/article_attachments/202761727/example_2.json',
cache: true,
'cf-cache-status': 'HIT',
'x-cache': undefined
}
still alive
[Map Iterator] {
'cacheable-request:GET:https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json',
'cacheable-request:GET:https://jsonplaceholder.typicode.com/todos/1',
'cacheable-request:GET:https://support.oneskyapp.com/hc/en-us/article_attachments/202761727/example_2.json'
}
requesting
{
url: 'https://danbooru.donmai.us/posts.json',
cache: false,
'cf-cache-status': 'DYNAMIC',
'x-cache': undefined
}
still alive
[Map Iterator] {
'cacheable-request:GET:https://gist.githubusercontent.com/Aericio/3f20a5b4447b1ca6a7348f13987dc706/raw/d1a1001f17065a88acb71ff448de4a1a76db7077/data.json',
'cacheable-request:GET:https://jsonplaceholder.typicode.com/todos/1',
'cacheable-request:GET:https://support.oneskyapp.com/hc/en-us/article_attachments/202761727/example_2.json',
'cacheable-request:GET:https://danbooru.donmai.us/posts.json'
}
requesting
Pay now to fund the work behind this issue.
Get updates on progress being made.
Maintainer is rewarded once the issue is completed.
You're funding impactful open source efforts
You want to contribute to this effort
You want to get funding like this too