Fresh pageres-cli installed today, it hammers the server with repeat requests before ultimately stopping and successfully producing the screenshot.
I noticed the issue on a production server when the generated screenshots were all errors with HTTP code 429 ("too many requests").
$ pageres http://www.mytestsite.com
Server logs, see timestamps (goes much faster on production server)
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:55 +0300] "GET / HTTP/1.1" 200 5736 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.045 "- : 0.044"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:55 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.042 "- : 0.040"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:56 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.046 "- : 0.048"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:56 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.047 "- : 0.048"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:56 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.148 "- : 0.148"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:56 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.049 "- : 0.048"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:56 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.046 "- : 0.048"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:56 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.044 "- : 0.044"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:57 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.045 "- : 0.044"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:57 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.061 "- : 0.060"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:57 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.055 "- : 0.056"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:57 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.044 "- : 0.044"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:57 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.064 "- : 0.064"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:57 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.052 "- : 0.052"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:58 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.052 "- : 0.052"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:58 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.054 "- : 0.052"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:58 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.052 "- : 0.052"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:58 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.045 "- : 0.044"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:58 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.043 "- : 0.044"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:58 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.042 "- : 0.040"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:58 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.051 "- : 0.048"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:58 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.048 "- : 0.048"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:59 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.039 "- : 0.040"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:59 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.056 "- : 0.056"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:59 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.055 "- : 0.052"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:59 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.044 "- : 0.044"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:59 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.042 "- : 0.044"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:08:59 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.042 "- : 0.044"
www.mytestsite.com "127.0.0.1" - - [21/Jun/2021:14:09:00 +0300] "GET / HTTP/1.1" 304 502 "http://www.mytestsite.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36" 0.063 "- : 0.064"
[...]
How does funding with Polar work?
Pay now to fund the work behind this issue.
Get updates on progress being made.
Maintainer is rewarded once the issue is completed.
Backer
You're funding impactful open source efforts
Contributor
You want to contribute to this effort
Maintainer
You want to get funding like this too