master
.Server will hit some limit (I don't know what it is) and stop accept any new request.
Looks like Some connection is always alive
Host a server and keep making request on it after around 24H.
The connection should be close after 60 sec.
Get asyncio:socket.accept() out of system resource
and OSError: [Errno 24] Too many open files
error log when try to make new request on server.
I already make sure everything or browser is closed. But looks like some connection is still alive.
The ESTABLISHED
will increase by time.
run lsof
in container:
COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME
uvicorn 1 root 0u CHR 1,3 0t0 6 /dev/null
uvicorn 1 root 1w FIFO 0,13 0t0 18371653 pipe
uvicorn 1 root 2w FIFO 0,13 0t0 18371654 pipe
uvicorn 1 root 3u a_inode 0,14 0 13401 [eventpoll]
uvicorn 1 root 4u unix 0x0000000000000000 0t0 18369360 type=STREAM
uvicorn 1 root 5u unix 0x0000000000000000 0t0 18369361 type=STREAM
uvicorn 1 root 6u IPv4 18369363 0t0 TCP *:https (LISTEN)
uvicorn 1 root 7u IPv4 18413076 0t0 TCP 59c08b6aac89:https->61-66-209-161.askey.com.tw:27651 (ESTABLISHED)
uvicorn 1 root 8u IPv4 18369366 0t0 TCP 59c08b6aac89:33846->ip-172-31-28-203.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn 1 root 9u IPv4 18600291 0t0 TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:61830 (ESTABLISHED)
uvicorn 1 root 10u IPv4 18384947 0t0 TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:14377 (ESTABLISHED)
uvicorn 1 root 11u IPv4 18388349 0t0 TCP 59c08b6aac89:https->210.241.98.253:12903 (ESTABLISHED)
uvicorn 1 root 12u IPv4 18402240 0t0 TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:57692 (ESTABLISHED)
uvicorn 1 root 13u IPv4 18370708 0t0 TCP 59c08b6aac89:33856->ip-172-31-28-203.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn 1 root 14u IPv4 18369763 0t0 TCP 59c08b6aac89:56040->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn 1 root 15u IPv4 18369373 0t0 TCP 59c08b6aac89:56042->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn 1 root 16u IPv4 18370707 0t0 TCP 59c08b6aac89:56044->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn 1 root 17u IPv4 18402915 0t0 TCP 59c08b6aac89:https->61-30-51-61.static.tfn.net.tw:26547 (ESTABLISHED)
uvicorn 1 root 18u IPv4 18373916 0t0 TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:14034 (ESTABLISHED)
uvicorn 1 root 19u IPv4 18479991 0t0 TCP 59c08b6aac89:https->125-230-66-32.dynamic-ip.hinet.net:46420 (ESTABLISHED)
uvicorn 1 root 20u IPv4 18604769 0t0 TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:61864 (ESTABLISHED)
uvicorn 1 root 21u IPv4 18509169 0t0 TCP 59c08b6aac89:https->114-137-84-64.emome-ip.hinet.net:54518 (ESTABLISHED)
uvicorn 1 root 22u IPv4 18668143 0t0 TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:63229 (ESTABLISHED)
uvicorn 1 root 23u IPv4 18520950 0t0 TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:60139 (ESTABLISHED)
uvicorn 1 root 24u IPv4 18374580 0t0 TCP 59c08b6aac89:https->211-75-187-47.HINET-IP.hinet.net:56651 (ESTABLISHED)
uvicorn 1 root 25u IPv4 18376897 0t0 TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:14167 (ESTABLISHED)
uvicorn 1 root 26u IPv4 18386427 0t0 TCP 59c08b6aac89:https->210.241.98.253:12898 (ESTABLISHED)
uvicorn 1 root 27u IPv4 18399495 0t0 TCP 59c08b6aac89:https->101.10.59.24:19382 (ESTABLISHED)
uvicorn 1 root 28u IPv4 18409532 0t0 TCP 59c08b6aac89:https->36-224-32-223.dynamic-ip.hinet.net:1918 (ESTABLISHED)
uvicorn 1 root 29u IPv4 18386429 0t0 TCP 59c08b6aac89:https->210.241.98.253:12899 (ESTABLISHED)
uvicorn 1 root 30u IPv4 18388350 0t0 TCP 59c08b6aac89:https->210.241.98.253:12900 (ESTABLISHED)
uvicorn 1 root 31u IPv4 18388352 0t0 TCP 59c08b6aac89:https->210.241.98.253:12901 (ESTABLISHED)
uvicorn 1 root 32u IPv4 18494758 0t0 TCP 59c08b6aac89:https->125-230-66-32.dynamic-ip.hinet.net:47366 (ESTABLISHED)
uvicorn 1 root 33u IPv4 18372777 0t0 TCP 59c08b6aac89:56334->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn 1 root 34u IPv4 18386431 0t0 TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:14590 (ESTABLISHED)
uvicorn 1 root 35u IPv4 18370173 0t0 TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:13950 (ESTABLISHED)
uvicorn 1 root 36u IPv4 18372577 0t0 TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:13988 (ESTABLISHED)
uvicorn 1 root 37u IPv4 18530325 0t0 TCP 59c08b6aac89:https->49.216.39.2:8508 (ESTABLISHED)
uvicorn 1 root 38u IPv4 18370760 0t0 TCP 59c08b6aac89:56102->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn 1 root 39u IPv4 18388354 0t0 TCP 59c08b6aac89:https->210.241.98.253:12902 (ESTABLISHED)
uvicorn 1 root 40u IPv4 18554686 0t0 TCP 59c08b6aac89:https->210-209-175-100.veetime.com:9852 (ESTABLISHED)
uvicorn 1 root 41u IPv4 18441399 0t0 TCP 59c08b6aac89:https->36-227-149-58.dynamic-ip.hinet.net:63682 (ESTABLISHED)
...
...
...
...
There shows like 500u
or more after server up time for 24h
When it hit the limit, the server no longer can accept any new request.
docker logs
shows below:
ERROR:asyncio:socket.accept() out of system resource
socket: <asyncio.TransportSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 443)>
Traceback (most recent call last):
File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 164, in _accept_connection
conn, addr = sock.accept()
File "/usr/local/lib/python3.9/socket.py", line 293, in accept
fd, addr = self._accept()
OSError: [Errno 24] Too many open files
I have to down
and up
the docker container again.
And the number will go down to only like 10u
Run ulimit -a
in container:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63280
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 63280
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I already tried increase the open files (-n) 1024.
But it didn't resolve the problem.
Only increase the time to happen this problem.
Run server by following command:
uvicorn main:app --host 0.0.0.0 --port 443 --proxy-headers --timeout-keep-alive 60 --limit-concurrency 1000 --ssl-keyfile=./xxxx.key --ssl-certfile=./xxxx.cer
uvicorn --version
:
Running uvicorn 0.13.4 with CPython 3.9.4 on Linux
python --version
:
Python 3.9.4
cat /etc/os-release
in container:
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Pay now to fund the work behind this issue.
Get updates on progress being made.
Maintainer is rewarded once the issue is completed.
You're funding impactful open source efforts
You want to contribute to this effort
You want to get funding like this too