The default value for listen.backlog isn't exactly "unlimited".
It's 128 on some operating systems, and -1 (which doesn't mean "unlimited" as well, but is an alias to a hard limit) on other systems.
Check for a sysctl value like kern.somaxconn (OpenBSD) or net.core.somaxconn (Linux).
Crank it up if you need more PHP workers than the default value. Then adjust listen.backlog in your php-fpm configuration file to the same value.