From the pi forum thread, your problem isn’t your computer’s RAM, nothing to do with DHCP (or even IP protocol/addressing). It’s at a lower layer in the Wifi chip (separate from the Pi CPU). It’s not even getting to the Pi’s operating system. The nth PB just can’t associate with the network since the wifi chip is out of memory/resources.
For what it’s worth, Pixelblaze in AP mode has similar limitations (10 clients for recent V3 versions).
The wifi chip seems to load a firmware when the driver loads, and the suggestions on the pi forum point to a minimal firmware for the wifi chip that should leave it with more resources. I’d give that a try. If you follow those links, the github readme here claims that the minimal firmware supports up to 19 clients.
If that doesn’t work out, you can get a cheap wifi router ($15-$25 or so) and use that for hosting the network instead of the pi.
I’ve used these TP link routers, even powering them from the PI’s USB for small portable setups, though these aren’t the cheapest anymore, still not bad for $25: