Based on the DOCs for deploying a AP you need to manually add the IP addresses for ESXi hosts allowed to use the proxy:
Using vSphere Authentication Proxy
You can set up other hosts to use vSphere Authentication proxy if you want to make it possible for the host to join the domain without using Active Directory credentials. That means you do not need to transmit Active Directory credentials to the host, and you do not save Active Directory credentials in the host profile.
In that case, you add the host's IP address to the vSphere Authentication Proxy access control list, and vSphere Authentication Proxy authorizes the host based on its IP address by default. You can enable client authentication to have vSphere Authentication Proxy check the host's certificate.
However for the life of me I can't find out how do to that on the Appliance. I've checked the UI (Flash and HTML5), the shell and the service config but there's nothing with regards to adding these IPs. All this has been done as the SSO admin as well. Whatever info I can find pertains to vCenter running on Windows and therefore using IIS.