IIS + IP Security / IP and Domain Restrictions: Only allow localhost to access internal sites but allow external traffic through Default Web Site

I have an IIS web server that has Default Web Site open to external calls on port 443/80. Via URL Rewrite and rewrite rules, I'm routing external calls that enter through Default Web Site to the the internal web sties on other non 443/80 ports. What I want to be able to do is lock down those internal sites so that they aren't accessible to external calls. This scenario would be possible if someone off box found out what ports my internal sites have open.

I have the IP Security and IP and Domain Restrictions installed; these are for different operating systems but seem to work similarly. I am also using URL Rewrite and Application Request Routing as the reverse proxy in IIS.

Right now, with the config below, I am able to make HTTP(s) call to my Default Web Site and the requests get routed to the internal sites without problem if I am on the same box as the server.

The issue is that from another box, when trying to call Default Web Site, I still get Forbiddens.

I've tried enabling proxy mode but that didn't seem to work. I've looked at several articles but have found any that are quite like this scenario.

Is it possible to have Default Web Site open to off box calls but have my internal sites locked down so they aren't directly callable?

My applicationHost.config looks like this:

<system.webServer>
  <security>
    <ipSecurity allowUnlisted="true" /> //This is for Default Web Site
  </security>
</system.webServer>

<location path="Internal Site 1">
    <system.webServer>
        <security>
            <ipSecurity allowUnlisted="false">
                <add ipAddress="127.0.0.1" allowed="true" />
            </ipSecurity>
        </security>
    </system.webServer>
</location>
<location path="Internal Site 2">
    <system.webServer>
        <security>
            <ipSecurity allowUnlisted="false">
                <add ipAddress="127.0.0.1" allowed="true" />
            </ipSecurity>
        </security>
    </system.webServer>
</location>
<location path="Internal Site 3">
    <system.webServer>
        <security>
            <ipSecurity allowUnlisted="false" >
                <add ipAddress="127.0.0.1" allowed="true" />
            </ipSecurity>
        </security>
    </system.webServer>
</location>

Edit: My rules work without having the IP Security feature, mentioned above, on and configured.

I've also tried changing my port site bindings to have my internal HTTP port bindings set to 127.0.0.1 and [::1]. I get a 400 Bad Request, even if I try to call one of my internal sites directly, to by pass the rules, on the same box.