A private endpoint works from inside the VNet but on-prem users still hit the public IP. Why, and how do you fix it?
Show the answer
Say this first
Private endpoints depend on DNS. The public name has a CNAME to a privatelink name, and only networks that use the private DNS zone get the private A record. On-prem DNS asks the internet and gets the public IP. The fix is a conditional forwarder on-prem for the service's zone to a resolver inside the cloud (Azure DNS Private Resolver inbound endpoint or Route 53 Resolver inbound endpoint), so the query is answered with the private zone.
Go deeper
Azure example
mystore.blob.core.windows.net
CNAME mystore.privatelink.blob.core.windows.net
inside a VNet linked to privatelink.blob.core.windows.net -> A 10.50.2.4
anywhere else (public DNS) -> public IP- Create the private DNS zone
privatelink.blob.core.windows.net, link it to the hub VNet, and let the private endpoint register its record there. - Deploy Azure DNS Private Resolver with an inbound endpoint in the hub (or a DNS forwarder VM that uses
168.63.129.16). - On-prem DNS: conditional forwarder for
blob.core.windows.netto the inbound endpoint IP. Microsoft recommends forwarding the public zone name, not the privatelink name, so the CNAME chain is followed inside Azure.
168.63.129.16 itself is only reachable from inside Azure, which is why on-prem cannot forward to it
directly.
AWS equivalent
An interface endpoint with private DNS creates a hidden private hosted zone for the service name in that VPC. On-prem forwards the service domain to a Route 53 Resolver inbound endpoint. For endpoints shared from a central VPC, turn private DNS off on the endpoint, create your own private hosted zone for the service name with an alias to the endpoint, and associate that zone with every spoke VPC.
Design rule
One private DNS zone per privatelink name for the whole organisation, linked to the VNets or VPCs that need it. Duplicate zones in different hubs give different answers depending on where you ask.
They will ask next
The storage account has public network access disabled. What does the on-prem user see before the fix?
The name resolves to the public IP, the connection reaches the public endpoint, and the service rejects it, usually with a 403 authorization error. It looks like a permissions problem but it is DNS.
How do you prove it is DNS?
nslookup mystore.blob.core.windows.netfrom on-prem and from a VM in the VNet. Different answers (public vs10.50.2.4) tell you resolution is the issue.
Traps
- Forwarding only the privatelink zone and breaking the CNAME chain for some clients.
- Opening the firewall to the public endpoint "to make it work", which defeats the private endpoint.