Hi all,
After a lot of troubleshooting on our side, we found the cause of the problem and applied a fix. It was not a Three outage. It comes from a very unusual network implementation at Three Ireland, combined with standard firewall behaviour. I'm posting the details here in case anyone else runs into the same problem.
Background: public vs private IP addresses
Public IP: routable on the internet and used to reach services from outside.
Private IP (RFC 1918): 10.0.0.0/8 (10.0.0.0–10.255.255.255), 172.16.0.0/12 (172.16.0.0–172.31.255.255) and 192.168.0.0/16 (192.168.0.0–192.168.255.255). There is also 100.64.0.0/10, the carrier-grade NAT (CGNAT) range for operators (RFC 6598).
Our setup
We have a Three 5G broadband connection with a public IP, using the open.internet.public APN. On our firewall we use port forwarding to publish a web portal for our customers and ourselves. The same situation can apply to CCTV recorders, NAS devices or any other service you expose this way.
What we saw
- From a Three mobile phone (internet APN) to the Three 5G broadband public IP on ports 80/443, the connection failed.
- From any non-Three connection (another mobile operator or broadband) to the same public IP and ports, the connection worked.
Root cause
Three smartphones use the internet APN by default. This APN gives the phone a private IP in the 10.0.0.0/8 range, and Three's core network applies CGNAT. That means many customers share a single public IPv4 address, which saves public IPv4 addresses because they are scarce.
Normally, all traffic from a phone behind CGNAT should be translated to a public IP before it reaches any destination. Three Ireland does something very unusual here. When a Three mobile phone connects to another Three customer's public IP, the traffic seems to be routed internally without going through the CGNAT. So the request reaches our firewall with a 10.x.x.x private source address on the WAN interface.
Like most firewall vendors, our firewall assumes private IPs can only come from the LAN, never from the WAN. This follows RFC 1918, which says border routers should filter packets from outside that claim to come from private address space, because they are considered spoofed. The firewall dropped the traffic before the port forward (NAT) rule was applied. In other words, the firewall did exactly what the standards expect.
What Three should ideally do
Using the RFC 6598 range (100.64.0.0/10) for CGNAT instead of 10.0.0.0/8 would avoid overlap with customers' own private networks. On its own, though, it would not solve this problem. The correct behaviour is for mobile traffic to go through the CGNAT and reach other Three customers' public IPs with a public source address, as RFC 1918 and NAT best practices (RFC 4787 / RFC 6888) expect. At the very least, this behaviour should be documented.
How to fix it on your side
Allow traffic from 10.0.0.0/8 arriving on the WAN interface, limited to the published ports only (for example, 80/443 to the server). Depending on your vendor, you may also need to adjust the anti-spoofing or reverse-path check on that WAN interface. Keep the exception as narrow as possible.
Summary
If your service on a Three Ireland public IP works from everywhere except Three mobile phones, check whether your firewall is dropping private source IPs (10.x.x.x) on the WAN interface. This is caused by the way Three Ireland routes internal traffic between its mobile and broadband customers, and it is not documented anywhere.
Hope this saves someone a few hours!
Note for Three @Three_Niall
I know this may be quite technical and detailed, but I think it should be passed on to Three's engineering team. In my view, this points to a design flaw in the Three Ireland core network, and I'm not sure whether the engineering team is aware of it. All three business customers are affected by this bug/issue.
If you need any clarification or would like me to explain anything, feel free to send me a DM.
