The user is asking a classic networking problem: NAT hairpinning (also known as NAT loopback, hairpin NAT, or U-turn NAT). Internal clients connecting to the public IP of their own network fail because the edge device doesn't support or isn't configured for hairpin NAT. External clients work because normal destination NAT + port forward works fine. Let me think through the topology: - Public IP: 139.99.148.90 (this is the edge/public-facing IP) - Internal host: 10.10.10.100, service on port 7701 - The public IP:443 maps (DNAT/port forward) to 10.10.10.100:7701 presumably. The requirement: "A service at 7701 on internal host 10.10.10.100 must be reachable from outside via public 139.99.148.90:443." So there's a DNAT rule: 139.99.148.90:443 → 10.10.10.100:7701. - Internal clients connecting to 139.99.148.90:443 hang. - External clients work. Why do internal clients hang? 1. **No hairpin NAT**: The edge router/firewall receives a packet from an internal source (10.10.10.x) destined to its own public IP 139.99.148.90:443. If the device doesn't support hairpinning, it either: - Drops the packet (connection times out → "hang") - Performs DNAT but then the server replies directly to the internal client with source 10.10.10.100:7701; client expects traffic from 139.99.148.90:443, sees reply from different IP, sends RST or drops → connection hangs. - Asymmetric routing: client's SYN goes out the default gateway to the edge, gets DNAT'ed, server replies to client directly via local subnet (because 10.10.10.x is directly connected), bypassing the edge, so the NAT translation is never traversed in the return path → connection established but hangs/reset. Actually with TCP, the SYN-ACK would come from 10.10.10.100:7701 to client, but client's socket was opened to 139.99.148.90:443; kernel sees SYN-ACK from a different tuple, sends RST. Client sees "connection reset" or hangs depending on firewall. 2. **Split-horizon DNS absent**: Many designs fix this by internal DNS returning the internal IP for the service name. But the question says clients connect to the public IP directly, so DNS isn't necessarily in play. But it's worth mentioning as an alternative fix (split DNS) — but the requirement is that internal clients connect to the public IP; the fix plan should keep external access working. So the root cause: lack of NAT hairpin (loopback) support/configuration on the edge device. Ranked root-cause hypotheses: 1. **No hairpin NAT (NAT loopback) on the edge firewall/router** — the DNAT rule only matches packets arriving on the WAN interface; LAN-sourced packets to the public IP are either not matched, or the reverse path is asymmetric. Probability: very high (~85%). 2. **Partial hairpin: DNAT matches but no matching SNAT (masquerade) for internal→internal flow**, causing asymmetric return path: server replies directly to client over LAN, client kernel rejects the SYN-ACK (tuple mismatch) → hang/RST. Probability: ~10%. Actually this is a subcase of hairpin config: the fix is adding SNAT/masquerade rule for LAN→LAN-NAT flows. 3. **Asymmetric routing/return path via different gateway** (e.g., internal client default gateway is a different L3 device than the NAT device; the NAT device's rule for hairpin never sees traffic or the return path bypasses it). Probability: small. 4. **Firewall zone policy blocking LAN→WAN-IP or LAN→LAN via NAT** (interzone rules, anti-spoofing "block traffic to WAN IP from LAN", uRPF strict mode dropping packets from LAN source arriving at WAN-destined IP, martian packet drops). Probability: small but plausible. 5. **NAT reflection misconfiguration**: NAT reflection enabled but with wrong translation (e.g., reflection translates destination but not source; or reflection rule maps port incorrectly). Also possible: the device translates 443→7701 but the reflected connection returns source port 7701 which the client... actually that's fine. Hmm, but a common issue with reflection: the reflected DNAT changes destination to 10.10.10.100:7701, but the return packet from the server to the internal client is not SNAT'ed back, so the client sees source 10.10.10.100:7701 instead of 139.99.148.90:443 → kernel sends RST → connection "hangs"/resets. So hypothesis 2 covers this. 6. **STP/asymmetric path or host firewall**: 10.10.10.100 host firewall allows only source from WAN interface or only public sources; internal client traffic arrives from 10.10.10.x and is dropped by host firewall → hang. Worth verifying. 7. **Client-side proxy/PAC forcing traffic out and back** — less likely. Verification commands per hop: Hop 1: Internal client (e.g., 10.10.10.50): - `curl -v --connect-timeout 5 https://139.99.148.90:443/` (with -k for TLS) → observe hang (SYN sent, no response) or reset. - Packet capture: `sudo tcpdump -ni any host 139.99.148.90` or better `tcpdump -ni eth0 'tcp port 443 and host 139.99.148.90'` → see SYN retransmissions, no SYN-ACK. Or see SYN-ACK from 10.10.10.100:7701 (indicates asymmetry). Hop 2: On-server capture on 10.10.10.100: - `sudo tcpdump -ni any 'tcp port 7701'` while client tests. - Interpret: - If SYN arrives with source 10.10.10.50 (client's real IP) and server replies SYN-ACK → asymmetric return path (no SNAT on hairpin). The server's SYN-ACK goes directly to client; client expects 139.99.148.90:443 → RST/hang. - If SYN arrives with source = edge's inside IP (e.g., 10.10.10.1) → hairpin SNAT is working; then problem elsewhere (server firewall?). - If no SYN arrives at all → DNAT/reflection not matching LAN-sourced traffic (hairpin unsupported) or edge drops. Hop 3: Edge device: - Check NAT rules: `iptables -t nat -L -nv` / `nft list ruleset`, or vendor CLI: e.g., `show nat rules` / policy lookup. On pfSense: NAT reflection settings; on Cisco ASA: `show nat`, hairpin u-turn with `same-security-traffic permit intra-interface`, NAT hairpinning config; on Linux: check that DNAT rule `iptables -t nat -A PREROUTING -d 139.99.148.90 -p tcp --dport 443 -j DNAT --to-destination 10.10.10.100:7701` exists and whether there's a matching rule in POSTROUTING/OUTPUT with SNAT. - Key check: does the PREROUTING DNAT rule match only on WAN ingress (`-i eth0`)? If it has `-i wan0`, LAN-sourced packets arriving on LAN interface won't match → no DNAT → packet routed to default gateway? Actually if the destination is the edge's own IP, it goes to INPUT chain, not forwarded → nothing listening on 443 on the edge → RST or drop → hang. If DNAT is in PREROUTING without interface match, it would apply. - Capture on edge: `tcpdump -ni lan0 host 139.99.148.90` and `tcpdump -ni wan0 port 443` to see whether the SYN reaches edge, whether DNAT happens, and whether the SYN-ACK comes back through edge. Hop 4: Intermediate L3 switch (if any): check routing for 139.99.148.90 — likely default route to edge. `show ip route 139.99.148.90` or `ip route get 139.99.148.90` from a Linux intermediary. Also verify from external vantage that it still works: `curl -v https://139.99.148.90:443` from an external host/phone hotspot — baseline control to ensure changes don't break external. MTU issues? No — SYN hang is not MTU. Brasero... okay let me structure the fix plan. Fix options (ranked): **Fix A (preferred, keeps external intact): Enable/configure hairpin NAT on edge.** Generic Linux (iptables/nftables) model — many edge devices are Linux-based: 1. DNAT rule should match LAN-sourced traffic to public IP: either remove `-i wan` constraint or add a second rule: `iptables -t nat -A PREROUTING -i lan0 -d 139.99.148.90 -p tcp --dport 443 -j DNAT --to-destination 10.10.10.100:7701` 2. Add SNAT/masquerade for hairpin flow so replies traverse the edge: `iptables -t nat -A POSTROUTING -o lan0 -s 10.10.10.0/24 -d 10.10.10.100 -p tcp --dport 7701 -j SNAT --to-source 10.10.10.1` (edge's LAN IP) or `... -j MASQUERADE` (but SNAT is cleaner/deterministic). 3. Ensure FORWARD chain allows the flow: `iptables -A FORWARD -i lan0 -o lan0 -d 10.10.10.100 -p tcp --dport 7701 -j ACCEPT` and return traffic (stateful rule `-m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT` usually exists). pfSense/OPNsense: Firewall → NAT → Port Forward: check "NAT Reflection" — enable "NAT Reflection mode: NAT" (pure NAT) and "Automatic outbound NAT for reflection" (or add manual outbound rule). Note: "Enable NAT Reflection mode for 1:1 and Port Forwards". Alternatively add a rule on LAN: source LAN net, destination WAN address port 443 → use "NAT reflection". In OPNsense: Services → DHCP... no, Firewall → Settings → NAT Reflection: "Enable NAT Reflection for 1:1 and Port Forward" + "Automatic outbound NAT for reflection". Cisco ASA: `same-security-traffic permit intra-interface` + NAT (u-turn): object NAT for hairpin: e.g., ``` object network SERVICE host 10.10.10.100 nat (inside,inside) static 139.99.148.90 service tcp 443 7701 ``` plus `same-security-traffic permit intra-interface`. FortiGate: VIP with "nat44" hairpin — FortiGate supports hairpin if internal clients' route to VIP goes through FortiGate; may need "VIP matching" and no special config, but often requires disabling "set nat-source-vip"? Actually for FortiGate hairpin, the VIP works if traffic goes through the FortiGate; add `config system virtual-ip` with `set portforward enable`. Some need `set src-filter`? The generic plan should be vendor-neutral with examples. Juniper SRX: NAT with `junos-` hairpin requires `nat` rule in both directions plus `security policies`; SRX supports hairpinning if route exists. MikroTik: dst-nat rule + masquerade rule for src-address=LAN dst-address=internal host. **Fix B (avoids NAT entirely, zero risk to external): Split-horizon DNS.** Internal DNS zone overrides the FQDN to return 10.10.10.100 (or the edge's LAN IP with a second forward on 443? No—service is 7701 internally, so point DNS to 10.10.10.100 and clients must connect to :7701, which changes the port... unless internal clients should still use :443 — then point DNS to edge LAN IP and add port-forward on LAN IP:443→10.10.10.100:7701, or just document internal URL with port 7701). But the problem statement says internal clients connect to the public IP; DNS fix only works if they connect by name. Mention as alternative if the app uses DNS name. Since the ask emphasizes "does not break external access", Fix B is inherently safe but may change internal URL/port. Keep as option. **Fix C: NAT-on-a-stick / policy routing alternative**: If edge can't do hairpin, add a static route so the internal client's traffic to 139.99.148.90 loops through a Linux router that does the DNAT+SNAT. Complex; not preferred. Rollback plan: - For iptables: capture baseline `iptables-save > /root/iptables-before.txt` (and `nft list ruleset > nft-before.txt`), make changes with numbered inserts or a test window, and rollback via `iptables-restore /root/iptables-before.txt` (or `nft -f nft-before.txt`). Test: external curl before/after each change. - For pfSense: note the NAT reflection settings before change; revert by setting reflection back to "default/disable"; config backups via Diagnostics > Backup, or `copy running-config` style for CLI vendors: Cisco ASA `copy running-config startup-config` before? Rollback: `configure terminal` then remove the added NAT line and `no same-security-traffic permit intra-interface` — but careful: removing `intra-interface` doesn't affect external. For any change, keep the exact pre-change config file and the exact commands to undo. - Timing: make one change at a time; verify external access after each change (`curl https://139.99.148.90:443` from external vantage point), plus internal test. Order of operations for verification (per hop): 1. From internal client: establish failure mode precisely: - `curl -vk --connect-timeout 5 https://139.99.148.90:443/` → hang (timeout) vs RST. - `sudo tcpdump -ni host 139.99.148.90` → SYN going out? retransmits? any SYN-ACK? From what source IP? Interpretation matrix: - SYN sent, no SYN-ACK, captures on server show nothing → edge not translating (no hairpin support/rule mismatch). - SYN-ACK arrives from 10.10.10.100:7701 → asymmetric return (SNAT missing). - SYN-ACK arrives from 139.99.148.90:443 but TLS hangs later → MTU/MSS or TLS interception (less likely; note it). - ICMP unreachable → routing/firewall drop (look at type/code: admin prohibited → firewall rule). 2. On server 10.10.10.100: `sudo tcpdump -ni any 'tcp port 7701'` during internal client test. Confirms whether traffic arrives and what source it carries. 3. On edge (LAN interface): `tcpdump -ni lan0 'host 139.99.148.90 or host 10.10.10.100'` → does the edge see the SYN? Does it send anything back? Check NAT table state: `conntrack -L | grep 139.99.148.90` (Linux-based) or vendor equivalent (`show nat translations`, `show xlate` on ASA: `show xlate detail`, `show nat detail`). 4. Check for drops: edge logs; Linux: `iptables -L FORWARD -nv -x` counters before/after test, `journalctl -k`, or `conntrack events`. ASA: `packet-tracer input inside tcp 10.10.10.50 12345 139.99.148.90 443` — brilliant diagnostic on ASA. FortiGate: `diag sniffer packet`, `diag firewall sniffer`? and `diag sys nattrace`. Juniper: `security flow traceoptions`. Provide a couple of vendor-specific examples since we don't know the vendor. 5. Control tests: from external host: `curl -v https://139.99.148.90:443/` works (baseline). From internal host directly to service: `curl -v --connect-timeout 5 http://10.10.10.100:7701/` (or `nc -vz 10.10.10.100 7701`) → service itself is up internally, eliminating server-side cause. Also `curl` from another internal subnet/VLAN if applicable to see if same-subnet vs cross-subnet differs (hairpin often fails only via edge; if client is on same subnet as server, traffic normally never hits the edge — but to the public IP it must). Also check server firewall: `sudo iptables -nvL` on 10.10.10.100 or `ss -tlnp | grep 7701` to confirm listening. But since external works, service and DNAT are fine — the failure is specific to internal-origin path. That's the strong evidence pointing to hairpinning. Root-cause ranked list (with rough confidence): 1. ~80%: Edge lacks NAT loopback/hairpin for LAN-originated traffic to its own public IP (DNAT rule bound to WAN interface, or device simply drops "martian"/its-own-IP traffic from LAN). Symptom matches: external OK, internal SYN goes out, never comes back → hang. 2. ~15%: Hairpin DNAT occurs but no SNAT back → asymmetric path; server SYN-ACK bypasses edge; client RSTs (may manifest as hang or immediate reset). Distinguishable by server-side capture seeing client's real source IP. 3. ~3%: Interzone/anti-spoof/uRPF drop on edge for LAN→WAN-IP (martian filtering), or explicit firewall rule blocking LAN to WAN address zone. Log evidence. 4. ~2%: Path/other: server host firewall drops LAN-source connections (but external works, so host firewall would have to be source-selective), NAT translation table exhaustion (no — external works), MTU (no — SYN-level hang). Fix plan (stepwise, lowest risk first), with explicit "does not break external access" guarantees: Step 0: Change-control baseline: - Record configs: `iptables-save`, `nft list ruleset`, vendor `show running-config`, pfSense XML backup. - Baseline tests: external curl OK; internal curl fails (document). - Schedule brief change window; changes are additive so external impact is nil, but still verify. Step 1: Add hairpin DNAT (only if step shows rule bound to WAN interface): - Linux/nftables: ``` # iptables -t nat -A PREROUTING -i lan0 -d 139.99.148.90 -p tcp --dport 443 -j DNAT --to-destination 10.10.10.100:7701 # iptables -t nat -A POSTROUTING -d 10.10.10.100 -p tcp --dport 7701 -s 10.10.10.0/24 -j SNAT --to-source 10.10.10.1 ``` nftables equivalent: ``` table ip nat { chain prerouting { type nat hook prerouting priority -100; iif "lan0" ip daddr 139.99.148.90 tcp dport 443 dnat to 10.10.10.100:7701 } chain postrouting { type nat hook postrouting priority 100; ip saddr 10.10.10.0/24 ip daddr 10.10.10.100 tcp dport 7701 snat to 10.10.10.1 } } ``` Wait — need the actual inside interface names; use placeholders and note to replace. Also note: on many setups the generic DNAT rule without `-i` already covers hairpin, and only SNAT is missing. Step 2 (if vendor appliance): - pfSense/OPNsense: enable NAT Reflection (mode: NAT) + automatic outbound NAT for reflection. Zero impact on external rules (reflection only affects LAN-originated traffic). - Cisco ASA: `same-security-traffic permit intra-interface` + `nat (inside,inside) static ... service tcp 443 7701` in an object. External unaffected (outside rules untouched). - FortiGate: ensure internal clients route to VIP through FortiGate; VIPs hairpin by default when routing is correct; verify with `diag sniffer packet lan 'port 443' 4` and `diag firewall proute list`? Keep it brief. - MikroTik: `/ip firewall nat add chain=dstnat dst-address=139.99.148.90 protocol=tcp dst-port=443 in-interface=bridge-lan action=dst-nat to-addresses=10.10.10.100 to-ports=7701` + `/ip firewall nat add chain=srcnat src-address=10.10.10.0/24 dst-address=10.10.10.100 protocol=tcp dst-port=7701 action=masquerade`. Step 3: Allow forward across same zone (LAN→LAN) if policy engine requires it; add stateful rule; note many firewalls by default allow established/related. Step 4: Verify: - Internal: `curl -vk --connect-timeout 5 https://139.99.148.90:443/` → TLS handshake proceeds; expect cert name mismatch warning (use -k) — service reachable. Or if app-level check: whatever the service speaks on 7701; the external fronting is 443. - External: repeat external curl — must still work; also test after each change. - Check conntrack entry exists for hairpin flow: `conntrack -L -d 10.10.10.100 | grep `. - Edge counters increase on the new rule. Step 5: Persistence: save rules (`iptables-save > /etc/iptables/rules.v4` or netfilter-persistent / firewalld rich rules / vendor config save `copy run start` / pfSense auto-persists). Rollback: - Linux: `iptables-restore < /root/nat-before.v4` (or `nft -f nft-before.txt`), restore prior file. Because changes are additive with unique comments, individual removal is possible: `iptables -t nat -D PREROUTING -i lan0 ...`. - pfSense: disable NAT reflection / restore config backup. - ASA: `no object network ...; no nat ...; no same-security-traffic permit intra-interface` (removing last doesn't affect external flows). - MikroTik: remove the two rules by ID. - Verification after rollback: external OK (unchanged), internal returns to prior state (documented). Also mention pitfalls: - Don't "fix" by disabling `rp_filter`? Actually uRPF strict on the edge could drop LAN→own-IP? Not typical; the packets arrive on the correct interface for their source. Skip or brief. - Beware double-NAT of client source if masquerade rules for outbound already SNAT LAN→WAN; a POSTROUTING masquerade with `-o wan0` won't catch `-o lan0` flows, so a dedicated rule is needed. Good. - TTL/loop: hairpin can create loops if edge forwards LAN→WAN-IP→DNAT back out LAN; with DNAT+SNAT done properly in one device, no loop. If hairpin unsupported and device forwards to upstream ISP, the ISP will drop (uRPF) → hang — this also explains hang with a device that doesn't intercept its own IP on LAN ingress. Worth noting in hypothesis 1 details. - TCP MSS/TLS issues not relevant at SYN stage. Alternative fix (if hairpin is impossible on the box): internal DNS split-horizon returning 10.10.10.100 for the service FQDN; internal clients use `http://name:7701` (or the LAN IP of the edge with a second forward). Zero external risk. Also: add a second DNAT rule for edge's LAN IP `10.10.10.1:443 → 10.10.10.100:7701` so internal clients can still use :443 with a LAN address. Mention as Plan B with its own rollback (delete DNS record / restore A record TTL considerations — low TTL beforehand is wise). Also a last-resort: configure clients' hosts file (not scalable). Now the report format: "Report: root-cause hypothesis ranked, verification command per hop, fix, rollback." So deliver a structured incident-style report: 1. Topology & symptom summary 2. Ranked hypotheses (with likelihood, signature, discriminating evidence) 3. Hop-by-hop verification matrix (hop, command, what to look for, decision) 4. Fix plan (primary + alternate), per-vendor snippets 5. Verification after fix (internal + external) 6. Rollback plan (per step, commands) 7. Risks & notes (change window, one change at a time, persistence) Length: This is a technical deep-dive; detailed is appropriate. Use headers, tables, code blocks. Markdown appropriate. Should be thorough but not bloated — aim for a solid, organized report. Probably ~800–1200 words with command blocks. Let me be careful to keep commands realistic and mark placeholders (IF_LAN, GW_LAN_IP) clearly. Decision tree interpretation of tcpdump results — key discriminating evidence: | Evidence at server capture | Meaning | |---|---| | No SYN ever arrives | Edge isn't translating/reflection rule mismatch (H1) or drops | | SYN arrives with client real source IP 10.10.10.x | DNAT happened but no SNAT → asymmetric return (H2) | | SYN arrives with edge's LAN IP as source | SNAT working; check server firewall/app | And client-side tcpdump: | Evidence at client | Meaning | |---|---| | SYN retrans, no reply | traffic swallowed (H1/H4) | | SYN-ACK from 10.10.10.100:7701 | asymmetric (H2) — kernel RSTs | | ICMP net unreachable/admin prohibited | firewall drop (H4) | Edge-side checks per vendor — include generic + examples. Also, quick note: because external works, we KNOW the DNAT 139.99.148.90:443→10.10.10.100:7701 and the service are fine; the failing variable is the ingress path (LAN vs WAN). That inference should be stated early — it narrows diagnosis massively and is good analytical hygiene. Let me also include the ASA packet-tracer and FortiGate diag sniffer, MikroTik torch/tail, pfSense states listing. Keep them short. Let me draft. --- # Diagnosis report — hairpin NAT failure ## 1. Situation summary - DNAT in place (external works): 139.99.148.90:443 → 10.10.10.100:7701 - Failure domain: only LAN-originated clients hitting the public VIP. Since service + DNAT + policy for WAN→LAN path are proven good, root cause is almost certainly in how the edge handles traffic from LAN to its own public address — i.e., NAT loopback/hairpin. ## 2. Ranked hypotheses H1 (≈80%): No NAT reflection/hairpin. DNAT rule matches WAN ingress only (or device refuses to route its own IP from LAN). SYN goes to edge and either dies in INPUT (nothing on 443) or is forwarded to the ISP and uRPF-dropped. Symptom: silent timeout. Discriminator: no SYN on server; no conntrack entry. H2 (≈15%): Reflection translates destination but not source (no SNAT for LAN→VIP-NAT flows). Asymmetric return: SYN-ACK from 10.10.10.100:7701 goes straight over LAN to client; client's socket expects 139.99.148.90:443 → kernel sends RST, app sees hang/reset. Discriminator: server capture shows client's real source IP; client capture shows SYN-ACK from internal IP. H3 (≈4%): Policy drop: interzone rules, anti-martian/uRPF, or explicit "LAN → WAN-address" block on the edge. Discriminator: drop counters/logs; ICMP admin-prohibited. H4 (≈1%): Path-external-to-NAT: client default route via a different L3 hop that sends 139.99.148.90 out the ISP without loopback (edge never sees it), or server host firewall with source-based rules. Discriminators: `ip route get`, server iptables counters. (Also note H2′: hairpin enabled but reflection NAT mode uses "pure NAT" vs "split DNS"... skip; keep 4 hypotheses.) ## 3. Verification per hop Run internal test while capturing. One test run, three captures in parallel. Hop A — internal client 10.10.10.50: ``` curl -vk --connect-timeout 5 https://139.99.148.90:443/ sudo tcpdump -ni any 'host 139.99.148.90 and tcp port 443' ``` Interpretation table (hang vs RST vs ICMP). Hop B — edge, LAN iface: ``` tcpdump -ni 'host 139.99.148.90' conntrack -L 2>/dev/null | grep '139.99.148.90' # Linux-based edges iptables -t nat -S | grep -i 139.99.148.90 # rule binding? -i wan? ``` Vendor one-liners: ASA `packet-tracer input inside tcp 10.10.10.50 40000 139.99.148.90 443`; pfSense Diagnostics > States filter; MikroTik `/ip firewall connection print` + `/tool sniffer`. Hop C — server 10.10.10.100: ``` sudo tcpdump -ni any 'tcp port 7701' ss -tlnp | grep 7701 # service sanity ``` Hop D — control from outside (baseline before any change): ``` curl -v https://139.99.148.90:443/ # must succeed — re-run after every change ``` and same-subnet/local direct: ``` nc -vz 10.10.10.100 7701 ``` Decision tree mapping to H1..H4. ## 4. Fix Primary: enable hairpin on edge — two rules (DNAT for LAN-sourced hits on the VIP; SNAT for LAN→server 7701 flows), plus allow LAN→LAN forward. Additive only; external path untouched. Provide Linux/nft, pfSense/OPNsense, ASA, MikroTik snippets. Notes: use SNAT to edge LAN IP rather than masquerade for determinism; if a generic DNAT without `-i` exists, only SNAT missing. Fallback if edge can't hairpin: Plan B DNS split-horizon (internal A record → 10.10.10.100, clients to :7701) or DNAT on edge LAN IP:443 → keep 443. Zero external risk. ## 5. Post-fix verification - internal curl to https://139.99.148.90:443 (expect TLS OK or app banner; -k for self-signed) - external curl (unchanged, must still pass) - conntrack entry with both LAN src and dst NAT'ed - new rule counters increment ## 6. Rollback - Before: capture `iptables-save`, `nft list ruleset`, vendor config export, pfSense XML backup; low TTL on DNS if Plan B. - iptables: delete the two added rules by exact spec or full restore file. - nft: `nft delete rule ... handle ` or restore file. - pfSense: set reflection back to default/disable; restore backup. - ASA: negate added commands. - MikroTik: remove by ID. - Confirm external still OK after rollback (it never depended on any of it — but verify). ## 7. Pitfalls - Do both DNAT and SNAT in same device to avoid asymmetry. - Don't add SNAT with `-o wan` masquerade — it won't match LAN egress; needs `-o ` or plain dst-based match. - Keep one change per test; re-run external control each time. - Persist rules after validation. - Watch out for double NAT on client (client masquerade rules with -o wan0 don't touch these flows) — fine. - If app is HTTPS with SNI/vhost expecting public hostname — cert/SNI unaffected since client still targets 139.99.148.90:443 with hairpin. I think including a short "interpretation matrix" mapping observed evidence → hypothesis is valuable and differentiates a good report. One more consideration: The hang specifically ("hang" not RST) is most consistent with H1 (no SYN-ACK ever, client retransmits until timeout) or H2 (SYN-ACK from wrong tuple → client RST → curl may show "connection reset by peer" quickly, or hang if RST is filtered). The user says "hang," so H1 leads. Good, matches ranking. Also mention: on some edge platforms, "NAT reflection" via "split DNS" mode is distinct from "pure NAT" mode — pure NAT is what we want when clients use the raw IP (split DNS only helps name-based). Word budget: aim for a tight but complete report. Use tables where they compress info. I'll write it now. Also note assumption placeholders: 10.10.10.50 as representative client, IF_LAN as LAN interface name, 10.10.10.1 as edge LAN IP — call these out. Format check: headers, small tables, code blocks. Long-form technical answer is appropriate here; the user explicitly requested a structured report with four sections. I'll keep prose lean and put detail in commands/tables. # Diagnosis Report — Internal clients hang on public VIP 139.99.148.90:443 ## 0. Framing (what external access already proves) Since **external clients work**, the following are already validated and can be ruled out: - Service up and listening on `10.10.10.100:7701` - DNAT rule `139.99.148.90:443 → 10.10.10.100:7701` is correct - Edge firewall policy for **WAN→LAN** direction is correct - No server-side connectivity problem at all — the **only failing variable is ingress path** (packets originating from LAN toward the edge's *own public IP*) This collapses the diagnosis space to: **how the edge handles LAN-originated traffic addressed to its own public IP** — i.e., NAT hairpinning / loopback. --- ## 1. Root-cause hypotheses (ranked) | # | Hypothesis | Likelihood | Signature | |---|---|---|---| | **H1** | **No NAT reflection/hairpin.** DNAT rule is bound to WAN ingress (`-i wan0`), or the device refuses to route its own IP from LAN. LAN SYN either dies in the edge's INPUT chain (nothing listens on 443 locally) or is forwarded upstream to the ISP and uRPF-dropped. | **~80%** | Client retransmits SYN forever (clean hang — matches symptom); **no SYN ever reaches server**; no conntrack entry on edge | | **H2** | **Partial hairpin: DNAT fires, but no SNAT** for LAN→VIP flows. Asymmetric return: server's SYN-ACK (src `10.10.10.100:7701`) goes directly over the LAN, bypassing the edge. Client kernel expects `139.99.148.90:443`, tuple mismatch → RST/drop. | ~15% | Server capture shows client's **real source IP**; client capture shows SYN-ACK arriving from an internal IP | | **H3** | **Policy drop on edge**: interzone rule, anti-martian/uRPF, or explicit "LAN → WAN-address" block. | ~4% | ICMP admin-prohibited / drop-counter increments in logs at exact test time | | **H4** | **Off-device cause**: internal client's default route sends `139.99.148.90` out via a different L3 hop (edge never sees it), or server host firewall has source-selective rules. | ~1% | Edge capture shows **no SYN arriving at all** on any interface | The "hang" symptom (vs. instant reset) leads with H1; H2 usually presents as reset or slow hang. --- ## 2. Verification — one test run, captures per hop Start the internal test, then read the captures: **Hop A — internal client (e.g., 10.10.10.50):** ```bash curl -vk --connect-timeout 5 https://139.99.148.90:443/ sudo tcpdump -ni any 'host 139.99.148.90 and tcp port 443' ``` **Hop B — edge device, LAN interface:** ```bash tcpdump -ni 'host 139.99.148.90' conntrack -L 2>/dev/null | grep 139.99.148.90 # Linux-based edges iptables -t nat -S | grep 139.99.148.90 # KEY: does DNAT rule have "-i wan0"? ``` Vendor equivalents: **ASA:** `packet-tracer input inside tcp 10.10.10.50 40000 139.99.148.90 443` · **FortiGate:** `diagnose sniffer packet 'host 139.99.148.90' 4` · **MikroTik:** `/ip firewall connection print where dst-address~"139.99.148.90"` · **pfSense:** Diagnostics → States, filter VIP. **Hop C — server 10.10.10.100:** ```bash sudo tcpdump -ni any 'tcp port 7701' ss -tlnp | grep 7701 ``` **Hop D — control (before every change):** ```bash # From an EXTERNAL vantage point (must succeed — this is the baseline we protect): curl -v https://139.99.148.90:443/ # From internal client, direct (proves service reachability inside): nc -vz 10.10.10.100 7701 ``` **Interpretation matrix:** | Observed evidence | Conclusion | |---|---| | No SYN on server; no conntrack entry; SYN dies at edge INPUT or exits WAN | **H1** | | Server sees SYN with client's real IP; client sees SYN-ACK from `10.10.10.100:7701` | **H2** | | ICMP unreachable / admin-prohibited; edge drop counters increment | **H3** | | Edge LAN capture shows no SYN | **H4** | --- ## 3. Fix plan ### Primary fix: enable hairpin NAT on the edge (additive only — external path untouched) Two elements are required: **(a)** DNAT matching LAN-sourced traffic to the VIP, **(b)** SNAT so the return path traverses the edge (prevents H2 asymmetry). Replace `IF_LAN` / `10.10.10.1` (edge's LAN IP) as appropriate. **Linux (iptables / nftables) edge:** ```bash # (a) hairpin DNAT — only needed if the existing rule is bound to WAN ingress iptables -t nat -A PREROUTING -i IF_LAN -d 139.99.148.90 -p tcp --dport 443 \ -j DNAT --to-destination 10.10.10.100:7701 # (b) SNAT for the hairpin flow so replies come back through the edge iptables -t nat -A POSTROUTING -o IF_LAN -s 10.10.10.0/24 -d 10.10.10.100 \ -p tcp --dport 7701 -j SNAT --to-source 10.10.10.1 # (c) allow LAN→LAN forward if policy engine requires it (stateful return rule usually exists) iptables -A FORWARD -i IF_LAN -o IF_LAN -d 10.10.10.100 -p tcp --dport 7701 -j ACCEPT ``` > Note: existing `-o wan0` masquerade rules will **not** match these flows — the dedicated SNAT rule is mandatory. If the platform can't hairpin at all, see Plan B. **pfSense/OPNsense:** NAT reflection mode **"NAT" (pure NAT)** — *not* split-DNS mode, since clients use the raw IP — plus "Automatic outbound NAT for reflection". Reflection only affects LAN-originated traffic; WAN rules untouched. **Cisco ASA:** ``` same-security-traffic permit intra-interface object network HAIRPIN-SVC host 10.10.10.100 nat (inside,inside) static 139.99.148.90 service tcp 443 7701 ``` **MikroTik:** ``` /ip firewall nat add chain=dstnat in-interface=IF_LAN dst-address=139.99.148.90 \ protocol=tcp dst-port=443 action=dst-nat to-addresses=10.10.10.100 to-ports=7701 /ip firewall nat add chain=srcnat src-address=10.10.10.0/24 dst-address=10.10.10.100 \ protocol=tcp dst-port=7701 action=masquerade ``` ### Plan B (if the edge cannot hairpin) — zero external risk Split-horizon DNS: internal A record for the service FQDN → `10.10.10.100` (clients use `:7701`), **or** add DNAT on the edge's *LAN* IP (`10.10.10.1:443 → 10.10.10.100:7701`) so internal clients keep port 443. Low DNS TTL (300s) before cutover. Only viable if clients connect by name — not raw IP. ### Order of operations 1. Export baseline configs (see §4), record external curl success. 2. Apply **one** change; re-run Hop A internal curl + **Hop D external curl after every single change**. 3. Confirm conntrack shows both translations on hairpin flows: `conntrack -L | grep 139.99.148.90` (expect internal src rewritten to `10.10.10.1`). 4. Only after validation: persist rules (`iptables-save > /etc/iptables/rules.v4`, `copy run start`, pfSense auto-persists, etc.). --- ## 4. Rollback **Pre-change snapshots (mandatory):** ```bash iptables-save > /root/nat-before-$(date +%F).v4 nft list ruleset > /root/nft-before-$(date +%F).nft # vendor: export running-config / pfSense Diagnostics→Backup XML ``` **Rollback commands per platform:** - iptables: `iptables-restore < /root/nat-before-.v4` (or delete individually: `iptables -t nat -D PREROUTING -i IF_LAN -d 139.99.148.90 …`) - nftables: `nft -f /root/nft-before-.nft` - pfSense/OPNsense: set NAT Reflection back to previous value / restore XML backup - ASA: `no object network HAIRPIN-SVC` + `no same-security-traffic permit intra-interface` - MikroTik: `/ip firewall nat remove ,` - Plan B DNS: restore prior A record (TTL ≤300s bounds exposure) **Post-rollback check:** external `curl https://139.99.148.90:443/` still works (it never depended on any added rule — verify anyway), internal clients return to documented prior state. --- ## 5. Pitfalls - **DNAT without SNAT = H2 regression**: both rules must live on the same box, or the return path goes asymmetric again. - Don't SNAT with `-o wan0`-style masquerade — it won't match LAN-egress hairpin flows. - Change one thing at a time; re-run the **external control test** after each change (it's additive config, but verify, don't assume). - SNI/TLS is unaffected: clients still target `139.99.148.90:443`; hairpin only rewrites L3/L4 in transit. - If the app checks source IPs, it will now see `10.10.10.1` (edge SNAT) instead of client IPs on hairpin flows — whitelist accordingly if needed.

SCB TRUSTED MUSE TEST