The user is asking about a classic networking problem: NAT hairpinning / NAT loopback issue. Setup: - Internal host: 10.10.10.100, service on port 7701 - Public IP: 139.99.148.90, external port 443 - External clients: work fine (port forward/NAT DNAT works from outside) - Internal clients: connect to the public IP and hang This is the textbook NAT hairpinning (NAT loopback / reflected NAT) problem. When an internal client connects to the public IP, the packet goes to the firewall/router. The DNAT rule translates destination to 10.10.10.100:7701. The packet is forwarded to the internal host. But the source IP is still the internal client's IP (e.g., 10.10.10.50). The reply from 10.10.10.100 goes directly to 10.10.10.50 via the internal route — never traversing the firewall — so the client sent a packet from its public-IP-mapped connection and receives a reply from a different IP path (asymmetric routing). The TCP handshake fails: SYN goes to public IP, SYN-ACK comes back from 10.10.10.100 directly to client (source 10.10.10.100:7701), client's socket is connected to 139.99.148.90:443, so kernel sees an out-of-window/RST or drops; connection hangs (retransmitting SYN). The fix: enable NAT hairpinning / loopback NAT — add a SNAT (masquerade) rule so traffic from internal clients to internal server (after DNAT) also gets source-NATted to the firewall's internal interface IP. Then the server replies to the firewall, which un-NATs both directions. Possible other root causes ranked: 1. NAT hairpinning not configured (most likely — matches symptoms exactly: external works, internal hangs). 2. Split-horizon DNS missing / clients using public DNS — but they said clients connect to the public IP directly; if they used DNS, the alternative fix would be split DNS. But the symptom states they connect to the public IP. So hairpin NAT is the issue. However, ranked hypotheses should include split-DNS as alternative remediation approach (not root cause though). 3. Asymmetric routing due to firewall rule ordering (DNAT applied but SNAT only on WAN interface / outbound NAT rule tied to WAN out interface, so hairpin traffic isn't SNATted). This is basically the mechanism of #1. 4. Firewall zone policy blocking internal→WAN-IP loop (interzone rule). Some firewalls (e.g., with zones) block u-turn traffic. 5. MTU/timeout — unlikely; hang from start suggests handshake failure. 6. The service binds to specific IP or has allowlist — external works, so no. 7. TCP MSS clamping issues — external works, so no. Actually let me think about ranked root-cause hypotheses more carefully: Hypothesis 1 (most likely): No NAT loopback/hairpin rule. Evidence: external works, internal hangs. Mechanism: asymmetric path after DNAT; server replies directly to client, client expects replies from 139.99.148.90:443 → receives RST/ignored, SYN retransmits, hangs. Hypothesis 2: Hairpin DNAT exists but outbound SNAT/masquerade excludes the internal interface (masquerade rule scoped to WAN out-interface only), so reply path is asymmetric — same net effect. Hypothesis 3: Zone-based firewall / security policy blocks "U-turn" or intra-zone traffic through the firewall (e.g., internal zone → internal zone via firewall denied), or strict RPF drops. Hypothesis 4: DNS-based expectation: clients actually resolve a name to public IP; proper fix is split-horizon DNS (internal DNS returns 10.10.10.100). This is an alternative architecture fix rather than root cause. But if the requirement is strictly "connecting to the public IP must work", then hairpin NAT is required. Hypothesis 5: Less likely: the public IP is on the router itself and there is a local input vs forward chain issue; or the firewall's NAT only matches in-interface WAN. Verification commands per hop: Hop A — Client side (10.10.10.x): - `ping 139.99.148.90` (confirm reachability of firewall) - TCP test: from Linux: `timeout 5 bash -c 'cat < /dev/null > /dev/tcp/139.99.148.90/443' ; echo $?` or use nc: `nc -vz -w 5 139.99.148.90 443` - Watch handshake: `tcpdump -ni any host 139.99.148.90 and port 443` on client — will show SYN retransmissions with no SYN-ACK, or SYN-ACK with wrong source (RST). - On Windows: `Test-NetConnection 139.99.148.90 -Port 443` Hop B — Firewall/router: - View counters for DNAT rule: `iptables -t nat -L PREROUTING -v -n` or `nft list ruleset` ; pfSense: states view `pfctl -ss`. - Packet capture on firewall internal interface: `tcpdump -ni host 10.10.10.100 and port 7701` — key evidence: you see client→server SYN (src client 10.10.10.x, dst 10.10.10.100:7701 after DNAT) and server SYN-ACK going directly back to client without traversing firewall (asymmetry), or if firewall is not on the path you see nothing. - Check conntrack: `conntrack -L | grep 443` or `cat /proc/net/nf_conntrack | grep 139.99.148.90` — expect only UNREPLIED or the entry to never become ESTABLISHED. Hop C — Server side (10.10.10.100): - `tcpdump -ni any tcp port 7701` — during an internal client attempt: you should see SYN from client IP 10.10.10.x (not from firewall IP) and the server sending SYN-ACK back directly to client — this proves hairpin/asymmetric issue. If SNAT were applied, source would be firewall LAN IP. - Confirm listening: `ss -lntp | grep 7701`. Hop D — Return path evidence: - On client, `tcpdump` shows SYN-ACK arriving from 10.10.10.100:7701 while socket is to 139.99.148.90:443 → client sends RST; hang. That's the smoking gun. Fix plans (platform-dependent, provide generic + common platforms): Generic Linux iptables (router/firewall with DNAT existing): Assume existing DNAT: `iptables -t nat -A PREROUTING -i eth0 -d 139.99.148.90 -p tcp --dport 443 -j DNAT --to-destination 10.10.10.100:7701` Add hairpin SNAT: ``` iptables -t nat -A POSTROUTING -s -d 10.10.10.100 -p tcp --dport 7701 -j SNAT --to-source ``` or `-j MASQUERADE` (with `-o `). Note ordering: this SNAT rule must come before any generic masquerade; it's specific so fine. Also ensure the DNAT PREROUTING rule applies to packets arriving on the LAN interface too, or a separate DNAT for `-i -d 139.99.148.90`. On many setups the DNAT is `-i eth0` (WAN) only, so internal packets to the public IP aren't even DNATted — they'd hit the firewall and... if the public IP is assigned on the WAN interface, packet to 139.99.148.90 arrives at firewall; if no DNAT matches (because -i eth0), it goes to input chain → dropped/reject or handled by firewall's own services. That's another variant: internal clients hang because DNAT only on WAN ingress. Fix: add DNAT for LAN interface too or remove interface constraint. pfSense/OPNsense: - Firewall → NAT → Port Forward: existing rule for WAN. Add second rule: Interface = LAN, source = LAN net, destination = WAN address, dest port 443, redirect to 10.10.10.100:7701, and check "NAT reflection" or add outbound NAT rule: Interface LAN, source LAN net, destination 10.10.10.100, port 7701, translation address = Interface address. Alternatively System → Advanced → Firewall & NAT → enable "Pure NAT" NAT reflection + "automatic outbound NAT for reflection". - Command-line verification: `pfctl -sn`, `pfctl -ss | grep 443`. Cisco ASA: `same-security-traffic permit intra-interface` + NAT hairpin config (object NAT with `nat (inside,inside)`). Verification: `packet-tracer inside inside tcp 10.10.10.50 12345 139.99.148.90 443`. MikroTik: dstnat rule matches in-interface; add dstnat for LAN in-interface + srcnat masquerade to internal server; verify `/ip firewall nat print stats`, `/tool sniffer`, `/ip firewall connection print`. Cisco IOS: NVI (`ip nat enable`) vs legacy. Since the platform isn't specified, I should present a vendor-neutral ranked diagnosis plus generic fix (iptables/nftables) with notes for common platforms. The user wants: "Report: root-cause hypothesis ranked, verification command per hop, fix, rollback." Format: This should be a structured report. Use headers, tables maybe. Keep it organized: 1. Symptom summary & traffic model (brief) 2. Ranked root-cause hypotheses with likelihood and mechanism 3. Verification plan per hop (client, firewall, server, return path) — commands and expected outputs distinguishing hypotheses 4. Fix plan (primary fix: hairpin NAT; alternative: split DNS) — with config examples for generic iptables/nftables + common platforms briefly 5. Safety: do not break external access — test plan after change (external test from outside, e.g., use external vantage or online port checker; verify DNAT counters) 6. Rollback plan — remove added rules, commands, verify external again 7. Residual considerations: port 443 on internal 7701 — TLS SNI cert mismatch? Not our concern; hairpin double NAT breaks source-IP visibility (server sees firewall IP) — mention implication for ACLs/logging. Also note: hanging (not refused) is consistent with SYN retransmits due to asymmetric reply or drop. If it were refused, firewall would REJECT. Verification command per hop — they explicitly want a command for each hop: Hop 1: Client (10.10.10.50 example): - `nc -vz -w 5 139.99.148.90 443` → hangs/timeout - `sudo tcpdump -ni any 'host 139.99.148.90 or host 10.10.10.100' and tcp port 443` → expect: SYN out to 139.99.148.90:443; either nothing back, or SYN-ACK from 10.10.10.100:7701 (if server answers directly) → client RST. Distinguishes Hyp1 vs drop-at-firewall. Hop 2: Firewall: - Confirm DNAT hit from outside vs inside: `nft list ruleset | grep -A2 7701` or `iptables -t nat -vnL PREROUTING` (watch counters while a LAN client tests). - conntrack: `conntrack -L -d 139.99.148.90 -p tcp --dport 443` → entries from LAN clients UNREPLIED. - Capture on LAN iface: `tcpdump -ni 'tcp port 7701'` → see SYN from 10.10.10.50 → 10.10.10.100:7701 and NO return traffic through firewall (server replies direct) → confirms asymmetric hairpin. Hop 3: Server 10.10.10.100: - `sudo tcpdump -ni any 'tcp port 7701 and host 10.10.10.50'` → see SYN from 10.10.10.50 and SYN-ACK sent directly back to 10.10.10.50 → proves reply bypasses firewall (asymmetric) → hairpin problem. - `ss -lnt | grep 7701` confirm listening. Hop 4: Return path / routing on server: - `ip route get 10.10.10.50` → shows direct route via LAN, not via firewall — mechanism confirmation. That's a strong evidence chain. Also hypothesis: client uses the public IP but the router does proxy ARP? Not needed. Also consider: maybe the public IP is not on the edge firewall but at an upstream provider doing DNAT (e.g., hosted/1:1 NAT at provider). Then internal packets to 139.99.148.90 may leave the site toward the provider and be DNATted there, coming back — if provider device does hairpin, could work; if their edge blocks, hang. Verification: traceroute from client to 139.99.148.90 — see if it stays internal. Include `traceroute -n 139.99.148.90` at client hop as a check for where DNAT lives. Good to include: if public IP is provider-hosted (e.g., VPS provider OVH — 139.99.x is OVH/OVHcloud range!). 139.99.148.90 is OVH Singapore/OVH range. So the "public IP" might be on-premises? If internal network uses public IP on their own firewall, fine. But the OVH range suggests maybe it's a cloud setup where 139.99.148.90 is a cloud server and 10.10.10.100 is internal? Then "outside via public" implies port-forward at some gateway. Either way the analysis stands; but the possibility that the public IP is announced/routed upstream and NAT occurs upstream should be a lower-ranked hypothesis. I'll mention traceroute verification. Ranked hypotheses: H1 (High, ~85%): No NAT hairpin/loopback: DNAT exists only on WAN ingress; internal→public-IP traffic either (a) not DNATted (destined to firewall WAN IP on input chain → filtered) or (b) DNATted but not SNATted → server replies directly to client → asymmetric → client kernel drops/RST → SYN retries → hang. Actually (a) and (b) are two sub-mechanisms; both fall under "hairpin NAT missing/incomplete". I'll present H1 with two variants and evidence to distinguish (does server see the SYN? does firewall LAN capture show DNATted SYN?). H2 (Medium): Hairpin rules exist but firewall policy blocks intra-zone/u-turn (evidence: firewall capture shows drop, conntrack no entry, firewall logs). H3 (Lower): Public IP NAT is upstream (provider/cloud LB); internal traffic egresses and gets dropped or unreachable loop (evidence: traceroute leaves site; tcpdump on LAN shows traffic leaving WAN). H4 (Low): DNS/hybrid: clients sometimes resolve name → different path; or hosts file. Not really since they said IP. Skip or low. H5 (Very low): MSS/MTU or asymmetric MTU causing hang after handshake — but hang is at connection start; external works. Skip; maybe mention as excluded. Fix plan: Primary (vendor-neutral, iptables example): 1. Ensure DNAT applies on internal ingress too (or is interface-agnostic): ``` iptables -t nat -A PREROUTING -i -d 139.99.148.90 -p tcp --dport 443 -j DNAT --to-destination 10.10.10.100:7701 ``` (or modify existing rule to drop `-i` constraint if policy permits; better to add a parallel rule to avoid touching the working external rule) 2. SNAT so replies come back via firewall: ``` iptables -t nat -A POSTROUTING -o -d 10.10.10.100 -p tcp --dport 7701 -j SNAT --to-source ``` Scope it tightly: `-s 10.10.10.0/24` to avoid affecting other traffic. Note MASQUERADE alternative. 3. Filter chain: ensure FORWARD rules allow LAN→LAN via firewall (if zone policy requires). nftables equivalent: ``` nft add rule ip nat prerouting iifname "lan0" ip daddr 139.99.148.90 tcp dport 443 dnat to 10.10.10.100:7701 nft add rule ip nat postrouting oifname "lan0" ip saddr 10.10.10.0/24 ip daddr 10.10.10.100 tcp dport 7701 snat to 10.10.10.1 ``` Platform notes: - pfSense/OPNsense: enable NAT reflection (System > Advanced > Firewall & NAT: "Pure NAT" + create rule) or add LAN-interface port forward + Outbound NAT rule (hybrid outbound, LAN → 10.10.10.100:7701 → translate to interface address). - MikroTik: add dstnat in-interface= + srcnat masquerade rule scoped to dst 10.10.10.100 port 7701 src LAN subnet. - Cisco ASA: `same-security-traffic permit intra-interface` + hairpin NAT object. - UniFi/EdgeOS: port-forward has hairpin option (EdgeOS: port-forward hairpin enable) / UniFi "Port Forward + Firewall rule" and NAT loopback setting. Don't-break-external guarantee: add NEW rules; do not modify the existing WAN DNAT or its position; new SNAT scoped to dst 10.10.10.100 + dport 7701 so it can't affect external flows. Pre-change baseline: record external test (from external vantage: `curl -kv --connect-timeout 5 https://139.99.148.90:443/ -H 'Host: x'` from a phone LTE/online port check), save rule backups (`iptables-save > /root/nat-backup-$(date +%F).rules`, `pfctl -sr` etc.). Post-change tests: - External: repeat external vantage test — must still pass; check DNAT counters increased. - Internal: `nc -vz -w5 139.99.148.90 443` → open; `curl -kv https://139.99.148.90:443/`. - conntrack shows ESTABLISHED both directions; server tcpdump now shows source = fw LAN IP. - Monitor: retransmits, conntrack table growth (hairpin consumes conntrack entries on fw — both directions tracked once). Rollback: - Remove added rules by exact spec (not flush!): ``` iptables -t nat -D PREROUTING -i -d 139.99.148.90 -p tcp --dport 443 -j DNAT --to-destination 10.10.10.100:7701 iptables -t nat -D POSTROUTING -o -s 10.10.10.0/24 -d 10.10.10.100 -p tcp --dport 7701 -j SNAT --to-source 10.10.10.1 ``` - Restore from backup file if needed: `iptables-restore < backup` (note: use `iptables-restore` carefully; safer to delete specific rules). Clear affected conntrack: `conntrack -D -d 139.99.148.90` to avoid stale states during rollback. - Re-verify external still works (unchanged rules), internal reverts to hang (known prior state). Caveats to note: - SNAT breaks source IP visibility on server (logs/ACLs see firewall IP) — mitigation: proxy protocol not applicable to raw NAT; use fw LAN IP allowlist on server; or prefer split-horizon DNS as alternative (client→10.10.10.100:7701 direct, source preserved, no NAT load) — but requirement says public IP must work, so hairpin is the fix; split-DNS can be complementary. - Double-NAT reflection on some consumer gear. - If service is TLS on 443→7701, ensure cert/SNI unchanged (NAT is L4, no impact). Also the "hang" vs "refused": hang = SYN no response (drop or asymmetric), consistent. Format thoughts: This is a technical runbook request. Use headings, concise bullet lists, code blocks for commands. A table for ranked hypotheses could work well (rank, hypothesis, mechanism, likelihood, discriminating evidence). Then per-hop verification with commands and expected outputs. Fix with numbered steps and code. Rollback with code. Keep it tight but complete. The user is clearly a network engineer; use precise terminology (hairpin/loopback NAT, asymmetric routing, conntrack). Length: fairly detailed but avoid fluff. Markdown appropriate. I should assume a Linux/iptables firewall as default and give one-line notes for pfSense/MikroTik/ASA since platform unknown. Placeholders: LAN_IF, FW_LAN_IP (10.10.10.1 assumed), LAN net 10.10.10.0/24. Let me draft. Structure: 1. Problem model (2-3 lines + packet path diagram) 2. Ranked hypotheses (table) 3. Verification per hop (client → firewall → server → return path) with commands + expected findings mapped to hypotheses 4. Fix (primary hairpin, generic iptables/nft, platform notes), guarded so external can't break 5. Validation matrix (external + internal tests) 6. Rollback 7. Operational notes (source IP visibility, conntrack, alternative split-DNS) Packet path for the failure: ``` Client 10.10.10.50 ──SYN──▶ 139.99.148.90:443 fw DNAT ──▶ 10.10.10.100:7701 (src still 10.10.10.50) 10.10.10.100 ──SYN-ACK──▶ 10.10.10.50 directly (same subnet, no fw) Client socket expects src 139.99.148.90:443 → gets 10.10.10.100:7701 → RST/drop → SYN retrans → hang ``` Good. Write the report now. Keep commands accurate: Client hop: ``` nc -vz -w 5 139.99.148.90 443 # expect: timeout (hang), not refused traceroute -n 139.99.148.90 # confirm path stays on-prem (first hop = firewall) sudo tcpdump -ni any 'tcp port 443 and host 139.99.148.90' # smoking gun variant: SYN-ACK arrives from 10.10.10.100:7701 → kernel RSTs, SYN retransmits ``` Firewall hop: ``` # NAT rules + counters iptables -t nat -vnL PREROUTING | egrep '7701|139.99.148.90' iptables -t nat -vnL POSTROUTING nft list ruleset # state table conntrack -L -p tcp --dport 443 | grep -E '139.99.148.90|10.10.10.100' # LAN-side capture while an internal client tests tcpdump -ni 'tcp port 7701' # expect: SYN client→10.10.10.100:7701 (DNAT ok) but NO SYN-ACK returning through fw ``` Server hop: ``` ss -lnt | grep 7701 sudo tcpdump -ni any 'tcp port 7701' # expect: SYN from client RFC1918 source directly, SYN-ACK back to client direct → asymmetric proof ip route get # confirms reply leaves via LAN, bypasses fw ``` Distinguish H1a (no DNAT on LAN ingress) vs H1b (DNAT but no SNAT): if firewall LAN capture shows the SYN already translated to 10.10.10.100:7701 → DNAT happened, missing SNAT (H1b). If firewall LAN shows nothing and server sees dst 139.99.148.90 (only if routed there... actually if no DNAT, packet to fw WAN IP input chain → filtered → client sees silent drop; server sees nothing) → H1a. Fix code — generic: ``` # ---- hairpin: add-only, scoped; do not touch existing WAN DNAT ---- # 1) DNAT for packets arriving on LAN interface iptables -t nat -A PREROUTING -i "$LAN_IF" -d 139.99.148.90 -p tcp --dport 443 \ -j DNAT --to-destination 10.10.10.100:7701 # 2) SNAT so server replies return via the firewall iptables -t nat -A POSTROUTING -o "$LAN_IF" -s 10.10.10.0/24 -d 10.10.10.100 \ -p tcp --dport 7701 -j SNAT --to-source 10.10.10.1 # 3) ensure FORWARD permits LAN->LAN flow if policy firewall (usually already covered by ESTABLISHED/related + the same rules that serve DNAT'd WAN traffic — check) iptables -I FORWARD 1 -i "$LAN_IF" -o "$LAN_IF" -d 10.10.10.100 -p tcp --dport 7701 -m conntrack --ctstate NEW -j ACCEPT ``` Hmm, careful: adding FORWARD accept first might be unnecessary but safe; mention only if default FORWARD drop and no intra-zone allow. nftables version briefly. Platform one-liners: - pfSense/OPNsense: NAT reflection "Pure NAT" mode, or hybrid: Port Forward on LAN iface (dest "WAN address") + Outbound NAT (LAN net → 10.10.10.100:7701 → translate "Interface address"). - MikroTik: `/ip firewall nat add chain=dstnat in-interface= dst-address=139.99.148.90 protocol=tcp dst-port=443 action=dst-nat to-addresses=10.10.10.100 to-ports=7701` + `chain=srcnat src-address=10.10.10.0/24 dst-address=10.10.10.100 protocol=tcp dst-port=7701 action=masquerade`. - ASA: `same-security-traffic permit intra-interface` + twice NAT. - UniFi/EdgeOS: enable NAT loopback/hairpin on port-forward. Non-breaking guarantees: new rules appended; scoped by dst/dport so cannot intercept WAN traffic (PREROUTING LAN-iif rule only matches internal ingress; SNAT only matches dst 10.10.10.100:7701 from LAN). External path (WAN in → DNAT → forward → WAN out masquerade) untouched. Validation: External (from LTE hotspot / external vps): ``` curl -kvs --connect-timeout 5 https://139.99.148.90:443/ -o /dev/null ``` Internal: ``` nc -vz -w5 139.99.148.90 443 && echo OPEN curl -kvs --connect-timeout 5 https://139.99.148.90:443/ -o /dev/null conntrack -L -p tcp --dport 443 -d 139.99.148.90 # expect ESTABLISHED ``` Server: tcpdump now shows source 10.10.10.1 (fw LAN IP). Rollback: ``` iptables -t nat -D PREROUTING -i "$LAN_IF" -d 139.99.148.90 -p tcp --dport 443 -j DNAT --to-destination 10.10.10.100:7701 iptables -t nat -D POSTROUTING -o "$LAN_IF" -s 10.10.10.0/24 -d 10.10.10.100 -p tcp --dport 7701 -j SNAT --to-source 10.10.10.1 conntrack -D -d 139.99.148.90 # clear half-open/hairpin states ``` Or restore file: pre-change `iptables-save -t nat > /root/nat.pre-hairpin.v4`. Note: never `iptables -F -t nat` in prod. Operational notes: - Server will log firewall IP as client source; update app ACLs/allowlist accordingly; keep a fw→server allow for 10.10.10.1. - Hairpin states live in conntrack; sized fw required under load. - Long-term alternative: split-horizon DNS (internal name → 10.10.10.100) preserves source IPs and reduces NAT load; but explicit IP access requires hairpin regardless. Also note assumption placeholders: LAN subnet 10.10.10.0/24, FW LAN IP 10.10.10.1 — replace. One more check on hypothesis ranking: Is it possible that external works and internal hangs due to DNS? They said internal clients connect to public IP — so DNS isn't in path. Keep H-DNS as note only (if any clients use a hostname, split DNS is complementary). Also possibility: internal clients' traffic to public IP gets DNATted correctly AND SNATted but a server-side firewall drops fw's LAN IP... then external works (source WAN IP allowed) but hairpin fails with RST from server — symptom would be refused or hang if dropped silently. That's a fix-verification caveat rather than root cause; mention in validation: if after adding SNAT the connection is refused, check server-side ACL for 10.10.10.1. Actually good to include in validation step: "if nc now returns refused instead of open → server allowlist missing fw IP." Nice touch. Ranked table: | Rank | Hypothesis | Mechanism | Prior | Discriminator | |---|---|---|---|---| | 1 | Missing/incomplete NAT hairpin — DNAT applies only on WAN ingress (no LAN-side DNAT) | Packet to 139.99.148.90 hits fw input chain, silently filtered → SYN retries → hang | High | fw LAN capture shows nothing; conntrack shows no entry for LAN client | | 2 | Missing/incomplete hairpin — DNAT works but no SNAT on LAN egress | Asymmetric return: server SYN-ACKs straight to client; client kernel rejects (socket expects 139.99.148.90) → hang | High | fw LAN capture shows SYN translated to 10.10.10.100:7701 but no SYN-ACK; server capture shows direct reply | | 3 | Zone/inter-zone policy blocks u-turn traffic | fw drops LAN→LAN through itself | Medium-low | fw counters/logs drop rule; packet-tracer/test shows deny | | 4 | Public IP NAT performed upstream (provider/cloud), not on-prem | Internal packet leaves site; upstream device lacks loopback or ISP ingress filter drops own-prefix from inside | Low | traceroute from client leaves premises; nothing on fw | | 5 | Server ACL silently drops unexpected source (only if partial hairpin exists) | RST/timeout after translation | Very low | appears only during fix validation | Ranks 1 and 2 are the two sub-mechanisms of the same root cause; I'll frame rank 1 as umbrella "hairpin NAT absent" with two variants — actually cleaner: rank them as one root cause family but two distinct findings; the report asked for ranked hypotheses, so presenting the two variants separately with discriminators is more actionable. I'll present H1 umbrella with variants (a)/(b) — that's the most honest. Length: medium-long; it's a runbook. Use code blocks liberally, tables for hypotheses and validation. Headers per section. Avoid excessive prose. I'll write it now. # Diagnosis Report — Internal clients hang connecting to `139.99.148.90:443` (DNAT → `10.10.10.100:7701`) ## 1. Traffic model of the failure ``` Client 10.10.10.50 ── SYN ──▶ 139.99.148.90:443 │ [firewall DNAT] ▼ 10.10.10.100:7701 ◀── source still 10.10.10.50 (no SNAT on hairpin path) │ 10.10.10.100 ── SYN-ACK ──▶ 10.10.10.50 DIRECTLY (same subnet, reply never re-enters firewall) ``` Client's socket is bound to `139.99.148.90:443`; the SYN-ACK arrives from `10.10.10.100:7701`. Kernel treats it as bogus, sends RST (or drops it), and keeps retransmitting SYN → **hang, not refuse**. External clients work because both directions traverse the firewall. This is the classic **NAT hairpin / loopback** gap. --- ## 2. Ranked root-cause hypotheses | Rank | Hypothesis | Mechanism | Discriminating evidence | |---|---|---|---| | **1a** | **Hairpin NAT absent — DNAT rule bound to WAN in-interface only.** Internal packet to the public IP enters on LAN, matches no DNAT, lands in INPUT chain and is silently filtered. | SYN retransmits → hang | Firewall LAN capture shows **nothing**; no conntrack entry for the LAN client | | **1b** | **Hairpin DNAT occurs, but no SNAT on the LAN side** → asymmetric return path (diagram above) | Client kernel rejects SYN-ACK → hang | Firewall LAN capture shows SYN **already translated** to `10.10.10.100:7701` but **no SYN-ACK returning**; server capture shows direct reply to client | | **2** | Hairpin rules exist but zone/inter-zone firewall policy drops LAN→LAN ("u-turn") traffic through the firewall | Drop at FORWARD | Firewall drop counters/logs increment during test; conntrack entry never created | | **3** | NAT for the public IP is performed **upstream** (provider/cloud edge), not on-prem; upstream device lacks loopback or ingress-filters its own prefix from inside | Internal packets leave the site and die upstream | `traceroute` from client exits the premises; no relevant rules/captures on local fw | | **4** | Server-side ACL silently drops unexpected source (only relevant *after* a partial hairpin exists) | RST/timeout post-translation | Appears only during fix validation — see §5 note | 1a/1b are two variants of the same root cause (missing/incomplete hairpin NAT) — combined prior ~85%. Hypothesis 2 is the runner-up (~10%). Test order below resolves 1a vs 1b vs 2 in one pass. --- ## 3. Verification — one command per hop **Hop 1 — Internal client (e.g., 10.10.10.50):** ```bash nc -vz -w 5 139.99.148.90 443 # expect: timeout (hang), NOT "refused" traceroute -n 139.99.148.90 # first hop = firewall → NAT is on-prem (kills H3) sudo tcpdump -ni any 'tcp port 443 and host 139.99.148.90' ``` *Smoking gun:* SYN out, then either silence or a SYN-ACK arriving **from `10.10.10.100:7701`** followed by client RST. **Hop 2 — Firewall:** ```bash iptables -t nat -vnL PREROUTING | grep -E '7701|139.99.148.90' # check in-interface binding + counters (live, during a LAN test) iptables -t nat -vnL POSTROUTING | grep -E '10.10.10|MASQ' conntrack -L -p tcp --dport 443 | grep 139.99.148.90 # expect UNREPLIED or no entry tcpdump -ni 'tcp port 7701' # run during a LAN client test ``` *Decides 1a vs 1b:* translated SYN visible on LAN iface but no return SYN-ACK → **1b**. Nothing visible at all → **1a** (or H2 — then check drop counters: `iptables -vnL FORWARD`). **Hop 3 — Server (10.10.10.100):** ```bash ss -lnt | grep 7701 # confirm listening sudo tcpdump -ni any 'tcp port 7701' ``` *Expected during LAN test:* SYN from the client's RFC1918 source and SYN-ACK sent **straight back to the client** — proves the reply bypasses the firewall (asymmetry confirmed). **Hop 4 — Server return path:** ```bash ip route get 10.10.10.50 # shows direct LAN route, no firewall in return path ``` --- ## 4. Fix — add-only hairpin NAT (external path untouched) Placeholders: `LAN_IF` = LAN interface, LAN net `10.10.10.0/24`, firewall LAN IP `10.10.10.1` — adjust to your topology. **Linux / iptables (generic reference):** ```bash # Baseline backup first iptables-save -t nat > /root/nat.pre-hairpin.$(date +%F).rules # 1) DNAT for traffic arriving on the LAN interface (do NOT modify the existing WAN rule) iptables -t nat -A PREROUTING -i "$LAN_IF" -d 139.99.148.90 -p tcp --dport 443 \ -j DNAT --to-destination 10.10.10.100:7701 # 2) SNAT hairpin traffic so the server replies via the firewall (tightly scoped) iptables -t nat -A POSTROUTING -o "$LAN_IF" -s 10.10.10.0/24 -d 10.10.10.100 \ -p tcp --dport 7701 -j SNAT --to-source 10.10.10.1 # 3) Only if FORWARD policy is DROP and no intra-zone allow exists: iptables -I FORWARD 1 -i "$LAN_IF" -o "$LAN_IF" -d 10.10.10.100 -p tcp --dport 7701 \ -m conntrack --ctstate NEW -j ACCEPT ``` **Platform equivalents:** - **pfSense/OPNsense:** System → Advanced → Firewall & NAT → NAT Reflection mode = **Pure NAT** (enable automatic outbound NAT for reflection), or explicitly: Port Forward on **LAN** interface (dest = WAN address, port 443 → 10.10.10.100:7701) **plus** hybrid Outbound NAT rule (LAN net → 10.10.10.100:7701, translation = interface address). - **MikroTik:** add `chain=dstnat in-interface= dst-address=139.99.148.90 dst-port=443 action=dst-nat to-10.10.10.100:7701` + `chain=srcnat src-address=10.10.10.0/24 dst-address=10.10.10.100 dst-port=7701 action=masquerade`. - **Cisco ASA:** `same-security-traffic permit intra-interface` + hairpin twice-NAT. - **UniFi/EdgeOS:** enable NAT loopback/hairpin on the existing port-forward. **Why external access cannot break:** all changes are *appended* and scoped — the LAN-iif DNAT can only match internal ingress; the SNAT matches only dst `10.10.10.100:7701` from the LAN. The working WAN path (WAN in → existing DNAT → WAN out) is untouched. --- ## 5. Post-change validation | Test | Where | Command | Pass criteria | |---|---|---|---| | External still works | LTE hotspot / external VPS | `curl -kvs --connect-timeout 5 https://139.99.148.90/ -o /dev/null` | Handshake + response; DNAT counter increments | | Internal now works | LAN client | `nc -vz -w 5 139.99.148.90 443` | `succeeded` | | State sane | Firewall | `conntrack -L -p tcp --dport 443 -d 139.99.148.90` | `ESTABLISHED` both directions | | Path symmetric | Server | `tcpdump -ni any 'tcp port 7701'` | Source now = `10.10.10.1` (fw LAN IP) | ⚠ If internal test flips from *timeout* to ***refused***: hairpin works but the server/application allowlist rejects the firewall's IP — add `10.10.10.1` to the service ACL. --- ## 6. Rollback ```bash # Remove only the added rules by exact match — never flush the table in prod iptables -t nat -D PREROUTING -i "$LAN_IF" -d 139.99.148.90 -p tcp --dport 443 \ -j DNAT --to-destination 10.10.10.100:7701 iptables -t nat -D POSTROUTING -o "$LAN_IF" -s 10.10.10.0/24 -d 10.10.10.100 \ -p tcp --dport 7701 -j SNAT --to-source 10.10.10.1 iptables -D FORWARD -i "$LAN_IF" -o "$LAN_IF" -d 10.10.10.100 -p tcp --dport 7701 \ -m conntrack --ctstate NEW -j ACCEPT # only if step 3 was applied conntrack -D -d 139.99.148.90 # flush stale/half-open hairpin states ``` Or restore from baseline: `iptables-restore -t nat < /root/nat.pre-hairpin..rules`. Re-run the external test to confirm return to known-good state. --- ## 7. Operational notes - **Source-IP loss:** with hairpin SNAT, the server logs the firewall IP, not the real client. Update app logging/ACLs accordingly. If clients use a **hostname**, the complementary long-term fix is split-horizon DNS (internal zone → `10.10.10.100`), which preserves source IPs and offloads NAT — but explicit IP access still requires the hairpin rules. - **Conntrack load:** every hairpin flow is tracked on the firewall; fine at this scale, size accordingly if fanned out. - NAT is L4 — no TLS/cert impact from the 443→7701 remap.

SCB SCB-network-T1 RMUSE