Let me think carefully about this scenario. The user is asking about a networking problem, apparently a follow-up to a previous conversation ("Same topology, but...") that I don't have context for. The scenario described: 1. There's some NAT topology (presumably a Linux gateway/router doing NAT between internal and external networks) 2. External access "ALSO works sometimes from internal" — 1 in 5 attempts succeed 3. conntrack shows BOTH DNAT and SNAT hits Wait, let me re-read: "external access ALSO works sometimes from internal (1 in 5 attempts)". Hmm. This phrasing is ambiguous. Let me think about what the original scenario likely was. Common scenario: A gateway with DNAT rule forwarding external traffic to an internal server (port forwarding). The classic problem: external clients can reach the internal server via the public IP, but internal clients cannot reach the internal server via the public IP (hairpin NAT problem). The fix is to add a hairpin NAT rule: DNAT + SNAT (MASQUERADE) on the internal interface. Now the follow-up: "external access ALSO works sometimes from internal (1 in 5 attempts), and conntrack shows both DNAT and SNAT hits." So the user apparently added hairpin NAT rules, and now: - Internal clients accessing the service via the public IP sometimes work (1 in 5) - conntrack shows both DNAT and SNAT entries The question: "Diagnose with the ambiguity resolved by evidence ordering." Hmm, "ambiguity resolved by evidence ordering" — this is an interesting instruction. I think the user wants me to diagnose the fault, and where there are multiple possible causes (ambiguity), resolve the ambiguity by ordering the evidence — i.e., use the sequence/correlation of evidence (which evidence appeared/confirmed first, what evidence discriminates between hypotheses) to determine which cause is the actual one. Actually, maybe "evidence ordering" means: order the diagnostic steps by their evidentiary power — cheapest/most-discriminating first. Or it means: given the ambiguity in the symptom report ("external access works sometimes" could be several things), resolve it by ordering evidence collection. Let me think about the technical fault first. The symptom: internal → public IP → internal server works 1 in 5 times. conntrack shows both DNAT and SNAT. Classic hairpin NAT setup on Linux: ``` iptables -t nat -A PREROUTING -i eth0 -d $PUBLIC_IP -p tcp --dport 443 -j DNAT --to-destination 10.0.0.5 iptables -t nat -A POSTROUTING -o eth0 -s 10.0.0.0/24 -j MASQUERADE ``` Hairpin addition: ``` iptables -t nat -A PREROUTING -i eth1 -s 10.0.0.0/24 -d $PUBLIC_IP -p tcp --dport 443 -j DNAT --to-destination 10.0.0.5 iptables -t nat -A POSTROUTING -o eth1 -d 10.0.0.5 -p tcp --dport 443 -j SNAT --to-source 10.0.0.1 (or MASQUERADE) ``` Why would it work only 1 in 5 times? Intermittent hairpin NAT. Classic causes of intermittent hairpin NAT: 1. **Asymmetric routing**: The SNAT (masquerade) rule matches only sometimes. E.g., the POSTROUTING SNAT rule is conditional or the masqueraded source IP varies. If the reply from server goes directly to client at L2 (same subnet) when no SNAT applied — TCP reset/ignore because client thinks it's talking to public IP but receives packet from private IP — connection fails. If SNAT applies, it works. 2. **conntrack race / conntrack entry collision**: The DNAT in PREROUTING vs the conntrack entries — the "1 in 5" could be due to a race where two rules both match and the first packet of some connections takes a different path. 3. **MASQUERADE port collision / source port reuse**: With masquerade, source port might be preserved; if two internal clients use the same source port... but that wouldn't be 1 in 5 per attempt from one client. 4. **Multiple DNAT rules / rule ordering with generic DNAT**: If there's a DNAT rule in PREROUTING for -i eth0 (external interface) and internal traffic comes in on eth1, it wouldn't match... unless the rule has no interface match. Then internal traffic gets DNAT'd, but return traffic from the internal server to the internal client is NOT routed back through the gateway (direct L2), because dst IP is client's private IP on same subnet. So the TCP handshake breaks: client sends SYN to public IP, gateway DNATs to server (source remains client private IP), server replies directly to client at L2 with source = server private IP; client expects source = public IP → RST/ignored. Fails consistently, not 1 in 5. But 1 in 5 success suggests something that works sometimes. What changes between attempts? Key insight: **conntrack shows both DNAT and SNAT hits**. In `/proc/net/nf_conntrack` or `conntrack -L`, for a hairpinned connection you'd see something like: ``` tcp 6 431999 ESTABLISHED src=10.0.0.100 dst=203.0.113.10 sport=51000 dport=443 src=10.0.0.5 dst=10.0.0.1 sport=443 dport=51000 [ASSURED] mark=0 use=1 ``` The reply direction shows DNAT+SNAT applied: src=10.0.0.5 (server real IP after un-DNAT... wait). Actually in conntrack output, the first tuple is the original direction, second tuple is the reply direction showing the rewritten addresses. For a hairpin with both DNAT and SNAT: Original: src=10.0.0.100 dst=203.0.113.10 sport=X dport=443 Reply: src=10.0.0.5 dst=10.0.0.1 sport=443 dport=X Here the reply tuple shows that on the wire, packets from server appear as src=203.0.113.10 dst=10.0.0.100 (after un-NAT), i.e., the conntrack reply tuple shows the pre-translation view of the reply as it will be seen by the client... hmm, actually conntrack shows the reply tuple as expected to be seen: src=10.0.0.5 is the real server address as packets arrive at the gateway from the server; the conntrack entry records what it will be rewritten to. Let me be precise. conntrack -L displays: - Original: src=CLIENT dst=PUBLIC_IP (as the client sent it) - Reply expected: src=SERVER_REAL dst=SNAT_SOURCE (as packets come back), and conntrack will rewrite to src=PUBLIC_IP dst=CLIENT. If the SNAT to the gateway's internal IP (10.0.0.1) is applied, the reply tuple shows dst=10.0.0.1. If only DNAT (no SNAT) applied: Reply: src=10.0.0.5 dst=10.0.0.100 — meaning the gateway expects the server to reply to the client directly (off-box, asymmetric). In that case the packet from server to client doesn't traverse the gateway (same subnet), so conntrack entry never confirms/never sees reply, connection hangs and fails. Client's TCP stack receives SYN-ACK from 10.0.0.5:443 while its socket is connected to 203.0.113.10:443 → RST. So a purely DNAT hairpin fails 100%, deterministically, on a flat internal network. Not 1 in 5. What gives 1 in 5? Possibility A: **Both a hairpin DNAT+SNAT rule and a broad DNAT rule exist, and rule ordering/packet path varies.** E.g., DNAT rule without `-i` match placed before the hairpin rule; when the packet is DNAT'd by the first rule to the server's private IP, then POSTROUTING SNAT rule `-d serverIP -o lan`... it could still SNAT. Hmm. Possibility B: **MASQUERADE with multiple source IPs / multipath**: If the LAN interface has multiple addresses, or the masquerade picks different source... Not typically random. Possibility C: **The 1-in-5 is timer/conntrack expiry related**: e.g., nf_conntrack table full → new connections dropped → some succeed after entries expire. "conntrack table full, dropping packet" in dmesg. But then external access would also be intermittent. The user says "external access ALSO works sometimes from internal" — implying external access (from outside) works reliably? Actually "ALSO" suggests in the previous scenario, external access worked but internal didn't; now internal also works sometimes. Hmm, actually re-read: "external access ALSO works sometimes from internal (1 in 5 attempts)". Parse: "external access ... from internal" — from internal clients, accessing the external-facing service (via public IP) — now ALSO works, sometimes, 1 in 5 attempts. The "ALSO" refers to: previously (in the prior scenario) presumably nothing worked from internal; now external-style access (via the public IP) from internal also works intermittently. So the setup: hairpin NAT has been (partially) added, and it works 1 in 5. conntrack shows both DNAT and SNAT hits — meaning the rules exist and sometimes both apply. Why 1 in 5? Let me think about the classic flaky hairpin causes: 1. **Source-port collision on SNAT with connection reuse** — no. 2. **The DNAT rule matches only some packets because it's in the wrong hook/chain order relative to conntrack expectations** — conntrack decisions are per-connection; first packet decides. So per-connection randomness must come from the first packet taking different paths. 3. **ECMP / multiple default routes**: If the internal client's packet to the public IP is routed... no, the gateway does the DNAT. 4. **The client's traffic to the public IP sometimes leaves via a different path**: e.g., the internal host has its own route, or the switch occasionally... no. 5. **Asymmetric flows due to the SNAT rule having `-o` interface mismatch**: If the SNAT rule is `-o eth1` but... consistent. 6. **The server has multiple IPs / round-robin DNS**: If the public hostname resolves to multiple A records (round-robin), 1 in N attempts might hit the right IP... 1 in 5 would be odd (usually 2-4 records). 7. **Hairpin works only when the conntrack entry for the external direction exists / doesn't exist** — conntrack conflict: If an external client with the same tuple... unlikely to be 1 in 5. 8. **RP filter (rp_filter) dropping**: rp_filter on the internal interface: packets from server (10.0.0.5) to gateway internal IP arrive on eth1 — source 10.0.0.5, route to 10.0.0.5 via eth1 — fine. Not random. 9. **The masquerade source IP flapping because the gateway's internal interface IP changes (DHCP) or multiple IPs** — MASQUERADE picks the interface's primary address; stable. 10. **The 1-in-5 = DNS-based**: The client tries to reach the service by hostname; DNS resolves to public IP sometimes and split-horizon internal IP sometimes? Then "1 in 5" could be TTL caching windows. But conntrack shows DNAT hits, so DNAT is happening. 11. **Load balancer / multiple backend servers**: DNAT to a set of backends, one broken... no, classic DNAT is single target. 12. **Conntrack race between two DNAT rules producing different translations**: E.g., both a `PREROUTING` DNAT (public→server:443, interface-agnostic) and the hairpin SNAT. If the DNAT rule appears twice with different --to-destination (e.g., old stale rule pointing to a dead IP and new rule to live IP), the first matching rule wins deterministically though — nftables/iptables first match wins. Deterministic, not random. 13. **The "sometimes" is due to the TCP retransmission and conntrack timeout interplay**: Without SNAT, the SYN goes out DNAT'd, server replies directly to client, client RSTs or ignores; TCP retries SYN 5 times... all fail identically. Still deterministic fail. 14. **Hardware offloading / flowtable intermittently bypassing NAT**: Software flowtable (conntrack offload) can bypass the NAT hooks for established flows — but if the first packet is NAT'd correctly, offloaded flow preserves the NAT. If flowtable is misconfigured... this is exotic but real: with `flowtable` offload, packets bypass netfilter hooks, and there have been bugs where NAT wasn't applied to offloaded flows. Intermittency could correlate with offload engagement after N packets (flow offload engages after ~packets/time). Hmm, interesting but exotic. 15. **The real classic: hairpin SNAT applied only when source/dest in same subnet rule with `-d` matching, but the client and server on different subnets/VLANs, and inter-VLAN routing... consistent again. 16. **nf_conntrack hash race / "conntrack: table full"**: intermittent drops with 80% failure — plausible ratio. But conntrack shows both DNAT and SNAT hits — on successful connections. The failures wouldn't create entries (or would create UNREPLIED DNAT-only entries). "conntrack shows both DNAT and SNAT hits" — the user is seeing entries with both translations = the successful 20%. 17. **Nagle/... no. 18. **Two gateways / VRRP with asymmetric state (conntrack not synchronized)**: If there are two routers sharing a VIP (keepalived), and the client's traffic sometimes goes to gw1 and the server's reply... In a switched LAN, both gateways are on the same L2; the server replies to the SNAT source (the VIP or the specific gateway's IP?). If SNAT uses MASQUERADE on the VIP interface, replies go to the VIP → whichever master → conntrack entry is on gw1 but reply arrives at gw2 → gw2 has no entry → drops or sends RST → fail. If gw1 is master, works; after failover or per-flow... With VRRP, all traffic to the VIP goes to the master consistently. Unless there are two public IPs... Hmm, but "1 in 5" with two gateways doesn't fit naturally unless per-packet load balancing (GLBP-style) — Linux doesn't do that by default. 19. **The 1-in-5 ratio smells like SYN retransmission backoff eventually... no — if it fails it fails. 20. **Masquerade + conntrack source port collision with the SNAT'd connections to same server**: When masquerading multiple internal clients to the server through the gateway, if two concurrent connections use the same source port to the same dest, the second one gets its source port rewritten — fine, handled. 21. **The intermittent success correlates with whether the client's ARP cache has the gateway's MAC vs the server's MAC**: If the public IP is (mis)configured as a secondary IP ON the internal segment... wait — interesting: If someone assigned the public IP as an IP alias on the LAN (or the server itself has the public IP as secondary), then internal clients ARP for the public IP directly. The server might respond (proxy-ARP/secondary IP) — connection works direct. The gateway might also respond (if the public IP is on its external interface, it won't ARP on internal iface unless proxy_arp/arp_ignore settings). ARP flux: both gateway and server answer ARP at different times → 1 in 5 might reflect ARP cache lifetimes and who answers faster. This is a real classic: "ip_nonlocal_bind / arp_ignore / ARP flux" causing intermittent behavior where sometimes the client reaches the gateway (DNAT+SNAT entry: both DNAT and SNAT hits!) and sometimes reaches the server directly... Hmm, but if the server had the public IP, external access works direct too. Actually wait — reconsider. What does "conntrack shows both DNAT and SNAT hits" tell us? It tells us: for the flows that create conntrack entries, both translations are programmed. So the rules are firing. The 4-in-5 failures: are they connections that got both translations but still failed, or connections that never got entries? That's the key diagnostic fork — and the user asked to "resolve the ambiguity by evidence ordering." The ambiguity: Two competing diagnoses: - Hypothesis A: Both rules apply, but the reply path bypasses the gateway (asymmetric return), so only translations happen but no completion... no — if both DNAT and SNAT applied, the reply comes back to the gateway (dst = SNAT IP) and gets un-NAT'd; should work. - Hypothesis B: On failing attempts, only DNAT applies (SNAT rule didn't match), reply goes direct, fails. - Hypothesis C: Rules apply but something else (MTU, rp_filter, firewall INPUT vs FORWARD) breaks it. Evidence ordering approach: The discriminator is: **look at conntrack entries for failing attempts**. Run the failing connection, then `conntrack -L -d $PUBLIC_IP` or watch `conntrack -E` live. If failing attempts show entries with reply tuple `dst=10.0.0.100` (client's private IP, i.e., NO SNAT) vs successful showing `dst=10.0.0.1` (SNAT applied) — that resolves it: SNAT rule matches only sometimes. Why would SNAT match only sometimes? AH — here's the classic: **the SNAT rule is placed in POSTROUTING with `-o` matching the wrong interface, or MASQUERADE is bound to the external interface**. Actually, here's THE classic intermittent hairpin cause: **the DNAT rule is a port-forward that matches ALL source addresses (no -s restriction) — fine — but the SNAT rule was added only for `-s /24` while clients are in a different subnet (e.g., DHCP pool larger than the SNAT rule's subnet)**. Then clients in some subnets work, others don't — deterministic per client, not per attempt. What varies per attempt from the SAME client? Let me think harder. **Source port based!** Here's one: the hairpin SNAT rule might have a `--dport` or conntrack state match like `-m conntrack --ctstate NEW`... first packet only — fine for NEW; established packets use the conntrack entry. Hmm. **Rubber meets road: nftables vs iptables double-NAT**: If BOTH nftables NAT rules AND iptables-legacy rules exist (the dreaded split-brain), packets traverse only one of them (whichever registered first, per family/daemon startup order). Sometimes the machine loads rules via iptables-nft (both see them), sometimes... Actually iptables-nft and nftables native share the same netfilter core — rules from both are visible to both. iptables-legacy vs nft is the split. Not random per packet though. **Random per-attempt with 20% success — think about what's random in TCP**: the source port (ephemeral). If the SNAT rule matches only certain source ports? E.g., rule written as `-p tcp --sport 1024:65535`? Always true. If the client uses different destination... no. What if there are FIVE paths and one works? "1 in 5" — could be literal: 5 backend paths, e.g., DNAT to a range: `--to-destination 10.0.0.5-10.0.0.9` — DNAT with a range is deterministic per-conntrack (it hashes... actually DNAT range picks based on... it's deterministic-ish, pseudo-random hash). If 1 of 5 backends actually has the service up... but conntrack shows DNAT to varying IPs. The user didn't mention a range. Hmm, but maybe I'm overthinking the specific original topology. The user said "Same topology" referring to a prior conversation I can't see. The prior conversation probably involved: Linux gateway, one external interface with public IP, internal LAN, DNAT port-forward from public to internal server; internal clients couldn't access via public IP (hairpin problem); I likely recommended adding SNAT/MASQUERADE for hairpin. Now: it works 1 in 5 and conntrack shows both DNAT and SNAT hits. The "evidence ordering" instruction: The user wants a diagnosis where ambiguities (multiple possible causes) are resolved by ordering the evidence — i.e., present the diagnostic sequence in an order such that each piece of evidence maximally discriminates between remaining hypotheses, and use the given evidence (1-in-5, conntrack dual hits) in that order. So the deliverable: a structured diagnosis: 1. What the evidence already establishes (locked in): - Intermittency (1/5) rules out a purely deterministic mis-config like "missing SNAT rule" — a missing/never-matching SNAT gives 0% via hairpin (with direct-L2 reply), not 20%. - Wait, does it? If no SNAT, the reply from server to client is direct L2, client RSTs — 0% success... UNLESS the client and server are on different subnets, in which case the reply routes back through the gateway WITHOUT SNAT — then the reply arrives at the gateway from server (src 10.0.0.5) destined to client (10.0.0.100) on the same internal interface; gateway has conntrack entry (DNAT'd); the reply does NOT match the conntrack entry (wrong src: entry expects reply from DNAT'd dst... wait the DNAT rewrote dst to server; the reply from server matches the reverse tuple: src=server dst=client? The entry's reply tuple is src=server-priv dst=client... hmm let me redo. Conntrack entry after DNAT-only hairpin (client 10.0.0.100 → public 203.0.113.10:443, DNAT to server 10.0.0.5:443): - orig: src=10.0.0.100 dst=203.0.113.10 sport=X dport=443 - reply expected: src=10.0.0.5 dst=10.0.0.100 sport=443 dport=X If server replies and the packet routes back through the gateway (different subnets, gateway is the router), the packet src=10.0.0.5 dst=10.0.0.100 arrives at gateway; conntrack matches it as reply, un-DNATs: src becomes 203.0.113.10 → client sees SYN-ACK from public IP → works! So hairpin across subnets works with DNAT only. Hairpin within the same subnet fails without SNAT (asymmetric L2 bypass). So: same-subnet (flat LAN) hairpin without SNAT = 0%; with SNAT = 100%. Neither is 20%. Therefore intermittency comes from something that varies per attempt. 2. What varies per attempt — the candidates (the ambiguity): a. SNAT matches on some first-packets but not others (rule with variable match: e.g., `-m conntrack --ctstate DNAT` ordering, or multiple POSTROUTING rules where an earlier MASQUERADE for external traffic sometimes grabs it — e.g., rule `-o eth0 -j MASQUERADE` and the hairpin SNAT placed after it; but hairpin packets go -o eth1, so no). b. **Per-connection variance via source-port-dependent rule or DNAT pool**: e.g., `--to-destination` range or multiple DNAT rules with statistics module (`-m statistic --mode nth`)? Someone might have used `-m statistic` for load balancing — 1-in-5 could literally be a 5-way split with 1 healthy path. Hmm! "1 in 5" is suspiciously specific. But I shouldn't invent rules not in evidence. c. **Conntrack race/duplicate: two DNAT rules (one for -d PUBLIC_IP on external iface, one generic), plus NAT of the reply...** d. **The reply path sometimes DOES route through the gateway**: if the internal switch occasionally... no, L2 is stable. e. **The client's ephemeral port sometimes collides with an existing conntrack entry** — with masquerade source-port preservation, when two flows from same client port... The classic masquerade source-port collision: first flow keeps port X; a second concurrent flow from a different internal client to the same server with same port X gets rewritten — handled fine. Not 20%. f. **TIME_WAIT/conntrack conflict**: Retrying the same source port after a failed attempt: client retries with new source port each time. no. g. **rp_filter on the internal interface combined with the SNAT'd source**: Reply from client to gateway SNAT IP: src=10.0.0.100 dst=10.0.0.1 on eth1 — rp_filter fine. h. **The gateway itself is the client sometimes? no. i. **Load-balanced dual gateways (ECMP at L2 via same IP two MACs)**: If the LAN has TWO gateways both answering for the internal gateway IP (misconfigured VRRP both MASTER, or a cloned VM), then per-ARP-flux, flows split ~50/50 between gw A (has the rules) and gw B (doesn't). Flows through gw B: packet DNAT'd? gw B has no DNAT → forwards public dst to its default upstream → external router... breaks differently. If gw B has DNAT but no SNAT: 0%... The ratio 1/5 ≠ 50% though. Unless 1 of 5 gateways... too speculative. j. **conntrack tuple race with "DNAT + SNAT both hits"**: Actually, maybe the intended reading is different: the user sees in conntrack BOTH a DNAT entry AND a separate SNAT entry — i.e., two entries for related traffic, indicating DOUBLE NAT happening: hairpin SNAT rule uses MASQUERADE on internal iface AND the generic MASQUERADE also... no, one packet traverses POSTROUTING once; only one SNAT applied. Hmm, but "conntrack shows both DNAT and SNAT hits" might mean: `conntrack -S`/counters or `iptables -L -v` counters show both the DNAT rule and SNAT rule have hit counts > 0. And the hit counts could reveal: DNAT counter ≈ 5×SNAT counter → SNAT misses 4 of 5! That would be the decisive evidence ordering: compare rule counters. If DNAT rule hits = N and SNAT hits ≈ N/5, the SNAT rule fails to match 80% of the time → examine why: the SNAT rule's match criteria exclude most attempts. E.g., SNAT rule written with `-d $PUBLIC_IP` in POSTROUTING — WRONG: after DNAT, dst in POSTROUTING is already 10.0.0.5, so `-d $PUBLIC_IP` never matches... that would be 0% again. But if the SNAT rule is `-d 10.0.0.0/24`... matches. Hmm. What SNAT rule matches only sometimes for the same client/dest? **A rule matching on source port range vs the random ephemeral port**: e.g., `-p tcp --sport 32768:61000` (the old default ephemeral range in some docs) while the client's actual ephemeral range is 1024+ or net.ipv4.ip_local_port_range = 32768..60999 vs rule says 49152:65535 (IANA ephemeral)... If the client's ephemeral port falls in the rule's range sometimes → intermittent! Linux default ip_local_port_range is commonly 32768-60999. A doc-copied rule `--sport 1024:65535` always matches. But if the rule says `--sport 49152:65535`, matches ~? Ports 49152-60999 out of 32768-60999 = ~38% of attempts. Close-ish to 1-in-5? Eh, contrived but the MECHANISM (port-range-dependent match) is a real class: **per-attempt randomness lives in the source port** because it's the only per-attempt variable in the tuple. That's actually an elegant principle to teach: In a stable topology, the ONLY per-attempt varying fields in the first packet are the client's ephemeral source port (and TCP options/timestamps). Therefore any intermittent match/mismatch (20%) must key off the source port (or something derived from it, like conntrack hashing or a stat module). So evidence ordering: (1) confirm which rule translations applied per attempt (conntrack -E live), (2) record source ports of success vs failure, (3) diff the rules for port-dependent matches. But hold on — is there an even more canonical answer for "hairpin works sometimes, conntrack shows DNAT and SNAT"? Let me think about **the classic documented flaky case: hairpin with MASQUERADE where the client accesses by hostname that round-robins between public IP and another IP**... or **split DNS partially deployed**. Hmm. Another classic: **NAT loopback via the ISP router**: If the gateway's public IP is itself NAT'd by the upstream ISP (CGNAT), internal clients trying the public IP might reach the ISP's edge which sometimes hairpins (rare)... 20%? no. Let me reconsider: maybe the original topology wasn't hairpin at all. "External access works from external; internal can't reach the internal server via external IP" is the hairpin case — likely. The follow-up adds partial hairpin success. Alternatively the topology could be: internal clients NAT to internet via MASQUERADE (SNAT only). "External access" = internet access. Previously broken (no NAT), now "ALSO works sometimes from internal" — 1 in 5 attempts, conntrack shows both DNAT and SNAT hits. DNAT hits in a pure-MASQUERADE setup? If the user sees DNAT entries, there's a DNAT rule — supporting the port-forward interpretation. Given the follow-up phrasing "external access ALSO works sometimes from internal", the previous scenario likely was: external clients CAN access the internal service (DNAT works externally), internal CANNOT (hairpin missing). Now internal SOMETIMES can. So hairpin partially works. conntrack: both DNAT and SNAT hits → the hairpin SNAT rule exists and fires at least sometimes. Now the "1 in 5" with both DNAT and SNAT firing — the most defensible diagnosis with the given evidence: **The successful 20% have both translations; the failing 80% likely have DNAT only (SNAT missed) — OR the rules fire every time but a per-flow variable (source port) determines which of two conflicting rules wins.** The cleanest, most-likely real-world cause that yields "sometimes works, conntrack shows both DNAT and SNAT": **rule shadowing/ordering: an earlier, broader rule in POSTROUTING or a duplicate DNAT rule sometimes intercepts**. First-match semantics are deterministic per packet though. For per-attempt randomness with deterministic rule evaluation, the packet must differ — only the source port varies. So: two overlapping rules whose match boundary is the source port, OR conntrack-related: the masquerade chooses source port based on collisions... WAIT. I think I've got it — the truly canonical one for "works ~sometimes with conntrack showing both": **The hairpin SNAT rule was added to POSTROUTING with `--ctstate DNAT` or as MASQUERADE, but there's ALSO the external MASQUERADE `-o wan -j MASQUERADE`, and the LAN bridge/offload...** no. Let me think about **conntrack "hits" meaning iptables counters** and the 1-in-5 differently: Maybe the user tests with a browser; the page loads some resources direct (private IP via split-horizon DNS sometimes) — no. OK here's another strong classic I should weigh: **conntrack table full or per-protocol... no. **UDP vs TCP?** no. **The 1-in-5 might be "sometimes" ≈ after conntrack entry for the client→public tuple expires vs persists**: If an earlier BROKEN hairpin attempt (DNAT-only) created a conntrack entry in an unreplied state, and subsequent retries from the same client socket... each browser attempt = new socket = new tuple. no. **MSS clamping / PMTU**: works sometimes depending on payload sizes — small responses succeed, large fail (MTU blackhole). "1 in 5 attempts" if attempts are different operations (some small). conntrack would show both NATs on all. Symptoms: TCP connects but hangs on data. The user says "attempts" fail/succeed — could be page loads of varying size! But MTU issues aren't 20% random; they're size-dependent. Possible but I'd rank lower. Given I must "resolve the ambiguity by evidence ordering," the correct move is: lay out the hypotheses, then prescribe the ordered evidence sequence where each step eliminates branches, and explicitly commit to the diagnosis the given evidence supports. Let me now decide what the evidence given (1-in-5; conntrack both DNAT and SNAT hits) most supports: Key deduction chain: 1. Success at all from internal proves the hairpin path (DNAT+SNAT) is installed and functional end-to-end when fully applied. (Evidence: works sometimes + conntrack entries with both translations.) 2. Intermittency with a stable topology means the variable must be per-flow, and the only per-flow variable on the first SYN is the ephemeral source port (plus timing/collisions). 3. Therefore the discriminator: capture per-attempt conntrack tuples and rule counters; compare source ports of successes vs failures. Hmm, but honestly, in the field, the MOST COMMON cause of flaky hairpin NAT that I know of is **return-path asymmetry due to the SNAT rule not covering all client subnets** (deterministic per client) and **two gateways/ARP flux** (bimodal 50/50), and **conntrack race when the same tuple is used** ... none give 20%. 20% ≈ 1/5. What common mechanism gives exactly 1-in-N? **A DNAT address pool/range of N with one dead member**: `--to-destination 10.0.0.5-10.0.0.9` (5 addresses, round-robin per new connection... iptables DNAT with a range maps pseudo-randomly per connection). If one backend is down (or lacks the service / lacks the return route), ~1/5 of new connections fail... wait, 1 dead of 5 → 1/5 FAIL, 4/5 succeed. User says 1 in 5 SUCCEED. Inverse: 4 of 5 backends broken? Unlikely but the mechanism direction is wrong. What gives 1 in 5 success? One good path of five. Or a 20%-probability match. Or timing: something flaps with a period such that ~20% of attempts hit the good window (e.g., a cron job restarting the firewall every N minutes leaves a window where the new ruleset is loaded... or two rulesets alternate: nft and iptables-legacy both loaded, one has the full hairpin pair and one has only DNAT; WHICH ruleset processes the packet depends on... actually legacy and nftables are SEPARATE hook registrations; a packet traverses BOTH (both get the packet; both can mangle). If both a legacy DNAT and an nft DNAT exist for the same traffic: legacy DNAT applies, then nft rules see the ALREADY-DNAT'd packet (dst=10.0.0.5) so an nft rule matching `-d $PUBLIC_IP` won't match → its SNAT skipped → DNAT-only → fail. If only legacy has rules and nft empty → also DNAT-only... conversely if nft has the pair and legacy empty → both apply → success. For BOTH to sometimes... no, both always run. Deterministic again. Per-attempt randomness from timing: **client-side happy-eyeballs / DNS with multiple A records**: browser attempts IPv6 first sometimes? If the name has AAAA (broken) and A (works): happy eyeballs makes it near-always succeed. If the name has 5 A records, one pointing to the right public IP and others to wrong IPs: 1-in-5 success, and conntrack would show DNAT hits only on successes! The DNAT hit count ≈ 20% of attempts. And "conntrack shows both DNAT and SNAT hits" = on the successful flows both fired fine — the rules are FINE; the failure is upstream of the gateway (DNS). Evidence ordering resolves this: (1) count rule hits vs attempts: if DNAT rule counter increments on only ~20% of attempts while tcpdump on gateway sees 100% of SYNs to public IP arriving on internal iface... wait, if wrong A record, the SYN goes to a different IP entirely — may never reach the gateway, or reaches and is dropped (no DNAT rule) → conntrack NULL/drop. Evidence: tcpdump on gateway LAN iface for SYNs to public-IP:443 — do all 5 attempts arrive? If all arrive and only 1/5 get DNAT'd → rule-matching problem (port-keyed). If only 1/5 arrive → resolution/routing problem upstream (client-side). THIS is the cleanest first fork! Because it splits the world at the gateway boundary: attempts reaching the gateway vs not. So the evidence-ordered diagnosis: Step 0: Lock in what we know: both translations exist and can complete (the 20% successes prove the hairpin pair is installed, ordered, and functional). So stop auditing rule syntax for the success path. Step 1 (splits upstream-vs-gateway): On the gateway, `tcpdump -ni 'dst host and tcp dport '` while making 10 internal attempts. Two outcomes: - Only ~2/10 SYNs arrive at the gateway → the gateway is innocent; the attempt dies before it. Then the variability is in name resolution (multiple A records, stale split-DNS, /etc/hosts) or client routing (sometimes the client has a route/ARP to reach public IP elsewhere). Check: what IP did the failing attempt actually target? On the client: `getent hosts name`, browser dev tools, or tcpdump on the client. Correlate with conntrack: no DNAT entry for failures. Evidence order rationale: one packet capture at the single choke point distinguishes "gateway misbehaves" from "traffic never arrives" — highest information per step, non-invasive. - All 10 SYNs arrive → gateway-side per-flow nondeterminism; go to step 2. Step 2 (splits which translation applied): Run `conntrack -E -p tcp --dport 443` (event stream) during 10 attempts. Each attempt prints NEW with orig tuple and, as it confirms, the reply tuple: - Success pattern: orig src=CLIENT dst=PUB; reply src=SRV dst=GW-LAN-IP (SNAT applied) → then DESTROY/TIMEOUT... fine. - Failure pattern A: reply tuple shows dst=CLIENT (no SNAT) → SNAT rule missed; note the SYN's source port; check whether the SNAT rule contains a port-range/statistic/multi-match clause; check `iptables-save -t nat | `-v` counters: DNAT rule ≈ 10 hits, SNAT rule ≈ 2 hits confirms 20% SNAT match rate. Then fix the SNAT rule match. - Failure pattern B: entry exists with both translations but state stays SYN_SENT/UNREPLIED → both NATs applied but reply never returns to gateway → return-path asymmetry (server ARPs for client directly on same segment because... if SNAT applied, server replies to GW-LAN-IP — asymmetry impossible; unless SNAT'd to an IP the server doesn't route to, or gateway LAN iface has multiple IPs and ARP for the SNAT IP is answered by another device (IP conflict!) — ANOTHER classic: the SNAT source IP (gateway LAN IP) is DUPLICATED on the network (another host, or a VM clone), so ~4/5 ARPs answered by the wrong box → server sends SYN-ACK to the wrong MAC → fail; when the real gateway wins ARP (gratuitous refresh timing), success. ARP flux! That gives sub-100% success with BOTH translations present in conntrack but entries stuck in SYN_SENT (UNREPLIED) — and on the gateway you'd see SYN retransmits going out but no replies, plus `ip neigh` showing a flapping/wrong MAC for the SNAT IP... actually the gateway would see `ip neigh` for the SERVER fine; the problem is the SERVER's ARP cache for the gateway SNAT IP. Diagnose from the server: `ip neigh | grep ` — compare MAC to gateway's LAN MAC. THIS is a strong hypothesis for intermittent-with-both-NATs-applied! Evidence: conntrack entry created (both rules fired) but no reply packet ever hits the gateway → tcpdump on gateway LAN: see outbound SYN to server, no SYN-ACK back; tcpdump on server (or arp table) shows SYN-ACK sent to a different MAC. Hmm wait, but would an IP conflict yield exactly 1-in-5? It yields "sometimes," ratio depends on ARP dynamics — users report round numbers like "1 in 5." Plausible. Failure pattern C: NEW event shows NO entry at all for the failing SYN despite it arriving → the SYN was dropped before conntrack insert (raw table NOTRACK, or ingress drop) — but then successes and failures differ by... something keyed on the tuple (NOTRACK rule with port match!) — again port-keyed. So both step-2 failure modes funnel toward: **identify the per-attempt varying bit**. And the only per-attempt varying bits in a stable topology: (a) ephemeral source port, (b) ARP timing/caches, (c) name resolution (if attempts start at the client). And the evidence ordering cleanly discriminates: 1. tcpdump at gateway LAN ingress — did the failing SYN arrive? (fork: client-side vs gateway-side) 2. conntrack -E during attempts — which translations were programmed on failures? (fork: SNAT-miss vs both-applied-but-no-return) 3. If both-applied-but-no-return: tcpdump/ARP check on the return leg — is the SYN-ACK coming back to the gateway's MAC? (fork: ARP conflict / wrong SNAT IP vs server-side drop) 4. If SNAT-miss: dump nat table counters + exact failing tuples; diff successes vs failures (expect a port-keyed match boundary — e.g., stale rule copied with a sport range, or `-m statistic`, or a duplicate/shadowed rule whose position makes match depend on... no, position is deterministic). Also worth including: the "both DNAT and SNAT hits" evidence needs precise interpretation — "hits" in `iptables -t nat -L -v -n` counters are PER-RULE evaluations... actually in the nat table, rules are evaluated only for NEW connection first packets; counters increment per first packet matched. If DNAT counter = SNAT counter > 0 and both ≈ (number of successful + failed attempts that reached gateway), then on failures both fired → the failure is downstream (return path). If DNAT counter = attempts but SNAT counter ≈ 20% of it → SNAT rule match problem. THAT'S the decisive ordered evidence #2 actually — even before conntrack -E. Counter arithmetic is the cheapest discriminator. Let me also reconcile: what does conntrack showing "both DNAT and SNAT" per entry look like — I should show a sample line and how to read it, because the user's evidence claim hinges on reading it right: Successful hairpin entry: ``` tcp 6 431997 ESTABLISHED src=10.0.0.100 dst=203.0.113.10 sport=55001 dport=443 src=10.0.0.5 dst=10.0.0.1 sport=443 dport=55001 [ASSURED] use=1 ``` Reply tuple: src=10.0.0.5 (real server), dst=10.0.0.1 (the SNAT source = gateway LAN IP). The wire-level rewrite: outbound SYN seen by server: src=10.0.0.1:55001 dst=10.0.0.5:443; return SYN-ACK un-NAT'd to src=203.0.113.10 dst=10.0.0.100. Client sees full cone to public IP. DNAT-only entry (the smoking gun for SNAT miss): ``` ... src=10.0.0.100 dst=203.0.113.10 sport=55002 dport=443 src=10.0.0.5 dst=10.0.0.100 sport=443 dport=55002 [UNREPLIED] ``` Reply tuple dst = client's own IP → gateway expects the server to reply directly to the client — on the same L2 the reply never transits the gateway, entry ages out in ~120s (tcp SYN_SENT timeout), client RSTs. If the user sees UNREPLIED DNAT entries with 120s timers piling up during failures → diagnosis locked: SNAT rule intermittently missed. And why would SNAT miss 4/5 with identical-looking packets? The per-attempt variable = source port. Real rule examples that cause this: - `... --sport 32768:60999` (copied default ephemeral range) vs client actually using e.g. 49152+ (Windows/IANA) or an app binding low ports. Hmm wait — Linux default is 32768-60999; a rule `--sport 49152:65535` matches ~38%... A rule `--sport 60000:60999` matches ~3%... exact 20% is coincidence-prone; I shouldn't claim exactness, just mechanism: match keyed on a per-flow variable. - `-m statistic --mode random --probability 0.2` (someone "load testing" or a leftover) - Two SNAT rules where one is conditionally wrong IP (SNAT to wrong IP always fails though). - Actually another real one: **SNAT rule uses `-m conntrack --ctstate DNAT` placed in POSTROUTING... fine, matches all DNAT'd flows. - **MASQUERADE with `--random-fully`... no. Alternatively if counters show BOTH rules hit on ~all attempts (both ≈ attempts), failures are return-path: conntrack -E will show entries created but stuck UNREPLIED / SYN_SENT with both translations; then tcpdump on gateway LAN during failures: SYN out to server MAC (verify with `-e`), no SYN-ACK in. Then check on the server: does SYN-ACK go to the gateway MAC? `ip neigh show ` on server; compare with gateway's LAN MAC (`ip link show `). Mismatch or flapping → duplicate IP / ARP flux (e.g., a VM clone, a second device misconfigured with the gateway's LAN IP, or the public IP also ARPing on LAN via arp_ignore=0+proxy... ). Also: if SNAT is to the gateway's LAN IP and the server has a HOST FIREWALL allowing only certain sources... deterministic though. Also possible return-path asymmetry variant: SNAT rule SNATs to an IP that isn't locally configured on the gateway (e.g., SNAT to a stale old LAN IP) — server ARPs for it; gateway (not owning the IP, arp_ignore semantics) doesn't answer → sometimes another device answers → 20%? eh. One more important check in ordering: **confirm the 1-in-5 isn't measurement artifact** — e.g., the client retries transparently (TCP retrans can make a "failed page load" actually contain 5 SYNs of which... no—retransmits don't help if path broken). Actually here's a subtle one: if failures manifest as long timeouts (client gives up), the observed "1 in 5" might really be "success after ~4 timeouts" pattern from ARP cache refresh timing (stale ARP for gateway on SERVER refreshed by the occasional successful flow's traffic... ARP cache entries for the gateway IP on the server expire after ~60s(reachable timeout ~30s + gc); if most flows fail because server ARPs and the WRONG device answers, but the successful flow refreshes... circular. I think for the answer I should: A) State the diagnosis space compressed to three live hypotheses given the evidence, and explicitly kill the dead ones with reasoning: - Killed: "missing hairpin SNAT rule" (would be 0%, not 20%, on flat LAN; and conntrack shows SNAT hits). - Killed: "wrong DNAT target/port" (would be 0%). - Killed: "conntrack table full" (would hit external flows too; also user sees normal entries — check dmesg anyway? optional). - Live H1: SNAT rule matches only a per-flow-varying subset (port-keyed match / shadowing / statistic module) → signature: DNAT counter = all attempts; SNAT counter ≈ attempts/5; failing entries are DNAT-only UNREPLIED. - Live H2: both NATs always applied; return leg lost → signature: counters equal; failing entries have both tuples but stuck SYN_SENT/UNREPLIED; SYN-ACK never ingresses gateway → ARP/IP-conflict on the SNAT source IP, or server-side policy. - Live H3: only 20% of attempts ever reach the gateway → signature: tcpdump sees 20% of SYNs; cause upstream: DNS (multiple A/AAAA), stale client ARP to a different router, client-side proxy/VPN toggle. Note: conntrack DNAT "hits" then naturally = 20% — the dual hits are just the healthy path; user's evidence is consistent with this too! This is important: "conntrack shows both DNAT and SNAT hits" does NOT prove the rules fire on every attempt — counters are cumulative; 1 success in 5 produces both-hit entries for that 1. The user may be over-reading the evidence. Pointing this out IS the ambiguity resolution: the given evidence has a selection-bias ambiguity — conntrack entries only exist for flows that got translated; you cannot conclude per-attempt behavior from entry presence; you must ORDER the evidence to de-bias. That's the crux insight for "resolve the ambiguity by evidence ordering": The provided evidence (conntrack dual hits) is selected-on-the-dependent-variable — it's consistent with ALL THREE hypotheses, hence ambiguous. The 1-in-5 ratio tells us it's per-flow nondeterminism or per-flow targeting variance, not a deterministic config error. The ordering that resolves it: (1) ingress capture at the gateway (does the failing SYN arrive?) → splits H3 from {H1,H2} using ~1 command; (2) nat-table counter arithmetic + conntrack -E (did SNAT fire on failures?) → splits H1 from H2; (3) return-leg capture/ARP (does SYN-ACK come back to the gateway MAC?) → confirms H2 and localizes to ARP conflict vs server policy. Each step is ordered by information gain: step 1 partitions the space with a single passive capture at the choke point; step 2 uses counters already available (cheapest) plus an event stream; step 3 only runs if needed. B) Give the likely root causes per branch with the fix: - H1 fix: find the over-constrained/shadowed match in the POSTROUTING hairpin rule; canonical correct rule: `iptables -t nat -A POSTROUTING -o -s -d -p tcp --dport -j MASQUERADE` (or SNAT --to-source ) — crucially: match on the POST-DNAT destination (server IP), NOT the public IP; no sport ranges; placed with no earlier overlapping rule. Also ensure not duplicated across iptables-legacy AND nftables (double registration causes shadowing weirdness; check `iptables-save`, `nft list ruleset`). - H2 fix: resolve duplicate IP / ARP flux: `arping -D` the SNAT IP from the server side... or arp_scan; ensure the SNAT source IP is the gateway's primary LAN IP; set `arp_ignore=1, arp_announce=2` if multiple IPs; remove the clone. - H3 fix: split-horizon DNS (internal view returns private IP) or add a hosts entry; the cleaner long-term fix: internal DNS A record → server private IP, making hairpin unnecessary for internal clients; hairpin retained for unmanaged clients. C) Note the ratio forensics: 1-in-5 ≈ per-flow variable (source port, DNS record selection, ARP winner), NOT a global gate. And a "verification" close-out: after fix, `conntrack -E` shows every attempt with reply-tuple dst=gw and ESTABLISHED; counter ratio 1:1; 10/10 success ×3 runs; also confirm established-flow offload isn't bypassing... skip. Also worth a small note on why "both DNAT and SNAT hits" is EXPECTED on the healthy path (hairpin requires both by design — DNAT alone can't work same-L2), so that evidence alone was never diagnostic; its only value is proving the ruleset contains a viable hairpin pair. This respects the user's framing. Format considerations: The user speaks in terse expert shorthand ("Diagnose with the ambiguity resolved by evidence ordering") — they want a tight, technical, structured response. Likely a network engineer. I should: - Be concise but complete; use headers/ordered steps; include exact commands; include sample conntrack lines and how to read them; a decision tree; root causes & fixes per branch; and the explicit "evidence ordering" rationale (why this order: information gain / cheap-first / non-invasive-first). - Length: this is a complex diagnostic; medium-long is justified, but keep it disciplined — no fluff. Use code blocks for commands and conntrack samples. Maybe a compact table for hypotheses vs signatures. A table would serve well: hypothesis | signature in ordered evidence | fix. - Avoid overclaiming a single root cause since evidence is genuinely ambiguous — the user explicitly wants ambiguity resolved BY the ordering, i.e., show the fork logic and what each outcome locks in. But I should still commit: state which branch the given evidence most favors. Does the given evidence favor anything? "conntrack shows both DNAT and SNAT hits" — if the user counted hits and the ratio DNAT:SNAT isn't stated... If they saw per-entry dual translation, that only covers successes. I'd say the evidence is compatible with all three; slight tilt: if the user had seen piles of DNAT-only UNREPLIED entries they'd probably have said so → weakly disfavors H1? Not necessarily; UNREPLIED DNAT-only entries persist 120s+ and pile up visibly — absence of mention weakly favors H2 or H3. Hmm, honestly H3 (DNS/round-robin) or H2 (ARP conflict) both plausible. I'll present the fork honestly with a "most probable" note: in the field, with hairpin newly added and intermittency from day one, the most common are (a) the SNAT rule scoped wrong so it only catches some subnets/ports, (b) return-path ARP issues, (c) the test client resolving multiple IPs. But I'll keep the fork primary per instructions. Wait — one more classic I should double check for the "sometimes works from internal via public IP" with EXACTLY intermittent behavior: **the conntrack race between the outbound MASQUERADE for internet traffic and the hairpin SNAT**. Scenario: client's SYN to public IP egresses... no, hairpin SYN goes out the LAN iface. If the gateway LACKS the hairpin DNAT for `-i lan` and relies on the external DNAT rule without `-i` match: SYN from client arrives on lan, matches PREROUTING DNAT (no iface match) → DNAT'd → fine; then POSTROUTING: generic `-o wan MASQUERADE` doesn't match (-o lan); hairpin SNAT rule... if it exists it fires. Deterministic. **flowtable/software offload**: If `flowtable` offload is enabled (nftables or mt76/etc.), after the flow's first packets, it's offloaded; NAT is baked into the offload — fine if first packets NAT'd right. Known bugs: hairpin flows + flowtable caused breaks; intermittency would correlate with offload engagement (~after 4 packets? nft flowtable offloads after the first... it's after a few packets/MSL). For short-lived attempts (connection succeeds at TCP level then breaks) — user reports attempt-level success/failure; if TCP establishes then stalls, they'd describe "loads partially." Probably skip or one-line mention. Actually, worth ONE line under H2 ("both applied, reply lost") alternatives: TLS/TCP establishes but stalls → check flowtable/offload & MSS clamping. Keep it brief. Also: **the 1-in-5 could be `nf_conntrack_tcp_loose`/beelib... skip. Let me also prepare exact command set: Ordered evidence collection (run during 10 attempts): 0. Establish baseline variables to hold constant: test from ONE client, `curl -v --connect-timeout 3 http://PUBLIC:PORT/` in a loop of 10, log `sport` per attempt? curl doesn't print sport; tcpdump does. Fine. 1. Ingress test (gateway, LAN iface): ``` tcpdump -ni -e 'tcp port and (dst host or src host )' ``` counting SYNs to PUB_IP vs SYN-ACKs from SRV_IP arriving at gateway. - Failures with SYN present but no SYN-ACK → branch {H1,H2} - Failures with NO SYN → H3 (upstream). Confirm on client: `tcpdump -ni any 'dst host '` and `getent ahosts `; check for extra A records / IPv6. 2. Translation test (which NATs fired per attempt): ``` conntrack -E -p tcp --dport ``` plus counters: ``` iptables -vxnL PREROUTING -t nat; iptables -vxnL POSTROUTING -t nat ``` (or `nft list chain inet filter/... `). Read: DNAT pkts vs SNAT pkts ratio ≈ 1:1 or ≈5:1. - SNAT counter ≈ DNAT/5 → H1; failing conntrack events show reply tuple dst= (SNAT skipped), UNREPLIED. → inspect SNAT rule matches; look for sport/ctstate/statistic/multiport clauses and rule order (first overlapping POSTROUTING match wins); check duplicate rulesets (iptables-legacy vs nft). - Counters 1:1 and failing events show reply dst= but entry stuck SYN_SENT [UNREPLIED] → H2: return leg lost despite full NAT → check server-side: `ip neigh ` MAC vs gateway LAN MAC; `tcpdump -e` on gateway shows SYN-ACK never ingresses; suspect duplicate SNAT IP / ARP flux / server egress ACL; also MSS/flowtable if it ESTABLISHes then stalls. 3. (only for H2 confirmation) on the SERVER during a failing attempt: `tcpdump -ni -e 'host and tcp port '` — SYN-ACK egressing? to which MAC? `arping -I ` from a third host twice, compare MACs → duplicate IP evidence. Also note evidence-ordering rationale explicitly: each step is placed to maximize branch elimination per unit cost, passive before active, cumulative counters before live tracing, gateway before servers/clients... Actually ingress capture is arguably equally cheap as counters and more decisive; ordering: (1) counters [zero-cost, already-collected], (2) ingress capture [passive, one box], (3) conntrack -E [passive, one box], (4) active ARP probes. Hmm, but counter arithmetic only works if user stops other traffic — note that caveat (isolate the test client; clear/record counter baseline; run exactly 10 attempts; re-read counters). That's clean experimental design and matches "evidence ordering": sequence such that each result prunes the hypothesis tree. I might structure the answer as: 1. What the evidence already locks in (3 bullets: ruleset contains functional hairpin pair; failure is per-flow not static-config; the dual-hit evidence is survivorship-biased → it cannot distinguish "fires always, fails later" from "fires sometimes" — this is THE ambiguity to resolve). 2. The three surviving hypotheses (table: H1 SNAT under-matching, H2 return-leg blackhole, H3 attempts never reach gateway) with expected conntrack signatures. 3. Ordered evidence procedure (step, command, outcomes → which hypotheses die). Emphasize controlled 10-attempt run with counter baselines. 4. Per-branch root causes and fixes (canonical rules; ARP/IP-conflict; split DNS). 5. Confirmation criteria (what "fixed" looks like in the same instruments). Length check: this could balloon. Keep each section tight; use code blocks minimally but concretely. The user is clearly expert; no hand-holding on basics (skip explaining what DNAT is). I'll include sample conntrack lines since reading them IS the evidence. One more consideration on "evidence ordering": could the user mean "order the evidence by reliability/timestamps"? "Diagnose with the ambiguity resolved by evidence ordering" — I'll interpret as: the diagnostic must present an ordered evidence sequence where each step's outcome disambiguates, rather than guessing a single cause. I'll also note timestamp/correlation ordering where relevant (correlate conntrack -E event timestamps with attempt timestamps — e.g., if successes cluster periodically, suspect a flapping process (cron reloading firewalls, VRRP transitions) — add as a secondary observation: log `conntrack -E` timestamps; if the 1-in-5 correlates with a ~periodic window, pivot to "something flaps" (refire of rules, VRRP master flips, conntrack flush cron). Good addition — intermittency that's periodic vs IID changes the hypothesis space. Check `journalctl -u firewalld/ufw/netfilter-persistent` for reload churn; a broken reload that transiently flushes nat could give windows of success... Actually inverse: rules PRESENT = success window; if a config management tool re-applies rules every N min and a race leaves SNAT missing 80% of the time... plausible! `iptables-save | md5sum` polled during attempts, or watch counter deltas. Include as H4? Keep as a note within the procedure (if success correlates with time windows → poll ruleset hash; suspects: config mgmt reload races, `iptables-restore` clobbering, VRRP transitions). Hmm, keep H4 as a short "if evidence shows periodicity" pivot — the ordered procedure naturally surfaces it (success timestamps clustering). Also the dual-ruleset check (iptables-legacy + nft + iptables-nft mixed) deserves explicit mention in H1 causes: `iptables-save` (legacy) vs `iptables-nft-save` vs `nft list ruleset` — if hairpin pair lives in legacy but a flush tool manages nft (or vice versa), reload churn explains intermittency windows. Good. Now, what's my "most likely" commitment? Given "conntrack shows both DNAT and SNAT hits" was offered as if it's a surprising/clue-bearing fact, and the 1-in-5: I'd commit to: the rules fire on the successful flows only is NOT yet established — but if I must bet: in practice I most often find (a) SNAT rule match scoping (subnet/port) excluding most clients — but that's deterministic per client, so from ONE test client you'd get 0% or 100%, NOT 20%. For 20% from one client, the per-flow variable must matter → port-keyed match (uncommon) or return-path ARP (common in the field) or multi-A-record DNS (common!). Between ARP-conflict and DNS: if external access from outside works fine and clients are "internal users hitting the public name," split-brain DNS or the name having multiple A records is very common. But then some attempts go to an IP with NO listener → fast fail (RST/timeout)... The user would probably notice failing attempts targeting a different IP only with evidence step. I'll keep commitment soft: present forks; state that if I had to rank priors: H3 (only some attempts reach the gateway — DNS/round-robin) and H2 (ARP winner variance on the return leg) are the common field findings; H1 requires an unusual rule shape but is instantly confirmed by counter arithmetic. Actually, hmm — one more very common field cause of exactly "hairpin works sometimes": **the client sometimes gets the hairpin'd path and sometimes a direct cached path**... no. Or **browser proxy/PAC toggling**. Enough. Let me also make sure to answer with the diagnosis framing they asked: "Diagnose with the ambiguity resolved by evidence ordering." → The deliverable: a diagnosis procedure where ambiguity is resolved by ordered evidence, plus what each order outcome means, plus fixes. I'll also seed priors. Sample conntrack outputs to include: Success (both applied): ``` tcp 6 431999 ESTABLISHED src=10.20.0.50 dst=203.0.113.10 sport=51514 dport=443 \ src=10.20.0.10 dst=10.20.0.1 sport=443 dport=51514 [ASSURED] mark=0 use=1 ``` Read: reply tuple dst=10.20.0.1 → SNAT to gateway LAN IP → return leg must come to the gateway → healthy. Failure under H1 (SNAT skipped): ``` tcp 6 119 SYN_SENT src=10.20.0.50 dst=203.0.113.10 sport=48211 dport=443 \ src=10.20.0.10 dst=10.20.0.50 sport=443 dport=48211 [UNREPLIED] mark=0 use=1 ``` Read: reply tuple dst=10.20.0.50 = CLIENT → no SNAT → gateway expects server→client direct; on flat L2 that packet bypasses the gateway; entry dies at ~120s. (Also note: even if it did transit, un-DNAT would... if it transited, conntrack WOULD match it as reply and rewrite src to public IP — so on routed (non-flat) LANs DNAT-only works; flat LAN is where DNAT-only = fail. Mention this dependency: whether H1 causes visible failures depends on L2 adjacency of server and client... careful: reply tuple dst=client; the server sends SYN-ACK to client MAC directly (same subnet) → client discards (socket expects 203.0.113.10) → client retries SYN → gateway sees retransmits, same conntrack entry → still DNAT-only → loop → fail. On a flat net: deterministic fail.) Failure under H2 (both applied, return lost): same as success line but SYN_SENT/[UNREPLIED] and reply dst=GW_IP; SYN retransmits egress gateway; no SYN-ACK ingress. Good — the [UNREPLIED] + SYN_SENT + timeout read is the core signature craft. Counter arithmetic caveat: nat table counters count only NEW first-packets. So over exactly N isolated attempts: DNAT hit delta should = N (attempts reaching gw), SNAT delta = successes (if H1) or N (if H2) or ≤successes (if H3). That arithmetic alone nearly solves it! Actually elegant: three hypotheses give three distinct (DNATΔ, SNATΔ, client-attempts) triples: - H3: (S, S) with S ≈ N/5; missing SYNs in tcpdump. - H1: (N, S) SNATΔ ≈ successes. - H2: (N, N). That's the crispest formulation of the evidence ordering! Lead with it. The captures then localize WITHIN the winning branch. So evidence order: Step 1: isolate test client; record nat counter baseline; run exactly 10 attempts; read deltas → triple classification. (Zero packet capture; uses counters user already cited.) Step 2a (if (N,S)): dump exact failing tuples via conntrack -E / conntrack -L grep UNREPLIED; inspect the SNAT rule for per-flow-keyed matches (sport ranges, -m statistic, multiport, owner, connlimit, ctstate) and shadowing order; check dual rulesets (legacy vs nft) & reload churn (poll `iptables-save|md5sum` across attempts). Fix: canonical pair. Step 2b (if (N,N)): entries exist with both tuples → tcpdump -e on LAN: SYN-ACK ingress? No → server ARP cache / duplicate SNAT IP / server egress ACL; arping -D tests; fix ARP/clone or ACL. If SYN-ACK ingresses but still fails → ingress ACL on gateway INPUT for... no, reply to NAT'd flow goes through FORWARD/filter — check FORWARD rules conntrack ESTABLISHED accept; MSS/flowtable if established-then-stall. Step 2c (if (S,S)): gateway innocent → client-side: which dst did failures use? client tcpdump + `getent ahosts`; fix DNS (split-horizon A record to private IP — also makes hairpin moot), or hairpin via internal DNS. Confirmation/close: re-run 10×: expect (N,N) with 0 UNREPLIED and 10 ESTABLISHED; repeat ×3 including from a second client subnet (catches per-subnet scoping bugs in H1); verify from external still OK (guard against the fix breaking the external DNAT path — e.g., don't over-broaden SNAT... note: adding `-s lan -d srv SNAT` must not catch external reply traffic — POSTROUTING only sees egress; external→server replies egress via... server→external replies: src=srv dst=external — hairpin SNAT rule matches -d srv? No: -d srv won't match (dst is external client). Safe if rule requires -s .) Also small but important: the rule-order/first-match note: in nat PREROUTING, only the first packet of a conn is NAT'd; a wrong earlier ACCEPT... in nat table, ACCEPT stops traversal? In nat table, once a connection is NAT'd, later packets skip nat; rule traversal within nat for the first packet continues until first terminal target; DNAT/SNAT are terminal per... they're terminal targets within the chain? DNAT in PREROUTING terminates nat traversal (becomes the conn's nat binding). Two overlapping DNATs → first wins deterministically. Fine — mention only in H1 (shadowing) if SNAT keyed weirdly... SNAT also terminal. Shadowing matters for which of two SNAT rules applies (e.g., generic `-o wan MASQUERADE` before hairpin rule — but -o differs so no overlap; a generic `-o lan MASQUERADE` for guest→lan would overlap! If a pre-existing `-o -j MASQUERADE` for some guest subnet sits BEFORE the hairpin SNAT and also matches this traffic, the hairpin traffic gets masqueraded to... same effect (SNAT to gw IP) — success anyway. Unless the generic rule is a full-cone `-o lan -j MASQUERADE` — masquerade picks the lan iface primary IP = same result. So shadowing rarely breaks hairpin SNAT semantics. Drop that nuance, keep "port/statistic-keyed matches" + "ruleset churn/dual backends" as H1 causes. Under H1, one more genuinely common shape: the SNAT rule scoped to `-s 10.20.0.0/24` while test client... deterministic per client again. Only port-keyed gives per-attempt variance from a single client. OR the DNAT rule has `-m recent`/limits (`--hitlimit`)? `hashlimit` on the DNAT/port-forward for anti-DDoS: `--hashlimit-above 1/min` would make attempts 2-5 within a minute NOT DNAT → they'd... no DNAT → dst=public on gateway → forwarded out WAN → fails; 1 per minute succeeds → if user's "5 attempts" span >1 min → 1-in-5! Rate limiting (hashlimit/recent) on the port-forward is a REAL pattern for "works sometimes." conntrack: failures would show NO entry or a public-dst forwarded conn (SNAT'd by MASQUERADE out wan → conntrack shows SNAT-only hits!). Hmm! That gives a 4th signature: DNATΔ < attempts, SNATΔ from the masq... the user said "both DNAT and SNAT hits" — in H4(limiter), failures produce SNAT-only entries (client masqueraded out WAN to public IP → some routers/ISP loop... likely dies upstream). Deltas: DNATΔ ≈ S, SNAT(wan masq)Δ ≈ F. Distinguishable in step-1 arithmetic if you read per-rule, not per-translation. Good—include limiter (`hashlimit`, `recent`, `limit`) in H1's cause list, and note failures may show masquerade-out-WAN entries. Right — time to also sanity-check H2's ARP-conflict math: not exactly 20% but "sometimes"; users round. Fine. And note that H2's failure wouldn't create UNREPLIED... it WOULD: entry exists (both NATs programmed) but no reply packet ever seen → stays SYN_SENT [UNREPLIED] until 120s timeout. Client behavior: curl hangs ~2min+ or RST... client SYN retries ~63s then gives up. Yes. Also for completeness in H2: another return-leg killer with both NATs applied: **the SNAT IP chosen is not on the LAN subnet** (e.g., SNAT to the gateway's WAN IP or a VIP not ARP-announced on LAN): server ARPs for 203.0.113.10-ish or VIP → no answer (or upstream answers) → SYN-ACK never sent. Same evidence path (no SYN-ACK ingress at gw; server side shows incomplete ARP). Include as cause. And the policy-based-routing alternative under H2: ip rule on gateway sending LAN→LAN... skip; keep causes tight: (i) duplicate/contested SNAT IP (ARP), (ii) SNAT source not owned/announced, (iii) server egress firewall/ACL, (iv) offload/MSS if established-then-stall. For H3 causes: multiple A/AAAA (round-robin incl. CDN name), stale split-DNS/hosts entry, PAC/proxy, client VPN route flapping. Evidence: client-side capture; fix: internal DNS override (split-horizon) — the canonical "right" fix that eliminates hairpin dependence. Now the response format: I'll write: - **What the current evidence already proves** (short bullets). - **The ambiguity**: dual-hit conntrack evidence is survivorship-biased; three hypotheses survive; each predicts a DISTINCT counter signature → this is why we order evidence: one controlled run + counter arithmetic partitions the space. - **Ordered procedure** Step 1 (controlled run + per-rule deltas, with the (DNATΔ, SNATΔ) → branch table), Step 2a/2b/2c with commands + reading + root causes + fixes. - **Priors**: what I'd bet given field frequency. - **Confirmation** (definition of done). Use a small table for the branch signatures — tables render well and this is tabular (hypothesis × signature × meaning). Keep code blocks tight. Total maybe 500-700 words plus code/tables. The topic is deep; the user's message is expert-compressed; a thorough but disciplined answer is appropriate. Aim for high signal density, no filler. Let me draft the conntrack reading rules precisely: How to read a hairpin entry (LAN 10.20.0.0/24, gw LAN IP 10.20.0.1, client .50, server .10, public 203.0.113.10, port 443): - Tuple 1 (orig): src=10.20.0.50 dst=203.0.113.10 — client's SYN as sent. - Tuple 2 (reply expected): what the gateway must see coming back, pre-un-NAT: - src=10.20.0.10 dst=10.20.0.1 → both DNAT+SNAT wired; healthy path requires this. - src=10.20.0.10 dst=10.20.0.50 → SNAT never fired; on flat L2 the reply bypasses the gateway → dead. - State SYN_SENT + [UNREPLIED] for >5s during active retries = return path dead OR peer silent; correlate with tcpdump to distinguish. Commands: ``` # baseline iptables -vxnL PREROUTING -t nat; iptables -vxnL POSTROUTING -t nat # or nft list ruleset # controlled run from ONE client for i in $(seq 10); do curl -so /dev/null -w "%{http_code} %{time_connect}\n" --connect-timeout 5 https://203.0.113.10/ -k; sleep 1; done # deltas iptables -vxnL PREROUTING -t nat; iptables -vxnL POSTROUTING -t nat # live per-flow view during run conntrack -E -p tcp --dport 443 ``` Branch table: | Deltas over 10 attempts | Meaning | Branch | | DNAT=10, SNAT=10 | both fired every time → failure is post-NAT return path | H2 | | DNAT=10, SNAT≈2 | SNAT rule matches only a per-flow subset (port-keyed/statistic/limit) | H1 | | DNAT≈2, SNAT≈2 | 80% of SYNs never reached the gateway | H3 | | DNAT≈2, SNAT(wan masq)≈8 | DNAT rate-limited; failures masq'd out WAN | H1 variant (limiter) | Then per-branch: H1 (SNAT under-match): find why per-flow: `iptables-save -t nat` inspect the hairpin SNAT for `--sport`, `-m statistic`, `--hashlimit`, `--recent`, `--ctstate` restrictions; check dual backends: `iptables-legacy-save` vs `nft list ruleset` (a reload tool may manage one while rules live in the other, with churn windows). Canonical fix: ``` iptables -t nat -A PREROUTING -i -d 203.0.113.10 -p tcp --dport 443 -j DNAT --to-destination 10.20.0.10 iptables -t nat -A POSTROUTING -o -s 10.20.0.0/24 -d 10.20.0.10 -p tcp --dport 443 -j SNAT --to-source 10.20.0.1 ``` (match on post-DNAT dest; no per-flow keyed matches; SNAT to a LAN-owned IP.) H2 (return lost): confirm with `tcpdump -ni -e 'host 10.20.0.10 and tcp port 443'` — SYN egress, no SYN-ACK ingress. Then on server: `ip neigh show 10.20.0.1` vs gateway MAC; `arping -D`... causes: duplicate 10.20.0.1 (VM clone/appliance), SNAT IP not owned on LAN, server egress ACL. If ESTABLISHED then stall: MSS clamp `--clamp-mss-to-pmtu` / flowtable offload interaction. H3 (never arrives): client-side: `tcpdump -ni any 'host 203.0.113.10'`, `getent ahosts `; multiple A records → successes are the ones hitting the DNAT'd IP. Fix: split-horizon DNS A → 10.20.0.10 (makes hairpin irrelevant internally; keep DNAT for external). Priors: H3 and H2 most common in field; H1's limiter variant if the port-forward carries anti-abuse modules. But the counter arithmetic decides, not priors — that's the point of the ordering. Definition of done: three consecutive 10-attempt runs (two clients, incl. different subnet) all (DNAT=N,SNAT=N), zero UNREPLIED, external path regression-tested. Also a one-liner on why 1-in-5 ⇒ per-flow variable: stable topology + deterministic first-match NAT ⇒ per-attempt randomness can only enter via the variable fields of the flow (ephemeral sport), via contest/limiter state, via ARP winner, or via per-attempt destination selection upstream. The ordering above exactly discriminates these four. Nice crisp principle to state up front. Length control: I'll keep prose minimal, rely on structure. Use headers: "Locked in", "The ambiguity", "Ordered procedure", "Branch resolution", "Priors", "Done when". Tables: one. Code: three short blocks. This is a technical deep-dive request; ~700-900 words total is fine; avoid bloat. Let me also double check the conntrack reply-tuple semantics one more time so my sample lines are correct. For a DNAT+SNAT'd hairpin flow: Packet from client: 10.20.0.50:51514 → 203.0.113.10:443 arrives on LAN. PREROUTING DNAT → 10.20.0.10:443, POSTROUTING SNAT → src 10.20.0.1. On the wire server sees: 10.20.0.1:51514 → 10.20.0.10:443. conntrack -L shows: `src=10.20.0.50 dst=203.0.113.10 sport=51514 dport=443` (original, pre-NAT) `src=10.20.0.10 dst=10.20.0.1 sport=443 dport=51514` (reply, as-arriving-pre-unNAT: server's SYN-ACK 10.20.0.10:443→10.20.0.1:51514; un-NAT on output makes client see 203.0.113.10:443→10.20.0.50:51514). ✔️ My sample is right. DNAT-only: reply tuple `src=10.20.0.10 dst=10.20.0.50`. ✔️ Also note for the user: "both DNAT and SNAT hits" per-entry = the healthy shape; the diagnostic value is in the RATIO across attempts and the state/timer of failed entries, not their existence. One more possible reading of the user's "conntrack shows both DNAT and SNAT hits": they might mean `conntrack -S` or they grepped and saw two entries per connection? Two entries can't happen for one flow. Or they see DNAT entries (orig dst=public) AND separate SNAT-masq entries for other traffic — ambiguous but my treatment covers it via the ratio approach. I'll note the per-entry read explicitly so misreads get corrected. Trim: skip flowtable/MSS to a parenthetical. Skip conntrack-full (external would degrade too) — one line in "locked in"/eliminated list? Good to show elimination reasoning briefly: static misconfigs (missing rule, wrong target, wrong subnet scope) eliminated by the successes; global resource exhaustion (nf_conntrack full) eliminated because external path stays clean + entries are normal; deterministic first-match issues eliminated by intermittency ⇒ per-flow variables. That's tight elimination demonstrating "evidence ordering" thinking. Now write it. Keep the voice expert, terse, structured. Markdown headers + one table + fenced blocks. I'll name interfaces generically (lan/wan) and note placeholders. ## What the current evidence already locks in - **The hairpin pair exists and is functional end-to-end** — the 1-in-5 successes prove DNAT→SNAT→un-NAT completes when fully applied. Stop auditing rule syntax for the *success* path. - **Deterministic misconfigs are eliminated.** Missing SNAT, wrong DNAT target, wrong subnet scope = 0% from a given client, not 20%. Global resource exhaustion (conntrack table full) is eliminated because the external path stays clean and entries look normal. - **Therefore the randomness is per-flow.** With a stable topology and first-match NAT, per-attempt variance can only enter via: (a) the flow's variable fields (ephemeral source port), (b) contested/limited state, (c) ARP winner on the return leg, or (d) the *destination* differing per attempt upstream of the gateway. ## The ambiguity you must resolve "conntrack shows both DNAT and SNAT hits" is **survivorship-biased evidence**: conntrack entries only exist for flows that got translated. One healthy flow in five produces exactly this signature. It cannot distinguish "rules fire always, failure is downstream" from "rules fire sometimes." That's the ambiguity the ordering below resolves. ## Ordered procedure — one controlled run partitions the space Isolate one test client, record nat-table counter baselines (nat counters count **first packets only** — perfect for this), run exactly 10 attempts, read deltas: ```bash iptables -vxnL PREROUTING -t nat; iptables -vxnL POSTROUTING -t nat # baseline for i in $(seq 10); do curl -skso /dev/null -w "%{http_code} %{time_connect}\n" \ --connect-timeout 5 https://203.0.113.10/; sleep 1; done iptables -vxnL PREROUTING -t nat; iptables -vxnL POSTROUTING -t nat # deltas conntrack -E -p tcp --dport 443 # live, during run ``` | Deltas over 10 attempts | Meaning | Branch | |---|---|---| | DNAT=10, SNAT=10 | Both fire every time → failure is **post-NAT return path** | **H2** | | DNAT=10, SNAT≈2 | SNAT matches a per-flow subset only | **H1** | | DNAT≈2, SNAT≈2 | 80% of SYNs never reached the gateway | **H3** | | DNAT≈2, wan-MASQ≈8 | DNAT is rate-limited (`hashlimit`/`recent`); failures masquerade out WAN | **H1 variant** | **Why this order:** zero captures, uses counters you already have, and the three live hypotheses predict *distinct* triples. One run decides. ## Branch resolution **H2 — both translations applied, return leg black-holed.** Signature in `conntrack -E`: entry created with both tuples but stuck `SYN_SENT ... [UNREPLIED]`: ``` tcp 6 119 SYN_SENT src=10.20.0.50 dst=203.0.113.10 sport=51514 dport=443 src=10.20.0.10 dst=10.20.0.1 sport=443 dport=51514 [UNREPLIED] ``` Reply tuple `dst=10.20.0.1` (gateway LAN IP) = healthy wiring; the SYN-ACK just never comes back. Confirm: `tcpdump -ni -e 'host 10.20.0.10 and tcp port 443'` → SYN egress, no SYN-ACK ingress. Then on the **server**: `ip neigh show 10.20.0.1` vs the gateway's real MAC. Classic causes: **duplicate/contested SNAT IP** (VM clone, second appliance answering ARP), SNAT to an IP not owned/announced on the LAN, or server-side egress ACL. (Established-then-stall instead → MSS clamp / flowtable offload.) **H1 — SNAT rule under-matches.** Signature: failures show reply tuple `dst=10.20.0.50` (the *client*) — SNAT skipped — and on a flat L2 that reply bypasses the gateway, so the client discards it (socket expects 203.0.113.10): ``` tcp 6 119 SYN_SENT src=10.20.0.50 dst=203.0.113.10 sport=48211 dport=443 src=10.20.0.10 dst=10.20.0.50 sport=443 dport=48211 [UNREPLIED] ``` Since source port is the only per-attempt variable from one client, grep the hairpin SNAT for port-keyed or stateful matches: `--sport` ranges, `-m statistic`, `--hashlimit`, `--recent`. Also check **split backends and churn**: `iptables-legacy-save` vs `nft list ruleset` — if a config-management reload manages one while rules live in the other, you get intermittent windows; poll `iptables-save | md5sum` across attempts. Canonical fix, no per-flow keys, SNAT to a LAN-owned IP, matching the **post-DNAT** destination: ```bash iptables -t nat -A POSTROUTING -o -s 10.20.0.0/24 -d 10.20.0.10 \ -p tcp --dport 443 -j SNAT --to-source 10.20.0.1 ``` **H3 — attempts never reach the gateway.** Client-side capture shows failures targeting a different address: multiple A records (round-robin/CDN name), stale split-DNS, hosts entry, or PAC/proxy toggling. Fix: split-horizon DNS A → 10.20.0.10 for internal views — which also makes the hairpin path irrelevant for managed clients while the DNAT keeps serving external ones. ## Priors vs. verdict In the field, H3 (DNS round-robin) and H2 (ARP winner variance) are the most common findings for "hairpin works sometimes"; H1's limiter variant if the port-forward carries anti-abuse modules. But the counter arithmetic decides, not priors — that's the point of the ordering. ## Done when Three consecutive 10-attempt runs (include a second client on a different subnet — catches scope bugs H1 alone would hide) show `(DNAT=N, SNAT=N)`, zero `UNREPLIED` entries, and a regression pass on the external DNAT path.

SCB SCB-network-T2 MUSE