Browse Source

contrib/sni-router: use host networking for HAProxy to preserve client IPs

Move HAProxy into the host network namespace so it sees the real
client source IP on inbound connections.  With bridge networking +
published ports the source IP is rewritten to the bridge gateway by
the runtime (Docker's userland-proxy, rootless Podman's slirp4netns
or pasta), and the PROXY v2 header HAProxy then sends to mtg and
Caddy carries that useless address.

mtg and Caddy stay on the compose bridge and publish their ports on
host loopback; the host-mode HAProxy dials them at 127.0.0.1.  Caddy's
proxy_protocol allow list is tightened to loopback only.

The 'sysctls: net.ipv4.ip_unprivileged_port_start=80' line is removed
because Docker refuses to apply namespaced sysctls when the netns is
shared with the host.  Rootless Podman users binding the privileged
ports need the equivalent host-side sysctl once; this is documented
in README.md.

Fixes #498.
contrib/sni-router-host-mode
dolonet 1 month ago
parent
commit
b86a9cf85d

+ 4
- 4
contrib/sni-router/Caddyfile View File

10
 	# to Caddy's access log.  The `tls` wrapper must follow so that TLS
10
 	# to Caddy's access log.  The `tls` wrapper must follow so that TLS
11
 	# is terminated on the unwrapped connection.
11
 	# is terminated on the unwrapped connection.
12
 	#
12
 	#
13
-	# `allow` lists the networks permitted to send PROXY headers.  These
14
-	# ranges cover docker compose's default bridge networks; tighten
15
-	# them if you pin a specific subnet in docker-compose.yml.
13
+	# `allow` lists the networks permitted to send PROXY headers.
14
+	# HAProxy runs in the host netns and reaches Caddy via host loopback
15
+	# (see docker-compose.yml), so the only legitimate peer is loopback.
16
 	servers :8443 {
16
 	servers :8443 {
17
 		listener_wrappers {
17
 		listener_wrappers {
18
 			proxy_protocol {
18
 			proxy_protocol {
19
 				timeout 5s
19
 				timeout 5s
20
-				allow 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
20
+				allow 127.0.0.0/8 ::1/128
21
 			}
21
 			}
22
 			tls
22
 			tls
23
 		}
23
 		}

+ 24
- 0
contrib/sni-router/README.md View File

58
 If you disable one, disable all four, otherwise the backend will fail
58
 If you disable one, disable all four, otherwise the backend will fail
59
 to parse the connection.
59
 to parse the connection.
60
 
60
 
61
+## Why host networking for HAProxy
62
+
63
+HAProxy runs in the host network namespace (`network_mode: host` in
64
+`docker-compose.yml`) so it sees the real client source IP on every
65
+inbound connection.  With the default bridge networking + published
66
+ports the source IP is rewritten to the bridge gateway — by Docker's
67
+userland proxy (`docker-proxy`), by rootless Podman's `slirp4netns`
68
+or `pasta`, or by NAT on the Docker host — and the PROXY v2 header
69
+HAProxy then sends to mtg and Caddy carries that useless address.
70
+Host networking lifts HAProxy out of the rewrite path; mtg and Caddy
71
+stay on the compose bridge and HAProxy dials them via host loopback
72
+(`127.0.0.1`).
73
+
74
+Trade-off: HAProxy occupies the host's `:443` and `:80` directly, so
75
+nothing else on the host may listen on those ports.  For a dedicated
76
+mtg/SNI-router host that is the intended layout.
77
+
78
+Rootless Podman users binding the privileged ports `:80`/`:443` need
79
+the host-side sysctl once (rootful Docker handles this implicitly):
80
+
81
+```bash
82
+sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
83
+```
84
+
61
 ## Fronting loop (why `[domain-fronting]` is set explicitly)
85
 ## Fronting loop (why `[domain-fronting]` is set explicitly)
62
 
86
 
63
 When mtg sees TLS that isn't valid Telegram (a probe or a browser
87
 When mtg sees TLS that isn't valid Telegram (a probe or a browser

+ 15
- 10
contrib/sni-router/docker-compose.yml View File

24
 services:
24
 services:
25
   haproxy:
25
   haproxy:
26
     image: haproxy:lts-alpine
26
     image: haproxy:lts-alpine
27
-    ports:
28
-      - "443:443"
29
-      - "80:80"
27
+    # Host networking so HAProxy sees the real client source IP on
28
+    # inbound (bridge networking with published ports rewrites it to
29
+    # the bridge gateway under both Docker's userland-proxy and
30
+    # rootless Podman's slirp4netns/pasta).  See "Why host networking"
31
+    # in README.md.
32
+    network_mode: host
30
     volumes:
33
     volumes:
31
       - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro,Z
34
       - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro,Z
32
     environment:
35
     environment:
35
       - mtg
38
       - mtg
36
       - web
39
       - web
37
     restart: unless-stopped
40
     restart: unless-stopped
38
-    sysctls:
39
-      - net.ipv4.ip_unprivileged_port_start=80
40
 
41
 
41
   mtg:
42
   mtg:
42
     image: nineseconds/mtg:2
43
     image: nineseconds/mtg:2
43
     volumes:
44
     volumes:
44
       - ./mtg-config.toml:/config/config.toml:ro,Z
45
       - ./mtg-config.toml:/config/config.toml:ro,Z
45
-    expose:
46
-      - "3128"
46
+    # Publish on host loopback so the host-mode HAProxy can reach it.
47
+    ports:
48
+      - "127.0.0.1:3128:3128"
47
     restart: unless-stopped
49
     restart: unless-stopped
48
     extra_hosts:
50
     extra_hosts:
49
       - "host.containers.internal:host-gateway"
51
       - "host.containers.internal:host-gateway"
54
       - ./Caddyfile:/etc/caddy/Caddyfile:ro,Z
56
       - ./Caddyfile:/etc/caddy/Caddyfile:ro,Z
55
       - caddy_data:/data
57
       - caddy_data:/data
56
       - ./www:/srv:ro,Z
58
       - ./www:/srv:ro,Z
57
-    expose:
58
-      - "80"
59
-      - "8443"
59
+    # Publish on host loopback so the host-mode HAProxy can reach it.
60
+    # Caddy's HTTP listener is mapped off :80 (occupied by HAProxy)
61
+    # to :8080; haproxy.cfg dials it on 127.0.0.1:8080 for ACME.
62
+    ports:
63
+      - "127.0.0.1:8080:80"
64
+      - "127.0.0.1:8443:8443"
60
     environment:
65
     environment:
61
       <<: *domain-env
66
       <<: *domain-env
62
     restart: unless-stopped
67
     restart: unless-stopped

+ 8
- 3
contrib/sni-router/haproxy.cfg View File

50
     # send-proxy-v2 prepends a PROXY protocol v2 header so mtg sees the
50
     # send-proxy-v2 prepends a PROXY protocol v2 header so mtg sees the
51
     # real client IP instead of HAProxy's.  mtg must have
51
     # real client IP instead of HAProxy's.  mtg must have
52
     # `proxy-protocol-listener = true` in its config.
52
     # `proxy-protocol-listener = true` in its config.
53
-    server mtg mtg:3128 send-proxy-v2
53
+    #
54
+    # HAProxy runs in the host network namespace (see docker-compose.yml)
55
+    # and mtg publishes :3128 on host loopback, so we dial 127.0.0.1.
56
+    server mtg 127.0.0.1:3128 send-proxy-v2
54
 
57
 
55
 backend web
58
 backend web
56
     # send-proxy-v2 prepends a PROXY protocol v2 header so Caddy logs the
59
     # send-proxy-v2 prepends a PROXY protocol v2 header so Caddy logs the
57
     # real client IP instead of HAProxy's.  Caddy must enable the
60
     # real client IP instead of HAProxy's.  Caddy must enable the
58
     # proxy_protocol listener wrapper on :8443 (see Caddyfile).
61
     # proxy_protocol listener wrapper on :8443 (see Caddyfile).
59
-    server web web:8443 send-proxy-v2
62
+    server web 127.0.0.1:8443 send-proxy-v2
60
 
63
 
61
 backend web_acme
64
 backend web_acme
62
     mode http
65
     mode http
63
-    server web web:80
66
+    # Caddy's :80 is published on host 127.0.0.1:8080 (HAProxy occupies
67
+    # the host's :80 itself).
68
+    server web 127.0.0.1:8080

Loading…
Cancel
Save