Просмотр исходного кода

sni-router: switch HAProxy to host networking for real client IPs

Bridge ingress (Docker's docker-proxy userland forwarder, Podman's
slirp4netns/pasta) rewrites the source IP of inbound connections on a
published port to the bridge gateway address.  HAProxy then stamps that
gateway address into the PROXY v2 header it forwards to mtg and Caddy,
so neither backend ever sees a real client IP.

Move HAProxy into the host netns (network_mode: host) so it binds
:443/:80 directly with no NAT in the path.  mtg and Caddy stay on the
compose bridge and are published on 127.0.0.1 only; HAProxy reaches
them via host loopback and PROXY v2 carries the real client IP (v4 or
v6) end-to-end.

Also accept IPv6 clients explicitly on the HAProxy frontends — `bind
*:443` is IPv4-only and missed v6 clients on hosts where the previous
example happened to "work" only because of dual-stack quirks.

Add 127.0.0.0/8 to Caddy's PROXY allow-list to cover the new loopback
hop from HAProxy.  README gains a short subsection explaining the
host-mode choice and its trade-off (HAProxy occupies host :443/:80).

Diagnosed and tested by @bam80 on Fedora + Docker 29.  Fixes #498.
pull/522/head
Alexey Dolotov 2 месяцев назад
Родитель
Сommit
4a4e001980
4 измененных файлов: 59 добавлений и 19 удалений
  1. 6
    4
      contrib/sni-router/Caddyfile
  2. 20
    0
      contrib/sni-router/README.md
  3. 18
    10
      contrib/sni-router/docker-compose.yml
  4. 15
    5
      contrib/sni-router/haproxy.cfg

+ 6
- 4
contrib/sni-router/Caddyfile Просмотреть файл

10
 	# to Caddy's access log.  The `tls` wrapper must follow so that TLS
10
 	# to Caddy's access log.  The `tls` wrapper must follow so that TLS
11
 	# is terminated on the unwrapped connection.
11
 	# is terminated on the unwrapped connection.
12
 	#
12
 	#
13
-	# `allow` lists the networks permitted to send PROXY headers.  These
14
-	# ranges cover docker compose's default bridge networks; tighten
15
-	# them if you pin a specific subnet in docker-compose.yml.
13
+	# `allow` lists the networks permitted to send PROXY headers.
14
+	# 127.0.0.0/8 covers HAProxy reaching Caddy over host loopback (HAProxy
15
+	# runs in network_mode: host and connects to the published 127.0.0.1
16
+	# port).  The RFC1918 ranges cover mtg → Caddy on the compose bridge
17
+	# (fronting path; see "Fronting loop" in README.md).
16
 	servers :8443 {
18
 	servers :8443 {
17
 		listener_wrappers {
19
 		listener_wrappers {
18
 			proxy_protocol {
20
 			proxy_protocol {
19
 				timeout 5s
21
 				timeout 5s
20
-				allow 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
22
+				allow 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
21
 			}
23
 			}
22
 			tls
24
 			tls
23
 		}
25
 		}

+ 20
- 0
contrib/sni-router/README.md Просмотреть файл

63
 If you disable one, disable all four, otherwise the backend will fail
63
 If you disable one, disable all four, otherwise the backend will fail
64
 to parse the connection.
64
 to parse the connection.
65
 
65
 
66
+### Why HAProxy uses `network_mode: host`
67
+
68
+When a container is on a bridge network and a port is published with
69
+`ports: "443:443"`, the source IP of inbound connections is rewritten
70
+to the bridge gateway before HAProxy sees it — Docker's `docker-proxy`
71
+userland forwarder accepts on the host and re-opens the connection
72
+from the gateway; Podman's `slirp4netns` / `pasta` does the same in
73
+rootless mode.  The PROXY v2 header HAProxy then sends downstream
74
+carries that gateway address (e.g. `172.x.x.1`), not the real client.
75
+
76
+`network_mode: host` puts HAProxy in the host network namespace, so it
77
+binds `:443` / `:80` directly with no NAT in the path and observes the
78
+true source address of every connection.  mtg and Caddy stay on the
79
+compose bridge and are published only on `127.0.0.1` — HAProxy reaches
80
+them via host loopback, and the PROXY v2 header carries the real
81
+client IP (v4 or v6) end-to-end.
82
+
83
+Trade-off: HAProxy occupies the host's `:443` and `:80`.  Don't run
84
+anything else on those ports on the same host.
85
+
66
 ## Fronting loop (why `[domain-fronting]` is set explicitly)
86
 ## Fronting loop (why `[domain-fronting]` is set explicitly)
67
 
87
 
68
 When mtg sees TLS that isn't valid Telegram (a probe or a browser
88
 When mtg sees TLS that isn't valid Telegram (a probe or a browser

+ 18
- 10
contrib/sni-router/docker-compose.yml Просмотреть файл

27
 services:
27
 services:
28
   haproxy:
28
   haproxy:
29
     image: haproxy:lts-alpine
29
     image: haproxy:lts-alpine
30
-    ports:
31
-      - "443:443"
32
-      - "80:80"
30
+    # network_mode: host lets HAProxy see real client source IPs (v4 and v6)
31
+    # instead of the docker/podman bridge gateway.  Bridge ingress (docker-proxy
32
+    # userland forwarder, podman slirp4netns/pasta) rewrites the source address
33
+    # of inbound connections to the gateway; with host networking HAProxy binds
34
+    # in the host netns directly and the rewrite never happens.  See the
35
+    # "Real client IPs" section of README.md.
36
+    #
37
+    # Trade-off: HAProxy occupies host :443 and :80.  Don't run anything else
38
+    # on those ports.
39
+    network_mode: host
33
     volumes:
40
     volumes:
34
       - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro,Z
41
       - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro,Z
35
     environment:
42
     environment:
38
       - mtg
45
       - mtg
39
       - web
46
       - web
40
     restart: unless-stopped
47
     restart: unless-stopped
41
-    sysctls:
42
-      - net.ipv4.ip_unprivileged_port_start=80
43
 
48
 
44
   mtg:
49
   mtg:
45
     # FIXME: :master until #480 lands in a tagged release; switch back to :2/:3 after release
50
     # FIXME: :master until #480 lands in a tagged release; switch back to :2/:3 after release
46
     image: nineseconds/mtg:master
51
     image: nineseconds/mtg:master
47
     volumes:
52
     volumes:
48
       - ./mtg-config.toml:/config/config.toml:ro,Z
53
       - ./mtg-config.toml:/config/config.toml:ro,Z
49
-    expose:
50
-      - "3128"
54
+    # Published on host loopback only — HAProxy (host netns) reaches it via
55
+    # 127.0.0.1:3128.  Not exposed on any public interface.
56
+    ports:
57
+      - "127.0.0.1:3128:3128"
51
     restart: unless-stopped
58
     restart: unless-stopped
52
     extra_hosts:
59
     extra_hosts:
53
       - "host.containers.internal:host-gateway"
60
       - "host.containers.internal:host-gateway"
58
       - ./Caddyfile:/etc/caddy/Caddyfile:ro,Z
65
       - ./Caddyfile:/etc/caddy/Caddyfile:ro,Z
59
       - caddy_data:/data
66
       - caddy_data:/data
60
       - ./www:/srv:ro,Z
67
       - ./www:/srv:ro,Z
61
-    expose:
62
-      - "80"
63
-      - "8443"
68
+    # Published on host loopback only — HAProxy reaches Caddy on 127.0.0.1.
69
+    ports:
70
+      - "127.0.0.1:8080:80"
71
+      - "127.0.0.1:8443:8443"
64
     environment:
72
     environment:
65
       <<: *domain-env
73
       <<: *domain-env
66
     restart: unless-stopped
74
     restart: unless-stopped

+ 15
- 5
contrib/sni-router/haproxy.cfg Просмотреть файл

23
 # --- HTTP :80 — ACME challenges + redirect -----------------------------------
23
 # --- HTTP :80 — ACME challenges + redirect -----------------------------------
24
 
24
 
25
 frontend http
25
 frontend http
26
-    bind *:80
26
+    # Explicit v4 + v6 binds so IPv6 clients are accepted regardless of the
27
+    # host's IPV6_V6ONLY sysctl.  v6only on the v6 bind avoids the
28
+    # "address in use" overlap on dual-stack hosts.
29
+    bind 0.0.0.0:80
30
+    bind [::]:80 v6only
27
     mode http
31
     mode http
28
 
32
 
29
     # Let Caddy answer ACME HTTP-01 challenges for Let's Encrypt.
33
     # Let Caddy answer ACME HTTP-01 challenges for Let's Encrypt.
35
 # --- TLS :443 — SNI-based routing -------------------------------------------
39
 # --- TLS :443 — SNI-based routing -------------------------------------------
36
 
40
 
37
 frontend tls
41
 frontend tls
38
-    bind *:443
42
+    bind 0.0.0.0:443
43
+    bind [::]:443 v6only
39
     tcp-request inspect-delay 5s
44
     tcp-request inspect-delay 5s
40
     tcp-request content accept if { req_ssl_hello_type 1 }
45
     tcp-request content accept if { req_ssl_hello_type 1 }
41
 
46
 
46
 
51
 
47
     default_backend web
52
     default_backend web
48
 
53
 
54
+# Backends reach mtg and web on host loopback — they publish to 127.0.0.1
55
+# (see docker-compose.yml), and HAProxy runs in the host netns
56
+# (network_mode: host).  PROXY v2 still carries the real client address
57
+# (v4 or v6) end-to-end, independent of the loopback transport.
58
+
49
 backend mtg
59
 backend mtg
50
     # send-proxy-v2 prepends a PROXY protocol v2 header so mtg sees the
60
     # send-proxy-v2 prepends a PROXY protocol v2 header so mtg sees the
51
     # real client IP instead of HAProxy's.  mtg must have
61
     # real client IP instead of HAProxy's.  mtg must have
52
     # `proxy-protocol-listener = true` in its config.
62
     # `proxy-protocol-listener = true` in its config.
53
-    server mtg mtg:3128 send-proxy-v2
63
+    server mtg 127.0.0.1:3128 send-proxy-v2
54
 
64
 
55
 backend web
65
 backend web
56
     # send-proxy-v2 prepends a PROXY protocol v2 header so Caddy logs the
66
     # send-proxy-v2 prepends a PROXY protocol v2 header so Caddy logs the
57
     # real client IP instead of HAProxy's.  Caddy must enable the
67
     # real client IP instead of HAProxy's.  Caddy must enable the
58
     # proxy_protocol listener wrapper on :8443 (see Caddyfile).
68
     # proxy_protocol listener wrapper on :8443 (see Caddyfile).
59
-    server web web:8443 send-proxy-v2
69
+    server web 127.0.0.1:8443 send-proxy-v2
60
 
70
 
61
 backend web_acme
71
 backend web_acme
62
     mode http
72
     mode http
63
-    server web web:80
73
+    server web 127.0.0.1:8080

Загрузка…
Отмена
Сохранить