AGH + Traefik (DNS-over-TLS) : real client IPs are not shown

[Wonderfall]

Prerequisites

Please answer the following questions for yourself before submitting an issue. YOU MAY DELETE THE PREREQUISITES SECTION.

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed

Issue Details

• Version of AdGuard Home server:
  • v0.102.0
• How did you setup DNS configuration:
  • Traefik 853 (TCP routing) -> AdGuard 53 (Docker)
• Operating system and version:
  • Docker

Expected Behavior

AdGuard should see what the real clients IP are.

Actual Behavior

AdGuard can only see Traefik's IP, so no real clients. Evertyhing else is working as intended (tested with Private DNS on Android 10).

I know that HTTP routing makes use of X-Forwaded-For header, but is it possible to do that at all with TCP? Is it even something on AdGuard Home side?

While I could use AGH's own DoT server (it'd likely solve the issue), I prefer Traefik for some reasons (routing for specific domain, automatic certificates renewal, and so on).

Additional Information

  adguard:
    image: adguard/adguardhome
    container_name: adguard
    restart: unless-stopped
    user: 3211:3211 # Rootless container, likely not related to this issue
    networks:
      - http_network
    depends_on:
      - unbound
    volumes:
      - /home/docker/adguard/config:/opt/adguardhome/conf
      - /home/docker/adguard/data:/opt/adguardhome/work
    labels:
      - traefik.enable=true
      - traefik.docker.network=http_network

      # HTTP related (0.0.0.0:3001 is admin interface)
      - traefik.http.routers.adguard.entrypoints=http
      - traefik.http.routers.adguard.rule=Host(`dns.domain.tld`)
      - traefik.http.routers.adguard.middlewares=https-redirect@file
      - traefik.http.routers.adguard-secure.entrypoints=https
      - traefik.http.routers.adguard-secure.rule=Host(`dns.domain.tld`)
      - traefik.http.routers.adguard-secure.tls=true
      - traefik.http.routers.adguard-secure.tls.certresolver=http
      - traefik.http.routers.adguard-secure.middlewares=secure-headers@file,hsts-headers@file
      - traefik.http.routers.adguard-secure.service=adguard-web
      - traefik.http.services.adguard-web.loadbalancer.server.port=3001

      # DNS-over-TLS (0.0.0.0:5053 is DNS)
      - traefik.tcp.routers.adguard-dot.rule=HostSNI(`dns.domain.tld`)
      - traefik.tcp.routers.adguard-dot.entrypoints=dot
      - traefik.tcp.routers.adguard-dot.tls.certresolver=http
      - traefik.tcp.routers.adguard-dot.service=adguard
      - traefik.tcp.services.adguard.loadbalancer.server.port=5053

[Potterli20]

#1787 #1383

[ameshkov]

I know that HTTP routing makes use of X-Forwaded-For header, but is it possible to do that at all with TCP? Is it even something on AdGuard Home side?

Unfortunately, there are no such options for TLS/TCP.

As @Potterli20 stated, #1383 seems relevant, but it'd require using a wildcard cert.

[Wonderfall]

I see, I was afraid it was a TCP limitation (I observed the same issue with Pi-Hole and a similar setup). Seems like we hit a trade-off between certificate convenience and this.

One workaround would be to use the built-in DNS-over-TLS server and make a script that can extract the certificates from acme.json (V2 format).

I'm going to close the issue since it doesn't seem to be resolvable as it is, thanks for responding though!

[Akruidenberg]

@Wonderfall are you still using this setup? I have practicily the same docker compose config as you, but my browser is very slow.
However, the buid-in dot server of adguard is working fine. My android phone and desktop have the same issue. In the beginning a part of website is loading, but then, the browsers keeps the session open the load the rest of the page. After a minute the rest of page is loading.

its very frustrating because the rest is working fine. Here is my docker compose file:

  traefik:
    container_name: traefik
    image: traefik:v2.2.4
    restart: always
    command:
      - --global.checkNewVersion=true
      - --global.sendAnonymousUsage=false     
      - --entryPoints.http.address=:80
      # https://www.reddit.com/r/docker/comments/c1wrep/traefik_reverse_proxy_question_docker_overlay/
#      - --entrypoints.http.forwardedHeaders.trustedIPs=127.0.0.1/31, 192.168.90.1/24
#      - --entrypoints.http.proxyProtocol=true
#      - --entrypoints.http.proxyProtocol.trustedIPs=127.0.0.1/31, 192.168.90.1/24
      - --entryPoints.https.address=:443
      - --entryPoints.urbackup.address=:55415
      - --entryPoints.tls.address=:853
      - --api=true
#     - --api.insecure=true
#      - --serversTransport.insecureSkipVerify=true
      - --log=true
      - --log.level=INFO # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
      - --accessLog=true
      - --accessLog.filePath=/var/log/docker/traefik.log
      - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
      - --accessLog.filters.statusCodes=400-499
      - --providers.docker=true
#      - --providers.docker.defaultrule=Host(`{{ index .Labels "com.docker.compose.service" }}.$DOMAINNAME`)
      - --providers.docker.exposedByDefault=false
      - --entrypoints.https.http.tls.certresolver=ovh
      - --entrypoints.https.http.tls.domains[0].main=$DOMAINNAME
      - --entrypoints.https.http.tls.domains[0].sans=*.$DOMAINNAME    
      - --providers.docker.network=reverseproxy
      - --providers.docker.endpoint=tcp://dockersock-proxy:2375
      - --providers.docker.swarmMode=false
      - --providers.file.directory=/rules
      - --providers.file.watch=true
      - --certificatesresolvers.ovh.acme.dnschallenge=true
#      - --certificatesResolvers.ovh.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory # uncomment when testing
      - --certificatesResolvers.ovh.acme.email=$SMTP_DEST_EMAIL
      - --certificatesResolvers.ovh.acme.storage=/acme.json
      - --certificatesResolvers.ovh.acme.dnsChallenge.provider=ovh
#      - --certificatesResolvers.ovh.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
      - --certificatesResolvers.ovh.acme.dnsChallenge.delayBeforeCheck=90 # To delay DNS check and reduce LE hitrate
    networks:
      - reverseproxy
      - dockersock-proxy
    security_opt:
      - no-new-privileges:true  
    user: $PUID
    ports:
      - "80:80"
      - "443:443"
      - "55415:55415"
      - "853:853"
    volumes:
      - $USERDIR/traefik/rules:/rules
      - $USERDIR/traefik/acme.json:/acme.json
      - $USERDIR/traefik/traefik.log:/var/log/docker/traefik.log
    environment:
      OVH_ENDPOINT: ovh-eu
      OVH_APPLICATION_KEY: $OVH_APPLICATION_KEY
      OVH_APPLICATION_SECRET: $OVH_APPLICATION_SECRET
      OVH_CONSUMER_KEY: $OVH_CONSUMER_KEY
    sysctls:
      net.ipv4.ip_unprivileged_port_start: "0"
    labels:
      traefik.enable: "true"
      # HTTP-to-HTTPS Redirect
      traefik.http.routers.http_catchall.rule: HostRegexp(`{any:.+}`)
      traefik.http.routers.http_catchall.entrypoints: http
      traefik.http.routers.http_catchall.middlewares: https_redirect@docker
      traefik.http.middlewares.https_redirect.redirectscheme.scheme: https
      traefik.http.middlewares.https_redirect.redirectscheme.permanent: "true"
      # HTTP Routers
      traefik.http.routers.traefik-rtr.entrypoints: https
      traefik.http.routers.traefik-rtr.rule: HostHeader(`traefik.$DOMAINNAME`)
      ## Middlewares
      traefik.http.routers.traefik-rtr.middlewares: chain-oauth@file
      ## Services - API
      traefik.http.routers.traefik-rtr.service: api@internal
      ## watchtower      
      com.centurylinklabs.watchtower.enable: "false"     

  adguard:
    container_name: adguardhome
    restart: unless-stopped
    image: adguard/adguardhome
    hostname: adguard
    ports:
      - "53:53"
      - "53:53/udp"
    expose:
      - "3000"
      - "443"
      - "80"
    volumes:
      - $USERDIR/adguard/conf:/opt/adguardhome/conf
      - $USERDIR/adguard/work:/opt/adguardhome/work
      - $USERDIR/traefik/certs:/certs:ro
    sysctls:
      net.ipv4.ip_unprivileged_port_start: "0"  
    user: $PUID
    networks:
      - reverseproxy
    labels:
      traefik.enable: "true"
      ## HTTP Routers
      traefik.http.routers.adguard-rtr.entrypoints: https
      traefik.http.routers.adguard-rtr.rule: HostHeader(`dns.$DOMAINNAME`)
      ## Middlewares
      traefik.http.routers.adguard-rtr.middlewares: chain-no-auth@file        
      ## HTTP Services
      traefik.http.routers.adguard-rtr.service: adguard-svc
      traefik.http.services.adguard-svc.loadbalancer.server.port: 3000  
      ###TCP
      traefik.tcp.routers.adguard-tls.entrypoints: tls
      traefik.tcp.routers.adguard-tls.rule: HostSNI(`dns.$DOMAINNAME`)  
      ## tcp Services 
      traefik.tcp.routers.adguard-tls.service: adguard-svc-tls   
      
      traefik.tcp.routers.adguard-tls.tls: "true"                 
      traefik.tcp.services.adguard-svc-tls.loadbalancer.server.port: 53  
      ## watchtower       
      com.centurylinklabs.watchtower.enable: "false"        
      

See #1495 for my issue

[FrancYescO]

@Wonderfall did u finally made the script? or @Akruidenberg have you any update on the slowness?
looks i have also another issue.. using:
https://medium.com/@nykolas.z/troubleshooting-dns-over-tls-e7ca570b6337
to check if DoT was served okay, looks like the certificate on the 853 is not served (it uses the "TRAEFIK DEFAULT CERT") if using the direct IP when connecting to the DNS server and i assume that this certificate validity is the check that chrome does when is on the default settings

ie. if i have on my OS, 192.168.1.2 as primary dns and 8.8.8.8 as secondary it will stick to use 8.8.8.8 or at least it will keep skipping the custom rewrites in adguard :/

EDIT1:
After few more research seems there is an issue with comunication btw traefik and agh, i'll get this on each ip from client on trefik log: http: TLS handshake error from 123.12.12.23:54493: tls: client requested unsupported application protocols ([dot])

EDIT2:
finally traefik/traefik#7906 (comment) give me an hint, due to the strict check of alpn added in the newest traefik versions..
you should add "dot" to the supported protocols adding

tls:
  options:
    default:
      alpnProtocols:
        - http/1.1
        - h2
        - acme-tls/1
        - dot

to your provider file... i still to understand why is serving default traefik certificate over the 853 port instead of the one in the acme.json...

[yfhyou]

@FrancYescO

to your provider file... i still to understand why is serving default traefik certificate over the 853 port instead of the one in the acme.json...

Did you ever figure out why it serves the default certificate instead of a generated one? I think it might be a bug possibly in traefik?

When I set up a TCP router with the dot ALPN and run a test I get the following:

ubuntu@home-build:~$ kdig -d @192.168.2.3 +tls-ca +tls-host=dot.my.domain google.com
;; DEBUG: Querying for owner(google.com.), class(1), type(1), server(192.168.2.3), port(853), protocol(TCP)
;; DEBUG: TLS, imported 137 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG:  #1, CN=TRAEFIK DEFAULT CERT
;; DEBUG:      SHA-256 PIN: gg+69u7KAm7oKxdmXu1uGgv2i7vL1boHRVD0vMjjUSU=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is NOT trusted. The certificate issuer is unknown. The name in the certificate does not match the expected.
;; WARNING: TLS, handshake failed (Error in the certificate.)
;; ERROR: failed to query server 192.168.2.3@853(TCP)

[FrancYescO]

Nope, i still have as workaround directly exposing adguard (and "stealing" the trafik generated certficate)