traefik & Docker Swarm으로 서버 구축하기

Tue Jan 12 2021

참고 참고2

신경쓸게 너무 많아서 한꺼번에 정리하는 글. 배경 설명은 다 집어치우고 해야 할 것만 골라서 간단하게 설명한다.

누가 이걸 볼까 싶지만 따라하고 싶다면 traefik(+ Docker swarm)에 대한 기초 이해는 가지고 시작하는 것이 좋을 듯.

Docker Swarm Init

# Docker 설치 필요
docker swarm init

Docker Network 설정

목적에 따라 서로 다른 네트워크 3개를 구성할 것이다.

  1. cloud-edge (for Traefik 2)
  2. cloud-public (인터넷에 노출시킬 서비스용)
  3. cloud-socket-proxy (docker socket proxy - 보안 목적)
# ingress 네트워크 삭제 후 재생성
docker network rm ingress
docker network create --ingress --driver overlay \
   --opt encrypted --subnet 10.10.0.0./16 ingress

# cloud-edge / cloud-public / cloud-socket-proxy 생성
docker network create --subnet 10.11.0.0/16 --driver overlay \
  --scope swarm --opt encrypted --attachable cloud-edge

docker network create --subnet 10.12.0.0/16 --driver overlay \
  --scope swarm --opt encrypted --attachable cloud-socket-proxy

docker network create --subnet 10.13.0.0/16 --driver overlay \
  --scope swarm --opt encrypted --attachable cloud-public

Node Label & 환경 변수 추가

export NODE_ID=$(docker info -f '{{.Swarm.NodeID}}')
docker node update --label-add cloud-public.traefik-certificates=true $NODE_ID

이어서 DOMAIN, USERNAME, EMAIL, TRAEFIKADMINS, WHITELISTIP 환경변수가 필요한데 그 전에 인증용 비밀번호 해쉬툴이 필요하다.

# 데비안 계열
sudo apt update && sudo apt install -y apache2-utils

# RHEL
sudo yum install -y httpd-tools

WHITELIST_IP는 dashboard에 접근 가능한 IP를 지정하는 용도.

export DOMAIN="<domain here>"
export EMAIL="<email for letsencrypt certificates here>"
export WHITELIST_IP="<your public ip>/32"
export USERNAME="<username here>"
export TRAEFIK_ADMINS=$(htpasswd -nBC 10 $USERNAME)

TLS

TLS는 dynamic configuration을 통해 설정하는데, /home/$USER/treafik_conf/dynamic_conf.toml 파일을 생성한 다음 폴더째로 docker에 바인딩할 것이다.

dynamic_conf.toml

[tls.options]
  [tls.options.default]
    minVersion = "VersionTLS13"
    sniStrict = true

  [tls.options.tls12]
    minVersion = "VersionTLS12"
    sinStrict = true
    cipherSuites = [
      "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
      "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
      "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
      "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
      "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
      "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"
    ]

Docker Compose

version: '3.8'

# Let's Encrypt의 인증서를 저장하는 volume
volumes:
  traefik-certificates:

networks:
  cloud-edge:
    external: true
  cloud-public:
    external: true
  cloud-socket-proxy:
    external: true

services:
  reverse-proxy:
    image: traefik:v2.2
    command:
      - --providers.docker
      # Use the secure docker socket proxy
      - --providers.docker.endpoint=tcp://socket-proxy:2375
      # Add a constraint to only use services with the label "traefik.constraint-label=cloud-public"
      - --providers.docker.constraints=Label(`traefik.constraint-label`, `cloud-public`)
      # Don't expose containers per default
      - --providers.docker.exposedByDefault=false
      - --providers.docker.swarmMode=true
      # fileprovider needed for TLS configuration
      # see https://github.com/containous/traefik/issues/5507
      - --providers.file.filename=traefik_conf/dynamic_conf.toml
      # Entrypoints (ports) for the routers
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      # Entrypoint for the dashboard on port 8000
      - --entrypoints.api.address=:8000
      # Create the certificate resolver "letsencrypt" for Let's Encrypt, uses the environment variable EMAIL
      - --certificatesresolvers.letsencrypt.acme.email=${EMAIL?Variable not set}
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.letsencrypt.acme.tlschallenge=true
      # Let's encrypt 인증 제한을 피하기 위해서 개발 도중에 사용되는 caServer
      - --certificatesresolvers.letsencrypt.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory

      # Logging
      - --accesslog
      - --log.level=debug

      # Enable the dashboard
      - --api
    deploy:
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.labels.cloud-public.traefik-certificates == true
          - node.role == manager
      labels:
        # traefik.enable is required because we don't expose all containers automatically
        - traefik.enable=true
        - traefik.docker.network=cloud-public
        - traefik.constraint-label=cloud-public

        # Global redirection: HTTP to HTTPS
        - traefik.http.routers.http-redirects.entrypoints=web
        - traefik.http.routers.http-redirects.rule=hostregexp(`{host:(www\.)?.+}`)
        - traefik.http.routers.http-redirects.middlewares=traefik-ratelimit,redirect-to-non-www-https

        # Global redirection: HTTPS www to HTTPS non-www
        - traefik.http.routers.www-redirects.entrypoints=websecure
        - traefik.http.routers.www-redirects.rule=hostregexp(`{host:(www\.).+}`)
        - traefik.http.routers.www-redirects.tls=true
        - traefik.http.routers.www-redirects.tls.options=default
        - traefik.http.routers.www-redirects.middlewares=traefik-ratelimit,redirect-to-non-www-https

        # Middleware to redirect to bare https
        - traefik.http.middlewares.redirect-to-non-www-https.redirectregex.regex=^https?://(?:www\.)?(.+)
        - traefik.http.middlewares.redirect-to-non-www-https.redirectregex.replacement=https://$${1}
        - traefik.http.middlewares.redirect-to-non-www-https.redirectregex.permanent=true

        # Dashboard on port 8000
        - traefik.http.routers.api.entrypoints=api
        - traefik.http.routers.api.rule=Host(`${DOMAIN?Variable not set}`)
        - traefik.http.routers.api.service=api@internal
        - traefik.http.routers.api.tls=true
        - traefik.http.routers.api.tls.options=default
        - traefik.http.routers.api.tls.certresolver=letsencrypt

        # middlewares:
        # IP whitelist, ratelimit and basic authentication
        - traefik.http.routers.api.middlewares=api-ipwhitelist,traefik-ratelimit,api-auth
        - traefik.http.middlewares.api-auth.basicauth.users=${TRAEFIK_ADMINS?Variable not set}
        # 접속 가능한 white list IP 지정
        - traefik.http.middlewares.api-ipwhitelist.ipwhitelist.sourcerange=${WHITELIST_IP?Variable not set}
        - traefik.http.services.api.loadbalancer.server.port=8000

        # Extra middleware (ratelimit, ip whitelisting)
        - traefik.http.middlewares.traefik-ratelimit.ratelimit.average=100
        - traefik.http.middlewares.traefik-ratelimit.ratelimit.burst=50
    # use host mode for network ports for ip whitelisting
    # see https://community.containo.us/t/whitelist-swarm-cant-get-real-source-ip/3897
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8000
        published: 8000
        protocol: tcp
        mode: host
    volumes:
      # storage for the SSL certificates
      - traefik-certificates:/letsencrypt
      # bind mount the directory for your traefik configuration
      - /home/$USER/traefik_conf:/traefik_conf
    networks:
      - cloud-edge
      - cloud-public
      - cloud-socket-proxy

  socket-proxy:
    image: tecnativa/docker-socket-proxy:latest
    deploy:
      restart_policy:
        condition: on-failure
      placement:
        constraints: [node.role == manager]
    environment:
      # permssions needed
      NETWORKS: 1
      SERVICES: 1
      TASKS: 1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      cloud-socket-proxy:
        aliases:
          - socket-proxy

그리고 나서 docker stack deploy -c <stack name>을 입력하면 된다.

letsencrypt caServer가 staging 서버로 되어 있어서 dashboard에 접속하면 브라우저가 안전하지 않은 연결이라고 경고를 띄울텐데 개발 도중에 계속해서 인증서를 다시 발급받다가 인증 제한에 걸리는 사태를 피하기 위한 설정이다.

다음 설정값을 삭제하거나 주석처리하면 정상 동작한다.

—certificatesresolvers.letsencrypt.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory

하지만 기존에 발급된 테스트용 인증서가 traefik-certifcates 볼륨에 남아있을 수 있기 때문에 이걸 삭제하고 다시 배포해야 한다.

서비스 배포

whoami.mytraefik.com이라는 URL에 새 서비스를 배포하고 싶다면 다음과 같이 docker compose 파일을 작성한다.

version: '3.8'

services:
  whoami:
    image: containous/whoami:latest
    command:
      - --port=8082
    deploy:
      labels:
        # traefik.enable=true and constraint label are needed because
        # of the restrictions we enforced in our traefik configuration
        - traefik.enable=true
        - traefik.constraint-label=cloud-public
        - traefik.http.routers.whoami.entrypoints=websecure
        - traefik.http.routers.whoami.rule=Host(`whoami.mytraefik.com`)
        - traefik.http.routers.whoami.tls=true
        # min TLS version
        - traefik.http.routers.whoami.tls.options=tls12@file
        - traefik.http.routers.whoami.tls.certresolver=letsencrypt
        - traefik.http.routers.whoami.middlewares=traefik-ratelimit
        - traefik.http.services.whoami.loadbalancer.server.port=8082
    networks:
      - cloud-public

networks:
  cloud-public:
    external: true

그리고 나서 배포 명령어 실행

docker stack deploy -c whoami.yaml <name-of-your-swarm>
Loading...
snark

닉네임은 Snark이지만 어째선지 썸네일은 Jabberwocky