Skip to content

Instantly share code, notes, and snippets.

@sbonds
Last active February 6, 2019 00:54
Show Gist options
  • Save sbonds/dc72b458848e8b72aba6028d140d72c4 to your computer and use it in GitHub Desktop.
Save sbonds/dc72b458848e8b72aba6028d140d72c4 to your computer and use it in GitHub Desktop.
LearnChef: Try Inspec Docker DNS Issues

On the step:

# git clone https://github.com/learn-chef/auditd.git

I get:

Cloning into 'auditd'...
fatal: unable to access 'https://github.com/learn-chef/auditd.git/': Could not resolve host: github.com

The DNS config inside the workstation looks like this:

# cat /etc/resolv.conf
search <my internal domain>
nameserver 127.0.0.11
options ndots:0

The setup for this is as follows:

Host OS: CentOS7

$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

Docker version

$ docker --version
Docker version 1.13.1, build 774336d/1.13.1

User setup and initial config:

# useradd -m learnchef
# usermod --groups docker learnchef
# su - learnchef
$ pip install --user docker-compose
$ vi ~/.bashrc
export PATH=$PATH:$HOME/.local/bin
$ . ~/.bashrc
$ mkdir learn-inspec
$ cd learn-inspec
$ curl -C - -O  https://raw.githubusercontent.com/learn-chef/inspec/master/docker-compose.yml
$ cat docker-compose.yml
version: '3'
services:
  workstation:
    container_name: workstation
    image: learnchef/inspec_workstation
    stdin_open: true
    tty: true
    links:
      - target
    volumes:
      - .:/root
  target:
    image: learnchef/inspec_target
    stdin_open: true
    tty: true

$ docker-compose version
docker-compose version 1.23.2, build 1110ad0
docker-py version: 3.7.0
CPython version: 2.7.5
OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Getting the Docker images going

$ docker-compose pull
Pulling target      ... done
Pulling workstation ... done

$ docker-compose up -d
Creating network "learn-inspec_default" with the default driver
Creating learn-inspec_target_1 ... done
Creating workstation           ... done

$ docker exec -it workstation bash
root@e81dcdc8f026:/#
root@e81dcdc8f026:/# inspec version
2.0.17

# git clone https://github.com/learn-chef/auditd.git
Cloning into 'auditd'...
fatal: unable to access 'https://github.com/learn-chef/auditd.git/': Could not resolve host: github.com
@sbonds
Copy link
Author

sbonds commented Feb 5, 2019

Some good info about the inner workings of Docker's DNS was found here: docker/for-linux#179

@sbonds
Copy link
Author

sbonds commented Feb 6, 2019

For some reason, when I tried things again I had a completely different failure:

$ docker-compose up -d
Creating network "learn-inspec_default" with the default driver
Creating learn-inspec_target_1 ... error

ERROR: for learn-inspec_target_1  Cannot start service target: shim error: docker-runc not installed on system

ERROR: for target  Cannot start service target: shim error: docker-runc not installed on system
ERROR: Encountered errors while bringing up the project.

A bit of Google later, and the solution was (on the CentOS 7 Docker host server):

 # systemctl stop docker
 # cd /usr/libexec/docker
 # ln -s docker-runc-current docker-runc
 # ln -s docker-init-current docker-init
 # ln -s docker-proxy-current docker-proxy
 # systemctl start docker

@sbonds
Copy link
Author

sbonds commented Feb 6, 2019

I also found this document on how to enable debugging in Docker itself so I can see what's going on:

https://docs.docker.com/config/daemon/

In particular, I changed /etc/docker/daemon.json from:

{}

to:

{
  "debug": true
}

When I run:

$ docker-compose up -d
$ docker exec -it workstation bash
$ git clone https://github.com/learn-chef/auditd.git

The following appears in /var/log/messages during the "git clone" operation:

Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.987904044-08:00" level=debug msg="Name To resolve: github.com."
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.987904489-08:00" level=debug msg="Name To resolve: github.com."
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.990566690-08:00" level=debug msg="Query github.com.[28] from 172.18.0.3:48427, forwarding to udp:192.168.1.252"
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.990566783-08:00" level=debug msg="Query github.com.[1] from 172.18.0.3:44050, forwarding to udp:192.168.1.252"
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.991841353-08:00" level=debug msg="Name To resolve: github.com."
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.992139402-08:00" level=debug msg="Query github.com.[1] from 172.18.0.3:33356, forwarding to udp:192.168.1.252"
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.992376243-08:00" level=debug msg="Name To resolve: github.com."
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.992537400-08:00" level=debug msg="Query github.com.[28] from 172.18.0.3:53133, forwarding to udp:192.168.1.252"
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.993047498-08:00" level=debug msg="Name To resolve: github.com.internal.net."
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.993201750-08:00" level=debug msg="Query github.com.internal.net.[28] from 172.18.0.3:38682, forwarding to udp:192.168.1.252"
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.993376021-08:00" level=debug msg="Name To resolve: github.com.internal.net."
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.993525378-08:00" level=debug msg="Query github.com.internal.net.[1] from 172.18.0.3:52108, forwarding to udp:192.168.1.252"
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.994730233-08:00" level=debug msg="Name To resolve: github.com.internal.net."
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.994877283-08:00" level=debug msg="Query github.com.internal.net.[28] from 172.18.0.3:44166, forwarding to udp:192.168.1.252"
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.995013650-08:00" level=debug msg="Name To resolve: github.com.internal.net."
Feb  5 16:30:01 dockerhost dockerd-current: time="2019-02-05T16:30:01.995165252-08:00" level=debug msg="Query github.com.internal.net.[1] from 172.18.0.3:39069, forwarding to udp:192.168.1.252"

Which is somewhat good news-- it suggests that the Docker image itself is fine and the problem is somewhere within my extremely default Docker server configuration.

@sbonds
Copy link
Author

sbonds commented Feb 6, 2019

I found some good reading in the pull request where the embedded DNS server was created, but no smoking gun for why it seems unable to forward to a working DNS IP: moby/libnetwork#841

@sbonds
Copy link
Author

sbonds commented Feb 6, 2019

FOUND IT

The nameserver was configured to only respond to local subnets, not random bridged IP ranges. Those were added to named.conf in the allow-recursion section and (no surprise) DNS works now.

I also disabled Docker debugging and restarted it before I forgot and came back to gigabyte sized log files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment