Skip to content

Instantly share code, notes, and snippets.

@mcculloughsean
Last active August 29, 2015 14:04
Show Gist options
  • Save mcculloughsean/5c1dd3cc674e6fd583f5 to your computer and use it in GitHub Desktop.
Save mcculloughsean/5c1dd3cc674e6fd583f5 to your computer and use it in GitHub Desktop.

Refactoring Docker Remote API Client

What's wrong

The docker remote api isn't easy to use because:

  • There's no official client to the API
  • Much of the validation the CLI provides is in the CLI code, not the API layer
  • The documentation doesn't always show a proper mapping of CLI commands to API calls (e.g. docker run involves multiple steps)
  • The CLI code is cluttered with many concerns and makes understanding how to use the API properly by example difficult
  • Marshaling/Unmarshaling in Go should use the same structures as the internal code for easier code sharing (i.e. dont require consumers to implement a type Containers struct for unmarshalling the output of docker ps)

How to fix

Propose:

  • Creating an official API client for use in the CLI tool
  • Exposing proper validation from the API layer or in the API client
  • Simplifying the CLI code and breaking commands down into smaller files so they're easier to understand.
  • Leverage pre-existing types when (un)serializing calls from the API

Creating an official client

Composes 'Call' with commands to create a fetcher per endpoint in the API with proper unmarshalling.

Reimplement CLI in terms of API client

Use existing types for marshaling/unmarshaling JSON data for use in Go code.

Move formatting to methods for CLI stringification.

Create helper methods to generate valid query strings.

Push validations down to API as much as possible

Remove all checks from the CLI level that can easily live in the API layer.

Create a common pattern for passing validation errors back up through the API pipeline (maybe by switching unmarshaling strategy based on http status code)

Ensure that validation failure messages are verbose. e.g.:

$curl -v -H "Content-Type: application/json" -d "{\"Hostname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"AttachStdin\":false,\"AttachStdout\":true,\"AttachStderr\":true,\"PortSpecs\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":null,\"Cmd\":[\"date\"],\"Image\":\"repository.snc1/candyland/echo-fedora\",\"Volumes\":{\"/tmp\":{}},\"WorkingDir\":\"\",\"DisableNetwork\":false,\"ExposedPorts\":{\"22/tcp\":{}}}" http://localhost:12345/containers/create?name=foo
* About to connect() to localhost port 12345 (#0)
*   Trying ::1... Connection refused
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 12345 (#0)
> POST /containers/create?name=foo HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:12345
> Accept: */*
> Content-Type: application/json
> Content-Length: 340
>
< HTTP/1.1 201 Created
< Content-Type: application/json
< Date: Thu, 31 Jul 2014 21:54:33 GMT
< Content-Length: 90
<
{"Id":"2d3a1faaa6d83680ddb4164dc39755ef18fab37b4bf66e88f62bd00168547faa","Warnings":null}
* Connection #0 to host localhost left intact
* Closing connection #0

$curl -v -H "Content-Type: application/json" -d "{\"Hostname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"AttachStdin\":false,\"AttachStdout\":true,\"AttachStderr\":true,\"PortSpecs\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":null,\"Cmd\":[\"date\"],\"Image\":\"repository.snc1/candyland/echo-fedora\",\"Volumes\":{\"/tmp\":{}},\"WorkingDir\":\"\",\"DisableNetwork\":false,\"ExposedPorts\":{\"22/tcp\":{}}}" http://localhost:12345/containers/create?name=foo
* About to connect() to localhost port 12345 (#0)
*   Trying ::1... Connection refused
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 12345 (#0)
> POST /containers/create?name=foo HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:12345
> Accept: */*
> Content-Type: application/json
> Content-Length: 340
>
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Date: Thu, 31 Jul 2014 21:54:35 GMT
< Content-Length: 53
<
Abort due to constraint violation: constraint failed

The second error message isn't useful. The dockerd logs themselves aren't useful either:

[debug] server.go:999 Calling POST /containers/create
2014/07/31 21:54:33 POST /containers/create?name=foo
[ead17ee3] +job create(foo)
[error] mount.go:11 [warning]: couldn't run auplink before unmount:
exec: "auplink": executable file not found in $PATH
[ead17ee3] -job create(foo) = OK (0)
[debug] server.go:999 Calling POST /containers/create
2014/07/31 21:54:35 POST /containers/create?name=foo
[ead17ee3] +job create(foo)
Abort due to constraint violation: constraint failed
[ead17ee3] -job create(foo) = ERR (1)
[error] server.go:1025 Error: Abort due to constraint violation:
constraint failed
[error] server.go:90 HTTP Error: statusCode=500 Abort due to constraint
violation: constraint failed
package api
import (
"crypto/tls"
"io"
)
// So i don't go crazy with stupid gofmt warnings
type Container struct {
}
var registry interface {
ConfigFile
}
// Create an Api Client that holds onto state about how to connect to the remote api
type ApiClient struct {
// ...
}
func NewApiClient(proto, addr string, tlsConfig *tls.Config) *ApiClient {
// ...
return &ApiClient{}
}
// move call over to the ApiClient as a private method
func (client *ApiClient) call(method, path string, data interface{}, passAuthInfo bool) (io.ReadCloser, int, error)
// in a api/client package with each endpoint in separate file
type PsFlags struct {
Quiet bool
Size bool
All bool
NoTrunc bool
NLatest bool
Since string
Before string
Last int
}
// Separate out encoding to a method
func (flags PsFlags) UrlEncode() string {
v := url.Values{}
if flags.Last == -1 && flags.NLatest {
flags.Last = 1
}
if flags.All {
v.Set("all", "1")
}
if flags.Last != -1 {
v.Set("limit", strconv.Itoa(flgags.Last))
}
if flags.Since != "" {
v.Set("since", flags.Since)
}
if flags.Before != "" {
v.Set("before", flags.Before)
}
if flags.Size {
v.Set("size", "1")
}
return v.Encode()
}
// Handle logic around managing connection to remote API
func (client *ApiClient) CallPs(flags PsFlags) (containers []*Container, e error) {
body, _, err := readBody(cli.call("GET", "/containers/json?"+v.Encode(), nil, false))
if err != nil {
return nil, err
}
if err := json.Unmarshal(body, containers); err != nil {
return nil, err
}
return containers, nil
}
// in the main client package, remove raw proto state and include a reference to the client
type DockerCli struct {
configFile *registry.ConfigFile
in io.ReadCloser
out io.Writer
err io.Writer
isTerminal bool
terminalFd uintptr
tlsConfig *tls.Config
scheme string
Client *ApiClient
}
// Refactored version of the CLI command
func (cli *DockerCli) CmdPs(args ...string) error {
// parse out the cli options
cmd := cli.Subcmd("ps", "[OPTIONS]", "List containers")
quiet := cmd.Bool([]string{"q", "-quiet"}, false, "Only display numeric IDs")
size := cmd.Bool([]string{"s", "-size"}, false, "Display sizes")
all := cmd.Bool([]string{"a", "-all"}, false, "Show all containers. Only running containers are shown by default.")
noTrunc := cmd.Bool([]string{"#notrunc", "-no-trunc"}, false, "Don't truncate output")
nLatest := cmd.Bool([]string{"l", "-latest"}, false, "Show only the latest created container, include non-running ones.")
since := cmd.String([]string{"#sinceId", "#-since-id", "-since"}, "", "Show only containers created since Id or Name, include non-running ones.")
before := cmd.String([]string{"#beforeId", "#-before-id", "-before"}, "", "Show only container created before Id or Name, include non-running ones.")
last := cmd.Int([]string{"n"}, -1, "Show n last created containers, include non-running ones.")
if err := cmd.Parse(args); err != nil {
return nil
}
// hand those options into a standard interface to the API client
flags = PsFlags{
Quiet: *quiet,
Size: *size,
All: *all,
NoTrunc: *noTrunc,
NLatest: *nLatest,
Since: *since,
Before: *before,
Last: *last,
}
// call the api client endpoint, get a list of containers back
containers, err := cli.Client.CallPs(flags)
if err != nil {
return err
}
// delegate to something else to format the output
printContainerTable(containers)
// or just define a useful String() on Containers struct
fmt.Println(containers)
return nil
}
@shykes
Copy link

shykes commented Jul 31, 2014

Perhaps printContainerTable could be passed as an argument to NewDockerCli?

@shykes
Copy link

shykes commented Jul 31, 2014

Actually ignore my last comment. It's orthogonal.

@shykes
Copy link

shykes commented Jul 31, 2014

I like it.

Copy link

ghost commented Aug 15, 2014

Searching for docker run wasn't working and failing with obtuse error messages like:

[error] mount.go:11 [warning]: couldn't run auplink before unmount: exec: "auplink": executable file not found in $PATH

I found this gist. Better error reporting is a big +1 from me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment