Skip to content

Instantly share code, notes, and snippets.

@ikenna

ikenna/extract-endpoints.js

Created June 5, 2023 13:02
Show Gist options
  • Save ikenna/d7834f652f14b93e9e3df7698a486b52 to your computer and use it in GitHub Desktop.
Save ikenna/d7834f652f14b93e9e3df7698a486b52 to your computer and use it in GitHub Desktop.
Download ZIP
Extract endpoints from OpenAPI YAML file
Raw
extract-endpoints.js
const fs = require('fs');
const yaml = require('js-yaml');
function listEndpoints(openapiFile) {
try {
const openapiData = yaml.safeLoad(fs.readFileSync(openapiFile, 'utf8'));
const paths = openapiData.paths || {};
const endpoints = [];
for (const path in paths) {
const methods = paths[path];
for (const method in methods) {
const endpoint = `${method.toUpperCase()} ${path}`;
endpoints.push(endpoint);
}
}
return endpoints;
} catch (error) {
console.error(`Error loading OpenAPI file: ${error}`);
return [];
}
}
// Usage example
const openapiFile = 'your-openapi.yaml';
const endpoints = listEndpoints(openapiFile);
for (const endpoint of endpoints) {
console.log(endpoint);
}
@ikenna
Copy link
Author

ikenna commented Jun 5, 2023

To use this script, replace 'your-openapi.yaml' with the path to your actual OpenAPI YAML file. The script uses the fs module to read the file, js-yaml to parse the YAML data, and then iterates over the paths section to extract the endpoints in the format HTTP_METHOD PATH. Finally, it prints each endpoint.

Make sure you have the js-yaml package installed before running the script. You can install it using npm install js-yaml.

@polarathene
Copy link

polarathene commented Sep 15, 2024 •
edited
Loading

You can use the yq CLI tool as another way which can be simpler in this case (you only need to change the yaml filename):

# Take the array items of paths then output each path key (an endpoint) with an array of it's keys (methods):
yq '.paths[] | {key: keys}' your-openapi.yaml

For example on the Docker Engine API:

# ...
/containers/{id}:
  - delete
/containers/{id}/archive:
  - head
  - get
  - put
/containers/prune:
  - post
/images/json:
  - get
/build:
  - post
# ...

If you need another output format like JSON you can use -o json 👍

The above may not be as friendly if you need an object with standard keys to query.

A more complex example that outputs an array of objects with an additional id field that better identifies the endpoint:

# Entire yq expression wrapped with `[` + `]` for a top-level array
# Each paths element creates an object with `endpoint` + `methods` keys
# Each `methods` key has an array of objects for the method kind and associated id
# Added example of uppercase conversion.
# NOTE: `|` is the pipe operator, similar to `map()` in JS. I used it more extensively here and with nested contexts.
# `.` refers to the data piped in, `.[]` is for selecting all direct array elements, `key` refers to the current key (within the scope of the pipe).
yq '[.paths[] | {"endpoint": key, "methods": [.[] | {"kind": key | upcase, "id": .operationId}] }]' api.yaml

Output example:

# ...
- endpoint: /secrets
  methods:
    - kind: GET
      id: "SecretList"
- endpoint: /secrets/create
  methods:
    - kind: POST
      id: "SecretCreate"
- endpoint: /secrets/{id}
  methods:
    - kind: GET
      id: "SecretInspect"
    - kind: DELETE
      id: "SecretDelete"
- endpoint: /secrets/{id}/update
  methods:
    - kind: POST
      id: "SecretUpdate"
# ...
With `yq -o=json` for JSON output 
[
// ...
  {
    "endpoint": "/secrets",
    "methods": [
      {
        "kind": "GET",
        "id": "SecretList"
      }
    ]
  },
  {
    "endpoint": "/secrets/create",
    "methods": [
      {
        "kind": "POST",
        "id": "SecretCreate"
      }
    ]
  },
  {
    "endpoint": "/secrets/{id}",
    "methods": [
      {
        "kind": "GET",
        "id": "SecretInspect"
      },
      {
        "kind": "DELETE",
        "id": "SecretDelete"
      }
    ]
  },
  {
    "endpoint": "/secrets/{id}/update",
    "methods": [
      {
        "kind": "POST",
        "id": "SecretUpdate"
      }
    ]
  },
// ...
]

yq starts to be less pleasant the more complex the expression gets, especially if you need more flexibility there the approach with JS is easier to grok :)

@polarathene
Copy link

polarathene commented Sep 17, 2024 •
edited
Loading

This might be of interest for anyone landing here too.

yq + qsv + csview - Extract to CSV + markdown table output

Using the same Docker OpenAPI document (direct YAML link) as before, here is a way to extract the desired data with yq and do some extra processing with qsv to format it:

# yq will filter and format the data into an array of values we want from the input
# The comments within the string value are safe to copy/paste as a shell command.
# yq outputs CSV for qsv to then adds the column headers and sort by column order.
# csview then renders this out in the preferred style such as formatted markdown.
yq -o=csv '.paths[] | .[] | [
  .tags | join(" "), # An array, can be used to assign the endpoint to a group
  (parent | key),    # The URL path for this endpoint
  key | upcase,      # The HTTP method uppercased `get => GET`
  .operationId       # Unique identifier for the endpoint
]' api.yaml \
| qsv rename --no-headers GROUP,PATH,METHOD,ID \
| qsv sort \
| csview --style markdown

This will output a table like this:

Markdown output (rendered)
GROUP PATH METHOD ID
Config /configs GET ConfigList
Config /configs/create POST ConfigCreate
Config /configs/{id} DELETE ConfigDelete
Config /configs/{id} GET ConfigInspect
Config /configs/{id}/update POST ConfigUpdate
Container /containers/create POST ContainerCreate
Container /containers/json GET ContainerList
Container /containers/prune POST ContainerPrune
Container /containers/{id} DELETE ContainerDelete
Container /containers/{id}/archive GET ContainerArchive
Container /containers/{id}/archive HEAD ContainerArchiveInfo
Container /containers/{id}/archive PUT PutContainerArchive
Container /containers/{id}/attach POST ContainerAttach
Container /containers/{id}/attach/ws GET ContainerAttachWebsocket
Container /containers/{id}/changes GET ContainerChanges
Container /containers/{id}/export GET ContainerExport
Container /containers/{id}/json GET ContainerInspect
Container /containers/{id}/kill POST ContainerKill
Container /containers/{id}/logs GET ContainerLogs
Container /containers/{id}/pause POST ContainerPause
Container /containers/{id}/rename POST ContainerRename
Container /containers/{id}/resize POST ContainerResize
Container /containers/{id}/restart POST ContainerRestart
Container /containers/{id}/start POST ContainerStart
Container /containers/{id}/stats GET ContainerStats
Container /containers/{id}/stop POST ContainerStop
Container /containers/{id}/top GET ContainerTop
Container /containers/{id}/unpause POST ContainerUnpause
Container /containers/{id}/update POST ContainerUpdate
Container /containers/{id}/wait POST ContainerWait
Distribution /distribution/{name}/json GET DistributionInspect
Exec /containers/{id}/exec POST ContainerExec
Exec /exec/{id}/json GET ExecInspect
Exec /exec/{id}/resize POST ExecResize
Exec /exec/{id}/start POST ExecStart
Image /build POST ImageBuild
Image /build/prune POST BuildPrune
Image /commit POST ImageCommit
Image /images/create POST ImageCreate
Image /images/get GET ImageGetAll
Image /images/json GET ImageList
Image /images/load POST ImageLoad
Image /images/prune POST ImagePrune
Image /images/search GET ImageSearch
Image /images/{name} DELETE ImageDelete
Image /images/{name}/get GET ImageGet
Image /images/{name}/history GET ImageHistory
Image /images/{name}/json GET ImageInspect
Image /images/{name}/push POST ImagePush
Image /images/{name}/tag POST ImageTag
Network /networks GET NetworkList
Network /networks/create POST NetworkCreate
Network /networks/prune POST NetworkPrune
Network /networks/{id} DELETE NetworkDelete
Network /networks/{id} GET NetworkInspect
Network /networks/{id}/connect POST NetworkConnect
Network /networks/{id}/disconnect POST NetworkDisconnect
Node /nodes GET NodeList
Node /nodes/{id} DELETE NodeDelete
Node /nodes/{id} GET NodeInspect
Node /nodes/{id}/update POST NodeUpdate
Plugin /plugins GET PluginList
Plugin /plugins/create POST PluginCreate
Plugin /plugins/privileges GET GetPluginPrivileges
Plugin /plugins/pull POST PluginPull
Plugin /plugins/{name} DELETE PluginDelete
Plugin /plugins/{name}/disable POST PluginDisable
Plugin /plugins/{name}/enable POST PluginEnable
Plugin /plugins/{name}/json GET PluginInspect
Plugin /plugins/{name}/push POST PluginPush
Plugin /plugins/{name}/set POST PluginSet
Plugin /plugins/{name}/upgrade POST PluginUpgrade
Secret /secrets GET SecretList
Secret /secrets/create POST SecretCreate
Secret /secrets/{id} DELETE SecretDelete
Secret /secrets/{id} GET SecretInspect
Secret /secrets/{id}/update POST SecretUpdate
Service /services GET ServiceList
Service /services/create POST ServiceCreate
Service /services/{id} DELETE ServiceDelete
Service /services/{id} GET ServiceInspect
Service /services/{id}/logs GET ServiceLogs
Service /services/{id}/update POST ServiceUpdate
Session /session POST Session
Swarm /swarm GET SwarmInspect
Swarm /swarm/init POST SwarmInit
Swarm /swarm/join POST SwarmJoin
Swarm /swarm/leave POST SwarmLeave
Swarm /swarm/unlock POST SwarmUnlock
Swarm /swarm/unlockkey GET SwarmUnlockkey
Swarm /swarm/update POST SwarmUpdate
System /_ping GET SystemPing
System /_ping HEAD SystemPingHead
System /auth POST SystemAuth
System /events GET SystemEvents
System /info GET SystemInfo
System /system/df GET SystemDataUsage
System /version GET SystemVersion
Task /tasks GET TaskList
Task /tasks/{id} GET TaskInspect
Task /tasks/{id}/logs GET TaskLogs
Volume /volumes GET VolumeList
Volume /volumes/create POST VolumeCreate
Volume /volumes/prune POST VolumePrune
Volume /volumes/{name} DELETE VolumeDelete
Volume /volumes/{name} GET VolumeInspect
Volume /volumes/{name} PUT VolumeUpdate
Markdown output (raw) 
|    GROUP     |            PATH            | METHOD |            ID            |
|--------------|----------------------------|--------|--------------------------|
| Config       | /configs                   | GET    | ConfigList               |
| Config       | /configs/create            | POST    | ConfigCreate             |
| Config       | /configs/{id}              | DELETE  | ConfigDelete             |
| Config       | /configs/{id}              | GET     | ConfigInspect            |
| Config       | /configs/{id}/update       | POST    | ConfigUpdate             |
| Container    | /containers/create         | POST    | ContainerCreate          |
| Container    | /containers/json           | GET     | ContainerList            |
| Container    | /containers/prune          | POST    | ContainerPrune           |
| Container    | /containers/{id}           | DELETE  | ContainerDelete          |
| Container    | /containers/{id}/archive   | GET     | ContainerArchive         |
| Container    | /containers/{id}/archive   | HEAD    | ContainerArchiveInfo     |
| Container    | /containers/{id}/archive   | PUT     | PutContainerArchive      |
| Container    | /containers/{id}/attach    | POST    | ContainerAttach          |
| Container    | /containers/{id}/attach/ws | GET     | ContainerAttachWebsocket |
| Container    | /containers/{id}/changes   | GET     | ContainerChanges         |
| Container    | /containers/{id}/export    | GET     | ContainerExport          |
| Container    | /containers/{id}/json      | GET     | ContainerInspect         |
| Container    | /containers/{id}/kill      | POST    | ContainerKill            |
| Container    | /containers/{id}/logs      | GET     | ContainerLogs            |
| Container    | /containers/{id}/pause     | POST    | ContainerPause           |
| Container    | /containers/{id}/rename    | POST    | ContainerRename          |
| Container    | /containers/{id}/resize    | POST    | ContainerResize          |
| Container    | /containers/{id}/restart   | POST    | ContainerRestart         |
| Container    | /containers/{id}/start     | POST    | ContainerStart           |
| Container    | /containers/{id}/stats     | GET     | ContainerStats           |
| Container    | /containers/{id}/stop      | POST    | ContainerStop            |
| Container    | /containers/{id}/top       | GET     | ContainerTop             |
| Container    | /containers/{id}/unpause   | POST    | ContainerUnpause         |
| Container    | /containers/{id}/update    | POST    | ContainerUpdate          |
| Container    | /containers/{id}/wait      | POST    | ContainerWait            |
| Distribution | /distribution/{name}/json  | GET     | DistributionInspect      |
| Exec         | /containers/{id}/exec      | POST    | ContainerExec            |
| Exec         | /exec/{id}/json            | GET     | ExecInspect              |
| Exec         | /exec/{id}/resize          | POST    | ExecResize               |
| Exec         | /exec/{id}/start           | POST    | ExecStart                |
| Image        | /build                     | POST    | ImageBuild               |
| Image        | /build/prune               | POST    | BuildPrune               |
| Image        | /commit                    | POST    | ImageCommit              |
| Image        | /images/create             | POST    | ImageCreate              |
| Image        | /images/get                | GET     | ImageGetAll              |
| Image        | /images/json               | GET     | ImageList                |
| Image        | /images/load               | POST    | ImageLoad                |
| Image        | /images/prune              | POST    | ImagePrune               |
| Image        | /images/search             | GET     | ImageSearch              |
| Image        | /images/{name}             | DELETE  | ImageDelete              |
| Image        | /images/{name}/get         | GET     | ImageGet                 |
| Image        | /images/{name}/history     | GET     | ImageHistory             |
| Image        | /images/{name}/json        | GET     | ImageInspect             |
| Image        | /images/{name}/push        | POST    | ImagePush                |
| Image        | /images/{name}/tag         | POST    | ImageTag                 |
| Network      | /networks                  | GET     | NetworkList              |
| Network      | /networks/create           | POST    | NetworkCreate            |
| Network      | /networks/prune            | POST    | NetworkPrune             |
| Network      | /networks/{id}             | DELETE  | NetworkDelete            |
| Network      | /networks/{id}             | GET     | NetworkInspect           |
| Network      | /networks/{id}/connect     | POST    | NetworkConnect           |
| Network      | /networks/{id}/disconnect  | POST    | NetworkDisconnect        |
| Node         | /nodes                     | GET     | NodeList                 |
| Node         | /nodes/{id}                | DELETE  | NodeDelete               |
| Node         | /nodes/{id}                | GET     | NodeInspect              |
| Node         | /nodes/{id}/update         | POST    | NodeUpdate               |
| Plugin       | /plugins                   | GET     | PluginList               |
| Plugin       | /plugins/create            | POST    | PluginCreate             |
| Plugin       | /plugins/privileges        | GET     | GetPluginPrivileges      |
| Plugin       | /plugins/pull              | POST    | PluginPull               |
| Plugin       | /plugins/{name}            | DELETE  | PluginDelete             |
| Plugin       | /plugins/{name}/disable    | POST    | PluginDisable            |
| Plugin       | /plugins/{name}/enable     | POST    | PluginEnable             |
| Plugin       | /plugins/{name}/json       | GET     | PluginInspect            |
| Plugin       | /plugins/{name}/push       | POST    | PluginPush               |
| Plugin       | /plugins/{name}/set        | POST    | PluginSet                |
| Plugin       | /plugins/{name}/upgrade    | POST    | PluginUpgrade            |
| Secret       | /secrets                   | GET     | SecretList               |
| Secret       | /secrets/create            | POST    | SecretCreate             |
| Secret       | /secrets/{id}              | DELETE  | SecretDelete             |
| Secret       | /secrets/{id}              | GET     | SecretInspect            |
| Secret       | /secrets/{id}/update       | POST    | SecretUpdate             |
| Service      | /services                  | GET     | ServiceList              |
| Service      | /services/create           | POST    | ServiceCreate            |
| Service      | /services/{id}             | DELETE  | ServiceDelete            |
| Service      | /services/{id}             | GET     | ServiceInspect           |
| Service      | /services/{id}/logs        | GET     | ServiceLogs              |
| Service      | /services/{id}/update      | POST    | ServiceUpdate            |
| Session      | /session                   | POST    | Session                  |
| Swarm        | /swarm                     | GET     | SwarmInspect             |
| Swarm        | /swarm/init                | POST    | SwarmInit                |
| Swarm        | /swarm/join                | POST    | SwarmJoin                |
| Swarm        | /swarm/leave               | POST    | SwarmLeave               |
| Swarm        | /swarm/unlock              | POST    | SwarmUnlock              |
| Swarm        | /swarm/unlockkey           | GET     | SwarmUnlockkey           |
| Swarm        | /swarm/update              | POST    | SwarmUpdate              |
| System       | /_ping                     | GET     | SystemPing               |
| System       | /_ping                     | HEAD    | SystemPingHead           |
| System       | /auth                      | POST    | SystemAuth               |
| System       | /events                    | GET     | SystemEvents             |
| System       | /info                      | GET     | SystemInfo               |
| System       | /system/df                 | GET     | SystemDataUsage          |
| System       | /version                   | GET     | SystemVersion            |
| Task         | /tasks                     | GET     | TaskList                 |
| Task         | /tasks/{id}                | GET     | TaskInspect              |
| Task         | /tasks/{id}/logs           | GET     | TaskLogs                 |
| Volume       | /volumes                   | GET     | VolumeList               |
| Volume       | /volumes/create            | POST    | VolumeCreate             |
| Volume       | /volumes/prune             | POST    | VolumePrune              |
| Volume       | /volumes/{name}            | DELETE  | VolumeDelete             |
| Volume       | /volumes/{name}            | GET     | VolumeInspect            |
| Volume       | /volumes/{name}            | PUT     | VolumeUpdate             |

With qsv you can rearrange the columns, omit columns, etc quite easily.

Advanced processing with QSV

This example that transforms that same CSV output from yq to a filtered subset rearranged and sorted to different requirements:

# Same yq operation as before, but now qsv will filter rows of the GROUP column
# Then for any PATH values that have a URL parameter substring, replace it with `*`
# The METHOD and PATH columns are extracted via `select` with new column positions
# Instead of sorting by the 1st colum (METHOD), sort by 2nd colum (PATH)
# Finally `table` will output an unstyled table of the data, tail to remove headers
yq -o=csv '.paths[] | .[] | [
  .tags | join(" "), # An array, can be used to assign the endpoint to a group
  (parent | key),    # The URL path for this endpoint
  key | upcase,      # The HTTP method uppercased `get => GET`
  .operationId       # Unique identifier for the endpoint
]' api.yaml \
| qsv rename --no-headers GROUP,PATH,METHOD,ID \
| qsv search --ignore-case --literal --select GROUP secret \
| qsv replace --quiet --select PATH '\{.+\}' '*' \
| qsv select METHOD,PATH \
| qsv sort --select PATH \
| qsv table | tail -n +2

# Result:
GET     /secrets
GET     /secrets/*
DELETE  /secrets/*
POST    /secrets/*/update
POST    /secrets/create

That output is more aligned with the original JS example (extra processing aside).

Swapping qsv table | tail -n +2 for csview --style markdown we get the same data formatted for rendering:

METHOD PATH
GET /secrets
GET /secrets/*
DELETE /secrets/*
POST /secrets/*/update
POST /secrets/create

JS to yq port

The actual minimal equivalent for the JS example output would be:

# Output as space delimited strings:
yq '.paths[][] | (key|upcase) + " " + (parent|key)' api.yaml

# Result:
GET /secrets
POST /secrets/create
GET /secrets/{id}
DELETE /secrets/{id}
POST /secrets/{id}/update
# With QSV for better column aligned output, uses arrays with CSV output:
yq -o=csv '.paths[] | [.[] | [
  (key|upcase), # The HTTP method uppercased `get => GET`
  (parent|key)  # The URL path for this endpoint
]]' api.yaml | qsv table

# Result:
GET     /secrets
POST    /secrets/create
GET     /secrets/{id}
DELETE  /secrets/{id}
POST    /secrets/{id}/update

NOTE: Result outputs were filtered with grep secrets for brevity, unlike the prior advanced example which actually filtered results by specific column via qsv search.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment