Skip to content

Instantly share code, notes, and snippets.

@cvan
Last active September 3, 2024 01:43
Show Gist options
  • Save cvan/03ffa0c71317cb6b0b95a41ab189b097 to your computer and use it in GitHub Desktop.
Save cvan/03ffa0c71317cb6b0b95a41ab189b097 to your computer and use it in GitHub Desktop.
get all URLs of all network requests made on a web page from the Chrome Dev Tools (from an exported HAR)

Setup

Using Homebrew on Mac OS X:

brew install jq

Add these aliases to your profile (e.g., ~/.zshrc, ~/.bashrc, ~/.profile, etc.):

alias hurlp='pbpaste | jq ".log.entries" | tee >(jq --raw-output "[.[] | .request.url] | sort | unique | .[]")'
alias hurld='pbpaste | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" | harurls | tee >(xargs -n 1 curl -O $1)'

Usage

  1. Open the Chrome Developer Tools (Command+Option+i)
  2. Click the Network tab
  3. Reload the page (Command+r)
  4. In the Network tab, hover over any area in the table that is not a link (e.g., the Status column)
  5. Right click (Control+click) to open the context menu and click Copy All as HAR
  6. Open your favourite command-line interface (e.g., Terminal)
  7. Enter hurlp to print the URLs or hurld to download the files
@z333d
Copy link

z333d commented Feb 26, 2020

Here is my solution:

  1. Goto Network tab, right click to open context menu and click "Copy all as HAR"
    image
  2. Switch to Console tab, input a = , and click Command + v(or Control + v on Windows)
    image
  3. In Console tab, keep input copy(a.log.entries.map(e => e.request.url).join('\n'))
  4. Paste results to anywhere you want

@akvotil
Copy link

akvotil commented Oct 3, 2021

Here is my solution:
....
4. Paste results to anywhere you want

The result contains all urls from source page. How to get only urls from filtered url list in network tab?

@akvotil
Copy link

akvotil commented Oct 3, 2021

Anyway my HAR is too large to be processed.

Снимок экрана 2021-10-03 в 19 51 27

@akshay-kakade-dev
Copy link

thanks, @z333d for the solution.

@akshay-kakade-dev
Copy link

Anyway my HAR is too large to be processed.

Снимок экрана 2021-10-03 в 19 51 27

+1
Same problem for me @z333d Could you please provide any solution?

@smartexpert
Copy link

Copy and paste HAR to a text file.

Let's assume you call this file requests.json

Then run the following on your terminal:

cat requests.json | jq '.log.entries|.[].request.url'

This assumes you have jq installed on your system

@Sifasol
Copy link

Sifasol commented May 18, 2022

Thanks smartexpert, I followed your instructions and I have jq installed. It returned a lot of data all in green, but then where are my results now? I paste the clipboard and still have your cat entry there. Thanks again.

@jgrisham
Copy link

jgrisham commented Mar 7, 2024

@Sifasol - you probably need to append something like | pbcopy to the end of hurlp to place the results back on the clipboard; the commands above just display the results (for hurlp) or download their contents as individual files (for hurld).

For a more universal solution (that works on Linux, et. al.), try:

  1. Use the Network tab of Chrome/Edge/etc. Developer Tools as described above, except use the 'Save all as HAR with content' option to save it as a .har file.

  2. Run one of the following from the directory where you saved that file, replacing requests.json with the filename you used when saving that .har file (note: these were tested in the bash shell):
    a. View the results (same results as 'hurlp' above):

    • cat requests.json | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" | less

    b. Save the results to a text file:

    • cat requests.json | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" >> sorted-urls-from-requests_$(date +%Y-%m-%d_%H:%M).txt

@ixqbar
Copy link

ixqbar commented Sep 3, 2024

what's harurls ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment