Skip to content

Instantly share code, notes, and snippets.

@andrewheiss

andrewheiss/get_distances.R

Last active August 12, 2019 16:31
Show Gist options
  • Save andrewheiss/867036aeac7a589e6818 to your computer and use it in GitHub Desktop.
Save andrewheiss/867036aeac7a589e6818 to your computer and use it in GitHub Desktop.
Download ZIP
Calculate driving distance with Mapzen/Valhalla
Raw
get_distances.R
# DISTANCE CALCULATING MADNESS
#
# Mapzen has a bunch of super useful (and free and open source!) APIs for
# working with maps, including geoencoding addresses and names of locations
# and getting directions between multiple points. Their turn-by-turn service
# is named Valhalla.
#
# You can access the Mapzen/Valhalla API using a URL with a bunch of parameters,
# basically structured like this:
#
# https://valhalla.mapzen.com/route?json=blah&api_key=blah
#
# The JSON variable is for JSON-encoded data, while the API key is a string
# that allows you to access the API.
#
# IMPORTANT:
# Get an API key for Valhalla/Turn by Turn at https://mapzen.com/developers
#
# JSON is a structured data format that most web services use for transferring
# data behind the scenes. At minimum, your JSON data for Valhalla should look
# like this:
#
# {
# "locations": [
# {
# "lat": xxx,
# "lon": xxx
# },
# {
# "lat": yyy,
# "lon": yyy
# }
# ],
# "directions_options": {
# "units": "miles"
# }
# }
#
# The first lat/lon pair is the origin; the second is the destination. If you
# include others in the "locations" list, it'll route you through those points.
#
# You can include a ton of other options in the reqest too, all documented here:
# https://github.com/valhalla/valhalla-docs/blob/master/api-reference.md
#
# A typical request looks like this:
# {
# "locations": [
# {
# "lat": 42.358528,
# "lon": -83.271400
# },
# {
# "lat": 42.996613,
# "lon": -78.749855
# }
# ],
# "costing": "auto",
# "costing_options": {
# "auto": {
# "country_crossing_penalty": 2000.0
# }
# },
# "directions_options": {
# "units": "miles"
# }
# }
#
# JSON doesn't need the line breaks---those just make it nice to read. That
# same data looks like this when turned into a URL:
#
# https://valhalla.mapzen.com/route?json={"locations":[{"lat":-14.9718,"lon":28.1588},{"lat":-15.4289,"lon":28.3277}],"costing":"auto","costing_options":{"auto":{"country_crossing_penalty":2000}},"directions_options":{"units":"miles"}}&api_key=ADD_API_KEY_HERE
#
# Paste that into a browser after adding your own API key and you'll see the results
# from the API, which are also encoded as JSON. Your browser will show ugly results.
# Here's what they look like prettified (TONS OF LINES OF CODE: SCROLL DOWN REALLY FAR):
#
# {
# "trip": {
# "status": 0,
# "status_message": "Found route between points",
# "legs": [
# {
# "shape": "vgdr[avjtt@dhHgsHjsAwcBzqA{qBltA}hCziBg`Er{@sdCbo@qzCp~@crGpc@amDlX{sC`oF_gp@bTqrCtTujCnJsx@tJmc@rWew@hUkg@nYwc@b_@af@ps@eq@~k@}^vg@mUno@iThi@gLjh@qH|va@u~DzuJcaA`mD}a@zb@iK|s@wW`kGggDhpBkb@`tByEhrBzVhaB|e@blEr{AxqC~t@nhBnIhs[rm@fcCqJb}Boo@`lWisOnzBmmAf{A}j@zdc@ubNlo@{OnmFy`AroC}s@nmf@qxQvoBe^jr@yCjrFfLxiBXx{BoEngFoa@xjDyk@`fQ{bEdoGcnAtnQ}sD~gMo|CrdBiWfoD_Ot{F~DlfFoBdlCcXpmh@soI|zc@wdHb}B}^h}AuWvqHcjA`hEkr@tbNgzBzkp@erK|tG}eAp}Jw`BhfBcS`cB}JttM}^ljAcDxJYxe@sA~_EgL~p@mBdhE}LzpDeK`oBwFpv@yBxf@sAfUq@tc@wAvIYjp@mBvM[pMe@di@sAxs@uBrmBoFvmIuTrEKjs@iBr@Cb{@}BpuDeK~cC}Gpg@sAbg@}A~Na@dTSza@a@`~@p@rfDvFl`@r@tdAnB|z@nAjtDxG~tDxGho@zAh|EdJzmCpElrAzAtiArAbN[jh@gApR_B|YcD|u@qObk@kPn\\uOxtAks@rCuAdEsBhQkIp]wKdg@gNxg@_LnWqEro@mIzt@mH`b@aEz}@{IdfAiJnj@mElk@aCheAoDfqAmEjhAnBjn@Obm@l@fp@Opi@?zu@|@nh@rAxjAbIvDPvu@lHvD\\nwBfQb~B|RpyBjSzgAoDz|AmMzmA{KlsAwMl~@{IfU{Ezg@yPeA{ELeFpAqEnCmDfEkBd@CtEi@pFrBdDdEbVsGrcJ}z@d~@_I`WkDcDe_@oRo_CmCwX{Hi`AaK}hAuAgNsIa|@mDqn@g@_`@bFqlDrDw~A|FyxB~AePbWkj@jO{ZlCwFhh@u~@xl@ieA~n@gjA|m@qgA|o@}hAiTibA_CkGeFaEeHqE}DyBoC{DkA{EPmH`AgDzB_DbDqBdd@}q@jAkCrCsDnEyBpFm@rFp@pEdCh\\ud@rbA_vAlOkQ{CqHr@wHlOiTpP}UhF{CzFGrDnApHkHhJ}M|Y{_@nVe_@t]ii@`Vs^|DaHvDqJzNuu@~Wc}@v]qjAle@_|Aj}AshFx`@gqAfP{g@plAe}DzCcKpWsz@e|A}f@",
# "summary": {
# "length": 36.832,
# "time": 2772
# },
# "maneuvers": [
# {
# "begin_shape_index": 0,
# "length": 22.862,
# "time": 0,
# "type": 3,
# "end_shape_index": 72,
# "instruction": "Go southeast on T2.",
# "verbal_pre_transition_instruction": "Go southeast on T2 for 22.9 miles.",
# "street_names": [
# "T2"
# ]
# },
# {
# "begin_shape_index": 72,
# "street_names": [
# "Great North Road"
# ],
# "time": 760,
# "type": 8,
# "end_shape_index": 166,
# "instruction": "Continue on Great North Road.",
# "length": 9.878,
# "verbal_transition_alert_instruction": "Continue on Great North Road.",
# "begin_street_names": [
# "T2",
# "Great North Road"
# ],
# "verbal_pre_transition_instruction": "Continue on Great North Road for 9.9 miles."
# },
# {
# "roundabout_exit_count": 2,
# "begin_shape_index": 166,
# "street_names": [
# "Northend Roundabout"
# ],
# "time": 14,
# "type": 26,
# "end_shape_index": 175,
# "instruction": "Enter the roundabout and take the 2nd exit.",
# "length": 0.066,
# "verbal_transition_alert_instruction": "Enter roundabout.",
# "verbal_pre_transition_instruction": "Enter the roundabout and take the 2nd exit."
# },
# {
# "begin_shape_index": 175,
# "length": 0.525,
# "time": 51,
# "type": 27,
# "end_shape_index": 179,
# "instruction": "Exit the roundabout.",
# "verbal_post_transition_instruction": "Continue for a half mile.",
# "verbal_pre_transition_instruction": "Exit the roundabout."
# },
# {
# "begin_shape_index": 179,
# "street_names": [
# "Church Road"
# ],
# "verbal_post_transition_instruction": "Continue for 200 feet.",
# "time": 12,
# "type": 15,
# "end_shape_index": 180,
# "instruction": "Turn left onto Church Road.",
# "length": 0.035,
# "verbal_transition_alert_instruction": "Turn left onto Church Road.",
# "verbal_pre_transition_instruction": "Turn left onto Church Road."
# },
# {
# "begin_shape_index": 180,
# "time": 13,
# "type": 8,
# "end_shape_index": 181,
# "instruction": "Continue.",
# "length": 0.139,
# "verbal_transition_alert_instruction": "Continue.",
# "verbal_pre_transition_instruction": "Continue for 1 tenth of a mile."
# },
# {
# "begin_shape_index": 181,
# "street_names": [
# "Church Road"
# ],
# "time": 163,
# "type": 8,
# "end_shape_index": 200,
# "instruction": "Continue on Church Road.",
# "length": 1.341,
# "verbal_transition_alert_instruction": "Continue on Church Road.",
# "verbal_pre_transition_instruction": "Continue on Church Road for 1.3 miles."
# },
# {
# "begin_shape_index": 200,
# "street_names": [
# "Independence Avenue"
# ],
# "verbal_post_transition_instruction": "Continue for 400 feet.",
# "time": 16,
# "type": 15,
# "end_shape_index": 201,
# "instruction": "Turn left onto Independence Avenue.",
# "length": 0.076,
# "verbal_transition_alert_instruction": "Turn left onto Independence Avenue.",
# "verbal_pre_transition_instruction": "Turn left onto Independence Avenue."
# },
# {
# "begin_shape_index": 201,
# "time": 9,
# "type": 17,
# "end_shape_index": 204,
# "instruction": "Stay straight to take the ramp.",
# "length": 0.032,
# "verbal_transition_alert_instruction": "Stay straight to take the ramp.",
# "verbal_pre_transition_instruction": "Stay straight to take the ramp."
# },
# {
# "begin_shape_index": 204,
# "time": 5,
# "type": 8,
# "end_shape_index": 205,
# "instruction": "Continue.",
# "length": 0.007,
# "verbal_transition_alert_instruction": "Continue.",
# "verbal_pre_transition_instruction": "Continue for 40 feet."
# },
# {
# "begin_shape_index": 205,
# "verbal_post_transition_instruction": "Continue for 200 feet.",
# "time": 3,
# "type": 9,
# "end_shape_index": 209,
# "instruction": "Bear right.",
# "length": 0.032,
# "verbal_transition_alert_instruction": "Bear right.",
# "verbal_pre_transition_instruction": "Bear right."
# },
# {
# "begin_shape_index": 209,
# "verbal_post_transition_instruction": "Continue for 1 tenth of a mile.",
# "time": 29,
# "type": 9,
# "end_shape_index": 218,
# "instruction": "Bear right.",
# "length": 0.129,
# "verbal_transition_alert_instruction": "Bear right.",
# "verbal_pre_transition_instruction": "Bear right."
# },
# {
# "begin_shape_index": 218,
# "street_names": [
# "Independence Avenue"
# ],
# "verbal_post_transition_instruction": "Continue for 2 tenths of a mile.",
# "time": 21,
# "type": 15,
# "end_shape_index": 221,
# "instruction": "Turn left onto Independence Avenue.",
# "length": 0.198,
# "verbal_transition_alert_instruction": "Turn left onto Independence Avenue.",
# "verbal_pre_transition_instruction": "Turn left onto Independence Avenue."
# },
# {
# "begin_shape_index": 221,
# "verbal_post_transition_instruction": "Continue for 1 tenth of a mile.",
# "time": 20,
# "type": 16,
# "end_shape_index": 228,
# "instruction": "Bear left.",
# "length": 0.108,
# "verbal_transition_alert_instruction": "Bear left.",
# "verbal_pre_transition_instruction": "Bear left."
# },
# {
# "begin_shape_index": 228,
# "verbal_post_transition_instruction": "Continue for 2 tenths of a mile.",
# "time": 39,
# "type": 15,
# "end_shape_index": 236,
# "instruction": "Turn left.",
# "length": 0.248,
# "verbal_transition_alert_instruction": "Turn left.",
# "verbal_pre_transition_instruction": "Turn left."
# },
# {
# "begin_shape_index": 236,
# "verbal_post_transition_instruction": "Continue for 1 mile.",
# "time": 120,
# "type": 16,
# "end_shape_index": 246,
# "instruction": "Bear left.",
# "length": 1.045,
# "verbal_transition_alert_instruction": "Bear left.",
# "verbal_pre_transition_instruction": "Bear left."
# },
# {
# "begin_shape_index": 246,
# "verbal_post_transition_instruction": "Continue for 1 tenth of a mile.",
# "time": 26,
# "type": 15,
# "end_shape_index": 247,
# "instruction": "Turn left.",
# "length": 0.112,
# "verbal_transition_alert_instruction": "Turn left.",
# "verbal_pre_transition_instruction": "Turn left."
# },
# {
# "begin_shape_index": 247,
# "time": 0,
# "type": 4,
# "end_shape_index": 247,
# "instruction": "You have arrived at your destination.",
# "length": 0.000,
# "verbal_transition_alert_instruction": "You will arrive at your destination.",
# "verbal_pre_transition_instruction": "You have arrived at your destination."
# }
# ]
# }
# ],
# "units": "miles",
# "summary": {
# "length": 36.832,
# "time": 2772
# },
# "locations": [
# {
# "side_of_street": "left",
# "lon": 28.158800,
# "lat": -14.971800,
# "type": "break"
# },
# {
# "lon": 28.327700,
# "lat": -15.428900,
# "type": "break"
# }
# ]
# }
# }
#
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!S!T!O!P!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!STOP!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!S!T!O!P!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# STOP SCROLLING!!!
# STOP
# STOP
# STOP
#
# The API returns complete turn-by-turn directions in case you want to build
# your own GPS or whatever. Super cool. What we care about the most, though is
# the summary section, or the part that looks like this:
#
# "summary": {
# "length": 36.832,
# "time": 2772
# },
#
# That shows that it takes 2,772 seconds to drive the 36.832 miles from some
# random point near Lusaka to the Zambia State House (where the president lives).
#
# Magic.
#
# So, here's an example of how to automate all of this with R. Run this script
# line-by-line in RStudio to see how it all works.
#
# You need to install two really useful packages that let you (1) convert and
# encode data as JSON (jsonlite), and (2) send that JSON data as an HTTP
# request (httr). # Install the dplyr package too, since it makes working with
# dataframes really easy
install.packages(c("dplyr", "httr", "jsonlite"))
# Load libraries
library(dplyr)
library(httr)
library(jsonlite)
# The jsonlite library includes the toJSON function which converts a list
# variable into JSON, like so:
x <- list(thing1 = "Testing", thing2 = c("Thing 1", "Thing 2"))
toJSON(x, auto_unbox=TRUE) # Basic single-line JSON
toJSON(x, auto_unbox=TRUE, pretty=TRUE) # Pretty JSON
# We can automate this for data with a bunch of observations. First make some
# fake data:
data_full <- data.frame(id = 1:3,
lat = c(-14.971822, -11.176554, -15.292965),
lon = c(28.158848, 28.968746, 23.189938),
other_var = c(1, 5, 9))
data_full
# Make a varible of the lat/lon for the end point (in this case the
# Zambian State House)
lusaka <- data.frame(lat = -15.428902, lon = 28.327689)
# This function takes the lat/long values from a single row of the dataset and
# creates a full JSON-encoded request
make_json <- function(i, df, endpoint) {
# Combine the ith row of the dataframe with the given endpoint, resulting in
# a small 2x2 dataframe
start_to_end <- rbind(df %>% select(lat, lon) %>% slice(i),
endpoint)
# A list of all the JSON parameters
request_full <- list(locations = start_to_end,
costing = "auto",
costing_options = list(
auto = list(
country_crossing_penalty = 2000)),
directions_options = list(
units = "miles"))
# Convert list to actual JSON and return
return(toJSON(request_full, auto_unbox = TRUE))
}
# You can fun that function on just one row of your data:
make_json(3, data_full, lusaka)
# Prettified result:
# {
# "locations": [
# {
# "lat": -15.293,
# "lon": 23.1899
# },
# {
# "lat": -15.4289,
# "lon": 28.3277
# }
# ],
# "costing": "auto",
# "costing_options": {
# "auto": {
# "country_crossing_penalty": 2000
# }
# },
# "directions_options": {
# "units": "miles"
# }
# }
# You can apply that function to every row in the dataframe using sapply():
sapply(1:nrow(data_full), FUN=make_json, df=data_full, endpoint=lusaka)
# For fun, you can add a new column to the dataset with the full JSON
data_full$json <- sapply(1:nrow(data_full), FUN=make_json,
df=data_full, endpoint=lusaka)
View(data_full)
# With the data structured as JSON, all you need to do is send that data to the
# API with the HTTP request. The GET() function from the httr package does this
# for you---it generates a URL given a bunch of parameters
base_url <- "https://valhalla.mapzen.com/route"
api_key <- "API_KEY_HERE" # GET THIS AT https://mapzen.com/developers
# For fun, manually make the URL and paste it in a browser:
cat(base_url, "?json=", slice(data_full, 1)$json, "&api_key=", api_key, sep="")
# It's better to automate that, though. So...
# Select just the first row
first_row <- slice(data_full, 1)
# Make the HTTP GET request
request <- GET(base_url, query = list(json = first_row$json, api_key = api_key))
request # It returned a bunch of JSON and HTTP codes
# Read just the JSON content
json <- content(request, as="text")
json # Lots of text
# Convert that text into something R can use
results <- fromJSON(json)
results
# Everything seems to be stored in the "trip" level, so save just that to a variable
trip <- results$trip
trip
# You can access any of the results in the trip object like so:
trip$status_message
trip$summary$length
trip$summary$time
# Super magic.
# You can get the distance between Lusaka and each row of the dataset pretty
# easily. First, wrap all the request stuff inside a function:
make_request <- function(id, json_data) {
# Wait for a couple seconds, since Mapzen only allows 2 queries per second
Sys.sleep(2)
# Make the HTTP GET request
request <- GET(base_url, query = list(json = json_data, api_key = api_key))
json <- content(request, as="text") # Read just the JSON content
results <- fromJSON(json)$trip # Convert that text into something R can use
return(list(id = id,
num_seconds = results$summary$time,
distance = results$summary$length))
}
# This function takes the row id and the full JSON-encoded data and returns
# the id, number of seconds, and distance
first_row <- slice(data_full, 1)
make_request(first_row$id, first_row$json)
# You can then apply the function to each row in the dataset
raw_results <- data_full %>%
rowwise() %>%
do(results = make_request(.$id, .$json))
# That yields a bunch of lists. Convert them into a dataframe
final_results <- bind_rows(raw_results$results)
View(final_results) # Voila!
# Merge those distances and times back into the actual data
final_data <- data_full %>%
left_join(final_results, by="id") %>%
select(-json)
View(final_data)
# And you're done!
# This process isn't the most efficient way to do this, but it's hopefully the
# most didactic.
#
# In real life you'd probably want to combine the make_json()
# and make_request() functions or add error checking or do other things to
# clean everything up. But this works.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment