Skip to content

Instantly share code, notes, and snippets.

@serenamm

serenamm/create_table.py

Last active September 9, 2019 18:41
Show Gist options
  • Save serenamm/b2ec6fb24253229e65e9e0ff27537a66 to your computer and use it in GitHub Desktop.
Save serenamm/b2ec6fb24253229e65e9e0ff27537a66 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
create_table.py
def create_new_table(spark, table_paths, params, same_category_q):
similarity_table = table_paths["product_similarity"]["table"]
created_table = spark.sql(create_table_query.format(similarity_table=similarity_table,
same_category_q=same_category_q,
num_items=params["num_items"]))
# Write table to some path
created_table.coalesce(1).write.save(table_paths["created_table"]["path"],
format="orc", mode="Overwrite")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment