Hugh Saalmans minus34
hakib
/ beers.py
Last active
September 10, 2024 19:47
Source for article at https://hakibenita.com/fast-load-data-python-postgresql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # https://hakibenita.com/fast-load-data-python-postgresql | |
| from typing import Iterator, Dict, Any, Optional | |
| from urllib.parse import urlencode | |
| import datetime | |
| #------------------------ Profile | |
| import time |
gwangjinkim
/ install-and-start-postgresql-in-conda-locally
Last active
September 18, 2024 15:05
How to install and run postgresql in conda
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| This gist I write, because I couldn't find step by step instructions | |
| how to install and start postgresql locally (using conda within a conda environment - with the result | |
| that you can run it without sudo/admin rights on your machine!) | |
| and not globally in the operating system (which requires sudo/admin rights on that machine). | |
| I hope, this will help especially people new to postgresql (and those who don't have sudo/admin rights on a specific machine but want | |
| to run postgresql there)! | |
| #################################### | |
| # create conda environment |
pavlov99
/ haversine.scala
Created
December 19, 2016 07:52
Spherical distance calcualtion based on latitude and longitude with Apache Spark
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // Based on following links: | |
| // http://andrew.hedges.name/experiments/haversine/ | |
| // http://www.movable-type.co.uk/scripts/latlong.html | |
| df | |
| .withColumn("a", pow(sin(toRadians($"destination_latitude" - $"origin_latitude") / 2), 2) + cos(toRadians($"origin_latitude")) * cos(toRadians($"destination_latitude")) * pow(sin(toRadians($"destination_longitude" - $"origin_longitude") / 2), 2)) | |
| .withColumn("distance", atan2(sqrt($"a"), sqrt(-$"a" + 1)) * 2 * 6371) | |
| >>> | |
| +--------------+-------------------+-------------+----------------+---------------+----------------+--------------------+---------------------+--------------------+------------------+ | |
| |origin_airport|destination_airport| origin_city|destination_city|origin_latitude|origin_longitude|destination_latitude|destination_longitude| a| distance| |