Skip to content

Instantly share code, notes, and snippets.

View taiwotman's full-sized avatar
🎯
Focusing

Taiwo O. Adetiloye taiwotman

🎯
Focusing
View GitHub Profile
@taiwotman
taiwotman / gist:8b11f34cccfd824676c070c032a5afc6
Last active May 11, 2024 23:57
Communicate complex concepts in a clear
Describe a complex data modeling challenge you faced in a previous project. How did you approach the problem, and what factors influenced your decisions regarding the data model?
A complex data modelling challenge I faced was designing a data model for a multinational e-commerce company that wanted to analyze customer behaviour across multiple countries.
The complexity in this scenario arises from the need to handle diverse data sources, different types of data (structured, semi-structured, and unstructured), and the requirement to support multi-language data. Additionally, I was required to implement the data model to be scalable to handle the increasing volume of data as the company grows.
My first approach was understanding the business requirements and the type of analysis that the company wants to perform. Second, I explored the available data sources to understand the data’s structure, quality, and content. Third, based on the requirements and the nature of the data, as a data engineer, I chose a hy
docker compose up run
$ docker build -t <image-name> .
@taiwotman
taiwotman / Dockerfile
Created October 5, 2022 19:46
Enabling Microservices on Postgres using Docker and Docker Compose.
FROM python:latest
WORKDIR app
COPY . /app
RUN python3 -m pip install -r requirements.txt
psycopg2==2.9.3
pytest== 7.1.2
version: "3.9"
services:
postgres:
image: postgres:10.5
restart: always
environment:
- POSTGRES_USER=<username>
- POSTGRES_PASSWORD=<password>
ports:
- '5433:5432'
@taiwotman
taiwotman / docker commands
Last active October 6, 2022 01:49
gist_docker-build
## Key Docker Commands
docker build --pull --no-cache --tag=<image-name> .
docker-compose up
docker-compose down
@taiwotman
taiwotman / capacity-optimization-heuristics-approach-for-the-bus-loading-problem.ipynb
Last active January 13, 2022 14:27
Capacity Optimization Heuristic for the Bus Loading Problem
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
#Transformation
df = df.withColumn("latitude", col("latitude").cast(DoubleType()))\
.withColumn("longitude", col("longitude").cast(DoubleType()))
df.printSchema()
df.show(10)
#More Insight
from pyspark.sql import SparkSession
from pyspark.sql.types import DoubleType
from pyspark.sql.functions import col
jar_path = 'rds_jar_driver.jar'
spark = SparkSession \
.builder \
.appName("AWS REDSHIFT PYSPARK APP") \
.config("spark.jars", jar_path)\
.config('spark.driver.extraClassPath', jar_path) \