Skip to content

Instantly share code, notes, and snippets.

View bennyistanto's full-sized avatar

Benny Istanto bennyistanto

View GitHub Profile
@bennyistanto
bennyistanto / geest_ntl.md
Last active August 27, 2024 13:15
Test NTL data process for GEEST

PyQGIS NTL Clipping, Classification, and Visualization

Screenshot 2024-08-27 194434

Overview

This script is designed to run within the PyQGIS console and provides a complete workflow for processing Night Time Lights (NTL) data. The script performs the following tasks:

  1. Clipping - Clips the global NTL raster data to the boundary of a specified country or region.
  2. Classification - Classifies the clipped NTL data into 6 classes using the GEEST Safety Classification standard.
@bennyistanto
bennyistanto / geest_color.md
Last active August 27, 2024 23:24
Colorblind safe for GEEST classification

Colorblind safe for GEEST classification

geest_normal

The existing color scheme in the image uses a predominantly orange-red palette for most of the map, with some yellows, greens, and blues appearing minimally. While this scheme does show variation, it has several limitations. The heavy use of similar orange and red hues makes it difficult to distinguish between different levels of enablement and population, especially in the low to moderate ranges. Additionally, this color scheme may not be easily interpretable for individuals with color vision deficiencies.

I propose changing to a more effective color scheme based on the ColorBrewer RdYlBu (Red-Yellow-Blue) 5-class diverging color palette.

Baseline color 5-class RdYlBu

@bennyistanto
bennyistanto / Enhanced_NTL_class_for_GEEST.md
Last active August 24, 2024 07:46
Enhanced Night Time Light (NTL) Classification for GEEST

Enhanced Night Time Light (NTL) Classification for GEEST

Approach and Rationale

We propose refining the NTL classification in GEEST to better reflect local conditions, particularly for small island countries. Instead of using fixed thresholds or coverage percentages, we'll employ local statistics to create a more contextually relevant classification. This approach is supported by Elvidge et al. (2013), who emphasize the importance of considering local context in night-time light analysis.

Standard Classification Procedure:

  1. Clip the global NTL raster to the country boundary.
  2. Calculate statistics for the clipped raster: min, max, mean, median, and 75th percentile.
@bennyistanto
bennyistanto / 0_rainfall_distribution.md
Last active August 21, 2024 11:22
Rainfall distribution analysis and visualisation

Rainfall Distribution Analysis

This rainfall distribution analysis script provides a comprehensive examination of observed and satellite-based daily precipitation data, offering valuable insights for hydrologists, climatologists, and water resource managers. The analysis begins by generating synthetic rainfall data to simulate both ground observations and satellite estimates, including the introduction of extreme events. This approach allows for a controlled comparison between different data sources and methods of analysis.

The script employs a range of advanced statistical techniques to characterize precipitation patterns and extreme events. It utilizes the method of L-moments to fit Gamma distributions to the overall rainfall patterns, providing a robust representation of the general precipitation behavior. For extreme rainfall events, the script applies Generalized Pareto Distributions (GPD) and incorporates cross-validation techniques to ensure reliable modeling of these critical occurrences. This dua

@bennyistanto
bennyistanto / moving_window.md
Created July 2, 2024 07:18
The moving window approach is used to balance the advantages of capturing both spatial and temporal variations in precipitation patterns

Moving Window illustration

You can paste the below code into an online Python compiler like https://python-fiddle.com/ and grab the result instantly.

This script will generate a plot that illustrates:

  1. This highlighting the spatial moving window on the top layer. The selected pixel at (4,5) will be red, and the 24 surrounding pixels within the 5x5 window will be pink.
  2. This illustrate how the temporal moving window considers data from multiple layers (time steps) when smoothing the value for a single pixel.

The moving window approach is used to balance the advantages of capturing both spatial and temporal variations in precipitation patterns. By considering both spatial and temporal windows, the bias correction method can address local variations and seasonal changes more effectively.

LSEQM Bias Correction

Bias correction of satellite-based precipitation estimates is crucial in climate studies to enhance their accuracy and reliability. This exercise explores a method that combines Linear Scaling (LS) and Empirical Quantile Mapping (EQM) with tail adjustment using the Generalized Pareto Distribution (GPD). This hybrid approach aims to correct both the mean bias and the distributional discrepancies between satellite estimates and ground-based observations.

lseqm_xkcd_style_b

The following sections will describe the methodology, the role of different functions, their strengths, and weaknesses in achieving accurate bias correction.

1 Linear Scaling Approach

@bennyistanto
bennyistanto / flood_probability_model.md
Created June 25, 2024 05:58
Visual representation of flood probability model, illustrating the different scenarios

Visual representation of flood probability model

You can paste the below code into an online Python compiler like https://python-fiddle.com/ and grab the result instantly.

This script will generate a plot that illustrates:

  1. The original model (blue line).
  2. A scenario with a steeper slope (red line), showing a more rapid increase in flood risk with rainfall.
  3. A scenario with a gentler slope (green line), showing a more gradual increase in risk.
  4. A scenario with a less negative intercept (purple line), resulting in higher probabilities at lower rainfall amounts.
@bennyistanto
bennyistanto / gev_threshold.py
Created June 23, 2024 04:17
Simple Generalized Extreme Value (GEV) from set of data
# Copy and paste the code below into https://python-fiddle.com/
# You will get instant result: GEV 25-year threshold: 96.12570045491432
import numpy as np
from scipy.stats import genextreme
# This data is the maximum daily precipitation in a month (let say May) for the period
# 2001-2023
data = [
35.809994, 55.120003, 97.575005, 17.01, 43.260006, 50.285004, 51.489998, 59.05999,
58.339996, 82.835, 42.875, 55.574997, 105.149994, 61.155, 83.68001, 73.56,
@bennyistanto
bennyistanto / precip_pettitt.md
Last active June 12, 2024 13:46
Simulate how Pettitt test detects inhomogeneous or homogeneous on precipitation data

xkcd-style Pettitt's illustration on Homogeneity testing

The script below try to simulate how Pettitt test detects inhomogeneous or homogeneous on precipitation data. Here's the updated approach:

  1. Daily Precipitation Data:

    • We have daily precipitation.
  2. Generate Monthly Maximum Precipitation Values:

  • Create synthetic monthly maxima with distinct differences to ensure inhomogeneity.
@bennyistanto
bennyistanto / imerg_thresholdbymonth.py
Created June 8, 2024 07:04
Calculates the thresholds for extreme rainfall events using both percentile-based and Generalized Extreme Value (GEV) methods
import xarray as xr
import numpy as np
import os
import glob
import pandas as pd
from tqdm import tqdm
from scipy.stats import genextreme
# Define directories and filenames for threshold calculations
threshold_calculations = [