Skip to content

Instantly share code, notes, and snippets.

View davins90's full-sized avatar

Daniele D'Avino davins90

View GitHub Profile
@davins90
davins90 / crisp_ml.md
Created September 8, 2024 14:21
CRISP-ML(Q) Framework for data science and ML project

Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology

  • Proposes CRISP-ML(Q), a process model for developing machine learning applications with quality assurance methodology
  • Extends CRISP-DM by adding:
    • Quality assurance methods to mitigate risks in each phase
    • A monitoring and maintenance phase post-deployment
    • Merging of business and data understanding phases
  • Provides best practices and guidelines for each phase of the ML process
  • Aims to increase success rate and efficiency of ML projects in industrial settings
  • Covers the entire ML development lifecycle, from defining objectives to maintenance

Reflections on AI

Below, I present some of my thoughts on artificial intelligence:

  1. AI technology is different from AI applications. It's the latter that should be regulated, not the former.

  2. AI is a means to reach a solution, not the solution itself. It's similar to electricity in this regard.

Docker Container for Sniffing Activities with Scapy in JupyterLab

This Docker setup provides a robust environment for network sniffing activities using Scapy within JupyterLab. Here's a quick overview of the key components:

Dockerfile

The Dockerfile builds upon the jupyter/datascience-notebook image, adding:

  • libpcap-dev for packet capture capabilities
  • Custom Python requirements (scapy)

Parallel API Calls

ThreadPoolExecutor

I recently used ThreadPoolExecutor for concurrent API calls. Here's a snippet from my code:

def enrich_dataframe(df, cidr_column='cidr_range', max_workers=8):
    total_cidrs = len(df[cidr_column])
 
@davins90
davins90 / github_pipeline.md
Last active December 11, 2023 09:42
github_pipeline.md

Create GitHub repo online, pull it locally, upload local code

  1. git init -b main
  2. git remote add origin URL
  3. git pull origin main
  4. git add .
  5. git commit -m "MESSAGE"
  6. git push origin main
@davins90
davins90 / docker_linux.md
Last active August 12, 2023 15:12
Docker images managed by Docker Compose with user permission (from root)

Setting Up JupyterLab in Docker on Ubuntu

This guide outlines the steps to build and run a JupyterLab container with specific directories and permissions, following a common structure needed for data science projects.

Prerequisites

  • Docker and Docker Compose installed on an Ubuntu system.
  • Your project should include a Dockerfile and docker-compose.yml file, as outlined earlier in the discussion.

Step-by-Step Guide

  1. Create the Source Directories: Create the src directory along with its subdirectories. These will be used as volumes in your Docker containers.
@davins90
davins90 / heroku_streamlit_app.md
Created January 8, 2021 20:45
How to create a webapp and deploy it - 28/8/2020

This is my tutorial on how to create and publish a webapp written in Python, interfaced with Streamlit and brought online with Heroku.

Introduction

Creating algorithms and applications has a precise purpose: to use them outside of your PC. Using them "locally" risks being a big limitation. Hence my need to find a way to bring small applications online, so that they can be used by more people, without having to go through me or my PC.

To do this you need 3 ingredients: GitHub, Streamlit and Heroku.

The process

GitHub

@davins90
davins90 / pelican_blog_pt2.md
Last active January 8, 2021 20:44
How to create a blog with Pelican? (Part 2) - 11/7/2020

Introduction

In the previous article (Part 1) we've seen how to build a website, now it's time to publish it! Let's start!

The publishing process

The publication process consists of two phases:

  1. the first online publication;

  2. the second, third, etc.. publication based on updates, new posts and so on.

@davins90
davins90 / pelican_blog_pt1.md
Last active January 8, 2021 20:42
How to create a blog with Pelican? (Part 1) - 28/6/2020

Introduction

With modern technological tools it is becoming increasingly easy to build your own web identity that is not only linked to the world of social networks. In this post we will see how to build a blog using Pelican, a way to build static websites based on Python, one of the most popular languages of the moment.

Building it will be the first step, making it operational will be the second, and for this goal we make use of the possibilities offered by GitHub, the most known code-repository that allows to host for each user its own web page.

The purpose? From creating an alternative Curriculum Vitae, to telling and spreading examples of work, projects or simple tutorials, like this one. Let's get started with the first step!

The building process

@davins90
davins90 / virtual_environment.md
Last active January 8, 2021 20:36
How to create a virtual environment on your PC - 01/06/2020

Why a virtual environment?

A virtual environment is useful in case you want to try particular libraries and extensions of a particular programming language (eg Python), without having to worry about creating conflicts with previous (or later) versions already installed on your machine.

For example: the XY library only works with version 3.0 of the Z tool, but you have version 4.0 installed on your PC. To avoid creating conflicts by uninstalling the current version, creating a virtual environment is the ideal solution!

The procedure

These steps can be performed comfortably via command prompts. Let's get started!

  1. First of all you need to install the package that allows the creation of a virtual environment. This is possible by running the following code: