Skip to content

Instantly share code, notes, and snippets.

@Nullcaller
Last active September 1, 2024 01:26
Show Gist options
  • Save Nullcaller/ec00480aa8ef9cf20d0f8835eff52896 to your computer and use it in GitHub Desktop.
Save Nullcaller/ec00480aa8ef9cf20d0f8835eff52896 to your computer and use it in GitHub Desktop.
A RouterOS v7.8-compatible script for automatic generation of an address list containing all Facebook CDN addresses required for maintaining a normal connection from the DNS cache entries. Designed for decent stability and low CPU usage spikes. This is a combination of scripts by @jgrossiord and @ZloyXEP with some additions.
:delay delay-time=#REPLACE_THIS_WITH_SOME_ARBITRARY_RESULT_OF_HITTING_YOUR_NUMPAD_WITH_A_FIST#ms;
:foreach i in=[/ip dns cache find] do={
# Turn this on while turning off other delays for ease of manual use. Don't forget to reenable later.
# :delay delay-time=5ms;
:delay delay-time=50ms;
:local cacheName [/ip dns cache all get $i name];
:local cacheType [/ip dns cache all get $i type];
:if (($cacheType="A" or $cacheType="CNAME") and (($cacheName~"^([0-9A-Za-z\\-]*\\.)*facebook(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*fbcdn(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*fbstatic(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*fbexternal(\\.[0-9A-Za-z\\-]*)*\\.?\$"))) do={
# IP-based rule
:if ($cacheType="A") do={
:local cacheData [/ip dns cache all get $i data];
:local record [/ip firewall address-list find where (address=$cacheData) and (list=vpn_loopback_sites_dynamic)];
# One day I just randomly found 0.0.0.0 in the list. I'm not sure how it got there, so I introduced this safety measure.
:if (($cacheData~"^0\\.") or ($cacheData~"^10\\.") or ($cacheData~"^127\\.") or ($cacheData~"^192\\.168\\.")) do={
:put ("script tried putting $cacheData into the address list, was prevented");
} else={
:if ($record="") do={
:put ("add: $cacheName $cacheType $cacheData");
/ip firewall address-list add address=$cacheData comment=$cacheName timeout=14d list=vpn_loopback_sites_dynamic
} else={
:put ("renew: $cacheName $cacheType $cacheData");
/ip firewall address-list set $record timeout=14d;
}
}
}
# Domain-based rule
:if ($cacheType="A") do={
:local record [/ip firewall address-list find where (address=$cacheName) and (list=vpn_loopback_sites_dynamic)];
:if ($record="") do={
:put ("add: $cacheName $cacheType");
/ip firewall address-list add address=$cacheName timeout=14d list=vpn_loopback_sites_dynamic
} else={
:put ("renew: $cacheName $cacheType");
/ip firewall address-list set $record timeout=14d;
}
}
:if ($cacheType="CNAME") do={
:local cacheData [/ip dns cache all get $i data];
:local record [/ip firewall address-list find where (address=$cacheData) and (list=vpn_loopback_sites_dynamic)];
:if ($record="") do={
:put ("add: $cacheName $cacheType $cacheData");
/ip firewall address-list add address=$cacheData comment=$cacheName timeout=14d list=vpn_loopback_sites_dynamic
} else={
:put ("renew: $cacheName $cacheType $cacheData");
/ip firewall address-list set $record timeout=14d;
}
:local record [/ip firewall address-list find where (address=$cacheName) and (list=vpn_loopback_sites_dynamic)];
:if ($record="") do={
:put ("add: $cacheName $cacheType");
/ip firewall address-list add address=$cacheName timeout=14d list=vpn_loopback_sites_dynamic
} else={
:put ("renew: $cacheName $cacheType");
/ip firewall address-list set $record timeout=14d;
}
}
}
}
@Nullcaller
Copy link
Author

Nullcaller commented Mar 30, 2023

Original scripts can be found here.

Differences from original scripts:

  • Uses actual regexp for domain recognition, so that you don't accidentally ban half the internet by blocking t.co in such a way that it blocks microsof**t.co**m like Roskomnadzor did.
  • Generates domain names as well as IP addresses in the address list, so as to allow MikroTik to automatically resolve potential IP changes.
  • Designed for stability of the address list. After some initial easing-in, provided you give the script a somewhat decent supply of information, using Facebook (or whatever other service you use this script for) at least once or twice a week, at least for fifteen minutes or so, the services that depend on the list should never stop working.
  • Designed for low CPU usage spikes so as not to kill it with expansion and contraction of the silicon. By the way, don't schedule this script to run every five seconds. It won't be able to finish executing, and there's no reason to do so. Every five minutes is plenty, every ten should be enough.

Remember to insert some arbitrary number of milliseconds of delay at line 1 of the script, so that multiple instances of this script don't run simultaneously.

If the script runs too slowly or too quickly, adjust the delay value at line 3.

@Nullcaller
Copy link
Author

Nullcaller commented Mar 30, 2023

Here's line 6 for Twitter CDN:

:if (($cacheType="A" or $cacheType="CNAME") and (($cacheName~"^([0-9A-Za-z\\-]*\\.)*twitter(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*twimg(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*twttr(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*t\\.co.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*twtr(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*tweetdeck(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*vine(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*vineapp(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*mopub(\\.[0-9A-Za-z\\-]*)*\\.?\$"))) do={

(Yes, I know, Daddy Elon has a lot of servers. Big, juicy, muscular servers)

@Nullcaller
Copy link
Author

Here's line 6 for all of Meta's services:

:if (($cacheType="A" or $cacheType="CNAME") and (($cacheName~"^([0-9A-Za-z\\-]*\\.)*meta\\.com.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*facebook(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*fbcdn(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*fbstatic(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*fbexternal(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*instagram(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*cdninstagram(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*oculus(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*whatsapp(\\.[0-9A-Za-z\\-]*)*\\.?\$"))) do={

@Nullcaller
Copy link
Author

Nullcaller commented Mar 31, 2023

My current line 6 that is supposed to hopefully maybe with some luck and chance and if the stars align add Microsoft's CDN to the list, except for OneDrive, Windows Update and Office:

:if (($cacheType="A" or $cacheType="CNAME") and (($cacheName~"^([0-9A-Za-z\\-]*\\.)*microsoft(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*bing(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*msn(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*msftauth(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*virtualearth(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*bingtravel(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*msidentity(\\.[0-9A-Za-z\\-]*)*\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*[0-9A-Za-z\\-]*msn[0-9A-Za-z\\-]*\\.([0-9A-Za-z\\-]*\\.)*akamaized\\.net\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*[0-9A-Za-z\\-]*microsoft[0-9A-Za-z\\-]*\\.([0-9A-Za-z\\-]*\\.)*akamaized\\.net\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*live.com\\.?\$") or ($cacheName~"^([0-9A-Za-z\\-]*\\.)*takelessons.com\\.?\$")) and !(($cacheName~"storage") or ($cacheName~"1drv") or ($cacheName~"onedrive") or ($cacheName~"office") or ($cacheName~"sharepoint") or ($cacheName~"windows") or ($cacheName~"windowsupdate") or ($cacheName~"ntservicepack") or ($cacheName~"update\\.microsoft\\.com\\.?\$") or ($cacheName~"download\\.microsoft\\.com\\.?\$"))) do={

I don't know if it'll work correctly though. Currently testing it. Microsoft seems to have a very disorganized decentralized CDN compared to Facebook and Twitter. Facebook and Twitter settled neatly into a few domains, while Microsoft just can't get enough of them.

@klugev303
Copy link

Please help me to create a script to determine YouTube IP addresses using a script in Mikrotik (RouterOS7.15.3)
Doesn't work with the code below, thanks in advance!

:delay delay-time=246ms;
:foreach i in=[/ip dns cache find] do={
:delay delay-time=50ms;
:local cacheName [/ip dns cache all get $i name];
:local cacheType [/ip dns cache all get $i type];
:if (($cacheType="A" or $cacheType="CNAME") and (($cacheName~"^([0-9A-Za-z\-]\.)googlevideo(\.[0-9A-Za-z\-])\.?$") or ($cacheName~"^([0-9A-Za-z\-]\.)youtube(\.[0-9A-Za-z\-])\.?$") or ($cacheName~"^([0-9A-Za-z\-]\.)ytimg(\.[0-9A-Za-z\-])\.?$") or ($cacheName~"^([0-9A-Za-z\-]\.)youtu(\.[0-9A-Za-z\-])\.?$"))) do={
# IP-based rule
:if ($cacheType="A") do={
:local cacheData [/ip dns cache all get $i data];
:local record [/ip firewall address-list find where (address=$cacheData) and (list=Youtube_IP)];
# One day I just randomly found 0.0.0.0 in the list. I'm not sure how it got there, so I introduced this safety measure.
:if (($cacheData~"^0\.") or ($cacheData~"^10\.") or ($cacheData~"^127\.") or ($cacheData~"^192\.168\.")) do={
:put ("script tried putting $cacheData into the address list, was prevented");
} else={
:if ($record="") do={
:put ("add: $cacheName $cacheType $cacheData");
/ip firewall address-list add address=$cacheData comment=$cacheName timeout=14d list=Youtube_IP
} else={
:put ("renew: $cacheName $cacheType $cacheData");
/ip firewall address-list set $record timeout=14d;
}
}
}

	# Domain-based rule
	:if ($cacheType="A") do={
		:local record [/ip firewall address-list find where (address=$cacheName) and (list=Youtube_IP)];
		:if ($record="") do={
			:put ("add: $cacheName $cacheType");
			/ip firewall address-list add address=$cacheName timeout=14d list=Youtube_IP
		} else={
			:put ("renew: $cacheName $cacheType");
			/ip firewall address-list set $record timeout=14d;
		}
	}
	:if ($cacheType="CNAME") do={
		:local cacheData [/ip dns cache all get $i data];
		:local record [/ip firewall address-list find where (address=$cacheData) and (list=Youtube_IP)];
		:if ($record="") do={
			:put ("add: $cacheName $cacheType $cacheData");
			/ip firewall address-list add address=$cacheData comment=$cacheName timeout=14d list=Youtube_IP
		} else={
			:put ("renew: $cacheName $cacheType $cacheData");
			/ip firewall address-list set $record timeout=14d;
		}
		:local record [/ip firewall address-list find where (address=$cacheName) and (list=Youtube_IP)];
		:if ($record="") do={
			:put ("add: $cacheName $cacheType");
			/ip firewall address-list add address=$cacheName timeout=14d list=Youtube_IP
		} else={
			:put ("renew: $cacheName $cacheType");
			/ip firewall address-list set $record timeout=14d;
		}
	}
}

}

@Nullcaller
Copy link
Author

The idea is that you use this line of code:

:if (($cacheType="A" or $cacheType="CNAME") and (#matchers#)) do={

where #matchers# is replaced with a string that looks like

(#matcher_1#) or (#matcher_2#) or ... or (#matcher_n#)

where #matcher_i# for i=1..n is replaced with

$cacheName~"#regexp_i#"

and #regexp_i# is a properly-escaped regex pattern that the domain names you'd like to put into the list vpn_loopback_sites_dynamic should match. (well, one of the patterns)

Regex101 is a great resource to test your regular expressions, and Regex Learn is a great resource if you don't even know how to make them.

Otherwise, look at my code to understand how to properly escape them for MikroTik. Good luck!

@Nullcaller
Copy link
Author

I may or may not make a similar script for YouTube myself in the future, though.

@klugev303
Copy link

I may or may not make a similar script for YouTube myself in the future, though.

I'm afraid I won't figure it out myself(
I ask for help, in the current situation of YouTube blocking, I'm looking for the best solution to this problem through Mikrotik, Thanks in advance, I'll be waiting impatiently!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment