Tags: scraper

3

sparkline

Sunday, January 7th, 2024

RSS Anything

Next time you’re frustrated by a website that doesn’t provide an RSS feed, try using this tool:

Transform any old website with a list of links into an RSS Feed

Saturday, December 30th, 2023

AI Agents robots.txt Builder | Dark Visitors

A handy resource for keeping your blocklist up to date in your robots.txt file.

Though the name of the website is unfortunate with its racism-via-laziness nomenclature.

Saturday, September 30th, 2023

Robots.txt - Jim Nielsen’s Blog

I realized why I hadn’t yet added any rules to my robots.txt: I have zero faith in it.