Tags: scraper

RSS Anything

Next time you’re frustrated by a website that doesn’t provide an RSS feed, try using this tool:

Transform any old website with a list of links into an RSS Feed

AI Agents robots.txt Builder | Dark Visitors

A handy resource for keeping your blocklist up to date in your robots.txt file.

Though the name of the website is unfortunate with its racism-via-laziness nomenclature.

Robots.txt - Jim Nielsen’s Blog

I realized why I hadn’t yet added any rules to my robots.txt: I have zero faith in it.