[go: up one dir, main page]
Menu ▾ ▴

Commit [r619]  Maximize  Restore  History

added robots.txt support, more options, more file extensions to ignore

nabber00 2010-03-24

Browse code at this revision

Parent: [r618]

Child: [r620]

changed /crawler/crawler.py
/crawler/crawler.py Diff Switch to side-by-side view
Loading...