Menu
▾
▴
Tree [r713] / crawler / History
| File | Date | Author | Commit |
|---|---|---|---|
| README.txt | 2010-01-23 | nabber00 | [r557] added project description |
| config.ini | 2010-03-24 | nabber00 | [r620] added more mirrorbrain based websites, option t... |
| crawler.py | 2010-03-24 | nabber00 | [r619] added robots.txt support, more options, more fi... |
Read Me
This is designed to go out and grab metalink files from known metalink websites. This index can then be shared and indexed by Google for advanced searches.