[go: up one dir, main page]

File: robots.txt

package info (click to toggle)
linkchecker 4.9-1
  • links: PTS
  • area: main
  • in suites: lenny
  • size: 3,308 kB
  • ctags: 3,736
  • sloc: python: 21,328; lex: 1,114; yacc: 781; ansic: 551; makefile: 244; sh: 100; sql: 19; awk: 4
file content (12 lines) | stat: -rw-r--r-- 312 bytes parent folder | download | duplicates (2) 
1
2
3
4
5
6
7
8
9
10
11
12
 
# A test robots.txt file. It resides in the root directory to be found
# by the unittest HTTP server.
# Note that while unit tests are running, the contents of this file
# can change.

User-agent: LinkChecker
Crawl-delay: 1
Disallow: /secret

# The following rule should not match
User-agent: *
Crawl-delay: 100