robotstxt (Q100982)
From MaRDI portal
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
| Language | Label | Description | Also known as |
|---|---|---|---|
| English | robotstxt |
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker |
Statements
3 September 2020
0 references
