robotstxt (Q100982)
From MaRDI portal
!
This is the item page for this Wikibase entity, intended for internal use and editing purposes. Please use the normal view instead:
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
| Language | Label | Description | Also known as |
|---|---|---|---|
| default for all languages | No label defined |
||
| English | robotstxt |
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker |
Statements
29 August 2024
0 references
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
0 references
