robotstxt (Q100982)

From MaRDI portal
Revision as of 22:40, 15 October 2023 by Swh import (talk | contribs) (SWHID from Software Heritage)
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Language Label Description Also known as
English
robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

    Statements