robotstxt
From MaRDI portal
Software:100982
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Last update: 3 September 2020
Software version identifier: 0.7.13
Copyright license: MIT license, File License
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Last update: 3 September 2020
Software version identifier: 0.7.13
Copyright license: MIT license, File License