robotstxt

Authors Peter Meissner, Pedro Baltazar, Kun Ren

Last update 29 August 2024

Copyright license MIT license, File License

Software version identifier 0.7.13, PACKAGES.rds, 0.1.2, 0.3.2, 0.4.0, 0.4.1, 0.5.2, 0.6.0, 0.6.2, 0.7.4, 0.7.7, 0.7.8, 0.7.15

Description

Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.

Cited in

(2)

This page was built for software: robotstxt