Protego is a pure-Python robots.txt parser with support for modern conventions.
It is mainly used by Scrapy application when crawling websites.