PuchoPkBot
PuchoPkBot collects public pages and PDFs from official Pakistani government websites so that people can search them in one place. It always links back to the original page.
- It identifies itself as
PuchoPkBot/1.0 (+https://pucho.pk/government/about-crawler). - It reads
/robots.txtfirst and obeysDisallowandCrawl-delay. If robots.txt cannot be read, the site is not crawled. - It waits at least five seconds between requests to one website, and stops for the day if a site answers with “too many requests” or refuses access. It never uses logins, never solves CAPTCHAs and never works around a block.
- It only requests public pages and PDFs on the same website, and re-checks unchanged pages with conditional requests.
- Pages marked
noindexare not stored.
A government body that wants a site removed, slowed down or corrected can write to [email protected]. Removal takes effect immediately in the index.