summaryrefslogtreecommitdiff
BranchCommit messageAuthorAge
masteradd verbose search to robots.txtRené Wagner6 days
 
 
AgeCommit messageAuthor
6 daysadd verbose search to robots.txtHEADmasterRené Wagner
2021-02-12correctly handle robots.txtRené Wagner
2021-02-10limit max_crawl_depth to 100 for normal crawlRené Wagner
2021-02-10increase frequency to avoid rescanning within a single crawlRené Wagner
2021-02-07Add a few more url parsing test casesNatalie Pendragon
2021-02-07Update to Python 3.9 compatibilityNatalie Pendragon
2021-02-07update python depsRené Wagner
2021-02-07introduce systemd-unit for indexerRené Wagner
2021-01-30Make README heading lines more consistentNatalie Pendragon
2021-01-30add systemd-units for automatic crawlingRené Wagner
[...]