BranchCommit messageAuthorAge
masteradd verbose search to robots.txtRené Wagner11 months
AgeCommit messageAuthor
2021-02-21add verbose search to robots.txtHEADmasterRené Wagner
2021-02-12correctly handle robots.txtRené Wagner
2021-02-10limit max_crawl_depth to 100 for normal crawlRené Wagner
2021-02-10increase frequency to avoid rescanning within a single crawlRené Wagner
2021-02-07Add a few more url parsing test casesNatalie Pendragon
2021-02-07Update to Python 3.9 compatibilityNatalie Pendragon
2021-02-07update python depsRené Wagner
2021-02-07introduce systemd-unit for indexerRené Wagner
2021-01-30Make README heading lines more consistentNatalie Pendragon
2021-01-30add systemd-units for automatic crawlingRené Wagner