3 commits

Author	SHA1	Message	Date
Infinity Search	6cbc3bb463	Fixed possible bug for sites_to_crawl.txt files that ended with a newline	2021-03-14 14:25:05 -05:00
Infinity Search	d348ed2f62	Created a robot parser that supports * and $	2021-02-11 11:22:51 -06:00
Infinity Search	cc3e10b1b1	Added everything to remote	2021-02-05 13:53:28 -06:00