5 Commits

Author	SHA1	Message	Date
Dan Milne	88c7dc67f2	Add code and tests to handle a disallowed clause with value	2024-12-24 10:43:17 +11:00
Dan Milne	71bbd4d1ad	Support for mulitple sitemaps. Sitemaps should be absolute, so we'll try and make them absolute - but if we're passed the Robots.txt as text we can't determine the host.	2023-09-10 13:17:53 +10:00
Dan Milne	dd4a874f3f	Use lower case name	2023-09-10 10:36:23 +10:00
Dan Milne	27a47700e1	Gemspec adds the version, but tests also need it.	2023-09-10 10:31:46 +10:00
Dan Milne	5774ca2836	Switch to Net::HTTP and set the request header to match the query header	2023-09-10 10:02:52 +10:00