Hang fetching tree for www.govern.ad

[donbowman]

Site URL

https://www.govern.ad/

Description

The robots.txt suggests http:// for the sitemap, rather than htttps:
when fetching the root w/ http:, you get a 301 w/ Location of https:, but not for the sitemap
It hangs since http://www.govern.ad/sitemap-index.xml does not complete, and there appears to be no timeout (it retries it)

It then gets stuck fetching the sitemap with parameters that are a guid, over and over, forever like:

DEBUG:usp.helpers:Testing if URL 'https://www.govern.ad/sitemap.xml?p_l_id=3072&layoutUuid=6df02011-37ee-5914-c1ac-0320e3be7f3c&groupId=1898932&privateLayout=false' is HTTP(s) URL
INFO:usp.fetch_parse:Fetching level 1 sitemap from https://www.govern.ad/sitemap.xml?p_l_id=3072&layoutUuid=6df02011-37ee-5914-c1ac-0320e3be7f3c&groupId=1898932&privateLayout=false...
INFO:usp.helpers:Fetching URL https://www.govern.ad/sitemap.xml?p_l_id=3072&layoutUuid=6df02011-37ee-5914-c1ac-0320e3be7f3c&groupId=1898932&privateLayout=false...
DEBUG:urllib3.connectionpool:https://www.govern.ad:443 "GET /sitemap.xml?p_l_id=3072&layoutUuid=6df02011-37ee-5914-c1ac-0320e3be7f3c&groupId=1898932&privateLayout=false HTTP/1.1" 200 0
DEBUG:usp.fetch_parse:Response URL is https://www.govern.ad/sitemap.xml?p_l_id=3072&layoutUuid=6df02011-37ee-5914-c1ac-0320e3be7f3c&groupId=1898932&privateLayout=false
INFO:usp.fetch_parse:Parsing sitemap from URL https://www.govern.ad/sitemap.xml?p_l_id=3072&layoutUuid=6df02011-37ee-5914-c1ac-0320e3be7f3c&groupId=1898932&privateLayout=false...
DEBUG:usp.fetch_parse:Parent URLs is {'http://www.govern.ad/sitemap-index.xml', 'https://www.govern.ad/sitemap.xml'}
DEBUG:usp.helpers:Testing if URL 'https://www.govern.ad/sitemap.xml?p_l_id=3076&layoutUuid=eb0783e2-72fa-8783-df88-d57ba12196db&groupId=1898932&privateLayout=false' is HTTP(s) URL
INFO:usp.fetch_parse:Fetching level 1 sitemap from https://www.govern.ad/sitemap.xml?p_l_id=3076&layoutUuid=eb0783e2-72fa-8783-df88-d57ba12196db&groupId=1898932&privateLayout=false...
INFO:usp.helpers:Fetching URL https://www.govern.ad/sitemap.xml?p_l_id=3076&layoutUuid=eb0783e2-72fa-8783-df88-d57ba12196db&groupId=1898932&privateLayout=false...

there is a valid sitemap, at https://www.govern.ad/sitemap-index.xml, but the package doesn't find it.

See attached

output.txt

Environment

• Python version: Python 3.12.3
• USP version: 1.4.0

Log and Output Files

• Output log:
• Output text:

[freddyheppell]

Sorry it's taken a while to get back to you on this! It may have been this was just taking a very long time due to the way timeouts were set in the current release version. I should be able to release the fix for this today. done in 1.5.0

When I tried this on the current main version:

• it checked robots.txt and found that HTTP http://www.govern.ad/sitemap-index.xml URL
• It times out and retries (but the timeout for you would've been longer)
• It then moved onto trying known sitemap paths
• It tries https://www.govern.ad/sitemap.xml which contains around 2k sub-sitemaps with those weird parameters in the URL. All of these resolved and were blank
• It then tried https://www.govern.ad/sitemap-index.xml and discovered the actual sitemap tree

I'm not sure that USP can do anything differently here (other than the timeouts), because the actual sitemap structure is just a little strange. There's not really a way of knowing that the HTTP URL in robots.txt is incorrect (equally the opposite could've been true, that it only resolves over HTTP not HTTPS), or that the dynamic URLs in sitemap.xml are all empty and seemingly pointless.