JS.ORG | high traffic for `/phishing/**/*.json`

[indus]

Hi Polkadot.js team,

As the founder of JS.ORG, which has provided polkadot.js.org with its subdomain since 2017, I wanted to share some observations regarding the traffic of polkadot.js.org based on Cloudflare analytics.

Observations

• Your subdomain receives over 6 million requests per day, accounting for about 15% of all requests across JS.ORG. Roughly 150 GB of data is transferred daily, which represents approximately 30% of the total data transferred across all JS.ORG-provided subdomains (over 3500).

• Over 90% of the traffic to your subdomain comes from your blacklisting JSONs served under the various /phishing paths.

• Cloudflare’s cache status is consistently dynamic, indicating that these requests are not currently being served from Cloudflare’s edge cache but forwarded directly to the origin (GitHub Pages).

Questions

I’m unsure whether Cloudflare might have concerns if this request volume continues to grow in the future, especially since JS.ORG is using a free plan with extended quotas. Do you see this level of traffic and the current lack of caching as a potential issue for origin load or performance?

If so, do you see any options or strategies to reduce this load?

While caching could probably be implemented at the Cloudflare level to help reduce origin requests, it’s understood this may not align with your need to always serve the most up-to-date phishing data.

Let me know what you think of if or if you need more information (e.g. screenshots of the statistics)

Best,
Stefan

[TarikGul]

@indus Thanks so much for the thorough issue.

Over 90% of the traffic to your subdomain comes from your blacklisting JSONs served under the various /phishing paths.

I am going to look into this specifically and see what various solutions we can put together. We'll use this issue as the source of truth :)