Currently, each file discovered in the .tekton directory requires a REST API request to pull the contents. In order to prevent hitting rate limits, repositories are forced to have fewer large files rather than splitting them into many smaller files.
The GraphQL APIs for both GitLab and GitHub would allow PaC to pull batches of file contents at once. GitLab roughly 80 files, and I couldn't identify a limit with GitHub. Since the GitLab GraphQL API (and I think the GitHub API) is available unauthenticated, this should be a seamless change.