New feature: validator API service

[themightychris]

Describe the problem

For a recent demo, I needed to be able to run the the validator as a service and execute validations through synchronous API calls.

I initially tried to just add a /validate endpoint to do the job to the existing spring server under web/service but that quickly became a rabbit hole because I wanted to be able to run this without all the Google infrastructure for storing data and doing pub/sub and be able to build it entirely within a container and found that project to be very far from being set up for that. I ended up just building a bare-bones spring server under /api that only exposed the /validate endpoint

Proposed solution

If I end up building this thing for real, I'll want to move away from working out of a fork so there won't be any barrier to the API moving to new validator versions. I'm interested in the community/maintainer opinions on the best route:

1. Contribution to MobilityData/gtfs-validator: refactor web/service so that it can build entirely in a container and so that the GCP services are an optional dependency for the job-oriented endpoints, and add the /validate endpoint to it
2. Contribution to MobilityData/gtfs-validator: add something like the separate minimal API service I built somewhere else in the tree
3. Just build a separate repo and publish it either under my own org or contribute it to the MobilityData org the defines the API service and consumes the validator JAR

Alternatives you've considered

No response

Additional context

No response

[davidgamez]

Hi @themightychris,

Thanks for opening this and for sharing your approach. We’ve been planning along similar lines as part of #1911, since the current web/service project is tightly coupled to the major validator releases. That coupling slows down enhancements and makes it less flexible for demos or lightweight deployments.

1. Contribution to MobilityData/gtfs-validator: refactor web/service so that it can build entirely in a container and so that the GCP services are an optional dependency for the job-oriented endpoints, and add the /validate endpoint to it

I agree with you here. The API layer should be flexible and not tied to any specific vendor. Storage should be configurable, with local storage as the default when nothing else is set. Making GCP integrations optional would make the service usable in both demo and production environments.

2. Contribution to MobilityData/gtfs-validator: add something like the separate minimal API service I built somewhere else in the tree

In order to keep this repository clean, I would prefer to have an API service as an independent repository(option 3 of your proposal).

3. Just build a separate repo and publish it either under my own org or contribute it to the MobilityData org the defines the API service and consumes the validator JAR

This aligns with the original intent in #1911. A dedicated repository would make it easier to iterate on the API layer at its own pace, while the API code would only depend on the published Maven artifacts from the validator. As part of this, we’d also like to take the opportunity to refactor the existing Swagger API
so that current users can migrate smoothly to the new endpoints if they choose to.

So, our preference is option 3: creating a dedicated repository for the API service. We’d like to have this project under the MobilityData organization so that it’s easier to maintain, stays aligned with validator releases, and remains open to community contributions.

[themightychris]

The part I'm interested in wouldn't require coupling with any cloud vendor or need storage. I just want to be able to run a container and then throw GTFS ZIPs at an endpoint and get the validation report back in the response whenever it's done. I think that outside MobilityData's use case of running a global public validator service, other users wouldn't have need for the queue and storage features so I'm not sure that what I'm describing here has any hard need to be coupled with what #1911 is describing.

It wouldn't be crazy to do them both in the same app/repo—the connection to cloud queue+storage services could be optional environment vars that flip on those endpoints. But I don't see the business case for investing in supporting multiple cloud vendors there because no one else would want to run it the same way MD runs the public one and need to engineer for unpredictable global loads

Any mature cloud container runner service provides all the capabilities anyone with simpler use cases would need to autoscale based on long-running HTTP requests per job