cli: Add ability to toggle whether tasks get sent to builders or not

This could be useful for builder test and maintenance purposes. I imagine it could work so that no _new_ jobs get pushed to builders after the current jobs are finished (or failed).