Consider using NON_INTERACTIVE_SHELL from context to detect event as coming from CI

[Martinsos]

In wasp-lang/wasp#1586 we added new "NON_INTERACTIVE_SHELL" value to our telemetry. We think it should directly correspond with if Wasp is running in CI or not, but we are not yet 100% sure, so we want to first see how it behaves in wild.

We should look at which events in PostHog have this value, and if it indeed corresponds to what I wrote above. If that behaves as it should, then we should start using it in our analytics here in wasp-bot to categorize all events that have this value in their context to be recognized as CI events!

[sodic]

Turns out this wasn't the case. Here's the rundown of recent discoveries.

NON_INTERACTIVE_SHELL does not detect what we wanted. We populate it based on the return value of SIO.hIsTerminalDevice SIO.stdout, which only checks whether the commands stdout is connected to the terminal.

This value doesn't say whether a Wasp command is run from an interactive shell. It only says whether the wasp command output was redirected to something other than the terminal's STDOUT. Examples when NON_INTERACTIVE_SHELL is true:

• Piped to another program: wasp start | less
• Redirected into a file: wasp start > log

It seems there's no easy way to detect what we actually need (whether wasp is run from a non-interactive environment). All libraries that do this just have a dictionary of env variables used to identify specific CI systems.

So, we should probably stop relying on NON_INTERACTIVE_SHELL when it comes to CI detection.

Interestingly enough, we had some weird activity heavily (but not fully) correlated with non-interactive shell. It might be a spurious correlation, we're not sure.

Some points to add. The main questions behind this weird activity are:

• How is one project ID connected to multiple different user IDs in a short time span?
• How is one project ID (or user ID) connected to multiple differnt IPs in a short time span?

More details in the discord thread: https://discord.com/channels/686873244791210014/835130205928030279/1347225711332692039