Mocked time

[scf37]

Add delay and now functions. Add test runtime with emulated clock to test code with delays without spending actual time on delays.

Would be invaluable for testing asynchronous code, not sure is this even possible with current Loom design but might be worth a try.

There is similar feature in cats-effect, separate effect evaluator with emulated time: https://typelevel.org/cats-effect/docs/core/test-runtime#mocking-time

[adamw]

I agree, that would be a great feature to have. One problem that I'm currently not sure is possible to solve right now is detecting when time can be "pushed forward". This should be done only when all threads of execution have suspended (either on delay or some condition - e.g. reading from a blocking queue). In cats or ZIO, that's possible as you are in control of the runtime, so you know when all of your fibers are suspended, and none can be further executed unless the time progresses. With Loom, you'd have to somehow inspect the state of all currently running virtual threads. I don't think there's an API for that currently, and even it if where, there would also be the problem of scoping the checks: we only really want to wait until the virtual threads created by some parent scope are suspended, not necessarily all in the whole JVM.

[adamw]

A project with similar goals (though no longer developed / maintained) is https://github.com/devexperts/time-test, but probably doesn't work with Loom

[nob13]

IMHO this can only work if getting current time, and waiting for somehting comes together, e.g.

class Clock {
  // Returns simulated time
  def currentTime: Instant
  // Wait for specific amount of time
  def wait(duration: FiniteDuration): Unit
  // Wait until a time is met.
  def waitUntil(i: Instant): Unit 
}

In my opinion, it shouldn't be necessary to track all other threads suspensions state, as long as the clock is monotonically increasing, because other threads can't make more assumptions. And if something crashes because the time advanced, well, that's exactly what a test should figure out.

On the other hand, all (testable) access to the time/wait must then use this clock, maybe in some scoped value?

[adamw]

Yes, we can make this available via a scoped value, but what would the wait method do then? It wouldn't know if it's fine to advance the clock (& "pretend" that e.g. 10 seconds has passed), as other threads might be runnable and have something to do in the "current" time. So such a test would behave very differently from one ran using wall-clock time.

[scf37]

Idea is to write tests assuming execution of non-blocking code takes zero time. Which is true in most practical cases involving wait(duration) calls

[nob13]

I think it depends on the test scope. But if wait immediately returns and stores advanced time, it should work for some reasons. For "smaller tests" this could already be useful, e.g.

class RetryPolicy {
  def retry3Times(f: => T): T = { .. }
}

// The test
val retryPolicy = RetryPolicy (/* Configure*/)

intercept[CouldNotRetryAnymore]{
  retryPolicy.retry3Times { throw RuntimeException() }
}

clock.waitedTime: FiniteDuration shouldBe 3 * retryPolicy.retryTime

[adamw]

Ah, good to share some examples :) Something that I had in mind:

val i = AtomicInteger(0)
fork {
  i.getAndAdd(1)
}

sleep(10.seconds)
assert(i.get() == 1)

In (almost ;) ) every "normal" execution that test would pass. But if sleeps don't sleep, it would fail.

[scf37]

Isn't that good rather than bad? :-)
This is data race 'solved' by sleep call instead of joining forked thread. Both linearizations are valid and test one just uncovers illegal synchronization. Therefore my point is: emulated sleeps do not introduce new linearizations but shift their probabilities.

[adamw]

Yes, you're right in principle, but then ... what are the usual cases for having sleeps in your code in the first place?

One non-artificial example that comes to my mind is sending pings over a web socket:

fork {
  forever {
    ws.send(Ping())
    sleep(1.second)
  }
}

ws.send(msgsToSendChannel.receive())

with the sleep-doesn't-sleep, the test might end up sending an infinite stream of pings, and no messages altogether. Or vice versa.

[nob13]

I see two alternatives in your example

• We wait a minimal time (like 5ms), so that a parallel thread can check exit-assumptions, but that wouldn't be very good unit tests as we need statistical results
• The clock has some upper boundery for simulated time, e.g and we stop the children thread if this time is over. Then no real waiting has to be done.

val clock = SimualtedClock(simulatedTime = 1.minute)
clock.run {
  // Adams Example
}
pingReceived.count shouldBe 60

In the end it looks like, multiple clock implementations can be discussed, depending on the way of testing methods.

[scf37]

Cases I've encountered (and written tests for!) are:

• retries
• rate control (call remote service with at most 10 rps)
• scheduling

Additionally, from my experience of writing emulated time tests (for Future though, it is much easier):
All waits are coordinated by the test. When code under test calls wait() function, it block indefinitely until:
a) test calls tick() funciton which releases single wait call with closest timeline
b) test calls waitUntil(time) function which releases all waits before given timestamp

edit: here is some inspiration: https://gist.github.com/scf37/4071839f25e197e38e4b070cfbed977b

[nob13]

@scf37 this sounds more or less like mocking of the clock for me, so that the test itself is capable of releasing waits.

This is a perfect use case for complicated things, which involves multiple threads.

In my small unit test thinking I would like to get rid of threads all together to get a deterministic behaviour. We can run a function, which itself calls wait/waitFor and has some stopping kriteria (e.g. when simulated time is out or some or some explicit behavior).

Consequently this leads to multiple clock implementations.

[scf37]

I would like to get rid of threads all together to get a deterministic behaviour

Provided Mock does exactly that - it performs linearization by adding chunks of code to single queue executed by test.

Unfortunately there is no API in Loom to access such "chunks" (AKA coroutines/continuations) therefore I don't know how to implement deterministic tests outside of Futures. Still, emulated delays are much better than nothing.

[adamw]

I was hoping that now when Ox manages all threads by itself, it would be possible to implement such a functionality. However, I think there's one major problem which forbids doing so. I do have a prototype implementation on the https://github.com/softwaremill/ox/tree/mock-time branch, but the flaw is still there.

The core problem is that we are unable to reliably detect, that all threads (within a concurrency scope, but that's a detail) are blocked (waiting on some condition). There's Thread.getState(), but there are no guarantees as to when it's updated. So even if it reports as WAITING, it might as well happen that the thread can run, but simply wasn't yet scheduled by the VM to run.

If we don't know when "all threads are blocked", we cannot determine that we should advance mock time, or return control to the user so that they can call the next .tick().

So unless there's some API in virtual threads (that is available now, or becomes available in the future) that allows to determine "this thread is blocked on a condition, and won't run until this condition is met" I don't think it's possible to implement a mock timer.

[mushtaq]

Scoped Values (https://openjdk.org/jeps/506) will be part of Java25 next month.

Is there any chance we can use that to inject Clock/Time in Ox and then use mocked version of Clock/Time in tests @adamw ?