Performance ideas/roadmap

[lmcd]

I see often in benchmarks that SwiftNIO seems to perform worst than most other things out there.
I expect this will probably get better over time.

It would be very useful for me and other advocates of swift-nio to have a write-up somewhere in the repo/wiki of the current state of play, performance penalties etc, some potential strategies to overcome these challenges, future Swift features that may improve things, how things like Io_uring could change the performance picture, what impact a full transition to Swift Concurrency might have etc.

I occasionally see threads like these on the Swift forums and find them very detailed and insightful. Things like this are beyond my usual level of expertise, so I think it'd be useful for someone with this information to present it for others to see.

Just a thought! 🙂

[Lukasa]

This is not a bad idea.

For what it's worth, SwiftNIO's perceived performance issues are not really reflective of where it's at. Instead, they're reflective of the size of the relative communities. Producing a good benchmark result is less about the framework and much more about tuning to the benchmark. NIO's defaults are designed to make it safe, not to make it fast, and so without some work to configure it NIO doesn't necessarily perform to its best.

But even then, the reality is that NIO is faster than essentially anyone in the community needs it to be. My view is that performance is not NIO's problem: actually, we could probably burn some performance at the cost of ergonomics.

[Lukasa]

A very useful thing community members could do, actually, is to write implementations of particular benchmarks and propose them for review here, or inclusion as examples in swift-nio-examples. This would allow for community collaboration on those benchmarks, which can help with turning around that benchmark perception.

[lmcd]

But even then, the reality is that NIO is faster than essentially anyone in the community needs it to be.

We're trying to push many Gbps through it 🙂 As in large, concurrent file transfer operations. We're not hitting any walls yet, but may eventually turn our focus to throughput as we scale.

[Lukasa]

That should be no problem whatsoever. When you do start hitting some bottlenecks, let us know.

[RaghavRoy145]

A very useful thing community members could do, actually, is to write implementations of particular benchmarks and propose them for review here, or inclusion as examples in swift-nio-examples. This would allow for community collaboration on those benchmarks, which can help with turning around that benchmark perception.

Hello @Lukasa, I was playing around with the current Benchmarks in place, and I was able to run the ones under the Benchmarks/ directory (haven't been able to get the ones under Sources/NIOPerformanceTester/ to run yet using the same commands).

I wanted to know if there is already active work in this area. There were a few interesting benchmarks that I wanted to propose, ones that cover more "real-world" functionality like large file transfers and lock-contention around that, partial reads and writes, or high volume concurrency that can be because of many messages being sent. I'm sure theres more/better scenarios that may be interesting to benchmark, or if there's gaps.

Since there is a lot of the framework set-up I'd love to know what are potentially good ones to write, or if there's existing ones that need extending?

Further, I'd like to know if it makes sense to benchmark cases which get around swift's safety mechanisms for speed, like grouping multiple related steps under a single actor to reduce the number of executor hops - but potentially increasing data race occurrences. If these are ways people may write their swift code, it could make sense to benchmark it, if only to show that the gains are not worth the pains (unless ofcourse like you mentioned, the reality is that NIO is faster than essentially anyone in the community needs it to be , and its just a fringe case)

[Lukasa]

I wanted to know if there is already active work in this area. There were a few interesting benchmarks that I wanted to propose, ones that cover more "real-world" functionality like large file transfers and lock-contention around that, partial reads and writes, or high volume concurrency that can be because of many messages being sent. I'm sure theres more/better scenarios that may be interesting to benchmark, or if there's gaps.

Since there is a lot of the framework set-up I'd love to know what are potentially good ones to write, or if there's existing ones that need extending?

Many of the benchmarks that would be most interesting don't fit into the microbenchmark context of the Benchmarks directory, or even really of NIOPerformanceTester which is still very micro-benchmark focused. Instead, a better example might be things like writing a canonical NIO implementation of certain kinds of HTTP server. This is why I offered swift-nio-examples as a good use-case. Benchmarking and optimizing those things is very possible.

A good example would be the HTTP responsiveness server that @ehaydenr contributed to the examples. Having someone sit down and do some performance work on this tool would be very interesting.

get around swift's safety mechanisms for speed

This is a really useful call-out, so I should explain a NIO principle that we don't always articulate. For swift-nio, our view is that using unsafe code for performance reasons is only acceptable in cases where there is absolutely no safe alternative, and the safe code is so slow as to be unusable.

Otherwise, our view is that it must be possible to write fast code in safe Swift, and an inability to do so is fundamentally a Swift bug. We'd treat it as such: filing the bug report on Swift, and where possible avoiding the construction. So in general this kind of benchmark is not something we're particularly interested in, except in cases where it can be shown that a) the performance limit is very substantial, and b) no safe alternative exists.

[RaghavRoy145]

Many of the benchmarks that would be most interesting don't fit into the microbenchmark context of the Benchmarks directory, or even really of NIOPerformanceTester which is still very micro-benchmark focused. Instead, a better example might be things like writing a canonical NIO implementation of certain kinds of HTTP server. This is why I offered swift-nio-examples as a good use-case. Benchmarking and optimizing those things is very possible.

A good example would be the HTTP responsiveness server that @ehaydenr contributed to the examples. Having someone sit down and do some performance work on this tool would be very interesting.

Got it, I think I'll work on an example app that has the option to collect benchmarks built in as a flag or something of some simple functionality, and have it be configurable in some way so that things like "file size" for transfer or "number of concurrent threads" can be knobs. Would be interesting to see how it scales, and also this would make the app extensible in a way that people can simply add their own functionality to this; which can be wrapped using the same benchmarking helpers.

This is a really useful call-out, so I should explain a NIO principle that we don't always articulate. For swift-nio, our view is that using unsafe code for performance reasons is only acceptable in cases where there is absolutely no safe alternative, and the safe code is so slow as to be unusable.

Otherwise, our view is that it must be possible to write fast code in safe Swift, and an inability to do so is fundamentally a Swift bug. We'd treat it as such: filing the bug report on Swift, and where possible avoiding the construction. So in general this kind of benchmark is not something we're particularly interested in, except in cases where it can be shown that a) the performance limit is very substantial, and b) no safe alternative exists.

This is really cool info about NIO in general. I'll keep my implementations of the simple functions idiomatic in that case.

Thanks for this!