Plain array is incomparable in the comparisons #4
Description
Heya!
Could plain array be changed to allocate a page worth of bytes memory instead of n -longs? The comparison readme at the moment looks as if a normal array would be faster than a buffer, which is simply not the case. The other buffers have a maximum byte capacity of one page size, which makes them comparable to each other as their capacity is the same and take roughly the same amount of time to construct. It would be a lot of work changing the other buffers compared like go.d and rwqueue.d to allocate a billion bytes to achieve better performance from their current one pagesize. Here is a simple way of changing the page size for the plain array:
import core.memory : minimumPageSize;
long[] arr = new long[minimumPageSize/long.sizeof];The reason why buffers are usually just a pagesize or a couple of pagesizes is that memory is very fragmented and allocating all of the size takes up a really long time. The OS struggles to find that many bytes next to each other, without there already being an allocation somewhere in between the first and the billionth byte as 244140 pages of memory are required. It may not even be possible to allocate that much at all if the computer has run for long enough for programs to have badly fragmented the memory , if it is a machine that only has a gigabyte of memory or if there is memory, but the OS and its programs are already taking much of the memory.
Another issue is that buffers are usually used in tasks when you do not know how many bytes you'll end up receiving to the buffer, as is often the case with IO. In which case it is not possible to predict how large the buffer needs to be. If you know how much you'll receive in one task, in the next task the plain array could need more data which will cause a buffer overflow.
Why even use the buffer? It'd be even more efficient if it just incremented the sum. 😉
All jokes aside, the largest issue buffers try to solve is what do you do when: You've read the buffer full, then popped half of the items from the front and cannot make sense of the rest of the data without reading more to the end of the array. This needs to be done while preserving the FIFO -nature of the buffer for it to be useful for understanding the received data. With normal arrays the only possibility is copying the existing data to the start of the buffer and then reading more. The buffers compared are performing this task in the background when needed, so that the user does not need to care. The plain D array script does not try to solve this issue.
By the way, as Elembuf is also a plain D array, the performance is the same with equal buffer size. It just has an improved O(1) concat and built-in optional threading.