FixedQueue

[conartist6]

I can't tell what you were going for with the FixedQueue class. It has some beautiful ASCII art in it, but it seems to be part of a design that was not the final one. In particular it seems to assume that it will be fed data character by character, and thus that a pure linked list would have too much memory overhead. In fact though only chunks are ever stored in it so it could be trivially replaced with something like @iter-tools/queue, which would also allow you to remove the queueSize variable.

I also think the dynamic requires on the queue are premature optimization.

[Farenheith]

I took FixedQueue directly from NodeJs source code. I used it because I intend to open a PR to them and this queue implementation showed a very good performance. It's not actually my code.
Unfortunately the implementation on this package doesn't completely cover the test cases from the NodeJs project so I declined the idea of the PR

[conartist6]

I know, I was reading the thread.

I could be open to giving a place in the iter-tools organization though.

[Farenheith]

@conartist6 I took a look on the @iter-tools/queue and it actually followed a logic quite similar to a minimalist implementation I did first (just on my machine):

function simpleQueue() {
  let first;
  let last;
  return {
    isEmpty() {
      return !first;
    },
    push(value) {
      const node = { value };
      if (last) {
        last.next = last = node;
      } else {
        last = first = node;
      }
    },
    shift() {
      if (first) {
        const { value } = first;
        first = first.next;
        if (!first) {
          last = first;
        }
        return value;
      }
    }
  };
}

With this implementation, the performance seemed a bit worse in the benchmark than with the FixedQueue, or it was at least the same. As internally node already had that implementation in hand and the intention of this package was to make an easy copy and paste code, I decided to use it. I trusted that the NodeJs team came up with this solution after some studies involving the V8 nature and memory management, so, this is a question I really want to get deep into, later.

About using your package, I don't have anything against it. I can do that but I really want to take some time to understand better the results I got on my preliminary tests.

[conartist6]

Your code does its best to keep no more than a single item in the buffer. Anything should perform well I'd think, even an array.

[Farenheith]

Actually, my code tries to pause the readable interface when the queue reaches 2048 items, so, it looks like this will be the biggest size of the queue and I'll have only one array in memory, but it's not quite like that. I notice that, even after the call for the pause method, I can still receive line events, so It's not guaranteed that I'll stay with 2048 items, probably due to the size of the chunks readline is reading. In my tests I actually got almost 5000, so with this FixedList implementation, I got 3 arrays of 2048 in memory in that situation, and probably it performed better because the allocation of an array of that size is optimized in the V8, compared to creating 5000 linked objects.

But I'm really curious about it and I'll try to understand it better when I got some time.

[conartist6]

The readable interface says that the data event is called with a chunk. You're saying the queue contains 2048 chunks? Or have I misunderstood something else?

[conartist6]

Also to your point of the @iter-tools/queue code being like your minimalist implementation, that is very much on purpose. My goal with many of these projects is to ensure that they are not significantly more complicated to understand or use than just writing your own copy, because often it will be easy to build a simple version of the functionality I offer.

[Farenheith]

In the FixedQueue implementation, instead of creating one object for each node, it starts creating an array of 2048 items that will be used as a circular queue. If you need more than 2048 items in the queue, FixedQueue creates another array, managing them as a "queue of queues".
About the chunks, the readable interface implementation receives chunk by chunk from the original stream and, for each chunk, he splits it into lines. What I meant is that, probably, the test I mentioned got big chunks that came out to have thousands of lines each one. For example, he could get the first chunk with 2000 lines, split the lines, and a second one with 3000 lines, split the lines, and then my package called pause. The pause here will prevent it from getting further chunks, but will not prevent it from processing the current split lines, so, even though I called pause when I got 2048 lines, I'll still receive 2952 additional lines from the readline interface.
The effect I got in the FixedQueue implementation is that it will instantiate three circular queues of 2048 items so it can deal with all the 5000 chunks. I don't know how much memory it uses but, let's say is 2048 * 16 bytes for each array, so, 32Kb. The decision for this strategy is from the NodeJs team, but I believe that it's pretty optimized to run in the V8 engine.

I pointed that out because you said my code keeps no more than a single item in the buffer, but it's not actually true, because I got almost 5000 items in one of my tests. Or maybe I understood something wrong from what you said

[conartist6]

You're missing the point. A chunk is by default 64K. If you fill up even one circular buffer (2048 items) that would be 131M of data sitting in buffers.

[Farenheith]

What I mean by chunk is each message that is read from the original stream. I'll not have chunks in the queue, I'll have lines, which are small smaller strings. Maybe we're having some language difficult here, as English is not my first language

[conartist6]

Your english is great!

I think my difference stems from the fact that I would prefer to see splitting into lines happen as an operation on an iterable, not on a stream. But the code you've written, and particularly if it were adopted into node, has little control over what size chunks the stream is split into, meaning it must work equally well for big or small chunks.

[conartist6]

What I really want to be writing is openFile('foo.js').splitOnAny(['\r\n', '\n']). But I'm held up because it implies tiny chunks (individual characters) in the output iterable, something which cannot possibly be sufficiently performant without optimizations to avoid promises and intermediate objects.

[conartist6]

But it's the only way you can write any kind of code without creating new edge cases when things you need to understand cross chunk boundaries.