'pause' not sent to underlying stream

[dpcollins-google]

The 'pause' method on the MessageStream class is not proxied to the underlying grpc PullStreams. This means that messages continuously back up at the MessageStream layer without ever triggering grpc channel backpressure, causing infinite buildup irrespective of flow control settings.

[callmehiphop]

@dpcollins-google the underlying grpc streams get piped into the message stream, so there shouldn't be a need to manually call pause/resume since streams have backpressuring built in.

Are you experiencing an issue where you see a message pile up or memory leak, etc.?

[dpcollins-google]

I am experiencing a memory backup. The subscriber code is here. When publishing with a message size of 9MB, my subscriber processes (one per core on a 16 core GCE machine with 60GB of ram) get killed by the OOM killer, even though I've set a flow control byte limit of 500 MB.

Even presuming that there are 16 messages per stream (80 per process) backed up in the GRPC layer as described here, this amounts to 1.2GB/core total, still not enough to account for the 60GB of ram.

There doesn't seem to be anywhere other than in the stream layer that these messages could be backing up, no references to them outlive the on('message') callback.

[callmehiphop]

@dpcollins-google I think there are a few things worth mentioning

• messages pass through grpc streams in batches, so I think they will be closer to 20MB per object
• we monkey patch grpc streams so there should never be more than 1 batch per grpc stream buffered on the JS side
• The C++ portion the grpc addon also buffers messages, however I don't have any insight into how many and I don't believe this is configurable

You might try testing the js version of gRPC which should buffer fewer messages.

[dpcollins-google]

Given that I have a repeatable scenario that can trigger this, do you have any suggestions for how to determine in which of these layers the backup is? Doing something other than the default client setup is a non-starter for me.

[dpcollins-google]

I've attached a heap dump when 50/60 GB of memory was in use. It doesn't look like the primary memory usage is inside of node, but perhaps someone who understands the library better could determine whether there is a leak of node managed grpc resources?
dump.heapsnapshot.zip

To clarify, this is a heapdump of one of the 16 node processes.