streaming control

[berry4u]

Hi
I'm trying to use this node to read big files, passing them to an http streamer (as blocks).
Is there a way to manage the flow control in order to wait the reception confirmation of a block to proceed in reading the next one.

Thanks

[Jacques44]

Hello, here is a solution: every node I know including node-red-contrib-traffic I’ve made or others use memory to store flow messages. This may be not acceptable for very very big files. The solution here relies on a function which stacks messages en send them only if it receives a « go » control message You just have to fill your filename in this flow in the node named « filename" [{"id":"e00a8e65.41d33","type":"bigfile","z":"81e867ca.e36108","name":"","filename":"","nopayload":false,"flow":"blocks","highWaterMark":16,"encoding":"utf8","format":"utf8","keepEmptyLines":false,"x":338,"y":227,"wires":[["35e3993.af63466"],[]]},{"id":"de189b80.166b98","type":"inject","z":"81e867ca.e36108","name":"filename","topic":"","payload":"fill here your filename","payloadType":"str","repeat":"","crontab":"","once":false,"x":130,"y":226,"wires":[["e00a8e65.41d33"]]},{"id":"62ef45a4.78087c","type":"debug","z":"81e867ca.e36108","name":"","active":true,"console":"false","complete":"false","x":1030,"y":221,"wires":[]},{"id":"372697ee.a46be8","type":"function","z":"81e867ca.e36108","name":"next","func":"msg.payload = \"go\";\nmsg.control = true;\nreturn msg;\n","outputs":1,"noerr":0,"x":574,"y":121,"wires":[["35e3993.af63466"]]},{"id":"d106356f.8d6008","type":"inject","z":"81e867ca.e36108","name":"","topic":"","payload":"next","payloadType":"str","repeat":"","crontab":"","once":false,"x":283,"y":122,"wires":[["372697ee.a46be8"]]},{"id":"35e3993.af63466","type":"function","z":"81e867ca.e36108","name":"control","func":"// Get stack from context\nvar stack = flow.get('stat');\nstack = stack || [];\n\n// If data message, store it in stack\nif (!msg.control) stack.push(msg);\n\n// If \"next\" control message\nif (msg.control && msg.payload === \"go\") {\n msg = stack.pop();\n} else {\n msg = undefined;\n}\n\n// Store stack in node context\nflow.set('stat', stack);\n\n// Node status\nif (stack.length) {\n node.status({fill:\"blue\",shape:\"dot\",text:\"has \" + stack.length + \" messages\"});\n} else {\n node.status({fill:\"green\",shape:\"dot\",text:\"has not any messages\"});\n}\n\nif (msg) return msg;\n","outputs":1,"noerr":0,"x":734.5,"y":219,"wires":[["62ef45a4.78087c"]]},{"id":"4882fe.18c85d04","type":"inject","z":"81e867ca.e36108","name":"","topic":"","payload":"clear","payloadType":"str","repeat":"","crontab":"","once":false,"x":281,"y":66,"wires":[["f43a9c08.bbffe"]]},{"id":"f43a9c08.bbffe","type":"function","z":"81e867ca.e36108","name":"clear","func":"flow.set('stat', []);\nreturn msg;","outputs":1,"noerr":0,"x":425,"y":66,"wires":[["372697ee.a46be8"]]}]

…

Le 25 août 2019 à 17:13, berry4u ***@***.***> a écrit : Hi I'm trying to use this node to read big files, passing them to an http streamer (as blocks). Is there a way to manage the flow control in order to wait the reception confirmation of a block to proceed in reading the next one. Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7?email_source=notifications&email_token=AEKS6M5ETKWYMIBBSDFSUHDQGKOQ7A5CNFSM4IPJK322YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HHHUFTQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEKS6M5KWR2MPKC2DNYGW3TQGKOQ7ANCNFSM4IPJK32Q>.

[berry4u]

Thank you for answering.
Yes, your hint works fine and as you pointed out it is suitable for big but not huge files.
For those, I'm wondering if it's feasible to build a loop with "big file reader node" by setting the start and end reading positions in order to load only a couple of blocks per time in memory.

[Jacques44]

Hello, I thought of offering a way to flow blocks through the big file node. In fact it relies ont .fs.createReadStream which streams block and I’ve no control on it.

…

Le 26 août 2019 à 19:58, berry4u ***@***.***> a écrit : Thank you for answering. Yes, your hint works fine and as you pointed out it is suitable for big but not huge files. For those, I'm wondering if it's feasible to build a loop with "big file reader node" by setting the start and end reading positions in order to load only a couple of blocks per time in memory. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7?email_source=notifications&email_token=AEKS6M4UIDBC2WKKGP5WRWLQGQKVVA5CNFSM4IPJK322YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FEUEY#issuecomment-524962323>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEKS6M6N6XYTC3P6DBWN7A3QGQKVVANCNFSM4IPJK32Q>.