You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some thoughts on how to make Transform streams faster.
Part of the overhead of Transform (and PassThrough) is that it is actually 2 streams, one Writable and one Readable, both with buffering and state management, which are connected together.
We could try to skip this and implement Transform as a Readable which implements the Writable interface and proxies the naming.
e.g.
classFastTransformextendsReadable{constructor(options){super(options)this._writableState={length: 0,needDrain: false,ended: false,finished: false}}getwritableEnded(){returnthis._writableState.ended}getwritableFinished(){returnthis._writableState.finished}_read(){constrState=this._readableStateconstwState=this._writableStateif(!wState.needDrain){return}if(wState.length+rState.length>rState.highWaterMark){return}wState.needDrain=falsethis.emit('drain')}write(chunk){constrState=this._readableStateconstwState=this._writableStateconstlen=chunk.lengthwState.length+=lenthis._transform(chunk,null,(err,data)=>{wState.length-=lenif(err){this.destroy(err)}elseif(data!=null){this.push(data)}this._read()})wState.needDrain=wState.length+rState.length>rState.highWaterMarkreturnwState.needDrain}end(){constwState=this._writableStatewState.ended=trueif(this._flush){this._flush(chunk,(err,data)=>{constwState=this._writableStateif(err){this.destroy(err)}else{if(data!=null){this.push(data)}this.push(null)wState.finished=truethis.emit('finish')}})}else{this.push(null)wState.finished=truethis.emit('finish')}}}// TODO: Make Writable[Symbol.hasInstance] recognize `FastTransform`.
Making this fully backwards compatible with Transform might be difficult.
$ time node async-generator.js
real 0m0.788s
user 0m0.772s
sys 0m0.075s
$ time node fast-transform.js
real 0m1.612s
user 0m1.729s
sys 0m0.169s
So using an async generator in pipeline might be significantly faster. Is it worth further optimizing Transform? Note that an optimization as suggested here does make it harder to maintain than today's rather trivial implementation.
pipeline will convert a Readable source into an async iterator and read from it that way. Might introduce unnecessary overhead. Will investigate that first.
Activity
ronag commentedon May 14, 2020
Thoughts @nodejs/streams?
mcollina commentedon May 14, 2020
How much will we gain from this? I'm +1, possibly with a better name.
ronag commentedon May 14, 2020
@mcollina: ~60%
FasterTransform? 😄targos commentedon May 14, 2020
How does it compare to transforming with an async generator function?
ronag commentedon May 14, 2020
So using an async generator in
pipelinemight be significantly faster. Is it worth further optimizingTransform? Note that an optimization as suggested here does make it harder to maintain than today's rather trivial implementation.ronag commentedon May 17, 2020
pipelinewill convert aReadablesource into an async iterator and read from it that way. Might introduce unnecessary overhead. Will investigate that first.clshortfuse commentedon Jun 16, 2020
Apparently I should be using
pipeline. This does seem like a good replacement forPassThrough.