import WARCStreamTransform from 'node-warc/lib/parsers/warcStreamTransform.js'

public class | source

WARCStreamTransform

Extends:

Transform → WARCStreamTransform

Transforms a WARC file ReadStream into its individual WARCRecords

Example:

 fs.createReadStream('someWARC.warc')
   .pipe(new WARCStreamTransform())
   .on('data', record => { console.log(record) })

 fs.createReadStream('someWARC.warc.gz')
   .pipe(zlib.createGunzip())
   .pipe(new WARCStreamTransform())
   .on('data', record => { console.log(record) })

Constructor Summary

Public Constructor
public	constructor() Create a new WARCStreamTransform

Member Summary

Public Members
public	buffered: Buffer
public	builder: RecordBuilder
public	sepLen: number

Method Summary

Private Methods
private	_consumeChunk(chunk: Buffer, done: function, pushLast: boolean) Process the supplied chunk
private	_flush(done: function) Flushes any remaining data
private	_transform(buf: Buffer, enc: string, done: function) Process a chunk

Public Constructors

public constructor() source

Create a new WARCStreamTransform

Public Members

public buffered: Buffer source

public builder: RecordBuilder source

public sepLen: number source

Private Methods

private _consumeChunk(chunk: Buffer, done: function, pushLast: boolean) source

Process the supplied chunk

Params:

Name	Type	Attribute	Description
chunk	Buffer		The chunk to be processed
done	function		Function used to indicate we are done processing the chunk
pushLast	boolean	optional	Boolean indicating if we attempt to build a record and push it once we are done processing the chunk IFF a record was built. Is only true when called from _flush

private _flush(done: function) source

Flushes any remaining data

Params:

Name	Type	Attribute	Description
done	function		Function used to indicate we are done processing the chunk

private _transform(buf: Buffer, enc: string, done: function) source

Process a chunk

Params:

Name	Type	Attribute	Description
buf	Buffer		The chunk to be processed
enc	string		The encoding of the chunk
done	function		Function used to indicate we are done processing the chunk