Don't support reading "snappy framed" format - bug or feature?

[grimen]

Motivation: I am trying to use snappy and snappy-stream along each other they write different output because of the framed format, I would assume snappy library should handle the framed format case ideally? At minimum would make sense to detect it and raise a more developer friendly error message than "Invalid input"?

const fs = require('fs')
const snappy = require('snappy')
const snappyStream = require('snappy-stream')

// ==================================
//      SNAPPY
// ------------------------------

const inData = fs.readFileSync('data.txt')
const snappyCompressData = snappy.compressSync(inData)

console.log('snappy.compressSync DATA', snappyCompressData)


// ==================================
//      SNAPPY-STREAM
// ------------------------------

const snappyCompressStream = snappyStream.createCompressStream()

const inStream = fs.createReadStream('./data.txt')
const outStream = fs.createWriteStream('./compressed.sz')

// NOTE: "snappy" and "snappy framed" are not identical: https://github.com/google/snappy/blob/master/framing_format.txt

inStream
    .pipe(snappyCompressStream)
    .on('data', (chunk) => {
        this.buffers = this.buffers || []
        this.buffers.push(chunk)
    })
    .on('end', () => {
        const data = Buffer.concat(this.buffers)

        console.log('snappyCompressStream DATA', data, snappy.uncompressSync(data))
        // console.log('snappyCompressStream DATA', data)
    })
    .pipe(outStream)

Output:

➜  node-snappy-stream-test node compress.js
snappy.compressSync DATA <Buffer 13 48 7b 0a 20 20 20 20 22 66 6f 6f 22 3a 20 62 61 72 0a 7d 0a>
/home/jonas/Desktop/node-snappy-stream-test/node_modules/snappy/snappy.js:49
  return binding.uncompressSync(compressed, uncompressOpts(opts));
                 ^

Error: Invalid input
    at Object.exports.uncompressSync (/home/jonas/Desktop/node-snappy-stream-test/node_modules/snappy/snappy.js:49:18)
    at CompressStream.<anonymous> (/home/jonas/Desktop/node-snappy-stream-test/compress.js:36:63)
    at CompressStream.emit (events.js:208:15)
    at endReadableNT (_stream_readable.js:1154:12)
    at processTicksAndRejections (internal/process/task_queues.js:77:11)

[grimen]

Implementation of the framing format is optional for Snappy compressors and
decompressor; it is not part of the Snappy core specification.

From: https://github.com/google/snappy/blob/master/framing_format.txt

Well, the Google do point out that it is not part of core specification but rather an optional - but yea, would be neat if it was handled. Not sure otherwise how to ensure data integrity. 🤔

[kesla]

hey,

Sorry for late response.

yeah I think you're right. I think that the best way forward here will be to support the framing format in snappy, so merge in https://github.com/kesla/node-snappy-stream.

I will then change it so that .createUncompressStream() support output from "normal" snappy and .uncompress() can handle data in the framing format.