JavaScript traditionally didn’t had great support for binary data. So, the Node developers had to tackle this problem and primarily for performance reason and this is where the Buffer data type came into picture.

They are specifically also useful when working with TCP streams and file operations.

Buffers are raw allocations of the heap and contains binary data, which just consist of bits, i.e. o or a 1.

The buffer documentation is here https://nodejs.org/dist/latest-v9.x/docs/api/buffer.html

They are available globally and thus don’t need to be imported and can be treated like any other JavaScript type.

We will understand buffer by following the below activities.

• Creating Buffer from string
• Creating Buffer from an array
• Convert Buffer to JSON
• Concatenating Buffer
• Compare Buffers
• Converting a Buffer to different encodings
• Using the Buffer API to transform a binary file to JSON
• Encoding and decoding your own binary protocol

Changing data encodings

When working with file if no encoding is given, file operations and many network operations will return data as a Buffer.

For example, take this fs.readFile (Take a look at the source code, buffer folder)

let fs = require('fs');
fs.readFile('./phone.txt', function (err, buf) {
   console.log(Buffer.isBuffer(buf)); // true
});

Running this program as node readfile.js will give the below output

Example 1: Turn a Buffer into plain text

Let us take the example of a text file, phone.txt, which contains the below data

name,mobile
a,987837373
b,897878787
c,383783737

By default if we load this file, without specifying any encoding we will get a Buffer object by default.

The output is shown below:

But in the above example, since we already know that this data is only comprised of ASCII characters, we could also get a performance benefit by changing the encoding to ASCII rather than UTF-8.

To do this, we will provide the type of encoding as the first argument to toString as shown below.

let fs = require('fs');
fs.readFile('./phone.txt', function (err, buf) {
   console.log(buf.toString('ascii'));
});

Running this, will result in the same output as before, but for larger files will be much more performant.

The Buffer API provides other encodings such as base64, hex, utf16le etc. More details about it can be found in the Node documentation as well as http://unicode.org/faq/utf_bom.html.

Changing string encodings using buffers

In addition to converting buffers, we can also utilize buffers to turn one string encoding into another.

Example 1: You want to change from one string encoding to another.

Let us take a dig at the above problem by creating a basic authentication header.

Sometimes it is very helpful to build a string of data and then change its encoding. For example, if you wanted to request data from a server that uses Basic authentication, you would need to send your credentials, like username and password encoded using Base64:

Authorization: Basic dG9tbXk6c2VjcmV0

Before Base64 encoding is applied, Basic authentication credentials combine the username and password, separating the two using a : (colon). For our example, let us use the following credentials:

username: tommy
password: secret

let username = 'tommy';
let password = 'secret';
let authvalue = username + ':' + password;

Let us now convert this into a Buffer in order to change it into another encoding. Buffer can be allocated by bytes and also can be allocated by passing in string data.

// Let's convert the string value to a Buffer
let buf = new Buffer(authvalue);
let encoded = buf.toString('base64');  
console.log(encoded); // outputs: dG9tbXk6c2VjcmV0

NOTE: By default when strings are used to allocate a Buffer, they are assumed to be UTF-8 strings. But we can also specify the encoding of the incoming data using a second, encoding argument as shown below

new Buffer('dG9tbXk6c2VjcmV0', 'base64')

Example 2: Embedding a PNG as a Data URI

Data URI’s are another example of when using the Buffer API can be helpful. Data URIs allow a resource to be embedded inline on a web page using the following scheme:

data:[MIME-type][;charset=<encoding>[;base64],<data>

For example, the following kungu pada https://static.wixstatic.com/media/2cd43b_d1ecd76051de458b9041402093ffeb54~mv2.png image can be represented as a data URI:

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQ...

Let’s check the code to accomplish this. Let us put this code in a file named png-to-datauri.js

Example 3: Creating an image file from the Data URI

Lets’ take the data uri created above and reverse it to a png file. To get the complete data uri you can redirect the output to a file as shown below.

Running the above command will save png as base64 in a file named base64png.txt.

Lets have a look at the code to reverse the process and create the png. (file: datauri-to-png.js)

The important part of the code is uri.split, and then we take the base64 encoded string and create a buffer.

NOTE: We are using the sync method of fs for demonstration (for real apps use the async version whereever applicable)

Running the command

creates a file named secondpanda.png.

Example 4: Compressing files using gzip (buffer)

(Also check the stream version)

The code is small and easy to understand and hence I am putting it down for your reference.

We import the required libraries and store the file name that is passed in the command line into a varible named file.

The steps to gzip using buffer is outlined below.

• Use fs to readFile
• Use zlib.gzip to compress the buffer passed from the readFile method.
• Use fs writeFile to store the file to disk.

You can run the following command from the terminal.

node gzipbuffer.js <filenametocompress>