View on GitHub

Accel.js

Accelerating your Node.js program by parallelizing computation with multicores and multinodes.

Index Documentation Archive for 15-418 Competition

// bar.js
function foo(arg1, arg2) {
    // expensive computation
    return result;
}
$(foo);
for (var i=0; i<100; i++)
    for (var j=0; j<100; j++) {
        // Launch 10,000 jobs
        $.foo(i, j, (res) => {
            console.log("The result is", res);
        });
    }

# bash
npm install accel -g
acceljs bar.js

Comparison

	Accel.js	Parallel.js (Rockstar with 2K+ stars)	Cluster (Node.js original)
Computational Resources	Local Cores & Cluster Cores	Local Cores	Local Cores
Lines of Code for Similar Tasks (Mandelbrot, ~20 for serial)	~40	~60	~80
Load Balancing Assignment	YES	YES	NO
Manual Assignment	YES	NO	YES
Asynchronous Job Support	YES	NO	YES
Overhead per Job	LOW	HIGH	LOW
Data Synchronization Mechanism	Transparent sync environment	One string only	One string only
Communication	Anytime, to Anyone (via Channel)	On Spawn only, Master-Worker only	Anytime, Master-Worker only
MPI Support	YES	NO	YES
Data-Parallel Support	On the way!	YES	NO

Wonderful Features

One-step Wrapper to Parallelize Function

Simply by wrapping one function with $(), you can get one distributed version and invoke it on workers. Note that in wrapped functions, you can only refer to $ (synchronized environment) and accel (package itself).

Easy to Go Distributed

All you need to do to parallelize your Node.js application to multiple machines is modify your launching command, e.g., from acceljs bar.js to acceljs -l 4 -r m1:15618 4 bar.js. (For detail refer to acceljs --help)

Environment Synchronization

Accel.js performs smart (delta sync) and transparent environment synchronization to help you transfer all data you want to workers. Since it is delta synchronization, you won't pay for extra communication costs once transferred.

const accel=require("accel");
const $=accel();

$.alpha=0.6;
$.mixNumber=(c1, c2) => $.alpha*c1+(1-$.alpha)*c2;
$(function foo(n) {
    // you can use $.mixNumber and $.alpha here
});

Asynchronous Function on Worker

Although Accel.js by default treats one function as completed in return, you can explicitly assign an asynchronous function to worker and it will not be marked completed until callback is invoked.

function bar(time, callback) {
    setTimeout(() => callback("Finishes"), time);
}
$(bar, "async")(1000, (res) => {
    // will get here after 1000ms
});

Universal Dependency

When you want to require a dependency for worker, you don't have to do it once in every worker code. Instead, take our universal dependency and use it everywhere.

$.os=accel.require("os");    // introduce an universal dependency
function printOS() {
    return $.os.platform();
}
console.log(printOS);    // you can use it on master
$(printOS)((res) => console.log(res));  // or use it on worker

MPI Support

Accel.js also have brilliant support for MPI jobs, where you assign long running jobs for each node (instead of assigned by job queues) and each job communicate with each other.

$.ch1=new accel.Channel();  // a multi-producer-multi-consumer channel
function MPIJob(callback) {
    var nodeId=accel.id;
    accel.Recv((msg, fromNode) => {  // receive a message from pre-defined channel
        // ...
    });
    $.ch1.Send(someMessage);  // or send one message to self-defined broadcast channel
}
$(MPIJob, "async").toAll()(() => {
    // since MPI job do not complete, the callback will never get invoked
});

More...

For complete list of features chck out documentation

Evaluation

Mandelbrot Graph Calculation

1. All experiments are conducted on 4-core machines. Task to be done is rendering 26k x 4k mandelbrot graph and Accel-k denotes the performance of Accel.js on k machines. (Since Parallel.js cannot utilize multiple machines, we ignore Parallel-k)
2. Comparison between Accel-1 and Parallel.js, with trailing -k as the number of subtasks.

Webfont Subsetting Service

1. All experiments are conducted on one 4-core machine.
2. Webfont subsetting service is a Google Font like web app, which receives an URL, crawls its data and subset a large font file. The task is both Disk-I/O-intensive (read and write the font file (~10MBs)) and CPU-intensive (parse the font and setup index in memory)
3. To simulate real workload, all servers are implemented with cache, and it takes almost no time to serve a cache hit.