Communicating between Node.js and Python

Can a node.js server call Python functions? Yes, yes it can.

In a recent project, I needed to do some natural language processing. Since I was running a node server, one solution would have been to use the natural node module and do the computation entirely in Javascript. Natural looks promising, but I preferred to use the more established Natural Language Toolkit (nltk) Python library, which has some excellent, free documentation in the form of Natural Language Processing with Python.

How can node and Python talk to each other? The answer depends on whether the node and Python processes are being run on the same machine or separate machines.

For node and Python on the same computer, there are two solutions: (1) node could talk to Python via Unix domain sockets, or (2) node could call the Python functions as child processes.

But running node and Python on the same machine isn’t a scalable solution: as the natural language processing in Python becomes more and more processor- and memory-intensive, those resources would be diverted away from serving web requests. The node server’s response time would suffer and users would go elsewhere.

The scalable solution is to have a separate machine dedicated to the Python work. The node server can talk to this machine via TCP sockets. Rather than work with sockets directly and deal with problems like what happens when a packet gets dropped, I chose to use a messaging framework which takes care of that messy work. ZeroMQ is a messaging framework which has been described as “sockets on steroids”. There is some debate over just how much of an upgrade ZeroMQ is over sockets. I have nothing to add to that debate, but it seems generally agreed that there is an advantage in the robustness of its queuing for a variety of messaging patterns.

So ZeroMQ is a framework for exchanging messages, with implementations for 40+ languages including Javascript and Python. But my goal was not just to exchange messages between my node server and a Python listener, but for the node server to actually invoke Python code and get the result back. Fortunately there is ZeroRPC: a library for node.js and Python built on top of ZeroMQ that enables remote procedure calls (RPC).

Using ZeroRPC is a breeze. Following the steps in their Hello, World example, we can see that on the Python side of things, all we have to do is run a Python script which creates a ZeroRPC server object out of a Python class (in this case the HelloRPC class), and set that server to listen on port 4242 for incoming connections from any IP.

import zerorpc

class HelloRPC(object):
    def hello(self, name):
        return "Hello, %s" % name

s = zerorpc.Server(HelloRPC())
s.bind("tcp://0.0.0.0:4242")
s.run()

On our node server, we create a ZeroRPC client object which connects to the listening Python server and can invoke any of the functions in the class with which the server was created. In this case, we invoke the hello function and pass along the "World!" string as its argument.

var zerorpc = require("zerorpc");

var client = new zerorpc.Client();
client.connect("tcp://127.0.0.1:4242");

client.invoke("hello", "World!", function(error, res, more) {
    console.log(res);
});

Hello, World! gets returned to our node server.

In this example, the Python server was created on the same machine as the node server, so we connected to it at localhost (127.0.0.1). If we were running Python on a different machine, as I was doing in my caracol project, communicating with that machine is as simple as replacing that address.

And there you have it: node.js and Python communicating in a service-oriented architecture. Brilliant!

A final thing to note: ZeroRPC depends on libevent and ZeroMQ, so you’ll need to install those before installing ZeroRPC. Here is a shell script for Ubuntu 12.04 LTS which installs everything you need. For Mac OS X, you can install the system dependencies using Homebrew.

#!/bin/bash

# Simple script for installing ZeroRPC on Ubuntu 12.04 LTS

# System dependencies

# First install ZeroMQ
sudo apt-get install libzmq-dev

# Next install libevent, an event notification library required by zerorpc
sudo apt-get install libevent

# Python dependencies

# Now install pyzmq: Python bindings for ZeroMQ
# If you don't already have pip installed:
sudo apt-get install python-setuptools
sudo apt-get install python-pip
sudo pip install pyzmq

# Now we can install ZeroRPC
sudo pip install zerorpc

# Node.js dependencies

# Just install the ZeroRPC node module
sudo npm install -g zerorpc

Reposted From : Ian Hinsdale