Setting up a mirroring cluster

General information

The Github API limit (currently, 5000 reqs/sec) makes it impossible to retrieve all data linked from events on a single node. For that, GHTorrent was designed to work on multiple phases in a distributed fashion. Depending on the data you want to collect, a cluster setup may be necessary.

A full GHTorrent cluster consists of the following types of nodes:

• Event retrieval nodes: Nodes that query the public Github event API for new events. More than one instances are required to both ensure that no events are lost due to spikes in event generation and that machine or network malfunctions the event collection machine do not affect the service.
• Linked data retrieval nodes: Retrieval of data linked by events is where the Github API is imposing the most significant restrictions.
• MongoDB shards: A MongoDB installation can be sharded (have the data spread on multiple nodes) on a per collection basis. Sharing MongoDB helps with both distributing the storage requirements and faster querying. Sharding is transparent to the application and therefore no modification is required to GHTorrent to work with a shared MongoDB. See more on MongoDB sharding here
• RabbitMQ active-active mirrors: RabbitMQ can work in cluster mode for high availability. GHTorrent will include basic support for this by reconnecting to a node from a list of active nodes.
• MySQL nodes: MySQL can be configured with master/slave replication. As failover in such clusters is automatic, GHTorrent does not include any special support, at the cost of single transaction lost (it will be retried as the transaction failure will trigger a requeuing of the message that initiated it).

Recommended setup

To setup a GHTorrent cluster, at least 2 nodes are required. With one node, chances are that events might be lost during periods of extreme Github load. The more the available nodes, the more data can be collected and the more resilient to external errors or mirroring script bugs the cluster will be. The following sections assume a Debian-based operating system.

Setting up software dependencies

Debian stable's versions of RabbitMQ and MongoDB are ancient. Fresh versions need to be installed from the repositories provided by VMWare and 10gen, respectively. The MySQL version on Debian stable is adequate for the purposes of GHTorrent.

We describe briefly how each system can be setup. For more advanced scenarios (clustering etc) please refer to each software's manual.

MongoDB

• Add MongoDB repository and install MongoDB v > 2.0

sudo echo "deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen" >/etc/apt/sources.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
sudo apt-get install mongodb-10gen

• Create a MongoDB ghtorrent account

$ mongo admin
> db.addUser('ghtorrent', 'ghtorrent')
> use github
> db.addUser('ghtorrent', 'ghtorrent')

RabbitMQ

• Add RabbitMQ repository and install RabbitMQ v > 2.8

sudo echo "deb http://www.rabbitmq.com/debian/ testing main" > /etc/apt/sources.list
sudo wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc; apt-key add rabbitmq-signing-key-public.asc
apt-get install -t=testing rabbitmq-server

• Configure a RabbitMQ ghtorrent user

$ rabbitmqctl add_user ghtorrent YOUR_PASSWORD
$ rabbitmqctl set_permissions -p / ghtorrent ".*" ".*" ".*"

# The following will enable the RabbitMQ web admin for the github user
# Not necessary to have, but good to debug and diagnose problems
$ rabbitmq-plugins enable rabbitmq_management
$ rabbitmqctl set_user_tags ghtorrent administrator

MySQL

• Create a MySQL ghtorrent account with localhost access

mysql -u root -p
mysql> create user 'ghtorrent'@'localhost' identified by 'ghtorrent';
mysql> create database ghtorrent;
mysql> GRANT ALL PRIVILEGES ON ghtorrent.* to ghtorrent@'localhost';
mysql> flush privileges;

Setting up a GHTorrent node

The following steps will create a new GHTorrent cluster node. To ensure that the mirroring operations will continue even if there are bugs in the code, we use the supervise program (from D.J. Bernstein's daemontools package) which monitors processes and restarts them in case of errors.

• Install the necessary dependencies

apt-get update
apt-get -y install ruby rubygems git daemontools screen sudo

• Add a user for the mirroring operations.

adduser ghtorrent

• Install ghtorrent

sudo gem install ghtorrent

• Select whether the node will be a data retrieval or event mirror node and setup process supervision:

mkdir ghtorrent

ln -s /var/lib/gems/1.8/bin/ght-mirror-events ghtorrent/run

# or

ln -s /var/lib/gems/1.8/bin/ght-data-retrieval ghtorrent/run

• Use the config.yaml file to configure the mirroring tools with the connection details for MongoDB and RabbitMQ. If MongoDB and/or RabbitMQ hosts run in a separate network from the host you are currently configuring (or even on the same network, since MongoDB does not provide for SSL encrypted sockets), you can use ssh port forwarding to setup a secure channel between the local machine and the MongoDB/RabbitMQ host(s). To do so, create a github user on the machine where MongoDB/RabbitMQ runs and do the following:

ssh -fN -L 5672:rabbithost:5672  ghtorrent@rabbithost ls
ssh -fN -L 27017:mongohost:27017 ghtorrent@mongohost ls

The created ports will not be deleted after the controlling terminal exits.

• Start mirroring

supervise ghtorrent

To keep the mirroring session open on terminal disconnects, use screen or an equivalent program.