This python script generates a graph of Youtube by scraping the related channels from the "about" page using requests
and lxml
because unfortunately this data is not provided by the Youtube API.
The gathered data is streamed to Gephi for visualization using the gephistreamer library.
The scraping is coordinated by a size limited, thread-safe python set
as the queue to keep track of the channels to explore. It makes use of multithreading and a simple history to accelerate the exploration.
Example map created in about 14 min of scraping consisting of about 10000 channels. With the ForceAtlas 2 layout in LinLog mode one can observe different youtube communities mostly grouped around one or more central hubs (node color shows its in-degree, red is higher).
For data streaming to Gephi you need the Graph Streaming
plugin available in Gephis plugin manager. When installed you will need to rightclick -> start the streaming master server.