I’ve managed to get all of the various components wired in the right locations. Data is now flowing from Twitter into RabbitMQ, through a pair of workers, past a sentiment analysis API, and through to an in-browser visualization. The results are gorgeous!


The animation above is just a preview of what the data looks like. Because I don’t have access to the whole Twitter firehose, I’m limited to just a subset of the sample data (roughly 1% of total traffic). I then further limit myself to just English tweets so the sentiment analysis API works appropriately. Then, because we’re looking now at only 1% of 20% of the data, I limit even further with a geographic bounding box around North America.

Not every tweet has geolocation data, so I can only plot the ones that do. This is a fun chart showing relative, geographical sentiment, but unfortunately isn’t enough data at a high enough bitrate to really satisfy my itch.

I might try to take this a bit further and do some keyword latency/frequency mapping with sentiment as well – think of a live tag cloud where terms grow with usage – so I won’t be bound to geolocation …


Browser-based Visualization

For the sake of simplicity, I’m just flat out stealing the markup and styling from Automattic’s activity visualization. It’s relatively simple, and the heaviest components are the synchronization from the server to the browser. Automattic is doing AJAX pulls of cached information, I’ll be instead opening a persistent websocket and dispatching data across the wire.


Just realized I forgot to wire the workers to the socket …

Oh how I hate hidden dependencies! It seems the PHP library for RabbitMQ interaction requires the bcmath extension. Unfortunately, none of the stock PHP7 containers were compiled with bcmath support, so I’m building my own forked from the stock Alpine container. The only difference is a tiny little “–enable-bcmath” flag.


I’ve got Docker Compose set up with RabbitMQ (using the management plugin so I can see how many messages get queued up). Next step is to script and Dockerize the PHP thread that ingests the firehose.

I’ve decided to create a GitHub repo for the project: ericmann/fireviz. It will likely be subdivided into different “iteration” folders, each with its own docker-compose config so I can play with different language setups.