PARADOX
FIND WHAT’S HIP AND WHERE, WITH THE FRIENDSHIP PARADOX
Our journey into building the PARADOX app at Data61
For the complete story, please visit Manuel’s Medium page.
Our research demonstrated the potential to apply the Friendship Paradox and the new techniques developed by computational social science to derive very real social benefits, but these benefits can only be realized with systems to ingest, process and analyze large volumes of data, along with sophisticated software tools and expertise to extract valuable and actionable information.
We have managed to combine the power of cloud computing and smartphones to create an iPhone app, PARADOX, that can analyse Twitter feeds and exploit the Friendship Paradox to deliver useful information to people who would like to be better connected socially.
PARADOX works for seven cities — Los Angeles, Melbourne, New York, San Francisco, Singapore, Sydney and Tokyo. It analyses the Twitter feed from people located in these cities, along with other social network feeds, and uses these data to locate trending venues and other social developments before they become mainstream.
We have spent about six months developing the app and faced quite a few challenges. Twitter makes available to external parties one percent of all tweets and also the approximately one percent that contain location information — these are tweets where the tweeter has attached geotagging information. All up, we are getting between two and three million tweets every day from all seven cities. We tried to expand the city list, but the computational capacity we had available could not accommodate it, nor did we have access to storage sufficient for the volume of tweets that would have collected. We need to buy more servers!
What Twitter does not provide, and what is essential to the application of the Friendship Paradox, is any information on who follows whom on Twitter. We used our own analysis to infer these networks, and now has a database of more than one billion links between Twitter users.
Although the app shows only trending locations in the seven cities, We have information on more than 5,000 locations across these cities. A possible enhancement to the app would be to allow users to select which cities they want information on.
The app revealed the power of the Friendship Paradox. Locations could be easily divided into three categories based on their propagation patterns on Twitter (two distinct patterns are shown in the figure above):
- Adventure spots which hip people were always the first to discover, followed by ‘the crowd’;
- Popular spots that both hip people and the crowd visited at roughly the same time;
- Tourism places that hip people and the crowd visited at roughly the same time, but where the sizes of sensor groups — the better-connected people — and randomly selected control groups were comparably smaller than the total number of visitors. For instance, a tourism place could have 10,000 individual visitors every week, only a very small portion of whom had social links on Twitter, say 1%, i.e., 100 users. Because the target users of PARADOX are ‘hipsters’, those tourism places are not shown.