{"id":4993,"date":"2017-03-24T16:13:45","date_gmt":"2017-03-24T21:13:45","guid":{"rendered":"http:\/\/blog.lib.uiowa.edu\/studio\/?p=4993"},"modified":"2018-10-31T13:50:23","modified_gmt":"2018-10-31T18:50:23","slug":"subreddit-algebra","status":"publish","type":"post","link":"https:\/\/blog.lib.uiowa.edu\/studio\/2017\/03\/24\/subreddit-algebra\/","title":{"rendered":"Subreddit Algebra"},"content":{"rendered":"
\"\"<\/a>
photo credit: Laura Crossett<\/figcaption><\/figure>\n

Yesterday, FiveThirtyEight featured a fantastic article by Trevor Martin, a Ph.D student in Computational Biology at Stanford University. Martin\u2019s piece, <\/span>Dissecting Trump\u2019s Most Rabid Online Following<\/span><\/i><\/a>, looked at the toxic communities surrounding Donald Trump, notably r\/The_Donald, by using a machine learning technique called <\/span>latent semantic analysis<\/span><\/a>. LSA uses\u00a0words and concepts from two sets of documents and shows how closely they are related. Martin used this process to find the overlap between different subreddits; two different subreddits are more similar if users comment in both. He then goes further to use what he calls \u201csubreddit algebra\u201d. By adding or subtracting the subreddits together, other related subreddits can be revealed. For example, r\/nba + r\/minnesota = r\/timberwolves. If you\u2019re interested in semantic vector math, there\u2019s a <\/span>fun twitter bot<\/span><\/a> that does this algebra several times per day.<\/span><\/p>\n

As with all FiveThirtyEight\u2019s data stories, they make their code freely available for readers to try out themselves. I thought it\u2019d be interesting to take a peek at some subreddits that are a little closer to home (and a whole lot less racist and sexist). If you don\u2019t want to run this yourself, feel free to skip to the results below.<\/span><\/p>\n

The Setup<\/b><\/h2>\n

If you want to follow along, you\u2019ll need some familiarity with the <\/span>Google Cloud Platform<\/span><\/a> since that\u2019s where everything will be run. Specifically, you\u2019ll be using their BigQuery service<\/a>, which is a tool for working with massive datasets. You\u2019ll also want to set up a bucket in Google Storage<\/a>. Your outputs will be quite large and they don\u2019t allow you to export directly to your local file system. Finally, you\u2019ll need some basic familiarity with the R language and an environment to run R scripts. <\/span>RStudio<\/span><\/a> is a great tool for this.<\/span><\/p>\n

First, from your Google Cloud console, create a new project to contain the various tables you\u2019ll be generating. Next, head over to <\/span>BigQuery<\/span> and create a new dataset under your project. You could call this something like \u2018reddit\u2019. This will hold your results. You\u2019ll be querying against <\/span>fh-bigquery:reddit_comments <\/span><\/code>set that is made available to you by default. Click on the Compose Query button and use <\/span>this code from the fivethirtyeight GitHub repository<\/span><\/a>. Change line 19 to the path of your own dataset you just created.<\/span><\/p>\n

Take the resulting dataset that this query generates and export it to the storage bucket you created. From there, you can download it as a CSV file.<\/span><\/p>\n

Now, in RStudio, load the <\/span>vector analysis script from the repository.<\/span><\/a> You\u2019ll need to change the path to the CSV file on line 20 to your exported CSV. And, of course, change the various subreddits after line 59. Now the fun begins!<\/span><\/p>\n

The Results<\/b><\/h2>\n

The first obvious search is for similar subreddits to r\/IowaCity<\/a>. What kinds of things do Iowa City folks post about? The higher the number, the more related the subreddits are.<\/span><\/p>\n

Cedarrapids<\/strong> 0.4627451
\n<\/span>Madisonwi<\/strong> 0.4278260
\n<\/span>Uiowa<\/strong> 0.4216467
\n<\/span>Milwaukee<\/strong> 0.4069844
\n<\/span>Homebrewing<\/strong> 0.3992629
\n<\/span>Beer<\/strong> 0.3941419
\n<\/span>Chicago<\/strong> 0.3916151
\n<\/span>Indianapolis<\/strong> 0.3868063
\n<\/span>Iowa<\/strong> 0.3850677
\n<\/span>Smoking<\/strong> 0.3823774<\/span><\/p>\n

Ok, not surprising. Surrounding cities plus beer drinking and smoking meats. Iowa City redditors are a chill bunch. What about the uiowa subreddit?<\/span><\/p>\n

IowaCity<\/strong> 0.4216467
\n<\/span>Mazdaspeed6<\/strong> 0.2913548
\n<\/span>Swimming<\/strong> 0.2766708
\n<\/span>Projectcar<\/strong> 0.2719264
\n<\/span>Madisonwi<\/strong> 0.2699070
\nCartalk<\/strong> 0.2696891
\n<\/span>College<\/strong> 0.2646985
\n<\/span>Cars<\/strong> 0.2642775
\n<\/span>Civilengineering<\/strong> 0.2637309
\n<\/span>Milwaukee<\/strong> 0.2634588<\/span><\/p>\n

I\u2019ll admit, there are a surprising amount of car discussion going on. Perhaps not when you see some of the cars downtown.<\/span><\/p>\n

What happens when we take the uiowa out of Iowa City? IowaCity – uiowa =<\/span><\/p>\n

PoGoIC<\/strong> 0.2447359
\n<\/span>Smoking<\/strong> 0.2135908
\n<\/span>Homebrewing<\/strong> 0.2053004
\n<\/span>BBQ<\/strong> 0.2028280
\n<\/span>Grilling<\/strong> 0.1997918
\n<\/span>Sousvide<\/strong> 0.1983743
\n<\/span>Wine<\/strong> 0.1961068
\n<\/span>Cedarrapids<\/strong> 0.1937385
\n<\/span>Bourbon<\/strong> 0.1917187
\n<\/span>Spicy<\/strong> 0.1895046<\/span><\/p>\n

Iowa City likes to grill out and drink. And play Poekmon Go. Let\u2019s see what librarians are up to. From r\/Libraries<\/a>:<\/span><\/p>\n

Librarians<\/strong> 0.6681721
\n<\/span>Teachers<\/strong> 0.6463503
\n<\/span>Knitting<\/strong> 0.6231567
\n<\/span>Parenting<\/strong> 0.6165957
\nWeddingplanning<\/strong> 0.6118699
\n<\/span>Genealogy<\/strong> 0.6118073
\nWedding<\/strong> 0.6039990
\nFemalefashionadvice<\/strong> 0.6024974
\n<\/span>Crochet<\/strong> 0.6010991
\n<\/span>Vegetarian<\/strong> 0.5975182<\/span><\/p>\n

Congratulations, librarians, on your marriage and children! And your new fiber arts project. What happens when we remove\u00a0the wedding planning from librarians\u2019 reddit posts?<\/span><\/p>\n

Corruption<\/strong> 0.3048685
\n<\/span>HistoryofIdeas<\/strong> 0.2961678
\nCornbreadLiberals<\/strong> 0.2932469
\n<\/span>TrueProgressive<\/strong> 0.2924358
\n<\/span>Scifi<\/strong> 0.2919506
\n<\/span>Media<\/strong> 0.2833257
\n<\/span>WarOnComcast<\/strong> 0.2797392
\nTechNewsToday<\/strong> 0.2789546
\n<\/span>InCaseYouMissedIt<\/strong> 0.2789388
\n<\/span>Obama<\/strong> 0.2774487<\/span><\/p>\n

What other interesting algebra problems\u00a0could we think up? Send me an email<\/a> and I\u2019ll try to post a few next week. After all, it\u2019s Friday and I\u2019m off to drink beer, grill some vegetarian food, and read sci-fi after I\u2019m done parenting for the day. This weekend might be a good time to pick up knitting.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Yesterday, FiveThirtyEight featured a fantastic article by Trevor Martin, a Ph.D student in Computational Biology at Stanford University. Martin\u2019s piece, Dissecting Trump\u2019s Most Rabid Online Following, looked at the toxic communities surrounding Donald Trump, notably r\/The_Donald, by using a machine learning technique called latent semantic analysis. LSA uses\u00a0words and concepts from two sets of documentsContinue reading “Subreddit Algebra”<\/span><\/a><\/p>\n","protected":false},"author":153,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"syndication":[],"_links":{"self":[{"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/posts\/4993"}],"collection":[{"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/users\/153"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/comments?post=4993"}],"version-history":[{"count":11,"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/posts\/4993\/revisions"}],"predecessor-version":[{"id":5007,"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/posts\/4993\/revisions\/5007"}],"wp:attachment":[{"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/media?parent=4993"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/categories?post=4993"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/tags?post=4993"},{"taxonomy":"syndication","embeddable":true,"href":"https:\/\/blog.lib.uiowa.edu\/studio\/wp-json\/wp\/v2\/syndication?post=4993"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}