The following passage from Slate’s “The Year In Push Alerts” touched a nerve with me:

As a computer programmer, White said, he gets frequent 30-second breaks while the software he’s working on is loading, rendering, and searching—and during those tiny intervals he feels helplessly drawn to the news. “[I’ll see a] tweet about some bizarre behavior… Look at the article. Click through to another article. Post that on Twitter. Get a like. Look at that person’s feed. See another take on how awful Trump is. Click on it. Feel guilty. Try to focus on work. Someone walks into my office and says, ‘Can you believe that Gorsuch says …’ And so on.” During the first few months of the administration, White said, he was losing approximately half of his work time falling down these miserable holes.

For me, this pattern started to take hold during the immediate aftermath of the Brexit vote and I’ve had to work hard to break out of it. It’s part of the reason why I stopped being a smartphone person and took a step back from Twitter.

Oh yes, mentioning Twitter reminds me of this thing I saw about Twitter, on Twitter. It’s depressingly accurate and makes me even more keen to run screaming from the noisy blue bird:

“How a day on British Twitter works”, by @TechnicallyRon. Click for full version

There was a time when most of the posts on this blog were about Twitter.

It started more than ten years ago with me writing about how I didn’t get Twitter before later posts revealed my growing enthusiasm for the whole thing. I savoured the endearing banality of strangers on buses, smirked at “brands” who were botching their forays on to the platform (as if I knew any better), and eventually I was posting semi-technical instructions about how to extract data from Twitter (Twitter eventually stopped that working) or share your loved tracks from Last.fm (I don’t know if Last.fm still exists). I even designed the service that put a British rail company on Twitter for the first time.

You might remember that back in the late 2000s and early 2010s many people blithely dismissed Twitter as a tool for nothing more than knowing what Stephen Fry had for breakfast. The harrumphing Twitter sceptics were everywhere with their gruff dismissals of what they saw as lightweight ephemera. I made a point of taking them on and trying to convince them of the platform’s value. If you had told me back then that, within the next decade, the White House would play host to a president who used Twitter as his or her primary mode of communication — that “the first Twitter President” was going to be a thing — I’d have been genuinely excited. “Stick that in your pipe and smoke it!”, I would have said to those sceptics.

In the end the sceptics lost the debate and they lost it hard. Worse, most of them ended up on Twitter themselves, in time to take part in the eventual death spiral that is now well underway. Because Twitter has become a dumpster fire: a raging, unstoppable inferno, casting toxic fumes and molten plastic blobs in all directions.

If this was the mid-2000s still, it wouldn’t be a problem. We’d just move on. If a forum’s admin turned out to be a Nazi sympathiser, or stood by while Nazis polluted all the threads with swastikas and frog cartoons, people would just leave. I’ve been involved in online communities for long enough to have seen numerous such mass migrations, where a critical mass of contributors moved off a mailing list or message board, leading to the whole community embarking on an exodus to the new, unblemished territory. Renewal and rejuvenation of the community usually followed.

Unfortunately it’s hard to imagine this happening today, now that Twitter and Facebook have formed what is essentially a duopoly in online community. Users of Twitter look to its owners to sort out the problems it has, and there’s a lot of anger at the business for its inability or unwillingness to do so. But what is yet to dawn on us, as a group, is that the onus is not on Jack Dorsey to change Twitter; the onus is on us to do what we would have done in the mid-2000s and just leave.

This is easier said than done though, because leaving Twitter at the moment feels like leaving the internet itself. It’s like you’ve made yourself invisible. And besides, Twitter remains the best place to go to complain or joke about how bad Twitter has become.

So I’ve resolved to start avoiding Twitter as much as I can, just for my own personal reasons. I could do without the feelings of anxiety, rage, frustration and depression that come from scrolling through the timeline that has become the moronic inferno made real.

But can I find a way to step back from Twitter without stepping back from the internet? If I find enough other people who are trying to do the same thing, then perhaps I will.

Kevin at Strange Attractor on what the tactics of Gamergaters tell us about the future of online political discourse.

When you look at the techniques being used by some of these groups, you quickly get a sense of how the next partisan political scorched earth campaign will be fought. Sockpuppets will become the weaponised drones of popular opinion, amplifying marginal views so that they swamp mainstream opinion.

The article’s broadly pessimistic tone resonates with me, but I feel more positive when thinking about the countermeasures that could protect against these tactics. The Eliza chatbot set up to engage these sock puppets is just one example, acting more as chaff—attracting attackers and wasting their resources—than as an offensive weapon.

If ideas like that proliferate and evolve we might just escape a future where we have to abandon the web to swarms of hate-spewing bots.

Update (5th December 2017): Several years ago, Twitter changed its API in a way that completely broke the process I describe below. I don’t know how you’d do the same thing today. It would probably help if you were some kind of white supremacist, going by where Twitter’s moral compass seems to be pointing.

Last weekend I was looking for ways to extract Twitter search data in a structured, easily manageable format. The two APIs I was using (Twitter Search and Backtweets) were giving good results – but as a non-developer I couldn’t do much with the raw data they returned. Instead, I needed to get the data into a format like CSV or XLS.

Some extensive googling led me to this extremely useful post on Labnol, where I learnt about how to use the ImportXML function in Google Spreadsheets. Before too long I’d cracked my problem. In this post I’m going to explain how you can do it too.

Data you can extract from Twitter

This walkthrough will teach you how to extract two types of Twitter data using Google Spreadsheets – tweets and links.

Tweets are extracted using the Twitter Search API in conjunction with ImportFeed. This allows Twitter search results to be extracted into a spreadsheet format.

Links are extracted using the Backtweets API in conjunction with ImportXML. The Backtweets API allows you to find any links posted on Twitter even if they’ve been shortened using services like bit.ly or tinyurl.

I’m in a hurry, can I just do this right now?

If you just want to do it – instead of learn how to do it – just open this Google spreadsheet I’ve created.  You’ll need to make your own local copy so you can edit it. Instructions can be found in the spreadsheet itself.

How to extract tweets containing links

The instructions below will help you create a Google Spreadsheet that pulls in and displays the time, username and text of all tweets containing links to a specified page. Because it uses Backtweets, these tweets will be retrieved even if they used shortened URLs from services like bit.ly or tinyurl.

1. Create a new spreadsheet in Google Documents.
2. Enter column labels in this order: “Search criteria”, “Timestamp”, “Username” and “Tweet text” in cells A1 to D1.
3. In cell B2, underneath Timestamp, insert the following formula:

=ImportXML("http://backtweets.com/search.xml?itemsperpage=100&since_id=1255588696&key=key&q="&A2,"//tweet_created_at")

4. In cell C2, underneath Username, insert the following formula:

   =ImportXML("http://backtweets.com/search.xml?itemsperpage=100&since_id=1255588696&key=key&q="&A2,"//tweet_from_user")

5. In cell D2, underneath Tweet Text, insert the following formula:

   =ImportXML("http://backtweets.com/search.xml?itemsperpage=100&since_id=1255588696&key=key&q="&A2,"//tweet_text")

6. Now paste a search query into cell A2 – say, http://www.google.com. After a few seconds, you should see columns B, C and D fill up with tweets, looking something like the image below:



7. The formulas pasted into cells B2, C2 and D2 all reference the URL in cell A2. This means that whenever you paste anything new into A2, the search results should refresh.
8. Also, you can paste parts of URLs into A2 – not just entire ones. This is useful for seeing all links to a specific directory on your site, for example.

Finally, this tool can only extract 100 results at a time – but it is possible to set it up to retrieve more than that. Look at my sample Google Spreadsheet if you want to do this.

Extracting tweets from Twitter search results

The method for doing this is identical to the above, but uses the ImportFeed function instead of ImportXML.

1. Create a new spreadsheet in Google Documents.
2. Enter column labels in this order: “Search criteria”, “Timestamp”, “Username” and “Tweet text”. For the rest of this walkthrough, I’m going to assume that these labels are in cells A1 to D1, but in reality you can put them wherever you like
3. In cell B2, underneath Timestamp, insert the following formula:

   =ImportFeed("http://search.twitter.com/search.atom?rpp=20&page=1&q="&A2, "items created")

4. In cell C2, underneath Username, insert the following formula:

   =ImportFeed("http://search.twitter.com/search.atom?rpp=20&page=1&q="&A2, "items author")

5. In cell D2, underneath Tweet Text, insert the following formula:

   =ImportFeed("http://search.twitter.com/search.atom?rpp=20&page=1&q="&A2, "items title")

6. Type a search query into cell A2 - say, "Hoth." Hit enter and the results will load. It should look something like this:
7. Things will go wrong if you insert characters like # or @ into the search query. To get around this, type %23 instead of # and %40 instead of @. This will allow you to search for hash tags and usernames.

I haven't been successful in generating more than 20 search results per request, but you can get around this using the page number parameter in the ImportFeed query string. See my own Google spreadsheet to find out how to do this.

I hope these instructions are useful - if you have any comments, questions or feedback, please let me know in the comments.