Is RSS the “vinyl” of digital media?
Jan 26th
For large stretches of my life, I’ve allowed my obsession with music to burn up huge chunks of my time as well as my money. Illness, poverty, hangovers, rain – none of these things would stop me leaving the house and spending whole weekends wandering London, going from record shop to record shop. Over time my vinyl collection grew while my bank balance fell, but I didn’t mind – because that collection of vinyl was (and still is) valuable in lots of ways. I didn’t just enjoy listening to those records – I also enjoyed playing them out. I played in clubs, made compilation tapes and distributed mixes over the internet.
My vinyl collection helped me evangelise the music I loved to like-minded people. And before the worlds of music and the internet collided back in 1999, this sort of behaviour occupied a useful niche in the music ecosystem. Vast numbers of releases, especially in genres that flew under the radar of mainstream promotion, were filtered, curated and recompiled, helping normal people – who had better things to do than waste their lives exploring dusty record shops or compiling mixtapes in their bedrooms – explore obscure fields of new music. In this way vinyl kept influencing the public’s relationship with music long after it stopped being a mainstream format.
It’s a common mistake, especially when thinking about media formats, to see things in a binary way where the only two states are ubiquity and death. Many made this mistake when vinyl was eclipsed by the CD, thinking that its death was just around the corner. But this thinking was wrong. Although vinyl sales fell, its role remained important and it still is today – in fact, vinyl sales in the US actually increased by 33% in 2009.
RSS, unlike vinyl, isn’t a formerly dominant format that’s finding a smaller niche. Instead, it’s a new format that’s failed to go mainstream: usage of RSS readers is in decline and Twitter is supplanting it as a mass-market feed delivery channel. But there are definitely similarities between the formats, and the role they play in their respective ecosystems.
You can’t ask mainstream users whether or not they use RSS in their daily course of Internet usage any more than you can ask the average couch potato whether or not they use Cathode Ray Tubes or Liquid Crystal Displays – Mashable, October 2008
Not everyone wants to get to grips with concepts like Atom or OPML, learn how to use an RSS reader and incorporate it into their daily routine. That’s understandable: I know lots of voracious online readers who’ve never got to grips with RSS. Similarly, many people in the 1990s, despite loving music genres that released mainly on vinyl, didn’t want to join the anorak-wearing record shop brigade and start buying expensive import 12″s.
But for media owners (whether websites or record labels) that vinyl-buying, RSS-reading audience is worth reaching if only because they’re in the habit of evangelising. A heavy RSS user is more likely to run their own website on which they’ll compile and re-publish that content, just as turntable owners are more likely to create mixes that showcase obscure records to a larger audience. RSS heavily influences how information moves online, and plays an indirect role in shaping the online experiences even of those who have no idea what it is.
So even if RSS is never destined to become a mainstream format for delivering content online, reports of its death will prove to be greatly exaggerated. The internet needs a format which, like vinyl, appeals to the obsessives and whose very nature encourages compilation and re-transmission.
UK unemployment drops… unexpectedly?
Jan 21st
The UK Office for National Statistics announced yesterday that unemployment had dropped for the first time in 18 months. BBC News reported this as a “surprise”:
The number of people unemployed in the UK has fallen unexpectedly for the first time in 18 months… George Buckley [of Deutsche Bank] admitted previous predictions of the unemployment rate reaching 10% now looked unrealistic… [The figures] came as a surprise to many analysts.
But to regular readers of this blog, this news is anything but unexpected: in December 2009, my analysis of unemployment-related search trends clearly indicated that the unemployment rate was about to fall.
So could this be an example of search trends providing early insight into economic data? Possibly, but it’s only one month’s figures we’re talking about. A sustained track record of successful projection is needed to demonstrate that search analysis can yield valuable insights.
Over 2010 I’ll be keeping an eye on the data to see what happens. In the meantime, if you can think of other real-world metrics that might be suitable subjects for search trends analysis, get in touch.
Can search predict the future?
Dec 23rd
Today, we often search for information about upcoming major events in our lives – both good and bad – before we experience them. When facing financial difficulty or unemployment, many of us will go online at the earliest opportunity to look for help and guidance. And when we’re considering major financial decisions such as buying a house, search engines are usually consulted before estate agents are called.
Traditional economic reports, on the other hand, look at events that have taken place. Unemployment figures tell us how many people are claiming benefits rather than how many people have been put at risk of redundancy. Average house prices are based on completed transactions, not how many people are currently looking to buy. So while we can be fairly confident of these reports, they don’t provide us with particularly current insights.
This trade-off between confidence and currency was, in the past, largely academic as analysing current data was almost impossible. But in the age of the real-time web, this might be about to change: maybe patterns in search behaviour can give us a glimpse of future patterns in the economy.
We first became interested in this topic back in spring 2009, so we analysed search patterns for two sets of keywords as the UK economy went into recession. We looked for relationships between these search patterns and related economic indicators, and listed some tentative predictions based on what we observed.
House prices
In April 2009, we looked at volumes for 23 keywords that homebuyers might use, including buying a home, cheap mortgage and mortgage providers. UK search volumes for these keywords were then compared to house prices.
House prices charted against search volumes for 23 related keywords, from January 2004 to April 2009. Sources: Nationwide, Google Insights
Searches typically decline as autumn ends before rebounding in January. But in 2008, the January rebound was lacklustre and the decline came in spring – much earlier than usual. This was in line with house prices, which peaked in late 2007 and dropped severely from spring 2008.
In the first few months of 2009, however, search volumes enjoyed a far stronger January rebound than in the previous year – so we hypothesised that house prices would bottom out or even start to rise again in the middle of 2009. Let’s look at how accurate that hypothesis turned out to be.
House prices charted against search volumes for 23 related keywords, from January 2004 to December 2009 Sources: Nationwide, Google Insights
Sure enough, the search volume resurgence was accompanied by house price growth throughout 2009. But you’ll notice that search volumes soon tapered off, with a particularly steep fall after August. Our revised hypothesis, then, is that house prices will initially plateau and then drop again. We’ll revisit the statistics in spring 2010 to see how things turn out.
Financial difficulties
The second set of keywords we analysed was related to impending financial difficulties such as joblessness, debt and insolvency. They included signing on, mortgage arrears and debt problems, and were compared to the UK jobless rate.
UK unemployment rate charted against search volumes for 24 related keywords, from January 2004 to April 2009 Sources: Office for National Statistics, Google Insights
These search volumes dip at the end of each year before rising in January – and the rise in early 2008 was more pronounced than in previous years. The jobless rate started climbing three months later, suggesting that in this case search patterns might anticipate economic statistics. We observed that search volumes had dropped significantly in the first few months of 2009, so our hypothesis was that the jobless rate would stabilise but not drop between April and July. The chart below shows what actually happened.
UK unemployment rate charted against search volumes for 24 related keywords, from January 2004 to April 2009 Sources: Office for National Statistics, Google Insights
The unemployment rate has indeed stabilised, wavering between 7.7% and 7.8% since early June, suggesting that our original hypothesis was valid. And search volumes have kept on dropping throughout 2009. If search trends do anticipate economic reports in this case, we should see the unemployment rate drop steadily between now and spring 2010. Again, we’ll revisit these figures in April to see if this happens.
Conclusion
Our hypotheses from April 2009 were largely borne out as the year progressed: the drop in house prices was reversed and unemployment rates stabilised. So maybe there is some truth to the notion that search patterns can shed some light on forthcoming economic change.
But these hypotheses were in tune with the economic mood of the time. Many commentators were talking about green shoots and a V-shaped recession – there was a feeling that recovery was just around the corner. Today, we remain in what has become the longest-running recession in recorded history and there is considerable uncertainty about what 2010 will bring.
Our new hypotheses are less likely to be tainted by current economic consensus, precisely because no real consensus seems to exist right now. For this reason, the idea of search predicting the future will be seriously tested as the year unfolds. Don’t forget to come back in April 2010 to see the results for yourself.
An open assault on the walled garden
Dec 21st
Mobile telcos charge us for the texts, minutes and megabytes we use. They buy our loyalty by heavily subsidising our increasingly expensive phones. And they’re terrified of becoming like the people who supply our electricity or gas. They’re terrified that one day they’ll be nothing but interchangeable providers of a commodity, irrelevant logos printed on tedious, humdrum bills.
This is why their marketing focuses so much on music, culture and lifestyle. It’s why O2 customers get priority tickets to concerts at the arenas bearing their name. It’s why Orange customers get half-price cinema tickets on Wednesdays. And it’s why T-Mobile runs that insufferable campaign about Josh and his ever-growing band.
Join the b(r)and: T-Mobile want to be associated with music and lifestyle
Customers are being encouraged to associate the brands of mobile operators with a certain type of lifestyle experience instead of just voice and data. This experience extends from the marketing to exclusive content services and even the interfaces and feature sets of the handsets themselves.
In this sense, mobile telcos are offering their customers a walled garden, in which the mobile internet is presented as part of a convenient package branded Orange, AT&T, T-Mobile or O2. If your internet memory goes back as far as the mid-1990s this might sound slightly familiar. But in the next ten years this walled garden is due to come under direct assault.
Charlie Stross has posted an excellent, thought-provoking piece looking at how the next ten years might pan out for the mobile industry – and making it sound in some ways like a technology rehash of the Great Game, with Apple and Google as the chief protagonists.
As Stross sees it, Apple and Google both want to destroy the walled garden built by telcos but for different reasons and in different ways. As a premium marque, Apple wants to work with telcos while preventing their brands from adulterating the Apple experience:
Apple don’t want to destroy the telcos; they just want to use them as a conduit to sell their user experience… [they] want to maintain the high quality Apple-centric user experience and sell stuff to their users through the walled garden of the App Store and the iTunes music/video store
Google, on the other hand, wants people to view more of its ads. To make this happen, Google wants to fundamentally reshape the mobile industry:
I think Google are pursuing a grand strategic vision of destroying the cellco’s entire business model… turning 3G data service into a commodity… getting consumers to buy unlocked SIM-free handsets [like the Nexus One]… and ultimately do the Google thing to all your voice messages [through Google Voice] as well as your email and web access.
These distinct strategies both threaten the mobile telcos, who stand to lose any emotional connection they have with their customers either way. But this doesn’t mean that Apple and Google are going to be bedfellows:
Apple’s iPhone has been good for Google: iPhone users do far more web surfing — and Google ad-eyeballing — than regular phone users. But Apple want to maintain… the walled garden of the App Store and iTunes… [and] Google can’t slap their ads all over those media. So it’s going to end in handbags at dawn … eventually.
The piece (here’s the link again by the way) has me thinking that the coming decade in mobile networks will be much like the previous decade was in land-line internet service provision.
If Charlie Stross is right, the idea of the telco as provider of an experience will not last the decade, meaning that flash mobs, Orange Rock Corps and Josh Ward will become nothing but a dim and distant memory. And customers will hopefully have greater choice over how they use mobile networks, which would be nothing but a good thing in my opinion.
Are mobile apps here to stay?
Dec 17th
A few weeks ago a guest speaker came to our office to talk about mobile apps. His company produced a lot of them, for pretty big brands. He knew his stuff: the team here was both impressed and engaged.
But an exchange during the following Q&A session stuck in my mind later. One of our directors asked a question: is the mobile app destined to be a transitory phenomenon, something that will fade away as mobile browsers become capable of delivering the same functionality?
The speaker was adamant that this was not the case and that mobile apps were here to stay. He felt that Google’s increasing preference for mobile browser apps over native apps was misguided and that Google were wrong on this one. Mobile browsers were so far from rivalling the functionality of native apps that it wasn’t even worth thinking about.
I was tempted to counter this point by bringing up the iPhone’s support for HTML 5 and starting a detailed discussion about in-browser capabilities. But this wasn’t the main subject of the talk and I’m in no way an expert on HTML 5, so I decided to keep my mouth shut instead.
In the weeks since the talk, however, I’ve often found myself turning this question over and over again in my head. And the more I think about it, the more I feel that mobile apps are basically doomed – or at least I hope they are.
Don’t get me wrong – they play an important role. It’s good that so many people today see phones as devices for more than just calling or texting, and the iPhone and its suite of native apps is largely to thank for this. But in the longer run, the publication and distribution model they are based on has to go.
The idea of tying software to a single hardware platform is anachronistic, uncompetitive and limits user choice. This is bad enough when you’re dealing with computers, but it’s even worse when the devices are as personal as mobile phones. People should be free to choose a different phone without needing to buy new versions of the software tools that have become integral to their lives.
Aside from user choice, there’s a more practical reason why the native app model is unsustainable. Developers won’t want to keep maintaining multiple codebases for the apps they produce, especially when there’s the option of building an equally functional in-browser app which any standards-based client can run. And although Apple might hope to render this point irrelevant by establishing monopolistic domination of the smartphone market, relieving developers of the need to consider other platforms, current research indicates that they won’t succeed.
The smartphone OS market will be more fragmented in 2012 than in 2009
A more fragmented smartphone OS market will increasingly compel developers to support separate codebases for Windows Mobile, RIM, Android, Symbian and the iPhone. But as mobile browsers become capable of delivering similar interactivity, serious developers will become inclined to start using the browser as the platform instead. This will be a good thing for users and the industry alike.
If I’m correct and native apps do fade away over time, we may look back on the era of pointless mobile apps as just one among many strange blips in the history of technology. But despite some early rumblings from notable developers, native mobile apps will be with us for some time yet – and, in the medium term at least, they still have an important role to play in encouraging mainstream adoption of the mobile internet.
Edit: This article was later reposted on Android and Me and attracted numerous comments. Click here to see the conversation on Android and Me
Edit 2: Stephen Fulljames shared a couple of links related to this post. PhoneGap is a toolkit for developing mobile apps in HTML & JavaScript. And this post from front-end consultant Peter-Paul Koch provides some background to his work with Vodafone on mobile browser compatibility and W3C widgets.
How to post your Last.fm loved tracks to Twitter
Dec 8th
I remember when Twitter was still quite new. Back then, a lot of people were still trying to think of uses for it and one thing that was fairly common was to plug it into your Last.fm account.
In retrospect I can see why that was seen as a good idea. Twitter was supposed to be about broadcasting minor ephemeral details, and the music you were currently listening to definitely fell into that category. But there was a downside. People listen to a lot of music and, with a Twitter post for each track played, that added up to a lot of useless information on Twitter. Thankfully, the practise of scrobbling directly to Twitter soon faded out.
Today there are some more useful and less irritating ways of posting information from Last.fm (or, indeed, its open source alternative Libre.fm to your Twitter account. One of them, Tweekly.fm, produces an automated weekly tweet of your top three artists. Another one, which I’m going to explain here, involves posting tracks that you “love” on Last.fm to your Twitter account.
Here’s how it works:
- If you don’t have a Last.f account, create one here
- Get the URL of your “Loved tracks” RSS feed. This is easy: just change “USERNAME” in the URL below for your Last.fm username.
http://ws.audioscrobbler.com/2.0/user/USERNAME/lovedtracks.rss
- Test the URL by opening it in a browser. You should see something that looks a bit like this:
- If it works, go to Twitterfeed.com and create an account if necessary
- Once logged in to Twitterfeed, click on the “Create new feed” button to the top-right of the screen
- In “Step 1: Send Feed To”, select Twitter. Click on the large “Authenticate Twitter” button and enter your Twitter account details. You’ll then be directed back to Twitterfeed.com
- In “Step 2: Name feed & source URL”, enter a name for the feed – this can be anything you like. In the “RSS Feed URL” field, paste the URL of your RSS feed
- Click on the “test feed” button to make sure the feed is valid
- Click “Advanced settings”. A bunch of new options will appear underneath. Here’s a screenshot with the things you need to check circled in red:
- In “Post content”, select “Title Only”. This will ensure that the posts to your Twitter account only contain the artist, title and shortened URL to the track you loved
- Make sure “Post link” is checked and a URL shortening service is selected
- You might also want to enter some text in the “Post Prefix” or “Post Suffix” fields, otherwise your tweets might be slightly baffling
- You’re done – just click “Create feed” and that’s it set up.
Now whenever you “love” a track on Last.fm, your Twitter account will post a link to it. This makes Last.fm’s “love” feature a bit more useful when it comes to recommending music to other people – especially people who don’t use Last.fm. And as long as you don’t love everything you listen to you won’t be clogging up your Twitter feed.
Using Google Spreadsheets to extract Twitter data
Nov 20th
Last weekend I was looking for ways to extract Twitter search data in a structured, easily manageable format. The two APIs I was using (Twitter Search and Backtweets) were giving good results – but as a non-developer I couldn’t do much with the raw data they returned. Instead, I needed to get the data into a format like CSV or XLS.
Some extensive googling led me to this extremely useful post on Labnol, where I learnt about how to use the ImportXML function in Google Spreadsheets. Before too long I’d cracked my problem. In this post I’m going to explain how you can do it too.
Data you can extract from Twitter
This walkthrough will teach you how to extract two types of Twitter data using Google Spreadsheets – tweets and links.
Tweets are extracted using the Twitter Search API in conjunction with ImportFeed. This allows Twitter search results to be extracted into a spreadsheet format.
Links are extracted using the Backtweets API in conjunction with ImportXML. The Backtweets API allows you to find any links posted on Twitter even if they’ve been shortened using services like bit.ly or tinyurl.
I’m in a hurry, can I just do this right now?
If you just want to do it – instead of learn how to do it – just open this Google spreadsheet I’ve created. You’ll need to make your own local copy so you can edit it. Instructions can be found in the spreadsheet itself.
How to extract tweets containing links
The instructions below will help you create a Google Spreadsheet that pulls in and displays the time, username and text of all tweets containing links to a specified page. Because it uses Backtweets, these tweets will be retrieved even if they used shortened URLs from services like bit.ly or tinyurl.
- Create a new spreadsheet in Google Documents.
- Enter column labels in this order: “Search criteria”, “Timestamp”, “Username” and “Tweet text” in cells A1 to D1.
- In cell B2, underneath Timestamp, insert the following formula:
=ImportXML(“http://backtweets.com/search.xml?itemsperpage=100&since_id=1255588696&key=key&q=”&A2,”//tweet_created_at”)
- In cell C2, underneath Username, insert the following formula:
=ImportXML(“http://backtweets.com/search.xml?itemsperpage=100&since_id=1255588696&key=key&q=”&A2,”//tweet_from_user”)
- In cell D2, underneath Tweet Text, insert the following formula:
=ImportXML(“http://backtweets.com/search.xml?itemsperpage=100&since_id=1255588696&key=key&q=”&A2,”//tweet_text”)
- Now paste a search query into cell A2 – say, http://www.google.com. After a few seconds, you should see columns B, C and D fill up with tweets, looking something like the image below:
- The formulas pasted into cells B2, C2 and D2 all reference the URL in cell A2. This means that whenever you paste anything new into A2, the search results should refresh.
- Also, you can paste parts of URLs into A2 – not just entire ones. This is useful for seeing all links to a specific directory on your site, for example.
Finally, this tool can only extract 100 results at a time – but it is possible to set it up to retrieve more than that. Look at my sample Google Spreadsheet if you want to do this.
Extracting tweets from Twitter search results
The method for doing this is identical to the above, but uses the ImportFeed function instead of ImportXML.
- Create a new spreadsheet in Google Documents.
- Enter column labels in this order: “Search criteria”, “Timestamp”, “Username” and “Tweet text”. For the rest of this walkthrough, I’m going to assume that these labels are in cells A1 to D1, but in reality you can put them wherever you like
- In cell B2, underneath Timestamp, insert the following formula:
=ImportFeed(“http://search.twitter.com/search.atom?rpp=20&page=1&q=”&A2, “items created”)
- In cell C2, underneath Username, insert the following formula:
=ImportFeed(“http://search.twitter.com/search.atom?rpp=20&page=1&q=”&A2, “items author”)
- In cell D2, underneath Tweet Text, insert the following formula:
=ImportFeed(“http://search.twitter.com/search.atom?rpp=20&page=1&q=”&A2, “items title”)
- Type a search query into cell A2 – say, “Hoth.” Hit enter and the results will load. It should look something like this:
Things will go wrong if you insert characters like # or @ into the search query. To get around this, type %23 instead of # and %40 instead of @. This will allow you to search for hash tags and usernames.
I haven’t been successful in generating more than 20 search results per request, but you can get around this using the page number parameter in the ImportFeed query string. See my own Google spreadsheet to find out how to do this.
I hope these instructions are useful – if you have any comments, questions or feedback, please let me know in the comments.
Readability of online text
Nov 10th
I’ve been trying to codify some guidelines for writing for the web recently, and came across this study (PDF) by Wichita University’s Software Usability Research Laboratory in 2005.
The study involved 66 graduate students with either normal or corrected vision being given a short story to read online. A preliminary reading test was carried out on participants so the study could predetermine their reading speed. Different text layouts were used, such as multiple column, full justification and so on. Study participants were tested for both reading speed and reading comprehension.
- Reading speed: Multiple-column layouts impaired reading speed when text was left-justified. However, left-justified text was read more quickly in a single column layout than full-justified text. The highest reading speed was 269.33 words per minute for two-column, full-justified text.
- Reading comprehension: No significant variation was found across the different text formats.
- Fast versus slow readers: Faster readers benefited most from the 2-column, fully-justified layout. Slow readers benefited from 1-column, left-justified text.
The study was perhaps limited by the fact that the participants, as undergraduates, were heavier readers of online text than the average member of the population. I’d be interested to see if any similar studies have been carried out with a larger sample size, broader age range and a more representative mix of internet ‘natives’ versus internet ‘newbies’. Does anyone know of any? If I find some I’ll post them here.
links for 2009-11-06
Nov 6th
-
Website that digs up and displays conversations between any two Twitter users. Handy for stalking & memory-jogging. I like the fact that it bugs you for money while loading up tweets.
-
T-Mobile (US) have released Android statistics. myTouch owners seem keen on browsing, apps and so on – 80% of them use the browser once a day, and 66% use it more than once a day. Indicates that smartphone buyers make good use of their features
links for 2009-10-30
Oct 30th
-
Visual timeline of internet meme history, to before the creation of the first emoticon. If you ask me, though, the fun doesn't really begin until 1990 when the term Godwin's Law was first coined. Of course everything went haywire in 1993 once Eternal September began.
