1. Google launches Google Correlate, a new tool to support search trend analysis

    Posted May 25, 2011 in research  |  1 Comment so far

    Yesterday I wrote about this Twitter-based hedge fund, and connected it to the broader area of large-scale online analytics being used to anticipate real-world events. And today Google has announced a new tool, Google Correlate, which has been built to do just that.

    When I was dabbling in this area with search data and unemployment statistics I was using Google Insights, which made the process pretty long-winded – it produced a lot of messy data which only became useful after a few hours of macro-writing in Excel. So it was encouraging to read, in Google’s official post about Correlate, that:

    [T]ools… such as Google Trends or Google Insights for Search weren’t designed with this type of research in mind. Those systems allow you to enter a search term and see the trend; but researchers told us they want to enter the trend of some real world activity and see which search terms best match that trend… This is now possible with Google Correlate, which we’re launching today on Google Labs.

    I’m looking forward to giving Google Correlate a try, from what I’ve read it seems like it still only represents the tip of a very big iceberg, a glimpse through a keyhole into a big world of data that only Google is allowed to explore. Hopefully I’m wrong and it does go deeper than that though. I’ll post more about it when I’ve had a chance to look around…

  2. UK unemployment drops… unexpectedly?

    Posted January 21, 2010 in research, strategy  |  No Comments so far

    The UK Office for National Statistics announced yesterday that unemployment had dropped for the first time in 18 months. BBC News reported this as a “surprise”:

    The number of people unemployed in the UK has fallen unexpectedly for the first time in 18 months… George Buckley [of Deutsche Bank] admitted previous predictions of the unemployment rate reaching 10% now looked unrealistic… [The figures] came as a surprise to many analysts.

    But to regular readers of this blog, this news is anything but unexpected: in December 2009, my analysis of unemployment-related search trends clearly indicated that the unemployment rate was about to fall.

    So could this be an example of search trends providing early insight into economic data? Possibly, but it’s only one month’s figures we’re talking about. A sustained track record of successful projection is needed to demonstrate that search analysis can yield valuable insights.

    Over 2010 I’ll be keeping an eye on the data to see what happens. In the meantime, if you can think of other real-world metrics that might be suitable subjects for search trends analysis, get in touch.

  3. Can search predict the future?

    Posted December 23, 2009 in research, strategy  |  2 Comments so far

    Today, we often  search for information about upcoming major events in our lives – both good and bad – before we experience them. When facing financial difficulty or unemployment, many of us will go online at the earliest opportunity to look for help and guidance. And when we’re considering major financial decisions such as buying a house, search engines are usually consulted before estate agents are called.

    Traditional economic reports, on the other hand, look at events that have taken place. Unemployment figures tell us how many people are claiming benefits rather than how many people have been put at risk of redundancy. Average house prices are based on completed transactions, not how many people are currently looking to buy. So while we can be fairly confident of these reports, they don’t provide us with particularly current insights.

    This trade-off between confidence and currency was, in the past, largely academic as analysing current data was almost impossible. But in the age of the real-time web, this might be about to change: maybe patterns in search behaviour can give us a glimpse of future patterns in the economy.

    We first became interested in this topic back in spring 2009, so we analysed search patterns for two sets of keywords as the UK economy went into recession. We looked for relationships between these search patterns and related economic indicators, and listed some tentative predictions based on what we observed.

    House prices

    In April 2009, we looked at volumes for 23 keywords that homebuyers might use, including buying a home, cheap mortgage and mortgage providers. UK search volumes for these keywords were then compared to house prices.

    House prices charted against search volumes for 23 related keywords, from January 2004 to April 2009. Sources: Nationwide, Google Insights

    Searches typically decline as autumn ends before rebounding in January. But in 2008, the January rebound was lacklustre and the decline came in spring – much earlier than usual. This was in line with house prices, which peaked in late 2007 and dropped severely from spring 2008.

    In the first few months of 2009, however, search volumes enjoyed a far stronger January rebound than in the previous year – so we hypothesised that house prices would bottom out or even start to rise again in the middle of 2009. Let’s look at how accurate that hypothesis turned out to be.

    House price data to the present

    House prices charted against search volumes for 23 related keywords, from January 2004 to December 2009 Sources: Nationwide, Google Insights

    Sure enough, the search volume resurgence was accompanied by house price growth throughout 2009. But you’ll notice that search volumes soon tapered off, with a particularly steep fall after August. Our revised hypothesis, then, is that house prices will initially plateau and then drop again. We’ll revisit the statistics in spring 2010 to see how things turn out.

    Financial difficulties

    The second set of keywords we analysed was related to impending financial difficulties such as joblessness, debt and insolvency. They included signing on, mortgage arrears and debt problems, and were compared to the UK jobless rate.

    Financial problems searches versus unemployment

    UK unemployment rate charted against search volumes for 24 related keywords, from January 2004 to April 2009 Sources: Office for National Statistics, Google Insights

    These search volumes dip at the end of each year before rising in January – and the rise in early 2008 was more pronounced than in previous years. The jobless rate started climbing three months later, suggesting that in this case search patterns might anticipate economic statistics. We observed that search volumes had dropped significantly in the first few months of 2009, so our hypothesis was that the jobless rate would stabilise but not drop between April and July. The chart below shows what actually happened.

    Financial difficulties searches versus unemployment, until now

    UK unemployment rate charted against search volumes for 24 related keywords, from January 2004 to April 2009 Sources: Office for National Statistics, Google Insights

    The unemployment rate has indeed stabilised, wavering between 7.7% and 7.8% since early June, suggesting that our original hypothesis was valid. And search volumes have kept on dropping throughout 2009. If search trends do anticipate economic reports in this case, we should see the unemployment rate drop steadily between now and spring 2010. Again, we’ll revisit these figures in April to see if this happens.


    Our hypotheses from April 2009 were largely borne out as the year progressed: the drop in house prices was reversed and unemployment rates stabilised. So maybe there is some truth to the notion that search patterns can shed some light on forthcoming economic change.

    But these hypotheses were in tune with the economic mood of the time. Many commentators were talking about green shoots and a V-shaped recession – there was a feeling that recovery was just around the corner. Today, we remain in what has become the longest-running recession in recorded history and there is considerable uncertainty about what 2010 will bring.

    Our new hypotheses are less likely to be tainted by current economic consensus, precisely because no real consensus seems to exist right now. For this reason, the idea of search predicting the future will be seriously tested as the year unfolds. Don’t forget to come back in April 2010 to see the results for yourself.

  4. Google Flu Trends

    Posted November 13, 2008 in strategy, webapps  |  No Comments so far

    In this post, I’m going to try to outline a convergence between two separate trains of thoughts. It might get messy, so bear with me.

    Train one (think of this as the Edgware branch of the Northern Line) is search engine optimisation.

    One of the areas I’ve been working in a lot recently is search engine optimisation. I’ve carried out three fairly in-depth assessments of different search markets in the last few weeks.

    It’s been an interesting learning experience in a lot of ways—the last time I was heavily involved in SEO was a few years ago and the tools available for carrying out analysis have come a long way since then. Perhaps the most potent new weapon in the arsenal of a search market analyst is Google Trends. Try it, it’s fun.

    Train two (this is the High Barnet branch) is corporate social responsibility (CSR).

    For a while now I’ve held the view that companies are not doing enough just throwing money at CSR initiatives—donating to charity, that sort of thing. After all, money isn’t the only thing that successful companies have to contribute. They are also rich in expertise and capability. Companies should therefore look for ways to apply their know-how to social problems.

    An example of this that I often refer to is TNT Express Worldwide’s work with the World Food Programme. It assigns staff to work with the WFP and contributes its expertise in the fields of distribution and logistics, helping to manage the distribution of food in geographically remote and challenging regions. The value of this contribution is inestimably higher than it would be if it were purely financial.

    And here’s where the two trains of thought converge. To torture an already stretched metaphor, imagine this as being Camden Town station.

    Google launched Google.org some time ago as its philanthropic arm. It’s headed by epidemiologist/technologist Dr Larry Brilliant and seeks to do the sort of thing that TNT are doing with the WFP, namely using Google’s unique capabilities to bring a fresh approach to various social problems.

    A great example of this is the recently launched Google Flu Trends, an analysis of how Google Trends can help point to flu outbreaks around two weeks than conventional epidemiological analysis.

    It’s nice to see companies bringing knowledge and not just money to the table when it comes to health, hunger and other real-world problems.