If you’re a Simpsons fan you’ll probably recognise this pearl of fatherly wisdom that Homer once shared with Bart:

“No matter how good you are at something, there’s always about a million people better than you.”

Like lots of good lines in the Simpsons, it’s funny cos it’s true comically demoralising. If all you can hope for is to be the millionth-best person at something, why bother? But maybe the gag isn’t as depressing as it used to be. Maybe being the millionth-best at something these days is something to be proud of.

The episode in question (Homer At The Bat, fact fans) aired in February 1992, before the internet really got going. If you were an aspiring yo-yo artist in 1992, coping with the realisation that a million other yo-yoers had you beat, you could try to live your life in blissful ignorance of them. And if you lived in a remote enough area, that just might work.

If you were a young yo-yoer today, on the other hand, these million people would be an achingly visible blight on your career. You’d go to a party, start talking about yo-yos, and before you got a chance to show off your new trick, everyone would be gathered round a laptop watching videos of amateurs far more practised than yourself. Those million people? They’re just a click away now, and you’re going to be compared to them, even if you’ll never meet any of them in real life.

It used to take lots of effort to jump from personal to local to global context. Now we just get swept out there as soon as we start doing anything

The amount of effort you need to make to enter the global arena, in most walks of life anyway, is far lower than it used to be. Musicians used to have to record demo tapes, haggle with labels and play thankless gigs in sullen backwaters just to get some sort of exposure. Now you just need to get on Bandcamp and all of a sudden you’re playing with the big dogs.

The first people to really feel this effect, I would say, were computer gamers in the late 1990s, when online gaming started to kick off. In the pre-internet era, a teenager into computer games would only ever see their immediate friends and schoolmates play these games. So when you watched the school’s best Buggy Boy player doing their thing, you might as well have been watching the world champion.

Buggy Boy: global fame is but one lap away

Then the internet came along and put us all in our places. Gamers were suddenly thrust into this grand global arena, a colosseum where wins and losses were mercilessly quantified over the years and the leaderboards were calculated. Before long the true champions emerged. You became able to say “I’m in the top 20,000 players of X Wing Versus TIE Fighter” and you wouldn’t be lying. Yes, it was kind of demoralising, but gamers had to adjust – this was how things were going to be from now on. Everyone knew precisely where they stood in relation to everyone else, ambitions were recalibrated, you had lots of people to learn from – and, most importantly, gaming was still fun.

Over the last decade this phenomenon has extended beyond gaming and other nerdy pastimes. Internet video and the consolidation of once-fragmented online communities on to a small number of social networks means that the competent amateurs, struggling beginners, and reigning champions are out there and equally easy to find.

Yo-yoers, violinists, singers, underwater jugglers – in all of these fields, an aspiring newcomer will be able to go online and find those million people that are better than them. But they shouldn’t let this put them off. In a world of nearly 7 billion people, more of whom are coming online every day, being the millionth-best maybe isn’t too shabby after all.

Using online analytics and social media trends to predict real-world events is nothing new. Twitter’s been used to predict box-office sales (story link, detailed paper) and Google search data has been telling us about future flu epidemics for a while now.

Even I got in the act, demonstrating back in 2009 that Google Insights could anticipate changes in UK unemployment figures.

UK unemployment rate charted against search volumes for 24 related keywords, from January 2004 to April 2009 Sources: Office for National Statistics, Google Insights

Maybe I should have followed through with that idea, because there’s now a hedge fund that bases its investment decisions on data from Twitter. It’s called Derwent Capital Markets, it opened for business last week, and if its managers end up making a mint there might well be a new bandwagon in town.

So how do you run a hedge fund based on tweets? From what I understand of Derwent’s methodology, their algorithms measure the “calmness” of the Twittersphere – presumably based on sentiment analysis, which I’m a bit skeptical about. This is used to estimate the volatility of the Dow Jones Industrial Average index, with a three-day time lag.

This leaves a lot of unanswered questions. Does a non-calm day of Twitter conversations always correspond to a drop in the DJIA, or just volatility? Are they trying to predict metrics like trade volume and so on as well as broader day-to-day movements in the overall index? And are they ranking Twitter users based on credibility, or are spam bots equal to financial journalists, economists, and prominent investors?

Obviously algorithmic hedge funds aren’t about to disclose their inner workings so questions like this will have to remain unanswered for now. But what of the other, larger, question – isn’t the whole idea just, well, a bit… silly?

I can see why people might react in this way, and even I feel a bit skeptical about something describing itself as a “social media-based hedge fund” and that apparently pulls data only from Twitter, when there are lots of other sources that could be tapped. But it would be wrong to dismiss the basic concept.

Our everyday activities – web searches, page views, purchases, things we say on open social networks – leave a trail of data behind, which we tend to see as ephemeral or throwaway. We severely underestimate the value of this data but Google doesn’t, Facebook doesn’t, and we shouldn’t either. This data becomes even more valuable when aggregated across entire countries, continents, or the planet as a whole. In fact, it could be argued that the predictive potential of aggregated global real-time data has yet to be fully imagined, let alone realised.

The biggest problem with this resource is that we don’t really know how to exploit it yet. Things like Google Flu Trends or this Twitter-based hedge fund may be crude and experimental, and will definitely look even more so in five years time. Along the way there will be hype, bandwagonism, maybe even a stock market bubble, resulting from the application of real-time data to real-world problems.

But we need to make a start somewhere, and as silly as a Twitter-based hedge fund might sound, it’s as good a place to begin as any.

One of LinkedIn’s strengths is its “how you’re connected” feature, which shows how you’re linked to second degree contacts. Seeing who you have in common with someone helps you understand who they are, what they’re like, and whether it’s worth getting to know them. It’s often more informative than the blurbs people write about themselves.

"Any friend of Joe's is a friend of mine"

But this LinkedIn feature is becoming less useful due to an insidious form of network pollution. Like coastal erosion, this network pollution is a slow process that’s barely noticeable from one day to the next, but could be hugely damaging in the longer term. And I think I know who’s responsible for this network pollution – recruiters.

Before I continue, I should say that this isn’t an anti-recruiter rant. Recruiters may be responsible for this network pollution, but the blame lies with LinkedIn, and I’ll talk more about this later. Building a big contact list is essential to a recruiter’s job and they can’t be expected not to do this. But this is what’s weakening the value of LinkedIn’s “how you’re connected” feature, and quite possibly its network as a whole.

If you’re a LinkedIn user, you’re not just a person – you’re a “node”, which is a fancy way of saying that you can connect people to one another. If one of your contacts finds another one of your contacts on LinkedIn, you will be the node that connects them. And as a connecting node, your usefulness comes from the quality of your relationships with those two individuals. If the person searching knows that you’re picky about who you connect with (which you clearly are, only highly discerning people read this blog after all), your connection to that person is itself a notable endorsement.

If you were Marlo, you'd probably be more interested in people you knew through Prop Joe than through McNulty

Not every “node” on LinkedIn is as discerning and useful as you are, though. Some nodes are far more promiscuous, connecting to lots of people they’ve never met, let alone worked with, and the more promiscuous someone is the less useful they become as a LinkedIn node. This is where recruiters come in. They hoover up connections, which means that you often find your second degree contacts are connected to you through recruiters. But as connecting nodes, the recruiters aren’t all that useful because they’re not very choosy about who they connect with.

Bubbles pollutes Marlo's network because he knows so many people. Now everyone's a second degree contact

OK, maybe I’m stretching the analogy by comparing Bubbles to a recruiter, so I’ll drop it now. The general principle is that, if you’re connected to more than a couple of recruiters, searching LinkedIn will turn up more and more people who are second degree contacts, but that you only know through recruiters. The value of someone being a second degree contact slowly declines, because when a recruiter is the common contact you learn nothing more meaningful than that you both once looked for a job, or once tried to hire people.

It’s like sharing a mild dislike of rain – common ground, yes, but not very meaningful. This is what I mean by “network pollution”. The value or interestingness of the network is dropping because of recruiters and other “super-nodes” who are turning nearly everybody into your second degree contacts.

LinkedIn isn’t the only service susceptible to this kind of network pollution. Twitter will sometimes recommend another user to you because you have a “follow” in common. And if that “follow” is, say, your best friend, that’s good grounds for a recommendation. But if the common follow is Stephen Fry, Barack Obama, or any other celebrity account with millions of followers, that’s pretty useless. If Last.fm recommended someone to you because you both listened to the Beatles, that would be pretty useless too (which is why music recommendation algorithms are hard to get right). All social networks have to deal with problems like this where “super-nodes” undermine the value of recommendations based on shared connections.

So as I said earlier, this is a LinkedIn issue and not the fault of recruiters who are simply trying to do their jobs. Recruiters will continue to add connections, other people will continue to accept them, and the usefulness of “how you’re connected” will continue to drop. It’s not a very serious problem right now, but LinkedIn needs to think of how it can design for this aspect of its social graph, which is something it seems to take pretty seriously – and rightly so.

I use Dreamhost to host this site. They’re pretty good but occasionally things go wrong, so I’ve got a saved Twitter search for “dreamhost” that lets me know when they’re having problems. Earlier today the saved search started turning up lots of identical tweets about Dreamhost from a load of young female users:

"Online buzz" for Dreamhost

Maybe I’m a cynic but I wonder if all these young girls really posted exactly the same tweet at the same time about dedicated hosting. I mean, it’s an exciting topic and everything, but it still seems a bit fishy to me.

So what’s going on? Is Dreamhost using bot accounts to spam Twitter?

You might not have heard of HBGary Federal before. I certainly hadn’t, or at least not until February 4th when their CEO Aaron Barr boasted to the press that he had unmasked members of Anonymous and was going to pass their details to the FBI. This was presumably in retaliation for Anonymous having slowed down the servers of Visa, Mastercard and Paypal for a few hours back in December 2010, a crime that will no doubt live in infamy.

As it turns out HBGary Federal is a computer security consultancy that does a lot of work for the US government, trading on a reputation as experts in the field. Their CEO was obviously looking to generate headlines with his Anonymous story. And he succeeded, but not quite in the way he was expecting.

Within a few hours of his boasting to the press about having “infiltrated” Anonymous, Anonymous struck back. And they struck back hard. The HBGary Federal website was compromised and defaced, Aaron Barr’s Twitter and Facebook profiles were hijacked, and – most damagingly for HBGary – the company’s email server was breached, the emails extracted and put into the public domain via BitTorrent.

At this point, the damage done to HBGary was already severe. How could “experts” in information security be so thoroughly compromised, so quickly, and in such a humiliating manner? As Aaron Barr put it, soon after the attack took place:

I knew some folks would take my research as some kind of personal attack which it absolutely was not. I thought they might take down our Web site with a DDoS attack. I did not prepare for them to do what they did…

But the worst was yet to come. It took a few days for the contents of the email dump to be reviewed, and what it revealed was even more damning – not just for HBGary Federal, but for the shady culture of impunity it portrayed among firms contracting for the US government.

The new twist in the tale came when a project proposal was discovered among the emails. The proposal, titled “The Wikileaks Threat” (link to the full presentation), had been created by HBGary Federal in conjunction with two other companies for Hunton & Williams, a law firm that works with Bank of America. It outlined a systematic plan of attack against Wikileaks and its supporters which included tactics ranging from DDoS attacks, falsification of information, and what could be seen as extortion of prominent free-speech supporters such as Salon writer Glenn Greenwald. The exact quote about people in this category was that they could be pushed to “choose career preservation over cause”.

Slide from the Palantir, HBGary and Berico proposal

If you want to know more without reading the whole thing, this Tech Herald article has a good overview, but you should definitely read Glenn Greenwald’s response over at Salon:

The very idea of trying to threaten the careers of journalists and activists to punish and deter their advocacy is self-evidently pernicious; that it’s being so freely and casually proposed to groups as powerful as the Bank of America, the Chamber of Commerce, and the DOJ-recommended Hunton & Williams demonstrates how common this is. These highly experienced firms included such proposals because they assumed those deep-pocket organizations would approve and it would make their hiring more likely.

To put it mildly, the tactics outlined in this proposal are indefensible and the other companies involved have since apologised to the proposed victims and distanced themselves from HBGary Federal. Indeed the chief of Berico has called the proposal “reprehensible” (PDF link to company statement).

But this doesn’t bring the matter to a close. The leaked proposal is almost certainly the tip of a very large iceberg, giving us a glimpse of a corporate culture surrounding the US government that has grown accustomed to operating outside the law. As Glenn Greenwald puts it:

The exemption from the rule of law has been fully transferred from the highest level political elites to their counterparts in the private sector. “Law” is something used to restrain ordinary Americans and especially those who oppose this consortium of government and corporate power, but it manifestly does not apply to restrain these elites.

The story began with a so-called security expert bragging to the media and has ended with the disgrace of his company. Andy Greenberg at Forbes:

Rarely in the history of the cybersecurity industry has a company become so toxic so quickly as HBGary Federal …many of the firm’s closest partners and largest clients have cut ties with the Sacramento startup. And now it’s cancelled all public appearances by its executives at the industry’s biggest conference in the hopes of ducking a scandal that seems to grow daily as more of its questionable practices come to light.

These questionable practises, which are still being uncovered, are too many to list here, but this timeline over at Ars Technica is worth a read if you want to know more about Aaron Barr’s techniques.

It’s a shame that this story isn’t getting more press attention, because it reveals a lot about what’s happening on the front line of the struggle for internet freedom – and by “front line” I mean the hand-to-hand trench combat as opposed to the high-profile court cases taking place in the US and in the UK.

But it’s unlikely to get much coverage because it’s a messy, data-intensive, and fast-changing story; in other words, the type of story that is extremely difficult to get across within the constraints of traditional news media forms. Traditional media seems to be more comfortable talking about Julian Assange’s personal hygiene or Downing Street’s new cat than covering this sort of thing.

I’m posting this from my phone, so apologies in advance for any typos. But I wanted to share this article from Felix Salmon on how the Wired/Wikileaks discussions of the last few days have highlighted a problem with Twitter’s new role in online debates:
http://blogs.reuters.com/felix-salmon/2010/12/30/the-evanescence-of-twitter-debates/

As commentators use their blogs for increasingly journalistic content, the conversational aspect of blogging moves on to Twitter. This leads to two problems.

First, these conversations become very hard to join mid-stream. If you weren’t following from the beginning, you’ll have a hard time catching up. This is especially true of conversations that involve more than two people, as the “in reply to” functionality is no help. A commment thread on a blog or forum, on the other hand, can be read from the beginning even if you’re coming late to the party, and its linear structure makes it easy to catch up.

The second problem is that Twitter loses these discussions after a couple of months, so they’re not available for future reference. This ephemerality is part of Twitter’s appeal for users, but from an archiving point of view it’s definitely a weakness. It’s good to be able to look back on how topics were discussed in their time, but Twitter currently doesn’t let us do that.

Maybe Twitter will evolve to address these problems over time. If it doesn’t, however, there could be an opportunity for third party products that do.

I’m not sure exactly how much I’ve spent with Amazon in the last year, but it’s a lot. If I buy something online, I’ll probably buy it from Amazon even if it’s slightly cheaper elsewhere. I buy books, MP3s and big-ticket items like computers too. So I guess I have a strong “brand relationship” with Amazon.

Like many people, I’m re-evaluating this relationship after Amazon dropped Wikileaks in an apparent concession to US government pressure (their official statement didn’t impress me much either) and I may stop buying things from them.

But here’s a good question – if you plan on boycotting Amazon for not hosting Wikileaks, why not boycott every firm that doesn’t host Wikileaks? This is my answer, and it’s grounded in Amazon’s ambitions (specifically the Kindle) rather than a general sense of corporate morality.

The Kindle strategy: mediate between reader and book

When Amazon started out, it just sold books. As it grew it started selling lots of other stuff (encountering more than a few UX problems along the way) but books and their readers remained key to its identity, as was affirmed by the launch of the Kindle in 2007.

Before the Kindle, Amazon’s relationship with the reader began with browsing for a new book and ended soon after it arrived. The packaging discarded, the book was opened and Amazon was forgotten: the relationship was now directly between the reader and the book.

With the Kindle, this relationship was to change. Rather than just enabling the book’s purchase, Amazon would remain in the equation while the book was being read. The relationship, instead of being a direct one between reader and book, would – through the Kindle – be mediated by Amazon, who would enjoy a more meaningful connection with the reader.

It’s a great strategy, and well-executed too: the Kindle is a joy to use. But underlying this strategy – and this is where Wikileaks becomes relevant again – is the increased need for trust between Amazon and the reader.

Trust, neutrality, and moral failure

Trust isn’t important when Amazon sells me a book. I need to trust that they won’t rip me off, yes, but that would be illegal – the trust is backed up by law. And once I’ve got the book in my hands, what can Amazon do? They can’t stop me reading it.

In the world of the Kindle, however, trust changes and becomes absolutely essential. This is because, in this transformed relationship where Amazon is the mediator, Amazon can remove books from your Kindle. It can do so remotely, without warning, at its own discretion, even if you paid for them or got them elsewhere. The reader must therefore trust Amazon not to do this. If she doesn’t, her relationship with the written word is no longer free.

When Amazon remotely wiped 1984 and Animal Farm from Kindles in 2009  this trust was damaged. That was due to rights & ownership problems – it wasn’t political, it was commercial. But the Wikileaks incident shows that Amazon will remove content for reasons that are ultimately political.

This doesn’t just damage that trust, it destroys it completely, and with it Amazon’s credibility as an organisation fit to mediate my relationship with the book. What if there was political uproar over a controversial novel, and Amazon was pressurised to remove it from the possession of people who had paid for it? We know now that they’d do it, and the implications are depressing.

In fact they’re so depressing that I feel glad that the Kindle wasn’t invented a century earlier. How much more effective would Soviet suppression of samizdat have been if the Kindle was in widespread use back then? What would have happened to Lolita, Lady Chatterly’s Lover, Ulysses, or any of the hundreds of books that were banned and burnt in supposedly less enlightened eras? How much would we have lost?

The banning of Wikileaks raises questions that are particularly sensitive given Amazon’s lofty aspirations. How can you promise to manage someone’s relationship with the written word – and therefore with culture, politics, literature, and arguably thought itself – when you can’t be trusted to remain neutral and impartial? Amazon has to be held to a particular moral standard, and it is this standard it has failed to meet. We were within our rights to expect more.

Yesterday I was wondering if it was really a good thing that we seem to be engaging less with technology while at the same time becoming ever more dependent on it.

That post was inspired by Stuxnet, the ultra-advanced software weapon seemingly aimed at Iran’s nuclear facilities. But a more everyday example of the risks technology can pose to overly oblivious users has appeared on the BBC’s website, with Rory Cellan-Jones discovering how easy it is to compromise an iPhone 4 and steal personal information.

[Security experts] used a netbook computer to set up a wireless access point. They called it “BTOpenzone”, a network my phone and many others look out for and join. I watched as they showed me a range of devices in their office in London’s Soho looking at the network – including my phone.

This wasn’t the only exploit used – the demonstration also included the iPhone 4 PIN hack, SMS number spoofing, and the interception of cookies sent via Facebook. As you’d expect, Cellan-Jones is at pains to mollify Apple and Facebook, the two companies whose products are shown to be compromised in the article. But none of this stuff is hyper-technical – for a hacker to pull this stuff off is relatively trivial.

The demonstration and the article as a whole is a great example of how blind, unquestioning trust in the technology we use can expose us to massive risks, not just from uber-hackers but from anyone with malicious intent and basic networking knowledge. It reinforces the point that we could do well to understand the technology that surrounds us a bit more than we currently do.

Stuxnet is a piece of extremely advanced attack software, currently active in several Iranian nuclear facilities while being studied intently by malware experts around the world. No-one knows who made it. It’s completely unprecedented – a militarised program, engineered to near perfection, something that’s more accurately described not as a computer virus, but as a weapon.

Langner, a security consultancy that’s been analysing the code, recently described Stuxnet as being “like the arrival of an F-35 fighter jet on a World War I battlefield.” Kaspersky Labs called it “a working and fearsome prototype of a cyber-weapon that will lead to the creation of a new arms race.” These claims sound hyperbolic, but the more I learn about Stuxnet, the more inclined I am to agree.

I’m interested in Stuxnet not so much because I have plans to disrupt centrifuge controllers in distant nuclear power plants (well, not in the near future anyway), but because it’s an example of how invisible technologies can have such a concrete effect on the world.

We’re living in a period where our use of technology and our dependence on it is growing. But at the same time our technology is disappearing from view. It gets smaller, it gets lighter, it gets better at understanding us without the aid of clunky input devices, it gradually disappears – as Adam Greenfield describes the phenomenon in his book Everyware, it “dissolves in behaviour”. We use technology, but we’re becoming less engaged with it.

We’re becoming a bit like the Eloi in The Time Machine, completely dependent on things we don’t understand. You can see the results of this pattern wherever you look. Whether it’s Microsoft’s adverts for Windows Phone 7 whose core message could be translated as “phones suck”, or workplace cultures where it’s embarrassing to be seen as technologically adept, there’s a strong theme of technology as an enabler, but still something that should be on the fringes of our lives.

I’m not setting out to criticise this trend or pattern, however, or argue that everyone should become a hardcore techie. If we were burdened with a detailed knowledge of every technological process we initiate in the course of a normal day, we’d probably all suffer from constant migraines. It’s good that, say, checking Twitter on my mobile phone feels like a casual and trivial  thing to do, and that we’re not forced to confront and experience the mind-boggling combination of technologies that are actually invoked when we do it. Technology couldn’t be as ubiquitous as it is if it hadn’t developed this Houdini-like talent for making itself invisible.

But then, when I read about Stuxnet, I’m reminded that these deeper layers of technology are still there, still real, and still have a concrete and tangible effect on our lives. We might push technology aside and keep it out of view, but there are others – like the organisation behind the Stuxnet worm – who obviously aren’t, and their ability to change the world should not be underestimated.