Free 'sentiment' tool don't work

More and more companies are now tracking social media, to see how they are being discussed on twitter, Facebook, forums, video sites, and so on.  

So far, so good.  However lots of people think that you can automatically assess the sentiment of these conversations.  You can't.

Let's look at the current situation in Japan.  I looked at two tools that focus on twitter, TwitterSentiment and Twittratr.

First, TwitterSentiment:

Currently 67% negative, 33% positive, TwitterSentiment thinks that the following tweets are negative for Japan:

Mourinho on Real not winning CL: "if it doesn't happen that's not a serious problem. A serious problem is what is happening in Japan"

Finding it hard to concentrate at work, while all the horrible disasters are happening in #Japan.

Talked to a homeboy in Japan last night... and is only concern was getting back for soon fried chicken & a blunt. Smh #hoodbooger

Dumping water operation by helicopter was stopped due to high?radiation above the plant... #Japan #quake #nuke

I am still in shock after what has happened in Japan it just seems to be getting worse as each day go`s on wish I could help

& these are positive:

Official propaganda and misinformation from the Prime Minister's office of Japan - now in English! Follow @JPN_PMO

We should see if we can get enough people together to do a charity concert to raise money for #Japan. #JapanCharityConcert

Good to hear they are ok. How are you? My thoughts are with the people in Japan.

Toyota aims to decide today when it can resume production at automotive and parts assembly plants in Japan

Tune in at 6 this morning for the kickoff of the @KOMO phonathon to support Red Cross relief efforts in Japan! #HelpJapan

Similarly, over at Twittrratr, they think that 9% of tweets are positive, 88% neutral, and 3% positive

These ones are negative for Japan:

you're going to japan?! i guess you won't get this til you're back! have fun! 

watching "queen of atlantis", possibly the worst sci-fi film i've ever seen. worse even than the giant starfish movie from japan! 

[world news update] japan economy 'not in bad condition': aso (afp): afp - japan's eco.. http://tinyurl.com/5gcrop

is wondering what they are doing on this show in japan were they point at stuff in the newspaper, showing bad grammar maybe? 

talking about moving to japan. why is it so expensive to live there? we decided that @theantialliance is just gonna have to buy us a house.

& these ones are positive:

on weekend america tomorrow: ben folds talks keytar, david rakoff gets scary, flying monkeys, and obama japan. plus: porn. kind of. 

there's the lady in japan who is 120. come on people

top story: the shinto religion at japan blog: a nice quick overview of shinto by maki at japan.. http://tinyurl.com/5fu8g3

keigo is such bullshit. i can't believe we have to learn that super formal nonsense. japan needs to open up it's tight ass, jeez.

is anyone really surprised that wii music debuted at #1 in japan? i mean it's been the only game released for the console in months

Crazy, and almost totally random, and totally worthless.

With Twittrratr, it seems that the presence of words like 'bad' or 'anti' (even if in a longer word like 'Atlantis' signifies that the tweet is negative.  

These are two free tools, so to a degree you get what you pay for, but I'd challenge any of the paid tools to show me their results.  Deciding whether a mention is positive or negative in this instance is pretty hard.

The only way to measure sentiment correctly is to take a random sample (say 200 mentions) and then go through them manually and work out a percentage positive and percentage negative.