Search - for a better way

WARNING - BRAIN DUMP AHEAD!

This post is rather long, unwieldy and scattered, not to mention incomplete... Read it at your own risk! ;)

I dont have time these days to blog about all the things I would like to... this subject has been on my mind the past couple of days. A couple of blogs that I have been checking recently are talking about search. The key personalities involved are Robert Scoble, Dave Winer and Seth Godin.

Dave Winer posted that he thought Google were being careless with their search, making some errors in judgement and not focussing where they should be. Interestingly (to me) his brief comment on Google's blogsearch (trying to segregate RSS instead of embracing it) would seem to be in line with my own views on it; as I posted in Tom R's comments

Do we really need a seperate blog search from Google?
Why not just integrate it into the regular search? After all the so called 'blogosphere' is not really distinct from the 'interweb' it's just a way of publishing to it.[...]I guess that your point that it's a feed search and not a blog search is probably the real issue, but you see where I'm coming from...

To me, whether a site/page has a feed or not is a seperate issue to be worked out by searches, not whether it's a blog or not

But the main point here was that Dave Winer was one of several people who seem to be becoming critical of the lack of forward progress on the search front.

I noticed that Scoble was discussing search, and it was interesting - at first I thought it was baloney, but the more I thought about it the more I... went around in circles.

My interpretation of Scoble's argument

Scoble's argument seems to have two key points, illustrated by the following.

1. Do a Google search for HDTV: where are the manufacturers?
2. Do a Google search for Robert: why is scoble more relevant than, say, Robert Redford?

Where are the manufacturers?

This point is based around search engines not returning links to sites which a human would consider highly relevant.
The manufacturers example is interesting. Certainly you would imagine they would be important results in a search for HDTV. Scoble points out that users search using 1 or 2 words generally. It is difficult to find the manufacturers using 1 or 2 word searches. However I tried 'role-playing' some HDTV research and by modifying Scobles search to 'buy hdtv' (which I felt was closer to what he wanted) I was able to get manufacturer information in two or three clicks, by doing a search, following a relevant looking link from a reasonably trustworthy source and then following the navigation in that site through to HDTV models endorsed by that site.

Interestingly, when I later I read Seth Godin's ebook, his new internet venture is based on the fact that this is how searching on the internet works... more on that later!

So search had worked for me in my role-play through logical use of it. And as I commented on Scoble's blog, if I were buying a HDTV the manufacturers would not be my most trusted source. I would rather research and then go to manufacturers. And as Scoble pointed out, part of the problem with Sony's site was that they developed an unfindable flash site.

Is this an integral part of the problem? Search engines search the internet. If companies fail to understand the nature of the internet and how sites are best constructed to allow content to be found and accessed easily, how can search engines be expected to return them as relevant matches for searches?

Would better adherence to web standards in development of all websites show an improvement in search results. Most likely. Almost certainly if coupled with enough meaningful content to demonstrate that your site is an authority on a particular subject.

Is this a utopian vision of the internet? I don't think so. But surely web standards and valuable content are the only problems here? Why don't the manufacturers come up on a search for HDTV? We'd have to ask Google and they tend to be pretty secretive.

Is it fair to lumber the search engines with trying to decipher what it is we want from a search? Yahoo Mindset begins to implement a way to narrow down your results by allowing you to quantify whether your search results should be for research or purchasing purposes.

But to do this properly, I believe, could result in a fairly complex search interface - and I love the simplicity of Google. As must everyone considering their market share.

However, having done some work on Hotel search engines, I am willing to admit there may be a nice solution. I just haven't had the time to even begin to consider what it might be. For example look at Kayak.com and do a hotel search, it has a similar interface to yahoo mindset but with more options. What would the options be on a generic search in order to return more valuable results?

Which Robert is more relevant?

This point revolves around search engines giving higher priority to sites which are of lesser relevance to the searcher.
This is also an interesting point. If you do a Google search for Robert, Robert Scoble's site comes back as number one. Why? I assume because the people publishing content these days, especially with the advent of blogs, are technical types and they have an interest in Robert Scoble and link to him a lot and so Google interprets Scoble's site as being important. Which is fair enough, but you can see how this skew isn't going to suit everyone.

On the other hand, do a Google search for 'Robert actor' and you get a Robert Redford relevant result back as number one, followed closely by other Roberts who are actors such as Rob Lowe and Robert De Niro.

So being a little more specific about what you are looking for helps. I use search a lot and have become quite good at finding what I want by refining my search as I go. But Scoble maintains that educating users is not viable. And I understand where he's coming from, but if a user wants to find Robert Redford related material, and they can't remember his second name, surely they know he's an actor and would know to add that to their search? If the don't, what are the chances they could use a more complex search engine interface?

And how can one develop a more relevant search without making the interface more complex?

How can you discover whether a user wants HDTV manufacturers from a search for 'HDTV' or whether a search for 'Robert' is looking for a friend, an artist, a musician, an actor or a technologist?

Is it all in the algorythm?

Maybe I'm on the wrong track... maybe the answer is not in the interface but in the algorythm. Now, to date people seem pretty impressed with Google's algorythm. But Scoble brought up another interesting point a couple of days after his initial point. He said:

I think bloggers aren't looking at search critically because we LOVE the traffic that Google brings! Translation: we're addicts and we don't want to turn the drug off. Well, I want a better search engine. I don't care anymore if I'm at the top of the list.

Interesting point. After all bloggers have managed to leverage Google's algorythm to their benefit - this is one likely reason why Scoble is number one as he points out. But he wants a better search he says. So the algorythm would have to change.

Well, fair enough. I am not about to start into arguing algoryhtms, but I often find material on the web by searching, finding a blog that posted about the material I am looking for, and am then able to follow a link from that blog to the material I wanted. Often the actual page I am looking for is not as findable as the blog that pointed to it.

Scoble would argue, I'm sure, that a search engine should be built that could find the page I'm looking for directly - and I'm not going to argue that (it would be lovely) but I wonder how you do it. If the page I'm looking for is a flash based page with no search engine optimisation how do we determine that page is relevant? A web standard blog entry which has an amount of relevant content to my search query and points me to that flash page is a life saver to me in this situation.

What about NOT changing algorythms or interfaces?

All this brings me to Seth Godin. As I was thinking about search and whether it needed to be improved and whether the current algorythms gave users relevant results, I downloaded Seth Godin's latest ebook from his blog.
As I read it I was surprised to find it was bizarrely tied into all my thoughts on search! Read this edited extract:

You go to Google. And you type in “buy espresso machine." [...] Of course, you're not ready to buy an espresso machine right this second. [...] Right now, you're just looking. You just want to learn about what's going on. [...]

A few sites down the list, I found that Engadget.com, a site I know and trust, has an article. So you click on it.
It's a pretty worthless article. But you notice that there are literally hundreds of comments [...]

about five comments down, you discover a long, thoughtful post by someone who knows all about espresso
machines. Not everyone is seduced by rational textual argument, but you are, so you get excited. Finally! You're
starting to understand. [...]

So you go to www.coffeegeek.com , which you find through another comment. Nirvana! This is the site that
should have been #1. But alas, it's disorganized and hard to follow. So you spend three hours (I'm not kidding,
three hours) reading up on espresso. Now you're informed, you know what's out there and you've read a few
reviews of different machines. Finally, you know enough to think about buying. [...]

So you go back to your original Google search. And now you click on an ad. You look at that site for a while, hit
Back, click on another ad. After you've clicked on six ads, you decide to go back to coffeegeek and buy a $1,400
espresso machine.

This is how Seth describes the current search cycle during research on the web. Which is at once both similar to my defense of current search engines AND highlights the issues Robert Scoble brings up.

Seth Godin's response is similar to the bloggers response: leverage on existing search algorythms to point to relevant material on a subject and therefore (hopefully) improve search results.

Seth's approach is either genius or idiocy. I don't know whether to admire his solution or jump up and shout that the emporer has no clothes! he advocates a new type of website or web page called 'a lens'. It is basically a one pager on a subject you hold dear to your heart where you provide relevant links and rss feeds. That's it.

UNLIKE A BLOG, just about every single item in a lens is connected to something on the Web. Lenses don't hold content. They point to content. And like all good guides, they comment on what they point to.
So your lens can point to blogs or to predefined Yahoo! searches or to a MapQuest map to your favorite restaurant. Your lens can point to the weather report or to treasured books on Amazon or to your wedding pictures on Flickr. A lens isn't filled with content. It points to content.
And your lens also points to other lenses. Lenses on similar topics. Lenses by people you know and trust. Lenses that are highly rated by Web surfers, and lenses that a lot of other people have linked to.

Seth plans to make it easy to build a lens by providing a service called Squidoo.com. By building this network of lenses with it's own search, Seth hopes people can leverage of the research of their peers to find relevant information.

I hope it works because Seth also hopes to raise a lot of money for charity through this venture - and that I applaud. And it could be the driver to make this crazy idea work too.

Conclusion

Are you crazy? You think I concluded anything? I warned you this was a brain dump!!
I think it's great people are pushing for more search development. I would love to see a better search, but we need to identify the problem with the existing search engines first. Some of the more obvious paths being discussed will, I fear, result in horrible search engines. I do worry that the user trend is to do a one word search and only look at the first couple of results returned. It is not surprising that many techies call for user education rather than changing the search! All in all Google does provide me with the tools to do a good search I think, but in order to do it I have to be willing to refine my search using longer phrases and look beyond the initial page of results. This is not typical searcher behaviour it would seem, but how do you develop a search that caters for such a limited user input while delivering better results?

Ok, ok so my conlusions are as follows:

• I am willing to concede that, like all things, search could be improved.
• I don't think anyone has identified the exact problem yet.
• I don't think the solution will become available until we identify that exact problem.
• I believe that web standard implementation of ALL websites will help create better search results.
• I worry about making search engines over complex
• I worry about pandering to lowest common denominator in user behaviour

Ok... I have to leave this for now, I am just going around in circles and boring anyone who chose to read this to tears. Perhaps I will come back in a few days, read this and clarifiy some of my thoughts.