author

Sections

Galleries

The ringmaster of the blogosphere

Technorati.com began as a case of 'scratching a personal itch' and has grown to be a

vital tool for web watchers

Glyn Moody

Posted on Thursday 16 February 2006 6.45 America/Los_Angeles in Technology

If life without Google is hard to imagine for most web users, life without the blog

search engine Technorati.com is probably unthinkable for inhabitants of the blogosphere.

Where Google lets users find what they are looking for, Technorati enables bloggers to

find out something even more precious: who has been linking to their posts.

Technorati is a classic case of scratching a personal itch. Back in 2002, David Sifry was

a 33-year-old serial entrepreneur in Silicon Valley. After gaining a degree in computer

science, a stint in Japan and another on Wall Street, he'd already launched a couple of

companies and was busy setting up another. Along the way, he'd discovered blogs.

But, as Sifry says, blogging inevitably raises two questions: "What's everyone saying

about me - what's the conversation going on around my stuff" He tried to find out using

traditional search engines: "They all failed miserably," he says. They failed because

their approach - based around counting the number of links to a web page - "is

fundamentally biased against things that are new, because it takes time for links to

accumulate pointing to something." But in the blog world, it is precisely the new things -

the most recent posts on other blogs - that are the most interesting, especially if you

want to join in the conversation.

Traditional search engines send out a software "spider" to index all the pages on a

website, whether or not they have changed. This is a huge task, so the time between visits

of the spider is usually days - hopeless for the fast-moving world of blogs. A better

method would be to index blogs as soon as they are updated, and only then. Sifry realised

that a mechanism was already in place that would allow him to do precisely that.

'Ping'

"A huge amount of credit has to go to Dave Winer," Sifry says. Winer was one of the ur-

bloggers and also the author of an early blogging program called Radio Userland. Winer had

built a feature into Radio Userland that would prove key to Sifry's new kind of blog

search. Winer "created something known as a ping," Sifry explains. "Whenever anybody who

was using his tool, Radio Userland would update their blog, [the software] would

automatically update a site called weblogs.com," owned by Winer. "It would send off a

little notification message" - the ping - "saying 'hello, Bob's thoughts just got

updated'." So all Sifry had to do was create a spider that went to weblogs.com, downloaded

which blogs had been updated and then indexed those new blog entries. This formed the

basis of what became Technorati (chosen because an earlier idea, "Digerati", was

unavailable as a domain name).

Sifry then approached all the leading blog software vendors and said: "Look, if you guys

build this [ping mechanism] into your tool, and have it ping Technorati [as well as

weblogs.com] we'll go out and index your content literally within seconds of when the user

posts." The software companies readily agreed. First, because "everybody who writes a blog

wants to get found," as Sifry points out. Second, he says, because "I literally would

write the code for people."

Rapid programming was crucial for the creation of Technorati itself, which began in

November 2002 as Sifry's "weekend project" in his basement, alongside his latest startup,

consulting to pay the bills and the arrival of his second child. Sifry's hacker past

helped - one of his previous startups was Linuxcare, which had employed half the coding

stars of the open-source world - but so did the ready availability of well-honed GNU/Linux

software such as the Apache web server, the MySQL database and the PHP and Perl scripting

languages, collectively known as the LAMP stack.

The constant time pressure Sifry was under had an important consequence. A few months

after Technorati was launched, the 2003 invasion of Iraq took place. "It hit me like a

lightning bolt," Sifry explains. "I don't have time to figure out what's the most

interesting stuff to read, but there's like a million other bloggers, they know.

"What if I took the four thousand or so different mainstream news sources that are out

there," he continues, "and then looked at all the people in the blogophere: what are the

stories they are linking to the most in the last three hours" The result was the first of

what Sifry calls "discovery products", which appear on Technorati's home page.

"This is one of those things that I think is fundamentally different about Technorati,"

compared to Google or Yahoo, he says, since it is based on "understanding people and

understanding time" - not just on static links between web pages.

Unique insight

Technorati's way of tracking blogs - 27m of them at last count - gives Sifry a unique

insight into this world, which he reports on in his occasional State of the Blogosphere,

posted on the Technorati blog. As well as problems like splogs (spam blogs) and spings

(spam pings), one last week touched on the issue of blogfade: the fact that 50% of new

bloggers have given up after three months. Sifry is not unduly worried. "The number that I

find indicative of the true growth of the blogosphere is the number of daily posts. This

is a much better way of being able to watch how much time people in aggregate are putting

into writing on their blogs. Today, we're tracking about 1.2m posts a day." A year ago, it

was a third of that.

Other notable trends include the growth of blogging in eastern Asia: last month, more

posts were made in Japanese than in any other language - English-language posts represent

about 28% of the blogosphere. In part, this is because Japanese bloggers tend to post lots

of shorter entries, a habit that is spreading, Sifry says. "More people are using blogs as

a sort of conversational medium, as opposed to the long-winded 'here's my 500 to 1,000-

word essay' medium." As a result, the average number of links in each post is dropping.

Another development concerns who is using Technorati, alongside all those bloggers

checking their links and the time-challenged looking for news. "We have a lot of companies

that use this service who want to be able to track 'what are people saying about my

company What are they saying about my products Hell, what are they saying about my

competitor's products'" Other regular visitors include the "cool-hunters", desperate for

the next trend in music or fashion, and journalists seeking stories and sources.

Technorati makes its money from sponsorship and advertising on the main site, as well as

syndication revenues from titles such as the Washington Post, Newsweek and Der Spiegel:

alongside a news story, Technorati provides real-time links to who is blogging about it.

Sifry's company is still privately held: "We did a little bit of an angel round with some

smart Silicon Valley investors in 2003," he explains, and in 2004 he got funding from two

venture capital firms.

Sifry is evasive about any possible stock market float, and laughs uproariously at the

idea of being made an offer he can't refuse by somebody like Google. But given the rise of

blogging as the new face of the web, and Technorati's central position in the blogosphere,

going public or being acquired is surely just a matter of time.

Curriculum Vitae

Age 37

Education Graduated in 1991 with a degree in computer science from Johns Hopkins

University, Baltimore, Maryland.

Career

Contact this candidate