wanderingstan

Can a ranking system be transparent, inclusive, and successful? That was the topic of a long conversation last week with Lijit’s senior developer Derek Greentree. We kept coming back to questions about transparency and the cost of acquiring votes. And in the end we decided that this is some sort of rule:

The maximum success possible for a system is a function of the transparency of the algorithms and the cost of acquiring votes.

Consider this rough chart:

I hear you asking, “Why do Digg and Google get “Low” marks for transparency?”

Digg ranks news stories by the number of members which vote for (“digg”) each candidate. It’s pretty much a pure democracy, with an added time component: old articles are worth less. On the other hand, Google ranks pages by a more complicated algorithm known as PageRank, which treats links on web pages as “votes” for other pages and some pages’ votes worth more than others. It’s a bit like the electoral college, with an added semantic component: pages not related to the search query are worth less.

Do those descriptions sound about right? The thing is, neither is true these days. PageRank is now only one small ingredient of a page’s search ranking. Anyone who pays attention to their page in search listings is familiar with the “Google Dance” when ranking can change unpredictably and sometimes unfairly. Google has become a black box. Digg’s newfound popularity has it struggling to deal with spammers, and has also begun to shroud its algorithms in secret. The most recent Wired magazine, has an article “Herding the Mob” quotes Digg founder Kevin Rose as saying there are antihacking techniques that he can’t talk about. 

Jay and Kevin said they couldn’t explicitly detail how Digg’s ranking algorithm works because it would be used by those who want to game the system (the aiding the enemy defense is popular these days), but they gave enough information to understand the basics of how Digg’s version of a democracy works.

So what we see is that these two popular online ranking systems began with public algorithms, but have retreated into secrecy. 

On the other hand, systems like the US election process remain part of the public record. Of course, in a democracy it costs a lot to get a vote. For one thing, you have to be born. And if you really want to cheat, you have to mess around with getting the ID’s of dead people and other very messy activities. In the recent Iraq elections the took the extra measure of dipping each voter’s finger in permanent ink to prevent double voting. 

Is this trend necessary? What are the underlying principles?

The trend seems to be that to thwart spammer in popular systems, transparency must go down or cost must go up. And in the online world, costs are dropping so low that transparency is being forced down as well.

The web has seen a lot of systems that begin with low costs and high transparency. That’s the very definition of openness. But as the systems experience success, they have 3 choices:

• Raise the costs. E.g. SomethingAwful.com added a $10 cover charge to participate in voting. Metafilter added a $5 cover charge.
• Obscure the algorithms. E.g. Digg adding secret “anti-gaming” algorithms
• Become irrelevant. E.g. Usenet forums overrun with spammers

The most popular choice seems to be obscuring the algorithms.

Should we be alarmed at this? Imagine if the US government took the same approach: they will tell us who won the election, but the exact algorithm used to determine the winner can’t be revealed! One can argue that getting on the front page of a Google search or the front page of Digg is not nearly as important as an election. But the value of such positioning is only increasing in value, and the bad guys are already trying to rig these elections!

I would argue that low transparency is a form of editing. When Digg or Google says that they must keep their algorithms secret, they are in effect saying “Our algorithms are fair, but we can’t tell them to you. You can trust us.” But do we really trust them? Should we? If some quirk of Google’s algorithms somehow helps a company they have a partnership with, how motivated will they be to fix it? 

Anyways, those are some beginning thoughts on the subject. Any ideas from you would be appreciated, as I feel there is a lot more to explore here.

Who are the influential bloggers? Which blogs matter? What metrics would you use to even begin to answer these questions?

I’ve been exploring alternate methods of ranking in the past months. The best results are coming from examining Blogrolls. When you think about it, blogrolls compromise the links in a huge implicit trust network. For now I’m calling the calculated score “PeopleRank”. It’s kinda like PageRank, in that blogroll links from higher PeopleRank-ed blogs count more. E.g. if Om Malik has you on his blogroll, that counts a lot more for your ranking than the blogroll of your niece on Livejournal. (No offense to your niece.)

So here are the top 50 blogs as ranked by the preliminary algorithm: (Commentary and caveats follow)

Caveats of this calculation:

• Results with ~5K blogs crawled.
• Blogroll Count = Number of blogrolls this blog appears on = How many people publicly admit to reading this blog.
• The interesting datapoints are where the PeopleRank ordering puts a blog higher in the list than one with a higher blogroll count — those fewer subscribers must be “more important”.
• This crawl took Lijit user blogs as the starting seeds giving an overall tech bias.
• However, there was a period when the crawler went unchecked into what can only be called “The Mommy-o-sphere” so there is an over representation of Mom-blogs in teh dataset.
• Our blogroll detector algorithm still gets false positives, thus the high rank for “Subscribe to Feedburner” and the multiple ColoradoStartups.com listings.
• Some blogs use a Blogrolling widget for a “Web Ring” functionality, thus erroneously appearing as blogrolls. This explains most of the 100+ blogroll counts.
• We need better de-duping. Several blogs appeared until multiple URL’s, reducing the overall score.

So how is this different from existing rankings? Til now, the most common methods have fallen into one of two camps:

1. Number of subscribers. I.e. a pure democracy. Use some combination of Feedburner (for RSS readers) and some web analytics (for web readers) to count the raw number of people reading a blog.
2. Raw number of incoming links (citations). This is similar, except that links are counted instead of subscribers.

Note that neither method discriminates between the blogs “casting the votes”. It doesn’t matter if that 24th reader of your blog happens to be Scoble. Nor does it matter if those 3 citations to your blog in the last month (Technorati defines this as “very low authority”) came from Seth Godin, Fred Wilson, and Guy Kawasaki.

Initial results are encouraging, and I hope to do more analysis this week. What do you think? If you have any suggestions or ideas, please get in touch with me.