I’ll state it as clearly as I can: that definition of relevance is completely useless for anyone using the system, for anyone evaluating it, and for anyone maintaining it. It leaves a gaping and obvious hole in the results: what about documents clearly relevant to a given *meme* (e.g. Harvard for the zip code example) but which don’t incorporate the *meme id*, and are hence not in the results of a search?

Any sort of “meme id adoption rate” would reduce to the same sort of relevance judgements (which applications of the id are spam, and which aren’t?), and would simply define a ceiling on an appropriately-defined recall measure - you can’t get away from human-level relevance in a search system.

By relying so heavily on the meme id, you’re cutting out what could be potentially the most beneficial aspect of the system, and the one that has the most resonance: the meme itself as a locus of meaning for the meme id. What does it do in the search? What effect does it have on results? As far as I can tell, the answer to both of those questions is “nothing”.

Thanks for your comments and your long thoughtful post on shiftless.

Relevance has always been subjective and in the eye of the beholder. As you say, it should be in relation to a human need. We are twisting it, I know, but for memography, relevance is “some human tagged this page with the meme I am looking for.”

Note that we will still only have high precision and recall, since some taggers may make mistakes, there will be spammers, etc.

Perhaps we should call it “relevence” to distinguish it (with deference to Jacques Derrida and his “differance” that made such a difference, and to the “Relevence of Misspelling,” a topic I will address shortly).

Bob Doyle

Long and rambling response: http://www.shiftless.org/archives/2005/11/memetic_web_ann.html

Summary: I think you’re misapplying the recall measure (ratio of relevant documents retrieved to total relevant documents). A good definition of “relevant documents” should be in relation to a particular human-level information need, not simply “contains the query string”. In the terms you’ve defined, documents should be relevant in relation to the *meme*, not simply the meme id. Defining relevance in terms of the meme id only shows that Google crawls regularly and indexes very rare strings.

There was an announcement for the Memetic Web broadcast today on the SIG-IRList, a moderated search and information-retrieval mailing list I subscribe to. The basic idea seems to be a compromise on the semantic web and tagging (Flickr, technorati, del….