How To Compare Two Ranking Algorithms?

[mk65]

Hello all,
 
I want to compare two ranking algorithms. In these algorithms, client specifies some conditions in his/her search. According to the client`s requirements, these algorithm should assign a score for each items in data base and retrieve items with highest scores.
 
I have searched the net. According to my searches, the most relevant article which explains about some metrics for comparing ranking algorithms, was this:
http://www.icml2010.org/papers/504.pdf
I think prec@k, MAP, MRR, and NDCG, are good metrics to use, but I have a problem:
 
My algorithm sort results, so the first item in my result list is the best one with highest score, the second result have the second top score, and so on. I limit my search algorithm to for example find 5 best results.The results are the most top 5 items. So,precision will be 1. When I limit my search to find best result,It finds the best one. Again, precision will be 1.But the problem is that,it is unacceptable for people who see this result.
 
What can I do? How I can I compare these algorithms and show one is better than the other?
Please help me. Thanks a million.

[Jan]

Hello mk65,

 

If I understand correctly, it is better to look for "the best algorithm" for you, because algorithms were created to solve specific tasks. So, some algorithm are better for something, and other are better for others.

 

Do you want to use algorithms on your Search DataPages?

[mk65]

Hello mk65,

 

If I understand correctly, it is better to look for "the best algorithm" for you, because algorithms were created to solve specific tasks. So, some algorithm are better for something, and other are better for others.

 

Do you want to use algorithms on your Search DataPages?

 

Hello.

Thanks for your attention.

No. I don`t want to use a search algorithm.

I have written an algorithm for a specific search in a topic, which have used ranking algorithms. In addition, there is another algorithm in this topic. I have implemented both of these algorithms.  I should prove my algorithm is better than the other. I think prec@k, MAP, MRR, and NDCG, are good metrics to use, but I have the  problem mentioned in my question. I don`t know how to solve this? How to show my algorithm is better than the other in this topic?

Please help me.

Thanks a lot.

[futurist]

Hi @mk65

You can do something like this: