Evaluator

Category	Evaluation
Description	Description Evaluator is a module to evaluate given question answering system, based on the user's choice of modules. Users can make use of configuration options to choose which modules to use for the steps of QA system. Once evalutor runs, it sends a natural language question in answer dataset to Controller one-by-one. It compares true answer of the dataset with the answer list from Controller to calculate accuracy of the system. Function Evaluator evaluates the overall QA system using answer set. For instance, let the answer set contains only two sentences, "Which rivers flow through Seoul?" and "Who is the author of 'Samguk Sagi'?" In addition, suppose that Controller success to give correct answer for the former question, but fails for the latter one. Then, Evaluator would return accuracy 1/2 = 0.5 to the user. Scope and limit The only dataset usable for the Evalator is NLQ-1 dataset, which contains question-answer pair for only Korean and English at the moment (March, 2016) with 40 questions in the set. Since Evaluator works for measuring accuracy of the given QA system, it is not suitable for analyzing each test case. For the purpose, you may use website for Controller or website for Evaluator (which supports dump file for each question). Issues and discussion * Is it possible to make use of another answer set like QALD-4? Not yet (January 13, 2016, OKBQA3.5). However, supporting mult-answer sets would be added at OKBQA4(http://4.okbqa.org) * Can I test my own QA system? Since Evaluator calls Controller to retrieve answers of each questions, only QA system following OKBQA system structure (TGM, DM, AGM) is applicable. This structure is targetted for those who want to add their own modules to the OKBQA system structure, and checks the performance of it. Maintainer: Sang-min An {asm427@kaist.ac.kr}
Maintainer	asm427@kaist.ac.kr
Source-code URL	https://github.com/machinereading/OKBQA_7_TASK1_EVALUATOR
Homepage URL	http://ws.okbqa.org/~b15_2015/
Web service URL	http://ws.okbqa.org:31999/evaluation
Sample cURL command	curl -i -H "Content-Type: application/json" -X POST -d '_sample_input_' _ws_url_ Test
Sample input	{ "language": "en", "data": [ { "id": "82", "lang": "en", "question": "In which programming language is GIMP written?", "answer": [ "http://dbpedia.org/resource/C_(programming_language)", "http://dbpedia.org/resource/GTK+" ], "type": "resource", "sparql": "PREFIX dbo: <http://dbpedia.org/ontology/>\nPREFIX res: <http://dbpedia.org/resource/>\nSELECT DISTINCT ?uri \nWHERE { \n\tres:GIMP dbo:programmingLanguage ?uri .\n}" } ], "config": { "address": { "TGM": [ "http://ws.okbqa.org:1515/templategeneration/rocknrole" ], "DM": [ "http://ws.okbqa.org:2357/agdistis/run" ], "QGM": [ "http://ws.okbqa.org:38401/queries" ], "AGM": [ "http://ws.okbqa.org:7745/agm" ], "KB": [ [ "http://kbox.kaist.ac.kr:5889/sparql", "http://en.dbpedia2014.kaist.ac.kr" ] ] }, "sequence": [ "TGM", "DM", "QGM", "AGM" ], "timelimit": 100, "n": 5 } }
Sample output	{ "config": { "KB": [ [ "http://kbox.kaist.ac.kr:5889/sparql", "http://en.dbpedia2014.kaist.ac.kr" ] ], "DM": [ "http://ws.okbqa.org:2357/agdistis/run" ], "TGM": [ "http://ws.okbqa.org:1515/templategeneration/rocknrole" ], "QGM": [ "http://ws.okbqa.org:38401/queries" ], "AGM": [ "http://ws.okbqa.org:7745/agm" ], "timelimit": 100, "N": 5, "sequence": [ "TGM", "DM", "QGM", "AGM" ] }, "result": { "timestamp": "18-08-03 12:34:40", "time_lapsed": 2.0503299236297607, "correct_answer": [ ], "partial_answer": [ { "id": "82", "question": "In which programming language is GIMP written?" } ], "false_negative_answer": [ { "id": "82", "question": "In which programming language is GIMP written?", "true_positives": [ "http://dbpedia.org/resource/C_(programming_language)" ], "false_negatives_up_to_n": [ "http://dbpedia.org/resource/GTK+" ], "false_negatives_number": 1 } ], "false_positive_answer": [ { "id": "82", "question": "In which programming language is GIMP written?", "true_positives": [ "http://dbpedia.org/resource/C_(programming_language)" ], "false_positives_up_to_n": [ "http://dbpedia.org/resource/ALGOL", "http://dbpedia.org/resource/IBM_RPG", "http://dbpedia.org/resource/Ruby_(programming_language)", "http://dbpedia.org/resource/C_Sharp_(programming_language)", "http://dbpedia.org/resource/ALGOL_60" ], "false_positives_number": 29 } ], "failure": [ ], "time_out": [ ], "correct_answer_rate": 0.0, "partial_answer_rate": 1.0, "false_negative_answer_rate": 1.0, "false_positive_answer_rate": 1.0, "failure_rate": 0.0, "time_out_rate": 0.0 } }