Motif Technologies, which was dropped in the second round of South Korea's sovereign AI foundation model project, formally filed an objection with the Ministry of Science and ICT on the 27th. With Motif requesting a reassessment, the dispute over the fairness of the evaluation that flared up around the announcement of the second-round results is dragging on.
In a statement released the same day, Motif said it had "formally filed an objection regarding the results of the second round of selection." The company added that it was "requesting detailed disclosure of the evaluation scores and a reassessment."
Motif pointed to the gap between its score on the Artificial Analysis Intelligence Index (AAII), a global AI benchmark, and its marks from expert users and members of the public, arguing that the government should disclose the detailed scores. Motif's AI model scored 41 on the AAII, the highest among the four companies competing in the second round. Upstage followed with 37, SK Telecom (017670) with 35 and LG AI Research with 31. But Motif posted the lowest overall score of the four and was eliminated, touching off debate in the information technology industry over the evaluation criteria.
In response, the ministry on the 20th disclosed each evaluation category, its weighting and the top-scoring company in each. According to the ministry, Motif recorded the highest mark of 11.9 in the AAII category, which carried 25 points, while LG AI Research led the expert evaluation, worth 35 points, with 29.5. SK Telecom posted the top scores in the benchmark assessment by the National Information Society Agency (NIA), worth 15 points, and in the evaluation by professional AI users, also worth 15 points, with 13.4 and 11.6 respectively. LG AI Research ranked first in the public evaluation panel category, worth 10 points, with 7.6.
Motif says the divergence between the AAII scores and the usability assessments is hard to accept. While saying it "respects and accepts the evaluation results themselves," the company stressed that "the AAII is not a simple benchmark but a comprehensive performance assessment that also covers usability." It added that "the core of this second-round evaluation is a comparison between AI models, and there needs to be a more concrete explanation, one the industry can understand, of why and how the results of a major global benchmark and the expert evaluation came out in opposite directions."
Motif said that "the essential improvement of AI model performance is what best fits the goals of the sovereign AI foundation model project," and that "from that perspective, Motif is raising an objection as to whether this review outcome and the ongoing discussion of the direction of the third-round evaluation are consistent with the goals the project should be pursuing." The company drew a line, however, saying the objection was not a claim that Motif should be chosen as a final operator of the project.
