Science1 publisher2 min readPublished
A QUT model solves the whole expert panel at once to keep rankings from flipping
Small changes in expert answers can flip a multi-criteria ranking. Two QUT researchers propose solving the entire panel in one optimisation and favouring the experts who agree, and report steadier results without giving a magnitude.
The Scientist · Science desk
What happened
- QUT PhD researchers Omid Motamedisedeh and Faranak Zagia have published the Group-Consistency Best-Worst Method in the journal Array, aimed at group decisions on major infrastructure projects and public policy.
- Lead author Motamedisedeh said existing approaches can be highly sensitive to minor errors or changes in expert responses, and that the new model consistently produced more stable and reliable rankings.
- The model picks out a subset of respondents whose judgments are both individually consistent and closely aligned with one another, cutting the influence of noisy or conflicting responses.
- Co-researcher Zagia said a key finding was that the rankings stayed stable even when expert responses were deliberately altered during testing.
Compiled by The ScientistSomething wrong?How this is made
Why it matters
- constraint Stability under perturbation and accuracy are different properties, so a ranking that survives altered answers has not yet been shown to be the right ranking.
- exposure A dissenting expert who happens to be right is down-weighted by the same rule that suppresses fatigue and misread questions.
- decision An agency that adopts this has to publish which respondents the optimisation retained, because the weights no longer rest on the full panel it convened.
- precedent Once a published method reports passing perturbation tests, buyers of weighting exercises can ask any consultant for the same result.
The method solves the whole panel in one optimisation. In most group applications the panel is pooled first: the raw comparisons are averaged before the weights are solved, or each expert's solved weights are averaged afterwards [5]. Either way, the record of who agreed with whom is gone by the time a ranking appears. GC-BWM leaves every respondent in one problem, so agreement between respondents can enter the objective [5].
Motamedisedeh said: "Our research shows there is a way to make group decision-making more robust by looking not only at whether an individual's responses are internally consistent, but also how well those responses align with the broader group." [7]
The testing was adversarial by design. Zagia said, "We conducted extensive simulations and sensitivity testing to see how the model would perform when judgments changed," and that the method "maintained stable rankings significantly more often than conventional methods, even when individual responses were modified" [8][9]. The QUT announcement leaves both the number and the definition of a stable ranking to the paper, DOI 10.1016/j.array.2026.101176 [17][2].
Stability under perturbation is worth measuring. It is also the property a consensus filter gets most cheaply: any rule that narrows a panel toward mutual agreement shrinks the spread of the inputs, and inputs with a smaller spread move the output less when one response is nudged. Whether the retained subset sits closer to the correct ranking is a separate question, and simulation can answer it, because a simulated panel has a designed answer to check against. Motamedisedeh credited the "wisdom of crowds", where collective judgments can outperform individual assessments "when combined effectively" [13]. Selecting for agreement is one way of combining them, and it reduces the diversity of what gets combined.
Motamedisedeh named what the filter is aimed at: responses "affected by misunderstanding, fatigue or other forms of decision-making noise" [12]. "The goal is not to force everyone to agree," he said [11]. Misreading a pairwise question and holding a minority view about whether flood risk should outrank capital cost both show up in the data as distance from the group.
Zagia listed infrastructure planning, transportation, energy, sustainability assessment, risk analysis and policy development among the fields where the framework could be used [14]. On what it costs to run, she said: "Our research demonstrated that more reliable group decisions could be achieved without requiring experts to provide additional information or complete more complex assessments." [15]
What to watch
- Replication on real elicitation data rather than perturbed simulations, and whether the retained subset changes which project wins.
- Whether the Array paper reports the share of a typical panel the optimisation keeps.
- A third-party comparison on the same panel data against the aggregation methods agencies use now.