Incorporating stakeholders’ knowledge in group decision-making

The objective of this paper is to characterize the group decision-making process. Three basic questions should be considered in group decision-making: (1) how to extract stakeholders’ knowledge and preferences, (2) how to combine these preferences and knowledge coherently and (3) how to conduct discussions and resolve conflicts. To respond to these questions, this paper distinguishes and discusses several elements characterizing the group decision-making process. More specifically, the paper discusses the following topics: techniques for acquiring stakeholders’ knowledge, aggregation strategies and rules, preference parameters elicitation techniques, stakeholder weighting and conflict resolution. The paper also includes a brief discussion of several real-world case studies in which the authors of this study participated.


Introduction
Decision-making can be regarded as the cognitive process resulting in the selection of a course of action from several alternative scenarios. Decision-making problems range from everyday decisions, such as the selection of an itinerary to go from one place to another, to more complex decisions involving a large number of individuals, associations, or socio-economic groups. Complex decisions generally require the development of formal methods to handle them. Several methods have been proposed in the literature, including cost-benefit analysis, statistical techniques, evolutionary algorithms and multicriteria analysis. The majority of decision-making methods assume a single decision maker for simplicity, whereas real-world problems requiring decisions naturally imply multiple decision makers with conflicting objectives and distinct value systems. This multiplicity of objectives and value systems derives directly from the multidimensional nature of decision problems. The situation is complicated by what Hendriks (1995) calls the 'social context' of the decision problem that creates conflicting objectives, reflecting the diversity of interests and concerns of the stakeholders in the decision-making process. Indeed, the identical decision problem is perceived, designed, formalized and modeled differently by people with different perspectives, such as environmentalists, politicians or economists. Each stakeholder in the decision-making process has a different perception of the decision problem according to his own objectives and concerns.
The objective of this paper is to characterize the group decision-making process. Three basic questions should be considered in group decision-making: (1) how to extract stakeholders' knowledge and preferences, (2) how to combine these preferences and knowledge coherently and (3) how to conduct discussions and resolve conflict situations. In this paper, we discuss the following topics: techniques for acquiring stakeholders' knowledge, aggregation strategies, aggregation rules, preference parameter elicitation techniques, weighting stakeholders and conflict resolution. The first topic is related to the first question that should be considered in characterizing group decisionmaking. The last topic corresponds to the third question. The remaining topics are related to the second question regarding the characterization of the group decision-making process. The paper also includes a brief discussion of several real-world case studies in which the authors of this study participated. It is important to mention that this paper is neither a survey nor a comparative study of group decision-making. Rather, it represents the authors' point of view based on their personal research and experience. It is also important to mention that the terms 'stakeholder', 'decision maker' and 'expert' are used in various manners in this paper. Indeed, in real-world decision problems, these terms may refer to different individuals or groups of individuals who may or may not be directly involved in the decision-making process.
The rest of the paper is organized as follows: Section 2 discusses different techniques for acquiring stakeholders' knowledge. Section 3 presents the different aggregation strategies. Section 4 summarizes the different aggregation techniques and rules in group decision-making. Section 5 discusses the different methods to elicit preference parameters. Section 6 addresses the weighting of stakeholders. Section 7 discusses conflict resolution. Section 8 discusses several real-world case studies. Section 9 presents the paper's conclusions.

Acquiring experts' knowledge
The aim of this section is to discuss the different techniques used to extract and represent stakeholders' knowledge and preferences in group decision-making. Several techniques have been proposed in the literature to acquire and represent experts' knowledge. In addition, a multitude of strategies have been advanced to manage certain conflicts in expertise and reflect experts' knowledge. One strategy identifies the expert most competent to acquire and model knowledge. Another strategy aims at focusing only on the consensual knowledge of the experts, knowledge on which various experts agree. Other strategies have also been proposed solely to acquire knowledge such as 'the strategy of the biggest generalization'. This strategy focuses only on the most general knowledge and avoids specific knowledge. In this context, Labidi (1996) advances two approaches to acquire knowledge from multiple experts: Incremental acquisition of knowledge: This technique focuses on first acquiring knowledge from one expert and then progressively refining the knowledge with subsequent experts. Combination of expertise models: This technique focuses on acquiring knowledge separately from each expert. Next, the knowledge engineer combines the various experts' models to create an integrated model that is based on consensual knowledge combining different points of view. The 'technique of consensual decision-making' proposed in McGraw and Seale (1987) allows identifying the optimal solution to a problem. This technique is based on the vote of a group of experts on a solution identified during, for example, a brainstorming session. This technique comprises several steps: (1) Presentation of the technique to the experts.
(2) Identification of the problem and possible solutions presented by the knowledge engineer.
(3) A first round of voting: Each expert may choose three solutions with at most two rationales per option. Propositions that receive a low number of rationales are eliminated. (4) A second round of voting: if, after being reduced, the number of solutions remains too high, experts proceed to a second round of voting. In this round, each expert must present no more than two options with no more than one rationale per option. Solutions that have less than a certain number of votes are eliminated; a discussion can occur if necessary. (5) Continuity of the voting process until a unanimous option is agreed upon. (6) Finally, the knowledge engineer checks the degree of confidence of everyone regarding the obtained option. Huseman (1977) proposed the 'Approach of the Nominal Group'. The experts meet together but work independently. Every expert 'silently' creates a certain number of solutions to a problem. Next, the knowledge engineer selects the most pertinent ideas. These ideas are then expanded. A secret vote to prioritize ideas from the best to the worst then occurs. At the end, to reach a consensual solution by experts, other discussions may follow.
The method of 'Brainwriting' was proposed by Boy (1988). This strategy gathers knowledge from different experts to produce ideas within a group. It is based on the formalization of the experts' expertise on a precise subject. At the beginning, the subject is described on sheets of paper available in the middle of the table. Each expert takes a sheet, reads it, writes some notes on it and places it back in the middle of the table. Each expert must write his ideas on all of the papers. Next, a collation of the principal ideas is created. The consensus rate is thus calculated by comparison. This method permits the gathering of a great many ideas.
Moreover, other methods of the 'logic design' type have been proposed in the literature to model the knowledge used or created in a project and formalize memories of projects. These methods allow the preservation of all information related to decisions made by designers on a predefined project. This information concerns the reasons why certain solution options were explored, why some were eliminated, the project context and what occurred during the project.
These methods were first developed to document the decision process in the design field. Today, however, such methods permit the capitalization of knowledge in a project memory (Dieng-Kuntz & Matta, 2002). Of these methods, we will deal only with those that utilize applications such as the Issue-Based Information System (IBIS) and the Question, Options, Criteria method (QOC).
The IBIS method was proposed by Horst Rittel et al. in the 1970s (Buckingham-Shum, MacLean, Bellotti, & Hammond, 1997 to support communication within designers' groups and formalize decision-making in a framework of design projects. The IBIS is a process-oriented approach, structured according to the three different steps of the decision-making process: issues (questions), positions (positions) and arguments (arguments). In fact, when a primary question is presented, a participant in the decision-making process takes a certain position not only by suggesting a solution to the issue presented but also by justifying his position using arguments. Other participants, however, present other positions by proposing other solutions to the identical problem and developing other arguments. These arguments can reinforce a position or oppose other positions. During discussion, new issues may appear.
The QOC method supports the decision-making process in a design activity and is also a process-oriented approach. The QOC allows the presentation of different solutions and includes the advantages as well as the drawbacks of each solution. The use of the QOC method requires precisely specifying the problem, whether the idea concerns an issue, an option or a criterion. Then, the idea must be formalized and well-structured to be added to the diagram.
The different methods previously proposed by knowledge engineering and the 'Rational design' methods are quite appropriate for the acquisition and preservation of the knowledge used and produced by a multitude of experts. However, the application of these methods becomes impractical in complex projects (a great many professions and/or experts, collaboration with the external environment involving many partners and suppliers). Moreover, experts must be mobilized to acquire their knowledge and then validate its modeling, which indicates that experts may be diverted for a certain period of time from performing tasks that are of immediate value to the company to accomplish operations of knowledge capitalization that generate value for the long term.

Perspective aggregation strategies
Two main strategies that may be used to combine the judgements and perspectives of decision makers in group decision-making are at the input level or the output level (Dias & Climaco, 2000). It is also possible to combine the input and output levels to reduce or avoid their respective shortcomings, as will be explained later in this section. In the rest of this section, utilizing h ≥ 2, we design the number of decision makers.
The input-oriented aggregation strategy appears to be technically and practically easier to implement. However, this strategy assumes that decision makers can collaborate effectively and that there is little conflict regarding input data.

Output aggregation strategy
The output level aggregation strategy proceeds as follows ( Figure 1b): (1) the operator w(.) yields the individual results R i (i = 1,...,h), and (2) an operator k(.) aggregates the individual results R i (i =1,...,h) into a final result R. The operator w(.) may be any type of approach that yields the desired result. The aggregation performed by k(.) may comprise averaging, minimizing a distance measure, voting, etc. Previous research includes examples of proposals based on this strategy (Bregar, Gyorkos, & Juric 2008;Greco, Mousseau, & Slowinski, 2009;Jabeur & Martel, 2007b).
The output-oriented strategy appears to be more flexible than the input-oriented strategy because the implied decision makers can work independently. This is an important aspect of the decision-making process because it is appropriate for decision makers who are geographically distributed and/or who have time constraints. Furthermore, the output-oriented aggregation strategy appears to be more appropriate for decision situations in which the judgements and objectives of implied participants are conflicting. However, the main shortcoming of the output aggregation strategy is its technical complexity and difficulty of use in practice.

Mixed input-output aggregation strategy
To take advantage of both aggregation strategies and to avoid or reduce their respective shortcomings, we designed a mixed aggregation strategy that is shown in Figure 1c. The proposed mixed input-output aggregation strategy proceeds as follows: (1) An operator w(.) yields the individual results R i (i = 1,...,h); (2) an operator a(.) uses the common input information I (e.g., common preference parameters) and the individual results R i (i = 1,...,h) to construct the collective input S; (3) an operator w'(.) yields the final result R.
As shown in Figure 1c, the first phase of the mixed aggregation strategy corresponds to the first phase of the output aggregation strategy, and the third phase of the mixed aggregation strategy corresponds to the second phase of the input aggregation strategy. The second phase of the mixed aggregation strategy combines the first phase of the input strategy and the second phase of the output strategy in the sense that it leads to the collective input S (as in the first phase of the input aggregation strategy) by aggregating the output of individual results R i (i = 1,..., h) (as in the second phase of the output aggregation strategy).
We think that the mixed aggregation strategy combines the advantages of both the input and output aggregation strategies. In particular, the decision makers can work independently during the first phase; however, all of them should agree on the final result of the third phase. This agreement requires some collaboration between decision makers. The second phase is an automatic phase, and there is no need for intervention from the decision makers.

Aggregation rules
One of the most important issues in group decision-making is defining appropriate techniques to combine individual preferences. The importance of this issue has been addressed in previous research as in Ben Khelifa and Martel (2001), Cook (2006), Martel (2007a, 2007b). Ideally, one can define an aggregation rule that reproduces the 'totality' of individual preferences. Unfortunately, this is not often possible in practice because decision makers generally have conflicting objectives and preferences.
Three categories of aggregation rules may be distinguished (Azibi & Vanderpooten, 2002): statistical, functional and rule-based techniques. The main advantages of statistical operators are their compactness and simplicity. The functional aggregation rules are based on using functions, i.e., a weighted-sum or distance measure. Researchers such as Brigui-Chtioui and Saad (2011), andMartel (2007b) have presented examples of proposals based on functional aggregation. Rule-based aggregation techniques are based on using Boolean and/or 'IF...,THEN...' rules. These rules apply to complex situations for which statistical or functional aggregation rules cannot be applied (Azibi and Vanderpooten, 2002). An example of rule-based aggregation techniques use is presented in Azibi and Vanderpooten (2002).
In some situations, aggregating the input data is simply based on discussion among different decision makers as in Damart et al. (2007), Mercat-Rommens, Chakhar, Chojnacki and Mousseau (2010) and Saad, Rosenthal-Sabroux, and Grundstein (2005). In Mercat-Rommens et al. (2010), for instance, although several decision makers have been implied, the collective input has been defined by the responsible project manager based on her expertise and the information presented by different decision makers.
The aggregation rule should permit reaching consensus by considering the points of view of decision makers that are favorable and those that are not. The majority rule, which is characterized by its simplicity, anonymity and neutrality and its low demand in terms of computational time, appears to be an excellent candidate for the aggregation mechanism. There are different manners in which to implement the majority/veto principle. Generally speaking, the majority/veto principle requires two conditions: (1) there is a 'sufficient' majority of decision makers who are favorable to a given decision, and (2) when the first condition holds, none of the minority of decision makers presents an 'important' opposition to this decision.
Let us assume that the decision problem concerns the assignment of decision objects to different decision classes, Cl 1 ,...,Cl n . These decision classes may, for instance, represent different levels of risk as in Chakhar and Saad (2012). The above conditions can be implemented as follows: an object x is assigned to a class Cl t if and only if (1) at least λ = 50% (any other value in the range of 50-100% may also apply) of decision makers agree with this decision and (2) at most θ = 25% (any other percentage in the range of 0-49% may also apply) of the minority of decision makers oppose this decision (i.e., assignment of x a class Cl t ). This idea has been applied in Chakhar and Saad (2012).

Preference parameter elicitation approach
In decision-making, decision makers are often called upon to specify parameters required by the aggregation rule such as the weight of various criteria. This remains the case with group decision-making. We distinguish two major approaches to specifying preference parameters (Dias & Climaco, 2000): direct or indirect. It is also possible to combine these two strategies.
The direct elicitation approach is the most used in practice. In this approach, decision makers explicitly specify the values for all preference parameters. This is not an obvious exercise in practice, especially in group decision-making. Indeed, specifying these parameters requires a significant cognitive effort from the experts. In addition, in this approach we assume that all implied experts understand the meaning and role of these parameters and that all of them agree on the values of these parameters.
In the indirect elicitation approach, parameter values are implicitly obtained. The basic idea of this approach is to deduce values for preferred parameters from holistic information supplied by the decision makers. A well-known approach to indirect elicitation is the case-based reasoning method. Some authors design the case-based reasoning approach by aggregation/disaggregation as in Dias, Mousseau, Figueira, and Climaco (2002). Case-based reasoning can be structured in three stages: (1) definition of the holistic information, (2) inference of preference values, and (3) use of inferred values. The objective of the first stage is to specify the holistic information. Holistic information is global judgements on decision objects and thus represents aggregated information. The objective of the second stage is to infer the values of preferred parameters using the holistic information as input. The values of inferred parameters are obtained by disaggregating the global information provided by the decision makers. At the end of this stage, the decision makers should agree on the inferred values. Otherwise, the two first stages can be repeated using different input data. The objective of the third stage is to use the obtained results to apply the aggregation rule. This approach has been used in different real-world decision problems (e.g. Chakhar & Saad, 2012;Damart et al., 2007;Guay, Joerin, Chakhar, Villeneuve & Lavoie, 2011;Mercat-Rommens et al., 2010).
The case-based reasoning approach has two main advantages. First, it substantially reduces computing time. Second, case-based reasoning reduces the cognitive effort required from the experts. This second advantage is particularly important in practice. Indeed, we think that experts are more cooperative in producing assignment examples than giving exact values for the different parameters. This fact has been confirmed by real-world decision problems (see Chakhar & Saad 2012;Guay et al., 2011).
The direct and indirect elicitation approaches can also be used jointly as in Mercat-Rommens et al. (2010). The basic idea of a mixed direct-indirect elicitation approach is to fix the values of certain parameters and to infer the values of others. The process is repeated several times to ensure the stability of the inferred values. The main argument for using the mixed approach is the substantial gain in terms of computing time compared to the use of the indirect approach alone and the relatively low cognitive effort compared with the use of the direct elicitation approach alone.
In spite of the attractiveness of case-based reasoning, one should mention that the quality of the output depends largely on the quality of the input.

Weighting stakeholders
In group decision-making, it is often assumed that the decision makers have different 'powers' or 'weights'. This fact is recognized by different authors, including Cook (2006) and Martel (2007a, 2007b). We may distinguish several manners in which to define these weights: (1) weights are defined explicitly by a mediator or an external person; (2) weights are defined based on the hierarchical levels of implied decision makers; (3) weights are defined explicitly using a given method; (4) weights are defined implicitly based on input.
Although the first method appears to be quite subjective, it may be useful in practice, especially when the expertise and neutrality of the mediator is well recognized by the different implied decision makers. An example of using this method is provided by Leyva-Lopez and Fernandez (2003). In the second case, weight generally reflects the hierarchical levels of decision makers in the organization. This method can be justified in some decision situations, especially when one or some of the decision makers are financially involved and the final decision has serious consequences for their investment. In addition to the usefulness of the two first cases in practice, there is a need for more formal and objective methods, as indicated by Cook (2006). Researchers such as Ramanathan and Ganesh (1994) thus propose using the Analytic Hierarchy Process (AHP) (Saaty, 1980) method to derive members' weight. Martel and Ben Khelifa (2000) proposed a method of using individual outranking indices. Jabeur et al. (2004) proposed using a previous method (Zeleny, 1982) to integrate the relative importance of the groups' members into the consensus construction.
In some situations, the weight is not used, as in Saad and Chakhar (2009) and Greco et al. (2006). However, this may be perceived as an implicit use of weight in the sense that all decision makers are assumed to have identical power.
We think that the most important characteristic of weight definition methods is the objectiveness of weight. In this respect, Cook (2006) advocated that the question is not how to use weight, but rather how to objectively quantify weight. In Chakhar and Saad (2012), for instance, the contribution of each decision maker to the collective decision is measured by the quality of input data provided by the decision maker. We think this a more objective manner than more conventional methods. Indeed, generally, more experienced decision makers provide input data and knowledge of higher quality. This leads to more 'democratic' decisions because decision makers are only differentiated based on their level of 'expertise' and not on their hierarchical levels. This is confirmed in different real-world applications in which the authors were engaged; see Mercat-Rommens et al. (2010), Saad et al. (2005) and Saad and Chakhar (2009).

Conflict resolution
Supporting conflict resolution is an important characteristic of group decision-making. In some proposals, conflict resolution is simply addressed by discussion. This is the case with Damart et al. (2007) and Saad et al. (2005). Some proposals use advanced tools to resolve the conflict between different decision makers. For instance, Brigui-Chtioui and Saad (2011) proposed to use multi-agent models to address conflict situations. Another example is the ELECTRE-GD (Elimination and Choice Expressing Reality-Group Decision making) method proposed by Leyva-Lopez and Fernandez (2003). This method contains two phases. The objective of the first phase is to solve the conflict between the information provided by individual rankings and fuzzy relations among preferences. The objective of the second phase is to define a fuzzy relation that is highly ranked for a Supra Decision Maker, which should be exploited in some rational manner to derive a final group ranking. Finally, some proposals do not support any conflict resolution, such as in Bi and Chen (2007), Chakhar and Saad (2012) and Greco et al. (2006).

Case studies
The objective of this section is to discuss some real-world case studies in which the authors have been involved. The description of these cases with respect to the discussed characteristics of group decision-making is summarized in Table 1. Further information can be found in the given references.

Management of post-accident nuclear risk
The PRIME project (research project on the application of radioecological sensitivity indicators and multi-criteria methods to industrial site environments) considered here concerns the management of post-accident nuclear risk in southern France. A full description of the project is available in Mercat-Rommens et al. (2010). The objective of PRIME was for the decision makers, the stakeholders and representatives of the territory to jointly develop a multicriteria evaluation approach to analyze and characterize the contaminated territory. This evaluation would be utilized individually by the managers of the risk. The evaluation approach would allocate to each district of the study area a degree of weight representing the risk to the district of a nuclear accident resulting in contaminants released into the atmosphere. The PRIME project was a stimulating project and an excellent opportunity to apply decision-making tools in practice and to discuss issues and meet with different stakeholders who naturally have different objectives and preferences. The project was also an important occasion on which to address and discuss different conceptual and technical problems. One of the most important problems encountered during the PRIME project concerned the definition of preference parameters. In the PRIME project, the multicriteria classification method ELECTRE TRI (sorting) (Figueira, Mousseau, & Roy, 2005) was used for different simulations. During PRIME, preference parameters (especially criteria weight) required by ELECTRE TRI were inferred using the IRIS software (Dias & Mousseau, 2003), which supports the aggregation/disaggregation approach proposed in Dias et al. (2002).
The second thoroughly discussed issue during the PRIME project was the manner in which the preference information of the different implied stakeholders was incorporated into the decision-making process. The aggregation/disaggregation approach of Dias et al. (2002) is designed for single decision makers and thus could not be applied in this case. For simulation purposes, the project manager responsible informally aggregated the assignment examples provided by the decision makers, and then the aggregation/disaggregation approach of Dias et al. (2002) was used.

Heat islands exposition in the QMC
This case study was conducted as part of an action-research project whose purpose was to strengthen the resilience of the Québec Metropolitan Community (QMC, Québec, Canada) to climate change (Dubois, Bergeron, Potvin, & Adolphe, 2012). We used a methodology of four phases. The objective of the first phase was to assess and structure the input data. It involved the identification and assessment of evaluation factors and the definition of a risk scale. In this particular application, a list of seven factors was used. Next, the experts jointly designed an ordinal relative risk scale of six levels from the lower relative risk (1) to the highest relative risk (6) to compare areas in the QMC. In the second phase, each team (1) identified a set of learning examples and (2) used the Dominance Rough Set Approach (DRSA) (Greco et al., 2001) to approximate and summarize the information provided by the experts. The objective of the third phase was to combine the output of the previous phase and then infer a set of collective decision rules. The fourth phase utilized the inferred decision rules to classify all sectors of the QMC.
In this application, the result obtained by the aggregation procedure and the results directly and jointly specified by the experts during data collection sessions coincide with approximately 65% of the decision objects. One of the practical problems faced during this application was the lack of information. Indeed, initially, a large number of attributes were identified by the experts. However, only seven attributes for which data were available for all districts of the QMC were used to produce final risk maps. Another problem concerned the quality of information. In fact, the resolution of the data used was 250 × 250 m. The effect of low resolution explains the observed difference between temperature maps executed by the National Institute of Public Health of Québec and the maps obtained by the proposed risk exposition approach. Furthermore, although the aggregation procedure requires the definition of several parameters, experts may not agree on these parameters' values. A possible solution to this problem could be the use of an indirect elicitation approach.

Management of company's crucial knowledge
Given the high cost of the operations of capitalization as well as limited resources, a French car manufacturer must focus the investment he intends to make on the knowledge that he deems 'unavoidable knowledge' (tacit and explicit) to render it accessible to those who need it. It is a question, for him, of not losing touch with this type of knowledge, to preserve the feedback on experiments on the projects and to transfer this knowledge to less-experienced engineers. Thus, the problem of the car manufacturer was to determine the crucial knowledge mobilized and produced during the realization of the development projects of the car products. In this context, we have proposed a multicriteria method to enable those in charge of the projects to present arguments regarding what knowledge should be preserved and shared. Moreover, the choice of this knowledge results from a process of collective decision-making established by a combination of multiple project deciders' viewpoints.
Thus, we have based our method on the use of models and tools reserved for the modelling of the processes for the acquisition of knowledge held by the experts to identify knowledge (potential actions) and build criteria.
Depending on the evaluations of knowledge using the different criteria specified by the deciders, the mediator asks each decision maker to assign the 'reference knowledge' to only one of the following ordered decision classes: Cl1: decision class of 'non-crucial knowledge', corresponding to knowledge whose inclusion has been shown to be unnecessary, or Cl2: decision class of 'crucial knowledge', corresponding to knowledge shown to be necessary for inclusion.
The decision is to assign knowledge to the decision classes of 'crucial' or 'non-crucial' knowledge. Hence, we obtain a number of decision tables that equal that of the decision makers. Each decision table contains the values F (Ki, gj) corresponding to the evaluation of each item of knowledge Ki by each criterion gj as well as its association with either decision class.
The decision table corresponding to a decider comprises 32 lines and 15 columns. Each line is associated with one of the 32 pieces of 'reference knowledge' that we want to order and is characterized by 15 criteria. The last column is reserved for the decision made by the decider concerning each piece of 'reference knowledge'. From the decision tables of each decider (expert), the mediator has used one of the inference algorithms (DOMLEM [Learning from Examples Module based on Dominance relation], Explore) proposed in the DRSA method to infer for each decider the decision rules consistent with his allocation examples. The mediator analyzes, with each decider, the set of rules inferred from examples presented by each of the deciders. The mediator first determines whether the rules show some defect. These defects may originate from (1) the hesitation of the decider while allocating knowledge in the decision class, (2) the change in the deciders' viewpoints during the decision process. During experiments in the field, we noticed that the decider can change his mind concerning the evaluation of knowledge over a criterion (depending, for instance, on the validity of his knowledge); or (3) the inconsistency of the criteria (a missing criterion, an extra criterion, etc.).
Once the origin of the defect is identified, the mediator corrects it with the decider. He proceeds incrementally because defects are identified in the rules base and the decider intends to modify the allocation examples and/or the criteria. This iterative process allows a better understanding of the decision rules chosen by each decider. In addition, the analysis of these rules can be conducted during individual meetings of an average duration of 2 hours. For each decider, two types of decision rules are determined: those associated with knowledge belonging certainly to the decision class of 'crucial knowledge' and those rules concerning knowledge that belongs to the 'crucial knowledge' class. Among these rules, the presider must only retain the 'reference knowledge' clearly belonging to the decision class of 'crucial knowledge'. After analyzing the different inferred rules, a set of rules is maintained for each decider.
The mediator has defined a unique set of rules corresponding to a set of rules collectively accepted by the deciders using decision rules retained for each one of them. The construction of this set is conducted using a constructive approach based on Belton and Pictet's (1997) work. In effect, we have used the comparison technique proposed by Belton and Pictet (1997). Thus, the mediator has helped deciders determine a set of decision rules that are collectively accepted and based on the different rules retained by each one of them. The quality of these rules has been determined while testing them on examples of allocations of new knowledge by the same deciders.

Conclusions
In this paper, the authors have provided a personal view of acquiring knowledge in group decision-making. The paper discusses several points that are directly related to three basic questions in group decision-making: (1) extracting stakeholders' knowledge and preferences, (2) combining these preferences and knowledge and (3) resolution of conflict. The second question has been addressed in terms of aggregation strategies and rules, preference parameter elicitation techniques and weighting of stakeholders. The paper also includes a brief discussion of several real-world case studies in which the authors of this study participated. This paper may serve as a departure point for a survey and review paper. It will also be interesting to compare the different methods and approaches discussed more formally and in further detail.