Int J Adv Manuf Technol (2008) 39:168–181 DOI 10. 1007/s00170-007-1189-3 ORIGINAL ARTICLE Evaluating the performance of an ERP system based on the knowledge of ERP implementation objectives Chun-Chin Wei Received: 4 September 2006 / Accepted: 25 July 2007 / Published online: 9 September 2007 # Springer-Verlag London Limited 2007 Abstract Success is often defined as a favorable or satisfactory result or outcome.
Since the objectives of an Enterprise Resource Planning (ERP) implementation project reveal exactly the states of an ERP system, which managers wish to realize, the knowledge of these objectives can indicate how outcomes of an ERP system execution must be measured in the ERP success analysis. This paper proposes a comprehensive framework for assessing the performance of an adopted ERP system.
Based on the knowledge of ERP implementation objectives, the framework can systematically identify the appropriate ERP performance indicators, construct the performance indicator structure, and set up consistent evaluation standards for facilitating the complex ERP performance evaluation process. A real-world example demonstrates the feasibility of the proposed framework. Keywords Enterprise Resource Planning system . Performance assessment . Information system . Decision-making usiness processes integration to automate the flow of material, information, and financial resources among all functions within an organization using a common database [1]. A comprehensive ERP implementation project involves selecting an ERP software system and a cooperative vendor, managing business processes change (BPC), implementing the system, and examining the practicality of the adopted ERP system [2]. Many companies devote their attention to selecting and implementing an ERP system, but they ignore evaluating the effectiveness of the adopted ERP system.
Unfortunately, information managers are often swamped by the diverse requirements of users, instead of evaluating the advantages and drawbacks of the ERP system and further improving its performance. However, given the considerable financial and labor investment and potential risks and benefits, the importance of a pertinent ERP system performance evaluation cannot be overemphasized. The key reasons why a company should assess the performance of their ERP system are presented below. 1. ERP systems are more complicated than any traditional information system (IS). Installing an ERP system requires large investments of money, time, and energy. . All business processes of an organization are involved in ERP systems. The adopted ERP system will influence all business operations, even strategies, in the future. 3. ERP systems are built on generic business rules and procedures. Implementing an ERP system requires customizing and tailoring the work processes based on the business practices of the company. Many current business processes may be reengineered and changed. 4. A successful ERP system cannot only meet the requirements of current business functions, but also needs to satisfy the future requirements.
Continuous upgrade and maintenance are very important. 1 Introduction Due to intense market competition and the progress of information technology (IT), many companies have implemented Enterprise Resource Planning (ERP) systems to enhance their competitiveness. An ERP system is a complex enterprise information system (EIS) based on the C. -C. Wei (*) Department of Industrial Engineering and Management, Ching Yun University, Chung Li 320 Taiwan, Republic of China e-mail: d887801@cyu. edu. tw Int J Adv Manuf Technol (2008) 39:168–181 169
Additionally, after implementation of the ERP system, managers need to ensure that the adopted system supports the competitive environment and fit the essential system characteristics. Furthermore, managers would also like to understand which parts of their ERP system need improvement and to determine the direction for the system to take. Success is often defined as a favorable or satisfactory result or outcome [3]. The objectives of an ERP implementation project describe exactly the states of an ERP system which managers wish to realize. Knowledge of the objectives can help in dentifying and selecting ERP alternatives, and beyond that, the objectives indicate how outcomes of an ERP system execution must be measured and what kinds of uncertainties should be considered in the ERP success analysis. Then, the explicit definition of “the success of an ERP system” can be defined that the adopted ERP system really achieves the objectives that the managers strive for. An ERP performance evaluation is necessary to align the ERP performance indicators (PIs) with the ERP implementation objectives and to highlight the effectiveness of the ERP system.
Since the business environment is characterized by high uncertainty and ERP systems become more organizationally encompassing, the process of ERP system performance assessment involves numerous problems. First, Motwani et al. [4] noted that ERP implementation involves initiating appropriate business process changes as well as IT changes in order to significantly enhance performance, quality, costs, flexibility, and responsiveness. The performance evaluation of an ERP system should incorporate multiobjective and multi-criteria, including system functions and organization impact dimensions.
Such a method makes it possible to include multiple aspects of ERP system success and perform longitudinal trace. Second, Jurison [5] emphasized that complex interactions between technology and organizational users during system implementation make it difficult to compare and develop a consistent theory. Evaluating PIs tends to be inconsistently defined or measured, due to common criteria that do not satisfy various considerations of different companies. The third problem arises from the adoption of conventional inputoutput accounting methods, e. g. , turnover, return on investment (ROI), and so on [6].
Many PIs are difficult to quantify in terms of these economic efficiency statistics. Finally, the question is how to link the objectives that managers want to attain and the PIs used to measure whether the adopted ERP system accomplishes these objectives. This study aims to construct a framework to elaborate the process of the ERP PIs development to link the ERP performance-measurement contents with the ERP system implementation considerations. The framework allows managers to extend the objectives of the ERP implemen- tation project into suitable ERP PIs and to evaluate whether those objectives have been achieved.
The PIs are specified to provide detailed guidance for the ERP system performance assessment. Hence, the proposed methodology can verify that the ERP performance evaluation process is in line with the goals of the enterprise and guide the strategic plan for the ERP system improvement. An empirical case in Taiwan is described to demonstrate the practical viability of the proposed method. 2 IT performance evaluation method review Various methods have been applied to evaluate the performance of ERP or other IS. For example, the financial analysis is a conventional means, like return on assets (ROA), ROI, and cost–benefit analysis [7, 8].
Most of the researches reveal that investment in IS pays off only in the long term [9]. Even Hunton et al. [7] found no pre- to postadoption improvement in financial performance in firms that adopted ERP, but that the financial performance of nonadopters decreased over time while it held steady for adopters. However, the financial analysis may seldom suffice, ignoring other critical qualitative aspects, like system quality and its impact on the organization and the individual. Instead, some subjective and surrogate measures need to be developed.
The output of an IS or ERP system can be measured in different aspects, like technical factors, effectiveness factors, and users’ experience factors. A large numbers of studies have been proposed and identified many success variables. Some researchers and practitioners have been calling for the use of techniques that focused on case studies, exploratory method, and meta-methodology [1, 4, 6, 10]. Empirical explorations are helpful to clarify the vague business environment or situation and contribute some critical evaluation guides and PIs in the ERP or IS performance assessment process.
A great deal of effort has been made on the system and information quality assessment of IS [11–14]. The quality assessment reflects the engineering-oriented performance characteristics of the system itself [15] and the quality of information and data. Researchers believe that a conscientious system development process leads to a high quality IS whose use has a positive impact on the organization. User satisfaction and usage are still critical issues [3, 16, 17]. Pertinent studies evaluated the IS performance by using the experience and perspective of various users, like employees, middle managers, top managers and system engineers.
The perceived satisfaction is a paramount requisite to the eventual success of an IS, including overall satisfaction, information satisfaction, software and hardware satisfaction, and decision-making satisfaction. 170 Int J Adv Manuf Technol (2008) 39:168–181 DeLone and McLean [15] surveyed 180 articles from seven publications to categorize IS success measures into six major dimensions, including system quality, information quality, use, user satisfaction, individual impact, and organization impact. They created an IS success model and proposed that a systematic combination of these six measures can create a measurement instrument.
This study offers many critical directions to the later IS performance research efforts [6, 18]. Although the performance of IS has been an objective of study for a long time, there is little literature to report how to develop tailored PIs and assess the performance of an ERP system practically. Recently, a few popular techniques in measuring the performance of an IS have been reported. For example, Hagood and Friedman [19] implemented the Balanced Scorecard-based performance-measurement system for assessing the performance of a human resource information system.
Stensrud and Myrtveit [20] applied Data Envelopment Analysis (DEA) to model the productivity measurements of outstanding ERP projects. Lin et al. [21] applied a statistical method to ERP implementation by providing a pair of performance indices, i. e. , action and importance. They used quality function deployment (QFD) to remedy the weakness and develop corporate strategies to improve the ERP implementation. As mentioned earlier, previous literature has provided many useful PIs for ERP or IS performance evaluation.
However, the most frequently adopted PIs refer to the common indices without developing tailor-made measures echoing the objectives of the ERP implementation project. Additionally, the relationship between these objectives and the ERP PIs developed in the ERP use phase has seldom been addressed. Specially, there is a lack of an analytic framework that can serve as a guide for identifying measures and assessing the performance of ERP systems practically. In this study, a systematic framework is proposed to develop the appropriate PIs and construct the PI structure taking into ccount the objectives of the ERP implementation project and thus extract the associated contents of the PIs for evaluating the performance of the adopted ERP system. Then, the procedure can help managers understand the current condition of the ERP system and how to improve the system in the future. Step 3. Survey and add other appropriate PIs on the ERP output view. Step 4. Reduce the number of PIs using the quantitative content validity method. Step 5. Construct the PI structure based on the knowledge of the ERP implementation objectives. Step 6. Develop the detailed contents and evaluation guidelines of ERP PIs. Step 7.
Evaluate the performance of the ERP system. Step 8. Undertake the periodic measurement. The procedure is illustrated in Fig. 1 and elaborated as follows. 3. 1 Form an ERP performance indicator content development team The first step is to form an ERP PI content development team involving critical managers, user representatives, and ERP system experts to formulate an ERP system performance assessment plan, identify appropriate PIs and their contents, and develop consistent evaluation guidance. Notably, these team members may not be the same individuals as those who will undertake the ERP performance evaluation in the future.
However, they are better incorporated into the ERP performance-measurement team. Form an ERP PI content development team Expand the ERP implementation objectives to suitable ERP PIs Survey and add other appropriate PIs on the ERP output view Reduce the number of PI Construct the PI structure based on the knowledge of the ERP implementation objectives Develop the detailed performance evaluation method 3 An ERP performance-measurement framework To clearly present the proposed ERP performance-measurement framework, a stepwise procedure is first described.
Step 1. Form an ERP PI content development team. Step 2. Expand the ERP implementation objectives to suitable ERP PIs. Evaluate the performance of the ERP system Undertake the periodic measurement Fig. 1 ERP system performance evaluation framework Int J Adv Manuf Technol (2008) 39:168–181 171 3. 2 Expand the ERP implementation objectives to suitable ERP PIs For evaluating the ERP performance, it is important to incorporate appropriate PIs that are linked to the system’s implementation objectives, or even the strategic objectives.
The objectives of the ERP implementation project indicate the direction in which managers should strive to perform better and the ERP PIs reveal whether the objectives have been achieved. Therefore, the team members should transform these objectives into some appropriate PIs to link the input factors of ERP implementation with the output factors of ERP execution and recognize the gap between what the managers want and what the system performs. In the ERP implementation objective structure, two kinds of objectives derived from the strategic objectives were structured systematically.
The fundamental objectives are those that are important to specify the goal of the ERP implementation project. The objectives highlight why the managers are concerned with the ERP system implementation and what attributes the managers should be reviewing to select an adequate ERP system. They are organized into a hierarchy. Furthermore, the means objectives specify how to accomplish the desired fundamental objectives. Having sorted them, the managers can evaluate various alternatives whose conditions are consistent with the company’s concerns. They are organized into networks.
Readers can refer to Wei et al. [22] about the detailed ERP implementation objective development and construction process. Based on the definitions of the fundamental objectives and the means objectives, this study finds that the fundamental objectives indicate the critical directions of ERP performance evaluation and some means objectives themselves are suitable to be ERP PIs to evaluate whether the fundamental objectives have been accomplished as promised. Different companies may adopt their ERP systems for completely different reasons and objectives, including both technical and business reasons.
To ensure the process progresses smoothly, a systematic ERP PI discussion process is employed. The team can start the discussion with a nearest means-objective of a bottom-level fundamental objective in the ERP implementation objective structure to discuss, “Whether the means objective can be used as a suitable ERP performance indicator? ” If the means-objective is an appropriate PI, then it is added to the PI set. If it is not suitable, the team can further discuss, “How to evaluate whether this means-objective has been achieved? The answers reveal more detailed and new PIs, which can be incorporated into the PI set. The project team should discuss the suitability of each PI repeatedly. Some similar PIs may be combined or canceled. Additionally, some means objectives can not easily find its corresponding PIs, the members can develop some proxy PIs to assess the achieveness of its fundamental-objective. The process is repeated with all means objectives, resulting in an original PI set being defined. The systematic ERP PI discussion technique is summarized in Fig. 2. 3. Survey and add other appropriate PIs on the ERP output view The original ERP PI set is extracted from the ERP implementation objectives. These PIs elucidate many critical aspects that are considered by the managers in the ERP implementation phase. However, they cannot sufficiently involve all considerations to measure the outputs of ERP system. The team should survey other proper PIs based on the usage and output aspects of the adopted ERP system execution. A wide range of information concerning ERP or other IS PIs can be found from the relevant literatures or their concernment.
Then, some critical PIs can be added into the PI set. 3. 4 Reduce the number of performance indicators A very large number of PIs would require numerous evaluations, making the process very inefficient. This study undertakes a survey “quantitative content validity method” [23] to modify the PI set. The team members are asked to rate each PI using the three-point scale of “not relevant” (value 1), “important but not essential” (2), and “essential” (3). A content validity ratio (CVR) is calculated by the responses from all members for each PI: CVR ? ?n A N =2? ? N =2? ?1? where N denotes the number of team members and n is the number of team members who give the PI a value of 2 or 3. All CVRs of every PI are confirmed with a cut value calculated according to the significant level ? =0. 05. The Start from a nearest means-objective of a bottom level fundamental-objective How to evaluate whether the means-objective has been achieved? No Whether this means-objective can be used as a suitable PI? Yes Generate a PI and add it into the original ERP PI set Fig. 2 Systematic ERP PI discussion technique 172 Fig. Fundamental-objective hierarchy of system factor and its means-objective network [22] Int J Adv Manuf Technol (2008) 39:168–181 Fundamental-objective On schedule Implementation time Price Maintenance costs Total costs Consultant expenses Infrastructure costs Means-objective Payment Discount Number of modules Provision of consultants Price of vendor’s quotation Price of consultants Infrastructure integration Satisfies requirements Clear requirements Hardware support Other systems support Multi-language Multi-currency Multi-site Cooperation with the consultant companies
Degree of employee cooperation Degree of customization Module completion Functionality Comparison with existing processes Parameters setting Function fitness Security Permission management System & database protection Globalization Graphic interface User friendliness Ease of operation Ease of learning Step-by-step command Provision of a guidebook Online learning Online help Upgrade ability Ease of integration Ease of in-house development Common programming language Platform independence Ease of integration with other systems System maturity Few “bugs” Flexibility Stability Reliability Recovery ability
Minimum of system break down Automatic data backup Automatic data recovery Int J Adv Manuf Technol (2008) 39:168–181 Fig. 4 PI diagram of the major objective, “Functionality” Degree of workflow support 173 Satisfies requirements Information timeliness Function fitness Functionality Comparison with existing processes Information aggregation Parameter setting Frequency of special function requests Permission management Security System & database protection Means-objective ERP PI PIs whose CVRs are less than the cut value can be eliminated thereby reducing the number of PIs. 3. Construct the PI structure based on the knowledge of the ERP implementation objectives After identifying the ERP PIs, the team can construct the PI structure pertinent to evaluating the success of the adopted ERP system. This study categorizes the PIs to be compatible with the ERP implementation objective structure as follows: (1) Project management factors: PIs involved in ERP project management, such as total cost and project time. (2) System factors: PIs for evaluating the practicality of the ERP system, including functionality, reliability, user friendliness, and flexibility. 3) Vendor factors: PIs for assessing the performance of the ERP vendor, such as technology capability and service. (4) Impact factors: the impact of ERP system on the organization, individual, and customer. The members can review the remaining PIs in the set and place them into the four dimensions. The knowledge of the fundamental-objective hierarchy and means-objective network can assist them in constructing the ERP PI structure. The PIs at the lowest level are specific and operational in practice and are perceived as the means to achieve those fundamental objectives.
Therefore, the PIs will be consistent with the ERP implementation objectives. The ERP PI hierarchy should be iteratively examined and modified so that it is complete, decomposable, nonredundant, measurable, and minimal [24]. After specifying the hierarchy, the team may find themselves refining the content and establishing the evaluation directions. 3. 6 Develop the detailed performance evaluation method Different stakeholders may hold different perceptions about how to assess the ERP system performance. Explicit knowledge and understanding of how a PI will be measured is necessary.
The team should investigate what data they need to collect and how to collect it for evaluating each PI. A standard form can help them to collect the data and identify the detailed evaluation method. Additionally, the knowledge of the ERP implementation objective structure can not only help in identifying the PI hierarchy, but also in indicating how the outcomes are assessed and what kinds of uncertainties should be considered in the ERP performance evaluation. Then, the detailed evaluation contents and an assessment questionnaire or form for each PI can be developed.
The members array the data for each PI so that it can be charily measured with a consistent baseline. Weightings associated with PIs can be assigned simultaneously. The weight of each PI can be set by direct assignment or indirect pairwise comparisons. 3. 7 Evaluate the performance of the ERP system The ERP performance-measurement team is responsible for assessing the performance of the adopted ERP system. The team examines the data to rate each PI according to the current condition of the ERP system. However, most PIs may not be easily quantifiable.
Since a score is easy for managers to understand and communicate to each other, the members assess the PIs with a simple rating questionnaire or form by scoring them from 0 to 100. On the other hand, the values of the quantitative PIs are also converted into the corresponding scores to ensure that the compatibility with the ratings of the qualitative PIs. Then, the ratings can be aggregated to obtain the scores of the four critical dimensions and the ERP system. 174 Table 1 ERP PIs extracted from the means objectives and their CVRs Main bjective Cost Fundamental-objective Software cost control Maintenance cost control Consultant cost control Infrastructure cost control Schedule control Performance indicator Int J Adv Manuf Technol (2008) 39:168–181 CVR 1. 00 1. 00 1. 00 1. 00 1. 00 0. 75 0. 50 1. 00 1. 00 1. 00 0. 75 1. 00 0. 75 1. 00 0. 75 0. 50 0. 75 0. 75 0. 75 1. 00 1. 00 1. 00 1. 00 0. 75 1. 00 0. 75 1. 00 1. 00 1. 00 0. 75 0. 50 1. 00 0. 50 0. 75 1. 00 1. 00 1. 00 0. 50 Time Functionality Module completion Functionality fitness Security User friendliness Ease of operation
Ease of learning Flexibility Upgrade ability Ease of Integration Ease of in-house development Reliability Stability Recovery ability Technology capability Technology support Service Training service Service ability Gap between the software budget and real expense Gap between the maintenance budget and real expense Gap between the consultant budget and real expense Gap between the infrastructure budget and real expense 1. Gap between the schedule and real time taken 2. Degree of customization 3. Degree of employee cooperation 1. System completeness 2. Global task performance 1. Degree of workflow support 2.
Parameter setting functions 1. System and database protection 2. Permission management 1. Ease of operation 2. E-guidebook usefulness 3. Step-by-step guiding 1. Online learning 2. Online help 1. Upgrade technology support 2. Upgrade service performance 1. Ease of integration with other systems 2. Ease of communication with other platform 1. Ease of maintenance 2. Ease of modification 1. Minimum of system break down 2. System maturity 1. Recovery ability 2. Automatic data backup ability 1. Technology development 2. Diverse product introduction 3. Engineer stability and experience enhancement 1.
Effective training lessons 2. Sufficient training time 1. Online service 2. Solving problem ability 3. Consultant service ability 4. Service speed 5. Warranty satisfaction After assessment, individual graphs and reports are created to show the condition of each main PI dimension and show whether the overall ERP system is making progress or losing ground. By studying the results and trends, the managers can set meaningful targets and plans for improvement. 3. 8 Establish the periodic measurement Due to inevasible changes in the ERP system and its environment, the measurement framework is dynamic and evolutionary.
Additionally, periodic ERP performance assessments should be undertaken to provide the basis for the practice of continuous improvement. This framework is conducted whenever the need for a new PI is realized. Such a modification process aligns the existing performance evaluation system through a combination of system configuration and organizational change. 4 Practical example The case company used in this study is in the business of modular microwave communication system design, manufacturing, repair and service.
The company’s manufacturing facilities are located in Taiwan and China, and some servicing depots and offices are distributed in the USA, Europe, and Brazil. The company’s products are exported to Int J Adv Manuf Technol (2008) 39:168–181 Table 2 Additional PIs and their CVRs ERP performance indicator Data completeness Information aggregation Response time Information timeliness Frequency of requested for special functions Frequency of use Convenience of access information Acceptance of reports Frequency of e-learning use Data error rate BPR ability Management bility enhancement Cycle time reduction Work process standardization Quality of decision-making Personal productivity and efficiency improvement Employee satisfaction Customer satisfaction Efficiency of ERP system Company’s fame enhancement Financial condition improvement On time delivery Response time to customer CVR 0. 75 0. 75 1. 00 1. 00 0. 75 0. 75 0. 50 0. 75 0. 75 1. 00 0. 50 1. 00 1. 00 1. 00 0. 75 1. 00 1. 00 0. 75 1. 00 0. 50 0. 50 0. 75 0. 75 175 After the ERP system was adopted, both the efficiency and quality of business operations were improved.
However, tedious requirements from user departments often bothered the information managers. They believed that a process needed to be developed to determine how the ERP system was currently performing and how it should be performing at a future point of time. To justify the success and the value-added contribution of the ERP system to accomplish the objectives set up in the ERP system implementation stage, the proposed framework helped them in developing an ERP performance evaluation framework and focused the efforts on improving the system. . 1 Form an ERP performance indicator content development team An ERP PI content development team with six members was formed, including five major managers and an information manager, with the responsibility to formulate the project plan, identify the appropriate ERP PIs, and develop the consistent evaluation guideline of each PI. Representatives of different user departments were also chosen to participate in the project team for offering their experience of using the ERP system and detailed evaluation data.
Notably, to avoid the resistance and misunderstanding of employees, the project team held several promotional workshops. These workshops revealed the aims of measuring the ERP system performance. The results will indicate that the functionality and service of the ERP system can be trusted and that high system performance standards can be maintained. Additionally, the measurement results are not used to punish any individual or any department. The ultimate purpose is to capture and prioritize the improvement requirements for incorporation within a future modification of the ERP system. . 2 Extend the objectives of the ERP implementation project to suitable performance indicators The objectives of ERP implementation have been established in the ERP implementation phase. The fundamentalobjective hierarchy and the means-objective network can be found in Wei et al. [22]. For example, Fig. 3 shows the fundamental-objective hierarchy and means-objective network of the system factor. First, according to the systematic ERP PI discussion technique (Fig. 2), the members discussed whether the existing means objectives in the objective structure were suitable PIs.
For example, the means objectives of a fundamental-objective “security” were “permission management” and “system and database protection”. The members were led to discuss “Whether ‘permission management’ and ‘system and database protection’ can be used as suitable ERP PIs? ” The members the USA, South America, Europe, and mainland China. The company seeks to maintain its competitive advantage by improving the effectiveness of its global logistics and the efficiency of its response to customer demand.
The top management announced the launch of a series of projects including the re-engineering of the complex business process and the implementation of an ERP system in the end of 2002. A local ERP system was implemented using a phased approach. Some legacy systems operating at the company were also integrated with the ERP system. Overall modules of the ERP system have been implemented in 2004. Some important strategic objectives of the ERP implementation project are listed as below [22]: (1) To satisfy business strategy: to satisfy the industrial characteristics and the business goals and adapt to dynamic business environment. 2) To enhance business process performance: to integrate business systems and procedures and enhance information transparency. (3) To improve operations quality and efficiency: to standardize and simplify operations flow, improve quality and reduce lead times. (4) To shorten turn-around time to the customer: to efficiently analyze customer information from various markets and quickly respond to customer demands. (5) To support globalization development: to support business operations worldwide. 176 Fig. 5 ERP PI structure and the corresponding weights
Category Major objective Fundamentalobjective Software cost Maintenance cost Project management Time Cost Consultant cost Infrastructure cost Int J Adv Manuf Technol (2008) 39:168–181 PI Gap between budge and real expense Gap between budge and real expense Gap between budge and real expense Gap between budge and real expense Gap between schedule and real time taken Degree of workflow support Information timeliness Information aggregation 0. 48 0. 24 0. 18 Implementation time Function fitness 0. 75 Functionality 0. 53 Frequencyof special function requests 0. 11 Security System & database protection 0. 25 0. 75 0. 25 0. 50 0. 10 0. 25 0. 5 1. 00 Permission management Ease of operation E-guidebook usefulness Acceptance of reports Online learning System ERP success assessment 0. 54 User friendliness 0. 24 Ease of operation and learning 1. 00 Upgradation ability 0. 20 Upgrade service performance Ease of integration with other systems 0. 50 Flexibility 0. 12 Ease of integration 0. 50 Ease of communication with other platforms 0. 50 Ease of maintenance Ease of modification Frequency of system error 0. 75 0. 25 0. 50 0. 50 1. 00 1. 00 1. 00 0. 33 0. 33 0. 34 0. 12 0. 20 0. 27 0. 41 0. 25 Ease of in-house development 0. 30 Reliability 0. 11 Stability 0. 67 Recovery ability 0. 3 Technology support 1. 00 Data error rate Mean recovery time Diverse product introduction Effective training lessons Technology capability Vendor 0. 16 0. 33 Training support 0. 33 Solving problem ability Consultant service ability Service 0. 67 Service ability 0. 67 Service speed Management enhancement Organization 0. 44 Cycle time reduction Workflow standardization Efficiency of system Impact 0. 30 Individual 0. 17 Quality of decision-making Personal productivity improvement 0. 59 Employ satisfaction 0. 16 0. 33 0. 67 Customer 0. 39 Response time to customer On time delivery thought that they were very critical and suitable to be PIs.
Some means objectives were combined. For example, “required function satisfaction” and “business process support” could be grouped into “degree of workflow support” to support the fundamental-objective “function fitness”. The means-objective “parameter setting” can make the ERP system “function fitness”. However, the concept of parameter setting meant a tool and was not a good and measurable PI. The members used “frequency of special function requests” as a proxy PI. If the number of special function requests within specified timeframe was high, then the function of parameter setting did not help the system to achieve function completeness.
Some diagrams were developed for each of the three main categories to reveal the linkages between the major objectives, fundamental objectives, and ERP PIs based on the knowledge of the ERP implementation objective structure and the results of the systematic ERP PI discussion technique. The diagrams also indicated the detail assessment contents of ERP PIs. Figure 4 illustrates the linkage of the major objective, functionality, its fundamental objectives, PIs and the contents of these PIs. Repeated the systematic ERP PI discussion process through all means objectives of each fundamental-objective.
The result of this examination derived core contents of the original ERP PI set. Initially, the original set of 38 PIs was extracted from relevant means-objective network. Int J Adv Manuf Technol (2008) 39:168–181 Table 3 Examples of PI details Fundamental-objective: Function fitness Degree of workflow support Information timeliness Information aggregation Frequency of special function requests Quantitative PI: Number of special function requests per month 0–1 2–3 4–5 6–8 9–10 11–15 16–20 21–25 26–30 31–40 Above 40 80 0. 1 100 90 80 70 60 50 40 30 20 10 0 177 Score conversion Qualitative PI: Average value based on ratings made on a five-point scale 5 4. 5 4 3. 5 3 2. 5 2 1. 5 1 0. 5 0 76 0. 48 Qualitative PI: Average value based on ratings made on a five-point scale 5 4. 5 4 3. 5 3 2. 5 2 1. 5 1 0. 5 0 81 0. 24 Qualitative PI: Average value based on ratings made on a five-point scale 5 4. 5 4 3. 5 3 2. 5 2 1. 5 1 0. 5 0 74 0. 18 78. 04 Score Weight Total score Table 1 reveals that the ERP PIs can align with the objectives of the ERP implementation project.
The alignment exercise indicated that while many of the PIs supported one or more fundamental objectives, some objectives needed new measures created in order to ensure the objective could be assessed. 4. 3 Add other crucial ERP performance indicators Some additional ERP PIs surveyed from prior literatures were recommended to compensate the PI set. After examining the necessity of these PIs with the members, 23 PIs were incorporated into the original set, increasing the number of ERP PIs to 61. Table 2 lists the added ERP PIs. 4. 4 Reduce the number of performance indicators and modify the indicators The ERP PI set must be refined.
Employing the quantitative content validity method, the members were asked to rate each PI using the three-point scale, so that the CVR of each PI could be obtained. Tables 1 and 2 summarize the CVRs of those PIs. Based on the significant level ? =0. 05, the cutoff value of CVR was 0. 75. The PIs whose CVRs were less than 0. 75 were deleted. The members also discussed how some of the PIs whose CVRs were greater than or equal to 0. 75 could be meaningfully grouped into those of the other PIs. Similarly, “ease of operation” could include “convenience of access information”. After discussion, 37 PIs were selected for further consideration.
Additionally, some PIs needed modification. For example, “number of system or database protection failure” was used instead of “system and database protection” to record the density of system and database protection failure exactly. 4. 5 Construct the PI structure based on the knowledge of the ERP implementation objectives A hierarchy was constructed from the remaining PIs based on the four main ERP PI categories. The ERP PI hierarchy comprised five levels, as illustrated in Fig. 5. Level 1 reveals the ultimate mission for assessing the performance of the adopted ERP system.
Level 2 consists of four main PI categories: project management, system, vendor, and impact factors. Level 3 contains the major objectives. Level 4 describes the basic fundamental objectives that the managers hoped to achieve. The bottom level comprises the associated PIs that were used to measure the performance of the adopted ERP system. Then, this ERP PI hierarchy is aligned with the fundamental-objective hierarchy of the ERP implementation project. Significantly, once the ERP implementation project was complete, the project management PIs and relative critical problems about the project should be examined immediately.
Since the project management PIs had been assessed 178 Fig. 6 Score trend of the three main PI categories 100 Int J Adv Manuf Technol (2008) 39:168–181 90 80 score 70 60 50 1 2 3 4 month 5 6 7 system vendor impact after the ERP implementation project finishing and then it was not necessary to be assessed again during the ERP execution. Thus, the last three main dimensions comprised the ERP PI structure. For reducing the loading of the PIs’ importance comparison process, this case followed the analytic hierarchy process (AHP) methodology.
That is, the relative weights of ERP PIs and fundamental objectives were determined by using AHP method [25]. The paired comparisons process was repeated for each PI by all decision makers and converted to a numerical scale of 1–9 based on the largest eigen-value method. The software Expert Choice was then used to determine the normalized weights and synthesize the results. The results were shown in the frame of each PI and fundamental-objective in Fig. 5. It must be noted that the weight in the ERP PI hierarchy forming a standpoint of output measurement would be different from those in the objective structure of ERP implementation.
Fig. 7 Score trend of the vendor PIs 80 75 70 score 65 60 55 50 1 2 3 4 month 5 6 7 Technology support Training support Service ability Int J Adv Manuf Technol (2008) 39:168–181 179 4. 6 Develop the detailed performance evaluation method The members discussed how to measure the PIs and how to collect the data of the ERP system performed. Using the above PI diagrams, we assisted the members to establish the specific data requirements of each PI. The detailed evaluation guidance and an assessment form for each PI were also developed.
For example, Table 3 presents the detailed descriptions of PIs corresponding to a fundamental-objective, “function fitness”. All measured values of qualitative and quantitative PIs can be translated into the score of 0–100. The members determined the best possible performance they believed that the ERP system might achieve, and set it as 100. Then they discussed reasonable score conversion of each PI in the rest of the scores using every ten intervals from 100 down to 0. Furthermore, the members paid attention to the reliability and usefulness of each data set, as well as its correspondence with certain PIs.
This process can ensure that everyone follows the same PIs in the performancemeasurement process consistently. 4. 7 Evaluate the performance of the ERP system The members of the ERP performance-measurement team assessed the current conditions of the ERP system to determine the score of each PI. A rating example of a specific time is shown in Table 3. The average scores of the ratings of all decision makers were obtained and then the average scores of PIs were aggregated by multiplying the scores by the weights to obtain the scores of fundamental objectives. The score of the fundamental-objective “function fitness” was 78. 04.
Then, average scores of each fundamentalobjective was multiplied by its weight and all scores were summarized to obtain the aggregated scores of the corresponding major objective. These scores were rolled to yield the scores of system, vendor, and impact PI categories. Finally, a total value was obtained that combined the three main PI categories into a single score to indicate whether the ERP system was improving over time. 4. 8 Establish the periodic measurement The scores of each fundamental-objective, major objective and entire ERP system can display that the results are raising or descending compared with that in the preceding period.
They indicated the trends of different fundamental objectives and PI categories. Figure 6 shows the score trends of the system, vendor, and impact categories 1 to 6 months after the ERP performance-measurement system was established. The ERP system had made significant progress on the system and impact categories, but had not improved on the ERP vendor category. Checking the score of the vendor PIs showed that the fundamental objectives “training support” and “service ability” related PIs had made regression (Fig. 7).
Since there were a few information engineers in the company, the managers hoped that the ERP vendor could provide more support and service to continuously improve the system. The IT department initiated a project to strengthen the relationship with the ERP vendor. Some actions were implemented, including: (1) Revising the contract with the ERP vendor to improve the support of ERP vendor and highlight what the company really wanted to support. (2) Establishing a problem feedback mechanism and a solving problem process immediately with the ERP vendor. 3) Undertaking a training plan to train the IT employees to improve the maintenance ability in-house. 5 Discussion ERP systems make possible deep changes in employee behaviors, company culture, and supply-chain relationships that can be crucial sources of advantage in the knowledge economy. The purpose of the PI development process is that people can use a systematic method to assess whether the ERP system achieves its implementation objectives based on the knowledge and guidelines of these objectives, and then continuously improve the performance of the ERP system.
The ERP PI should indicate how the ERP system is performing, but they must also be designed so as to encourage the desired behaviors by all functions and individuals. Additionally, as the PIs are aligned with the ERP implementation objectives, the ERP performance evaluation can ensure that the continuous improvement is executed towards the directions to which the company wishes to strive. The scores of the entire ERP system and the three PI categories can show how the ERP system is currently performing and whether its performance is moving forward over time.
The scores are easy to understand and to be communicated with others. The trends of those scores allow the managers to identify which aspects of the ERP system need to have their attention and resources allocated to improve its performance. Some ERP PI classifications were arbitrary, because they did not obviously fit into any one specific category. However, it was not a big question in practice, as most members could achieve a consensus on each classification. The precision with which the members of the ERP performance-measurement team could provide an evaluation was limited by their knowledge, experience, and even 80 Int J Adv Manuf Technol (2008) 39:168–181 cognitive biases, as well as by the complexity of the ERP system execution. To avoid inconsistency among semantic descriptions and score assignments to the PIs, the members were encouraged to discuss and examine the PIs development and its detailed evaluation standards. During the assessment process, consistency checks were conducted and the members of ERP performance evaluation team in some cases were asked to provide reasons and detailed explanations to justify and refine their assessments.
The managers were impressed with the analysis that we presented. They stated that they have now become more aware of the benefits and shortcomings of their ERP system. The PI development process took about 2 months. The proposed framework facilitated the complex ERP performance mechanism establishment process and focused their discussions on the real problems and objectives. 6 Conclusions The installed ERP system is not an ending but instead is continuously working and improving over time and across the organization in a complex exercise.
A very significant challenge faced by information managers today is to justify the value-added contribution of their ERP systems. Without the ability of assessing the performance of the ERP system, they cannot evaluate its status and monitor its improvement. An ERP performance-measurement framework should establish a feedback mechanism between the desired objectives of ERP adoption and the substantial effects of ERP execution. This study proposed a framework that developed an ERP performance-measurement system based on the knowledge of ERP implementation objectives.
The proposed procedure expands the structure of fundamental objectives and means objectives, which had been constructed in the ERP system implementation methodology to identify the ERP PI hierarchy and formulate the detailed evaluation guidance. The pertinent contents for assessing the performance of the adopted ERP system can be derived according to the relationship diagrams of objectives, fundamental objectives, and PIs. Then, the evaluation results can truly reflect the current situation of the adopted ERP system and the accomplishment of the expected objectives.
Following the analysis, the managers can recognize the directions of ERP system improvement and the strategies of corporate information system in the future. The proposed framework has the following advantages for the companies: (1) Due to extension from the ERP implementation objectives to obtain the PIs, the proposed framework ensures that the PIs using to assess the ERP system performance are consistent with the ERP implementation objectives. The project team can evaluate and understand whether the ERP implementation objectives have been achieved. 2) The project team can easily and systematically identify suitable PIs and develop their detailed contents by following the stepwise procedure to facilitate the complex ERP PI development process. The systematic ERP PI discussion process can make the team members to focus their discussions on what they want. Additionally, the project team can analyze the relationships among different objectives to identify their corresponding PIs based on the fundamental-objective hierarchy and means-objective network and model these PIs to the hierarchical structures. 3) The trends of those scores allow the members to monitor how the ERP system performs and identify which aspects of the ERP system need to improve. (4) The proposed framework can be computerized to establish an ERP performance-measurement system accompanying with the ERP system. The evaluation results can offer applicable feedbacks to the managers for improving the ERP system performance. References 1. Kumar V, Maheshwari B, Kumar U (2002) Enterprise resource planning systems adoption process: a survey of Canadian organizations.
Int J Prod Res 40:509–523 2. Wei CC, Wang MJJ (2004) A comprehensive framework for selecting an ERP system. Int J Project Manage 22:161–169 3. Saarinen T (1996) An expanded instrument for evaluating information system success. Inf Manage 31:103–118 4. Motwani J, Mirchandani D, Madan M, Gunsasekaran A (2002) Successful implementation of ERP projects: evidence from two case studies. Int J Prod Econ 75:83–96 5. Jurison J (1996) The temporal nature of IS benefits: a longitudinal study. Inf Manage 30(2):75–79 6.
Heo J, Han I (2003) Performance measure of information systems (IS) in evolving computing environments: an empirical investigation. Inf Manage 40:243–256 7. Hunton JE, Lippincott B, Reck JL (2003) Enterprise resource planning systems: comparing firm performance of adopters and nonadopters. Int J Account Inf Syst 4:165–184 8. Kivijarvi H, Saarinen T (1995) Investment in information systems and the financial performance of the firm. Inf Manage 28:143–163 9. David G, Ragowsky A (2005) A multi-level approach to measuring the benefits of an ERP system in manufacturing firms.
Inf Syst Manage 22(1):18–25 10. Salmeron JL (2003) EIS success: keys and difficulties in major companies. Technovation 23:35–38 11. Scott JE (2005) Post-implementation usability of ERP training manuals: the user’s perspective. Inf Syst Manage 22(2):67–77 12. Lee YW, Strong DM, Kahn BK, Wang RY (2002) AIMQ: a methodology for information quality assessment. Inf Manage 40:133–146 Int J Adv Manuf Technol (2008) 39:168–181 13. Xu H, Nord JH, Brown N, Nord GD (2002) Data quality issues in implementing an ERP. Ind Manage Data Syst 102(1):47–58 14.
Palvia SC, Sharma RS, Conrath DW (2001) A socio-technical framework for quality assessment of computer information systems. Ind Manage Data Syst 101(5):237–251 15. Delone WH, McLean ER (1992) Information systems success: the quest for the dependent variable. Inf Syst Res 3:60–95 16. McHaney R, Hightower R, Pearson J (2002) A validation of the end-user computing satisfaction instrument in Taiwan. Inf Manage 39:503–511 17. Doll WJ, Xia W, Torkzadeh G (1994) A confirmatory factor analysis of the end-user computing satisfaction instrument.
MIS Q 18(4):453–461 18. Choe JM (1996) The relationships among performance of accounting information systems, influence factors, and evolution level of information systems. J Manage Inf Syst 12(4): 215–239 181 19. Hagood WO, Friedman L (2002) Using the balanced scorecard to measure the performance of your HR information system. Public Pers Manage 31(4):543–557 20. Stensrud E, Myrtveit I (2003) Identifying high performance ERP projects. IEEE Trans Softw Eng 29(5):398–416 21.
Lin WT, Chen SH, Lin MY, Wu HH (2006) A study on performance of introducing ERP to semiconductor related industries in Taiwan. Int J Adv Manuf Technol 29:89–98 22. Wei CC, Chien CF, Wang MJJ (2005) An AHP-based approach to ERP system selection. Int J Prod Econ 96:47–62 23. Lawshe CH (1975) A quantitative approach to content validity. Pers Psychol 28:563–575 24. Keeney RL, Raiffa H (1993) Decisions with multiple objectives: preferences and value tradeoffs. Cambridge University Press, Cambridge 25. Saaty TL (1980) The analytic hierarchy process. McGraw-Hill, New York
