CASE FOUR:MEASUREMENT AND VALIDATION| Case requirements: 1. Assess the practical and statistical significance of a proposed set of hiring tools, and make recommendations regarding how adopting these new hiring methods might benefit stores. 2. Make suggestions to Tanglewood regarding which subset of predictors is most likely to improve the effectiveness of selection without creating an administrative burden. 3. Assess the content validity of various proposed selection techniques by determining how well they match the general requirements of the job. 4. Estimate how well the test sample results will generalize to other locations.
Case Objectives Once concrete goals for hiring have been established and applicants have been generated, the most important part of the staffing process is developing methods to identify individuals who will be the best performers on the job. Anything that identifies good potential performers is a “predictor. ” This includes interviews, standardized tests of knowledge, personality measures, job trials, and so on. Although finding good predictors requires intuition about the job, the organization, and the type of people who are going to apply, demonstrating which predictors are most effective requires clear quantitative skills as well.
In this instance, you will review several types of evidence related to predictors and job performance and select a mix of predictors you think will work well. Developing a good selection strategy also means thinking of the process from the applicant’s point of view. The greatest selection system in the world is not effective if it scares the best applicants away. This case is an opportunity to look at the types of measures (found in Appendix C) that are often used in the selection process and determine how applicants might react to them. Hiring for the Store Associate Position
Two years ago, Marilyn Anchley instigated a thorough assessment of the hiring practices in the Tanglewood stores following complaints from many store managers regarding the quality of employees. Results were not encouraging. The current methods for selection received negative reports from managers. Many have noted that their current employees deliver suboptimal work, fail to appreciate the organization’s culture, and have difficulty working in teams. This sometimes means that they have to fire poor performers, which is not good for morale in a team-based organization.
There are few selection methods traditionally used at all Tanglewood stores. The first method for selection is an application blank as shown in Appendix C. Applicants provide some basic information regarding their employment history and education, along with other simple contact information. Ideally, Tanglewood would prefer to contact former employers to get job performance history information, but in practice, very few former employers give much more than dates worked and job titles because of concerns about being sued for disseminating damaging information about their former employees.
In practice, then, the only useful information Tanglewood obtains from the application blank is the number of years of work experience a person has and the highest degree they have completed. The process of selection begins when the applicants turn in their application blanks. These forms are reviewed by the Assistant Store Manager for Operations and HR, who will also conduct brief interviews. The initial application interview is generally brief and consists mostly of efforts to confirm and clarify information in the application blank. Those who make it past this stage of the process are termed “candidates. ”
A more substantial interview occurs with the candidates who are referred to the department manager. There is a relatively loose protocol for how managers should conduct the interviews. Tanglewood provides a list of suggested interview topics, including, “tell me about your previous work performance,” “explain your philosophy of customer service,” and “describe a time when you worked on a team. ” Managers are encouraged to develop a warm atmosphere for the interview to send a positive message about the company culture as well. Questions are formalized, although there are no keys for managers regarding what the “right” answers are.
The form used for interviews is also supplied in Appendix C. Approximately 30% of candidates who make it to this interview phase go on to become finalists. Final approval for hires also must come from each store’s manager for operations and human resources. The last stage before job offers are provided includes a background check and some paperwork. Nearly all finalists go on to receive offers. Proposing an Alternative to the Current System Based on negative feedback from managers, a corporate committee determined that an ideal solution would require an organization-wide shift to a more detailed selection system.
The interview process has been deemed “essential” by most managers, and they are resistant to anything that would modify the interview process. As a result, approximately one year ago the staffing services department initiated a search for good predictors of sales associate performance that would create minimal additional administrative burdens for managers. In the course of this search a number of new methods were brought to the organization’s attention. All of the materials described in this section are presented in Appendix C. Retail Market Knowledge Exam
The retail market knowledge exam is a set of questions related to the retail industry and Tanglewood’s unique position in the industry. Several of the questions are related to basic knowledge of marketing principles, while others address the factors that separate Tanglewood from other competitors in the industry. This exam was developed in house by the staffing services and marketing divisions. Marshfield Customer Service Biodata Questionnaire and Essay Biodata exams are questions for significant life experiences that are potentially associated with performance at work.
Marshfield Testing Corporation has given life experiences interviews to thousands of individuals, and based on this huge sample of responses, developed a set of occupation-specific life experiences that they have identified as being associated with job performance. The test costs $10 per applicant. Newer versions of the test are available which can be administered via computer, or through the stores’ in house kiosk system. The newer versions carry a more substantial fixed startup cost, but do not carry a “per applicant” cost. The essays are machine scored in a manner similar to resume scoring software.
The customer service questionnaire and essay is two times longer than the sample shown here, but the topics are the same. Marshfield Applicant Exam The Marshfield applicant exam is a test designed to capture problem solving abilities, fluency with numerical processes, and work comprehension. Marshfield Testing Corporation will administer variations on this test to several thousand individuals per year in contexts ranging from managerial selection to staffing for clerks at convenience stores. Like the Marshfield Biodata Questionnaire and Essay, this test can be administered either online or in person.
The customer service questionnaire and essay is four times longer than the sample shown here, but the basic topic items are the same. Personality Exam Donald Penchiala conducted a thorough reading of the literature on personality psychology during his undergraduate major (he had a concentration in industrial/organizational psychology at Michigan State University). Based on this knowledge, he independently developed a 20 item measure designed to capture the constructs of conscientiousness and extraversion. He believes that these are the two personality traits that will be most relevant for the position of a retail clerk.
The Validation Procedure After assembling these new predictors, 10 stores were selected from the Seattle area to serve as a “test” area. Over the course of the last year, all proposed employees have been administered all the new selection tools during the hiring process. Stores were informed that they would be part of the new staffing system through a corporate memo personally signed by Emerson and Wood. This personalized memo was designed to impress on store managers the importance of gathering complete and accurate data on all employees.
As part of the trial process, Tanglewood has also collected information from all stores that were not engaged in the pilot program. The reason for this use of supplemental data is to investigate whether the pilot was conducted on a representative sample of stores. Each store compiled all the archival data from their computer databases and then sent it to the corporate offices. In the course of this process, it was noted that stores were often not keeping very complete data, so that information on selection outcomes was available for only about half of the nearly 25,000 employees hired in the past year.
Several types of performance measures are kept for all employees at Tanglewood as part of their annual performance reviews. The first measure is citizenship performance, which refers to how well employees perform well as team members, cooperate with store policies, and generally set a positive tone for the workplace. Citizenship for each employees is rated on a scale from “1=very poor organizational citizen” to “5=excellent organizational citizen. Absence is a simple count of the number of days on which an employee did not show up for a scheduled work shift or was more than ? hour late for a scheduled shift. Performance is a direct measure of completion of assigned work tasks and effort on the sales floor, as rated by the employee’s supervisor on a scale from “1=very poor task performance” to “5=excellent task performance. ” Finally, promotion potential is a manager’s subjective impression of how likely it is that they will recommend an employee for promotion to a higher level in the organization at some point n the future. To estimate the validity of the current and proposed selection methods, the data regarding employee performance were taken from the organizations’ human resources information systems and several statistical analyses were performed. The statistical analyses were performed in two separate stages. Both steps used a predictive validation design, meaning that predictor data measured at the point of hire was correlated with performance data one year later.
For the predictive design, only interviews and application blanks were actually used as part of the selection process. The first stage of the statistical analysis was to estimate correlations between the archival measures of education, work experience, and managerial interviews and the measures of citizenship, absence, performance, and promotion potential. These data are shown in Table 3. 1. The top row for each predictor shows the correlation between the predictor and the corresponding work outcome. The p-values represent the statistical significance of each correlation.
The second stage of the statistical analysis was to estimate correlations between the traditional predictors (education, work experience, and managerial interviews) and the measures of citizenship, absence, performance, and promotion potential, along with correlations between proposed predictors (retail knowledge, biodata, applicant exam, conscientiousness, and extraversion). These data are shown in Table 3. 2. Table 3. 1 Data from Stores Employing the Traditional Selection Method | | Citizenship| Absence| Performance| Promotion potential| Education| Correlation| 0. 01| -0. 01| 0. 08| 0. 14| | p-value| 0. 26| 0. 26| < 0. 1| < 0. 01| | | | | | | Work experience| Correlation| 0. 04| -0. 04| 0. 16| 0. 18| | p-value| < 0. 01| < 0. 01| < 0. 01| < 0. 01| | | | | | | Interview score| Correlation| -0. 02| 0. 03| 0. 01| 0. 16| | p-value| 0. 03| < 0. 01| 0. 26| < 0. 01| N=12510| | | | | | Table 3. 2 Data from stores employing the proposed method | | Citizenship| Absence| Performance| Promotion potential| Education| Correlation| 0. 03| -0. 02| 0. 15| 0. 17| | p-value| 0. 39| 0. 57| < 0. 01| < 0. 01| | | | | | | Work experience| Correlation| 0. 17| 0. 07| 0. 22| 0. 25| | p-value| < 0. 01| 0. 04| < 0. 01| < 0. 1| | | | | | | Interview score| Correlation| 0. 13| 0. 01| 0. 04| 0. 32| | p-value| < 0. 01| 0. 77| 0. 25| < 0. 01| | | | | | | Retail knowledge| Correlation| 0. 02| -0. 07| 0. 12| 0. 33| | p-value| 0. 57| 0. 04| < 0. 01| < 0. 01| | | | | | | Biodata| Correlation| 0. 22| -0. 17| 0. 17| 0. 34| | p-value| < 0. 01| < 0. 01| < 0. 01| 0. 0000| | | | | | | Applicant exam| Correlation| 0. 03| -0. 02| 0. 34| 0. 26| | p-value| 0. 39| 0. 57| < 0. 01| < 0. 01| | | | | | | Conscientiousness| Correlation| 0. 18| -0. 33| 0. 13| 0. 29| | p-value| < 0. 01| < 0. 01| < 0. 01| < 0. 01| | | | | | |
Extraversion| Correlation| 0. 09| 0. 09| 0. 22| 0. 06| | p-value| 0. 01| 0. 01| < 0. 01| 0. 09| N=832 Specific Assignment Details Marilyn Anchley and Donald Penchiala have asked you to complete the following steps to evaluate the soundness of the proposed selection plan, and provide them with a report. They are especially interested in learning what your results suggest about the validity of various selection methods. 1. Examine the data provided in Table 3. 1 and 3. 2. Write a one page memo describing what the results show. Be certain to emphasize both the practical and statistical and statistical significance of the results.
Also, be certain to note which of the predictors is most related to which of the relevant outcomes. 2. One key question for selection methods is the content validity of selection methods. For each of the scales proposed by Tanglewood, assess how well it matches the content that it claims to measure, and how well it corresponds to the specific job of store associate. Make suggestions for how each method could better capture the content it seeks to measure. 3. After summarizing the overall results of the staffing system, write a description of what Tanglewood should do if it wants to find good candidates.
The company would also prefer to use only two or three predictors. Based on the analyses above and the data provided in the case, describe which predictors you would recommend for this job, and explain why these predictors are the best choices. 4. Based on the description of the experimental validation procedure, do you believe that the observed validity estimates will generalize to other stores? Provide an explanation for why the traditional method for collecting validation evidence and the experimental procedure might yield different results.
Which method do you believe is more accurate? Why or why not? What additional information might you use to determine if the results will generalize? Reminder: Your audience is not particularly familiar with validation techniques, so your grade for this assignment will be based, in large measure, on how well you can guide the reader through the basic information. Staffing Services will look very negatively on a report that is difficult to read, presents contradictory information, or that includes
