| Psy 401 – Personality Research Methods | Dr. John A. Johnson |
| Instructions for Final Project: | Due December 17, 1998 |
SCALE CONSTRUCTION AND VALIDATION
Objective
To demonstrate ability to conduct and interpret item analysis, reliability analysis, scale revision, and construct validation for a newly-created self-report personality scale.
Overview of Procedure
The procedure for this project consists of five steps, although the fourth step is COMPLETELY optional. First, you will receive an SPSS framework program from Dr. Johnson, modify it slightly according to instructions right in the program, and run it to determine the reliability of your scale and the properties of the individual items (frequency of Trues and Falses; item-total correlations, alpha if deleted).
Second, you will revise your scale by eliminating what appear to be bad items. Third, you will assess the reliability of your revised scale. (Steps 2 and 3 may be repeated if desired.)
Fourth, you may examine the relationship between your scale and other students’ items; this is purely optional. Finally, you will examine the construct validity of your revised scale by examining correlations between it and the CPI, HPI, and NEO scales.
Procedure, Step by Step
Step 1: Assessing Properties of Your Initial Scale and Items
First, link to the SPSS program, invspss.txt, and save it to either a floppy or the hard disk of your computer. Open it with Notepad, Wordpad, or the word processor of your choice. (After editing the file you will want to save it as a plain text file, so you might review the recommendations about word processing for plain text files from the second lab, especially if you are considering using a text editor other than Notepad.)
Follow the instructions for modifying the file written directly in the program and indicated with three asterisks at the beginning of the line. *** A particularly important modification to the program is telling it which of your items should be “reversed-scored,” that is, items where a “False” response should count as a point toward what is being measured. Before you begin to edit the SPSS program, it is set up to give a point for “True” responses to all items. If you were measuring, say, depression, “True” should be counted as a point for an item such as, “I often feel blue for no reason.” However, an item such as “Most of the time I am pretty happy” needs to be recoded so that a “False” counts as a point toward depression.
The SPSS program is all set up for you to recode your reverse-scored items. As it stands, the program has a line that begins: RECODE which needs to be completed. Let’s say you want to recode items 80 and 107 to be reverse-scored. You would complete the RECODE line as follows:
RECODE I80 I107 (1=2) (2=1)
This, in essence, tells the program that a True (1) means False (2) and vice-versa for these items. To determine the item numbers for the items which you need to recode to score in reverse, refer to the section of the program marked ***VARIABLE LABELS; this contains a list of your 24 items with their item numbers from the full inventory, inventory.txt. Once you have found the items that need to be reverse-scored, you can complete the RECODE statement.
After you have made the modifications indicated by the *** instructions in the invspss.txt file, save the modified file. Log on to PSUVM and upload your text file, specifying its name on the mainframe as INV SPSS.
From a FILELIST of all your files’ names, type SPSS in the prefix area to the left of the program’s name [followed by ENTER or RETURN]. When the program is done executing, MORE . . . will appear in the right of your screen; clear the screen with the CLEAR key to return to your list of files. REFRESH your list of files with the F2 key; a new file called INV LISTING should appear. You can PEEK at it with F11 and/or download it and print it out.
By looking at the output either on the screen or printout you will be ready to answer the following questions that constitute part of this assignment.
Answers to all questions should be sent in a separate email note to me, j5j@psu.edu.
Questions for Step 1
From PSUVM, please SENDFILE to me the INV LISTING file you use to answer the following questions.
- Referring to your FREQUENCIES printout, which–if any–of your items seem to be seriously skewed from a 50-50 response distribution? Do you have any hunches about why these items are so skewed? Do you think that their skewedness warrants their elimination from your scale?
- Referring to your RELIABILITY printout, what is the current alpha reliability coefficient? Are you satisfied with its value? Why or why not?
- Citing the actual figures in your reliability printout, explain which items seem not to be working well. Which of these items to you plan to eliminate for your next RELIABILITY run?
Step 2: Eliminating Bad Items
Eliminate the items you indicated might be poor in your answer to the above question. This is done simply by xediting the INV SPSS program, finding the line where your scale score is computed, and typing over the items you wish to eliminate with the space bar. For example, let’s say that student ABC123 had the following COMPUTE statement to calculate a scale score:
COMPUTE ABC123=I8 +I39 +I70 +I99 +I123+I153+I175+I197+I222+I251+I274+I303 +I338+I365+I387+I412+I438+I476+I493+I517+I548+I572+I613+I624-K
and the student wanted to toss out items 99, 197, 365, 493, and 624.
After spacing over those items, the new COMPUTE statement will look like this:
COMPUTE ABC123=I8 +I39 +I70 +I123+I153+I175 +I222+I251+I274+I303
+I338 +I387+I412+I438+I476 +I517+I548+I572+I613 -K
Although it isn’t absolutely critical, if you drop some items, you should change the COMPUTE K= statement to reflect the new total number of items. Student ABC123’s statement would be changed to:
COMPUTE K=19
After editing out any items, you should probably save your SPSS program under a new name, say, INV2 SPSS. That way, when you run it, the resulting listing output file (INV2 LISTING in this example) will not overwrite any existing INV LISTING file. So, instead of just typing =====> FILE to save the changed file, type =====> SAVE INV2 SPSS (or whatever you want to call your new SPSS file).
Step 3: Reliability Analysis of Revised Scale
In this step you will repeat the RELIABILITY analysis after eliminating what you believe to be inadequate items (based on serious skewedness and/or poor item/total correlations). To do this you simply type SPSS next to INV2 SPSS (or whatever you’ve named your revised SPSS file) in your filelist. When the program has finished executing, PEEK and/or download and print the resulting LISTING file to answer the questions below.
Questions for Steps 2 and 3
(The answers should be included in the same email note that contains the answers to Questions for Step 1.)
- How much has your alpha reliability improved with your scale revisions? Do your item-total correlations indicate that additional revisions might improve your scale?
- Are you satisfied with the present reliability of your revised scale, or do you wish to delete additional items or replace items you had deleted?
- Remember, there is a trade-off between deleting possibly irrelevant items and losing variance and reliability with a scale that is too short. If you decide to revise your scale further, repeat the procedures for this step, including answering these questions. If you decide to stop revising here, explain why.
Step 4: Adding other Students’ Items to Your Scale
This step is completely and totally optional. There is no penalty for skipping it. I would not undertake this step if I were you unless I were very confident and comfortable with computer work and had some free time during finals week.
There are two options for adding other students’ items to your scale–the rational approach and the empirical approach. One could also use a combination of both approaches. The rational approach is to look at the other students’ items from inventory.txt and choose some whose content seems relevant to the construct you are attempting to measure with your scale.
Let’s say that ABC123 thought items 7, 49, 212, and 439 looked like they went with her items. If any of the items would need to be reverse-scored, it should be added to the RECODE statement line. Also, the COMPUTE K= statement would be changed to reflect the total number of items. Then, the new, trial items can be added to ABC123’s COMPUTE statement as follows:
COMPUTE ABC123=I8 +I39 +I70 +I123+I153+I175 +I222+I251+I274+I303
+I338 +I387+I412+I438+I476 +I517+I548+I572+I613 -K
+I7+I49+I212+I439
Note that the items don’t need to be in special order and can be put on a new line as long as at least one blank is in the first column. A new RELIABILITY analysis should be run on the experimentally revised scale to see if the new items contribute to the alpha reliability.
An alternative to a purely rational approach for finding new items is a blind empirical strategy. In this case we want to identify items that show significant correlations with our own scale. To find such items, one would have to use the following instruction line somewhere near the end of the program (right before the RELIABILITY statement would be fine):
CORRELATIONS VARIABLES=ABC123 with I1 to I720
(where ABC123 is your userid). With this new line in your SPSS program, save it [again, it would probably be best to use a new name like INVEXP SPSS rather than existing names like INV SPSS or INV2 SPSS], and run SPSS on it. You’ll probably want to download and print the resulting LISTING output file because there will be a lot of correlations coefficients to look at–720 to be exact. Normally, researchers do not try out every single item whose correlation reaches “statistical significance” (p<.05) because 5% or 35 of the 720 correlations will appear to be significant due to chance alone. Instead, a researcher usually scans through the correlations, noting items showing a strong correlation with the experimental scale, and then looks at the actual content of these items before trying to add them to the scale. The test items are then evaluated just as before with the RELIABILITY program, modified to include the new items.
Should you decide to attempt to add other students’ items to your scale, please send a separate email note describing briefly what you did along with a copy of the relevant LISTING output files. In the email note, indicate the name of the exact name of the LISTING file (INV3 LISTING, INVEXP LISTING, or whatever).
Step 5: Examining the Construct Validity of your Scale
In serious research, construct validation is a time consuming process of relating your scale to various life, observer, self-report, and test criteria. For our class we have time only to correlate our experimental scales with several standard personality scales. Despite being self-report measures, these established scales have themselves been linked to various real-life criteria and therefore can be used to begin to validate your scale.
The first step in validation entails making predictions about how your scale will correlate with the established scales. These predictive hypotheses might come from your own experience or common sense, from talking with others, from another psychology course, or from reading psychological literature. To make these predictions, you will also need to know what the established scales measure. Some information about these established scales (especially the CPI) are in the required readings, but the following summary might be a helpful guide to making predictions:
Factor I (Extraversion) Scales:
CPI Scales Associated real-life correlates of high scorers
Do Dominance assertive, talkative, self-confident leadership
Cs Capacity for Status ambitious, intelligent status achievement
Sy Sociability outgoing, enjoys social participation
Sp Social Presence energetic, very confident and socially at ease
Sa Self-Acceptance very similar to Sp
In Independence verbal, assertive, self-confident
Em Empathy verbal, animated, good communicator
NEO Extraversion
friendly, warm, affectionate
sociable, outgoing, spontaneous
assertive, forceful, confident
energetic, hurried, enthusiastic
pleasure-seeking, daring, charming
humorous, optimistic, jolly
HPI Scales
SOC Sociability talkative, outgoing, show-off
AMB Ambition assertive, forceful, active
SLS Sales demonstrated sales ability
MAN Managing demonstrated ability to manage others
Factor II (Agreeableness) Scales:
CPI Scales
TL Tolerance open-minded, accepting, rational
FM Femininity gentle, sensitive, considerate
NEO Agreeableness
forgiving, trusting
not complicated or shrewd
gentle, generous, tolerant
not stubborn, headstrong, or impatient
not show-off, assertive, or aggressive
friendly, sympathetic, kind
HPI Scales
LIK Likeability sympathetic, praising, sensitive, pleasant
SOI Service Orientation highly-rated in human service occupations
Factor III (Conscientiousness, Planful Purposefulness)Scales:
CPI Scales
Re Responsibility reliable, methodical, conscientious
So Socialization dependable, serious-minded, ethical
Sc Self-Control controlled, responsible, fastidious
Gi Good Impression conventional, moderate, conservative
Cm Communality organized, stable, realistic
NEO Conscientiousness
efficient, thorough, resourceful
organized, precise, methodical
not careless or lazy or absent-minded
ambitious, industrious, enterprising
organized, energetic, thorough
not hasty, careless, or impatient
HPI Scales
PRU Prudence wise, stable, cautious, practical
RLB Reliability demonstrated reliable behavior in workplace
Factor IV (Emotional Stability)Scales:
CPI Scales
Wb Well-Being poised, unconcerned, clear-thinking
NEO Neuroticism
anxious, fearful, worrying, tense
irritable, impatient, exciteable, moody
worrying, pessimistic, not contented
shy, timid, defensive, inhibited
irritable, sarcastic, loud, hasty
not clear-thinking, confident, or alert
HPI Scales
Adjustment not tense, worrying, moody or unstable
Resiliency not subject to getting stress illnesses at work
Factor V (Intellect-Openness to Experience)Scales:
CPI Scales
Ai Achievement-Independence intelligent, intellectual, wide interests
Ac Achievement-Conformance industrious, productive, persevering
Ie Intellectual Efficiency similar to Ai
Py Psychological Mindedness objective, rational, logical, clear-thinking
Fx Flexibility unusual thinking, imaginative, likes fantasy
NEO Openness to Experience
dreamy, imaginative, mischievous, complicated
artistic, original, inventive, idealistic
spontaneous, insightful, affectionate
wide interests, adventurous, optimistic
inventive, curious, original, insightful
unconventional, flirtatious, not cautious
HPI Scales
INT Intellectance ingenious, artistic, imaginative
SCH School Success insightful, foresighted, clever
After reviewing nature of these scales, decide which of these scales should, theoretically, correlate (either positively or negatively) with your scale. Then look at the results of the last instruction:
CORRELATIONS VARIABLES=ABC123 WITH GENDER TO C
when run with your final, revised scale. This will show all the correlations between your scale (represented here by ABC123) and all other scales. Gender is coded 1=male, 2=female, so a significant positive correlation indicates that females score higher, a negative correlation says the opposite. Examine all correlations, particularly the ones relevant to your hypotheses.
Then answer the questions.
Questions for Step 5
(Include in the same email containing the answers to questions from the earlier steps.)
- What do the Pearson correlations say about the construct validity of your scale? If you were to continue to research your scale, would you revise it? Explain.
- What other steps would you take to evaluate the construct validity of your scale?
- After completing to your satisfaction the construct validation of your scale, for what research or applied purposes would you use the scale?
For Fun
If you’ve answered all of the questions and still want to play around with your data (well, it could happen), see if you can figure out what is being measured by the scales I did not define above (e.g., V1, V2, V3).