Saturday, September 3, 2011

Validity in Assessment

Definition
            While designing a test, it is essential for it to be valid. Validity can be defined in different ways.
“It is the extent to which a test measures what it is supposed to measure”.
“Validity is the subjective judgment made on the basis of experience and empirical indicators”.
Validity is “the agreement between test score or measure and the quality it is believed to measure”. (Kaplan and Saccuzzo 2001)
            In simple words we can say that validity refers to the meaningfulness of the test. This meaningfulness can work at two levels. At the level of the design, the design of the test should be according to the requirements. At the level of context, the test should be used for the specific purpose for which it is designed. We cannot use a mathematics test to test the writing skills of the learner, it is against the context, the test is used other than its context and will not be valid.
            There are various kinds and aspects of validity. Here they are tried to discuss according to their relevancy and importance.
1. Face Validity
            In fact it is not a kind of scientific aspect of validity. Face Validity refers to the face of the test among general public, test-takers and other lay people/non-related persons etc. Face validity means the test should have certain characteristics. These characteristics are those which the people expect about the test. They include proper printing, a governing body, an appropriate manner of test taking, subjective/essay type of questions etc. Thus a mathematics test will not be considered valid according to face if it has not numerical questions. Numerical Question is the expectation of the lay people from a mathematics test.
            A test that does not have such evidences may be rejected by the test takers and the governing body may bear criticism. Face Validity is nothing in the opinion of the expert but a test should look like a test a common man thinks. So face validity becomes an issue for the test designer and paper setter.

2. Predictive Validity
            Predictive Validity refers to the future performance and success of the learner. This aspect of validity ensures that the test is providing valid information about the future performance of the learner. So it includes all those situations or skills for testing which the learner encounter or perform in his future. The example of the tests must have predictive validity are entry tests and selection tests.
            Language aptitude tests should have predictive validity because they test present skills for future performance. Proficiency tests also use predictive validity. In diagnostic and achievement tests although there are other types of validity involved but they should also have predictive validity. As they are also liked with the future performance of the learner.
            Predictive Validity is calculated by statistical co-relations. The validity coefficient is calculated usually by comparing success in the test with the success in job. Thus the validity of the test is checked and improved for future tests. A 0.6 value of the coefficient is considered high which indicates that all tests do not have predictive validity.

3. Concurrent Validity
            There is  no major difference between the two validity types i.e. Predictive Validity and Concurrent Validity except time. Predictive Validity is related to futures while concurrent validity is related to present. When the future becomes present the predictive validity becomes the concurrent validity. We compare two tests instead of a test with future or job performance.  This comparison is of two test taken usually simultaneously, one written and other oral or spoken usually. It is used to limit the criterion related errors. Most suitable situation is the comparison of a new test with already established test/criterion to find its validity and meaningfulness.
            So a test having concurrent validity will show its validity in a given field. Concurrent validity is  a statistical measure which requires a quantifiable criterion. Although all the criterion are not quantifiable but statistical approach assumes that they are quantifiable. Co-efficient of validity is used to compare the two tests.

4. Content Validity
            It is the appealing aspect for the expert. It seeks the extent upto which the test represents the content from which it is constructed. 
            It is required in achievement tests. They represent a content/syllabus and they should be constituted from the given syllabus and content. Similarly diagnostic tests should also have content validity because they seek certain deficiencies of the learner from a given set of skills or syllabus. The chief examiner, advisor etc. can check that the test is representing the content which it is going to test.
            Teaching materials should have their own validation i.e. predictive and construct validity. Otherwise the content validity of the test will not be fruitful. So the teaching of speaking skills should involve such materials and the examples from native speakers which teach the student appropriate speaking/spoken skills.
            In proficiency tests the content validity can also be employed. Learners will have to perform in certain situations so the test can be a representative of those skills testing. Those areas can be specified before the exam just like the syllabus of the achievement tests. This is a guesswork as compared to other tests where we have syllabuses.
5. Construct Validity
             A construct is a theory, usually a psychological one, which explains certain mental process say learning. It will thus say how the learner learns the language and what are the factors involved and what is the nature of language etc.
            On the base of such theory the test is constructed to evaluate certain factors/indicators of the learner language to measure his ability. Thus the test will have the construct validity if it represents the aspects of that particular theory on which it is based.
            Here a point should be kept in mind that a construct may be wrong. So a test having construct validity will become meaningless due to the false theory on the basis of which it is constituted. Here the problem will be with that construct not with the test. Materials and syllabuses should also be evaluated on the basis of construct validity to know if they represent and teach the language according to the theory of language and language learning. 
Different questions relating each validity evidence are presented in this table.
Content
1.     Do the evaluation criteria address any extraneous content?
2.     Do the evaluation criteria of the test address all aspects of the intended content?
3.     Is there any content addressed in the task that should be evaluated through the test, but is not?
Construct
1.     Are all of the important facets of the intended construct evaluated through the scoring criteria?
2.     Is any of the evaluation criteria irrelevant to the construct of interest?

Criterion (Predictive + Concurrent)
1.     How do the scoring criteria reflect competencies that would suggest success on future or related performances?
2.     What are the important components of the future or related performance that may be evaluated through the use of the assessment instrument?
3.     How do the scoring criteria measure the important components of the future or related performance?
4.     Are there any facets of the future or related performance that are not reflected in the scoring criteria?

Validity is a pre-test concern. We should develop tests in such manner that they have the validity and meaningfulness. In this regard a three step approach can be helpful. 
1. First, clearly state the purpose and objectives of the assessment.
2. Next, develop scoring criteria that address each objective.
3. If one of the objectives is not represented in the score categories, then the rubric is unlikely to provide the evidence necessary to examine the given objective. If some of the scoring criteria are not related to the objectives, then, once again, the appropriateness of the assessment and the test is in question.
Sources of Invalidity
            Validity can suffer due to various factors some of which are discussed.
1.      Lack of reliability indicates that the test is not valid. Although the contrary may also be true, that is, a test is reliable and consistent in its results but it may be meaningless in certain context and irrelevant. Reliability should be seen to prevent invalidity.
2.        Content and Construct under-representation is a situation in which important aspects of the content and construct are not included in the test. Thus the results are unlikely to reveal the true abilities of the student's abilities within that construct or content which were indicated and having been measured by the test.
3.      Content and Construct over-representation is a situation in which the aspects of the content and construct are represented in the test in excess, that is, irrelevant part are also included in the test.  This can further be divided in two situations:
1.      One where the over-representation leads the test to easiness and the learner or test-taker can get some clues from the test to solve some problems, thus guessing increases invalidity.
2.      Other situation can lead the test to difficulty and it becomes difficult for the student to score appropriately. It is not the fault of the student but the construction of the test affects him and he cannot perform well.

No comments: