This is part of the ANKN Logo This is part of the ANKN Banner
This is part of the ANKN Logo This is part of the ANKN Banner Home Page About ANKN Publications Academic Programs Curriculum Resources Calendar of Events Announcements Site Index This is part of the ANKN Banner
This is part of the ANKN Logo This is part of the ANKN Banner This is part of the ANKN Banner
This is part of the ANKN Logo This is part of the ANKN Banner This is part of the ANKN Banner
Native Pathways to Education
Alaska Native Cultural Resources
Indigenous Knowledge Systems
Indigenous Education Worldwide
 

HOW WELL ARE INDIAN
CHILDREN EDUCATED?

Chapter Five
RESULTS ON TESTS CONSTRUCTED
FOR THE USIS PROGRAM

Two types of tests were constructed for use in the Indian Service testing program. In the first group, the Rural Practices tests were designed to measure the students' understanding of some of those things emphasized in the Indian School curricula. It was recognized that most Indian environments fail to provide certain types of training which most rural white children are believed to receive in their home and out-of-school experiences. The Rural Practices tests include those objectives dealing with the students' understanding of the uses and conservation of natural resources, health and safety factors, home economics, and credit. The second type of test included instruments designed to measure the students' ability to write correctly. The figures in this chapter describe the achievement of the students in the various groups and areas in those fields in which these special examinations were constructed.

In the 1945 program a general resources test was administered to all of the eighth grade students and regional resources tests were given to the boys in three areas. Because of the limited usefulness of the regional tests, they were not repeated in the 1946 program, though many of the items were rewritten and utilized in the extension of the general test.**

Figure V-1 gives the norms for the eighth and twelfth grade students on the extended general resources test, which was administered in 1946. The medians of the various groups and areas vary considerably, with the public school group at both eighth and twelfth grade levels demonstrating the greatest competence. The superior performance of the public school students is interesting and surprising in view of the nature of the test, which like the other rural practices tests, was constructed to measure concepts specifically stressed in the Indian school curriculum. It was thought that the Indian students in public schools would constitute something of a control group in measuring the degree of effectiveness of the instruction in resources given in the Federal schools. Although the Indian students in public schools at both the eighth and twelfth grade levels did less well than white children, they demonstrated higher competence than the students in any of the Indian schools. This would indicate that the formal instruction given in the Indian schools does not equal in effectiveness the training in general resources which is gained through informal channels by students attending the public schools. In other words, results indicate that the training in resources provided by Indian Schools is not sufficient to compensate for the lack of out of school experiences.

Figure V-1

It was thought that this trend, as indicated in the 1945 battery, might be partly explained by the fact that the items included in this examination were those which were of such a general nature that they were labeled as suitable for Indian students in all of the areas. Such criterion of selection would naturally produce a test that would include a wide sampling of items, most of which would be relatively easy. If this sample of items were fairly easy, it would be highly probable that most of these same facts and principles had been discussed and described in story books, science magazines, story magazines, magazine advertisements and newspapers, thereby giving any student who did a fair amount of reading as good a chance for knowing these items as the student who had experienced these facts first-hand on the farm. Therefore, it was thought possible that the test, as developed in 1945, might not have been discriminating between actual firsthand training and experience and the kind of information that could be gained by reading magazines and newspapers. It was for this reason that the 1945 general resources test was extended, making it three times as long as the original test. When it was observed that the data from the 1946 program revealed the same trends, it was concluded that the construction of the original test and the seeming simplicity of some of the items probably could not account for the better performance of the public school students. In other words, the differences indicated are probably due to a failure on the part of the Indian Schools, despite their recognition of the problem, to provide instruction that will as yet offset or balance that training which the public school student receives in his out-of-school experiences.

Tables V-1 and V-2 graphically present data on the relative difficulty and discriminating value at the eighth grade level on each of the twenty items of the 1945 General Resources Test. These twenty items are identical with the first items in the 1946 test. Table V-1 presents the data for the Southwest Area and Table V-2 for the Dakota Area. These two areas were selected for comparison because they constitute relatively large groups, each having recognized differences. Also, at the eighth grade level, each of the two areas includes a fairly good representation of the different types of schools, although for this comparison, data from only the public, day and non-reservation schools have been used.

Table V-1

Table V-2

Per cent of Correct Responses means the per cent of students answering any particular item correctly. For example, in item number 3, which calls for a definition of irrigation, more than 3/a of the public school students answered it correctly, while between 1/4 and %2 of the day school students and %2 to 3/a of the non-reservation students were able to answer it. In this particular item there is a difference between the achievement as demonstrated by the students in the three different types of schools. Table V-2 shows that the item has about the same difficulty in the Dakota area as in the Southwest for both the public and non-reservation students, but the day school students in the Dakota area found the item easier than did the day school students in the Southwest. In instances such as item number 15, on the value of the poultry flock to the grain former, the item was extremely easy for the students in all three types of schools and in both areas. This type of information relative to individual items allows the instructor and the supervisor to tell something about the specific items and objectives that are apparently being taught more effectively in one type of school or in one area than in another.

In general, it may be concluded that while some specific items show considerable variability in difficulty from school to school and area to area, there are a great many items that are mastered about equally well in the various types of schools and in the different areas. While the public school students in the Southwest do equally as well or better than the other two groups, there are only two items-numbers 7 and 11-in which Indian training seems to have resulted in superior performance. Items numbered 2, 7, 14 and 19 in the Dakota area show the Indian students doing better than the public school students.

The discriminating value or item-test correlation provides an index as to relative usefulness of the item in selecting the students who have performed very well I on the test as a whole from those who have not done so well. In this particular instance, these data were computed by comparing the performance on the basis of this particular test, of the students in the top half of the class, with those in the bottom half of the class. Items such as those numbered 3 and 18 have item-test correlation values of better than .70, which means that there is a very high correlation between the public school student's performance on these items alone and what he is able to do on the test as a whole. It is interesting to note that items that are discriminating for public school students are relatively discriminating for the Indian school students as well. In other words, items that will select able students in one type of school are nearly as useful in other types of schools for selecting the good students from the poor ones. It is also gratifying to note that a relatively high percentage of the items have discriminating values of better than .25, for some test builders employ this figure, including as good items those having higher values than .25. There are relatively few instances in which a zero or negative relationship is exhibited. A zero relationship might indicate that the material had not been taught and a negative correlation might indicate that the instruction concerning the item was contrary to the response considered correct by those who constructed the test. Of course, zero or negative correlations may also be interpreted as indicative of weaknesses in the item, or in the way that it has been stated, but in view of the fact that all of these items have merit in most schools and areas, the negative correlations probably do not reflect weaknesses in the test item itself. Item 16 from the Dakota area is a case in point. The item is very discriminating for the public school students and somewhat so for the non-reservation school students, but for the day school students it may point to the fact that this item is not taught or that a certain amount of misinformation may have confused the students.

Figure V-2

Figure V-2 is the norm sheet for the Health and Safety Test. Tables V-3 and V-4 describe the content of all fifty items as well as showing the relative difficulty of the various items and their degrees of discrimination.

Figure V-2 reflects tremendously wide ranges of performance between the students in each of the groups but in most instances the differences between the medians of the groups are not large enough to be important. Though the students in the day schools demonstrate the poorest performance at the eighth grade level, there is almost no difference between the students in the various types of schools at the twelfth grade level. . In the Dakota region the twelfth grade students in Indian schools did a little better than the students in public schools, and in Oklahoma, students in the two types of schools did equally well. The results would indicate that health and safety instruction in the Indian schools has been sufficiently effective to compensate for the students' lack of home training.

This test furnishes a good example of the usefulness of item-by-item analysis of the test data. The instructor will find much of interest to him in Tables V-3 and V-4, for these tables reveal the specific topics in which the eighth grade students from the three types of schools of the Southwest and Dakota areas encountered varying degrees of difficulty. For example, item 10 measures the students' understanding of the health and safety factors contributed by the "purification of water." In the Southwest it will be noted that the item was very easy for the public school students, having been answered correctly by more than three-fourths of them. In the day and non-reservation schools, the students found this same item somewhat more difficult. The symbols in the right hand columns describe the discrimination values of the items. Among day school students, item 10 was highly discriminating, for it was easy for the good students and difficult for the poor students. In the cases of both the public school and non-reservation students, the item is shown to possess negative or zero discriminating value. For the public school students, this lack of discrimination should modify the interpretation that has probably already been given by those who have observed that the public school students are well informed on the item of water purification. While it is still true that they are relatively well informed, it is also of importance to know that the poorest students in this group do just about as well as the best students in the class. When an item has been thoroughly and effectively taught to a class, it is often found that both the poor and good students, as measured on the test, show near perfect performance on the item. On the other hand, the fact that good and poor students do equally well could also mean that this item was not taught in school but that the students have acquired the information elsewhere. For the non-reservation students in the Southwest item 10, on water purification, appears to be relatively easy, but is not discriminating. One would conclude, therefore, that in this area, the understanding of water purification among non-reservation students is not a good index as to whether they understand all of those factors which have been included in this test on health and safety. Specific teaching of the water purification topic should have the effect of increasing the discriminative value of this item the next time such a test is given, for the better students will in all probability have their scores improved much more than will the slower or poorer students. The discriminating value of item 10 has not been computed for the twelfth grade data. It is possible that the teaching in the four-year period between eighth and twelfth grades will be found to have produced a higher discriminating value for the item on the twelfth grade tests.

Table V-3

Table V-3

Table V-4

Table V-4

The non-reservation students in the Dakota area find item 10 about as easy as do the students in the Southwest, but in the Dakota area, the item is highly discriminating. This could be interpreted that there has been a more definite effort to teach water purification to the non-reservation students in the Dakota area than to the students in the Southwest.

Figure V-3 shows the distribution of test scores for the Home Economics Test. This test was designed as a companion for the General Resources Test and was given to all of the eighth and twelfth grade girls. No regional tests were prepared because the subject matter included was judged to be of importance to the girls in all areas. The performance of the girls on the Home Economics Test is about the same for all types of schools with the exception of the eighth grade girls in mission schools who fall somewhat lower. The performance of the Oklahoma Indian school group, at both the eighth and twelfth grade levels, is higher than that of the total public school group at comparable grade levels. This is of special interest when one considers that the test was prepared to measure the degree to which the Indian Schools have been successful in their attempt to compensate for the lack of home training which their students receive in relation to good homemaking practices. The fact that this performance is due to instruction in Indian schools is pointed to by the fact that the public school Indian students do less well than the non-Indian students.

Figure V-3

Tables V-5 and V-6 present an evaluation of the items in the Home Economics Test. As in the previous item evaluation, these data have been computed for the Dakota and Southwest areas. Examination of these figures will disclose much more agreement than might be expected. The few items, such as numbers 2, 9, 21 and 28, that are very difficult for all the students seem to be difficult for the students of both areas and for the students in all three of the types of schools. Therefore, if this difficulty is due to the material not being taught or being taught incorrectly, the fault is not characteristic of a single type of school or of the schools in either of these areas. The same thing may be said for the items which appear to have been easy, such as numbers 3, 5, 7, 12, 14, 16, 25, 27, 33, 35, and 38. All of these items appear to be fairly easy for students in all three types of schools and in both areas, again showing no appreciable deviations.

The curriculum expert would probably look for more instances such as items numbered 18, 26, and 37 in the Southwest, for here in Table V-6 are examples of the three types of schools in this area demonstrating three levels of proficiency. While these three items will undoubtedly be of passing interest, the fact that only three items have such ranges is hardly enough to make anyone think that there are significant differences in the attention given to Home Economics. On the whole, the discriminating values for the Indian School groups are higher than for the public school students, and this is additional evidence that the test measures achievement of topics stressed in the Indian School curriculum.

Figure V-4 is a norm sheet describing the achievement of students on a test which has been called the Credit Test. The purpose of this examination was to determine the kinds of conditions or situations which the students felt would justify the granting of credit or loans. Various situations were described in each item of the examination and in each case the student was expected to decide whether or not the situation was such as to warrant the giving of credit. The situations were so set up and described as to convince those responsible for the test that the situations were typical and that all of the items could be keyed. The situations chosen were those which students might have observed and which due to their own training, lack of training, or prejudice might answer incorrectly. The results from a test of this sort therefore reflect a great many things. Certainly it would not be wise to interpret the results as a measure purely of educational opportunity or direct training in the classroom, for the training, whatever it might be, is modified so much by home experiences. Therefore, it is possible to think of this test partly as one measuring attitudes and its results should be examined diagnostically with that in mind.

Table V-5

Table V-5

Table V-6

Table V-6

Figure V-4 illustrates that there is relatively little variability between the median attainment of the different types of schools, but somewhat more variability between the Indian schools in the different regions. In this test the Oklahoma group is highest, even higher than the median for all public schools. Analysis of the test items showed that none of the items gave negative correlation values. In fact, most of the correlation values were relatively high, showing a high degree of discrimination. However, the credit test was not repeated in 1946 for it was planned to incorporate these, or modified items, in a longer test designed to measure a wide scope of attitudes. This test is still under construction.

A Rural Practices Vocabulary Test was constructed for administration in the 1946 program. Its purpose was not that of a general vocabulary test, but rather that of measuring the degree to which the students understood the vocabulary being used in the rural practices tests. The purpose of the test has been fulfilled when it has been discovered that:

1) the achievement of all students on this select vocabulary is relatively high, thereby indicating that the word difficulty of the rural practices tests was not a serious handicap;

2) a positive correlation between the vocabulary -and rural practices scores exists; and

3) the results have been made available to the teacher in order that the scores on the special vocabulary may be used in the analysis of individual student scores on the rural practices tests.

The second type of tests constructed for use in this study are known as the Free Writing Tests. It was decided that the standardized "proof reading" type of English test was inadequate to measure the students' skill in written expression. The standardized test measures the students' ability to detect errors in printed material; the Free Writing Tests measure the students' ability to use English in expressing his own thoughts clearly and correctly.

In 1945 two forms of the Free Writing Tests were given to each of the eighth grade students. Because Form A of the test appeared to motivate a freer type of expression, this form of the test was given in 1946 to all of the fourth and twelfth grade students. Unfortunately, the extent to which the method of interpretation of results would be altered was not anticipated in time to indicate the desirability of repeating the test at the eighth grade level.

Figure V-4

Objective scoring of the Free Writing Tests is somewhat more difficult than the scoring of most standardized tests, but the results indicate that it is a better measure of the type of language skill which Indian schools are attempting to develop. Each of the 1946 papers was scored in three ways to determine:

1) the total number of errors in spelling, punctuation, usage and sentence structure,

2) the maturity of English construction attempted, and

3) the types of errors most common at different grade levels and in different geographic areas.

A fourth type of analysis which is recognized as important, but which has not been undertaken thus far, is a qualitative analysis of the ideas expressed-the abiIity of the writer to sustain the reader's interest, his imagination and creativeness.

Figure V-5 shows the results or norms for the Free Writing Tests-Total Errors Per Sixty Words for the fourth and twelfth grade students in 1946. The distribution of scores on the 1945 study are not shown because the revision in method of analysis yielded scores which are not comparable to those reported in 1945. The unit of sixty words in each composition was decided upon because the opportunity for error increases in proportion to the length of the composition. Experimentation indicated that there is considerable consistency in the number of errors which occur throughout the composition. A correlation of .872 was found between the error score based upon sixty words of a composition and the error score of the entire composition. It was therefore decided that sixty words was a sufficiently reliable sample of the whole.

The opportunity for error likewise increased in proportion to the increase in complexity of the composition. For example, a student may entirely eliminate errors in either omission or wrong use of commas by using only simple sentences where periods are adequate. There are fewer opportunities for errors in sentence structure if he uses short, choppy sentences.

The following fourth grade compositions serve to illustrate the importance of considering complexity as well as the total number of errors in making comparisons between the achievement in the Free Writing skill of different students.

(Example)

The man is riding on a horse. A dog is going with him. The man is going to the hills. A girl is standing by the door. She is telling him to be back before sundown. There is a little house a little way from the house. The dog stays in the little house. There are trees… Hemill bring a rabbit.

Figure V-5

In contrast to the simplicity of this paper, the following sample illustrates the opportunities for error, and resultant errors which occurred through the introduction of complex and compound sentences. (The 60 word sample in heavy face type indicates the 60 word sample which was used as a basis of computing the total error score.)

(Example)

This Indian is going out toward the mountains to kill large game for his family, and he is taking one of his best dogs with him. He is going out because his family do not have anything to eat in their house for supper.

He is the only Indian warrior in the camp, because just his family and him are the only ones there.

The little girl at the door is trying to tell him to bring back a live baby deer for her pet.

After he's gone a little way She tells him Goodbye and then go back into their little house.

The mountains are high and rocky but the Indian is going out to hunt. The Indian man will return soon, from his long hunt in the mountains. The Indian man told the little girl that he would kill a deer and bring back two baby deer for the little Indian girl's pet. The little girl is so happy that she claps her hand and tells him to take care of himself on the journey.

After along time the little girl wonders why her father hasn't returned yet. But her questioned is soon answered, for out in the open spaces comes some Kiwoa Indians to the camp to get the women and children and make slaves out of then and take them home to their houses.

But that was not all because the little girl saw out in the opening saw her father coming across the broard plan with something on the pony's back and two baby wolfs in his arms instead of deer.

When the Kiowa Indians saw him coming they ran off and were never seen again because the man had brought back other warriors who were part of the people. There were about 49 people who were with them then. and then they lived happily together and when the wolf got big she littered.

Since many errors in punctuation result frorn wrong use, it was decided to show the opportunity for errors in punctuation by counting the number of different kinds of punctuation marks which the student attempted to use. Only those which were used in the 60 word sample were considered in determining the qualitative punctuation score.

The number of different words of more than one syllable was used as an index of the student's attempt to use complex words. Obviously a student with limited spelling ability might make relatively few errors if he used only simple words which were easy to spell. Since a misspelled word was counted an error only once, the use of an index which indicated different words was suggested since repetition of the same word decreased the opportunity for spelling errors.

The third type of analysis of the Free Writing papers involved a careful study of the specific types of errors found in the papers of the different Indian students. This study was made by means of careful sampling of the papers of both fourth and twelfth grade students in each of the geographic areas. The samples were grouped so that the findings indicated whether an error was typical of all students at this grade level, or only of those who made many errors.

It was found that there was marked similarity, both in type and in frequency of errors, between the poorest twelfth grade and the best fourth grade papers: With the exception of errors in the use of the comma, the frequency of each type of error was considerably higher among the fourth grade students. In the case of the comma errors, it seems probable that the opportunity for comma errors was markedly increased by the greater complexity of the twelfth grade compositions, for the twelfth grade students made more errors in the use of the comma than did the fourth grade students. The largest percentage of errors among all fourth grade papers were those of omission of capital letters at the beginning of sentences. Almost as many errors at the fourth grade level were in the capitalization of common nouns and the use of a capital letter at the beginning of each line: The total number of errors in the wrong use of the capital letter was greater than those resulting from omission of capital letters. This difficulty in the use of capital letters in the fourth grade may spring from the fact that most Indian Schools do not place as much emphasis on the proper use of capital letters in the lower grades as is customary in the average public school. This omission is deliberate, and designed to simplify the problem of alphabet recognition for pupils coming from nonreading homes. The most frequent errors in sentence structure were classified as "awkward and dangling sentences and modifiers." Among the poorer twelfth grade, and most of the fourth grade papers, the "run-on sentence" accounted for a large percentage of errors. The verb error which occurred with greatest frequency at both grade levels was in lack of agreement with the subject in person or number. At the fourth grade level the omission of the verb or auxiliary and the omission of an "ing" ending was common. The most frequent noun errors were those which resulted from a confusion of plural and singular forms. A relatively large percentage of errors resulted from mistaken identity of words. It is interesting to note that almost no double negative errors occurred at either grade level.

It was noted by those who scored the 1945 Free Writing tests that the illegibility of the penmanship sometimes made it difficult to determine whether or not the student had misspelled a word. It was decided that the quality of student penmanship should be studied in 1946.

Because it has been suggested that there is no merit in forcing a child to transfer from manuscript to cursive writing, students in many of the Indian schools continue to use manuscript writing throughout their entire period of school attendance. It was therefore necessary to employ two separate Handwriting Scales?one for judging the quality of cursive writing and one for manuscript writing.

A large number of scales were examined for their possible usefulness in evaluating the quality of handwriting. It was necessary that the range of the scale should be wide in order that it would extend from the poorest penmanship of the fourth grade students to the best penmanship of the twelfth grade students. All of the scales examined appeared to discriminate between finer differences in quality than seemed either justifiable or necessary. An attempt was made not to require the distinction between extremely small differences so that there would be no disagreement between the judges. Cursive writing was evaluated by comparison with samples selected from the Thorndike Handwriting Scale, shown in Figure V-6. The samples from the Bronxville Scale of Manuscript Writing, by which the quality of the manuscript penmanship was judged, are shown in Figure V-7.

Three classroom teachers were selected to rate the penmanship of all students who took the Free Writing tests. In order to evaluate this method of scoring penmanship, one hundred samples of twelfth grade penmanship were compared with the selected samples by each of the three judges. The rating of each judge was so indicated that his rating could not be seen by the next judge. The independent ratings of the three judges were then compared. It was found that in no instance did all three judges disagree as to the quality of the penmanship and, in no instance did the judge who disagreed with the other two deviate more than one point on the selected scale. In seventeen instances, one of the three judges disagreed with the other two. In 83 instances there was unanimous agreement. The method was repeated with 100 samples of the writing of fourth grade students, again with 100 samples of students of different grade levels, and again with students who used manuscript writing. The results each time were almost identical. The disagreements never exceeded that of the first experiment and there were never less than fifteen disagreements.

Thorndike Handwriting Scale

Thorndike Handwriting Scale

Thorndike Handwriting Scale

Bronxvville Manuscript Writing Scale

Bronxvville Manuscript Writing Scale

Bronxvville Manuscript Writing Scale

Bronxvville Manuscript Writing Scale

Bronxvville Manuscript Writing Scale

Figure V-8

Figure V-9

It was found that some of the disagreement resulted from difference in opinions of the judges in regard to the merit or lack of merit of flourishes, and "Spencerian type" of writing which did not fit on the scales selected. The primary grade teacher was somewhat more sensitive to "drawn" writing than were the secondary teachers. Recording first impressions appeared to result in at least as high reliability as careful analysis of the sample.

The rating assigned to the penmanship represents the independent agreement of at least two of the three judges. It is of interest to note that the time required to obtain three independent judgments of approximately. 4,000 papers, including the recording of the ratings, was 10 days, or 240 man-hours.

Norms for the quality of penmanship of the fourth and twelfth grade students are shown by Figures V-8 and V-9. The number of students at the twelfth grade level who elected to use manuscript penmanship was too small to permit computation of norms for that grade.

** See Appendix A for details of the revision of the test

 

 

Go to University of AlaskaThe University of Alaska Fairbanks is an affirmative action/equal opportunity employer and educational institution and is a part of the University of Alaska system.

 


Alaska Native Knowledge Network
University of Alaska Fairbanks
PO Box 756730
Fairbanks  AK 99775-6730
Phone (907) 474.1902
Fax (907) 474.1957
Questions or comments?
Contact
ANKN
Last modified August 17, 2006