Project OATS Final Report

I.   Purpose:

Project OATS (Outcomes of Assistive Technology in the Schools) was designed in response to the need to know the impact of assistive technology (AT) devices and services in the schools. Project OATS sought to develop and test instrumentation useful for assessing assistive technology outcomes in the school settings. Furthermore, Project OATS began identifying database fields useful for planning more effective interventions by linking outcomes to AT services.

II.   Chronology of Project

  1. Year 1 (1997-1998) Activities
    1. National Survey of AT Outcome Instruments
      The assessment instrumentation component of the research project included identification and piloting of existing instruments. After a thorough literature search and consultation with experts in the field, the Project OATS team compiled a list of 27 assessment instruments that had potential for measuring assistive technology in the schools. Next, the 27 instruments were reduced to 9 by the Project OATS Team, based on current use in educational settings around the country and similarity to existing special education and therapy assessments normally used for evaluation of children with special needs within the public school system. They included: Lifespace Access Profile, AT Screener, ASNAT, Assessment for AT, PEDI, SFA, QUEST, MPT, SchoolFact and were field tested to identify if any one or combination of them may be useful in AT outcomes measurement in the schools. Special educational professionals administered one of the nine assessments to students within their regular caseload. The special education professionals included 9 speech and language pathologists, 7 educators and 6 occupational therapists. Data were collected to identify assessments most sensitive to assistive technology process and outcome assessment, most generic (exportable to other schools), most feasible to administer, and closest to the construct of predicting and measuring educational outcome. After administering the assessment, each participant completed a questionnaire regarding opinions about the instrument. Next, the participants came together to discuss the instrument in small groups and present their opinions to the others. The major finding from this study was identification of The School Function Assessment (SFA) and SchoolFact assessments as those that best fit the identified criteria for further investigation as a possible AT outcomes instrument in the schools.
  2. Year 2 1998-1999 Activities:
    1. Development of AT Supplement to the SFA
      The SFA was chosen for further investigation over SchoolFact because it was already being used by therapists and educators in the Wisconsin schools. Information from the field testing in year one was combined with scrutiny by the Project OATS Team to determine if in fact the SFA could be used to measure AT outcomes in the schools. A number of potential problems were identified with the SFA. They included:

      1. Medical model nature of SFA
      2. Difficulty in separating AT from other interventions provided
      3. Not enough specific information about AT to determine the outcomes of specific devices.These problems were discussed and possible solutions identified. The solutions required including modifications to the SFA creating a new supplement to the SFA specific to AT.
    2. Assessing the validity of the AT Supplement to the SFA
      Traditional test and measurement theory requires the study of the reliability and validity of any new instrument. Even though, the AT Supplement to the SFA is a modification, it is still a new measure and required scrutiny. This following study of the AT supplement to the SFA was restricted because of size to a review of only one important aspect of validity known as consequential validity. The concept of consequential validity grew out of construct validity asking the question: Does an instrument measure what it purports to measure? Literature to support this definition emerged in the education psychology over the past decade. The concept of “Consequential validity” grew out of construct validity and includes the validity of the meaning or interpretation of the scores. The following Project OATS study examined this aspect of the technical adequacy of the AT Supplement to the SFA. It also investigated whether the AT Supplement to the SFA provides more information than the SFA in identifying effectiveness of assistive technology intervention. Participants targeted for inclusion in the study were school practitioners with varying backgrounds in assistive technology and from a variety of locations. As the literature suggested, this study examined the interpretations made by therapists who have reviewed a completed AT Supplement to the SFA or a completed SFA. Analysis of interpretive data and questionnaire responses was conducted by comparing the number of specific interpretations related to assistive technology made by the participants to interpretations made by assistive technology specialists. Results did not support the hypothesis that practitioner’s perceptions of the overall quality of assistive technology interpretations are different when using the AT Supplement to the SFA. Results did support the hypothesis that the number of matching interpretations about the outcome of AT implementation between participants and AT experts differ between participants using the AT Supplement to the SFA and those using the SFA.There are four important implications for practice related to the results of this study.

      1. Therapists in both groups may have been overconfident in their ability to identify assistive technology outcome interpretations.
      2. Inequality in experience with the instrument may have contributed to the insignificant results found in the first t-test.
      3. The results from the second analysis suggest that therapists using the AT Supplement to the SFA will formulate better interpretations based on AT outcome than those using the SFA.
      4. The second analysis upholds an important aspect of construct validity of the AT Supplement to the SFA, lending support that the AT Supplement to the SFA may accurately measure the construct of AT outcomes in schools.
    3. Focus Group for Database Field Identification
      During year 2 a focus group was held to begin identifying fields for the database. A group from the CESA #1 AT taskforce was invited to participate. The question “what are the important factors of successful assistive technology programs for children 3-21” was posed. Participants were given 15 minutes to respond to the question on paper. Next, the group facilitator conducted a round robin to obtain answers from all participants. Finally, the participants grouped the list of answers into major categories. The major categories include: equipment, training, staff competence and staffing issues, staff attitude, IEP planning/assessment and follow up, team process, financial issues, student centered issues, support from administration and program evaluation.
  3. Year 3 (1999-2000) Activities:
    1. Further Investigation of the AT Supplement to the SFA
      1. Can a part of the SFA be used to measure AT outcomes?
        The next Project OATS study specifically examined the SFA Task Supports Adaptations Scale of Part II. The Adaptations Scale includes but does not distinguish assistive technology from other adaptations. Also, the SFA Checklist contains adaptation interventions including assistive technology devices, environmental adaptations, and adaptations to the task. Examining the relationship between the Adaptations Scale and the assistive technology specific adaptations of the Checklist helped determine whether the Adaptations Score includes only assistive technology adaptations or includes other constructs. This study focused on examining the relationship between the assistive technology component of the Checklist and the Adaptations Scale of the Task Supports section of the SFA to assess the construct validity of the Adaptations Scale of the SFA as a measure of assistive technology outcome.Results of this study were mixed. The first analysis demonstrated that there was a significant relationship between the weighted number of assistive technology devices and the Physical Tasks Adaptation Total Raw Score. Interestingly, the second analysis did not show a significant relationship between the number of weighted assistive technology devices and the Adaptations Total Raw Score of the Cognitive/Behavioral Tasks. There are three conclusions that can be made regarding the use of the Task Supports section of the SFA as an outcome instrument for assistive technology interventions. 1) Assistive technology is a significant factor in the Physical Tasks Adaptations Scales, but it is not the only factor. Therefore, the Adaptations Scale also consists of other components. The use of this scale as an assistive technology outcome instrument is questionable. 2) The Adaptations Scale is not applicable to all scoring situations. When a student is not using adaptations and not participating in activities as same aged peers inconsistencies in scoring may occur. 3) The wording in the rating scale and the directions is confusing; often interchanging the words “uses” and “requires.”
    2. Is the AT Supplement to the SFA sensitive enough to discriminate change based on AT intervention?A study is currently underway to determine if the scales on the AT Supplement to the SFA are sensitive enough to discriminate change in function based on AT intervention. The study solicited participants from Wisconsin to administer the AT Supplement to the SFA at two points in time with students who received some type of AT intervention. The current data include 26 completed AT Supplement to the SFA forms. While the study is currently in progress, preliminary results indicate that the AT Supplement to the SFA is in fact sensitive enough to detect change due to AT intervention.
    1. Database planning
      During year three, Project OATS staff used the focus group information as a springboard to began planning the database intended to combine data from the outcomes instrument and information about AT services provided in the school setting. During year three, a database questionnaire was developed for administration locally to AT teams and nationally to special education directors. The database was designed such that it could be administered in questionnaire form over the phone if needed.

III.   Summary of Deliverables

The primary deliverables from Project OATS include:

  1. A listing with descriptions of the 27 instruments initially reviewed by the Project OATS team
  2. The AT Supplement to the SFA including directions for use and scoring
  3. The Database fields
  4. An extended list of Publications and Presentations at numerous local and national conferences
  5. A brochure was also developed to provide more information about Project OATS to study participants as well as attendants at conferences

IV.    Summary of Findings and Clinical Utility

  1. The SFA and SchoolFact were identified as 2 potential AT outcome measures for the schools from a national set of 27.
  2. Both the SFA and SchooFact were determined to require modification in order to begin to measure such outcomes
  3. The modifications made to the SFA, called the AT Supplement to the SFA were effective in producing an instrument allowing therapists to make accurate interpretations about AT outcomes in the school setting.
  4. Assistive technology is a significant factor in the Physical Tasks Adaptations Scales of the SFA but it is not the only factor. Therefore, the use of this scale as an assistive technology outcome instrument is questionable.
  5. The Adaptations Scale is not applicable to all scoring situations. When a student is currently not using adaptations and not participating in activities as same aged peers inconsistencies in scoring may occur.
  6. Preliminary data suggest that the AT Supplement to the SFA is sensitive enough to measure functional change due to AT intervention.
  7. Pending confirmation of the preliminary statistical analysis (summer 2004), the AT Supplement to the SFA (Part III and checklist) would appear to measure AT outcomes in the schools until a better measure is available.
  8. There appears to be substantial difference in AT contributions to performance between Physical and Cognitive/Behavioral Tasks as defined in the SFA. Our data suggest that AT alters Physical Task performance, but does not seem to with Cognitive/Behavioral Task performance.

V.   Implications For Future Work

  1. Obtain further funding for investigation of AT outcomes based on the data collected during the course of the project.
  2. Further develop the database model.
  3. Investigate ways to improve the AT Supplement to the SFA
  4. Further investigate the potential of SchoolFact for measuring AT outcomes.
  5. Further study needs to clarify why AT has not seemed to impact Cognitive/Behavioral performance. Theories include: a) Participants came from CESA #1 where there are a higher number of children with low incidence disabilities focusing on physical impairment; b) Cognitive / Behavioral interventions tend to focus more on adaptations to program or environment then devices; c) Therapists and special education teams are not well trained to apply AT for cognitive problems; d) AT devices for cognitive and behavioral interventions are in development and are not yet available for use.
  6. Related to Project OATS, this team was energized and is running a Federal Grant (ATOMS-Assistive Technology Outcomes Measurement Systems Project) which is in the process of systematically exploring, piloting, and testing AT outcome measurement ideas in order to recommend the next generation outcome system for AT. To find out more about this project please visit the ATOMS Project website (http://www.atoms.uwm.edu/).