Description Each question needs to be answered individually. The rubric is down below with the ques ...
Description Each question needs to be answered individually. The rubric is down below with the question prompts. There is a video to help answer the questions. Here is the link: UNFORMATTED ATTACHMENT PREVIEW C 2015 Wiley Periodicals, Inc. Psychology in the Schools, Vol. 52(4), 2015 View this article online at wileyonlinelibrary.com/journal/pits DOI: 10.1002/pits.21829 A COMPARISON OF MOMENTARY TIME SAMPLING AND PARTIAL-INTERVAL RECORDING FOR ASSESSMENT OF EFFECTS OF SOCIAL SKILLS TRAINING KEITH C. RADLEY, RODERICK D. O’HANDLEY, AND ZACHARY C. LABROT University of Southern Mississippi Assessment in social skills training often utilizes procedures such as partial-interval recording (PIR) and momentary time sampling (MTS) to estimate changes in duration in social engagements due to intervention. Although previous research suggests PIR to be more inaccurate than MTS in estimating levels of behavior, treatment analysis decisions have been found to be similar across procedures. To addresses limitations of previous studies that did not find substantial differences in treatment analysis decisions, PIR and MTS estimates were compared to those of continuous duration recording of social engagements to determine the effect on treatment interpretation. Five previously conducted social skills training data sets were coded using PIR, MTS, and duration recording. Treatment analysis interpretations using MTS were found to closely match those made using duration recording, but interpretations using PIR significantly overestimated the effectiveness of the intervention. Implications of findings are discussed in terms of assessment for social skills C 2015 Wiley Periodicals, Inc. training in both research and applied settings. Social impairments have been described as the most salient characteristic of autism spectrum disorders (ASD) (Carter, Davis, Klin, & Volkmar, 2005). Deficits in social functioning are observable from early in life (Landa, Holman, & Garrett-Mayer, 2007; Sigman, Dijamco, Gratier, & Rozga, 2005), with impairments becoming more apparent throughout the school years. Children with ASD spend substantially less time engaged with peers than typically developing children (Bauminger, Shulman, & Agam, 2003; Landa et al., 2007). Failure to successfully engage with peers may inhibit the development of intelligence, language, and other important skills, contributing to poor outcomes throughout the lifespan (Cederlund, Hagberg, Billstedt, Gillberg, & Gillberg, 2008; Howlin, Mawhood, & Rutter, 2000; Strain & Schwartz, 2001). For these reasons, social skills interventions are among the most frequently utilized therapies for individuals with ASD (Goin-Kochel, Myers, & Mackintosh, 2007; Reichow & Volkmar, 2010). Accurate assessment of change in social engagement skills is an essential component of intervention (Bellini, 2006), and is mandated by the Individuals with Disabilities Education Improvement Act (2004). Naturalistic, direct observation is most likely to provide a valid assessment of social engagement of children with ASD in comparison to indirect measures (Elliott & Gresham, 1987). Direct observation benefits from being sensitive to change in social engagement behavior (Cunningham, 2012), which is essential for monitoring response to intervention. Continuous recording methods (e.g., continuous duration recording) may be utilized to directly assess clinically meaningful dimensions of target social behaviors, and are preferred to other methods of measurement as they produce complete records of behavior during an observational period (Johnston & Pennypacker, 1993). However, as continuous recording methods require a dedicated observer, logistical and measurement challenges (e.g., observer fatigue, observer drift) often discourage their utilization in both research and applied settings (Gardenier, MacDonald, & Green, 2004). To mitigate logistical challenges associated with continuous recording methods, both partialinterval recording (PIR) and momentary time sampling (MTS) are commonly utilized in social skills training to monitor progress (e.g., Bellini, Akullian, & Hopf, 2007; Betz, Higbee, & Reagon, 2008; Hughes et al., 2013; Walton & Ingersoll, 2012), with PIR often favored in assessment of social Correspondence to: Keith C. Radley, Department of Psychology, University of Southern Mississippi, 118 College Drive #5025, Hattiesburg, MS 39406-001. E-mail: keith.radley@usm.edu 363 364 Radley et al. engagement skills (Merrell, 2001). PIR involves recording a behavior as having occurred if the target behavior is observed at any point during an interval. MTS, in which an occurrence is recorded if the target behavior occurs at a predetermined moment, is a sampling method that may also be applied to assess social engagement skills of children with ASD. Despite their widespread use, these sampling methods may yield inaccurate estimates of actual social engagement, as they do not allow for recording of every occurrence of target behaviors (Gardenier et al., 2004). Error in observation when using PIR and MTS methods is the result of mixed intervals— intervals in which the target behavior only occurs during a portion of the interval (Suen, Ary, & Covalt, 1991). Use of PIR results in all mixed intervals being recorded as an occurrence, which may produce inflated estimations of rate and duration of target behaviors. Using MTS, mixed intervals may either be coded as an occurrence or a nonoccurrence depending on whether or not the target behavior occurs during the predetermined moment, producing random error. As both PIR and MTS methods entail an inherent degree of inaccuracy, it is essential to know how these methods estimate actual levels of target behavior and thus, evaluation of treatment effects. Substantial research has been devoted to evaluation of the accuracy of PIR and MTS methods of recording behavior. Studies have demonstrated that PIR overestimates the actual duration of behavior (Gardenier et al., 2004; Powell, Martindale, Kulp, Martindale, & Bauman, 1977), which is particularly important as social skills training often targets duration of social engagement as a primary outcome of intervention. Additionally, PIR has been found to become increasingly inaccurate as the length of intervals is extended (Ary, 1984; Powell et al., 1977). Although MTS has been found to both overestimate and underestimate duration, margin of error is smaller than that of PIR (Gardenier et al., 2004; Powell, Martindale, & Kulp, 1975) and error remains minimal as interval length is increased (Kearns, Edwards, & Tingstrom, 1990). Additionally, Gardenier et al. (2004) suggested that MTS with relatively small intervals may serve as a substitute for continuous duration recording when evaluating duration-based events. Nevertheless, there will inevitably be error in recording procedures that are not continuous in nature. Error associated with sampling methods may inadvertently inflate or diminish baseline and intervention outcomes when evaluated using visual analysis methodology. As such, it is important to evaluate the extent to which error in sampling methods influences clinical judgment of response to intervention. Although the majority of studies comparing the accuracy of PIR and MTS have utilized data during a baseline condition only (e.g., Gardenier et al., 2004), Rapp, Colby, Vollmer, Roane, Lomas, and Britton (2007) compared PIR and MTS procedures to duration recording in reversal and multielement designs. Findings from Rapp et al. suggest that data paths generated through MTS and PIR were often similar to those created using duration recording, suggesting that similar conclusions of treatment effectiveness would be reached regardless of sampling procedure utilized. MeanyDaboul, Roscoe, Bourret, and Ahearn (2007) compared the accuracy of 10-second PIR and 10-second MTS procedures for recording clinically meaningful behaviors (i.e., stereotypy) during baseline and intervention conditions to determine whether recording method affected treatment analysis decisions (e.g., intervention evaluated as more or less effective due to choice of observational procedure). Although treatment effectiveness decisions using MTS were found to be slightly more likely to match continuous recording than PIR, overall results indicated that treatment analysis interpretations based on both sampling procedures often matched those based on continuous recording methods. However, due to small sample size, Meany-Daboul et al. (2007) suggest that results be interpreted with caution and replicated to increase generalizability Although Meany-Daboul and colleagues (2007) added to the observational procedures literature through their utilization of an expert panel to evaluate effects of diverse sampling procedures on treatment analysis, it should be noted that members of the expert panel were Psychology in the Schools DOI: 10.1002/pits Comparison of Momentary 365 required to make dichotomous yes/no decisions regarding presence of treatment effect. Utilization of dichotomous yes/no decisions regarding presence of treatment effect is dissimilar to real-world assessment, in which practitioners utilize visual analysis to draw conclusions regarding the degree of effectiveness instead of simple presence of treatment effect. Additionally, utilization of simple dichotomous yes/no decisions may have contributed to a lack of more significant discrepancies in treatment analysis decisions when comparing PIR, MTS, and continuous data recording. Instead, Brossart, Parker, Olson, and Mahadevan (2006) suggested that members of expert panels conducting treatment analyses be asked to determine degree of effectiveness, which more appropriately imitates decisions that practitioners make when evaluating intervention effects. In summary, the results of many studies suggest that MTS is a more accurate measure of behavior duration than PIR. Despite findings suggesting increased measurement error associated with PIR, studies suggest similarity in data paths (Rapp et al., 2007) and treatment effectiveness decisions (Meany-Daboul et al., 2007) regardless of observational procedure utilized. However, as Rapp et al. is limited by the utilization of graphs frequently depicting clear treatment effects, it is unknown whether findings would generalize to social skills assessment, in which intervention effects are often less clear (e.g., Bellini, Peters, Benner, & Hopf, 2007; Gresham, Cook, Crews, & Kern, 2004). Additionally, Meany-Daboul et al.’s (2007) finding that use of diverse observational procedures often results in similar treatment analysis decisions is limited in that use of dichotomous yes/no decisions regarding presence of treatment effect is dissimilar to real-world practice and does not follow expert panel guidelines proposed by Brossart et al. (2006). Taken together, it is hypothesized that limitations in these studies contributed to a lack of more discrepant treatment analysis decisions when diverse observational procedures are utilized. The purpose of the current study was to replicate and extend previous research comparing observational sampling and continuous duration methods to determine whether observational procedure affects treatment effectiveness decisions. First, the study serves as a replication of Gardenier et al. (2004), evaluating the accuracy of PIR and MTS at various interval lengths in comparison to continuous duration recording for assessing social engagement behaviors. Second, the study addresses limitations of Rapp et al. (2007) through utilization of data sets more similar to those found in real-world social skills training (i.e., AB graphs with less-than-clear intervention effects), imitating decisions that practitioners are likely to make regarding treatment effectiveness. The current study systematically replicates Meany-Daboul et al. (2007) by allowing expert panel members to make treatment analysis decisions regarding continuous levels of treatment effectiveness, hypothesized to enhance the ability to detect discrepant treatment analysis decisions. As procedures in the current study more closely approximate real-world practice in social skills training through utilization of graphs depicting less-than-clear intervention effects and by allowing expert panel members to make treatment analysis decisions regarding treatment effect, the current study provides an important evaluation of the effect of diverse observational procedures on practitioner interpretation of data. M ETHOD Participants and Setting Five elementary aged children who had been previously diagnosed with ASD, and who had been included in social skills training served as participants in the present study. Data for Ryan and Jeff (pseudonyms used throughout) have been published (see Participant 1 and Participant 2 in Radley, Jenson, Clark, & O’Neill, 2014), whereas data from other participants represent unpublished data. Ryan was a 6.9-year-old male with a diagnosis of autism. Jeff and Steve were 5.1-year-old males with a diagnosis of Asperger Syndrome. Nicole was a 4.9-year-old female with a diagnosis of Psychology in the Schools DOI: 10.1002/pits 366 Radley et al. autism. Bradley was a 4.8-year-old male with a diagnosis of Pervasive Developmental Disorder-Not Otherwise Specified. Social Skills Training Procedures Participants in the current study attended social skills groups utilizing the Superheroes Social Skills program (Jenson et al., 2011). Ryan and Jeff, two participants from a larger multiple-baseline design study evaluating the utility of parent-facilitated social skills training, attended an 8-week clinic-based social skills group. Steve, Nicole, and Bradley attended an 8-week school-based social skills group. Although both groups of participants received similar interventions, a detailed evaluation of treatment effects of the intervention is beyond the scope of the current study (see Radley et al., 2014). During the baseline condition, participants with ASD were observed during three 10-minute free play sessions. Free play sessions took place in the room in which social skills training took place. Three peers were present during free play with Ryan and Jeff, and two peers were included with Steve, Nicole, and Bradley. Peers were nominated for inclusion by teachers for demonstrating appropriate social skills, were consistent across free play sessions, and present during each session. During the free play sessions, toys were available for participants to play with in groups or solitarily. Using cameras placed in the free play setting, play sessions were filmed and used to code levels of social engagement of participants. During the intervention phase, participants in each group were introduced to the 10-minute free play session immediately following each social skills training session. The number of intervention phase observations of free play sessions ranged from 7 to 16. Assessment of Social Engagement Social engagement of participants with ASD was assessed using definitions adapted from the Playground Observation of Peer Engagement (POPE; Kasari, Rotheram-Fuller, Locke, & Guiou, 2009). The POPE allows for coding of six states of engagement: solitary, in which the child plays alone with no other peers within 3 feet; proximity, in which the child plays alone within 3 feet of a peer; onlooker, in which the child watches a peer or group of peers; parallel, in which the target child and a peer are engaged in a similar activity (e.g., playing with the same toy) yet fail to engage socially; and joint engagement, in which the target child and a peer engage in direct social behavior (e.g., giving, sharing, or showing an object, engaging in conversation, playing a game). Previous studies that have utilized the POPE have found the scheme to be coded with high levels of interobserver agreement (IOA) (e.g., Kasari, Rotheram-Fuller, Locke, & Gulsrud, 2012). Observer Training and IOA The primary researcher trained two school psychology doctoral students to accurately and reliably record states of engagement using duration recording, and PIR and MTS at 10- and 15second intervals. Prior to coding videos of participants, the primary researcher reviewed social engagement codes with the doctoral students. Doctoral students then viewed training videos that depicted elementary aged children in a free play setting, coded for social engagement using PIR, MTS, and continuous duration recording, and compared their results with the primary researcher. Doctoral students were required to obtain a minimum 85% IOA score with the primary researcher on two consecutive coding observations for each recording method prior to independent coding of videos of participants. Once the data collection process began, IOA was determined by having two raters independently score filmed free play sessions. Agreement for PIR and MTS was calculated by dividing the number of intervals of agreement by the number of intervals of agreements and disagreements Psychology in the Schools DOI: 10.1002/pits Comparison of Momentary 367 and multiplying by 100. Agreement for duration recording was calculated by dividing the smaller of the duration count by the larger count and multiplying by 100. Agreement was measured across each coding procedure for each participant during 36.3%, 33.3%, 30.0%, 31.5%, and 30.7% of filmed free play sessions for Ryan, Jeff, Steve, Nicole, and Bradley, respectively. Agreement across coding procedures averaged 91.0% (range = 85.0–98.3%) for Ryan, 89.6% (range = 80.0–96.6%) for Jeff, 91.9% (range = 87.5–98.3%) for Steve, 91.5% (range = 87.5–96.6%) for Nicole, and 94.6% (range = 87.5–96.6%) for Bradley. Coding of Social Engagement Video-recorded free play sessions were divided into both 60 (10-second) intervals and into 40 (15-second) intervals for both PIR and MTS observations. Two data sheets, one marking 60 intervals and the other marking 40 intervals, were constructed to code social engagement and compare coding procedures. Two school psychology doctoral students observed filmed free play sessions. Both observers had extensive experience in utilizing PIR, MTS, and continuous duration recording data in school and clinic settings. Videos of free play sessions were randomly assigned to observers using a random number generator. To obtain PIR data, the observers coded the occurrence of social engagement if it was observed at any point during the 10- or 15-second interval, with intervals designated by an audio cue. To obtain MTS data, observers noted the occurrence or nonoccurrence of social engagement when prompted by an audio cue at the end of the 10- or 15-second interval. PIR and MTS data were summarized and graphically displayed as percentage of intervals by dividing the number of intervals coded as containing social engagement by total number of intervals and multiplying by 100. Using a stopwatch, total duration of social engagement was coded and used as a standard for which to compare 10- and 15-second PIR and MTS coding procedures. Total duration data for each participant were summarized and presented graphically by dividing the total number of seconds of social engagement by the total number of seconds in an observation session (i.e., 600), and multiplying the result by 100. Use of five disparate coding procedures resulted in five separate graphs being made for each participant, with each graph depicting baseline and intervention data points. Specifically, for each participant: one graph displayed data when measured using PIR at 10-second intervals, one displayed data when measured using PIR at 15-second intervals, one displayed data when measured using MTS at 10-second intervals, one displayed data when measured using 15-second MTS, and one displayed total duration of social engagement. Expert Panel Review Four individuals served as members of an expert panel to visually inspect individual graphic displays of each participant’s percentage of social engagement, as measured by each observation procedure. All panel members had a Ph.D. in psychology, served as faculty members in an American Psychological Association-accredited and National Association of School Psychologists-approved school psychology doctoral program, and had extensive experience (i.e., mean of 20 years) utilizing visual analysis to evaluate treatment effects in both research and applied settings. Expert panel protocol was adapted from Meany-Daboul et al. (2007), with a minor modification to allow raters to assess degree of treatment effectiveness in addition to dichotomous yes/no decisions regarding presence of treatment effect. Members of the expert panel were interviewed by a doctoral student and shown each of the 25 graphs. The doctoral student provided background information to each member of the expert panel by explaining the x- and y-axis, the purpose of the intervention, and the operational definition of social engagement. In addition, the doctoral student informed panel members of the different coding Psychology in the Schools DOI: 10.1002/pits 368 Radley et al. procedures utilized, without divulging which graphs corresponded to which coding procedure (this information was also withheld from the doctoral student). Graphs were presented one by one in a randomized order across participants and recording methods. Once presented with a graph, the expert was asked, “Is there an effect?” Using visual analysis of changes in level, trend, and variability from baseline to intervention, expert panel members determined whether an effect was present (e.g., increasing trend and/or level of data points in intervention). If panel members responded “No,” they were asked the same question regarding the next graph. If panel members responded “Yes,” they were subsequently asked “What degree of treatment effectiveness is observed in this graph: ‘Small,’ ‘moderate,’ or ‘large?’” Panel members were allowed to review graphs previously presented, but no other information was provided. Panel members’ responses were recorded on a data sheet. Data Analysis To replicate findings of Gardenier et al. (2004), the extent to which estimates of social engagement produced by PIR and MTS procedures differed from actual duration of social engagement was calculated by determining the difference between actual duration and the duration estimated by either PIR or MTS procedures (Murphy & Goodall, 1980). Additionally, the percent difference relative to the duration of social engagement was calculated for PIR and MTS using procedures described by Gardenier et al. Relative percent difference was calculated by determining the difference between actual duration and estimated duration derived from either PIR or MTS, dividing the difference by the actual duration, and multiplying by 100%. Data obtained through expert review were assessed by comparing panelists’ interpretations of PIR and MTS data sets to their interpretations of total duration data sets. Identical responses for perceived dichotomous and continuous levels of treatment effectiveness were coded as agreements and displayed as a percentage. In addition, comparisons of expert panel interpretations of continuous levels of treatment effectiveness across coding procedures were analyzed using a one-way repeated measures analysis of variance (ANOVA) and subsequent planned pairwise comparisons. R ESULTS Relative Differences Figures 1–5 depict the level of social engagement for each participant as estimated by each coding procedure in comparison to continuous duration recording. Each figure demonstrates that both 10-second PIR and 15-second PIR estimates consistently overestimated the duration of social engagement across participants. Ten-second MTS and 15-second MTS also overestimated duration of social engagement, but less frequently, and to a much lesser extent than either PIR procedure. Both MTS procedures also underestimated the actual duration of social engagement. The extent to which estimates of social engagement produced by PIR and MTS differed from actual duration of social engagement was calculated in terms of measurement error. Estimates of social engagement produced by 15-second PIR and 10-second PIR coding procedures resulted in higher levels of measurement error (15.41% and 10.16%, respectively) relative to 15-second MTS and 10-second MTS (–0.90% and –0.21%, respectively). The average percent difference relative to the percent of actual duration of social engagement was calculated for each coding procedure. PIR estimates greatly overestimated duration of social engagement relative to MTS, which yielded both over- and underestimates of social engagement. Specifically, 15-second PIR produced estimates of social engagement that were an average of 140.51% greater than duration recordings of the same observation (range = 68.69–244.56%); 10-second PIR overestimated social engagement by an average of 93.78% across participants (range = 34.97–160.32%); 15-second MTS overestimated social engagement by an average of 0.58% across participants (range = –20.28–24.63%); and Psychology in the Schools DOI: 10.1002/pits Comparison of Momentary 369 FIGURE 1. Intervention data based on PIR, MTS, and duration for Ryan. Psychology in the Schools DOI: 10.1002/pits 370 Radley et al. FIGURE 2. Intervention data based on PIR, MTS, and duration for Jeff. Psychology in the Schools DOI: 10.1002/pits Comparison of Momentary 371 FIGURE 3. Intervention data based on PIR, MTS, and duration for Steve. Psychology in the Schools DOI: 10.1002/pits 372 Radley et al. FIGURE 4. Intervention data based on PIR, MTS, and duration for Nicole. Psychology in the Schools DOI: 10.1002/pits Comparison of Momentary 373 FIGURE 5. Intervention data based on PIR, MTS, and duration for Bradley. Psychology in the Schools DOI: 10.1002/pits 374 Radley et al. FIGURE 6. Correspondence data from the expert panel evaluating dichotomous assessment of effect presence and continuous levels of treatment effectiveness when comparing PIR and MTS to duration. 10-second MTS underestimated social engagement by an average of –2.70% across participants (range = –22.59–36.42). Expert Panel Review Results pertaining to the expert panel review are depicted in Figure 6. Across coding procedures, dichotomous interpretations of sampling procedures provided by panel members were generally consistent with their dichotomous interpretations of duration data sets (range = 93.33–100% across coding procedures). Panelists’ interpretations of continuous levels of treatment effectiveness for each respective coding procedure were less consistent with their interpretations of duration data sets. Specifically, duration interpretations agreed most often with those from 15-second MTS and 10-second MTS graphic displays of social engagement (85% agreement) than with 15-second PIR (55%) and 10-second PIR (60%) graphic displays. Psychology in the Schools DOI: 10.1002/pits Comparison of Momentary 375 A one-way repeated measures ANOVA with Greenhouse–Geisser correction was used to determine whether differences in responses between panelists’ analysis of continuous levels of treatment effectiveness across coding procedures varied significantly. Results of the ANOVA revealed a significant main effect for coding procedure (F(2.94, 0.254) = 11.57, p = .0005, ?2 = .18). Bonferroni post-hoc tests were conducted between treatment analyses of each coding procedure relative to panel members’ duration responses. Panel members’ duration interpretations differed significantly when compared to 15-second PIR (p = .016) and 10-second PIR interpretations (p = .035), but not when compared to either 15-second MTS (p = .828) or 10-second MTS interpretations (p = 1.0). Significant differences were also obtained between 15-second PIR and 15-second MTS responses (p = .004); 15-second PIR and 10-second MTS responses (p = .009); and 10-second PIR and 15-second MTS responses (p = .009). Due to modest sample size, a post-hoc power analysis was performed, revealing a within-groups comparison effect size of d = .82, indicating adequate statistical power (Cohen, 1988). D ISCUSSION The purpose of the current study was to systematically replicate and extend previous research comparing PIR and MTS to continuous duration recording across treatment conditions to assess accuracy of sampling procedures used for assessment in social skills training. Similar to Gardenier et al. (2004), the current study found that MTS both under- and overestimated actual duration of social engagement in children with ASD by small margins. PIR, however, was found to substantially overestimate actual duration of social engagement. As accuracy of social skills assessment is essential to intervention planning and progress monitoring (Bellini, 2008), these findings suggest that MTS should be utilized by practitioners and researchers evaluating interventions aimed at increasing duration of social engagement of children with ASD. The current study also tested the effect of different commonly utilized interval lengths on treatment analysis decisions. Results of the study suggest that utilization of 10-second MTS and 15-second MTS did not result in significantly differing treatment analysis decisions. However, 15-second PIR was found to be slightly more inaccurate than 10-second PIR in comparison to continuous duration recording. This finding supports previous research that has demonstrated that PIR becomes increasingly inaccurate as interval length increases (Ary, 1984), whereas MTS appears to remain relatively accurate as interval length increases (Kearns et al., 1990). Although the current study replicated procedures assessing the accuracy of sampling methods, the primary purpose was to compare interval methods to continuous recording across treatment conditions to determine whether method of recoding affects treatment effectiveness interpretation. Meany-Daboul et al. (2007) found MTS to be slightly more accurate in reflecting continuous data recording than PIR, with overall findings that both methods produced treatment analysis decisions that matched those based on continuous data recording. Similar to Meany-Daboul et al., the current study found dichotomous yes/no decisions to often match regardless of observational procedure utilized. However, the current study found that treatment interpretations based on MTS to be a significantly better representation of continuous data recording than PIR when utilizing real-world treatment analysis procedures in which practitioners determine degree of treatment effectiveness. Unlike Meany-Daboul et al., which found both treatment evaluations based on MTS and PIR to often match continuous data recording, the findings of the current study suggest that utilization of PIR results in intervention effects being judged as significantly larger than when MTS or continuous duration recording is utilized. Treatment interpretations based on MTS were not found to be significantly different than continuous duration recording. These findings confirmed the authors’ hypothesis that utilization of dichotomous yes/no decisions regarding treatment effectiveness limited the ability of Meany-Daboul et al. to detect treatment analysis discrepancies. Results of the current study indicate Psychology in the Schools DOI: 10.1002/pits 376 Radley et al. that practitioners are likely to overestimate treatment effectiveness when PIR is utilized to assess duration-based events. In contrast, MTS produced treatment effectiveness interpretations that more closely approximate interpretations of duration of social engagement. In addition, the variability in responses resulting from the use of either 10- or 15-second PIR observation procedures were large enough to yield significant response differences when directly compared to MTS procedures, suggesting that MTS is a more conservative method for social skills assessment. Although Rapp et al. (2007) found that sampling methods produce data paths that are similar to those of duration measures, it should be noted that participants included in their analysis demonstrated clear treatment effects. Meta-analyses of social skills interventions indicate that interventions often do not produce clear intervention effects, with effects ranging from minimal (Bellini, Peters, et al., 2007; Wilson & Lipsey, 2007) to substantial (Gresham et al., 2004). As effect of intervention was often less clear for participants included in the present study, results of the current study suggest that the effect of data recording procedure may be exacerbated for data that contain marginal or moderate treatment effects. Findings of the current study suggest t
