Technical Overview

Introduction

Multi-raters are plentiful in the marketplace. They come in three categories:

  1. Theory-based tools
  2. Context-specific tools
  3. Competency-based tools

Theory-based tools are developed on the idea of a theory about the nature of behavior. Context-specific tools are developed to provide information about a specific environment or organization. Competency-based tools assume that a set of competencies apply to a particular role or position.

Type 360® is a theory-based tool built on the theory of psychological types by Carl Jung. The theory proposes eight psychological energies are used to adapt to the demands of everyday life. Leader Agility 360™ is a competency-based tool that uses factorial research to identify eight dimensions of leader behavior.

The objective of this technical guide is to provide information on the development of the assessment and the psychometric values determined through rigorous scientific methods which have been recognized by professionals.

History

During the 1990s research articles reported a consistent relationship between personality variables and multi-rater behaviors. Building on this research, the author, Roger R. Pearman, Ed.D., completed a variety of studies using the database from the Center for Creative Leadership. From these findings, initial groundwork was created that would result in the development of Type 360® and Leader Agility 360™. Research began in 2002 to develop the tool which was eventually released in 2004, and after peer review, promptly won the 2004 Innovation Award by the Executive Human Resources Management Association.

Key benefit: research on this tool began in 1994 and continues to 2011 to ensure accuracy and developmental value.

Item Development

Research on creating items began by generating a list of all of the behaviors associated with personality type as defined by Carl Jung and eight leadership domains which were identified and published in HardWired Leadership (1998), by Roger R. Pearman. Through the use of a panel of experts on personality type, the initial list was reduced to 192 items for testing. Using groups of individuals in academic and corporate settings, an item analysis resulted in sixty-four items being selected which satisfied two measurement goals: behaviors that expressed personality patterns and were demonstrated aspects of leadership behavior.

It is fashionable for multi-raters to have common denominator items on which everyone receives a 3 or higher score. Scores lower than 3 are rare. With Type 360® and Leader Agility 360™, a deliberate effort was made to select items that reflected genuine behavior differences. It is unreasonable to assume that an individual is actively using all available psychological resources as suggested by psychological type; thus, having high scores on all eight dimensions is unlikely. By extension, those aspects of leader agility dimensions tied to personality resources that are infrequently used would also receive lower ratings. For this reason, this multi-rater will produce relatively large scoring ranges which would inform the user that the assessment is working. By calculating ranges of scores for each dimension, the developer was able to determine that the items were appropriately differentiating in populations.

Key benefit: items were carefully analyzed to meet criteria related to personality and leader-related behaviors.

Scales

Carl Jung proposed eight mental resources as necessary for managing everyday challenges. His model suggests that the eight resources are used in varying degrees by each person and that a pattern exists where individuals prefer to use a select number of these mental resources. This pattern produces the "type." Table 1 provides a bulleted list of key behaviors associated with each of the mental processes. The goal of Type 360® is to identify how these behaviors are displayed in an individual's daily behavior. Presumably all eight processes are of value and accessible to individuals, though each individual may be unaware of the degree of use. Type 360® allows for a discovery of what individuals observe and what an individual being rated might consider using in more deliberate ways. Jung noted that "we hardly know the affect of our behavior on others." With Type 360®, individuals can get a glimmer of how their behavior is perceived.

The selected items needed to serve a dual purpose of reflecting behaviors associated with the eight dimensions of leader effectiveness. Table 2 provides the definitions of the eight dimensions on which Leader Agility 360™ was developed. Essentially, the review of literature for the dissertation of the developer of the tool began a systematic process of critiquing all of the published research on leader effectiveness and performance available at that time. Ongoing literature reviews confirm the eight dimensions of leadership as the core aspects of effectiveness.

Key benefit: the scales that make up the two reports are based on well established and accepted aspects of behavior from either a personality or leader frame of reference.

Ratings

An observation by the developer of Type 360® and Leader Agility 360™ is that most multi-raters do not provide specific information about the behavior being rated in terms of frequency, perceived effectiveness, and importance. Having coached individuals for 30 years, developer Roger R. Pearman noted that sometimes a behavior needs to be increased or decreased in frequency, sometimes a behavior needs to be more fully developed to be more effective in its use, and sometimes frequency or effectiveness are not important. The individual being rated easily sees a score as a measure of "goodness" rather than an instruction about the behavior, which is in part the fault of other multi-rater designs.

In order to provide specific development guidance to the individual being rated, the developer decided to ask raters to identify how frequently each behavior was displayed, how effective the behavior is experienced, and the degree of importance the behavior has for the individual at this time. Research shows that a behavior may be frequent, effective, and important, and may be overdone. For this reason, raters are asked to indicate if the behavior is "overdone".

Key benefit: ratings are designed to provide instructive guidance in terms of the frequency, effectiveness, and importance of a given behavior.

Reports

Three reports are available from the sixty-four items that can be rated in terms of frequency, effectiveness, and importance.

Type 360® provides specific scores on the sixty-four behaviors and the type patterns these represent.

Leader Agility 360™ provides specific scores on the sixty-four behaviors and the leader dimensions they represent.

The Combined Report links the personality type and leader dimensions in a single report.

Each of the reports has the following sections:

In the case of Leader Agility 360™, Sections 8-14 relate to the eight dimensions of leader behavior defined in Table 2.

Key benefit: three reports are based on the same sixty-four items which provides maximum flexibility for the users of the tool

Reliability and Validity

Reliability in traditional assessment tools is determined by internal consistency — showing how responses to items are similar and test-retest correlations. With multi-rater tools, there is an assumption that if you get feedback, you are likely to adjust your behavior, thus a test-retest would show change rather than consistency. For this reason, multi-rater guides do not show test-retest analysis as this would be pointless. However, it is important to consider how raters respond to items that are grouped into each scale. If the items differentiate among important differences in an aspect of behavior, then a perfect correlation is not desired. For example, Communication has a number of aspects to it, and it is likely that an individual may be effective with some of the behaviors and not with others. Thus, when correlating among all of the items for Communication, a high correlation may not be good news for instrument construction as it would suggest the items are measuring the same thing. Table 3 provides you with a summary of inter-item correlations. In general, the correlations need to be greater than .60 but smaller than .80 to reflect consistency and differentiation. As shown in Table 3, the scales hold up quite well to these standards. Keep in mind that these analyses combine ratings from different groups from 2002 until February 2011.

Validity is traditionally shown by correlating scales with scales of other instruments with similar content, by contrasting group scores to show how one group is different from another, and statistical manipulations such as factor analyses. In this instance, it is important for individuals who know their type very well to identify whether the items and scales fit their understanding of their type. Groups of individuals who knew their type well and who had multiple sources of feedback about their leadership behavior were asked to rate the accuracy of their reports from Type 360® and Leader Agility 360™. Table 4 shows the percentage of agreement on each of the dimensions of the reports. Individuals were asked to read report statements and determine if the information was accurate. If not accurate, feedback was requested for use in potential modification of the report mechanism.

In a corporate-wide talent management study that stretched over four years, performance data and other multi-rater data were used to (a) correlated with Type 360® or Leader Agility 360™ data and (b) to do comparison studies of high and low-end performers to insure appropriate differentiation in the ratings. Table 3 shows the scoring ranges for each dimension.

Interpretation Validity

Regardless of statistical measures, the validity of the result of any assessment is how accurately it fits with the individual in his or her context. How do the behaviors work for or against the effectiveness and well being who was assessed? Because this is a developmental multi-rater, the interpreter should make every effort to frame the information as useful material to clarify both strengths and areas of potential growth.

Key benefit: statistical analyses reveal consistency within the items and valid performance of the scales as predicted across groups. With multi-rater tools, it is important to have interpretation validity from the learner's perspective and confirming evidence from independent factors, which is true for both Type 360® and Leader Agility 360™.

Development Tips and Applications

Type 360® and Leader Agility 360™ reports provide specific development tips for the personality aspects and leader dimensions rated by observers. The suggestions and tips are based on learning strategies reported by leaders as most useful in becoming more effective. In addition, specific chapters of books that have extensive development suggestions are provided.

For Type 360®, the development tips are extended into the book, YOU: BEING MORE EFFECTIVE IN YOUR MBTI® TYPE, with attention to the suggestions for those types who share a specific process (e.g. extraverted thinking) as the dominant aspect of the personality.

For Leader Agility 360™, the development tips are extended into the book, FYI: FOR YOUR IMPROVEMENT, with attention to those development chapters that share specific content with each leader agility dimension.

Administrators of the tool also have access to career development and learning by experience maps which correlate the needed learning experience with each of the personality or leader dimensions. Learning through experiences is the focus of these materials to aid interpretation and utilization of the reports.

Administration

Type 360® and Leader Agility 360™ are designed for easy administration, scoring, and utilization. Once an administrator is certified, an account is activated and completely controlled by the professional using the tool. The certified professional can set up groups, enter email addresses, print reports, send reminders to raters, and reschedule processes when convenient. With the Type 360® and Leader Agility 360™ administrative processes, there are no "middle men" to manage your client's experience or controlling when and how reports are generated. Both tools are administered from www.type360.com.

Contact us for more information.

(336) 774-0330 or info@type360.com