2.1.1

The Curriculum-Instruction-Assessment Paradigm

Pellegrino^[11] has argued that three key elements exist in the education of the American workforce: curriculum, instruction, and assessment (CIA). These elements should be explicitly linked and any investigation of interdependencies between curriculum-instruction-assessment should begin with the exploration of students’ conceptions (including their misconceptions) of the subject matter. Programs that generate materials and approaches that integrate each of these aspects of the teaching and learning experience, build an implicit coherence into their educational approach. Analogous to the case of comparing total available energy from coherent versus incoherent optical signals, coherent links between curriculum (content), instruction (pedagogy), and assessment may provide wonderful opportunities for deep learning experiences for students. Figure 1 presents an overview of the three interdependent components of NSU design.

Figure 1.

Overview of Curriculum – Instruction – Assessment Paradigm

As suggested above, the NSU assessment plan also incorporates a developmental cognitive model into its design. The ground-breaking work highlighted in How People Learn^[10], reveals that the process of moving from novice to expert skills and abilities takes place over a number of authentic learning experiences that build new knowledge from an existing context. The pre-existing context (beliefs) that students bring to any new experience significantly influences their interpretation of the new information. Moreover, even if preexisting beliefs are in fact false (wrong), students may be able to provide apparently correct interpretations of the new information without correcting the pre-existing beliefs. The new information will be based on a faulty foundation that becomes increasingly entrenched over time.

This model of developing new skills and knowledge based on an existing context is combined with a semiquantitative framework that characterizes the degree of complexity that students demonstrate as they apply their knowledge. The Bloom’s Learning Taxonomy defines complexity in learning that begins with factual knowledge and increases in comprehension, application, analysis, synthesis, and evaluation^[3]. Today, the Bloom’s Taxonomy has become increasingly popular among engineering educators as a framework that characterizes complexity and higher-order thinking. Moreover, Besterfield et al developed a comprehensive framework for application of Bloom’s Taxonomy in the assessment of ABET-defined program outcomes^[4],[5].

Table 2.

Bloom’s Taxonomy

2.1.2

Practical Guidelines for Application of Bloom’s Taxonomy in an Embedded Assessment Program

Building the CIA paradigm into the teaching and learning experience was more effectively achieved by an examination of the current curricular design using the CIA filter. In our examination, it was found that a number of practical guidelines could help in simplifying the actions needed to meet our assessment and evaluation objectives.

First, practically speaking, the ABET defined program outcomes can be divided into two categories: (1) skills and abilities associated with technical proficiency, and (2) skills and abilities that complement technical skills, termed professional skills. The technical outcomes are summarized in Table 3a and the professional outcomes are summarized in Table 3b.

Table 3a.

ABET Defined Technical Outcomes

Table 3b.

ABET Defined Professional Outcomes

Given that this program was being developed from the ground up, the courses involved in our assessment plan only includes courses that are taught by engineering faculty. A curriculum mapping exercise was therefore conducted with two primary goals in mind. An initial review of courses was conducted during which faculty were asked to determine which ABET outcomes would likely be addressed through the normal activities of the course. It should be noted that at the time of this exercise, the Optical Engineering program was only in its second academic year of existence. Several courses from the junior and senior year had not yet been taught. Nevertheless, the curriculum mapping exercise would provide a snap shot of where we stand in light of the program outcomes required for accreditation. Not surprisingly, it was found that the technical outcomes, and outcomes a, b, and e in particular, were well emphasized in the curriculum. It was also found that the professional outcomes were underemphasized. While it was assumed that most of the professional skills would be addressed through general education courses that are taught outside of the Engineering Department, the faculty also agreed that opportunities to demonstrate the professional skills within the context of an engineering course would be favorable. In particular, assessment would be simplified if our data on professional outcomes could be collected from engineering course materials.

Second, once the inherent imbalance in the coverage of program outcomes had been summarized, the faculty assigned outcomes to courses to achieve balanced coverage of the outcomes across the full engineering curriculum. It was decided that our assigned distribution of outcomes should achieve balanced coverage of all technical and professional outcomes within the lower-division and upper-division curriculum. Hence, students would have opportunities to learn about and demonstrate their skills at an introductory and advanced stage of their academic experience.

It is expectation that students will achieve increasing ability to demonstrate higher-order cognition as they progress through the curriculum. It was therefore further decided that at the introductory level (lower-division), students should at a minimum demonstrate skills at the Bloom’s Taxonomy level of application, or higher. Accordingly, embedded assignments should challenge students to demonstrate application skills, as well as additional assignments for which analysis and evaluation may be demonstrated. For outcomes assessment purposes, application abilities are acceptable levels of demonstrated skills. Once students are into the upper-division, the minimum requirement for program outcomes is the Bloom’s Taxonomy level of analysis or higher (analysis being the minimum acceptable skill level). Effectively, faculty would be expected to help students achieve domain-specific content mastery and accreditation related skill mastery in their courses (see figure 2). The conventional grading structure would continue to be applied to the domain-specific content. However, the program outcomes would be evaluated using a different scale, and results of these evaluations of program outcomes would be used to support the case for accreditation.

Figure 2.

Dual course objectives include domain-specific content and program outcome skills

2.1.3

Generating Embedded Assignments

Several critical tasks are needed to achieve an effective assessment program based on embedded assignments. These include development of specific criteria that clarify what specific abilities are expected, and developing rubrics that identify the differences between poor, average, and very good abilities. The criteria state what students must do, the rubrics help faculty recognize the difference between average and good skills and abilities.

The approach taken for generating the criteria mimics the models described by Felder^[1] and Besterfield et al^[5]. Each outcome is independently reviewed, and where appropriate, the steps needed to achieve the stated outcome are identified. Once these necessary steps are agreed upon, the faculty must decide what specifically is the distinction between acceptable and unacceptable demonstration of the skill. A well developed criteria and rubrics should also be helpful to faculty instructors as they develop assignments based on the criteria and rubrics.

The rubric may be highly definitive, prescriptive with little ambiguity. Or It may be general enough to leave space for interpretation. In our case, the fact that a given outcome could be assigned to several varied course types, has led us to opt for a loose definition of the rubric. Hence, we have taken the approach of looking first at whether or not students provide evidence that each task for a given outcome has been completed, then the quality of performance for each task is considered.

As an example, the ABET outcomes b and e are reviewed. Table 4 lists the example outcomes (b and e) and a set of tasks (referred to as criteria) needed to successfully achieve the selected outcomes. Faculty may review the criteria to recognize the intermediate skills they must help students demonstrate, and the faculty may also use the criteria as a guideline for generating appropriate assignments. The instructor might then review the criteria for outcome b and develop an assignment that requires generation of a design to measure a parameter, presentation of evidence that the prescribed plan was completed, presentation of the data gathered while executing the plan, and results that indicate the collected data has been useful in drawing the conclusions. In practice, we have observed that student may choose to leave out one or more procedures in their presentation materials. Students may also attempt to carry out each task, but may have difficulty completing each task – for example, not knowing how to generate a logarithmic graph, or not realizing that a logarithmic presentation would help the reader interpret the data.

Table 4.

Outcome Criteria

The final step needed to complete the assignment is the development of rubrics that may be used to score the assignment. This may or may not be a separate activity from grading an assignment for content mastery. In our experience however, grading for content is rarely done simultaneously with the scoring for outcomes assessment. A simplistic rubric for outcomes b and e would be to establish a category for each task, then describe characteristics that reflect poor, acceptable, and good performance for each case. The fact that these outcomes include analysis, design, and evaluation components mean that these outcomes are compatible with efforts to determine higher-order cognitive skills in students. Finally, a requirement that students achieve acceptable or better scores in selected categories is considered reasonable evidence that their Bloom’s-based rating is sufficient.

Table 5 outlines the scoring rubric for outcome b. Note however, that there is a final level of detail that is dependent on the actual problem chosen. In a course on fiber-optics, a design plan would differ depending on the specifications given. Is the student designing with emphasis on size? Is the design emphasis on performance? Is the design emphasis on reliability? These factors make the scoring rubric domain specific and problem specific. Our approach has been to ask the faculty instructor to complete a detailed scoring rubric for their embedded assignment, but the rubric should be guided by the framework presented in Table 5. The table pairs shown for outcome b must be developed for all outcomes being evaluated. We have developed table pairs for outcomes that are evaluated to the level of Analysis and higher. These include outcomes b, c (design with realistic constraints), e (engineering problem solving), and f (ethics and professional responsibility). We specify below that selected outcomes are only evaluated to the Bloom’s level of Application.

Table 5a.

Outcome b Scoring Rubric

Table 5b.

Outcome b Scoring Rubric

2.1.4

Application Notes

After two iterations of our assessment activity, the faculty decided upon several application guidelines to help the process achieve greater efficiency and relevance.

It was decided that a few of the outcomes are particularly suited for specialized courses. For example, outcome b, an ability to design and conduct experiments, and to analyze and interpret data, is well-suited for laboratory courses. Similarly, outcome k, an ability to use techniques and tools necessary for modern engineering practice, is readily assessed in laboratory courses. The faculty therefore decided that laboratory courses would be used to support outcomes b and k. These outcomes are not assigned to lecture or other courses that are taught within the engineering department.

For selected outcomes, demonstration of the skill beyond Bloom’s Level of comprehension is not indicated. In some cases, the faculty have determined that students need only establish a knowledge-base to draw upon, and the appropriate standard for learning is the level of comprehension. The outcomes that fall into this category include outcomes a (application of math, science, and engineering), h (engineering solutions in a global/societal context), i (lifelong learning), j (contemporary issues), and k (tools for modern engineering practice).

It was quickly observed that embedded assignments that challenge students to the Bloom’s level of Application or higher require a minimum knowledge base. It is considered helpful to delay the administration of embedded assignments to the second half (or latter portion) of a school term so that adjustments to teaching styles, and an appropriate knowledge-base may be established. Other design or other assignments may be given early in the school term, but those used for program assessment will be delayed.

Inherent in the embedded assessment strategy is the need to have a fully engaged faculty. Developing embedded assignments for our program requires familiarity with the Bloom’s Taxonomy, a well developed criteria for each program outcome, and a well developed rubric to score the embedded assignments. The overall requirement therefore includes professional development, and a corporate approach to development and evaluation of the criteria and rubrics on the part of the faculty. Faculty are also asked to submit their embedded assignment to an assessment committee at an early point in the school term. The assignments are not screened, but the requirement does help facilitate a thoughtfully constructed assignment.