In 1999, ABET introduced significant reform to its accreditation process with the introduction of Engineering Criteria 2000 (EC2000). In 2005, ten years after the release of the American Society for Engineering Education’s (ASEE) Green Report calling for changes in engineering education leading to the attainment of critical skills and knowledge in engineering graduates, ABET commissioned its own early evaluation of EC2000’s impact on engineering graduates. As with many immature efforts in educational settings, more time and more information are needed to fully understand the impact of EC2000, and the subsequent EC2XXX1 criteria. A few basic themes have however emerged, as evidenced by annual professional meetings sponsored by ABET, the American Society for Engineering Education (ASEE), and virtually all of the engineering professional societies. First, the EC2XXX criteria on assessment and evaluation of program outcomes and program objectives remain problematic areas for programs seeking accreditation. Specifically, programs have difficulty understanding how to demonstrate a well-documented process for assessment and evaluation, with a clear path for revision based on the evaluation resultsError! Reference source not found.. Indeed, some now assert that the accreditation focus has evolved into one that is attentive to the assessment and evaluation process, rather than a focus on demonstrated outcomes. Second, clear distinctions between program objectives and program outcomes continue to exist. Third, non-uniformity among ABET program evaluators pose significant uncertainties for programs as they prepare for the accreditation evaluation experience.
Norfolk State University (NSU) launched programs in Electronics (EEN) and Optical (OEN) Engineering in 2003. The two programs reside within a Department of Engineering, which staffs eleven full-time faculty members. The department averages about 50 incoming freshmen students each year, and in Spring 2009, a total of 165 undergraduate students were enrolled in the two engineering programs. The department also offers Master of Science degrees in Electronics and Optical Engineering, and selected faculty within the department also support a doctoral degree program in Materials Science and Engineering. The doctoral program is a multidisciplinary program that is administered in cooperation with faculty that have been appointed to the Physics and Chemistry Departments.
The preparation for NSU’s first ABET evaluation presented several exciting challenges, including the challenge of establishing program missions, objectives, and outcomes for the two undergraduate degree programs. To facilitate these actions, the engineering faculty launched a Faculty Advance process. The Faculty Advance is a retreat-like activity that brings the faculty group together for one to two days in a remote (preferably off-campus) setting. The Advance typically includes invited presentations, small-group sessions where details of proposed ideas and activities can be refined, and full-group sessions where final actions can be debated and agreed upon.
The Advance sessions began in fall 2004 and the faculty decided in spring 2005 to launch a year-long planning effort to research best practices in outcomes assessment, and to identify a favorable plan to implement within the department. The faculty agreed to immediately adopt the existing ABET program outcomes in their entirety. It was agreed that adjustments to this decision could be explored in the future. As a first step, self-reportable student and faculty surveys would be used to measure attainment of the ABET defined outcomes. This was not seen as a long-term activity, but it would be done so that the data collection process could begin. At the conclusion of the research and preliminary assessment activities, a formal assessment and evaluation process was reviewed and ratified by the engineering faculty.
METHODOLOGY: THE NSU ASSESSMENT AND EVALUATION PLAN
The NSU Assessment and Evaluation Plan (AEP) is based on the principles and best practices that have been promoted through ABET’s regular suite of assessment workshops, and in the teaching and learning literature. Specifically, the NSU AEP is intended to enhance student learning with respect to the intended learning outcomes, it is conducted in unison with other evaluative processes within the department, and assessment results provide meaningful feedback to faculty as they continuously improve and refine the program offerings.
The current NSU AEP is summarized in the Table 1, below. Over time, the plan is expected to evolve as the faculty better understand critical time cycles for meaningful evaluation based upon collected data and practical experience. A more detailed examination of the plan for program outcomes assessment follows:
Overview of Assessment and Evaluation Plan
|Academic Functions||Engineering Courses||Students, faculty||annually|
|Engineering Program Outcomes (a-k)||Students, faculty, employers||two years|
|Engineering Program Mission and Objectives||Employers, faculty, advisory board, alumni||three years|
|NSU Core Competencies2||Students, faculty||annually,|
|Physical Infrastructure||Students, faculty, advisory board,||two years|
|Administration / Organizational Structures||faculty, advisory board||three years|
Beyond the general actions that are outlined in Table 1, the faculty must decide on a detailed plan for data collection that informs the task of evaluating the program objectives and the program outcomes. The focus of this paper is on the assessment and evaluation of program outcomes. As discussed above, our initial choice has been to adopt the ABET defined program outcomes, commonly referred to as the a through k criteria. The balance of this paper outlines our current activities related to this task.
More importantly, the detailed performance criteria associated with each of the learning outcomes will evolve based on student academic level (e.g. freshman, sophomore, junior, or senior standing), course content, and the resources available to the faculty. Hence, there is a critical need to fully engage the faculty as the AEP becomes fully operationalized into the standard practices for the department faculty.
Assessment of Engineering Program (Learning) Outcomes3
The NSU plan for assessment of program outcomes is inspired by work completed at the University Pittsburgh, and by the knowledge and experiences gained from ABET sponsored workshops on program and outcomes assessment. Specifically, the NSU engineering department has adopted an embedded assessment strategy that reflects a cognitive developmental approach to teaching and learning. Each semester, embedded assignments are generated in selected courses within the lower- and upper-division engineering curriculum. The assignments focus on specific skills needed to achieve the learning outcomes associated with ABET accreditation. The embedded assignments are intended to be transparent to students in that they are administered within the normal context of a course assignment and they require students to demonstrate knowledge and skills related to the course content. That is, a course in fiber optics understandably may ask students demonstrate their knowledge of step-index and graded-index waveguides. However, when faculty also ask students to design a system that guides uses step- or graded-index fibers for an outlined purpose, and to discuss the performance differences between the two designed systems, that assignment addresses domain specific learning outcomes, as well as professional outcomes. The professional outcomes might include: the ability to apply knowledge of math, science, and engineering; the ability to design a system or process; the ability to identify, formulate, and solve engineering problems; etc… This dual-perspective on the use of problem- and project-based assignments that typically comprise the engineering curriculum underscore an important principle in the NSU assessment strategy. Problem- and project-based assignments present an opportunity to combine standard educational activities with our assessment objectives. Moreover, we may refine the ways that we construct our problem- and project-based assignments to better support our commitment to help students achieve domain-specific knowledge and skills, as well as skills necessary for modern engineering practice as established by ABET.
The Curriculum-Instruction-Assessment Paradigm
Pellegrino has argued that three key elements exist in the education of the American workforce: curriculum, instruction, and assessment (CIA). These elements should be explicitly linked and any investigation of interdependencies between curriculum-instruction-assessment should begin with the exploration of students’ conceptions (including their misconceptions) of the subject matter. Programs that generate materials and approaches that integrate each of these aspects of the teaching and learning experience, build an implicit coherence into their educational approach. Analogous to the case of comparing total available energy from coherent versus incoherent optical signals, coherent links between curriculum (content), instruction (pedagogy), and assessment may provide wonderful opportunities for deep learning experiences for students. Figure 1 presents an overview of the three interdependent components of NSU design.
As suggested above, the NSU assessment plan also incorporates a developmental cognitive model into its design. The ground-breaking work highlighted in How People Learn, reveals that the process of moving from novice to expert skills and abilities takes place over a number of authentic learning experiences that build new knowledge from an existing context. The pre-existing context (beliefs) that students bring to any new experience significantly influences their interpretation of the new information. Moreover, even if preexisting beliefs are in fact false (wrong), students may be able to provide apparently correct interpretations of the new information without correcting the pre-existing beliefs. The new information will be based on a faulty foundation that becomes increasingly entrenched over time.
This model of developing new skills and knowledge based on an existing context is combined with a semiquantitative framework that characterizes the degree of complexity that students demonstrate as they apply their knowledge. The Bloom’s Learning Taxonomy defines complexity in learning that begins with factual knowledge and increases in comprehension, application, analysis, synthesis, and evaluation. Today, the Bloom’s Taxonomy has become increasingly popular among engineering educators as a framework that characterizes complexity and higher-order thinking. Moreover, Besterfield et al developed a comprehensive framework for application of Bloom’s Taxonomy in the assessment of ABET-defined program outcomes,.
Practical Guidelines for Application of Bloom’s Taxonomy in an Embedded Assessment Program
Building the CIA paradigm into the teaching and learning experience was more effectively achieved by an examination of the current curricular design using the CIA filter. In our examination, it was found that a number of practical guidelines could help in simplifying the actions needed to meet our assessment and evaluation objectives.
First, practically speaking, the ABET defined program outcomes can be divided into two categories: (1) skills and abilities associated with technical proficiency, and (2) skills and abilities that complement technical skills, termed professional skills. The technical outcomes are summarized in Table 3a and the professional outcomes are summarized in Table 3b.
ABET Defined Technical Outcomes
|a||an ability to apply knowledge of mathematics, science, and engineering|
|b||an ability to design and conduct experiments, as well as to analyze and interpret data|
|c||an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability|
|e||an ability to identify, formulate, and solve engineering problems|
|k||an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice|
ABET Defined Professional Outcomes
|d||an ability to function in multidisciplinary teams|
|f||an understanding of ethical and professional responsibilities|
|g||an ability to effectively communicate|
|h||broad education necessary to understand the impact of engineering solutions in a global/societal context|
|i||recognition of the need for and an ability to engage in lifelong learning|
|j||A knowledge of contemporary issues|
Given that this program was being developed from the ground up, the courses involved in our assessment plan only includes courses that are taught by engineering faculty. A curriculum mapping exercise was therefore conducted with two primary goals in mind. An initial review of courses was conducted during which faculty were asked to determine which ABET outcomes would likely be addressed through the normal activities of the course. It should be noted that at the time of this exercise, the Optical Engineering program was only in its second academic year of existence. Several courses from the junior and senior year had not yet been taught. Nevertheless, the curriculum mapping exercise would provide a snap shot of where we stand in light of the program outcomes required for accreditation. Not surprisingly, it was found that the technical outcomes, and outcomes a, b, and e in particular, were well emphasized in the curriculum. It was also found that the professional outcomes were underemphasized. While it was assumed that most of the professional skills would be addressed through general education courses that are taught outside of the Engineering Department, the faculty also agreed that opportunities to demonstrate the professional skills within the context of an engineering course would be favorable. In particular, assessment would be simplified if our data on professional outcomes could be collected from engineering course materials.
Second, once the inherent imbalance in the coverage of program outcomes had been summarized, the faculty assigned outcomes to courses to achieve balanced coverage of the outcomes across the full engineering curriculum. It was decided that our assigned distribution of outcomes should achieve balanced coverage of all technical and professional outcomes within the lower-division and upper-division curriculum. Hence, students would have opportunities to learn about and demonstrate their skills at an introductory and advanced stage of their academic experience.
It is expectation that students will achieve increasing ability to demonstrate higher-order cognition as they progress through the curriculum. It was therefore further decided that at the introductory level (lower-division), students should at a minimum demonstrate skills at the Bloom’s Taxonomy level of application, or higher. Accordingly, embedded assignments should challenge students to demonstrate application skills, as well as additional assignments for which analysis and evaluation may be demonstrated. For outcomes assessment purposes, application abilities are acceptable levels of demonstrated skills. Once students are into the upper-division, the minimum requirement for program outcomes is the Bloom’s Taxonomy level of analysis or higher (analysis being the minimum acceptable skill level). Effectively, faculty would be expected to help students achieve domain-specific content mastery and accreditation related skill mastery in their courses (see figure 2). The conventional grading structure would continue to be applied to the domain-specific content. However, the program outcomes would be evaluated using a different scale, and results of these evaluations of program outcomes would be used to support the case for accreditation.
Generating Embedded Assignments
Several critical tasks are needed to achieve an effective assessment program based on embedded assignments. These include development of specific criteria that clarify what specific abilities are expected, and developing rubrics that identify the differences between poor, average, and very good abilities. The criteria state what students must do, the rubrics help faculty recognize the difference between average and good skills and abilities.
The approach taken for generating the criteria mimics the models described by Felder and Besterfield et al. Each outcome is independently reviewed, and where appropriate, the steps needed to achieve the stated outcome are identified. Once these necessary steps are agreed upon, the faculty must decide what specifically is the distinction between acceptable and unacceptable demonstration of the skill. A well developed criteria and rubrics should also be helpful to faculty instructors as they develop assignments based on the criteria and rubrics.
The rubric may be highly definitive, prescriptive with little ambiguity. Or It may be general enough to leave space for interpretation. In our case, the fact that a given outcome could be assigned to several varied course types, has led us to opt for a loose definition of the rubric. Hence, we have taken the approach of looking first at whether or not students provide evidence that each task for a given outcome has been completed, then the quality of performance for each task is considered.
As an example, the ABET outcomes b and e are reviewed. Table 4 lists the example outcomes (b and e) and a set of tasks (referred to as criteria) needed to successfully achieve the selected outcomes. Faculty may review the criteria to recognize the intermediate skills they must help students demonstrate, and the faculty may also use the criteria as a guideline for generating appropriate assignments. The instructor might then review the criteria for outcome b and develop an assignment that requires generation of a design to measure a parameter, presentation of evidence that the prescribed plan was completed, presentation of the data gathered while executing the plan, and results that indicate the collected data has been useful in drawing the conclusions. In practice, we have observed that student may choose to leave out one or more procedures in their presentation materials. Students may also attempt to carry out each task, but may have difficulty completing each task – for example, not knowing how to generate a logarithmic graph, or not realizing that a logarithmic presentation would help the reader interpret the data.
The final step needed to complete the assignment is the development of rubrics that may be used to score the assignment. This may or may not be a separate activity from grading an assignment for content mastery. In our experience however, grading for content is rarely done simultaneously with the scoring for outcomes assessment. A simplistic rubric for outcomes b and e would be to establish a category for each task, then describe characteristics that reflect poor, acceptable, and good performance for each case. The fact that these outcomes include analysis, design, and evaluation components mean that these outcomes are compatible with efforts to determine higher-order cognitive skills in students. Finally, a requirement that students achieve acceptable or better scores in selected categories is considered reasonable evidence that their Bloom’s-based rating is sufficient.
Table 5 outlines the scoring rubric for outcome b. Note however, that there is a final level of detail that is dependent on the actual problem chosen. In a course on fiber-optics, a design plan would differ depending on the specifications given. Is the student designing with emphasis on size? Is the design emphasis on performance? Is the design emphasis on reliability? These factors make the scoring rubric domain specific and problem specific. Our approach has been to ask the faculty instructor to complete a detailed scoring rubric for their embedded assignment, but the rubric should be guided by the framework presented in Table 5. The table pairs shown for outcome b must be developed for all outcomes being evaluated. We have developed table pairs for outcomes that are evaluated to the level of Analysis and higher. These include outcomes b, c (design with realistic constraints), e (engineering problem solving), and f (ethics and professional responsibility). We specify below that selected outcomes are only evaluated to the Bloom’s level of Application.
After two iterations of our assessment activity, the faculty decided upon several application guidelines to help the process achieve greater efficiency and relevance.
It was decided that a few of the outcomes are particularly suited for specialized courses. For example, outcome b, an ability to design and conduct experiments, and to analyze and interpret data, is well-suited for laboratory courses. Similarly, outcome k, an ability to use techniques and tools necessary for modern engineering practice, is readily assessed in laboratory courses. The faculty therefore decided that laboratory courses would be used to support outcomes b and k. These outcomes are not assigned to lecture or other courses that are taught within the engineering department.
For selected outcomes, demonstration of the skill beyond Bloom’s Level of comprehension is not indicated. In some cases, the faculty have determined that students need only establish a knowledge-base to draw upon, and the appropriate standard for learning is the level of comprehension. The outcomes that fall into this category include outcomes a (application of math, science, and engineering), h (engineering solutions in a global/societal context), i (lifelong learning), j (contemporary issues), and k (tools for modern engineering practice).
It was quickly observed that embedded assignments that challenge students to the Bloom’s level of Application or higher require a minimum knowledge base. It is considered helpful to delay the administration of embedded assignments to the second half (or latter portion) of a school term so that adjustments to teaching styles, and an appropriate knowledge-base may be established. Other design or other assignments may be given early in the school term, but those used for program assessment will be delayed.
Inherent in the embedded assessment strategy is the need to have a fully engaged faculty. Developing embedded assignments for our program requires familiarity with the Bloom’s Taxonomy, a well developed criteria for each program outcome, and a well developed rubric to score the embedded assignments. The overall requirement therefore includes professional development, and a corporate approach to development and evaluation of the criteria and rubrics on the part of the faculty. Faculty are also asked to submit their embedded assignment to an assessment committee at an early point in the school term. The assignments are not screened, but the requirement does help facilitate a thoughtfully constructed assignment.
The Norfolk State University Engineering Department has implemented an ambitious program of outcomes assessment based on embedded assignments informed by the Bloom’s Learning Taxonomy and the How People Learn philosophy of cognitive development. The program design effectively combines the curriculum, instructional emphases, and the assessment strategy to achieve a coherent package for effective teaching and learning. The program also provides mechanisms for the development of a faculty community around the goal of outcomes assessment. Although a space of time is recommended before wide-spread proficiency can be expected, we have observed that faculty gain in confidence and efficiency as they implement the embedded assignments. Given the appropriate time and resource, the embedded approach may readily become a systematically supported feature of the curriculum, and the faculty expertise also becomes integrated – perhaps to the point of transparency.
This work has been supported in part by the National Science Foundation HBCU-UP Program through NSF0625105. The authors would like to thank the full engineering faculty for their contributions and support. The authors particularly thank Drs. Rasha Morsi, Frances, Williams, and Arlene Maclin, and Mr. Kennon Outlaw.
 ABET increments the official title of its criteria document annually. For example, the current version is EC2009. The designation EC2XXX is used as an indication of this annual change.
 The terms program outcome, educational outcome, and learning outcome are used interchangeably throughout this document
 Criteria listed for outcome b have been taken from model developed by Felder
Pellegrino, J. W., (2006). Rethinking and Redesigning Curriculum, Instruction and Assessment: What Contemporary Research and Theory Suggests. A Paper Commissioned by the National Center on Education and the Economy for the New Commission on the Skills of the American Workforce.Pellegrino, J. W., (2006). Rethinking and Redesigning Curriculum, Instruction and Assessment: What Contemporary Research and Theory Suggests. A Paper Commissioned by the National Center on Education and the Economy for the New Commission on the Skills of the American Workforce.