“Small World” Does Not Mean Small View
Time to think globally and embrace outcome-based
assessment
Marilee J. Bresciani, assistant vice president for
institutional
assessment and visiting associate professor at Texas
A&M University, speaks
with Higher Education Digest
When you travel
metaphorically with Disneyland in the ride “It’s a small world,” you’re transplanted
to a place of harmony. Smiling, happy people greet you in their native
costumes. You see the differences in their dress and physical features, but in
their song the melody is as one. The world comes together and isn’t so big when
we all join in song: that’s the message you take home as you leave the
Disneyland ride.
The same thing is happening in higher education. The world
is getting even smaller now that Europe has the edge in educating international
students. Where we once ranked at the top, now U.S. universities are less
dominant than most would have predicted, according to the World University Rankings put out by the London Times in November
2004.
Recently, Higher Education Digest interviewed Marilee J.
Bresciani, assistant vice president for institutional assessment and visiting associate professor
at Texas A&M University, to get her perspective
on how institutions in the U.S. can regain a position of leadership through
implementation of outcome-based assessment.
Higher Education
Digest: How is Europe competing against
U.S. institutions of higher education?
|
White Paper
Continuous Quality Improvement and
Organizational Effectiveness
by Kendell Rice, Ph.D., Datatel, Inc.
Quality is a concept that is hard to define but easy to recognize. We know it
when we see it. Quality is frequently defined as "a high standard of excellence." Like many accolades used in higher education,
quality and excellence are often simply hyperbole. But, in the new lexicon of management tools, quality really means something, and
increasing numbers of colleges and universities are embracing the philosophy of Continuous Quality Improvement.
|
 |
this
complimentary white paper. |
|
Marilee
J. Bresciani: Institutions of higher education
have all competed for the same students for a long time. Homeland Security, as you can imagine, has
made it more difficult for international students to study in the U.S. Therefore, it makes other country’s higher education systems more attractive.
Europe’s current efforts to compete
against U.S. institutions of higher education stem from both the European Erasmus Mundus Programme
and the Bologna Declaration of 1999. Specifically, the Erasmus Mundus Programme
is an effort to promote quality in higher education with a
distinct European added value to appeal to students within the European Union
and beyond its borders. This program also encourages co-operation between EU
and third-country institutions to improve accessibility and enhance the profile
and visibility of higher education in the EU.
With the
Bologna Declaration of 1999 students can now take a class at the University of
Cologne and other courses at Oxford and the Sorbonne all offered by different
higher education institutions. Mobility of this sort is fostered through a system
of credits known as the ECTS system. These courses are standardized across
various European countries.
HED: What can we do to gain the
competitive edge?
Bresciani:
U.S.
institutions of higher education can attempt to continue to compete on reputation.
However, as the London Times suggests our reputation has less meaning in the
international context than it once did.
Yet, we could compete more effectively and combat reputation concerns
for quality of undergraduate and graduate learning by presenting evidence of
our educational excellence. We can
demonstrate genuine educational quality by using outcome-based assessment,
which has met with much success in Australia, South Africa and other
countries.
Think of
it in this way, rather than understanding whether your son or daughter or niece
or nephew will get a good quality education based on an institution’s faculty
to student ratio (e.g., easy to identify indicators), wouldn’t it be great to
actually know what your student would be expected to learn in any specific
major and what the level of learning within that major was for students who
came before? And if you are wondering
why the level of student learning is what it is, you could look deeper to see
what the plans for improving the learning are at each institution. You may then realize a progressive
institution striving to achieve more may be better suited to your student than
one that had already established itself.
This kind of data could transcend legacy reputations and allow
institutions to compete for international and national students based on the
facts and the end results of learning.
HED: Why are institutions
concerned about assessment?
Bresciani: Institutional reputation, branding concerns, some grant funding
agencies, regional and professional accreditation requirements and some ranking
algorithms require outcomes-based assessment results.
Since 1985, there has been an increased
call for accountability in the United States, which has centered on an
institution’s ability to demonstrate that they are achieving what they say they
are (e.g., outcomes). Additional requests
for accountability based on outcomes could be driven by an institution’s governing
board, or state government.
The federal
government also appears to be interested in demonstrating a certain level of
performance as well. The “No Child Left Behind Act (NCLB) which pertains to
K-12 education seems to have influenced conversations in the higher education
arena in regards to the reauthorizations acts."
Even though NCLB gave authority to the states, as the U.S.A. has no
national Ministry of Higher Education, the inclination to create an easy to
administer and comparable testing methodology has appeared to sweep the nation
creating “apples to oranges” comparisons.
While some of the data from these tests have undoubtedly improved the
quality of education for some institutions’ students, the ability to perform
the types of improvements that are necessary for every type of learner remains
unseen. Rather, it appears standardized
tests are driving our pedagogical and learning outcomes conversations. Think for a moment if this type of mentality
were to successfully creep into higher education. We would have out-dated curriculum in higher learning as tests
could not be standardized efficiently to keep up with emerging research in
certain fields. We would further fall
behind as a nation.
Institutions
can accommodate and more importantly inform national standards through
outcomes-based assessment where multiple methods are designed and used by the
disciplinary, multi-disciplinary and co-curricular experts.
HED: What is the difference
between test- and outcome-based assessment?
Bresciani: Tests are a method to evaluate
students’ knowledge of a particular course within a curriculum or an entire
curriculum. A test could be unrelated to outcomes and not in alignment with the
course or curriculum. If this is the case, then it’s obviously not a good
measurement of what the student has been taught and consequently would not demonstrate
what the student learned.
A test
based on regurgitation/memorization doesn’t reflect the whole picture or whether
the students can use what they’ve learned through cognitive thought processes.
For example, how do we know if an engineer or a literary scholar has the
requisite knowledge to perform well in his or her chosen fields? Instead of
being dependent on a standardized test as a mode of assessment, find out what
experts (e.g., scholars and practitioners) in these fields think makes a good
engineer and literary scholar. Design the curriculum and the corresponding assessment
of student learning around the experts’ criterion and desired outcomes.
With
outcome-based assessment the institution has the opportunity to decide what the
intended outcome is before a course is taught. When the class is completed, the
student’s performance is measured based on authentic assessment methods and the
pre-set outcome.
Tests are
not an ineffective way to evaluate certain levels of student learning. However, good practice research tells us
never to use one research method upon which to base all of our
conclusions. We know this, so why do we
do it when it comes to education? Why
do we determine, for instance, the quality of an entire K-12 school district
based on one test? When we do this, we
are not only practicing poor research, we are demonstrating poor quality
decision-making.
HED: How does an institution
implement outcome-based assessment?
Bresciani: Your
organization can plan for and implement an outcome-based assessment process
through the following 10 steps.
-
Acknowledge why you are engaging
in outcomes-based assessment
-
Define assessment and your shared
conceptual understanding of assessment
-
Define a common language
-
Acknowledge your political
environment
-
Identify everyone’s roles and
responsibilities in the assessment process
-
Articulate your assessment expectation(s)
-
Identify what you have already
done that is evaluation/assessment/planning
-
Identify easy-to-access resources
(data, assessment tools, people, technology, etc.)
-
Establish a support system
-
Just dive in — assessment is an iterative process
(Bresciani,
2003a)
As you
develop a continuous and on-going
assessment process keep these key assessment cycle questions in mind.
-
What are we trying to do and why?
-
What is my program supposed to
accomplish?
-
How well are we doing it?
-
How do we know we are accomplishing our
goals?
-
How do we use this information to
improve or celebrate successes?
-
Do the improvements we make work?
(Bresciani, 2002,
p.1)
HED: Is outcome-based assessment
widespread?
Bresciani: The practice is not readily
pervasive in the U.S., primarily because faculty members have not been taught
how to implement outcomes-based assessment. We tend to hire faculty because
they are experts in their fields. When
we hire these experts, we don’t typically pay full on attention to whether they
know how to share that expertise. Thus,
faculty have to be educated on how to effectively share what they know and then
educated about how to efficiently and effectively evaluate whether their
students have learned what they taught.
Most of
our institutions’ award systems don’t recognize that we need to teach faculty ways
to evaluate student learning. Often the faculty does not have release time to
do this. Rather, the institutional focus is on rewarding research productivity,
and promotions are often based on the volume of research an individual faculty
member produces.
Research
is not disconnected from student learning, yet our means of promoting faculty
does disconnect these. We don’t reward
faculty for collaborating on curriculum design and the creation of authentic
means to evaluate how their students learn about the applications of the
faculty member’s research. We tend to
only reward whether the individual faculty member published the latest
research.
Quality
teaching and learning often requires faculty collaboration and the application
of their research to their classroom.
Where is the collaboration on curriculum design and student learning
evaluation line and how much does it count for in faculty member’s individual
promotion and tenure portfolios?
HED: How do you get faculty
focused on assessment?
Bresciani: Regardless
of whether faculty are individually rewarded for their attention to student
learning most faculty care about it because they are innately intellectually
curious about whether what they do actually works. If faculty is provided with instruction on how to evaluate
student learning and how to use the information to improve what they are doing,
we may actually see more of this occurring.
While we wouldn’t expect students to learn math algorithms without
teaching what they are first, at most institutions, we do expect faculty to meaningfully
engage in assessment, even though we never teach them how to successfully do
it.
It is also
time to rethink the model for instructional costs and better plan around or
within it. For instance, K-12 instructional costs include (or at least they use
to): faculty development, curriculum development, and assessment of student
learning. In higher education, our instructional costs models do not account
for pedagogical differences, nor do they account for the inclusion of authentic
assessment. Furthermore we don’t budget
for collaborative meetings of faculty to design outcomes-based general
education programs, let alone build in time for them to discuss learning
outcomes and means of assessment for each degree major. We certainly don’t account for the time to
document the results and decisions made.
Here are
some strategies for implementing outcome-based assessment:
-
Appeal
to faculty members’ values.
-
Respect
faculty autonomy.
-
Remain
flexible and adaptable.
-
Work
collaboratively toward assessment.
-
Empower
faculty to conduct outcome-based assessment at their level, present the results
and try to get other faculty members on board.
-
Educate
higher level administrators so they see the effective use of results.
It’s not just
a small world; it’s a world of opportunity. Let’s work together to make the
U.S. a leader in quality education again by embracing outcome-based assessment.
More on
this topic can be found in Dr. Bresciani’s forthcoming book from Stylus
Publications.
 |
Dr.
Marilee J. Bresciani has been in higher education administration and faculty positions for
over 18 years. In those positions, she has conducted enrollment management
research, quantitative and qualitative institutional research, course-embedded
assessment, and academic and administrative program assessment. As assistant
vice president for institutional assessment at Texas A&M University, Dr. Bresciani enjoys assisting several units and departments with the development
of their assessment plans, the identification and development of assessment tools
and methods, and the use of their data for continuous improvement of student
learning and development. Also in this role, she coordinates the university
processes for assessment of programs and student learning and development as
well as coordinating effective support for the embedding of those processes in
faculty and staff’s day-to-day doing. In collaboration with others, Dr.
Bresciani conducts outcomes assessment for specific programs and courses as
well.
Formerly
the director of assessment at North Carolina State University, Dr. Bresciani
has been invited to present on assessment nationally and internationally, has a
number of invited publications, is a leading author of a book on assessing
student learning and development, is authoring two other books on program assessment
and general education assessment, has developed and delivered several courses
on assessment of student learning, and serves on the editorial board of the
NASPA Journal. Dr. Bresciani is a reviewer for the Australian Quality Assurance
Agency and is also a managing partner in an international assessment and
enrollment
management consulting firm.
In
her spare time, Dr. Bresciani enjoys technical rock climbing, backpacking,
kayaking, scuba diving, gardening, music and cooking for friends, family and her
two yellow labs. She holds a Ph.D. in administration, curriculum, and
instruction from the University of Nebraska and a master's degree in arts in teaching
from Hastings College.
