close

Se connecter

Se connecter avec OpenID

Annual Outcomes Assessment Report Workshop: …

IntégréTéléchargement
Annual Outcomes Assessment
Report Workshop:
Academic Programs
Office of Institutional Planning, Research and Effectiveness (OIPRE)
October 6 – 8, 2015
Drake LRC
1
Workshop Objectives
1.
2.
3.
4.
Provide a rationale for engaging in outcomes assessment
Describe the concept of institutional effectiveness
Outline steps in the outcomes assessment process
Distinguish between student learning outcomes and program
outcomes
5. Describe AAMU outcomes assessment process requirements
6. Outline annual outcomes assessment report timeline
2
Rationale for Assessment
• Outcomes assessment is a formative process
• Outcomes assessment is not an end; rather, it is a means for improving
student learning
• Motivation to engage in outcomes assessment
• Intrinsic value
• External accountability demands
3
Rationale for Assessment
• External accountability demands
• Outcomes assessment is a requirement of our regional accreditor – SACSCOC
• Without regional accreditation our students are not eligible for federal
student aid
• Intrinsic value
• Outcomes assessment allows us to determine how well we are living up to the
implicit promise we make to our students when they enroll and where
appropriate, make the necessary changes to curriculum and/or pedagogy to
ensure students reach the expected performance levels
4
Institutional Effectiveness
• Institutional effectiveness is the systematic, explicit, and documented
process of measuring institutional performance against mission in all
aspects of an institution (p. 16).
• A commitment to continuous improvement is at the heart of an on-going
planning and evaluation process. It is a continuous, cyclical process that is
participative, strategic, flexible, relevant and responsive (p. 16).
• This process [outcomes assessment] serves as the cornerstone of
institutional effectiveness. Institutional Effectiveness focuses on the design
and improvement of educational experiences to enhance student learning
(pp. 48 – 49).
Source: Southern Association of Colleges and Schools Commission on Colleges, Resource Manual for the Principles of Accreditation
5
University Mission Statement
Alabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to access and opportunity, of knowledge
and excellence in teaching, research and service is responsive to the needs of a diverse student population and the social and economic needs of the state and
region. The university offers contemporary baccalaureate, master’s educational specialist and doctoral level degrees to prepare student for careers in the arts,
sciences, business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a student-centered
educational environment for the emergence of scholars, scientist, leaders and critical thinkers, who are equipped to excel through their contribution and
leadership in a 21st century national and global society.
Strategic Priority 1:
Enhance AAMU’s
Distinctiveness
and
Competitiveness
PROGRAM 1
MISSION
Strategic Priority 2:
Strengthen
Structures,
Operations, and
Systems to
Promote and
Support
Organizational
Excellence and
Stewardship
PROGRAM 2
MISSION
Strategic Priority 3:
Strategic Priority 4:
Strategic Priority 5:
Strategic Priority 6:
Upgrade
University
Infrastructure and
Facilities
Secure the
University’s
Financial Future
Enhance the
University’s Image
and Recognition
Enhance university
Engagement
through Expanded
Outreach
PROGRAM 3
MISSION
PROGRAM 4
MISSION
PROGRAM 5
MISSION
6
Institutional Effectiveness
• Comprehensive Standard 3.3.1 (Institutional Effectiveness – unit level)
The institution 1) identifies expected outcomes*, 2) assesses the extent
to which it achieves these outcomes, and 3) provides evidence of
improvement based on analysis of results.
*For academic programs this includes student learning outcomes.
7
Assessment Steps
1.
2.
3.
4.
5.
6.
Develop program mission statement
Develop outcomes
Develop appropriate assessment measures
Develop desired performance targets
Analyze results
Develop and implement improvement plans
8
Assessment Steps
Mission
Statement
Step 1
Improvement
Plans
Outcomes
Step 2
Step 6
Assessment
Measures
Analysis
Step 5
Step 3
Performance
Targets
Step 4
Program Mission Statements
• Mission statements should…
•
•
•
•
•
•
Be specific, distinctive
Clearly state the purpose of the academic program
Identify the program’s stakeholders
State the values and principles that guide the curriculum/program goals
Speak to what the program is currently doing
Align with AAMU’s strategic goals
10
Mission Statement Template
• The mission of <<enter program name>> is to <<program’s primary
purpose(s)>> by providing <<program’s primary functions or
activities>>.
11
Sample Mission Statements
• “The International Baccalaureate® aims to develop inquiring,
knowledgeable and caring young people who help to create a better
and more peaceful world through intercultural understanding and
respect. To this end the organization works with schools,
governments and international organizations to develop challenging
programmes of international education and rigorous assessment.”
• “The mission of the Department of Physician Assistant Studies is to
educate patient-centered medical providers of the highest quality
who are critical thinkers, leaders, and life-long learners.”
12
Types of Outcomes
• Student learning outcomes (SLOs)
• The knowledge, skills or abilities students acquire through the successful
completion of a course or academic program (what do you expect your
graduates to know, think or be able to do upon graduation?)
• Unit of analysis = students enrolled in your program
• Program outcomes
• Illustrate what you want your program to do. These outcomes differ from
learning outcomes in that you discuss what it is that you want your program
to accomplish (Bresciani, 2001)
• Unit of analysis = the program
13
Outcomes and the Annual Assessment Report
• For the annual assessment report, academic programs must report
on:
• Three (3) student learning outcomes
• Two (2) program outcomes
14
Writing Student Learning Outcomes
• SLO statements should…
• Specify what students are expected to know, think or be able to do upon
program completion
• Address only one outcome per statement
• Be under the control/responsibility of the program
• Be measurable
• Yield non “Yes/no” results
• Be at the program level
15
Student Learning Outcomes Template
• Graduates of the <<program name>> will be able to/will demonstrate
<<skill, knowledge, value>>.
16
Student Learning Outcomes Examples
• Graduates of the B.S. Nursing
Program will have published at
least one article or presented a
paper at a conference.
• The program has no control over
a student’s paper or conference
presentation proposal being
accepted.
• Graduates of the B.S. Nursing
Program will be able to integrate
evidence, critical thinking and
patient preference in evaluating
options for care.
17
Student Learning Outcomes Examples
• Graduates in Food and Beverage
Management will appreciate the
value of customer service.
• Graduates in Food and Beverage
Management will demonstrate
their understanding of the value
of customer service.
• This outcome is not measurable.
18
Student Learning Outcomes Examples
• Graduates in Computer Systems will
be able to develop web pages, write
and debug programs in multiple
languages, understand basic computer
concepts, and be able to describe the
impact of technology on society.
• This outcome is bundled – the
statement addresses more than one
outcome.
• Graduates in Computer Systems
will be able to describe the
impact of technology on society.
• Graduates in Computer Systems
will be able to analyze technical
requirements and select the
most appropriate solution.
19
Student Learning Outcomes Examples
• Graduates in Social Work will
have successfully completed an
internship.
• Graduates in Social Work will be
able to apply the knowledge and
skills of a generalist social
worker.
• This is a yes/no outcome that
will not necessarily yield data for
improvement. Also, completion
of the internship is not under
the program’s control.
20
Program Outcome Examples
• Program Outcomes measure things other than direct student learning
• Enrollment levels, # of graduates, course completion rates
• Alumni satisfaction, employer satisfaction, satisfaction with advising
• Licensure pass rates
• Job/graduate school placement rates
• Faculty productivity, research grant funding levels
21
Outcomes and the Annual Assessment Report
• For the annual assessment report, academic programs must report
on:
• Three (3) student learning outcomes
• Two (2) program outcomes
22
Assessment Measures
• Now that you have developed sound outcomes, you must
• Indicate how the outcome will be assessed
23
Assessment Measures
• Assessment measures should be designed to yield meaningful,
actionable data that can lead to improvements.
24
Assessment Measures
• Should…
•
•
•
•
•
•
Be directly related to the outcome
Address all aspects of the outcome over time
Provide measurable results
Provide adequate data for analysis
Provide actionable data
Focus on student work at or near graduation
25
Assessment Measures
• Should…
• Outline a systematic way to assess the outcome
• Include the following details:
•
•
•
•
What will be collected
When the data will be collected
What assessment tool will be used
How will the data be collected
26
Choosing Assessment Measures
• Direct Assessments
• Are based on analysis of student work, performances or behaviors
• Can be
• Quantitative – correct/incorrect (e.g., multiple choice, true/false, problem sets, etc.)
• Qualitative – requires judgment on the part of the evaluator (e.g., term paper, short
answer/essay exam, project, performance, exhibits, etc.)
27
Choosing Assessment Measures
• Indirect Assessments
• Based on reported perceptions of student learning (e.g., student survey)
• Only appropriate as a secondary assessment method
28
Choosing Assessment Methods
• When assessing a Student Learning Outcome
• Multiple (at least two) assessment methods should be used where possible
• Direct assessments should be the primary assessment method utilized
29
Direct Assessment Measures
• Tests (completion, essay, matching, multiple choice, true-false)
• Embedded assignments and course activities
•
•
•
•
•
•
Classroom assessment activities
Capstone projects (e.g., senior theses)
Homework assignments
Class presentations
Poster presentations
Recitals/exhibitions
• Portfolios
30
Grades as Assessment Measures
• While grades are appropriate as class assessment measures, they are
not acceptable as program assessment measures.
• Why is that?
• Grades take the individual student as the unit of analysis, not the program
• May include factors not related to student performance (e.g., attendance,
participation)
• Don’t necessarily map neatly on to specific student learning outcomes and
therefore don’t lead to actionable data
31
Performance Targets
• Why do we need performance targets?
• Allows the program to identify desired performance levels
• Provide standard for determining success
• Puts data analysis in perspective
32
Performance Targets
• Targets should…
• Use component (not student or respondent) as unit of analysis
• Relate to outcome and assessment method
• Be specific (“x% of responses will be correct for each question” or “y% of
scores for each item on rubric or survey will be at least 4”
• Avoid vague words such as “most” or “majority”
• Avoid “all” or “100%” targets
33
Assessment Measures and Performance
Target Example
• We will analyze final exams from
the capstone course. 85% of our
majors will pass the final with a
score of at least 60%.
• This measure uses students
rather than learning
components as the unit of
analysis. As such, the measure
will not lead to data that can
inform improvement plans.
• We will analyze final exams from
the capstone course. 80% of the
responses for each of the ten
questions will be correct.
• In this revised measure and
target, each of the ten exam
items represents a learning
component.
34
Assessment Measures and Performance
Target Example
• We will keep a record of the
number of majors who
participate in internships. The
criterion for success is that all
majors will successfully
complete an internship.
• Example of a 100% target. Also
the measure – internship
completion – is not under the
program’s control.
• Each intern supervisor will be
asked to complete an evaluation
form that directly measures the
intern’s performance.
At least 85% of the scores for
each item on the evaluation form
will be “satisfactory” or “highly
satisfactory.”
35
Assessment Measures and Performance
Target Example
• A random sample of lab reports
from the sophomore lab courses
will be graded using a rubric. The
target is that 75% of lab reports
will be judged as “satisfactory” or
“excellent.”
• The assessment measure is
conducted early in the program
rather than near graduation. Also,
the performance target focuses on
the assignment grade rather than
the learning components evaluated
in the assignment.
• A random sample of lab reports
from the capstone lab course will
be graded using a rubric. The target
is that at least 75% of scores for
each item on the rubric will be
“satisfactory” or “excellent.”
36
Assessment Measures and Performance
Target Example
• A survey will be conducted of
graduating majors during the spring
semester. The average score on the
question related to “overall
satisfaction with the program” will be
at 3.5 out of a possible 5 points.
• This is an indirect measure (should
only be used in concert with a direct
measure). Target uses averages which
can be skewed by outlier scores.
Instead we should focus on the
proportion of students meeting the
performance criteria.
• A survey will be conducted of
graduating majors during the
spring semester. At least 80% of
the response for each question
on the survey will be “satisfied”
or “highly satisfied.”
37
Analysis of Results
• Here we move from planning the assessment to conducting it
• Provides data on which you can base changes
38
Analysis of Results
Analysis should…
• Provide detailed data (avoid “a majority” or “most”)
• Use specific numbers (no rounding)
• Avoid technical language
• Be clearly and succinctly presented
• Align with outcome and target
• Support actions taken later to foster improvement
39
Improvement Plans
• Should…
• Address outcomes where the performance target was not met
• State what future actions will take place in the future to address the outcome
40
Types of Improvement Plans
• Curriculum revision
• Additional assignment
• Course modification
• Instructional methodology
change
• Sequence change
• Technology update
• Assessment method change
• Target adjustment
• Faculty development
• Procedure, process change
41
Example for Discussion
Student Learning Outcome
Biology graduates will demonstrate an understanding of cell structure
and function.
42
Example for Discussion
Assessment Measure
• Students in BIOL 432 will submit a research project based on cell
structure and function. A faculty panel will use a rubric that has a
scale of 1 to 5 (where 5 is excellent) to assess the five components of
the project. Panel scores will be analyzed by each component across
all student work.
43
Example for Discussion
Target
• 80% of scores provided by the faculty panel for each of the five
components of the Biology Project Rubric will be 4 (very good) or 5
(excellent).
We will revisit this in a few slides.
44
Example for Discussion
Biology Project Results
Component
# Scores 4 or 5
Total # Scores
% Scores 4 or 5
Sources
68
72
94.4%
Methods
45
72
62.5%
Analysis
67
72
93.1%
Conclusion
48
72
66.7%
Organization
59
72
81.9%
45
Example for Discussion
Analysis
The 24 biology projects were assessed by a three-member faculty panel
for each of the five components of the rubric and for overall
performance. The target was not met for the Methods (62.5%) and
Conclusion (66.7%) components of the projects.
46
Example for Discussion
Improvement Plans
Fewer than 80% of scores provided by the faculty panel were 4 (very
good) or 5 (excellent) for the Methods and Conclusion components of
the Biology project. The department will develop a short video
detailing the process of writing a methods section and make this
available to BIOL 432 students through the course Blackboard site. A
new assignment requiring students to critique the conclusion sections
of sample lab reports will also be introduced in BIOL 432.
47
Annual Assessment Report Template
University Mission: Alabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to
access and opportunity, and dedicated to intellectual inquiry. The application of knowledge and excellence in teaching, research and service
is responsive to the needs of a diverse student population and the social and economic needs of the state and region. The University offers
contemporary baccalaureate, master’s, educational specialist and doctoral level degrees to prepare students for careers in the arts, sciences,
business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a studentcentered educational environment for the emergence of scholars, scientists, leaders and critical thinkers, who are equipped to excel through
their contributions and leadership in a 21st century national and global society.
Program Purpose: Step 1
Student Learning
Outcomes
Assessment Methods
Performance Criteria
Performance
Target
Assessment Results
Improvement Plans
Improvements
Implemented
Step 2
Step 3
Step 4
Step 5
Step 6
Step 6
Program Outcomes
Assessment Methods
Performance Criteria
Performance
Target
Assessment Results
Improvement Plans
Improvements
Implemented
Step 2
Step 3
Step 4
Step 5
Step 6
Step 6
48
Assessment Dates
• Assessment Plans Due
• Steps 1 – 4
• Mission, outcomes, measures and targets
• Friday, November 13, 2015
• Assessment Reports Due
• Steps 5 – 6
• Findings, improvement plans
• May ??, 2016 (specific date to be determined)
49
OIPRE Contact Information
Office of Institutional Planning, Research and Assessment
306 Patton Hall
James Walke, Director
james.walke@aamu.edu
256-372-8876
Dianne Bozeman, Assessment Coordinator
dianne.bozeman@aamu.edu 256-372-8644
50
Auteur
Документ
Catégorie
Без категории
Affichages
5
Taille du fichier
274 Кб
Étiquettes
1/--Pages
signaler