The Evaluation phase of the instructional design process

During the implementation phase of the ADDIE process some evaluation of the course was done while it was implemented. The evaluation phase done during this last phase of the ADDIE process consists in doing a final, overall evaluation, making some adjustments to the course and releasing the project. The evaluation process continues by doing some personal reflections about the project.

During the the evaluation phase the following actions will be taken:

• Conduct a final overall action of the course
• Collect data and summarize the responses
• Adjust the course design and some parts of the media
• Release the project
• Write a summary and some reflections about anything learned for doing the project

Some interesting questions to ask :

• Did the learners achieve the goals of instruction?
• What did they like best or least about the course?
• What kinds of responses did one get when they rated parts of the work?
• What did one learn from all of this?
• What did one like or didn't like about designing one's own course?

The answers to some of these questions allow to know if learning happens effectively or not. Evaluation provides feedback and helps to improve the course. It can also lead to refine or reconceptualize and redo some elements of the analysis part of the ADDIE process

During the evaluation process the following steps will be taken:

Step 1: Plan and conduct the project evaluation

One will need to find out if the course fits the learners. How to determine that? The actions that the learners do as a result of taking the course can help to evaluate the learner. If the learners understand the lessons and do all the requirements correctly it means that they are doing well. However if If they don't understand the course and don't show anything related to their understanding it means that they are not doing well.

A course evaluation enables to find out the following information:

• All parts of the course are appropriate and useful
• The course parts flow in a convenient sequence
• All topics are suitable to the course
• Learners gain skills and knowledge
• Learning and instruction activities are appropriate. effective and efficient

One makes a copy of this worksheet for each people who participates in the course. One asks 5 people who participate in the Beta test evaluation in the implementation phase to fill out the evaluation form and to return it.

The results of the evaluation form allow to:

• Study the opinons and results of the evaluation
• Allow to adjust the project design and some parts of the course media
• Release the project
• Evaluate one's performance about doing the project 

Step 2: Write project reflections

After learning the main phases of the ADDIE process and doing this project it is time to reflect. The following questions can help write about these reflections:

• What were the surprises?
• What can one remember about doing this project?
• What is the favorite part or achievement in doing the ADDIE process?
• What parts of the projects one desires to go back and do better if one has the chance? This worksheet can help to write and summarize the reflections about the project experiences

The implementation phase of the the Instructional Design process

The instructional design process is similar to the design process of a house. In the building process the engineer or builder does some undertakings concerning the plan of the ground on which the building will be erected and some analysis of the soil. He closes the lot and starts preparing the design of the house. Then he develops and implements the design. When he finishes building the house the owners are ready to move. The builder will be satisfied with his work when everything works properly in the house. Usually things don't work well and the owners have to call the builder in order to have other workers to fix whatever that doesn't work properly..

The same thing happens in the ADDIE process. After the development process things might not work the way they were designed. That's the reason why some experiments have to be done to make sure that things work the way they are supposed to. Learner problems and challenges may arise. Some bugs in the media may be found. In this case flexibility is recommended  because as mentioned before things might not work the way they were planned, This situation can cause to redesign and redevelop some elements.

The experimentation phase consists in asking a few students to use the project. You watch them to see if they can use the elements of the project. You ask them questions and write down their answers. You find out what they like about the learning experiences and what is confusing to them. You also ask them about what they like you to fix about the project.

During the implementation phase you will have to do the following steps after developing the project enough to try it out:

• Conduct an alpha test with a few people
• Make revisions of the course based on suggestions from the alpha test
• Conduct a beta test with 5 or people in your target audience

Some considerations to take into account during the implementation phase are listed below. They are what you would like to achieve with your project and the reason why you are doing this experiment or pilot test.

• All parts of the lesson are appropriate and useful
• The lesson or course flows in a satisfactory sequence
• All relevant topics are included
• Learners gain skills and knowledge
• Instructions and learning activities are appropriate, appealing, effective and efficient.

Step One: Conduct an Alpha Test. These instructions have to be followed:

• Find three or five people to try the project
• Give each of them a copy of this Evaluation form and have them return it to you
• Write a list of questions to ask them at the same time they are trying the project
• Write down the answers to these questions in order to make the revisions.

Observe the learners to find out :

• They move through the lessons and media materials
• What could be causing negative feelings or confusion
• What are the positive feelings and behavior about this project and the learning experience

Step Two: Make revisions

Use the notes from the previous step to plan changes about some specific areas of the project. Keep them for redoing later some parts of the project.

Use this worksheet  to make the revisions that will improve the project. The assignment consists in:

• Making lists of all the revisions to be made
• Do the improvements
• Try out the revised project before conducting the Beta test.

Step Three: Conduct the Beta Test

The beta test allows to make a few other changes that couldn't be done during the alpha test. Use this worksheet to list the changes necessary to make the project better. The assignment consists in doing the following checklist:

• Make the lists of changes base on feedback received.
• Do the improvements
• Write 3 things you learned from doing this project and describe them.

The third phase of the instructional design process: the Development

The purpose of the design phase of the ADDIE process is to give designers an idea of the structure of the course. During the development phase designers actually create the materials of the course. This process is similar to the building process of a house. One starts by creating a blueprint. Then during the executing phase one starts by acquiring and creating the materials necessary to build the house.

In the development phase instructional designers do the following things:

• Develop all the materials for a prototype lesson or multimedia presentation
• Create a file directory and store all files
• Create a prototype lesson 
• Check copyright permissions and list the credits
• Use a production checklist to double check that everything looks and functions properly.

This is done by doing the following steps:

Step 1 Acquire, Produce Media and Store the Project Files

In this phase one designs the web page(s), text, images, audio and video necessary to teach the lessons. One chooses the appropriate media for these lessons according to the design process. One can create a web page with text, images, video. etc. It could be a PowerPoint slide. a single video or a combination of the two. It could be a web page with a combination of text and links. Think about it. We live in an evolving world and one can't limit oneself to a single process. There are various learning theories and each of them brings different learning approaches. If you think that a video is the best learning approach you can do it. But if you want to experiment and innovate you need to try different ways. With the internet and the development of web technologies there are different ways of teaching and learning.

The first thing to do is to create a directory structure to store all the project files and that might not be necessary if one doesn't produce the media. An example of the directory structure is provided in this worksheet. The next thing to do is to go to the storyboard created during the design phase to decide the type of files that have to be produced. One can produce the text and media files or one can acquire them. In the design phase a storyboard or a sort of blueprint of the materials that need to be produced was created. The storyboard holds also a web page design.  The web page design is a plan of the different web pages for the project. One can go to the home page for example and decide what type of media and text that can be put on this page. One can acquire the text and media or produce them. The pages can contain text, video, images, etc. This worksheet is an example of a web page(s) design that can be used as a reference to decide what type of of media that can go on the page(s). The storyboard is where one decides what type of materials need to be created. For me there is no fixed storyboard design as there is no unique design for all the houses that were built and that can be built. Houses are built according to the needs and preferences of the owner. The instructional designer designs instruction according to the needs of the learner. He also brings his own preferences according to what he thinks is best for the learner.

Three types of designs were created during the the design process: task design, information design and user interface or graphic design. In the task design one asks the following question: What are the steps the learner has to follow in order to perform a particular task? In the information design one decides the amount of content and the type of design that will draw attention to important information. During the user interface design one decides the type of handout, web page or PowerPoint slides that will best convey the information. These media should be easy for the learner to use. Feedback for the prototype created is gathered from targeted learners and subject matter experts to see if the design reflects the goals of instruction. This prototype is preliminary and will be evaluated in the first implementation of instruction. It is recommended to plan for feedback and evaluation throughout the design process so that the evaluation phase can reflect the goals and objectives for the redesigned instruction and materials.

Step 2 Create a prototype lesson

The preceding paragraph dealt with the design process. It reminded of certain important considerations in that phase. During the development phase one creates the materials such as web page, handout, PowerPoint slides or online activities that will be used with the learners in the first implementation of the lesson (s).  Media and technology specialized knowledge may be required during the development phase if the web materials are created from scratch. However armed with the necessary information the instructional designer can create everything by himself. Some instructional platforms give instructions to anybody who wants to  teach without having a lot of knowledge in web development and instructional design. It means that to teach a course online one doesn't need to be an expert in web development, media technology and instructional design. These platforms have already their web developers, media technologists and instructional designers. When you want to teach you only have to follow the instructions to design a course. What is deceiving is like the instructor has to follow these instructions and doesn't have the maneuver to bring his own preferences.

Step 3 Check resource permissions and write the credits

In this phase one determines the amount of copyrighted work that will be used without copyright infringement. According to an instructional design course of Utah State University 10% of a song or article can be used to stay in the limits of non violation copyrights. One can always use open source and open educational resources, which is a better option. One writes the credits and includes them in an appropriate place of the media used.

Step 4 Use the Production checklist

The ADDIE production checklist has the following elements:
Management.
This section deals with the management of the project: hard drive file structure, backup system
Treatments.
In this section is found the theme/motif of the project, the project metaphor that supports the theme, the style, color scheme. Pacing and tone match learner's skills, views and goals for the project.
Media.
This section deals with the requirements for graphics, audio, video and text. Use a media log to track filenames of files created or acquired. An copy of the course can also be created in a CD or DVD for learners who don't have access to high speed internet.
Project authoring.
Elements considered are: formatting styles, typeface, unit consistency, menu consistency, learner activities, script, test items, caption, instruction or directions
Navigation
Considerations are given to buttons, links, site map, exit button
Language check
Spell check, language easy to understand, absence of implicit or explicit bias are considered
Copyright check
Citation of sources for graphics, sound, video and text. correctness of materials quoted, permission for use of copyrighted material, attribution of credits to contributors are taken into consideration
Double checking of the total project
Critique of pages, media log, accuracy of information, feedback from experts, debugging, proofreading, media testing, clarity of objectives are considered. Detailed information about the production checklist can be found in this workshet
      

The second phase of the instructional Design process (continued)

Step 2 Create a course map
A course map is the foundation of the instructional deign process. This is also a course map that allows to get a clearer picture of the course. It shows all the pieces of a courses. Some of the pieces can be: Overview, Introduction, Pre-test, Unit, Lesson, Post-test, Summary, Glossary, etc. Before laying out the course map one can do two things:
1. Lengthen the list of objectives by dividing topics in subtopics and writing each subtopic as an objective.
2. Refine the objectives by making them more specific
One can define a strategy component corresponding to a teaching strategy for each specific objective.
Step 3: Define a project style guide
In this step one decides of the theme, color scheme, metaphor, the look and feel of the project.
Step 4: Design lesson strategies, events and practice activities
In this step one creates the blueprint design document that shows the plan for the entire course. First the course is created in one's mind and one paper. Then one creates it in the computer. It is a question of putting together steps one and two in order to show the plan for the entire course.
Step 5: Evaluation strategy
In this phase one plans a way to measure how well the learners accomplish the goals and objectives of the course or lesson. One plans the lesson evaluation at the end of the lesson plan. The assessment can take several forms: a portfolio, a project, a test,  etc. A test is usually measured quantitatively. But other forms of evaluation can be used qualitatively. A portfolio and a project can be measured qualitatively where one can write a report on how well the learner reaches the goals and objectives of the lesson or course. The quantitative measure is very traditional and is still maintained as the most correct measure of evaluation. This measure is ridiculous because it requires the evaluation to be done by an instructor. But what's the evaluation measure of a self-learner and a lifelong learner? Out of school who evaluates the learning while one continues to learn everyday by experiences, reading and self-study?. I think in this case self-evaluation and society's evaluation are some measures of one's evaluation. For example one can use one's own judgement to determine if one has done well in a subject area or even in life. For example one can have good grades in college and not perform well in the exercise of one's profession. If you want to find a good physician you don't research his scores at the school of medicine but you research how well he has treated other patients. A good physician gets his patients by word of mouth. Once he treats them well  these patients refer other people to him. This medical doctor continues to learn by experiences, reading, attending seminars. The feedback that he gets from his patients and a sense of confidence in himself create a value for his competence that cannot be equated to a grade mark.
In a traditional measure of evaluation you give a test at the end of the lesson to see to what degree the learners consume the knowledge, skills and attitudes of what they are supposed to learn. When a student first take a course it is sometimes good to test the prerequisites to see how well they do. This can be pre-test. Depending on the results of the test one reviews the prerequisites before starting the course. All the tests of the actual course can be considered as post tests of this course. One test for a lesson can be considered as a pre-test for the second lesson when the lessons of the course are related.
I think a pretest and a posttest can be given for each lesson if the lessons of the  same course are not related.  

The second phase of the instructional design process: the design

Engineers create a blueprint before undertaking any construction project. Similarly the instructional designer creates a "design blueprint" or "course architecture" before physically creating a course. This phase is the second phase of the ADDIE process defined in the previous post. It is the planning phase of a course and requires brainstorming and creative thinking. The course design blueprint includes a course map, lesson event strategies and treatments, graphical user interface design, assessment plan and storyboard. The following steps are used in this phase:
Step 1: Refine Course Scope and Strategy
Step 2: Create Course Map
Step 3: Define Project Style Guide
Step 4: Design Lesson Strategies, Events and Practices
Step 5: Plan Evaluation Strategy
Step 6: Design Storyboard and layout screens.

Refine Course Scope and Strategy

In this step one focuses on the goal of the course. One reduces the content to some specific topics and some particular aspects of a topic or topics.

This step consists in two tasks: Reduction of topics and organization of  the overall teaching strategy.

Reduction of topics

This task consists in reducing the topics of the course and organizing them in sequence or putting them in a certain order. One can assimilate this organization to a table of contents. The content is set according to the objectives and tasks. The content is arranged according to some guidelines: general to specific, frequency (first skills used first), simple to complex or logical sequence.

Organization of the overall teaching strategy

This task contains the following components:
1. Have a  clear idea of the objectives of the course. One can check the objectives set previously
2. Define the strategies to help students learn. These strategies and learning events depend on the content topics and objectives. Not all events and strategies are introduced in a lesson. These strategies are divided in : pre-instructional events, instructional events. Strategies are set for each objective.

Pre-instructional events help focus on the overview of your topic. They set up all the events that follow.
a. Gain attention. Can be motivational. State a problem to solve. A dramatic statement or question. Show a real object, a model, a video.
Use actions (a demonstration, a song, quotation, a survey of learner’s opinions, some gimmick or unexpected event).
b. Tell or show what is expected (the objectives)
c. Remind them of things they already know about the new topic or task. Make a bridge to your message & the learners’experience.
Instructional events (decide how much to do with students and in what sequence). Decide the scope of each activity. Sequence the events.
a. Present new knowledge, skills, and attitudes. Work with only 3- 5 key messages or new steps in each lesson.
b. Provide guided learning (interactive), such as: practice, examples, nonexamples, embedded assessment, questions and answers,
analogies, demonstrations, presentations, discussions, interviews, games, role playing, scavenger hunts, and application exercises.
Give frequent examples to help students visualize what you mean. Give tips and summaries often.
Follow-through events
a.  Summarize learning points (use a dramatic statement that sums up your key messages)
b. Final lesson assessment (students will act and respond to quizzes, questions, problems to solve, projects and worksheets to complete

3. Break the instruction into manageable pieces incorporating objectives and strategies
4. Putting these pieces in a logical order as follows:

• Introduction, motivation, attention getting, review
• Statement of objectives
• Learning activity
• Summary
• Assessments. Assessments can also include non conventional forms: portfolio, projects, reflections, etc.

(To be continued)
   

Realization of the tasks of the Open PhD challenge in the learning contract

In continuation with a PhD I started a few years ago and for the fulfillment of the requirements of the OnPhD candidacy challenge at Peer-to-Peer University I am submitting the following links for each task:

 Learning history. Identification of knowledge, skills and abilities from prior work and experiences. The learning history is part of the learning contract I set in my OnPhD page at Wikiversity. The elements of the learning history are: formal learning, informal learning, background summary,  specialties, interests and professional experiences and educational portfolio.
 Domain of Study
Contributions
Research Methodology
Skills and Knowledge development
Personal learning networks
Supervision and endorsements

The instructional design process of a multimedia project

Instructional technology is a systematic way of designing, developing, implementing and evaluating the process of teaching and learning. It uses a combination of technical and human resources to make instruction more effective. It includes some elements of the instructional design, product development, interactive learning technologies, multimedia, distance education. library and information literacy. Each of these elements makes its contribution to the teaching and learning learning process. Instructional design is based on a process that makes learning more efficient. It consists in identifying the learner needs, setting some instructional goals and designing a teaching process that addresses those needs. The instructional method uses the ADDIE model that includes six phases: Analysis, Design Development, Implementation and Evaluation. The first step is developed below

Analysis. This phase comprises 8 steps:

1. Define the Purpose
2. Identify the Needs
3. Identify the Learner characteristics
4. Identify the Learning environment
5. Discover the Technical constraints
6. Write objectives
7. Decide the content
8. Prepare a brief project plan

Defining the purpose

In this phase the purpose of instruction is defined. The purpose includes the benefits that the learner will get from the instruction. Example the social studies teacher is teaching about the Industrial Revolution so that students understand how the Industrial Revolution influenced people at that time and how it impacts people's lives today.

Identifying the needs

In this phase is identified what the learner needs to learn or do in order to reach the goal of instruction. The learning prerequisites are identified. The gap between what students can actually do and what they need to do define the knowledge and skills that the students need to acquire. Example: In order to achieve a level of understanding of the past influence and the present impact of the Industrial Revolution the students need to describe some major aspects of the Industrial Revolution. Actually they lack the learning prerequisites about the Industrial Revolution or they vaguely remember the major aspects of it if they already learn about it. The focus of the Instruction at this point is to give the specific details about some major aspects of the industrial Revolution. This is what the students need to know at this point.

Learner characteristics

In this phase the designer learns about the learner's characteristics in order to match the instruction to the goals, attitude and abilities of the students. The essential information required is : age and grade, attitude, background experience, etc. the student's attitude determines if the students are interested or not about the subject. The background experience includes their current knowledge level about the topic to be taught. Other characteristics include the ability to learn through different media (audio, video, text), reading level, computer literacy, student's aptitude to learn the content, etc.

Learning environment

Some aspects of the learning environment include:
- location, facilities for instruction
- delivery platforms and medium
- teacher availability, involvement during instruction
- number of students involved
- learning done individually, in small groups or in class

Technical constraints

In this phase is analyzed the availability of equipment and program. For example: availability of computers, internet access, projector, etc. In cases where equipments and programs are not available, plans can be made to provide or substitute them.

Writing of objectives  

In this phase are written the objectives which are the actions the learner are supposed to perform. The objectives are also skills related to the topics that have to be taught. They are also based on course goals and what is needed for the students to accomplish.  Objectives need to be observable and measurable.

Content

In this phase are outlined the topics needed to be taught. They are based on the objectives stated.

Brief project plan

A brief project plan includes the project and main phases due dates., people involved and their role.