The Evaluation phase of the instructional design process

During the implementation phase of the ADDIE process some evaluation of the course was done while it was implemented. The evaluation phase done during this last phase of the ADDIE process consists in doing a final, overall evaluation, making some adjustments to the course and releasing the project. The evaluation process continues by doing some personal reflections about the project.

During the the evaluation phase the following actions will be taken:

• Conduct a final overall action of the course
• Collect data and summarize the responses
• Adjust the course design and some parts of the media
• Release the project
• Write a summary and some reflections about anything learned for doing the project

Some interesting questions to ask :

• Did the learners achieve the goals of instruction?
• What did they like best or least about the course?
• What kinds of responses did one get when they rated parts of the work?
• What did one learn from all of this?
• What did one like or didn't like about designing one's own course?

The answers to some of these questions allow to know if learning happens effectively or not. Evaluation provides feedback and helps to improve the course. It can also lead to refine or reconceptualize and redo some elements of the analysis part of the ADDIE process

During the evaluation process the following steps will be taken:

Step 1: Plan and conduct the project evaluation

One will need to find out if the course fits the learners. How to determine that? The actions that the learners do as a result of taking the course can help to evaluate the learner. If the learners understand the lessons and do all the requirements correctly it means that they are doing well. However if If they don't understand the course and don't show anything related to their understanding it means that they are not doing well.

A course evaluation enables to find out the following information:

• All parts of the course are appropriate and useful
• The course parts flow in a convenient sequence
• All topics are suitable to the course
• Learners gain skills and knowledge
• Learning and instruction activities are appropriate. effective and efficient

One makes a copy of this worksheet for each people who participates in the course. One asks 5 people who participate in the Beta test evaluation in the implementation phase to fill out the evaluation form and to return it.

The results of the evaluation form allow to:

• Study the opinons and results of the evaluation
• Allow to adjust the project design and some parts of the course media
• Release the project
• Evaluate one's performance about doing the project 

Step 2: Write project reflections

After learning the main phases of the ADDIE process and doing this project it is time to reflect. The following questions can help write about these reflections:

• What were the surprises?
• What can one remember about doing this project?
• What is the favorite part or achievement in doing the ADDIE process?
• What parts of the projects one desires to go back and do better if one has the chance? This worksheet can help to write and summarize the reflections about the project experiences

The implementation phase of the the Instructional Design process

The instructional design process is similar to the design process of a house. In the building process the engineer or builder does some undertakings concerning the plan of the ground on which the building will be erected and some analysis of the soil. He closes the lot and starts preparing the design of the house. Then he develops and implements the design. When he finishes building the house the owners are ready to move. The builder will be satisfied with his work when everything works properly in the house. Usually things don't work well and the owners have to call the builder in order to have other workers to fix whatever that doesn't work properly..

The same thing happens in the ADDIE process. After the development process things might not work the way they were designed. That's the reason why some experiments have to be done to make sure that things work the way they are supposed to. Learner problems and challenges may arise. Some bugs in the media may be found. In this case flexibility is recommended  because as mentioned before things might not work the way they were planned, This situation can cause to redesign and redevelop some elements.

The experimentation phase consists in asking a few students to use the project. You watch them to see if they can use the elements of the project. You ask them questions and write down their answers. You find out what they like about the learning experiences and what is confusing to them. You also ask them about what they like you to fix about the project.

During the implementation phase you will have to do the following steps after developing the project enough to try it out:

• Conduct an alpha test with a few people
• Make revisions of the course based on suggestions from the alpha test
• Conduct a beta test with 5 or people in your target audience

Some considerations to take into account during the implementation phase are listed below. They are what you would like to achieve with your project and the reason why you are doing this experiment or pilot test.

• All parts of the lesson are appropriate and useful
• The lesson or course flows in a satisfactory sequence
• All relevant topics are included
• Learners gain skills and knowledge
• Instructions and learning activities are appropriate, appealing, effective and efficient.

Step One: Conduct an Alpha Test. These instructions have to be followed:

• Find three or five people to try the project
• Give each of them a copy of this Evaluation form and have them return it to you
• Write a list of questions to ask them at the same time they are trying the project
• Write down the answers to these questions in order to make the revisions.

Observe the learners to find out :

• They move through the lessons and media materials
• What could be causing negative feelings or confusion
• What are the positive feelings and behavior about this project and the learning experience

Step Two: Make revisions

Use the notes from the previous step to plan changes about some specific areas of the project. Keep them for redoing later some parts of the project.

Use this worksheet  to make the revisions that will improve the project. The assignment consists in:

• Making lists of all the revisions to be made
• Do the improvements
• Try out the revised project before conducting the Beta test.

Step Three: Conduct the Beta Test

The beta test allows to make a few other changes that couldn't be done during the alpha test. Use this worksheet to list the changes necessary to make the project better. The assignment consists in doing the following checklist:

• Make the lists of changes base on feedback received.
• Do the improvements
• Write 3 things you learned from doing this project and describe them.

The third phase of the instructional design process: the Development

The purpose of the design phase of the ADDIE process is to give designers an idea of the structure of the course. During the development phase designers actually create the materials of the course. This process is similar to the building process of a house. One starts by creating a blueprint. Then during the executing phase one starts by acquiring and creating the materials necessary to build the house.

In the development phase instructional designers do the following things:

• Develop all the materials for a prototype lesson or multimedia presentation
• Create a file directory and store all files
• Create a prototype lesson 
• Check copyright permissions and list the credits
• Use a production checklist to double check that everything looks and functions properly.

This is done by doing the following steps:

Step 1 Acquire, Produce Media and Store the Project Files

In this phase one designs the web page(s), text, images, audio and video necessary to teach the lessons. One chooses the appropriate media for these lessons according to the design process. One can create a web page with text, images, video. etc. It could be a PowerPoint slide. a single video or a combination of the two. It could be a web page with a combination of text and links. Think about it. We live in an evolving world and one can't limit oneself to a single process. There are various learning theories and each of them brings different learning approaches. If you think that a video is the best learning approach you can do it. But if you want to experiment and innovate you need to try different ways. With the internet and the development of web technologies there are different ways of teaching and learning.

The first thing to do is to create a directory structure to store all the project files and that might not be necessary if one doesn't produce the media. An example of the directory structure is provided in this worksheet. The next thing to do is to go to the storyboard created during the design phase to decide the type of files that have to be produced. One can produce the text and media files or one can acquire them. In the design phase a storyboard or a sort of blueprint of the materials that need to be produced was created. The storyboard holds also a web page design.  The web page design is a plan of the different web pages for the project. One can go to the home page for example and decide what type of media and text that can be put on this page. One can acquire the text and media or produce them. The pages can contain text, video, images, etc. This worksheet is an example of a web page(s) design that can be used as a reference to decide what type of of media that can go on the page(s). The storyboard is where one decides what type of materials need to be created. For me there is no fixed storyboard design as there is no unique design for all the houses that were built and that can be built. Houses are built according to the needs and preferences of the owner. The instructional designer designs instruction according to the needs of the learner. He also brings his own preferences according to what he thinks is best for the learner.

Three types of designs were created during the the design process: task design, information design and user interface or graphic design. In the task design one asks the following question: What are the steps the learner has to follow in order to perform a particular task? In the information design one decides the amount of content and the type of design that will draw attention to important information. During the user interface design one decides the type of handout, web page or PowerPoint slides that will best convey the information. These media should be easy for the learner to use. Feedback for the prototype created is gathered from targeted learners and subject matter experts to see if the design reflects the goals of instruction. This prototype is preliminary and will be evaluated in the first implementation of instruction. It is recommended to plan for feedback and evaluation throughout the design process so that the evaluation phase can reflect the goals and objectives for the redesigned instruction and materials.

Step 2 Create a prototype lesson

The preceding paragraph dealt with the design process. It reminded of certain important considerations in that phase. During the development phase one creates the materials such as web page, handout, PowerPoint slides or online activities that will be used with the learners in the first implementation of the lesson (s).  Media and technology specialized knowledge may be required during the development phase if the web materials are created from scratch. However armed with the necessary information the instructional designer can create everything by himself. Some instructional platforms give instructions to anybody who wants to  teach without having a lot of knowledge in web development and instructional design. It means that to teach a course online one doesn't need to be an expert in web development, media technology and instructional design. These platforms have already their web developers, media technologists and instructional designers. When you want to teach you only have to follow the instructions to design a course. What is deceiving is like the instructor has to follow these instructions and doesn't have the maneuver to bring his own preferences.

Step 3 Check resource permissions and write the credits

In this phase one determines the amount of copyrighted work that will be used without copyright infringement. According to an instructional design course of Utah State University 10% of a song or article can be used to stay in the limits of non violation copyrights. One can always use open source and open educational resources, which is a better option. One writes the credits and includes them in an appropriate place of the media used.

Step 4 Use the Production checklist

The ADDIE production checklist has the following elements:
Management.
This section deals with the management of the project: hard drive file structure, backup system
Treatments.
In this section is found the theme/motif of the project, the project metaphor that supports the theme, the style, color scheme. Pacing and tone match learner's skills, views and goals for the project.
Media.
This section deals with the requirements for graphics, audio, video and text. Use a media log to track filenames of files created or acquired. An copy of the course can also be created in a CD or DVD for learners who don't have access to high speed internet.
Project authoring.
Elements considered are: formatting styles, typeface, unit consistency, menu consistency, learner activities, script, test items, caption, instruction or directions
Navigation
Considerations are given to buttons, links, site map, exit button
Language check
Spell check, language easy to understand, absence of implicit or explicit bias are considered
Copyright check
Citation of sources for graphics, sound, video and text. correctness of materials quoted, permission for use of copyrighted material, attribution of credits to contributors are taken into consideration
Double checking of the total project
Critique of pages, media log, accuracy of information, feedback from experts, debugging, proofreading, media testing, clarity of objectives are considered. Detailed information about the production checklist can be found in this workshet
      

The instructional design process of a multimedia project

Instructional technology is a systematic way of designing, developing, implementing and evaluating the process of teaching and learning. It uses a combination of technical and human resources to make instruction more effective. It includes some elements of the instructional design, product development, interactive learning technologies, multimedia, distance education. library and information literacy. Each of these elements makes its contribution to the teaching and learning learning process. Instructional design is based on a process that makes learning more efficient. It consists in identifying the learner needs, setting some instructional goals and designing a teaching process that addresses those needs. The instructional method uses the ADDIE model that includes six phases: Analysis, Design Development, Implementation and Evaluation. The first step is developed below

Analysis. This phase comprises 8 steps:

1. Define the Purpose
2. Identify the Needs
3. Identify the Learner characteristics
4. Identify the Learning environment
5. Discover the Technical constraints
6. Write objectives
7. Decide the content
8. Prepare a brief project plan

Defining the purpose

In this phase the purpose of instruction is defined. The purpose includes the benefits that the learner will get from the instruction. Example the social studies teacher is teaching about the Industrial Revolution so that students understand how the Industrial Revolution influenced people at that time and how it impacts people's lives today.

Identifying the needs

In this phase is identified what the learner needs to learn or do in order to reach the goal of instruction. The learning prerequisites are identified. The gap between what students can actually do and what they need to do define the knowledge and skills that the students need to acquire. Example: In order to achieve a level of understanding of the past influence and the present impact of the Industrial Revolution the students need to describe some major aspects of the Industrial Revolution. Actually they lack the learning prerequisites about the Industrial Revolution or they vaguely remember the major aspects of it if they already learn about it. The focus of the Instruction at this point is to give the specific details about some major aspects of the industrial Revolution. This is what the students need to know at this point.

Learner characteristics

In this phase the designer learns about the learner's characteristics in order to match the instruction to the goals, attitude and abilities of the students. The essential information required is : age and grade, attitude, background experience, etc. the student's attitude determines if the students are interested or not about the subject. The background experience includes their current knowledge level about the topic to be taught. Other characteristics include the ability to learn through different media (audio, video, text), reading level, computer literacy, student's aptitude to learn the content, etc.

Learning environment

Some aspects of the learning environment include:
- location, facilities for instruction
- delivery platforms and medium
- teacher availability, involvement during instruction
- number of students involved
- learning done individually, in small groups or in class

Technical constraints

In this phase is analyzed the availability of equipment and program. For example: availability of computers, internet access, projector, etc. In cases where equipments and programs are not available, plans can be made to provide or substitute them.

Writing of objectives  

In this phase are written the objectives which are the actions the learner are supposed to perform. The objectives are also skills related to the topics that have to be taught. They are also based on course goals and what is needed for the students to accomplish.  Objectives need to be observable and measurable.

Content

In this phase are outlined the topics needed to be taught. They are based on the objectives stated.

Brief project plan

A brief project plan includes the project and main phases due dates., people involved and their role. 

The new role of the instructional designer

The role of the instructional designer should now focus on new ways of learning made possible by different technology devices and ways of using them.

"When designing for mLearning we need to put ourselves in the shoes of the user who is walking around with a powerful tablet and smart phone. That person today has different expectations that are informed by social media and You Tube. They expect instantaneous access to what they want, where they are, on the device they have in their hand, And learning is not a one-way push anymore"
(When the teacher is learner, do we need instructional designer?).

"The camera, voice recorder and collaboration features of the devices drive user generated content that enable every user to be both teacher and student" (When the teacher is learner do we need instructional designer?).

The ADDIE process way of creating courses doesn't work anymore. The role of the elearning designer should be to facilitate learning and collaboration instead of dictating it. How can the learning designer create learning experiences that fit the needs of the learner where he is? First the content should should be created in granular and modular ways so that it can be delivered in mobile devices. Second it should not be tied to a certain platform because the latter can change. Third it should be created and managed in XML where the presentation is different from the content. A granular content is stored in the cloud for everyone to access it. An app is the delivery channel.

The designer should facilitate user-generated content. His goal ought to design an entire learning experience not just a piece of content. He has to know his audience in order to crate engaging learning experiences. He creates entire programs combining instruction, social collaboration and self-study.

Social commenting, rating and analytics substitute for need analysis. "The users tell what they want, what's working and where the gaps are. Analytics will tell what's being used and how effective it is" (When the learner is the teacher, do we need instructional designer?)

More on E-learning