Amira G. Seif El Din, Faten A. Kamel, Randa M. Youssef and Hoda Y. Atta
Abstract: A six-day workshop was conducted to develop trainers in mental health by providing physicians participating in the programme with the necessary educational knowledge and skills. A significant improvement was observed in the participants' knowledge and skills after the workshop. A significant correlation was found between the increase in knowledge and that of performance. The increase was affected neither by the years of work experience nor by the postgraduate degree held. This observed benefit and the satisfaction expressed by participants encourage the replication of such a programme in order to develop qualified health personnel capable of expanding mental health services for children.
Evaluation d'un programme de formation pédagogique pour la formation de formateurs dans le domaine de la santé mentale infantile à Alexandrie
RESUME: Un séminaire-atelier d'une durée de six jours a été organisé pour former des formateurs en santé mentale en donnant aux médecins participant à ce programme les connaissances et aptitudes pédagogiques voulues. On a observé une amélioration considérable des connaissances et des aptitudes des participants après ce séminaire-atelier. On a découvert une corrélation importante entre le gain de connaissances et de meilleures performances, indépendamment du nombre d'années d'expérience professionnelle et du diplôme d'études universitaires supérieures obtenu. Le bénéfice constaté et la satisfaction exprimée par les participants encouragent la réédition de cette expérience afin de former des personnels de santé qualifiés capables de développer les services de santé mentale destinés aux enfants.
Introduction
Existing health care delivery systems, including mental health care, have largely failed to meet the needs of most of the world's population. Many of these systems are centralized, hospital-based and disease-oriented, with care delivered by medical personnel in a one-to-one doctor-patient relationship [1].
Member States of the World Health Organization (WHO) have agreed that the key to achieving the goal of health for all by the year 2000 is primary health care. This care is based on the needs of the population rather than on the needs of health structures and centralized specialized facilities. This decentralized care requires the active participation of community and family members, and should be undertaken by nonspecialized general health workers in collaboration with governmental and nongovernmental sectors. Therefore, training of general health workers in the use of simple but effective techniques that are widely applicable is essential. These techniques include the stimulation of self-help groups and providing health education activities with particular emphasis on health promotion and disease prevention [2]. Since about half the population of developing countries consists of children and adolescents, specific programmes directed at this age group can be highly effective.
In 1987, a school mental health programme was launched in Alexandria, Egypt by a team from the Ministry of Health, Ministry of Education and the Faculty of Medicine with the technical collaboration of the WHO. Its aim was to train school personnel in mental health promotion, and early detection of problems and referral to a specialist [3]. Several training courses were conducted over 18 months to equip mental health physicians with the necessary knowledge and skills in child psychiatric examination. These courses covered a wide range of childhood disorders such as developmental disorders, emotional and behavioural problems, school backwardness, epilepsy, smoking, drug dependency and psychosis. Skills of basic communication were also covered [4].
Although this programme was launched a considerable time ago, the coverage is still low. This is attributed mainly to the limited number of faculty members, who can therefore train only a small number of school physicians and social workers annually. The development of trainers in the area of child mental health is crucial for the expansion of the programme. Hence, a six-day workshop was planned to provide primary care physicians with the educational skills to carry out the role of trainer.
Development of the training programme
An educational workshop was held in August 1995 over a six-day period. By the end of the workshop, participants were expected to be able to:
write specific objectives related to a professional task containing all the requisite elements;
develop a content appropriate to the previously set objectives;
adopt the most effective educational methods and techniques;
develop an evaluation tool for a teaching programme considering all its elements.
Three months were spent in the planning of the workshop, and the following activities were considered:
determination of specific objectives and development of appropriate content;
preparation of teaching materials and training modules;
development of evaluation tools;
selection of participants and task assignment.
Determination of specific objectives and development of appropriate content
The workshop was divided into three main areas: educational objectives and content development; teaching methods and techniques; and educational evaluation. Specific objectives were set for each area which led to the intermediate objectives as follows
Educational objectives and content development:
to describe the process of education;
to explain the difference between education, teaching and learning;
to draw the cycle of education;
to define the term "educational objectives";
to list three methods that determine the selection of educational objectives;
to differentiate between the three types of educational objectives;
to write an educational objective considering its quality and elements;
to illustrate, with an example, the three domains of educational objectives;
to design a content appropriate to a previously set educational objective;
to present a talk specifying the learning objectives and the appropriate content in seven minutes.
Teaching methods and techniques:
to communicate effectively with the learners;
to make a good presentation;
to select the educational technique most relevant to the stated educational objectives;
to use appropriate audiovisual aids effectively.
Educational evaluation:
to identify the relationship between evaluation and other parts of the educational process;
to define the purpose and aim of evaluation;
to describe the difference between formative and certifying evaluation;
to list the good and bad features of a test;
to compare the advantages and disadvantages of tests in current use;
to define the following terms: validity, reliability, objectivity; and describe the relation that exists between them;
to choose an appropriate evaluation method for measuring the specific educational objective.
According to these objectives, a relevant content was developed and logically arranged, proceeding from basic to advanced level of knowledge and skills.
Preparation of training modules and teaching materials
Three modules were prepared covering the three areas, to be distributed after the corresponding sessions. The modules consisted of the specific learning objectives for each area and a relevant content, in addition to illustrative diagrams, examples and a list for further reading. Audiovisual materials, including transparencies, slides and videotapes, were prepared to facilitate learning.
Development of evaluation tools
Performance evaluation checklist
A modified checklist based on a WHO format [5] was used to rate the participants' performance during a teaching session before and after the training workshop. The performance checklist consisted of three sections with a total of 27 tasks measured as not performed or performed, scoring 0 and 1 respectively, with an overall score ranging from 0 to 27.
Section I: Student-teacher interaction. This section measured the interaction of the participant with the audience during a teaching session. It consisted of 13 tasks reflecting the ability of the participant to use appropriate gestures during presentation, share ideas with the audience, encourage discussion among them, ask relevant questions, appreciate good ideas and correct without devaluating, give feedback and show concern for all the audience. Voice modulation, speech fluency, ability to get and maintain an audience's attention and to keep order were also assessed.
Section II: Presentation and teaching techniques. A total of 11 tasks measured the ability of participants to give a preliminary overview of the subject and state clearly the objectives of their talk. The adequacy of the participant's knowledge of the subject presented was also considered, together with his/her ability to select the appropriate teaching technique, and to present the subject in a simple, organized and understandable manner with emphasis on the important points. The ability of the participant to provide the audience with illustrative examples, to relate theory to practice and to refer them to additional sources of information were also assessed. Finally, the effective utilization of the allocated time was considered.
Section III: The use of audiovisual aids. This section consisted of three tasks—the appropriate selection of audiovisual aids, their clarity and their effective use.
Pencil and paper test
The pencil and paper test was designed to test participants' knowledge of principles of education before and after training. It consisted of 29 multiple choice, matching and free-listing questions with a subset of questions covering the areas of educational objectives and content development, teaching methods and techniques, and educational evaluation. All questions applied to common childhood mental health problems in Egypt such as nocturnal enuresis, childhood emotional problems (anxiety and depression), epilepsy, mental retardation and school backwardness. A mark was given for each correct answer, and the overall test marks ranged from 0 to 69.
Workshop evaluation by participants
A modified evaluation form, based on WHO workshop evaluation format [6], was used to reveal the participants' opinions of the workshop. It consisted of 19 questions measured on a scale of strongly disagree, disagree, agree and strongly agree. The evaluation form covered the following areas: workshop planning (3 questions); relevance and utility of the working methods (7 questions); mode of conducting the workshop and attitudes of organizers (2 questions); organization of activities in relation to the time available (3 questions); and benefit gained by participants (4 questions).
Selection of participants and task assignment
The selected participants were physicians affiliated to the Ministry of Health and the Health Insurance Organization participating in the Alexandria child mental health programme. Selection of participants was based on their achievement in the seven-week advanced training in child mental health that followed the basic training in this area. Those who achieved 80% or more in the advanced training course were invited to join the training on educational skills. A total of 34 were selected, and four psychiatrists attended as observers.
All participants agreed to take part in the workshop and to take the pre- and post-tests. On the other hand, only 68% of them agreed to participate in performance testing.
The participants were informed of the workshop objectives, content and detailed activities three weeks in advance. Each participant was assigned a topic addressing common childhood mental health problems such as anxiety, depression, epilepsy, nocturnal enuresis, mental retardation and school backwardness. They were requested to prepare a 15-minute presentation on their assigned topic directed at physicians or school teachers.
Workshop schedule
Registration of participants and the opening ceremony took place on the first day. This was followed by the participants' presentations of previously assigned topics, and pre-training evaluation of performance was carried out by three assessors. All presentations were videorecorded to serve as an additional source for performance evaluation.
Assessment of the participants' knowledge of the principles of education prior to the workshop was scheduled for the second day.
From the second to the fourth day, three presentations were given: learning objectives and content development; teaching methods and educational techniques; and educational evaluation. Each lecture was followed by group work where participants were divided into five groups, each of five to six trainees. The group activities entailed writing the educational objectives and developing an appropriate content based on the previously assigned topics addressing the different mental health problems. The corresponding audiovisual materials, including flip charts, posters and transparencies, and the evaluation tool were prepared during the group activities. The group activities were presented which provided feedback, day-to-day evaluation of participants and their motivation.
The final presentations of participants were scheduled for the last two days for post-training evaluation of performance by assessors. Post-testing of knowledge using the same pretest form was also carried out at this time. Finally, participants were requested to evaluate the workshop.
Data analysis
After data collection, the three study instruments were coded and analysed using the Statistical package for social science (SPSS version 6). Checking for typing mistakes was carried out by file printing, frequency distribution and cross tabulation.
The satisfaction index [6] and the Spearman rank correlation were computed. The Kruskal-Wallis test was used to test the difference in the total performance scores given by the three assessors. The Wilcoxon signed rank test was used to test the difference in participants' knowledge and performance before and after the workshop. The 5% level of significance was taken to test the significance of the results obtained.
Three scales were generated in performance evaluation.
Student-teacher interaction. This scale was generated by summing the scores for the 13 tasks measuring the ability of participants to interact with students, with a minimum of zero and a maximum of 13. Cronbach's alpha reliability for this scale was 0.95 for the pretest and 0.92 for the post-test.
Presentation and teaching techniques. The scores for 11 tasks measuring the presentation and techniques used by the participants were summed for the generation of this scale, with a minimum of zero and a maximum of 11. Cronbach's alpha reliability for this scale was 0.78 for the pretest and 0.72 for the post-test.
Use of audiovisual aids. This scale was generated by summing the scores for three tasks related to the adequate preparation of audiovisual aids, their clarity and effective use, with a minimum of zero and a maximum of 3. Cronbach's alpha reliability was 0.68 for the pretest and 0.73 for the post-test.
The three assessors were in accord with regard to the evaluation of participants' performance, as proved by the insignificant difference of the mean scores of the 23 participants given by the three assessors in the pre- (χ2 = 0.1039, P = 0.0404) and post- (χ2 = 0.358, P = 0.3614) evaluation of performance. Furthermore, in the pretest, the participants' scores given by the first assessor significantly correlated with that of the second (r = 0.9237) and the third assessor (r = 0.8966), and the score given by the second assessor correlated with the third assessor (r = 0.7687). The same was observed in the post-test (r = 0.7672, r = 0.8154 and r = 0.9229 respectively). Hence, the average scores of the three assessors were computed to represent each participant's performance.
Results
The majority of participants were males (68.4%). The overall mean age was 41.0 ± 5.11 years. Nearly one-half of the participants were working in a school health facility (44.7%), 34.2% in a maternal and child health centre, and a minority in paediatric hospitals (15.8%) and the local directorate of health affairs (5.3%). The majority held a diploma (39.5%) or a masters degree (34.2%), while 26.3% had no postgraduate degree. Their mean number of years of work experience was 16.6 ± 5.01.
Table 1 presents the marks obtained by the participants by quartile in knowledge testing before and after the training. In the pretest, all participants (n = 34) obtained less than 50% of the total marks, but the majority of them (85.3%) got more than 50% in the post-test. The overall mean percentage of marks obtained by the participants was 29.33 ± 10.17 for the pretest; in the post-test, it increased significantly to 64.1 ± 16.76 (Z = 5.0862, P < 0.0001). For the pre- and post-testing of knowledge, the highest mean percentage marks obtained by the participants was in the area of teaching methods and techniques (42.5 + 15.67 and 67.2 + 12.20, respectively), followed by educational objectives and content development (25.7 + 11.62 and 65.4 + 21.32, respectively) and educational evaluation (15.9 + 13.13 and 58.4 + 25.64, respectively). The increase in percentage marks between pretest and post-test was highest for educational evaluation (42.5%), followed by educational objectives and content development (39.7%) and teaching methods and techniques (24.7%). This gain in marks after the training was statistically significant as indicated by the Wilcoxon signed rank test (Table 2).
In observing the participants' performance in a teaching session, 56.5% of them scored less than 50% of the total test scores before the training. After the training, the scores of 82.6% of participants were 75% or more of the total test scores (Table 1).
Table 3 presents the mean percentage scores of participants in performance evaluation. There was a significant increase in the mean percentage scores of participants before (46.3 ± 20.58) and after the training (84.9 ± 12.88) (Z = 4.1668, P < 0.0001). Similarly, a significant increase in the mean percentage scores obtained by the participants in the pre- and post-evaluation of performance was observed in the areas of student-teacher interaction (Z = 4.0744, P = 0.0001), presentation and teaching techniques (Z = 4.0420, P < 0.0002) and the effective use of audiovisual aids (Z = 4.0420, P < 0.0002). The highest mean increase in percentage scores was in the effective use of audiovisual aids (60.3%), followed by student-teacher interaction (42.2%) and presentation and teaching techniques (28.4%).
Table 4 illustrates a significant positive correlation between the percentage difference in participants' knowledge and performance (r = 0.4850, P < 0.022). On the other hand, the percentage difference of participants' knowledge and performance were insignificantly correlated with the years of work experience or the postgraduate degree obtained.
Regarding the participants' opinions of the workshop, none of them strongly disagreed on any of the areas related to the evaluation of the workshop. A minority disagreed on areas related to workshop planning (5.26%), the way the workshop was conducted and the attitudes of organizers (2.63%), organization of activities in the time available (7.90%) and the evaluation of the workshop (2.63%). All participants either agreed or strongly agreed on the relevance and utility of the working methods adopted (39.47% and 60.53%, respectively) and the benefit gained from the workshop (21.05% and 78.95%, respectively) (Table 5).
Table 6 presents the satisfaction index of participants in the 19 items related to workshop evaluation. The satisfaction index was lowest for the following items: understanding of the active participation required in the workshop (55.26%); sufficient information provided on the aims and methods of the workshop before arrival (66.84%); and enough time given for practical work (69.47%). The satisfaction index was highest for the contribution of the workshop to the development of favourable attitudes towards the systematic approach to the educational problem (91.05%), encouragement of participants to put the knowledge gained into practice (90.53%) and the usefulness of the daily evaluation session (88.95%).
Discussion
Training and continuing education of health personnel and primary health care workers are key strategies for the attainment of the WHO goal of health for all by the year 2000, through providing a forum whereby practitioners can augment their knowledge, as well as maintain and improve their practice standards [5,7]. Promoting the teaching capabilities of trainers of health personnel is emphasized by the WHO as an essential component in the development of human resources for health. Moreover, it is a great advantage if trainers are health workers in the same setting as trainees, as they will have a clear and detailed idea of what the trainee will be doing in a real situation, as well as being able to identify community needs and priorities [8].
This workshop aimed at equipping physicians with educational knowledge and skills to carry out their role as trainers of staff at peripheral health centres in the field of mental health. Although this training workshop was planned on implied needs, these needs were confirmed by knowledge testing and performance appraisal conducted prior to the training. All participants scored less than 50% in knowledge testing and nearly half of them (57%) did so in performance appraisal.
Prior to the training, the low mean percentage of total marks (29.3%) obtained by participants in knowledge testing pointed to the deficiency in educational knowledge. The marked deficiency was in the areas of educational evaluation (15.9%) and objectives definition (25.7%). In fact, these areas are essential components of the educational spiral. Prior definition of educational objectives is a precondition to ensure the relevancy of training programmes. Moreover, setting criteria for a minimum level of performance expected from trainees is the basis for value judgements that permit better educational decision-making [6,9]. The relatively higher marks (42.5%) obtained by participants in the area of teaching methods and techniques could be attributed to the training previously received in basic communication, and the role they play as counsellors and health educators.
The deficiency in educational knowledge certainly influenced the participants' performance where the mean total score was 46.3%. Although audiovisual aids represent an important tool in health education, the mean percentage score for their use was only 18%. Moreover, in the area of student-teacher interaction, the mean score of performance was 45.38%. At the beginning of the workshop, most participants read from papers and thus lost eye contact or interaction with their audience.
A significant improvement in all areas of knowledge and performance was observed after implementation of the training workshop. The difference in the mean total percentage marks and scores was 34.7% for knowledge and 38.6% in performance. Moreover, with regard to the use of audiovisual aids, the rise was 60.3%. The increase in knowledge and performance indicates that the participants benefited from the handouts provided, the atmosphere of the workshop, group activities, practical exercises and the daily evaluation sessions, for which the satisfaction index was over 80%.
In fact, knowledge and performance are interrelated, as proved by the positive correlation between the percentage gain in knowledge and performance. Adequate knowledge about the educational process provides the confidence needed for efficient performance. Neither years of work experience nor postgraduate degrees influenced such a gain. All participants were physicians and all benefited from the workshop. This observed benefit and the satisfaction of the participants with the benefit gained indicate the success of the workshop and encourage its replication.
The development of strategies for enhancing the training of trainers is recommended. All physicians could be eligible for educational training programmes, regardless of the postgraduate degrees they hold or the years of work experience they have, in order to equip them with educational capabilities. Such training programmes could be replicated for mental health workers and expanded to cover all health workers and all health problems.
References
- The introduction of a mental health component into primary health care. Geneva, World Health Organization, 1990.
- Community-based education for health personnel. Report of a WHO study group. Geneva, World Health Organization, 1987 (WHO Technical Report Series No.746).
- Seif El Din A et al. A multisectoral approach to school mental health, Alexandria, Egypt. Eastern Mediterranean Region health services journal, 1991, 11:24-30.
- Seif El Din A et al. A multisectoral approach to school mental health, Alexandria, Egypt, part II. Health services journal of the Eastern Mediterranean Region, 1993, 7(2):34-40.
- Report on the intercountry workshop on research studies on teaching/learning material and training of trainers of health personnel in WHO Eastern Mediterranean Region. Alexandria, World Health Organization Regional Office for the Eastern Mediterranean, February 1993.
- Educational handbook for health personnel, 6th ed. Geneva, World Health Organization, 1987 (WHO offset publication No.35).
- Morrow NC, Speedy P, Totten C. Health education perspectives in pharmacist continuing education programme. In: Proceedings of the 12th world conference on health education, 1-6 September 1985, Dublin, Ireland: 866-72.
- Abbatt FR. Teaching for better learning: a guide for teachers of primary health care staff. Geneva, World Health Organization, 1980.
- Gleit CJ, Graham BA. Preparation of nurses for the teaching role. In: Proceedings of the 12th world conference on health education, 1-6 September 1985, Dublin, Ireland: 542-5.