An Investigation of Learning Efficacy , Management Difficulties and Improvements in Tertiary CLIL ( Content and Language Integrated Learning ) Programmes in Taiwan : A Survey of Stakeholder Perspectives

In 2011, Taiwan’s Ministry of Education conducted a national-scale appraisal of 92 CLIL programmes. However, we lack an effective model for examining by precisely how much improvement in the quality of the CLIL programmes will rise as a consequence of the increased language proficiency and the acquisition of disciplinary knowledge. To gain greater insight into the relationship between the execution and appraisal results of CLIL and the facilitation of content and foreign language acquisition, we researched the stakeholders’ perceptions of and attitudes towards CLIL in order to create a reference for the national appraisal results. Our aim was to collect both quantitative and qualitative data on the programme managers, teachers and learners in 12 CLIL programmes nationwide by way of a questionnaire survey and interviews. We assessed these data to answer our main research questions regarding the efficacy and effectiveness of employing CLIL education in higher education in Taiwan. In total, 53 undergraduates and postgraduate CLIL students completed a self-designed questionnaire survey, investigating their perceptions of and attitudes towards CLIL education. In addition, interviews with CLIL programme managers and student focus-groups were also conducted to further probe their opinions on CLIL. The findings mainly revealed that the learners’ satisfaction with the CLIL approach is greatly affected by their level of language proficiency. Our findings can significantly advance our understanding of the current situation of CLIL education and the likely effects of changing the curricula and directions of delivering content and foreign language courses at the tertiary level in Taiwan.


INTRODUCTION 1
Content and Language Integrated Learning (CLIL) is a generic term, referring to an innovative educational approach in which a subject is taught in an additional language (Lasagabaster & Sierra, 2010).CLIL is always dual-focused on both content and language.Depending on the model used, its aims can be very subtle in terms of helping learners understand the point of learning a language and developing their 'can-do' attitude as language learners (Marsh, 2009).This purpose is crucial for EFL contexts such as Taiwan or Japan where English is mainly used in classrooms rather than in the students' real lives, and where many EFL learners are eagerly attempting to advance their English proficiency.On the other hand, its aims can be subliminal in having both language and content teachers "change teaching practices, or socially-oriented, in boosting levels of harmony between inter-ethnic groups" (Marsh, 2009: 1).This purpose is also vital for those contexts in which CLIL programmes have a combination of international and local students such as in the cases researched in the present study.Contributed to by the strong driver of socio-economic globalisation, the number of CLIL programmes has increased rapidly in the last decade, and CLIL has become a mainstream form of education in many European countries (Maljers, Marsh, & Wolff, 2007;Gefaell & Unterberger, 2010).Similarly, in tertiary education in Taiwan, the establishment of CLIL programmes has been encouraged by the educational authorities, mainly the Ministry of Education (MOE), to push the goal of internationalised higher education forward.The preeminent additional language taught by CLIL programmes is English (Lasagabaster & Sierra, 2010).In order to evaluate the effectiveness and management of offering CLIL education, the MOE conducted its first large-scale appraisal in 2011, which is exceptional in that it was a nationwide appraisal conducted by the top educational authority of the country rather than by teachers, schools, researchers, or local governments, as in the European context.
Previous studies on evaluating CLIL education have mainly been conducted from three dimensions: firstly, an examination of the effectiveness of CLIL education by comparing learners' progress in the (additional) language proficiency before and after the implementation (e.g., Connon, 2007;Dalton-Puffer, 2008, 2009); secondly, an evaluation of the CLIL implementation, mainly in secondary schools in European contexts, by applying a sole research method (e.g., Jappinen, 2005); and thirdly the evaluation of CLIL education mainly initiated by the micro-involvers, namely, CLIL teachers and researchers (e.g., Perez-Canado, 2012).
However, apart from European contexts, investigations of implementing and evaluating CLIL programmes on a large scale are apparently rarely documented in the literature (Perez-Canado, 2012), in particular, in Asian EFL situations where CLIL education is increasingly obtaining the attention of educational authorities who hope to establish close connections to globalisation by integrating English, the most dominant foreign language, into disciplines at the tertiary level.Therefore, this study aims to bridge the gap by examining how EFL learners view the new learning approach of CLIL based on their expectations and experiences with a quantitative survey, and probing CLIL learners' and practitioners' deeper attitudes and perceptions of CLIL during qualitative interviews to see if their opinions correspond to the official results and previous studies.

LITERATURE REVIEW 1
As Coyle (2007b) suggested, effective CLIL takes place "through progression in knowledge, skills and understanding of the content, engagement in associated cognitive processing, interaction in the communicative context, the development of appropriate language knowledge and skills as well as experiencing a deepening intercultural awareness" (p.550) and thus much current research focuses on evaluating the relationship between input and outcome in both content knowledge and linguistic competence, while still other studies spotlight the evaluation of CLIL programmes as a whole in various settings, and identify their potential problems and the difficulties of implementation.

YANG
In the first instance, a number of studies in European contexts have examined the effectiveness of using the CLIL approach to teach subjects other than language courses at different levels of education.For instance, in Jappinen's (2005) study, the results reveal that the students in the experimental CLIL group had more significant cognitional development in learning mathematics and science than those in the control group, and the research confirms that "the Finnish CLIL environments in public mainstream L1 education have succeeded" (p.161).However, she also warned that abstract topics may not be well suited for young CLIL learners in her context.
Similarly, Burston and Kyprianou (2009) also concluded that their CLIL programmes in teaching architecture, biology and law in a Cyprus university were holistically successful in that the participants were very motivated and satisfied with the programmes, the departments were satisfied, and most importantly, the students' grades had improved.
In contrast to the above two studies, Dalton-Puffer ( 2008) examined the outcome of CLIL education from the perspectives of language competencies and use in the classroom.She argued that "CLIL significantly enhances the language skills of the broad group of students whose foreign language talents or interest are average" (p. 5) where the favourably affected language competencies include receptive skills, vocabulary, morphology, creativity, risk-taking, fluency, quantity and emotive/affective outcomes.
In addition, in CLIL classrooms, teachers tend to provide elaborate feedback and additionally reserve more space and time for students' comments and interaction.At the end of her paper, Dalton-Puffer (2008) pinpointed the fact that investigation of students' perceptions of how they view their CLIL teachers' (including content teachers and EFL teachers) language problems and corrections in instruction is still lacking, so the present research aims to attend to this concern.Coonan (2007), on the other hand, studied the outcomes of both language competence and non-language effects in CLIL classrooms through teachers' self-observation and introspection.Her results are similar to Dalton-Puffer's (2008) in that she found that learners' linguistic receptive skills are promoted more than their productive skills.Besides, she emphasised that CLIL positively affects the way students learn the content because extra care is taken by teachers to help them overcome the hurdles and to nurture language growth through the content.
Nevertheless, the claimed dual effects of CLIL are also questioned in some studies, and some disadvantages have been unexpectedly generated in some European contexts.For example, Seikkula-Leino (2007), who researched CLIL learners' affective performance and attitudes, noted that although pupils in CLIL programmes were more motivated to study and to use the foreign language (FL), they sometimes felt incompetent and inadequate in learning, a situation brought about by the complexities and difficulties of learning through a FL.CLIL caused low self-esteem regarding the students' FL ability.Bruton (2011a.b.) not only questioned the results of improved FL ability in the experiments comparing students' performance in CLIL and non-CLIL groups, but raised critical concerns about the real effect of the development of the dual focus for average students (not selected or elite students) in CLIL education.Furthermore, he also highlighted the concern of quality control in CLIL, including CLIL teacher training.
In addition to the European cases, a number of CLIL studies have also been conducted in Asian contexts, but their attitudes are more cautious.For example, in Japan the implementation of CLIL has aroused concerns that FL may replace the priority of learning their mother tongue for young learners (Sasajima, Ikeda, Hemmi, & Reilly, 2011).Likewise, a number of studies (Lee & Chang, 2008;Mackenzie, 2008;Marsh & Hood, 2008) have also contended that as a potentially new teaching approach in Asian contexts, CLIL has motivational advantages, develops multiple intelligences, and acts as an enabler for Asian EFL learners to achieve positive atti-YANG tudes towards the relevance of English, but still, CLIL teacher preparation is apparently insufficient in this area.
Feng (2010) also described the diverse models of CLIL adopted in the greater Chinese circle (Hong Kong, Mainland China, Macau and Taiwan).He argued that CLIL exists as a continuum from language-driven to content-driven ends, and each context is mapped diversely on this continuum according to differences in cultural, governmental or economic factors.Similar discussion of the probabilities of implementing CLIL in Taiwan and China has confirmed that, in contrast to the traditional ELT approaches, CLIL can be an alternative for accommodating students' subject knowledge and linguistic competence, although there is still the issue of training qualified CLIL teachers as in other contexts (Hou, 2007;Lo, 2007;Luo, 2006).Besides, the nature of the different subjects taught in English, the English acquired in the English language classes, and the students' English proficiency entry level can also greatly affect the teaching efficacy in Asian CLIL education, as in the case of Hong Kong (Marsh, Hau, & Kong, 2000).To the best of my knowledge, currently, the only empirical study investigating CLIL education in Taiwan is Yang and Gosling's research (2013).After examining a single highly-recommended CLIL programme, they concluded that CLIL teacher development is an urgent need in Taiwan, further consideration of whether all students in EFL settings can equally learn or thrive in a CLIL programme is warranted, and accommodation should be made for language support.However, compared to the rich CLIL research documented in European contexts, the practical outcomes of CLIL education in Asian contexts, in particular Taiwan, are still relatively understudied.
Regarding the evaluation of CLIL in various settings, Perez-Canado (2012) conducted an extensive review of the cases across European countries.She concluded that, not surprisingly, a great majority of the CLIL programmes have been championed in Europe, but unfortunately the fact is that these programmes are full of methodological flaws in their design.Most of the evaluations are stand-alone quantitative studies which do not take dynamic variables or statistical analysis into consideration.Thus, she advises that future studies evaluating CLIL should rigorously investigate "whether the gains observed are truly ascribed to CLIL practices" (p.16).
So far, most of the experimental studies used learners' language output and knowledge competency as the measures to establish the effectiveness of the programmes.However, students, as "the target group and intended beneficiary of CLIL education, have at this point not had much voice in the development of and discourse on CLIL in general" (Dalton-Puffer et al., 2009).Students' attitudes and perceptions of CLIL education are relatively understudied, except for a few studies (e.g., Dalton-Puffer et al., 2009;Gefaell & Unterberger;2010;Yassin, Marsh, Tek, & Ying, 2009).
Although CLIL is still in its infancy in Taiwan, several studies have been conducted by the author and colleague to unveil its current implementations and effects.They, for instance, studied why a CLIL programme would be highly recognised or not recommended by Taiwan's MOE after its first official appraisal of all the CLIL programmes in Taiwan universities (Yang & Gosling, 2013, 2014).They concluded that CLIL learners' attitudes towards CLIL education can be a key factor, affecting the results of the programme appraisal.Those who hold a relatively positive attitude towards the CLIL approach may help the programme pass the appraisal and make it highly recommended by the authorities, and vice versa.In addition, Yang found that proficient CLIL learners tend to use indirect language learning strategies relatively more frequently in order to adapt to the highly collaborative learning environment of tertiary CLIL education (Yang, 2016 b).His studies (Yang, 2014a(Yang, , 2015)), furthermore, confirmed that CLIL undergraduates did outperform their non-CLIL counterparts in both content achievement and linguistic outcomes, and thus also showed higher mobility and employability in the globalised job market.These studies, so far, have clearly evidenced the positive benefits of CLIL education realised in an Asian EFL context.
However, in Taiwan, although CLIL programmes are encouraged by the MOE, currently an objective and systematic evaluation of both students' and teachers' performance in CLIL education is still seriously lacking.Thus, the issue of whether or not the quality of CLIL lessons can be controlled is doubted by the public (Chen, 2012).In other words, the issue of how stakeholders such as students or programme managers evaluate their CLIL education is generally overlooked.Thus, to bridge the gap, this study attempted YANG to combine both qualitative and quantitative methods to evaluate CLIL education along with considering the stakeholders' variables, and to triangulate the evaluation results from various sources of data by comparing and contrasting a number of CLIL programmes in Taiwan.Hopefully, the study can highlight implications for managing CLIL programmes, and provide practical suggestions for how to evaluate students' performance in terms of both content and FL in EFL contexts in Taiwan.
To be specific, this research mainly explores: firstly, how the CLIL learners at tertiary level view the effects of CLIL education; secondly, how the stakeholders of the CLIL programmes, namely, the managers and learners, view the claimed dual benefits of CLIL.To be specific, the research focuses on the following questions: What are the students' perceptions of and attitudes towards CLIL education?
What are the differences or similarities in the views of the students and managers of the implementations of the CLIL education in Taiwan?

A mixed-methods research framework
As suggested by Perez-Canado (2012), the research design and methodology of future studies on CLIL should combine mixed methods, adopt triangulation, and employ multivariate procedures.Hence, the present study used both quantitative and qualitative methods to collect and analyse data, as the study's focus lent itself to a mixed methods approach using qualitative and quantitative approaches in combination, and drew upon the advantages of each.This method can serve to provide a better understanding than either approach alone (Creswell & Plano, 2007).Using the notation proposed by Morse (1991Morse ( , 2003)), this is a QUANQUAL explanatory mixed-methods design.The results are discussed using the quantitative results in combination with the qualitative data to enhance the analysis and further illuminate some of the findings.This methodology allows the integration of data and responses from the various stakeholders, from the MOE, through the programme directors, the teaching staff and the students as outlined below.
A quantitative approach was employed with the students who were given a Chinese close-ended questionnaire with a Likert scale five-point response (from 5, strongly agree to 1, strongly disagree) designed for the purpose of the study.The question items in the questionnaire are divided into two major sections and five sub-categories, that is, linguistic improvement, content knowledge learning, classroom practices, supports and strategies, and affective attitudes.Before the students completed the questionnaire, it had been piloted by one L2 teacher, one L1 teacher and a number of students in the selected programmes to ensure its comprehensibility and readability.The reliability of Cronbach's alpha reaches .856.
The qualitative data came from several sources in addition to a reference to the MOE appraisal report.Interviews were held with the teaching staff, where each teacher and programme director was individually interviewed for about half an hour in their respective first languages.Chinese interviews were transcribed into English and checked by independent translators.In addition, the students responded to open-ended questions on the given questionnaire.

The researched context
In a globalised society, politics, culture and economics are becoming closely dependent on each other, and higher education, which is responsible for cultivating professionals and transmitting professional knowledge, is also facing competitiveness and challenges from other countries.Consequently, to facilitate cultural exchange, to publicise the features of Taiwanese higher education, to increase the standards of tertiary education, to solidify the competitiveness of universities, and to broaden college students' global vision and language proficiency, the MOE of Taiwan has long encouraged institutes in higher education to set up courses which use English as a medium of instructing disciplinary knowledge.The final purposes of these programmes are to recruit foreign students to study in Taiwan, to increase the competitiveness of Taiwanese higher education, and to develop professionals with the ability of international mobility.
In addition to the numerous CLIL courses, there are 92 CLIL degree programmes in 29 universities2 conferring bachelor, master and doctoral degrees.The earliest programmes were established in 1979 and the latest were set up in 2010.All of the programmes are expected to use English as the only language of instruction in the classroom.The MOE provides incentives of up to 10 million NT dollars per university to encourage the establishment of more CLIL degree programmes, while also conducting a national-scale appraisal of all the programmes to ensure their quality in October 2011.The 92 programmes were classified into three different rankings, namely, highly recommended, recommended, and not recommended.

Sampling CLIL programmes and participants
This research sampled the programmes based on several categories.Firstly, all of the current CLIL programmes were divided into undergraduate (UG) and postgraduate (PG) programmes.Then, the results of the national appraisal were used to further classify the two major categories into highly recommended (HR), recommended (RD), and not recommended (NR) programmes, which resulted in six categories in total.Next, one programme (G) was randomly selected as the study context from each of these six categories; therefore, six CLIL programmes were investigated in this study.Approximately ten learners in each programme were invited to complete the questionnaire, and the return rate reached 90% (i.e., N=54 in total).Of the participants, 38 were undergraduate students while the rest were postgraduates (including master and doctoral students) from six different universities island-wide (5 comprehensive, 1 polytechnic, 2 national, 4 private), all of whom are Taiwan nationals.A majority of them (40.7%,N=22) were in the final year of their programme, the first year (7.4%, N=4), the second year (33.3%, N=18), and the third year (18.5%,N=10); 81.5% (N=44) of them came from normal senior high schools while only 18.5% (N=10) had studied at a vocational high school.Normally, senior high school students' English competency is higher than that of polytechnic students in Taiwan; therefore, CLIL programmes are more commonly implemented in the former than in the latter.Hence, over half of the respondents (57.4%) had reached equivalent of the CEFR B2 level in an English proficiency test (over TOEIC 750) while 40.7% had reached CEFR B1 (TOEIC 500-750), and only 1 participant had lower competency (less than TOEIC 500).Generally, the present participants had better command of English compared to their non-CLIL peers, and had achieved the benchmark of English language proficiency required by most Taiwanese universities.
Finally, the directors (D) and also the teachers of the six programmes were interviewed together with four randomly-selected students (S) who helped complete the questionnaire survey from each programme.Totally, 24 students, making up six focus groups, joined the interviews for this study.Since the CLIL education investigated in the present study was focused on degree-based programmes rather than departments, all the programme managers (or chairs) and teachers came from other departments to support the programme, so there were no full-time teaching faculty Similarly, all the programme chairs (4 males, 2 females) were Taiwanese teachers with proficient English skills due to their education background in English-speaking countries.All of them, during the research stage, had taken the position for one term (i.e., 3 years).The student interviewees constituted four undergraduate groups and two postgraduate groups.All the interviews lasted for about one hour and were conducted in Mandarin Chinese to facilitate contribution and avoid any misunderstandings.

Data analysis
To answer research questions 1, a Chinese close-ended questionnaire (see Appendix 1 for the English version, Yang & Gosling, 2013) on a five-point Likert scale was designed, and the collected data were analysed with the help of the statistical software, SPSS 14.0.In addition to the descriptive analysis, t-tests and one-way ANOVA were also performed to gauge any significant differences in the variables.Then, to answer research question 2, further interviews with the directors, teachers and students were conducted.The interviews were recorded and then transcribed into English by the researcher and two research assistants together with the help of NS colleagues' proof-YANG reading.Then, all the English transcriptions were processed with the assistance of the qualitative data analysis software, WeftQDA.The study applied grounded theory, which is extensively used to analyse qualitative data by coding data with similar concepts into various categories (Glaser & Strauss, 1967).The coding was checked and confirmed by the research assistants of this study.
Tables 1 to 5 display the results of the perceived linguistic improvements under CLIL education.Surprisingly, different from the previous studies, a majority of the present participants generally did not show high agreement with the increment in linguistic skills under the CLIL approach.CLIL learners usually have better performance in receptive skills than in productive skills, which has been extensively documented in previous research (see Yang, 2014a for a detailed discussion).Yet, the current participants, in contrast, did not hold similar attitudes.It is assumed that the participants' overall English proficiency was already high; thus, it is difficult for them to perceive any significant change.In addition, it is likely that teaching linguistic elements is not stressed as equally as instructing content knowledge in Taiwan CLIL programmes; hence, the participants may not be clearly aware of the improvements in their English language skills.CLIL is sometimes criticised as an elitist approach (Burton, 2011a. b.) and Taiwan CLIL programmes also purposefully recruit highly proficient English learners due to the fear that lower English achievers may not be able to adapt well to courses instructed only in English.Ironically, this selective recruitment seems to lead to a lack of general agreement regarding linguistic improvement as a result of CLIL education.The following Tables 6 to 9 exhibit the participants' opinions on learning content knowledge via the additional language, English.However, the results show a different picture from the previous discussion.The majority of the respondents admitted to having difficulties understanding the courses delivered in English.In other words, half of the students indicated that learning content knowledge through an additional language is not as easy as they had believed, even though their command of English was generally rather good.There are various possible reasons for this.For instance, learning content and language simultaneously brings students double pressure, which may hinder the acquisition of content knowledge compared to learning in Chinese-speaking classrooms.Another reason might be the different purposes of using English in the classroom.Most Taiwanese EFL learners learn English for the purpose of taking tests, and the English curriculum is designed based on learning the language for general purposes, which does not sufficiently prepare learners for using English for academic purposes (EAP) at tertiary level; neither are there many EAP courses provided to bridge this gap in the CLIL programmes.Although CLIL is not regarded as providing language courses, and CLIL and ESP (English for specific purposes) are seemingly two extremes of the language learning curriculum, additional ESP courses can be offered as supplements and supports for CLIL learners (Yang, 2016 a), in particular in higher education where CLIL students have to learn academic disciplines via very academic English which clearly has some language conventions which must be followed but which are seldom taught in Taiwan's English learning context.
Thus, it is advisable that tailor-made ESP courses be designed and provided as additional support for CLIL learners.Regarding the third category, supports and strategies, Tables 10 to 14 show the results as follows.First, in line with the previous discussion, over half of the participants showed a need for extra supportive language courses to assist their CLIL learning.Although CLIL claims a dual-focused approach, catering for both content and language learning, content teaching is usually overemphasised in relation to language teaching.It is therefore argued that when CLIL is implemented in secondary or primary schools, the courses are less academic and extra language support to master the courses is also thus less necessary.Yet, in tertiary CLIL, nearly all the courses are academic-oriented and the English required to master the disciplines is more complicated, but also conventionalised with predictable peculiarities.Thus, extra academic English courses may be beneficial for lessening learners' anxiety and pressure of acquiring content knowledge via the additional language in CLIL contexts.Without appropriate scaffolding, learners may translate English into Chinese in order to learn the subject in the CLIL classroom, which is evidenced in the results of question items 28 and 29.
Second, the participants did not expect practitioners' timely use of the learners' mother tongue to help their learning and facilitate their understanding in CLIL classrooms, as shown in Table 14.In Taiwan, CLIL is classed as a high-exposure CLIL mode (Yang & Gosling, 2013, 2014) as English is often used for over 60% of the instructional time.Some local CLIL practitioners and learners may misinterpret that CLIL should be instructed using English only, or at least as much as possible; however, appropriately using the L1 (mother tongue) is helpful and is in fact encouraged in CLIL classrooms to keep learners motivated and engaged in CLIL learning (Lo, 2015), thus increasing the effect of CLIL education.Hence, timely code-switching while instructing or explaining difficult concepts in the disciplines should be adopted by local CLIL practitioners, as it is believed to be able to facilitate the understanding of the content knowledge and increase the satisfaction with the programme.
In the category of examining teachers' actual CLIL practices in class, a number of questions were asked.Table 15 lists the overall results.In general, the participants agreed relatively less with their teachers' actual practice in class.First, the present participants would prefer Taiwanese teachers to native English-speaking teachers to teach CLIL courses (Q11, Q12).They also showed less agreement with native English speaking teachers being more capable of teaching CLIL subjects (Q20, Q21).It is believed that local teachers share the same cultural and language backgrounds with the learners so interaction with each other would be easier.In addition, postgraduate students may receive many tutorials with their teachers to discuss their dissertations.Teachers who speak the same L1 are thus more popular with postgraduate students as their supervisors.
Second, a majority of the respondents did not agree that the teaching methods are different in CLIL class (Q10), which pinpoints a major problem in the current Taiwan CLIL education.That is, most CLIL teachers are content teachers who have a good command of English, but unfortunately lack proper training of teaching language.Thus, they tend to replicate the way they teach content in their L1 in the CLIL context, and make no obvious attempt to adjust their English to increase content comprehension, which is also evidenced in Q30 and Q35.In other words, insufficient CLIL teacher training leads to this duplication without proper adjustments to the teaching methods.This insufficiency also causes dissatisfaction with the assessments in CLIL courses.CLIL is dual-focused and differs from traditional education approaches; thus, assessing both language achievement and content learning outcomes is necessary, but this is often neglected by CLIL practitioners (Massler, 2010).Similarly, the results of question items 17, 18 and 19 also indicate that traditional lectures and group discussion are still the most common teaching methods used in the current CLIL programmes, but they are not very popular with the students.All these concerns mirror the necessity of offering CLIL teacher training for practitioners.However, nearly 90% of the respondents did not expect teachers' correction of their errors (Q34).Error correction may prevent learners from engaging in linguistic production, and discourage them from studying in a CLIL programme, for fear of losing face in public.Indeed, at the initial stage of CLIL education, fluency is more desired than accuracy in terms of learners' linguistic production.The respondents held a divided attitude towards Q26.They showed an uncertain attitude towards which language is preferred to learn content knowledge, L1 or English.CLIL is still a rather new educational approach and thus not much positive or empirical evidence can be found locally, except for Yang's research (2014aYang's research ( , 2015)).It is argued that learners are likely to have more positive attitudes towards CLIL once more local studies have confirmed its effectiveness and benefits.Table 16 concludes the last category of the questionnaire, i.e. affective attitudes, and presents the overall results.First, the reason why the students decided to study in a CLIL programme may be their parents' encouragement or their interest (Q13 and Q14).According to the MOE's policy, Taiwan CLIL programmes are established not only to attract international YANG students but also to develop local students' content knowledge and language proficiency, which is supposed to help increase their future employability and mobility in the globalised job market, and this non-linguistic benefit of CLIL education has also been confirmed in Yang's research (2015).Thus, it is self-evident that parents would encourage students to study in a CLIL programme.
However, after studying a CLIL programme for a period of time, it seems that it does not increase most students' motivation to learn either English or the content knowledge.It is possible that CLIL education is not as wonderful as they may have expected.Thus, in their overall judgement, they did not show a sufficiently supportive attitude to recommend this innovative approach to other students.It seems that the present CLIL learners, in general, are not very satisfied with the claimed benefits of CLIL education.Probably, the fact that they did not perceive a significant increase in their English proficiency is one of the reasons.Yet, in Yang's (2014a) study, he pointed out the interesting fact that CLIL learners tend to devalue the actual increment of their language achievements and content knowledge performance while in fact showing an increase in both areas according to tests.This also echoes the previous issue, namely the use of appropriate assessments in CLIL education.

Significant differences by variables
Differing from the descriptive data in section 4.1, the following discussion focuses on the significant differences caused by different variables.In the present study, question items regarding the demographic information of the questionnaire were used as the variables to gauge the statistical significance among various groups by performing t-tests and one-way ANOVA.First of all, only two question items showed a significant difference between female and male learners, i.e.Q16, satisfaction with teachers' assessments in CLIL courses (t=-.761,<.01) and Q17, the preference of teachers' lectures in the CLIL classroom (t=-1.175,<.01).Male learners showed significantly higher satisfaction with teachers' assessments and lectures than the female students did.It is possible that, since female EFL learners generally have a more positive attitude towards English learning and have higher English proficiency than their male peers (Kobayahsi, 2002), they would naturally care more about how they are assessed.Therefore, when English performance is not as highly stressed and properly assessed as the content knowledge is, as in the present context, they would feel dissatisfied.Similarly, female CLIL learners may expect more diverse teaching methods to be used in the CLIL classroom, and thus also showed a reluctant attitude if teachers' lectures still dominated as they do in traditional courses delivered in the L1.However, this issue may be worth further investigations.
There is only one item showing significant difference between undergraduate and postgraduate CLIL learners, i.e. improvement in writing skills (t=.640, <.05).The undergraduates had significantly higher satisfaction with this than the postgraduates did.It is assumed that English writing skills are more important for postgraduate study, and learners would show greater dissatisfaction if they felt that CLIL failed to improve their English writing skills, in particular the skills required for dissertation writing.This result indicates that when a CLIL programme is being planned, considering which linguistic elements would be more practically required by different learners is essential.For instance, CLIL implemented in postgraduate study should place much emphasis on integrating English writing with content learning, while listening and reading skills may be more highly required in undergraduate study.
For the variable of which year they were studying, no significant difference was found, which indicates that the respondents' concerns and attitudes towards CLIL education are generally consistent across the years of study.However, for the variable of their previous major in senior high school (senior high school without a specific major, vocational high school with a non-language major and with a language major), a number of items show statistical significance among the groups as shown in Table 17.In general, the CLIL students who came from senior high schools had a more negative attitude towards CLIL education.They were less satisfied with the improvement in their English skills and with the teachers' teaching methods in CLIL courses; thus, they showed less motivation and involvement in their CLIL courses, and also believed that Taiwanese teachers have difficulty teaching CLIL programmes.In contrast, those learners with a vocational high school background had a more welcoming and acknowledging attitude towards CLIL education, but also expressed greater need for extra English courses.The likely reasons might be similar to those discussed previously.That is, the higher English achievers may have a higher expectation of CLIL increasing their English proficiency, but they easily become dissatisfied if the linguistic elements are not equally addressed in the courses.On the other hand, students with intermediate or lower English proficiency may regard CLIL education as one way to improve their English abilities, so they showed more positive attitudes towards it.Again, the implementation of a CLIL course or programme should take contextual differences into consideration and require necessary adaptation to fit the local settings.This can also explain why CLIL, used as an umbrella term, exists in many different versions worldwide (Coyle, 2008).
The variable of learners' English proficiency led to the most significant differences in this study (see Table 18).Over half of the participants had a high command of English (CEFR, B2 level or above) while the rest were at the B1 level (with one student below).Those who had better English proficiency showed significantly higher agreement with their own performance and with native English-speaking teachers' capability in CLIL courses, but in contrast, lower satisfaction with the teachers' performance and the CLIL approach overall; thus, it is obviously difficult for them to make a recommendation for future students.Contrarily, those with lower English proficiency were more motivated to study in their CLIL programme, acknowledging its effects on helping increase their future employability, and recommending the approach to prospective students, although they had relatively more difficulties understanding the CLIL course materials.They tended to convert English into Chinese in CLIL class and also required additional language courses to bridge the gap in comprehending English-delivered lectures.Moreover, they preferred local teachers to teach CLIL courses probably because local teachers are more empathetic and understand learners' needs and difficulties when learning with an innovative approach (Árva & Medgyes, 2000;Medgyes, 1992).
In the last variable, the time duration of studying English, only one item was found to have significance across the groups, i.e.Q33.CLIL benefits to future job prospects (F(2:51)=3.662,P<.05).It was found that those who had studied for longer in the programme showed higher agreement with this benefit than junior CLIL learners did.The former learners were about to graduate from the programme and had to find a job, so the dual-focused CLIL education, developing both content knowledge and language competency, was expected to be helpful for their job hunting in a globalised market.Indeed, in Yang's (2015) research, he argues that increasing CLIL learners' mobility and employability are the two evidenced non-linguistic benefits of CLIL education.
A short conclusion to this questionnaire survey is that the implementation of CLIL education should be contextualised, depending on the learners' needs and customised curricula to accommodate both content and language learning.

Feedback from the interviews
From CLIL programme managers and teachers In total, 6 programme directors (also CLIL practitioners) were interviewed for the present study, and their qualitative contribution can be discussed under the following headings.
Dual benefits of CLIL: All of the interviewees expressed the opinion that CLIL has two major advantages.From the university perspective, CLIL brings more international students to study in Taiwan's tertiary education institutions, which is relatively more important for private universities.Internationalisation is one of the important indicators to appraise a university, and the number of international students is an essential criterion for evaluating a university's internationalisation, which would greatly affect the annual budget or subsidy a university can obtain from the MOE.From the students' perspective, CLIL creates an international learning environment for local students, which means that students can acquire intercultural knowledge and learn English without travelling or studying abroad.One director said, "The number of international students we can recruit is unlimited and the implementation of CLIL programmes can help me pass the MOE evaluation in terms of internationalisation." Insufficient and unstable CLIL teachers: This is the major concern raised by most interviewees.They argued that CLIL is the right approach and should be implemented, but the problem is that teachers might not be ready for this change.Qualified CLIL teachers are still rare.Most of the current CLIL teachers are either native English-speaking content teachers or local content teachers with proficient English competency.However, CLIL seems to only mean lectures given in English to them; therefore, the accommodation of both content and language teaching is always overlooked.This probably may be the major reason making the present learners dissatisfied with CLIL education.Overreliance on native speakers also leads to another problem, that is, "they are relatively unstable.I mean, they usually come to Taiwan for a short term and may terminate their contract anytime, which brings much uncertainty for the course director to arrange the future courses," as another programme director commented.
Problem-based learning oriented: Due to the novelty of the CLIL approach, traditional lectures may not be able to suit learners' needs and expectations.Thus, group discussion, class activities, collaboration, and interaction are the common features of a CLIL classroom.However, compared to a similar course delivered in Mandarin Chinese, these CLIL practices require much longer pre-class preparation by the teachers, so most universities are willing to pay double the salary for those CLIL teachers.One teacher remarked, "Unless you are also interested in and enthusiastic about CLIL education, preparing a CLIL course is really high pressure and causes anxiety.Honestly, double the pay is not a very attractive incentive.The time and effort spent on preparing the course is unbelievably immense." Unattended accuracy in English production: Nearly all the directors and teachers expressed the opinion that accuracy in learners' English production is not what they are concerned about; in contrast, they attend much YANG more to the accuracy in communicating content knowledge, not whether the content knowledge is presented grammatically correctly or not.Hence, they hardly ever offer feedback on language use to learners, and also seldom evaluate learners' English performance in assessments.Ironically, receiving teachers' feedback on their language use is what many learners yearn for in CLIL class.It is argued that this gap may distort the claimed dual-focuses of CLIL education and decrease learners' motivation and satisfaction with CLIL.
The enhancement of global vision and confidence: Since most of the CLIL programmes embrace international students and local students in the same classroom, they offer chances for local students to learn how to interact with the international students, which greatly helps in developing their intercultural awareness and communication, and builds up their confidence in using English for communication.According to one director's observation, "CLIL learners comparatively have more confidence in expressing their opinions in English, and the programme helps them to expand their global views as they spend time interacting with their international peers very often during and after class."One director was very proud of her students' performance and achievements in the CLIL programme and even claimed that CLIL will replace English departments in Taiwan tertiary education soon as she believes that CLIL learners perform better at content knowledge and language proficiency while English majors only focus on language skills.
Preparing qualified CLIL teachers: Although the interviewees expressed positive comments on the implementation of CLIL in Taiwan, they all suggested that it is indispensable to train qualified CLIL teachers immediately as the popularity of CLIL programmes is growing quickly in Taiwan's universities.They suggested that workshops addressing how to integrate language teaching into content teaching should be held, which is particularly urgent for content teachers as they usually have no previous formal training on language teaching.One director said, "A workshop on how to teach CLIL is not only needed but should also be provided periodically for a long-term period.It is a bit ridiculous that the University sometimes holds a workshop for simply two days and hopes that we, the content teachers, can become qualified CLIL teachers all of a sudden." All in all, compared to the preceding survey results, CLIL directors and teachers hold a much more positive attitude towards the effect of CLIL education.However, if teachers' anticipations and learners' expectations of CLIL are not aligned, complaints and dissatisfaction would unavoidably occur.Hence, communication between the two parties and modification of the programmes are essential to increase the future implementation of CLIL and ensure its claimed benefits.

From CLIL learners
In total, six focus groups of students were invited to join the interviews.Their opinions are categorised as the following headings, some of which are similar to the author's previous findings (Yang & Gosling, 2013, 2014).
'Asianalised' CLIL: Many students expressed the opinion that they supposed there would have been many native English-speaking peers in class, but the fact was that most of their foreign peers come from Asian countries, in particular the South-East Asian region where the first language is not English.Ideally, all the CLIL programmes would have half native English-speaking students and half local students to create an internationalised learning environment according to the MOE expectation, but in fact native English speakers are still very rare in the current programmes.Thus, students may feel that the actual situation is totally different from what they had been told, and then disappointment and dissatisfaction may occur.One student commented that "all the international students in my programme come from a single South-East Asian country and their English proficiency is much lower than ours.I cannot see that I can learn any English from them." Deficient in CLIL teaching faculty: 'Teacher and teaching quality' is another major student doubt about the current CLIL programmes.A majority of the interviewees indicated their preference for native English speakers as CLIL teachers and complained about the local non-native English speaking teachers' English accuracy, fluency and pronunciation, which would sometimes hinder understanding of the lectures.Although the differentiated CLIL classroom discourse between native and non-native speakers has been discussed in some research (e.g., Dafouz, Nunez, & Sancho, 2007;Llinares-García & Romero-Trillo, 2008), it is difficult to argue who, native or non-native teachers, is more qualified to teach CLIL courses.One interviewee commented that "Taiwanese teachers have difficulties delivering the course in English but they know our learning difficulties and can offer help.However, if we don't understand something, native teachers only explain it once again in English and we still cannot get it."Besides, the interviewees argue that those who can speak fluent English are not necessarily qualified CLIL teachers because many of them have no training in how to teach English.Thus, the dual-focused programme would sometimes end with only one focus, i.e. content knowledge.
The timely intrusion of L1 and language feedback: As discussed in the preceding sections, learners' L1 can play an important role in CLIL education (Lo, 2015) and the interviewees also expressed a similar opinion that L1 can be appropriately used to facilitate understanding."Sometimes we need Chinese translation for difficult terminology and concepts in class; otherwise, we will be stuck there and cannot move on," remarked one student.In addition, they argued that they expected some feedback on their language use so that they can improve it.However, feedback should be general, and not too specific or frequent.One learner said, "If teachers correct each of my mistakes or errors in English, I think I would refrain from using English next time.This would demotivate me." Progressive English receptive skills: Differing from their responses in the questionnaire survey, many of the interviewees agreed that their English proficiency had improved due to their CLIL education, in particular, their receptive skills, i.e. vocabulary, listening, and reading, as evidenced in the literature (e.g., Aguilar & Rodríguez, 2012;Dalton-Puffer, 2008).The materials students used in CLIL courses are authentic English language textbooks, and they are also expected to do a lot of reading after class; therefore, their reading speed and vocabulary size is expanded.However, they also hoped that an ESP course, teaching terminology and reading skills, would be offered before the CLIL course in order to shorten their time of adaptation.As one student remarked, "Teachers often use some professional terms in class but we cannot understand what they mean.So, we spend time looking up the meanings after class, but we actually already know these terms in Chinese.So, if teachers can provide us with Chinese translations of the terminology in advance, this will reduce our wrong guessing and facilitate our content comprehension." Divided opinions about the effect of CLIL: Regarding the overall evaluation of the CLIL approach, the respondents held a divided attitude.It was found that learners from a highly-recommended programme generally liked CLIL and would recommend it for future students, while those from programmes that were not recommended held a relatively negative attitude towards the effect of studying in a CLIL programme, and were hesitant to make recommendations.These responses are similar to the author's previous findings in a case study (Yang & Gosling, 2013, 2014) that students' satisfaction and the appraisal result of a CLIL programme are closely related.Furthermore, not all disciplines in universities are suitable to be conducted with a CLIL approach.For instance, disciplines in the humanities or regarding local features may not be adequately taught using the CLIL approach.

Conclusion of the research
This research investigated the effect of implementing CLIL education at the tertiary level in Taiwan.The research adopted a mixed-method research design to study the questions.The research instruments included a self-designed questionnaire and in-depth interviews with CLIL programme directors and learners.In total, 52 undergraduate and postgraduate CLIL learners island-wide were invited to join the questionnaire survey, while six programme directors and teachers and six focus groups of CLIL learners joined the semi-structured interviews.In the descriptive analysis, the results show that the present CLIL learners apparently do not have very high agreement or satisfaction with the claimed benefits of their CLIL education.The major cause may come from the imbalance in the teaching of the content and language areas.Usually, content teaching is highly stressed but linguistic elements are not, which makes the learners perceive no increment in their language proficiency, and thus they feel that the dual-focuses of CLIL are not equally achieved.However, in the interviews with the YANG CLIL directors and teachers, they showed a very positive and welcoming attitude towards CLIL.They believed that CLIL brings many advantages to both universities and students, but they also expressed one major concern, that is, the preparation of qualified CLIL teachers for the popularly-growing programmes at the tertiary level island-wide.In contrast with the responses expressed in the questionnaire survey, the interviews with the CLIL learners showed a slightly different attitude.In general, they acknowledged the increment in linguistic skills, in particular the receptive skills, preferred native English-speaking teachers, and developed intercultural awareness.In contrast to their less positive responses shown in the descriptive analysis, the learners expressed a more positive attitude towards the effect of CLIL education in the interviews.

Pedagogical implications
The present research also offers the following pedagogical implications.
First, communication should be well built between programme providers and CLIL learners.The results indicate that there is a gap regarding the effect and benefits of CLIL education between these two groups of stakeholders.The former hold positive attitudes towards CLIL while the latter do not.
Any educational innovation should be mutually communicated, understood among all the stakeholders and then implemented smoothly.CLIL, a new educational approach, required the involvement of the authorities, universities, programme directors, learners and their parents to make it a success.Next, teachers are a vital factor affecting the success or failure of CLIL education.Both the learners' and teachers' opinions show that there is a lack of qualified CLIL teachers in Taiwan's tertiary education sector.Most of the programme providers use the easiest way to staff the teaching faculty, that is, CLIL teachers are either local content teachers with proficient English or native English-speaking content teachers.Very few of them have a background in English language teaching (ELT).Hence, it is suggested that the programme providers offer potential CLIL teachers periodic training in CLIL education, in particular, the theories of ELT and how ELT can be well integrated with their content teaching.In addition, since the popularity of CLIL is growing dramatically in Taiwan's universities island-wide, some 'CLIL Centres' (in Northern, Central, Southern and Eastern Taiwan) can be instituted to be in charge of training certified CLIL teachers, providing teachers with resources and support, conducting CLIL research and exchanging experiences with other neighbouring countries where CLIL is also rapidly gaining in popularity, such as Japan or Hong Kong.
Furthermore, since CLIL teaching is always dual-focused, CLIL assessment is supposed to be dual-focused in the classroom, too.Too often, CLIL practitioners over-emphasise the assessment of content knowledge while overlooking the assessment of learners' linguistic performance.One major reason is that the content teachers may not have the appropriate knowledge of ELT assessment, and the assessment of CLIL is especially different and complicated (Massler, 2010), so assessing leaners' linguistic performance seems purposefully avoided.Yet, CLIL learners hope to know not only if they have learned the content knowledge, but also how much their English proficiency has progressed as a result of their CLIL education.Indeed, CLIL assessment should accommodate both elements and again proper training on how to assess learners' performance in both content knowledge and language competency is needed.
Lastly, appraising the effect of managing CLIL programmes should be multi-dimensional and contextualised.In the first nation-wide CLIL appraisal, the indicators mainly focused on the curriculum structure and administrative support; however, the teaching and learning performance in the CLIL context were not included as indicators.Hence, it is argued that in future appraisals, practitioners' practice in the classroom and learners' performance should also be evaluated, as these two dimensions reflect the core values of this innovative approach.Besides, the standards of CLIL appraisal could be amended in accordance with various CLIL settings.The YANG focuses of the appraisal should be differentiated, depending on the status of the programme providers, i.e. national vs. private universities and comprehensive vs. polytechnic universities.Different programme providers have various purposes of setting up CLIL programmes, and the sources of learners are also diverse, with different entry levels of English proficiency.Catering to both contextual diversities and learners' differences in appraisals can help avoid the criticism of CLIL being an elitist approach.After all, one suit cannot fit all.

Limitations and suggestions for future research
Since the present research may be one of the very few studies investigating the implementation and effect of the CLIL approach at the tertiary level in Taiwan, some future studies can be done to complement it.First, international students' opinions can be examined.They are not all native English speakers, and in fact most are also non-native English speakers like the present participants, and are mainly from Southeast Asian countries.How they see the actual effect of CLIL can provide another perspective on its implementation in Taiwan.Next, due to the small size of the current sample, and in order to generalise the findings, more participants should be invited to join future research.The CLIL learners from polytechnic universities seemed to be more satisfied with CLIL education, so they may be able to provide better reasons for why CLIL has become popular and successful if more of them are included in a future study.Finally, to better understand why there is reluctance about the adoption of CLIL, it is suggested that researchers investigate CLIL practitioners' actual practices in the classroom.Classroom observations may unveil the real teaching performance and learning outcomes in class, which would help researchers clearly identify how content and language teaching is executed in the classroom and what kinds of supports teachers and learners may need in order to implement CLIL successfully.
My writing proficiency is increased.

5.
Overall English proficiency is increased.
□5 □4 □3 □2 □1 Hence, good communication with all the stakeholders about what CLIL is and how it differs from traditional approaches is indispensable.If learners perceive that they are placed in an educational experiment without knowing what CLIL is, or if they are told how great CLIL is and what they will achieve, they would show great dissatisfaction with CLIL if the claimed benefits are not sensed or realised.