Evaluating Contextualized Content and Language Integrated Learning Materials at Tertiary Level

Evaluacion de materiales de aprendizaje integrado de contenidos y lenguas extranjeras contextualizados a nivel de educacion superior Avaliacao de materiais de aprendizagem integrada de conteudos e linguas estrangeiras contextualizados a nivel de ensino superior The Content and Language Integrated Learning (CLIL) approach has been broadly adopted and extensively researched at different educational levels across European contexts. It is also becoming popular in Asian settings due to its dual focus on developing both language skills and content knowledge. It aims to empower learners with high mobility and employability in the globalized job market. However, successful implementation of CLIL in these countries has been difficult for various reasons, including lack of suitable CLIL-based curricular materials. This study reports how an English for Specific Purposes (ESP) teacher, who is also a CLIL practitioner, worked collaboratively with an English learner who has professional, industrial internship experience, to produce contextualized CLIL learning materials for tertiary education. It also examines to what extent the self-designed CLIL materials satisfy standards of good quality. In total, 47 English-major first-year students joined the study in a national polytechnic university in Taiwan. The results reveal that CLIL learners have a fairly high expectation of quality materials, and our design received 5%–25% lower agreement than the criteria. Moreover, learners’ gender, language proficiency and previous secondary school major were factors that affected how they perceive the criteria of quality and our own CLIL materials. This demonstrates complexities of designing CLIL-based curricular materials addressing learners’ individual differences (especially in terms of needs and expectations) in a polytechnic university with pedagogic implications. To reference this article (APA) / Para citar este articulo (APA) / Para citar este artigo (APA) Yang, W. (2018). Evaluating Contextualized Content and Language Integrated Learning Materials at Tertiary Level. LACLIL, 11 (2), 236-274. DOI: 10.5294/laclil.2018.11.2.4 Received: 11/09/2018 Apporved: 30/11/2018


Introduction
The Content and Language Integrated Learning (CLIL) approach has been emerging rapidly in European, South American and Asian contexts, where English is used as a second language, and where there is a national drive to pursue economic development and social modernity (Lin, 2016).Its dual focus on accommodating both content and language achievements has drawn the attention of the government sector and scholars, as it is expected not only to equip learners with proficient language skills and content-based knowledge, but also to empower them with high competitiveness of employability and mobility in the globalized economy (Lo & Lin, 2015).Extensive evidence of successful CLIL implementations and their positive effects on developing linguistic performance, content achievements or learning motivation and "can-do" attitudes have been documented at various education levels across Europe (e.g., Breidbach & Viebrock, 2012;Cañado, 2018;Marsh, 2000;Navarro-Pablo & Jiménez, 2018) and in some Asian and South American contexts such as Japan, Taiwan, or Argentina (e.g., Banegas, 2011;Ikeda, 2013;Lo, 2015;Pinner, 2013;Riddlebarger, 2013;Yang, 2015;OEASOL, 2018) in addition to the fact that learning content knowledge through a foreign language may also bring learners additional cognitive and psychological loads (Roussel, Joulia, Tricot, & Sweller, 2017).
In Taiwan, at the tertiary level, different teaching methods or strategies such as English for specific purposes (ESP) and English used as a medium of instruction (EMI) have been growing rapidly around the country in response to the requirements of globalization.The quickly emerging CLIL approach, planned as a centralized object at institutional or regional levels (Morton, 2013), has been applied with the hope of learning English and subject knowledge concurrently.Yet, ESP and EMI stand for the two extremes of language-based and content-oriented courses in a continuum of integrating language and content learning (Yang, 2016), and the course books used under each may not be suitable for CLIL education.Thus, there is an urgent need to prepare tailor-made CLIL learning materials for its learners, although preparing them may increase teachers' workload and require a balanced expertise of both content and language (Lyster, 2007).To bridge this gap, this study reports how the CLIL materials in a Taiwan polytechnic university course were designed in collaboration with a student professional with an extensive industrial experience.Most importantly, discussing the standards of quality in designing materials to meet quality CLIL standards is also very important in this study, as the evaluation of such materials is scarce in the literature.To the best of our knowledge, our design could be the first contextualized set of CLIL materials intended for a non-language subject for which it was recommended to deliver the content using CLIL in higher education in Taiwan.The evaluation of this material can shed light on the uptake of CLIL curriculum planning and material development.

Literature Review
Currently, there is a shortage of CLIL course books or materials (Coyle, Hood, & Marsh, 2010), and CLIL material development is still in its infancy (Floimayr, 2010).It is still classed as part of the ELT course book market, as they bring an innovative and profitable component, different from general ELT course books (Banegas, 2014).Yet, these international series of EFL or CLIL-driven materials without much appropriate adaptation are usually not cognitively engaging or connected to the local context where they are used because they are intended to cater to a wide range of educational settings, and are, thus, generally not suitable for integrating subject matter and language learning (Banegas, 2014;Bell & Gower, 2011;Tomlinson, 2012).Coyle, Hood, and Marsh (2010)  bate.In the study of Baecher, Farnsworth, and Ediger (2014) on identifying the patterns of CLIL materials written by EFL teachers, it has been reported that it is much more difficult to include language target than content knowledge in the materials and that the writers tend to address the learners' four language skills, grammar, and vocabulary knowledge rather than content-based educational targets.On the other hand, grammatical structure, language functions and learning strategies are often overlooked.A similar situation was further confirmed by Banegas (2015), who found that language teachers focus more on content than on language knowledge when developing CLIL-based curricular material.In Nikula's (2012) study also reveals the same problem: The activities and tasks in the designed CLIL materials tend to offer more support for content learning, whereas language teaching is less apparent.
Meanwhile, Cummins' (1992) linguistic dichotomy, called Basic Interpersonal Communication Skills (BICS) and Cognitive Academic Language Proficiency (CALP), is used to ensure the balance between linguistic demands and content.Tasks are, thus, designed to support learners to achieve the required educational targets and to make linguistic environment from context-embedded to context-reduced by using language and integrating language with content, which promises pedagogic support from cognitively undemanding contexts to cognitively demanding contexts (Banegas, 2014).Besides, to achieve a balance between the content and language components, collaboration between content and language teachers is encouraged by CLIL scholars such as Pavón-Vázquez, Ávila-López, Gallego-Segador, and Espejo-Mohedano (2015) or Prochazkova (2013).Chien (2017) also suggested collaboration between native speakers of the target language and non-native speakers when producing CLIL materials.
Compared to the significant amount of research on how to evaluate course books, in particular for EFL and ESL, there are a few well-established criteria specifically for evaluating CLIL materials.As discussed by Banegas (2014), some ESL/EFL course books, e.g., for ESP purposes, may be treated as a weak form of CLIL materials where language classes are taught by CLIL language teachers with great use of content, aiming to develop the learners' content-based language proficiency (Ikeda, 2013).However, the criteria applied to evaluate them may not precisely fit evaluation of CLIL materials because the majority of these principles focus on language elements and presentation and tend not to engage much with disciplinary knowledge.For instance, commonly used criteria are those of Ur (1996), who offers a check-list for judging grammar and pronunciation presentations, vocabulary practice, grading, sequencing, cultural and pedagogical concerns and interesting topics.Morton (2013) surveyed European CLIL teachers' practices and perceptions in finding, adapting, creating and using materials in secondary education and found that a great majority of the teachers would be willing to create their own CLIL materials, although it consumes much effort and time.Yet, they also showed great concerns about appropriateness of materials for the learners they teach in terms of both content and language difficulty in their educational or cultural contexts.
Besides, Banegas (2014) used content analysis to describe CLIL-oriented EFL course books, and found that those publisher-made materials rarely provide connections between subject knowledge and school curricula in the learners' L1, have oversimplified subject content, and place much focus on training reading skills, with little chance for learners to develop high-order thinking skills.Thus, he concludes that CLIL elements in EFL materials are superficial, and there have been a few endeavors to promote bilingual education.However, one major difficulty of conducting an evaluation of contextualized materials and needs is that it creates extra workload for CLIL practitioners (Mehisto, 2008).
Although course-book evaluation might be considered a necessary evil, there is no fixed formula or system for judging course books (Sheldon, 1988).In addition, evaluating course books helps decision-makers become agents of change and enhances their standards of professionalism within their academic communities (McGrath, 2013).Ball, Kelly, and Clegg (2015) suggest seven principles for CLIL material design, including "the primacy of task, prioritising the three dimensions of content, guiding input and supporting output, scaffolding and embedding, making key language salient, the concept of difficulty in didactic materials, and thinking in sequences" (p.176).Mehisto (2012) proposes comprehensive standards exclusively for planning quality CLIL materials, suggesting that quality CLIL materials should make the learning intentions (language, content, learning skills) and process visible to students, systematically foster academic language proficiency, foster learning skills development and learner autonomy, include self, peer and other types of formative assessment, help create a safe learning environment, foster cooperative learning, seek ways of incorporating authentic language and authentic language use, foster critical thinking, foster cognitive fluency through scaffolding of a) content, b) language, c) learning skills development helping student to reach well beyond what they could do on their own, and help to make learning meaningful.(Mehisto, 2012, p. 17-25) However, no study so far has adopted these standards to appraise publisher-made or contextualized self-designed CLIL course books and materials.Besides, as Coyle, Hood, and Marsh (2010) argue, research is still needed to investigate the effectiveness of CLIL materials from the perspectives of design and task.
Hence, the present study hopes not only to bridge this gap, but also to focus CLIL scholars' and practitioners' attention on developing and appraising contextualized bilingual education learning materials by working with a team of students with 1 year of professional industrial internship experience, instead of content experts.
To be specific, our research aims to answer the following questions:

Research participants and context
The research context is a Taiwanese national polytechnic university featuring education in hospitality and tourism.It offers a well-known DOI: 10.5294/LACLIL.2018.11.2.4 PP. 236-274 "sandwich curriculum," which requires all undergraduate students to successfully complete one year of industrial placement, either domestically or overseas in their third year.They, then, return to the university to continue with their last year of studies.The teaching materials developed in this study are intended for the 47 first-year students majoring in Applied English Language Studies (AELS), with 34 female and 13 male students aged 18.Out of these students, 85% (40) majored in English/foreign language in their secondary education while the remaining seven students (15%) were not previously language majors.
The majority (82.6%) reached an average English proficiency level of between B2 and C1 according to the Common European Framework of Reference for languages (CEFR).The Applied English (AE) department has partnered with the International College of the University, and aims to prepare graduates with a high employability and mobility by integrating content and English teaching.Thus, nearly half of the students (48.9%) will consider going overseas (to English-speaking countries) to complete their industrial placement, while 21.3% would like to stay in Taiwan, and 29.6% had not yet decided at the time of this study.
One core content course required by the College is "Introduction to Hospitality and Tourism" (IHT).This course was previously delivered mainly in Mandarin Chinese, which is the participants' L1.However, in order to attract international students, empower local students' English skills and increase their future employability and global mobility, the college working together with its affiliated departments is attempting to conduct some content courses using the CLIL or EMI approaches.
The AELS students are the first participants to use these in-house tailor-made CLIL materials, and IHT could also be their first CLIL course, differing from their previous experiences of hospitality and tourism content courses.
The course is always delivered during their first year in University, and consists of 18 weeks with 3 hours per week, with the hope of equipping them with essential knowledge on Hospitality and Tourism (H&T) before their internships.Normally, these AELS students have a relatively high command of English compared to their peers at entry, and they are required to pass CEFR level B2 before graduation.Students in a Taiwan polytechnic university mainly come from vocational high schools, and 85% of the present participants had a foreign language major in high school; the remainder had a hospitality or tourism relevant major.In other words, the majority of the current learners have a strong language background but no H&T training, whereas some have H&T knowledge with a low command of English.This situation poses challenges for combining content and language in CLIL materials.The total number of learners taking the IHT class is 47, and they were invited to join the research as well during the 18 th week, the final week of the semester.

Developing CLIL materials for the course
The initiative of designing IHT materials for the CLIL approach was raised by one Taiwanese language teacher, also a CLIL practitioner in the AELS department, who was also the lecturer of the course and the researcher of the present action research study.Instead of using the Chinese textbooks or English content books designed for native speakers, which is the approach adopted by all the teachers teaching ICH across the university, he decided to develop his own CLIL materials for the course.Authentic materials were adapted in line with the teaching goals, which is believed to be the most promising option for developing CLIL materials (Moore & Lorenzo, 2007).The aim was to develop materials with a dual focus, i.e., language and content.Usually CLIL or EMI scholars (e.g., Lagabaster, 2018) argue for a close collaboration between content teachers and language experts in teaching or developing CLIL curricula and materials.However, due to the concerns of the teachers' various mindsets, time availability and how much teachers should be paid for their involvement, such collaborative teaching or cooperation appears to be rather difficult in Taiwan.
Hence, the teacher decided to work with a final year student to develop CLIL materials, because any final year student in the researched setting who has successfully completed his/her job placements, not only has good language skills, but also clearly understands what content knowledge is really useful for AELS undergraduates to acquire during their internship.This co-developer was also employed as the research assistant in the present study.He has an English proficiency level ranging from CEFR B2 to C1.Before starting the material design, he was instructed about the principles of CLIL and which activities can be useful to activate and sustain CLIL learning by following Bentley's DOI: 10.5294/LACLIL.2018.11.2.4 PP. 236-274 (2010) suggestions for designing language elements, communicative, cognitive and learning skills in CLIL materials.
The course materials are topic-based.In total, 14 topics in 14 units about hospitality, food and beverages (F&B) and tourism were selected.This student-professional worked cooperatively with the language teacher to select appropriate reading passages focusing on each topic.
The texts were mainly retrieved from the Internet and then purposefully re-written or adapted targeting specific linguistic elements for learning.In other words, the selected course and its designed CLIL materials not only match the needs of stakeholders and learners, but also concur with the school curriculum (Evans, Hartshorn, & Anderson, 2010).
Each unit follows the same sequence and patterns to display the content.They include warm-up questions, content reading, comprehension checks, technical vocabulary practices, two communicative activities for learners to exhibit their language skills and content knowledge, cognitive activities to engage them in using higher-order thinking skills, and, finally, learning activities to foster the learning skills that are helpful across the curriculum (see Appendix B for a sample unit).In general, each unit tries to accommodate the development of the 4Cs in the CLIL framework, i.e., content, communication, cognition, and culture (Coyle, 2008).The first four units were completed before the beginning of the course, and the rest were written while the course was ongoing, so the writers had chances to modify the materials after receiving comments and feedback from both learners and instructor.This collaborative, developing model is quite distinct from the previously mentioned CLIL-focused practices in the literature review.

Instrument and analysis
After one semester of trial use, a summative evaluation of whether the self-designed materials reached the pedagogical targets and the claimed CLIL benefits from the learners' perspectives was conducted.
A questionnaire was, thus, designed (see Appendix A for the English version).Since no well-established evaluation forms had been specifically designed for this purpose, our own form was developed by adopting Mehisto's (2012) criteria for producing quality CLIL materials.The criteria were divided into general and specific principles and other re-

UNIVERSIDAD DE LA SABANA DEPARTMENT OF FOREIGN LANGUAGES AND CULTURES
quirements.The general principles can be applied to appraise course books or materials in any subject, and include principles such as guiding learners to seek other resources for learning, assisting students in building learning skills or stimulating critical and creative thinking.
In addition, 10 specific standards were added, together with their rationales and examples for writing quality CLIL materials, including to help "learners reach well beyond what they could do on their own, build a sense of security in experimenting [with] language and content, or increase the likelihood that both content and language learning will be meaningful" (Mehisto, 2012, p. 17).Moreover, quality CLIL materials should conform to other requirements, such as the consideration of technical requirements and environmental or social issues, which are also viewed as the essential norms in any quality educational material design as CLIL material "does not operate in a world of its own" (Mehisto, 2012, p. 30).

Mehisto's checklist contains many key features of what makes
learning materials effective, and may be applicable to other educational contexts, but these features place more stress on main themes in second language pedagogy and pedagogical thinking (Morton, 2013).
These principles were transformed into our question items to evaluate materials.Questions eliciting the students' judgment of the weight of content and language in the materials were also added.
The bilingual questionnaire in both Mandarin Chinese and English is composed of four main sections: the participants' demographic information (4 items), their perceptions of the quality of the CLIL materials (16 items), the evaluation of the current CLIL materials (16 items), and one open-ended question, resulting in a total of 37 questions.We were also interested in knowing if their opinions would differ according to the variables of their present English proficiency, highschool major and intended location for future internship (in Taiwan or overseas).The questionnaire was provided in an online format and its reliability reaches Cronbach's alpha .964.At the end of the semester, all the first-year students who were taught the target CLIL course were invited to take the survey.After receiving raw data from the 47 firstyear undergraduates, SPSS 21.0 was used to process the descriptive analysis, and t tests and one-way ANOVA tests were performed.The statistical significance was set at .05.

Expectations of quality CLIL materials and the current CLIL design
In the second and third sections of the questionnaire, we asked the 47 learners what makes quality CLIL materials in their view, and their judgment of the extent to which the current materials meet standards of quality CLIL materials.It was also attempted to compare and contrast differences and similarities in the two sections.

Requirement of content more than language
When judging the balance between content and language, a great majority of respondents (87.2%) believed that, in quality CLIL materials, both content and language portions should weigh equally, but they also preferred to learn more content knowledge (82.9%) than language skills (48.5%).In other words, most learners regard a CLIL course as a content-oriented course, with language learning as an academic bonus while learning the disciplinary knowledge.These responses conform to what a CLIL course should be (Fernández, 2009).However, in contrast, the learners perceived our materials as placing nearly equal stress on content and language learning.We have to agree that the current materials, differing from the authentic textbooks used in the EMI method, tried to rebalance the weight of language and content.
As the materials were written by an English educator and an English major, it is understandable that linguistic elements were treated with an equally important status as the content area in these dual-focused materials.Although CLIL claims to be dual-focused, teachers' expertise, i.e., language or content, can greatly affect how CLIL materials are designed and instructed.

Making learning intentions and the learning process visible
To motivate learners in learning, goals should be explicitly specified, and thus visible (Gardner, 1985).In quality CLIL materials, learning outcomes integrating both language performance and content achieve-

UNIVERSIDAD DE LA SABANA DEPARTMENT OF FOREIGN LANGUAGES AND CULTURES
ments are planned and expected to be not only meaningful, but also challenging (Mehisto, 2012).Nearly all the respondents (97.7%) agreed with this principle, but the percentage of those who agreed with this general principle dropped to 72.4% when evaluating the present materials.Some of the learners may have been confused about the exact targets of this CLIL course, and they were also puzzled as to why the materials looked different from what they were used to, namely, either language skills or authentic content course books.It is inferred that, before the trial, much effort should have been invested in convincing the learners that CLIL is a dual-focused and beneficial approach to help them gain both linguistic command and content knowledge, since this may be their first time learning in a content-focused course under the CLIL approach and using CLIL materials.

A need for highly focus on systematically fostering English learning and proficiency
According to Mehisto (2012), English proficiency refers to academic purposes only, but in this survey it was modified to English for Specific Purposes because the learners in a polytechnic university are supposed to use English in specific working situations.Quality CLIL materials should be able to help learners scaffold their language learning and offer supplementary situated information to assist them in processing language (Mehisto, 2012).About 93.6% of the respondents agreed with this principle in assessing CLIL materials.Yet, once again, 72.2% of them, with a 21% decline, believed the present materials satisfy this standard.The current CLIL materials were written using a topic-based design, and the language parts were integrated into context, making language use dependent on different situations and, thus, probably lacking the logical order.In other words, this material differed from the four-skills language training course books, which usually arrange linguistic elements systematically, purposefully from the easy to the difficult.The results indicate that language elements will be learnt out of order if the content is selected as the priority when designing CLIL materials, as altering the content to fit systematic language learning can be very challenging.In contrast, locating key and common language use in the content is relatively easy.

Explicitly addressing learning skills development and learner autonomy
In addition to content and language learning, CLIL, like other educational approaches, should also be able to develop learners' learning skills, which can be applied to learning other subjects as well.In addition, quality CLIL materials, similar to designing language learning materials (Cotterall, 2000), should also help learners take responsibility for reflecting on their learning content and process, as successful learners are always autonomous (Little, 1995).A total of 83% of the respondents agreed that quality CLIL materials should have this goal, but 72.3% of them believed that it was achieved in the present materials.In fact, the last activity in each unit, i.e., students watching a short online video and then answering some questions, purposefully prepares learners to practice learning skills such as note-taking or memorization.
In addition, the comprehension check after reading the passage also helps them develop the skills of scanning, skimming, summarizing or searching for keywords.However, due to the fact that learning skills are usually integrated and instructed implicitly via classroom activities, only a few respondents perceived their explicit appearance.

Lack of interest in CLIL assessments
According to Mehisto (2013), assessments can generally help improve teaching practices in addition to the learners' attitudes and engagement in learning.He, therefore, argues that quality CLIL materials can include assessments in the areas of achievement of content and language goals, achievement of learning skills goals, use of language for various purposes, ability to work with authentic materials, as well as with native and non-native speakers of the CLIL language, willingness to experiment with content and language, and ongoing growth of language.(Mehisto, 2013, p. 20) However, designing CLIL assessments is also believed to be the most challenging part for teachers due to its dual focus on simultaneous evaluation of the learners' language and content performance, and difficulties of attaining different sources of evidence (Massler, 2011).
It is interesting that the respondents' agreement with this principle is higher regarding the present materials than ideal quality CLIL mate-UNIVERSIDAD DE LA SABANA DEPARTMENT OF FOREIGN LANGUAGES AND CULTURES rials (77.6% vs. 72.3%);besides, they showed low agreement with this standard compared to the other evaluated principles.It is assumed that first-year undergraduates dislike testing after taking many formal or informal tests in their secondary schooling in order to enter an ideal university.Yet, from the teacher's perspective, assessing learners' achievement is very natural and necessary after instruction, in particular in Taiwan.These contradictory expectations of CLIL learners and material designers may bring about some potential risks or difficulties in fully implementing the CLIL approach if not considered appropriately.

Learners are not accustomed to appraising the safety of the learning environment
It is essential for educationists to offer a safe and positive learning environment in order to foster and optimize interaction among learners (Clapper, 2010).Materials should be designed so that students can freely engage in learning without fear.Jakonen and Morton (2013) claim that a safe learning environment has a great effect on learning performance, and this principle is also applied to the design of CLIL materials because in the communication process of the CLIL classroom, CLIL learners use so-called linguistic, non-verbal and paralinguistic features to achieve the educational demands set in advance.In other words, quality CLIL materials are supposed to provide appropriate navigation support and respect diversity to foster inclusion, and thus, facilitate communication (Mehisto, 2013).A total of 72.3% of the respondents believed that the present materials conform to this criterion, which is slightly lower than their 78.8%agreement with necessity of statement.The relatively lower agreement, compared to the previous standards, may result from the fact that appraising materials is not customary for learners in Taiwan; the learning materials published for use in schools are reviewed by the government and selected by teachers.Thus, students may naturally assume that all of the materials used in classrooms are safe.However, another explanation can be that, although the materials provide a safe learning environment, instructors' lectures dominate the classroom and deprive learners from the opportunities to engage in communication, which is commonly seen in university content-based courses.

Materials facilitating cooperative learning
Peer-cooperative tasks have been viewed as essential and effective for motivating learners and helping them engage in meaningful interaction in language learning and CLIL situations (Goldenberg, 2008;Coyle et al., 2010).The designed CLIL materials incorporate several tasks involving communicative activities that require learners to complete missions collaboratively.In all, 78.8% of the learners regarded this principle as being important, and 74.5% of them agreed that the newly introduced materials fulfill this standard.Yet, the diversity is very narrow.These results acknowledge that the present CLIL materials provide learners with "language of learning" and "language for learning."For example, the corrective design of the materials incorporates a professional glossary, including necessary terminology to understand the concepts of the content, and a section of communicative activities, offering students chances to successfully utilize the target language in specific situations.

Demand for audio-visual resources
In quality CLIL materials, authentic texts incorporate electronic resources to present natural language as it can be used for orientation to authentic communication and intercultural learning (Dalton-Puffer, 2007;Mehisto, 2012;Sudhoff, 2010), and thus can motivate learners to learn (Pinner, 2013).In addition, the importance of visual supports in CLIL materials is viewed by teachers as essential (Morton, 2013).The newly introduced materials include resources available in the Internet, such as online videos, pictures or websites, to raise learners' interest and get them engaged with authentic and meaningful interaction in designed activities.Altogether, 95.7% of the respondents are convinced of the importance of this principle, and 38.3% view it as extremely important.Yet, 80.9% of them regarded the newly introduced course materials as meeting this standard, which is nearly 15% lower than those who agree with its importance.Actually, the content texts are authentic, but they were adapted and rewritten purposely in order to address learners' reading level, and also to highlight "language of learning."Thus, in CLIL materials, the adaptation of authentic texts may make the content look like reading passages in ESL/EFL textbooks when language focus is addressed.This concern was also expressed

Blurred concepts of critical thinking
Quality CLIL materials should help develop learners' higher order and critical thinking skills by helping them apply, analyze, evaluate and create something on the basis of the information provided in the materials (Mehisto, 2013), as these intellectual skills are the core of CLIL learning (Meyer, 2015).A total of 85.1% of the respondents agreed with this criterion, while about one quarter (only 74.5% agreed) did not believe our design could help foster critical thinking.Indeed, each unit of the current book contains one specific section to develop cognitive skills; it contains three basic questions that can easily be answered by looking at the pictures provided, and comparing, contrasting or summarizing, and another three deeper questions that require learners to "think behind the pictures" in order to answer them by synthesizing, evaluating, or predicting.The possible causes of the relatively lower agreement can be that, firstly, the course was dominated by the instructor's lectures without allowing time for learners to raise questions, and secondly, the learners may not clearly realize the meaning of "critical thinking," and thus cannot sense its actual training when the instructor did not explicitly specify the purposes of asking the three probing questions.In addition, judging whether the materials develop critical thinking skills is also "critical," as the evaluation of these skills may not yet be standardized in the literature.
The need to scaffold through material advancing learning Mehisto (2013) and Walqui (2006) argued that, compared to their non-CLIL counterparts, CLIL learners often suffer from comparatively large cognitive load in order to simultaneously process language and content.Therefore, quality CLIL materials should be able to offer additional scaffoldings to guide students in their learning.They suggest that CLIL materials and practitioners apply strategies of modeling, bridging, contextualizing, schema building, representing text and developing meta-cognition to help learners avoid cognitive overload.In all, 95.7% of the respondents believed that quality materials should help them reach well beyond what they could do on their own, and 83% of them agreed that the newly produced materials reach this standard.
At the beginning of each unit, the warm-up questions are forwarded in order to connect the learners' previous understanding or experiences with the target content.The learning activity section is also designed to equip them with learning skills to plan, monitor and evaluate on their own.Besides, one teaching assistant who was a final year student always sat in the class and was ready to provide real examples to help elaborate and clarify domain knowledge and concepts based on his internship experience in the hospitality and tourism industry.Hence, the majority of the respondents agreed that the current CLIL materials can be deemed as quality materials.

Meaningful learning gaps in the current CLIL materials
Making learning meaningful is the last specific principle of designing quality CLIL materials.CLIL materials should create meaningful and supportive learning experiences for learners (Marsh, Mehisto, Wolff, & Frigold Martin, 2012), helping them engage in authentic and meaningful communication in meaningful situations (Coyle, 2007;Ioannou-Georgiou, 2012).The materials should connect the learners' previous learning experiences, interests, lives and communities to the new content; quality CLIL materials can exhibit relevance of the intended learning (Mehisto, 2013).As suggested by Fortune (2000), the CLIL materials built on content-based thematic concepts can help achieve this linking.Nearly all the participants (97.9%) consented to the importance of this criterion; yet, 76.6% of them felt that the present materials met the principle, showing a 20% gap between their ideal materials and how they perceived the materials developed in this study.In fact, the majority of these students had no background in hospitality and tourism, and it was their first time for them learn-

UNIVERSIDAD DE LA SABANA DEPARTMENT OF FOREIGN LANGUAGES AND CULTURES
ing a content course under the CLIL approach, whereby they need to process content knowledge through a foreign language.This makes learning a little different, complicated and challenging, and they have to concentrate well in order to become successful CLIL learners (Seikkula-Leino, 2007).It is likely that, due to their dual stress, anxiety and uncertainty, they showed relatively less agreement with the newly produced materials.Apparently, the explicit linkages, connections or incorporations built on their previous experiences and then related to the intended learning should have been more boldly highlighted in the materials in this experimental trial.

A need for inclusion of ICT and integration of social issues
In addition to the above-mentioned specific criteria used to evaluate the CLIL materials, the learners were also asked to judge if the materials match appropriate technical requirements and if they include environmental and social issues, which can be generally applied to evaluating non-CLIL materials as well.A very high percentage (93.6%) of the students agreed on the need for inclusion of technical designs, but, to our surprise, a low percentage of the respondents agreed on the necessity of including environmental issues (72.4%) and social issues (76.6%) in the materials.They also did not agree that our design complied with these principles (68%, 68.1%, and 61.8%, respectively).
As previously discussed, integrating technical or electronic resources with materials is highly demanded by both learners and CLIL practitioners (Morton, 2013).Visualized materials not only motivate learners but also help them to connect to their previous experiences and to lower negative affective loads on learning content in a foreign language (Seikkula-Leino, 2007).Apparently, the newly designed material did not reach the learners' expectations due to the designers' limited IT skills, and the fact that the current materials are formatted as handouts for trial usage instead of being a formal printed copy.
In hospitality and tourism trends, environmental and socials issues are core concerns nowadays for both product sellers and buyers.Environmental-friendliness, social equality, and mutual respect to avoid any possible destruction and discrimination are becoming core values when hospitality and tourism businesses plan or operate their products.Our materials do, in fact, provide a number of critical topics such as eco-tourism, human activities in world heritage sites, working ethics, carbon footprint of food and beverages or tours for minorities for discussion and reflection in the classroom; however, our CLIL learners were probably not used to discussing or reflecting on these issues in class under their traditional education system and learning culture, which may have led them to believe that these requirements are not very important.

English level
In addition to the descriptive analysis of the data, t-tests and Oneway ANOVA were performed to examine if any significant differences existed among different variables when the respondents evaluated quality and the newly designed CLIL material.Four variables were determined in the survey: gender, high school major, English proficiency, and students' intended internship destination.All the variables revealed statistical differences except intended internship destination.
The learners' varied English proficiency had the greatest effect on the extent to which they agreed that the criteria applied to quality and to our CLIL materials.Tables 1 and 2 tabulate those differences according to the variables.In contrast to female learners, it appears that male CLIL learners would significantly emphasize the importance of cooperative learning in quality CLIL materials.Generally, female learners are better language learners in terms of motivation, confidence or proficiency than males are (Diab, 2000), and this proposition is applied to the current participants according to their entry English performance.In other words, it is assumed that female students may have fewer difficulties and have more positive attitudes towards using the target language to learn content knowledge than male students do.Thus, it is because of extra demands on learning content through a foreign language that male learners would naturally need reliance on cooperation with peers while learning the new content that is being taught under the CLIL approach.
However, when they judged the designed materials, there was no significance between male and female students indicating that the newly provided material helped foster cooperative learning for both genders.
Those undergraduates with non-language high school majors, i.e., hospitality and tourism majors, expressed significantly higher agreement than their language-major peers with the statement that quality CLIL materials should incorporate authentic language and its use.The former students had already learnt essential hospitality and tourism knowledge taught in Mandarin Chinese in high school, and thus, they naturally expressed a high expectation that, in the CLIL classroom, the content should address authentic situations with authentic communication to realize its dual-focused aim; otherwise, content instruction may become repetitive or even redundant for them.
As Table 2 shows, differences in the learners' English proficiency was the reason for the greatest divergences in their evaluation of quality and the present CLIL materials.CLIL has been criticized as an elite approach, favoring learners with high language proficiency and priv-Quality CLIL materials help students to reach well beyond what they could do on their own.F(4:42)=2.810,p<.05 The designed materials systematically foster English proficiency.F(4:42)=2.940,p<.05 The designed materials foster cooperative learning.F(4:42)=3.512,p<.05 ileged socio-economic backgrounds (e.g., Bruton, 2011Bruton, , 2013;;Fernández-Sanjurjo, Fernández-Costales, & Arias-Blanco, 2017).Our results reveal that those learners with low English proficiency and non-language backgrounds gave relatively high endorsements of the importance of materials systematically fostering English proficiency.As discussed previously, their purpose to study in the English Department seems to be to strengthen their language skills rather than the content knowledge, which they had very likely learnt in high school.The same concern may also explain their high demand for quality CLIL materials that foster learning skills and learner autonomy.Hence, they may need peer assistance to complete the class tasks cooperatively, particularly in the area of language output.
However, in contrast, those with high English proficiency perceived that the current materials offered them a few opportunities to work cooperatively with their classmates.It is presumed that the language intended for learning in the designed materials is not challenging enough because usually these high achievers are able to complete activities and tasks individually without relying on their peers' (those with comparatively poor English command) help.These responses indicate that self-designed CLIL material designers may attempt to ease the aim and scope of content and language in order to accommodate the dual focus of CLIL.In other words, learners' different needs of the CLIL courses and their different previous majors lead to a compromise of deciding the weight of content and language in the materials.This situation raises problems for CLIL implementation in Taiwan's polytechnic universities as students in these universities mainly come from two divergent tracks, relevant majors and non-relevant majors, in vocational high schools.

Conclusion with Implications
This study investigated CLIL learners' perceptions of quality CLIL materials and to what extent they agreed that the designed CLIL materials met the above standards.A survey was designed with this purpose, and in total, 47 English-major first-year undergraduates in a national polytechnic university in Taiwan joined the study.The CLIL materials were designed with the cooperation of a language teacher and a final-year student professional who had completed his one-year industry placement.The materials consisted of 14 theme-based units mainly addressing target subject knowledge of hospitality and tourism.The teacher designer, who is also the instructor of this CLIL course, administered the questionnaire in the final week of the semester after the first trial of the material.The results reveal that, in general, the respondents highly with the principles of quality CLIL materials, based on Mehisto's (2013) principles.When judging the current CLIL materials, their level of agreement dropped by 5%-25%, indicating that the newly used materials have room for improvement, and that CLIL is still in its infancy.It was also found that the variables of learners' gender, previous major at high school and English proficiency cause a number of significant differences among groups, such as the various levels of English proficiency, expectations regarding the course and their previous knowledge of the content and the target language, mirroring not only the importance, but also the difficulties of carrying out a needs analysis in CLIL execution in Taiwan's polytechnic universities.
The study also has the following pedagogical implications.Firstly, differing from the CLIL literature, it is believed that language experts working together with student professionals who have a target language major and industrial work experience are able to design CLIL materials.Other than the team cooperation between content and language experts, collaboration with student professionals is viewed as a new alternative for developing CLIL curriculum and design materials.
The production can be contextualized to fit what the learners actually need in the job market, and can be shared with other CLIL practitioners in similar contexts (Morton, 2013).Furthermore, IT experts could also be part of the design team, as earners today greatly rely on content integrating technology resources, which will help motivate them and connect them to their previous learning experiences.
Since CLIL is still a relatively new educational approach in Taiwan's tertiary education, it is suggested that its core stakeholders, i.e., the learners, should be well informed of its implementation beforehand.Integrating content and language could be a novel idea, but the learners should have a chance to fully realize what the major differ-ences and effects of the approach will bring, compared to what they are used to, helping lessen the likely psychological loads, such as uncertainty, stress, anxiety or resistance to learning disciplinary knowledge in a foreign language.
Following the preceding implications, offering bridge programs may be helpful to reduce cognitive loads.ESP courses have been suggested by several scholars (e.g., Arnó-Macià & Mancho-Barés, 2015; Yang, 2016) as being useful to scaffold learners, preparing them to make transition from being language learners to CLIL learners.Moreover, ESP practitioners can also play a mediating role in helping CLIL learners link language and content (Taillefer, 2013).Indeed, whether there is a provision of language bridging can be a determiner to accept or reject this approach if learners have relatively low language command.Finally, similar to ESP, needs analysis is also regarded as essential and beneficial in defining programs, designing CLIL curricula and materials, and preparing CLIL teacher training (Ruiz-Garrido & Fortanet-Gómez, 2009).This task is particularly important, though challenging, when learners have various course expectations, knowledge in previous academic backgrounds and English levels.Very often, curriculum reforms or material design are based on the policy and perceived needs of educators or teachers.However, it is not so often that the major stakeholders' needs, mainly those of learners, are analyzed and accommodated.Industry needs are also, sometimes, overlooked.A holistic needs analysis including all stakeholders, before an implementation, would help reduce doubts and increase feasibility in the eyes of the public.It is speculated that students' relatively low agreement with the newly provided CLIL material matching the principles of quality materials may be partly due to the insufficient preparation of the CLIL teachers.Thus, proper CLIL teacher training, which is scarcely available now in Taiwan at tertiary level, can also be designed and offered in advance.
To the best of the researcher's knowledge, the designed CLIL materials, their trial execution and this investigation may be one of the first such attempts in a Taiwanese university.Hence, several measures can be taken to complement it in the future.Firstly, an integration of qualitative designs is doable.The majority of the explanations or arguments proposed in the discussion were based on the researcher's experiences and inferences.Thus, further interviews with CLIL learners and student designers are suggested to deeply probe their viewpoints and concerns about CLIL material design.In addition, content teachers' perceptions can also be examined since a hard CLIL course is a content course, but in the current study, they were not considered at the beginning.They can also be invited to evaluate the designed CLIL materials because they may provide different viewpoints on how quality CLIL materials should be designed and appraised.The opinions of teachers who have to select, adapt, create and use CLIL materials should also be heard (Morton, 2013).Finally, an assessment of learners' performance in content and language achievements can be performed comparing three groups: those using the newly designed CLIL-oriented materials, those taught the same course using the EMI method where an authentic English textbook is adopted and those deliver the course in Mandarin Chinese using a Chinese textbook.This comparison would produce rich evidence of the effects and the purposes of designing CLIL materials.1. Language and content teaching is equally weighted in the current CLIL materials.
2. Language element outweighs content element in the CLIL current materials.
3. Content element outweighs language element in the CLIL current materials.
4. The current CLIL materials make the learning intentions and process visible to students.
5. The current CLIL materials systematically foster English proficiency.
6.The current CLIL materials foster learning skills development and learner autonomy.
7. The current CLIL materials include self, peer and other types of formative assessment.
8. The current CLIL materials help create a safe learning environment.
9. The current CLIL materials foster cooperative learning.
10.The current CLIL materials seek ways of incorporating authentic language and authentic language use.
11.The current CLIL materials foster critical thinking.
12. The current CLIL materials help students to reach well beyond what they could do on their own.
13.The current CLIL materials help to make learning meaningful.
also maintain that the CLIL materials produced under the Teaching English to Speakers of Other Languages (TESOL) umbrella tend to overlook the balance of content and linguistic presentation, aspects of courses, modules, and units.In other words, CLIL materials should be developed in accordance with the specific context, considering local school cultures and curricula and involving the efforts of CLIL practitioners.Contextualisation in Coyle's (2007) 4Cs framework is an important concept for ensuring success of CLIL-based learning across diverse contexts.The issues in relation to the human resources eligible to write CLIL materials and the weight of presenting each focus are subjects of de-UNIVERSIDAD DE LA SABANA DEPARTMENT OF FOREIGN LANGUAGES AND CULTURES

1 .
What are CLIL learners' perceptions of the principles of quality CLIL materials?2. To what extent do the contextualized self-produced CLIL materials conform to quality CLIL materials?3. Are there any significant differences in learners' perceptions according to variables such as gender, English proficiency, previous high-school major and intended destination for internship?If yes, what may cause them?
UNIVERSIDAD DE LA SABANA DEPARTMENT OF FOREIGN LANGUAGES AND CULTURES by two respondents in the open-ended question of the survey.Another limitation of the current materials could be the lack of diversity of electronic resources, such as social network platforms, which students nowadays frequently use.This may be owing to the fact that the language teacher and the learners have limited information technology (IT) skills of integrating technology with learning materials on a large scale in the current study.Thus, in addition to content knowledge, language teachers who think of designing their own CLIL materials should be equipped with extra IT knowledge, or at least cooperate with or seek assistance from experts.

Table 1 .
t-test on gender and high school major

Table 2 .
One-way ANOVA on English proficiency

Please write down any comments or suggestions for the current CLIL materials if any.
14.The current CLIL meets appropriate technical requirements, like pictures, format, or multimedia.15.The current CLIL materials include environmental issues.16.The current CLIL materials contain social issues like the elderly, the physically challenged, minorities, gender equality or general social concerns. D.