What is already known about this topic

• Language tests often do not combine all the skills: oral and written reception and production.

• Language tests often lack mediation according to the 2018 Companion Volume with new descriptors of the CEFR.

What this paper adds

• Detailed description of how to carry out a design of an original Initial Test of Integrated Skills (ITIS).

• AVT techniques included in the tailor-made test.

• Mediation included in the design of the integrated skills.

Implications for practice and/or policy

• Designing a tailor-made Initial Test of Integrated Skills can help language teachers better test their students’ language skills.

• The design and contents of the ITIS can be transferable and adapted to other assessment and placement tests.

TRADILEX project general framework

The present paper is framed within the TRADILEX Project (2020-2023) funded by the Spanish Ministry of Science, Innovation and Universities, aiming to study the scope and validity of audiovisual translation tools in developing the communicative competence of a foreign language.

The main goal of TRADILEX is to determine the degree of improvement in the Foreign Language (henceforth FL) learning process after including the pedagogical use of AVT as a teaching and learning tool. To this end, a thorough methodological proposal will be articulated including complete tasks which make use of diverse AVT modes such as subtitling, dubbing or audiodescription. These tasks are designed to enhance communicative competence in an integrated and differentiated manner. This methodology will be piloted with adult students in non-formal educational contexts, especially in language centres of the universities involved. In this regard, the researchers in TRADILEX will elaborate the necessary materials to conduct quasi-experimental research aiming to bring to the fore a representative indication in support of the potential benefits of this methodological approach.

The results will be reflected in the online platform that will disseminate the use of this didactic (teaching and learning) resource. Some of the tasks created within the platform are designed to be undertaken with the help of a subtitle editor and a voice recorder integrated in the web. It is expected that the benefits will then be shared among the population at large, thereby potentially promoting the development of plurilingualism among citizens that increasingly live their lives (and develop their lifelong learning) in a technological and online context. The main FL in the project will be English, given the current importance of this language in our globalised world, coupled with the necessity to make more frequent use of it.

Being TRADILEX a language related project, a common linguistic framework was required. Accordingly, the Common European Framework of Reference for Languages (CEFRL) was adopted as it provides an action-oriented approach aimed at promoting plurilingualism and describing the knowledge and skills that learners need to develop in order to communicate effectively. In a recent revision of that document (Council of Europe,2018 ) more weight is lent to the role of mediation in FL learning, understood as the language activity of rephrasing a text, orally or in writing, for the purpose of making it comprehensible to speakers of other languages. In mediation, the speaker stands at the centre of the process and collaborates with his/her interlocutors to construct meanings, helping to create the appropriate context for communication and learning to occur. In this sense, the TRADILEX Project includes mediation as a methodological basis through the AVT that students carry out in the corresponding tasks of subtitling, dubbing, audiodescription, etc., which is innovative. Additionally, we consider that the project is also in line with the most current trends in the field of FL teaching and learning and it is also fundamental from a methodological point of view. Students who perform AVT tasks in their learning become audiovisual mediators, by making accessible the video clips with which they interact, either from a plurilingual point of view (cross-linguistic translation) or considering specific needs of sensory accessibility (auditory or visual).

Teaching experience has demonstrated an increase in audiovisual communication. Most of the content produced and received daily comes from audiovisual formats and online environments. The active use of video within an Integrated Communication Technologies context (henceforth ICT), including mediation as a key element of the task, provides a high degree of authenticity to the FL learning process. By using AVT as a didactic resource, learners add subtitles and record their voices, and it will be demonstrated that learners are able to develop the necessary awareness of accessibility. They can also share their final products online with classmates, friends or family. In turn, this promotes the principles of "learner autonomy" and "learning to learn". They also may use the FL in order to facilitate the acquisition of knowledge, competences and skills, which are regarded as powerful and effective communicative tools. This AVT environment helps increase learners' interest in the FL by giving them a clear reason to communicate, including topics of their interest. This AVT environment also creates an easily recognisable, innovative, fully active, and motivating atmosphere. This provides greater self-reliance to the learner in their learning process (Incalcaterra, Lertola, & Talaván, 2018).

While until a few years ago FL classes were complemented with audio documents in order to improve students' communicative skills, nowadays it is difficult to find teachers who do not use audiovisual and technological material to illustrate the contents of each teaching and learning unit (Motteram, 2013; Torre, 2011). In this context, it is essential to develop projects that can provide pedagogical models and accurate data to bring ICTs and the educational potential of audiovisual materials and mediation to the largest possible number of users. The field of study dealing with the integration of video in FL teaching and learning has been with us for decades, since the appearance of VHS (Allan, 1985). At first, video was mainly used to work on listening comprehension; however, many other areas are now exploited, such as cultural aspects, linguistic, extralinguistic and paralinguistic information or vocabulary acquisition (, 2019; Sherman, 2003).

In recent decades, several authors have claimed that the use of video within the classroom has a very positive and motivating effect by offering variety and entertainment while maintaining the instructional character (Bobadilla, 2016; González-Vera & Hornero-Corisco, 2016; Herrero, Sánchez-Requena, & Escobar, 2018; Thaler, 2014; Vanderplank, 2010; Wagener, 2006; Wang, 2012). Thus, fragments of movies, series, commercials, short films, documentaries, cartoons or music video clips, are increasingly used more habitually and with greater ease for pedagogical purposes in FL learning (Sabaté-Carrove & Baiget, 2019). The current teaching and learning trend advocate the inclusion of video in class with active participation by students. This participation is also included in the pre-viewing stage, by exploiting the characters or the context, and also in the tasks that take place during the viewing, giving answers to questions about the content, or providing summaries in post-viewing phases (Talavan, 2007; Talaván, 2013). Moreover, the learner can manipulate the video itself thanks to the multimodal resource, and develop free and easy-to-use software for audio editing, include subtitles, or use platforms that integrate these functionalities, as is the case with Clipflair (Sokoli, 2015).

AVT is the professional and academic field in which the linguistic and cultural transfer of an audiovisual text is carried out and studied, that is, a text whose message arises from the combination and interaction between an acoustic and a visual channel (Talaván, Lertola, & Costal, 2016). Such a transfer can take place through electronic media (e.g. cinema, television, DVD and video games) or non-electronic media (e.g. theatre, opera, etc.). This broad framework of study is in turn divided into different subtypes ranging from traditional modalities (dubbing, subtitling or voiceover) to others less developed at this moment, but of growing interest, such as subtitling for the deaf, audiodescription for the blind and video game localization. This growing interest is in line with the increasing attention devoted to diversity, accessibility and inclusion demanded by 21^st century societies. The first two mentioned modalities are the most widespread: while dubbing includes a translation aimed at replacing the voices of the original actors with others that reproduce the script in another language, subtitling involves the inclusion of a written text within the image as a translation (either in FL or in the mother tongue) of the soundtrack of the audiovisual product. Today, AVT is possibly the linguistic-cultural transfer modality that has experienced the greatest boost, given that we are immersed in the global knowledge society and the digital and audiovisual revolution (Chaume, 2018; Díaz-Cintas, 2018). From the technological and software development point of view, the discipline has experienced a dizzying advance in the last decade, especially in terms of subtitling (Roales-Ruiz, 2018). For its part, it should be noted that today, much of the professional practice of this discipline is developed in online environments (Bolaños, 2018).

The didactic application of AVT to FL learning dates to the 1980s, when several educators and researchers began to analyse the use of subtitles as a support in language labs to improve various skills (Vanderplank, 1988; Vanderplank, 2016). AVT, as a didactic support resource and as an active tool, comes in multiple forms. As seen in the literature below, subtitling as a learning support resource has been one of the most researched modalities, showing that there are multiple benefits from various combinations of subtitles: interlingual or standard (FL-mother tongue), intralingual or bimodal (FL-FL). All of them facilitate FL teaching-learning and foster the development of communicative competence in an integrated way (Danan, 2015; Frumuselu, De-Maeyer, Donche, & Plana, 2015; Talaván, 2013). In the last thirty years, researchers' interest has grown, and this trend has resulted in an extensive number of studies that have demonstrated the benefits of AVT as a support in the development of various FL competences (Caimi, 2006; García-Escribano, 2017; Ghia, 2012; Holobow, Lambert, & Sayegh, 1984; Perego, Del-Missier, & Bottiroli, 2015; Vanderplank, 2016).

At the beginning of the 21st century, the active aspect of the didactic application of AVT to the FL teaching-learning environment began to be explored. Most of the modalities of AVT can be applied as a teaching-learning resource in this context and many of them are already put into practice in classrooms and online environments and have been investigated, with an increasing degree of interest, over the last 10 years (Incalcaterra et al., 2018; Sánchez-Requena, 2018). First, didactic AVT was analysed in terms of standard cross-linguistic (FL to mother tongue) subtitling, i.e., learners subtitling teacher-selected videos (Incalcaterra, 2009; Lertola, 2012; Sokoli, 2006; Talaván, 2010; Williams & Thorne, 2000). Later, other modalities such as dubbing, audiodescription or voiceover, as well as other combinations of subtitling also began to be used in teaching and research in academia (Danan, 2010; Requena, 2016; Talaván & Lertola, 2016; Talaván & Rodríguez-Arancón, 2014; Talaván & Rodríguez-Arancón, 2018; Talaván et al., 2016).

Thus, in the last decade, several studies have shown the benefits of using AVT as an active teaching resource, many of them published by researchers participating in the TRADILEX Project. Finally, it is worth noting other proposals that include either a combination of modalities (Lertola & Mariotti, 2017; Talaván & Ávila-Cabrera, 2015), proposals for integration into the curriculum (Incalcaterra & Lertola, 2014), or even their application to Content Language Integrated Learning (CLIL) environments (Couto-Cantero & Bobadilla, 2018; Fernández-Costales, 2017; Fernández-Sanjurjo, Fernández-Costales, Blanco, & M, 2019).

Several Spanish centres and researchers have participated in previous projects, but there is still no consolidated multidisciplinary group (with experts in the field of the didactic application of AVT, as well as specialists in FL didactics, in AVT and in the use of ICT in education), neither in our country nor at the European level, whose main line of work is precisely the pedagogical use of AVT. The implementation of this paper falls within the scope of the TRADILEX Project aims. The design and testing of the ITIS is a preliminary step within the project and the starting point of its ensuing research.

To sum up, this article will describe and analyse the preliminary design of an Initial Test of Integrated Skills (ITIS) within the context of the TRADILEX Project. In the following section, the main hypothesis and objectives of the ITIS design will be presented.

Hypothesis and objectives of the ITIS design

The starting hypothesis of this article is supported by the results and previous contributions of the researchers participating in this proposal, mentioned in previous paragraphs. Many of these contributions are reflected in a recent monograph written by a member of the work team (, 2019), which exhaustively reviews the literature in the area. The hypothesis consists in demonstrating that the design of a new test of integrated skills will help the assessment quality, as opposed to existing traditional assessment tests. To this end, the following objectives will be pursued falling in the intersecting areas of (i) foreign language teaching assessment standards, (ii) ICT and AVT as teaching tools, and (iii) mediation in integrated skills.

As regards the first objective, the ITIS is an essential preliminary step aiming to verify and certify that the students’ actual initial level of competence in EFL is in accordance with a B1 level (corresponding to independent users fluent enough to communicate without effort with native speakers) of the CEFRL as our chosen international standard for defining language competence in English. Moreover, this ITIS is in line with a trend in language assessment that evaluates the examinees’ abilities by integrating source reading or listening material into their writing and speaking performance in ways that simulate the cognitive, communication, and literacy demands of real‐life academic or vocational tasks (Cumming, 2013). This ITIS attempts to offer a diagnostic report of the actual English level of the participants of the project, as well as enhance authentic language‐use situations (Gebril, 2018; Plakans, 2021), as opposed to skills tested separately (Kormos, Brunfaut, & Michel, 2020).

The second objective lies in the fact that our methodology integrates ICT and AVT as didactic resources to improve the teaching and learning FL process. Also, the ITIS theoretical background builds on previous studies (Requena, 2016; Talaván, 2010) that have yielded promising results on the teaching-learning use of AVT in the FL classroom to validate and test the extent of the effectiveness of the proposed methodology. So far, most of the published results present small-scale independent studies and data obtained from short-term interventions. Additionally, unlike previous studies mentioned above whose focus is on technological aspects as software development, the ITIS design focuses more on methodological issues. Namely, these issues are based on the teaching-learning use and review that is intended to be implemented in an open access online platform for the TRADILEX Project.

The last and third objective revolves around mediation in integrated skills. Mediation, as set before, is understood as the language activity of rephrasing a text, orally or in writing, for the purpose of making it comprehensible to speakers of other languages. The audiovisual component brings pedagogical, linguistic, cultural and cognitive advantages to the teaching and learning process and it has a great educational potential specifically for young born-digital learners of a FL. Besides the revision of the Companion Volume (Council-of-Europe, 2018) in which the role of mediation in FL learning acquires a leading role, other integrated skills tests were consulted for inspiration, such as the specifications issued by the Trinity College London Examination Board (2020). In accordance with this, mediation standards and integrated skills were taken into account in the design of the ITIS B1. Therefore, instructions, suggestions, questions, and answers were elaborated in the light of mediation elements. It should be noted that this mediation was implemented and integrated in all the language skills of the ITIS. We undertook the task of designing an ITIS that, from the ground up, established a consistent methodological pattern that paved the way for the student to follow.

Participants

According to the planning and tasks distributed among all the members of the TRADILEX Project, the authors of this paper oversaw the design of the instrument (ITIS). As part of the previous tasks implemented, a 25-hour course about quantitative and qualitative research methods was completed by all the 20 members of the TRADILEX Project, in order to be better prepared to elaborate the above-mentioned instrument. Moreover, several international members of the Project took part during the process of the design when required to improve, assess or evaluate the instrument within its different phases. Their review reports will be described in the corresponding section of this article.

Other reports piloted by voluntary students from different language centres in Spain are still underway. Their suggestions and comments about the preliminary ITIS are encouraging. The results will be published in further papers as soon as they are available.

Design of the ITIS

As a starting point to design the instrument of this research, the authors counted on various models used previously and published by the researchers of the TRADILEX Project in other innovative proposals and papers related to this area of study (Martínez-Sierra, 2018; Pereira-Varela, 2014; Romero, Torres-Hostench, & Sokoli, 2011; Sokoli, 2018). Additionally, other integrated skills tests were consulted for inspiration, namely: TEAP, Test of English for Academic Purposes: Associated Examinations Board (Weir, 2014); IELTS, Writing Task 1: Cambridge English, and the specifications issued by the Trinity College London Examination Board, 2020. Nevertheless, it is worth mentioning that the ITIS has been originally and specifically elaborated for the TRADILEX Project, and it is expected that it will be used as a model for the future, as it has been receiving encouraging reports from external evaluators and peer participants. This point will be further developed in Section 2.4.

The ITIS was designed in three phases. First, the structure and sections of the template were discussed and designed in a Word document. After reaching the consensus of dividing the ITIS into 4 sections, the following order was established. To begin with, Oral Reception followed by Oral Production. Then, Written Reception and finally Written Production. Further information about the reason for this order will be delivered in the discussion section.

The topics of the contents of the tasks were selected according to appealing criteria and trying to capture the students’ attention and motivation. Some of the topics are the following: “Science is for everyone”, “5 reasons why you should sleep more”, “How whales can teach us about us”, “Make the Most of your Free Time”, “Country Kids” and “TV series”. The intention for the titles was to make them as descriptive and unambiguous as possible for the learner. Both the duration of the videos for the reception activities, and the duration of the production activities was commonly agreed among the designers of the instrument. The videos were designed to last between 1’.30’’ to 2’.00’’ bearing in mind that participants could listen to them twice. The overriding priority was for the whole ITIS not to exceed 2 hours.

Another aspect taken into account was the length of the writing tasks. The first proposal was for the student to write 200-250 words, but after consideration it was reduced to 100-150 words. In the discussion section, further details will be provided on this matter. Other technical issues were determined and agreed by the authors during the drafting process of the ITIS. The content, skills, and specific features of the template were discussed, and general instructions were presented as can be seen in Figure 1.

The second phase was directed to improve the first draft according to the mother tongue standards and the Project requirements. One of the main aspects dealt with was the number of questions and answers for each section and the type of response to be obtained (multiple choice, true/false answers, choose the correct answer, etc). Finally, it was decided that the proper number of questions for each section would be 5, with 3 possible answers. Additionally, the first draft was converted into a web-based online edition (Google Forms) to make the ITIS more appealing, easy to handle, to make it impossible for users to edit and to make it a reliable instrument (See Figure 2).

Finally, in the third and final phase, all suggestions and improvements recommended by the reviewers were considered to refine the ITIS prior to its final stage. For example, all reviewers strongly suggested the division of the instrument into 2 parts of 1 hour each. Figure 3 shows a sample screenshot of the final ITIS B1. The complete ITIS instrument can be found in the Appendix A.

Procedure

Among the TRADILEX Project members, the three authors of this paper were assigned the elaboration of the ITIS B1. Since they are established in different universities from the West, East and Centre of Spain and, due to the lockdown caused by the pandemic, it was agreed that the meeting place would be an online platform (Microsoft TEAMS). The meeting schedule would be from late January to March 2021 on a weekly basis. A total of 16 hours were devoted to justifying, creating, and deciding how to proceed according to the pre-stablished parameters of the Project.

After an in-depth bibliographic review of existing literature and reference sources, each member of the group made a proposal in line with the main elements of the theoretical foundations presented. The idea of elaborating an innovative ITIS that differed from existing traditional ones, like those issued by the Cambridge-Oxford-Trinity College Examination Board, was of paramount importance. The intention was to create a tool especially adapted to the needs of the project, the students and the scope outlined. Added to this, the ITIS had to be reliable, appealing, and appropriate for a CEFRL B1 level student. Above all, it was important that the design included integrated skills (productive and receptive, oral and written) with a special emphasis on mediation. For example, the design of the questions in the reception section included mediation as students had to rephrase a text for the purpose of making it comprehensible to other speakers. Finally, rigorous and adequate use of the English language in the final design of the ITIS was mandatory.

The three authors of this paper had regular meetings to solve and improve the design of the ITIS B1 and worked as a team throughout the process. However, each author oversaw a particular aspect of the design of the instrument, so that the researcher on AVT paid special attention to the selection and final election of the listenings, readings and audiovisual materials to be presented as part of the instrument. The second researcher undertook the task of elaborating the questions, activities and answers to be given out of the previously selected texts. Finally, the third researcher designed the basic structure of the test and tried to give coherence and integrate the different sections of the ITIS B1. A final consensus was then achieved by all the researchers at the end of the design process.

Regarding general questions and topics to deal with, both cultural and social issues were discussed, and these authors were very much concerned about selecting interesting topics to engage the participants. As already mentioned, the ITIS B1 was made out of a total of 4 sections with 2 parts for each section specifically designed to measure a B1 level of English Language Learning according to the CEFRL (2018). The maximum duration of the ITIS was 2 hours. The format of the questions as well as the answers to the questions combined multiple input and output tools, e.g., voice recording (by means of https://vocaroo.com), multiple choice questions, paragraph typing (100-150 words) and video playing (1’30”-2’ talk), as can be seen in the Appendix A.

Finally, it is also important to underline that all the materials used for the design of this ITIS were only used for educational purposes and the videos and texts used were adapted or trimmed from downloadable webpages.

Review of the instrument

Any instrument designed for a scientific study of this kind has to be validated by an expert panel and tested by a substantial number of participants in a preceding piloting phase. In the process of validation of this ITIS, which is underway, preliminary reports are being obtained from the expert panel, random report reviewers, and one technical collaborator. Timing of the test, contents, aesthetic appearance and any relevant detail to improve the design of the web-based instrument are being analysed by these parties in order to obtain the best results. The piloting stage is being implemented at the moment, and positive feedback and good results have so far arisen according to the reports from the advisors. (See Figure 5; Figure 4 below).

There are more corrections and suggestions on both reports, but most of them revolved around benefits for the sake of user-friendliness, optimisation of the web-based ITIS and other technical improvements for the interface. Since this stage of the project is still preliminary, further reports are due, and there are likely to be more changes.