Over the last decades, pilot training has been based on evidence of accidents driven by the belief that repeating tasks and manoeuvres, usually in ‘worst case’, would ensure the safe operation of aircraft. This has led to a programme increasingly crowded with new types of events, putting at risk its effectiveness.

The training method shift comes with the introduction of Evidence Based Training (EBT) that puts individual performance in the center. EBT is designed to develop crew competencies to deal with unexpected events and operate safely. The adoption of EBT for recurrent training at European level is recent and has been driven by EASA through the definition of a recommended roadmap for its progressive implementation: starting with the Mixed EBT phase to gain experience and then progressing to the Baseline EBT phase. EASA AIR OPS and AIR CREW regulations have been adapted following the European Decisions and Implementing Regulations1 in terms of requirements for flight crew competence and training methods to allow authorities to approve first the mixed EBT and then the baseline EBT, which replaces legacy recurrent training. Further work is foreseen for the expansion of EBT to initial type rating and other types of aircraft (e.g., helicopters and business jets).

1 ED Decision 2015/027/R, Commission Implementing Regulation (EU) 2020/2036 and Commission Implementing Regulation (EU) 2020/2193 and ED Decision 2021/002/R

The regulatory perspective was introduced in 2015 and operators have started to adopt it since 2017. Mature operators have acquired relevant experience and several more are on the way, according to a survey launched by EASA back in 2022, to which 9 authorities and 26 operators replied.

However, despite the progressive adoption of the programme and its maturity at industry level, one of the main challenges that stands in the way of its progress is the use of data. EBT is a deeply data-driven programme and therefore, the standardised and robust exploitation of data is fundamental to extract insights that are relevant for the continuous improvement of the pilot's competencies. 

This has been the focus of our research in the context of the DATAPP project 2, to address the adoption of EBT from the perspective of existing digital constraints and to propose solutions that push the concept forward. In this blog, we will describe some of them as well as potential solutions.

2 DATAPP is funded by the Horizon Europe Research & Innovation Programme

Empowering the use of data to implement and continuously enhance EBT programmes

Evidence stays at the core of EBT, the use of data and available digital capabilities being the main facilitators for programme design, implementation and enhancement. Training (inner loop) and operational data (inner & outer loop) drive the development of EBT programmes. 

In the context of DATAPP project, we have analysed the data related processes in which we have identified four key areas where limitations exist and where we propose a set of solutions to ease the adoption of EBT making maximum use of the data. The results summarised here reflect the information and feedback coming from the stakeholder consultation process performed under the scope of the project. The webinars and workshops held have allowed sharing the project results with the community and validating the proposed solutions through dedicated questionnaires and live surveys. The events materials and documentation delivered so far are available on the DATAPP project website.

1- Cooperation with safety department and programme’s customisation

EBT programmes are designed based on the selection and adaptation of the provided training topics and scenarios, which are then customised based on insights from the airline's operational data, the training data collected in the context of EBT and industry wide data external to the operator.

This customisation allows the programmes to better adapt their needs with the intention to develop pilot’s specific competencies for them to be prepared for any type of situation that they may encounter in operations. 

In the context of the programme customisation, the relationship between the training and the safety department is essential. The cooperation between these two departments faces limitations as there is an absence of a governance framework. To address this, proposed solutions include developing best practices specifying data sources to be shared, recommending cross-departmental initiatives, providing guidance on documentation creation, promoting training initiatives for inter-departmental familiarisation, and highlighting the importance of cross-profile personnel. With the provision of these guidelines, operators would have a reference and recommendations to establish a good cooperation between departments. This would also enhance their capacity to customise programmes in an effective manner through the safety department's input. Additionally, this would address the need to share more than serious occurrences, since the provision of these best-practices would encourage to exploiting and jointly discussing relevant safety data with an enhanced level of collaboration and integration between safety and training. 

The fact that governance is needed to share data in an effective way between safety and training also means that these departments need to communicate in a common language. There is no common taxonomy between training and safety departments. In most operators, the lack of this common taxonomy leads to an exchange of safety data that is not performed in the most efficient way. To address that, the proposed solution involves the development of best practices to standardise the taxonomy between Flight Data Monitoring (FDM) methods and EBT competencies. This includes mapping FDM events to EBT competencies, facilitating efficient data sharing between departments. By implementing that mapping, operators could smooth the sharing of data between the safety and the training departments. Furthermore, the training department would be able to easily identify which training topics should be introduced into the programmes to mitigate the risks shared by safety.

2- Evaluation of pilots and key training data gathering

Once the EBT programme has been designed and customised, the focus goes on the training assessment and evaluation, providing the key training data. The pilot's competencies are assessed by means of a grading system and through the observable behaviours defined for each of the 9 competencies. This implies the collection of grading metrics that will subsequently enable an analysis of training needs.

In this context, the main limitations are related to the complexity of the process, to the quality of the data generated and collected, and also the need for additional metrics helpful for the programmes.

One of the key limitations for this specific area is the lack of clarity on the assessment method. The current methodology for performing the assessment and the grading is standard, but open to interpretation by each operator. This fact conditions the assessment, as it becomes a subjective approach that can affect the quality of the output training data. For solving that, the proposed solution involves the provision of guidance material on how to satisfy the established Observable Behaviours. It is also vital to define in a clearer way how the assessment should be performed through the provision of guidelines for the standard application of the grading system and the assessment method and techniques, as it would contribute to avoid confusion among operators. 

Also in that context, another important limitation is the high workload of the instructors and the limited time for debriefing. Instructors are in charge of observing, recording and taking notes of everything that happens during the session to, subsequently, classify the pilot’s behaviours to determine a grade. That means that they have to deal with many things at the same time. To address this limitation regarding the workload, more importance should be given to the debriefing, perhaps even setting a reserved time slot to be able to discuss what has occurred and to end up assigning the grades with the sufficient time. 

Since the implementation of digital tools to support instructors in the process of assigning a grade to the competencies could also be useful, guidelines should be developed defining the requirements to be met by this software. The proposed solution suggests developing guidance material capturing the desirable capabilities for such tools. In addition, it is especially important to underline and capture the potential risks related to the use of this kind of tools, with the objective of preventing the excessive automation of certain processes which is another limitation.

And finally, the last limitation in this area is the lack of a metric to reflect the programme difficulty. Introducing this type of metric could be a valuable step for assessing and communicating the level of challenge or complexity associated with the programme, and it would allow to contextualise the pass-fail percentages, the grading data, and the concordance of the instructors. In addition, the aim of this metric would be to explain the potential discrepancies between the grades given by one instructor and another or the grades of a certain population of pilots.

3- Instructor Concordance Assurance Programme (ICAP)

During the research, we have seen that ICAP is possibly the most challenging area and perhaps the hardest to implement. The dimensions usually evaluated are the agreement, which reflects if instructors provide similar assessments, and the alignment, which reflects if instructors are aligned with company standards.

In the context of ICAP, operators must ensure that data generated and collected by the instructors is reliable, and therefore, measures must be taken to enhance the standardisation and concordance of the instructors. Instructors must be monitored in their assessment in terms of accuracy and homogeneity. This is done by measuring the concordance through the agreement and alignment dimensions.

In that context, best-practices are needed to define suitable and standardised metrics and methodologies to assess the agreement and the alignment through statistical methods, as the regulation does not clarify how the ICAP should work. In that way, operators would have a solid basis for ensuring the reliability of their instructors' data.

Operators consider the analysis for instructors alignment as very challenging. They usually hold exercises or workshops based on reference videos, also known as “Golden Standards”. These exercises serve as benchmarks for evaluating instructor performance and alignment with the established standards of the airline.

However, not all operators are able to develop them, given the significant costs in terms of resources and time that they entail. For that reason, one of the proposed solutions consists in the creation and provision of “Golden Standards” as reference videos to be used by any operator. These reference materials could be developed by the authorities or even by leveraging on industry’s stakeholders’ collaborations. This way, any operator seeking to implement an EBT programme could at least have generic reference videos to start assessing their instructors’ alignment.

In addition to clear guidance regarding the ICAP, operators need a tool to support the analysis of instructors' data, which allows them to intuitively draw conclusions in the form of potential improvements and mitigating measures that can be introduced into their programmes. The proposed solution to address that limitation is the implementation of a tool that allows the operators to manage the ICAP related data. One approach could be to provide recommendations for operators to develop their own tools, giving references on how to collect data, which metrics to use or which statistical methods and algorithms should be used. And as a complementary approach, the authority or a third-party could develop a tool that can be used by any operator, building upon a pre-defined template in which operators could provide concordance related data on a periodic basis to perform analysis.

Another limitation in the area of the instructor concordance is that although there are mature operators who perform normalisation of instructors' data to fairly compare the assessments, there are no indications on how this should be done. In that regard, the proposed solution is developing Guidance Material to enable operators to implement normalisation methods, defining the level of metrics and the normalisation methods to be used. And finally, the potential appearance of forced concordance is also among the limitations identified within ICAP.

That risk comes from the fact that instructors may assign certain grades just to avoid falling outside of the concordance and from the use of pre-filled templates when conducting the assessments, which should be discouraged. In addition to clearly defining how assessments should be performed, a framework of indicators or metrics should be defined and introduced to assess whether there is a forced concordance, aiming to prevent it.

4- Link with the authorities and their role in EBT

In the context of EBT, authorities are responsible for monitoring and verifying the correct functioning of the programmes. In addition, they should provide data to operators to design and improve their EBT programmes (e.g., Mandatory Occurrence Reporting, Annual Safety Review trends and Incident/Accident reports). On the operators side, they are required to provide indicators and metrics of their programmes to the authority on a regular basis or during the audits.

The relationship and communication between operators and authorities should be strengthened, as the external operational data such as regulator and industry wide data should be provided for customising the programmes. The communication should be bidirectional as operators are required to provide indicators and metrics of their programmes to the authority on a regular basis or during the audits conducted by the authority. 

As EBT implementation has been mainly driven by the operators, authorities have limited resources to keep the pace with the development. This results in not being able to provide data for EBT customisation, what translates into the operators relying on their own data for the identification of training needs. The proposed solution suggests the publication of industry best practices to encourage effective data sharing from authorities to operators, covering the data sources, frequency and methods to be considered. Also in that respect, a collaborative data-driven mechanism among authorities and operators should be incentivised. This type of initiative could provide data for the operators’ programmes at national or European level, enabling a better customisation of the programmes, even filling data gaps that some operators might have in their data.

Sharing metrics from the operator to the authority is a way for the authorities to check the correct implementation and functioning of EBT programmes. There is a need for Guidance Material on a recommended framework of KPIs for oversight, supporting the continuous evaluation of their effectiveness. This would facilitate the authority's monitoring work and would provide visibility on the programmes.

To address the current authorities’ limited capacity to exploit the data coming from the authorities having scattered data sources and the lack of staff, resources and expertise regarding EBT and data analytics, a potential solution would be supporting the definition of specific trainings for the enhancement of the authorities’ data analysis capabilities.

This current role of the authorities in the contact of EBT could be evolved and give them the possibility of ensuring the consistency of the programme once implemented. Mature operators note potential shifts in grading curves as instructors and pilots develop competencies, leading to over or downgrading. The proposed solution for that involves developing industry best practices for standardised metrics to monitor the consistency of the EBT programmes over time. The established framework or set of metrics would allow monitoring the evolution of the gradings and the concordance over time, and the flow between good and bad gradings and vice versa. This approach empowers authorities to ensure ongoing programme consistency among operators.

Digital solutions play an important role in the adoption, design and continuous improvement of operators' EBT programmes. They can ease the process, as data stays at the core of the programme. The challenges identified and proposed solutions are under discussion and represent a proposal of the researcher to EASA. Consolidated results will be shared in line with project’s progress. Keep track on DATAPP website.

About the authors

Núria Alsina

Aeronautical engineer with specialisation in air navigation and systems, certified in project management and scrum methodology. Principal at ALG and head of digitalisation and advanced analytics in the Transportation practice.

Antonio Cabeza

Aeronautical engineer with a MSc in Big Data and Advanced analytics. Specialises in strategical projects in airport and air traffic operations with wide expertise in digitalisation and regulatory related projects. Engagement Manager at ALG.

Andrada Bujor

Aeronautical engineer with a MSc in Business Intelligence and Big Data and expertise in strategic business projects, ATM research, impact assessment and digital initiatives mainly in the European context. Team Leader at ALG.

Alex Olivera

Aeronautical engineer with expertise in projects related to data analysis and information management, Business Intelligence, ATM and airports in the context of aviation and technology consulting. Consultant at ALG.