Response to discussion on machine learning for IRB models
Go back
As the report points out, as a counterpart to the significant improvement in the prediction of IRB models, the use of ML models in this context may also lead to a certain increase in the complexity of its interpretability and explicability. That is why it will be key to find a balance between these two extremes.
We would like to emphasize that, in order to guarantee a level playing field in the market, the principle “same activity, same risk, same regulation” needs to be applied, ensuring fair competition.
For all these reasons, we welcome the initiative of the EBA to discuss the implications of the use of ML in the IRB context and we encourage the EBA to build a clear and fear supervisory scheme that goes further and allows compliance with the proposed principles to be measured and finding the balance between the advantages and the new risks that ML brings.
To respond to the questions, in terms of usage, a number of ESBG members currently already use ML models in the context of IRB.
An additional challenge could be the extension of the terms that supervisors need to review ML models in the IRB context, given the increased complexity for their interpretation and understanding.
Additionally, the recommendations that will be developed should build a clear and fair supervisory framework, which includes some detail on compliance with the recommendations, to promote legal certainty for the financial entities using ML models in the IRB context.
1: Do you currently use or plan to use ML models in the context of IRB in your institution? If yes, please specify and answer questions 1.1, 1.2, 1.3. 1.4; if no, are there specific reasons not to use ML models? Please specify (e.g. too costly, interpretability concerns, certain regulatory requirements, etc.)
ESBG believes that the possibility of using machine learning (ML) techniques in the internal rating based (IRB) models’ context represents a significant improvement in the predictive power of these ML models compared to traditional IRB models. This improvement implies a significant enhancement in the performance of the models which has a direct impact on the income statements, solvency and on the future performance of the financial entities. Hence, it is important that the financial sector can also make use of this technology and make the most of the many advantages it offers, not only for a IRB models use, but also any other area (process optimization, cash management, etc.).As the report points out, as a counterpart to the significant improvement in the prediction of IRB models, the use of ML models in this context may also lead to a certain increase in the complexity of its interpretability and explicability. That is why it will be key to find a balance between these two extremes.
We would like to emphasize that, in order to guarantee a level playing field in the market, the principle “same activity, same risk, same regulation” needs to be applied, ensuring fair competition.
For all these reasons, we welcome the initiative of the EBA to discuss the implications of the use of ML in the IRB context and we encourage the EBA to build a clear and fear supervisory scheme that goes further and allows compliance with the proposed principles to be measured and finding the balance between the advantages and the new risks that ML brings.
To respond to the questions, in terms of usage, a number of ESBG members currently already use ML models in the context of IRB.
1.1: For the estimation of which parameters does your institution currently use or plan to use ML models, i.e. PD, LGD, ELBE, EAD, CCF?
ESBG is of the opinion that the use of ML could improve the estimation of any of the parameters (LGD, ELBE, EAD or CCF). The reason why ML models are not used by for the estimation of each of these parameters in the case of some institutions is because the identification of risk drivers for them may not need to be as granular as the ranking method.1.2: Can you specify for which specific purposes these ML models are used or planned to be used? Please specify at which stage of the estimation process they are used, i.e. data preparation, risk differentiation, risk quantification, validation.
Institutions us ML models in the stage of risk differentiation, for the purposes of giving support to the approval and monitoring of credit risk exposures.1.3: Please also specify the type of ML models and algorithms (e.g. random forest, k-nearest neighbours, etc.) you currently use or plan to use in the IRB context?
One of the ML models and algorithms currently used in IRB context by our institutions is the GBM (Gradient Boosting Machine).1.4: Are you using or planning to use unstructured data for these ML models? If yes, please specify what kind of data or type of data sources you use or are planning to use. How do you ensure an adequate data quality?
At the moment we haven’t noticed any intention to use unstructured data for ML models in the IRB context. However, this should not be dismissed, at least in a research context. The reason why an institution does not currently use unstructured data for ML models in the IRB context is because the need has not arisen, since the use of structured data may have been sufficient for the current intended functions. It should be noted that the use of unstructured data may be beneficial in areas other than the IRB models (for example, in recoveries), although the research is at a very early stage.2: Have you outsourced or are you planning to outsource the development and implementation of the ML models and, if yes, for which modelling phase? What are the main challenges you face in this regard?
Our members generally develop and implement ML models internally. At this time, there we haven’t noticed any intention to outsource any of the parts or phases of the process on the development and implementation of the ML models, which have been integrated within the entities themselves.3: Do you see or expect any challenges regarding the internal user acceptance of ML models (e.g. by credit officers responsible for credit approval)? What are the measures taken to ensure good knowledge of the ML models by their users (e.g. staff training, adapting required documentation to these new models)?
Some of our members have already been using ML models internally for estimating credit risk for a few years already. In their development and implementation, the main challenge was the technological adaptation of all lines of work and teams, and the development and acquisition of proper skills to work with the results of ML models.4: If you use or plan to use ML models in the context of IRB, can you please describe if and where (i.e. in which phase of the estimation process, e.g. development, application or both) human intervention is allowed and how it depends on the specific use of the ML model?
We believe that human intervention should be always present in all the phases IRB models involve and that there are multiple human control interventions at each step of the process; during the development of the models (ex-ante) and once implemented (ex-post). For example, in the development phase there must be an analyst who builds, reviews and validates the model. Also, in the application phase there is a person to evaluate the final results of an operation and is capable of giving an opinion or acting against the ML model result.5. Do you see any issues in the interaction between data retention requirements of GDPR and the CRR requirements on the length of the historical observation period?
We do not see issues in the interaction between data retention requirements of GDPR and the CRR requirements on the length of the historical observation period, since the latter are covered by a regulatory provision. In any case, the data retention requirements of GDPR are the same regardless of whether it is ML models or traditional models.6.a) Methodology (e.g. which tests to use/validation activities to perform).
Some ESBG members have already been using ML models internally for estimating credit risk for a few years In its development and implementation, the main challenges were having the necessary technological resources and the technological adaptation of all lines of work and teams. However, institutions have not experienced significant challenges in the use of ML models for estimating credit risk in relation to methodology and traceability, and overall we believe that ML models do not cause more issues than traditional models in this regard.6.b) Traceability (e.g. how to identify the root cause for an identified issue).
Some ESBG members have already been using ML models internally for estimating credit risk for a few years In its development and implementation, the main challenges were having the necessary technological resources and the technological adaptation of all lines of work and teams. However, institutions have not experienced significant challenges in the use of ML models for estimating credit risk in relation to methodology and traceability, and overall we believe that ML models do not cause more issues than traditional models in this regard.6.c) Knowledge needed by the validation function (e.g. specialised training sessions on ML techniques by an independent party).New textarea
Some ESBG members have already been using ML models internally for estimating credit risk for a few years In its development and implementation, the main challenges were having the necessary technological resources and the technological adaptation of all lines of work and teams. However, institutions have not experienced significant challenges in the use of ML models for estimating credit risk in relation to methodology and traceability, and overall we believe that ML models do not cause more issues than traditional models in this regard.6.d) Resources needed to perform the validation (e.g. more time needed for validation)?
Some ESBG members have already been using ML models internally for estimating credit risk for a few years In its development and implementation, the main challenges were having the necessary technological resources and the technological adaptation of all lines of work and teams. However, institutions have not experienced significant challenges in the use of ML models for estimating credit risk in relation to methodology and traceability, and overall we believe that ML models do not cause more issues than traditional models in this regard.7: Can you please elaborate on your strategy to overcome the overfitting issues related to ML models (e.g. cross-validation, regularisation)?
To overcome the overfitting issues related to ML models, institutions in some cases have developed a methodology to guarantee a monotonic relation between dependent and explanatory variables. Furthermore, it is also guaranteed that this relation is consistent with economic theory or intuition. This approach eases the explicability of results. In this respect, it is essential that the model has a business understanding, so overfitting may be dismissed in exchange for a better understanding of the model from the team of users.8: What are the specific challenges you see regarding the development, maintenance and control of ML models in the IRB context, e.g., when verifying the correct implementation of internal rating and risk parameters in IT systems, when monitoring the correct functioning of the models or when integrating control models for identifying possible incidences?
ESBG believes that that the greatest challenges regarding the development, maintenance and control of ML models in the IRB context are the knowledge and the technological resources of teams, both in the development phase and in the implementation phase. With regard to traceability, we think that ML models do not bring more issues than traditional models, although it is true that the explainability and fairness frameworks are new and specific for ML models.9: How often do you plan to update your ML models (e.g., by re estimating parameters of the model and/or its hyperparameters) Please explain any related challenges with particular reference to those related to ensuring compliance with Regulation (EU) No 529/2014 (i.e. materiality assessment of IRB model changes).
Our members using ML models have a model monitoring and governance framework, which makes it possible to observe a performance deterioration of the models and assess the need to update them. In this way, institutions do not have a predetermined update frequency, but updates are carried out when a deterioration in the model and a need for them is observed. This framework aims to develop stable models and update them only when necessary, in the same way that the consultation paper already recommends.10: Are you using or planning to use ML for credit risk apart from regulatory capital purposes? Please specify (i.e. loan origination, loan acquisition, provisioning, ICAAP).
ESBG believes that the opportunities for using ML models within the credit risk area and their advantages are endless. Some of our institutions already use ML for credit risk for purposes other than regulatory capital such as admission, monitoring, alert models, recoveries and defaulting, etc.11. Do you see any challenges in using ML in the context of IRB models stemming from the AI act?
Most of the obligations included in the IA Act that could be required to financial entities for their use of ML models in a IRB context were already set out in other applicable legal texts (for example, governance, monitoring or human intervention), so that they do not bring new challenges. It is true that the requirements regarding explainability and fairness are novel, however, we think they will not cause remarkable issues to institutions.12. Do you see any additional challenge or issue that is relevant for discussion related to the use of ML models in the IRB context?
We believe that an additional challenge or issue that is relevant for discussion related to the use of ML models in the IRB context is the classification of changes as material changes or non-material changes and the notification and inspection requirements that this entails. The use of ML models does not arise new problems in this respect than those that already exist today, but perhaps it does make them more complex. Adding more variables in the ML models can generate more doubts about the designation of changes as material or non-material.An additional challenge could be the extension of the terms that supervisors need to review ML models in the IRB context, given the increased complexity for their interpretation and understanding.
13: Are you using or planning to use ML for collateral valuation? Please specify.
We do not dismiss using ML for collateral valuation in the future as this could also improve processes in the area. However, at least some ESBG members, currently nor use nor plan to use ML for collateral valuation because currently they may also rely on appraisal values and updating through indexes.14. Do you see any other area where the use of ML models might be beneficial?
In ESBG’s opinion, the use of ML models could be beneficial in any other areas (e.g. process optimization, cash management, etc.). Some of our members are already using ML models in other areas such as recoveries, defaulting, creation of alerts, etc.15: What does your institution do to ensure explainability of the ML models, i.e. the use of ex post tools to describe the contribution of individual variables or the introduction of constraints in the algorithm to reduce complexity?
To ensure explainability of the ML models, our institutions using ML develop an explainability report and a support tool for each model.16. Are you concerned about how to share the information gathered on the interpretability with the different stakeholders (e.g. senior management)? What approaches do you think could be useful to address these issues?
In relation to the reporting of information on the interpretability of IRB models to the different stakeholders, we are no more concerned with what has to do with ML models than with traditional models and we believe that the challenge is the same. In general, to guarantee disclosure, banks are willing to dismiss the most technical aspects in exchange to ensure that the implications of each variable are well understood.17: Do you have any concern related to the principle-based recommendations?
ESBG welcomes the initiative to develop principle-based recommendations for ML models in IRB context. However, we think it is important not to over-regulate weak AI so it would prevent the private sector from developing their own AI applications, leaving the digital market to other countries. Besides, a level-playing field needs to be ensured and the principle “same activity, same risk, same regulation” to be applied, ensuring fair competition since these methodologies entail a significant improvement in the performance of the models which has a direct impact on the income statements, solvency and on the future performance of the entities.Additionally, the recommendations that will be developed should build a clear and fair supervisory framework, which includes some detail on compliance with the recommendations, to promote legal certainty for the financial entities using ML models in the IRB context.