Q&A with Vishal Srivastava, Vice President (Model Validation) at Citi

As workplaces start to open, a hybrid model—seems to be a new norm that provides flexibility for people to operate both from their homes and offices, as we emerge out of the pandemic period.

MEDIA 7: Could you please tell us a little bit about yourself and what made you choose this career path?
VISHAL SRIVASTAVA: Since my childhood, I have had a deep interest in math and science—which led me to pursue a bachelor’s in engineering degree at NIT Trichy (National Institute of Technology, Tiruchirappalli) in India. Later, to advance my knowledge, I pursued MBA and Ph.D. studies in the United States, with fully-funded university scholarships. During my Ph.D. research, I was intrigued by various applications of mathematics with which risk in engineering systems could be quantified. Thanks to my advisors Prof. Carolyn Koh and Prof. Luis Zerpa at the Colorado School of Mines, I got the opportunity to explore ideas—from first principles to machine learning—and to build risk modeling frameworks in high-pressure flow systems. As a result of my Ph.D., we were able to develop risk frameworks to be used by consortium partners that included leading global energy companies. My Ph.D. research in the quantification of risk was an intellectually stimulating experience that taught me that anything is possible if we let our focus and energy stick to a single idea over a reasonable time.

Due to the nature of my Ph.D. research—which included quantitative risk modeling—and my earlier degree of MBA in Finance, I was contacted by several risk management professionals for a potential job opportunity in the finance sector. From a mathematical standpoint, risk management in engineering and finance has a lot of overlap.  The computation of risks in engineering systems deals with investigating factors that can lead to a system failure, which can be predicted using first principles-based engineering methods or with statistical models that include the historical distribution of failure events. Similarly, credit risk management can be approached using the first principle-based mathematical methods or statistical models that forecast defaults as a function of macroeconomic or account level variables. In both cases, a binary classification model can be developed in modeling these default or failure events. I found it fascinating to explore the different avenues where a graduate study in risk engineering could be applied.

About six months before my Ph.D. defense, I had an offer from Bank of the West, BNP Paribas in model risk division. My job role as an Assistant Vice President in the Model Risk Team was to challenge the model-building process of credit and fraud risk models—both involved binary classification models. The credit risk models include a logistic regression framework which is a well-accepted industry methodology for classification and is easy to interpret. The fraud risk models included both—traditional rule-based models—and new age RNN (Recurrent Neural Network) based sequential models, which use complex and non-linear models. From this experience, I learned that from a regulatory standpoint—model explainability could be a key factor while selecting a model. This was a valuable experience, but I’ve always enjoyed challenging myself and moving out of my comfort zone. So, about one and half years later, I accepted an opportunity to work as Vice President with Citibank’s Model Risk Division in the Secured Loan team, where my responsibilities included working with the international model validation team-members to review International and US Mortgage default risk models. My focus at this job is to challenge mortgage default risk models across various continents to ensure that these models are regulatory compliant. This experience is extremely insightful due to the varied nature of credit default events across different continents as well as the homogeneity in the modeling approach towards developing a model.


M7: What are some of the means through which you select appropriate model validation methodology?
VS: In an increasingly competitive environment, financial institutions depend on models which help them optimize risks and make decisions that are well informed. Model validation managers need to ensure that every step in the model building process—data acquisition, conceptual soundness evaluation, model stability analysis, back-testing, performance assessment, model implementation testing—is well supported by a sound scientific framework. This is done to ensure that critical decisions such as loss estimates, capital allocation, and budget planning are taken based on scientific and mathematical reasoning rather than intuition. One key aspect in the whole model validation process is to ensure that the given model is compliant with the prevailing regulatory framework. In that regard, model developers present an assessment of all model usages and outputs. The performance assessment is conducted for all model usages and model outputs across all forecasting horizons. But one caveat of this process is that model risk assessment across all models can be cost-intensive. Therefore, the model review process is prioritized and models of higher importance—that are of substantial size and with significant risk contribution—are reviewed with a greater frequency. These are some of the key guidelines model validators keep in mind while performing model risk management activities.

The US economy has stayed resilient for most of 2021 when macroeconomic factors such as consumer spending and the unemployment rate have been showing promising trends.

M7: What are some of your go-to model validation techniques that help you effectively identify and manage model risk?
VS: There can be no fixed technique that can be homogeneously applied to evaluate if a model under review is totally fit for the purpose. However, at a high level, there can be some guiding principles that could be quite useful while deciding to approve or reject a model. The first check is to ensure if there has been enough analysis performed on the conceptual soundness of the final selected methodology which is proposed in the model. Here the goal is to ensure that there is sufficient evidence to justify if the selected methodology is indeed the right modeling approach. For example, for the scorecard model, one can use logistic regression, decision tree, or neural network model. In such a situation, the model validator would review if enough analysis has been performed to justify if the given modeling framework suits the given data best and if the selected model can be sufficiently explained to the regulators. 

Additionally, model developers also explore the alternative modeling framework to demonstrate why the selected modeling framework is superior to the alternative modeling methodologies. The next aspect in model validation is to find if there are any inadequacies towards analysis or model documentation. If that is observed during the validation, the same needs to be recorded in the model validation report as findings and recommendations. In the model validation report, the model validator provides a record of comprehensive documentation to record all model findings and recommendations. This serves later as a reference document for model developers when there is a need for future model enhancement. Next, model developers need to ensure if model assumptions continue to be reasonable and are based on sound theoretical appropriateness. Consequences of model assumptions violations can be expensive. As an example, during the financial crisis of 2007–2008, several modelers assumed that the housing market will continue to grow based on the historical performance and previous data. However, during the financial crisis of 2007–2008, the housing market plunged, and many assumptions of those times were violated. As a result, several companies had to face a huge financial loss. Hence, it is imperative that each of the model assumptions is carefully evaluated. Model validators also need to ensure if the data quality checks have been performed sufficiently. The goal here is to ensure a scientific approach towards data segmentation, data cleaning, data sampling methodology, missing values, and data outliers—which can severely affect the model forecasts. The model validator also needs to ensure if data sources—both internal and external (rating agencies, etc.) are well checked and properly recorded while clearly justifying all data exclusions. The model validator also needs to ensure if the model developer has performed a sound variable selection process and if all variable transformations are well documented. Many times, continuous variables are converted to a categorical variable by a process called binning, and dummy variables are created. Any discrepancy in the variable transformation in the modeling and implementation stage can lead to a big discrepancy between the modeling and production. Another very important part of the model validation exercise is model back-testing and performance analysis.  This is to ensure that model is still producing accurate forecasts even for the recent period with unseen data. As described, the three main pillars of the model validation process can be depicted as below:
 

Model validation managers need to ensure that every step in the model building process—data acquisition, conceptual soundness evaluation, model stability analysis, back-testing, performance assessment, model implementation testing—is well supported by a sound scientific framework.

Model validator reviews if the model developer has performed back-testing in OOT (out-of-time) and OOS (out-of-sample) data to ascertain if the model is still accurate when the sample is not from the data that was used in the original developmental period to rule out overfitting. Next, the model validator must ensure if the model is meeting all the necessary regulatory compliance and all the model document fully complies with the necessary regulatory requirements. Model validators also need to review model dependencies. For instance, if the output from one model goes as an input to the second model, and if there is a performance issue with the first model, the performance of the second model can be adversely affected due to model dependencies. These are some of the pointers that model validators use to review a given model. A summary of the model validation review process can be pictorially represented in the below diagram:

M7: What do you see as the most noticeable change right now happening in the workforce, encouraged by the rise of digital technologies?
VS: There is a Chinese proverb that says— “May you live in interesting times”. If we look around, we are rather living currently in transformational times that will redefine our future. Many banking tasks which earlier required physical proximity, are now being automated with digital innovations—that include advancements in computer vision and image recognition. Financial institutions have already introduced several innovative products—from automatic cheque deposits and online cash transfers to digital payments and transactions.  Additionally, the rise of digital technologies coupled with the changes due to the pandemic has brought irreversible changes in our workforce. As workplaces start to open, a hybrid model—seems to be a new norm that provides flexibility for people to operate both from their homes and offices, as we emerge out of the pandemic period.  There is an immense opportunity to retain the best parts of office culture while getting freedom from inefficient tasks and office meetings, which are unproductive.  This is resulting in the trend that commercial workplaces are moving into residential complexes as organizations are exploring new opportunities to be more efficient. We are seeing a new form of organizational agility, which is empowering teamwork across all disciplines and offshore locations. In my opinion, companies that quickly adapt to this remotely operated flexi-time organizational culture—rather than enforce the orthodoxy of 9-to-5 office-centric work—will have a clear competitive advantage in this new era of work. As digital transactions take precedence, many banking products such as payments and other forms of deposits—are fast becoming obsolete because people are able to use these applications on their cell phones. The ongoing pandemic has accelerated the adoption of automation and AI processes which were started in the pre-covid period. All these changes create immense opportunities in the financial sector in general.


M7: What are the top challenges you see for the industry in general?
VS: The year 2021 is full of changes in many aspects. First, due to the rapid increase in pandemic cases worldwide, many countries witnessed some sort of slow-down in their economy during last year. However, with the ongoing vaccination drive, and reopening of offices and workplaces, synchronous global recovery has also been witnessed in the recent period. The US economy has stayed resilient for most of 2021 when macroeconomic factors such as consumer spending and the unemployment rate have been showing promising trends. However, the unemployment rate in the US for last year was among the highest in the last several decades. The dynamics and volatility in macroeconomic drivers thus affected many modeling forecasts. This is one of the main challenges from a model risk standpoint when many traditional models don’t seem to work as well as they did during the pre-pandemic time. The rise in macroeconomic volatility in the wake of COVID-19 has increased the uncertainty in modeling forecasts. When this uncertainty is not handled in a sound manner, this could result in two things—An inaccurate forecast from a simple model or a need towards a more complex model, giving rise to overfitting problems. From a model risk validation standpoint, model complexity is a growing challenge in the current times as many products are seeing the adoption of AI and machine learning to make the best use of banking data for improving efficiencies and gaining competitive intelligence. For such models, there is a need for modelers to explain the working of the model not just the performance of the model. With greater use of AI and analytics in the model risk domain, model explainability becomes a challenge faced by modelers. However, there have been significant advancements in model interpretability aspects with Explainable AI due to techniques such as LIME (Local Interpretable Model-agnostic Explanations) and SHAP, which stands for SHapely Additive exPlanation. Nevertheless, it is a constant battle to strike the right balance between model accuracy and model explainability in the wake of regulatory requirements. From a compliance viewpoint, this could also result in an environment that requires greater regulatory intervention in the model risk domain. These are some of the main challenges faced in the model risk domain from a technical standpoint. From a human resource viewpoint, finding good talent in the model risk domain is a big challenge in current times when many technology companies are hiring data scientists for similar roles. All challenges however come with great opportunities. Financial institutions are innovating and offering products that are creative and user-friendly. The speed of innovation has improved and the future only looks more promising.


M7: When you are not working, what else are you seen doing?
VS: I love jogging and hiking in nature. I have recently finished a 100-day challenge of jogging 3 miles a day without missing a single day and I hope to take this to a next level by joining a marathon in Dallas when I move there next week. Apart from that, I love listening to podcasts on a variety of subjects. I have been recently listening to podcasts of Rich Rolls and Andrew Huberman, a neuroscientist from Stanford, who publicly presents his research about neuroscience and all the fun experiments his team performs at Stanford University. I also enjoy exploring different types of meditations and like to read about the healing effects of meditation. Other than these, I also enjoy swimming and vacationing to hilly places.

��