Unleashing the Power of ChatGPT: Company Line of Business
Introduction
In the rapidly evolving fintech world, accurate classification and categorization of data by various attributes is crucial in a number of business processes like market analysis, investment strategies, and regulatory compliance. Traditional methods of classification often fall short due to the complexity and dynamic nature of the industry. But now we all see the rise of game-changing tools that are set to transform the way we approach this task: these are the much talked-of Large Language Models (LLM) like GPTs by OpenAI or LLaMA by Meta. While being AI-based chat-bots, they have already brought to the table a great variety of emergent features. We have tried and tested one of these tools in the field, and are happy to share with you what we’ve got so far. So, let's dive in!
Premise
Just to get acquainted with the API of this mighty tool we decided to assign it a simple function: let’s say we have a list of companies that customers specify as their employer when filling in a credit application form. Customers may choose the line of business for their company from a given list of values. But along with the client-provided value we need our own tool to predict this value based on the company name.
In fact, we have already been using a third-party service for predictions, but our keen interest was to find out whether the AI-driven assistant could do it better.
Models applied
To categorize the companies in this way we have utilized the ChatGPT API with the ‘text-davinci-003’ and ‘gtp-3.5-turbo’ models. The turbo model was 10x cheaper yet not that efficient as the other model: davinci was good enough in understanding the data structure that we had expected from it, and rather accurate in processing the returned value.
Testing
Step 1
First thing, we just gave a list of total 3347 companies to ChatGPT with a prompt to categorize all of them using only values from a given taxonomy. It did so, and at least completed the task without omissions, which itself was a small success. Any response is better than no response, right? But the input list was too long to check in detail.
Step 2
Moving on, we picked three random companies from the original list and categorized each of them 100 times in order to find out whether the ChatGPT prediction rate is consistent. After that we calculated the degree of prediction accuracy based on the values provided by the customers (assuming that the customer choices are 100% correct). The first two samples gained 99% average accuracy across all iterations (98% and 100% correspondingly). But the third one turned a tough nut: in some cases the AI returned a value that was not present in the prescribed taxonomy, while in others it defined a completely wrong category for the company. Anyway, the accuracy rate here was 20%, which means that at least 20 times out of 100 ChatGPT categorized this company correctly.
Step 3
After some tuning and playing around with the model settings, we decided to go around and give it another try. This time we took 100 companies and made 500 requests, with 5 requests per company name, to compare the efficiency of the AI and the third-party categorizer.
The test revealed the AI’s prediction accuracy of 52%, while its competitor scored 56%. Not that striking as expected, right? Well, actually these figures include only unambiguous cases, where the category was obvious. Yet there were some companies that implied controversy (in case we question the correctness of customer choice). If we take those into account and consider the AI’s predictions as correct, the accuracy rate then hits 73%.
And that’s where it’s getting really engaging. The OpenAI’s model has shown at least similar performance as the third-party service, not to speak of the latter case, which is really impressive and leaves room for further consideration and training.
Prompt matters?
Another curious thing that we found had to do with the prompts we used. We tried various wording, and it seemed that ChatGPT understood us better when we used the combination ‘field of activity’ in our prompts. With average accuracy ranging from 55% to 70% when speaking about ‘taxonomy’, ‘categories’ or simply ‘list of areas’ in our requests, the usage of ‘field of activity’ slightly increased the accuracy up to 75%. Again, that’s not much, but when it comes to efficiency every point counts.
Closing thoughts
As we see, the usage of ChatGPT for classifying companies in fintech holds immense potential. Classification that used to be quite a time- and labor-intensive process and implied tedious data labeling and learning for human beings, can now be taken over by AI. Given the model was trained on a huge array of text data from the Internet, the chat-bot can now be used in data categorization just as efficiently and still require no specific fine-tuning. While there are challenges and limitations to overcome, the benefits of using ChatGPT, such as improved accuracy, scalability, and adaptability, outweigh these concerns. We believe, with further research and collaboration, large language models with ChatGPT among them can continue to revolutionize the way we handle regular processes, enabling better performance, decision-making and insights in the dynamic fintech landscape.
Disclaimer
The information and methods described in this article regarding the usage of ChatGPT in fintech processes are provided for research and scientific purposes only. With this article we do not present or offer any ready-made product or solution for implementation in a real-world fintech environment. While the content of this article is based on the capabilities and potential applications of ChatGPT, it is essential to understand that the described methods may require further development, customization, and rigorous testing. Implementing any AI technology, including ChatGPT, in a production environment requires careful consideration of various factors, including data privacy, security, regulatory compliance, and ethical considerations. The creators and authors of this article, as well as OpenAI, cannot be held responsible for any consequences arising from the use or interpretation of the information presented herein.