LLMs in Investment Research (III) - Summarisation of Earnings

neuralgap.io

LLMs in Investment Research (III) - Summarisation of Earnings

An earnings note serves as a vital tool for investors, distilling a company’s financial performance and future prospects into a concise yet comprehensive document. And more often than not, an analyst is under a lot of pressure to deliver this in a very short span of time. Crafting an effective earnings note requires a delicate balance of financial acumen, industry knowledge, and forward-thinking analysis. Even with the advent of modern LLMs, many challenges remain in supporting an analyst with the creation of these summaries. We explore the essential components of an earnings note, including key financial metrics, business highlights, management commentary, and risk factors, while also delving into the challenges LLMs face when attempting to summarize these reports. From limitations in recognizing relevant information to biases towards academic writing styles, the article examines the hurdles that must be overcome for LLMs to effectively mimic the work of experienced financial analysts.

What should an Earnings Note Capture?

An earnings note is a critical piece of investment research that summarizes a company’s financial performance and provides insights into its future prospects. To effectively inform investment decisions, an earnings note should generally capture the following key elements:

Financial Performance

Income statement, balance sheet, and cash flow-based analysis like revenue, earnings per share (EPS), and other key financial metrics
Comparison to previous periods and analyst expectations
Breakdown by business segments or product lines

Business Highlights and Management Commentary

Major achievements, milestones, or challenges during the reporting period
New product launches or significant contracts
Market share information
CEO and CFO statements on the company’s performance
Insights into the company’s strategy and future outlook

Guidance and KPIs

Forward-looking statements on expected performance and updates to full-year or next quarter forecasts
Industry-specific metrics relevant to the company’s performance

Risk Factors and Analyst Expectations

Any new or evolving risks that may impact future performance
How the results compare to consensus estimates
Key risks to investment conclusion
After-hours or pre-market trading reactions, if available

Moreover, we have to keep in mind that to extract the above core points, the analyst will source information from a few key documents:

Formal filings annual reports quarterly reports
Earnings presentation which usually contains extra information about revenue drivers
Earnings press release similarly containing key financial results and management quotes
Earnings call transcript which are the verbatim record of the management’s presentation and Q&A session with analysts
Supplementary financial data detailed breakdowns of financial performance, often in spreadsheet format
Previous guidance and analyst estimates
Relevant industry news and market conditions

A point to remember is that not all notes will cover all of these details, but will tend to have several of these as relevant to the company and industry. Most importantly, research analysts who create the notes will exercise ‘recognition of relevance’ when inserting this detail into the notes.

Challenges for LLMs when Creating Earnings Notes

When creating an earnings summary note, a key skill an analyst exercises is the recognition of relevance of a particular bit of information and whether that piece of information should make it to the note. Analysts also focus on identifying trends, understanding the implications of financial data, and connecting current performance to future projections. Essentially, an earnings note is a summarized snapshot of the most relevant information on a company that must be optimally crafted to manage an investor’s/portfolio manager’s short attention span in conveying only the information. Subsequently, this task is also one of the most challenging for an LLM to achieve. Let’s break down the core issues as to why this is so.

- Multi-headed Attention Limitations: This is the mechanism which LLMs use to focus on different parts of the input simultaneously. However, this can sometimes lead to a lack of coherence in understanding the overall context. As a result, they may miss subtle but crucial connections between different pieces of information in an earnings report. For example, an LLM might fail to connect a slight decrease in profit margins with a newly mentioned supply chain disruption, whereas a human analyst would quickly make this association.

- Generalized Training vs. Specialized Knowledge: Most LLMs have been trained on a vast generalized space of information. While this provides them with broad knowledge, it can make it challenging to discern what’s truly relevant in a specialized field like finance. Financial relevance often requires deep industry knowledge and an understanding of market dynamics that go beyond general language patterns. An experienced analyst might recognize the significance of a minor shift in inventory levels, while an LLM might overlook this detail due to its generalized training.

- Context Limitation: LLMs have a maximum ‘inferable’ context, which means that even though they can process large amounts of text, they may struggle with maintaining coherence and relevance across very long documents. In practical terms, this means that information presented early in an earnings report might be ‘forgotten’ or given less weight when the model reaches the end of the document. This is particularly problematic for earnings notes, where information from the beginning (like revenue figures) needs to be held in context while processing later sections (like future guidance).

- Academic Writing Style Bias: LLMs often exhibit a tendency towards more academic, retrospective writing styles rather than the forward-looking, interpretive format typically employed in financial analysis. This bias stems from several factors:
  - Training Data Composition: The training corpus frequently includes a higher proportion of academic texts, research papers, and historical analyses compared to forward-looking financial reports.
  - Risk Aversion in Training: Many LLMs, particularly those developed by major tech companies, are trained with filters and guidelines that prioritize factual statements over speculative ones.

- Temporal Context Limitations: LLMs often struggle with maintaining a consistent temporal perspective, especially when asked to project future outcomes based on current data. This limitation can result in a preference for describing past events rather than extrapolating future trends.
- Lack of Domain-Specific Inference Training: While LLMs possess broad knowledge, they may not be specifically trained in the types of inferences and projections that financial analysts routinely make.

Mitigating these Challenges

There is no actual way to circumvent all the above issues as it requires true human-level intelligence and experience (thankfully!). That being said, there are several techniques and methods that can be employed to improve LLM performance in summarizing earnings notes and producing more analyst-like outputs. These approaches aim to address the specific challenges we’ve identified:

- Few-shot Prompting: This technique involves providing the LLM with examples of well-written earnings note summaries before asking it to generate one. By including samples that demonstrate the desired balance of retrospective analysis and forward-looking statements, as well as showcasing examples that highlight key financial metrics and their interpretations, the LLM can better understand the expected output format and content.

- Tree of Thought for Aggregation: Implementing a tree of thought approach can help break down the earnings note analysis into separate “branches” of reasoning. Each branch could focus on a different aspect: financial metrics, business highlights, management commentary, etc. By aggregating these separate threads of ‘thought’, a more comprehensive and coherent summary can be created. This method can help address the multi-headed attention limitations by ensuring all crucial aspects are considered.

Fine-tuned Models: Fine-tuning is a process where a pre-trained language model is further trained on a specific dataset to adapt it for a particular task or domain. In the context of earnings note summarization, fine-tuning LLMs can significantly improve their performance by tailoring their capabilities to the specific nuances of financial reporting and analysis. This process involves exposing the model to a large number of earnings reports and their corresponding expert-written summaries, allowing it to learn the specific language, structure, and analytical approach used in financial analysis. However, it’s crucial to note that fine-tuning is not a trivial task. It requires significant computational resources, a large, high-quality dataset of earnings reports and summaries, and expertise in machine learning to avoid issues like overfitting or inadvertently introducing biases. The challenge lies in striking the right balance between specialization and maintaining the model’s general capabilities.

In conclusion, while LLMs have made significant strides in natural language processing and generation, the task of summarizing earnings notes presents unique challenges that highlight both the capabilities and limitations of these AI systems. The complexities involved in recognizing financial relevance, maintaining context over long documents, adopting appropriate writing styles, and providing forward-looking analysis underscore the continued importance of human expertise in financial analysis. We see the responsibility of the final checks always requiring human input, but the integration of LLMs and broader AI have the ability to drastically cut down the effort and time needed to turnaround these notes.