KGs for Financial Reasoning

🧪 Active Investigation Do knowledge graphs meaningfully improve LLM numerical reasoning over financial documents?

CURRENT POSITION LLMs are not inherently bad at math; failures mostly come from poor grounding, structure loss, and information selection in long, messy documents. Providing a structured world model (e.g., a knowledge graph) before reasoning materially improves reliability for multi-step numerical tasks.

KEY ASSUMPTIONS LLMs struggle with numerical and multi-hop reasoning when information is only in unstructured text. Financial documents contain implicit structure (tables, periods, units) that is lost when flattened to text. A domain-specific schema can capture most relevant financial facts needed for reasoning.

SUPPORTING EVIDENCE The arXiv paper 'Structure First, Reason Next' reports ~12% relative improvement on FinQA using a KG-enhanced pipeline. Empirical result in 'Structure First, Reason Next' shows ~12% relative improvement on FinQA when using a KG, suggesting structure and grounding, not raw computation, are the bottleneck. FinQA benchmark paper shows models often fail due to wrong number selection and multi-step reasoning over tables and text, not pure arithmetic.

OPEN QUESTIONS How well does this approach generalize beyond FinQA or beyond financial documents? Is the cost and complexity of building the knowledge graph worth it compared to just using larger or better LLMs? How robust is the pipeline to extraction errors when building the KG?

WANT TO EXPLORE DEEPER? Read Full Thought → by parag