Knowledge Graph Generation and Multimodal Fusion for Reasoning on Collaboration: A Hybrid Approach Combining Graph Neural Networks, Language Models, and Logical Inference

We are moving towards a world in which software development will increasingly rely on the collaboration of AI agents. This project is the first step in forging effective collaboration between humans and AI agents. By conducting in-depth analysis of online collaboration within large communities, we propose the use of a knowledge graph and reasoning capabilities to improve task management, expertise discovery, knowledge integration and governance transparency within open-source software (OSS) ecosystems. In the near future, when AI and humans develop software systems together, this will enable the incorporation of human accountability into the development process.

Open-source software (OSS) repositories thrive as vibrant, decentralized communities where developers collectively build, refine, and disseminate knowledge. They offer large amounts of semi-structured information to build a knowledge graph (KG) that captures collaboration; to enable advanced reasoning, fusing the disparate modalities into a unified representation is required. The way to encode all types of information (conversation networks (interactions in issues/PRs), unstructured text  (comments, descriptions), code structure, task  flow, etc.) and the fusion of these heterogenous sources of information constitute the main research avenue of our project. The state-of-the-art leverages multimodal fusion frameworks [Li & Tang, 2025]) including graph-based  neural architectures [Liu et al., 2020] [Liu et al., 2023][Wu et al., 2021], and LLMs [Yang et al., 2024] to achieve  holistic, informative, and actionable knowledge graphs that enable advanced reasoning. 

We propose a hybrid framework that integrates Graph Neural Networks (GNNs), Large Language Models (LLMs), and Natural Language Inference (NLI) to construct multimodal knowledge graphs from OSS data. Our approach advances the field through: 1) Novel fusion techniques for heterogeneous data without complete retraining, in particular via model ensembling or knowledge injection techniques, including the use of existing ontologies [Smith et al., 2011] [Nundlall & Nagowah, 2022], 2) NLI-driven methods to infer causal relationships from developer conversations and relationships between issues as workflows, 3) Scalable architectures for real-world deployment, 4) the release of a public benchmark for duplicate detection and causal relationships between issues and 5) Empirical evaluation of performance (accuracy, cost, robustness) on public datasets and industrial cases (use of LLM-based multi-agent (LLM-MA) for complex problem-solving and world simulation [Guo et al., 2024]).