This study introduces an innovative approach for constructing multimodal investor sentiment indices and explores their varying impacts on stock market returns. We employ the RoBERTa model to quantify text-based sentiment, the Google Inception(v3) model for image-based sentiment measurement, and a multimodal semantic correlation fusion model to comprehensively consider the interplay between textual and visual sentiment features. These sentiment indices are further categorised into industry-specific investor sentiment and market-wide investor sentiment, enabling separate analyses of their effects on stock markets. Furthermore, we leverage these indices to build a multifactor stock selection model and timing strategies. Our research findings demonstrate that multimodal sentiment analysis yields superior predictive accuracy. Industry-specific investor sentiment exerts bidirectional positive influences on stock market returns, whereas market-wide investor sentiment indices exhibit unidirectional impacts. Integrating industry-specific investor sentiment into our multifactor stock selection model effectively enhances portfolio returns. Furthermore, combining market-wide investor sentiment with timing strategy optimisation further augments this advantage.

Quantum theory provides a comprehensive framework for quantifying uncertainty, often applied in quantum finance to explore the stochastic nature of asset returns. This perspective likens returns to microscopic particle motion, governed by quantum probabilities akin to physical laws. However, such approaches presuppose specific microscopic quantum effects in return changes, a premise criticized for lack of guarantee. This paper diverges by asserting that quantum probability is a mathematical extension of classical probability to complex numbers. It isn’t exclusively tied to microscopic quantum phenomena, bypassing the need for quantum effects in returns.By directly linking quantum probability’s mathematical structure to traders’ decisions and market behaviors, it avoids assuming quantum effects for returns and invoking the wave function. The complex phase of quantum probability, capturing transitions between long and short decisions while considering information interaction among traders, offers an inherent advantage over classical probability in characterizing the multimodal distribution of asset returns.Utilizing Fourier decomposition, we derive a Schr¨odinger-like trading equation, where each term explicitly corresponds to implications of market trading. The equation indicates discrete energy levels in financial trading, with returns following a normal distribution at the lowest level. As the market transitions to higher trading levels, a phase shift occurs in the return distribution, leading to multimodality and fat tails. Empirical research on the Chinese stock market supports the existence of energy levels and multimodal distributions derived from this quantum probability asset returns model.