Can You Beat the Market? — The Cracks in the Efficient Market Hypothesis

There was once a proposition in the world of investment that was almost akin to faith: nobody can consistently beat the market. It means that stock prices already reflect all known information, making tomorrow's price as unpredictable as a coin toss. This idea is the Efficient Market Hypothesis (EMH). The theory that prices fluctuate randomly is the random walk. Beating the market here means consistently generating excess returns that outperform the average returns of the entire market (even after adjusting for risk). According to this hypothesis, even a fund manager's dazzling returns are ultimately just luck.

However, scholars who looked closely at the data began to discover peculiar cracks. The proposition that "the market is efficient" shook in front of the data.

There is one thing to clarify in advance. Almost all the papers cited here are preprints (arXiv) released before peer review. Therefore, not a single line should be read as a final conclusion. This very caution is part of this story.

Myth — Why We Believed You "Cannot Win"

The core of the Efficient Market Hypothesis is simple. If prices "fully reflect" all information, it becomes nearly impossible to trade on that information and make a profit. EMH is summarized by the proposition that "prices fully reflect the information held by traders (people who trade)." Its core depends on the process by which information flows into prices through trading strategies (arXiv:2501.07489).

There have also been attempts to sharpen the definition of efficiency more rigorously. One study pointed out that efficiency as "the absence of weak arbitrage opportunities" is not enough, and formulated that adding the condition that prices react sensitively to information flow becomes a necessary and sufficient condition for informational efficiency (arXiv:1302.3001). Arbitrage is a trade that makes a risk-free profit by exploiting price differences of the same asset.

There were also attempts to measure efficiency using information theory. When an efficiency metric was created using entropy (a measure of how hard something is to predict), it was equivalent to the classical definitions organized by Fama and Jensen, namely that prices reflect information (arXiv:1812.02371). A metric reflecting correlation structures and collective behavior (herding, where people follow each other) was applied to 41 stock indices. In this way, "how far [markets] are from an ideal efficient market" was measured as a distance (arXiv:1208.1298).

There is also a provocative theorem that pushes this conventional wisdom to the limit. It is a proof that if the market is weak-form efficient, then P=NP holds. Weak-form efficiency means that information from past prices is already fully contained in current prices. P=NP is the proposition that problems whose answers do not take long to verify also do not take long to find. However, P≠NP is very likely. Then, conversely, the conclusion comes out that markets are probably not perfectly efficient. The longer the time series and the higher the frequency, the greater the inefficiency becomes (arXiv:1002.2284).

One study defined efficiency using information theory. It views a market as efficient when the information that tradeable past data gives to the current price is zero. Under this definition, it was shown that a market with a GARCH (a structure where volatility varies from period to period) return process cannot be efficient (arXiv:0903.2243). Under this definition, perfect efficiency is difficult to hold. This article uses EMH not as an absolute truth, but as a benchmark for comparison.

Crack ① — Unexplained Rules, Anomalies

The real crack came from the data. If the EMH is right, no simple rule can consistently generate excess returns. Yet empirical studies have repeatedly found such rules. These regular patterns that cannot be explained by the efficient market hypothesis are called anomalies.

Momentum is a representative example. Stocks that have risen recently tend to keep rising for a while. Momentum also appears in factors, which are bundles of stocks sharing the same characteristics. Factor momentum has been reaffirmed across 72 factors. It worked both in the directional sense where rising factors keep rising, and cross-sectionally by comparing factors at the same point in time. We control for stock momentum and factor exposure. Then, statistically significant Sharpe ratios (returns per unit of risk) remained only when including the most recent month's return. This means the effect is concentrated in a short time lag, in other words, the near past (arXiv:2009.04824). The results of applying a physical model called 'symmetry breaking' to the momentum strategy have also been reported. It showed stronger performance and better risk metrics than simple momentum in US and Korean market data (arXiv:1107.5122).

Value is also a persistent signal. Robert Shiller's cyclically adjusted price-to-earnings ratio (CAPE) is a value dividing price by average earnings over multiple years. A model relying on the point that this ratio predicts long-term returns theoretically derived a linear relationship where the lower the initial CAPE (meaning the cheaper it is), the higher the subsequent long-term expected return. The momentum component can produce bubbles and crashes in the short and medium term. Even so, the long-term anchor is ultimately whether a stock is cheap or expensive compared to its earnings and assets, in other words, valuation (arXiv:1204.5055). Empirical evidence has also been added that a multivariate model combining value indicators such as dividends, earnings, and book value predicts future returns (arXiv:2110.01873).

There are also small-cap and low-price signals. 167 penny stocks (stocks with very low prices) on the National Stock Exchange of India were analyzed. The returns of four stock groups were significantly higher. They are stocks with low market capitalization (the total market value of a company), low P/E and P/B (values dividing price by earnings and net assets), and low prices (arXiv:1610.01338).

Anomalies were also hidden along the time axis. In intraday patterns, returns continued in the same direction every 30-minute interval, which is an exact multiple of the trading day. This pattern persisted for at least 40 trading days. It was a pattern not explained by trading volume or the bid-ask spread (arXiv:1005.3535). Calendar effects, where returns are skewed toward specific months, were also weakly detected. In the monthly returns of US and Turkish indices, Turkish stocks in particular showed a pronounced negative effect in May after having positive returns in April (arXiv:2003.05750).

New types of factors continue to be discovered. The intangible investment (investment in invisible assets) intensity factor had weak return predictability from 1963 to 1992. In the 1993–2022 period, it became stronger (arXiv:2505.16336). A long-short portfolio was created by buying stocks with high cyber risk measured from disclosure statements and selling stocks with low cyber risk. This portfolio yielded a significant risk premium of 6.93% per year (arXiv:2402.04775). When a network between stocks is built with over 1 million news articles, that 'network attention' predicted monthly returns cross-sectionally (arXiv:2108.05721). Whether this excess return is inefficiency or compensation for risk cannot be discerned from the abstract.

Crack ② — Human Behavioral Biases

If anomalies are evidence that "there are patterns in the market," behavioral finance says that "the cause of those patterns is human irrationality." Behavioral finance is a field that explains the market through investors' psychological biases.

The most famous is the disposition effect. It is the irrational tendency to hastily sell stocks that have gone up to lock in gains, while holding onto stocks that have fallen because one refuses to admit the loss. The disposition effect appeared even in cryptoasset markets like Bitcoin. However, it was not consistent throughout the entire observation period. It became distinct after the 2017 surge and crash (arXiv:2010.12415).

Overreaction and underreaction are also central. If information moves prices by more than it should, it is an overreaction; if by less, it is an underreaction. Actual return distributions show high kurtosis and fat tails. This means the center is sharp and extreme returns occur more frequently than in a normal distribution (a bell-shaped distribution). One model assumes that shocks from events and information follow a normal distribution. At this time, the return distribution was explained by a response function that underreacts to small information and overreacts to large information. This asymmetric response function was observed in Chinese, US, and Japanese indices (arXiv:2312.02472). The predictive power of investor sentiment showed different time lags depending on stock characteristics. In small-cap stocks and low book-to-market ratio stocks, strong predictive power appeared in both the short and long term (arXiv:1803.02962). LLMs are large language models like ChatGPT. Even this AI showed over-extrapolation, placing excessive weight on recent performance like a human when interpreting past returns. This is the tendency to view recent trends as if they will continue into the future. The predictions looked more optimistic than actual returns (arXiv:2409.11540).

Bubbles and crashes are generated in models not only from irrationality, but also from efficient mechanisms coupled with herding and optimism. In one dynamical model, individual investors adjust their opinions by combining private information and surrounding behavior (herding). When expectations were in balance, efficient prices emerged even in this model. Only when optimistic beliefs became dominant did a bull market form even if the majority held pessimistic information, ultimately leading to bubbles and crashes (arXiv:1106.1577). There is an even more surprising case. One sociological analysis took price bubbles as a case study. The market met textbook efficiency conditions: complete information, atomic actors, and the absence of uncertainty. Yet it reported that patterns consistent with the institutionalization process emerged rapidly (arXiv:1609.04629). There are also attempts to model irrationality. When a deep learning model learned "irrational factors" such as market sentiment, speculation, manipulation, and psychological biases, return prediction reportedly improved (arXiv:2502.04737).

Compromise — Efficiency Varies by Context

Then, is the market efficient, or inefficient? The most persuasive answer was "it depends." There is a view that efficiency is not a fixed property, but changes over time and across markets. This is the adaptive market hypothesis (AMH).

In pre-war and wartime Japanese stock markets analyzed with a new index, the degree of efficiency fluctuated over time and according to major historical events. The authors viewed this as a result supporting the AMH. As the war risk premium rose in the 1930s when government intervention strengthened, efficiency fell (arXiv:1911.04059). When looking at the Japanese TOPIX and TSE2 through an AR model that changes by period, the degree of efficiency varied over time in both markets. However, the efficiency of the TSE2 was lower than the TOPIX for most periods, and only the TOPIX evolved in efficiency while the TSE2 did not. The authors concluded that the AMH is supported in the better-performing market (TOPIX) (arXiv:1207.1842). In the cryptocurrency market (Bitcoin and Ethereum) as well, efficiency changed over time. For most periods, the efficiency of Bitcoin was higher than that of Ethereum, and efficiency was evolving in the market with high liquidity (how easy it is to buy and sell). The authors concluded that the AMH is supported in the most established cryptocurrency market (arXiv:1904.09403). The return predictability of the Chinese market grew stronger around periods of market upheaval (arXiv:1611.04090). The efficiency of major world markets viewed through permutation entropy (a measure quantifying the randomness of returns) was stable only on a scale of a few months (arXiv:2011.14809).

The U.S. market is not simple. The NYSE index was analyzed across 1950–2013. In the equal-weighted index (an index holding all stocks in equal proportions), the random walk hypothesis was rejected across the entire period. In the market-cap-weighted index (an index where larger companies have higher weights), the random walk began to be accepted starting in the 1990s (arXiv:1510.03926). However, the authors viewed this difference not as a change in efficiency, but as the combined result of a portfolio effect and a non-synchronous trading effect. Non-synchronous trading is a phenomenon where the last trading time differs for each stock, causing the index to move late. In a simulation assuming 2-stock and 6-stock portfolios, almost all of the difference between the two indices was explained. Therefore, this case is difficult to use alone as evidence for the AMH.

Why does efficiency change? One conceptual study conceptualized efficiency as a local and state-dependent property arising from the interaction of economic scale, institutional enforcement power, and geopolitical location. This is a paper that proposed a framework rather than an empirical test (arXiv:2601.05924). In this framework, efficiency can appear differently from market to market.

There are also reports that signals appear in markets where inefficiencies remain. In 8 years of data from the Brazilian market, a statistical relationship was reported between chart patterns and subsequent trends. The authors considered technical analysis (a method of reading the future through the shapes of price charts) valid as a means of predicting trends within that period (arXiv:1302.1228). In the Chinese market, the multifractal spectrum width was a significant predictor of excess returns in both in-sample and out-of-sample tests. Multifractal is a method of measuring the complex structure of price fluctuations (arXiv:1806.07604).

An Honest Twist — What Makes Winning Hard

Up to this point, you want to conclude that "there are patterns, so we can win." However, several factors that make this difficult are also reported. When putting together the research this article has read, evidence of inefficiency and evidence that its profits are limited appear together. Some studies have found that profits remain even after subtracting transaction costs (arXiv:2207.10476, arXiv:2601.11958), so conclusions are divided. Transaction costs are the costs incurred when buying and selling.

The first shadow is that predictability and profit potential are different. There is an argument that "predictability alone is not a sufficient measure of market efficiency." One evolutionary market model remeasured inefficiency by the maximum profit an ideal trader could actually extract. As a result, the feasible profit was very limited, and it shrank further as market participants increased (arXiv:cond-mat/0403621). Within this model, even when patterns existed, the amount that could be converted into money was small. The authors concluded that high levels of efficiency can coexist with predictability.

The second shadow is the self-destruction of strategies. In a ChatGPT experiment, GPT-4 predicted market reactions using news headlines. It also significantly predicted subsequent price drift (especially in small-cap stocks and negative news). Price drift is a phenomenon where prices continue to move in the same direction even after the initial reaction. However, as LLM adoption increased, strategy profits decreased. The authors interpreted this as consistent with improved price efficiency (arXiv:2304.07619). Although it is a case of a single strategy, it fits EMH's self-correcting logic that known inefficiencies shrink as they are exploited.

The third shadow is transaction costs. However, studies vary on how much transaction costs cut into profits. The Moscow Exchange was reported to have significantly low efficiency in most months from 2012 to 2021. The authors wrote that this inefficiency is a signal that strategies capable of generating profits even after subtracting transaction costs can be built (arXiv:2207.10476). However, the abstract merely writes this as a "signal of possibility" and does not say that the strategy was tested directly. Bitcoin was analyzed using neural networks. Even when input assets were increased, it consistently failed to beat a buy-and-hold strategy on data set aside for evaluation. This means evidence was found that the Bitcoin market was efficient in the EMH sense during the sample period (arXiv:2208.07254). There are studies that Indian and Chinese markets are weak-form inefficient (meaning you can predict them to some extent using past prices alone) (arXiv:1709.04059, arXiv:2012.01160). Based on this, the Indian market study concluded that the market can be beaten. However, the abstract did not account for transaction costs and used only statistical tests like run tests and autocorrelation tests. The China and India studies also wrote that efficiency seemed to increase after the 2008 financial crisis.

The fourth shadow is that the time scale over which inefficiencies are caught is too short. When examined in units of tens of seconds, it has been reported that stock bundles (baskets) where the autocorrelation structure (the relationship where a recent return connects to the next return) is not self-evident can be systematically created. This is a bundle violating the EMH, and that situation reportedly persists long enough to be predictable to some extent (arXiv:1006.5230). The time scale of tens of seconds makes it difficult to capture this inefficiency. When arbitrage was measured geometrically, on limited data, the market was efficient at time scales of a day or longer. Inefficiencies appeared in intraday high-frequency data and seemed to disappear in about a minute once generated (arXiv:0908.3043). These opportunities must be used with very short trading times, so how large the costs are cannot be known from the abstracts read by this article. In blockchain assets, physical friction called consensus delay, which takes time to finalize transaction records, restricts arbitrage. This "limit of arbitrage" teaches the same lesson (arXiv:1812.00595).

The fifth shadow is that efficiency is conditional. The Shanghai-Hong Kong Stock Connect is a financial liberalization policy that lowered the threshold for cross-border investment. In a 2026 analysis examining this, the policy widened the price difference between A-shares and H-shares of the same company (the A-H premium) by an average of 18.4%. The effect was larger in companies in less efficient markets. The authors concluded that liberalization does not necessarily produce price convergence (arXiv:2602.14754). There is also the "leverage efficiency" hypothesis. This is the hypothesis that the optimal value of leverage (the act of increasing investment scale with borrowed money) converges to 1. This hypothesis considers that applying leverage to the market portfolio does not yield long-term excess performance. As an implication of this hypothesis, the authors suggested that fraudulent funds have probabilistic properties different from those of legitimately traded assets, so it can be used for fraud detection. Analyzing price data for the S&P 500, Bitcoin, Berkshire Hathaway, and the Madoff fund reportedly supported this hypothesis (arXiv:1101.4548).

Even what behavioral finance has called "anomalies" leave room for alternative interpretations. The equity premium puzzle is the mystery of why stocks earn much more than safe assets without being explained. The volatility puzzle is the mystery of why stock price volatility is larger than fundamental value volatility. There is an argument that both of these and asset return predictability can be explained within rational finance theory without behavioral finance assumptions (arXiv:1710.03211). The authors stated that it is not the only explanation. There is also an argument that the stock predictability puzzle can be explained by statistical models within rational finance (arXiv:1911.02194). Over the same data, "evidence of irrationality" and explanations without behavioral finance assumptions contend with each other.

New Cracks in the Age of AI — And Doubting Them

Recently, AI is opening a new chapter in this age-old question. There is a 2026 experiment where LLMs were made to independently evaluate the Russell 1000 stocks (the constituent stocks of a major US large-cap index) every day. It is a fully real-time out-of-sample (a method of checking with new data not used in the analysis) design where future information leakage is structurally impossible. The results were nuanced. The authors evaluated that AI has a genuine stock-picking ability, but only when picking the "top-tier winners." A portfolio containing the top 20 stocks rated highest by the AI generated a significant alpha (excess returns not explained by market movements). Transaction costs were less than 10% of the total alpha, so the authors viewed it as an implementable strategy. However, expanding beyond that caused the alpha to dilute rapidly, and the lower-tier stocks were indistinguishable from the market. The observation period is short, starting after April 2025 (arXiv:2601.11958). "Can beat" and "cannot" coexist within a single study.

At the same time, extraordinary claims require extraordinary caution. One preprint activated value signals and short-term reversal signals only in a "drift regime." Value signals use the tendency for cheap stocks, and short-term reversal signals use the tendency for recently dropped stocks to soon bounce back. The drift regime is a period with many days on which individual stocks went up. They reported achieving an out-of-sample Sharpe ratio of 13 or higher and an annual return of 158.6% in that manner (arXiv:2511.12490). This preprint reportedly even passed 1,000 randomized tests and parameter perturbation. Even so, figures of this magnitude are targets for suspicion in themselves. Data mining (the act of sifting through data to find accidental patterns) and survivorship bias (distortion where only surviving stocks remain, making results look good) are representative traps. Future information leakage and capacity limits are also traps. Whether all of these were filtered out is something that peer review and independent replication will tell. The first warning of this primer—that a single line of a preprint cannot be read as a conclusion—rings the loudest in front of the most dazzling yields.

Conclusion

The conventional wisdom that "no one can beat it" showed cracks in the face of anomalies like momentum, value, and small caps, and behavioral biases like the disposition effect and overreaction. However, the opposite side claiming "you can beat it" could not easily boast either, in the shadow of trading costs, short time scales, and strategy self-destruction. The most honest compromise was the adaptive market hypothesis.

So the conclusion is a balance, not a certainty. Looking at the research this piece explored, there are studies reporting known inefficiencies, and also studies showing that those profits were limited (arXiv:cond-mat/0403621, within a model) or decreased as adoption increased (arXiv:2304.07619). Whether profits remain after subtracting trading costs varies from study to study. The market is neither perfectly efficient, nor inefficient enough to beat at will.

This article is for providing financial and economic information, and is not investment advice or a solicitation. Most of the cited evidence is preprints (arXiv) before peer review, and the details may be updated by subsequent research.

---

Source Papers (arXiv)

Common Belief — Efficient Market Hypothesis and Random Walk

• How EMH and AI universal approximators reshape efficiency (arXiv:2501.07489)
• Modern approach to EMH (absence of weak arbitrage + market sensitivity as a necessary and sufficient condition for information efficiency) (arXiv:1302.3001)
• Efficiency measure based on information entropy (equivalent to Fama and Jensen) (arXiv:1812.02371)
• Efficiency measurement of 41 indices (correlation structure and clustering) (arXiv:1208.1298)
• "If markets are efficient, P=NP" (arXiv:1002.2284)
• Practical information rate and market efficiency (GARCH is inefficient) (arXiv:0903.2243)

Crack ① Anomalies — Momentum, Value, Small Cap, Calendar, and New Factors

• Factor momentum versus stock momentum (arXiv:2009.04824)
• Symmetry-breaking momentum strategy (US and South Korea) (arXiv:1107.5122)
• CAPE valuation and long-term return forecasting (arXiv:1204.5055)
• Multivariate return forecasting model (dividends, E/P, B/M) (arXiv:2110.01873)
• Penny stock cross-sectional returns (small cap, low P/E, low P/B) (arXiv:1610.01338)
• Intraday patterns of cross-sectional returns (continuation and reversal) (arXiv:1005.3535)
• Monthly return calendar effect (May effect) (arXiv:2003.05750)
• Intangible investment intensity factor (60 years of evidence) (arXiv:2505.16336)
• Cyber risk and cross-sectional returns (arXiv:2402.04775)
• News network and cross-sectional predictive power (arXiv:2108.05721)

Crack ② Behavioral Biases — Disposition Effect, Overreaction, and Bubbles

• The disposition effect in Bitcoin (arXiv:2010.12415)
• Underreaction and overreaction response functions and return distribution (arXiv:2312.02472)
• Predictive horizon and stock characteristics of investor sentiment (arXiv:1803.02962)
• LLM over-extrapolation and miscalibration (human bias) (arXiv:2409.11540)
• Forecasting, clustering, and the formation of bubbles and crashes (arXiv:1106.1577)
• Price bubbles arising even under efficiency conditions (sociology) (arXiv:1609.04629)
• Multi-layer market irrationality factor learning (arXiv:2502.04737)

Compromise — Adaptive Market Hypothesis (AMH)

• Time-varying efficiency in pre-war and wartime Japan (AMH) (arXiv:1911.04059)
• AMH viewed through time-varying AR (Japan's TOPIX and TSE2, only TOPIX evolves) (arXiv:1207.1842)
• Evolution of cryptocurrency market efficiency (AMH) (arXiv:1904.09403)
• Time-varying predictive power in the Chinese market (arXiv:1611.04090)
• Collective dynamics of market efficiency (time-varying) (arXiv:2011.14809)
• NYSE index differences explained by nonsynchronous trading and portfolio effects (arXiv:1510.03926)
• Geopolitics, institutions, and adaptive efficiency (GAER) (arXiv:2601.05924)
• Statistical evidence of technical analysis (Brazil) (arXiv:1302.1228)
• Multifractals and return predictive power (China) (arXiv:1806.07604)

Honest Reversal — Costs, Time Scales, Self-Destruction, and Conditional Efficiency

• Limited profits in predictable markets (participants↑ → profits↓) (arXiv:cond-mat/0403621)
• ChatGPT strategy returns, vanishing as adoption↑ → price efficiency↑ (arXiv:2304.07619)
• Moscow Exchange inefficiency (strategy viability even after deducting transaction costs) (arXiv:2207.10476)
• Bitcoin EMH (neural networks fail to beat buy-and-hold) (arXiv:2208.07254)
• Weak-form inefficiency in Chinese and Indian markets (arXiv:1709.04059)
• Weak-form inefficiency in the Indian market (arXiv:2012.01160)
• EMH violations in baskets restricted to multi-second scales (arXiv:1006.5230)
• Arbitrage exists only in high-frequency trading under 1 minute (arXiv:0908.3043)
• Arbitrage limits in blockchain assets (consensus delay) (arXiv:1812.00595)
• Financial liberalization effects depend on prior efficiency (arXiv:2602.14754)
• Leverage efficiency (Madoff fraud detection) (arXiv:1101.4548)
• Rational financial explanation of behavioral anomalies (arXiv:1710.03211)
• Rational model of the stock predictability puzzle (arXiv:1911.02194)

New Cracks in the AI Era — and Caution

• Agent AI nowcasting (alpha goes only to top winners) (arXiv:2601.11958)
• Claims of out-of-sample Sharpe 13 factor (preprint caution) (arXiv:2511.12490)