Long-term and Short-term Stock Prices Prediction with Sentiment

NaN.

Long-term and Short-term Stock Prices Prediction with Sentiment[ Edit ]

As I mentioned in my last article, I will complete this sentiment study and share more interesting findings here with you. Based on the financial event study that I posted before and other research in the financial domain, there is increasing evidence that online sentiment can help predict subsequent market activity. But its effect, news to trade prices, is asymmetric, news with positive sentiment has been demonstrated to relate to a large price increase for a relatively short period of time; and negative sentiment, however, is linked to price decrease but with more prolonged effects. The next question would be how can we use sentiment to help us to make a prediction about stock prices? Based on the study of this topic, here is the summary of my study:

For a short-term period: sentiment score combined with impact score can provide us with intraday’s BUY or SELL signal — if sentiment score is positive and the impact score is high then it implies a BUY signal( there is a very high possibility that stock price will go up) and vice versa;
For a long-term period: we care more about the trend, however, daily sentiment score cannot provide us with much useful information about the stock trend. But if we can extract the trend signal of the stock prices from continuous cumulative or average sentiment(or long-term average or moving average) — if the sentiment scores are positive and corresponding impact score is high for some time we would have the high possibility that the stock prices will go up within this period, vice versa.
Sentiment signal seems always delayed for the public due to the asymmetric information (A situation in which one party in a transactionhas more or superior information compared to another.) It happens before and after the sentiment score is generated.

Before the article or the news released, some people have the information earlier so that they have the arbitrage opportunity. They can make arbitrage profits by betting on public’s reaction towards the news later — if this is good news the public tend to have a positive reaction and in turn, the stock prices will increase(so betting on the stock prices will increase) vice versa. Therefore, this arbitrage opportunity is actually obtained from betting on ‘social sentiment’ and its direct influence on stock prices.
After the news released, some people may detect the information earlier than others. Such as the twitter leak accident which only lasts for only 45 seconds and for those who detect the information at the very beginning can react quickly. Eventually, it leads 20% drop in Twitter’s stock price within this very short time. It happens every day and every moment and it seems like the timing is always the essence of sentiment. In other words, the phenomenon of asymmetric information among public would create some arbitrage opportunity for people who have access to the news even slightly earlier and it continues to prove that the social sentiment actually has a significant impact on stock prices even if mostly short term.

To sum up, for intraday traders the most recent sentiment can provide BUY or SELL signals of the stock if efficient market hypothesis(EMH) (An investment theory that states it is impossible to “beat the market” because stock market efficiency causes existing share prices to always incorporate and reflect all relevant information. ) stands here. Bear in mind that the signals have nothing to do with the trend of the stock prices since we can never know the magnitude of the UP/DOWN of the stock prices by simply relying on sentiment. For example, for last 5 days, we have positive sentiment scores for 4 days among the 5 days and only one day we got a negative sentiment score. If we purely rely on the sentiment and impact score we could choose to BUY the stock since we assume the sentiment signal somehow can determine the trend of the stock price. However, the stock price could possibly go DOWN since the stock price goes DOWN for $10 within 1 day(we assume), and the stock price only goes UP for $0.1 per day for rest 4 days(we can call this distribution is Negative Skewed). The stock prices could still go down and the sentiment can not give us any indication of it. Using short-term sentiment to capture the stock trend does not work for the Positive Skew distribution for the same reason.

Only if we can compute the average sentiment for a long enough time period and we may be able to grasp the general picture about stock prices trend moves such as if the sentiment is mostly positive for a long period it is highly possible that the stock prices have an upward trend. We can think backwards to prove the feasibility of this deduction:

— the stock price and sentiment can influence each other to some extent, thus, usually we assume the sentiment comes first and stock prices will reflect sentiment a little bit later. The negative sentiment will bring stock prices down for a while. However, the downward trend of the stock will also result in the negative sentiment afterwards. Therefore, if the average sentiment is keeping negative for a long enough time period, it implies that the stock price is most likely moving downwards and vice versa.

The stock prices trend is influenced by multiple factors and sentiment is one of them;

Stock prices and sentiment tend to influence each other mutually;
If you want to take advantage of the sentiment for the short-term period, timing is the essence;
If you would like to know about the trend of the stock price for a long-term period, the continuous sentiment and its impact are necessary. But still, they should only be a part of the overall analysis. I would recommend adding more factors to increase the dimensions and the accuracy of your analysis.

Based on the previous analysis, we can summarize that the sentiment has the direct connection with stock prices but it also has its own limits for prediction.The next step would be:

how can we prove whether the connection between sentiment and the stock prices truly exists or does it just conforms to our common sense. In order to check the accuracy of the signal that I generated from daily sentiment. The Null hypothesis here would be: there is no correlation between stock prices and sentiment.

In order to understand the relationship between them I have run the test on more than 500 stocks randomly and the conclusions are generally the same. I will use an example(using “AA” as usual) to show you if this hypothesis is wrong or not. I compare the signal with the price moving average crossover signal that I introduced before in my article.

The graphs below show that moving an average of the sentiment and signals that created from the sentiment moving averages. The sentiment tends to work well for a short-term period so I only use short-term moving average to generate signals. The signals are created by using distance metrics which is based on the hamming distance theory. Generally, if there are positive peaks — BUY signals; if the position of the signals is high — HOLD; reversely, if there are subsite spikes — SELL signals(* I multiply the stock prices by 5 to show the trend clearly). We can find that the signals are accurate overall — the stock prices increase or keep being stable right after the BUY signal(positive peaks); the stock tends to go down right after the SELL signals.

(**date: 2012–10–01–2016–02–03)

Let’s shorten our testing period to check in detail. The first graph below shows the signal that I generated from the daily sentiment score. The second one is generated from the price moving average (from 5 days to 20 days moving average). Prices moving average seems to capture more signals. But if you look closely(they are within the same testing time frame so you can compare them vertically), you can find that, except the huge signal displayed at the beginning of the first graph, which is not included in the second one(but given longer time frame to test the price moving average you will find price moving average crossover generated signals will capture this one too) the rest three signals are pretty much similar (happen at the same time).

I generated both graphs based on the Spearman Rank Correlation metrics(feel free to check my previous article for an explicit explanation for this measurement). In short, if there the moving average come across with each other there will be a peak on the graph since the p-value is close to 0 when all the moving averages are moving together and the only moments that p-value will increase is when they have crossovers.

From the comparison of these two graphs, our null hypothesis seems to be rejected. Sentiment moving average generated signals can be used to compare with price moving averages generated signals. But just for the graph below you can tell that the signals coming from the sentiment moving average seem to be a little bit delayed and they tend to last longer than the signals from prices moving average. Therefore, if we simply rely on sentiment to give us the signals we may end up adding some noises into our prediction.

(date:2015/10/01–2016/02/03)

As I explained before, the moving average of the sentiment is the relative long-term signal which can contain the useful information for us to detect the trend of the stock prices. I have shown in my previous article about the short-term(daily sentiment) sentiment signal and it’s correlation with the stock returns. The graph below is similar to the one I posted before and the only difference here is I add the product of the sentiment with impact score into the graph.(impact score won’t affect the changing of the trend but will affect the extent of the trend). It looks like the sentiment(or the product) seems to have similar moving pattern as the daily returns.

(**date: 2012–10–01–2016–02–11)
Then let’s look at the long-term sentiment and its connection with the stock returns. Based my analysis above, we try to test whether the cumulative sentiment and the average sentiment can help us capture the long-term or intermediate term’s stock trend.

From the graph below, we can find that the purple line represents the cumulative daily sentiment of AA and the red line represents the cumulative daily abnormal returns of the AA. We can see that when the cumulative sentiment is above 0 and high(implies the sentiment is positive in general and we can see the stock price is moving up when the long-term sentiment is generally positive). Around August 2014, the cumulative sentiment started to drop quickly towards 0(the social sentiment turns into very negative then drag all the cumulative sentiment down quickly). At the point at the beginning of 2015, the stock cumulative sentiment reaches 0 the stock prices starts to drop dramatically. The cumulative returns seem to keep being stable for a short time and started dropping quickly afterward. Then the cumulative sentiment just keeps being low and cumulative returns are still moving downwards with few some peaks.(I divided cumulative sentiment by 100 times for AA and 10 times for TSLA).

The effect of cumulative sentiment depends on the start point of both cumulative and cumulative returns. When the cumulative starts from a high positive number and only if it moves towards 0 means there are negative sentiments last for a certain amount of time drag the total cumulative level down and it possibly imply that the cumulative returns will go down sooner or later; If cumulative sentiment starts from low negative number and only if it moves towards 0 means there are positive sentiments show up for a certain amount fo time to drag the total cumulative level and the stock prices(returns) will go up sooner or later like the situation on the graph of FB. Some small vibration of cumulative sentiment may not be incorporated in the returns.

(**date: 2012–10–01–2016–02–11)

As for the 50 days sentiment moving average, we can find a similar pattern giventhe sentiment moving average is going down towards 0 or below 0(the sentiment scores are generally negative within this period) the cumulative returns will have the downward trend. When the sentiment moving average moving above 0 the cumulative returns will go up. The trend of 50 days sentiment moving average seems to show up faster on the stock prices than the cumulative sentiment’s.

In sum, the sentiment has a close and mutual relationship with stock prices(returns). Sentiment and its impact can generate fair signals for short-term(intraday) but we cannot claim that we get the trend information about stock prices from it. For long-term, the continuous sentiment score has to be calculated to identify the trends of stock prices. You can choose to take advantage of sentiment score based on your strategy but be aware of the limitations of the application of sentiment, obviously, my research isn’t the end all, but I do hope to shed light on more accurate uses of sentiment analysis.

(** all the sentiment related data I am testing and using here is from Accern’s database and you can find the detailed explanation here**)

Hope you enjoy the content. Please feel free to leave your comments or share your ideas here with me. Thank you!

Reference:

Category: Machine Learning