This post will describe how to process & store High Frequency financial data to support your Machine Learning Algorithm, based on how much information is contained in the data The post is directly based on content from the book "Advances in Financial Machine Learning" from Marcos Lopez de Prado Physical meaning: This approach is inspired by Claude Shannon's Information Theory, that is the basis of most compression algorithms. Shannon argued that only departure from the norm is new information. A famous example, is that the sun has risen in the morning. Yes, the event has happened, but because the probability of this event was 1.0, this event does not represent information. In information driven bars, we attempt to use the same insight, to compress the financial data. Here, we know that financial data is inherently noisy, and put guardrails around what level of variance we expect for a particular behavior. It is only when the variance exceeds these guardrails, that we ...
This post will describe an Information Theoretic approach to estimating a random variable. This approach again highlights how important Electrical & Electronics Engineering concepts are applicable to Capital Markets. 1. Caveats of correlation: Correlation is not a useful metric of co-dependency between two variables if: If there is a non-linear relationship between the variables If there are significant outliers that skew the correlation metric If the two variables do not jointly follow a bi-variate normal distribution If correlation is judged to be a useful metric instead of its usual form, it is worth using it in a modified form to judge the distance between two metrics 2. Shannon's Information-Theoretic approach: We need a metric that highlights the degree of randomness in a random variable. E.g. the degree of sunlight at a point throughout a 24 hour period is a random variable, but there is a low degree of randomness in this random variable. The entropy in a random variable...
This post will describe how to identify and remove noise from a Correlation or Covariance matrix Practical use: Correlation and Covariance matrices are used as the basis for many quant finance algorithms. If these matrices are corrupted through noise, the results could be invalid. If there is an easy workflow to identify the presence of noise in our dataset, we can then take steps to denoise. Physical meaning: We first need a template of what to examine, what good looks like, and how we can compare our covariance matrix against this template, to determine if action is required. What to examine: We will be examining the probability density function of the Eigen values of our covariance/correlation matrices. This is an excellent video on the meaning of probability density function: https://www.khanacademy.org/math/statistics-probability/random-variables-stats-library/random-variables-continuous/v/probability-density-functions What good looks like: The Marcenko–Pastur Theorem ...
Comments
Post a Comment