Identify noise in Correlation and Covariance matrices

This post will describe how to identify and remove noise from a Correlation or Covariance matrix

Practical use: Correlation and Covariance matrices are used as the basis for many quant finance algorithms. If these matrices are corrupted through noise, the results could be invalid. If there is an easy workflow to identify the presence of noise in our dataset, we can then take steps to denoise.

Physical meaning: We first need a template of what to examine, what good looks like, and how we can compare our covariance matrix against this template, to determine if action is required.

What to examine: We will be examining the probability density function of the Eigen values of our covariance/correlation matrices. This is an excellent video on the meaning of probability density function: https://www.khanacademy.org/math/statistics-probability/random-variables-stats-library/random-variables-continuous/v/probability-density-functions

What good looks like: The Marcenko–Pastur Theorem tells us that for independent and identically distributed random variables, the Eigen values of the covariance matrix will be distributed as per a particular profile. The Theorem allows us to generate a custom template profile for different values of number of independent variables and observations. This is the absolute genius of this Theorem.

How we can compare: We can determine the covariance matrix, compute the Eigen values, and then can compute the probability density function. This is an excellent article that explains how to compute the probability density function by hand, and the significance of the bandwidth attribute: https://medium.com/analytics-vidhya/kernel-density-estimation-kernel-construction-and-bandwidth-optimization-using-maximum-b1dfce127073

If the probability density function of our Eigen values, does not match the target probability density, then that means we should consider denoising algorithms for our covariance matrix.

Results: Figure 1 shows an instance where the Actual probability density function (PDF) of a covariance matrix matches well with the template  PDF predicted by the Marcenko-Pastur Theorem. The code for generating this is provided below 


Compute using Python:



Comments

Post a Comment

Popular posts from this blog

1.2 Structured Data: Information Driven Bars

Image
This post will describe how to process & store High Frequency financial data to support your Machine Learning Algorithm, based on how much information is contained in the data The post is directly based on content from the book "Advances in Financial Machine Learning" from Marcos Lopez de Prado Physical meaning: This approach is inspired by Claude Shannon's Information Theory, that is the basis of most compression algorithms. Shannon argued that only departure from the norm is new information. A famous example, is that the sun has risen in the morning. Yes, the event has happened, but because the probability of this event was 1.0, this event does not represent information. In information driven bars, we attempt to use the same insight, to compress the financial data. Here, we know that financial data is inherently noisy, and put guardrails around what level of variance we expect for a particular behavior. It is only when the variance exceeds these guardrails, that we ...
Read more

2.2 Labeling: Triple barrier method

Image
 This post will describe how to label a financial dataset to support supervised learning The post is directly based on content from the book "Advances in Financial Machine Learning" from Marcos Lopez de Prado Physical meaning: Algorithm description: Python code: # Import libraries import numpy as np import pandas as pd import matplotlib.pyplot as plt import math # Import data df = pd.read_csv( r'C:\Users\josde\OneDrive\Denny\Deep-learning\Data-sets\Trade-data\ES_Trades.csv' ) df = df.iloc[: , 0 : 5 ] df[ 'Dollar' ] = df[ 'Price' ] * df[ 'Volume' ] print (df.columns) # Generate thresholds d = pd.DataFrame(pd.pivot_table(df , values = 'Dollar' , aggfunc = 'sum' , index = 'Date' )) DOLLAR_THRESHOLD = ( 1 / 50 ) * np.average(d[ 'Dollar' ]) # Generate bars def bar_gen (df , DOLLAR_THRESHOLD): collector , dollarbar_tmp = [] , [] dollar_cusum = 0 for i , (price , dollar) in enumerate ( zip (df[ 'Price...
Read more

Denoise a Covariance Matrix using Constant Residual Eigen Value Method

Image
 This post will describe how to denoise a Covariance matrix using Constant Residual Eigen Value Method Practical use: The Eigen Values are naturally stored in decreasing order. Once we find the range of Eigen Values that correspond to signal (using fit to Marcenko Pastur Distribution), we can then reset all other eigen values below this threshold to a constant value. This basically vaccums all the eigen values containing random noise into a single Eigen value. These transformed Eigen values are used with the original Eigen Vectors, to recreate a denoised covariance matrix, that will be used for further processing. Compute using Python: # File: Denoise-Constant-Residual-Eigen.py import numpy as np,pandas as pd import matplotlib.pyplot as plt #--------------------------------------------------- def mpPDF(var,q,pts): # Marcenko-Pastur pdf # q=T/N eMin,eMax=var*( 1 -( 1. /q)** .5 )** 2 ,var*( 1 +( 1. /q)** .5 )** 2 eVal=np.linspace(eMin,eMax,pts) pdf=q/( 2 *np.pi*var*eVal)*...
Read more