Inferential Statistics¶

Inferential statistics is used for finding inferences on the data and make predictions about the data on a given sample of data.This uses probability to find conclusions.

There are possible methods to perform inferential statistics on the data. In this blog we will discuss about Z-Score, Z-Test, F-Test, Correlation Coefficients, chi-square Test for performing the analysis of the data and get a probable conclusion based on it.

When we use Inferential Statistics?

Inferential statistics mainly used for finding conclusions about the data, the data can be a sample or set of features so sometimes we use a large size of data for building a model at that time this inferential statistics comes in handy.

Contents¶

1. Z Scores, Z-Test

   1.1 Z Value

   1.2 Z Test

   1.3 Two-sided One-Sample t-Test

   1.4 Independent t-Test

   1.5 Paired T-test

1. F-test

1. Correlation Coefficients

1. Chi-Square Test

1. Z scores, Z-Test¶

1.1 Z Value¶

Z- Value/ Z- Score tells a value (x) is how many standard deviations below or above the population mean. If the Z value is positive the value/ score (x) is higher than the mean and if the Z value is negative the value is lesser than the mean

Z-Score can be calculated as follows

   z = (X – μ) / σ

   where,
       X : Single data value
       μ : Mean value
       σ : Standard Deviation

Z-score in python can be calcualted by using scipy.stats.zscore such as, scipy.stats.zscore(a, axis=0, ddof=0, nan_policy=’propagate’)

where,

a : array_like
An array like object containing the sample data.

axis : int or None, optional
Axis , either horizontal or vertical

ddof : int, optional
Degrees of freedom correction in standard deviation calcualtion. Default value is 0 (zero).

nan_policy : {‘propagate’, ‘raise’, ‘omit’}, optional
This field defines a way of handling when input contains nan.
Default value is propagate, which returns nan
The value raise, throws an error
The value omit, ignores nan values and performs the calculation.

Note: Whenever the value is omit, the nan values in the input propagate to the output, but these nan values 
do not affect the z-scores that's been computed for the non-nan values

Ex: a = [0.8976,0.9989,0.5678,0.1234,0.7765,1,1.675,1.456]

==> Mean (μ) = Sum of all the elements/N , where N = total number of elements

mean (μ)= (0.8976+0.9989+0.5678+0.1234,0.7765+1+1.675+1.456)/8 =0.9369

==> standard deviation (σ) = sqrt((X-μ)/N) , where X = element

standard deviation (σ) = sqrt((0.8976-0.9369)^2+(0.9989-0.9369)^2+(0.5678-0.9369)^2+(0.1234-0.9369)^2+(0.7765-0.9369)^2+(1-0.9369)^2+(1.675-0.9369)^2+(1.456-0.9369)^2))/8 = 0.45378

==> Z-score (z) = (X-μ)/σ

z = [(0.8976-0.9369)/0.4537,(0.9989-0.9369)/0.4537,(0.5678-0.9369)/0.4537,(0.1234-0.9369)/0.4537,(0.7765-0.9369)/0.4537,(1-0.9369)/0.4537,(1.675-0.9369)/0.4537,(1.456-0.9369)/0.4537]

Result is ==> z = [-0.0866,0.1357,-0.8135,-1.7930,-0.3535,0.1390,1.6268,1.144]

Computing z-score using defualt values¶

Computing z-score along specified axis using degrees of freedom¶

Computing z-score using nan_policy¶

1.2 Z-Test¶

Z- Test is to test the population proportion. Z-Test can be used to test the given mean, when the sample is large, which means the length of the data is more than 30 , and when the population standard deviation is known as well as variance is known. This test is perfromed to check if the 2 sample means are approximately equal or not.

To perfrom this z-test the samples should be taken at random from the population and the data should be normally distributed. If the data taken is larger than 30 then it is assumed that the data is normally distributed. If the sample size is less than 30 then the t-test is considered.

We check if the value obtained is approimately equal or not by considering hypothesis such as Null Hypothesis (H0) : If the value is equal to the other value, this hypothesis is accepted Alternate Hypothesis (HA) :If the values is not equal to the other value, this hypothesis is accepted

This Z-test is calculated by using the formula

After performing the z-test, the value obtained should be compared with the alpha value, which is assumed to be 0.05 in the z-score table, which is considered to be pvalue.

• This pvalue if it is less than the alpha value, then the Null Hypothesis is rejected which means the Alternate Hypothesis is considered. In other words the means of the two samples are not equal

• The pvalue if it is greater than the alpha value, then the Null Hypothesis is accepted. In other words the means or averages of the two samples are equal.

  In Machine Learning, we calculate z-test by using method ztest from statsmodels.stats.weightstats

       statsmodels.stats.weightstats.ztest(x1, x2=None, value=0, alternative='two-sided', usevar='pooled', ddof=1.0)
  
     where,
         x1,x2 are arrays
  
         value : float
         In the one sample case, value is the mean of x1 under the Null hypothesis. In the two sample case, value is the difference between mean of x1 and mean of x2 under the Null hypothesis. The test statistic is x1_mean - x2_mean - value.
  
         alternative : str
         The alternative hypothesis, H1, has to be one of the following
         ‘two-sided’: H1: difference in means not equal to value (default)
         ‘larger’ : H1: difference in means larger than value 
         ‘smaller’ : H1: difference in means smaller than value
  
         usevar : str, ‘pooled’
         Currently, only ‘pooled’ is implemented. If pooled, then the standard deviation of the samples is assumed to be the same. see CompareMeans.ztest_ind for different options.
  
         ddof : int
         Degrees of freedom use in the calculation of the variance of the mean estimate. In the case of comparing means this is one, however it can be adjusted for testing other statistics (proportion, correlation)

        Returns, 
            tstat : float
            test statistic

            pvalue : float
            pvalue of the t-test

Example¶

The first value from the above result is statistic value and the other value is pvalue. From the above output, we can understand that the pvalue of the taken data is 0.9 which is greater than the value of alpha which is 0.05. Hence we come to the output that Null Hypothesis is correct and it is accepted, which means that the given data and the assumed mean are approximately equal.

From the above output, we can understand that the Null Hypothesis is rejected and Alternate Hypothesis is accepted as the pvalue is less than the alpha value

T-Test¶

T-test, also known as Student’s T-test, is used to determine the difference among two groups of variables by comparing their mean values or the averages. This T-test not only determines the difference but also determines the significance of their differences. In other words, this test simply explains that the differeneces among the varibale groups is occurred by a chance or relevant to the data taken.

The 3 types of T-test are

        1. Independent T-test
        2. Paired Sample T-test
        3. One-Sample T-test

One-Sample T-test : This one-sample T-test is a t-test where the one group’s mean or avergae is compared with one significant value which is a mean of the population

     Types of One-Sample T-test are
         1. One tailed One-Sample T-test
         2. Two tailed One-Sample T-test
         3. Upper tailed One-Sample T-test
         4. Lower tailed One-Sample T-test

1.3 Two Sided One-Sample T-test¶

Two Sided One-Sample T-test or Two tailed One-Sample T-test ———————

1.4 Independent t-Test¶

Independent T-test, also known as Two Sample T-test is used to test whether the means of the taken 2 groups are equl or not. This Independent T-test assumes that the variances of the taken population has equal variance by defualt.

    In Machine Learning, we can perform this test using
              - Using scipy library
              - Using Statsmodels

Using scipy library¶

scipy.stats.ttest_ind(a, b, axis=0, equal_var=True) where, a, b are two arrays of 2 groups

axis : int or None, optional Axis along which to compute test. If None, compute over the whole arrays, a, and b.

equal_var : bool, optional If True (default), perform a standard independent 2 sample test that assumes equal population variances If False, perform Welch’s t-test, which does not assume equal population variance.

Returns statistic : float or array The calculated t-statistic.

pvalue : float or array The p-value.

Example¶

For example, we are given 2 different groups of data among which Bag-A has a bunch of apples and Bag-B has a bunch of mangoes. We need to check if the both bags have the same averages or means.

For this we need to assume 2 hypothesis, null hypothesis and alternate hypothesis H0 -> The means of two bags are equal HA -> The means of two bags are not equal

Two find which hypothesis is predicted to be different, we check the value of alpha, which is assumed to be 0.05, with the pvalue that is obtained after performing t-test. If the pvalue is less than the alpha value, then the HA is considered to be true If the pvalue is greater than the alpha value, then the H0 is considere to be true

Lets test the above example with ttest using scipy library

Understanding the result from the above test, assuming that the alpha value is 0.05, the both results assuming that the 2 groups have same variances and different variances, returns out pvalue which is less than the value of alpha. Hence the 2 bags or the 2 groups means are not supposed to be equal

Using statsmodels¶

statsmodels.stats.weightstats.ttest_ind(x1, x2)

where, 
    x1 and x2 are two array groups

Returns:
    tstat : float
    test statistic

    pvalue : float
    pvalue of the t-test

    df : int or float
    degrees of freedom used in the t-test

Understanding the result from the above test, assuming that the alpha value is 0.05, the resulted pvalue is 0.7 which is greater than the assumed alpha value. Hence it can be said that the assumption H0 is true, i.e., the means of the 2 groups are equal

1.5 Paired t-Test¶

A Paired t-Test explains the difference between two variables for the same subject. It compares one set of measurements with the second set from the same sample. This test is also known as Dependent Sample T-test

In simple words, this T-test measures the difference between two averages or means of two different groups. This test similar to other tests assumes that there are 2 hypothesis. Null Hypothesis (H0) : The difference between two means of the two groups is zero Alternate Hypothesis (HA) : The difference between two means of the two groups is not equal to zero.

In Machine Learning, this Paired T-test can be calculated by using ttest_rel() method defined in scipy.stats library

     scipy.stats.ttest_rel(a, b, axis=0, nan_policy='propagate', alternative='two-sided')

     where,

          a, b : array_like

          axis : int or None, optional
          Axis along which to compute test. If None, compute over the whole arrays, a, and b.

          nan_policy : {‘propagate’, ‘raise’, ‘omit’}, optional
          Defines how to handle when input contains nan. The following options are available (default is ‘propagate’):
          ‘propagate’: returns nan
          ‘raise’: throws an error
          ‘omit’: performs the calculations ignoring nan values

          alternative : {‘two-sided’, ‘less’, ‘greater’}, optional
          Defines the alternative hypothesis. The following options are available (default is ‘two-sided’):
          ‘two-sided’: the means of the distributions underlying the samples are unequal.
          ‘less’: the mean of the distribution underlying the first sample is less than the mean of the distribution underlying the second sample.
          ‘greater’: the mean of the distribution underlying the first sample is greater than the mean of the distribution underlying the second sample.

    Returns
        statistic : float or array
        t-statistic.

        pvalue : float or array
        The p-value.

From the above result, the assumed alpha value which is 0.05 is less than the obtained pvalue. Hence we can accpet the Null hypothesis H0, saying that the difference between two means of two groups is zero

2. F-Test¶

F-Test can be applied to test the significant difference between the variance of two populations, based on the small samples drawn from those populations. The test based on this statistic is known as F-Test.

Simply said, this F-test compares the variances of 2 values by perfroming division.The result of the f-test is always positive, because the variances are always positive. Let’s assume that the two variables are s1 and s2, the formula is considered as F = s1^2/s2^2

The Hypothesis for this F-test are defined as, Null Hypothesis (H0) : The variances of two variables are equal and there is no significant difference Alternate Hypothesis (HA) : The variances of two variables are not equal

F-Statistic, also known to be as F-Value is used in Analysis of Variance (ANOVA) and in regression models to find the significance between the means of the two populations by comparing variances. F-Statistic is used in F-test. F-Test is almost similar to T-test, except in F-test we check for the significance among group of variables, whereas in T-test, we check for the significance among 2 variables. F-test is used to check for the similarity among the means of different variables.

  For an F-test to be conducted we need to assume
       - that the data taken is normally distributed
       - The numerator variance should be larger and the denominator variance should be smaller

  There are many statistics in which F-Statistic is used, but mostly used F-test is Analysis Of Variance (ANOVA)

ANOVA : Analysis Of Variances, called as ANOVA, is used to test two or more than two groups of means differences. This ANOVA uses F-Statistic to calculate the difference between means of 2 groups or more than that

 The Hypothesis here is taken as,
         Null Hypothesis (H0) : The groups are significantly equal
         Alternate Hypothesis (H1) : The groups are not equal

   There are different types of ANOVA such as,
       - One-way ANOVA
       - Two-way ANOVA
       - Factorial ANOVA
       - Repeated Measures ANOVA
       - MANOVA etc.,

   The most used ANOVA is One-way ANOVA, which is used to compare groups mean with an independent variable to check whether the groups are likely or not 

   This test is performed by using a method f_oneway from scipy.stats as
       scipy.stats.f_oneway(*samples, axis=0)

       where, 
           samples can be any number of groups or array like variables
           axis defines along which the test to be performed, by default set to zero and is optional

       returns,
           statistic : float
           The computed F statistic of the test.

           pvalue : float
           The associated p-value from the F distribution.

 As per this if the pvalue is less than the alpha value (0.05) then the Null Hypothesis is rejected and if the pvalue is higher than the alpha value then the Null Hypothesis is accepted.

From the above obtained result, we can decide on to the output that the pvalue which is 0.4 is greater than alpha value (0.05). Hence we can say that there is no significant difference among the variances of different groups and are almost equal. And the Null Hypothesis is accepted

3. Correlation coefficients¶

The correlation coefficient is a statistical measure that shows the degree to which, changes to a value of one variable predict change to the value of another. The letter r is used to represent the correlation coefficient and the r is a unit-free value between -1 and 1.

Correlation coefficient measures the relatability among the data, the strength of that relationship is obtained by using these correlation coefficient formulas. The values obtained from these correlation coefficients are -1, 0 or 1 where, -1 represents a relationship which is negative and weak 1 represents a relationsip which is positive and strong 0 (zero) represents no relationship

Let’s suppose, there are two variables x and y, for which the correlation coefficient need to be found

• If the value of y goes up, when the value of x goes up, which means x is directly proportional to y then the correlation coefficient between x and y results out to be between 1 or positive values

• If the value of y goes down, whenever the value of x goes up or vice versa, which means x is inversely proportional to x then the correlation coefficient between x and y results to be -1 or negative values

• Though the value of y goes down, if there is no change in the other variable in our case it is x, then the correlation coefficient between these 2 variable results to be 0 (zero)

  For Example,

  Positive Correlation : If the quantity of milk increases, the price also increases Negative Correlation : If the price of a stock goes down, then the buying of that stock increases Zero Correlation : There is no relationship between score in video games and grades of an examination

  Before we dig into how the correlation among 2 or more coefficients is calculated. It is necessary to understand a term called covariance

  So, what is covariance? Covariance is a term that is used to describe the linear relationship between 2 variables. If the covariance is predicted to be positive, then the variables have a linear relationship i.e., both the variables can change in the same direction. If the covariance is predicted to be negative, then the variables don’t have that linear relationship i.e., both varibles tend to go in different directions

There are many types of correlation coefficients. But here we will discuss some of the important correlation coefficients that are widely being used

 1. Pearson’s r
         Pearson's r also known as Pearson's product momemnt correlation coefficient, is used for describing the strength of the linear relationship amomg 2 variables
         This correlation coefficient is used when the data follows normal distribution, with no outliers, with no skewed data and when you expect linear relationship among the 2 variables. 
         The Pearson'r formula is as follows

    The strength of the correlation is considered as
       - weak positive correlation as 0<r<0.3, weak negative correlation as -0.3<r<0
       - strong positive correlation as 0.5<r<1, strong negative correlation as -1<r<-0.5
       - no correlation as r=0

1. Spearman’s rho

Spearman’s rho, also known as Spearman’s rank correlation coefficient, is used as an alternative to the Pearson’s correlation coefficient. This correlation coefficient is a rank correlation coefficient as it uses the rankings to determine the strength of each varibal(say lowest to highest). Unlike Pearson’s r, Spearman’s rho is used to calculate monotonic relationship, which is Non-linear. Spearman’s rho formula is as follows

1. Kendall’s tau

Kendall’s tau is used for the calculation of correlation coefficient when there are 2 variables, which may be continuous variables with outliers or ordinals, but exhibiting monotonic relationship. Spearman’s rho and Kendall’s tau are almost similar but it is better to use Kendall’s tau for better results.

These 3 correlation coefficients can be calculated in Machine Learning by using a function in pandas which is

    DataFrame.corr(method='pearson', min_periods=1)

   Paramaters:
      method {'pearson','spearman','kendall')

Calculating correlation coefficient using pandas dataframe¶

4. Chi-Square Test¶

Chi-Square Test is a non-parametric test, which tests the significance of the difference between observed frequencies and theoretical frequencies of distribution without any assumption about the distribution of the population. In simple words, chi-square test is used to determine the difference between the expected data and the observed data

Chi-square test is also used to determine whether the built regression model is good fit or not by assessing train and test datasets. This test is used on categorical variables

Two Chi-Sqaure tests that are mostly used are

• Independence : As the name suggests, it describes the dependence of two variable sets

• Good Fit of data: This chi-square depicts whether the taken sample of a data is considered to be the representive sample that contributes to the good fit for the expected outcome from the taken population of data.

  The formula for the chi-square test is as follows

  In Chi-Square Test we assume two hypothesis,

     Null Hypothesis (H0) : The 2 variables are independent
     Alternate Hypothesis (HA) : The 2 variables are not independent
  
  

  We determine by performing chi-square test and come out to one reasonable hypothesis as the solution. This can be done by comparing the statistic value that is obtained after the test with the alpha value i.e., 0.05 using chi-square table with the help of pvalue.

  If the pvalue is less than alpha we accept the alternate hypothesis (HA). If the pvalue is greater than alpha then we accept the Null hypothesis (H0).

Expected value is calculated as Ei= (Row Total * Column Total)/Total no of observations

 In Machine learning, to perform chi-squared test we use a method named chisquare which is imported from scipy.stats.

       scipy.stats.chisquare(f_obs, f_exp=None, ddof=0, axis=0)
       where,
           f_obs - an array, with observed frequencies in each category
           f_exp - an array, optional; with expected frequencies in each category. If no array is given, it assumes the categories are almost likely
           ddof - int, optional; stands for delta degrees of freedom, which is used for adjustment to the degrees of freedom for obtaining p-value. The p-value is determined using a chi-squared distribution with k-1 delte degrees of freedom (ddof), where k is said to be the umber of observed frequencies. By defualt the ddof value is 0
           axis - int or None, optional; the axis along which to be applied test. If axis is given as none, then the f_obs values are treated as a single data set. Defaulted value is 0.
       Returns,
           chisq - float or ndarray, the value is a float if axis is None or f_obs or f_exp are 1-dimensional array
           p-value - float or ndarry, the values is a float if ddof and the return value chisq are scalars.

Importing Library¶

With f_obs values¶

With f_exp and f_obs values¶

With f_obs as 2d¶

With axis as None¶

With axis as 1¶

With ddof specified¶

The above results with the different degrees of freedom generates pvalues with values higher than the alpha value, so we accept the Null hypothesis (H0)