Linear Regression: Introduction

• Linear regression is a statistical method for fitting a line to data.

Linear Regression: Introduction

• Linear regression is a statistical method for fitting a line to data.

• Recall that a (non-vertical) line in the $x,y$-plane is determined by

$$y=\text{slope}\times x+\text{intercept}$$

Linear Regression: Introduction

• Linear regression is a statistical method for fitting a line to data.

• Recall that a (non-vertical) line in the $x,y$-plane is determined by

$$y=\text{slope}\times x+\text{intercept}$$

• There are two aspects to fitting a line to data that we will study:

Linear Regression: Introduction

• Linear regression is a statistical method for fitting a line to data.

• Recall that a (non-vertical) line in the $x,y$-plane is determined by

$$y=\text{slope}\times x+\text{intercept}$$

• There are two aspects to fitting a line to data that we will study:

  • Estimating the slope and intercept values, and

Linear Regression: Introduction

• Linear regression is a statistical method for fitting a line to data.

• Recall that a (non-vertical) line in the $x,y$-plane is determined by

$$y=\text{slope}\times x+\text{intercept}$$

• There are two aspects to fitting a line to data that we will study:

  • Estimating the slope and intercept values, and

  • Assessing the uncertainty of our estimates for the slope and intercept values.

Linear Regression: Introduction

• Linear regression is a statistical method for fitting a line to data.

• Recall that a (non-vertical) line in the $x,y$-plane is determined by

$$y=\text{slope}\times x+\text{intercept}$$

• There are two aspects to fitting a line to data that we will study:

  • Estimating the slope and intercept values, and

  • Assessing the uncertainty of our estimates for the slope and intercept values.

• In this lecture, we cover all of the concepts necessary to understand how to carry out and interpret linear regression.

Learning Objectives

• After this lecture, you should

Learning Objectives

• After this lecture, you should

  • Understand the basic principles of simple linear regression: parameter estimates, residuals, and correlation. (8.1, 8.2)

Learning Objectives

• After this lecture, you should

  • Understand the basic principles of simple linear regression: parameter estimates, residuals, and correlation. (8.1, 8.2)

  • Know the conditions for least squares regression: linearity, normality, constant variance, and independence. (8.2)

Learning Objectives

• After this lecture, you should

  • Understand the basic principles of simple linear regression: parameter estimates, residuals, and correlation. (8.1, 8.2)

  • Know the conditions for least squares regression: linearity, normality, constant variance, and independence. (8.2)

  • Know how to diagnose problems with a linear fit by least squares regression. (8.3)

Learning Objectives

• After this lecture, you should

  • Understand the basic principles of simple linear regression: parameter estimates, residuals, and correlation. (8.1, 8.2)

  • Know the conditions for least squares regression: linearity, normality, constant variance, and independence. (8.2)

  • Know how to diagnose problems with a linear fit by least squares regression. (8.3)

  • Understand the methods of inference of least squares regression. (8.4)

Learning Objectives

• After this lecture, you should

  • Understand the basic principles of simple linear regression: parameter estimates, residuals, and correlation. (8.1, 8.2)

  • Know the conditions for least squares regression: linearity, normality, constant variance, and independence. (8.2)

  • Know how to diagnose problems with a linear fit by least squares regression. (8.3)

  • Understand the methods of inference of least squares regression. (8.4)

  • Know how to obtain a linear fit using R with the lm function.

Simple Regression Model

• Simple linear regression models the relationship between two (numerical) variables $x$ and $y$ by the formula

$$y={\beta }_0+{\beta }_{1}x+ϵ$$ where

Simple Regression Model

• Simple linear regression models the relationship between two (numerical) variables $x$ and $y$ by the formula

$$y={\beta }_0+{\beta }_{1}x+ϵ$$ where

• ${\beta }_0$ (intercept) and ${\beta }_1$ (slope) are the model parameters

Simple Regression Model

• Simple linear regression models the relationship between two (numerical) variables $x$ and $y$ by the formula

$$y={\beta }_0+{\beta }_{1}x+ϵ$$ where

• ${\beta }_0$ (intercept) and ${\beta }_1$ (slope) are the model parameters

• $ϵ$ is the error

Simple Regression Model

• Simple linear regression models the relationship between two (numerical) variables $x$ and $y$ by the formula

$$y={\beta }_0+{\beta }_{1}x+ϵ$$ where

• ${\beta }_0$ (intercept) and ${\beta }_1$ (slope) are the model parameters

• $ϵ$ is the error

• The parameters are estimated using data and their point estimates are denoted by $b_0$ (intercept estimate) and $b_1$ (slope estimate).

Simple Regression Model

• Simple linear regression models the relationship between two (numerical) variables $x$ and $y$ by the formula

$$y={\beta }_0+{\beta }_{1}x+ϵ$$ where

• ${\beta }_0$ (intercept) and ${\beta }_1$ (slope) are the model parameters

• $ϵ$ is the error

• The parameters are estimated using data and their point estimates are denoted by $b_0$ (intercept estimate) and $b_1$ (slope estimate).

• In linear regression, $x$ is called the explanatory or predictor variable while $y$ is called the response variable.

Australian Brushtail Possum

Possum Data Example

• Suppose that we as researchers are interested to study the relationship between the head length (head_l) and total length (total_l) measurements of the brushtail possum of Australia.

Possum Data Example

• Suppose that we as researchers are interested to study the relationship between the head length (head_l) and total length (total_l) measurements of the brushtail possum of Australia.

• Note that head length (head_l) and total length (total_l) are both (continuous) numerical variables.

Possum Data Scatterplot

Possum Data Regression Line

• We have added the "best fit" line to the scatter plot of head_l versus total_l. Later we discuss how this line is obtained.

Possum Regression Residuals

Possum Regression Residual Plot

• A residual plot displays the residual values versus the $x$ values from the data.

Fits and Residual Plots

• Let's look some linear fits and their corresponding residual plots.

Fits and Residual Plots

• Let's look some linear fits and their corresponding residual plots.

Correlation

• Correlation, which always takes values between -1 and 1 is a statistic that describes the strength of the linear relationship between two variables. Correlation is denoted by $R$.

Correlation

• Correlation, which always takes values between -1 and 1 is a statistic that describes the strength of the linear relationship between two variables. Correlation is denoted by $R$.

• In R, correlation is computed with the cor command. For example, the correlation between the head_l and total_l variables in the possum data set is computed as

cor(possum$head_l,possum$total_l)

## [1] 0.6910937

Strongly Related Variables with Weak Correlations

• It is important to note that two variables may have a strong association even if their correlation is relatively weak. This is because correlation measures linear association and variables may be a strong nonlinear association.

Least Squares Regression

• We now begin to discuss the details of how to fit a simple linear regression model to data.

Least Squares Regression

• We now begin to discuss the details of how to fit a simple linear regression model to data.

• The approach we take is called least squares regression.

Least Squares Regression

• We now begin to discuss the details of how to fit a simple linear regression model to data.

• The approach we take is called least squares regression.

• The idea is to chose parameter estimates that minimize all of the residuals simultaneously. That is, for each observed data point $(x_i,y_i)$, we find $b_0$ and $b_1$ such that if ${\hat{y}}_i=b_0+b_1{x}_i$, then

$$RSS=\sum _{i=1}^n({\hat{y}}_i-y_i{)}^2$$

is as small as possible.

Conditions for Least Squares

• Linearity. The data should show a linear trend.

Conditions for Least Squares

• Linearity. The data should show a linear trend.

• Normality. Generally, the distribution of the residuals should be close to normal.

Conditions for Least Squares

• Linearity. The data should show a linear trend.

• Normality. Generally, the distribution of the residuals should be close to normal.

• Constant Variance. The variability of points around the least squares line remains roughly constant. Residual plots are a good way to check this condition.

Conditions for Least Squares

• Linearity. The data should show a linear trend.

• Normality. Generally, the distribution of the residuals should be close to normal.

• Constant Variance. The variability of points around the least squares line remains roughly constant. Residual plots are a good way to check this condition.

• Independence. We want to avoid fitting a line to data via least squares whenever there is dependence between consecutive data points.

Regression Assumption Failures

Regression Assumption Failures

• In the first column, the linearity condition fails. In the second column, the normality condition fails.

Regression Assumption Failures

• In the first column, the linearity condition fails. In the second column, the normality condition fails.

• In the third column, the constant variance condition fails. In the fourth column, the independence condition fails.

Fitting a Linear Model

• For simple least squares linear regression with one predictor variable ( $x$ ), one can fit the model to data "by hand". The mathematical formula is

$$b_1=\frac{s_y}{s_x}R,$$

$$b_0=\overline{y}-b_1\overline{x},$$

where

• $R$ is the correlation between $x$ and $y$,

Fitting a Linear Model

• For simple least squares linear regression with one predictor variable ( $x$ ), one can fit the model to data "by hand". The mathematical formula is

$$b_1=\frac{s_y}{s_x}R,$$

$$b_0=\overline{y}-b_1\overline{x},$$

where

• $R$ is the correlation between $x$ and $y$,

• $s_y$ and $s_x$ are the sample standard deviations for $y$ and $x$, and

Fitting a Linear Model

• For simple least squares linear regression with one predictor variable ( $x$ ), one can fit the model to data "by hand". The mathematical formula is

$$b_1=\frac{s_y}{s_x}R,$$

$$b_0=\overline{y}-b_1\overline{x},$$

where

• $R$ is the correlation between $x$ and $y$,

• $s_y$ and $s_x$ are the sample standard deviations for $y$ and $x$, and

• $\overline{y}$ and $\overline{x}$ are the sample means for $y$ and $x$.

Applying the Regression Formulas

• We need to compute the correlation, sample means, and sample standard deviations:

x <- possum$total_l; y <- possum$head_l
x_bar <- mean(x); y_bar <- mean(y)
s_x <- sd(x); s_y <- sd(y)
R <- cor(x,y)

Applying the Regression Formulas

• We need to compute the correlation, sample means, and sample standard deviations:

x <- possum$total_l; y <- possum$head_l
x_bar <- mean(x); y_bar <- mean(y)
s_x <- sd(x); s_y <- sd(y)
R <- cor(x,y)

• Now we can compute our estimates $b_0$ and $b_1$:

(b_1 <- (s_y/s_x)*R)

## [1] 0.5729013

(b_0 <- y_bar - b_1*x_bar)

## [1] 42.70979

The lm Command

• Let's see an example of how the lm command is used:

lm(head_l ~ total_l, data=possum)

## 
## Call:
## lm(formula = head_l ~ total_l, data = possum)
## 
## Coefficients:
## (Intercept)      total_l  
##     42.7098       0.5729

Interpreting Model Parameters

• For a linear model,

Interpreting Model Parameters

• For a linear model,

  • The slope describes the estimate difference in the $y$ variable if the explanatory variable $x$ for a case happened to be one unit larger.

Interpreting Model Parameters

• For a linear model,

  • The slope describes the estimate difference in the $y$ variable if the explanatory variable $x$ for a case happened to be one unit larger.

  • The intercept describes the average outcome of $y$ if $x=0$ and the linear model is valid all the way to $x=0$, which in many applications is not the case.

Interpreting Model Parameters

• For a linear model,

  • The slope describes the estimate difference in the $y$ variable if the explanatory variable $x$ for a case happened to be one unit larger.

  • The intercept describes the average outcome of $y$ if $x=0$ and the linear model is valid all the way to $x=0$, which in many applications is not the case.

• To evaluate the strength of a linear fit, we compute $R^2$ (R-squared). The value of $R^2$ tells us the percent of variation in the response that is explained by the explanatory variable.

Interpreting Model Parameters

• For a linear model,

  • The slope describes the estimate difference in the $y$ variable if the explanatory variable $x$ for a case happened to be one unit larger.

  • The intercept describes the average outcome of $y$ if $x=0$ and the linear model is valid all the way to $x=0$, which in many applications is not the case.

• To evaluate the strength of a linear fit, we compute $R^2$ (R-squared). The value of $R^2$ tells us the percent of variation in the response that is explained by the explanatory variable.

• There are some pitfalls in interpreting the results of a linear model. In particular,

Interpreting Model Parameters

• For a linear model,

  • The slope describes the estimate difference in the $y$ variable if the explanatory variable $x$ for a case happened to be one unit larger.

  • The intercept describes the average outcome of $y$ if $x=0$ and the linear model is valid all the way to $x=0$, which in many applications is not the case.

• To evaluate the strength of a linear fit, we compute $R^2$ (R-squared). The value of $R^2$ tells us the percent of variation in the response that is explained by the explanatory variable.

• There are some pitfalls in interpreting the results of a linear model. In particular,

  • Applying a model to estimate values outside of the realm of the original data is called extrapolation. Generally, extrapolation is unreliable.

Example

• Before we discuss further details of regression, let's look at a detailed example of fitting and interpreting a linear model.

Example

• Before we discuss further details of regression, let's look at a detailed example of fitting and interpreting a linear model.

• We will look at a linear model for the data set, cheddar from the faraway package.

Outlier Issues

• Outliers in regression are observations that fall far from the cloud of points.

Outlier Issues

• Outliers in regression are observations that fall far from the cloud of points.
• Outliers can have a strong influence on the least squares line.

Outlier Issues

• Outliers in regression are observations that fall far from the cloud of points.

• Outliers can have a strong influence on the least squares line.

• Points that fall horizontally away from the center of the cloud tend to pull harder on the line, so we call them points with high leverage.

Outlier Issues

• Outliers in regression are observations that fall far from the cloud of points.

• Outliers can have a strong influence on the least squares line.

• Points that fall horizontally away from the center of the cloud tend to pull harder on the line, so we call them points with high leverage.

• A data point is called an influential point if, had we fitted the line without it, the influential point would have been unusually far from the least squares line.

Inference for Regression

• Recall that a simple linear model has the form

$$y={\beta }_0+{\beta }_{1}x+ϵ$$

Inference for Regression

• Recall that a simple linear model has the form

$$y={\beta }_0+{\beta }_{1}x+ϵ$$

• Least squares is a method for obtaining point estimates $b_0$ and $b_1$ for the parameters ${\beta }_0$ and ${\beta }_1$. Thus, ${\beta }_0$ and ${\beta }_1$ are unknowns that correspond to population values that we want to infer information about.

Inference for Regression

• Recall that a simple linear model has the form

$$y={\beta }_0+{\beta }_{1}x+ϵ$$

• Least squares is a method for obtaining point estimates $b_0$ and $b_1$ for the parameters ${\beta }_0$ and ${\beta }_1$. Thus, ${\beta }_0$ and ${\beta }_1$ are unknowns that correspond to population values that we want to infer information about.

• A somewhat subtle point is that we also do not know the population standard deviation $\sigma$ for the error $ϵ$. This is an additional model parameter.

Inference for Regression

• Recall that a simple linear model has the form

$$y={\beta }_0+{\beta }_{1}x+ϵ$$

• Least squares is a method for obtaining point estimates $b_0$ and $b_1$ for the parameters ${\beta }_0$ and ${\beta }_1$. Thus, ${\beta }_0$ and ${\beta }_1$ are unknowns that correspond to population values that we want to infer information about.

• A somewhat subtle point is that we also do not know the population standard deviation $\sigma$ for the error $ϵ$. This is an additional model parameter.

• We would like answers to the following questions:

Inference for Regression

• Recall that a simple linear model has the form

$$y={\beta }_0+{\beta }_{1}x+ϵ$$

• Least squares is a method for obtaining point estimates $b_0$ and $b_1$ for the parameters ${\beta }_0$ and ${\beta }_1$. Thus, ${\beta }_0$ and ${\beta }_1$ are unknowns that correspond to population values that we want to infer information about.

• A somewhat subtle point is that we also do not know the population standard deviation $\sigma$ for the error $ϵ$. This is an additional model parameter.

• We would like answers to the following questions:

  • How do we obtain confidence intervals for ${\beta }_0$ and ${\beta }_1$, specifically how to we get the standard error?

Confidence Intervals for Model Coefficients

• Fact: The sampling distribution for estimates for ${\beta }_0$ and ${\beta }_1$ is a t-distribution. So we can obtain confidence intervals with

$$b_i\pm t_{\text{df}}^{\ast }\times S{E}_{b_i}$$

Confidence Intervals for Model Coefficients

• Fact: The sampling distribution for estimates for ${\beta }_0$ and ${\beta }_1$ is a t-distribution. So we can obtain confidence intervals with

$$b_i\pm t_{\text{df}}^{\ast }\times S{E}_{b_i}$$

• All the numerical information you need to obtain confidence intervals for ${\beta }_0$ and ${\beta }_1$ is provided in the output of the summary command for a linear model fit with lm.

Confidence Intervals for Model Coefficients

• Fact: The sampling distribution for estimates for ${\beta }_0$ and ${\beta }_1$ is a t-distribution. So we can obtain confidence intervals with

$$b_i\pm t_{\text{df}}^{\ast }\times S{E}_{b_i}$$

• All the numerical information you need to obtain confidence intervals for ${\beta }_0$ and ${\beta }_1$ is provided in the output of the summary command for a linear model fit with lm.

• Suppose we fit a linear model for the possum data again:

(lm_fit <- lm(head_l ~ total_l, data=possum))

## 
## Call:
## lm(formula = head_l ~ total_l, data = possum)
## 
## Coefficients:
## (Intercept)      total_l  
##     42.7098       0.5729

lm summary output

summary(lm_fit)

## 
## Call:
## lm(formula = head_l ~ total_l, data = possum)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -7.1877 -1.5340 -0.3345  1.2788  7.3968 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 42.70979    5.17281   8.257 5.66e-13 ***
## total_l      0.57290    0.05933   9.657 4.68e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.595 on 102 degrees of freedom
## Multiple R-squared:  0.4776,    Adjusted R-squared:  0.4725 
## F-statistic: 93.26 on 1 and 102 DF,  p-value: 4.681e-16

Regression CI Example

• Based on the information provided in the summary command output, we can construct confidence intervals for ${\beta }_0$ and ${\beta }_1$. First we observe that the degrees of freedom is 102, then the $t_{102}^{\ast }$ value for a 95% CI is

(t_ast <- -qt((1.0-0.95)/2,df=102))

## [1] 1.983495

Hypothesis Testing for Linear Regression

• There are actually several types of hypothesis tests one can conduct relating to linear regression models.

Hypothesis Testing for Linear Regression

• There are actually several types of hypothesis tests one can conduct relating to linear regression models.

• The most common test is of the form

  • $H_0:{\beta }_1=0$. The true linear model has slope zero. Versus

Hypothesis Testing for Linear Regression

• There are actually several types of hypothesis tests one can conduct relating to linear regression models.

• The most common test is of the form

  • $H_0:{\beta }_1=0$. The true linear model has slope zero. Versus

  • $H_A:{\beta }_1\ne 0$. The true linear model has a slope different than zero.

Hypothesis Test for possum Data

• Again, the output for summary(lm_fit) is

summary(lm_fit)

## 
## Call:
## lm(formula = head_l ~ total_l, data = possum)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -7.1877 -1.5340 -0.3345  1.2788  7.3968 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 42.70979    5.17281   8.257 5.66e-13 ***
## total_l      0.57290    0.05933   9.657 4.68e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.595 on 102 degrees of freedom
## Multiple R-squared:  0.4776,    Adjusted R-squared:  0.4725 
## F-statistic: 93.26 on 1 and 102 DF,  p-value: 4.681e-16

More R For Linear Models

• To fit a linear model in R we use the lm command. The necessary input is a formula of the form y ~ x and the data. The summary command outputs all of the information relevant for inferential purposes.

More R For Linear Models

• To fit a linear model in R we use the lm command. The necessary input is a formula of the form y ~ x and the data. The summary command outputs all of the information relevant for inferential purposes.

• However, the output from summary is not necessarily formatted in the most convenient way.

More R For Linear Models

• To fit a linear model in R we use the lm command. The necessary input is a formula of the form y ~ x and the data. The summary command outputs all of the information relevant for inferential purposes.

• However, the output from summary is not necessarily formatted in the most convenient way.

• Another approach to working with regression model output is provided by functions in the broom package. For example, the tidy function from broom displays the results of the model fit:

tidy(lm_fit)

## # A tibble: 2 x 5
##   term        estimate std.error statistic  p.value
##   <chr>          <dbl>     <dbl>     <dbl>    <dbl>
## 1 (Intercept)   42.7      5.17        8.26 5.66e-13
## 2 total_l        0.573    0.0593      9.66 4.68e-16

Comparing Many Means

• In previous lectures, we studied inference for a difference of means. There are also statistical methods for comparing more than two means. The primary method is called analysis of variance (ANOVA), see 7.5.

Comparing Many Means

• In previous lectures, we studied inference for a difference of means. There are also statistical methods for comparing more than two means. The primary method is called analysis of variance (ANOVA), see 7.5.

• ANOVA uses a single hypothesis to check whether the means across many groups are equal:

Comparing Many Means

• In previous lectures, we studied inference for a difference of means. There are also statistical methods for comparing more than two means. The primary method is called analysis of variance (ANOVA), see 7.5.

• ANOVA uses a single hypothesis to check whether the means across many groups are equal:

  • $H_0:{\mu }_1={\mu }_2=\cdots ={\mu }_k$. The mean outcome is the same across all groups. Versus

  • $H_A:$ At least one mean is different.

Comparing Many Means

• In previous lectures, we studied inference for a difference of means. There are also statistical methods for comparing more than two means. The primary method is called analysis of variance (ANOVA), see 7.5.

• ANOVA uses a single hypothesis to check whether the means across many groups are equal:

  • $H_0:{\mu }_1={\mu }_2=\cdots ={\mu }_k$. The mean outcome is the same across all groups. Versus

  • $H_A:$ At least one mean is different.

• We must check three conditions for ANOVA:

Example Data Motivating ANOVA

• Let's consider our chicken feed data again. The first few rows are shown below:

head(chickwts)

##   weight      feed
## 1    179 horsebean
## 2    160 horsebean
## 3    136 horsebean
## 4    227 horsebean
## 5    217 horsebean
## 6    168 horsebean

Plot of Data

chickwts %>% ggplot(aes(x=feed,y=weight)) + geom_boxplot()

The F Statistics and the F-Test

• Analysis of variance (ANOVA) is used to test whether the mean outcome differs across 2 or more groups.

The F Statistics and the F-Test

• Analysis of variance (ANOVA) is used to test whether the mean outcome differs across 2 or more groups.

• ANOVA uses a test statistic denoted by $F$, which represents a standardized ratio of variability in the sample means relative to the variability within groups.

The F Statistics and the F-Test

• Analysis of variance (ANOVA) is used to test whether the mean outcome differs across 2 or more groups.

• ANOVA uses a test statistic denoted by $F$, which represents a standardized ratio of variability in the sample means relative to the variability within groups.

• ANOVA uses an $F$ distribution to compute a p-value that corresponds to the probability of observing an $F$ statistic value that is as or more extreme than the sample $F$ statistic value under the assumption that the null hypothesis is true.

The F Statistics and the F-Test

• Analysis of variance (ANOVA) is used to test whether the mean outcome differs across 2 or more groups.

• ANOVA uses a test statistic denoted by $F$, which represents a standardized ratio of variability in the sample means relative to the variability within groups.

• ANOVA uses an $F$ distribution to compute a p-value that corresponds to the probability of observing an $F$ statistic value that is as or more extreme than the sample $F$ statistic value under the assumption that the null hypothesis is true.

• We will see how to conduct ANOVA and an $F$-test using R.

Conditions for ANOVA

• There are three conditions we must check for an ANOVA:

Conditions for ANOVA

• There are three conditions we must check for an ANOVA:

  • Independence. If the data are a simple random sample, this condition is satisfied.

Conditions for ANOVA

• There are three conditions we must check for an ANOVA:

  • Independence. If the data are a simple random sample, this condition is satisfied.

  • Normality. As with one- and two-sample testing for means, the normality assumption is especially important when the sample size is small. Grouped histograms are a good way to diagnose potential problems with the normality assumption for ANOVA.