Learning Objectives

• After this lecture, you should

Learning Objectives

• After this lecture, you should

  • know how to fit a multiple regression model using lm,

Learning Objectives

• After this lecture, you should

  • know how to fit a multiple regression model using lm,

  • understand and be able to interpret adjusted $R^2$, and

Motivating Data

• Consider the mariokart data set which consists of auction data from Ebay for the game Mario Kart for the Nintendo Wii. This data was collected in early October 2009.

Motivating Data

• Consider the mariokart data set which consists of auction data from Ebay for the game Mario Kart for the Nintendo Wii. This data was collected in early October 2009.

head(mariokart)

## # A tibble: 6 x 12
##          id duration n_bids cond  start_pr ship_pr total_pr ship_sp  seller_rate
##       <dbl>    <int>  <int> <fct>    <dbl>   <dbl>    <dbl> <fct>          <int>
## 1   1.50e11        3     20 new       0.99    4        51.6 standard        1580
## 2   2.60e11        7     13 used      0.99    3.99     37.0 firstCl~         365
## 3   3.20e11        3     16 new       0.99    3.5      45.5 firstCl~         998
## 4   2.80e11        3     18 new       0.99    0        44   standard           7
## 5   1.70e11        1     20 new       0.01    0        71   media            820
## 6   3.60e11        3     19 new       0.99    4        45   standard      270144
## # ... with 3 more variables: stock_photo <fct>, wheels <int>, title <fct>

Glimpse of mariokart

glimpse(mariokart)

## Rows: 141
## Columns: 12
## $ id          <dbl> 150377422259, 260483376854, 320432342985, 280405224677, 17~
## $ duration    <int> 3, 7, 3, 3, 1, 3, 1, 1, 3, 7, 1, 1, 1, 1, 7, 7, 3, 3, 1, 1~
## $ n_bids      <int> 20, 13, 16, 18, 20, 19, 13, 15, 29, 8, 15, 15, 13, 16, 6, ~
## $ cond        <fct> new, used, new, new, new, new, used, new, used, used, new,~
## $ start_pr    <dbl> 0.99, 0.99, 0.99, 0.99, 0.01, 0.99, 0.01, 1.00, 0.99, 19.9~
## $ ship_pr     <dbl> 4.00, 3.99, 3.50, 0.00, 0.00, 4.00, 0.00, 2.99, 4.00, 4.00~
## $ total_pr    <dbl> 51.55, 37.04, 45.50, 44.00, 71.00, 45.00, 37.02, 53.99, 47~
## $ ship_sp     <fct> standard, firstClass, firstClass, standard, media, standar~
## $ seller_rate <int> 1580, 365, 998, 7, 820, 270144, 7284, 4858, 27, 201, 4858,~
## $ stock_photo <fct> yes, yes, no, yes, yes, yes, yes, yes, yes, no, yes, yes, ~
## $ wheels      <int> 1, 1, 1, 1, 2, 0, 0, 2, 1, 1, 2, 2, 2, 2, 1, 0, 1, 1, 2, 0~
## $ title       <fct> "~~ Wii MARIO KART &amp; WHEEL ~ NINTENDO Wii ~ BRAND NEW ~

A First Model

• As a start, we fit a linear model with the game condition (cond) as the only predictor:

Results

• Our first fit predicts that a used game will, on average, go for $10.90 less than a new game will.

Results

• Our first fit predicts that a used game will, on average, go for $10.90 less than a new game will.

Adding Predictors

• As we will see, in R it is extremely easy to fit a model with many predictors. Why might we want to do this?

Adding Predictors

• As we will see, in R it is extremely easy to fit a model with many predictors. Why might we want to do this?

• We would like to fit a model that includes all potentially important variables simultaneously.

Adding Predictors

• As we will see, in R it is extremely easy to fit a model with many predictors. Why might we want to do this?

• We would like to fit a model that includes all potentially important variables simultaneously.

• Multiple regression can help us evaluate the relationship between a predictor variable and the outcome while controlling for the potential influence of other variables.

Multiple Regression Model

• A multiple regression model is a linear model with many predictors. In general, we write the model as

$$\hat{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_k{x}_k$$

when there are $k$ predictors. We always estimate the ${\beta }_i$ parameters using statistical software.

Multiple Regression Model

• A multiple regression model is a linear model with many predictors. In general, we write the model as

$$\hat{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_k{x}_k$$

when there are $k$ predictors. We always estimate the ${\beta }_i$ parameters using statistical software.

• For example, we may want to use cond, stock_photo (whether the auction feature photo was a stock photo or not), duration (auction length, in days), and wheels (number of Wii wheels included in the auction) all as predictors of price for the mariokart data.

Results

• Notice that when we have controlled for other features, the condition (new versus used) of the game has a smaller impact on the price of the game since the slope estimate has gone from -10.90 to -5.13.

Results

• Notice that when we have controlled for other features, the condition (new versus used) of the game has a smaller impact on the price of the game since the slope estimate has gone from -10.90 to -5.13.

• For simple linear regression, we used $R^2$ to determine the amount of variability in the response that was explained by the model. Recall that

$$R^2=1-\frac{\text{variability in residuals}}{\text{variability in the response}}$$

Adjusted $R^2$

• The adjusted $R^2$ is computed as

$$R_{\text{adj}}^2=1-\frac{s_{\text{residuals}}^2}{s_{\text{response}}^2}\frac{n-1}{n-k-1}$$ where $n$ is the number of observations and $k$ is the number of predictor variables. Remember that a categorical predictor with $p$ levels will contribute $p-1$ to the number of variables in the model.

Adjusted $R^2$

• The adjusted $R^2$ is computed as

$$R_{\text{adj}}^2=1-\frac{s_{\text{residuals}}^2}{s_{\text{response}}^2}\frac{n-1}{n-k-1}$$ where $n$ is the number of observations and $k$ is the number of predictor variables. Remember that a categorical predictor with $p$ levels will contribute $p-1$ to the number of variables in the model.

• Notice that the adjusted $R^2$ will be smaller than the unadjusted $R^2$.

Adjusted $R^2$

• The adjusted $R^2$ is computed as

$$R_{\text{adj}}^2=1-\frac{s_{\text{residuals}}^2}{s_{\text{response}}^2}\frac{n-1}{n-k-1}$$ where $n$ is the number of observations and $k$ is the number of predictor variables. Remember that a categorical predictor with $p$ levels will contribute $p-1$ to the number of variables in the model.

• Notice that the adjusted $R^2$ will be smaller than the unadjusted $R^2$.

• One of the main benefits of using adjusted $R^2$ for multiple regression is that it accounts for model complexity.

Model Selection

• Model selection seeks to identify variables in the model that may not be helpful.

Model Selection

• Model selection seeks to identify variables in the model that may not be helpful.

• The model that includes all available explanatory variables is referred to as the full model.

Model Selection Strategies

• Backward Elimination. In this approach, we would identify the predictor corresponding to the largest $p$-value. If the $p$-value is above the significance level (usually $\alpha =0.05$), then we drop that variable, refit the model, and repeat the process. If the largest $p$-value is less than the significance level, then we would not eliminate any predictors.

Model Selection Strategies

• Backward Elimination. In this approach, we would identify the predictor corresponding to the largest $p$-value. If the $p$-value is above the significance level (usually $\alpha =0.05$), then we drop that variable, refit the model, and repeat the process. If the largest $p$-value is less than the significance level, then we would not eliminate any predictors.

• Forward Selection. This approach begins with no predictors, then we fit a model with each individual predictor one at a time and keep the predictor that has the smallest $p$-value. Forward selection proceeds by continuing to add at each step a predictor that results in the smallest $p$-value that is less than the significance level. When none of the remaining predictors can be added to the model and have a $p$-value less than the significance level, we stop.

Model Selection Example

• Let's see both backward elimination and forward selection applied to the mariokart data. Note that the full model is (in R formula notation) total_pr ~ cond + stock_photo + duration + wheels.

Model Selection Example

• Let's see both backward elimination and forward selection applied to the mariokart data. Note that the full model is (in R formula notation) total_pr ~ cond + stock_photo + duration + wheels.

• Let's work out the details together in R.

Model Selection Example

• Let's see both backward elimination and forward selection applied to the mariokart data. Note that the full model is (in R formula notation) total_pr ~ cond + stock_photo + duration + wheels.

• Let's work out the details together in R.

• Backward elimination and forward selection use $p$-values in deciding which variables will make up the final model. However, there are other measures that are used in other approaches to model selection. For example, one could seek a model that has the largest adjusted $R^2$ value. Information theoretic measures such as AIC and BIC are also often used. A discussion on these matters falls outside the scope of this course.

Checking Model Conditions

• Multiple regression methods using the model

$$\hat{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_k{x}_k$$ generally depend on the following four conditions:

Checking Model Conditions

• Multiple regression methods using the model

$$\hat{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_k{x}_k$$ generally depend on the following four conditions:

• the residuals of the model are nearly normal,

Checking Model Conditions

• Multiple regression methods using the model

$$\hat{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_k{x}_k$$ generally depend on the following four conditions:

• the residuals of the model are nearly normal,

• the variability of the residuals is nearly constant,

Checking Model Conditions

• Multiple regression methods using the model

$$\hat{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_k{x}_k$$ generally depend on the following four conditions:

• the residuals of the model are nearly normal,

• the variability of the residuals is nearly constant,

• the residuals are independent, and

Checking Model Conditions

• Multiple regression methods using the model

$$\hat{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_k{x}_k$$ generally depend on the following four conditions:

• the residuals of the model are nearly normal,

• the variability of the residuals is nearly constant,

• the residuals are independent, and

• each variable is linearly related to the response.

Histogram of Residuals

• A histogram of the residuals can be used to check for outliers. For example, the residuals for the final model for the mariokart data have the following histogram:

Absolute Value of Residuals

• A plot of the absolute value of residuals versus fitted values is helpful to check the condition that the variance of residuals is approximately constant.

Absolute Value of Residuals

• A plot of the absolute value of residuals versus fitted values is helpful to check the condition that the variance of residuals is approximately constant.

Additional Diagnostic Plots

• It can also be useful to examine the following types of plots:

Additional Diagnostic Plots

• It can also be useful to examine the following types of plots:

  • Residuals in the order of their data collection. Such a plot is helpful in identifying any connection between cases that are close to one another.

Additional Diagnostic Plots

• It can also be useful to examine the following types of plots:

  • Residuals in the order of their data collection. Such a plot is helpful in identifying any connection between cases that are close to one another.

  • Residuals against each predictor. We are looking for any notable change in variability between groups.

Further Regression

• When it comes to regression, we have only scratched the surface. There is more we could discuss regarding multiple regression, and there are also other types of regression. The following video provides an introduction to logistic regression