United States District Court, District of Massachusetts
MEMORANDUM AND ORDER REGARDING DEFENDANTS’ MOTIONS TO EXCLUDE THE EXPERT TESTIMONY OF JOHN A. KILPATRICK AND STEVEN I. BUTLER (Dkt. Nos. 355 and 359)
MARK G. MASTROIANNI United States District Judge
Massachusetts Mutual Life Insurance Company (“Plaintiff”) brought eleven related actions against various defendants, asserting violations of Mass. Gen. Laws ch. 110A, § 410, the Massachusetts Uniform Securities Act (“MUSA”), for misstatements and omissions contained in the offering documents of certain residential mortgage-backed securities (“RMBS”). The instant action (11-cv-30039-MGM), brought against Deutsche Bank Securities Inc. (“DBSI”), Anilesh Ahuja, Michael Commaroto, Richard D’Albert, and Richard Ferguson (together, “Defendants”), was designated a “bellwether” case by Judge Saris on December 4, 2013. (See Dkt. No. 225.) Accordingly, it is scheduled to proceed through summary judgment and trial while the other cases are stayed.
Presently before the court are Defendants’ motions to exclude the expert testimony of John A. Kilpatrick and Steven I. Butler, under Federal Rule of Evidence 702 and Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579 (1993). For the following reasons, the court will grant in part and deny in part Defendants’ motion to exclude Dr. Kilpatrick, and deny the motion to exclude Mr. Butler.
Plaintiff brought these eleven actions in 2011, alleging that material misrepresentations were made in the sale of RMBS in violation of MUSA. RMBS are securities entitling the holder to income payments from pools of residential mortgage loans held by a trust. After the various mortgage loans are aggregated into loan pools and securitized, certificates representing the securitizations are sold to investors. Accordingly, the characteristics of the loans underlying the certificates, such as the loan-to-value (“LTV”) and debt-to-income (“DTI”) ratios, as well as the loan origination and underwriting standards,  affect the certificates’ value. In this case, Plaintiff alleges the offering materials for the certificates contained misrepresentations regarding (1) the underwriting guidelines and standards used to originate or acquire the mortgage loans underlying the certificates; (2) the appraisal standards used to value the mortgaged properties; and (3) the LTV ratios for the underlying loans.
On July 21, 2014, Plaintiff served the affirmative expert reports of Dr. Kilpatrick and Mr. Bulter, among others. Dr. Kilpatrick prepared two expert reports: in one, he assessed the accuracy of the original appraised values of a sample of properties underlying the certificates (“Sample Properties”) using an automated valuation model (“AVM Report”); in the other, he evaluated, for the subset of properties in the AVM Report which he found were significantly overvalued, whether the appraisals adhered to applicable appraisal standards using a credibility assessment model (“CAM Report”). Mr. Butler, for his part, prepared a report in which he analyzed, or “reunderwrote, ” the loans of the Sample Properties to determine whether they were originated or acquired in accordance with applicable underwriting guidelines in compliance with representations in the offering documents (“Butler Report”).
A. Dr. Kilpatrick
Dr. Kilpatrick has over 30 years of experience in finance and financial analysis, real estate, business development, statistical analysis, consulting, and teaching. (Dkt. No. 503, Decl. of John A. Kilpatrick (“Kilpatrick Decl.”) ¶ 6; Ex. 1 (“AVM Report”) at 3.) He earned a Bachelor of Science in business administration (accounting), a Master of Business Administration, and a Ph.D in real estate finance from the University of South Carolina. (Kilpatrick Decl. ¶ 7; AVM Report at 5; Ex. 3 at 2.) Dr. Kilpatrick currently teaches real estate finance as a visiting scholar at Baruch College, City University of New York. (Kilpatrick Decl. ¶ 7; AVM Report at 5.) In addition, he is an MAI designated member of the Appraisal Institute and is a state-certified (general) real estate appraiser in all 50 states as well as the District of Columbia. (Kilpatrick Decl. ¶ 8; AVM Report at 5.) In 2004, the Appraisal Qualifications Board (“AQB”) designated Dr. Kilpatrick as one of approximately 500 nationally certified appraisal standards instructors. (Kilpatrick Decl. ¶ 9; AVM Report at 6.) Dr. Kilpatrick also was nominated to the AQB and was named a Fellow of the Faculty of Valuation of the British Royal Institution of Chartered Surveyors. (Id.) He has authored numerous books and articles on real estate and serves as an editorial board member of two peer-reviewed publications: the Journal of Sustainable Real Estate and The Appraisal Journal. (Kilpatrick Decl. ¶ 10; AVM Report at 6.)
With regard to automated valuation models in particular, Dr. Kilpatrick has authored or co-authored several peer-reviewed publications on the subject. (Kilpatrick Decl. ¶ 12.) In addition to teaching, Dr. Kilpatrick is the Managing Director of Greenfield Advisors LLC, a real estate appraisal and economic consulting firm headquartered in Seattle, Washington. (AVM Report at 6.) Dr. Kilpatrick and the Greenfield Advisors staff have written and lectured extensively on the application of mass appraisal and statistical methods in class action litigation. (Id. at 6-7.) Moreover, Dr. Kilpatrick has testified as an expert in 41 actions in the past four years for both plaintiffs and defendants, and he has been qualified as a mass appraisal and real estate valuation expert in a number of cases. (Kilpatrick Decl. ¶ 13; AVM Report at 7; Ex. 5.)
Dr. Kilpatrick’s AVM Report describes his use of the Greenfield Automated Valuation Model (“GAVM”), an AVM he previously developed but adapted for this action, to assess the accuracy of the original appraised values of a sample of properties underlying the certificates. AVMs are computer programs that use statistical models to reach objective estimates of the market value of real property. (AVM Report at 27.) GAVM in particular uses a statistical hedonic regression model to perform retrospective real estate appraisals. (Kilpatrick Decl. ¶ 15.) Similar to a traditional appraisal, GAVM uses the sales prices of comparable properties to generate a value for the subject property. (Id. ¶ 18.) Unlike a traditional appraisal, however, GAVM uses 100 to 2, 000 sales transactions as comparables, rather than only three or six. (Id. ¶ 17; AVM Report at 30.) These comparable sales transactions are from the same county as the subject property, and the model limits the sales observation data to the geographically nearest sales transactions in the year preceding the effective date of the Sample Property’s appraisal. (Kilpatrick Decl. ¶ 19; AVM Report at 32.) As Dr. Kilpatrick explains in his AVM Report, “[b]ecause it relies on a larger number of comparisons and uses uniform computing logic, an AVM’s valuation results are inherently more accurate and objective” than a traditional appraisal, which is prone to subjectivity in selecting comparable properties and adjusting sales data. (AVM Report at 29-30.)
GAVM consists of two valuation sub-models: an ordinary least squares regression model (“OLS valuation model”), and an OLS model with a trend surface component (“OLSXY valuation model”). (Kilpatrick Decl. ¶ 19; AVM Report at 32.) The use of the trend surface component allows the model to incorporate spatial effects, i.e., value impacts due to location, into the valuation. (Id.) The required variables for the OLS model are tax assessed value, days between an origin date and sale, and days between an origin date and sale squared. (Kilpatrick Decl. ¶ 20; AVM Report at 41.) The OLS model can also accept home size, year built, lot size, and number of bathrooms, if sufficient data about these variables are available. (Id.) The required variables for the OLSXY model are tax assessed value, days between an origin date and sale, days between an origin date and sale squared, latitude, longitude, latitude squared, longitude squared, and latitude multiplied by longitude. (Kilpatrick Decl. ¶ 21; AVM Report at 41-42.) The OLSXY model can also accept home size, year built, lot size, and number of bathrooms, again, if these variables are available. (Id.) The data used to run the models was obtained from CoreLogic, which maintains the largest, most comprehensive property databases in the United States, as well as the loan tapes and loan files for the Sample Properties. (Kilpatrick Decl. ¶ 22; AVM Report at 20-22.) This data included previous sales prices, previous sales data, property characteristics such as footage, baths, age, and lot size, and location on a county-by-county basis, from January 1, 1998 through December 31, 2008. (Kilpatrick Decl. ¶ 22; AVM Report 22-23.)
Before running the Sample Properties through GAVM, Dr. Kilpatrick both calibrated and filtered the model using only the comparable properties. First, Dr. Kilpatrick “calibrated” GAVM, or tested its accuracy, using approximately 1.53 million sales from throughout the country. (Kilpatrick Decl. ¶¶ 26-27; AVM Report at 48.) For each of the 2, 783 counties in the dataset, he randomly split sales into a 10% set and a 90% set. (Id.) Then, he ran the 10% set through GAVM using the 90% set as the comparables. (Kilpatrick Decl. ¶ 28; AVM Report at 48.) Dr. Kilpatrick removed the following from his calibration analysis: transactions where the home was constructed after the sales date, because this indicated the sale was likely a land sale; transactions outside the middle 70th percentile of sales price to assessed value ratio, to eliminate suspect data and/or non-arm’s-length transactions; and properties with “errors” over 100%, because these likely indicated data errors. (Kilpatrick Decl. ¶ 28; AVM Report at 48-49.) Using his 10% set, Dr. Kilpatrick calculated GAVM’s coverage, or “hit rate, ” which measures how many of the properties the model was able to value, as well as the forecast standard deviation (“FSD”), which measures the model’s ability to predict known sales prices. (Kilpatrick Decl. ¶¶ 30-31; AVM Report at 50-52.) GAVM’s hit rate, without applying the 100% error filter, was 98.0%, which is much higher than the 80% hit rate Freddie Mac has noted is considered high in the industry. (Kilpatrick Decl. ¶ 30; AVM Report at 51.) Regarding FSD, of the approximately 1.53 million sales transactions valued by GAVM, 72.7% of the valuations were within 13% of the actual sales prices. (Kilpatrick Decl. ¶ 34; AVM Report at 52.) The FSD for GAVM was 15.1%, and the mean absolute error was 10.45%. (Id.)
Second, Dr. Kilpatrick used a “Cross-Validation Filter” to ensure that the comparable properties used in GAVM were properly valued. (Kilpatrick Decl. ¶ 42.) This filter eliminated comparable properties from GAVM if the sales price was at least 25% higher or lower than estimated using GAVM based on the remaining comparable properties in the same county. (Id.; AVM Report at 59-60.) Dr. Kilpatrick used this filter because he concluded that such properties either were not sold in arm’s-length transactions or had certain characteristics that significantly impacted the sales price which either were not knowable using the available data or which the GAVM model did not use. (Kilpatrick Decl. ¶ 42.) Accordingly, the Cross-Validation Filter was an attempt to remove outlier comparable properties which would unduly skew the analysis. (Id. ¶¶ 42-43.)
Dr. Kilpatrick then ran 924 of the 998 Sample Properties through both the OLS and OLSXY sub-models of GAVM. (AVM Report at 35-37, 60-62.) If only one sub-model produced a value, then that value constituted the final GAVM value prediction for the Sample Property, but if both sub-models produced a value, then the two values were averaged together to produce a final value prediction. (Id. at 35-36.) Using this process, Dr. Kilpatrick found the original appraised values of 211 of the 924 Sample Properties were more than 15.1% (the FSD discussed above) higher than their true, credible appraised values. (Kilpatrick Decl. ¶ 47; AVM Report at 62 n.158.) In addition, Dr. Kilpatrick found the original appraisals for the Sample Properties were systemically overvalued by a statistically significant average of 6.40% and that the 10% of supporting loan groups which were overvalued the most were overvalued by an average of 9%. (Kilpatrick Decl. ¶¶ 3, 47; AVM Report at 4, 63.) From this, Dr. Kilpatrick concluded that the LTV ratios for the Sample Properties were significantly higher than represented in the offering documents for the securitizations. (Id.) Dr. Kilpatrick further concluded that “no reasonable, competent appraisal professional adhering to appraisal standards applicable at the time could, in my opinion, validate the Deutsche Bank Appraisals as reliable, unbiased, or accurate.” (Kilpatrick Decl. ¶ 58; AVM Report at 66.)
In addition to his GAVM Report, Dr. Kilpatrick also prepared a CAM Report, in which he assessed the appraisals for 206 of the 211 Sample Properties identified in the GAVM Report as significantly overvalued. (Kilpatrick Decl. ¶¶ 59-60; Ex. 2 (“CAM Report”) at 2.) The purpose of the CAM Report was to determine whether these appraisals were “credible, ” i.e., whether they adhered to appraisal standards and practices pursuant to the Uniform Standards of Professional Appraisal Practice (“USPAP”). (Id.) To do this, Dr. Kilpatrick developed a Credibility Assessment Model (“CAM”) which evaluates the frequency and magnitude of an appraisal report’s deviation from settled appraisal standards and practice. (Kilpatrick Decl. ¶ 61; CAM Report at 2-3.) The CAM scores appraisals using a list of 31 questions based on USPAP and other professional guidance applicable to residential appraisals at the time. (Kilpatrick Decl. ¶ 67; CAM Report at 40.) The questions are phrased to elicit a “yes” or “no” response based on a review of information derived from loan tapes, loan files, and third-party sources. (Id.) A “no” response, which indicates that an appraisal process was not properly followed, is also given a specific weight or score based on the degree to which it affects the credibility of the appraisal. (Kilpatrick Decl. ¶ 69; CAM Report at 40 41.) Thus, a “no” response to one question may result in a score of 2.54, while a “no” response to another question may result in a score of 9.2. (Kilpatrick Decl. ¶ 72; CAM Report at 72, 89; Ex. 28.)
Dr. Kilpatrick also developed a scoring threshold, based on USPAP Standards Rules 1-1(b) and 1-1(c),  beyond which he concluded the adherence to applicable appraisal standards was so poor as to render the appraisal not “credible.” (Kilpatrick Decl. ¶ 74; CAM Report at 41, 98.) He concluded that a total score of 14.13 could render an appraisal not credible under USPAP due to either one major and one minor “no” response or a series of minor “no” responses. (Kilpatrick Decl. ¶ 76; CAM Report at 99.) In order to evaluate the appraisals in the most conservative light, Dr. Kilpatrick then raised this threshold score to 20, which means that an appraisal must elicit at least three “no” responses, two of which would have to be major questions, to be deemed not credible under the CAM. (Kilpatrick Decl. ¶ 80; CAM Report at 100-101.)
Of the 206 Sample Properties found to be overvalued by more than 15.1% using GAVM, Dr. Kilpatrick found 168 appraisals (81.55%) were not credible using CAM. (Kilpatrick Decl. ¶ 80; CAM Report at 101.) Dr. Kilpatrick explained that these appraisals contain sufficient errors such that, in his opinion, “no reasonable appraiser adhering to the appraisal standards and practices at the time could have believed the appraisals were credible.” (Kilpatrick Decl. ¶ 80; CAM Report at 101.) On average, the appraisals contained 6.38 errors, or “no” responses. (Kilpatrick Decl. ¶ 81; CAM Report at 102.)
In addition to using the CAM, Dr. Kilpatrick also had Recovco Mortgage Management, LLC (“Recovco”), a separate entity, independently review 205 of the appraisals for the Sample Properties found by GAVM to be significantly overvalued. (Kilpatrick Decl. ¶ 82; CAM Report at 103.) Recovco performed its standard collateral review to evaluate whether the appraisals adhered to USPAP, Fannie Mae, and industry standard appraisal guidelines. (Kilpatrick Decl. ¶ 83; CAM Report at 103.) Based on its evaluation, Recovco rendered a conclusion as to whether the appraisal was “weak” (not credible), “moderate” (of questionable credibility), or “strong” (credible). (Kilpatrick Decl. ¶ 83; CAM Report at 104.) It found 95.1% of the 205 appraisals were “weak” and demonstrated significant discrepancies indicative of a non-credible appraisal. (Kilpatrick Decl. ...