360
1 IN THE UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
2 EASTERN DIVISION
3 ERNEST T. BROWN, et al., )
)
4 Plaintiffs, )
) No. 95 C 1890
5 v. ) Chicago, Illinois
) November 20, 1997
6 CITY OF CHICAGO, ) 9:30 a.m.
)
7 Defendant. )
8
9 TRANSCRIPT OF PROCEEDINGS - TRIAL
10 BEFORE THE HONORABLE ROBERT W. GETTLEMAN
11 APPEARANCES:
12 For the Plaintiffs: KENNETH N. FLAXMAN, P.C.
122 South Michigan Avenue
13 Chicago, Illinois 60603-6107
BY: MR. KENNETH N. FLAXMAN
14
FUTTERMAN & HOWARD, CHTD
15 122 South Michigan Avenue
Chicago, Illinois 60603
16 BY: MR. CRAIG B. FUTTERMAN
17 For the Defendant: MAYER, BROWN & PLATT
190 South LaSalle Street
18 Chicago, Illinois 60603
BY: MR. JAMES HOLZHAUER
19 MR. JEFFREY S. PIELL
MS. ANGELA K. DORN
20 MR. ANDREW NICELY
21
22 Official Court Reporter: JENNIFER S. COSTALES, CSR, RMR
219 South Dearborn Street
23 Room 1744-A
Chicago, Illinois 60604
24 (312) 427-5351
25
361
Barrett - direct
1 (Proceedings heard in open court.)
2 THE CLERK: 95 C 1890, Ernest Brown versus City of
3 Chicago, on trial.
4 MR. HOLZHAUER: Your Honor, when we closed yesterday
5 you were asking us some questions about --
6 THE COURT: Somebody close the door, please.
7 MR. HOLZHAUER: You were asking us some questions
8 about schedule. Mr. Flaxman and I have conferred. Our
9 witnesses are all here and ready to go. I think, judging by
10 -- depending how long the cross is, we'll probably spend most
11 if not all of today with Barrett.
12 THE COURT: Remember, we break at 4:00 o'clock
13 today.
14 MR. HOLZHAUER: We break at 4:00. I think that will
15 probably be about the conclusion of Dr. Barrett, perhaps.
16 We have one long witness, Lieutenant Klein, who will
17 be going Monday. Our last two witnesses are very short
18 witnesses, depending on the length of cross.
19 We certainly feel we'll finish our case Monday or
20 perhaps Tuesday.
21 THE COURT: You know, if Dr. Barrett is almost
22 through today, but not quite, I had some matters cancel
23 tomorrow, so I could give you an hour or so tomorrow if you
24 just have to wrap him up.
25 MR. HOLZHAUER: All right. He has a plane going out
362
Barrett - direct
1 -- I think we'll be able to wrap him up.
2 THE COURT: I'm just saying that I have that
3 flexibility.
4 MR. FLAXMAN: It looks like we're going to end all
5 of the testimony a week from next Tuesday, which would give
6 you the Wednesday and Thursday of that week which you had
7 allocated to us to do other things.
8 THE COURT: Okay. Good.
9 MR. HOLZHAUER: I also understand, Mr. Flaxman, that
10 we might have mornings only on that Monday or Tuesday, judging
11 from your witnesses.
12 MR. FLAXMAN: We have one witness on Monday and one
13 on Tuesday, and we might have another one Monday afternoon.
14 And that's it.
15 THE COURT: All right. Well, fine. Okay.
16 Dr. Barrett, please take the stand. You are still
17 under oath, sir.
18 DR. GERALD V. BARRETT, DEFENDANT'S WITNESS, PREVIOUSLY SWORN
19 DIRECT EXAMINATION
20 RESUMED
21 BY MR. HOLZHAUER:
22 Q. Good morning, Dr. Barrett.
23 A. Good morning.
24 Q. When we ended yesterday, I believe we were reviewing the
25 oral briefing exercise; is that right?
363
Barrett - direct
1 A. Yes.
2 Q. How was the oral briefing exercise scored?
3 A. It was scored by three trained raters. The raters sat in
4 a room and independently rated each tape following a very
5 structured rating procedure.
6 After they had each rated each tape, they then
7 conferred and reached a consensus. If it was required, they
8 would play the tape back any number of times to be sure they
9 all agreed on the final score.
10 Q. You say a structured rating. Did they have a form that
11 they worked with?
12 A. Yes.
13 Q. How did you decide upon that scoring method for the oral
14 exercise?
15 A. It evolved over ten years in terms of our use of that
16 sort of procedure. Basically it's a very objective procedure
17 where there is little or no possibility of bias.
18 We have conducted research in the area of oral
19 communication, and we have found that in fact you can be
20 biased if there's -- if there is some lack of structure in the
21 rating process. So basically it's our -- based on our past
22 research and past experience, that's the reason we evolved
23 that sort of objective technique.
24 Q. You said that the raters first went through the
25 structured rating process. Was that independent, or did they
364
Barrett - direct
1 work together on that structured part?
2 A. That was independent.
3 Q. And then did they work together?
4 A. Yes. If there was any inconsistencies in their rating
5 process.
6 Q. Were the raters provided any information about the
7 identity of the voices on the tapes?
8 A. No. All you had was a test identification number.
9 Q. Was any analysis performed to assess whether the three
10 different versions of the exercise of the oral briefing
11 exercise were equivalent?
12 A. Yes.
13 Q. What was that?
14 A. We had forms X, Y and Z, and after all the ratings were
15 accomplished, we did a statistical approach to see if there
16 was any significant differences based upon form X, Y or Z. We
17 found no significant difference.
18 Q. Now, you testified earlier that the two other components
19 of the test were the written job knowledge component and the
20 written in-basket exercise. Were they administered on the
21 same day?
22 A. Yes.
23 Q. Well, let's start with the written job knowledge test.
24 Why did you decide to include a written job knowledge test in
25 the exam?
365
Barrett - direct
1 A. For several reasons. First, past research has shown the
2 importance of job knowledge for all jobs; and second, our job
3 analysis, which is consistent with other job analyses, showed
4 that in fact job analysis was critical and important for many
5 major work behaviors.
6 Q. Could you briefly describe for us the format of the test,
7 how many questions, how many choices, how much time and so
8 forth?
9 A. The job knowledge test consisted of 150 items. The
10 people, the candidates were allowed two and a half hours to
11 take the exam.
12 Q. Was this a multiple choice test?
13 A. Yes, it was a multiple choice test, a stem, and five
14 alternatives, one of which was correct.
15 Q. I'd like to show you a document that's been marked
16 Defendant's Exhibit 14, and it's been received in evidence.
17 I'd like you just to let me know what it is for
18 identification, please.
19 A. This is the Chicago police lieutenant written
20 examination.
21 Q. Thank you, Dr. Barrett.
22 Let's review the process by which you developed this
23 component of the exam.
24 Was the written exam designed to test any particular
25 body of job knowledge, skills and abilities?
366
Barrett - direct
1 A. Yes.
2 Q. What was it?
3 A. The domain of job knowledge was that which was contained
4 in our source material or reading list, which was given to all
5 candidates.
6 Q. How did you go about drafting those questions?
7 A. We had trained item writers who would review the
8 documents, review the job analysis and job descriptions, and
9 also look for areas where our job analysis indicated were
10 critical events, things which police lieutenants do do on the
11 job. So the stem often reflected an event, which did require
12 underlying knowledge to actually perform that activity.
13 Q. Dr. Barrett, you're using the term "stem." What do you
14 mean by that?
15 A. That's really the question, the question stem.
16 We had a very structured process in writing items.
17 Once an item was drafted, the alternatives, it was in a form,
18 and it was passed around and reviewed by item reviewers, it
19 was reviewed by four subject matter experts, and then the very
20 final review was my own review. I reviewed all the items.
21 Q. Who were these trained exam or question writers?
22 A. They were employees of Barrett and Associates. They all
23 had at least a master's degree in industrial psychology, had
24 all been trained to write items.
25 Q. Where were they from?
367
Barrett - direct
1 A. Akron, Ohio.
2 Q. They weren't Chicago police officials?
3 A. No.
4 Q. Did the item writers go through any particular training
5 for this job?
6 A. Yes, we have a structured proprietary training program.
7 Q. What did that consist of?
8 A. Basically it goes through all the item writing rules:
9 Here is how you should write an item in terms of the
10 structure, what the alternatives should be, how the
11 alternatives should be plausible, how you document the
12 process.
13 Q. Do they receive any diversity training?
14 A. Yes. We had a diversity training program for everyone,
15 just not item writers, but everyone who was working on the
16 project.
17 Q. Did you test for every bit of information on the reading
18 list material?
19 A. No.
20 Q. Would there be general orders and special orders and
21 other items on the reading list that were not the subject of
22 any test question?
23 A. Yes.
24 Q. How did that happen?
25 A. There's probably two main reasons. When you review the
368
Barrett - direct
1 material, you might find that the order is not amenable to
2 writing the question. Maybe it's a little bit vague or
3 whatever it might be.
4 A second consideration might be an item which was
5 written may be rejected as not being good enough by an
6 internal review process or by the subject matter experts. And
7 so you try to put in the most important material.
8 Q. Does the fact that some items on the reading list were
9 not tested diminish the validity of the test?
10 A. No.
11 MR. FLAXMAN: Objection. That assumes that there is
12 validity of the test.
13 THE COURT: I think the question is a fair
14 question. I understand the question. Overruled.
15 BY THE WITNESS:
16 A. No.
17 BY MR. HOLZHAUER:
18 Q. How did you decide what material to cover and how many
19 questions to ask regarding particular items on the reading
20 list?
21 A. Well, again, we tried to develop those situations for
22 which job knowledge was required and we could write a good,
23 professional item.
24 Q. Approximately how many written job knowledge questions
25 did your staff prepare?
369
Barrett - direct
1 A. We wrote several hundred. We pilot tested 186 before the
2 SMH reviewed it. So I don't know the exact number because
3 it's an additive process.
4 Q. Why did you select the multiple choice format?
5 A. This is a format which is very objective. There is no
6 possibility of bias.
7 Q. What do you mean by "objective"?
8 A. That means that there is one correct answer, it's
9 computer scored. It seemed to be very fair by candidates.
10 Q. Did you perform the same kind of linkage analysis that
11 you performed for the oral briefing exercise for the written
12 job knowledge test?
13 A. Yes.
14 Q. Once again, was the linking analysis -- what was the
15 linking analysis designed to demonstrate?
16 A. Just to show that everything we did test on was relevant,
17 we could link it to major work behaviors; that, in fact, one
18 concern is that you're going to be testing on something which
19 is not relevant to the job of the lieutenant. Of course, our
20 basic source knowledge list, of course, was by itself very
21 relevant, but just to be sure we were able to demonstrate this
22 was true.
23 Q. Were you able to link every one of the test items to
24 major work behaviors, knowledge, skills, and abilities?
25 A. Yes.
370
Barrett - direct
1 Q. Were the test questions and answers checked against the
2 source documents from the reading list?
3 A. Yes.
4 Q. By whom?
5 A. By our item writers and by the subject matter expert.
6 Q. Now, you mentioned a minute ago a pilot test. You ran a
7 pilot test of this exam?
8 A. Yes.
9 Q. How was that conducted?
10 A. We had ten, again, graduate students, eight whites, two
11 blacks, who went through the exam and took it looking for
12 inconsistencies, typos, various types of errors.
13 Q. Where did that take place?
14 A. That took place in Akron, Ohio, September 28 to 29th,
15 1994.
16 Q. Did you take any steps to ensure confidentiality of the
17 exam questions?
18 A. Yes. We took the same steps we did for the oral, of
19 people who were admitted into our secure quarters without any
20 sort of paper, pencil, et cetera. They were informed of the
21 confidential nature of what they were going to be doing. They
22 also signed a confidentiality agreement. And before they left
23 we were sure no pages of any of the tests were ever taken out
24 of the office.
25 Q. What use did you make of the pilot test results?
371
Barrett - direct
1 A. It helped us just to refine the test items.
2 Q. Did anyone from the Chicago Police Department review the
3 proposed test questions?
4 A. Yes.
5 Q. Who were they?
6 A. It was Chief Cadogan, Deputy Chief Shaw, Commander
7 DeLopez, and Lieutenant Klein. They came to Akron on October
8 1st through the 6th for the first review session.
9 Q. What did they do at that session?
10 A. We had the item writers individually go through each of
11 the items of the source material so they could see if in fact
12 the items were relevant. If they were any mistakes, they had
13 the source materials available, they could check the sources.
14 So, again, it was a check to be sure that the items
15 we wrote were relevant to the job of a Chicago police
16 lieutenant, and it was being -- the review was being performed
17 by people who were very knowledgeable about that position.
18 Q. You testified earlier that you actually prepared a good
19 deal of -- a number larger than 150 test questions initially.
20 Did the subject matter experts review all of those
21 questions?
22 A. Yes.
23 Q. Did they make recommendations that any of the questions
24 be revised or deleted?
25 A. Yes, they did.
372
Barrett - direct
1 Q. Did you follow those recommendations?
2 A. Yes, we did.
3 Q. Did they recommend any additional subject matter areas
4 that should be added to the test?
5 A. Yes.
6 Q. Did you follow those recommendations?
7 A. Yes.
8 Q. And you developed -- did you develop additional
9 questions?
10 A. Yes, as a matter of fact, we developed additional
11 questions, and in fact we revised for about two weeks, and
12 they came back for a second review session around the --
13 October 18th and 19th they came back, of 1994, for a second
14 review session, to look at the final form of the written job
15 knowledge test.
16 Q. So they reviewed the final form in that second review
17 session?
18 A. Yes.
19 Q. Did they review each and every item on the test?
20 A. Yes.
21 Q. You testified earlier that the written job exam was
22 administered on November 19, 1994. Were you present for that
23 administration?
24 A. Yes.
25 Q. Can you briefly describe how the test was administered?
373
Barrett - direct
1 A. Well, as always, the administration process was under the
2 control of Arthur Andersen. We came as a crisis team to
3 handle any unusual problems.
4 So again, it was a very structured process, highly
5 controlled, where we delivered the test booklets. We actually
6 produced the test booklets at our Akron office, transported by
7 truck with our people, taken to the Arthur Andersen facility,
8 which is also secure. It was delivered to the -- I believe it
9 was the Daley College, by Arthur Andersen.
10 Standardized instructions were given to the people,
11 the packets were passed out with the computer scoring sheet,
12 and they were given two and a half hours to complete.
13 There were monitors in all of the rooms. Security
14 was very, very tightly controlled.
15 Q. We'll come back to the timing in a second, but you
16 mentioned that you were there in case any particularly urgent
17 things came up.
18 Did any particularly urgent things come up during
19 the administration?
20 A. There were always perhaps a few things, nothing which was
21 -- compromised the validity of the testing process. There
22 were some complaints about a couple things about the rooms
23 perhaps, but nothing major for that written test.
24 Q. You said that -- how long did they have to complete that
25 exam?
374
Barrett - direct
1 A. That was -- the written test was two and a half hours.
2 Q. How did you decide on that time?
3 A. That's based on our past practice, and our pilot
4 studies.
5 Q. Did the Chicago Police Department subject matter experts
6 review the time allotted?
7 A. Yes.
8 Q. Did they give you any feedback on it?
9 A. They felt the time was correct, two and a half hours.
10 Q. Okay. Now, how was the test scored?
11 A. The test was computer scored by -- first I should say --
12 go back. What as I recall we did, the answer sheets were
13 scanned separately by ourselves and Arthur Anderson. We
14 brought a test scanner, and that data was entered onto a
15 computer file.
16 They separately -- Arthur Andersen scored the answer
17 sheets and retained the answer sheet. We went back to Akron
18 for our computer file. They went to their offices. They were
19 scored separately, and the scores were identical.
20 Q. So they were compared, the Arthur Andersen and the
21 Barrett and Associates scores were compared?
22 A. Yes.
23 Q. And you said they were identical?
24 A. Yes.
25 Q. Did the written job knowledge test test for knowledge
375
Barrett - direct
1 that a lieutenant must know in order to be effective on the
2 job?
3 MR. FLAXMAN: Objection. There is no basis for him
4 being able to answer this question.
5 MR. HOLZHAUER: This is based on his expertise, on
6 his job analysis, as well as a test developer.
7 MR. FLAXMAN: There should be a foundation for an
8 opinion of this nature.
9 THE COURT: I think he told us all the work he did
10 to develop the test, and how it related, in his opinion, to
11 the job of a police lieutenant. I think he is entitled to
12 give his opinion on this.
13 He may not like it or he may attack it, but I don't
14 understand why this would not be admissible.
15 MR. FLAXMAN: It's our position that there hasn't
16 been a foundation under Daubert, as some people pronounce it,
17 for this kind of opinion from this witness about this subject
18 matter.
19 THE COURT: I'm not asking you to respond. You're
20 reciting magic words here. What do you mean by that? He's
21 been -- don't you think he's qualified?
22 MR. FLAXMAN: No, I don't. I think there has to be
23 some real science before he can give an opinion about this.
24 He's being asked for his opinion about something which there
25 hasn't been an indication that there is a basis for the
376
Barrett - direct
1 opinion.
2 He's an expert about making up tests, but before he
3 can give an opinion that's admissible about whether something
4 is important for a lieutenant to know, there has to be some
5 scientific validity for his being able to give that opinion.
6 Daubert doesn't allow people who have gray hair and
7 who have been in the field 40 years to pontificate about
8 things without some scientific validity for their opinion.
9 THE COURT: Are you saying that a chemist couldn't
10 give an opinion about a chemical reaction unless he's actually
11 done some sort of experiment for me; is that it?
12 MR. FLAXMAN: No, a chemical reaction is much
13 different than this kind of social science.
14 THE COURT: Social science is subject to Daubert,
15 and it's subject to the normal inquiries about a person's
16 expertise. He's got the expertise. What would you have him
17 do?
18 MR. FLAXMAN: I think there has to be more of a
19 foundation. Certainly it would not be unreasonable for your
20 Honor to hear this subject to our further exploring the
21 Daubert foundation, but I don't think it would be --
22 THE COURT: You don't want to just let it in without
23 an objection, so that you can preserve your right later on to
24 cross examine him and attack his expertise and opinion?
25 MR. FLAXMAN: That's -- yes.
377
Barrett - direct
1 THE COURT: All right. I'm going to overrule the
2 objection.
3 MR. FLAXMAN: Can I have a standing objection?
4 THE COURT: Yes, you can have a standing objection.
5 Certainly I'm going to give you all the opportunity you want
6 to cross examine him and argue that the opinion should not be
7 given weight or whatever else you want to argue, or that it
8 doesn't meet Daubert. Right now I think it meets the
9 threshold of Daubert for me to hear it.
10 MR. FLAXMAN: I understand your Honor's ruling.
11 THE COURT: Thank you.
12 BY MR. HOLZHAUER:
13 Q. Dr. Barrett, did the written job knowledge test test for
14 knowledge that a lieutenant must know in order to be effective
15 on the job?
16 A. Yes.
17 Q. Now let's look briefly at the oral -- at the in-basket
18 exercise. Could you just briefly describe what the in-basket
19 exercise entailed?
20 A. The in-basket exercise entails a test of administrative
21 behavior, and the actual substance are a number of pages which
22 contains memos, directives, notes from a lieutenant.
23 The scenario is that a lieutenant becomes ill, he
24 becomes ill in the middle of a number of investigations, and
25 this is the instructions. So the new lieutenant is assigned
378
Barrett - direct
1 the job of taking over from the lieutenant who became ill, and
2 complete a number of assignments which the Chicago police
3 lieutenant actually perform on the job: review arrest
4 reports, for example, or analysis, apply rules. We included
5 in the packet various rules and regulations which were germane
6 to the problems we presented to the candidates.
7 They had two and a half hours to review the
8 material, read the memos. There were directives which said
9 basically -- from the commanding officer saying lieutenant,
10 role they are playing, you are to do X, Y, Z, various
11 functions.
12 And at the end of two and a half hours, they were
13 given 60 multiple choice questions based upon the events they
14 were doing, working on, the problems they were working on, and
15 they were given one and a half hours to complete that.
16 Q. Now, why did you decide to use an in-basket exercise?
17 A. For several reasons. Again, the general research shows
18 that in-basket exercise is the most frequently used assessment
19 center exercise, and there are numerous studies showing that
20 it's valid.
21 Second, our job analysis in the Chicago Police
22 Department showed the importance of these administrative
23 functions, and our past experience in police departments also
24 showed the importance of these functions.
25 Q. Did the in-basket exercise assess something different
379
Barrett - direct
1 from the written job knowledge test and the oral briefing
2 exercise?
3 A. Yes. This was all self-contained. You could answer all
4 the questions by the material in the packet they were given.
5 Q. What was it designed to assess?
6 A. The administrative ability, the ability to make these
7 decisions. Could they analyze, could they apply rules, could
8 they schedule.
9 Q. Dr. Barrett, I'd like to show you a document that's been
10 marked and received into evidence as Defendant's Exhibit 15,
11 just for identification purposes.
12 Can you tell me what that is?
13 A. This is a Chicago police lieutenant in-basket
14 simulation.
15 Q. So this is the in-basket test that you prepared?
16 A. Yes.
17 Q. What was the format for the answers to this exercise?
18 A. It was a multiple choice format with a computer score.
19 There was one right answer.
20 Q. Why did you --
21 A. I'm sorry. There was one right answer.
22 Q. Why did you decide to use a multiple choice format for
23 the in-basket exercise?
24 A. Because it's objective, there is no bias, and other
25 research had shown that it was a valid technique, and you
380
Barrett - direct
1 would obtain -- you obtain reliability and no possibility of
2 bias in the scoring of the test.
3 Q. Isn't it unusual to use multiple choice questions in
4 conjunction with an in-basket exercise?
5 A. It's not particularly unusual, no.
6 Q. Do you know whether other test developers use multiple
7 choice formats in in-basket exercises?
8 MR. FLAXMAN: Objection. This doesn't go to the
9 validity. This goes to what other people -- I don't know what
10 this goes to.
11 THE COURT: It goes to custom, usage.
12 MR. FLAXMAN: I don't know why that's relevant to
13 scientific validity.
14 THE COURT: I think your own witness testified about
15 very similar matters, answered very similar questions.
16 Overruled.
17 MR. FLAXMAN: Could we have more specificity rather
18 than just the general question about why other people do these
19 things: who they are, when they did it, where they did it?
20 THE COURT: I'm sure Mr. Holzhauer is going to
21 follow up to make it meaningful.
22 MR. HOLZHAUER: Yes.
23 BY MR. HOLZHAUER:
24 Q. Do you know whether other test developers used multiple
25 choice format for in-basket?
381
Barrett - direct
1 A. Yes.
2 Q. Can you tell us about that?
3 A. One of the first studies by Lopez in 1966, or 30 years
4 ago, used a structured format. In fact it was for, I believe,
5 a police lieutenant in the New York Port Authority.
6 And so that technique has been used right long as --
7 I suppose you could call it a psychometric technique, a
8 psychometric tradition for the in-basket.
9 There are articles more recent as 1977 by Haskinson,
10 which appeared in educational -- psychological measurement,
11 for example, showing the validity of a structured approach.
12 Q. Did you hear Dr. York testify that the test was not a
13 traditional in-basket test because it was a multiple choice
14 test?
15 A. Yes.
16 Q. Do you agree that it's not traditional?
17 A. Well again, the problem is at least with the word
18 traditional. As I said, there are at least in the literature
19 three different types. One is the more subjective rating
20 approach, but even that has what's called action items where
21 you actually rate the approach of psychometric approach, the
22 one I'm using, which is much more structured, and a
23 combination of the two.
24 I would say in the private sector you would often
25 have for hiring positions than what a structured approach
382
Barrett - direct
1 might be used.
2 Q. Does the fact that this used a multiple choice question
3 format diminish the reliability of the exam?
4 A. No.
5 Q. Does it change the reliability of the exam?
6 A. It tends to make the reliability higher.
7 Q. Why is that?
8 A. Because it's objective, objectively scored, and there is
9 no rater error in terms of two or three people rating and not
10 coinciding in the ratings of the responses.
11 Q. Does the fact that it used multiple choice questions
12 diminish its validity?
13 A. No.
14 Q. Why not?
15 A. Because, again, it's an objective approach, and when you
16 have a highly reliable objective approach, your validity tends
17 to increase.
18 Q. Did you hear Dr. York testify that in what he called
19 traditional in-basket exercises, after the exercise was
20 completed the raters discussed the results with the
21 candidate?
22 A. Yes.
23 Q. Did you consider using that approach for the Chicago
24 police lieutenants exam?
25 A. Yes.
383
Barrett - direct
1 Q. Why didn't you use it?
2 A. There was really no reason to use that approach. That is
3 a developmental approach, which is often a teaching approach
4 where you do an in-basket with managers in some organization,
5 and you then critique their style of management, for example,
6 and give feedback.
7 I've never heard of it being done in the public
8 sector. There is also practical considerations of the time
9 involved. It would be enormously expensive and
10 time-consuming.
11 Q. Did the candidates have to rely on job knowledge in the
12 in-basket test that was not included in the in-basket material
13 in order to complete the exercise?
14 A. No.
15 Q. Was this, in effect, an open-book test?
16 A. Yes.
17 Q. If it didn't require any knowledge that wasn't included
18 in the materials in front of the candidate, what did the test
19 assess? What did this exercise assess?
20 A. Well, it assessed basically their ability to perform
21 administrative functions of the Chicago police lieutenant.
22 Q. Were the materials patterned after materials actually
23 encountered on the job of the Chicago Police Department?
24 MR. FLAXMAN: Could we not have leading, Judge, at
25 this stage.
384
Barrett - direct
1 THE COURT: Sustained.
2 BY MR. HOLZHAUER:
3 Q. How were the materials designed?
4 A. As I previously testified, we had in our files all of the
5 arrest reports, various forms which were used by Chicago
6 police lieutenants.
7 We usually used the exact form that they would use
8 in the police department for our reports and other material.
9 The one exception, notable exception, would be the notes which
10 were left by the lieutenant who was ill. We thought about
11 having them handwritten. They were all typed.
12 In other words, for investigations they were
13 involved in, here were the notes the lieutenant had taken in
14 the process of that investigation but had not completed, so
15 the candidates could read the material.
16 Q. Why did you do it in that form, a typed out --
17 A. It's a very standardized way of doing it so every
18 candidate has the same -- pardon me. The same opportunity to
19 gain information.
20 It would be possible to give it in an oral fashion,
21 of course, but you have to have role players, which means it's
22 not as standardized.
23 Q. If you were to learn that watch commanders and other
24 Chicago lieutenants typically don't review a hundred pages of
25 documents at one time, would that reflect negatively on the
385
Barrett - direct
1 validity of your examination?
2 A. No.
3 Q. Why not?
4 A. Because every in-basket is a simulation. It's a test. I
5 don't care what type it is, what you are really doing is
6 compressing into a short period of time days or months of
7 work.
8 As an example, I started developing content valid
9 tests in 1962 while working for Goodyear Aerospace, and one of
10 my first jobs was to develop a test for the Mark 114 control
11 system. It's a control system for atomic submarines.
12 Now, you could have simulation where you're going
13 out in the Persian Gulf and you have no events. What you do
14 is you have model events occurring, so the person is very
15 active and busy.
16 The same philosophy is used in all in-baskets where
17 you in effect compress in a short period of time a lot of
18 material for the person to handle.
19 Q. Now, if you were to learn that watch commanders and other
20 Chicago police lieutenants don't typically review notes that
21 were prepared by other officers, would that reflect negatively
22 on the validity of the exam?
23 A. No.
24 Q. Why not?
25 A. Because the notes we had were standardized communication,
386
Barrett - direct
1 gave information which was needed for the candidate to solve
2 the problem, and it might be given in the actual situation in
3 oral form, but we wanted to be very standardized.
4 Q. Now, if you were to learn that watch commanders and other
5 Chicago lieutenants don't really have in-baskets on their
6 desks, would that reflect negatively on the validity of the
7 exam?
8 A. No.
9 Q. Why not?
10 A. Because we're talking about a process of solving
11 administrative problems.
12 Q. And if you were to learn that watch commanders and other
13 Chicago lieutenants don't have to answer multiple choice
14 questions on the job, would that reflect negatively on the
15 validity of the exam?
16 A. No.
17 Q. Why not?
18 A. Because to have a correct answer, they have to go through
19 and do the analysis, or do the scheduling, or apply the rules
20 to the problem. And once you have done that, then all they
21 have to do is, in fact, find in a multiple choice question,
22 mark the right response, correct response.
23 Q. You mentioned earlier that one simple example of a
24 content valid test would be a typing test for typists; is that
25 correct?
387
Barrett - direct
1 A. Yes.
2 Q. To be content valid, the tests have to replicate the
3 exact job --
4 MR. FLAXMAN: Object to the leading, Judge. The
5 witness can testify about this stuff, if that's his opinion.
6 MR. HOLZHAUER: I don't see that as leading, your
7 Honor.
8 THE COURT: I don't either. Overruled.
9 BY MR. HOLZHAUER:
10 Q. Do you recall the question?
11 THE COURT: Start over, please.
12 BY MR. HOLZHAUER:
13 Q. To be content valid, the tests have to replicate the
14 exact job tests, like a typing test does?
15 A. No, it does not.
16 Q. Why not?
17 A. Well, a typing test is a good example. There would be
18 very few typists who all they did was sit and type for eight
19 hours a day. In fact, clerical personnel perform a variety of
20 functions. They might answer the phone, might greet visitors,
21 they might do other clerical functions. And typing might be a
22 very important part of the job, but it's not all of the job.
23 Q. Was the in-basket exercise designed to be a precise
24 simulation of something Chicago police lieutenants actually
25 do?
388
Barrett - direct
1 A. It was a psychometric test. It was a simulation, yes.
2 Q. Was it designed to be similar to something a lieutenant
3 might do one day on the job, might happen to a lieutenant one
4 day on the job?
5 MR. FLAXMAN: Objection. Again, who's testifying
6 here?
7 THE COURT: Overruled.
8 BY THE WITNESS:
9 A. No. As I said before, it's a simulation, it's
10 compressed. It would not be one day on the job. It would be
11 a compressed activity which might occur over an extended
12 period of time to make it a psychometrically reliable and
13 valid test.
14 Q. How was the in-basket exercise developed?
15 A. We again relied upon our past experience, literature
16 reviews, and also, of course, the job analysis, which gave us
17 important information concerning the administrative activities
18 of a Chicago police lieutenant.
19 Q. Who designed the materials that were provided in the
20 in-basket?
21 A. We had a team who wrote or designed the in-basket, but
22 the actual material mostly came from passing out the forms,
23 and events. They came from our job analysis.
24 Q. Did you also conduct a pilot test of the in-basket
25 simulation?
389
Barrett - direct
1 A. Yes.
2 Q. Who took that pilot test?
3 A. Again, I believe there were nine graduate students, seven
4 whites and two blacks.
5 Q. Did you learn anything from that pilot test?
6 A. Yes.
7 Q. What did you learn?
8 A. We learned that, again, we fine-tuned and modified the
9 test.
10 Q. Did the pilot test-takers critique the exercise?
11 A. Yes.
12 Q. Did that result in any modification of the exercise?
13 A. Yes.
14 Q. Now, were the materials that were used and given to
15 candidates as part of the in-basket exercise, were they
16 reviewed by anyone at the Chicago Police Department?
17 A. Yes. Again, we had the two reviews those dates in
18 October, 4th, 5th and 6th were the first review where they
19 actually took the in-basket simulation, and that would be the
20 same four individuals: Chief Cadogan, Deputy Chief Shaw,
21 Commander DeLopez, and Lieutenant Klein. And they, based on
22 that, gave us feedback about the exercise.
23 Q. You mentioned they actually took the in-basket exercise
24 at that point?
25 A. Yes.
390
Barrett - direct
1 Q. Did they review all of the materials?
2 A. Yes.
3 Q. Did they give you any -- did they recommend any changes
4 in that material?
5 A. Yes, they recommended we cut down some of the materials,
6 not have so much material in the in-basket, and increase the
7 time.
8 Q. Did you follow that recommendation?
9 A. Yes, that's the reason they came back for a second review
10 period.
11 Q. Did they recommend that any of the materials that were
12 retained be modified?
13 A. What they would do is be sure it was realistic, yes. If
14 we were not quite on point, they wanted to be sure it
15 reflected what the Chicago police lieutenant actually did.
16 Q. You said they came back. When they came back, did they
17 review the actual exercise?
18 A. Yes.
19 Q. The exercise as given?
20 A. Yes.
21 Q. Did you perform a linkage analysis of the in-basket
22 exercise?
23 A. Yes.
24 Q. Were you able to link all of the test items to major work
25 behaviors and knowledge, skills, and abilities?
391
Barrett - direct
1 A. Yes.
2 Q. Is that linkage analysis contained in your validation
3 study?
4 A. Yes, as an outline in that study.
5 Q. I don't think I asked that question as to the job
6 knowledge test. Is the job knowledge test linkage contained
7 in your validation study?
8 A. Yes.
9 Q. What was the in-basket exercise designed to assess?
10 MR. FLAXMAN: Objection. Asked and answered.
11 THE COURT: Sustained.
12 BY MR. HOLZHAUER:
13 Q. You mentioned earlier that the in-basket exercise was
14 administered on the same day as the written job knowledge
15 exam.
16 Were you present for that exercise as well?
17 A. Yes.
18 Q. Can you briefly describe how the in-basket exercise was
19 administered?
20 A. Again, it was a very standardized process. The same
21 truck which brought the written job knowledge test brought the
22 in-basket.
23 Arthur Anderson had monitors, trained monitors at
24 the site. Each candidate was given their own desk in a room,
25 there was monitors in each and every room, and we were there
392
Barrett - direct
1 as a crisis team.
2 Q. Did any crises occur?
3 A. The usual administration complaints about maybe a piece
4 of paper is missing or something.
5 Q. Were those crises dealt with?
6 A. Yes.
7 Q. How long did the candidates have to review the
8 materials?
9 A. Two and a half hours.
10 Q. And how long did they have to complete the 60 questions?
11 MR. FLAXMAN: Objection. It's been covered. 60
12 minutes for the 60 questions -- an hour and a half.
13 THE COURT: An hour and a half.
14 MR. HOLZHAUER: I don't believe it has been covered
15 by this witness.
16 THE COURT: Yes, it has.
17 MR. HOLZHAUER: On this part?
18 THE COURT: Yes.
19 BY MR. HOLZHAUER:
20 Q. How did you decide on those times?
21 MR. FLAXMAN: That's been covered also.
22 MR. HOLZHAUER: Not on this exercise.
23 THE COURT: I don't think so, but let him go ahead.
24 If he did, he did.
25 BY THE WITNESS:
393
Barrett - direct
1 A. Based on our past experience, our pilot study and the
2 recommendations of the subject matter experts.
3 THE COURT: Were you there when the tests were
4 taken?
5 THE WITNESS: Oh, yes, I was physically there. All
6 the tests.
7 BY MR. HOLZHAUER:
8 Q. You mentioned the subject matter experts. Did they make
9 any recommendations on the timing?
10 MR. FLAXMAN: That I wrote down the answer to, and
11 they said it should be longer. That was two minutes ago.
12 THE COURT: Let him say it again just so I
13 understand it.
14 BY THE WITNESS:
15 A. Yes. Our first review they said we should increase the
16 time.
17 THE COURT: From what to what?
18 THE WITNESS: I don't recall the exact time.
19 THE COURT: And the time you ended up with was their
20 recommendation?
21 THE WITNESS: Yes.
22 BY MR. HOLZHAUER:
23 Q. Were the candidates allowed to refer to the materials
24 while they were completing the questions?
25 A. Yes.
394
Barrett - direct
1 Q. How many choices were there in the multiple choices?
2 A. Five.
3 Q. And how was this in-basket exercise scored?
4 A. It was computer scored also. The same process I
5 described for the written job knowledge test was used for the
6 in-basket, a computer scoring sheet. We individually scored
7 it in a personnel -- Arthur Andersen individually scored it on
8 their separate machines. We took our computer file back to
9 Akron, they took theirs to their offices, we scored them
10 separately, and we found they were identical.
11 Q. For both the written job knowledge test and the in-basket
12 exercise was there just one correct answer among the multiple
13 choice questions?
14 A. Yes.
15 Q. Did the in-basket exercise test for knowledge that a
16 Chicago police lieutenant needed to have in order to be
17 effective on the job?
18 MR. FLAXMAN: Objection. I think he's already
19 answered this question about what it was intended to test.
20 THE COURT: Overruled.
21 BY THE WITNESS:
22 A. Yes.
23 BY MR. HOLZHAUER:
24 Q. Dr. Barrett, I'd like to look at your report for a
25 minute, and I'd particularly like you to turn to Appendix L of
395
Barrett - direct
1 Exhibit 1.
2 Can you tell me what that is?
3 A. Item sample not used, item developed for the written job
4 knowledge examination.
5 Q. Turn to page L 2, please. You said, "item sample not
6 used." Was this a question not used then?
7 A. Yes.
8 Q. Why was it not used?
9 A. Because it overlapped with another item.
10 Q. Okay. Now, starting off on page L 2, there is a question
11 marked 52, and five possible multiple choice answers. Then at
12 the bottom it says 52 B. What does that mean, 52 B?
13 A. That means that the right answer is B.
14 Q. Following that it says general order 91-3, what does that
15 mean?
16 A. That's a source of the knowledge being tapped by this
17 item.
18 Q. Following that, there's a section called rationale, and
19 at the very end there are some -- a notation NWB: 2M4. Can
20 you tell us what all that means?
21 A. That refers to the major work behaviors which will be
22 found in the master job description we covered yesterday.
23 Q. Following that there's a section marked justification.
24 What does that entail or involve?
25 A. That's where the item writer goes through each of the
396
Barrett - direct
1 five alternatives and indicates why it is correct or not
2 correct.
3 Q. Now, the next page there is a second called reviewers
4 comments. What are they?
5 A. These are the reviewers comments which are internal, and
6 you'll also see SMEs which are the external reviewers.
7 Q. I notice in the reviewers comment section there is a
8 reviewer 2, reviewer 3, reviewer 4, reviewer 5. Who are they?
9 A. They are employees of Barrett and Associates.
10 Q. Different people?
11 A. Yes.
12 Q. Then there's a notation that says "929 pilot" and some
13 comments after that. What does that mean?
14 A. That was the result of our pilot study on that date, and
15 that is the P value or percent who got it correct, and looking
16 at the various alternatives, which were or were not endorsed.
17 Q. You said P value of .70. What does that mean?
18 A. That means that 70 percent of the people got it correct.
19 Q. Now, next there is -- it says 10-4 SME 1, and following
20 that 10-4 SME 2, and 10-10 SME 1 and 2. What is that all
21 about?
22 A. That refers to the subject matter experts. Those were
23 the four individuals from the Chicago Police Department who
24 reviewed all of the items. And what we did was we actually
25 wrote down their comments.
397
Barrett - direct
1 Q. Now, following there is something 10-13 reviewer No. 2.
2 What does that reflect?
3 A. Again, someone didn't like the item, liked something
4 better. Looks like we decided to use No. 18 instead of this
5 item.
6 Q. Now, was a similar documentation done for all of the
7 other written job knowledge questions?
8 A. Yes.
9 Q. Including the written job knowledge questions that
10 actually wound up on the exam?
11 A. Yes.
12 Q. Was a similar documentation done for the other components
13 of the exam?
14 A. Yes.
15 Q. Now --
16 THE COURT: Is this the only one that wasn't used?
17 THE WITNESS: No, no. Many were not used. There
18 were probably, I would guess, 50 which were not used.
19 BY MR. HOLZHAUER:
20 Q. Dr. Barrett, after all this was done, all this
21 administration took place, the three components were scored,
22 how did you deal with the scores?
23 A. The scores were standardized separately by ourselves and
24 Arthur Anderson, and they were added together.
25 Q. Standardized, what do you mean by that?
398
Barrett - direct
1 A. That means it's a process of taking one test, the written
2 job knowledge test had 150 items, maximum score, they all had
3 15. You put it on the same scale, basically, so it all had
4 the same weight. It's a standard procedure so that they all
5 have the same mean standard deviation.
6 Q. Is that a technical term for that standard procedure?
7 A. This was a Z score we used, transformation.
8 Q. Z score. Now, what was the purpose of exchanging the raw
9 score of each component to a Z score?
10 A. So we could equally weight the exams, so it would all be
11 on the same scale.
12 Q. How did you decide to equally weight the three
13 components?
14 A. Based on a number of factors. The first is the factor
15 that -- if you review the literature on weighting, you'll find
16 that there is a strong presumption from criterion-related
17 studies that equal weighting is the best way to proceed to
18 predict job performance.
19 The second is our job analysis, which indicated that
20 equal weighting could be appropriate. We also did a more
21 formal approach looking at the linkages from our job
22 analysis. We also did a computer simulation to look at some
23 possibilities.
24 Q. Was the weighting decided before or after the exam was
25 given?
399
Barrett - direct
1 A. Before.
2 Q. Why was that?
3 A. As part of the blue ribbon panel's recommendation, and as
4 our standard procedure, we always inform the candidates ahead
5 of time how the test is going to be weighted and scored.
6 Q. Was any thought given to modifying the weighting after
7 the exam results were obtained?
8 A. No.
9 Q. Did you run any computer simulations to test the
10 different weighting schemes?
11 A. Yes.
12 Q. Can you describe those computer simulations?
13 MR. FLAXMAN: Objection. He testified no thought
14 was given to it, so what he did is not relevant.
15 MR. HOLZHAUER: He testified that he decided on the
16 weightings based on computer simulations, among other things.
17 I would like to explore what those computer simulations were.
18 THE COURT: Overruled.
19 BY THE WITNESS:
20 A. For the sergeants examination we developed a theoretical
21 approach, really abstract, to try to take into account the
22 intercorrelations between various tests hypothetically before
23 the fact, their expected validity before the fact, and the
24 potential adverse impact. And we did a computer run using
25 those factors.
400
Barrett - direct
1 BY MR. HOLZHAUER:
2 Q. Were those simulations based on the actual exam results?
3 A. No.
4 Q. Were they based on the results of pilot exams?
5 A. No, not really.
6 Q. What do you mean by not really?
7 A. Well, we had some basic data from other tests and an
8 analysis, which we summarized. But it was not based on the
9 pilot data that we had collected in developing the test.
10 Q. I'd like you to turn to Appendix Q of the report that you
11 have in front of you.
12 What does that consist of?
13 A. That's our simulation. It's titled Chicago Police
14 Examination Weighting Rationale Final Report.
15 Q. Thank you. Following the administration of the test,
16 were any items challenged by candidates?
17 A. Yes.
18 Q. Did Barrett and Associates have any role in resolving
19 those challenges?
20 A. Yes.
21 Q. Can you describe how those challenges were resolved?
22 A. As I recall, from November 20th to about December 5th was
23 what was called an appeal period, where they could phone in to
24 Arthur Anderson objections or appeals.
25 I recall there were twelve appeals. Seven had to do
401
Barrett - direct
1 with administration of the test, four with the items itself.
2 There was one appeal concerning the in-basket, and three
3 concerning the written job knowledge test.
4 Q. Were any exam items modified or eliminated as a result of
5 the challenges?
6 A. Yes.
7 Q. Can you tell us about that?
8 A. A panel reviewed all the challenges, and I believe it was
9 a Chief Cadogan, Deputy Chief Shaw, Commander DeLopez, I
10 believe Commander Schraeder, and Dr. Cellar from our Chicago
11 office.
12 They found -- recommended to me that, I believe, two
13 items on the job knowledge test everyone should get correct
14 answers for it. And for the in-basket test, one item should
15 -- everyone should get a correct answer for it, points for
16 it, and one answer should have two correct answers.
17 Q. So for the job knowledge test everybody was given credit
18 for those two challenged questions?
19 A. Yes.
20 Q. And the other two were on the in-basket test?
21 A. Yes.
22 Q. For one, what happened?
23 A. For one everyone was given credit, and for the second
24 one, they were given two correct options. If you got A and B
25 correct, for example, you would get credit for that item.
402
Barrett - direct
1 Q. A or B?
2 A. Yes.
3 Q. Dr. Barrett, in your view, did the three-part examination
4 test a reasonable sample of the knowledge, skills, and
5 abilities required of police lieutenants?
6 MR. FLAXMAN: Could we note my objection on the
7 Daubert grounds?
8 THE COURT: I thought you said you wanted a standing
9 objection, and I said you could have one.
10 MR. FLAXMAN: I'll sit down. Thank you.
11 BY THE WITNESS:
12 A. Yes.
13 BY MR. HOLZHAUER:
14 Q. Did it test all of the knowledge, skills, and abilities
15 required of a Chicago police lieutenant?
16 A. No.
17 Q. Is it necessary, in your professional opinion, for a
18 promotional exam to test all of the knowledge, skills, and
19 abilities needed for the job?
20 A. No.
21 Q. Is it possible to assess all of the knowledge, skills,
22 and abilities required for a job?
23 A. Well, it's according to how complex a job it is.
24 Q. For the job of a Chicago police lieutenant.
25 A. I don't think you could assess everything, no.
403
Barrett - direct
1 Q. Could you give me a few examples of knowledge, skills and
2 abilities that you don't think it would be possible or
3 practical to assess?
4 A. Well, as Mr. Bishop indicated, he would like an integrity
5 test. I think it would be very difficult to develop an
6 integrity test and test the candidates' integrity to be a
7 Chicago police lieutenant.
8 Q. Is it possible to test for integrity?
9 A. We do have integrity tests, but there is an issue of its
10 reliability, its validity, and the stigma attached to someone
11 being labeled as lacking integrity to be a Chicago police
12 lieutenant.
13 Q. Dr. Barrett, your professional opinion, do the three
14 components that you developed collectively test for a
15 significant portion of job domain of lieutenant?
16 A. Yes.
17 Q. In your opinion, does the fact that certain knowledge,
18 skills, and abilities were not directly tested on the exam
19 diminish the content validity of the exam?
20 A. No.
21 Q. In your opinion, does the test that you developed conform
22 with the EEOC guidelines on selection procedure?
23 A. Yes.
24 Q. Do you discuss that in your report?
25 A. Yes.
404
Barrett - direct
1 Q. Dr. Barrett, I'd like to go over that section of your
2 report with you.
3 I believe it starts on page 137 of Exhibit 1. Is
4 that correct?
5 A. Yes, page 137 talks about professional guidelines.
6 MR. HOLZHAUER: I'm looking at page 137 of the
7 report, your Honor, which is Bates 144 of Exhibit 1.
8 BY MR. HOLZHAUER:
9 Q. I'd also like you to look at Appendix P of your report.
10 Stay on 137, but turn for a minute to Appendix P and tell me
11 what that is.
12 A. That Appendix P is a point-by-point reference to the EEOC
13 guidelines.
14 Q. Now, I see on page 137 at the end of the first paragraph
15 you indicate that a summary of a point-by-point reference to
16 the standards of the EEOC guidelines is presented in Appendix
17 O. Is that a typo?
18 A. Yes, it should be P.
19 Q. Now, I see there is a series of questions beginning on
20 page 137, with your answers. What are those questions based
21 on?
22 A. They're based upon a summary of the EEOC guidelines.
23 Q. Now, the first question is: "Were explicit linkages made
24 between the test's content and major work behaviors associated
25 with police lieutenants?" How do you answer that?
405
Barrett - direct
1 A. Yes.
2 Q. Are these the linkage tables you testified about
3 earlier?
4 A. Yes.
5 Q. The second question, which is on the next page, I
6 believe: "Were explicit linkages made between the frequency
7 or importance of major work behaviors for the master job
8 description and test items in the written job knowledge
9 examination?" How do you answer that?
10 A. Yes.
11 Q. Were explicit linkages made between the master job
12 description and the in-basket simulation and oral briefing
13 exercise?
14 MR. FLAXMAN: Objection. I think we covered this
15 once.
16 THE COURT: It's in the exhibit, the exhibit is in
17 evidence, right?
18 MR. HOLZHAUER: The exhibit is in evidence.
19 THE COURT: What good does it do to have him just
20 repeat it? If you want him to clarify something in it that
21 you think might need clarification or amplification, that's
22 fine, but let's not waste time.
23 BY MR. HOLZHAUER:
24 Q. Let me summarize it at least.
25 Dr. Barrett, did you go through the inquiries that
406
Barrett - direct
1 you believe were required by the EEOC guidelines in this
2 section of your report?
3 A. Yes.
4 Q. And did you believe that your test complied or met all of
5 those inquiries?
6 A. Yes.
7 Q. Now, in your opinion, does the test that you developed
8 conform with the APA standards for educational and
9 psychological testing?
10 A. Yes.
11 Q. In your opinion, does the test you developed conform with
12 the SIOP principles that you discussed earlier?
13 A. Yes.
14 Q. Dr. Barrett, you testified quite a bit ago that you were
15 using a content validation strategy to validate the
16 lieutenants exam; is that correct?
17 A. Yes.
18 Q. Were the candidates -- how were the candidates selected
19 for promotion off the list that resulted from this exam?
20 A. A rank order.
21 Q. Is content validation adequate to support use of an
22 examination for rank order promotions?
23 MR. FLAXMAN: Objection. The question is too
24 nonspecific to be relevant to this case. If it's made more
25 specific, then I stand on my Daubert objection, but I think
407
Barrett - direct
1 right now it's just posed as an abstract question which is not
2 necessarily material to this case.
3 THE COURT: I don't understand your objection.
4 MR. FLAXMAN: I think the general question of
5 whether content validity can be enough to support rank order
6 selection may or may not be -- is not what's at issue in this
7 case. What's at issue in this case is whether the content --
8 whether in this particular instance content validity is enough
9 for promotions in rank order.
10 And I would object to the general question without
11 being specific, and then I reassert the grounds that are
12 preserved with my standing objection.
13 THE COURT: Overruled. Ask the question again.
14 BY MR. HOLZHAUER:
15 Q. Is content validation adequate to support use of an
16 examination for rank order promotions?
17 A. Yes.
18 Q. On what do you base that opinion?
19 A. Base it on a number of factors. The first is the general
20 scientific literature which shows a positive relationship
21 between test scores and job performance.
22 Second, we have our job analysis, which shows in
23 some detail that more job knowledge, greater administrative
24 skills, oral communications, the more effective you are in
25 that, the more effective you expect to be on the job itself.
408
Barrett - direct
1 We also have the guidelines, EEOC guidelines,
2 standards and the SIOP principles all support that position.
3 We also have the fact that we did test a broad
4 representative basis of knowledge, skills, and abilities. In
5 other words, we did not just test job knowledge, we tested job
6 knowledge plus administrative skills, plus the oral
7 communication component of the job.
8 We also have evidence from the fact that once we
9 gave the test, it had the desirable psychometric properties
10 which allows you to rank order. For example, it was very
11 reliable. I think our overall reliability of test battery
12 is .93.
13 Second, we had good spread between the test scores
14 from high to low. Scores were not all bunched up at the top
15 and at the bottom end. So we did differentiate among the
16 candidates.
17 Those are the main supports for rank ordering.
18 Q. Are there any scholars, other scholars who support the
19 position that content validation strategy can be used for rank
20 order promotions?
21 A. Yes.
22 Q. Who are they?
23 A. Several come to mind. Dr. Landy, for example, Dr. Frank
24 Landy; Dr. Marvin Denette; Dr. Jenerette.
25 Q. Now, you mentioned the EEO guidelines. Did the EEOC
409
Barrett - direct
1 guidelines specifically address this issue?
2 A. Yes, they do.
3 Q. And the SIOP principles, do they specifically address
4 this issue?
5 A. Yes, they do.
6 Q. Do you recall briefly how the SIOP principles address
7 this issue?
8 A. I believe what they talk about in the SIOP principles is
9 if it's a broad representative sample of the job knowledge,
10 skills, and abilities, it being reliable, and that there is
11 some differentiation.
12 Q. Dr. Barrett, in an exam given to, let's say, 700 or more
13 candidates, would it be statistically possible to demonstrate
14 that the person who finished first on the list would perform
15 better than the person who finished second, who would perform
16 better than the person who finished third, and so on down to
17 No. 700?
18 A. Oh, in theory it's possible, but in reality it would
19 never happen. You'd have to have some indication that your
20 criterion or your major job performance is perfectly reliable,
21 your test is perfectly reliable, and your correlation between
22 test scores and job performance is 1.0. But that is not
23 feasible.
24 Q. Could you briefly describe what criterion -- did you
25 consider using a criterion-related strategy for this
410
Barrett - direct
1 examination?
2 A. Yes.
3 Q. Why didn't you use that kind of strategy?
4 A. Well, there were a number of reasons. As I described
5 yesterday, there are two different types of criterion-related
6 approaches. One would involve what's called concurrent
7 validation. I would have to take -- I have to develop my
8 test, give my test to all of the current lieutenants in
9 Chicago, give them the test, score it, and then obtain some
10 measure of their job performance.
11 The problem with that approach is at least twofold.
12 One is the confidentiality of the exam would be destroyed.
13 They would -- lieutenants would read questions, job knowledge
14 questions, they would memorize them, they would go back and
15 write them down and give them to their friends who are
16 sergeants.
17 And this effect occurred in the sergeants
18 examination. A matter of a very short time after we gave the
19 actual test, it was reproduced. So you have the idea of
20 confidentiality being destroyed.
21 The second problem in the modern police departments
22 is that people are very reluctant to rate people's job
23 performance knowing it might be reviewed in court. They feel
24 they might be accused of bias, for example. So that's a very
25 difficult process. It also would be a time-consuming process
411
Barrett - direct
1 to do that.
2 The second type of validation which is criterion
3 related would be a predictive study. That would mean we would
4 have to take a sample of sergeants with the test we have, a
5 random sample, give them the test, promote them to lieutenant,
6 and a year later see how well they perform as a lieutenant.
7 Again, we have the same issues about
8 confidentiality, and it's just not feasible nor practical.
9 Q. Dr. Barrett, did you ultimately see the results of this
10 exam?
11 A. Yes.
12 Q. Did you review the adverse impact of those results?
13 A. Yes.
14 Q. What were your reactions to the adverse impact?
15 A. I was disappointed that we had adverse impact.
16 Q. Now, after the results were received, did you do any
17 statistical analysis on alternative ways of using the test
18 results?
19 MR. FLAXMAN: Objection.
20 THE COURT: Before you object, I'd like to ask you a
21 question. Was there an adverse impact with all three
22 components?
23 THE WITNESS: Yes.
24 THE COURT: Okay. Thank you.
25 Ask your question again.
412
Barrett - direct
1 BY MR. HOLZHAUER:
2 Q. After the results were received, did you do any
3 statistical analysis on alternative ways of using the test
4 results?
5 MR. FLAXMAN: My objection is that he testified
6 previously that they never considered reweighting and any
7 statistical analysis would be irrelevant to have.
8 THE COURT: This is after the fact, and I think it
9 goes to your argument of alternatives, so I'm taking it in
10 that context.
11 BY MR. HOLZHAUER:
12 Q. Did you do any statistical analysis on alternative ways
13 of using the test results?
14 A. Yes.
15 Q. What are those alternative ways that you considered?
16 A. One way was a banding.
17 Q. Did you evaluate the adverse impact of banding?
18 A. Yes.
19 Q. Could you briefly describe what people in your field mean
20 when they refer to banding?
21 A. Well --
22 MR. FLAXMAN: Let me object and ask that he describe
23 what he means as banding.
24 THE COURT: That's a good objection. Sustained.
25 MR. FLAXMAN: Thank you.
413
Barrett - direct
1 BY MR. HOLZHAUER:
2 Q. Can you describe what you mean by banding? And I'll have
3 a follow-up question.
4 A. As Mr. Joyce described it yesterday, it's a relatively
5 new technique, and I call it a controversial technique, and
6 there is no one type of banding.
7 What you're in effect doing is using certain
8 statistical procedures to say, now I believe that the scores
9 between 90 and 100 should be treated the same. And in effect
10 you're, from my point of view, introducing random error into a
11 process. But that's the basic concept.
12 Q. Is there a substantial professional literature on
13 banding?
14 A. I'm not sure substantial, but at least there are a dozen
15 articles, I would guess, or half a dozen for sure.
16 Q. Does the substantial literature discuss banding in the
17 same way that you have discussed it?
18 A. They discussed a number of different types of banding,
19 yes.
20 Q. I would like you to look at Appendix R of your report.
21 Can you tell me what that is?
22 A. It's the Chicago police lieutenant banding report.
23 Q. Now, you mentioned that you evaluated the possibility of
24 banding for this test.
25 A. Yes.
414
Barrett - direct
1 Q. Look at page R 2. What size of band did you consider?
2 A. The first band included 50 people.
3 Q. How did you calculate that band size?
4 A. We went to the standard textbooks and used a standard
5 error of measurement to get a band, and we used -- applied
6 that to a lower band.
7 Q. And looking at page R 4, table R 5, did you examine what
8 the racial breakdown of that candidate in the first band would
9 be?
10 A. Yes. The first two bands you're saying, or first band?
11 Yes. The first band --
12 Q. Start with the first band.
13 A. The first band would be 47 white and three African
14 Americans, for a total of 50.
15 Q. Now, looking at that first band, would that banding
16 approach reduce adverse impact?
17 A. No, it would still be substantial adverse impact.
18 Q. Did you also calculate a second band?
19 A. Yes.
20 Q. What statistical technique did you use for that?
21 A. The same one we mentioned before.
22 Q. How many people are in that second band?
23 A. 153.
24 Q. Did you look at the racial breakdown of candidates in
25 that band as well?
415
Barrett - direct
1 A. Yes.
2 Q. Is that also in table R 5?
3 A. Yes.
4 Q. And what was that?
5 A. It would be 153 white, six African American, six
6 Hispanic, and no Asian Americans.
7 Q. Let's look back at table R 2. Can you tell me what that
8 shows?
9 A. This is the gender and ethnic group breakdown using a top
10 down selection procedure.
11 Q. Top down, do you mean the same thing as rank order?
12 A. Yes.
13 Q. And top down how far?
14 A. We were going down to 200.
15 Q. Did you compare that rank order group with what had
16 happened using statistical bands?
17 A. Yes.
18 Q. Can you look at tables R 6 and R 7 on page R 5. What are
19 they?
20 A. I'm sorry. What page are you on?
21 Q. Page R 5, tables R 6 and R 7.
22 A. We're looking at the expected numbers of each group from
23 band 2, R 7, expected number of candidates in each sub group,
24 approximately 200 candidates.
25 Q. Do they show the calculations determining the likely
416
Barrett - direct
1 composition of the first 200 promotees using the banding
2 approach?
3 A. Yes.
4 Q. And what's the result?
5 A. Well, basically you are actually -- have more adverse
6 impact. You have one less African American being chosen,
7 looks like.
8 Q. More adverse impact than what?
9 A. Than the top down.
10 Q. So would this banding approach reduce the adverse
11 impact?
12 A. No.
13 Q. From your analysis, was the adverse impact of this test
14 attributable to the fact that candidates were selected in rank
15 order?
16 A. No.
17 Q. In your professional opinion, do the three components of
18 the 1994 lieutenants exam collectively constitute a content
19 valid promotional exam?
20 A. Yes.
21 Q. In your opinion, do they collectively constitute a
22 content valid promotional exam when used for rank order
23 promotion?
24 A. Yes.
25 Q. In your professional opinion, was the written job
417
Barrett - direct
1 knowledge component a content valid exam component?
2 A. Yes.
3 MR. FLAXMAN: It's been asked and answered.
4 THE COURT: It has.
5 MR. HOLZHAUER: Have I asked it for all of the
6 components?
7 THE COURT: I think so.
8 BY MR. HOLZHAUER:
9 Q. May we establish that each of the components was content
10 valid, Dr. Barrett?
11 A. Yes.
12 Q. Thank you. In your professional opinion, would a
13 candidate receiving a higher score on this exam be likely to
14 show better job performance than one receiving a lower score
15 on the exam?
16 A. Yes.
17 Q. I'd like to show you a document we've labeled Defendant's
18 Exhibit 3. Can you tell me what that is, Dr. Barrett, when
19 you have a chance to look at it.
20 A. This is my response to the expert witness reports and
21 depositions of Drs. York, Koziol, and Levitt, and Mr. Bishop,
22 in the matter of Brown v. Chicago, October 13, 1997.
23 Q. So it includes your response to Mr. Bishop as well as the
24 others?
25 A. Yes.
418
Barrett - direct
1 Q. I'd like you to turn to page 52, table 5. Can you tell
2 me what that is?
3 A. This is the table rank of plaintiffs on job knowledge
4 from the Chicago police lieutenant test, where we eliminate
5 challenged items.
6 Q. What is column R 11 of this chart?
7 A. This is Bishop's critique of, I believe, all his
8 critiques, which includes test wiseness, not actual procedure,
9 reference to not essential, ambiguous, watch commander, police
10 officer, assorted items. I believe it's a compilation of all
11 his critiques.
12 Q. What does R 11 have -- do with those critiques?
13 A. What we're doing is -- say all right, what would happen
14 if we take out all the items that Mr. Bishop critiqued and
15 reranked the people on the new test.
16 In other words, we're saying all right, Mr. Bishop,
17 we're going to see if -- what's going to occur if we take out
18 all of the items you say are not good, will this actually
19 change or modify the ranking on this test.
20 MR. FLAXMAN: My objection to further questions
21 about this is it's not relevant now that we know what it's
22 about.
23 THE COURT: Why?
24 MR. FLAXMAN: We don't have to prove that anybody,
25 that any plaintiff would have been promoted but for any
419
Barrett - direct
1 particular questions. We have to prove that the test had an
2 adverse impact on minorities. We proved that, that's been
3 conceded. On the question of whether leaving out three
4 questions would have had a different result is not a defense.
5 It's not material to either the defense in this case or the
6 plaintiffs' case.
7 THE COURT: Well, you say that, but why? They have
8 to prove it's content valid, and they're doing that by showing
9 that it would be the same results, I assume, or similar
10 results even if they met your criticism, which you put on in
11 your case. They're meeting your case.
12 MR. FLAXMAN: I don't think that's part of our
13 case. Our case --
14 THE COURT: Well, that's what your witness testified
15 to. They went through these questions one by one and said
16 they were not related to the job. Now they're saying all
17 right, we'll take them out and let's see if it affects the job
18 at all. It may not be conclusive, but I think it's probative.
19 MR. FLAXMAN: I don't think it's even material. I
20 think this is something we have to educate you about as we go
21 on with this, and I --
22 MR. HOLZHAUER: Your Honor, we may be able to deal
23 with this in the briefs, but I think Mr. Flaxman takes the
24 position basically if he can show one question is not job
25 related, even if 100 percent of the exam candidates got it
420
Barrett - direct
1 right, that the exam is not valid. I don't agree with that.
2 THE COURT: I'm overruling the objection.
3 BY MR. HOLZHAUER:
4 Q. Dr. Barrett, as a result of eliminating all the questions
5 Mr. Bishop challenged in column R 11, did any of the
6 plaintiffs move up to the top 200 on that list?
7 A. No.
8 Q. Let's look at the first line, just as an example. We're
9 not going to go through this list, certainly.
10 The first line lists Diane Thompson, who testified
11 earlier. How would elimination of those questions affect her
12 ranking on the job knowledge test?
13 A. On R 1, on that column, across, you'll see she's ranked
14 200.5. You go across to R 11, her rank drops to 262. So she
15 would drop in rank when you eliminate the questions challenged
16 by Mr. Bishop.
17 Q. Were some of the other plaintiffs also moved down as a
18 result of eliminating those questions?
19 A. Yes.
20 Q. Were some of them moved up?
21 THE COURT: Excuse me. Can you tell me again, what
22 is R 1?
23 THE WITNESS: That's the actual rank as the test was
24 given, all 150 items. We're not taking any out. That's how
25 it was actually scored and used for rank orders.
421
Barrett - direct
1 THE COURT: And rank 11 would be taking -- rank 12
2 would be taking all of them out that they criticized?
3 THE WITNESS: Yes. Basically we're taking all the
4 items out in R 11, that column.
5 THE COURT: R 11?
6 THE WITNESS: Yes. R 12 I believe includes
7 plaintiff Holly Robinson had additional challenges, I
8 believe. I believe from her deposition, I assume.
9 THE COURT: Okay.
10 BY MR. HOLZHAUER:
11 Q. Now, I believe you testified that other plaintiffs did
12 move down. My next question was: Did other -- some of the
13 plaintiffs move up as a result?
14 A. Yes.
15 Q. Did you look at how many improved as a result and how
16 many lost ground?
17 A. Yes.
18 Q. What did you find?
19 A. About half improved, about half lost ground. I believe
20 one stayed the same.
21 Q. But nobody went into the top 200?
22 A. No.
23 Q. Dr. Barrett, were you in the courtroom when Diane
24 Thompson testified?
25 A. Yes.
422
Barrett - direct
1 Q. Did you hear her testify that she would have been
2 promoted had the in-basket been eliminated?
3 A. Yes.
4 MR. FLAXMAN: I don't think -- I didn't hear her
5 testify to that, Judge. She testified if she had gotten 60 in
6 the in-basket she would have been promoted.
7 THE COURT: I think you're wrong. I think she said
8 that: "If I hadn't taken it, I would have been promoted." I
9 remember that. Let's assume that's the case. If I'm wrong,
10 we'll do it as a hypothetical.
11 MR. FLAXMAN: If I'm wrong, I apologize.
12 BY MR. HOLZHAUER:
13 Q. Did you do any analysis over the last two days to see
14 what would have happened had the in-basket been eliminated?
15 A. Yes.
16 Q. Did you review that analysis in preparation for your
17 testimony today?
18 A. Yes.
19 MR. FLAXMAN: I would ask that we defer this
20 questioning until that analysis has been produced to me and I
21 have a chance to look at it and think about it rather than
22 having it been sprung in direct examination.
23 THE COURT: I think that's fair. Let's wait until
24 after lunch, and I'll let you reopen your examination. I
25 think that's fair.
423
Barrett - direct
1 MR. HOLZHAUER: Okay.
2 I have no further questions, Dr. Barrett.
3 MR. FLAXMAN: Can we take a short break before we
4 start.
5 THE COURT: Why don't we. We'll take a short
6 break.
7 Why don't you give him that analysis now,
8 Mr. Holzhauer. Maybe then we could start with you're asking
9 him those questions, and then go right into cross
10 examination.
11 Let's make it around five minutes.
12 (Short recess.)
13 THE COURT: Did you get your material, Mr. Flaxman?
14 MR. FLAXMAN: Yes. I'm satisfied that he could ask
15 those questions about the new exhibit.
16 THE COURT: All right. Dr. Barrett, please take the
17 stand. Please be seated.
18 All right. Mr. Holzhauer, you want to ask a few
19 more questions?
20 MR. HOLZHAUER: Yes, I do.
21 BY MR. HOLZHAUER:
22 Q. Dr. Barrett, if the in-basket had been eliminated, would
23 Diane Thompson have been promoted among the top 108 scores?
24 A. No.
25 Q. If the in-basket had been eliminated, would any of the 44
424
Barrett - direct - cross
1 plaintiffs have been promoted among the top 108 scores?
2 A. No.
3 Q. Did you see what happened or look to see what happened to
4 Sergeant DeSalvo if the in-basket was eliminated?
5 A. Yes.
6 Q. Do you recall what happened to Sergeant DeSalvo?
7 A. He would drop dramatically in the ranking.
8 Q. Did you also, over the last several days, look at the
9 rank ordering for the entire exam battery that would result
10 from elimination of the 25 items that Mr. Bishop challenged on
11 the stand?
12 A. Yes.
13 Q. As a result of elimination of those items, would any of
14 the plaintiffs have been promoted among the top 108 scores?
15 A. No.
16 MR. HOLZHAUER: No further questions, your Honor.
17 THE COURT: Mr. Flaxman?
18 MR. FLAXMAN: Thank you, Judge.
19 CROSS EXAMINATION
20 BY MR. FLAXMAN:
21 Q. Let's stay with this computation of scores while we're
22 there.
23 Did you have occasion to recompute where --
24 THE COURT: Are you talking about page 52 of
25 Exhibit --
425
Barrett - cross
1 MR. FLAXMAN: I'm not talking about any exhibit, I'm
2 just talking about the general question of recomputation.
3 BY MR. FLAXMAN:
4 Q. Did you have occasion to recompute where Ms. Thompson
5 would be if she scored 60 on the in-basket?
6 A. No.
7 Q. Is that possible to do, Dr. Barrett?
8 A. Yes.
9 Q. How would you go about doing that?
10 A. I would enter in the computer program for the in-basket
11 the score of 60, and have an appropriate Z score
12 transformation made and add them together, and get a final
13 rank order for her.
14 Q. Would you need a computer to do that, Dr. Barrett?
15 A. I would need a computer.
16 Q. Well, are you familiar with the procedures that the
17 computer would follow to come up with that final score?
18 A. In general.
19 Q. Well, specifically, the computer would compute the
20 standard score for Ms. Thompson's written score; is that
21 right?
22 A. You mean the job knowledge test?
23 Q. Right.
24 A. Yes.
25 Q. And it would compute the standard score for the orals,
426
Barrett - cross
1 for her oral -- for her score on the oral component; is that
2 right?
3 A. Yes.
4 Q. And the computer would also compute the in-basket
5 standard score for a score of 60; is that right?
6 A. Yes.
7 Q. And is it complicated to compute the standard score,
8 Dr. Barrett?
9 A. Not too complicated.
10 Q. Well, could you tell us how you would compute the
11 standard score for the written score, the score on the job
12 knowledge test?
13 A. You would take the mean of distribution minus the score
14 of the individual and derive the standard deviation in
15 general.
16 Q. Now, in your report, Defendant's Exhibit 1, you report,
17 don't you, what the mean of the job knowledge test score was,
18 don't you?
19 A. I think we do.
20 Q. And do you recall that the mean was 109.89?
21 A. No.
22 Q. Do you think if you looked at the report it would refresh
23 your recollection?
24 A. It probably would.
25 Q. Let me show you Defendant's Exhibit 1.
427
Barrett - cross
1 Do you already have a copy in front of you, sir?
2 A. The report, you mean?
3 Q. Yes.
4 A. Yes, it's here.
5 Q. Why don't you look at the one you have and see if you can
6 tell us if in that report you reported what the mean was on
7 the written job knowledge test.
8 MR. HOLZHAUER: Your Honor, while the doctor is
9 doing that, I had given Mr. Flaxman a new edition of this big
10 spread sheet, which I think is much clearer to read, and I
11 would give you a copy as well.
12 MR. FLAXMAN: This is Defendant's Exhibit --
13 MR. HOLZHAUER: 22.
14 MR. FLAXMAN: 22, and we are in agreement it's much
15 easier to read.
16 MR. HOLZHAUER: And it's been entered by
17 stipulation; is that correct?
18 MR. FLAXMAN: That's my understanding.
19 THE COURT: We're substituting it for Exhibit 22.
20 Is that right? This is Defendant's Exhibit 22, is it not?
21 Correct?
22 MR. HOLZHAUER: Correct.
23 THE COURT: Can you direct the witness's attention
24 to --
25 BY MR. FLAXMAN:
428
Barrett - cross
1 Q. Let me ask you to look at page 135, Dr. Barrett.
2 A. I'm in Appendix I 2, which says the mean is 109.89.
3 Q. And Appendix I 2 actually has that formula that you were
4 talking about; is that right?
5 A. Yes.
6 Q. So if you want to find the standard score for Diane
7 Thompson on the job knowledge test, and we've stipulated that
8 she received 121, you would take 121, subtract 109.89 from it,
9 and divide that by 14.95; is that right?
10 A. Yes.
11 Q. And we've stipulated that your chart shows that the
12 standard score for that is .74. Does that make sense to you,
13 Dr. Barrett?
14 A. Makes sense.
15 Q. And you could do the same thing with the oral briefing,
16 you have -- the mean is a different number. What's the mean,
17 Dr. Barrett?
18 A. I'm sorry.
19 Q. What is the mean? We use that word "mean." What does
20 that mean? What does the mean mean?
21 A. It's a number you divide by some numerator.
22 Q. Is the mean the average?
23 A. Yes, the average.
24 Q. So the mean on the oral briefing exercise was 12.29, that
25 means that the average score for the 765 candidates was 12.29;
429
Barrett - cross
1 is that right?
2 A. That's correct.
3 Q. And if the standard deviation -- what's the standard
4 deviation, Dr. Barrett?
5 A. That's a spread around the mean.
6 Q. Is it --
7 A. 66 percent of the score would fall around that -- plus or
8 minus that score.
9 Q. So if the -- could you tell us again what standard
10 deviation -- how do you compute the standard deviation,
11 Dr. Barrett?
12 A. Basically you're looking at the deviations around the
13 mean. So it's plus or minus -- that indicates that 66 percent
14 of the scores will be plus or minus one standard deviation.
15 Q. You have everybody's score on the written test; is that
16 right?
17 A. On the computer, yes.
18 Q. And on the computer how do you go about finding out what
19 the standard deviation is?
20 A. Put it in the program, and it comes out.
21 Q. Well, do you say computer, give me standard deviation?
22 A. Yes.
23 Q. Do you know of your own knowledge how to compute the
24 standard deviation?
25 A. I used to be able to do it.
430
Barrett - cross
1 Q. Is that something you learned when you went to school,
2 Dr. Barrett?
3 A. Yes, it is.
4 Q. That's something that you knew when you wrote your
5 dissertation, Dr. Barrett?
6 A. Yes.
7 Q. And it is something that you knew when you became a
8 diplomat of the SIOP?
9 A. Yes.
10 Q. But it's not something you know today when you're
11 testifying in court; is that right?
12 A. I know the concept.
13 Q. Well, could you explain to us in understandable words, if
14 you can, the concept of standard deviation?
15 A. As I said before, it's a spread around the mean, and it's
16 a number which indicates that you're going to have around that
17 mean 66 percent of the scores plus or minus.
18 Q. Well, let's say the standard deviation was 1.69 and the
19 mean was 12.29. What does that mean? What's the significance
20 of having a standard deviation of 1.69?
21 A. Well, you would add 1.69 to 12.29, and you should have 33
22 percent of the scores there. You subtract 1.69, you have 33
23 percent of the scores there, so you have between those two
24 bands 66 percent of the scores.
25 Q. Do you always have 66 percent of the scores between plus
431
Barrett - cross
1 or minus one standard deviation around the mean?
2 A. That's the concept, yes.
3 Q. My question, Dr. Barrett, is do you always have in real
4 world data 66 percent of the scores within plus or minus one
5 standard deviation around the mean?
6 A. Well, it's according to the distribution of scores. You
7 could have it very high, for example, and you would not have
8 that.
9 Q. So you're telling us that 66 percent is just some
10 abstract concept that may or may not apply to this data; is
11 that right?
12 A. No, it applies to this data.
13 Q. Where does it say in your report that it applies to this
14 data? Or does it say that in your report, that 66 percent
15 applies to this data?
16 A. I don't think it says anywhere in this report.
17 Q. And do you have any documentation that is in the report
18 that shows where you measured whether or not it applies to
19 this data?
20 A. Nothing at all.
21 Q. And it's just your opinion that it applies to this data;
22 is that right?
23 A. I'm not sure -- yes, it's my opinion.
24 Q. Now, let me ask you the same questions about the written
25 job knowledge test. I 2 says the standard deviation is 14.95;
432
Barrett - cross
1 is that right?
2 A. Yes.
3 Q. Do you have any data that shows that 66 percent of the
4 scores on the written job knowledge test are within plus or
5 minus 14.95 of the mean, which you say is 109.89?
6 A. No.
7 Q. And then the in-basket, it's on page I 3, you say the
8 mean is 45.02; is that right?
9 A. Yes.
10 Q. And you say -- you report the standard deviation as being
11 7.14; is that right?
12 A. Yes.
13 Q. And again, you don't have any analysis that shows that 66
14 percent of the scores on the in-basket are within plus or
15 minus 7.14 or 45.02, do you?
16 A. No.
17 Q. Now, to compute Diane Thompson's final Z score, if she
18 had scored 60 on the in-basket, would you take -- you take her
19 Z score on the written, Z score on the oral as reported in
20 Defendant's Exhibit 22, and then compute her Z score on the
21 in-basket, assuming that it was 60, and take those three
22 numbers and divide by three. Add those three numbers and
23 divide by three; is that right?
24 A. Yes.
25 Q. And you didn't try doing that, did you?
433
Barrett - cross
1 A. No.
2 Q. Okay. Do you think you'll have an opportunity to do that
3 over the lunch break and tell us what number you got?
4 A. I could try.
5 Q. Okay. Let me talk about Holly Robinson.
6 On Exhibit 22, Holly Robinson -- it shows that Holly
7 Robinson received 117 on the written and 56 on the in-basket.
8 And would it be possible -- it would be possible, would it
9 not, to take Holly Robinson's Z score on the written, or Z
10 score on the in-basket, and add to it a Z score for different
11 oral scores, let's say 13, 14 or 15, and see what Holly
12 Robinson's final Z score would have been if she had scored 13,
13 14 or 15 on the oral; is that right?
14 A. Yes.
15 Q. And do you think you'd have a chance to do that over the
16 lunch break?
17 A. Sounds like a lot of work over the lunch break.
18 Q. Well, what does it take to compute the Z score for one
19 exam component, Doctor? Do you have to take one number,
20 subtract another number, and then divide it?
21 THE COURT: What's the Z score again, Doctor?
22 THE WITNESS: A Z score is a standard technique for
23 making the scores equivalent when the raw scores are not
24 equivalent. It's spelled out, the formula is given in
25 Appendix I, which is --
434
Barrett - cross
1 BY MR. FLAXMAN:
2 Q. Now, when was it that you were retained to work on the
3 lieutenants test for the City of Chicago, Dr. Barrett?
4 A. We started, I believe, April of 1994, discussing it with
5 the City of Chicago Law Department.
6 Q. And did you submit a written proposal to the City of
7 Chicago for the contract to prepare the test?
8 A. Yes.
9 Q. Did you submit that written proposal before you had
10 prepared the test?
11 A. I don't know the exact date. The reason I don't, because
12 it is not unusual for us to begin work before a formal
13 contract is signed.
14 Q. Well, before submitting the formal proposal for the
15 lieutenants test, had you been hired by the city to prepare
16 the sergeants test?
17 A. Yes.
18 Q. And did you submit a formal proposal to do -- to prepare
19 the sergeants test?
20 A. Yes.
21 Q. And did you enter into the contract with the city to
22 prepare the sergeants test before you started work on the
23 sergeants test?
24 A. I don't recall. We may have.
25 Q. Okay. Did you propose to prepare the same kind of test
435
Barrett - cross
1 for lieutenant as you had proposed preparing for the
2 sergeants?
3 A. It was the same types of test, yes. Not the same type of
4 content.
5 Q. By the same type of test, do you mean that you proposed
6 preparing a test that included a job knowledge component, an
7 oral component, and an in-basket exercise?
8 A. Yes.
9 Q. As a matter of fact, that's the standard test that
10 Barrett and Associates has been administering for the last ten
11 years; is that right?
12 A. That standard format that we've used in many different
13 police and fire departments, yes.
14 Q. By Barrett and Associates; is that right?
15 A. Yes.
16 Q. And you've used that technique with the Akron police
17 sergeants; is that right?
18 A. As I recall we did, yes.
19 Q. You used it with Akron fire lieutenants; is that right?
20 A. As I recall we did, yes.
21 Q. And you've used it with Washington, D.C., Metropolitan
22 Transit police sergeants; is that right?
23 A. Yes.
24 Q. You also used it with the St. Louis Fire Department; is
25 that right?
436
Barrett - cross
1 A. Yes.
2 Q. And this three-component test is the Barrett standard
3 packet, is that fair to say?
4 A. No.
5 Q. Okay. Well, is it fair to say that in this
6 three-component test the in-basket exercise, whether it's used
7 in Akron or St. Louis, is intended to measure administrative
8 skills?
9 A. Yes.
10 Q. And there's an oral briefing exercise that was used in
11 Akron, and used in St. Louis and Chicago, and that was
12 intended to measure oral communication skills; is that right?
13 A. Yes.
14 Q. And the multiple choice job knowledge test, that's used
15 in -- that was used in Akron in police and fire; is that
16 right?
17 A. Yes.
18 Q. And it was used in the Washington Metropolitan Area
19 Transit Authority?
20 A. Yes.
21 Q. And it was used in St. Louis?
22 A. Yes.
23 Q. And it was used in Chicago?
24 A. Yes.
25 Q. And that's intended to measure job knowledge; is that
437
Barrett - cross
1 right?
2 A. Yes.
3 Q. And the view, your views that measuring those -- using
4 those three tests provides a good indication of who should be
5 promoted; is that right?
6 A. Yes.
7 Q. And that's true for the City of Chicago and its
8 lieutenants; is that right?
9 A. Yes.
10 Q. And it's true for Akron police sergeants; is that right?
11 A. Yes.
12 Q. Is the job of being a police sergeant in Akron, Ohio,
13 different than being a lieutenant in Chicago?
14 A. I'm not sure what you mean by "different." There are
15 certainly different contents in terms of the job knowledge, if
16 that's your question.
17 Q. Well, did you do a job analysis of the job of police
18 sergeant in Akron?
19 A. Yes.
20 Q. And from your job analysis, did you conclude that the job
21 of policing as a sergeant in Akron is the same as the job of
22 policing of a lieutenant in Chicago?
23 A. No.
24 Q. They're different jobs, aren't they?
25 A. Yes.
438
Barrett - cross
1 Q. And being a police -- being a fire lieutenant in Akron is
2 different than being a police lieutenant in Chicago; is that
3 right?
4 A. Yes.
5 Q. And being a fire fighter in St. Louis is different than
6 being a police lieutenant in Chicago; is that right?
7 A. Yes.
8 Q. And all of those jobs you assess with this three
9 component test, the job knowledge test, the in-basket exercise
10 and oral briefing exercise; is that right?
11 A. Those forms of tests were used in all those locations,
12 yes.
13 Q. Now, when you prepared the job knowledge test for the
14 police lieutenants in the City of Chicago, you had this source
15 list, this list of documents; is that right?
16 A. Yes.
17 Q. And were you the person who picked out those general
18 orders, and special orders, and statutes, and rules?
19 A. You mean personally picked them out?
20 Q. Yeah.
21 A. No.
22 Q. Who did?
23 A. Well, we had a team approach, and we had, as I recall,
24 around May, interviews with 11 different police lieutenants
25 who helped us generate potential relevant source documents.
439
Barrett - cross
1 We had our job analysis team, who also would often
2 have suggestions about relevant material. We had, of course,
3 the expert panel from the police department, which I recall
4 included Commander Wedgbury, Captain Montague, Lieutenant
5 Schraeder, I believe -- I think it's Commander Schraeder. We
6 also had input from Lieutenant Klein.
7 So we had multiple sources of what was relevant.
8 Q. Well, these 11 lieutenants that were interviewed, did you
9 interview any of those 11 lieutenants, Dr. Barrett?
10 A. I don't think I did.
11 Q. Do you have personal knowledge of what was said to any of
12 those 11 lieutenants?
13 A. Personal knowledge?
14 Q. Yes.
15 A. In the sense of -- no, I don't have personal knowledge.
16 Q. Do you know which, if any, of those 11 lieutenants were
17 asked to identify the important general orders of the Chicago
18 Police Department?
19 A. No, I don't.
20 Q. Do you know which if any of those 11 lieutenants were
21 asked to identify the important special orders of the police
22 department City of Chicago?
23 A. No, I don't.
24 Q. Now, you talked about the expert panel. Did you ever
25 meet with the expert panel?
440
Barrett - cross
1 A. No. Dr. Cellar met with the expert panel.
2 Q. Who is Dr. Cellar?
3 A. He's a vice president in Barrett and Associates. He has
4 his office here in Chicago.
5 Q. Is he one of your graduate students that went on to other
6 things?
7 A. Yes.
8 Q. Aside from his employment with Barrett and Associates,
9 does he have a day job?
10 A. Yes.
11 Q. What's that?
12 A. He's in DePaul University.
13 Q. Is he in the City of Chicago at this time?
14 A. I'm sorry?
15 Q. Is he in the City of Chicago at this time? Is he out of
16 the country, is he here?
17 A. I don't know if he's actually in the city. He might be
18 in his home, which is not in Chicago.
19 Q. Excuse me?
20 A. He might be at his home. I don't know where he is right
21 now.
22 Q. Is his home in the Chicago area?
23 A. Yes.
24 Q. Okay.
25 THE COURT: What is he doing at DePaul?
441
Barrett - cross
1 THE WITNESS: He's an associate professor.
2 BY MR. FLAXMAN:
3 Q. And he met with Wedgbury, Minogue, and who else did he
4 meet with?
5 A. I'm not sure. For some reason I think it was Commander
6 Schaffer or Schraeder. I've forgotten now.
7 Q. Anybody else?
8 A. Are you talking about now the review panel?
9 Q. Right.
10 A. Beyond the people I've already mentioned, is that what
11 you're asking?
12 Q. Well, I don't know. I don't think you've already
13 mentioned anybody else. I want to know who Dr. Cellar met
14 with.
15 A. On the review panel?
16 Q. Yes.
17 A. I thought I said Commander Wedgbury, Captain Montague, I
18 believe Shaffer or Schraeder --
19 Q. Anybody else?
20 A. Well, Dr. Cellar did a lot of interviews, as I recall, so
21 I don't know specifically who he met with.
22 Q. Well, do you have personal knowledge of whether anybody
23 that Dr. Cellar met with told him which orders were important?
24 A. I'm sorry.
25 Q. Do you have personal knowledge of whether anybody that
442
Barrett - cross
1 Dr. Cellar met with told him which general orders were
2 important for a lieutenant to know?
3 A. All I have is his report to me and the recommendations.
4 Q. Is his report to you and recommendation contained in
5 Defendant's Exhibit 1 that's been admitted into evidence?
6 A. I think the final recommendation would be whatever was
7 posted by the police department and sent to the candidates.
8 That would be the official list.
9 Q. Is that the form that you received your report from
10 Dr. Cellar?
11 A. I don't recall the exact form I received it in.
12 Q. Did you ever receive a report from Dr. Cellar?
13 A. Yes, I don't recall when. I recall receiving, yes,
14 here's is a recommendation in terms of the panel. And I
15 recall Lieutenant Klein made an addition to it.
16 Q. Well, let's go back to this report from Dr. Cellar. That
17 report is not part of Defendant's Exhibit 1; is that right?
18 A. It is in the sense I don't -- I assume, I don't know
19 what's in your exhibit, but as I recall, there is an official
20 posting. I don't know what exhibit number that is.
21 Q. Well, Doctor, we're trying to find out where that
22 official posting came from. Who prepared that official
23 posting?
24 A. I'm not sure what you mean.
25 Q. Who typed it? Where was it typed up?
443
Barrett - cross
1 A. I don't know where it was typed up.
2 Q. Do you know whether it was typed in Ohio or Chicago?
3 A. I don't know.
4 Q. Do you know whether it was prepared by Mr. Joyce or
5 somebody in his office?
6 A. Mr. Joyce testified yesterday he did not prepare it.
7 Q. That was not my question, Dr. Barrett.
8 Do you know whether it was prepared by Mr. Joyce or
9 somebody in his office?
10 MR. HOLZHAUER: Objection. He already testified he
11 doesn't know. How many times can he ask the same question a
12 different way?
13 THE COURT: I think you made your point.
14 BY MR. FLAXMAN:
15 Q. Did you talk to Lieutenant Klein about the important
16 general orders?
17 A. Yes, we discussed the important issues.
18 Q. When did you have this discussion with Lieutenant Klein
19 about which general orders were important?
20 A. I talked to Lieutenant Klein many times. I don't recall
21 any one specific date.
22 Q. Do you recall any particular conversation with Lieutenant
23 Klein about which general orders are important?
24 A. No.
25 Q. You told us that Lieutenant Klein made an addition to the
444
Barrett - cross
1 list. Do you recall that testimony?
2 A. Yes, I do.
3 Q. What did he add to the list? What addition did he
4 propose to the list?
5 A. I believe, now, this is -- I believe he suggested the
6 community policing bulletin, as I recall. Again, it's been
7 three years, so I could be wrong about that.
8 Q. Well, is Mr. Klein's recommendation about something to
9 add documented anywhere in Defendant's Exhibit 1?
10 A. Are you referring now -- is this my report you're
11 referring to?
12 Q. That's correct.
13 A. I doubt it.
14 Q. Well, when you agreed to prepare this test for the City
15 of Chicago, did you agree to create a paper trail of the work
16 that you were doing?
17 A. Yes.
18 Q. Could you tell us where the paper trail is for
19 Mr. Klein's recommendation about including community policing?
20 A. There is no -- as far as I know, there is no paper trail
21 on that.
22 Q. Did Mr. Klein make any other recommendations that things
23 should be added?
24 A. Again, I don't recall specifically what he -- beyond what
25 I just said, he might have added.
445
Barrett - cross
1 Q. In the course of your work in preparing this examination
2 and the sergeants examination, did you become aware of
3 something called the rules and regulations of the Chicago
4 Police Department?
5 A. Yes.
6 Q. And did you become aware that there are rules and
7 regulations that are promulgated by the Chicago police board?
8 A. I believe that's correct.
9 Q. Did you become aware that some of those rules and
10 regulations are vitally important for each police officer from
11 the rank of patrolman to the rank of superintendent?
12 A. I don't recall that.
13 Q. Well, did you conclude that those rules were not
14 important for Chicago police lieutenants to know?
15 A. Again, this is three years ago, but I think that there
16 was some belief that they were outdated. This is just my
17 recollection.
18 Q. Whose belief was that, Dr. Barrett?
19 A. Well, again, we relied upon the subject matter experts.
20 In other words, they are the ones who recommend to us these
21 sources.
22 Q. Did Mr. Klein tell you that the rules and regulations of
23 the Chicago Police Department were outdated?
24 A. I don't recall.
25 Q. Did Captain Minogue tell that you the rules and
446
Barrett - cross
1 regulations of the Chicago Police Department were outdated?
2 A. I don't recall.
3 Q. Did Commander Wedgbury tell you that the rules and
4 regulations of the Chicago Police Department were outdated?
5 A. I don't recall.
6 Q. Did Commander Schraeder tell you that the rules and
7 regulations of the Chicago Police Department were outdated?
8 A. I don't recall.
9 Q. And is it your opinion, based on whenever it's based on,
10 that those rules and regulations are outdated?
11 A. That was a decision that we reached, I'm sure if that's
12 what decision we did reach, at that point in time, yes.
13 Q. And the rules and regulations were not tested in the
14 lieutenants test; is that right?
15 A. I think that's correct.
16 Q. There wasn't any question about that on the written job
17 knowledge?
18 A. As I recall, that's correct.
19 Q. Now, in your report you include the job announcement from
20 the 1987 lieutenants test. Do you recall that inclusion?
21 A. Yes.
22 Q. That's at page Bates stamped 182, Defendant's Exhibit 1.
23 Why did you include that report? Excuse me. Why did you
24 include that job announcement in your report, Dr. Barrett?
25 A. I'm sorry. What page are you on?
447
Barrett - cross
1 Q. 182.
2 A. This was included because the EEOC regulations asked you
3 try to look at the previous exam process, and I believe there
4 was very little information obtained about that previous
5 process.
6 So the reason for including that is that's all we
7 could actually obtain at that point in time, because I don't
8 recall us being able to obtain a technical report, I believe
9 we asked for one, for that previous examination.
10 Q. Defendant's Exhibit 1 is dated on page 1 of that report,
11 October 10, 1995. Do you see that?
12 A. I'm sorry. What page are you on now?
13 Q. Page 1. The very first page. Page -- the cover of
14 Defendant's Exhibit 1 is what I'm referring to right now. See
15 where it says report on the development that administration --
16 do you see page 1?
17 A. Oh, you mean of the report itself?
18 Q. Yes.
19 A. I thought you meant of the -- the cover page you're
20 talking about?
21 Q. That's right. There's a date on the cover page; is that
22 right?
23 A. Yes.
24 Q. What's the date?
25 A. October 10, 1995.
448
Barrett - cross
1 Q. Was that after the exam had been administered?
2 A. The exam was administered on November 19th, 1994, so it
3 would be after the exam was administered.
4 Q. So it took 11 months to complete this report after the
5 exam was administered?
6 A. That's what the date says.
7 Q. Now, this lawsuit, do you know whether or not any
8 lawsuits had been filed about your Chicago promotional test
9 before October 10, 1995?
10 A. No, I don't know the dates when you actually filed the
11 suit.
12 Q. Well, did you ever become aware that lawsuits were filed
13 in Chicago about the promotional test?
14 A. Yes, I was -- became aware of that.
15 Q. Were any of those lawsuits filed before October 10,
16 1995?
17 A. I don't recall the exact date you filed the lawsuits.
18 Q. Did you learn about those lawsuits on or about the date
19 they were filed?
20 A. I assume that the City of Chicago law department called
21 me.
22 Q. Now, you told us on direct examination that you have been
23 paid -- maybe I misremembered or misheard it -- $130,000 for
24 your work in this litigation.
25 A. I think, again, I believe there is an invoice for that
449
Barrett - cross
1 amount outstanding, I think is what I said.
2 Q. Well, is that the total amount for which you billed the
3 City of Chicago for your post -- for your services in
4 connection with litigation concerning the test?
5 A. I don't believe so. I'm just -- other invoices which
6 have been paid.
7 Q. What's the total amount that you've billed the City of
8 Chicago for work that you've done after the tests were
9 administered, litigation work?
10 A. I don't recall the exact figure.
11 Q. Well, is it more than 130,000?
12 A. Yes.
13 Q. Is it more than 200,000?
14 A. I actually don't recall the exact figure.
15 Q. Is there any way that you could find out?
16 A. I thought we sent you the invoices.
17 Q. Well, my recollection is that those invoices show that
18 you are -- I'm not sure, but that there was like $80,000 over
19 the 130 that you've been paid. I don't want to testify, so
20 we'll move on.
21 That time has been billed at your rate of $250 an
22 hour?
23 A. Yes.
24 Q. So you worked nearly a thousand hours in connection with
25 the litigation that's been filed about your test?
450
Barrett - cross
1 A. No. That's not accurate.
2 Q. Well, how many hours have you worked?
3 A. I don't know.
4 Q. Well, if we take the total amount that you billed and
5 been paid and divide it by 250, would we determine that amount
6 of time?
7 A. No, it would not be accurate.
8 Q. Why is that?
9 A. Because we also billed for other people in the firm, at
10 which we always bill at the lowest possible rate.
11 Q. How much was Barrett and Associates paid to develop the
12 test?
13 A. I believe it was 350,000.
14 Q. It hasn't cost quite that much yet to defend the test,
15 has it, Dr. Barrett?
16 A. I don't know what cost it took to defend the test.
17 Q. You told us that it would be very expensive to administer
18 an assessment center where there was -- which was graded in a
19 different way than the in-basket exercise was in the test; is
20 that right?
21 A. Yes.
22 Q. Would it cost $150,000?
23 A. I'm not clear. You want me to estimate -- I made an
24 estimate, as I recall, in my report for Mr. York replying to
25 -- Dr. York, where I made an estimate of what it would cost.
451
Barrett - cross
1 Q. Let's go back to the 1987 job posting. Let's look at
2 page 183.
3 Did you look at that at any time before you
4 completed your work in the job knowledge test?
5 THE COURT: Page 183 of what?
6 MR. FLAXMAN: Of Defendant's Exhibit 1. I'm sorry.
7 MR. HOLZHAUER: What page?
8 MR. FLAXMAN: 183.
9 BY MR. FLAXMAN:
10 Q. Do you have that?
11 A. Yes.
12 Q. Did you look at that before the job -- before the reading
13 list was posted of the 1994 test?
14 A. I don't recall.
15 Q. Did you look at it after the reading list was posted of
16 the 1994 test?
17 A. I looked at it in some point in time. I don't recall the
18 exact time I saw it.
19 Q. Did you ever notice that some general orders were on the
20 1987 list that weren't on the 1994 list?
21 A. I never made a point-to-point comparison.
22 Q. Now, if you keep going in that exhibit to page 234, we
23 get to the reading list for the 1994 test; is that right?
24 A. Yes, it's labeled that way, yes.
25 Q. And then somewhere there is a linkage chart for the
452
Barrett - cross
1 written job knowledge test in this exhibit; is that right
2 also?
3 A. Yes.
4 Q. And the linkage chart tells you each question and what
5 general order it's based on.
6 Let me ask you to look at page 283. Is that the
7 linkage chart?
8 A. Yes.
9 Q. If you look at the linkage chart, question 54, that says
10 it's based on special order 7816; is that right?
11 A. Yes.
12 Q. And question 70 is also based on special order 7816; is
13 that right?
14 A. I'm sorry. I didn't hear your number.
15 Q. Question 70.
16 A. 70. Yes.
17 Q. Now, was there anything that came up in your job analysis
18 that demonstrated that there should be two questions on
19 general order 7816 on the job knowledge test?
20 A. I assume something on the job analysis may have come up
21 that way, or our review of the material may have found that it
22 was appropriate.
23 Q. Is there anything in your report that explains why there
24 are two questions on that general order?
25 A. No.
453
Barrett - cross
1 Q. As you sit here now, could you tell us why there are two
2 questions on that general order?
3 A. No.
4 Q. Now, the reading list, there is also a general order 86-2
5 called city license premises. Do you see that? It's on page
6 234 of Exhibit 1.
7 A. 234, and the question was --
8 Q. Do you see where the reading list says one of the general
9 orders is 86-2, city license premises?
10 A. 86-2. Yes, I see that.
11 Q. Are there any questions on the written test that were
12 linked to general order 86-2?
13 A. I don't know.
14 Q. Could you take a look and see if you find any?
15 A. Looks like 32.
16 Q. Okay. Anything else?
17 THE COURT: If it's there, can you point it out,
18 Mr. Flaxman?
19 MR. FLAXMAN: I'm proving a negative, Judge.
20 THE COURT: Well, it doesn't seem that it's a
21 negative.
22 THE WITNESS: I only found in my quick review one
23 item.
24 BY MR. FLAXMAN:
25 Q. How was this linkage chart prepared, Dr. Barrett?
454
Barrett - cross
1 A. One of my associates probably took from our review sheets
2 the item number and the source and linked that to the work
3 behavior.
4 Q. So are you telling us that if we look at general order
5 86-2, we'll find the answer to question 32; is that right?
6 A. Unless there's a mistake in the documentation.
7 Q. Are there any mistakes in your documentation,
8 Dr. Barrett?
9 A. Yes. Based upon the testimony of one of your sergeants,
10 I believe, we did find a mistake.
11 Q. Well, didn't you check all of your work before you gave
12 it into the City of Chicago?
13 A. Yes, we tried to check all our work.
14 Q. How many mistakes did you make, Dr. Barrett?
15 A. Well, it can only be one if I don't --
16 Q. How many lieutenants have been promoted because of a
17 mistake, Dr. Barrett?
18 A. I don't think any have been.
19 Q. You don't think 108 have been because of a mistake?
20 A. No.
21 Q. So this linkage chart is a mistake where it says question
22 32 is based on general orders 86- --
23 MR. HOLZHAUER: Objection. He didn't testify to
24 that. The jury is not impressed, Mr. Flaxman.
25 THE COURT: Sustained. Let's go to the facts.
455
Barrett - cross
1 You'll have a chance to argue. Let's go to the facts.
2 Is it or isn't it?
3 BY MR. FLAXMAN:
4 Q. Let me ask you to look at question -- the linkage chart
5 for item No. 5.
6 What general order does the linkage chart say that
7 item No. 5 is based on?
8 A. 93.3, addendum 5.
9 Q. And general order 93.3 is in fact on the reading list; is
10 that right?
11 A. Yes.
12 Q. So are you telling us then if we go look at general order
13 93-3 we could find the answer to question 5; is that right?
14 A. Unless there's a mistake. Yes.
15 Q. Well, there were only -- how many questions were there
16 based on general orders on this test, Dr. Barrett? 53?
17 A. Looks like there's 53.
18 Q. 53?
19 A. 53, yes, that's correct.
20 Q. And linking the general orders to the work behaviors was
21 an important part of the content validation process?
22 A. It's just one more piece of evidence to show -- you
23 really don't have to do it, but yes, it's a piece of evidence.
24 Q. How many mistakes were made in preparing that one piece
25 of evidence, Dr. Barrett?
456
Barrett - cross
1 A. I don't know.
2 Q. Well, was general order 84.7 on the reading list?
3 A. No.
4 Q. If I told --
5 A. Not on the list -- what we have here, this is not the
6 official list, by the way.
7 Q. The one we stipulated to is not the official list?
8 A. This was not the list which was sent out to the people.
9 It doesn't look like the list.
10 Q. Were different orders on the list that was sent out to
11 the people?
12 A. I don't know.
13 Q. Do you know who has the list that was sent out to the
14 people?
15 A. No.
16 THE COURT: What exhibit are you referring to, so
17 the record is clear on this?
18 MR. FLAXMAN: Exhibit 1.
19 THE COURT: Where is the list on Exhibit 1? The
20 witness is looking at a document. What document is he looking
21 at?
22 BY MR. FLAXMAN:
23 Q. Are you looking at page 234 of Exhibit 1, which is
24 entitled Chicago Police Lieutenant Promotional Exam Reading
25 List?
457
Barrett - cross
1 A. Yes. This is an appendix in our report, yes.
2 Q. And you're telling us that this is not the reading list
3 that was sent out to police lieutenants?
4 A. Well, I don't believe this -- it looks like something was
5 typed in our office, is my first thought for an appendix. It
6 does not look like what would be sent out. Usually in my
7 experience there is more on an official reading list sent out
8 by a city, but --
9 Q. Did you ever see what the city sent out to lieutenants,
10 to sergeants who wanted to take the lieutenants test?
11 A. Yes, I did.
12 Q. And was it different than this?
13 A. I didn't do a point-by-point correspondence check.
14 Q. Did you include a copy of what the city sent out in your
15 report, Defendant's Exhibit 1?
16 A. No, what we have is what I have here in this appendix F.
17 Q. Were there any differences between what the city sent out
18 and the reading list that's in your report?
19 A. As far as I know there's not, but there might be.
20 Q. Well, the reading list that's in your report does not
21 show general order 84-7 as being one of the things that
22 sergeants were told that the test would be based on; is that
23 right?
24 A. That's -- there is no 84.7 in our list, in the appendix,
25 that's correct.
458
Barrett - cross
1 Q. And it was your intent in preparing the written test that
2 each question on that written test would be based on materials
3 identified in the reading list; is that right?
4 A. Yes, the official reading list that went out, yes, that's
5 correct.
6 Q. So would it be another mistake if there was a test
7 question that was based on a general order that was not on the
8 reading list?
9 A. If it's not the official reading list, I would say yes,
10 it's a mistake.
11 Q. I'm not sure if I understood your answer, Dr. Barrett.
12 A. I'm saying if 84.7 is not on the list which was given to
13 the candidates, then it would be a mistake for us to have an
14 item written on that source.
15 THE COURT: Why don't we break for lunch. Be back
16 at 1:20.
17 Dr. Barrett, you are under cross examination, so
18 please don't talk to anybody else, anybody at all about the
19 test. Just have a nice, pleasant lunch, and I'll see you back
20 here at 1:20.
21 MR. FLAXMAN: What time did you say?
22 THE COURT: 1:20.
23 MR. HOLZHAUER: May I ask for point of
24 clarification, is Dr. Barrett supposed to be doing
25 calculations over his lunch hour?
459
Barrett - cross
1 THE COURT: Do you have any position on that?
2 MR. HOLZHAUER: I don't think it's appropriate. I
3 think Dr. Barrett should be allowed to have his lunch, and if
4 he wanted to proffer him as an expert, he could have done
5 that.
6 THE COURT: I agree. Enjoy your lunch.
7 (Recess at 12:15 p.m.)
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
460
1 IN THE UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
2 EASTERN DIVISION
3 ERNEST T. BROWN, et al., )
)
4 Plaintiffs, )
) No. 95 C 1890
5 v. ) Chicago, Illinois
) November 20, 1997
6 CITY OF CHICAGO, ) 1:30 p.m.
)
7 Defendant. )
8 VOLUME 3
9 TRANSCRIPT OF PROCEEDINGS - TRIAL
10 BEFORE THE HONORABLE ROBERT W. GETTLEMAN
11 APPEARANCES:
12 For the Plaintiffs: KENNETH N. FLAXMAN, P.C.
122 South Michigan Avenue
13 Suite 1850
Chicago, Illinois 60603-6107
14 BY: MR. KENNETH N. FLAXMAN
15 and
16 FUTTERMAN & HOWARD, CHTD.
122 South Michigan Avenue
17 Suite 1850
Chicago, Illinois 60603
18 BY: MR. CRAIG FUTTERMAN
19
For the Defendant: MAYER, BROWN & PLATT
20 190 South LaSalle Street
Chicago, Illinois 60603
21 BY: MR. JAMES HOLZHAUER
MR. JEFFREY S. PIELL
22 MS. ANGELA K. DORN
MR. ANDREW NICELY
23
Official Court Reporter: JENNIFER S. COSTALES, CSR, RMR
24 219 South Dearborn Street
Room 1744-A
25 Chicago, Illinois 60604
(312) 427-5351
461
Barrett - cross
1 (Proceedings heard in open court.)
2 THE COURT: Dr. Barrett. Please take the stand
3 again. You're still under oath.
4 Mr. Flaxman.
5 GERALD BARRETT, DEFENDANT'S WITNESS, PREVIOUSLY SWORN
6 CROSS-EXAMINATION (Resumed)
7 BY MR. FLAXMAN:
8 Q. Dr. Barrett, do you still have any of those exhibits up
9 there before you?
10 A. Yes, I do.
11 Q. Do you have that linkage chart that's at page 283 of
12 Defendant's Exhibit 1 before you?
13 A. Yes.
14 Q. And this was -- the left -- this chart has a table, is that
15 right?
16 A. Yes.
17 Q. And the left column of the table refers to a question
18 number on the written job knowledge test, is that right?
19 A. Yes.
20 Q. And the second column is entitled "Source," is that right?
21 A. Yes.
22 Q. And what information is contained in the column titled
23 "Source"?
24 A. The document from the reference list.
25 Q. And --
462
Barrett - cross
1 A. Those documents.
2 Q. And by documents from the reference list, do you mean the
3 general orders, special orders, statute, or municipal code
4 section, contract section, or CAPS booklet section from which
5 the question was derived?
6 A. Yes.
7 Q. Now, next to item 1 it says "GO 80-18." Does that mean
8 general order 80-18?
9 A. Yes.
10 Q. And am I correct that general order 80-18 was supposed to
11 have been on the reading list?
12 A. Yes.
13 Q. And if you go down to question 6, it says "General order
14 84-7, addendum 1A 1B5." Do you see that?
15 A. Yes.
16 Q. Does that relate to general order 84-7, addendum 1A 1B5?
17 A. Yes.
18 Q. And should that have been on the reading list?
19 A. Yes.
20 Q. And was it on the reading list?
21 A. I don't know.
22 Q. And could you -- to find out, would you look at the reading
23 list. Do you have the reading list in front of you?
24 A. No.
25 Q. No? Is it part of Defendant's Exhibit 1? Is it at page
463
Barrett - cross
1 234 of Defendant's Exhibit 1?
2 A. I'm sorry.
3 Q. Is it at page 234 of Defendant's Exhibit 1?
4 A. Yes.
5 Q. Do you see general order 84-7 on the reading list?
6 A. No.
7 Q. And were police sergeants who took the 1994 job knowledge
8 test given advance information about what rules and regulations
9 would be on the test?
10 A. Yes.
11 Q. And were all of the rules and regulations that were on the
12 test on the reading list?
13 A. The one they received I assume it was, yes.
14 Q. And the one they received, that's different than the one
15 you included in your report, is that your testimony,
16 Dr. Barrett?
17 A. No, it's not. I'm not sure. I'm saying this does not look
18 like it's the official list, I believe, put out by the City is
19 all I'm saying. This is our list.
20 Q. Could you tell us why you included something other than the
21 official list in your report, Dr. Barrett?
22 A. I would say it would be a mistake.
23 Q. Why did you prepare that validation report?
24 A. To document our process.
25 Q. Did you intend that the validation report be accurate?
464
Barrett - cross
1 A. Yes.
2 Q. Did you intend it to be complete?
3 A. Yes.
4 Q. And as a matter of fact, it's inaccurate, isn't that right,
5 Dr. Barrett?
6 A. It appears there is one missing goal statement, yes.
7 Q. Excuse me?
8 A. There appears to be one missing document here, yes.
9 Q. And what's the missing document?
10 A. Well, I think you put it 83 -- 84.7 is not on this list.
11 Q. Well, and if 84.7 wasn't on the list that was given to
12 police sergeants who studied for the test, then the test wasn't
13 fair, was it, Dr. Barrett?
14 A. This is not a test for police sergeants.
15 Q. Well, who was allowed to take this promotional test,
16 Dr. Barrett?
17 A. It's a test for lieutenants.
18 Q. Who was allowed to take the test, Dr. Barrett?
19 A. Sergeants.
20 Q. Police officer -- so it was a test for sergeants, isn't
21 that right?
22 A. Well, that's not the terminology you used.
23 Q. The test was taken by sergeants, is that right?
24 A. Yes, it was.
25 Q. And it was a test for sergeants, is that right?
465
Barrett - cross
1 A. It was for promotion to police lieutenant.
2 Q. And if you were testing these sergeants who were taking the
3 test for promotion to police lieutenants on items that were not
4 on the reading list, then the test would be unfair, isn't that
5 right?
6 A. There would be -- if, in fact, they did not receive notice
7 that 84.7 was a source, then that item should not be on them.
8 Q. Now, you told us that police sergeants who took the test
9 were allowed to make challenges after the test, is that right?
10 A. Yes.
11 Q. Were they allowed to taken the written test questions with
12 them to make challenges?
13 A. No.
14 Q. Were they given the linkage list of the source for where
15 the questions purportedly came from before they could make a
16 challenge?
17 A. No.
18 Q. In order to make a challenge, what did the police sergeant
19 have to do?
20 A. He would have to call in to Arthur Andersen.
21 Q. And would he have to tell Arthur Andersen the number of the
22 question he was challenging?
23 A. Some, some of the challenges were more general in nature.
24 Q. Well, could you answer my question, Dr. Barrett.
25 If a police officer wanted to challenge question 8 as
466
Barrett - cross
1 being based on something that wasn't on the reading material,
2 would he have to say, "I'm challenging question 8"?
3 A. Yes.
4 Q. Would he have to say what the question was?
5 A. I think the number would be enough.
6 Q. And in your professional career as a test validater, your
7 opinion is that this is a reasonable way to get fair challenges
8 to questions, by not giving people the questions and asking
9 them to challenge it later, telling them which question they
10 disagree with, is that right?
11 A. It's one, one process you can use.
12 Q. And it's a fair process, is that right, Dr. Barrett?
13 That's your opinion?
14 A. Everybody has an opportunity to call in and challenge any
15 question.
16 Q. And if somebody wants to challenge question number 112,
17 they would have to know that question, is that right?
18 A. All they have to do as far as I know is say, "I challenge
19 question 112."
20 Q. And you have to know what the question was, is that right?
21 A. Yes.
22 Q. You weren't allowed to make notes about which questions you
23 wanted to challenge to take with you, were you, Dr. Barrett?
24 A. No.
25 Q. As a matter of fact, weren't the people who took the test
467
Barrett - cross
1 asked not to tell anybody about the questions?
2 A. I don't recall that.
3 Q. Didn't you try to keep the questions a secret, Dr. Barrett?
4 A. I know that we say that for any of our tests. I'm not sure
5 if this is true in this test or not.
6 Q. All right. How many challenges were there to these
7 questions?
8 A. There were 12 overall challenges, as I recall, one was the
9 in-basket and three to the written test.
10 Q. Was there anybody who challenged the challenge process?
11 A. I don't recall anyone, no.
12 Q. Well, how were you involved in the challenge process,
13 Dr. Barrett?
14 A. We didn't handle the challenge process itself, the
15 procedure. That was Arthur Andersen. Our job was to review
16 the challenges.
17 Q. Well, so what do you mean, "review the challenges"?
18 A. Whatever Arthur Andersen gave to us as a challenge, we'd
19 review.
20 MR. FLAXMAN: I need a minute.
21 BY MR. FLAXMAN:
22 Q. When challenges were received by Arthur Andersen, were they
23 sent to Barrett & Associates in Akron, Ohio?
24 A. No.
25 Q. Excuse me?
468
Barrett - cross
1 A. No.
2 Q. Did you ever receive any challenges personally from Arthur
3 Andersen?
4 A. As I recall, one point, I reviewed all the challenges that
5 came in, I believe after December 15th when the committee met.
6 Q. Well, do you remember receiving a letter dated December
7 14th, 1994, from someone named Marion B-r-z-y-k-c-y?
8 A. Not specifically. I may have. Unless she worked for
9 Arthur Andersen.
10 Q. Okay. Do you know -- did you have any dealings with her in
11 the course of your work for the City of Chicago?
12 A. Yes.
13 Q. Do you remember receiving a letter from her in December of
14 1994 which contained a copy of a challenge to the challenge
15 system for the lieutenants test?
16 A. Not specifically, no, I don't.
17 Q. Is there another Dr. Gerald Barrett at Barrett &
18 Associates?
19 A. No, there's no one else.
20 Q. Let me show you what I will mark as Plaintiffs' Exhibit
21 151.
22 MR. FLAXMAN: And I'll hand a copy to the Court.
23 BY MR. FLAXMAN:
24 Q. Have you ever seen this exhibit before?
25 MR. HOLZHAUER: Mr. Flaxman, I believe he already has
469
Barrett - cross
1 a 151. The report on integrity, 157.
2 MR. FLAXMAN: May I renumber the exhibit, with the
3 permission of the Court, as 152?
4 THE COURT: Okay.
5 MR. FLAXMAN: Thank you.
6 BY MR. FLAXMAN:
7 Q. Have you ever seen Plaintiffs' Exhibit 152 before?
8 (Pause.)
9 BY MR. FLAXMAN:
10 Q. Have you ever seen it before, Dr. Barrett?
11 A. I don't specifically recall this communication, no.
12 Q. Okay. Do you recall how the challenge procedure was
13 announced to Chicago police sergeants who took this test?
14 A. I don't know any of the details about that process. It was
15 handled by the City and Arthur Andersen.
16 Q. And who at the City handled the challenge process or
17 handled -- excuse me -- who at the City handled notifying
18 police sergeants about the challenge process?
19 A. I don't know.
20 Q. Who at Arthur Andersen handled notifying sergeants who took
21 the test about the challenge process?
22 A. All I know is Arthur Andersen was in charge of the process.
23 I don't know how it actually worked.
24 Q. So you don't know whether or not sergeants were informed
25 about the challenge process, isn't that right, Dr. Barrett?
470
Barrett - cross
1 A. I have no personal knowledge of the challenge process.
2 Q. And if sergeants weren't notified about the challenge
3 process, you wouldn't be surprised if nobody made any
4 challenges, would you?
5 A. No. If no one was notified, I guess that would be true.
6 Q. Now, let's go back to that linkage chart at page 283 of
7 Exhibit 1. Let's look at question 8. No. Let's look at
8 question 13.
9 The second column says, "GO 87-7." Am I reading the
10 linkage chart correctly that if I look at the written test
11 question number 13, I would be able to find the answers to that
12 question in general order 87-7, is that right?
13 A. Unless there's a typing mistake. That's correct.
14 Q. Well, before you submitted this -- you were paid to do this
15 report, weren't you, Dr. Barrett?
16 A. Yes.
17 Q. And you had this report proofread, didn't you?
18 A. Yes.
19 Q. And efforts were made to be sure that there weren't
20 typographical mistakes, isn't that right?
21 A. Yes.
22 Q. And as far as you know, there weren't any typographical
23 mistakes, is that right?
24 A. Well, I know of one for sure.
25 Q. Which one is that, Dr. Barrett?
471
Barrett - cross
1 A. It appears in the appendix, which was placed O should have
2 been P.
3 Q. Well, aside from that typographical mistake, were there any
4 others?
5 A. I'm sure there were, but I don't know of them for a fact.
6 Q. Well, now, how is this linkage created?
7 A. Basically, they go back to -- in the appendix, I have the
8 process --
9 Q. Well, do you know?
10 A. I'm trying to find out what the next number it is.
11 Q. Can you recall of your own personal knowledge how that
12 linkage chart was prepared, Dr. Barrett?
13 A. Geographically we have the form we use. And it has on it
14 the item number.
15 Now, what sometimes occurs is this, that when we do a
16 final version, sometimes the item number might not be there.
17 But that should contain the item number and the source material
18 on that form. So that's what it's taken from.
19 Q. Is there a person who is in charge of doing this, getting
20 the item number and the source material?
21 A. Well, as I said before, our procedure is that there is a
22 form for every item. And when we go through the review
23 process, it's given a number at first. But once we pilot test,
24 the SMEs review it, then it's compiled into a final -- a test
25 format, those numbers change.
472
Barrett - cross
1 So that's what -- so they have to go back to that --
2 they should go back to that form which contains the final item
3 number, and it should contain the major work behaviors and the
4 source. So that's how it works.
5 Q. Are the people who are responsible for maintaining these
6 forms trained in writing test questions?
7 A. Well, some are more clerical, but, yes. People who
8 maintain -- make up the forms are trained in item writing.
9 Q. Is it part of the job of the people who maintain these
10 forms, these linkage forms to be careful and correct?
11 A. Yes.
12 Q. And is it important that the linkage form, when compiled
13 into a linkage chart, be accurate?
14 A. They should be accurate.
15 Q. And it's supposed to be accurate, because that's part of
16 the basis for a claim of content validation, isn't that,
17 Doctor -- isn't that true, Dr. Barrett?
18 A. It's just one piece of evidence which shows the test is
19 content valid.
20 Q. It's like one of the legs of a stool, is that right,
21 Dr. Barrett?
22 A. I wouldn't put -- use that analogy.
23 Q. Excuse me?
24 A. I wouldn't use that analogy.
25 Q. Is that because we've shown that there are inaccuracies in
473
Barrett - cross
1 this linkage chart, Dr. Barrett?
2 A. No, not really. It's because the content validity of the
3 test is relatively obvious in this case. In some cases it's
4 not very obvious.
5 So this is just a way to document the fact, yes,
6 these items are from a source. So it's not really critical in
7 the same sense as some other things might be critical.
8 Q. Let me ask you to look at question 13, which is in
9 Defendant's Exhibit 14, Bates stamped number 8364.
10 Have you had a chance to look at that question?
11 A. Yes.
12 Q. Is that one of the questions that you testified on direct
13 examination is important for a lieutenant to know when he's
14 promoted, he or she is promoted to lieutenant?
15 A. Yes, it's in -- all the questions are job relevant.
16 Q. My question, Dr. Barrett, isn't whether it is job relevant.
17 My question is whether that's one of the questions that you
18 told us was important for a lieutenant to know?
19 A. All the knowledge is important on this test, yes.
20 Q. My question is: Is the answer to question number 13
21 important for a lieutenant to know?
22 A. Yes.
23 Q. And are you telling us that a lieutenant can't do his or
24 her job without knowing the correct answer to that question
25 without -- without knowing the correct answer to that question,
474
Barrett - cross
1 aside from being able to look it up?
2 A. That part of the job they couldn't do, that's correct,
3 without knowing the answer.
4 Q. And a lieutenant -- that that would detract from the job
5 performance of the lieutenant if he or she had to look up the
6 answer to that question? Is that what you are telling us?
7 A. Yes. In other words, you should be able to have the
8 knowledge without looking up all the information.
9 Q. And according to your linkage part, page 283, you could
10 find the answer to that question in general order 87-7, is that
11 right?
12 A. That's what the chart says.
13 Q. Now, if I can find my copy of general order 87-7, which is
14 on the table.
15 MR. FLAXMAN: I thought I had it handy, Judge, but I
16 don't. If I could take a minute to find it. I apologize.
17 THE COURT: Where are the general orders? What
18 exhibit?
19 MR. PIELL: Seven.
20 THE COURT: Don't you have a book, Mr. Flaxman?
21 MR. FLAXMAN: Excuse me?
22 THE COURT: Don't you have these books?
23 MR. FLAXMAN: I took it out of the book so I would
24 have it handy for examination, and I don't.
25 MR. HOLZHAUER: Mr. Flaxman, would you like to borrow
475
Barrett - cross
1 one of ours?
2 MR. FLAXMAN: Thank you so much.
3 BY MR. FLAXMAN:
4 Q. Let me show you what your kind counsel has made available
5 to me as general order 87-7. And isn't it correct that the
6 answer to question 13 is nowhere to be found in general order
7 87-7?
8 (Pause.)
9 BY THE WITNESS:
10 A. No, I can't find the answer in this general order.
11 BY MR. FLAXMAN:
12 Q. So it appears that there is another mistake in the linkage
13 chart, is that right, Dr. Barrett?
14 A. Yes, it looks like the wrong general order has been
15 inserted in the column.
16 Q. Let me ask you, Dr. Barrett, to look at test question
17 number 34 which is Defendant's Exhibit 14, Bates stamped 8372.
18 Have you had a chance to look at that test question?
19 A. Yes.
20 Q. And this is another one of those test questions that you
21 believe is important for a lieutenant to know without having to
22 look up?
23 A. Yes.
24 Q. And according to the linkage chart, that's based on general
25 order 91-14, is that right?
476
Barrett - cross
1 A. Yes.
2 Q. Let me show you general order 91-14. I'd ask you if you
3 could find the answer to question 34 in that general order.
4 Can't find it, can you, Doctor?
5 A. I see one distractor in there it looks like. But I'm not
6 sure I have studied it well enough to know.
7 Q. Well, take your time, Doctor.
8 A. What's the correct answer?
9 Q. Excuse me?
10 A. What's the correct answer?
11 Q. Just my question, Dr. Barrett, is: Can you find the
12 correct answer to that question in general order 91-14?
13 A. It could be E. I'm not sure.
14 Q. Are you telling us that based on your review of general
15 order 91-14, the correct answer to test question 34 is E?
16 A. No, I'm not saying. I'm saying it's possible, because I'm
17 not -- you know, it would be much easier if I knew what the
18 correct answer was so I could scan correctly.
19 Q. Well, the lieutenants -- the sergeants who took this test
20 were supposed to study the general orders, is that right,
21 Dr. Barrett?
22 A. That's correct.
23 Q. And you gave them a list of general orders to study so that
24 they would know the answers to the question, is that right?
25 A. That's correct.
477
Barrett - cross
1 Q. And if they studied the general orders as hard as they
2 could, they should be able to answer all those questions, is
3 that right?
4 A. Yes.
5 Q. And your linkage list says that the answer to question 34
6 comes from general order 91-14, is that right?
7 A. Yes.
8 Q. Could you tell us where in general order 91-14 the answer
9 is to that question, if anywhere?
10 A. Well, if I'm reading it correctly, the answer is E.
11 Q. "Missing persons under 17 years of age"?
12 A. And there is something about notification of this, of
13 reports of missing persons under age of 17, under D-8. Now,
14 I'm not sure, again, I have to probably study it more
15 thoroughly. That's my first --
16 Q. So are you telling us you're not sure of the correct answer
17 after looking at that general order?
18 A. That's correct.
19 Q. But that your best guess is that the correct answer is E,
20 is that right?
21 A. Well, that's what mentions the words. I'm just saying at
22 this point in time it looks like "missing persons under the age
23 of 17," it's also in E. So if you are saying that that's a
24 possible correct answer. And the answer, it says, "watch
25 operation command" --
478
Barrett - cross
1 Q. Let me ask you to look at general order 85-1. Let me
2 direct your attention in general order 85-1 to section 1, Roman
3 IV, subsection E-2-B?
4 A. I'm sorry. I missed what you said.
5 Q. General order 87-1.
6 A. Right.
7 Q. Roman IV?
8 A. Roman IV.
9 Q. Letter E, number 2, letter B. Have you found that,
10 Dr. Barrett?
11 THE COURT: I haven't even found the order yet.
12 I mean, do you guys care whether I'm following along
13 with you, or do you just want to have this dialogue between the
14 two of you? Because neither one of you give me a chance to
15 find an exhibit when you are examining witnesses. And I'm
16 surrounded here by about ten binders, and I'm trying to find
17 these exhibits, and you guys go merrily along your way. I
18 mean, if you want -- I am the Judge, I should be looking at
19 these things with you, don't you think? Now I've lost my
20 place.
21 What are you looking for?
22 MR. FLAXMAN: General order 85-1.
23 THE COURT: 85-1.
24 Okay. And what are you looking at?
25 MR. FLAXMAN: Number 1, Roman IV, the letter E, the
479
Barrett - cross
1 number 2 and the letter B.
2 THE COURT: Four.
3 MR. FLAXMAN: 1 IV E-2-B.
4 THE COURT: What does 1 IV mean?
5 MR. FLAXMAN: I'm sorry, it's 85-1, roman IV, letter
6 E, 2-B.
7 BY MR. FLAXMAN:
8 Q. Do you see where it says "discharge weapons, watch
9 commander's responsibility"?
10 THE COURT: No.
11 I'm looking at Roman numeral IV, A, B and C. That's
12 all I see here. Then it starts again with an A, I see. Are we
13 missing some pages? I'm missing pages. Maybe I am. It looks
14 like I am.
15 MR. FLAXMAN: Could I -- well, could I see your --
16 THE COURT: These pages aren't even numbered.
17 Here is my copy, Mr. Flaxman. Is the defendant
18 having the same problem as I am or am I just not seeing this?
19 MR. PIELL: Is this the page that you're working
20 from?
21 THE COURT: Here is 84-1. Here is my copy of 84-1.
22 MR. FLAXMAN: 85-1.
23 THE COURT: 85-1, rather. Here is a Post It. Put
24 the Post It by the paragraph you're talking about. Make sure
25 the witness has it as well.
480
Barrett - cross
1 MR. HOLZHAUER: What section are we looking for?
2 MR. FLAXMAN: IV E-2-B. And I'm confused.
3 THE COURT: Is this different than the one you have
4 there, Mr. Flaxman? I can see from this distance it looks
5 different.
6 MR. FLAXMAN: Let me just do it over and do a
7 different line of questioning, because I'm confusing all of us,
8 and I apologize.
9 THE COURT: Well, no. I'm following your line of
10 questioning here. And I think it's an important line of
11 questioning. So, you know, if we have to find an exhibit,
12 let's go find the exhibit. I don't want to throw you off.
13 It's just that I want to follow along with the testimony. And
14 I've been having trouble ever since this trial began.
15 MR. FLAXMAN: I might have the wrong number, Judge,
16 which might be why everybody is confused which is why --
17 MR. HOLZHAUER: Is that what you are looking for?
18 MR. FLAXMAN: Yes.
19 MR. HOLZHAUER: It's on the third page of that
20 exhibit, physically the third page of that exhibit.
21 THE COURT: Dr. Barrett, have you been able to find
22 it?
23 MR. FLAXMAN: Well, I just took his exhibit.
24 THE COURT: He took it from you.
25 MR. HOLZHAUER: This is the first page, the second
481
Barrett - cross
1 page, this is the third page of the exhibit. B, that's what
2 you want, right?
3 MR. FLAXMAN: Is that the answer to the question?
4 MR. HOLZHAUER: Do you see that, Judge?
5 THE COURT: No. Mr. Flaxman is going to put the Post
6 It right by it.
7 MR. HOLZHAUER: Okay.
8 THE COURT: Then he's going to make sure that
9 Dr. Barrett has it as well.
10 MR. FLAXMAN: Now, I'm not sure which is yours and
11 which is Dr. Barrett's.
12 THE COURT: Mine is in the bold print, the bolder of
13 the two prints. There is three pages.
14 MR. FLAXMAN: I'm laughing because I don't know what
15 order they were in, not in.
16 THE COURT: You've put the Post It by it, though,
17 have you not?
18 MR. FLAXMAN: Now let me help Dr. Barrett find it.
19 BY MR. FLAXMAN:
20 Q. All right. I will direct your attention, Dr. Barrett, to
21 the third page of general order 85-1, where it says, "E, watch
22 commander's responsibilities, two, investigation locals, B,"
23 where it says, "prior to relocating a preliminary
24 investigation." Do you see that, Dr. Barrett?
25 A. Yes.
482
Barrett - cross
1 Q. And is that in fact the correct answer to test question 34?
2 A. It looks like it might be. It looks like what occurred
3 they typed the number in from the distractor, because all our
4 distractors also contain the orders.
5 So what it appears they did, they used a distractor
6 from 91-14, which is the missing person under 17 years of age,
7 and when they typed it, somehow they missed -- they typed that
8 distractor number.
9 Q. Let me, let me try to see if you answered my question. The
10 correct answer to question 34 is from general order 85-1, is
11 that right, Dr. Barrett?
12 A. My first looking at it, it looks like you are correct. But
13 what we don't -- this is a very long order. And I'd have to
14 really read and digest it. But I will accept the fact that,
15 yes, that's the right answer.
16 Q. Well, what does question 34 ask? What is the question?
17 A. "Watch operations Lieutenant Jones must obtain approval
18 from the on-duty assistant deputy superintendent from the
19 bureau of operations services, BOS, regarding incidents of."
20 Q. And general order 85-1, section IV E-2-B, does that answer
21 that question?
22 A. I'm saying it appears to, yes.
23 Q. Well, what do you mean, "it appears to"? Does it or
24 doesn't it, Dr. Barrett?
25 A. The reason I say it, because I have not gone through in
483
Barrett - cross
1 depth the order and the distractors. When I review things,
2 it's on a form and we have available to us the source materials
3 and the reasons why this is the correct answer.
4 Q. Well, could you -- this was an actual test question on the
5 lieutenants test, is that right?
6 A. Yes.
7 Q. And somebody who wanted to be a police lieutenant had to
8 take this test, is that right?
9 A. That's correct.
10 Q. And they had to choose choices A through E to answer
11 question 34 correctly?
12 A. That's correct.
13 Q. How much time did they have for the 150 questions on the
14 written multiple-choice test?
15 A. They had one minute for each question, two and half hours.
16 Q. In that one minute for each question, were they able to
17 look at any general orders?
18 A. No.
19 Q. So this was based on your memory, is that right?
20 A. What you're -- yes, long term memory of what you have, the
21 knowledge you have.
22 Q. Now, how long has it been since you read general order
23 85-1, when I first gave it to you? Is that about two minutes
24 ago?
25 MR. HOLZHAUER: Objection. I don't believe he's read
484
Barrett - cross
1 the order.
2 MR. FLAXMAN: Strike that. If I may withdraw that.
3 BY MR. FLAXMAN:
4 Q. How long has it been since you had a chance to read
5 paragraph 1 IV E-2-B?
6 A. I don't know.
7 Q. Well, did you have a fair chance to read it to your
8 satisfaction?
9 A. As I said before, if I were actually reviewing the item and
10 developing, I would read the whole order and try to understand
11 it in context.
12 But I would agree on my superficial look, that
13 appears to be the correct answer.
14 Q. But you're not absolutely sure, is that correct?
15 A. No. I haven't read the whole order.
16 Q. Well, why would you have to read the whole order to answer
17 the question about from whom watch operations Lieutenant Jones
18 must obtain approval?
19 A. Because there might be something in the order I don't
20 understand. There might be something which contradicts that in
21 the order in some way in some situation. I don't know.
22 Q. Well, did you read all these general orders?
23 A. Yes, at one point in time I did.
24 Q. Did you find any language in there that were words you
25 didn't know?
485
Barrett - cross
1 A. What I think we have found in every organization we've been
2 in in all the orders that at times there are things which are
3 somewhat confusing.
4 Q. So did you find the police department of the City of
5 Chicago's general orders somewhat confusing?
6 A. In general, they were not. But I think certain ones
7 probably were somewhat confusing.
8 Q. Did you find anything in general order 85-1 confusing?
9 A. Well, I haven't gone through this and read it.
10 Q. Is there anything in section IV E-2-B confusing?
11 A. Well, again, you're taking it out of context. So if you
12 want me to read the whole order, I can.
13 Q. No, you --
14 A. But at this point --
15 Q. Is there anything in that section, Dr. Barrett, that's
16 confusing?
17 A. Do you want me to read it again?
18 Q. If that will help you answer the question.
19 (Pause.)
20 BY THE WITNESS:
21 A. Yes, there is something confusing in that if you just read
22 that section out of context it would be confusing. You have to
23 go to the very first part of the order to read that is
24 concerning weapons discharge instance involving sworn members.
25 And you should probably read the purpose and the scope, and
486
Barrett - cross
1 then what you referred me to does make some sense. Out of
2 context, it did not really fit the stem. But now it does.
3 BY MR. FLAXMAN:
4 Q. So now after having -- after having had a chance to read as
5 much as you wanted to of general order 85-1, do you agree that
6 the answer to question 34 is found in general order 85-1?
7 A. As far as I can tell, it is.
8 Q. And it is not found in general order 91-14, is that right?
9 A. That's correct.
10 Q. And 91-14 is what's shown on that linkage chart, is that
11 right?
12 A. What's shown is the 91-14 is distractor E.
13 Q. Are you telling us that section -- that choice E comes from
14 general order 94-14 -- excuse me -- comes from general order
15 91-14?
16 A. No. 91-14, as I said before, it's 2-D-8 it looks like to
17 me.
18 Q. Now, you read all the general orders of the Chicago police
19 department, is that right?
20 A. At one point in time.
21 Q. And you formed an opinion about which are important for
22 lieutenants to know in order to be lieutenants, is that right?
23 A. Yes.
24 Q. And that's what you testified here today, is that right?
25 A. Yes.
487
Barrett - cross
1 Q. And you became somewhat expert with these general orders,
2 is that right?
3 A. I wouldn't call it expert in the general orders.
4 Q. Well, you just told us that answer E, choice E for question
5 34 is based on general order 91-14, is that right?
6 A. I'm sorry, would you --
7 Q. Did you just tell us that choice E on question 34 comes
8 from general order 91-14?
9 A. That's what it looks like, yes, at this point.
10 Q. Well, what do you mean, "that's what it looks like,"
11 Dr. Barrett?
12 A. That's what I just said. I reviewed 91-14 and the phrase
13 appears in 91-14. It appears to match alternative E on item
14 34.
15 Q. Could you point out to us where in general order 91-14
16 choice E on question 34 comes from?
17 MR. FLAXMAN: And maybe we should give the Judge a
18 chance to get to order 91-14 if he doesn't have it.
19 THE COURT: I saw it.
20 MR. FLAXMAN: Okay.
21 BY THE WITNESS:
22 A. It's, I said before, item 14, it's Roman numeral II D-E.
23 And it says, "Reports of missing persons under the age of 17."
24 So the phrase, "missing persons under the age --
25 under 17 years of age" appears to have come from there. I'm
488
Barrett - cross
1 just saying the reason perhaps it was typed -- the clerical
2 error would result was because they might have typed that
3 distractor instead of the correct answer when they typed it.
4 BY MR. FLAXMAN:
5 Q. Now --
6 A. Because for every distractor, we have where it might come
7 from, a general order.
8 THE COURT: I see. You're saying -- excuse me.
9 You're saying that E is wrong, but it's a distractor
10 and it came from another order that mentioned the subject but
11 not in the context of the question? Is that what you are
12 saying?
13 THE WITNESS: That's correct.
14 BY MR. FLAXMAN:
15 Q. And you're also saying that E came from general order
16 91-14, is that right?
17 A. I'm just saying it's one possibility. I'm just saying that
18 there is the same words appears in 91-14 as appears in E, very
19 similar words.
20 Q. Well, Dr. Barrett, when you made up -- when your people
21 made up these questions, they weren't trying to use similar
22 words, were they? They were trying to use the precise
23 terminology used by the Chicago Police Department, weren't
24 they?
25 A. Well, you're making, I think, a leap there. What we're
489
Barrett - cross
1 trying to do for the correct answer would give them, if they
2 have the knowledge, they could choose the correct answer. The
3 other four were plausible distractors. We were trying to put
4 in the distractors, which people might check if they did not
5 actually have the correct knowledge to answer the question.
6 Q. And you just told us that based on your reading of general
7 order 91-14, that's where question E came from, is that right?
8 A. No, I'm not saying that. I said it looks like it may have.
9 What I would need would be my form for question 34, which would
10 give more detail.
11 Q. Your form is different than the linkage chart?
12 A. Well, it's just -- I think I just testified one explanation
13 might be that they typed 91-14 from a distractor, a plausible
14 distractor.
15 Q. Now, that form you talked about, that's not part of the
16 validation report that's been received in evidence, is it?
17 A. I'm sorry, say that again.
18 Q. That form to which you just referred, that's not part of
19 the validation report, Defendant's Exhibit 1, is it?
20 A. We gave you a sample of it in one of the appendices.
21 Q. My question, Dr. Barrett, is whether that form is part of
22 Defendant's Exhibit 1?
23 A. No. We didn't put all the forms in our report.
24 Q. And as a matter of fact, choice E on question 34 has
25 nothing to do with general order 91-14. It's based on general
490
Barrett - cross
1 order 79-6, Roman II, C-3-E.
2 MR. FLAXMAN: Let me give you a copy of that general
3 order. Take a moment while we all find it.
4 THE COURT: Give it to me again, general order 79-6?
5 MR. FLAXMAN: 79-6, Roman II, C-3-E.
6 THE COURT: II C, did you say?
7 MR. FLAXMAN: I said Roman II, capital letter C-3-E.
8 BY MR. FLAXMAN:
9 Q. Do you see where it says, "Missing person section, upon
10 receiving notification from the preliminary investigator of a
11 bona fide missing person, make any other notification
12 necessary"? Have you found that, Dr. Barrett?
13 A. Yes, I have it.
14 Q. And that's the source for the distractor, what you called
15 the distractor choice E, isn't that right?
16 A. I guess I'm confused, because I don't know for a fact, as I
17 said before, where the distractor came from. I said I need my
18 form for item 34 to find out.
19 But I guess I'm missing your point, because the words
20 on E appear to match better 91-14 than they do 79-6 Roman
21 numeral II, C-3-E. Now, it's possible that they may have taken
22 it from there. But, you know, the words match the other part
23 closer or may have been from nowhere. I don't know.
24 Q. Are you telling us that you really don't know how these
25 questions were prepared?
491
Barrett - cross
1 A. No, I'm not saying that at all.
2 Q. Are you telling us that you don't know which general orders
3 these questions were based on?
4 A. I'm saying without my form, I don't recall from three years
5 ago where all these distractors came from, nor for that matter
6 where the correct answers came from.
7 Q. Let's do something different. Let me ask you to look at
8 question 18 on the written test, which is page 8366 of
9 Defendant's Exhibit 14.
10 Have you had a chance to look at the question,
11 Dr. Barrett?
12 A. Yes.
13 Q. And that test question is based on according to the linkage
14 chart general order 88-18, is that right?
15 A. Yes.
16 Q. Let me show you a copy, if we can find it, of general order
17 88-18. That's entitled "Lineup Procedures," is that right?
18 A. Yes. It's called "Lineup Procedures."
19 Q. Now, let me ask you to look at that.
20 THE COURT: Mr. Flaxman, again, help me again. Come
21 on.
22 MR. FLAXMAN: I'm sorry, Judge.
23 THE COURT: Okay.
24 BY MR. FLAXMAN:
25 Q. Let me ask you to look at that general order.
492
Barrett - cross
1 (Pause.)
2 BY MR. FLAXMAN:
3 Q. Have you had a chance to look at the general order?
4 A. I glanced at it, yes.
5 Q. Okay. Let me direct your attention to Roman II, the letter
6 E, where it says, "Lineup should not be conducted in district
7 lockups unless circumstances exist which preclude the use of
8 another area," is that right?
9 A. I'm sorry, Roman II E you said?
10 Q. Right.
11 Do you see where it says, "Lineup should not be
12 conducted in district lockups"?
13 A. In that one, yes, I see. It's the third sentence.
14 Q. Okay. That sentence tells you, does it not, that choice A
15 is incorrect?
16 A. It appears to, yes.
17 Q. Okay. Now, look at section II D. Do you see where it
18 says, "As far as physical and time circumstances will permit,
19 the victim and/or witness will view the lineup separately"?
20 A. Yes.
21 Q. And that's choice B in the question, is that right?
22 A. Yes.
23 Q. Now, II G, do you see where it refers to "at least five
24 persons"?
25 A. Right.
493
Barrett - cross
1 Q. And that says, does it not, that choice C is incorrect?
2 A. Right.
3 Q. Now let's look at section E -- choice E, "individuals
4 should be of different height and weight." That's also
5 incorrect based on the same section, II G?
6 A. Okay.
7 Q. Could you show us, if you can, where in that general order
8 it says that choice D is wrong?
9 (Pause.)
10 BY THE WITNESS:
11 A. I would say it's II B.
12 BY MR. FLAXMAN:
13 Q. Does the general order say that if a lawyer appears for a
14 suspect who hasn't been charged that the lawyer could not
15 appear -- view the lineup?
16 A. I'm a little confused. I'm sorry, counsel.
17 Q. Does the order say that if the lawyer appears for a suspect
18 who has not been charged, that the lawyer cannot view the
19 lineup?
20 THE COURT: That's not what B says, Mr. Flaxman.
21 MR. FLAXMAN: I know.
22 BY MR. FLAXMAN:
23 Q. Could you tell us why it is that it's important for a
24 lieutenant to know that a suspect -- whether or not a suspect
25 has a right to have counsel present during a lineup?
494
Barrett - cross
1 A. Well, it would certainly have an effect upon the arrest
2 procedure and how good the arrest procedure is in subsequent
3 litigation.
4 Q. Are you -- do you know how this question was answered by
5 the sergeants who took the tests? Do you know how many got it
6 wrong?
7 A. No.
8 Q. Did you ever look at that data in the course of your work
9 in preparing the test?
10 A. We wouldn't have that data in preparing the test.
11 Q. Well, after the test was scored, did you ever look at how
12 police sergeants who took the test did on particular questions?
13 A. Yes, in general, I looked at all the statistics.
14 Q. Did you find that there were some questions that more than
15 half the sergeants who took the test got wrong?
16 A. Yes. There were, as in any test, there were some items
17 where --
18 MR. FLAXMAN: Move to strike after "yes."
19 THE COURT: Calls for a yes or no.
20 BY MR. FLAXMAN:
21 Q. How many questions were there more than half of the
22 sergeants who took the test got the question wrong?
23 A. I don't recall.
24 Q. Was it more than one?
25 A. I just don't recall the number.
495
Barrett - cross
1 Q. Was it more than 75, Dr. Barrett?
2 A. I don't recall the number.
3 Q. Excuse me?
4 A. I don't recall.
5 Q. And that didn't change your opinion that the test was
6 content valid, did it?
7 A. No.
8 Q. The fact that more than half of the police sergeants who
9 took this test were unable to answer some particular number of
10 questions didn't cause you to question your conclusion that
11 knowing the correct answer to each question was important for a
12 lieutenant to know?
13 A. No.
14 Q. Did you tell the City of Chicago that they should do
15 remedial training for their sergeants on these areas that more
16 than 50 percent didn't know the correct answer?
17 A. No.
18 Q. Did you ever share that information with the City of
19 Chicago?
20 A. Well, the City of Chicago has the same information I have,
21 I believe.
22 Q. Did you ever tell them which questions more than 50 percent
23 of the sergeants that took the test couldn't get right?
24 A. Well, again, all I can say is that they have the same data
25 I have on the questions.
496
Barrett - cross
1 Q. My question, Dr. Barrett, is: Did you ever tell them which
2 questions?
3 A. No.
4 Q. When did you give that data to the City, if ever?
5 A. I don't know the exact date. But they have had the same
6 data for some time, at least the law department has.
7 Q. That was after this lawsuit was filed, is that right?
8 A. I don't know the exact date.
9 Q. Well, when was the written test taken, Dr. Barrett?
10 A. November 19th, 1994.
11 Q. And it was scored by computer, is that right?
12 A. Yes.
13 Q. So it took, what, ten minutes to score the test?
14 A. Well, about.
15 Q. When were the results given back to the City of Chicago,
16 Dr. Barrett?
17 A. I don't have a date.
18 Q. Was it in November of '94?
19 A. I have no idea.
20 Q. Was it in 1994?
21 A. I think we actually scored the test in January 5th of 1996.
22 Q. So you had the test papers for more than a year before you
23 scored them?
24 A. I'm not -- I'm sorry, '95. I made a mistake, I'm sorry,
25 1995.
497
Barrett - cross
1 Q. So you had the test papers for more than two months before
2 you scored them?
3 A. Well, as I recall, we did final -- I shouldn't say that,
4 because I don't know the exact date. But that date sticks in
5 my mind in terms of me producing the list for the City.
6 Q. Now, you told us that there were some questions that were
7 eliminated after the test had been administered, is that right?
8 A. Yes.
9 Q. And were those questions that had been -- were those
10 questions reviewed by the subject matter experts before they
11 had been included on the test?
12 A. Yes.
13 Q. Had they been reviewed by your skilled test item writers
14 before they had been included on the test?
15 A. Yes.
16 Q. Why were they included on the test?
17 A. Every test we give -- and this has been going on for over
18 25 years -- you're always going to have a few items which are
19 appealed. We find that for some reason the key correct answer
20 is not the correct answer.
21 Q. Well, let me direct your attention on the written test to
22 question 87, which is at page 8395 of Exhibit 14. Let me take
23 back my other pages.
24 That's one of the questions that was not counted, is
25 that right?
498
Barrett - cross
1 A. I think, as I recall, everyone got it correct, was given a
2 correct -- given a point for that question.
3 Q. And do you know why all the choices were graded correct for
4 question 87?
5 A. No, I don't.
6 Q. Was 87 reviewed by Mr. Klein?
7 A. Yes.
8 Q. Was 87 reviewed by Mr. Shaw?
9 A. I assume, again, that they were all reviewed by all four
10 members of that reviewing committee.
11 Q. And did Mr. Klein say that it was okay to include question
12 87 on the test?
13 A. Well, keep in mind I'm testifying in general to our
14 process, that if anyone of the three committees thought it was
15 not an appropriate question, we would have taken it out. So
16 I'm assuming they did approve it.
17 Q. So you don't really have any personal knowledge that
18 Mr. Klein approved question 87, do you?
19 A. I have personal knowledge of the process we went through.
20 But I can't recall each and every item. But I assume that was
21 approved because they reviewed all the items.
22 Q. And after whoever approved it approved it, it was included
23 on the test, is that right?
24 A. Yes. I always read it myself and gave the final approval.
25 Q. Could you tell us why it was that you approved question 87?
499
Barrett - cross
1 A. I assume that I thought it was one correct answer. And the
2 other alternatives were not plausible or plausible but
3 incorrect answers. I was mistaken.
4 Q. Well, question 87 asks about -- the correct answer to
5 question 87 as originally came out of your office was choice A,
6 deception. Do you recall that?
7 A. No, I don't.
8 Q. Okay. Do you recall what the correct answer was as it came
9 out of your office?
10 A. No.
11 Q. Well, if you look at the linkage chart, would you find
12 question 87 linked to any particular order or general order or
13 statute?
14 A. It's 720 ILCS 5/15-4.
15 Q. Did you ever in the course of your work for the City of
16 Chicago refer to the Illinois Compiled Statutes?
17 A. Yes.
18 Q. And did somebody check to see whether that question is
19 based on 720 ILCS 5- -- 5/15-4, Dr. Barrett?
20 A. I'm sorry, what?
21 Q. Did somebody check to see whether the correct answer to 87
22 comes from 720 ILCS 5/15-4?
23 A. Yes. I assume people checked it.
24 Q. Who would have checked that, Dr. Barrett?
25 A. Well, it would be on our form for item 87. I don't know
500
Barrett - cross
1 exactly who checked it.
2 Q. And when that person checked it, what would they have done?
3 A. They would have read the source document and read the item.
4 Q. Now and who -- who would have done this in the regular
5 course of the way your business works?
6 A. One of the item writers.
7 Q. These item writers aren't police officers, are they?
8 A. No.
9 Q. They're not lawyers, are they?
10 A. No.
11 Q. Are they graduate students?
12 A. No. They're employees.
13 Q. Well, before they became employees, did they go to school?
14 A. You mean at one point were they graduate students?
15 Q. Right.
16 A. Yes.
17 Q. What's the level of the education -- how many item writers
18 were there on this test, Dr. Barrett?
19 A. I believe there were ten item writers.
20 Q. Did they all have high school diplomas?
21 A. Yes.
22 Q. Did they all have bachelor's degrees?
23 A. Yes.
24 Q. Did they have all master's --
25 MR. HOLZHAUER: Your Honor, I object. He's harassing
501
Barrett - cross
1 the witness. He already said they were graduate students at
2 one point.
3 THE COURT: Overruled.
4 BY MR. FLAXMAN:
5 Q. Did they all have master's degrees?
6 A. As I recall, yes.
7 Q. Were any of them -- did any of them have PhDs?
8 A. I'm not really sure. I don't recall. One or two may have,
9 but I just cannot tell you for sure.
10 Q. Had any of them started PhD studies?
11 A. Probably they had, yes.
12 Q. Were any of them ABD, all but dissertation?
13 A. I think there were several like that.
14 Q. Okay. And these were the people who checked 720 ILCS
15 5/15-4, is that right?
16 A. Yes.
17 Q. And you also checked 720 ILCS 5/15-4, is that right?
18 A. That's correct.
19 Q. And when you checked that statute, did you -- well, let me
20 withdraw that.
21 You told us on direct examination that in addition to
22 being a psychologist you are a lawyer, is that right?
23 A. That's correct.
24 Q. And you're admitted to practice in the state of Ohio, is
25 that correct?
502
Barrett - cross
1 A. That's correct.
2 Q. So you're knowledgeable about reading state statutes, is
3 that right?
4 A. I have some knowledge of that.
5 Q. And when you read 720 ILCS 5/15-4, did you see that it
6 doesn't define an offense? And let me show you 720 ILCS
7 5/15-4, if I may?
8 THE COURT: Do you have any for us, for me, for your
9 opponent?
10 MR. FLAXMAN: There is an exhibit that's a defense
11 exhibit, which is this Illinois Criminal Law and Traffic
12 Matters.
13 THE COURT: Well, please tell me what it is,
14 Mr. Flaxman. I don't understand the way you guys are trying
15 this case. It's like I'm in the dark here. Tell me the
16 exhibit number. I'll try to find it.
17 MR. HOLZHAUER: We will.
18 MR. FLAXMAN: I believe it would be in a separate
19 book, Judge, that defendants gave you.
20 MR. FUTTERMAN: I think it's Defendant 12.
21 MR. FLAXMAN: I don't know how it was given to you,
22 whether it's in a book or not, a separate binder.
23 MR. HOLZHAUER: Your Honor, I have a copy here that
24 you can just take. It is in one of our binders, and I can give
25 you the number in a second.
503
Barrett - cross
1 THE COURT: Okay, thank you. This is not a book that
2 I need to keep handy anymore.
3 MR. FLAXMAN: It was a book that was given as part of
4 the defense exhibits.
5 THE COURT: All right. You've got to refer to the
6 exhibits, Mr. Flaxman.
7 MR. FLAXMAN: All right. It is Defendant's Exhibit
8 12, Your Honor.
9 MR. HOLZHAUER: That's correct, Your Honor, 12.
10 MR. FLAXMAN: And it's on page 579, to which I'm
11 directing the witness.
12 BY MR. FLAXMAN:
13 Q. Have you had a chance to look at that section, Dr. Barrett?
14 A. Yes.
15 Q. Now, question 87 on the test, which is page 228 -- no,
16 which is page 8396 of Defendant's Exhibit 14 asks the sergeant
17 taking the test whether certain elements constitute a
18 particular offense, is that right?
19 A. Well, I can read the stem. But I'm not sure what you are
20 asking.
21 Q. Why don't you take a moment and read the stem.
22 A. "A woman who works for a large private company confesses to
23 Police Officer Thompson" --
24 THE COURT: You don't have to read it into the
25 record, Doctor.
504
Barrett - cross
1 THE WITNESS: Okay.
2 THE COURT: I've read it. It's part of the record.
3 It's in evidence.
4 BY MR. FLAXMAN:
5 Q. Okay. The question, Dr. Barrett, is, question 87 asks the
6 police sergeant taking the test to identify what offense has
7 been committed from the facts set out in question 87, is that
8 right?
9 A. Yes.
10 Q. And, in fact, there is no offense of deception in 720 ILCS
11 5/15-4, is that right?
12 A. That's right.
13 Q. Deception is just a definition in that section, is that
14 right?
15 A. That's right.
16 Q. And your trained item writers and your subject matter
17 experts reviewed that question and allowed it to be given to
18 police sergeants for promotion to lieutenant, is that right?
19 A. That's correct.
20 Q. And is that one of the questions that when you approved the
21 test you believed a police sergeant should know the correct
22 answer to before being promoted to lieutenant?
23 A. Yes.
24 Q. And you believed it was important for a police lieutenant
25 to know the correct answer to that question, is that right?
505
Barrett - cross
1 A. At that point in time, yes.
2 Q. And let me ask you to look at question, test question 103,
3 which is in Defendant's Exhibit 14 at page 8402. And could you
4 tell us what the linkage chart says this question, question
5 103, is based on?
6 A. 725 ILCS 5/112A-3(4).
7 Q. And is it also -- is this question also one that was
8 reviewed by the item writers, reviewed by the subject matter
9 experts and reviewed by you before it was given as part of the
10 test?
11 A. Yes.
12 Q. And you concluded before the test was given, did you not,
13 that it was important for a police lieutenant to know the
14 correct answer to this question?
15 A. Yes.
16 Q. And this question asks again whether certain facts
17 constitute a particular offense, is that correct?
18 A. Yes.
19 Q. And let me direct your attention to Defendant's Exhibit 12
20 at page 755 and ask if that's where you can find the statute
21 that's referred to in the linkage chart for question 103?
22 A. Yes.
23 MR. FLAXMAN: I'll wait for Your Honor to --
24 THE COURT: I've got it.
25 MR. FLAXMAN: Okay.
506
Barrett - cross
1 BY MR. FLAXMAN:
2 Q. Again, that statute doesn't talk about an offense. That's
3 a definition statute, isn't that true?
4 A. Yes.
5 Q. And this question 103 that survived the screening was
6 another one of those questions where all choices were graded
7 correct, is that right?
8 A. Yes.
9 Q. Now, you told us before lunch that the subject matter
10 experts told you that the rules and regulations of the City of
11 Chicago Police Department were outdated and should not be
12 included on the test, is that right?
13 A. I think our records should reflect that.
14 Q. And did you ever review -- did you ever review those rules
15 and regulations?
16 A. I don't recall reviewing them, no.
17 Q. Well, did you ever learn that Rule 4 says that, "Any
18 conduct or action taken to use the official position for
19 personal gain or influence" is one of those rules and
20 regulations?
21 A. I don't recall.
22 Q. Is it your opinion as you sit here now that Rule 4 is
23 outdated?
24 A. I'm confused.
25 Q. Is it your opinion that a Chicago police lieutenant is not
507
Barrett - cross
1 required to know that "any conduct or action taken to use the
2 official position for personal gain or influence" is contrary
3 to the rules and regulations of the Chicago Police Department?
4 A. I don't know what the rules and regulations are of the
5 Chicago Police Department right now, if that's what you are
6 asking me. I assume it's reasonable for it to be still true,
7 but I don't know.
8 Q. Well, was there any question on your test asking the police
9 sergeant about whether or not it was contrary to the rules of
10 the Chicago Police Department to use the official position for
11 personal gain or influence?
12 A. I don't recall a question like that.
13 Q. Is that something that's important for a sergeant to know?
14 A. I would assume it would be important, yes.
15 Q. Is that important for a lieutenant to know?
16 A. I would assume it would be, yes.
17 Q. Does your job analysis, your master job analysis indicate
18 that that's a knowledge, skill or ability that a lieutenant has
19 to have?
20 A. I don't know if it does or not.
21 Q. Well, let's take a look at your job analysis. Do you have
22 that in front of you, Dr. Barrett? Do you have it,
23 Dr. Barrett?
24 A. What you are referring to is C, is that right?
25 Q. It's in Defendant's Exhibit 1, appendix C, starting at page
508
Barrett - cross
1 198.
2 A. Yes, I have it.
3 Q. All right. Could you find for us, if you can, any
4 reference to not using official position for personal gain or
5 influence?
6 A. You mean in terms of job description now?
7 Q. Right.
8 (Pause.)
9 BY THE WITNESS:
10 A. It probably would be best to follow under major work
11 behaviors in terms of -- it looks to be -- a footnote from
12 across would be Roman numeral I G.
13 THE COURT: Page?
14 THE WITNESS: It's 202. It's C-12 in the appendix.
15 BY MR. FLAXMAN:
16 Q. Anything else, Dr. Barrett?
17 A. You want me to find more behaviors like that, you're
18 asking? I'm sorry, what do you want me to do?
19 Q. Is it -- is that the extent of a job description?
20 A. Well, I thought you had attempted to identify that sort of
21 thing in the job description. Do you want me to look for more?
22 Also you could go to, I guess, I C, and I C --
23 THE COURT: Page?
24 THE WITNESS: 200.
25 BY THE WITNESS:
509
Barrett - cross
1 A. -- also speaks to these sorts of issues in terms of what
2 might be appropriate.
3 BY MR. FLAXMAN:
4 Q. What is -- what did you just say, Doctor, on page C, on
5 page 200?
6 A. "Provides counseling and coaches department members."
7 Q. And you're telling us that that's the same as not -- as
8 conduct or action -- as not taking any conduct or any action to
9 use the official position for personal gain or influence?
10 A. Well, if someone did that, I would assume they would at
11 least require some coaching or some counseling as a possibility
12 according on how severe it was.
13 Q. And this is a judgment you are making -- you're
14 interpreting your job description for us, is that right,
15 Dr. Barrett?
16 A. I'm reading from the job description fitting in your --
17 your comment to the categories of work behavior.
18 Q. And is it true that your claim of content validation is
19 based on judgment decisions you're making?
20 A. Basically content validity is based upon judgment, yes.
21 Q. And whose judgment is it that your claim of content
22 validity for this test is based on?
23 A. It's based upon the process we described in my direct
24 testimony starting --
25 Q. Whose judgment is it based on? Is it based on your
510
Barrett - cross
1 judgment?
2 A. Well, it's based on my judgment and others who work with me
3 in this process. It's based in part on the judgment of the
4 SMEs. It's based in part upon everybody we interviewed and
5 gave us input.
6 Q. Is it the same SMEs who approved that same question about
7 offenses that weren't offenses, is that right?
8 A. That's correct.
9 Q. Okay. And is it based on the same judgment of these item
10 writers or whoever it was who made the mistakes that you've
11 told us about in that linkage chart, is that right?
12 A. That's correct.
13 Q. Now, did you ever do any study to determine how much job
14 knowledge a Chicago police lieutenant needs to do his or her
15 job?
16 A. Yes.
17 Q. Well, did you give a job knowledge test to incumbent police
18 lieutenants?
19 A. No.
20 Q. Is that the way of determining a passing score on a job
21 knowledge test, to administer it to people who are performing
22 the job and see how they do on it?
23 A. That is a process I would not do or have not done. As I
24 mentioned before, the confidentiality issues preclude that sort
25 of approach.
511
Barrett - cross
1 Q. Putting aside the confidentiality issues, if you had a job
2 knowledge test and you want to determine what an appropriate
3 passing score is for people in a particular profession, would
4 you, would you administer it to incumbents and find out what
5 the minimum level of job knowledge is needed for those people
6 who were doing their job well?
7 A. I may or may not. It's one possible way of approaching a
8 problem like that.
9 Q. Is that a way of doing it, Dr. Barrett?
10 A. That's one possibility.
11 Q. And did you ever do anything to find out what the minimum
12 job knowledge was for somebody who could perform their job as a
13 Chicago police lieutenant?
14 A. No. I never made any determination of what the minimum
15 knowledge was in terms of requirements.
16 Q. And did you ever do any research to show that whether or
17 not it's true that somebody who knows the correct answers to
18 more of the questions on your test will do better on the job as
19 a police lieutenant than somebody who knows the answers to the
20 least questions?
21 A. Yes, we did an analysis like that.
22 Q. You did -- did you do data? Did you measure incumbent
23 police lieutenants?
24 A. I said before, we did not at any time measure incumbent
25 sergeants in terms of their job knowledge on this test.
512
Barrett - cross
1 Q. Well, you said you did data. Isn't it true, Dr. Barrett,
2 that you don't have any empirical evidence to show that a
3 police sergeant who scored higher on that written test of job
4 knowledge will do better as a police lieutenant than a police
5 sergeant who did not score as high?
6 A. That's not quite correct, again, in terms of how you are
7 defining empirical.
8 Q. Well --
9 A. If you define empirical merely as a criterion related to
10 study, I've said many times that, no, we never conducted a
11 criterion-related study with the lieutenants or sergeants in
12 the Chicago Police Department.
13 Q. Well, let me see if I can get you to answer the question,
14 Doctor.
15 Isn't it true that you don't know of any study in any
16 police department which provides empirical evidence that a
17 higher score on a written test of job knowledge will be
18 reflected in better performance when promoted?
19 A. I believe there's a study by Shoop of police sergeants
20 where they did correlate a content valid test with job
21 performance.
22 Q. Other than that one study by Shoop -- that was in 1962,
23 Dr. Barrett?
24 A. I don't recall the date.
25 Q. That involved about eight police lieutenants in a Michigan
513
Barrett - cross
1 police department?
2 A. I think it was more than that.
3 Q. Other than that one study by Shoop -- which hasn't been
4 published in a refereed journal, has it, Dr. Barrett?
5 A. No.
6 Q. That's an unpublished article, isn't that right?
7 A. It's a technical report.
8 Q. Excuse me?
9 A. It's a technical report, yes.
10 Q. So it hasn't been subject to peer review, right?
11 A. That's right.
12 Q. You don't know how many mistakes, if any, Dr. Shoop made in
13 that result?
14 A. No, I don't.
15 THE COURT: Is it Chute?
16 BY MR. FLAXMAN:
17 Q. S-h-o-u-p, Dr. Barrett?
18 A. I've forgotten. I thought it was S-h-o-o-p, but I could be
19 wrong.
20 Q. Other than that one unpublished study by Dr. Shoop, isn't
21 it true that there isn't any other study of any police
22 department which provides empirical evidence that a higher
23 score on a written test of job knowledge will be reflected in
24 better performance when promoted?
25 A. I don't know of any criterion-related study using job
514
Barrett - cross
1 knowledge in a police department beyond what I've told you.
2 Q. My question didn't mention the word "criterion-related
3 study," Dr. Barrett.
4 My question was, and I'll give it to you again:
5 Isn't it true, Dr. Barrett, that you don't know of any study in
6 any police department which provides empirical evidence that a
7 higher score on a written test of job knowledge will be
8 reflected in better performance when promoted?
9 A. No, I don't agree with that.
10 Q. Do you know of any studies other than that unpublished
11 study by Dr. Shoop?
12 A. I know the study -- the study we did, for example, the
13 study we've been doing for years using job analysis
14 information, using observation, and using, for example, in the
15 reply to Bishop, we have system flow diagrams for many of the
16 items showing in empirical fashion where you can see very
17 distinctly that here are the events which we have found from
18 our job analysis which are important for an officer -- for a
19 police lieutenant to do on the job, and we have shown the
20 underlying knowledge which is required for you to solve these
21 important problems. It is very easy to see that if you do not
22 have that knowledge, you can solve almost none of the problems
23 which are confronted by a police lieutenant.
24 Conversely, are -- those charts, for example, show
25 that, you know, if you do have the knowledge, these are
515
Barrett - cross
1 important problems that you can solve. That's empirical
2 evidence. You can observe it. It's a process. It goes even
3 beyond our job analysis evidence in terms of what you can see,
4 yes, here are the major work behaviors, and you can see
5 knowledge underlying that. But this is a probably more graphic
6 presentation of it.
7 Q. Well, Dr. Barrett, you testified in the preliminary
8 injunction hearing in this case. Do you remember that?
9 A. Yes, I do.
10 Q. And you were here before Judge Gettleman, is that right?
11 A. That's correct.
12 Q. And do you remember at page 265 I asked you the question
13 that I just asked you now: "Isn't it true, Doctor, that you
14 don't know of any study of any police department which provides
15 empirical evidence that a higher score on a written test of job
16 knowledge will be reflected in better performance when
17 promoted?"
18 And then you gave the answer: "No, I don't know that
19 any -- at this point any sort of --" court reporter wrote
20 "unintelligible study like that."
21 Do you remember that question and that answer?
22 BY THE WITNESS:
23 A. I think if you look before or after that question, you'll
24 see I define "empirical" as in a sense of criterion-related
25 study. That's what I recall empirical in the sense of
516
Barrett - cross
1 criterion-related study.
2 THE COURT: Hold on. All right. Go ahead.
3 MR. FLAXMAN: I would move to strike all of it after
4 when he starts explaining, Judge.
5 THE COURT: Overruled.
6 MR. FLAXMAN: Are you ready?
7 THE COURT: I'm ready.
8 BY MR. FLAXMAN:
9 Q. Were you finished, Dr. Barrett?
10 A. Yes.
11 Q. So you said we should look before that question and that
12 answer in the transcript?
13 A. I'm saying I recall that we had a problem about defining
14 empirical. And I was saying as I recall that empirical, if you
15 define it as criterion-related, then there is relatively little
16 criterion-related studies we call empirical which reflects that
17 issue.
18 Q. Well, before you testified here, did you have a chance to
19 look over your testimony from the preliminary injunction
20 hearing?
21 A. Yes.
22 Q. And did you look over where I asked you those questions?
23 A. Yes.
24 Q. And do you remember the question I asked you when I asked
25 you:
517
Barrett - cross
1 "As a matter of fact, there is no empirical evidence
2 to show that a police sergeant who scores higher on a
3 written test of job knowledge will do better as a
4 police lieutenant than a police sergeant who does not
5 score as high, isn't that true"?
6 And your answer was:
7 "I don't know of any study in the Chicago Police
8 Department which shows that a job knowledge test for
9 lieutenants relates to job performance, no, I don't.
10 That's true."
11 A. Yes. There is no criterion-related study taking a test, a
12 job knowledge test and relating it to job performance. I agree
13 with that, actual job performance.
14 Q. Well, you keep using criterion-related study, Dr. Barrett.
15 I'm asking you about empirical evidence. And my question then
16 was about empirical evidence. My question now is about
17 empirical evidence.
18 Is there any empirical evidence to show that a police
19 sergeant who scores higher on a written test of job knowledge
20 will do better as a police lieutenant than a police sergeant
21 who does not score as high other than that unpublished study by
22 Dr. Shoop?
23 A. My answer is the same as I recall in the preliminary
24 hearing and now, I'm saying if you define empirical as being
25 criterion-related, there is no study in the Chicago Police
518
Barrett - cross
1 Department.
2 And if you define empirical a different way, then the
3 answer is yes, there are studies, including our own.
4 Q. Okay. Well, let's talk about empirical as a
5 criterion-related study. What do you mean by a "criterion
6 related study"?
7 A. So we are clear, empirical now is going to mean
8 criterion-related from your point of view.
9 Q. No. My question, Dr. Barrett, you just used the word
10 "criterion-related study." What did you mean by that?
11 A. That's a technique of validation. There are two different
12 types of validation approaches under criterion-related: One is
13 concurrent validation approach, one is predictive. I described
14 them briefly, I believe, this morning. But I can -- I can
15 describe them again for you.
16 Q. Did you under -- did you propose to do a concurrent
17 validation study for the City of Chicago for promotions to
18 lieutenant?
19 A. I did not.
20 Q. Did you propose to do a predictive criterion validation
21 study for the City of Chicago?
22 A. I did not.
23 Q. Now, if you had done either kind of a criterion-related
24 study, would you have gotten a number for a validity
25 coefficient?
519
Barrett - cross
1 A. You're assuming that it's feasible to do a
2 criterion-related study in the City of Chicago.
3 Q. Well, my question doesn't assume anything, Dr. Barrett.
4 My question is: If you do a criterion-related study,
5 do you end up with something called the validity coefficient?
6 A. That's correct.
7 Q. What is a validity coefficient?
8 A. It's a correlation coefficient.
9 Q. Is it a number?
10 A. That's correct.
11 Q. It's a number between minus one and one, I guess?
12 A. That's correct.
13 Q. And is it something that you could write down on a piece of
14 paper, is that right?
15 A. Yes.
16 Q. And you don't have a validity coefficient for this test, do
17 you, Dr. Barrett?
18 A. No, I do not.
19 MR. FLAXMAN: Could we take a short break, Judge?
20 THE COURT: We're only going until 4:00, maybe even a
21 little before 4:00.
22 MR. FLAXMAN: A very short break. I just need to --
23 THE COURT: Are you almost done?
24 MR. FLAXMAN: Oh, no.
25 Do you want me to keep going?
520
Barrett - cross
1 THE COURT: Well, I would like to use the time.
2 MR. FLAXMAN: All right. Can I take a minute and
3 have a glass of water?
4 THE COURT: Sure.
5 MR. FLAXMAN: All right.
6 BY MR. FLAXMAN:
7 Q. Dr. Barrett, did you pilot test the written test with
8 police sergeants?
9 A. No.
10 Q. And the time limit, there was 150 minutes for 150
11 questions?
12 A. No.
13 Q. How long did the people have to do the written test
14 questions?
15 THE COURT: Which test are you talking about?
16 BY MR. FLAXMAN:
17 Q. The written test, the job, the job knowledge test, the
18 150-question multiple-choice test.
19 A. Two and a half hours, as I recall.
20 Q. And that's 150 minutes?
21 A. Yes.
22 Q. So that's one minute a question, is that right?
23 A. Yes.
24 Q. Did you ever do any research to see whether there would be
25 different passing -- different answer rates if people had two
521
Barrett - cross
1 minutes a question?
2 A. No.
3 Q. Did you ever pilot test 150 questions with anybody?
4 A. We have often used that size questions with that time limit
5 before.
6 Q. Well, did you ever pilot test a 150-question
7 multiple-choice test?
8 A. No. We've never pilot tested 150 choice -- I mean, a pilot
9 test for time, you're asking? We've pilot tested, yes. But
10 not in terms of your hypothetical.
11 Q. Is it just your judgment that there should be one minute a
12 question, Dr. Barrett?
13 A. I think it's a standard in the field, I believe. Also, as
14 I said, we have past experience in doing this.
15 Q. Well, is there any documentation that you can refer us to
16 where it says that one minute a question is the standard in the
17 field?
18 A. I don't -- at this point, I can't recall a good source for
19 you.
20 Q. Is that in the EEOC guidelines that you referred to,
21 Dr. Barrett?
22 A. No, it's not in the EEOC guidelines.
23 Q. Is it in the APA guidelines that you referred to?
24 A. No, it's not in the APA guidelines.
25 Q. Is it in the Society of Industrial and Organizational
522
Barrett - cross
1 Psychology guidelines that you referred to?
2 A. No, it's not in there.
3 Q. It's just something that you always used, is that right?
4 A. A study which we have used and found it works, yes.
5 Q. Did you ever do any study to see whether there would be
6 less disparate impact in this kind of test if there was two
7 minutes a question?
8 A. No.
9 Q. Did you ever do any part of your job analysis to see
10 whether lieutenants have to be able to answer these type of
11 questions in one minute?
12 A. Yes. Our general analysis did indicate that it was
13 important to give a response to citizens and to subordinates in
14 a timely fashion. Everyone said this in terms of, can you go
15 back and look up a book. No, they want an answer, an irate
16 citizen, arrest situation, whatever it might be, and that the
17 command structure demands that the response be one which does
18 take place in a relatively short time. In fact, a minute would
19 probably be too long.
20 Q. And the job analysis showed you -- are you telling us the
21 job analysis showed you that citizens get to make inquiries of
22 watch commanders or lieutenants about the type of questions
23 that were on the test?
24 A. They may. It's certainly -- it's possible. And I also
25 recall some talking about speaking to community groups, I
523
Barrett - cross
1 believe, also.
2 Q. Other than speaking to community groups, isn't it true,
3 Dr. Barrett, that nobody, none of these police lieutenants and
4 captains who were interviewed by Barrett & Associates in the
5 job analysis said that a police lieutenant has to be able to
6 answer questions from citizens about landlord/tenant
7 ordinances? Nobody said that, is that right?
8 A. I don't recall if that specific thing was brought up or
9 not. I think, I think Mr. Bishop, though, in his testimony was
10 very clear about --
11 MR. FLAXMAN: I move to strike the reference to
12 Mr. Bishop.
13 THE COURT: Sustained.
14 BY MR. FLAXMAN:
15 Q. Isn't it true, Dr. Barrett, that none of the subject matter
16 experts told you and your job analysts that Chicago police
17 lieutenants or watch commanders have to answer questions from
18 citizens about a hearing for a revocation of a weapon?
19 A. I don't recall.
20 Q. And there is nothing in your job analysis which shows that
21 that's something that field lieutenants or watch commanders
22 have to do, isn't that right?
23 A. No. We actually have in our job analysis many different
24 events, and a lot of the events which we have in our tests are
25 from the job analysis.
524
Barrett - cross
1 THE COURT: It's not a beeper. It's just --
2 MR. FLAXMAN: It's not going to blow up.
3 THE COURT: It's just an errant computer.
4 MR. FLAXMAN: I don't know why it is doing that, but
5 it will stop, I hope.
6 BY MR. FLAXMAN:
7 Q. Have you ever given a written job knowledge test where
8 there was a passing score, Dr. Barrett?
9 A. We may have. I don't recall. We may have. I don't recall
10 us doing it recently.
11 THE COURT: You mean a pass/fail?
12 MR. FLAXMAN: Where there is -- where there is a
13 score that you pass if you -- where there is a passing score.
14 THE COURT: I see.
15 BY MR. FLAXMAN:
16 Q. Do you know of any industrial psychologist who has prepared
17 a job knowledge test where there is a passing score,
18 Dr. Barrett?
19 A. I'm sure there have been.
20 Q. You didn't consider that for the 1994 lieutenants test, did
21 you?
22 A. No, we didn't have any passing score on any parts of the
23 tests.
24 Q. I didn't hear you, Dr. Barrett.
25 A. No, we did not have any passing score on --
525
Barrett - cross
1 Q. No. My question, Dr. Barrett, was: You did not consider
2 having a passing score for the job knowledge test, did you?
3 A. We probably considered it. But we didn't do it.
4 Q. Did you consider it when you made your proposal to the City
5 of Chicago for how you were going to prepare the job knowledge
6 test?
7 A. Yes.
8 Q. When you made that proposal, did you tell the City that you
9 were going to use the scores to make promotions in rank order?
10 A. I don't recall what we said. But it sounds like we
11 probably would have said that. I don't recall.
12 Q. Would anything refresh your recollection about what it was
13 that you said when you submitted your proposal to the City of
14 Chicago.
15 A. The actual proposal would refresh my memory.
16 Q. Excuse me?
17 A. The proposal itself would refresh my memory.
18 Q. Let me show you what's been marked as Plaintiffs' Exhibit
19 59. Is that the proposal?
20 A. It looks like a contract. It's a contract it looks like.
21 Q. Well, is the proposal included in the contract,
22 Dr. Barrett?
23 A. The back of Exhibit A, it's not labeled, the proposal.
24 Q. Have you found it?
25 A. Yes, I've found it. It's not labeled as a proposal. I'm
526
Barrett - cross
1 just surprised it's not.
2 Q. Are there Bates stamp numbers on the bottom right-hand
3 corner of those pages?
4 A. Yes. 7415.
5 Q. And is that the proposal that you submitted to the City of
6 Chicago in which you outlined that you were going to give a
7 three-part -- you were going to prepare a three-part test?
8 A. Yes.
9 Q. And nothing in there talks about a passing score on the
10 written test, does it, Dr. Barrett?
11 A. Well, I haven't read it yet.
12 Q. Well, take the chance to read it.
13 (Pause.)
14 BY MR. FLAXMAN:
15 Q. Are you familiar with that document, Dr. Barrett?
16 A. I was three years ago. It doesn't mention any cutoff
17 score.
18 Q. And what you just looked at was the proposal you submitted
19 to the City of Chicago for how you were going to do the test,
20 is that right?
21 A. That's correct.
22 Q. And so isn't it true, Dr. Barrett, that from the very
23 beginning of your work for the City of Chicago, you were -- it
24 was clear that you were not going to give a written test with a
25 cutoff score?
527
Barrett - cross
1 A. Yes. We didn't propose a cutoff score.
2 Q. And you didn't do any work to determine whether or not
3 there was some minimum level of knowledge that somebody needs
4 to know to be a lieutenant, is that right?
5 A. No, we don't do any study like that.
6 Q. And is it fair to say that your -- that the City's use of
7 scores on the written job knowledge test as one component of
8 making rank order selection is based on your recommendation
9 that somebody with the higher score on the written job
10 knowledge test is going to do better as a lieutenant than
11 somebody with a lower score?
12 A. Yes.
13 Q. Now, the oral component of the 1994 lieutenants test,
14 that's something else that you had proposed right at the
15 beginning when you got the contract with the City of Chicago,
16 is that right?
17 A. That's correct.
18 Q. And you made that proposal about the written test and the
19 oral test, and that included, also included an in-basket
20 exercise of the type that was actually administered, is that
21 right?
22 A. That's correct.
23 Q. You made that proposal, did you not, before doing the job
24 analysis, is that right?
25 A. That's correct.
528
Barrett - cross
1 Q. And that proposal was the standard three-part battery that
2 Barrett & Associates administers for public safety usage, is
3 that right?
4 A. That's correct.
5 Q. The oral briefing exercise, that was -- that was not
6 designed to mirror the actual oral communication that a
7 sergeant has to do on the job, does it, Dr. Barrett?
8 THE COURT: Do you mean lieutenant?
9 MR. FLAXMAN: I'm sorry. Let me rephrase the
10 question.
11 BY MR. FLAXMAN:
12 Q. The oral briefing exercise, that was not designed to mirror
13 the actual oral communication that a lieutenant has to do on
14 the job, was it, Dr. Barrett?
15 A. It was designed to be an exemplar of all communication --
16 THE COURT: Excuse me, Doctor. Can you answer the
17 question yes or no?
18 BY THE WITNESS:
19 A. No, I can't answer it yes or no.
20 BY MR. FLAXMAN:
21 Q. Excuse me?
22 A. I can't answer it yes or no.
23 Q. Well, was it designed to mirror the actual oral
24 communication that a lieutenant does on the job?
25 A. That some lieutenants might do on the job, yes.
529
Barrett - cross
1 Q. And it was designed before the job analysis, is that right?
2 A. We didn't actually design it before the job analysis. But
3 in terms of actual design of the content.
4 Q. Well, wasn't the oral test administered on June 1st, 1994?
5 A. That's correct.
6 Q. That was before the job description was prepared, isn't
7 that true, Dr. Barrett?
8 A. Well, I think early on we had some draft job descriptions.
9 You're correct, the final master job description was not
10 prepared until later probably.
11 Q. I didn't hear that.
12 A. I'm saying that the final master job description was not
13 prepared until later.
14 Q. Well, are you telling us that the oral examination was
15 designed to simulate the oral communications of some
16 lieutenants in the Chicago Police Department?
17 A. Well, it was designed to be a simulation, and it was
18 designed to be exemplar of oral communication in general, yes.
19 Q. Well, you told us before it was designed to simulate the
20 communication, oral communications of some lieutenants. Which
21 lieutenants, Dr. Barrett?
22 A. We found that there were different styles of giving a roll
23 call, for example. So for some lieutenants, it might be a much
24 more exact. Other lieutenants might have a different style.
25 Q. Well, was there any research that you did which showed that
530
Barrett - cross
1 that oral component of your test simulated the way in which
2 roll call was actually performed by any Chicago police
3 lieutenants or captains?
4 A. Just what they told us in their job analysis.
5 Q. Who told you that, Dr. Barrett?
6 A. I don't recall the exact name.
7 Q. Is there any documentation anywhere that reports that
8 somebody told you that's the way roll call is conducted?
9 A. No. I don't have any exact documentation.
10 Q. Can you recall the name of anybody who told you that?
11 A. No.
12 Q. Now, the oral test, how long did the person taking it have
13 to review those materials?
14 A. 25 minutes.
15 Q. And are you telling us that some lieutenants or captains
16 told you they take 25 minutes to review written materials
17 before conducting roll call?
18 A. Actually, I think what we have in our job description is
19 that they take less, they had less time than that. I think
20 they said to us -- I think what's documented in our job
21 description, they have a relatively small amount of time often
22 to review material and give feedback to their officers.
23 I believe it was more like ten minutes, I think, or
24 less. I believe it's somewhere in the job description.
25 Q. So your oral test gave police sergeants more time to review
531
Barrett - cross
1 the material than lieutenants actually have, is that right?
2 A. For some lieutenants, yes.
3 Q. Well, for how many lieutenants, Dr. Barrett?
4 A. I don't have the exact number.
5 Q. Does anybody have the exact number?
6 A. I doubt if anybody has the exact number.
7 Q. Why is that?
8 A. I'm not sure why anyone would want the exact number.
9 Q. Isn't that an important part of the paper trail you were
10 creating for this test, Dr. Barrett?
11 A. I think if you read the proposal, it says several
12 documents. It didn't say minute detail on every person we
13 interviewed. I think it's right here in the proposal. It says
14 we would generate several documents, and we have.
15 Q. Well, are you telling us, Dr. Barrett, that there is no
16 paper trail to show us who it was, if anyone, who told you that
17 the oral examination mirrors for some lieutenants the work
18 that's actually conducted in performing roll call?
19 A. No, I don't have any exact documentation beyond what's in
20 the job description.
21 Q. Now, the oral examination, how many documents did the
22 police sergeant have to review who was taking the test?
23 A. I don't recall.
24 Q. Was it the same number that some police lieutenants and
25 captains had told you they have to review before taking --
532
Barrett - cross
1 before giving roll call?
2 A. No.
3 Q. Was it more or less, Dr. Barrett?
4 A. We didn't get an exact number of pages.
5 Q. Well, the scoring system you told us about for the oral
6 test, had that been designed before the test was administered
7 in Chicago?
8 A. No. It was designed -- we say, again, we have a problem
9 probably in terminology. Our format of being objective is
10 designed ahead of time. The actual content of the scoring
11 system is designed and tailor made for Chicago police
12 lieutenants.
13 Q. Well, you say the format was objective. Does that mean
14 that there was a list that the raters had to follow in rating
15 the oral presentations?
16 A. Yes.
17 Q. And were there -- was that the same factors in that list
18 that were used when that oral test was used in other
19 municipalities?
20 A. I don't -- I'm not sure. I would assume it would probably
21 be very similar.
22 Q. Well --
23 A. The content would be different.
24 Q. Well, were there three, what you called scoring dimensions
25 on the oral test?
533
Barrett - cross
1 A. Yes.
2 Q. And was one of those scoring dimensions something you
3 called analysis?
4 A. Yes.
5 Q. And did you do any research in the Chicago Police
6 Department which led you to believe that analysis was an
7 appropriate dimension for measuring the oral communication
8 skills of Chicago police lieutenants?
9 A. Yes.
10 Q. What did you do, Dr. Barrett?
11 A. We had our job analysis interviews, which indicated people
12 do have to analyze material and they have to report on it to
13 other people.
14 Q. Well, how do you measure analysis, Dr. Barrett?
15 A. In terms of our standards.
16 Q. Well, are your standards from the EEOC guidelines?
17 A. No.
18 Q. Are your standards from the APA guidelines?
19 A. If you're talking about exact standards --
20 Q. It's a yes or no question, Dr. Barrett.
21 A. No.
22 Q. Are your standards from the SIOP guidelines?
23 A. I -- when I say no, I'm saying there is nothing specific.
24 They are content validity standards. That would be the only
25 content, if that's your question.
534
Barrett - cross
1 Q. Are the standards that you used for this analysis scale to
2 be found in the SIOP guidelines?
3 A. You won't find the scoring system in any of the guidelines
4 of any type.
5 Q. Find the scoring system in any recognized textbooks?
6 A. I don't think you'll find a scoring system in a textbook.
7 Q. Do you find the scoring system in any refereed articles,
8 articles that have been accepted in any refereed journals?
9 A. The scoring systems are usually not put in refereed
10 journals. So I don't know of any that you would see, find one.
11 Q. So the scoring system is an unvalidated instrument, is that
12 right, Dr. Barrett?
13 A. No, that's wrong.
14 Q. Well, has it been peer reviewed?
15 A. Yes, it has been, as a matter of fact.
16 Q. Who has it been peer reviewed by, Dr. Barrett?
17 A. Been peer reviewed in courtroom after courtroom by other
18 experts.
19 Q. Well, aside from other experts in other courtrooms, has it
20 been peer reviewed by any persons other than paid experts?
21 A. No. We have not sent it out to be peer reviewed. I would
22 think that sort of thing is -- that sort of thing is ever peer
23 reviewed, as far as I know.
24 Q. And is it correct, Dr. Barrett, that you don't have any
25 data which shows that a score on the analysis scoring dimension
535
Barrett - cross
1 of the oral briefing exercise is correlated with future
2 performance as a Chicago police lieutenant?
3 A. If you're using the word "correlated" to mean statistical
4 analyses criterion-related study, no, we have no
5 criterion-related study on that dimension.
6 Q. And if you did have a criterion-related study you'd have a
7 number, is that right?
8 A. We'd have a correlation, that's correct.
9 Q. And you don't have that. You just have your judgment, is
10 that right?
11 A. We have a content validity approach is what we have.
12 Q. Dr. Barrett, you don't have a number, is this right?
13 A. A number, no, we have no correlation.
14 Q. Now, one of the other dimensions on scoring the oral
15 briefing exercise is something that you call organization, is
16 that right?
17 A. That's correct.
18 Q. Did you do any study in the Chicago Police Department to
19 determine that your way of measuring organization is related to
20 performance as a Chicago police lieutenant?
21 A. Only from our job analysis we have that information.
22 Q. Well, how did you rate organization, Dr. Barrett, on the
23 oral test?
24 A. It's how they followed a directive. The directive, as I
25 recall, talked about a way you should organize the roll call
536
Barrett - cross
1 presentation.
2 Q. Well, who -- well, in the roll call -- in the oral
3 examination component, the person taking the test had some
4 written material to look over, is that right?
5 A. That's correct.
6 Q. And then they had to make a speech into a tape recorder, is
7 that right?
8 A. And they had to make -- yes, they had to give a talk into a
9 tape recorder.
10 Q. And then it was -- when the organization was rated, what
11 exactly happened? How was organization rated? Somebody
12 listened to it and say this is well organized?
13 A. No. It was a structured approach.
14 Q. Well, what do you mean, "structured approach"?
15 A. I have to recall, because I don't have it in front of me,
16 the scoring sheet. But I believe it had to do in terms of
17 having introduction of and going through in a structured way
18 and concluding with a summary, I believe.
19 Q. Now, at first, Dr. Barrett, let me show you what's been, I
20 think, received in evidence as Defendant's Exhibit 16. Is that
21 one of the oral exercises for the 1994 lieutenants exams?
22 I'm going to give you a hint, Dr. Barrett, that your
23 attorney has represented that that is what the oral test is.
24 A. Okay. If that's what it's represented as, yes, I agree.
25 Q. Now, in rating the what you call organization, did the
537
Barrett - cross
1 rater consider the order in which the person taking the test
2 mentioned specific facts?
3 A. As I recall, yes.
4 Q. And is there any data that you have that indicates that
5 mentioning facts in a particular order helps Chicago police
6 sergeants and police officers understand better what a
7 lieutenant is saying?
8 A. I think that if you are not organized in your approach,
9 that communication -- if you are disorganized in your approach
10 to giving the information to people, it's more difficult to
11 understand it.
12 Q. Other than what you think, Dr. Barrett, is there any data
13 to support that of which you're aware?
14 A. I don't recall any criterion-related study like that.
15 Q. Dr. Barrett, I'm not asking about criterion-related
16 studies. Is there any data at all other than your belief which
17 indicates that being organized in the manner in which you've
18 defined being organized makes oral communication more effective
19 for a Chicago police lieutenant?
20 A. Part of the exercise is --
21 Q. Dr. Barrett, that's a yes or no question. Yes or no,
22 Dr. Barrett?
23 A. What's the question again?
24 MR. FLAXMAN: Could you read it back?
25 May I have it read back?
538
Barrett - cross
1 (Question read.)
2 MR. FLAXMAN: And she read "doubt," but I meant
3 "data." I might have doubt.
4 BY MR. FLAXMAN:
5 Q. Did you understand the question, Dr. Barrett?
6 A. We have data from our job analysis which indicates that
7 effective communication is important, and part of that is being
8 organized in putting your presentation --
9 THE COURT: So the answer is yes?
10 THE WITNESS: Yes.
11 BY MR. FLAXMAN:
12 Q. What is that data from the job analysis?
13 A. Well, for example, I can still recall some lieutenants
14 saying basically that -- how important communication was to
15 communicate to your subordinates and to the people above them
16 about ongoing events.
17 Q. Well, I understand that, Dr. Barrett. But is there
18 anything on the job analysis that says that giving an
19 introduction first followed by descriptive information is a
20 more effective way of communicating than by giving descriptive
21 information first and introduction second?
22 A. I think you're missing the point that there is an
23 instruction. And following instructions in a paramilitary
24 organization is very important. And the instructions were to
25 follow this procedure when you give a presentation.
539
Barrett - cross
1 So basically it had that component of, all right,
2 here is the way it should be done, here are your instructions,
3 and see if, in fact, you can follow these instructions.
4 Q. Well, so you're telling us now that the oral examination
5 was to measure how well you could carry out instructions,
6 Dr. Barrett?
7 A. Well, that's part of it. In fact, if you disregard the
8 instructions, you would not do well on this exercise.
9 Q. Well, didn't you say on direct examination that the oral
10 examination was intended to measure oral communication skills,
11 Dr. Barrett?
12 A. Yes, that's correct.
13 Q. And it wasn't intended to measure the ability to follow
14 orders, was it?
15 A. Well, it's part of the process of communication if, in
16 fact, you're instructed to do things a certain way. In a
17 paramilitary organization, it's part of the whole process.
18 Q. Are you telling us that if somebody who gave the
19 descriptive information before they give the introduction will
20 be communicating more effectively as a lieutenant than somebody
21 who is a sergeant?
22 A. According to these instructions, that's correct.
23 Q. Is there any data that supports that, Dr. Barrett?
24 A. Well, again, if you're not following instructions in a
25 paramilitary organization, it's not particularly good. It's
540
Barrett - cross
1 certainly our job analysis indicated that.
2 Q. Well, are the instructions for Chicago police lieutenants
3 who conduct roll calls to give an introduction first followed
4 by descriptive information?
5 A. I'm sorry?
6 Q. Is the instruction for Chicago police lieutenants who
7 conduct actual roll calls to give introductory information
8 followed by descriptive information?
9 A. I don't know exactly what different lieutenants might
10 follow. Some might prefer them.
11 Q. Well, that was the instruction for your simulation, wasn't
12 it, Dr. Barrett?
13 A. Yes, it was.
14 Q. And you had no -- you made no effort to make your
15 simulation to be exactly the same as conducting roll call, did
16 you, Dr. Barrett?
17 A. No. We did not make it exactly like a roll call, that's
18 correct.
19 Q. You made it to be a simulation of oral communication
20 skills, is that right?
21 A. That's correct.
22 Q. And you wanted it to measure not just the ability to speak,
23 but the ability to understand what people said to you, is that
24 right?
25 A. I'm not quite following what you are saying.
541
Barrett - cross
1 Q. Well, I'll withdraw the question.
2 Let's talk about the way this was graded. And I want
3 to find an exhibit. Let me show you what's been marked as
4 Plaintiffs' Exhibit 140.
5 MR. FLAXMAN: Now I'll take a minute so we're all on
6 the same page.
7 MR. HOLZHAUER: Your Honor, we have an objection to
8 this exhibit. We might be able to deal with it through a
9 stipulation. But there is writing on the top, particularly the
10 name of a police officer. That certainly doesn't appear in any
11 of these exhibits that are provided during the testing process.
12 This exhibit is not, in effect, an accurate document,
13 correct?
14 (Discussion off the record.)
15 MR. FLAXMAN: Thank you.
16 I think we're going to stipulate to the admission
17 into evidence of this document with the understanding that the
18 handwritten notation in the top right-hand corner which has the
19 name and number -- or the name of Holly Robinson was not there
20 until it was produced in discovery in this case and I added it.
21 MR. HOLZHAUER: That would be fine, Your Honor.
22 THE COURT: Okay. It will be admitted.
23 (Plaintiffs' Exhibit 140 received in evidence.)
24 BY MR. FLAXMAN:
25 Q. Plaintiffs' Exhibit 140, are these, aside from the
542
Barrett - cross
1 handwritten notation, is this the form that was used to score
2 the oral briefing exercise by Barrett & Associates?
3 A. Yes.
4 Q. And there are three forms, is that right?
5 A. Yes.
6 Q. And is that -- are there three forms because there are
7 three raters?
8 A. That's correct.
9 Q. Now, at the top left-hand corner, it starts out
10 "Introduction," and it says "reason and structure." Do you see
11 that?
12 A. Yes.
13 Q. Was that something that the rater was supposed to listen
14 for?
15 A. Yes.
16 Q. And as a matter of fact, the rater was supposed to give
17 points for the candidate mentioning the topics in the
18 prescribed order, is that right?
19 A. Yes.
20 Q. And if the candidate gave the introduction followed by the
21 descriptive information followed by what's called procedural
22 information, they would get full credit for that section, is
23 that right?
24 A. I don't have in front of me the instructions. But I assume
25 it's correct.
543
Barrett - cross
1 Q. Okay. Now, in the instructions, there is something called
2 a transgression, is that right?
3 A. Yes.
4 Q. What's a transgression, Dr. Barrett?
5 A. I think when things are out of order, if I recall.
6 Q. Is there any data that indicates that communicating
7 information out of order is less efficient than communicating
8 in order when that communication is done by Chicago police
9 lieutenants?
10 A. It is when you are instructed to do it in a certain manner
11 on a form.
12 Q. Other than what the instructions are, Dr. Barrett, is there
13 any data that shows that actual police lieutenants in the field
14 do not communicate as effectively if they give you information
15 out of order?
16 A. I don't have any direct evidence of that.
17 Q. Okay. And then there were three people who listened to
18 each tape, is that right?
19 A. Yes.
20 Q. And they each scored one of these checklists?
21 A. Yes.
22 Q. There were three dimensions on the checklist, is that
23 right?
24 A. Yes.
25 Q. And we talked about two of them.
544
Barrett - cross
1 Do you remember -- do these three dimensions come
2 from the EEOC guidelines?
3 A. No.
4 Q. Do they come from the APA guidelines?
5 A. No.
6 Q. Do they come from the SIOP guidelines?
7 A. No.
8 Q. Do they come from any published articles?
9 A. No.
10 Q. They're something that Barrett & Associates produced
11 in-house for its use in administering tests, is that right?
12 A. Yes.
13 Q. And is there any data that Barrett & Associates has which
14 indicates that scoring somebody's oral communication skills in
15 accordance with this system is correlated with successful
16 performance as a Chicago police lieutenant?
17 A. We have no correlation coefficient within the criterion of
18 the related study.
19 Q. Now, were you involved in preparing the scoring protocol
20 for the oral test?
21 A. Dr. Doverspike, I recall, is the one who prepared these.
22 Q. Doctor who?
23 A. Dr. Doverspike, D-o-v-e-r-s-p-i-k-e.
24 Q. Did you review his or her work?
25 A. Yes.
545
Barrett - cross
1 Q. And it's your opinion that that's a reasonable way of
2 measuring oral communication skills?
3 A. Yes.
4 Q. If we listened to an actual oral interview tape, would you
5 be able to explain to us how that interview was rated?
6 A. No.
7 Q. Why is that?
8 A. I haven't been trained as a rater.
9 Q. So did you train raters, Dr. Barrett?
10 A. Yes, we trained the raters.
11 Q. No. Did you personally train raters?
12 A. No, I didn't personally train any rater.
13 Q. Who trained the raters, Dr. Barrett?
14 A. Dr. Doverspike, among others.
15 Q. And you weren't there when he trained the raters, were you?
16 A. No.
17 Q. So you don't know how he trained the raters, do you?
18 A. I didn't actually watch him train the raters.
19 Q. And you don't know what the raters actually did when they
20 rated these tapes, do you?
21 A. Well, I was in the room at times, so at various times
22 observed for short periods of time. But no, I didn't watch
23 each rating team rate each tape.
24 Q. So you don't know whether or not these tapes were rated in
25 accordance with whatever the instructions were that these
546
Barrett - cross
1 raters were given, do you?
2 A. I feel very confident they were following procedure.
3 Q. Other than every confidence, do you have any personal
4 knowledge that they were?
5 A. I did not personally sit in the room and watch them rate
6 the tape, no, I did not.
7 Q. So you -- and you can't -- now, I think on direct
8 examination Judge Gettleman asked you did each part of the test
9 have a disparate impact? And I think you said yes.
10 Did you consider whether or not the disparate impact
11 on the oral component of the test was caused by difference in
12 speech of African-American applicants?
13 A. This is always a concern. And that's the reason we went to
14 a very objective approach where it would be objective scoring
15 form like you see in front of you where you would check off and
16 say, all right, here is what was communicated.
17 And that's the very reason we went to that objective
18 approach instead of a more subjective approach where people
19 might be biased against some sort of ethnic group because of
20 their speech pattern.
21 Q. How could the applicant challenge his or her scoring on the
22 oral test? Could they go and listen to their tape and see how
23 their tape had been scored?
24 A. No.
25 Q. Is it true that there was no way that any applicant could
547
Barrett - cross
1 challenge the way his or her tape had been scored?
2 A. That's correct.
3 Q. So if any mistakes were made in scoring oral tapes, there
4 was nothing that anybody could do about it, is that right?
5 A. As far as I know, you're correct.
6 Q. Let's talk about the so-called in-basket, Dr. Barrett.
7 That consisted of 60 multiple-choice questions, is that right?
8 A. That's correct.
9 Q. And there were 90 minutes to answer those questions?
10 A. Yes.
11 Q. Barrett & Associates had administered that type of test
12 before it was administered to the Chicago police lieutenants,
13 is that right?
14 A. Yes.
15 Q. In any of the times that Barrett & Associates had used that
16 type of in-basket, had there ever been any data collected to
17 correlate performance on that test with subsequent performance
18 on the job?
19 A. No.
20 Q. As a matter of fact, this type of in-basket was something
21 that's been developed by Barrett & Associates in-house, is that
22 right?
23 A. No. I think I testified before that the type of structured
24 in-basket we're referring to goes back to at least 1966,
25 perhaps before that, perhaps late '50s. So that the specific
548
Barrett - cross
1 one you're referring to, of course, was developed by us.
2 Q. I'm referring to the specific one that was administered.
3 That was developed in-house by Barrett & Associates, is that
4 right?
5 A. Yes. This is tailor-made for the City of Chicago.
6 Q. Well, in your job proposal, Plaintiffs' Exhibit 60, did you
7 refer to that in-basket as an improvement over the traditional
8 in-basket that was designed in-house by Barrett & Associates?
9 A. Yes, I did.
10 Q. And you said it was something that improved upon the
11 traditional in-basket?
12 A. Yes, I did.
13 Q. And you'll agree, will you not, that the traditional
14 in-basket has been validated many times?
15 A. Yes.
16 Q. And there are numbers for correlation validation
17 coefficients for the traditional in-basket and subsequent job
18 performance, is that right?
19 A. Yes.
20 Q. And there is no number for the Barrett & Associates'
21 in-basket, is that right?
22 A. We have never performed a criterion-related study with our
23 in-basket.
24 Q. Now, that 90 minutes for 60 questions, did you ever perform
25 a study to see if people will do better on the test if they
549
Barrett - cross
1 have 120 minutes for 60 questions?
2 A. I don't have to. I know the answer.
3 Q. Well, is the answer no, you've never conducted a study?
4 A. No. I'm saying we have not conducted a study. But if you
5 give more time with all of the information to answer the
6 questions is there, certainly the more time you have the more
7 questions you are going to get correct, because all the
8 information -- you have the two and a half hours to work on the
9 problems. If you are given a limited amount of time, you are
10 going to answer more questions correctly.
11 Q. So if in Chicago there was two hours, 120 minutes, two
12 minutes a question, then more people would have done better?
13 Is that what you are telling us?
14 A. I'm saying that when you have an open book type test where
15 the answers are -- the material is there and you can keep
16 working on it, you can eventually get a correct answer for all
17 60 items.
18 Q. So are you also saying that if you had less time to answer
19 60 items, less people would have gotten as many correct?
20 A. In general, yes. Less the amount -- the less amount of
21 time you have, the fewer items they would get correct.
22 Q. Could you tell us what research you did to determine that
23 the 90 minutes was the appropriate time to use for Chicago
24 police sergeants who wanted to be promoted to lieutenant?
25 A. This is based upon our past experience in this in-basket in
550
Barrett - cross
1 different jurisdictions.
2 Q. Well, could you tell us what that past experience consists
3 of, Dr. Barrett?
4 A. Well, you mentioned many different places we have given the
5 in-basket in the past. Those same cities. And each case and
6 each time we've gotten good reliability. And if reliability is
7 low, this indicates to us that people were, in effect,
8 answering at random and guessing.
9 So we have confidence that the time limit was
10 appropriate for a psychometric test and that we have good
11 reliability on the test.
12 Q. Well, Dr. Barrett, you never gave this test to Chicago
13 police sergeants, did you?
14 A. No, I did not.
15 Q. You're not telling us, are you, that there is some job
16 requirement that Chicago police lieutenants be able to answer a
17 certain number of questions on this in-basket test in 90
18 minutes, are you?
19 A. I'm not sure what you mean by "requirement."
20 Q. Well, you're not telling us that in order to do your job as
21 a Chicago police lieutenant, you have to be able to answer a
22 certain number of questions on this in-basket in 90 minutes,
23 are you telling us that?
24 A. No. We didn't -- we didn't say that.
25 Q. And you're not telling us that if you could answer more
551
Barrett - cross
1 questions -- if you need two hours to do this test, you won't
2 be a good lieutenant?
3 A. No. What we are saying, that the more questions you get
4 correct in a stated amount of time, the higher probability you
5 will be a good lieutenant, particularly in the area of
6 administrative skills.
7 Q. And there is no data that you have to support this, is
8 there, Dr. Barrett?
9 A. We have the job analysis data. We have that sort of
10 information which tells us that is correct. We have the past
11 research on similar types of in-baskets, which is
12 criterion-related. As I mentioned before, the studies began in
13 1966 at least, and they continue until the present day.
14 Q. Well, 1966, was that that study by Lopez that you talked
15 about?
16 A. Yes.
17 Q. Have you ever read that study --
18 A. Yes.
19 Q. -- by Lopez, Doctor?
20 Did you read it in preparation for your testimony
21 here today?
22 A. No.
23 Q. When is the last time you read it?
24 A. I really don't know.
25 Q. Excuse me?
552
Barrett - cross
1 A. I don't know.
2 Q. Well, did Lopez use the same kind of exercise that you
3 used, Dr. Barrett?
4 A. He used a structured approach like we did, yes, that was
5 computer scored.
6 Q. What do you mean by "structured approach"?
7 A. There is a dimension going from constructive type responses
8 to very structured type responses. A constructive response
9 type of format would be an essay type test.
10 And as you go toward the other end of a more
11 structured, you have things like multiple choice, picking three
12 alternatives, that type of an approach, which can be basically
13 computer scored because there is a right or wrong answer.
14 Q. Well, let's talk about --
15 THE COURT: Let's not.
16 MR. FLAXMAN: Okay.
17 THE COURT: Let's not.
18 How much longer do you have?
19 MR. FLAXMAN: An hour.
20 THE COURT: All right. Do you want to do it tomorrow
21 morning?
22 MR. HOLZHAUER: Your Honor, I think I'm going to have
23 a substantial amount of redirect, particularly on those general
24 orders. And I understand Dr. Barrett, who doesn't have a hotel
25 room tonight, has to do a dissertation defense tomorrow
553
Barrett - cross
1 afternoon in Akron for one of his students.
2 THE COURT: I'm talking about tomorrow morning. I've
3 got a couple hours for you if you want it.
4 MR. HOLZHAUER: I believe that we can still finish
5 our case in chief on Monday and Tuesday of next week, and
6 Mr. Flaxman doesn't have any witnesses for the remainder of
7 next week anyway. And I believe that -- I think considering
8 the amount of material I have to go through, I'm not at all
9 sure I can get Dr. Barrett done in time to catch his plane and
10 do his dissertation work tomorrow afternoon.
11 THE COURT: Then how long do you have after that?
12 MR. FLAXMAN: We have witnesses Monday and Tuesday of
13 next week.
14 THE COURT: That's it?
15 MR. FLAXMAN: I hope so, yes.
16 MR. HOLZHAUER: Monday and Tuesday of the following
17 week.
18 THE COURT: The week of the 1st.
19 MR. FLAXMAN: That's correct.
20 THE COURT: All right. Then we'll adjourn and come
21 back, let's say, 9:30 on Monday.
22 MR. HOLZHAUER: That's fine.
23 THE COURT: We'll start at 9:30 on Monday.
24 Dr. Barrett, you are under adverse examination, so
25 you are still sequestered. That means you have the distinct
554
Barrett - cross
1 pleasure of not having to talk to anybody about this case until
2 you're through with the cross-examination.
3 MR. HOLZHAUER: Certainly no lawyers for the weekend.
4 THE COURT: No lawyers, no anybody. Don't talk to
5 anybody about the case at all, and I'll see you back here on
6 Monday.
7 (Adjournment at 4:00 o'clock p.m. until 9:30 a.m., Monday,
8 November 24, 1997.)
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
part 4
Main