165
1 IN THE UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
2 EASTERN DIVISION
3
ADAMS, et al., ) Docket No. 94 C 5727
4 )
Plaintiffs, )
5 )
vs. )
6 )
CITY OF CHICAGO, ) Chicago, Illinois
7 ) March 6, 1996
Defendant. ) 9:00 o'clock a.m.
8
9 VOLUME II
TRANSCRIPT OF PROCEEDINGS - Hearing
10 BEFORE THE HONORABLE JOHN A. NORDBERG
11
APPEARANCES:
12
13 For the Plaintiff: LAW OFFICES OF KENNETH N. FLAXMAN, P.C.
BY: MR. KENNETH N. FLAXMAN
14 122 South Michigan Avenue, Suite 1850
Chicago, Illinois 60603
15
16 For the Defendant: CITY OF CHICAGO
BY: MS. DARKA PAPUSHKEWYCH
17 MS. SHONA B. GLINK
MR. JAY KERTEZ
18 30 North LaSalle Street, Room 1020
Chicago, Illinois 60602
19
20 Court Reporter: MR. JOSEPH A. RICKHOFF
219 S. Dearborn Street, Room 1738
21 Chicago, Illinois 60604
22 * * * * * * * * * * * * * * * *
PROCEEDINGS RECORDED BY
23 MECHANICAL STENOGRAPHY
TRANSCRIPT PRODUCED BY COMPUTER
24
25
166
1 THE CLERK: 94 C 5727, Adams vs. City of Chicago.
2 Hearing.
3 THE COURT: Sorry for the delay, but I have already
4 had two emergency matters this morning. So, who knows what
5 lies in store for the rest of the day.
6 But, in any event, I know that it is important that we
7 move ahead on taking the testimony of the experts. And the
8 schedule that we would be following today is -- I was going to
9 say 9:00, but it would be 9:15 to 12:30. We will take one hour
10 for lunch. Part of that I have to use interviewing a
11 prospective law clerk. And, then, we would resume, again, at
12 1:30 and go to 4:30 in the afternoon. Then I have another
13 emergency hearing to undertake. So, if that -- hopefully, that
14 will give enough time to complete the examinations that we have
15 scheduled.
16 MR. FLAXMAN: We might be done before that. We're
17 conferring about what we have today, and it really --
18 THE COURT: Okay. All right. Very good.
19 THE COURT: I forget now exactly -- I know you had
20 completed your adverse examination. Do you have any additional
21 questions that --
22 MR. FLAXMAN: Of Dr. Barrett, no.
23 THE COURT: -- came to mind? All right.
24 So, we have some limited -- call it redirect -- of --
25 MS. PAPUSHKEWYCH: Right.
Barrett - cross
167
1 THE COURT: -- Dr. Barrett.
2 And, then, with the completion of his testimony, you
3 were going to call your expert then; is that right?
4 MR. FLAXMAN: That's right.
5 THE COURT: So, his testimony can be completed.
6 And we have agreed that all experts can sit in on the
7 proceedings.
8 All right. Dr. Barrett, if you will resume your
9 position at the stand, and you remain under oath.
10 DR. GERALD BARRETT, PLAINTIFFS' WITNESS, PREVIOUSLY SWORN
11 THE COURT: And, counsel, you may proceed when ready.
12 MS. GLINK: Shona Glink on behalf of the city.
13 CROSS-EXAMINATION
14 BY MS. GLINK:
15 Q. Dr. Barrett, do you recall yesterday testifying about
16 general cognitive ability tests?
17 A. Yes, I do.
18 Q. What is a general cognitive ability test?
19 A. A general cognitive ability test is one which basically
20 measures general intellectual functioning. And various tests
21 that exemplify this would be the Wechsler, for example,
22 Stanford-Binet, the Ravens -- are examples of general kinds of
23 ability tests. They can be taken by anybody in the population.
24 Q. Is the written job knowledge test that you developed as
25 part of your test battery for the 1993 police sergeant's
Barrett - cross
168
1 examination a general cognitive ability test?
2 A. No.
3 Q. In what ways was it not?
4 A. I can give an example. We could have a hundred Ph.D.'s
5 take the job knowledge test. Those hundred Ph.D.'s may be all
6 very proficient in their field. They would do very poorly on
7 the job knowledge test. They have no specific knowledge of the
8 Illinois Criminal Code, nor do they have specific knowledge of
9 the general orders or specific orders in Chicago Police
10 Department.
11 Conversely, a general cognitive ability test, if you
12 were to take one of those tests, they would have scored very
13 high on that test.
14 Q. How would you then describe your written job knowledge test
15 that you developed for the Chicago police sergeant's
16 examination?
17 MR. FLAXMAN: Object to the form of the question.
18 THE COURT: Objection overruled. He may answer.
19 BY THE WITNESS:
20 A. It's a specific job knowledge test tailored specifically to
21 the knowledge required of a Chicago police sergeant.
22 BY MS. GLINK:
23 Q. Was any portion of your examination a general cognitive
24 ability test?
25 A. No.
Barrett - cross
169
1 Q. Have you ever given a cognitive ability test as part of a
2 battery for a police sergeant's examination?
3 A. No.
4 Q. And why not?
5 MR. FLAXMAN: Objection. It's not relevant.
6 THE COURT: Well, this being the bench as opposed to a
7 jury, I am not going to consider anything that is not relevant,
8 but it is difficult at this stage to know exactly what may or
9 may not turn out to be relevant. So, I will overrule the
10 objection and permit you to proceed subject to a subsequent
11 motion to strike, if that is appropriate at some point.
12 You may proceed.
13 BY THE WITNESS:
14 A. First, a general cognitive ability test is apt to have
15 adverse impact. And, second, it is not pertinent to --
16 THE COURT: Will you explain what "apt to have adverse
17 impact" --
18 THE WITNESS: In other words, a general --
19 THE COURT: I do not want to interrupt your testimony,
20 but just be sure that we cover that.
21 MS. GLINK: I will follow up with that, Judge.
22 THE COURT: You may proceed.
23 BY THE WITNESS:
24 A. And, second, it's my belief in my analysis it's more
25 important to tap the specific knowledge required to perform a
Barrett - cross
170
1 job as a sergeant.
2 BY MS. GLINK:
3 Q. And when you said "adverse impact," what do you mean by
4 adverse impact?
5 A. I mean, differences -- mean differences between blacks and
6 whites who take the test.
7 Q. And what is your concern about adverse impact with
8 cognitive ability tests?
9 A. Well, there is a lot of research which indicates that it
10 might have or even maximize potential adverse impact between
11 groups.
12 Q. You also stated yesterday in your testimony that the
13 reliability coefficient on the three-part test that you
14 developed was, I believe you said, .87; is that correct?
15 A. Yes.
16 Q. And, in your opinion, based on your experience as a test
17 developer, is a .87 reliability coefficient a high coefficient?
18 A. Yes.
19 Q. You indicated yesterday, as well, that your initial
20 proposal to the City of Chicago included a recommendation that
21 the examination consist of three parts: A written job
22 knowledge test, an in-basket simulation and oral briefing
23 exercise; is that correct?
24 A. Yes.
25 Q. You further testified that you conducted a job analysis of
Barrett - cross
171
1 the job of Chicago Police Department -- police sergeant after
2 you submitted your proposal to the City of Chicago with that
3 recommendation; is that correct?
4 A. Yes.
5 Q. What was this initial recommendation based upon?
6 A. It was based upon over 20 years of experience in developing
7 tests for safety forces.
8 Q. And how were you able to determine based on this experience
9 that you should propose -- that your proposal should consist of
10 three -- a three-test battery?
11 A. Based on our past analyses -- job analyses of the job of
12 sergeants in other jurisdictions, this type of three-component
13 test battery covered the major work behaviors of a police
14 sergeant.
15 Q. At the time that you submitted your proposal, had you
16 developed any of the specific questions that would be used in
17 the testing process for the sergeant's examination?
18 A. No.
19 Q. Isn't it also true that your proposal states that the
20 development of the three test components would be based on the
21 results of your job analysis?
22 A. Yes.
23 Q. Did the job analysis that you conducted support your
24 initial proposal that the three-part test consist of a written
25 job knowledge portion, an in-basket simulation component and an
Barrett - cross
172
1 oral briefing exercise?
2 A. Yes.
3 Q. If the job analysis had not supported your initial
4 recommendation, would you have altered the test components to
5 conform with the results of your job analysis?
6 A. Yes.
7 Q. Do you know when the sergeant's examination announcement
8 was sent out to applicants in this instance?
9 A. I believe it was September or October of '93.
10 Q. Do you have an understanding as to whether the job
11 announcement contained a statement as to what components would
12 be included in the sergeant's examination?
13 A. Yes.
14 Q. And do you know what components that announcement stated?
15 A. It contained the components of job knowledge, the in-basket
16 and the oral briefing exercise.
17 Q. At the time that the examination announcement was sent out
18 to applicants, had you completed your job analysis for the job
19 of police sergeant?
20 A. Yes.
21 Q. And had you confirmed at that time that the job analysis
22 supported the three components that were listed on the
23 examination announcement?
24 A. Yes.
25 Q. Dr. Barrett, you also testified yesterday that the written
Barrett - cross
173
1 job knowledge test was administered on December 4th, and that
2 the in-baskets -- of 1993, excuse me, and that the in-basket
3 simulation was administered on December 5th of 1993; is that
4 correct?
5 A. Yes.
6 Q. And you further testified that you scored these
7 examinations before administering the oral briefing exercise;
8 is that also correct?
9 A. Yes.
10 Q. Dr. Barrett, did all candidates who took the written job
11 knowledge component of the examination and the in-basket
12 simulation component of the examination take the oral briefing
13 exercise?
14 A. No.
15 Q. How many individuals, if you know, took the first two
16 components of the examination?
17 A. 4,700, I believe.
18 Q. And how many individuals took the oral briefing exercise,
19 if you know?
20 A. As I recall, there were 2,044 who could have gone on, and I
21 think 1937 actually took the oral component.
22 Q. But 2,044 were invited, as far as you know, to take the
23 oral examination?
24 A. Yes.
25 Q. How did you determine who would be allowed to take the oral
Barrett - cross
174
1 briefing exercise?
2 A. It was really based upon three separate analyses we
3 conducted. The oral component is the most expensive component,
4 most time-consuming component. It's also the component where
5 there is the most problem with potential revealing the contents
6 of the oral briefing. So, in our proposal, we first, before we
7 submitted our proposal, we first did a simulation.
8 Q. Let me stop you there.
9 Can you explain to the Court what a simulation is?
10 A. Basically, it was a computer simulation to determine
11 hypothetically, if we obtained certain scores, what would
12 occur, so that we would pass through on to the oral portion
13 everybody who had any probability, based upon their test scores
14 from the first two components, of being selected as a police
15 sergeant based on their final score.
16 Q. How did you determine who had a probability of being
17 selected or had a probability of ultimately being selected for
18 promotion to sergeant based on the results of your simulation?
19 A. We wanted to pass through the person who had the -- any
20 score below the lowest score we passed through had no
21 probability, based on the fact they received a hundred percent
22 on the oral. Hypothetically, we said, "All right, let's assume
23 somebody would receive a score of 100 percent on the oral --
24 perfect score."
25 We would add that perfect score on to their scores
Barrett - cross
175
1 they obtained for the in-basket and the job knowledge test that
2 would give them a final score. And that final score would not
3 allow them, because it would be too low, to actually be
4 promoted.
5 THE COURT: Have we had any testimony about the
6 decision on having equal weight given to all three parts?
7 MS. GLINK: Not yet, your Honor. We intend to cover
8 that in our direct examination.
9 THE COURT: Because that seems to be the presumption
10 behind that last answer.
11 MS. GLINK: Would you like me to bring some of that
12 testimony out --
13 THE COURT: Well, I think --
14 MS. GLINK: -- for your understanding?
15 THE COURT: -- at some point, I think it needs -- the
16 timing needs to be brought out.
17 BY MS. GLINK:
18 Q. Well, Dr. Barrett, let me ask you, how was --
19 THE COURT: Mr. Flaxman may want to bring it out, too,
20 but why do we not --
21 MS. GLINK: Sure.
22 THE COURT: I just need to see what the timing is on
23 this.
24 MS. GLINK: Sure.
25 BY MS. GLINK:
Barrett - cross
176
1 Q. Dr. Barrett, at some point, did you determine how the
2 examination should be weighted?
3 A. Yes, we did.
4 Q. And at what point in your examination -- in the development
5 process did you determine how the three parts of your
6 examination should be weighted?
7 A. We made that determination after the job analysis was
8 completed and before the posting notice was sent out. It would
9 be sometime in September, as I recall.
10 Q. And that was before any component part of the examination
11 was administered; isn't that correct?
12 THE COURT: That's September of '93 we are dealing
13 with?
14 MS. GLINK: September of 1993.
15 BY MS. GLINK:
16 Q. Is that before any part of the examination had been
17 administered?
18 A. Yes.
19 Q. And how did you reach the decision to weight the three
20 parts of your examination equally?
21 A. There are a number of factors I considered. The first
22 factor is that there is a large body of professional literature
23 and scientific literature, which deals with the topic of
24 weighting of tests. And that general literature which I
25 reviewed indicated that the equal weighting model usually gave
Barrett - cross
177
1 you the best prediction. This is even if you might use
2 regression weights. Still, that's from a sample. Still, the
3 best prediction for the population is often found to be equal
4 weighting.
5 So, you have a strong presumption that that, from a
6 scientific point of view, is the best process.
7 Q. Let me stop you there for a minute, Dr. Barrett.
8 Did you submit an affidavit in this case?
9 A. Yes.
10 Q. And is that the affidavit that Mr. Flaxman referred to
11 yesterday --
12 A. I hope so.
13 Q. -- as the affidavit that was attached to our memorandum in
14 opposition to defendant's motion for summary judgment -- motion
15 for preliminary injunction, excuse me?
16 Let me show you --
17 MS. GLINK: Do you know what it was marked?
18 MR. FLAXMAN: Defendant's Exhibit E.
19 BY MS. GLINK:
20 Q. It was our Defendant's Exhibit E.
21 MS. GLINK: May I approach the bench?
22 THE COURT: Yes, you may.
23 MS. GLINK: Do you have that with you, your Honor?
24 THE COURT: Yes, I do.
25 BY MS. GLINK:
Barrett - cross
178
1 Q. I'm going to give you a copy of this.
2 (Document tendered.)
3 BY MS. GLINK:
4 Q. Dr. Barrett, is that the affidavit that you submitted in
5 this case?
6 A. Yes, it is.
7 Q. And is your scientific literature review set forth anywhere
8 in this affidavit?
9 A. Yes, it is.
10 Q. And can you tell me where in this affidavit you set forth
11 the articles that you reviewed on equal weighting of test
12 components?
13 A. It's in Appendix B, Page 26.
14 Q. And this is the literature that you're referring to -- that
15 you have just referred to in support -- as one basis for
16 supporting your decision to weight the three tests of the
17 components equally; is that correct?
18 A. Yes.
19 Q. Okay.
20 Other than the scientific literature that you
21 reviewed, what other bases, if any, did you have for
22 determining that the examination should be weighted equally?
23 A. The second basis was the fact I reviewed the job
24 description and the associated knowledge, skills and
25 abilities. And that also supported the equal weighting
Barrett - cross
179
1 approach.
2 Q. Can you explain that a little bit to me, Doctor?
3 A. Well, it's just more or less a judgmental approach by
4 looking at the various components, the major work behaviors,
5 the tasks performed. It was a judgmental approach. It was
6 nothing formal.
7 A third more formal approach, I had two of my job
8 analysts, who had completed the major part of the job analysis
9 of the police sergeant, do a more formal approach linking the
10 various work behaviors with the various test components. And,
11 again, we found support for the test weighting process. That's
12 in Appendix D.
13 Q. And that's also included in your affidavit?
14 A. Appendix D, Page 57, it starts.
15 As a last part of the process, we also did an abstract
16 computer simulation using a very theoretical approach, a very
17 abstract approach, which we had developed for this project,
18 actually.
19 Q. And was this the first time that you had ever used this
20 sort of computer simulation?
21 A. Yes, it is, for this project -- type of project; and, also,
22 gave support to the equal weighting approach.
23 Q. Let's go back to this judgmental or more formal weighting
24 rationale study that was conducted by the analysts that you
25 mentioned. Are these individuals who work for Barrett &
Barrett - cross
180
1 Associates?
2 A. Yes.
3 Q. And do you recall the names of the individuals who
4 conducted this analysis?
5 A. Ron Carabini, Sophia Siambekos, S-i-a-m-b-e-k-o-s.
6 Q. And are these two individuals -- do these two individuals
7 have specialized training in the field of industrial psychology
8 and psychometrics?
9 A. Yes.
10 Q. And are you familiar with their work?
11 A. Yes.
12 Q. And have you reviewed the work that's contained in this
13 affidavit?
14 A. Yes.
15 Q. And do you agree with the conclusions that are reached by
16 these two associates who work with you?
17 A. Yes.
18 Q. In your professional opinion, Dr. Barrett, is the decision
19 to weight the three test components of the police examination
20 valid?
21 A. Yes.
22 Q. And does the weighting scheme that you adopted in this
23 examination conform with the uniform guidelines on employee
24 selection procedure?
25 A. Yes.
Barrett - cross
181
1 Q. Now, this decision to weight the three parts of your
2 examination equally, that was made before -- was that made
3 before you conducted this computer simulation to determine who
4 should go on to the oral portion of the examination?
5 A. Well, it was made -- that -- the first one, of course,
6 before the proposal was submitted, was not based upon that, any
7 sort of weighting scheme, at that point in time, except we made
8 it broad enough. So, it could be equally or some variation
9 from that.
10 Q. Okay.
11 Could you explain that to me?
12 A. Well, in other words, you can start off by saying it's
13 equal. This is before a proposal was done. Then take a
14 ten-percent variation, what would occur. Say, we weighted the
15 oral higher -- ten percent higher, what would occur.
16 So, we built in a safety factor. But right before,
17 after we received the results, of course, and the weighting had
18 been done, then we could be more precise. And we verified the
19 fact that the 2,044 would, again, have -- not caused a problem
20 for us. No one who received a score below that would have any
21 probability of being promoted.
22 Q. Let me stop you there and see if I understand what you are
23 saying.
24 You conducted a computer simulation as part of your
25 initial proposal, and this simulation didn't have any set
Barrett - cross
182
1 weights, it had a variation of weights equal weighting plus ten
2 percent or minus ten percent --
3 A. Right.
4 Q. -- is that correct?
5 A. Yes.
6 Q. And when you actually received the results from your
7 written multiple choice and your in-basket simulation, you then
8 conducted a second computer simulation?
9 A. Yes, based on the actual data now. We had really the
10 actual information.
11 Q. And at the point that you conducted this second simulation,
12 you had, at this point, determined that equal weighting was the
13 most valid weighting scheme, in your opinion?
14 A. Yes, the posted notice had come out, and, specifically, it
15 said it's going to be equal weighted.
16 Q. And, so, this second simulation, you used the actual
17 results and you actually weighted the first two components
18 equally with the third one in determining who should -- who
19 would be able to move forward to the oral?
20 A. Yes.
21 Q. Did you conduct any other -- you mentioned a third computer
22 simulation. Did you conduct any other simulations with regard
23 to the determination of who should go on to the oral
24 examination?
25 A. Not before we obtained the actual results; but, the third,
Barrett - cross
183
1 we used a confirmation after we obtained the results from the
2 actual oral. And we saw, in fact, that we were correct. No
3 one with a low score --
4 Q. No one -- excuse me?
5 A. No one with a low score on the written test -- and we add
6 on the maximum you could actually receive in the oral -- would
7 have any probability of being selected as a police sergeant.
8 Q. What was your basis in terms of your analysis for the
9 probability of success or being selected? Did you have a
10 particular number in mind when you were conducting that
11 analysis?
12 A. As I recall, it was 500, and I -- as the number we had in
13 mind.
14 Q. Okay.
15 So, your analysis showed that -- am I correct that
16 your analysis showed that nobody who went on to take any --
17 everybody who went on to take the oral briefing exercise had
18 the potential of placing in the top -- at least in the top 500
19 of test takers once all three test components were given?
20 A. Well, we had actually a fudge factor in there. Some of
21 those people went on to the oral and probably had an extremely
22 low probability or no probability, we found out later, of being
23 promoted. I think it came down to 700 something.
24 So, we had an error factor in there that everybody
25 that went through -- I won't say had a probability of being
Barrett - cross
184
1 promoted, but the probability is almost 0 for some people, I
2 would say. But we wanted to be sure everybody had an
3 opportunity who had anything feasible because you can't be that
4 exact.
5 THE COURT: I am not following the rationale for this
6 at all.
7 MS. GLINK: Let me see if I can try and clarify.
8 THE COURT: And if you figure that maximum score on
9 the first two and a zero on the last score, if that is a base
10 that you are referring to with respect to the estimated number
11 that would be appointed from this list, I think I can
12 understand it.
13 But I am not quite sure now what he is saying. So,
14 perhaps --
15 MS. GLINK: I believe, your Honor, it's actually the
16 reverse.
17 BY MS. GLINK:
18 Q. Isn't that correct, Dr. Barrett? It's if you were to get a
19 maximum score on the oral?
20 A. Exactly.
21 Q. Could you have placed -- is this correct, Dr. Barrett: If
22 you were to have received a maximum score on the oral, you
23 allowed everybody who, if they had gotten a hundred on the
24 oral, would have had an opportunity of placing in the top -- or
25 had a potential of placing in the top 500 of test scores
Barrett - cross
185
1 overall?
2 A. That's correct.
3 Q. So that everybody who went on to the oral, you allowed to
4 go forward based on the assumption that, if they received a
5 perfect score on the oral component, given their performance on
6 the first two parts of the examination --
7 THE COURT: But assuming that others get 0, I take it,
8 in a sense.
9 MS. GLINK: I believe, your Honor --
10 BY MS. GLINK:
11 If this is correct, Dr. Barrett, that of anybody who
12 did not take -- everybody who took -- everybody who was not
13 allowed to take the oral, everybody who was not invited --
14 THE COURT: Right.
15 BY MS. GLINK:
16 Q. -- to take the oral, did not receive a high enough score on
17 the first two parts of the examination, even if they had scored
18 perfectly?
19 THE COURT: You assumed that they would get a perfect
20 one --
21 MS. GLINK: Right.
22 THE COURT: -- for purposes of the analysis?
23 MS. GLINK: Right.
24 So that their scores on the first two were not high
25 enough, that, if you were to add a hundred percent of the -- if
Barrett - cross
186
1 you were to give them a hundred percent on the oral, they still
2 would not have attained a high enough score to place them high
3 enough on the list to have a potential of being promoted.
4 THE COURT: But why do you not have to assume that
5 everybody is going to get a hundred for purposes of --
6 MS. GLINK: We are assuming that everybody will get a
7 hundred on the oral -- the computer simulation, if I'm correct,
8 Dr. Barrett --
9 THE COURT: So, there really is not a factor.
10 MS. GLINK: Everybody is --
11 THE COURT: The third score is not really a factor for
12 purposes of the analysis. You are only looking at the first
13 two, anyway.
14 MS. GLINK: For determining who goes on to the oral --
15 THE COURT: Right.
16 MS. GLINK: -- everybody was assumed to get a hundred
17 on the oral. And only those people whose combination of the
18 three component test scores was high enough to place them
19 ultimately down the road given that everybody gets a hundred on
20 the oral were allowed to go -- to move forward to take the oral
21 examination. Because anybody who received a hundred on the
22 oral examination whose scores were too low on the first two
23 components that they would not have had a potential of being
24 promoted were not allowed to go forward.
25 THE COURT: Yes, the only purpose -- the question I
Barrett - cross
187
1 have in analyzing it, it seems to me that it does not make --
2 you do not even use the third score of a hundred for your
3 analysis because you give it to everybody. So, there is no
4 differentiation there at all. You are only looking to what the
5 test scores show --
6 MS. GLINK: But the way to determine --
7 THE COURT: And you want to be sure that -- so, you
8 are grading it depending upon what percentage of the total
9 score, so that you would determine what ranking they would be
10 able to achieve given that everybody is going to do perfect on
11 the third. And I am not sure that that -- what the reason for
12 the validity of that is.
13 THE WITNESS: Your Honor, may I try and explain it?
14 THE COURT: Yes.
15 THE WITNESS: It's not everyone, of course. You are
16 absolutely correct. If you add a constant to all the test
17 scores, a hundred percent to everyone -- the top 2,000, doesn't
18 change anything. The real concern is adding a hundred percent
19 to the lowest score and seeing where that would put you. If
20 you add a hundred percent to the 2,000th person, how high will
21 that put them? And it won't put them very high because the
22 other people have very high scores on the first two
23 components.
24 So, you don't assume that the top ranking people on
25 the job knowledge test and the in-basket are -- will receive a
Barrett - cross
188
1 hundred percent because your analysis is correct. That would
2 make no sense.
3 So, we were really concerned that everybody -- the
4 lowest scores was where we add the hundred percent, not the top
5 scores.
6 THE COURT: Now, if I could ask why 500 was used? Was
7 there some determination made in advance that they were only
8 going to appoint 500 from this test or --
9 MS. GLINK: Based on our past experience, there was a
10 --
11 THE COURT: Well, I mean --
12 BY MS. GLINK:
13 Q. Did you have any understanding --
14 THE COURT: Wait a second.
15 BY MS. GLINK:
16 Q. -- of where that 500 number came?
17 A. Yes, that was the maximum number I was told which would
18 ever be promoted from this list. In fact, we now know that
19 it's going to be less than 500, but that was a maximum number
20 that we used.
21 Q. And who told you this?
22 A. It came from someone in the law department or police
23 department. I'm not sure which area it came from.
24 Q. So, at the time you were conducting this analysis at the
25 proposal stage, did you know -- did you have an understanding
Barrett - cross
189
1 at that stage as to how many the City of Chicago intended to
2 promote as a maximum number off of this list?
3 A. That was our understanding, it would be 500 is a maximum
4 number.
5 Q. So, your analysis was based on that?
6 A. Yes.
7 Q. Okay.
8 MS. GLINK: Has this been sufficiently clarified?
9 THE COURT: I think so.
10 I hesitate to interrupt, but I just was not following
11 -- or I want to be sure that I had the right view of the
12 analysis.
13 So, proceed.
14 BY MS. GLINK:
15 Q. Was the determination of who would advance to the oral
16 based on any measure of an applicant's minimum qualifications
17 for the job?
18 A. No, it was not.
19 Q. Dr. Barrett, when you conduct a job analysis, would you
20 agree that it's important to ask a sergeant about the content
21 of his job, meaning what he does on a daily basis?
22 A. Yes.
23 Q. And why is this important?
24 A. This is how we basically determine what the major work
25 behaviors are for a job description, and how -- what knowledge
Barrett - cross
190
1 is important.
2 And, also, it's critical for us to develop the job
3 knowledge test, the in-basket simulation and the oral briefing
4 exercise, because all based upon the content of the police
5 sergeant's job. That's how we develop these tests.
6 Q. Do you also believe that it's important to ask supervisors
7 of lieutenants what -- I mean, supervisors of sergeants, like
8 lieutenants and captains, what a sergeant does on a daily
9 basis?
10 A. Yes.
11 Q. And why is that important?
12 A. Well, this tells you, again, a different perspective,
13 because they review the sergeant's work and they have a good
14 concept of what a sergeant does.
15 And, also, it tells you what we might call critical
16 incidents which occur, which differentiate people. In other
17 words, a lieutenant might say, "Well, we have a problem with
18 some sergeants because, when they review reports, they don't
19 have" -- "are not clear about assuring patrol officers put down
20 all the elements of a crime," for example.
21 So, they tell you things which causes problems for
22 them as lieutenants.
23 Q. And once you've identified these critical incidents, how do
24 they play a factor in the development of a test process?
25 A. Well, it helps you pinpoint the areas of knowledge which
Barrett - cross
191
1 are important, the areas of administrative decision making and
2 the type of oral communication, which are important.
3 Q. And did you do this? Ask sergeants, lieutenants and
4 captains about the job of sergeant during your job analysis
5 process?
6 A. Yes.
7 Q. I'm going to show you what's been marked as Defendant's
8 Exhibit D, which was attached to our brief in opposition to
9 this preliminary injunction hearing.
10 (Document tendered.)
11 BY MS. GLINK:
12 Q. Can you tell me what this is, Dr. Barrett?
13 A. This is a report on the development of the administration
14 of the promotion process for the job of Chicago police
15 sergeant.
16 Q. And is this the technical report that you prepared for the
17 City of Chicago, based on your development of that examination?
18 A. Yes.
19 MS. GLINK: Can you give me a second, your Honor?
20 THE COURT: Okay.
21 (Brief pause.)
22 THE COURT: While she is looking, just to clarify,
23 this is a report based upon the December '93 examination that
24 was undertaken?
25 THE WITNESS: Yes, and the oral, also.
Barrett - cross
192
1 THE COURT: And this is prepared roughly a year later,
2 right?
3 THE WITNESS: Well, the oral was not given until, I
4 believe, March of 1994. So, it would be prepared -- well, it
5 was partially prepared, of course, but it was finalized in
6 December 13th, 1994.
7 THE COURT: Right.
8 BY MS. GLINK:
9 Q. Does this report contain the master job description that
10 you prepared at the end of your job analysis process?
11 A. Yes.
12 Q. Okay.
13 And let me direct you to Appendix C at Page -- at the
14 end of the report. It's starting at C-1.
15 A. Yes.
16 Q. Is that the master job description that you prepared for --
17 at the end of your job analysis process?
18 A. Yes.
19 Q. And does this master job description also include a listing
20 of those knowledge, skills and abilities that you identified
21 during your job analysis process as important to the job of
22 sergeant?
23 A. Yes.
24 Q. And does it also include a listing of all the individuals
25 that you interviewed during your job analysis process?
Barrett - cross
193
1 A. Yes.
2 Q. Okay.
3 Do you also recall yesterday testifying that you
4 linked all of the items on your written job knowledge test to
5 the major work behaviors and tasks identified in your master
6 job description?
7 A. Yes.
8 Q. And isn't that, when you said that you were referring to
9 this master job description, Appendix C; is that correct?
10 A. Yes.
11 Q. Okay.
12 Did you also link the items on your written job
13 knowledge test to the knowledges identified in your master job
14 description?
15 A. Yes.
16 Q. And those knowledges are set forth at Page C-18 through --
17 C-18 and C-19; is that correct?
18 A. Yes.
19 Q. I'm going to direct you now to Appendix I, as contained in
20 your technical report.
21 A. Yes.
22 Q. Is that the linkage chart that you were referring to
23 yesterday when Mr. Flaxman was asking you questions about what
24 tasks were being tapped by your written job knowledge test?
25 A. Yes.
Barrett - cross
194
1 Q. Can you describe for me and for the Court what this chart
2 -- what information is set forth in this chart?
3 A. This was --
4 THE COURT: Can I have the identification of where it
5 is located, again.
6 MS. GLINK: It's Appendix I of the technical report.
7 THE COURT: Okay.
8 MS. GLINK: That's Exhibit D.
9 THE COURT: That's Exhibit D. So, it's Appendix I to
10 Exhibit D.
11 Is there -- is it -- do we have a page number on
12 this?
13 MS. GLINK: It's just I-2, beginning at I-2. There's
14 not a --
15 THE COURT: Oh, okay. All right. I see now how it is
16 -- some of it is on the side here.
17 All right. I have it. Go ahead.
18 BY MS. GLINK:
19 Q. Can you briefly describe for me what this -- what
20 information is contained in this chart?
21 A. The left-hand column is the test item number from the job
22 knowledge test, and you'll see it contains 150 separate items.
23 The right-hand column is simply our linkage to the
24 work behavior in the master job description indicating that
25 this is a link between the knowledge tapped by that test item
Barrett - cross
195
1 and the work behavior and knowledge acquired there.
2 This is not required by a procedure. We just wanted
3 to illustrate the fact that each item is relevant, it's not an
4 irrelevant item, it does relate to the actual work behavior of
5 a police sergeant.
6 Q. Okay.
7 And in the column on the right-hand side, when it says
8 Roman numeral I-A-4, that is a reference back to Appendix C, is
9 it not, which lists the actual tasks and work behaviors that
10 you identified?
11 A. Yes, it is.
12 Q. So, for instance, Item No. 1, which is tapping section
13 I-A-4 of your master job description, would be linked back to
14 1, which is: "Monitors and supervises the activities of police
15 officers; A, monitors radio communication, communicates with
16 police officers by radio and telephone," and, then, Section 4
17 of that, is that correct, of that section?
18 A. That's correct.
19 Q. And can you read for me what Section I-A-4 -- what tasks
20 are listed in Section I-A-4?
21 A. I-A-4 is: "Assist officers, if needed, by answering
22 questions via radio (e.g., how to interview reluctant,
23 disoriented, injured, mentally ill and minor witnesses or
24 witnesses requiring translators; how to appropriately
25 categorize incidents for reporting and notification purposes;
Barrett - cross
196
1 how to identify whether or not elements of a particular
2 incident exist to indicate probable cause, justification for an
3 arrest or that charge is appropriate, how to identify which
4 report should be used for a particular situation in order to
5 avoid incomplete report packages; how to process arrestees; how
6 to inventory evidence; how to deal with situations involving
7 tender-aged children; refer to departmental guidelines when
8 necessary.
9 Q. So, Section I-A-4 includes all of those tasks that you've
10 just read?
11 A. Yes.
12 Q. You testified yesterday that you wouldn't be surprised if
13 43 of your 150 questions on your written job knowledge test
14 tested for the tasks set forth in Section I-A-4; is that
15 correct?
16 A. That's correct.
17 Q. You also said that you wouldn't be surprised if 56 of the
18 150 questions on your written job knowledge test tested for
19 tasks set forth in Section I-A; and, that is, in broad terms,
20 "Monitors radio communications, communicates with police
21 officers by radio or telephone."
22 And that includes three subparts -- or four subparts,
23 I apologize, underneath that subheading; isn't that correct?
24 A. That's correct.
25 Q. And one of those subheadings is the one that you have just
Barrett - cross
197
1 read into the record?
2 A. That's correct.
3 Q. And you further said that you wouldn't be surprised if 29
4 of the items specifically tap Section I-A-3 of the master job
5 description. Can you just briefly describe the kinds of tasks
6 that are set out in Section I-A-3?
7 A. Listens to radio communications in order to monitor
8 officer's behavior and to oversee dispatcher's assignments of
9 cars and jobs.
10 And below that are a number of specific tasks going A
11 through D.
12 Q. Okay.
13 Isn't it a fact, Dr. Barrett, that many of the
14 questions on your 150-item test were linked to more than one
15 task on your master job description?
16 A. Yes.
17 Q. And how is it possible for an item or one question on an
18 examination to be linked to more than one major work behavior
19 of a sergeant?
20 A. Because the same knowledge underlies that behavior.
21 Q. Can you give me an example?
22 A. Well, for example, like many of the items, as I recall,
23 have to do with the elements of a crime from the Illinois Code,
24 and that you will find, for example, I think, Item 90, for
25 example -- let's see, Item 90.
Barrett - cross
198
1 Q. And that's at I-5 of Appendix I -- I mean, I-6, excuse me,
2 of Appendix --
3 A. Yes.
4 And, for example, we have under that, it was rated as
5 "1-A and; 2." And, so, clearly we have two major work
6 behaviors there, and that sort of knowledge of the elements of
7 a crime underlie both types of behavior.
8 Q. What is Section 2 -- what tasks are set forth in Section 2
9 of your master job description?
10 A. "Reviews and forwards forms and reports submitted by police
11 officers."
12 Q. And there are two major subsections in that, an A and a B,
13 as well as Subsections 1 through 4 under A; is that correct?
14 A. Yes.
15 Q. So, many tasks are set forth in Section 2?
16 A. That's correct.
17 Q. Dr. Barrett, are there certain tasks and major work
18 behaviors that are not tapped by this 150-item job knowledge
19 test that you developed?
20 A. Yes.
21 Q. In addition to the written job knowledge test for police
22 sergeant, your test included an in-basket simulation test, as
23 well as an oral briefing exercise; isn't that correct?
24 A. Yes.
25 Q. Were these three test components designed to tap for
Barrett - cross
199
1 different parts of the job of sergeant?
2 A. Yes.
3 Q. And are some of the tasks and major work behaviors not
4 covered by your written job knowledge test tapped by the other
5 two components of your examination?
6 A. Yes.
7 Q. The in-basket and the oral briefing exercise?
8 A. Yes.
9 Q. What major work behaviors and knowledge skills and
10 abilities are tapped by the in-basket simulation?
11 A. Basically, the administrative decision-making aspect is
12 tapped, which includes analysis of review of reports, that sort
13 of type of decision making.
14 The oral briefing, of course, taps the oral
15 communication aspect of the job.
16 Q. Did you also create linkage charts similar to Appendix I
17 for the written job knowledge test for the in-basket simulation
18 test?
19 A. Yes.
20 Q. And let me direct you to Appendix L of your report.
21 MS. GLINK: Have you located that, your Honor?
22 THE COURT: Yes, I have it here.
23 BY MS. GLINK:
24 Q. And is Appendix L the linkage tables for the in-basket
25 simulation that you developed?
Barrett - cross
200
1 A. Yes.
2 Q. Can you briefly describe for me what information is
3 contained on Page L-2 of your in-basket simulation linkage
4 chart?
5 A. The L-2 table -- is the table you're referring to? Table
6 L-2?
7 Q. Table L-1 on Page L-2, excuse me.
8 A. Yes.
9 Table L-1, it has the dimensions in the left-hand
10 column "Analysis," "Administrative Verification."
11 Q. And what do you mean by "dimensions"?
12 A. That's the dimensions on -- in that we were tapping
13 underneath the in-basket in terms of the broad category of
14 administrative decision making. And each of these three
15 dimensions -- analysis, administrative verification,
16 applications of rules and procedures -- then goes across and
17 links it to the abilities, the rationale, and the related major
18 work behaviors and the in-basket modules themselves.
19 Q. So, Column No. 2, which is labeled, "Ability," what kind of
20 information is contained in that column?
21 A. Just describes what the police sergeant has to do. For
22 example, we see that, to gather relevant information, compare
23 information from various sources, identify critical
24 information, so forth, as being under -- defining that analysis
25 section.
Barrett - cross
201
1 Q. Okay.
2 And is that information you obtained through your job
3 analysis process?
4 A. Yes.
5 Q. And the second -- the third column, which is labeled,
6 "Rationale," can you explain what information is contained in
7 that column?
8 A. Here we describe what actually a sergeant does, in terms of
9 a sergeant, for example, analyzes statements and actions of
10 suspects, et cetera.
11 Q. And where does the information contained in this rationale
12 come from?
13 A. Again, this comes from a job analysis.
14 Q. The fourth column is labeled, "Related Major Work
15 Behaviors."
16 Can you explain what information is contained in this
17 column?
18 A. This shows that this sort of analysis is required in these
19 major work behaviors.
20 Q. And these major work behaviors are, again, the major work
21 behaviors identified in your master job description; is that
22 correct?
23 A. That's correct.
24 Q. So, when you say Section 1, "Monitors and supervises
25 activities of police officers," this was identified as a major
Barrett - cross
202
1 task -- those major tasks set forth in Section 1 in your master
2 job description. Analysis is important to be -- analytical
3 abilities are important to be able to successfully perform
4 those tasks; is that correct?
5 A. That's correct.
6 Q. And the final column is labeled, "In-basket Modules." Can
7 you please explain for me what is in this column?
8 A. This is basically the situations which represented the
9 testees, which indicates the type of issues they had to deal
10 with.
11 For example, case reports had to be reviewed and
12 analyzed. Actual reports which follow the exact forms which
13 are used by the Chicago police sergeants.
14 Q. Okay.
15 And everything in your in-basket simulation was linked
16 back to your master job description; is that correct?
17 A. That's correct.
18 Q. Okay.
19 What is table -- what is the table that's labeled
20 "L-2"? It's entitled, "Linkages Between In-basket Exercise,
21 Major Work Behaviors and Three Dimensions Tapped by the
22 In-basket Simulation Exercise"?
23 A. It's just a summary, basically, of the exercise. For
24 example, case reports, again, the major work behaviors and the
25 dimensions.
Barrett - cross
203
1 Q. So, it's, again, showing that these three -- these parts of
2 your exercise, part of the parts of the simulation, are linked
3 back to the master job description?
4 A. Yes.
5 Q. Okay.
6 And Table L-3, which is entitled, "In-basket
7 Simulation Questions Tapped by Dimension and Module," can you
8 explain for me the information that's contained in Chart L-3 --
9 or Table L-3?
10 A. This just links the actual test items. We had 60 items on
11 the in-basket. It links it very specifically to the dimension
12 and the actual exercise which was conducted. So, you can see
13 that, in essence, there were three dimensions and 20 items for
14 each of the dimensions.
15 You can see, for example, the first module said
16 "Vehicle Accident," where there is an actual paperwork on a
17 vehicle accident which the testee had to make decisions about.
18 And there were -- Question 36, for example, related that
19 vehicle accident.
20 Q. Did you also create a similar linkage chart for the oral
21 briefing exercise?
22 A. Yes.
23 Q. Let me direct your attention to Appendix O of your report.
24 MS. GLINK: I will give the Court a second to get
25 there.
Barrett - cross
204
1 THE COURT: I have it.
2 MS. GLINK: Your Honor, do you have the --
3 THE COURT: I have it. I need a magnifying glass, but
4 go ahead.
5 (Laughter.)
6 BY MS. GLINK:
7 Q. Can you briefly describe -- this is the linkage chart that
8 you created for the oral briefing exercise?
9 A. Yes.
10 Q. And can you briefly describe, in general, the information
11 contained in this chart?
12 A. Again, it's very similar to the other charts. We are just
13 looking at the dimensions. And we see there's analysis,
14 organization and oral communication are important dimensions.
15 And we see that -- I wish I could read it, also. I can't read
16 it. But it describes the definition, the ability, again, the
17 major work behaviors and our rationale for testing this
18 dimension.
19 Q. And those are the same columns and information that were
20 contained that we've already gone through with the in-basket?
21 A. Yes.
22 Q. So, again, you ensured that the purpose -- what was the
23 purpose of creating this chart?
24 A. Again, the purpose is just a rational way to show that what
25 we are doing is not arbitrary. It is part of the content of
Barrett - cross
205
1 the job of a Chicago police sergeant; that we are tapping
2 relevant dimensions of the police sergeant's job.
3 Q. And, in this case, in the oral case, your dimensions that
4 you're tapping are analysis, organization and oral
5 communication; is that correct?
6 A. Yes.
7 Q. Do you recall testifying yesterday that the written job
8 knowledge test did not directly tap the tasks set forth in
9 Section 1-E of your master job description?
10 A. Yes, I do.
11 Q. Let me direct you back to Section 1-E of that master job
12 description. Can you read into the record what Section 1-E is,
13 what tasks are set forth in Section 1-E of your master job
14 description?
15 A. "Participates in community meetings and activities,
16 different from community interactions while on patrol; attends
17 planned events where police participation is requested for
18 public relations purposes in the district or sector."
19 Q. Now, are these tasks tapped by your oral briefing exercise?
20 A. Yes.
21 Q. And let me direct you to the third column on the bottom
22 under "Oral Communication." Can you read for me what's
23 contained in that box?
24 A. I'm sorry, I'm not --
25 Q. Back on -- excuse me, Appendix O, under Dimension 3, "Oral
Barrett - cross
206
1 Communication"?
2 A. Where are you now?
3 Q. Can you read for me the column -- under the column labeled,
4 "Ability," under Section 3, "Oral Communication," can you read
5 for me what's listed there?
6 A. My copy is very indistinct, but it looks like, "Oral
7 communication ability to interact and direct officers" -- I
8 really can't -- it's too indistinct for me to --
9 THE COURT: Are we on Page O-2 --
10 THE WITNESS: O-2, yes.
11 MS. GLINK: O-2.
12 THE COURT: 3, "Oral Communication Ability"?
13 BY MS. GLINK:
14 Q. Let me give you my copy and see if you can read that more
15 clearly.
16 THE COURT: I think I can read it, too.
17 (Document tendered.)
18 BY THE WITNESS:
19 A. "Oral communication ability to direct and instruct
20 officers, communicate with peers, communicate with the public,
21 and inform superior officers of activities, ability to transmit
22 information in a verbal form that is clearly understood at an
23 appropriate rate of speech, and using proper terminology."
24 BY MS. GLINK:
25 Q. Thank you.
Barrett - redirect
207
1 MS. GLINK: I have nothing further, your Honor.
2 THE COURT: All right.
3 Any re-adverse?
4 MR. FLAXMAN: Thank you.
5 REDIRECT EXAMINATION
6 BY MR. FLAXMAN:
7 Q. Am I correct that what you said on redirect or direct was
8 that, of the people who were selected to go on to the oral
9 interview after taking the written job knowledge test and the
10 in-basket test, only 1400 were actually mathematically capable
11 of getting into the top 500?
12 MS. GLINK: Your Honor, I object to the form of the
13 question. I believe he's mischaracterizing the testimony
14 that's been before this Court.
15 THE COURT: If you are relating to what he testified
16 to today, I do not think 1400 is the right figure.
17 BY MR. FLAXMAN:
18 Q. Well, you told us that --
19 THE COURT: Why do you not state it in a more general
20 way.
21 MR. FLAXMAN: Okay.
22 BY MR. FLAXMAN:
23 Q. You took the people with the highest scores on the written
24 job knowledge test and the in-basket test, and you took about
25 2200 of those to go on to the oral test; is that right?
Barrett - redirect
208
1 A. As I recall, 2,044 were invited to go on. I believe 1937
2 actually appeared at the oral.
3 Q. Now, of that 1937, there was some smaller number that you
4 later determined, did you not, that was mathematically capable
5 of getting into the top 500?
6 A. I mean, later on, after the fact?
7 Q. After the fact, when you had all the data and you analyzed
8 it?
9 A. As I recall, there was something that no one -- the bottom
10 scores would not have been able to go on for sure, yes.
11 Q. And of the top -- when you say "bottom scores," you mean,
12 the bottom scores of those 1937; is that right?
13 A. Well, I'm getting confused because I'm not sure.
14 Q. Okay.
15 You took 1937 people who had the highest scores on the
16 written job knowledge test and the in-basket test and they took
17 the oral test; is that right?
18 A. Yes, the top 1937 people took all three components.
19 Q. Of that 1937, how many actually, when you analyzed all the
20 data, had a chance to be in the top 500?
21 A. I don't recall.
22 Q. It was less than 1937, wasn't it?
23 A. Yes, I'm sure it was.
24 Q. And it was less than a 1500, wasn't it?
25 A. I assume so, yes.
Barrett - redirect
209
1 Q. Okay.
2 It was about 1200; is that right?
3 A. It could be -- yes, I think you're right, because I think
4 there's 700 or some -- yes, it's probably true.
5 Q. So, in actuality, what you did was to set a passing score
6 for the written job knowledge test and the in-basket test of
7 the -- the 1200 highest scorers would have a chance to be
8 promoted?
9 A. We didn't set a passing score in the traditional sense. It
10 was only done for administrative convenience in terms of our
11 2,044.
12 Q. Well, operationally, did it have the same effect as a
13 passing score?
14 A. What do you mean "operationally"?
15 Q. Well, I mean, if you weren't in the top 1200, you didn't
16 have any chance of being promoted in the top 500; is that
17 right?
18 A. Well, after the fact, you're saying that administratively
19 we could go back and look and see what actually occurred, this
20 is probably true, yes.
21 Q. Now, of the top 1200, you don't have any empirical data to
22 show that, if you're lower than the top 1200 on the written job
23 knowledge and the in-basket test, you're not capable of
24 performing as a sergeant; is that right?
25 A. Are you asking if I have done an empirical criterion study
Barrett - redirect
210
1 of the top 1200 people?
2 Q. I'm asking you if you have any empirical data to support a
3 decision to take the top 1200 scorers on the written job
4 knowledge test and the in-basket test and say, "These are the
5 people who are best qualified to be sergeants"?
6 A. We have a lot of data information from past empirical
7 studies. If empirical, you mean, criterion-related, we have a
8 problem about how you define "empirical."
9 Q. Well, you've used the phrase "empirical," haven't you?
10 A. I have used the phrase "empirical." Other people have,
11 too, yes.
12 Q. And you've used the phrase "empirical data," haven't you?
13 A. Yes.
14 Q. And, as a matter of fact, didn't you use that on Page 27 of
15 your affidavit, Defendant's Exhibit 5?
16 A. I don't recall.
17 Q. Well, do you remember writing in that affidavit that the
18 unit weighting approach is recommended for situations where
19 there is no empirical data on the criterion? Do you remember
20 writing that?
21 A. No, I don't recall writing those exact words. It could
22 well be true, though, but I would have to inspect the document.
23 Q. Well, I think you still have it in front of you.
24 A. Okay.
25 Q. Would you turn to Page 27, the second paragraph.
Barrett - redirect
211
1 THE COURT: We are talking about Exhibit E now to
2 this?
3 MR. FLAXMAN: That's correct.
4 THE COURT: You are at Page 27?
5 MR. FLAXMAN: That's correct. Second sentence of the
6 second paragraph.
7 BY THE WITNESS:
8 A. Yes, this refers specifically to a criterion-related study.
9 BY MR. FLAXMAN:
10 Q. Okay.
11 Now, you don't have any empirical data to show that
12 the top 1200 on the written job knowledge test and the written
13 in-basket test are going to do better than the next 1200 if
14 promoted to sergeant; isn't that correct?
15 A. If we're using empirical data strictly in the sense of a
16 criterion-related study dealing with police sergeants, a
17 particular concurrent study on these 1200 individuals, that's
18 correct.
19 Q. Okay.
20 A. But we do have empirical data from hundreds of other
21 studies dealing with job knowledge tests and other types of
22 tests, which indicate that there is a positive correlation
23 between test scores and job performance.
24 Q. Now, these other studies, these other studies, in your
25 view, show that general cognitive ability tests predict job
Barrett - redirect
212
1 performance; isn't that right?
2 A. Yes. There are many studies which do show that general
3 cognitive ability tests do predict job performance.
4 Q. And you relied upon that belief in the work that you did
5 for the City of Chicago; isn't that right?
6 A. No.
7 Q. You didn't -- did you put that out of your head when you
8 were working for the City of Chicago?
9 A. I didn't put it out of my head. But I did not rely upon
10 that information in constructing the job knowledge tests, the
11 in-basket simulation or the oral briefing.
12 Q. Now, this -- the linkages that you were looking at for the
13 oral test and the in-basket test, those were something called
14 judgmental linkages; is that right?
15 A. Yes.
16 Q. That somebody made a judgment about whether something
17 relates to a different part of a job description; is that
18 right?
19 A. That's correct.
20 Q. There's no empirical data for those judgmental decisions;
21 is that right?
22 A. Well, again, it's how you define "empirical." Professor
23 Jencks, in his deposition, for example, defined "empirical" as
24 a process where you can reproduce information in a consistent
25 fashion and it would be possible, for example, to go back; and,
Barrett - redirect
213
1 so, if someone else would take the same information and
2 reproduce what we have done and agree or disagree.
3 Q. Okay.
4 You don't have any of that kind of data for this test,
5 do you?
6 A. Well, we have already conducted our analysis. It would be
7 for you, it seems to me, to agree or disagree.
8 Q. Now, this exhibit, this affidavit, Defendant's Exhibit 5,
9 was that prepared on or about March 3rd of 1996?
10 A. Yes.
11 Q. That was prepared specifically in response to this
12 litigation; is that right?
13 A. Yes and no. In other words, some had been prepared before
14 this. Some has been taken out of the -- our report on
15 development administration of a promotion process for the job
16 of Chicago police sergeant dated December 13th, 1994.
17 Q. Well, that weighting review that's Appendix B, Page 26 and
18 27 and 28, does that appear in the report that you gave to the
19 City of Chicago in December of 1994?
20 A. No, it does not.
21 Q. As a matter of fact, there's nothing of that report you did
22 in December of 1994 that indicated you were adopting a unit
23 weighting approach because there was no empirical data on the
24 criterion; is that right?
25 A. I don't recall those exact words in the report, no.
Barrett - redirect
214
1 Q. And the report December of '94, that was after the first
2 promotions had been made to sergeant; is that right?
3 A. I don't recall the date of the first promotion to sergeant.
4 Q. Well, if the first promotions were made in August, then
5 your report was given to the City after August of 1994; is that
6 right?
7 A. Yes.
8 Q. And that report that you gave to the City in December of
9 1994, that was after this lawsuit had been filed; is that
10 right?
11 A. I assume so. I don't know the exact date the lawsuit was
12 filed.
13 Q. Well, did you learn that this lawsuit had been filed
14 shortly after it had been filed?
15 A. I assume I did, yes.
16 Q. And you were faxed a copy of the complaint, weren't you?
17 A. I assume I was, yes.
18 Q. And you had conversations with lawyers from the City of
19 Chicago about the lawsuit; is that right?
20 A. I probably did, yes.
21 Q. And after those conversations, you completed your report,
22 this Defendant's Exhibit D; is that right?
23 A. That's correct.
24 Q. And when you completed that report, you talked about how
25 you reached your decision to weight each component of the test
Barrett - redirect
215
1 equally; is that right?
2 A. I may have. I don't recall.
3 Q. Well, do you remember there's an appendix in that report --
4 I think it's Appendix R, which is entitled, "Weighting
5 Rationale"?
6 A. Yes, it's probably our computer simulation.
7 Q. Okay.
8 That's the only explanation in the report that you
9 gave to the City in December of '94 for your decision to choose
10 the unit weighting approach; isn't that right?
11 A. I don't recall.
12 Q. Well, do you have the exhibit -- do you have the report in
13 front of you?
14 A. Which report are you referring to?
15 Q. Defendant's Exhibit D, as in David.
16 Now, if you turn to Roman III of that, there's an
17 Appendix. There's a list of appendices, and Appendix R is
18 Report of Rationale for Weighting Procedure Used. Do you see
19 that?
20 A. Yes.
21 Q. And, then, Appendix R talks about a meta-analysis and
22 computer simulations and some other work; is that right?
23 A. That's correct.
24 Q. There's nothing in Appendix R about the literature that
25 supports the unit weighting approach, is there?
Barrett - redirect
216
1 A. No, I don't believe there is.
2 Q. And there's nothing in Appendix R about the job analysis
3 supports the unit weighting approach; is that right?
4 A. No, I don't believe there is.
5 Q. And, as a matter of fact, there's nothing in -- anywhere in
6 Defendant's Exhibit D about weighting other than what's in
7 Appendix R, which is that computer simulation; is that right?
8 A. That may be true.
9 Q. Now, have you ever recommended that there be further
10 research to gather concrete data about considerations that are
11 used in employment decisions?
12 A. I'm not sure. It's sort of a general question. Again,
13 I've written over 130 articles in this area. And, so, it's
14 hard for me to recall what I may or may not have said.
15 Q. Well, did you ever recommend to the City of Chicago that
16 further work be done to gather concrete data regarding the link
17 between scores on your test and performance as a sergeant?
18 A. You mean, did I recommend a criterion-related study for
19 police sergeant?
20 Q. No. Did you ever recommend that any work be done by the
21 City of Chicago or for the City of Chicago to gather more
22 concrete data regarding the link between scores on your test
23 and ability to perform as a sergeant? Did you ever recommend
24 that?
25 A. I don't recall if I did or did not.
Barrett - redirect
217
1 Q. Did you ever recommend that the written job knowledge test
2 be administered to a sample of incumbent sergeants to see how
3 well they did on it?
4 A. I would -- I don't think I would recommend that.
5 Q. Did you ever recommend that the in-basket test be given to
6 a sample of incumbent sergeants to see how well they do on it?
7 A. I don't think I ever recommended that.
8 Q. Did you ever recommend that the oral examination be given
9 to a group of incumbent sergeants to see how well they do on
10 it?
11 A. I don't think I'd ever recommend that.
12 Q. Now, the oral exam on the sergeants, is that the same
13 format of test as the oral exam on the lieutenant's test that
14 you did for the City?
15 A. If you --
16 MS. GLINK: Objection, your Honor. Assumes facts not
17 in evidence.
18 BY MR. FLAXMAN:
19 Q. Did you also develop --
20 MR. FLAXMAN: I will withdraw the question.
21 THE COURT: Okay.
22 BY MR. FLAXMAN:
23 Q. Did you also develop the lieutenant's test for the City of
24 Chicago?
25 A. Yes, I did.
Barrett - redirect
218
1 Q. And that was used for promotion to the position of
2 lieutenant; is that right?
3 A. Yes, that's correct.
4 Q. And did that test include an oral test in the same format
5 as the oral test used in the sergeant's promotional test?
6 A. Yes, it was a similar format.
7 Q. And was the -- was there an in-basket component to the
8 lieutenant's test?
9 A. Yes, there was.
10 Q. And was that a similar format to the in-basket component of
11 the sergeant's test?
12 A. Yes.
13 Q. Could you tell us why it is that you did not recommend to
14 the City of Chicago that there be further research to offer
15 more concrete data regarding the link between scores on your
16 test and performance as a sergeant?
17 A. Again, I assume you're referring to a criterion-related
18 study; and, certainly, we considered the possibility of a
19 concurrent study. A concurrent validation study would involve
20 giving the sergeant's test -- test battery, all three
21 components to incumbent sergeants and, at the same time,
22 obtaining some measure of job performance for these sergeants.
23 There are a number of reasons why that was not a feasible
24 approach.
25 First, if we were to give the three test battery of
Barrett - redirect
219
1 three components to incumbent sergeants, say, 200 sergeants,
2 then the actual job knowledge test items and the actual items
3 on the in-basket and the actual process for the oral would be
4 revealed to these 200 incumbent sergeants. And we know that
5 when you give a test like this, it can be reproduced. So, the
6 confidentiality of the test process will be compromised. In
7 other words, the people who would be going to take this test in
8 the future would know the exact items and, thereby, destroy any
9 validity of that testing process. So, for that reason alone, I
10 would not recommend a concurrent validation study.
11 A second reason we did not -- I would not recommend
12 that approach is that it has been my experience in working with
13 safety forces that supervisors are very reluctant to give
14 performance appraisal ratings because they are worried about
15 being called biased or unfair.
16 Q. Did you ever gather any concrete data to see whether that
17 generalized belief was applicable to the police department of
18 the City of Chicago?
19 A. Yes, we had some very concrete data, as a matter of fact,
20 concerning that issue. Shortly after we gave the sergeant's
21 job knowledge test, the test was reproduced and all 150 items
22 or some semblance of them became available. In fact, it became
23 available to the Inspector General of the City of Chicago.
24 This resulted in an investigation to determine if
25 whether or not the job knowledge test had somehow been
Barrett - redirect
220
1 compromised and leaked to the testees before the test was
2 given. A complete investigation was conducted by the inspector
3 general. And, in fact, it was determined or decided that there
4 was no compromise of the testing process. What had occurred
5 was that the testees, after taking the test, had been able to
6 reproduce in a fairly good fashion the 150 items or some
7 resemblance of those 150 items.
8 So, this is very concrete evidence that we would not
9 want to conduct a concurrent validation study using the actual
10 police sergeants in the City of Chicago because we know there
11 would be a high probability that the test would be reproduced,
12 and, therefore, the validity of the testing process would be
13 destroyed.
14 Q. Now, did you ever think about giving the test to the
15 sergeants at the same time as you were giving the test to the
16 police officers?
17 A. I fail to understand the value of that sort of process.
18 Q. Okay.
19 But you never thought about that, did you?
20 A. I thought about it, but I can't understand the value of
21 that process.
22 Q. Did you ever consider giving the test to the sergeants
23 after the test had been given to the police officers?
24 A. I have thought about it; but, again, I don't understand the
25 value of that process.
Barrett - redirect
221
1 Q. Okay.
2 Did you ever -- well, are you telling us that the
3 test, as far as you're concerned, is no longer secret; it's a
4 public record that's been -- that its confidentiality has been
5 compromised?
6 A. Well, I think it's even worse than that. Its
7 confidentiality has been compromised for some people. Some
8 group of people, for example, has reproduced the job knowledge
9 test. We don't know who those people are.
10 Q. So --
11 A. So, it's some, not everybody has access to it. Some people
12 have, obviously, have reproduced a part of the job knowledge
13 test.
14 Q. So, would you recommend to the City that they use that job
15 knowledge test, again, in the future?
16 A. No, I would not recommend to the City that they would use
17 that same job knowledge test in the future.
18 Q. Okay.
19 Now, you also talked about, in this concurrent
20 criterion validity study, about performance ratings. And you
21 said, in your experience, that people don't like to give
22 performance ratings.
23 Did you ever gather any concrete data from the police
24 department or the City of Chicago about its ability or lack of
25 ability to rate how its incumbent police officers are
Barrett - redirect
222
1 performing their job?
2 A. No, I've never attempted to devise a performance appraisal
3 device for police sergeants in the City of Chicago and
4 determine whether or not the superior officers would be willing
5 or not to rate the incumbent police sergeants.
6 Q. At some point, did you learn that there is in existence in
7 the police department of the City of Chicago a formal mechanism
8 for identifying police officers who are performing their job at
9 an exceptional level of competence?
10 A. Yes.
11 Q. And did you ever consider using those people who had been
12 selected through that system performing their job at an
13 exceptional level of competence and administering your test to
14 them?
15 MS. GLINK: Objection, your Honor, to the extent that
16 it has not been established what system he's talking about at
17 this point.
18 THE COURT: Yes, it seems to me it is so vague.
19 MR. FLAXMAN: All right.
20 THE COURT: At least I will permit you to proceed in
21 this same subject matter area, but it seems to me this
22 particular question is very difficult for the witness to
23 understand to be able to answer intelligently.
24 BY MR. FLAXMAN:
25 Q. At some point you learned that there's a merit selection
Barrett - redirect
223
1 system in existence in the City of Chicago for promotion to
2 detective; is that right?
3 A. Yes. In general, I've heard that there is a mechanism like
4 this.
5 Q. And did you also learn that there's a merit selection
6 system which had been intended to be used in 20 percent of the
7 promotions to lieutenant?
8 A. Yes, I have learned that that was a process which was not
9 accepted by a court.
10 Q. And did you ever consider administering your test to those
11 persons who had been identified through the merit selection
12 process for promotion to see how well they did on these --
13 MS. GLINK: I'm going to object, your Honor. Counsel
14 hasn't established when he found out about this selection
15 process and whether it was even in existence --
16 THE COURT: Yes.
17 MS. GLINK: -- at the time that Dr. Barrett --
18 THE COURT: I will sustain the objection for lack of
19 foundation as to the relevancy of these questions.
20 MR. FLAXMAN: May I have just a minute, your Honor?
21 THE COURT: Yes.
22 (Brief pause.)
23 THE COURT: One thing that is not clear to me is
24 whether your questions are designed to relate to what was done
25 in preparation for preparing the existing test or whether you
Barrett - redirect
224
1 are asking him why he did not do all sorts of things to seek to
2 prove the validity of the test after the test had been taken.
3 It seems to me those are two different areas; and, to
4 intermingle them, it seems to me, is misleading.
5 MR. FLAXMAN: I don't want to be misleading or
6 confusing and --
7 THE COURT: Did you understand the point that I was
8 making, though?
9 MR. FLAXMAN: I understood the point, your Honor, and
10 I'm not trying to be misleading. It might be confusing, but
11 misleading is not productive.
12 BY MR. FLAXMAN:
13 Q. Let's go back to March of 1993, when Barrett & Associates
14 submitted its proposal to develop the sergeant's test for the
15 City of Chicago.
16 At that time, was it your plan, Barrett & Associates'
17 plan, that there would be no minimal -- no cutoff point --
18 MR. FLAXMAN: Strike that.
19 BY MR. FLAXMAN:
20 Q. At that time, back in March of 1993, was there a -- was
21 part of the proposal to do a concurrent validation study?
22 A. I don't recall ever proposing that.
23 Q. Okay.
24 Did you ever discuss doing that with the City of
25 Chicago back in March of 1993?
Barrett - redirect
225
1 A. I don't recall discussing -- I shouldn't say -- I just
2 don't recall. We may have, but I don't recall it.
3 Q. Well, when you were talking about identification of
4 minimally competent persons in the written test development in
5 the proposal, did that involve a concurrent validation study of
6 administering the test to incumbents?
7 A. Well, again, I don't recall. As I recall, you pointed out
8 somewhere in an appendix to a contract, and, again, you have to
9 show me what you are referring to.
10 Q. Let me show you what's previously been marked as
11 Plaintiffs' Exhibit 12 and direct your attention to the page
12 numbered 17, which is Bates stamped number 50, and ask you to
13 look where it says -- under where it says, "Written Test
14 Development."
15 (Document tendered.)
16 BY MR. FLAXMAN:
17 Q. Do you see that?
18 A. Yes, I do.
19 Q. Now, was that a reference to a concurrent validation study?
20 A. No, it was not.
21 Q. Well, how were you going to identify -- could I have it
22 back?
23 A. Well, okay.
24 (Document tendered.)
25 BY MR. FLAXMAN:
Barrett - redirect
226
1 Q. How were you going to identify the minimally competent
2 people?
3 A. Well, again, I think in the body of the proposal, there is
4 no mention of anything like that. So, I don't know what it
5 refers to in the sense of our project because it's not at all
6 mentioned in our main body of our proposal.
7 All we'd say is that I recall that we would take the
8 top 2,000 people, I believe, on to the oral.
9 Q. Now, do you remember an article about the effects of age
10 stereotyping in a simulated interview that you co-authored back
11 in 1987?
12 A. No, I don't recall the details of it. I recall authoring
13 an article -- or author of an article like that.
14 Q. You do or you don't? I'm sorry.
15 A. I'm sorry?
16 Q. You do or you do not recall?
17 A. I recall an article like that. I don't recall the details
18 of the article.
19 Q. Do you know someone named Bruce J. Avolia (phonetic)?
20 A. Yes, I do.
21 Q. Is he one of your former students?
22 A. Yes.
23 Q. And was he -- did you co-author an article with him?
24 A. Yes, I did.
25 Q. And do you remember in that article concluding that further
Barrett - redirect
227
1 research is needed to offer more concrete data regarding the
2 link between chronological age and various aspects of job
3 performance?
4 MS. GLINK: I'm going to object, your Honor, on the
5 grounds of relevance. This is an article written about age,
6 not about promotions of police officers.
7 THE COURT: I am going to overrule the objection.
8 It seems to me that it is related to the defendants.
9 I mean, the plaintiffs' attempt to show that specific testing
10 is necessary to establish the validity of certain principles
11 that were used in preparing the test, and the fact that age is
12 not a factor in this particular case does not mean that does
13 not establish a principle that might relate to the case. So,
14 on that rounded basis, I will overrule the objection and permit
15 him to proceed.
16 MS. GLINK: I am going to ask, your Honor, that the
17 plaintiffs' counsel provide a copy of this article to
18 Dr. Barrett, so that he has a chance to look at it and refer to
19 it while Mr. Flaxman is questioning him on it.
20 THE COURT: Well, if he is going to ask him about the
21 content, obviously, if the witness wishes to request -- review
22 the article, he has the right to do that. And, similarly,
23 counsel should give opposing counsel a copy of the article, if
24 you have it.
25 MR. FLAXMAN: I have one copy, Judge.
Barrett - redirect
228
1 THE COURT: That is -- well, then we will take a break
2 and let opposing counsel have a chance to look at it first
3 before you proceed with your questions.
4 MS. GLINK: Thank you.
5 THE COURT: In fact, why do we not take a mid-morning
6 break at this time, and we will try to hold it to ten minutes.
7 Then we will go on to 12:30, we will take an hour break; and,
8 then, we will resume and go until.
9 MS. GLINK: Your Honor, is there anybody in your
10 chambers who could make a couple copies, so that Dr. Barrett
11 could also review this during the break?
12 THE COURT: How many pages do we have?
13 MS. GLINK: It's not very long. About ten,
14 approximately.
15 THE COURT: I will ask Laurie here to assist and show
16 you where our -- we have a copy machine on this floor, so that
17 she will assist.
18 MR. FLAXMAN: Okay.
19 THE COURT: We will make -- how many copies, two,
20 anyway? Three?
21 MS. GLINK: One for the Court, one for us, I would
22 assume, and one for Dr. Barrett.
23 Thank you.
24 THE COURT: All right.
25 We will take a short recess then at this time.
Barrett - redirect
229
1 (Whereupon, a recess was taken.)
2 THE COURT: Court is back in session. Everybody
3 please be seated.
4 With the copying of the article, you may proceed now
5 with --
6 MR. FLAXMAN: Okay.
7 THE COURT: -- relevant questions, anyway.
8 MR. FLAXMAN: All right.
9 Let me just move away from the article for a moment.
10 BY MR. FLAXMAN:
11 Q. Did you, when you prepared your proposal for the City of
12 Chicago, examine or know about something called the Report of a
13 Blue Ribbon Panel on Police Testing, Hiring and Promotion?
14 A. I'm not sure of the exact time I knew about the Blue Ribbon
15 Panel of 1990 Report, but I know certainly before -- once we
16 started the project, I recall reviewing that report.
17 Q. And did you try to follow the recommendations of the Blue
18 Ribbon Panel on police promotion?
19 A. Yes.
20 Q. Well, did you follow the recommendation of the Blue Ribbon
21 Panel that performance evaluations play a role in promotions?
22 A. I'd have to read the exact section to know what you're
23 referring to.
24 Q. Well, let me show you what I will mark as Plaintiffs'
25 Exhibit 60 and ask you to look at Page 11.
Barrett - redirect
230
1 MR. FLAXMAN: And I think we have a copy for the Court.
2 (Document tendered to the Court.)
3 THE COURT: I take it the city has got a copy of this,
4 as well?
5 MS. PAPUSHKEWYCH: Yes.
6 THE COURT: Okay.
7 (Document tendered.)
8 BY MR. FLAXMAN:
9 Q. Exhibit 60 is the Report of the Blue Ribbon Panel; is that
10 right?
11 A. Yes.
12 Q. And Page 11, there's a Paragraph 4, which has the fourth
13 recommendation about promotions; is that right?
14 A. Yes.
15 Q. And Paragraph 4 is a recommendation that performance
16 evaluations should play a role in promotions?
17 A. Yes.
18 Q. And, then, the next paragraph is that some way should
19 weight should also be given to the applicant's attendance,
20 medical and disciplinary records?
21 A. Yes.
22 Q. Your test doesn't give any weight to attendance, medical
23 and disciplinary records, does it?
24 A. No, it does not.
25 Q. And your test doesn't include -- doesn't give any role to
Barrett - redirect
231
1 performance evaluations; is that right?
2 A. That's correct.
3 Q. Now, did you ever become aware of the work that had been
4 done for the City of Chicago by the consulting firm of
5 Richardson, Bellows, Henry & Company?
6 A. I know that consulting firm, yes.
7 Q. Okay.
8 Did you know that they did a concurrent validation
9 study of its -- of a performance evaluation system with
10 incumbent Chicago police officers?
11 A. I don't recall reading a report like that.
12 Q. Well, do you see on -- did you read -- when you read this
13 Blue Ribbon Panel Report for the first time, did you see where
14 it says on Pages 11 to 12, the reference to Richardson,
15 Bellows, Henry & Company administering an instrument to current
16 officers who would be taking the test for validation purposes?
17 A. Yes, I do see that.
18 Q. Well, when you saw that and read that, you were aware that
19 at least one concurrent validation study had been performed for
20 the City of Chicago with respect to validating a test for
21 hiring police officers; is that right?
22 A. It doesn't say that in this report. They just mention
23 concurrent validation study. I don't believe --
24 Q. Well, it says that, "Current officers who would be taking
25 the test for validation purposes." Do you see that at the top
Barrett - redirect
232
1 of Page 12?
2 A. Yes, I see those words, but it doesn't say the concurrent
3 study.
4 Q. Well, okay.
5 But it does say --
6 A. I have not reviewed any report of that study --
7 Q. Okay.
8 A. -- as far as I recall. I don't recall reviewing a report
9 of that study.
10 Q. So, as you sit here now, you don't know whether or not the
11 City of Chicago administered its entry-level examination to
12 current officers who took the test for validation purposes?
13 A. I'm sorry. Say that, again.
14 Q. As you sit here now, do you or do you not know that the
15 City of Chicago used its entry-level test on current officers
16 who took the test for validation purposes?
17 MS. GLINK: I object to the extent that it assumes
18 that that is, in fact, a true statement.
19 THE COURT: I will sustain the objection as to the
20 form of the question, but not the subject matter. But you are
21 referring to Exhibit G now as --
22 MR. FLAXMAN: No, this is --
23 THE COURT: -- the test?
24 MR. FLAXMAN: -- called Exhibit 60, Plaintiffs'
25 Exhibit 60.
Barrett - redirect
233
1 THE COURT: Right. No, no, but I am talking about the
2 inner report as you have referred to. At the top of Page 12,
3 it refers to --
4 MR. FLAXMAN: Oh.
5 THE COURT: -- Exhibit G --
6 MR. FLAXMAN: That's correct.
7 THE COURT: -- as that evaluation record. And you are
8 asking a question as to whether this has been used, as I
9 understand it?
10 MR. FLAXMAN: Well, did he know that it was used is
11 what the question was.
12 MS. GLINK: I, again, object to it the grounds that it
13 assumes a fact not in evidence.
14 THE COURT: Well, this is something we will have to
15 deal with, admissibility of the exhibits that are being offered
16 and referred to here. But it seems to me as long as the
17 question makes it clear you are referring to the knowledge that
18 he had prior to some specific time period, so we know in what
19 context it is, I will overrule the objection and permit you to
20 proceed.
21 BY MR. FLAXMAN:
22 Q. Before you submitted your final report in December of 1994
23 to the City of Chicago, were you aware of this Blue Ribbon
24 Panel Report, Plaintiffs' Exhibit 60?
25 A. Yes.
Barrett - redirect
234
1 Q. And were you aware of the Blue Ribbon Panel Report before
2 December of 1993 when the written test was administered?
3 A. Yes.
4 Q. And when you read the Blue Ribbon Report -- well, when you
5 became aware of the Blue Ribbon Report, did you read it?
6 A. Yes.
7 Q. And did you read Page 11 and 12?
8 A. Yes.
9 Q. And when you read Page 11 and 12, did you read the sentence
10 that said that, "To evaluate the current officers who would be
11 taking the test for validation purposes"?
12 A. It's a very ambiguous statement. Yes, I read the
13 statement.
14 Q. Well, when you read that statement, did you know that the
15 entry-level examination --
16 THE COURT: I think it would be helpful for the record
17 if you read the entire question.
18 MR. FLAXMAN: Okay. Well, let me --
19 THE COURT: I mean, the entire sentence starting on
20 Page 11, just so we get the context down.
21 BY MR. FLAXMAN:
22 Q. Did you read this sentence: "In conjunction with the
23 recent entry-level examination (the law enforcement candidate
24 record), the consulting firm of Richardson, Bellows, Henry &
25 Company (RBH) used a very interesting and sensitive performance
Barrett - redirect
235
1 evaluation instrument to evaluate the current officers who
2 would be taking the test for validation purposes."
3 Did you read that sentence?
4 A. Yes.
5 Q. And when you read that sentence, did you learn that the law
6 enforcement candidate record, the entry-level examination, was
7 being administered to current officers who would be taking the
8 test for validation purposes?
9 A. No, I didn't understand it in that context. I wasn't sure
10 what it meant.
11 Q. Well, did you ask anybody at the City what it meant?
12 A. No.
13 Q. Did you ask anybody on the Blue Ribbon Panel what it meant?
14 A. No.
15 Q. You didn't administer your test for validation purposes to
16 any incumbent police sergeants, did you?
17 A. No, I did not.
18 Q. Now, during the break, did you have a chance to review that
19 article that's been marked as Plaintiffs' Exhibit 58?
20 A. No, I didn't have a chance to review the whole article.
21 Q. Well, did you have a chance to review Page 62, the
22 paragraph before, "Potential Limitations"?
23 A. Which paragraph do you want me to review?
24 Q. The paragraph that starts, "The inconsistencies that
25 presently exist"; do you see that?
Barrett - redirect
236
1 A. Yes, I see.
2 Q. Could you review that paragraph for us.
3 (Brief pause.)
4 BY THE WITNESS:
5 A. Okay.
6 BY MR. FLAXMAN:
7 Q. Now, this Plaintiffs' Exhibit 58, I think, is an article
8 that appeared in a referee journal; is that right?
9 A. Yes, it did.
10 Q. And you are the co-author of that article; is that right?
11 A. Yes.
12 Q. And for an article to appear in a referee journal, does it
13 mean that your peers in the field of psychology have reviewed
14 the article and approved it for publication?
15 A. Yes.
16 Q. And in that article, when you -- is it correct --
17 THE COURT: Can you read the date of that?
18 MR. FLAXMAN: I'm sorry.
19 BY MR. FLAXMAN:
20 Q. Is it correct that that article appears in a journal
21 called, "Psychology and Aging," in March 1987?
22 A. Yes.
23 Q. And your article appears starting at Page 56; is that
24 right?
25 A. Yes.
Barrett - redirect
237
1 Q. And did you and your co-author write in that article that,
2 "Further research is needed to offer more concrete data
3 regarding the link between chronological age and various
4 aspects of job performance"?
5 A. Yes.
6 Q. And when you wrote that or co-wrote that sentence, was it
7 correct?
8 A. Correct in the sense -- yes, it's a correct statement.
9 Q. Okay.
10 And is it also correct that further research is needed
11 to offer more concrete data regarding the link between scores
12 on the sergeant's promotional test and aspects of job
13 performance as a police sergeant?
14 A. No.
15 Q. There is no data now; isn't that correct?
16 A. I'm confused. I don't see the analogy. I'm sorry.
17 Q. Okay.
18 Well, do you have any --
19 THE COURT: Your question -- this is
20 cross-examination. It's adverse examination. But it cannot --
21 your question cannot be misleading as to what he has testified
22 to. He has testified to the fact that there are over a hundred
23 tests that he is aware of with respect to the validation of a
24 content aspect test that we are dealing with here. And, so,
25 what you are asking -- what your question, I think, was
Barrett - recross
238
1 intended to be that there has been no testing of the Chicago
2 Police Department --
3 MR. FLAXMAN: That's what I --
4 THE COURT: -- with respect to a content aspect test.
5 MR. FLAXMAN: That's --
6 THE COURT: Okay. And I take it the answer is "No" to
7 that?
8 THE WITNESS: Yes, the answer is "No."
9 MR. FLAXMAN: You just saved me. Thank you. I have
10 nothing further.
11 MS. GLINK: I just have a couple of questions.
12 RECROSS EXAMINATION
13 BY MS. GLINK:
14 Q. Dr. Barrett, I am just going to draw your attention back to
15 the Blue Ribbon Report that you were just referring to, and I'm
16 going to call your attention to Page 11 and 12. That would be
17 Paragraph 4 of that report.
18 A. Yes.
19 Q. Can you read for me the second full paragraph that begins
20 with "We recommend" on Page 12 of that report?
21 A. "We recommend hiring a nationally-recognized police
22 consultant to develop a fair performance evaluation procedure
23 (perhaps using the RBH performance evaluation instrument) and
24 to consult generally on the promotion process. We have
25 consulted with a Carroll Buracker, the former chief of police
Barrett - recross
239
1 of Fairfax, Virginia and head of the consulting firm, Carroll
2 Buracker & Associates, and were very impressed with him and his
3 firm. They have developed promotional processes (including
4 performance evaluation measures) for other large police
5 departments, and could do the same for Chicago."
6 Q. At the time that you were -- Dr. Barrett, were you hired to
7 develop this fair promotion evaluation procedure that the Blue
8 Ribbon Panel is referring to on Page 12?
9 A. Yes.
10 Q. And what do you mean by that?
11 A. A fair --
12 Q. Were you hired to develop a performance evaluation
13 procedure?
14 A. No.
15 Q. What were you hired by the City of Chicago to develop?
16 A. A promotional examination for police sergeant.
17 Q. And at the time that you developed your promotional
18 examination for police sergeant, was there in place a
19 performance evaluation procedure like the one recommended by
20 the Blue Ribbon Panel?
21 A. As far as I know, there was not.
22 Q. Do you know whether the City has developed such a promotion
23 evaluation system today?
24 A. As far as I know, they have not.
25 Q. And, in your opinion, are there any problems associated
Barrett - recross
240
1 with developing a performance evaluation procedure like the one
2 referred to in the Blue Ribbon Panel?
3 A. Yes, there are many major problems.
4 Q. And what are those problems?
5 A. Well, the first problem with any performance appraisal
6 system is the issue of reliability. They tend to have low
7 reliability.
8 A second problem, of course, is perceived fairness.
9 When you have a supervisor making judgments about subordinates,
10 there is always the issue of subjectivity, friendship, and
11 other factors entering into the process.
12 In fact, I have personal experience of attempting to
13 develop performance appraisal techniques for police sergeants.
14 And while I was very successful in the pilot stages of
15 developing a reliable and valid performance appraisal
16 technique, once it was learned that it would actually be used
17 to help promote from patrol officer to sergeant, the process
18 fell apart because the supervisors were reluctant to
19 differentiate among individuals because they know that it would
20 be revealed who they rated higher. And, so, we had to abandon
21 that process.
22 Q. And for which jurisdiction did you attempt to develop such
23 a performance appraisal system?
24 A. The City of Akron's police department.
25 MS. GLINK: Can you give me one minute, your Honor?
Barrett - recross
241
1 THE COURT: Yes.
2 BY MS. GLINK:
3 Q. Dr. Barrett, in your opinion, is there a difference between
4 conducting a concurrent validation study for a police entrance
5 level versus for a concurrent validation study for a police
6 promotional rank?
7 A. Yes.
8 Q. And what are the differences?
9 A. Well, one, it's always a different level in the
10 organization; and, for a police entrance examination, you would
11 develop a test which was quite different and the process would
12 be quite different in terms of the type of test you could
13 develop.
14 For example, for a police entrance examination, you
15 could not use the job knowledge test because a job knowledge
16 test is information which was learned after you are hired as a
17 patrol officer, and you study for it to be promoted to police
18 sergeant. So, you have to use a different type of test
19 entirely in that context.
20 Also, there's the issue of -- confidentiality is
21 somewhat different because in the case of the concurrent study
22 for police sergeants, you would have to use incumbent police
23 sergeants who have friends who are police patrol officers who
24 they would want to give information to.
25 Conversely, the problem is still there, but it's not
Barrett - recross
242
1 quite as bad, for police entrants because you have a broader
2 group, in other words, and your test is somewhat different in
3 terms of the type of test you are going to be giving.
4 Q. Dr. Barrett, do the uniform guidelines on employee
5 selection procedure recognize more than one type of validation
6 of a promotional examination?
7 MR. FLAXMAN: Objection. This is beyond the scope of
8 re-adverse.
9 THE COURT: Well, it seems to me it deals with the
10 general subject matter. He is going to be called, again, to
11 testify, as I understand it. So, it seems to me that -- and he
12 is being questioned as though on redirect. So, I will overrule
13 the objection and permit you to proceed.
14 MS. GLINK: Well, I'm happy to cover this material in
15 our direct. I just am concerned we won't get to it this
16 afternoon.
17 THE COURT: Well, I think one of the problems is that
18 the other expert is not going to be available after today, as I
19 understand it. And I do not think that is the same problem.
20 MS. GLINK: Dr. Barrett has to leave by noon tomorrow.
21 THE COURT: All right.
22 MS. GLINK: Why don't we move on. I will save this
23 and we can go into this on direct.
24 THE COURT: All right.
25 MS. GLINK: I have nothing further.
Barrett - recross
243
1 THE COURT: Any additional questions?
2 MR. FLAXMAN: No.
3 THE COURT: All right. That will conclude your
4 testimony, then, at this time.
5 (Witness excused.)
6 THE COURT: And we can call the next witness, then.
7 MR. FLAXMAN: Our next witness, your Honor, is
8 Christopher Jencks.
9 THE COURT: All right. Sir, if you will stand and
10 raise your right-hand.
11 CHRISTOPHER JENCKS, PLAINTIFF'S WITNESS, SWORN
12 THE COURT: You may fire when ready.
13 MR. FLAXMAN: Thank you, Judge.
14 DIRECT EXAMINATION
15 BY MR. FLAXMAN:
16 Q. Could you state your full name and spell your last name,
17 please?
18 A. My name is Christopher Jencks, spelled J-e-n-c-k-s.
19 Q. And what's your business or occupation?
20 A. I'm a professor of sociology and urban affairs at
21 Northwestern University.
22 Q. Let me show you what's been marked as Plaintiffs' Exhibit
23 59.
24 (Document tendered.)
25 MR. FLAXMAN: And I have a copy for your Honor and for
Jencks - direct
244
1 --
2 (Document tendered to the Court.)
3 BY MR. FLAXMAN:
4 Q. Is Plaintiffs' Exhibit 59 a copy of your current curriculum
5 vitae?
6 A. Since it's dated June of 1995, I guess the answer is not
7 quite current.
8 Q. As of June, 1995 --
9 A. Yes.
10 Q. -- was it your curriculum vitae?
11 And does this list your books and reports and chapters
12 in books and articles in journals?
13 A. It does.
14 Q. Okay.
15 What's your -- do you have an area of concentration or
16 specialty?
17 A. The principal areas in which I've done research over the
18 years are race and economic inequality, education and testing.
19 Q. Now, is it accurate to refer to you as a quantitative
20 social scientist?
21 A. I would say that's accurate, yes.
22 Q. And does your work involve data?
23 A. Very large quantities of data, yes.
24 Q. In your work, have you ever been called upon to advise
25 public officials on matters of public concern?
Jencks - direct
245
1 A. Yes.
2 Q. Could you tell us briefly what that's been?
3 A. In the last few years, the principal activity that I've
4 engaged in advising public officials has had to do with giving
5 advice up to both the Congress and to various -- some
6 committees in the executive branch of the federal government
7 about the reform of Aid to Families with Dependent Children
8 and, particularly, about income sources of welfare recipients
9 that are not reported to the Department of Public Welfare,
10 typically. We have been doing research on unreported income,
11 which proves to be quite relevant to changes in welfare laws.
12 Q. Now, you're not an industrial psychologist, are you?
13 A. I am not.
14 Q. And you're not a full-time psychometrician?
15 A. No, I'm not.
16 Q. And you are not in the business of preparing promotional
17 tests for municipalities?
18 A. No.
19 Q. And you're not a police officer?
20 A. No.
21 Q. And you've never been a police officer?
22 A. No.
23 Q. Have you ever done any research involving testing?
24 A. Yes. I've written several books that dealt with testing,
25 and I'm currently working on a book dealing with racial
Jencks - direct
246
1 differences in test performance.
2 Q. When you say you're "working on a book," what exactly are
3 you doing?
4 A. Last year when I was teaching at the University of Chicago
5 as a visitor, I ran a conference with a colleague on racial
6 differences in performance on academic tests and IQ tests and
7 tests of that sort, involving experts from around the country
8 who came and gave papers. And we then decided that we would
9 turn that into a book.
10 I've been editing the papers, writing a long
11 introduction, collecting additional papers on some areas that
12 were neglected in the conference, as it seemed to me. And we
13 will probably publish that book with Harvard University Press
14 next year.
15 Q. Now, how long have you been at Northwestern?
16 A. I've been at Northwestern since 1979.
17 Q. And what's your current position at Northwestern?
18 A. I am the John D. MacArthur Professor of Sociology.
19 Q. I understand that you'll soon be leaving the Midwest.
20 A. That's correct.
21 Q. Where will you be going?
22 A. I've accepted a position at the Kennedy School of
23 Government at Harvard.
24 Q. And what will you be doing at the Kennedy School of
25 Government?
Jencks - direct
247
1 A. I will be teaching courses on social policy, poverty,
2 welfare, race-related matters.
3 Q. Do you have a consulting business you run on the side?
4 A. I do not.
5 Q. But I have promised to pay you for your time that you are
6 working on this case?
7 A. You have so promised, yes.
8 Q. Thank you.
9 In connection with the work that you've done on this
10 case, did you get a chance to look at the actual test
11 instruments?
12 A. Yes, I did.
13 Q. And before you did that, did you sign this protective
14 order?
15 A. I did.
16 Q. And have you abided by the protective order?
17 A. I have.
18 THE COURT: First of all, do we have any objection to
19 his being considered as an expert in the area of testing or
20 industrial promotional testing?
21 MS. GLINK: Yes, your Honor, we have a very large
22 objection because as Professor Jencks has stated, he has
23 absolutely no background in industrial psychology or test
24 development at all. And, in fact --
25 THE COURT: Well, of course, he has not been asked
Jencks - direct
248
1 that yet, and I wondered --
2 MR. FLAXMAN: I'm not calling -- we are not going to
3 be seeking to offer those kinds of opinions.
4 MS. GLINK: We would object on foundation if he did.
5 MR. FLAXMAN: I think, when I ask him the opinion
6 question, we can --
7 THE COURT: I was going to ask if he was aware of what
8 the leading treatises are in the area and --
9 MR. FLAXMAN: That's not his expertise, and he is not
10 being offered for that kind of opinion.
11 THE COURT: What is he being offered for, then?
12 MR. FLAXMAN: As a --
13 THE COURT: I take it he is being offered as an
14 expert?
15 MR. FLAXMAN: He is being offered as a quantitative --
16 as an expert in quantitative social sciences.
17 THE COURT: I am not aware that that is an established
18 area of social science. If quantitative means that he is aware
19 of statistical evidence and statistical reports that are made,
20 that is one thing; but, I am not sure how it relates to
21 testing.
22 MR. FLAXMAN: Well, let me --
23 THE COURT: I am not trying to be difficult.
24 MR. FLAXMAN: Let me ask some more foundation
25 questions.
Jencks - direct
249
1 THE COURT: I am trying to make as good a record as I
2 can for this case.
3 BY MR. FLAXMAN:
4 Q. In the course of your professional work, have you been
5 involved in testing?
6 A. I have been involved in analyzing the results of testing.
7 I have not been involved in administering tests.
8 Q. When did you first become involved in analyzing the results
9 of testing?
10 A. 1966.
11 Q. And what kind of test was that that you were involved in
12 analyzing the results of?
13 A. Those were tests administered to children enrolled in
14 school at all grade levels and a survey administered nationally
15 in 1965 to determine why black children were not achieving as
16 well as white children.
17 Q. Was there a particular name for that survey?
18 A. This was the survey -- a quality of opportunity survey
19 commonly known as the Coleman Survey, produced the Coleman
20 Report.
21 Q. Where were you when you were working on analyzing that
22 data?
23 A. I was -- at the time that I began working on that issue, I
24 was at the Institute for Policy Studies in Washington, and,
25 soon after that, I moved to Harvard University, where I was for
Jencks - direct
250
1 the next six years.
2 Q. In the course of analyzing that data, what kind of
3 techniques did you use?
4 A. We used a wide range of statistical techniques. We looked
5 at the questions of validity and reliability of the tests. We
6 -- most of the analysis was done with what were then standard
7 methods of multiple regression analysis.
8 Q. Now, when you said you looked at the reliability of the
9 test, what do you mean by the reliability of the test?
10 A. By "reliability" in this context, I mean that the different
11 items in the test appear to measure the same underlying skills
12 or, in some case, appear to measure different underlying
13 skills. But it's an effort to determine whether or not the
14 test measures one thing or many different things.
15 Q. And what do you mean by "the validity of the test"?
16 A. The validity of the test means that we investigated the
17 degree to which the tests that were being administered to the
18 students in school predicted anything that we might be
19 interested in after they left school; whether they went to
20 college, how well they did in college, whether they got a good
21 job, how much they earned. That kind of thing.
22 Q. Now, did the work that you did on that project get written
23 up in any form?
24 A. We did a number of book chapters and a couple of
25 professional articles out of it and, also, a full-length book
Jencks - direct
251
1 that dealt extensively with that.
2 Q. When you say "we," who is the "we"?
3 A. There was a group of people at the Center for Educational
4 Policy Research at Harvard, which I had founded when I went
5 there. There were seven of us, if I recall correctly, who
6 worked on producing this book. And I was the first author of
7 that book, but there were a number of other co-authors.
8 Q. And what's the name of that book?
9 A. The name of that book is "Inequality."
10 Q. Now, after that work, did you do any other work involving
11 tests?
12 A. Yes.
13 We -- I continued to be interested in the question of
14 whether tests played an important role in predicting how well
15 individuals did in the labor market and published another book
16 in 1979 titled, "Who Gets Ahead?", which dealt with that
17 question. That book also had a number of co-authors, but it
18 had -- it considered the role of tests in predicting wages and
19 the occupations that people ended up in.
20 And, then, during the last few years, I've returned to
21 the set of issues, because of an interest, in the role of tests
22 in explaining differences in wages between black and white
23 workers.
24 Q. Now, when you did the work on how people do in the labor
25 market in tests, was there data that you analyzed for that?
Jencks - direct
252
1 A. There are a large number of surveys in which it's -- either
2 tests have been administered to the respondents or in which
3 it's possible to link up respondents' names to tests that they
4 took, for example, when they went into the armed services and
5 took the Armed Forces qualification test.
6 So, we did work with a large number of quantitative
7 surveys of those issues.
8 Q. And what kind of tools did you use in analyzing that data?
9 A. I'm not exactly sure what you mean there. Most of those
10 analyses are done with computers and data tape and statistical
11 methods, which I could describe if you want, but I'm not
12 certain --
13 Q. Well, do you also teach courses in quantitative methods?
14 A. I have taught courses in quantitative methods, yes.
15 THE COURT: By "quantitative," you mean statistical
16 methods; is that --
17 MR. FLAXMAN: Yes.
18 BY MR. FLAXMAN:
19 Q. Have you taught courses?
20 A. Yes, I have.
21 Q. Okay.
22 And have you --
23 THE COURT: Okay.
24 If I can -- because time is of the essence, do you
25 have any additional questions as to his qualifications that you
Jencks - voir dire
253
1 wish to ask at this time?
2 MS. GLINK: Your Honor, we do.
3 THE COURT: I will permit you to do that, so we can
4 try to -- because we are running into a time problem here. So,
5 I want to be sure that --
6 VOIR DIRE EXAMINATION
7 BY MS. GLINK:
8 Q. You've already acknowledged, Professor Jencks, that you do
9 not have a background in industrial psychology or testing. And
10 isn't also true that you've never developed a test for a police
11 department?
12 A. That's correct.
13 Q. Or a fire department?
14 A. Yes.
15 Q. Or for any safety force position?
16 A. That's correct.
17 Q. And that you've also never developed a test for any
18 employment job; isn't that correct?
19 A. That's correct, yes.
20 Q. Isn't it also correct that you've never evaluated the
21 validity of any promotional examination given for promotion in
22 a police or fire department?
23 A. That's correct.
24 Q. And you also similarly never evaluated the validity of a
25 promotional examination given in an employment position?
Jencks - voir dire
254
1 A. That's correct, yes.
2 Q. Isn't it also true, Professor Jencks, that you've never
3 reviewed any studies which specifically addressed the
4 predictability of promotional tests in the context of police
5 departments?
6 A. Other than one study that I reviewed in connection with
7 this trial, that's correct.
8 Q. And that study did not directly deal with the predicted
9 value of a content valid test in a police department, did it?
10 A. That's correct.
11 Q. Isn't it also true that you've never conducted a job
12 analysis of the job of police sergeant?
13 A. That's correct, yes.
14 Q. Or for any police position?
15 A. That's correct.
16 Q. And that you've never even interviewed a Chicago police
17 sergeant about his job duties or responsibilities?
18 A. That's correct.
19 Q. And that your only knowledge about the job of police
20 sergeant is based on your readings of the documents that were
21 given to you in preparation for this litigation; isn't that
22 correct?
23 A. That's correct.
24 Q. And it would also be fair, would it not, to characterize
25 your knowledge of the EEOC Guidelines on Employee Selection
Jencks - voir dire
255
1 Procedure and your knowledge of the standards for industrial
2 psychologists in developing tests as being general?
3 A. I think that's a reasonable characterization, yes.
4 Q. And isn't it also true that the tests that you have
5 analyzed in the past, the ones that you were just referring to,
6 were generally scholastic aptitude tests or achievement tests
7 in the school setting?
8 A. Many of the tests are scholastic aptitude and achievement
9 tests in the school setting, but some -- quite a number of
10 these tests are also tests that are used for job selection.
11 For instance, the Armed Forces qualification test is a test of
12 that kind and is used for selection of recruits into the armed
13 forces. And that's probably the test that is most -- I've
14 spent the most time working on.
15 Q. And your other work in the area has been on either aptitude
16 tests or cognitive ability tests; is that right?
17 A. That's correct.
18 Q. And that you've never actually analyzed any job knowledge
19 test that's given in the employment sector for promotion to a
20 specific position; isn't that correct?
21 A. That's correct.
22 MS. GLINK: Based on his answers here today, we would
23 move to strike him as an expert with regard to any issue
24 regarding the items that have been developed in this test,
25 their validity, their predicted validity, and anything having
Jencks - voir dire
256
1 to do with the field of industrial psychology or psychometrics.
2 THE COURT: All right.
3 I think we have brought out all of the necessary
4 information from the witness. Rule 702 of the Federal Rules of
5 Evidence has, I guess, what has been characterized as a
6 relatively low level of requirement to qualify for testimony as
7 an expert. The language being, "If scientific, technical or
8 other specialized knowledge will assist the trier of fact to
9 understand the evidence or to determine a fact in issue, a
10 witness qualified as an expert by knowledge, skill, experience,
11 training or education may testify thereto in the form of an
12 opinion or otherwise."
13 I will find that for purposes of this witness'
14 testimony in this hearing for preliminary injunction that the
15 witness does qualify as an expert in the stated area, and that
16 the stated area is sufficiently relevant to the issues in the
17 case; that his testimony would be admissible.
18 So, I will -- I take it, with the offering of this
19 witness as an expert in what is called quantitative sociology,
20 assuming that that is an accepted area of sociology that
21 involves the use of statistics, regression analyses, analysis
22 of at least certain forms of tests, I would find that this
23 witness is an expert in that area and may give his opinion
24 pursuant to the Federal Rules of Evidence.
25 MR. FLAXMAN: Thank you, your Honor.
Jencks - direct
257
1 DIRECT EXAMINATION (Resumed)
2 BY MR. FLAXMAN:
3 Q. Now, in the course of your work in this case, did you have
4 occasion to review the 150-page -- 150-question short answer
5 written job knowledge test?
6 A. Yes, I did.
7 Q. And based on your review of that test of 150 questions, did
8 you --
9 THE COURT: One thing -- I am sorry to interrupt, but
10 I take it there has been some discussion with the witness about
11 the order that has been entered seeking to preserve the
12 confidentiality --
13 MR. FLAXMAN: I am not asking --
14 THE COURT: -- of the questions on the test?
15 MR. FLAXMAN: Yes.
16 THE COURT: And if there is a problem or a need at
17 some point to be specific as to an exact question, I think we
18 need to have a sidebar first to see how we are going to handle
19 it.
20 All right. You may proceed.
21 BY MR. FLAXMAN:
22 Q. Based on your review of those 150 questions, did you form
23 an opinion, based on your experience with race differences on
24 written tests, as to whether or not those 150 multiple-choice
25 questions were likely to have race differences when they're
Jencks - direct
258
1 taken by a group of different races?
2 A. I would certainly have expected to see a racial difference
3 in performance on those tests, yes.
4 Q. And why is that?
5 A. Because many other tests that -- of the same character --
6 that is, tests that measure knowledge and information about
7 specific areas and require protracted study of subject matter
8 materials in preparation for the examination -- have tended to
9 show a pattern of substantial racial differences.
10 Indeed, it's very difficult to find examples of tests
11 that don't show fairly sizable differences between blacks and
12 whites and their performance on an exam of this type.
13 Q. And does anybody know why there are these differences?
14 A. That's obviously an immensely controversial question.
15 There is not a short answer to that question.
16 Q. Did you also have occasion to look at the 60 questions that
17 were associated with the in-basket exercise?
18 A. Yes, I did.
19 Q. And those were 60 short answer questions, also?
20 A. They were.
21 Q. Did you form an opinion as to whether or not you would
22 expect the results on that test to have race differences?
23 A. I did form an opinion. And I would have expected to find
24 race differences, although not -- I wouldn't have been as
25 confident about that as I was with the first test.
Jencks - direct
259
1 Q. And why is that?
2 A. Because the character of the in-basket test is -- differs
3 more noticeably from the character of the large range of tests
4 from which I would be generalizing from than does the first
5 one. And the more dissimilar two tests are, the more hesitant
6 you are to make generalizations about the fact that the results
7 from one set of tests will carry over to another one.
8 Q. Now, did you also examine the oral component -- the written
9 -- the oral portion of the test?
10 A. I did.
11 Q. And did you form an opinion as to whether that was likely
12 to have a race impact?
13 A. I formed a much less confident opinion than on the other
14 two tests, I would say.
15 Q. Now, in the course of your work --
16 THE COURT: I do not find that he has been responsive
17 to the last two portions of it.
18 MR. FLAXMAN: Well --
19 THE COURT: I do not know what he means by he "formed
20 a less confident opinion." I do not know what his opinion is
21 and I do not know why he does not have any confidence in his
22 opinion.
23 MR. FLAXMAN: All right.
24 THE COURT: It is to the point where it is so vague, I
25 do not know what conclusion to draw so far from his testimony.
Jencks - direct
260
1 MR. FLAXMAN: I --
2 THE COURT: Okay.
3 BY MR. FLAXMAN:
4 Q. When you formed the less confident opinion about the oral
5 test, what was that opinion?
6 A. My guess about the oral test would have been that there
7 would have been some racial difference in performance, but it
8 was a guess based on very little information about other
9 comparable tests.
10 THE COURT: Again, I want to make it clear, Doctor,
11 that we do not want any guessing or conjecture in your
12 testimony, you know.
13 MR. FLAXMAN: Well, I am not offering -- I think I'll
14 -- well --
15 THE COURT: Okay.
16 BY MR. FLAXMAN:
17 Q. In the course of your work as a quantitative social
18 scientist, have you been called upon, other than testifying
19 here today, to offer opinions about things?
20 A. Yes.
21 Q. And have you been called upon to offer opinions about
22 questions for which you did not have research data?
23 A. Yes.
24 Q. And how would you answer those questions?
25 A. In general, you try to say what data you have and why you
Jencks - direct