555
1 IN THE UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
2 EASTERN DIVISION
3 ERNEST T. BROWN, et al., )
)
4 Plaintiffs, )
) No. 95 C 1890
5 v. ) Chicago, Illinois
) November 24, 1997
6 CITY OF CHICAGO, ) 9:30 a.m.
)
7 Defendant. )
8 VOLUME 4
9 TRANSCRIPT OF PROCEEDINGS - TRIAL
10 BEFORE THE HONORABLE ROBERT W. GETTLEMAN
11 APPEARANCES:
12 For the Plaintiffs: KENNETH N. FLAXMAN, P.C.
122 South Michigan Avenue
13 Suite 1850
Chicago, Illinois 60603-6107
14 BY: MR. KENNETH N. FLAXMAN
15 and
16 FUTTERMAN & HOWARD, CHTD.
122 South Michigan Avenue
17 Suite 1850
Chicago, Illinois 60603
18 BY: MR. CRAIG FUTTERMAN
19
For the Defendant: MAYER, BROWN & PLATT
20 190 South LaSalle Street
Chicago, Illinois 60603
21 BY: MR. JAMES HOLZHAUER
MR. JEFFREY S. PIELL
22 MS. ANGELA K. DORN
MR. ANDREW NICELY
23
Official Court Reporter: JENNIFER S. COSTALES, CSR, RMR
24 219 South Dearborn Street
Room 1744-A
25 Chicago, Illinois 60604
(312) 427-5351
556
1 (Proceedings heard in open court.)
2 THE CLERK: 95 C 1890, Ernest Brown versus City of
3 Chicago; on trial.
4 THE COURT: Are we ready?
5 MR. HOLZHAUER: Your Honor, we have one brief -- good
6 morning. We have one brief preliminary matter that Mr. Flaxman
7 and I --
8 THE COURT: Dr. Barrett, you can have a seat.
9 MR. HOLZHAUER: Sorry.
10 THE COURT: Go ahead.
11 MR. HOLZHAUER: I was informed after evidence closed
12 last week that the paralegal whose working with us on this
13 matter, one of the paralegals, Colleen Manley, is a good
14 personal friend of Katy Blakey, who I understand is a law clerk
15 of yours. I don't think that presents a problem. Mr. Flaxman
16 agrees, but I thought it would be --
17 THE COURT: Do you want me to fire Katy, is that it?
18 MR. HOLZHAUER: We're certainly not proposing
19 anything like that, sir.
20 MR. FLAXMAN: I doubt that the friendship will have
21 any impact on your decision in this case.
22 THE COURT: Some of my best former friends have
23 appeared before me as lawyers. I don't think those sorts of
24 things can get in the way. As long as they're not parties, I
25 don't have a problem.
557
Barrett - cross
1 MR. HOLZHAUER: Were they former friends before or
2 after they --
3 THE COURT: I chose my words carefully.
4 MR. HOLZHAUER: Thank you, Your Honor.
5 THE COURT: Thank you for informing me. I don't
6 think it's a problem. All right.
7 Dr. Barrett, good morning. You're still under oath.
8 GERALD BARRETT, DEFENDANT'S WITNESS, PREVIOUSLY SWORN
9 CROSS-EXAMINATION (Resumed)
10 BY MR. FLAXMAN:
11 Q. When we left, Dr. Barrett, we were talking about the
12 in-basket as I recall.
13 Was the in-basket exercise intended to measure
14 judgment?
15 A. To some degree, yes.
16 Q. Excuse me?
17 A. To some degree, yes, in terms of analysis of material.
18 Q. Was it intended to measure decision making?
19 A. Yes, they had to make decisions.
20 Q. Isn't it correct, Dr. Barrett, that a content validity
21 approach to test validation is not appropriate for a test which
22 is intended to measure judgment?
23 A. No, that's not correct.
24 Q. Isn't that what the EEOC guidelines say, Dr. Barrett?
25 A. No. I think you're misinterpreting the guidelines and the
558
Barrett - cross
1 standards and professional practice in the field.
2 Q. Well, the guidelines are written down, aren't they,
3 Dr. Barrett?
4 A. Yes.
5 Q. And as a matter of fact, they have been marked as an
6 exhibit, is that right?
7 A. That's correct.
8 Q. And I think they have been marked as exhibit number --
9 well, I don't know, 24 I think.
10 MR. FLAXMAN: Is that correct?
11 THE COURT: Whose exhibit 24?
12 MR. FLAXMAN: Defendant's exhibit 24 -- no. It's not
13 24. 27.
14 MR. PIELL: That's correct.
15 MR. HOLZHAUER: That's correct.
16 MR. FLAXMAN: Should I wait for Your Honor?
17 THE COURT: I have it.
18 MR. FLAXMAN: Okay.
19 BY MR. FLAXMAN:
20 Q. There is sections of EEOC guidelines that deal with content
21 validity, isn't that right?
22 A. Yes.
23 Q. And there is a section that deals with technical standards
24 for content validity, is that right?
25 A. Yes.
559
Barrett - cross
1 Q. And is there a sentence in the guidelines on page 208 of
2 the exhibit that says that, "A selection procedure based upon
3 inferences about mental processes cannot be supported solely or
4 primarily on the basis of content validity"?
5 A. There may be.
6 THE COURT: Can you direct my attention to that?
7 MR. FLAXMAN: Yes, page 208, where on the left column
8 there is a heading "Technical Standards for Content
9 Validation." Let me direct everyone's attention to the next
10 paragraph which starts out, "A selection procedure."
11 BY MR. FLAXMAN:
12 Q. Do you see that sentence that I just read, Dr. Barrett?
13 A. Yes.
14 Q. Have you found that?
15 A. Yes.
16 Q. Do you also see the sentence that says, "Thus, a content
17 strategy is not appropriate for demonstrating validity of
18 selection procedures which purport to measure traits or
19 constructs such as intelligence, aptitude, personality, common
20 sense, judgment, leadership and spatial ability"?
21 A. Yes.
22 Q. So your opinion, Dr. Barrett, is different than EEOC
23 guidelines about content validity measuring judgment, is that
24 right?
25 A. No, it's not.
560
Barrett - cross
1 Q. Okay. And we've established, you've agreed, don't you,
2 that the in-basket measured judgment, is that right?
3 A. Yes. Partly it measures decision making in that context,
4 in the context of specific content valid problems.
5 Q. Well, Dr. Barrett -- could I have the guidelines back?
6 Is it your testimony that the in-basket reflects
7 actual tasks performed by Chicago police lieutenants?
8 A. I'm saying that those are problems in the in-basket which
9 are representative of problems faced by Chicago police
10 lieutenants.
11 Q. Dr. Barrett, my question is: Was the in-basket intended to
12 reflect actual tasks performed by Chicago police lieutenants?
13 A. I'm not sure how you're defining "tasks." It was actual
14 work behavior problems which they face on the job.
15 Q. Well, Dr. Barrett, my question is: Was the in-basket
16 intended to reflect actual tasks performed by Chicago police
17 lieutenants?
18 A. As a simulation it was, a content valid simulation, yes.
19 Q. Dr. Barrett, can you answer my question?
20 A. Well, I'm trying to. See, I'm not sure what you are saying
21 in terms of it certainly was a simulation of the tasks, of the
22 work behavior of the Chicago police lieutenants.
23 Q. Dr. Barrett, my precise question is: Was the in-basket
24 intended to reflect the actual tasks performed by Chicago
25 police lieutenants?
561
Barrett - cross
1 A. It reflects some actual tasks performed by Chicago police
2 lieutenants.
3 Q. It was intended to reflect the actual tasks?
4 A. To some degree, yes. It's a simulation. I'm just saying,
5 you say "actual." I'm not sure what you're meaning. If we're
6 going out in the field in the context of it's not in the sense
7 of -- it is still a test.
8 Q. Well, so it wasn't intended to reflect the actual tasks, is
9 that what you are telling us?
10 A. It was reflected to be a content-valid test reflecting the
11 tasks which are performed by Chicago police lieutenants.
12 Q. Let me rephrase the question. Maybe it will help you.
13 Was the in-basket intended to approximate a realistic
14 work situation for a Chicago police lieutenant?
15 A. It was a simulation of realistic work situations, yes.
16 Q. Well, what do you mean, "simulation"? It means it was
17 different than what lieutenants really do, is that what you are
18 saying?
19 A. No. I'm saying that it is still a test, all right, with
20 all the limitations of a test. It's not actually walking out
21 into the field, if that's what you are implying, and going into
22 a patrol car and watching the behavior. It's not that sort of
23 thing. It is a simulation of the behavior and tasks.
24 Q. Well, when we talked about content validity as a typing
25 test, typing is an actual test done by typists, is that right?
562
Barrett - cross
1 A. That's correct.
2 Q. And that you could have a content-valid test if you give a
3 typing test to someone who is going to be a typist, is that
4 right?
5 A. Yes, you can give a content valid typing test to typists.
6 Q. Well, doesn't content validity require a close link between
7 the test content and the job content?
8 A. Yes. There should be a link between the two.
9 Q. And that to be content valid, the tests should consist of
10 representative samples of tasks, behaviors, or knowledge
11 actually used on the job, isn't that right?
12 A. Yes.
13 Q. And to be content valid, the job domain should be defined
14 through job analysis by identifying the important tasks,
15 behaviors, or knowledges of which the test is a representative
16 sample of, is that right?
17 A. Yes.
18 Q. That's not what the in-basket is, is it, Dr. Barrett? The
19 in-basket is a simulation?
20 A. It is a content-valid test which does simulate those
21 important activities which involve administrative skills of a
22 Chicago police lieutenant.
23 Q. Well, that's different than content validity as it's
24 understood in the SIOP guidelines, isn't that right,
25 Dr. Barrett?
563
Barrett - cross
1 A. No, that's not correct at all.
2 Q. Are you familiar with the SIOP guidelines?
3 A. Yes, I am.
4 Q. Isn't it true that the SIOP guidelines refer to what you
5 are defining as content validity as something that's treated as
6 construct validity?
7 A. No.
8 Q. It's not true?
9 A. Not true at all.
10 Q. Okay. Don't you believe that all validity is construct
11 validity?
12 A. I believe that the basis -- and this is certainly the
13 modern psychometric view -- that the basis of all of our tests
14 are first, identify some constructs in a general sense. And
15 that is reinforced by the standards, for example, on page 11,
16 where it says, "You should not make artificial distinctions
17 between content and constructs." You would go to Messick,
18 1989, which has --
19 MR. FLAXMAN: I move to strike the answer as
20 nonresponsive. I asked him, "Isn't it true that," and I'm
21 getting a lecture.
22 MR. HOLZHAUER: Well, I believe --
23 THE COURT: "Don't you believe" -- and I think he's
24 trying to tell you what he believes. You basically asked him
25 for his beliefs. I think you opened the door to this one.
564
Barrett - cross
1 MR. FLAXMAN: All right.
2 THE COURT: Do you want to finish? Were you
3 finished?
4 THE WITNESS: No, I'm not finished.
5 THE COURT: Go ahead.
6 BY THE WITNESS:
7 A. Messick, 1989, for example, has a treatise on construct
8 validity which takes the modern approach to the issue. And
9 clearly constructs underlie all of our tests.
10 And I wrote a paper in 1992 which is more specific to
11 the area of industrial psychology, also reiterating that point
12 of view. In question 81 of the EEOC guidelines, there is a
13 definition and a distinction which talks about
14 transportability, construct validity, and how these are
15 distinguished from criteria-related validity. And clearly
16 there are differences.
17 But one could argue, for example, based upon the EEOC
18 guidelines, based upon the past research, that I could defend
19 the in-basket we developed solely on construct validity based
20 on the past research.
21 BY MR. FLAXMAN:
22 Q. Doesn't construct validity require an empirical
23 relationship between test scores and actual performance?
24 A. I think, again, you should refer yourself to question 81 of
25 the EEOC guidelines questions and answers.
565
Barrett - cross
1 MR. FLAXMAN: Judge, could I have an answer to this
2 question?
3 THE COURT: Can you answer that yes or no?
4 BY MR. FLAXMAN:
5 Q. Doesn't construct validity require an empirical
6 relationship between test scores and job performance?
7 A. No.
8 Q. Doesn't construct validity require empirical data?
9 A. It's according, again, how you define empirical data and
10 how you define construct validity, because I think what you are
11 saying is that you have the same misconception Dr. York had.
12 He said you have to have a criterion-related study for that
13 job. That is not accurate. And this conforms to guidelines of
14 professional practice.
15 Q. Well, you keep saying that, Dr. Barrett.
16 Now, there aren't any questions in the EEOC
17 guidelines, are there?
18 A. In the questions and answers which came out in 1979 and
19 1980, they clarified the 1978 EEOC guidelines.
20 Q. So you're referring to questions and answers. You're not
21 referring to the EEOC guidelines when you talk about questions,
22 is that right?
23 A. Yes, I'm referring to the questions and answers which were
24 put out by the EEOC to clarify the 1978 guidelines.
25 Q. Now, let's -- are you familiar with the materials that made
566
Barrett - cross
1 up the in-basket?
2 A. In general, yes.
3 Q. Were you responsible for approving those before they were
4 given?
5 A. Yes.
6 Q. And did you conclude that the in-basket is representative
7 of important aspects of performance on the job for which police
8 lieutenants do?
9 A. Yes.
10 Q. Let me ask you to look at the in-basket test, which is
11 Defendant's Exhibit 15. If you turn to page, Bates stamped
12 page 08452, do you see the instructions that were given to the
13 person taking the in-basket?
14 A. Yes.
15 Q. And those instructions tell the person taking the test to,
16 "Assume that you're a watch commander who is taking over at
17 11:30 in the morning for another watch commander who became ill
18 at 11:00 o'clock," is that right?
19 A. Yes.
20 Q. Does your job analysis indicate how often that situation
21 occurs?
22 A. No.
23 Q. Do you have any personal knowledge of whether or not that
24 situation has ever occurred in the history of the Chicago
25 police department?
567
Barrett - cross
1 A. I've been told it has.
2 Q. You've been told it has?
3 A. Yes.
4 Q. Is that documented anywhere in your report that you've been
5 told it has?
6 A. No.
7 Q. Okay. Now, the instructions continue by saying that "You
8 can sort the in-basket materials in" -- there is another
9 instruction that "do not separate any stapled pages." Do you
10 see that? Have you found that, Dr. Barrett?
11 A. Is it on the next page?
12 Q. Yes.
13 A. Yes.
14 Q. Does your job analysis indicate that watch commanders are
15 not permitted to unstaple material -- unstaple materials when
16 they do their work?
17 A. No.
18 Q. Now, just to clarify, when this material was given to the
19 person taking the test, they didn't receive it in the form that
20 you have it, did they, Dr. Barrett? Some of the pages were
21 stapled together, is that right?
22 A. I don't recall exactly, but I assume so.
23 Q. Well, let me show you what I have one copy of, which I'll
24 call Plaintiffs' Exhibit 160. I'll put it in a folder.
25 MR. HOLZHAUER: Your Honor, we don't have a copy of
568
Barrett - cross
1 this as it is. We'll stipulate that things were stapled
2 together.
3 THE COURT: What is it?
4 MR. FLAXMAN: This is the actual in-basket that was
5 given to me in discovery. Actually, a duplicate original that
6 was actually administered.
7 THE COURT: This is Defendant's Exhibit 16 as it
8 existed at the time the test was given?
9 MR. FLAXMAN: That's correct.
10 THE COURT: Is that accurate, Mr. Holzhauer? Take a
11 look at it.
12 MR. HOLZHAUER: Can I look at it, please?
13 MR. FLAXMAN: Sure.
14 THE COURT: Let's make sure it's the right thing.
15 (Pause.)
16 MR. HOLZHAUER: It appears to be correct, Your Honor.
17 THE COURT: Okay.
18 BY MR. FLAXMAN:
19 Q. Let me show you Plaintiffs' Exhibit 160, which we've now
20 agreed is the in-basket that was actually administered. There
21 are some packets in there that are stapled together, is that
22 right?
23 A. Yes.
24 Q. And if you look through it, you see there is some scratch
25 paper, is that right?
569
Barrett - cross
1 A. Yes.
2 Q. Then you get to page 7. There is a to/from report, is that
3 right?
4 A. Yes.
5 Q. And did anything in your job analysis indicate that watch
6 commanders start their tour of duty with files in an in-basket?
7 A. Not in that format, no.
8 Q. Well, that's a yes or no question, Dr. Barrett.
9 Did anything in your job analysis indicate that watch
10 commanders start their tour of duty with files in an in-basket?
11 MR. HOLZHAUER: Asked and answered, Your Honor. He
12 said no.
13 THE COURT: I think he said no.
14 MR. FLAXMAN: Thank you.
15 THE COURT: Is that what you meant?
16 THE WITNESS: Yes.
17 BY MR. FLAXMAN:
18 Q. Now, in those materials, there are two weapons discharge
19 investigations for the watch commander. Do you see that?
20 A. What page are you on now? Oh, page 10?
21 Q. Well, it's the to/from on page 7 of the material.
22 A. This seems to be out of order. This is out of order. I
23 understand. Yes, I see. You're talking about page 7, the
24 first one, weapons discharge investigations.
25 Q. Right. That gives -- now, that's on Bates stamp 8462 of
570
Barrett - cross
1 the exhibit, I believe. That assigns six major tasks that the
2 watch commander has to complete, is that right?
3 A. Yes.
4 Q. Is there anything in your job analysis that indicates that
5 watch commanders do two weapons discharge investigations on a
6 daily basis?
7 A. No.
8 Q. And if we look further on that sheet, there is an activity
9 called scheduling. Do you see that, No. 6?
10 A. Yes.
11 Q. Does anything in your job analysis indicate that watch
12 commanders do scheduling without clerical assistance?
13 A. No.
14 Q. And to answer the questions that were later asked on the
15 in-basket, the person taking the test had to do scheduling
16 without clerical assistance, is that right?
17 A. That's correct.
18 Q. And you were trying to measure the ability to do
19 scheduling, is that right?
20 A. Or supervise scheduling, yes.
21 Q. You're trying to measure supervisory skills, is that right?
22 A. Supervisory behavior would be more accurate probably.
23 Q. Let's look at page 16 of the materials which is Bates
24 stamped 8471. Do you see that? That's a witness statement
25 from Paul Phillips?
571
Barrett - cross
1 THE COURT: What was the page?
2 MR. FLAXMAN: 8471.
3 BY MR. FLAXMAN:
4 Q. Have you found that?
5 A. Yes.
6 Q. Does anything in your job analysis indicate that watch
7 commanders review witness statements in this format?
8 A. In this format?
9 Q. That's correct, Dr. Barrett.
10 A. I'm not sure of the exact format. But they do review
11 witness statements.
12 Q. Anything in your job analysis indicate that this is the
13 format of Bates stamp 8471 in which watch commanders review
14 witness statements?
15 A. I don't recall anything specific in my job analysis, my job
16 description that says that, no.
17 Q. As a matter of fact, this isn't the form as used in the
18 Chicago Police Department, is it, Dr. Barrett?
19 A. I don't recall.
20 Q. Now, let me ask you to go on to page 20, which is Bates
21 stamped 8475, that's called "Lieutenant Roberta Joseph's
22 notes." Have you found that, Dr. Barrett?
23 A. Page 20, yes.
24 Q. Anything in your job analysis indicate that watch
25 commanders maintain notes in this type of format?
572
Barrett - cross
1 A. This type of format, no.
2 Q. As a matter of fact, this format was made up for your
3 simulation test, is that right?
4 A. We used this for our simulation, that's correct.
5 Q. Now, to take the in-basket, the person had to review the
6 materials and then answer multiple-choice questions, is that
7 right?
8 A. Yes.
9 Q. And there were 60 multiple-choice questions?
10 A. Yes.
11 Q. And you had 90 minutes to do those, is that right?
12 A. Yes.
13 Q. Do you have -- now, if you look at that exhibit, I think
14 you'll find the questions. They're in a separate packet. And
15 they start at Bates stamp 8403 of the exhibit. Let me ask you
16 to look at --
17 A. I'm sorry. I'm still -- I have the original.
18 Q. Well, if you look through the original, I think you'll find
19 the exact format in which the questions exist.
20 THE COURT: 8403?
21 MR. FLAXMAN: 8430.
22 BY MR. FLAXMAN:
23 Q. If you'd like me to find it for you, Dr. Barrett, I'll be
24 glad to point it out.
25 A. Yes, sure.
573
Barrett - cross
1 Q. I'm not pointing it out. Let me show you 8430. Do you see
2 question 3?
3 A. Yes.
4 Q. Anything in your job analysis which indicate that Chicago
5 watch commanders use language of "a discrepancy among the five
6 statements which may impact the investigation"?
7 A. No.
8 Q. You don't know, do you, whether or not that language is
9 used by Chicago police lieutenants, do you?
10 A. That specific language, no.
11 Q. You don't know whether -- that's not -- as far as you know,
12 that's not Chicago police jargon to determine whether there is
13 "a discrepancy among statements which may impact an
14 investigation"?
15 A. I don't recall anything like that, no.
16 Q. Okay. And that same language about "a discrepancy which
17 may impact the investigation," that was used in question 6 on
18 Bates stamp 8431, is that right?
19 A. I don't know.
20 Q. Let me show you question 6.
21 A. Yes, "discrepancy" is a term, "discrepancy" is in question
22 6, that's correct.
23 Q. And "a discrepancy which would impact the investigation,"
24 is that the phrase that was used in question 6?
25 A. No.
574
Barrett - cross
1 Q. What phrase was used about discrepancy, Dr. Barrett?
2 A. "There is discrepancy in the witness statements regarding."
3 Q. Do you know if Chicago police lieutenants talk about
4 discrepancies?
5 A. No.
6 Q. Let me show you question 11, Bates page 8433. Is that a
7 question which asked again about "a discrepancy which would
8 impact an investigation"?
9 A. Yes. It says "a major discrepancy" on item 11.
10 Q. Then the same thing as to question 13, about another
11 "discrepancy which would impact an investigation"?
12 A. Yes, number 13 says "a major discrepancy."
13 Q. Now, so this isn't the kind of language that Chicago police
14 watch commanders deal with, is that right?
15 A. I said I didn't know. It doesn't seem an usual term to me,
16 but I don't have any direct knowledge of the use of that term.
17 Q. As a matter of fact, you don't have any direct knowledge at
18 all of what's important for a Chicago police lieutenant to
19 know, do you, Dr. Barrett?
20 A. That's not correct.
21 Q. Okay. Well, let me ask you to look at question 47 which is
22 on Bates page 8445. Have you had a chance to look at that?
23 A. Yes.
24 Q. You don't have any personal knowledge, do you, Dr. Barrett,
25 that Chicago watch commanders actually make the kind of
575
Barrett - cross
1 decision that's asked for in question 47?
2 A. From our job analysis and job description, we know that
3 they do review that material, supervise that material, and they
4 may, in fact, make decisions based upon this material.
5 Q. They may, in fact, Dr. Barrett? Is that your testimony?
6 A. Yes.
7 Q. So as far as you know, all those decisions, all that work
8 is done by a watch secretary, is that right, by a clerical
9 person?
10 A. I'm not saying that.
11 Q. Well, you don't know whether or not it's done by the watch
12 commander or by a civilian personnel, do you?
13 A. I know it's all reviewed by the watch commander.
14 Q. Now, does that question ask the watch commander to review
15 something? What does question 47 say?
16 A. "According to Lieutenant Joseph's notes, Sergeant Woodward
17 has requested one day compensatory time in December to visit
18 her mother. The best day for Sergeant Woodward's day off is
19 blank December '94."
20 Q. Now, as far as you know, no watch commander has ever made
21 that kind of decision in the first instance in the history of
22 the Chicago Police Department?
23 A. I'm not -- that exact decision?
24 Q. That's right.
25 A. No, probably not. I don't have any knowledge of that.
576
Barrett - cross
1 Q. How about question 48? What's question 48, Dr. Barrett?
2 A. "In order to satisfy the personnel needs for security at
3 the Kraft Festival on 13 December '94, one of the day off
4 requests by the officers would need to be denied. Request of
5 Officer Blank should be denied first."
6 Q. And, again, you don't have any data from your job analysis
7 to show that watch commanders make that kind of decision in
8 their work as watch commanders?
9 A. We have data they, in fact, review decisions and may make
10 this decision, yes.
11 Q. And may make this decision?
12 A. Yes.
13 Q. What data do you have that shows they may make this
14 decision?
15 A. I don't have any exact data that I can point to in terms of
16 our job analysis in terms of saying this is some statement that
17 they made that certain decision on a certain day, no.
18 We do have reviews by subject matter experts, though,
19 which indicated that was the sort of behavior which watch
20 commanders did and were involved in.
21 Q. Was Mr. Klein one of those subject matter experts?
22 A. Yes.
23 Q. Did you ask Mr. Klein whether or not watch commanders make
24 the kind of decision that's called for in question 48?
25 A. I don't recall an exact question. We ask in general, and
577
Barrett - cross
1 we went through each and every item.
2 Q. Did you ask him specifically about the questions that were
3 used on the test, the in-basket test?
4 A. Yes.
5 Q. And did he tell you specifically that watch commanders
6 answer the kind of question that's asked for in question 48?
7 A. I don't recall that specific question, no.
8 Q. Well, was Commander Cadogan another one of the subject
9 matter experts who reviewed the in-basket?
10 A. You mean Chief Cadogan?
11 Q. That's right.
12 A. Yes, he reviewed the in-basket.
13 Q. Did you ask Chief Cadogan whether watch commanders did the
14 kind of scheduling that's called for in question 47?
15 A. We asked in general if the behavior we're testing for was
16 important and relevant.
17 Q. Did you ask him --
18 A. I don't recall exactly asking that specific question, no.
19 Q. Did you ask him specifically whether or not this is the
20 kind of thing that a lieutenant does?
21 A. I don't recall asking that specific question, no.
22 Q. I've asked you the same about Mr. Shaw. Did you ask him
23 specifically whether question 47 is the kind of thing that a
24 lieutenant does? Would your answer be the same, "I don't
25 recall"?
578
Barrett - cross
1 A. No. I don't recall asking specifically about 47, no.
2 Q. Now, after the in-basket was administered, you had an
3 opportunity to look at the results, is that right?
4 A. Yes.
5 Q. And you saw what percent got each question right, is that
6 right?
7 A. Yes.
8 Q. And some of the questions 90 percent of the people taking
9 the test got right?
10 A. I don't recall the exact figures, but it sounds accurate.
11 Q. And some of the questions less than 50 percent of the
12 people taking the test got right, is that right?
13 A. I assume that's correct, yes.
14 Q. Do you know that 30 percent of the people taking the test
15 got question 48 correct, 70 percent got it wrong?
16 A. No, I didn't know that. Well, I shouldn't say I didn't
17 know it. I may have known it at one point in time. Sitting
18 here today, I don't know that.
19 Q. I think there is a stipulated exhibit which shows that.
20 Now, your in-basket used an objective scoring system,
21 is that right?
22 A. Yes.
23 Q. And the scoring was based on how many questions somebody
24 answered correct on a multiple-choice test, is that right?
25 A. Yes.
579
Barrett - cross
1 Q. There was no penalty for guessing, is that correct?
2 A. Yes, that's correct. There was no penalty for guessing.
3 Q. You told us, I think when we broke last week, that your
4 objective scoring system was based on research that had been
5 reported by a man named Lopez in 1962, is that right?
6 A. Well, that's not quite accurate. I think I referenced
7 Lopez as a research article, research which in fact did use an
8 objective technique. I'm not saying I used his technique, I
9 don't believe.
10 Q. So you just cited it because he used objective scoring, is
11 that -- is that what your testimony is, Dr. Barrett?
12 A. Well, I don't recall your exact question, but I think there
13 was a discussion about traditional, objective and other types
14 of in-baskets.
15 And I just mentioned the fact that there is a long
16 history of psychometric, objective type scoring systems for
17 in-baskets. I think that's what I recall I said.
18 Q. Well, are you telling us that Lopez reported an objective
19 scoring system for an in-basket that was similar to the one
20 that you used in the 1994 police lieutenants test?
21 A. I'm not sure how you are using the word "similar," because
22 I'm sure, or as I recall, he used a variety of objective
23 in-baskets.
24 In fact, he reported that one of his concerns was
25 that the more mechanical approach might turn people off, like
580
Barrett - cross
1 vice-presidents. As I recall, he said even vice-presidents
2 accepted the more objective approach as being appropriate. So
3 I'm not sure what you are asking, I guess.
4 Q. Well, I'm asking you, Dr. Barrett, is your scoring system
5 based on anything that Felix M. Lopez reported back in the
6 1960s?
7 A. I'm not sure if it's based upon. I'm saying that, yes, he
8 used an objective approach also, which was computer scored as I
9 recall. If that's what you are asking, there is that
10 similarity.
11 Q. Let me take that back.
12 Let me show you what I'll mark as Plaintiffs' Exhibit
13 161. Is this the Lopez article that you are referring to,
14 Dr. Barrett?
15 A. Yes.
16 Q. Now, that's not an article that appeared in a peer-reviewed
17 journal, is it, Dr. Barrett?
18 A. No. This appeared as basically a technical report put out
19 by the American Management Association.
20 Q. Now, when you talked about the in-basket test, you said it
21 was a simulation of work that was performed by lieutenants, is
22 that right?
23 A. Yes.
24 Q. Do you agree with the Lopez definition of simulation as "an
25 operating representation of central features of reality"?
581
Barrett - cross
1 A. That could be one definition. I don't recall it, what he
2 said.
3 Q. Well, let me ask you to look at page 14 and see if you can
4 find where it says, "Social scientists define simulation simply
5 as an operating representation of central features of reality"?
6 A. I see "'Simulation,' 'role playing' and 'game' are terms
7 often used interchangeably in the literature to denote various
8 types of training and testing methods."
9 Is that what you mean?
10 Q. Yeah. Then he says, "Social scientists define simulation
11 simply as an operating representation of central features of
12 reality."
13 A. Yes, I see that.
14 Q. Now, is that -- do you agree with that definition of
15 simulation, Dr. Barrett?
16 A. I don't see it's -- yeah, I don't see anything wrong with
17 that definition particularly. It's his definition.
18 Q. And the in-basket was a simulation, is that right?
19 A. Sure. Yes, it was.
20 Q. And the oral exercise was also a simulation, isn't that
21 right.
22 A. Yes.
23 Q. Now, Mr. Lopez defines an in-basket at page 17 where he
24 says, "The in-basket exercise presents the participant with the
25 hypothetical work situation in which he just makes decisions on
582
Barrett - cross
1 a series of letters, memos, and other documents deposited as
2 incoming mail on his in-basket."
3 Do you see that?
4 THE COURT: What?
5 BY THE WITNESS:
6 A. No.
7 THE COURT: What page was that?
8 MR. FLAXMAN: Page 17.
9 THE COURT: Where on the page?
10 MR. FLAXMAN: I'm short one copy.
11 THE COURT: You don't have a copy?
12 MR. HOLZHAUER: The last paragraph of the second
13 column.
14 THE COURT: The last paragraph.
15 MR. HOLZHAUER: Second to last, the last full
16 paragraph, it starts with "the in-basket exercise," and you
17 split the last sentence, it starts about six lines from the
18 bottom of that paragraph, the words ETS.
19 THE COURT: I have it. Thank you.
20 BY MR. FLAXMAN:
21 Q. Have you found that, Dr. Barrett?
22 A. I find -- the one you're talking about ETS, that paragraph?
23 Q. "The in-basket exercise presents the participant with the
24 hypothetical work situation in which he just makes decisions on
25 a series of letters, memos and other documents deposited as
583
Barrett - cross
1 incoming mail in his in-basket."
2 A. Yes, I see that.
3 Q. That's not what happened in your in-basket for the 1994
4 lieutenants test, did it, Dr. Barrett?
5 A. Let me see now. It was a hypothetical work situation.
6 Q. Could you just --
7 A. That is true. You're asking me, I assume, the parts say
8 does this correspond with what we did? Yes. It was a
9 hypothetical work situation, our in-basket. Do you make
10 decisions? Yes. And they were based upon written
11 documentation. That's true.
12 Q. So in your opinion answering the multiple-choice questions
13 is the same as making decisions on a series of letters, memos
14 and other documents as in the in-basket that Mr. DeLopez --
15 Mr. Lopez is describing, is that what you are telling us?
16 A. I'm saying they could not answer the questions without
17 making decisions. Yes, they made decisions based on that
18 material. If they didn't have the material, they could not
19 make decisions.
20 Q. Now, Dr. Barrett, are you telling us that it is your
21 opinion that answering those 60 multiple-choice questions on
22 your in-basket is the same as what Mr. Lopez is describing at
23 page 17, the excerpt that I just read? Is that what you are
24 telling us?
25 A. What I've read so far, it seems to be the same.
584
Barrett - cross
1 Q. And do you remember Mr. Lopez describes an in-basket
2 exercise with lieutenants in the Port Authority Police
3 Department, is that right?
4 A. Yes.
5 Q. Is it your opinion that the in-basket for those lieutenants
6 was the same as your 60 multiple-choice question in-basket?
7 A. I've never seen the actual Port Authority police lieutenant
8 in-basket, so I would have to see it to compare. I don't know
9 exactly what --
10 Q. Well, isn't it described in Mr. Lopez's article,
11 Dr. Barrett?
12 A. It's described in general, yes.
13 Q. And it's not at all answering 60 multiple-choice questions,
14 is it, Dr. Barrett?
15 A. I don't recall how many questions they had to answer.
16 Q. Do you recall how long, how much time there was to do the
17 Port Authority in-basket?
18 A. No, I don't.
19 Q. Do you recall what format the answers took for the Port
20 Authority lieutenant test, in-basket test?
21 A. Again, it's been some time since I read it, but I think
22 they had to mark the reasons for the various actions, I
23 believe. But I could be mistaken.
24 Q. They had to complete a "reasons for action report,"
25 Dr. Barrett, isn't that correct?
585
Barrett - cross
1 A. Well, again, as I said, that's what I recall. They had to
2 indicate what were the appropriate actions to be taken.
3 Q. They had to explain what action they would take on
4 particular, on a particular situation, is that right?
5 A. No.
6 Q. Excuse me?
7 A. No.
8 Q. They didn't have to explain what action they took for each
9 piece of work and why they did it?
10 A. No.
11 Q. That's, that's -- okay.
12 A. I'm using the word "explain" to say they had made no verbal
13 explanation or written explanation. They just marked, "this is
14 the explanation I use for this decision," as I recall. So I'm
15 saying I think you're taking it out of context what they
16 actually did.
17 Q. Let me ask you to look at page 75 of the Lopez report.
18 There is some material I think on the right-hand column which
19 is quoted from one of the people who took the test where he's
20 describing his experiences on the test. Have you found that,
21 Dr. Barrett?
22 A. You're on page -- it says, "like a group of doomed men who
23 proceeded into the testing room." Is that what you are
24 referring to?
25 Q. Yes.
586
Barrett - cross
1 Do you see where it says, "We also had to explain
2 what action we took with each piece of work and why we did so"?
3 THE COURT: Where are you?
4 MR. FLAXMAN: There is a right-hand column. There is
5 excerpted material.
6 THE COURT: Where are you in that?
7 MR. FLAXMAN: If I could have somebody's copy, I'll
8 --
9 THE COURT: I see it.
10 MR. FLAXMAN: All right.
11 THE COURT: Second to the last paragraph, the last
12 sentence.
13 BY MR. FLAXMAN:
14 Q. Do you see that, Dr. Barrett?
15 A. Yes. I see, "After three strenuous hours during which --
16 three strenuous hours," this is what you are referring to?
17 Q. No. I'm referring to the sentence, "We also had to explain
18 what action we took with each piece of work and why we did so."
19 A. Yes, I see that sentence.
20 Q. In your in-basket for the 1992 lieutenants test, the only
21 explanation that the people taking the test could make was
22 whether -- which choice to make on the 60 questions, is that
23 right?
24 A. I'm sorry. What's your question?
25 Q. On your in-basket, the in-basket simulation, there was no
587
Barrett - cross
1 place to write why a particular applicant thought a question
2 was making a particular choice, is that right?
3 A. That's correct.
4 Q. That's different than the lieutenants test for the Port
5 Authority described in the Lopez article, isn't that right?
6 A. Well, again, I think I have to read it in context and go to
7 page 74. I think they might have been -- this might have been
8 the situation where they were comparing both the more objective
9 form and the more subjective form and showing of the same.
10 So if you start on page 74, I think you'll see, in
11 fact, at the top of page 75, they talk about the 47 problems
12 that they had to solve. And so I think what it could be would
13 be the fact that they're comparing the two. I don't know. I'd
14 have to read it in context, because what they did is show that
15 in fact we could do a rating or subjective approach, but you
16 would obtain the same results if you would just check off here
17 are the reasons for actions after they made a decision. Your
18 more objective approach which could be computer scored. So
19 again, I'd have to read the whole report in context, but I
20 think that's what he's referring to.
21 Q. Well --
22 A. I'd have to read on.
23 Q. Well, why don't you read on, Dr. Barrett, to page 79.
24 A. Okay.
25 Q. Where there is -- page 79 to 80. Do you see the
588
Barrett - cross
1 description there of the objective in-basket and a reference to
2 an 867-item questionnaire?
3 THE COURT: Where are you?
4 MR. FLAXMAN: I knew I needed one more copy.
5 BY MR. FLAXMAN:
6 Q. Do you see on page 80 underneath the horizontal line, the
7 third line down, by filling out the in-basket action report, an
8 867-item questionnaire?
9 A. Yes, I see. I believe he described his handling of the
10 problem by filling out the in-basket action report, an 867,
11 867-item questionnaire.
12 Q. That's different than a 60-question multiple-choice test,
13 isn't it, Dr. Barrett?
14 A. Yes. That's probably much more difficult.
15 Q. It's different, isn't it, Dr. Barrett?
16 THE COURT: Don't raise your voice, please.
17 MR. FLAXMAN: I'm sorry.
18 MR. HOLZHAUER: Asked and answered. He said yes.
19 MR. FLAXMAN: All right.
20 BY MR. FLAXMAN:
21 Q. And if you keep looking at page 80, do you see where it
22 says, "If a breakthrough had occurred in the scoring
23 procedures, it may be possible to develop in-baskets for other
24 positions and perhaps to use it to evaluate the effectiveness
25 of management training programs"?
589
Barrett - cross
1 A. Yes.
2 Q. Lopez wasn't saying he had made a breakthrough, was he,
3 Dr. Barrett?
4 A. He's saying what he's saying.
5 Q. All right. And if you go on in the Lopez, go to page 94
6 where he talks about another Port Authority test. Do you see
7 where he says, "The scoring system was a modified version of
8 the one developed in earlier research efforts. It consisted of
9 an analysis of two separate variables: One, the actual content
10 of the response, including the propriety and accuracy of the
11 actions taken by the candidate and his knowledge of the rules,
12 regulations, and technical aspects of police work; and, two,
13 the style with which he performed his actions, such as the way
14 he communicated, organized his work, made decisions and
15 delegated"?
16 Do you see that, Dr. Barrett?
17 A. No, I'm sorry. I didn't catch exactly -- which column are
18 you in, the first or second column?
19 Q. I'm on the left column of page 94.
20 A. Left column, okay.
21 Q. The first full paragraph.
22 A. First full paragraph, okay.
23 Q. Starting with the second sentence, "The scoring system"?
24 A. "As administrating of the test," is that what you're
25 concerned, that paragraph?
590
Barrett - cross
1 Q. The scoring system utilized in the police lieutenant
2 examination was a modified version?
3 A. Yes.
4 Q. Now, why don't you just read that sentence and the next
5 sentence?
6 A. "The scoring system utilized" --
7 Q. You can read it to yourself.
8 (Pause.)
9 BY MR. FLAXMAN:
10 Q. Have you had a chance to read that?
11 A. Yes.
12 Q. That's not the way the 1994 in-basket for the Chicago
13 police lieutenant was scored, is it, Dr. Barrett?
14 A. It's not the way it was scored, that's correct.
15 Q. And in your -- the report that you wrote about the 1994
16 lieutenants test, do you recall in appendix Q you refer to some
17 other scholarly works about in-baskets?
18 A. I guess I recall we may have discussed these issues.
19 Q. Well, let me direct your attention to page 321 of
20 Defendant's Exhibit 1, appendix Q, page 4. Do you see that,
21 Dr. Barrett?
22 A. Yes.
23 Q. And that was a reference to some earlier work that had been
24 done with in-baskets, is that right?
25 A. Yes. This is part of our weighting rationale report.
591
Barrett - cross
1 Q. And that earlier work with in-baskets, some of it achieved
2 something -- the validity coefficient that we talked about
3 before, is that right?
4 A. Yes.
5 Q. What's the first in-basket study that you refer to in that
6 appendix? I think it's Q-4?
7 A. Q-4 -- well, table 1, number 5, the correlation between the
8 in-basket and job performance?
9 Q. Right.
10 Well, do you refer -- could you tell us what the
11 first in-basket study is that you are referring to?
12 A. We have there Bentz 1968.
13 Q. Was the Bentz 1968 in-basket a 60-multiple-choice-question
14 test?
15 A. No.
16 Q. Was the Bentz in-basket what Dr. York referred to as the
17 "traditional in-basket"?
18 A. As I recall, it was more traditional, yes.
19 Q. Excuse me?
20 A. I believe it was.
21 Q. Is that one of those tests where the individual was asked
22 to assume that he had just been appointed manager of a store,
23 the previous manager had died, and the new manager had to come
24 in and perform many tasks? Is that the Bentz in-basket?
25 A. It sounds like the format that John Bentz used to use, yes.
592
Barrett - cross
1 Q. Okay. That's different than the format that Gerald Barrett
2 used in 1994 in the City of Chicago for promotions to
3 lieutenant, correct?
4 A. The instructions are almost identical.
5 Q. Excuse me?
6 A. The instructions are almost identical in terms of the
7 format. Just take the word "lieutenant" and substitute "store
8 manager," and it would be identical instructions.
9 Q. Well, was the Bentz in-basket graded by answering
10 multiple-choice questions?
11 A. I don't believe it was.
12 Q. As a matter of fact, it was graded by looking at the way in
13 which the work was performed, isn't that right?
14 A. I don't recall exactly now, but it could be true.
15 Q. Okay. Well, is there another -- there is another in-basket
16 study that you referred to in that report, is that right?
17 A. Yes.
18 Q. And what's the next one?
19 A. Kesselman 1982.
20 Q. Was the Kesselman 1982 the same type of in-basket as used
21 in the 1994 police lieutenants exam?
22 A. I need to see the exact reference. It's not on this paper,
23 so I can't --
24 Q. So as you sit here now, you don't know whether or not it
25 was, is that right?
593
Barrett - cross
1 A. Well, the reason I'm saying this is because I believe the
2 Kesselman articles appeared in Public Administration Quarterly,
3 I think it might have been, which did use an objective
4 in-basket.
5 And if this is the same one -- again, I'm not sure, I
6 don't recall right now, because it's been some time, but --
7 Q. Well, did you also rely on an in-basket study by Wallowick?
8 A. Yes. Is that one of those 60 multiple-choice
9 objectively-graded in-baskets?
10 A. I believe that was the IBM research. And I don't think
11 they did use objectives.
12 Q. Excuse me?
13 A. I don't believe they did at IBM use an objective in-basket
14 at that point in time.
15 Q. As a matter of fact, it wasn't an objective in-basket. It
16 was what Dr. York referred to as the traditional in-basket, is
17 that right?
18 A. As I recall, I think that's correct.
19 Q. Do you recall that Mr. Lopez in that 1960 article described
20 objective scoring "as quite promising but not overwhelmingly
21 successful"?
22 A. No.
23 Q. Do you still have it in front of you, Dr. Barrett?
24 A. Yes.
25 Q. Let me ask you to look at page 111 right at the top. Do
594
Barrett - cross
1 you see where it says, "In-basket exercise was deemed to be
2 quite promising"?
3 A. Yes.
4 It starts on page 110. It says, "This first approach
5 to group scoring of an in-basket exercise was deemed to be
6 quite promising, although not overwhelmingly successful."
7 Q. Thank you.
8 Now, do you agree that paper-and-pencil tests --
9 well, do you agree that the in-basket was a paper-and-pencil
10 test?
11 A. No.
12 Q. Excuse me?
13 A. No.
14 Q. Do you agree that the written job knowledge test was a
15 paper-and-pencil test?
16 A. Yes.
17 Q. Okay. Was reading comprehension important in doing the
18 in-basket?
19 A. I'm not sure what you mean by "important." But you had to
20 be able to read and comprehend the material in the in-basket,
21 yes.
22 Q. Well, would you agree that someone with better reading
23 comprehension would do better on that in-basket test?
24 A. I haven't done any studies that show it to be true. It was
25 written at the eleventh grade level, the instructions and the
595
Barrett - cross
1 question. So if the person was a high school graduate, that
2 would be adequate reading comprehension to perform the task.
3 So I'm not sure how much better they would do with better
4 reading comprehension.
5 Q. Well, you told us, I think, that you believe that if you
6 had more time to do the in-basket, you'd do better on it?
7 A. I don't think there is any question that since it's an
8 open-book test and the fact that if you have an unlimited
9 amount of time, you could get all of the 60 questions correct.
10 Q. And you can't -- you don't have any opinion at all about
11 whether reading comprehension, good reading comprehension would
12 help you do that test in the 90 minutes, answer those 60
13 questions?
14 A. Well, I said I've done no empirical study on that. But I
15 did say that I have -- we did check the reading level. It's
16 eleventh grade reading level.
17 Q. Well, you've done no empirical study on whether the final
18 scores predict performance as a police lieutenant, isn't that
19 right?
20 A. I said that we had accomplished no criterion-related study.
21 I didn't say we didn't do any empirical study.
22 Q. Now, the written-multiple-choice test was a job knowledge
23 test, is that right?
24 A. Yes.
25 Q. And do you have any evidence from your job analysis that
596
Barrett - cross
1 somebody who can answer 150 questions on that test correctly
2 would do better as a lieutenant than somebody who can answer
3 149?
4 A. Yes.
5 Q. Do you have any -- and do you have any data that shows
6 that, Dr. Barrett?
7 A. Well, I think not only from the job analysis, but also what
8 we did was to go back and for the challenged items of
9 Mr. Bishop, we devised our flow charts, which shows quite
10 graphically both the complexity of the knowledge required for
11 the police lieutenant's job in order to solve these problems
12 successfully.
13 Q. Maybe you didn't understand.
14 A. Okay.
15 Q. I'm sorry. Maybe you didn't understand my question.
16 Do you have any empirical evidence that knowing the
17 correct answers to 150 questions is correlated with better
18 performance as a police lieutenant?
19 A. Mr. Flaxman, again, you are saying correlated is empirical.
20 If that's what you are saying, if you're asking is there a
21 criterion-related study, I'm saying we have no
22 criterion-related study on the job knowledge test.
23 Q. I'm asking you about empirical evidence.
24 A. I'm saying, yes, we have empirical evidence from the job
25 analysis and from the flow charts.
597
Barrett - cross
1 Q. And the job analysis is this judgment that you're making,
2 is that right?
3 A. It's observation of behavior, observation of interviews.
4 It's -- it is evidence which we can use.
5 Q. Now, you told us that you don't have a criterion-valid
6 study or you didn't try to do criterion validation, is that
7 right?
8 A. I said we did not do a criterion-related study, that's
9 correct.
10 Q. And if you had done a criterion-related study you would
11 have ended up with a validity coefficient, is that right, or
12 number?
13 A. That's correct.
14 Q. And you briefly said on direct that it was impossible to do
15 a criterion-valid study, is that right?
16 A. I'm not sure I used the word "impossible." I'm not sure I
17 used that term.
18 Q. Well --
19 A. I might have said it was not feasible. There are many ways
20 that you could do it. I think I might have said if you did do
21 it, under even the best conditions, you would compromise the
22 test. But I'm not sure I said impossible.
23 Q. Well, you said it's not feasible and would compromise the
24 test? Did you propose to the City of Chicago that you
25 validate -- that you prepare a promotional test on a
598
Barrett - cross
1 criterion-valid theory?
2 A. I did not propose to the City of Chicago that I would do a
3 criterion-related study, that's correct.
4 Q. Did you consider doing a criterion-valid study?
5 A. Yes.
6 Q. And why did you not do a criterion-valid study?
7 A. I didn't do a criterion-related study for a number of
8 reasons. There are two types of criterion-related studies.
9 One is called predictive. To do a predictive study, I would
10 had to have -- what is called a perfect predictive study,
11 randomly selected sergeants --
12 THE COURT: I think we've been into this. I've heard
13 this before.
14 MR. FLAXMAN: I didn't ask the question.
15 THE COURT: Well, you asked him why he didn't do it.
16 Now he's telling you. I think he's already said that once.
17 MR. FLAXMAN: Well, he didn't tell you why he
18 couldn't do it on cross-examination. He gave his explanation
19 of predictive and -- but he didn't --
20 MR. HOLZHAUER: I believe he did once on direct and
21 once on cross.
22 MR. FLAXMAN: I don't think I've done this on cross.
23 I was saving this for the end.
24 THE COURT: Well, since you say it's the end, I'm
25 going to let you do it.
599
Barrett - cross
1 MR. FLAXMAN: All right.
2 BY MR. FLAXMAN:
3 Q. Without giving us the explanation that you were just giving
4 us, why did you not attempt to do a criterion-validated study
5 for the City --
6 MR. HOLZHAUER: Object, Your Honor. I don't think he
7 can instruct him not to give the explanation that answers the
8 questions he's just given.
9 THE COURT: Well, he could say, other than what
10 you've already said, is there any other reason?
11 MR. HOLZHAUER: Okay.
12 BY MR. FLAXMAN:
13 Q. Other than what you've already said, is there any other
14 reason why you did not do a criterion-valid study?
15 A. Not having recalled what I said last week, I think that
16 some of the main points had to do with compromising the test.
17 Q. Is it --
18 A. The feasibility, the issues having to do with being able to
19 obtain an adequate measure of job performance from the
20 supervisors. So there are a variety of reasons why we could
21 not do it.
22 Q. Well, are you telling us that you can't measure how well
23 somebody is doing on the job as a police lieutenant?
24 A. I didn't say that. I said it's very difficult in the
25 present climate to get reliable measures of job performance.
600
Barrett - cross
1 I usually say that I had tried this approach in the
2 past in the case in Akron where I attempted to obtain a good
3 measure of job performance. We pilot tested it. But once we
4 started -- tried to use it operationally, there were charges of
5 bias, and not only charges of bias, but there were grouping of
6 scores.
7 In effect, the whole process fell apart because of
8 the issues of a supervisor having to rate subordinates and
9 potentially being accused of bias in that rating process.
10 Q. So are you telling us that as a practical matter you can't
11 measure job performance of police lieutenants?
12 A. I'm not saying -- I'm saying as a practical matter it's
13 very difficult when it's being used in the way it's being used
14 in the courtrooms.
15 Q. Didn't you do a criterion-valid study for Akron police
16 officers?
17 A. Yes, in the early 1970s I did a number of criterion-related
18 studies for police and fire. I did for promotional exams also,
19 I did criterion-related studies, yes, I did.
20 Q. Did you do a test that was administered in June of 1974 to
21 police force applicants in the City of Akron?
22 A. That sounds like the right date. Yes, I probably did.
23 Q. And you did a criterion-valid study with that test, didn't
24 you, Dr. Barrett?
25 A. As I recall, I did, yes.
601
Barrett - cross
1 Q. And that criterion-valid test was challenged in court, is
2 that right?
3 A. Yes, it was.
4 Q. And that criterion-valid test was upheld by the Court,
5 isn't that right, Dr. Barrett?
6 A. I believe that was Arnold v. Ballard in 1975.
7 Q. Did you also, after Arnold v. Ballard, do a criterion-valid
8 test for police sergeants in the City of Akron?
9 A. As I recall, I did, yes.
10 Q. And that criterion-valid test was also upheld by the
11 courts, isn't that right?
12 A. That's correct.
13 Q. You also did a criterion valid -- criterion-related test
14 for police lieutenants in the City of Akron, is that right?
15 A. Yes, I did.
16 Q. And that criterion-related test was also upheld by the
17 courts, is that right?
18 A. Yes, it was.
19 Q. And you also did a criterion-related test for entry-level
20 fire fighters in Akron, is that right?
21 A. Yes, I did.
22 Q. That was also upheld by the courts, is that right?
23 A. Yes, it was.
24 Q. You did another criterion-related test for fire lieutenant,
25 is that right?
602
Barrett - cross
1 A. Yes, I did.
2 Q. That was in Akron, is that right?
3 A. That's correct.
4 Q. That was also upheld by the courts?
5 A. That's correct.
6 Q. And you did another criterion-related test for fire
7 captains in Akron, is that right?
8 A. Yes.
9 Q. That was also upheld by the courts?
10 A. That's correct.
11 Q. And then you come to Chicago, Dr. Barrett, and you can't do
12 any criterion-related studies, is that right?
13 A. There's been a time lapse of 20 years, I think, between the
14 ones I did in 1974 and the present, 1994.
15 As I said in that time period things have changed in
16 terms of the perception and the cooperation from safety forces.
17 It's been due to the fact of the continued litigation over this
18 past 20 years.
19 Q. Well, Dr. Barrett, the decision about police officers in
20 Akron, that was in 1975, is that right?
21 A. Yes.
22 Q. And the decision about police sergeants in Akron, that was
23 also in 1975, is that right?
24 A. Yes.
25 THE COURT: The decision or the test?
603
Barrett - cross
1 MR. FLAXMAN: The decision.
2 BY THE WITNESS:
3 A. I'm not sure. It's probably correct.
4 BY MR. FLAXMAN:
5 Q. Now, police lieutenants in Akron, that was 1987, isn't
6 it -- wasn't it?
7 A. I think you're referring to when we changed the format to
8 the format we have now in terms of the oral, job knowledge and
9 in-basket. I think that would be '87. But I'm not -- I cannot
10 be 100 percent sure.
11 Q. Didn't you do a test in 1996, a criterion-related test in
12 Akron for fire captain?
13 A. No.
14 There was a decision in Black Fire Fighters
15 Association v. the City of Akron, a Sixth Circuit appeal case,
16 (unintelligible) 1996, which I believe -- which did verify that
17 our tests were valid. But those tests were the ones, same
18 format we used here. It was based on content validity. It was
19 not based upon criterion-related validity.
20 Q. They weren't criterion-related tests, Dr. Barrett?
21 A. Not the ones we used -- not for the decision in 1996, as I
22 recall.
23 Q. Well, are you familiar -- you're a member of the SIOP, is
24 that right?
25 A. Yes.
604
Barrett - cross
1 Q. And the SIOP is -- for the court reporter -- the Society of
2 Industrial and Organizational Psychologists, is that right?
3 A. Yes.
4 Q. And the SIOP has a newsletter, is that right, Dr. Barrett?
5 A. Yes, it does.
6 Q. What's it called?
7 A. The TIP, I believe
8 Q. The Industrial Psychologist?
9 A. Yes.
10 Q. And from time to time you write for that newsletter, don't
11 you, Dr. Barrett?
12 A. Yes, I do.
13 Q. And when you write for that newsletter, you're truthful, is
14 that right?
15 A. Yes.
16 Q. And you're accurate, is that right?
17 A. I try to be.
18 Q. And everything you say in the articles you write for the
19 newspaper is correct, is that right?
20 A. Well, I wouldn't go quite that far. I try to be accurate.
21 Q. Well, you don't lie in the newsletter, do you?
22 A. No.
23 Q. Now, do you have an article in, I think it's the October
24 1997 TIP, called "An Historical Perspective on the Nassau
25 County Police Entrance Examination, Arnold versus Ballard,
605
Barrett - cross
1 B-a-l-l-a-r-d, 1975 Revisited"?
2 A. Yes, I do.
3 Q. And in that article, you refer to the work you did in Akron
4 as criterion-related studies, isn't that right?
5 A. I think I refer to some of the studies I did as being
6 criterion related, that's correct.
7 Q. Well, you refer to Akron -- cases that were reported as
8 Akron Fire Fighters versus City of Akron as a criterion-related
9 study, is that right?
10 A. Some of the early cases were criterion related, there is no
11 question about it.
12 Q. Well, Barrett & Associates is your organization, is that
13 right?
14 A. That's correct.
15 Q. And when they send out faxes, do they put their fax phone
16 number at the top?
17 A. I don't know.
18 Q. Do you know what your fax phone number is at Barrett &
19 Associates?
20 A. No.
21 Q. All right. Let me show you what we'll mark as 162.
22 MR. HOLZHAUER: Do you have one?
23 MR. FLAXMAN: I only have one.
24 MR. HOLZHAUER: Your Honor --
25 THE COURT: Take a look at it.
606
Barrett - cross
1 Mr. Flaxman, if you are going to be using exhibits,
2 you ought to have more than one copy. You ought to have one
3 for your opponent, one for the witness, one for me and one for
4 yourself.
5 MR. FLAXMAN: You're right. And I don't. And I was
6 doing better before. And I blew it with this one. I'm sorry.
7 THE COURT: Are you almost finished with cross?
8 MR. FLAXMAN: Well, I'm finished. And I want to
9 confer, but I think I'm done.
10 MR. HOLZHAUER: Are you just going to refer to this
11 paragraph right here?
12 MR. FLAXMAN: Yes.
13 BY MR. FLAXMAN:
14 Q. Is Plaintiffs' 162 a copy of your article that appeared in
15 TIP?
16 A. This doesn't look like a copy of an article. It looks like
17 a draft at least. But it's not -- is this the actual article
18 that appeared?
19 THE COURT: Keep your voice up a little bit.
20 THE WITNESS: I'm sorry.
21 BY THE WITNESS:
22 A. I'm not sure it's the actual article that's printed.
23 BY MR. FLAXMAN:
24 Q. Well, in that draft, do you see on the first page where it
25 refers to criterion-valid tests that you developed for the City
607
Barrett - cross
1 of Akron in police and fire?
2 A. I see that the concurrent validation study was completed in
3 six months and resulted in a battery of four tests. The test
4 was administered in June of 1974 to police force applicants, et
5 cetera. Is that what you mean?
6 Q. Yes. In that article you don't refer to content-valid work
7 that you did in Akron, do you?
8 A. No. I refer at that point, right in this paragraph, to
9 1974, what we did, yes.
10 Q. And you continue with cases that are reported through 1996,
11 is that right?
12 A. I say, for example, on page 2, "The police preacademy
13 course was conducted for six weeks. I administered a
14 content-valid test."
15 Q. No. Dr. Barrett, you're not --
16 A. Well, you asked in terms of what I talked about. It seems
17 like I talked about both.
18 Q. My question, Dr. Barrett, is: In this article, somewhere
19 in the article, isn't it true that you refer to the work that
20 you did in Akron as involving criterion-related studies?
21 A. In 1974, that's correct.
22 Q. And in that article, don't you also refer to, after the
23 reference to the 1974 test, to subsequent criterion-related
24 studies?
25 A. I probably did, yes. We did a subsequent criterion-related
608
Barrett - cross
1 study, that's correct.
2 Q. And some of those subsequent criterion-related studies
3 extend through the 1990s, isn't that right, Dr. Barrett?
4 A. No, that's not correct. As far as I can recall, we didn't
5 have any criterion-related studies conducted in 1990s. No,
6 that's not correct.
7 MR. FLAXMAN: I think I'm done, but I want to confer.
8 THE COURT: Let's take a short break.
9 I had the Criminal Code up here. Does anybody know
10 what happened to it? You didn't take it back, did you?
11 MR. PIELL: I think we might have taken it back after
12 it was handed up.
13 THE COURT: Don't come up to the bench and take
14 things off the bench without asking me first.
15 MR. HOLZHAUER: I think George gave that to --
16 THE COURT: Well, I want it back. Ask me, not
17 George. All right. Five minutes.
18 (Recess.)
19 MR. FLAXMAN: Very few.
20 THE COURT: Proceed.
21 BY MR. FLAXMAN:
22 Q. Let's go back to the in-basket very briefly, Dr. Barrett.
23 The in-basket that you administered in Chicago was
24 the same type of in-basket that you had administered to Akron
25 fire lieutenants, is that right?
609
Barrett - cross
1 A. The same format, yes.
2 Q. Did you ever do any investigation to determine if there is
3 a relationship between scores on the in-basket and performance
4 as an Akron police lieutenant?
5 A. I'm not sure I did one for Akron. I thought for fire. I'm
6 not sure I did it at this point. No, I've never done a
7 criterion-related study on the in-basket in the City of Akron.
8 Q. Did you also administer the same type of in-basket to
9 persons employed by the Washington Metropolitan Area Transit
10 Authority?
11 A. Yes.
12 Q. Did you ever do any investigation to determine whether or
13 not there was a relationship between scores on the in-basket
14 and performance in the Washington transit police at the
15 Washington Metropolitan Area Transit Authority?
16 A. We conducted no criterion, or criterion-related study in
17 Washington, D.C., with the transit police.
18 Q. Now, the in-basket when it was administered for the Akron
19 lieutenants and the Washington Metropolitan Area Transit
20 Authority was given under different physical conditions than in
21 Chicago, isn't that right, Dr. Barrett?
22 A. They were all administered, as I recall, in a setting of a
23 classroom-type setting or training room-type setting.
24 Q. Did the people taking the test in Akron and in Washington,
25 D.C., have a tablet chair to put their materials on?
610
Barrett - cross
1 A. I'm not sure. I don't believe so. I think we had tables
2 in other places.
3 Q. You didn't have tables in Chicago, did you, Dr. Barrett?
4 A. No, we did not.
5 Q. And isn't it correct, Dr. Barrett, that because of
6 logistics problems in the City of Chicago, you modified the
7 administration process for the in-basket from the way it had
8 been -- you had administered it in other cities to the way it
9 was administered in the City of Chicago?
10 A. I don't think -- I think our administration procedures were
11 exactly the same.
12 Now, I think what we did do in the City of Chicago, I
13 think what we did is I recall buying a lot of clipboards. I
14 think that we bought a lot of clipboards to help individuals
15 who were taking the in-basket test in the City of Chicago. I'm
16 not sure we had clipboards in other locations.
17 Q. Okay. Did you have Manila folders in any other location?
18 A. I don't recall.
19 Q. Isn't it true, Dr. Barrett, that each person taking the
20 Chicago test was given 15 Manila folders?
21 A. At this point, I don't recall the exact number. But that
22 could be correct if that was part of the instructions.
23 Q. And that was different than it had been in Akron and in
24 Washington, D.C., is that right?
25 A. I don't, I don't recall if we used Manila folders or not in
611
Barrett - cross
1 those two locations.
2 Q. Now, the Manila folders in Chicago, were your instructions
3 that each person should receive 15 folders, all with the same
4 tab? I mean 15 folders for the left tab or 15 folders for the
5 right tab or 15 folders with the tab in the middle?
6 A. I don't recall.
7 Q. Let me show you what's been marked as Plaintiffs' 162.
8 Have you ever seen this before?
9 A. Yes.
10 Q. Could you tell us what it is.
11 A. It's a letter from one of my associates, senior associates,
12 Mr. Carabini, to a Marion Brzykcy, B-r-z-y-k-c-y, of Arthur
13 Andersen. It's in regard to a telephone call received
14 concerning administration of the test.
15 Q. And the first sentence refers to a telephone call you
16 received from Adrian Brandt about one tablet chair, is that
17 right?
18 A. Yes.
19 Q. You had wanted, had you not, more than one tablet chair for
20 each person taking the test, is that right?
21 A. Actually, as I recall, what I wanted was a table.
22 Q. And you didn't get a table, did you?
23 A. No, I did not.
24 Q. And then because you weren't getting a table -- well, let's
25 go back.
612
Barrett - cross
1 In Akron when the test was administered, did the
2 people taking the test have a table?
3 A. I can -- I don't recall exactly, but I think we always
4 requested a table. I know that when the test was given in our
5 office building we had, we had folding tables. I recall that.
6 But I don't recall for each and every application, if we did or
7 not.
8 Q. Now, so when you learned that Chicago wasn't providing
9 tables, you modified the administration process, is that right?
10 A. I wouldn't call it modifying the administration process.
11 We provided additional aids to the people.
12 Q. Well, were the modifications set out in this letter made in
13 the administration process?
14 A. Yes. We said we would provide additional items. This
15 would be one clipboard and 15 Manila file folders.
16 Q. The paragraph after that talks about each person getting 15
17 folders with the same position tab, one person would get right
18 tab folders and another person left tab folders, a third person
19 middle tab folders, is that right?
20 A. Yes.
21 Q. Now, you're not telling us, are you, that your job analysis
22 indicated that watch commanders have to work with file folders
23 that are all tabbed in the same place, are you, Dr. Barrett?
24 A. No, I don't recall that.
25 Q. Okay. And did you do any study to determine whether or not
613
Barrett - cross
1 administering the test this way with the clipboard and file
2 folders would provide the same simulation as administering the
3 test with the table?
4 A. We didn't do any criterion-related studies, if you're
5 asking that question.
6 Q. Did you pilot test it with the clipboard and file folders?
7 A. Yes.
8 Q. Was that in November of 1994?
9 A. Once we learned about that, we -- as I recall, we said what
10 can we do to aid the individuals? Of course, it's the same
11 administration, the same clipboard, the same file folders for
12 everyone. So everyone has the same materials. But we said
13 what can we do to make it perhaps easier?
14 And so we decided to -- decided to use clipboards and
15 file folders. As a matter of fact, that's now become our
16 standard procedure since that time where, in fact, we do
17 provide those things to people.
18 Q. And you don't have any data which shows whether or not
19 administering the test with the table or with 15 file folders
20 and a clipboard produces the same result?
21 A. No. I've done no study of that type.
22 Q. And that's not a criterion type of study, is it,
23 Dr. Barrett?
24 A. Well, it could be if you wanted to make it one.
25 Q. Well, but you could do that -- that kind of study without
614
Barrett - cross
1 any criterion, what you talked about criterion validation,
2 isn't that right?
3 A. I'm not sure what you are asking. It would be, it would be
4 -- if you want to try to formulate a study at this point in
5 time?
6 Q. Well, what I mean, Dr. Barrett, is you could have two
7 groups of people. One group could take the test with a table
8 and another group could take the test with the clipboard and
9 file folders, and you can see if there is any differences in
10 how they do on the test. Is that a traditional kind of study,
11 Dr. Barrett, testing a hypotheses?
12 A. It's traditional only in the sense, yes, you could do a
13 study. I've never heard of anyone worried about if you use a
14 clipboard and file folders if there is going to be a
15 difference, because the conditions are the same for everyone.
16 If there is any difference, it's a constant across individuals.
17 Q. Now, you're not saying that working on a table is the same
18 work sample as working on a clipboard with 15 file folders, are
19 you, Dr. Barrett?
20 A. I'm saying it's the very same decision-making process, the
21 same administrative skills are called for to make decisions.
22 Q. And you could, that's a hypothesis that could be tested, is
23 that right?
24 A. I'm not sure what you are asking. Again, if you are asking
25 if we took 100 people and used a table, 100 people using the
615
Barrett - cross
1 tabular chair, clipboards and file folders, we could do a study
2 to see if there's a mean difference between those two groups in
3 terms of number correct, for example. Yes, we could do a study
4 like that. It's possible.
5 Q. You haven't done one, have you, Doctor?
6 A. No, I have not.
7 Q. Thank you.
8 Now, in the course of your work in preparing the 1994
9 lieutenants test for the City of Chicago, did you become aware
10 of something called the Mayor's Blue Ribbon Panel on Police
11 Testing and Hiring and Promotion, Reports and Recommendations?
12 A. Yes.
13 Q. And as a matter of fact, in your report about your test,
14 you make reference to that Mayor's Blue Ribbon Panel report, is
15 that right?
16 A. Yes.
17 Q. Did you consider that report when you devised your test?
18 A. Yes.
19 Q. Now, as you sit there now, were you aware that the Blue
20 Ribbon Panel recommended that attendance, medical and
21 disciplinary records should play a role in promotions?
22 A. I believe there was testimony earlier on in the trial, I
23 believe, about that, yes.
24 Q. Let me show you what's been marked as Plaintiffs' Exhibit
25 60.
616
Barrett - cross
1 MR. FLAXMAN: I think we all have a copy of this?
2 THE COURT: What is this?
3 MR. FLAXMAN: This is the Mayor's Blue Ribbon Panel
4 on Police Testing, Hiring and Promotion.
5 THE COURT: Defendant's 60?
6 MR. FLAXMAN: Plaintiffs' 60.
7 THE COURT: Go ahead.
8 BY MR. FLAXMAN:
9 Q. Do you see there's a second sentence under number 4 where
10 it says, "Some weight should also be given to the applicant's
11 attendance, medical, and disciplinary records"?
12 A. Yes.
13 Q. You didn't follow that recommendation, did you,
14 Dr. Barrett?
15 A. No.
16 Q. Now, also in that report there is a statement that at page
17 6 talking about hiring that, "The nearly unanimous view of
18 testing experts is that it is plainly incorrect to regard one
19 candidate as better qualified than another or more deserving of
20 a job simply because he or she scored slightly higher on a
21 written test."
22 Do you see that?
23 A. Yes.
24 Q. In the 1994 lieutenants promotional test, is it correct
25 that one candidate is ranked higher than another because he or
617
Barrett - cross
1 she scored higher on the test?
2 A. That's correct.
3 Q. And you're not one of those testing experts, are you, who
4 agrees that it's plainly incorrect to regard one candidate as
5 better qualified than another simply because he or she scored
6 slightly higher on a written test?
7 A. No.
8 Q. As a matter of fact, you're one of those testing experts
9 who believes that somebody who scored 99.91 will do better on
10 the job than somebody who scored 99.90 on one of your tests, is
11 that right?
12 A. What I said it is a general scientific proposition,
13 following the linear model, that the probability is that the
14 people who score higher on the test in general will score --
15 will perform better on the job, is what I've said.
16 Q. And it's your opinion that people who got a higher final
17 score on your test will do better on the job if promoted to be
18 lieutenant, is that right?
19 A. As a generalization, a scientific generalization, that's
20 correct.
21 Q. And you don't have any empirical data to support that
22 opinion, do you, Dr. Barrett?
23 A. Yes, I do.
24 Q. Do you have any measure of police performance that you
25 could relate scores on the test to, Dr. Barrett?
618
Barrett - redirect
1 A. If you're asking, again, if I did a criterion-related study
2 in Chicago, I never have.
3 Q. No.
4 My question, Dr. Barrett, is: You don't have any
5 measure of police performance to relate to scores on your test,
6 do you?
7 A. No, I've never measured police performance in Chicago.
8 MR. FLAXMAN: Thank you. I have nothing further.
9 REDIRECT EXAMINATION
10 BY MR. HOLZHAUER:
11 Q. Dr. Barrett, I believe you testified on cross-examination
12 that it would be a mistake if the written job knowledge exam
13 contained a question based on a general order that was not on
14 the reading list, is that correct?
15 A. Yes.
16 Q. And you were shown the linkage charts indicating that one
17 of the questions was linked to general order 84-7, is that
18 correct?
19 A. Yes.
20 Q. And you were also shown the version of the reading list
21 that appeared in your report which did not include general
22 order 84-7, is that correct?
23 A. Yes.
24 Q. I'd like to show you several exhibits now. And because
25 they come from different binders, I've made new copies for you
619
Barrett - redirect
1 and new copies for the Court and new copies for Mr. Flaxman.
2 I need the reading list, which is appendix F of your
3 report, the linkage chart, appendix K of your report, general
4 orders 84-7 to 94-5. This is the reading list, the linkage
5 chart and the general orders, general orders 84-7 to 94-5.
6 MR. FLAXMAN: Could we have a minute break, Judge?
7 Maybe I could just let him lead and not object or make a
8 representation, if I could just have a minute to look this over
9 before he asks questions. It's not a jury trial. If there is
10 a mistake --
11 THE COURT: But it is a trial, and my time is short.
12 Go ahead and take a look at it.
13 Aren't these all from the books?
14 MR. HOLZHAUER: These are all from the books, Your
15 Honor. I just thought it would be hard to have four or five
16 books going at once.
17 THE COURT: I appreciate that. It is.
18 (Pause.)
19 MR. FLAXMAN: Thank you very much, Your Honor.
20 MR. HOLZHAUER: I have another document, Your Honor,
21 that we've marked Defendant's Exhibit 29. Mr. Flaxman has that
22 as well. This is new.
23 BY MR. HOLZHAUER:
24 Q. Dr. Barrett, first I'd like for you to look at the linkage
25 chart that I handed you. Do you have that up there?
620
Barrett - redirect
1 A. Yes.
2 Q. In particular, look at question 8.
3 A. Yes.
4 Q. Okay. Can you tell me --
5 THE COURT: What exhibit is this from?
6 MR. HOLZHAUER: This is appendix K of Exhibit 1, Your
7 Honor.
8 BY MR. HOLZHAUER:
9 Q. Can you tell me what question -- what general order
10 question 8 is linked to in that linkage report?
11 A. It's linked to GO 84-7.
12 Q. Okay. Now, I'd like you to look for a second at the
13 reading list, which was part of appendix F of your report.
14 Does general order 84-7 appear on that reading list?
15 A. No, it does not.
16 Q. And Mr. Flaxman questioned you about that, is that correct?
17 A. That's correct.
18 Q. Looking at the reading list, what does it say -- perhaps
19 you can just read for us what it says in that paragraph on the
20 top that's marked number 1?
21 A. "Department directives as listed blow. The written
22 examination will be based only on directives issued prior to
23 December 1st, 1994, with the exception of GO 94-5 (payroll and
24 timekeeping), which will be issued September 27th, 1994."
25 Q. Thank you.
621
Barrett - redirect
1 Now, I'd like you quickly to look at general order
2 84-7 that I handed you a minute ago. That's the one you
3 couldn't find on the reading list. Do you have that there?
4 A. Yes, I do.
5 Q. What's the topic of that general order?
6 A. It's payroll and timekeeping.
7 Q. There is something handwritten on page 1. Can you read
8 that for us?
9 THE COURT: Maybe I'm having trouble here.
10 MR. HOLZHAUER: I'm sorry, Your Honor. It's the
11 white general order.
12 THE COURT: It's the white one?
13 MR. HOLZHAUER: Yes.
14 THE COURT: All right. Sorry.
15 BY THE WITNESS:
16 A. It says, "Rescinded by general order 94-5 on 27 September
17 '94."
18 BY MR. HOLZHAUER:
19 Q. Okay. Now, I'd like you to look at 94-5, the blue general
20 order that I handed you. What's the topic of that general
21 order?
22 A. Payroll and timekeeping.
23 Q. Now, I'd like you to look at what I've marked as
24 Defendant's Exhibit 29, what I've marked and handed up to you
25 with the new sticker on it.
622
Barrett - redirect
1 A. Yes.
2 Q. Can you tell me what that is?
3 A. That's question 8, which goes through our justification
4 procedure and also indicates the source is GO 94-5.
5 Q. Did you keep this -- is this a record that you kept as part
6 of the process of test development?
7 A. Yes.
8 Q. Were these entries made at the time of the events that are
9 reflected on the form?
10 MR. FLAXMAN: There is no time on the form.
11 MR. HOLZHAUER: The dates.
12 MR. FLAXMAN: There is no date. I'm looking at Bates
13 stamp 7488. I don't see a date.
14 THE COURT: 7488?
15 MR. FLAXMAN: Right, but I will agree that --
16 MR. HOLZHAUER: This comes up.
17 MR. FLAXMAN: Well, I'll agree that I was wrong about
18 the cross-examination, about the question, if that's what the
19 point of all this is.
20 THE COURT: Okay.
21 MR. HOLZHAUER: Well, that's the point.
22 But I would like to move Exhibit 29 into evidence so
23 that we have it. It is -- it comes in as a record. It also
24 comes in under 106. It's part of the set of documents that
25 you've already produced for a variety of questions.
623
Barrett - redirect
1 THE COURT: You're stipulating that you were wrong
2 regarding the cross-examination on question 8?
3 MR. FLAXMAN: That's correct, Your Honor. That it
4 is, in fact, based upon a general order that was on the reading
5 list. And I apologize for missing that.
6 THE COURT: Okay.
7 MR. HOLZHAUER: I'd like to move Exhibit 29 into
8 admission.
9 THE COURT: All right. It will be admitted.
10 (Defendant's Exhibit 29 received in evidence.)
11 BY MR. HOLZHAUER:
12 Q. Give me back that Exhibit 29 and keep the other things that
13 you have in front of you.
14 Now, looking back at that linkage chart that was from
15 appendix K, do you recall Mr. Flaxman asking you some questions
16 about question 5 on the linkage chart?
17 A. Vaguely.
18 Q. Do you recall acknowledging that the linkage for question 5
19 seemed to be in error?
20 A. Yes.
21 Q. What source is question 5 linked to?
22 A. It's linked to 93-3, addendum 5, two -- Roman numeral II,
23 D, capital D-2.
24 Q. I'd like to show you another new exhibit which we've marked
25 Defendant's Exhibit 30. Can you tell me what that is?
624
Barrett - redirect
1 A. This is our documentation for item No. 5.
2 Q. This is similar to the record we just looked at for item
3 No. 8?
4 A. Yes.
5 Q. And it relates to question -- which question?
6 A. This question No. 5.
7 MR. HOLZHAUER: I'd like to move Defendant's Exhibit
8 30 into evidence, Your Honor.
9 MR. FLAXMAN: I think there should be more foundation
10 for this document. There is no date. There is no --
11 MR. HOLZHAUER: I can go through it.
12 MR. FLAXMAN: There is nothing about it.
13 THE COURT: What is this document, Dr. Barrett?
14 THE WITNESS: This is our documentation of our
15 rationale and our justification for an item which we generate
16 as we write an item.
17 And the first part is the stem of the item, and then
18 we have the five alternatives. The star is the correct answer.
19 And it indicates that the source was GO 84-7.
20 Then we have the rationale and justification of why
21 the answer is correct and why the other alternatives or all
22 alternatives in this case, B, C, D and E, are not correct.
23 MR. HOLZHAUER: Your Honor, we were about to go
24 through the foundation of this for the first exhibit, which was
25 identical to this or identical as to another question when
625
Barrett - redirect
1 Mr. Flaxman stipulated that that was okay.
2 These are exactly the same as a series of documents
3 that Mr. Flaxman has introduced as to the questions that
4 Mr. Bishop has challenged. And I'll be glad to lay the
5 foundation if Mr. Flaxman would like me to.
6 THE COURT: Are they part of your report, Doctor?
7 THE WITNESS: No.
8 THE COURT: What are they? Where are they kept?
9 THE WITNESS: They're kept as working files in our
10 offices, in our office. And we gave one example, I believe, of
11 this in the report in the appendix.
12 THE COURT: Are they kept in the ordinary course of
13 your business?
14 THE WITNESS: Yes.
15 THE COURT: As part of your regular business?
16 THE WITNESS: Yes.
17 THE COURT: Are you in charge of the keeping of these
18 documents?
19 THE WITNESS: Yes.
20 THE COURT: I'm trying to save some time here,
21 Mr. Flaxman.
22 MR. FLAXMAN: I just would like to -- the missing
23 link is when was this thing prepared, at what point in the test
24 process?
25 THE COURT: That's a fair question.
626
Barrett - redirect
1 When was it prepared, Dr. Barrett?
2 THE WITNESS: I can't give you the exact date. It
3 was prepared after we had determined the source list and before
4 the final exam date, this ongoing process between those dates.
5 MR. FLAXMAN: I don't think that's enough foundation
6 to actually get it in as a business record.
7 MR. HOLZHAUER: I think it is, Your Honor, and --
8 THE COURT: I do, too. I do, too.
9 MR. HOLZHAUER: It's ongoing process.
10 THE COURT: It will be admitted.
11 (Defendant's Exhibit 30 received in evidence.)
12 BY MR. HOLZHAUER:
13 Q. Okay. Looking at the first page on which there is writing,
14 Bates stamp 7470, can you tell me what the correct answer to
15 this question is?
16 A. The correct answer has been marked as A.
17 Q. And can you tell from the form what general order the
18 answer is linked to?
19 A. Yes, it's GO 84-7.
20 Q. Now, looking back at your reading list, is 84-7 on the
21 list?
22 A. No.
23 Q. Is this the order that we just discussed that was
24 superseded by 94-5?
25 A. Yes.
627
Barrett - redirect
1 Q. Is 94-5 on the list?
2 A. Yes.
3 Q. Now, back to the linkage chart, could you look at question
4 6.
5 A. Yes.
6 Q. What source is question 6 linked to on the linkage chart?
7 A. 84-7.
8 Q. I'd like to show you another exhibit that I've marked
9 Defendant's Exhibit 31. Can you tell us what that is?
10 A. This is item 6 -- it's our documentation for item 6, which
11 contains the stem, correct answers and alternatives.
12 Q. Can you tell me, is this the same form, same business
13 record that we discussed and identified as No. 29 and No. 30?
14 A. Yes.
15 Q. And this is as to a different question?
16 A. Yes.
17 Q. No. 6?
18 MR. HOLZHAUER: I'd like to move 31 into evidence.
19 MR. FLAXMAN: Same objection.
20 THE COURT: Same ruling. It will be admitted.
21 (Defendant's Exhibit 31 received in evidence.)
22 BY MR. HOLZHAUER:
23 Q. Looking at the first page, Bates stamped 7476, can you tell
24 me what general order this is linked to?
25 A. 93-3.
628
Barrett - redirect
1 Q. Okay. Back to the linkage chart. Again, looking at No. 5
2 and No. 6.
3 A. Yes.
4 Q. Can you now tell me what the error is in No. 5 and in No.
5 6?
6 A. It looks like they are reversed in terms of the source;
7 that the source for 5 should be 6, and the source for 6 should
8 be 5.
9 Q. Those two items were transposed?
10 A. It looks that way, yes.
11 Q. Thank you.
12 THE COURT: You're saying they all have 93-3 as their
13 source?
14 THE WITNESS: No. I'm just saying that they're
15 transposed.
16 THE COURT: But all these questions seem to have 93-3
17 as their source, is that right? Am I wrong about that?
18 MR. HOLZHAUER: All of the alternatives?
19 THE COURT: Question 8, I'm sorry, question 8 is
20 94-5.
21 MR. HOLZHAUER: Right.
22 THE COURT: Question 5, question 5 and 6 both have
23 93-3 as their source.
24 MR. HOLZHAUER: Right.
25 No, Your Honor, I think question 5 has 94-5, which
629
Barrett - redirect
1 used to be called 84-7, as its source on the form. Question 6
2 has 93-3 on this form. You look at the linkage chart, what
3 appears next to 5 should appear next to 6, and what appears
4 next to 6 should appear next to 5.
5 THE COURT: I see. Okay.
6 BY MR. HOLZHAUER:
7 Q. Dr. Barrett, I'd like you to look at question 13 on the
8 linkage chart.
9 A. Okay.
10 Q. Do you recall Mr. Flaxman asking you some questions about
11 the chart relating to question 13?
12 A. Yes.
13 Q. Do you recall acknowledging that the linkage for question
14 13 also seemed to be in error?
15 A. Yes.
16 Q. What is the source listed for question 13 on your chart?
17 A. 87-7.
18 Q. Dr. Barrett, I would like to show you another document that
19 I have marked Defendant's Exhibit 32. Can you tell me what
20 that is?
21 A. This is item 13. And, again, it's the same business record
22 we keep as we generate our questions.
23 Q. As to which item?
24 A. It's to item 13.
25 MR. HOLZHAUER: Your Honor, I'd like to move
630
Barrett - redirect
1 Defendant's 13 -- 32 into evidence.
2 MR. FLAXMAN: Same objection.
3 THE COURT: Same ruling.
4 (Defendant's Exhibit 32 received in evidence.)
5 BY MR. HOLZHAUER:
6 Q. Now, looking at the first page, Bates stamped number 7522,
7 can you tell me from what general order -- what general order
8 this answer is linked to?
9 A. 88-15.
10 Q. Okay. Now, looking back at your reading list, is 88-15 on
11 the reading list?
12 A. Yes.
13 Q. I'd like to show you another document which is marked
14 Defendant's Exhibit 33.
15 But before I do that, I'd like you to look at your
16 linkage chart for No. 14.
17 A. Yes.
18 Q. Can you tell me what source No. 14 is linked to?
19 A. 88-15.
20 Q. Dr. Barrett, I just handed you a document I've marked as
21 Defendant's Exhibit 33. Can you tell me what that is?
22 A. This is our business record for item 14.
23 Q. The same kind of business record we've discussed for the
24 previous questions?
25 A. Yes.
631
Barrett - redirect
1 MR. HOLZHAUER: Your Honor, I'd like to move
2 Defendant's 33 into evidence.
3 MR. FLAXMAN: Same objection.
4 THE COURT: It will be admitted.
5 (Defendant's Exhibit 33 received in evidence.)
6 BY MR. HOLZHAUER:
7 Q. Looking at the first Bates stamped -- first page with words
8 on it, Bates stamped 7527, can you tell me from this form what
9 general order No. 14 is linked to?
10 A. 87-7.
11 Q. Now, again looking at your linkage chart, can you explain
12 the error that Mr. Flaxman found?
13 A. Again, it appears to be an identical error transposing 13
14 and 14 in terms of the source.
15 Q. Okay. I'd like you to turn to the second page of your
16 linkage chart. Look at No. 34.
17 A. Yes.
18 Q. Can you tell me what -- do you recall Mr. Flaxman asking
19 you questions about No. 34?
20 A. Yes.
21 Q. What source is question 34 linked to on the chart?
22 A. 91-14.
23 Q. I'd like to show you a document that we've marked
24 Defendant's Exhibit 34. Dr. Barrett, can you tell us what this
25 exhibit is?
632
Barrett - redirect
1 A. This is a business record we maintain for writing items.
2 In this case it's item No. 34.
3 MR. HOLZHAUER: Your Honor, I'd like to move
4 Defendant's Exhibit 34 into evidence.
5 THE COURT: It will be admitted.
6 MR. FLAXMAN: Same objection.
7 THE COURT: It will be admitted.
8 (Defendant's Exhibit 34 received in evidence.)
9 THE COURT: You have a standing objection,
10 Mr. Flaxman.
11 MR. FLAXMAN: Thank you.
12 BY MR. HOLZHAUER:
13 Q. Looking at the first page with words, it's Bates stamped
14 7624. Can you tell me what general order this item should be
15 linked to?
16 A. 85-1-IV-E-2-b.
17 Q. Looking back to the linkage chart, is the linkage chart
18 incorrect as to question 34?
19 A. Yes.
20 Q. Looking back at your reading list, is 85-1 on your reading
21 list?
22 A. Yes.
23 Q. So question 34 is based on a source in the reading list?
24 A. Yes.
25 Q. Okay. Let's go on to a new topic. Perhaps it would be
633
Barrett - redirect
1 easier if you just handed me back those forms.
2 Dr. Barrett, do you recall that on cross-examination
3 Mr. Flaxman asked you whether Diane Thompson would have been
4 promoted had she achieved a score of 60 on the in-basket?
5 A. Yes.
6 Q. Do you recall how many questions there were on the
7 in-basket exercise?
8 A. 60.
9 Q. So would 60 have been a perfect score?
10 A. Yes.
11 Q. Do you know if anybody received a 60?
12 A. I don't believe so.
13 Q. I'd like to show you Defendant's Exhibit 22.
14 MR. HOLZHAUER: That's the big spreadsheet, Your
15 Honor.
16 BY MR. HOLZHAUER:
17 Q. Dr. Barrett, we've stipulated that that's a list of scores
18 for all candidates. Do you see the column setting out the
19 in-basket scores?
20 A. Yes.
21 Q. Which column is it? It might be easier if you count from
22 the right, I think.
23 A. One, two, three, four, five, six.
24 Q. Can you go through that list now and tell me if anyone got
25 a perfect score of 60?
634
Barrett - redirect
1 MR. FLAXMAN: Judge, I think we all know what the
2 highest score was. We can stipulate to that.
3 THE COURT: Go ahead. Do it.
4 MR. FLAXMAN: 56.
5 MR. HOLZHAUER: Go ahead.
6 MR. FLAXMAN: I believe 56 was the highest score
7 anybody got on the in-basket.
8 MR. HOLZHAUER: We'll agree to that.
9 MR. FLAXMAN: Am I wrong?
10 MR. HOLZHAUER: I think it's right. I haven't gone
11 through it myself.
12 THE COURT: It's not 60?
13 MR. HOLZHAUER: It's not 60.
14 BY MR. HOLZHAUER:
15 Q. Dr. Barrett, do you remember Mr. Flaxman asking you about
16 your fees in this matter?
17 A. Yes.
18 Q. On direct I asked you about any outstanding invoices to the
19 City. Do you recall what your answer to that was?
20 A. I think it was 136 --
21 Q. Okay. I'd like to show you -- excuse me?
22 A. 136,000, as I recall.
23 Q. I'd like to show you a set of documents, Dr. Barrett. Can
24 you look at those and tell me what they are?
25 A. These -- it's a letter and an invoice sent to the -- from
635
Barrett - redirect
1 my office to the City of Chicago.
2 THE COURT: Are these marked?
3 MR. HOLZHAUER: No. I haven't marked these as
4 exhibits, Your Honor.
5 THE COURT: For what purpose are you showing them to
6 the witness?
7 MR. HOLZHAUER: I'm trying to -- we could mark them
8 as exhibits. But we're trying to clarify the point that
9 Mr. Flaxman made about additional --
10 THE COURT: You don't have to say anything else. Are
11 you refreshing his memory?
12 MR. HOLZHAUER: Yes.
13 THE COURT: Have you exhausted his memory?
14 MR. HOLZHAUER: I can exhaust his memory.
15 THE COURT: All right. Do you want to stipulate to
16 these?
17 MR. FLAXMAN: I've never seen these before.
18 THE COURT: All right. You don't want to stipulate
19 to these. Sit down, sit down. You don't want to stipulate.
20 Sit down. Exhaust his memory and use them properly. But mark
21 them. Give me a number.
22 MR. FLAXMAN: I want to object to their being used.
23 THE COURT: Well, he's not using them until -- sir, I
24 want you to turn them over right now. They're not marked.
25 First, he has to exhaust his recollection. Then he's allowed
636
Barrett - redirect
1 to use anything he wants to refresh it.
2 MR. FLAXMAN: Well, I'm not sure if he could use
3 things that were requested in discovery that were said we don't
4 have them and then produced them in trial.
5 MR. HOLZHAUER: Your Honor, Mr. Flaxman is incorrect
6 about that. We did produce these documents to Mr. Flaxman.
7 THE COURT: Well, I'm not getting into that. He can
8 use anything he wants to refresh the witness' recollection. It
9 doesn't mean they're coming into evidence, Mr. Flaxman.
10 He can use, you know, yesterday's Tribune if he wanted to.
11 MR. HOLZHAUER: That's next.
12 THE COURT: I hope not.
13 BY MR. HOLZHAUER:
14 Q. Dr. Barrett, do you recall on cross-examination that you
15 were asked whether there were any additional amounts you were
16 paid already for your work since the exam administration
17 regarding participation and your role in this litigation above
18 and beyond the $136,000 invoice already outstanding?
19 A. Are you referring to the 350, $350,000 for the actual
20 project itself?
21 Q. No. This $350,000 for the project?
22 A. Right.
23 Q. There is a $136,000 outstanding invoice now?
24 A. Yes.
25 Q. I believe Mr. Flaxman asked you a series of questions that
637
Barrett - redirect
1 related to other amounts that you had been paid above and
2 beyond the 350,000 that related to your work on this case since
3 the exam was given. Do you recall that?
4 A. Vaguely, yes.
5 Q. Do you recall whether you have sent the City invoices for
6 additional amounts of work other than the 350,000 and other
7 than the 136,000?
8 A. Yes, we have.
9 Q. Have you been paid for those amounts?
10 A. As I recall, yes.
11 Q. Do you recall what those amounts were?
12 A. No.
13 Q. I'd like to show you some documents to refresh your
14 recollection about that. Can you tell me what those documents
15 are?
16 A. The first document is a letter from, from one my associates
17 to Mr. Pat Rocks, which includes an invoice for $21,000. And
18 the date is 2-20-96.
19 Q. That's for additional work on this, this matter?
20 A. Yes. It says, "Work completed in the matter of Brown v.
21 City of Chicago."
22 Q. Okay. Would you keep going, please.
23 A. There's a second letter dated March 21st, 1996, again from
24 an associate, with an invoice which is in the amount of over
25 $54,000. And, again, it's for work completed in the matter of
638
Barrett - redirect
1 Brown v. City of Chicago.
2 Q. Okay. Can you keep going through all of these documents,
3 please.
4 A. The next is a letter of July 6th, 1995, from one of my
5 associates, including an invoice. The invoice is for a little
6 over $3,000. And again, it has to do with Brown v. City of
7 Chicago.
8 There is a letter, again from my associate dated
9 August 9th, 1995, which includes an invoice, and the invoice
10 amount is $62.50, again, for Brown v. City of Chicago.
11 And the last letter is from an associate which
12 includes an invoice, and I don't see a total. Oh, there it is.
13 Total invoice, $136,000, again for Brown v. City of Chicago.
14 Q. Is this last one the amount that's still outstanding?
15 A. Yes.
16 Q. So the others are amounts that have already been paid?
17 A. Yes.
18 Q. Looking at those together, you've been paid roughly $80,000
19 in addition to the 350,000 in addition to the 136,000, is that
20 correct?
21 A. That sounds correct.
22 Q. Do you recall Mr. Flaxman then trying to draw a comparison
23 between the amounts the City has paid you and the additional
24 cost of subjective scoring that Dr. York said would be needed
25 to make your test the traditional in-basket test?
639
Barrett - redirect
1 A. Yes.
2 Q. I'd like to show you a document we marked Defendant's
3 Exhibit 3. Can you tell me what that is?
4 A. This is a response to the expert witness reports and
5 depositions of Dr. York, Koziol, K-o-z-i-o-l, and Levitt, and
6 Mr. Bishop in the matter of Brown v. City of Chicago. The date
7 is October 13th, 1997.
8 Q. I'd like you to turn to page 27 and 28 of that report.
9 A. Yes.
10 Q. Will you tell me what the analysis on those pages is?
11 A. I made an analysis of what it would cost to use live
12 assessors in what Dr. York has termed a more traditional
13 process. And there I used, I believe, his rate of compensation
14 and determined what it would cost the City to perform that
15 process.
16 Q. And what was your estimate, Dr. Barrett?
17 A. Yes, it was approximately $6 million.
18 Q. Now, can you tell us what the table 3 on page 28 is?
19 A. That's my estimated costs of a traditional in-basket.
20 Q. Now, can you tell me approximately how many hours would be
21 spent on scoring the traditional in-basket eliminating the
22 training time of the raters?
23 A. How many hours, with no training you're saying?
24 Q. Yes. Eliminate the training.
25 A. It's 12, 15, 18,000 hours.
640
Barrett - redirect
1 Q. And do you know how much that would come to at --
2 A. That would be $3.6 million.
3 Q. Okay. Do you know what hourly rate you generally charge
4 for raters?
5 A. How I charge?
6 Q. Yes.
7 A. It would be -- according to who is doing it, our rate would
8 vary between probably 70 to $150 an hour.
9 Q. So to calculate what it would cost for your raters to do
10 this job, you would multiply something between 70 and 150 times
11 18,000?
12 A. Yes, if I used those people. But I'm talking about rating
13 for -- in our shop, yes.
14 Q. Was cost the primary reason for using multiple-choice
15 answers for your in-basket?
16 A. No.
17 Q. What was the primary reason?
18 A. In terms of objectivity and reliability, elimination of
19 bias.
20 Q. Let me take that book out of your way.
21 On the written job knowledge test, you testified
22 earlier that there was a pilot test, is that correct?
23 A. Yes.
24 Q. Do you recall Mr. Flaxman asking you a number of questions
25 about the time allotted for the various exam components?
641
Barrett - redirect
1 A. Yes.
2 Q. I'd like to have you look at your report, Exhibit 1. Look
3 at page 116. Can you tell us what is described on page 116?
4 A. The pilot testing of the written job knowledge exam.
5 Q. How many questions were on the pilot test?
6 A. 186.
7 Q. Do you know how much time the test takers were given?
8 A. Three hours.
9 Q. How does that work out in minutes per question?
10 A. One minute.
11 Q. Did you consider the pilot test results when you decided
12 whether the timing of the job knowledge test was appropriate?
13 A. Yes.
14 Q. Was there a problem with the amount of time given during
15 the pilot test?
16 A. No.
17 Q. Dr. Barrett, do you recall that Mr. Flaxman asked you on
18 cross-examination whether a three-part test consisting of an
19 in-basket exercise, an oral briefing exercise, and a job
20 knowledge component is the standard package offered by Barrett
21 & Associates?
22 A. Yes.
23 Q. Does every promotional exam developed by Barrett &
24 Associates consist of an in-basket exercise, an oral briefing
25 exercise and a written job knowledge test?
642
Barrett - redirect
1 A. No.
2 Q. What other exam designs have you used?
3 A. The most common exam is often -- a very common exam is the
4 job knowledge test by itself.
5 Q. In what circumstances do you use that kind of exam?
6 A. There is -- I've got to be very specific, I guess. In the
7 City of Jacksonville, Florida, for example, that's the design
8 we used, have been using for a number of years, because there
9 is a concern by the fire fighters that any other process will
10 not be as objective.
11 MR. FLAXMAN: Can I object to the -- his explanation
12 because of what other people think?
13 THE COURT: Hearsay?
14 MR. FLAXMAN: Yes.
15 THE COURT: All right. It won't be accepted for its
16 truth. It's accepted for his reasons for doing it, though.
17 Whether it's true or not doesn't matter.
18 BY MR. HOLZHAUER:
19 Q. So that was in the City of Jacksonville, did you say?
20 A. Yes.
21 Q. And in what force was that?
22 A. It's the fire fighters.
23 Q. Was that promotional or entry level?
24 A. It's promotional exams.
25 Q. Okay. And when was that exam given?
643
Barrett - redirect
1 A. We've been giving that exam now for a number of years. I'm
2 not sure exactly when we started, around early 1990s, I
3 believe.
4 Q. How recently has it been given?
5 A. As recently as about two months ago, I think.
6 Q. So all of the recent public safety promotional exams you've
7 given have not used the similar three-part exam?
8 A. That's correct.
9 Q. Is it possible that a job analysis of some police or fire
10 job might reveal that that three-part structure would not be
11 appropriate for that job?
12 A. You could infer, yes.
13 Q. What would you do under those circumstances?
14 A. Only use what was appropriate.
15 Q. When you develop promotional exams for different police and
16 fire departments, do you use the same questions and the same
17 resource materials each time?
18 A. No.
19 Q. Has your standard three-component test been found to be
20 content valid in every case in which its validity was
21 challenged?
22 MR. FLAXMAN: Objection, not material, especially if
23 they're different questions, different tests. It's his
24 reputation that he's trying to prove up here.
25 MR. HOLZHAUER: Let me ask one foundational question
644
Barrett - redirect
1 and try it again, Your Honor.
2 THE COURT: Okay.
3 BY MR. HOLZHAUER:
4 Q. Do you follow the same content validation process in the
5 other promotional exams for police and fire departments?
6 MR. FLAXMAN: Objection. This sounds like habit
7 testimony.
8 THE COURT: It sounds like what?
9 MR. FLAXMAN: Habit testimony.
10 THE COURT: Habit.
11 MR. HOLZHAUER: The issue in this case is whether his
12 process is appropriate.
13 THE COURT: I agree. Overruled.
14 BY THE WITNESS:
15 A. Yes.
16 BY MR. HOLZHAUER:
17 Q. Has your standard three-component test been found to be
18 valid content in every case where its validity has been
19 challenged?
20 MR. FLAXMAN: Same objection.
21 THE COURT: Overruled.
22 BY THE WITNESS:
23 A. Yes.
24 THE COURT: Your objection is to weight more than
25 admissibility, I think.
645
Barrett - redirect
1 BY MR. HOLZHAUER:
2 Q. Dr. Barrett, Mr. Flaxman asked you several questions about
3 the challenge procedure administered by Arthur Andersen. Do
4 you recall that?
5 A. Yes.
6 Q. What is the Educational Testing Service?
7 A. It's a nonprofit organization which develops a wide variety
8 of tests in the academic area and in the public and private
9 sectors.
10 Q. Could you tell us what the best known Educational Testing
11 Service tests are?
12 A. Probably the Scholastic Aptitude Test, the SAT, is probably
13 the widest known.
14 Q. Does it also do employment tests?
15 A. Yes. They do employment tests.
16 Q. Do you have any knowledge -- is the Educational Testing
17 Service a large organization?
18 A. It's very large.
19 Q. Among organizations that design and develop tests, how
20 would you say that its size ranks?
21 A. I would have to estimate, make an estimate from -- I've
22 been invited to its campus in New Jersey as a discussant on a
23 panel. They just do a wide variety. It's a very large group
24 of -- there's hundreds of people involved in that organization.
25 Q. Do you have any knowledge about the challenge procedures
646
Barrett - redirect
1 that are implemented by the Educational Testing Service?
2 MR. FLAXMAN: Objection, relevance.
3 THE COURT: Sustained, sustained.
4 BY MR. HOLZHAUER:
5 Q. Dr. Barrett, to your knowledge, did some candidates
6 challenge particular questions on the lieutenants exam?
7 A. Yes.
8 Q. As a result of those challenges, were some questions
9 eliminated?
10 A. Yes.
11 Q. Were some questions modified as to the correct answer?
12 A. Yes.
13 Q. What form did that modification take?
14 A. I believe in one case two alternatives, A and B, for
15 example, were both counted as correct.
16 Q. Do you recall for how many questions were either -- were
17 eliminated?
18 MR. FLAXMAN: I think we did this on direct.
19 THE COURT: Do it on direct or cross? I know it
20 sounds very familiar.
21 MR. HOLZHAUER: It is. And I have one follow-up
22 question on it. I think it will be useful to have the evidence
23 of the number of questions, Your Honor.
24 THE COURT: All right. Go ahead.
25 BY THE WITNESS:
647
Barrett - redirect
1 A. As I recall, there were two on the job knowledge test and
2 two on the in-basket which were challenged and we made some --
3 some modifications were made, either in terms of giving
4 everyone, giving everyone points or giving two options of the
5 points.
6 BY MR. HOLZHAUER:
7 Q. So three questions altogether?
8 A. Four.
9 Q. Four.
10 Did it surprise you that changes were necessary in
11 that many questions?
12 MR. FLAXMAN: Objection.
13 THE COURT: Sustained.
14 BY MR. HOLZHAUER:
15 Q. Does the number of challenges in this case, how does the
16 number of challenges in this case compare with your experience
17 in other cases?
18 MR. FLAXMAN: Objection.
19 THE COURT: Overruled.
20 BY THE WITNESS:
21 A. It's far fewer.
22 MR. HOLZHAUER: Thank you.
23 BY MR. HOLZHAUER:
24 Q. Now, you testified on cross that candidates were not
25 allowed to take their answer sheets with them or to make notes
648
Barrett - redirect
1 and take them out of the test -- test room. Do you recall
2 that?
3 A. Yes.
4 Q. Why do you do that?
5 A. To maintain the confidentiality of the test and the
6 procedure.
7 The a COURT: How much longer do you have?
8 MR. HOLZHAUER: Probably a very few minutes. Less
9 than five minutes, I think.
10 BY MR. HOLZHAUER:
11 Q. I'd like to refer you to Defendant's Exhibit 27. I can
12 take that other book.
13 Dr. Barrett, before we get into that exhibit, would
14 you just tell me why you maintain the confidentiality of test
15 questions?
16 A. It's often the standard procedure of a civil service
17 organization to do that. It's more of a policy decision of the
18 organization. It's not always done on a content-valid test in
19 terms of maintaining confidentiality.
20 Q. I'd like you to look at page 208 of this document.
21 A. Okay.
22 Q. What is this document?
23 A. This is the EEOC guidelines of 1978.
24 Q. Do you recall Mr. Flaxman asking you questions about
25 whether content validation can be used on the in-basket tests
649
Barrett - redirect
1 insofar as it involved judgment and decision making?
2 A. Yes.
3 Q. And you answered that it could?
4 A. Yes.
5 Q. And you also answered you didn't think your view was
6 contrary to the EEOC guidelines?
7 A. Yes.
8 Q. Can you explain that for me.
9 A. In that section he was referring to, it was, I think,
10 referring to the type of test which you take off the shelf,
11 such as a general intelligence test or a general test of
12 judgment of a very generic nature, where you're using a test to
13 measure a construct.
14 Again, the best example is an intelligence test of
15 where you are measuring a construct called intellectual
16 ability, and there's a standardized test which you can take off
17 the shelf and use for selection purposes.
18 Q. Thank you. I can take that from you.
19 Dr. Barrett, do you recall testifying both on direct
20 and cross-examination that your in-basket exam is a simulation?
21 A. Yes.
22 Q. And do you recall that Mr. Flaxman pointed to the in-basket
23 exercise and asked you whether you knew if a lieutenant would
24 have to do two weapons discharge incidents in one day?
25 A. Yes.
650
Barrett - redirect
1 Q. And you also asked him whether they would ordinarily be
2 doing a number of scheduling matters without clerical
3 assistance in that day?
4 A. Yes.
5 Q. Why do you test on more than what would occur in a
6 particular day?
7 A. Because it is a psychometric test. As I said early on last
8 week, it's a compression into a short period of time a number
9 of events which the lieutenant would actually do on the job.
10 But you're compressing it into a short period of time and
11 making it into a psychometric test.
12 THE COURT: Psychometric?
13 THE WITNESS: Psychometric, yes.
14 BY MR. HOLZHAUER:
15 Q. Dr. Barrett, do you recall questions about question No. 3,
16 No. 6, No. 11, and No. 13 of the in-basket exam that used the
17 term "discrepancy," and "discrepancy that had an impact"?
18 A. Yes.
19 Q. Was the term "discrepancy" in those questions intended by
20 you to be a replication of police jargon?
21 A. Not necessarily, no.
22 Q. What was it intended to be?
23 A. Give clear directions to the testees in terms of what was
24 being asked.
25 Q. Now, Mr. Flaxman also asked you questions about question 47
651
Barrett - redirect
1 and 48 and said that some of the issues as to scheduling and
2 time off that were raised by 47 and 48 would be made by
3 clerical employees, not by lieutenants. Do you recall that?
4 A. Yes.
5 Q. If they are made by clerical employees, why would they be
6 on your test?
7 A. Well, one reason, if in fact he's correct, that the
8 responsibility of lieutenant is to supervise these activities,
9 and, in fact, ensure that what has ever been done by clerical
10 employees is accurate and correct.
11 Q. Thank you.
12 Now, you testified, you were asked on
13 cross-examination at length about the Lopez monograph, is that
14 correct?
15 A. Yes.
16 Q. Were there any other studies of objective scoring systems
17 with in-basket that you relied upon?
18 A. Yes.
19 Q. Can you tell us what they were?
20 A. As I said, there is a long history of that starting with
21 Fredrickson and Lopez, of course. Kesselman has an article, as
22 I recall, was one which had it. And since that time, there has
23 been research by Hakstian up to the present day looking at the
24 more objective in-baskets.
25 Q. The Hakstian article was published in what year, do you
652
Barrett - redirect
1 recall?
2 A. 1997 is the most recent. But he had one in the late 1990s.
3 Of course, there is a review also by Shipman, et al.,
4 in Personnel Psychology, I believe, which is a review of
5 various in-basket techniques where they do mention the
6 objective approach.
7 MR. HOLZHAUER: Two more questions.
8 BY MR. HOLZHAUER:
9 Q. Do you recall that there were questions regarding reading
10 comprehension that you were asked by Mr. Flaxman?
11 A. Yes.
12 Q. And you testified that you looked at the reading level of
13 the exam questions?
14 A. Yes.
15 Q. Can you tell us how you did that with the process of
16 evaluating what the reading level was?
17 A. It's a standard Flesch count type approach where you
18 basically let the computer scan the material and you get an
19 index in terms of grade level.
20 Q. Did you match the grade level of the exam questions with
21 the grade level of something else?
22 A. Yes, I did.
23 Q. What was that something else?
24 MR. FLAXMAN: Objection. This is now hearsay about
25 what the computer told him something is. I don't know if Your
653
Barrett - redirect
1 Honor heard the beginning of --
2 THE COURT: I'm reading it.
3 (Pause.)
4 THE COURT: Overruled.
5 BY THE WITNESS:
6 A. I compared it with the source lists we used.
7 BY MR. HOLZHAUER:
8 Q. The material that lieutenants read on the job?
9 A. Yes.
10 Q. And how did it compare?
11 A. The reading level of our test is lower than the reading
12 level of material, material the lieutenants are required to
13 read on their job.
14 Q. Is that all described in your report?
15 A. Yes, it is.
16 MR. HOLZHAUER: Thank you. No further questions.
17 THE COURT: All right. We'll take an hour break.
18 (Recess at 12:25 p.m. until 1:25 p.m.)
19
20
21
22
23
24
25
654
Barrett - redirect
1 IN THE UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF ILLINOIS
2 EASTERN DIVISION
3 ERNEST T. BROWN, et al., )
)
4 Plaintiffs, )
) No. 95 C 1890
5 v. ) Chicago, Illinois
) November 24, 1997
6 CITY OF CHICAGO, ) 1:30 p.m.
)
7 Defendant. )
8 VOLUME 4
9 TRANSCRIPT OF PROCEEDINGS - TRIAL
10 BEFORE THE HONORABLE ROBERT W. GETTLEMAN
11 APPEARANCES:
12 For the Plaintiffs: KENNETH N. FLAXMAN, P.C.
122 South Michigan Avenue
13 Suite 1850
Chicago, Illinois 60603-6107
14 BY: MR. KENNETH N. FLAXMAN
15 and
16 FUTTERMAN & HOWARD, CHTD.
122 South Michigan Avenue
17 Suite 1850
Chicago, Illinois 60603
18 BY: MR. CRAIG FUTTERMAN
19
For the Defendant: MAYER, BROWN & PLATT
20 190 South LaSalle Street
Chicago, Illinois 60603
21 BY: MR. JAMES HOLZHAUER
MR. JEFFREY S. PIELL
22 MS. ANGELA K. DORN
MR. ANDREW NICELY
23
Official Court Reporter: JENNIFER S. COSTALES, CSR, RMR
24 219 South Dearborn Street
Room 1744-A
25 Chicago, Illinois 60604
(312) 427-5351
655
Barrett - recross
1 (Proceedings heard in open court.)
2 THE CLERK: 95 C 1890, Ernest Brown versus City of
3 Chicago, on trial.
4 MR. HOLZHAUER: Your Honor, we went back and checked,
5 I believe Mr. Flaxman did as well, on that scoring sheet. We
6 discovered one person who had a 59, two 58s and a 57 there.
7 But we didn't find any 60s.
8 THE COURT: Close, close.
9 MR. FLAXMAN: I think it's actually one 59, four 58s,
10 five 57s.
11 MR. HOLZHAUER: Really?
12 MR. FLAXMAN: Yes.
13 MR. HOLZHAUER: We didn't find all those. Good.
14 That's fine.
15 THE COURT: All right.
16 Your witness, Mr. Flaxman.
17 MR. FLAXMAN: Thank you.
18 GERALD BARRETT, DEFENDANT'S WITNESS, PREVIOUSLY SWORN
19 RECROSS EXAMINATION
20 BY MR. FLAXMAN:
21 Q. You were asked about Diane Thompson and the in-basket,
22 whether she would have been promoted if she got 60. She would
23 have been promoted if she got 56 correct on the in-basket,
24 isn't that right?
25 A. I didn't check.
656
Barrett - recross
1 Q. Okay. Now, you told us on redirect you talked about pilot
2 testing. You said you pilot tested the written test, is that
3 right?
4 A. Yeah.
5 Q. And were those police officers who took the written test?
6 A. No.
7 Q. Were they fire fighters who took the written test?
8 A. No.
9 Q. Were they graduate students who took the written test?
10 A. Yes.
11 Q. Graduate students, those are people who have taken a lot of
12 tests in their professional careers, is that right?
13 A. Yes.
14 Q. People who took the SAT to get into college, is that right?
15 A. They might have.
16 Q. Took the graduate record exam to get into graduate school?
17 A. That's correct.
18 Q. And they take tests in their courses every, every semester
19 or quarter, is that right?
20 A. Yes.
21 Q. And do you think that graduate students are maybe better at
22 taking written-multiple-choice tests than Chicago police
23 sergeants?
24 A. I don't know if they are or not.
25 Q. You don't have any opinion at all about that, is that
657
Barrett - recross
1 right?
2 A. That's correct.
3 Q. You also told us on redirect that when you devised the
4 test, you used an in-basket, an oral test and a written job
5 knowledge test because it was appropriate based on the job
6 analysis, is that right?
7 A. Yes.
8 Q. Didn't you propose doing an in-basket test, an oral test
9 and a written job knowledge test when you first got the
10 contract with the City of Chicago to do promotional tests here?
11 A. Yes.
12 Q. That was before any job analysis, isn't that right?
13 A. No.
14 Q. It was before any job analysis that you had conducted in
15 the City of Chicago, isn't that right?
16 A. No.
17 Q. Well, before you did this test, the lieutenants test, you
18 did a sergeants test, is that right?
19 A. Yes.
20 Q. And the sergeants test you made the same proposal of a
21 three-part test, is that right?
22 A. Yes.
23 Q. That was an in-basket, an oral test and a written job
24 knowledge test, is that right?
25 A. Yes.
658
Barrett - recross
1 Q. That was the first test you prepared for the City of
2 Chicago, is that right?
3 A. Yes.
4 Q. And you didn't do any work for the City of Chicago before
5 you did the work for the sergeants case -- test, is that right?
6 A. Yes.
7 Q. Now, when we talked about the in-basket, you talked about
8 articles that supported objective scoring, is that right? Do
9 you remember that?
10 A. Yes.
11 Q. None of those articles supported a 60-question
12 multiple-choice test, is that right?
13 A. I don't recall the exact number of items they may have had
14 on the other test.
15 Q. Well, isn't it true that neither of the articles that you
16 cited used a multiple-choice test in their in-basket
17 instrument?
18 A. I think they use a multiple-choice format as I recall, yes,
19 they used an objective type format.
20 Q. Well, was one of the studies or articles to which you refer
21 something by Kesselman, Lopez and Lopez called the Development
22 and Validation of a Self-Report Scored In-Basket Test in an
23 Assessment Center Setting?
24 A. Yes.
25 Q. Is that the Kesselman study to which you referred?
659
Barrett - recross
1 A. I believe so.
2 Q. Was that published in a refereed journal?
3 A. I thought it was.
4 Q. Let me show you what has been marked as Plaintiff's Exhibit
5 3. Is this the copy of the Kesselman article to which you
6 referred?
7 A. Yes.
8 Q. And you also refer to a Hakstian, H-a-k-s-t-i-a-n, article,
9 is that right?
10 A. Yes.
11 Q. Hakstian didn't use a multiple-choice test in his
12 in-basket, did he?
13 A. He used an objective format.
14 Q. Well, Dr. Barrett, my question is: Did he use a
15 multiple-choice test?
16 A. As I define it, yes, he used an objective multiple-choice
17 type format.
18 Q. Well, were there questions, and you had to choose the
19 correct answer?
20 A. Yes, you had to choose the correct answer among
21 alternatives as I recall.
22 Q. Let me show you what has been marked Plaintiffs' Exhibit
23 163. Is this the Hakstian study to which you refer?
24 A. Yes.
25 MR. FLAXMAN: I would move the admission into
660
Barrett - recross
1 evidence of these last two exhibits, 163 and 3.
2 MR. HOLZHAUER: No objection.
3 THE COURT: They will be admitted.
4 (Plaintiffs' Exhibits 3 and 163 received in evidence.)
5 BY MR. FLAXMAN:
6 Q. Now, you also talked about the cost of doing a traditional
7 in-basket. And you told us that it would cost 3.4 -- $3.6
8 million, is that right?
9 A. I believe if we eliminated the training of the assessors,
10 if I recall.
11 Q. Now, in a traditional in-basket as Dr. York described it,
12 the assessors review the work product of the person taking the
13 test, is that right?
14 A. Among other things, yes.
15 Q. The assessors in the traditional in-basket, as Dr. York
16 described it, don't watch the person take the test for three
17 hours?
18 A. It sounded like that's what Dr. York was saying in his
19 deposition and in his report.
20 Q. Well, what he said in court and what you understand this
21 traditional in-baskets was that the assessors rate the work
22 product of the people who take the test, isn't that right,
23 Dr. Barrett?
24 A. Yes. I disagree if Dr. York's characterization in his
25 report of what --
661
Barrett - recross
1 Q. So --
2 A. -- what the traditional in-basket consists of. I don't
3 recall anyone standing there and watching somebody actually
4 write out the memos or make the decisions. But he, in fact,
5 has a point in his report concerning that issue.
6 Q. When you did your cost estimate of $3.6 million, you costed
7 in three people watching for three hours each candidate take
8 the test, is that right?
9 A. That's correct.
10 Q. And you assumed that each of those three people who are
11 watching the person take the test would be billed at $200 an
12 hour, is that right?
13 A. Yes. I used Dr. York's rate.
14 Q. Well, I'm asking if you used your rate of $200 an hour for
15 people watching somebody take the test, is that right?
16 A. In my report I was very clear saying I was using the rate
17 of Dr. York.
18 Q. Well, you came to $1.8 million in your cost estimate for
19 having three people watch 1,000 people take a 3-hour test, is
20 that right?
21 A. It sounds correct.
22 Q. Okay. Now, in the traditional in-basket, the one that was
23 described by Mr. Bentz or Dr. Bentz in his article or monograph
24 --
25 A. Mr. Bentz is correct.
662
Barrett - recross
1 Q. Mr. Bentz in his article back in April of 1968, the scoring
2 was done just on the work product of what the people did, isn't
3 that right?
4 A. I don't recall exactly. Probably true.
5 Q. Okay. And it takes maybe half an hour to an hour to score
6 that work product for each candidate, is that right?
7 A. It depends. Yes, it could take an hour.
8 Q. So if it takes an hour -- and you could have two persons
9 rate each in-basket, isn't that right?
10 A. You could have one person rate. Yes, you could.
11 Q. Okay. So you have one person rate each in-basket. That's
12 1,000 hours, is that right?
13 A. Yes, that's correct.
14 Q. Now, there is no reason, is there, that everybody who wants
15 to be a lieutenant has to take an in-basket test?
16 A. Well, I'm not sure what you are asking, because I'm --
17 Q. Well, specifically what I mean is you could have given a
18 written job knowledge test as a screening device and say there
19 is a passing score and 500 people are going to pass, is that
20 right?
21 A. Anything is possible. Yes, I could have.
22 Q. So if you had done that, then you could have had 500 people
23 going on to take the in-basket, is that right?
24 A. That's a possibility.
25 Q. Then if you have 500 people take the in-basket, that takes
663
Barrett - recross
1 one hour to score by one person, we've got 500 hours involved
2 in scoring, is that right?
3 A. Under your hypothetical, that's correct.
4 Q. Okay. Now, do you know somebody named Doverspike?
5 A. Yes.
6 Q. Does he work for you at Barrett & Associates?
7 A. Yes.
8 Q. Does he -- what's the extent of his professional education?
9 A. He has a PhD in industrial psychology.
10 Q. Has he published any articles in industrial psychology?
11 A. Yes, he has.
12 Q. What's his billing rate, Dr. Barrett?
13 A. I think it's 150.
14 Q. Excuse me?
15 A. I believe it's 150. I'm not sure.
16 Q. Well, let me show you one of these documents, and I'll even
17 put a number on it. We'll call it Plaintiff's Exhibit 164. Is
18 that my next number? And I only have one copy.
19 This is one of your billing statements to the City of
20 Chicago, is that right?
21 A. Yes.
22 Q. And that billing statement sets out the rate at which you
23 billed the City of Chicago for Dr. Doverspike, is that right?
24 A. As of the date, yes. This is a correct -- this was --
25 Q. Could you tell us what Dr. Doverspike's billing rate was to
664
Barrett -
1 the City of Chicago?
2 A. It was $110 per hour in 1996.
3 Q. So if Dr. Doverspike spent 500 hours scoring the in-baskets
4 at that rate, that wouldn't cost $3.4 million or $3.6 million,
5 would it, Dr. Barrett?
6 A. No, it wouldn't.
7 Q. It would cost less than what the City of Chicago has paid
8 you in connection with your testimonial services in this case,
9 isn't that right?
10 A. Yes.
11 Q. It would cost about $55,000?
12 A. That's correct.
13 MR. FLAXMAN: Nothing further.
14 THE COURT: Thank you. I have some questions. We
15 don't do this.
16 MR. HOLZHAUER: Okay.
17 THE COURT: We don't do recross and re-recross and
18 re-redirect. This is not a deposition.
19 But I have some questions, and then I'll give you a
20 chance to ask some follow-ups based on what I ask Dr. Barrett,
21 if you have them.
22 Dr. Barrett, when you devised the in-basket test, did
23 you know what the physical conditions would be for the people
24 who took it?
25 THE WITNESS: Did I know, no. We recommended tables.
665
Barrett -
1 THE COURT: All right. But you didn't know for sure
2 whether they would have tables or something else?
3 THE WITNESS: Well, a representation was made we
4 would have tables. We thought we would have tables, I can tell
5 you, at that point in time.
6 THE COURT: Okay. So if you knew that they took it
7 in the conditions -- if you knew at the time you were devising
8 the test that the candidates would take the exam under the
9 conditions they, in fact, ended up taking it, would you have
10 done it any differently?
11 THE WITNESS: No. It's nothing we could have done
12 differently, except as a remedial thing I talked about us
13 having a clipboard and the file folders.
14 THE COURT: And that was remedial after you learned
15 they weren't having tables?
16 THE WITNESS: Exactly right.
17 THE COURT: All right. You heard some testimony --
18 because you've been sitting here for the whole trial, right?
19 THE WITNESS: Yes.
20 THE COURT: You've heard some testimony from people
21 who took the test about the trouble they had putting papers on
22 the floor and spreading them out. And one person even said he
23 thought there might have been some cheating going on as a
24 result of having to spread the papers out. Did you hear that
25 testimony?
666
Barrett -
1 THE WITNESS: Yes, I did.
2 THE COURT: What's your opinion of the way they gave
3 this test?
4 THE WITNESS: Well, I think that everyone had the
5 same opportunity to use the materials, the clipboards, the file
6 folders and the floor, as everyone else. I observed it. We
7 tried to make plenty of room around the people, because we
8 tried to crowd people in the classroom.
9 And also, I guess one other factor is I understand
10 that often you do do paperwork in cramped quarters as a Chicago
11 police lieutenant, perhaps in a patrol car. There is no
12 question, I did recommend a table. But we couldn't have a
13 table.
14 THE COURT: Why did you recommend a table?
15 THE WITNESS: To give people more room and wouldn't
16 have these complaints about putting stuff on the floor.
17 THE COURT: Have you ever given a test where they had
18 to do with the way they did it in Chicago in '94? Either
19 before or after that time?
20 THE WITNESS: I don't recall that we have, no.
21 THE COURT: All right. We've had some testimony and
22 discussion about merit selection, or what I'll call merit
23 selection. Do you have an opinion as to whether merit
24 selection is valid or not?
25 THE WITNESS: This goes back to my testimony of --
667
Barrett -
1 concerning performance appraisals.
2 THE COURT: Of what?
3 THE WITNESS: Performance appraisals and how you
4 develop a criterion-related validation study. That, in
5 essence, is merit approach.
6 In other words, what you have to do is somehow rate
7 the job performance of present sergeants. In general, I've
8 always said I think it's an excellent idea that you do want, if
9 you can, as some weighting on previous job performance as a
10 sergeant, have some weight if it's practical and feasible in
11 the selection process.
12 The problem I have had in my -- there is two, two
13 basic problems. One is the general problem of devising a
14 reliable measure of job performance. If you have two raters,
15 let's assume we had two lieutenants rating the sergeants' job
16 performance, if you could find two, and you need at least two
17 to have some check. Their correlation or interrater
18 reliability is about .5. It's very low. So you have that
19 general problem of getting a reliable measure which is not
20 going to be (unintelligible) biased.
21 The second practical problem is the reluctance of
22 people who are superior officers in the safety forces to make
23 ratings of their subordinates' job performance in a way which
24 would have an impact upon the promotion -- potential promotion.
25 They are worried about the reaction of their subordinates in
668
Barrett -
1 this important decision. They are worried about going to court
2 and being accused of discrimination.
3 And I perhaps mentioned that I tried this process in
4 the City of Akron. I tried it with promotion to police
5 sergeant. We spent over a year devising a good,
6 psychometrically sound measure of job performance of a police
7 officer. And the concept was you take the test for sergeant,
8 and at the same time we would have the performance of patrol
9 officers rated. And in trials it worked. In pilot studies
10 when there was no consequence, no one knew what was -- they
11 were rating people psychometrically. It was reliable and so
12 forth.
13 Once it was going to be used for potentially
14 promotion to sergeant, in other words, we'd take that measure
15 of job performance of a police officer and give it, say, a
16 10-percent weight, the whole process fell apart. And, in fact,
17 the district court, the federal district court said we couldn't
18 use it because of the problems with that approach.
19 A second type of merit approach would be some more
20 objective measure. But here you have the same types of issues
21 and problems in a little different way. For example, it's been
22 suggested many times arrest records, for example, numbers of
23 arrests might be appropriate. But we know from our studies,
24 which we have done and I've actually done, the number of
25 arrests are often related very highly to where you are
669
Barrett -
1 assigned, what district you are in, what unit you are in. So
2 as you go down the list of potential ways of doing merit
3 promotion, it becomes much more difficult.
4 Now, I would say this, perhaps at the very extremes
5 it's easier to do. But as a general process, you have enormous
6 practical problems and problems which we have not solved, and
7 I'm not sure we can ever solve, in terms of measuring job
8 performance in a reliable fashion.
9 THE COURT: Okay. Do you believe that there is any
10 alternative to the type of tests that you give that would be
11 equally valid and maybe less, have a less adverse impact than
12 this test?
13 THE WITNESS: In my review of the literature and what
14 I've seen, I don't know of any alternatives which is equally
15 valid and has less adverse impact.
16 THE COURT: I might have asked you this at the
17 preliminary injunction hearing, but I'll ask you again: What
18 do you attribute the adverse impact to in this case?
19 THE WITNESS: I recall you asking that question at
20 the preliminary hearing as a matter of fact. And I didn't
21 probably give you a very good response.
22 THE COURT: I don't remember. It's been a long time.
23 THE WITNESS: Okay. It goes back to what's been
24 occurring in the public sector since 1973 and since Title VII
25 was applied to the public sector. There has been continual
670
Barrett -
1 lawsuits. And early on in 1970s, in many places, including the
2 City of Chicago, where the federal courts found there was past
3 discrimination, based on that past discrimination, the remedy
4 was a quota.
5 And a quota itself did not take people who were equal
6 in terms of the knowledge, skills, and abilities required to be
7 successful in the police department or safety forces. In other
8 words, what you might have is -- I'll just give a hypothetical.
9 You might have in the City of Akron, for example, a 25 percent
10 black population. The quota might be 50 percent. So what I
11 did was a study using these sorts of figures. It's called
12 Kreck, Barrett, and Alexander, published in the Journal of
13 Applied Psychology about 1983.
14 And what I did was a computer simulation using these
15 figures. So you have these mean differences between groups.
16 The federal courts come in and give a quota. What you are
17 going to do is that's all very nice for entrants. Then you do
18 it for sergeants. And then you come to lieutenants, for
19 example. And now you have a different pool of minority and
20 nonminorities.
21 You can predict, because of those differences and
22 fundamental knowledge, reading comprehension, et cetera, you
23 are going to have differences on promotional tests. So for the
24 last at least 15 years I've been predicting that what's going
25 to occur is you are going to have continued adverse impact as
671
Barrett -
1 you go up the ranks in safety forces, because once you have
2 imposed a quota at the beginning level, it's going to go up all
3 the way through the system.
4 And I, in fact, in my last sentence of that paper
5 said as long as the federal courts keep doing this, there is
6 going to be a continued burden on the courts, because you can
7 predict that you are going to have adverse impact all the way
8 up the line.
9 There is a second part, to answer your question also,
10 and it has to do what went wrong, what can you do. In fact,
11 here you have 44 plaintiffs right now. I matched those
12 plaintiffs first on the job knowledge test with 44
13 nonminorities who have exactly the same scores. Now, I looked
14 at each item, 150 items. I couldn't match one person. That
15 person had a score of 57 on the job knowledge test. The mean
16 on a job knowledge test is around 109. So this person is very,
17 very low.
18 Why is that person very low, that plaintiff? The
19 obvious reason is these are tests which depend upon studying,
20 motivation to study, to do well. And one of the problems I
21 have right now in the public sector is if you want to have
22 adverse impact, all you have to have is have a group of
23 individuals who say, "I'm not going to study very hard. I'm
24 going to get a low score on the job knowledge test," which is
25 based upon if you read the material.
672
Barrett -
1 We had testimony from one sergeant who said, in fact,
2 she had not read the material. If that plaintiff gets a low
3 score, I'm not very surprised. There is nothing in the tests
4 themselves which are biased or discriminatory.
5 I did the same thing with the in-basket test. I
6 again matched 44 plaintiffs with the corresponding nonminority
7 individuals, looked at each item, and you see an even
8 distribution. Some get more, some less. They're
9 indistinguishable basically in that regard. So there is
10 nothing in the test which is discriminatory or biased.
11 In fact -- well, this is -- so what I'm basically
12 saying is at this point, we call it -- we use the word "race
13 discrimination," but it's not really that at all from my point
14 of view. What went wrong -- see, you still have to go back
15 again into Griggs versus Duke Power in 1971. And where the
16 spirit impact --
17 THE COURT: I'm sorry, I didn't catch that.
18 THE WITNESS: Okay. In 1971, the Supreme Court had a
19 case called Griggs versus Duke Power. This is where the
20 doctrine of disparate impact or adverse impact was formulated.
21 And you had -- Duke Power used two tests for laborers. One was
22 a general intelligence test called the Wonderlich. The second
23 test was the Bennett Mechanical Comprehension Test.
24 Now, they applied these two standardized tests off
25 the shelf for which there was no evidence that they were valid.
673
Barrett -
1 And they applied these for jobs of laborers where you did not
2 need a high school education. And this is where the Court
3 really devised the concept of adverse impact or disparate
4 impact, because as you might guess, there was a difference in
5 the test scores of the blacks and the test scores of the
6 whites. And this, they said, is evidence of adverse impact.
7 And, therefore, you have to show that the Wonderlich test,
8 measured general IQ, and the Bennett General Test of Mechanical
9 Comprehension are, in fact, valid. And I think that was very
10 appropriate for those types of tests.
11 But we're talking now about types of tests which are
12 no longer used in the public sector. We're using content-valid
13 tests, tests which are obviously job relevant.
14 THE COURT: You're getting beyond my question now.
15 THE WITNESS: Okay.
16 THE COURT: Okay. I have a couple more questions.
17 As I understand it, in order to take this test, you
18 have to memorize all the general orders, right, or you have to
19 study them and know them pretty well?
20 THE WITNESS: I would say, yes, correct.
21 THE COURT: All right. And those are, those are
22 Plaintiff's Exhibit 7. I'm holding this book up. It's about,
23 oh, three inches thick, it looks like, right? This is the book
24 you have to learn, right?
25 THE WITNESS: Yes.
674
Barrett -
1 THE COURT: Okay. You have to also either memorize
2 or know very well all of the special orders and department
3 notices, right?
4 THE WITNESS: That's correct.
5 THE COURT: I'm holding up Exhibit 8 is special
6 orders, about, oh, about two inches thick, maybe a little more;
7 exhibit 9 are notices, it's about an inch and a half thick,
8 right?
9 THE WITNESS: Yes.
10 THE COURT: How about the labor contract, that's
11 Exhibit 10, do you have to know that, too?
12 THE WITNESS: Yes.
13 THE COURT: All right. That looks like it's about,
14 oh, inch and a half thick. The strategic plan, do you have to
15 learn that, too?
16 THE WITNESS: Yes.
17 THE COURT: Exhibit 11, October 1993. That's about a
18 half an inch thick maybe. And then you have to learn the
19 municipal code, traffic law, and the criminal statutes, right?
20 THE WITNESS: Yes.
21 THE COURT: And that's this book that was so kindly
22 provided to me by defense counsel, which is about, it's a West,
23 it looks like a West statute book, it's a little over a
24 thousand pages in thickness. You'd have to learn this, too?
25 THE WITNESS: I would think --
675
Barrett -
1 THE COURT: Or only selected parts?
2 THE WITNESS: -- there's only selected parts.
3 THE COURT: The parts on your reading list, right?
4 THE WITNESS: Yes.
5 THE COURT: Now, was I wrong when I said you have to
6 learn all the general and special orders or you only have to
7 the learn the ones on the reading list?
8 THE WITNESS: Just the ones on the reading list, of
9 course.
10 THE COURT: Okay. And what proportion of this book
11 that was given me here of the general orders -- or are these
12 all of the general orders that were on the reading list?
13 THE WITNESS: I don't know what you have in front of
14 you, to be honest about it.
15 THE COURT: What I'm trying to get at is: How big a
16 job was this to memorize -- there is a lot of materials you
17 have to memorize or you have to know just about as well as
18 memorizing, right?
19 THE WITNESS: That's correct.
20 THE COURT: I'm trying to get an idea of what we are
21 talking about here because, I mean, this isn't the bar exam
22 that we're talking about. We're talking about a police
23 examination. I'm just trying to get an idea of what you expect
24 these folks to do. I mean, how -- would you say this is rather
25 typical?
676
Barrett -
1 THE WITNESS: Oh, it's very typical. It's the most
2 typical thing -- I don't know of any safety force which does
3 not concentrate upon job knowledge.
4 THE COURT: So do you believe that the ability to
5 either memorize or come close to memorizing that type of
6 material, that volume of material indicates the ability to do a
7 good job as a lieutenant? That's your belief, isn't it?
8 THE WITNESS: Yes, yes.
9 THE COURT: Okay. And that somebody who can memorize
10 that material better than somebody else can do a better job as
11 a lieutenant than somebody else, right?
12 THE WITNESS: Memorize and use it, yes.
13 THE COURT: Okay.
14 Now, I want to talk about the challenge process for a
15 moment. Tell me again, why didn't you allow the people who
16 took the test to take it home or to take notes about it home?
17 THE WITNESS: Again, as I said before, this is really
18 the policy of the personnel department of the City of Chicago.
19 THE COURT: You had nothing to say about it?
20 THE WITNESS: Not really in terms of my -- in other
21 words, I was not involved in the appeal process. It was Arthur
22 Andersen and the City that set up that procedure.
23 THE COURT: I see. So you were really uninvolved at
24 all?
25 THE WITNESS: Except in the process of reviewing the
677
Barrett -
1 challenge.
2 THE COURT: I see. So once the challenge came in,
3 you got involved. Until then you didn't help design it?
4 THE WITNESS: No, I did not.
5 THE COURT: You weren't asked for your opinion?
6 THE WITNESS: No.
7 THE COURT: Have you ever done any work on any of
8 these other tests where you had any part in the challenge
9 process, or I should say in designing the challenge process?
10 THE WITNESS: Usually, I'm trying -- I'm not trying
11 to be evasive, but usually there is a policy by the personnel
12 department which we follow about the appeals. And it's often
13 it's a policy which is for all tests, just not safety force
14 tests. So the City has a policy. We are going to do X, Y and
15 Z for an appeal process. And we follow that policy.
16 THE COURT: Well, do you have an opinion as to
17 whether you can have an effective challenge process without
18 allowing the candidates to take home at least notes about the
19 questions they had problems with?
20 THE WITNESS: What I would say is this, I think it
21 would be effective for the things which stick in their mind. I
22 think when you take a test in the safety forces, you say, gee,
23 item 55, that seems crazy to me. I'm going to call in and
24 challenge. You might talk to your friends who took the test
25 and say, I don't believe this is the correct answer. So I
678
Barrett -
1 think, yes, it can be -- for the major issues, I think it can
2 be effective. You can recall that. You tend to recall bad
3 questions.
4 THE COURT: After taking a test of this magnitude
5 with that type of material to either study or have a command
6 of?
7 THE WITNESS: See, afterwards, to give you an
8 example, after two and a half hours of taking the 150-item
9 test, you have a lunch break. They can go outside the room and
10 write down whatever they want to. They can write down item 10
11 was bad because of this. They can go out and do this. So
12 that's, that's certainly a choice they have.
13 THE COURT: So your opinion is that there is nothing
14 wrong with this challenge process?
15 THE WITNESS: I'm not saying there is nothing wrong
16 with it. I'm just saying that it is one which can, can
17 potentially work, yes.
18 THE COURT: Okay. But you don't let them take the
19 materials out, because you don't want them or somebody --
20 whoever does that presumably doesn't want them to take the
21 materials home with them or notes of the materials, because
22 they don't want to have them pass on these materials to
23 somebody else, right?
24 THE WITNESS: Yeah, I assume so, yes.
25 THE COURT: Meaning that they really can't remember
679
Barrett - recross
1 the materials unless they were to write them down?
2 THE WITNESS: They can write them down after the
3 test, yes. There is no -- in other words, once you take the
4 test, you can write them down.
5 THE COURT: I meant write them down during the test.
6 THE WITNESS: No, you can't write them down during
7 the test.
8 THE COURT: Okay. That's all I have.
9 Are there any questions that either of you want to
10 ask based only on what I've asked? I don't want you to plow
11 old ground here.
12 Mr. Flaxman, you're the plaintiff -- well, no. He's
13 your witness, Mr. Holzhauer. I guess you should go first.
14 MR. HOLZHAUER: I have none.
15 MR. FLAXMAN: I just have one point.
16 RECROSS-EXAMINATION
17 BY MR. FLAXMAN:
18 Q. I think I heard you say that reading comprehension will
19 have a difference on promotional tests?
20 A. I don't think I said that. I don't recall saying that.
21 Q. Well, if you didn't -- well, I can't -- all right.
22 THE COURT: I don't remember it either.
23 MR. FLAXMAN: Thank you.
24 THE COURT: You may be right, Mr. Flaxman. All
25 right.
680
Klein - direct
1 Dr. Barrett, you're excused. Thank you very much.
2 THE WITNESS: Thank you.
3 (Witness excused.)
4 THE COURT: All right. Do you have your next
5 witness?
6 MR. HOLZHAUER: Yes. We call Lieutenant Klein.
7 THE COURT: Please raise your right hand.
8 JOHN KLEIN, DEFENDANT'S WITNESS, SWORN
9 DIRECT EXAMINATION
10 BY MR. HOLZHAUER:
11 Q. Good afternoon, Lieutenant Klein.
12 A. Good afternoon.
13 Q. Would you please state your name and spell your last name
14 for the record, please.
15 A. John Klein, K-l-e-i-n.
16 Q. Where are you employed and in what capacity?
17 A. I'm employed as a lieutenant with the Chicago Police
18 Department.
19 Q. Could you briefly describe your employment history with the
20 police department.
21 A. I was hired as a probationary officer on February 19th,
22 1973. After my academy training, I was assigned to the 7th
23 police district on the south side of the City of Chicago, where
24 I stayed for approximately four years. After that time, or
25 during that time, I was a patrol officer in a beat car, I
681
Klein - direct
1 worked desk duties, lockup duties, tactical duties.
2 After that assignment, I was transferred to the
3 special operations section, also on the south side of the city,
4 where I was responsible for high crime enforcement in all south
5 side districts.
6 I stayed there for approximately three years, after
7 which time I was transferred to the research and development
8 division. While in the research and development division, I
9 was responsible for researching and drafting department
10 directives.
11 After that time I was promoted to the rank of youth
12 officer. I spent approximately six months in Area 1 youth
13 division. In 1981 or so, I was again transferred to be
14 administrative officer for a deputy superintendent in our
15 administration bureau. I stayed there through 1985 when I was
16 promoted to the rank of sergeant. After a short time I was
17 reassigned to the administration bureau as a sergeant.
18 I stayed there until approximately 1988, when I was
19 promoted to the rank of lieutenant. Upon promotion to the rank
20 of lieutenant, I was assigned as the commanding officer of the
21 office of legal affairs, which is the department's legal
22 services bureau. I was in that assignment for approximately
23 four, four and a half years, after which time I was again
24 reassigned and promoted to the rank of assistant deputy
25 superintendent.
682
Klein - direct
1 And I served as the general counsel to the
2 superintendent of police for approximately two and one-half
3 years. In 1995, I was again reassigned to our education and
4 training division where I presently serve as a coordinator
5 responsible for education and training in the Chicago Police
6 Department.
7 Q. Now, in addition to your work with the Chicago Police
8 Department, do you do any work related to police?
9 A. I am a part-time lecturer for the Northwestern University
10 Traffic Institute. I teach courses on policy and procedural
11 and legal issues.
12 Q. Lieutenant Klein, would you briefly summarize your
13 educational background starting with college.
14 A. I received my bachelor's degree in business management from
15 DePaul University in 1994 -- I'm sorry -- 1974. In 1978, I
16 received my master's degree in public administration from
17 Roosevelt University.
18 In 1985, I graduated from DePaul University Law
19 School and was admitted to the Illinois bar that same year. In
20 1987 I attended the 11-week FBI National Academy in Quantico,
21 Virginia, which was a school focusing on police management
22 issues.
23 In 1988, I attended the FBI's National Law Institute,
24 which is a week-long program with legal advisors from
25 throughout the country addressing and discussing contemporary
683
Klein - direct
1 legal issues in policing.
2 I have continued during that time and the intervening
3 years to attend continuing legal education and various police
4 management courses.
5 Q. Are you still a licensed attorney?
6 A. Yes, I am.
7 Q. What are your duties at the police academy?
8 A. My title and role there is that of coordinator. I'm
9 basically the number two person in our education and training
10 division. I have oversight and developmental responsibilities
11 for all training as it affects newly-hired officers, as it
12 impacts encumbent officers in the department, and supervisors.
13 Q. Now, when I think of the police academy, I think of the
14 training program for recruits. Do officers already on the job
15 go to the training academy?
16 A. We call that in-service training. We provide a vast
17 variety of courses and in-service, technical skills,
18 investigative skills, homicide investigations, for example, as
19 well as providing continuing education and executive
20 development for all our supervisory and command personnel.
21 Q. Do you officers who have just been promoted to a new rank
22 go through additional training at the academy?
23 A. All officers who are promoted to a higher rank go through a
24 training program at the academy.
25 Q. Are you involved in the training program for newly-promoted
684
Klein - direct
1 lieutenants?
2 A. Yes, I am.
3 Q. What is your role?
4 A. Because we promote lieutenants and supervisors on an
5 infrequent basis, each time we have a free service class, which
6 is what we call our promotional classes for new promotees. We
7 basically have to conduct a review and analysis of the training
8 program. Things change as time goes on, policies and
9 procedures change, management issues change, our policing
10 strategy has changed.
11 So what we do is we assess, in conjunction with the
12 patrol division, what the education and training needs are of
13 the new supervisor so that we can provide them with some of the
14 management tools they will need to go perform their functions.
15 Q. Can you describe what the lieutenants' preservice training
16 program involves?
17 A. It's a two-week program, two full weeks, eight hours a day.
18 And it is intended to provide a general overview of some of the
19 major policy and procedural substantive areas for which a
20 lieutenant is responsible.
21 There is presumption that the new lieutenants have a
22 strong base of technical skills. So what we try to do in the
23 two-week preservice course is to give them exposure to
24 management techniques, executive development issues, give them
25 some skill sets and tools to deal with the personnel issues
685
Klein - direct
1 with which they'll be dealing.
2 We try to apprise them or familiarize them, or
3 familiarize them or acquaint them with topical issues in the
4 department, issues such as sexual harassment, family and
5 domestic violence, things that are of significant concern in
6 the department at the time of their promotion.
7 Q. Is the City continuing to promote lieutenants off the 1994
8 list?
9 A. It's my understanding the City is not intending to make any
10 more promotions off that list.
11 Q. As a result of your work, are you familiar with the
12 organizational structure of the police department?
13 A. Yes.
14 Q. Let's start with geography. How are the functions of the
15 police department divided geographically?
16 A. We have 25 police districts that are geographically
17 interspersed throughout the corporate limits of the city. Each
18 of those -- or actually those 25 districts are subdivided into
19 five areas, five police areas. So each area, one through five,
20 has five districts in it. Therefore, an area --
21 Q. There are five areas?
22 A. Five areas, 25 districts.
23 Q. Five districts each, for 25 total?
24 A. Yes.
25 Q. Are all the functions of the police department administered
686
Klein - direct
1 through this 25-district structure?
2 A. No. Organizationally we have five bureaus in the police
3 department. That's the highest level, hierarchical structure.
4 There is an administrative bureau, which basically
5 handles such matters as finance and data processing, things of
6 that nature. There is a technical bureau, which handles
7 communications and support services, equipment and supplies,
8 things of that nature. There is a bureau of staff services,
9 which houses the research and development division, the
10 management and labor affairs section, the training division, of
11 which I'm a member.
12 And then there are two principal field units, the
13 bureau of operational services and the bureau of investigative
14 services. Investigative services are people such as our gang
15 crimes people, our youth officers, our detectives that have
16 field responsibilities throughout the city. They work out of
17 those areas. They don't work in police districts. They report
18 to and work out of a geographic area.
19 And the bureau of operational services is our largest
20 field unit. In that bureau is the patrol division, the patrol
21 division being the 25 districts I referred to that have the
22 basic responsibility for responding to police calls for service
23 throughout the city.
24 And also housed in that bureau of operational
25 services are some of our specialized units, our public
687
Klein - direct
1 transportation section for the CTA, our public housing section
2 for public housing developments, our canine unit, our mounted
3 units, some of those things.
4 Q. You mentioned the five areas into which the five districts
5 for each report. Who is in charge of each of those five areas?
6 A. There is a deputy chief that commands the area.
7 Q. Okay.
8 A. In the patrol division.
9 Q. And who is in charge of each of the 25 districts under
10 them?
11 A. There is a district commander that is responsible for each
12 of those 25 districts.
13 Q. Focusing on those districts, on one of those districts, how
14 are the districts staffed and administered?
15 A. The chain of command is basically the same, or is the same
16 for all 25 districts. There is a district commander who is
17 responsible for overall district operations. The districts are
18 divided into three shifts or watches as we call them. Each
19 watch has a watch commander.
20 The watch commander historically has been a captain,
21 but more so now there is a lieutenant, because we are attriting
22 the rank of captain and are no longer testing for that career
23 service position. Below that captain or lieutenant are
24 sergeants, and below the sergeants are the police officers.
25 Q. Now, is there any difference between what a captain does as
688
Klein - direct
1 a watch commander and what a lieutenant would do as a watch
2 commander?
3 A. No.
4 Q. Could you describe a typical day in the life of a
5 lieutenant serving as watch commander?
6 MR. FLAXMAN: I don't think this is material to the
7 case.
8 THE COURT: Why?
9 Your whole point is that these tests don't reflect
10 the reality of the job. He's asking him to describe a typical
11 day in the job. How can it not be more relevant than that?
12 MR. FLAXMAN: My objection then is lack of foundation
13 that he doesn't know. I don't think he has ever worked as a
14 district commander or a watch commander based on the testimony
15 I've heard today. He is a lieutenant. He worked in legal
16 services.
17 THE COURT: Well, lay a better foundation.
18 BY MR. HOLZHAUER:
19 Q. You testified earlier that you are working at the training
20 academy, is that correct?
21 A. That's correct.
22 Q. You testified earlier that part of your responsibility is
23 to develop the training program for lieutenants, is that
24 correct?
25 A. That's correct.
689
Klein - direct
1 Q. As part of that responsibility, have you taken it upon
2 yourself or been required to learn what the job of a watch
3 commander is?
4 A. Yes.
5 MR. FLAXMAN: Object to leading. I'd like to -- I
6 would really like to know how he has learned about the job of
7 watch commander, but I would like him to testify about it.
8 MR. HOLZHAUER: Well, I think you can ask that on
9 cross. I think I'm entitled to ask him whether he knows this
10 thing as a foundational question. He's the man who designs
11 these programs.
12 THE COURT: Well, why don't you just ask him how he
13 knows about the job. I think we would cut through a lot of
14 this, and there wouldn't be any need for objections, and it
15 would tell me that he's qualified to do it.
16 BY MR. HOLZHAUER:
17 Q. How do you know about the job of watch lieutenant, watch
18 commander?
19 A. Well, first of all, I served in a police district for four
20 years. I served in the special operations unit which operates
21 in police districts for approximately three years. I spent
22 those seven years in district facilities interacting with
23 sergeants and with lieutenants and with watch commanders and
24 observing the conditions under which they work and becoming
25 familiar with what their duties and responsibilities were.
690
Klein - direct
1 Beyond that personal experience, I was acting watch
2 commander when I worked in what is called special employment.
3 When I was a sergeant and a lieutenant, I worked in districts
4 and in specialized units as a supervising and in some cases a
5 watch commander on my off days. We're allowed to work
6 specialized employment, and you receive additional compensation
7 for that. And I did that with the CHA and with the CTA and
8 with districts as well.
9 Beyond those personal experiences and personal
10 exposure, as an analyst and a person responsible for policy
11 development from 1978, roughly, through 1988, one of my
12 responsibilities was to review all policy and procedures that
13 were being promulgated by the department. This required a
14 regular relationship and constant contact with field officers,
15 watch commanders, district commanders so that I could be aware
16 of the problems that they were experiencing and we could draft
17 appropriate policy and procedures to address those problems.
18 When I was the commanding officer of the office of
19 legal affairs, I was again involved in the defense of
20 litigation involving the Chicago Police Department which
21 required me to converse on a regular basis with supervisors
22 throughout the department to address and respond to complaints
23 in litigation.
24 As the counsel to the superintendent, I not only was
25 engaged in that same process and procedure regarding approval
691
Klein - direct
1 of department directives, but I was also responsible for
2 reviewing all sustained complaints against department
3 employees, which made me very familiar with the practices and
4 procedures that were employed in our patrol division.
5 And now as the coordinator of education and training,
6 in order to devise training programs for new officers as well
7 as supervisors, I am in constant contact conducting needs
8 assessments to determine and be able to validate that the
9 programs that we are providing for our new hires and for our
10 supervisors and encumbents are relevant and provide the skills
11 that they need to operate in the field.
12 Q. Thank you.
13 Could you describe a typical day in the life of a
14 lieutenant serving as watch commander?
15 MR. FLAXMAN: I still don't think he has enough
16 foundation to say he knows what they actually do.
17 THE COURT: I disagree. Overruled.
18 BY THE WITNESS:
19 A. A watch commander in the patrol division works basically an
20 eight-hour shift. They start their tour of duty approximately
21 two hours, hour and a half, two hours before the police
22 officers and the supervisors start. They start earlier than
23 the supervisors and police officers for a variety of reasons.
24 When they arrive at their assignment on a daily
25 basis, one of the first things that they're supposed to do is
692
Klein - direct
1 to converse with the off-going, the supervisor, the watch
2 commander who is leaving at the end of the tour of duty to see
3 if there are any particular problems that he needs -- he or she
4 needs to be aware of during their tour of duty.
5 They are supposed to inspect the lockup facilities to
6 see if there are any prisoners remaining who have not yet been
7 bonded out from the prior tour of duty. They are supposed to
8 and do then enter their office and examine the scheduling for
9 that particular day to ensure that the appropriate number of
10 department vehicles, police cars are staffed and fully manned,
11 and to deal with any unanticipated absences which may require
12 them to make some adjustments in scheduling.
13 They are to look through the commanding officer's
14 book and any paperwork or communications which have been left
15 for them, either by the off-going watch commander or by the
16 district commander or by any other command member of the
17 department, to ascertain any particular duties and
18 responsibilities which may be foreseen for that particular tour
19 of duty for them.
20 After having done these things, they will conduct
21 their first roll call of the day. There are two roll calls for
22 each shift, sometimes more than two roll calls. During this
23 roll call process, they physically go to a roll call room where
24 the police officers and supervisors who are working on that
25 shift are present. They conduct an inspection to determine and
693
Klein - direct
1 ascertain that everyone has the appropriate uniform and they
2 meet our grooming and appearance standards.
3 On the Wednesday each week they conduct a weapons and
4 ammunition inspection of all personnel to make sure that the
5 weapons and ammunition that they are carrying conform to
6 department policies. They, after the inspection process, give
7 the officers and supervisors their assignments for the day.
8 They consult the commanding officer's book to provide any
9 notifications that the officers may need, notifications
10 regarding court appearances or special work assignments that
11 they may have or items of interest which are occurring in the
12 district or other places in the city.
13 After they conduct this process, they periodically
14 are required to air what's called a Vista program. Vista is a
15 closed-circuit training network that we in the department use.
16 The division in which I work, the education and training
17 division, produces training materials on many subject matter
18 areas. And we broadcast those via the closed-circuit system
19 throughout the city on a daily basis. And periodically the
20 watch commander is supposed to be providing that training and
21 answering any related questions during the roll call process.
22 After the roll call is concluded and after the second
23 roll call is also concluded, during which time the same process
24 would occur for the officers who are working the later segment
25 of that shift, the officer or the watch commander would
694
Klein - direct
1 generally return to their office, handle any administrative
2 issues that arose during the course of the roll call.
3 They are available in their office, because they have
4 to review and approve arrest reports for individuals who are
5 taken into police custody. They need to personally investigate
6 issues involving police officers. If police officers allege
7 that they were battered during the course of an arrest, for
8 example, the watch commander must personally conduct that
9 investigation.
10 The watch commanders constantly monitor via the radio
11 the activities that are occurring in the field during their
12 course of duty, because they are personally responsible for
13 responding to the scene of major incidents such as police
14 shootings or hostage incidents or major crimes which may occur
15 during their shift of responsibility.
16 They are required to periodically tour their
17 district, go out in the police vehicle and physically drive
18 through the police district to provide some command presence
19 during the course of their tour of duty. And they are
20 generally available to respond to any problems that occur in
21 the district in that they supervise the desk personnel and are
22 responsible for the proper bonding and letting to bail of
23 people taken into police custody.
24 BY MR. HOLZHAUER:
25 Q. Lieutenant Klein, are you familiar with the watch
695
Klein - direct
1 commander's office?
2 A. Yes.
3 Q. How are you familiar with watch commanders' offices?
4 A. Over the course of my nearly 25 years with the Chicago
5 Police Department I have personally been in -- I counted them
6 yesterday, actually -- I have personally been in 18 of those
7 25.
8 Q. Can you describe briefly the working environment of the
9 watch commander's office?
10 A. They're all somewhat different. Many of our facilities are
11 very old. Our 1st District, our 6th District, 7th District,
12 8th, 9th, 10th Districts, 23rd District and others were
13 constructed in the 1920s. They did not anticipate the
14 expansion and the increase in the number of people that we have
15 staffing these facilities. They're antiquated.
16 The ones that I just mentioned and some others are
17 not air conditioned. The ventilation is not the best in the
18 world. The offices are generally small. They house the watch
19 commander himself, perhaps a district secretary. There are
20 filing cabinets in the room. It may be adjacent to or
21 connected to a locker room where the supervisors have their
22 lockers. There are newer facilities that have a little more
23 space available to them. A substantial number of the ones we
24 have are somewhat cluttered.
25 Because the watch commander's office is, beyond the
696
Klein - direct
1 desk, is the focal point of some of the more significant
2 activities that occur in the district, there are people pretty
3 much coming and going on a regular basis seeking approval of
4 charges, advising the watch commander of occurrences throughout
5 the district, handling administrative matters. So there is
6 generally a constant traffic pattern occurring there.
7 As I indicated, they're constantly monitoring the
8 police radio so that they can be mindful and cognizant of the
9 conditions that are evolving in the field during the course of
10 their tour of duty.
11 That's pretty much a general picture.
12 Q. Are you familiar with the work of field lieutenants?
13 A. Yes, I am.
14 Q. Can you tell us how you've become familiar with the work of
15 field lieutenants?
16 A. Personally because I did serve as a -- working in the
17 special employment capacity, I worked in the field as a
18 lieutenant, and along the lines of some of the things that I've
19 just talked about being involved in drafting, drafting
20 directives which govern what they are and are not supposed to
21 be doing during the course of their day, being involved in
22 litigation involving supervisors in the department, and from
23 all my personal observations during the seven or so years when
24 I was working in the patrol division.
25 Q. Can you describe to us a typical day in the life of a field
697
Klein - direct
1 lieutenant?
2 A. A field lieutenant comes to work every day about an hour
3 and a half or so after the watch commander starts. The reason
4 he comes later is because that field lieutenant is responsible
5 for being there at the conclusion of the tour of duty of the
6 officers, because he has some responsibilities to ensure that
7 they complete their tour of duty and report information on what
8 their activities were during the course of that tour.
9 The lieutenant would also report to the district
10 facility, go into the watch commander's office, or if there
11 were an adjacent area where there was a locker room or
12 something, and get into the appropriate uniform if he or she
13 was not properly attired already. Many of them change, change
14 clothes at work and don't travel to and from in uniform.
15 They would check their mailbox to see if they had any
16 pressing matters. They would get their materials organized for
17 the tour of duty, tasks that they wanted to perform. They
18 would probably check the daily watch assignment sheets to see
19 who the supervisors were that were working and who the police
20 officers were that were working and to be brought up to date on
21 any particular occurrences which they could anticipate to
22 happen during their tour of duty.
23 They would then attend the roll call, where they
24 would assist the watch commander in conducting the inspection
25 and providing information to the officers present at the roll
698
Klein - direct
1 call. At the conclusion of the roll call, they would obtain
2 the keys for their assigned vehicle for that tour of duty.
3 They don't always have a vehicle, these vehicles are rotated.
4 At the end of their tour, they would give it to another
5 lieutenant who in turn would give it to another lieutenant.
6 They would identify the vehicle to which they were assigned for
7 that tour. They would get their radio. And they would go out
8 to their car.
9 And what they are supposed to be doing and actually
10 do do during the course of their tour of duty is they are the
11 field supervisor. They are responsible for all the activities
12 that occur in the field out on the street during that tour of
13 duty. They supervise the sergeants under their command.
14 They are supposed to respond to calls for service,
15 in-progress calls, particular calls, such as domestic violence
16 incidents. They are responsible for personally conducting
17 investigations in cases where a police officer discharged their
18 weapon or are fired upon. They are personally responsible for
19 conducting investigations of licensed premises. We have many
20 criminal incidents that occur in taverns or liquor
21 establishments. Because of the importance of those
22 investigations, we have decided that the lieutenants must
23 personally conduct those investigations.
24 On occasion, when we are very busy, the lieutenants
25 respond to calls for service. It doesn't happen with any
699
Klein - direct
1 degree of regularity, but they have responsibility for
2 providing police services just like any other police officer on
3 the Chicago Police Department. And they are available as a
4 supervisor for the police officers and sergeants that are in
5 the field.
6 Q. Now, do they have offices?
7 A. No.
8 Q. Do they work primarily out of their cars?
9 A. Yes.
10 Q. Now, you mentioned that they check out their radios as part
11 of their process. What are these radios used for?
12 A. It's a personal radio. It's about nine inches long by
13 about three inches wide by about three quarters of an inch or
14 an inch deep. And it's attached to your belt with a cord that
15 goes up to your shoulder area.
16 And the radio is basically a lifeline for our police
17 officers and our supervisors in the field. It informs them of
18 all the activities, all the calls that are being dispatched,
19 the status of all the officers. If there are emergency
20 situations, in-progress calls, calls for police officers
21 seeking assistance, all that information is constantly being
22 transmitted over the radio.
23 And this is the way by which the lieutenant
24 communicates with our communications center and the way in
25 which the lieutenant would communicate with any of the officers
700
Klein - direct
1 in the field.
2 Q. Do they carry out lengthy conversations on the radio?
3 A. Very, very, very abbreviated conversations. We may have
4 10, 20, 30 officers assigned to any given area. And it's
5 generally what's called a single radio frequency, a single
6 zone, and only one person can talk at any given time.
7 So conservation of resources is very important, which
8 is why we have numerical codes, alphabetical codes, which
9 communicate information over the radio in a very, very
10 efficient and prompt manner, because as I say, when one person
11 is talking, someone else can't speak. So if there is an
12 emergency, you always want to make certain that you have the
13 opportunity to use the radio to get through for help.
14 Q. Would a new lieutenant be expected to serve as a watch
15 commander during the first weeks or months of being -- after
16 being promoted?
17 A. Many years ago, that would not have been the case. But in
18 1991, a determination was made that the Chicago Police
19 Department was no longer going to promote or test for the rank
20 of captain.
21 As a consequence of that decision and during the
22 intervening years, we have attrited out, due to retirement or
23 whatever reason, slightly more than half of the 100 captains
24 which we used to have. The captains are the ones that were
25 primarily historically responsible for staffing those watch
701
Klein - direct
1 commander positions.
2 Given the fact that we have less than half of that
3 complement presently available to us and given the fact that as
4 we sit here today there are no more competitive tests or lists
5 available for promotion to the rank of captain, that function
6 is performed on a regular basis by lieutenants.
7 Q. And new lieutenants would do it?
8 A. Yes.
9 Q. How about the position of field lieutenant, do new
10 lieutenants serve in that role?
11 A. Yes.
12 Q. Lieutenant Klein, what is the Chicago Alternative Policing
13 Strategy?
14 A. The acronym for that strategy is CAPS, which stands for
15 Chicago Alternative Policing Strategy.
16 With the coming in of the Daley administration and
17 the administration of Superintendent Rodriguez, a very careful,
18 thoughtful look was given to the way the Chicago Police
19 Department provides police services. This is consistent with a
20 national trend in law enforcement.
21 For many, many years, departments have been incident
22 driven. Officers get into a police car, they respond to 911
23 calls. They run from call to call during the course of their
24 tour of duty, having little time to do proactive policing, if
25 you will.
702
Klein - direct
1 And our analysis showed that while this practice or
2 policing style resulted in thousands of arrests and the
3 recovery of thousands and thousands of weapons and a
4 significant number of convictions, it had no long-lasting
5 effect on crime or on the perceptions of safety and security in
6 the neighborhoods throughout the city.
7 So a conscious decision was made to alter that
8 strategy. And through a long process beginning in 1991 and
9 continuing today and will continue in the future, the style of
10 policing has changed. The new style of policing -- and this is
11 consistent with what the national trend is -- involves a
12 partnership between the police department, members in the
13 community, and city agencies.
14 The way the partnership works is rather than being
15 incident driven and simply responding to 911 calls for service,
16 the police officers now interact on a daily basis with the
17 residents of the area, the geographic area in which they serve,
18 we call it a beat. And through this interaction, there is a
19 problem solving and identification process.
20 Cooperatively the officers and the people in the
21 community discuss, talk about, articulate problems of crime and
22 disorder in their neighborhoods. After articulating these
23 problems, they agree cooperatively, they, the police and the
24 community, agree cooperatively on some priorities amongst those
25 problems of crime and disorder.
703
Klein - direct
1 After establishing those priorities, they again
2 cooperatively agree on some strategies, how they intend to
3 approach those problems to impact upon or reduce or eliminate
4 those problems. The community has a role in that process as
5 does the police department as does city services. City
6 services, for example, if there are problems with drugs being
7 sold at a given location in a neighborhood, city task forces
8 can seek to bring to responsibility the landlord. If buildings
9 are abandoned, they can seek to raze buildings so that those
10 opportunities for criminal activities are no longer present.
11 It's a partnership bringing to bear a multiple level of
12 resources on problems of crime and disorder.
13 And the shift to this policing strategy, which we
14 believe to be effective in the long term, has resulted in a
15 changing of roles for all of our police officers and
16 supervisors. The skill sets which we used to transfer to our
17 police officers involve technical skills, mechanics of arrests,
18 search, seizure, things of that nature, fundamentals of
19 investigations.
20 The skill sets which we now have been and continue to
21 try to transfer to our officers and supervisors relate to
22 problem solving, interpersonal communication, conflict
23 resolution, team building, motivation leadership, these types
24 of management skills which are required to work in cooperation
25 with the community to address some of these problems of crime
704
Klein - direct
1 and disorder.
2 Q. What is the role of a lieutenant in the CAPS strategy?
3 A. A lieutenant plays a very significant role. The process
4 that I just set forth was an overview of the planning process,
5 if you will.
6 Police officers on their individual beats work
7 through the process I just described to develop what's called a
8 beat plan. This is a physical document that chronicles the
9 problems of crime and disorder, sets forth priorities,
10 strategies, and has status reports on progress toward
11 addressing those problems.
12 When you aggregate all these beat plans, and there
13 may be, generally speaking, there are nine beats in a district
14 on each watch, each shift, when you aggregate these nine beat
15 plans, it is the responsibility of the lieutenant to review
16 these plans to ensure that the strategies that are being
17 recommended are realistic strategies to ensure that the
18 problems being identified are those types of problems that lend
19 themselves well to police or community activity, that they are
20 not related to root causes such as lack of education or lack of
21 employment or those types, lack of access to schools or jobs,
22 those types of things.
23 It is their job to basically coordinate and focus all
24 the resources at their command to address these problems of
25 crime and disorder. This is a key part of the process, because
705
Klein - direct
1 after the lieutenant has approved these beat plans for the
2 deployment of resources and the utilization of resources in a
3 police district, this plan then gets coupled with the plan that
4 was created on the other two shifts for those same beats and
5 then becomes a district-wide plan which then moves upward to
6 eventually become a citywide plan for dealing with problems of
7 crime and disorder and the utilization of resources throughout
8 the City of Chicago.
9 Q. Lieutenant Klein, you've been here through most of the
10 proceedings?
11 A. Yes, sir.
12 Q. Did you hear some of the sergeants testify that they have
13 fulfilled some of the duties of watch commander in the youth
14 division or housing or other special departments?
15 A. Yes, sir.
16 Q. Do those positions, being watch commander in those
17 divisions, is that the same as being a watch commander in the
18 patrol division?
19 A. No.
20 MR. FLAXMAN: Objection that we don't have foundation
21 for this. There is a limit, I think, to how much Mr. Klein
22 knows given his experience. And I don't think he knows
23 everything. But yet he's being asked questions as if he knows
24 everything.
25 THE COURT: It seems to me he's been all over the
706
Klein - direct
1 police department. He was a youth officer. He was a
2 detective. I mean, you know, I think you can cross-examine him
3 on these. But I don't see why he's not as capable as anybody
4 to tell me about the basics of the job as he knows it.
5 If you don't know something, Officer, you'll say
6 that.
7 But I think that he's got the knowledge to do this,
8 just based on what I've heard so far. So overruled.
9 But don't testify to something you don't know.
10 THE WITNESS: No, sir.
11 THE COURT: Okay.
12 BY MR. HOLZHAUER:
13 Q. Do the responsibilities of watch commanders in those
14 special divisions, are they the same as the responsibilities in
15 the patrol division?
16 A. Some of the things watch commanders in specialized units do
17 are similar to what watch commanders in the patrol division do.
18 They have some general responsibilities for
19 scheduling. They certainly supervise subordinate personnel.
20 They're responsible for conducting disciplinary investigations
21 regarding their subordinates. They grant overtime and approve
22 overtime. They give assignments to their subordinate
23 personnel.
24 The differences lie in the magnitude of the
25 responsibility. Specialized units are by their term, by their
707
Klein - direct
1 description, specialized. The youth division watch commanders
2 are concerned only with activities related to juveniles.
3 Public housing division watch commanders have a little more --
4 they have more varied responsibility, because they are dealing
5 with issues of crime in public housing developments.
6 But some of -- the fundamental difference I would
7 suggest are that they supervise less people than a watch
8 commander does. They don't have responsibility for responding
9 to certain types of crimes, weapons discharge incidents,
10 hostage incidents, licensed premise investigations.
11 They don't have responsibility for the day-to-day
12 provision of police services throughout the city responding to
13 calls for service, et cetera, like the people in the patrol
14 division do.
15 Q. Lieutenant Klein, in the various positions that you've held
16 for the Chicago Police Department, have you had any
17 responsibility for the development of departmental directives?
18 A. Yes.
19 Q. Can you describe what that responsibility has been?
20 A. From 1978 through approximately 1991, I was a police
21 officer assigned as an analyst in the research and development
22 division. The research and development division is that branch
23 of our department which has the sole responsibility for the
24 research, development and promulgation of department
25 directives.
708
Klein - direct
1 During that time I personally researched, developed
2 and wrote department directives which were later published.
3 After performing that activity or that task for several years,
4 as an administrative supervisor in the administration section
5 of the department, research and development division was one of
6 those divisions in my bureau and I retained, because of my
7 prior experience, oversight responsibility for all department
8 directives which were being developed, as well as training
9 bulletins and other training education materials.
10 Following that experience in the administration
11 section, as both the commanding officer of our legal section
12 and as general counsel to the superintendent, it was my
13 responsibility to personally review all department notices,
14 department special orders and department general orders prior
15 to them being reviewed and signed by the superintendent.
16 Q. Can you tell me what types of directives are in place?
17 A. The Chicago Police Department has three general categories
18 of directives. They are general orders, which are generally
19 the blue pieces of paper that you see. There are special
20 orders, which are the yellow pieces of paper that you see, and
21 there are department notices, which are generally gray.
22 In the hierarchy of importance, that replicates the
23 heirarchy. General orders are the most important because they
24 set forth policy, department policy. Special orders are next
25 in the heirarchy. They basically are setting forth department
709
Klein - direct
1 procedures. And department notices are more informational in
2 nature. They convey information about changes in statutes and
3 ordinances and the municipal code and convey information about
4 events of interest throughout the city.
5 Q. Why does the police department issue directives?
6 A. The Chicago Police Department is a large organization.
7 Approximately 16,000 personnel, 13,500 or so of which are sworn
8 officers and supervisors.
9 We cover a substantial geographic area. Much of what
10 we do is, unlike other organizations, dictated by ordinances,
11 statutes, the General Assembly, by judicial decision, by state
12 and federal law, by considerations related to medical and
13 health care. Because of --
14 THE COURT: Did you say "unlike other organizations"?
15 THE WITNESS: Unlike many other organizations.
16 THE COURT: Unlike, okay.
17 BY THE WITNESS:
18 A. Although there is a significant amount of discretion in the
19 police department, there is also a very prescribed set of
20 policies and procedure which all our officers must adhere to
21 for a variety of reasons related to safety, health care,
22 litigation, such overriding concerns as those.
23 Because we are such a large organization, because we
24 work 24 hours a day 7 days a week 365 days a year, and because
25 of the diversity that's present in the city with respect to its
710
Klein - direct
1 demographics, it's very important that our officers engage in
2 conduct that is consistent and uniform, not only for reasons
3 related to litigation or compliance with existing statutes and
4 laws, but for purposes of ensuring that people are treated
5 fairly and equitably throughout the City of Chicago by police
6 officers.
7 It is, therefore, important for us to have a uniform
8 and consistent body of rules or body of law by which the
9 overwhelming majority of our activities are conducted.
10 BY MR. HOLZHAUER:
11 Q. Could you describe for us how these directives are
12 developed and drafted?
13 A. It's a very detailed and thorough process, in my view.
14 What generally happens -- and the genesis of a
15 department directive can come from a variety of places. It can
16 come from a need to publish a new department policy, such as
17 our new policing strategy. It can come from some legislative
18 change. It can come as a consequence of litigation or a
19 decision of court. It can come from a variety of venues.
20 Once that, once that spark, once that genesis has
21 occurred, the research and development division comes into
22 play. As I indicated, they are the people with the
23 responsibility for crafting department directives.
24 When the germ or the seed for a directive is
25 transmitted to the department research and development
711
Klein - direct
1 division, a supervisor will review the task and ascertain which
2 analyst may have particular skills in that subject matter area
3 and then will assign a task to a given analyst. The analyst
4 may be a police officer, may be a civilian member, may be a
5 supervisor depending on complexity of the subject matter.
6 When that individual receives the task, they
7 basically do a literature search. Is this a subject for a
8 directive that is merely an addendum to an existing directive?
9 It is completely new subject matter? Based upon what the
10 nature of the need is, that analyst will conduct a literature
11 search. They'll try and amass any relevant or related
12 department directives that are out there, assemble any law
13 which may be related, any just general information in the
14 literature which would relate to the task at hand.
15 That analyst will then set forth a statement of a
16 problem and a proposed course of action. They'll identify what
17 they believe the problem is which requires the creation of a
18 department directive, and they'll set forth the steps by which
19 they intend to proceed to create that directive.
20 That plan of action, if you will, is then approved by
21 the supervisors in the research and development. And once the
22 approvals are obtained and any modifications are made, the
23 analyst goes to work. More of a research -- or more of a
24 literature search may be necessary depending on any
25 modifications to the plan of action.
712
Klein - direct
1 That analyst will contact experts to begin drafting
2 the skeletal of the directives. Experts could be in the
3 department in the form of department personnel who have subject
4 matter responsibility for this particular area. They could be,
5 if it's a matter related to legal issues, they would contact
6 our legal office. They would contact the department of law.
7 If there are union considerations regarding the subject matter,
8 our management labor affairs section would be consulted for
9 input. If it involves a matter related to health care or
10 safety, we have a liaison with the hospital counsel in the
11 greater Cook County area. There is a variety of sources of
12 information which that analyst is required to conduct in and
13 then document that he or she has talked with those people to
14 get relevant input from subject matter experts.
15 Having done these things, the analyst then makes a
16 draft of a directive. Using the schematic of the way we
17 develop directives in the department, following the general
18 format that we use, he or she would set forth what they believe
19 are the general terms of the directive.
20 Having done that, it goes through an internal review
21 in the research and development division, supervisory review.
22 And whatever comments or suggestions are made in that process
23 would be noted and changes would be made.
24 Once that process is completed in-house, we have
25 what's called a staffing. That document, that draft document
713
Klein - direct
1 is then distributed to all department command personnel, all of
2 our exempt personnel, approximately 90 or so of them, for their
3 review and input. This draft document would also go to
4 relevant subject matter people, such as attorneys in the
5 department of law or people in health care or whomever it may
6 be, who would have an opportunity to provide input as well.
7 When all this input is received, the analyst again
8 sits down, looks at the comments, either incorporates the
9 comments if he or she believes that they're appropriate or
10 necessary, has some discussion with the person making the
11 comments to try and reconcile things about which there may be
12 some difference of opinion, and addresses each and every one of
13 the comments that were made during the course of this staffing.
14 When all those comments have been satisfactorily
15 resolved to the satisfaction of the director of the research
16 and development division, that directive then goes through
17 legal review, it would have gone to me when I was the
18 commanding officer of legal affairs, and then to the general
19 counsel to ensure that all the Is are dotted, all the Ts are
20 crossed, that there are no longer larger issues out there that
21 were missed during this, the developmental process.
22 When the general counsel is satisfied that all those
23 concerns have been properly addressed and that the order is
24 suitable for publication, it would then go to the
25 superintendent of police, who would sign off on it. It would
714
Klein - direct
1 then become a department directive.
2 Q. Thank you.
3 Lieutenant Klein, in your various positions in the
4 police department, have you had any role in the development of
5 promotional exams?
6 A. Yes.
7 Q. First, can you tell us what exams you've played a role in?
8 A. I was a subject matter expert and departmental coordinator
9 for the -- it's called the D-2 exam. It's our youth officer,
10 gang crime specialist and detective examination.
11 I was also a subject matter --
12 Q. What year was that?
13 A. Pardon me?
14 Q. What year was that?
15 A. I believe it was 1993.
16 I was also a subject matter expert in the development
17 of the 1993 sergeants examination and the development of the
18 1994 lieutenants examination.
19 Q. Now, as a subject matter expert in the 1994 lieutenants
20 exam, did you review all three components of the exam?
21 A. Yes, I did.
22 Q. Were there other subject matter experts?
23 A. Yes.
24 Q. Who were they?
25 A. Chief of Patrol John Cadogan, Deputy Chief of the Detective
715
Klein - direct
1 Division William Shaw, and Commander Joseph DeLopez.
2 Q. Let's start with the written job knowledge component of the
3 exam.
4 Can you briefly describe what you did as a subject
5 matter expert during the development of that component?
6 A. In my capacity as a subject matter expert for the written
7 job knowledge test of the 1994 lieutenants examination, I went
8 to Akron, Ohio to the work offices of Barrett & Associates,
9 where I reviewed several hundred proposed test items.
10 The purpose of my review was to ensure, to the best
11 of my ability and based upon my experience in the department,
12 that the items as written tested subject matter areas which I
13 believed were important to the job and duties of a police
14 lieutenant in the Chicago Police Department; to ensure that the
15 questions and subject matter being tested were crafted in such
16 a way as to utilize language that was familiar to police
17 officers and supervisors in the Chicago Police Department; to
18 ensure that the test items and answer choices comported with
19 departmental policies and procedures and the relevant
20 contractual provisions, municipal code, statutory provisions in
21 effect in the state of Illinois; and to basically ensure that
22 in my experience, having taken exams and having been on the
23 department for 20 some years, that in my view that the
24 examination was fair with respect to its complexity and its
25 length.
716
Klein - direct
1 Q. Now, in evaluating all these particular questions, did you
2 rely upon any job analysis done by Dr. Barrett or HR
3 Strategies?
4 A. No, I did not.
5 Q. What did you rely upon?
6 A. I relied upon my education and my experience in my 20 some
7 years with the Chicago Police Department.
8 Q. Did you suggest any changes to the job knowledge test?
9 A. Yes.
10 Q. Did you suggest that questions be deleted?
11 A. Yes.
12 Q. Did you recommend additional questions?
13 A. Yes.
14 Q. Did the four subject matter experts confer together or work
15 independently? How did they work?
16 A. It was a combination of conferring and working
17 individually.
18 What happened was each of us was paired with one or
19 more of Barrett's staff and segregated in different rooms to
20 individually work through all the test items and make our
21 comments to Barrett staff members. On occasion, there were
22 questions for which each of the subject matter experts, myself
23 and my three associates, may have had some more experience or
24 some more specialized knowledge. So if a question arose as to
25 a particular question, we would occasionally consult and come
717
Klein - direct
1 to some agreement upon the best way to deal with that
2 particular question or deal with the issues that were raised
3 during the course of the review.
4 Q. Did you assess whether each of the exam questions called
5 for knowledge that a lieutenant should possess to perform the
6 job?
7 MR. FLAXMAN: Objection, because we don't know what
8 "exam question" means. The questions that were actually given
9 on the test or what he was looking at back in Ohio?
10 THE COURT: Good point.
11 BY MR. HOLZHAUER:
12 Q. Did you assess whether each of the exam questions you
13 reviewed called for knowledge that a lieutenant should possess
14 to perform the job?
15 MR. FLAXMAN: I'm still not sure if they're exam
16 questions or --
17 THE COURT: You mean the ones that he looked at
18 before the test was put together?
19 MR. HOLZHAUER: That's correct.
20 THE COURT: The universe of potential questions he
21 was given.
22 BY THE WITNESS:
23 A. Yes.
24 BY MR. HOLZHAUER:
25 Q. What was your conclusion?
718
Klein - direct
1 A. Based upon my experience in the Chicago Police Department
2 and my 20 some years as a police officer and a supervisor,
3 after having conducted my review and made the comments and
4 suggestions that I made, I believed that the questions that
5 remained after my review were reasonable questions, were
6 questions that I believed it fair to expect a police lieutenant
7 on the Chicago Police Department to know, and that the
8 questions sought to elicit information with respect to subject
9 matter which was -- which was fair, which was equitable, which
10 was consistent with department policies and procedures and
11 practices.
12 Q. So there were some questions in your initial run through
13 that didn't call for a knowledge that a lieutenant should
14 possess on the job?
15 A. There were a number of questions that I believe, while
16 taken from the subject matter taken from the -- contained on
17 the reading list, some of the questions were somewhat ambiguous
18 or somewhat vague or sought to test subject matter which
19 occurred with such a degree of infrequency and lack of
20 criticality as to not be worthy of having a test item.
21 Q. What was your recommendation as to those?
22 A. I would ask that they be deleted from the bank of
23 questions.
24 Q. Did you review the in-basket component of the exam?
25 A. Yes, I did.
719
Klein - direct
1 Q. Did the other three subject matter experts review the
2 in-basket component?
3 A. Yes.
4 Q. What did your review of that component consist of?
5 A. We reviewed -- I reviewed all the material that was
6 proposed to be presented to the candidates. I looked at the
7 material to see if I thought that this was material related to
8 what a police lieutenant in Chicago does. I looked at the
9 material to see if the forms and the paperwork that were being
10 presented to the candidate were reasonably consistent with what
11 we use in the Chicago Police Department.
12 It was my intent to not have -- not to confuse the
13 candidates. I didn't want the test instrument to contain forms
14 which were completely foreign to the officers. I knew that
15 they were going to have -- be under enough pressure as it was
16 taking the exam. They didn't need to take time to learn new
17 forms. So I wanted to make sure to the extent practicable we
18 utilized forms that we utilized in the Chicago Police
19 Department and we utilized processes of conveying information
20 that were consistent with what we do in the Chicago Police
21 Department.
22 I looked at the bank of multiple-choice questions
23 which were related to the materials. I looked at the general
24 subject matter area which was being -- which was being tested
25 for the purpose of ensuring that it represented duties and
720
Klein - direct
1 responsibilities of a lieutenant, things they would do on a
2 regular basis, and things that were very important for a
3 lieutenant to know.
4 Q. Did you recommend any changes to the in-basket component?
5 A. Yes.
6 Q. Were those changes made?
7 MR. FLAXMAN: Objection. No foundation for whether
8 or not -- how he knows how many changes were made.
9 THE COURT: Well, did you see the ultimate test?
10 THE WITNESS: No, sir.
11 BY MR. HOLZHAUER:
12 Q. Did you evaluate the time allotted to the exercise?
13 A. Yes.
14 Q. Did you make any recommendations with regard to the time?
15 A. It was my belief, based upon my understanding of the
16 testing environment and having been through the testing
17 environment a few times in my career, that we were not
18 providing enough time to the candidates and that we were giving
19 them too much material for them to reasonably be able to work
20 through in the time allotted.
21 So it was my recommendation that the time for that
22 component of the examination be increased and that the number
23 of documents and the number of tasks the candidates were being
24 requested to perform was reduced.
25 Q. Do you know if those recommendations were followed?
721
Klein - direct
1 A. Yes.
2 Q. How do you know whether they were followed?
3 A. Because I know that the time allowed to take the
4 examination was increased. I believe it was a half hour for
5 the in-basket component. And I know that those areas which we
6 suggested they not include or reduce in complexity were either
7 not included or made simpler.
8 Q. Did you also review the oral briefing exercise?
9 A. Yes.
10 Q. Did the other subject matter experts review that as well?
11 A. Yes.
12 Q. What did your review consist of?
13 A. We looked at the process. Historically, we have had
14 problems with promotional examinations in the City of Chicago.
15 In the last few examinations, some of the problems that
16 occurred related to the oral component of the examination. So
17 we tried to learn from our experience and from past mistakes
18 that have occurred.
19 So we looked at the process itself to ensure to the
20 best of our ability that it was fair; that it reduced or
21 eliminated any likelihood of rater bias or favoritism, which
22 were some of the complaints we had received in the past. We
23 looked at the process to ensure that it was as objective as
24 possible.
25 We liked the fact that they were using outside raters
722
Klein - direct
1 rather than department personnel, because that has caused us a
2 lot of problems in the past when we utilized department command
3 members as raters, claims of favoritism, et cetera, were
4 rampant in the last examination process.
5 Beyond looking at the process for those types of
6 things which we thought we wanted to try and adjust to make
7 sure the exam was perceived as fair by all the candidates and
8 to the extent at all possible that it was objective for the
9 candidates, we looked at what they were being asked to do in
10 this period of time. Was the amount of time they were given
11 reasonable? Is the task that they were being asked to perform
12 something a lieutenant should be able to do right away? Was
13 the material complex? Was the material esoteric? Was it
14 something that our lieutenants would come into contact with on
15 a regular basis?
16 We looked at all those types of issues, and we made
17 some recommendations as to changes in time frames and mechanics
18 as to process and made also recommendations regarding some of
19 the subject matter that was contained which was ultimately to
20 result in the oral presentation which the candidates would
21 make.
22 Q. Can you briefly tell me what you recommended with regard to
23 the time allotted to the exam?
24 A. As with the in-basket component, it's my recollection that,
25 you know, based upon my experience and having worked here for a
723
Klein - direct
1 very long time in the Chicago Police Department, we wanted to
2 give them a little more time.
3 We didn't feel that the time allotted by Barrett &
4 Associates in their original recommendations gave the
5 candidates sufficient time given the constraints and given the
6 pressure that they would be under during the examination
7 process.
8 Q. Did you assess whether the oral briefing exercise was
9 similar to tasks lieutenants would face on the job?
10 A. I believe it's similar. It's not something that a
11 lieutenant would do every day on the job. But the fact
12 patterns involve criminal activity occurring in their district.
13 These things arise every day throughout the city in varying
14 degrees of frequency, so we believed -- I believed that it was
15 reasonable for a lieutenant to be able to assimilate some
16 information regarding criminal activity and be able to convey
17 that information to his subordinates.
18 Q. Did you review the reading list that candidates were given
19 for the lieutenants exam?
20 A. Yes.
21 Q. What did that reading list include?
22 A. It included references to subsets of our directives, some
23 of our general orders, some of our more important general
24 orders, some of our special orders.
25 I believe there were some notices, references to
724
Klein - direct
1 certain sections of the Illinois Compiled Statutes, to certain
2 limited sections of the Municipal Code of the City of Chicago,
3 to certain discrete sections of the collective bargaining
4 agreement between the department and the Fraternal Order of
5 Police, and to a document called "Together We Can," which is a
6 document authored by our police superintendent which sets forth
7 the conceptual and philosophical parameters of our new policing
8 strategy.
9 Q. Now, what did you review that reading list for?
10 A. I looked at it to see if in my view it contained reference
11 or contained materials which were important for police
12 lieutenants to know, which were reasonable to expect police
13 lieutenants to know.
14 And also in my capacity as general counsel at that
15 time, I was aware of issues affecting the department or
16 impacting the department that perhaps some of the subject
17 matter experts may not have had access to, issues related to
18 the First Amendment consent decree which the department is
19 subject to, issues related to the new policing strategy as it
20 evolved.
21 So I believe that I made some recommendations that
22 there be some subject matter added which would address some of
23 those matters for which I had information, which I believed
24 that some of the subject matter experts may not have had that
25 information.
725
Klein - direct
1 Q. Was the CAPS report on the original reading list?
2 A. I don't really recall.
3 Q. In your opinion, were the items on the source list
4 important for the performance of the job of lieutenant?
5 A. Absolutely.
6 Q. You testified earlier about the general orders and special
7 orders and directives. Were all of those on the reading list?
8 A. No. There was a subset. I couldn't give you a proportion
9 to give you a sense of perspective. But over time historically
10 in the department we've, based upon the problems associated
11 with promotional examinations in the city and in our
12 department, as I say, we try to learn from past, past mistakes
13 that we've made.
14 For example, when I took my sergeants examination, we
15 were required to study the universe of directives. It was not
16 these particular general orders or special orders or municipal
17 code sections or whatever it may be. We were required to study
18 everything. And I always thought that that was patently
19 unfair, because some of what we've got is highly technical in
20 nature. You never or very, very seldom would run into it. And
21 some of the stuff is so historically dated that it really bore
22 no relevance or meaning to the department. So I always felt
23 personally somewhat nonplussed that we had to study this
24 universe of materials.
25 And as the last couple examinations have been
726
Klein - direct
1 administered, I think we've become much more reasonable in our
2 approach and try to focus the candidates' attention on those
3 directives and those provisions of directives and statutes and
4 ordinances and the contract with which they will come into
5 contact on a regular basis or which contain such a high degree
6 of importance that they needed to be familiar with the
7 information.
8 MR. HOLZHAUER: Thank you.
9 Your Honor and Mr. Flaxman, I'm about to go
10 through -- we have the general orders and special orders in the
11 record already. I'm about to go through the notices which are
12 Defendant's Exhibit 9; the FOP contract, 10; CAP strategy, 11;
13 criminal code, 12; municipal code, 13. Then I'll have
14 Mr. Klein identify them and to offer them into evidence, unless
15 we can just get them in by stipulation.
16 MR. FLAXMAN: I thought we already did that.
17 MR. HOLZHAUER: We haven't.
18 MR. FLAXMAN: Well --
19 MR. HOLZHAUER: They were objected to.
20 MR. FLAXMAN: We can -- I'm sure we can work this out
21 without Your Honor sitting on the bench and a witness on the
22 stand.
23 THE COURT: Well, is there any reason not to admit
24 those into evidence? They are what they are.
25 MR. FLAXMAN: That's correct.
727
Klein - direct
1 THE COURT: The reading list is what the reading list
2 is. The test is what the test is. I think we're -- I think
3 we're sort of digressing here a bit this afternoon. You know,
4 I think we should try to tighten this up and get into the
5 record what we need to get into the record here.
6 Do you have any objection to the admission of the
7 documents Mr. Holzhauer just described?
8 MR. FLAXMAN: No.
9 THE COURT: All right. They're admitted.
10 (Defendant's Exhibit 9, 10, 11, 12, and 13 received in
11 evidence.)
12 MR. HOLZHAUER: Your Honor, I'm about to go through a
13 lot of the general orders that have been examined on direct and
14 on cross of other people. I wonder if this is an appropriate
15 breaking time?
16 THE COURT: It probably is. All right. We'll start
17 at 10:00 o'clock tomorrow.
18 How are you doing on time?
19 MR. HOLZHAUER: I think I probably have about -- I
20 would say about half of -- maybe a third of Mr. Klein is done
21 at this point. So I'm going to have another hour, let's say,
22 on Mr. Klein. Then I have two very short witnesses after that
23 tomorrow.
24 MR. FLAXMAN: How much time do we have tomorrow?
25 THE COURT: Until 4:00, okay. All right. We'll
728
Klein - direct
1 start at 10:00. We'll quit at 4:00. I have a judges' meeting
2 tomorrow. So we'll have a little longer lunch break than
3 usual. But we should be able to finish your case tomorrow very
4 easily.
5 MR. HOLZHAUER: I believe so, yes.
6 THE COURT: All right. Let's try to tighten it up.
7 I think we all know what we need to put into this record.
8 Then you are going to wait until next week to put on
9 your remaining witnesses, and you think you have two days?
10 MR. FLAXMAN: Yes.
11 THE COURT: I'm already filling up those other two
12 days that we had set aside.
13 MR. FLAXMAN: If I'm not done in two days, I won't
14 put anything else on.
15 THE COURT: Okay. You heard him. I'll see you
16 tomorrow.
17 (Adjournment at 3:15 p.m. until 10:00 o'clock a.m.,
18 November 25, 1997.)
19
20
21
22
23
24
25
part 5
Main