How to Measure
A cursory review of the above criteria demonstrates the traditional three-pronged method of vetting potential candidates is seriously insufficient. Can a resume, an in-person interview, and a perfunctory reference check provide sufficient insight into a candidate’s ability in all of these categories, or even some of them? Let’s review the measuring tools we have at hand (in the order in which they should be used) to improve the vetting process.
3.1 The Resume
Every evaluation of a potential candidate begins with a review of the resume (we will assume for purposes of this chapter that the resume describes a skill set that is relevant to the open position). And, while limited in its utility, the resume remains the single most (although not exclusive) tool for measuring one aspect of aptitude: advanced legal reasoning. The experience of most general counsels and hiring managers is that academic credentials and law school performance are reasonably reliable indicators of high achievement in the area of advanced legal reasoning. A good law school and strong performance within law school generally correlate to the ability to navigate complex legal concepts and wed them to complex fact patterns to arrive at sound legal results. After all, in order to achieve these credentials, candidates have already gone through a battery of advanced reasoning tests to be admitted to law school and to perform well once they are there.
However, the resume is a poor measuring device to assess almost every other attribute that will be important to achieving success in a legal department. Many general counsels are tempted to also assume that, if the candidate’s early career has been spent at a top-flight law firm, s/he will necessarily have acquired the training to become an adept drafter of contracts and other legal documents. Unfortunately, the correlation between high prestige law firm training and high aptitude for drafting skills is substantially more tenuous than the correlation between advance legal reasoning and academic credentials.
As a result, many of our clients now look to a customized writing exercise (discussed below) as better tool for measuring drafting performance.Many clients are also tempted to look at the work history on a resume and draw conclusions about a candidate’s employment stability. Lots of moves tend to be a red flag for many clients, suggesting that the candidate might be a job-hopping malcontent. However, in today's world, there are so many perfectly legitimate reasons for cycling through several positions that drawing such a conclusion from the resume would not be fair to the candidate.
And, of course, with respect to every other attribute that will be measured (risk assessment/business literacy, communication, collaboration, time management, etc.), the resume is practically useless as a vetting tool. Fortunately, there are other tools available to measure these other important qualities.
3.2 The Written Exercise/Drafting Skills
It is important at the outset to distinguish the written exercise from a writing sample. Many clients (although certainly not the majority) do ask candidates to provide a writing sample, typically after the initial interview. This request gives the candidate wide latitude to choose a writing sample that may or may not have been written under a tight deadline, may or may not have been written with the help of others, and may or may not have been edited by a third party. In short, the writing sample may not either be representative of the type of drafting the candidate will be asked to undertake in the company, or even be substantially the candidate's own work product. There is a much better way.
We are seeing a clear trend among our clients to ask candidates to take a written exercise (many still shy away from the word “test”). The exercise can take several forms. Some clients ask the candidates to mark up a draft agreement that the client may have purposefully sprinkled with errors, both substantive and typographical. Other clients present the candidate with a term sheet and ask the candidate to draft an agreement reflective of the terms.
Each client can determine what is the best type of exercise to present, but it is important that the exercise be designed to reveal drafting skills of the candidates that are relevant to the work they will be asked to perform. Typically, the candidate is given a time deadline in which complete the exercise (often the candidate will be asked to block out a two hour window where the exercise will be emailed to them and they will have to email it back within the two hours).Our clients who use these written exercises have found them to be hugely valuable vetting tools that are most appropriately administered before the in-person interviews (although potentially after a phone screen). These are so valuable that we also recommend that the clients administer another much briefer written exercise to measure the candidate's ability to write internal communications to non-lawyers explaining a legal concept. This is usually a brief two or three paragraph email the candidate might be asked to write to the head of sales explaining new revenue recognition rules, for example.
3.3 Psycho-Metric Testing: Advanced Reasoning and Personality
These types of tests have been around for a long time and have made very few in-roads into the vetting process of lawyers. Part of the reason may be that the credentials on the resume are usually viewed as sufficient for establishing advanced reasoning capabilities. Another reason is that lawyers are skeptical of personality tests generally and feel they are too generic and easily gamed.
These objections have some force to them but we believe these tests can be helpful in certain circumstances, particularly in those companies where the tests have been found to be useful in vetting candidates for other functions. The personality tests can help identify potential strengths or weaknesses in the area of collaboration and teamwork. The advanced reasoning tests can also help to clarify the nature and scope of a candidate’s reasoning abilities, particularly if a candidate possesses less than stellar academic credentials but otherwise appears to have solid and relevant experience.
Rather than simply rejecting such a candidate, an advanced reasoning test might prove useful in establishing that candidate’s capabilities. Each client will have to determine for itself whether the use of such tests is appropriate, but they should be considered as part of the mix.3.4 The Interview
Now comes the truly hard part. There is simply no way around the acknowledgement that the interview is the single most important, and single most squandered, tool for vetting candidates. Properly conducted, the interview can shed light on nearly every category of importance. Unfortunately, more often than not, the interview devolves into a rambling exchange of potentially relevant (or not) anecdotes. The interviewer either achieves that level of “comfort” with the candidate or not. More often than not, the main criteria that are so important to the metrics-driven environment in which we operate are barely explored.
Interviewers need to have a plan for each interview. Our own experience teaches us that the interview is particularly valuable as a measuring tool for the following criteria: risk assessment/business literacy; communication skills; responsiveness/ time management; cost containment/management of outside counsel; and collaboration and teamwork. Of course, the interview can also probe the issues of advanced reasoning and the candidate’s understanding and mastery of relevant legal principles.
Time and space constraints prohibit setting forth a comprehensive list of all the questions that could be asked in an interview to elicit valuable information, but when formulating an approach to the interview, the interviewer might consider alternating between more general questions and more specific ones. In my own experience, I have found that when testing for risk assessment and business literacy, open-ended questions tend to elicit very useful answers. For example, I might ask the candidates to generally describe the business models of the various companies where they have worked.
In their answers I look for an understanding of how the products are developed, how they come to market, how they are priced, how they are sold and distributed. In short I’m seeking from the candidate evidence that the candidate understands the business context within which the legal advice is rendered. Similarly, I might ask an open ended question to a candidate who worked on a M&A deal: “Why did your client buy that company?” A surprising number of candidates can struggle with that answer.Conversely, I have found more specific questions are useful when measuring the criteria of responsiveness/time management and cost containment/management of outside counsel. For example, I might give the candidate a list of three or four projects that could be on their plate and ask them to specifically prioritize their approach and give me an idea of how long they might take to complete each task. I will also ask them to give me hard numbers on the cost of outside counsel for various assignments and what they think is reasonable for outside counsel to charge. Of course, they must demonstrate an understanding of the market for fixed fee arrangements and discounts.
When it comes to communication, there is nothing like the art of listening. A candidate won't be good at communicating if they aren't good at listening (and an interviewer won't be good at measuring a candidate's communication skills if they aren't good at listening either). The threshold issue is this: is the candidate listening to the question and responding to the question that is asked rather than answering some unasked question that the candidate would prefer to answer? If they are on track, then I have found it useful to ask a very direct question: “Pretend I am on the sales team, please describe to me the potential risk to the company if we agree to this indemnity provision (insert an appropriate indemnity clause).” Do they respond clearly, concisely, respectfully and in a manner that a non-lawyer can understand?
It is important to remember that communication within a company takes place in three principal settings: the group meeting; one-on-ones; and written communications.
We have already tested for written communication and the interview handles the one-on-one. How about the meeting setting? How can we measure whether a candidate will function effectively in the group meeting with all of its unpredictable conversational flows, protocols, and frustrations! The group interview is the answer. As part of the interview process, many of our clients have found that in addition to one-on-ones, it is valuable to subject the candidate to a group interview to see how they handle over-talk and whether they can be inclusive and respectful when fielding comments or questions they might find irrelevant. Do they interrupt? How do they handle interruption? Can they effectively insert themselves during cross-talk? Will they let people have their say or try to shortcut the conversation? Success in meetings is a key component that informs how the client will evaluate a member of the legal department. The interview process should attempt to partially replicate that setting to see how the candidate manages it.Collaboration and teamwork are the most difficult criteria to measure. As noted above, psycho-metric personality tests can be potentially useful. The interview can also give you a sense of a candidate's collaborative skills, but it is very unlikely that a candidate will overtly demonstrate bad teamwork tendencies. Since this is such an important criterion, rather than just trust to hope, it is important to turn to third party testimony. This is the one area where the under-used reference check can be very valuable.
3.5 The Reference Check
As noted above, it is important to conduct the reference check at a point in the process where it can be of use, rather than as an afterthought once the offer has been extended. Ideally, in addition to the references that the candidate provides, it is very useful to “back channel” the candidate with others who may have worked with the candidate but who are not on the official reference list. Of course, this can be tricky, and clients must be mindful of keeping a candidacy confidential. But if recourse to an unofficial reference is available and appropriate, it is wise to contact that reference.
Once the list of references is complete, what is the best way of utilizing this resource? Again, it is important to have a plan and prepared questions rather than the open-ended “how was Jane Doe to work with, was she a good colleague?” We have found that the best approach is to ask the reference to cite specific examples of a candidate’s performance in each of the relevant categories. For example, “can you give me an example of an instance where Jane Doe ran into some uncooperative colleagues on a particular matter and describe how she navigated to a successful outcome?” “How does she perform in a meeting context, and can you give me an example that illustrates her style of communication within the meeting context?”
3.6 Damning with Faint Praise
The way references answer these questions can be very illuminating. Very rarely will they say anything overtly negative. However, if they can’t readily retrieve any examples that illustrate the topic at hand or if they don’t seem very interested in even trying, that can be an indicator of a lukewarm feeling about the candidate. Of course, they just may be too busy to invest the time in giving you a substantive answer in which case it is best to move on to another reference. More often, a reference who is not that enthusiastic about a candidate, will heavily qualify their praise of a candidate. A heavily qualified reference is usually a warning sign.
4