I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be. – Baron William Thomson (Lord Kelvin), Popular Lectures and Addresses, p.73, “Electrical Units of Measurement” (May, 3, 1883).
Most resources have a set of metrics to measure performance, effectiveness, and value. Law firms and departments predominately focus on financial performance metrics (e.g., profit, revenue, WIP, matter value). These metrics are invaluable for good reason – they help leaders manage the business of law. By contrast, when it comes of the practice of law, such as the drafting of contracts, data is infrequently captured and few metrics are identified.
Precedents banks, forms and templates (collectively “precedent collections”) serve as a foundation for legal drafting. These collections enable lawyers to generate high quality contracts. Lawyers intuitively understand the practical value of “standard” and “market” legal terms. In addition to supporting a more efficient practice, ready access enables the lawyer and client to focus on business issues and outcomes. But absent meaningful metrics, how do you know if your precedent collection strategy is successful?
Why are you in the precedent collection business? Is it because a specific practice or lawyer wants it? Because that is the way everyone does it? There has to be more to it than that if you are going to (continue to) invest money and time driving processes that connect people, content, and systems to create and maintain precedent.
Ideally, you already focus on specific metrics and goals to measure the success of your precedent efforts. The resulting analysis may support continued funding for and enhancements to your existing resources. Your analysis may even help you deliver new, insightful products and services to your lawyers and clients based on your capture of business terms and best practices. If are not focused on metrics, now is the time to start.
Every organization has a different culture and strategy. Although you may be inclined to focus on a range of metrics associated with your organization’s strategic goals, start with the fundamentals — metrics focused on precedent creation, maintenance, and use. These metrics will help you manage your precedent collection and may also help you measure its business value.
Precedent Creation & Maintenance
Effective precedent collections depend on a seamless blend of people, systems, and processes. When all three are aligned, the collection is consistently relied upon to generate contracts. At the other end of the spectrum, the collection can simply be a costly investment that occupies impressive space on the virtual shelf. In order to understand if you have allocated your resources efficiently, focus on four (4) metrics – output, input, cycle time, and cost.
- Output: Identify which precedent templates are necessary to run your business. What document types will deliver the greatest value to your users? More complex documents will likely require a greater investment of time and/or people with specific experience. Additionally, over time you may generate a lower volume of more complex documents, suggesting a different schedule for updates by more senior staff. Consider the number of templates you will produce (and separately update) in a given period (month/quarter/year). Separately, understand your how many templates remain a work-in-progress as you will need to manage them and user expectations.
- Input: Identify the people, content, and systems that are necessary to create and maintain precedent. How many people are devoted to precedent creation and maintenance? Your strategy may involve professional support lawyers (PSLs), knowledge management staff, paralegals, or volunteer/voluntold lawyers. What are their roles, practices, and locations? Will you staff your efforts internally or do you outsource, either with people or licensed content? Explore if you can use systems to identify standard clauses and documents to make the process more efficient.
- Cycle time: Calculate how long it takes you to develop a precedent from inception to completion. Does this cycle vary by author, practice, subject matter, or location? In order to align your resources efficiently you should understand the cycle time and the reasons for any variances associated with these factors.
- Cost: Consider the cost to operate your precedent collection based on your input, output, and cycle time metrics. Track the hours and associated cost rates for the people involved in the process. Add the cost of licenses and systems to calculate your total cost to produce and maintain a template. Again, understand if there is there a difference based on the document type, practice, or location that will use the templates.
Does precedent use match your expectations and investments? Are your users actually using your precedent as intended? Your users may rely on templates to generate first drafts of contracts or your users may simply exploit your precedent collection as a clause bank. Does the latter use case align with your goals or would reliance on search be a better investment? Focus on four (4) quantitative metrics (baseline, usage, timing, and conformity) and one (1) qualitative metric (alignment) to understand how your precedent collection is used.
- Baseline: Capture your baseline numbers – those associated with contract drafting in the absence of a precedent collection. Lawyers typically start their initial contract drafts from a previous agreement or by locating similar documents via e-mail or search. Understand the process, the average number of hours and versions, the levels of the people doing the work, and the associated costs. Although difficult to capture, this metric will help you gauge how your precedent collection impacts the way your users draft contracts.
- Usage: Track the numbers associated with precedent and template use. How many people are using the collection and what is the experience level of the individuals who are doing this work? Are your templates intended for more junior/senior lawyers or non-lawyers? How many templates are used and how often? Which ones? Your analysis may suggest focusing on different template types or audiences.
- Timing: Collect data about the number of hours it takes to draft a contract based on your precedent collection. Increased efficiency is one benefit of a precedent collection. What is the turnaround time for a first draft? A final contract? Can you track the number of days/hours through time entries, document history, or data from your document management sytem? Review your data by practice, location, and individual to understand if any specific attention is warranted. For example, limited use by an individual may suggest the need for additional training. Although variations based on location or practice can also indicate training issues, they may also suggest market-specific factors require template adjustments.
- Conformity: Compare your final executed contracts against your templates. Where does the contract language conform to the template and where does it diverge? Analysis by contract type, author, location, or practice can again provide insights. A lack of conformity may suggest that a template needs revision or that additional outreach and education is necessary.
- Alignment: Canvass your users — lawyers, clients, and non-lawyers — to identify practical concerns associated with your precedent collection. Are they satisfied with the collection (access, ease of use, quality)? Do your users know how to find and use your precedent? Separately, do your users believe that the quality of your precedent collection meets their needs? Understand if your templates strike the right balance. If they are too conservative, a contract based on a template may require prolonged negotiation and modification. User feedback may indicate that template clauses in support of or in lieu of a template contract will address the primary pain points.
Beyond the basics, you may want to identify additional metrics focusing on the business value driving your precedent collection efforts. The definition(s) of “business value” necessarily varies by firm/department size, location, and overall business strategies. By way of example, a precedent collection can impart “business value” in one or more of the following ways:
- consistently higher quality contracts
- new products and services
- increase (or maintain)
- client satisfaction
- market share
- scale/yield greater efficiency
- location or office
- administrative functions
- best practices
- young lawyers/non-lawyers
- business model of a particular practice and/or location
- manage (or reduce) risk.
Your definition of “business value” will help you identify specific metrics and more granular goals for your precedent collection (e.g., 8% increase in matters meeting a fixed budget/fee, 20% reduction in time to create a first draft, 10% increase in number of contracts/period).
Gathering the Data
Identify what data you have before you finalize your metrics and goals. You (should) have access to lots of data, structured and unstructured. Lawyers find, create, edit, store, process, manage, and share contracts and precedent content using various systems. However, system metadata is typically designed to support and maintain the system, not your precedent collection. Understand what data is relevant to support both your precedent collection and the underlying system(s). If you cannot collect accurate or relevant data, you may not be able to produce a meaningful analysis.
The list of potential data sources varies by organization, system design, and precedent collection strategy. You may focus on one or more of the following systems:
In light of increasing requests for dashboards and big data analytics to help manage the business, the temptation is to measure everything. For every element of your precedent collection processes and systems, there may be scores of metrics. Separately, different stakeholders may find value in different metrics. As you consider precedent collection metrics, keep your focus on the metrics that will help you understand and manage your precedent collection resources. In this way you will be positioned to improve your collection and truly understand if it is delivering your law firm/department the promised business value.
- Although precedent collections also support litigation practices, this article focuses on the use of precedent collections to support contract drafting. ↑