Here are excerpts from the post:
In “Measuring Effectiveness of Access to Justice Initiatives,” a panel discussion [at the Canadian Bar Association's Envisioning Equal Justice Summit] held Saturday morning, there was a discussion about the lack of standard definitions, which limits organizations’ ability to evaluate and compare initiatives’ effectiveness. For example, what is a “case” for the purpose of analyzing effectiveness of a program? Legal professionals tend to think of a “case” as an individual matter and would generally measure it that way, but the speakers observed that individuals interacting with the justice system tend to think of all interactions with the court system as one experience, so criminal, family law, landlord and tenant, and employment proceedings may all affect perceptions of each other. How can cause and effect be measured when there is no standard linking system among matters that are linked in the minds of the parties? Because participants’ perceptions of the assistance provided by organizations are a component of success, this is not an irrelevant consideration in development of analysis techniques for access to justice initiatives.
Some data surrounding court processes is starting to be made available online. For example, British Columbia’s DataBC lists the following datasets from Court Services. This is meaningful progress, but the available datasets don’t provide detailed information that would enable analysis of the social aspects of the justice system, though a simple analysis of numbers of matters opened and closed would be possible, which is one possible measure of productivity. Court Services Online, the closed system providing access to internal court documentation and information, provides much more extensive information on a cost per use basis. It is likely privacy constraints will continue to make full public access to information in a format that is easily linked with individuals undesirable, but that doesn’t mean organizations and researchers looking into access to justice issues shouldn’t collect the information to be used internally and potentially shared with other organizations and researchers.
This lack of shared definitions of elements in the legal environment also extends to matters’ outcomes. Contrary to the expectations created by Hollywood movies, where so many lawyers “have never lost a case,” legal outcomes are complex and can be considered wins or losses depending on context. It is difficult to evaluate the effectiveness of interventions when situations and outcomes are so varied.
Legal information also tends to be distributed in semi-structured formats, which makes it difficult to extract outcomes without manual examination of the text. Some information can be extracted using automated algorithms or techniques like regular expressions, but these require an available set of standard documents to work from. This documentation is not always readily available for the kinds of issues most commonly dealt with in access to justice initiatives, as written judgments are often not published for routine matters. [...]
For more details, please see the complete post.