4. Evaluating and Understanding Impact

Questions regarding an expansive digital publication’s impact and value often arise in tenure or promotion reviews. The wider scholarly community, however, understands these publications in terms of how they contribute to relevant fields of inquiry. Remarkable or pivotal contributions in one field may be commonplace in another; evaluation of digital work, as for any scholarly product, depends crucially on scholarly context, whether disciplinary or interdisciplinary.

In the case of expansive digital publications, the scholarly community also needs processes for gauging a work’s broader influence. Given their potential to engage audiences beyond the typical readership of scholarly books and journals, expansive digital publications require useful ways to capture this reach and impact. As other sections of this report describe, such assessment may also provide key data for decisions about funding, resource allocation, and long-term support.

Challenges and approaches to assessing impact

In order to address the challenges of evaluating expansive digital publications, we must clarify the intent of assessment: what purpose and audience does it serve?

• Highlighting a faculty member’s effort as part of the promotion and tenure process

• Identifying how the publication contributes to a discipline or emerging junctures between or among disciplines

• Tracking the publication’s influence more broadly, whether among the broad public or narrower constituencies, such as K-12 educators, teachers at the tertiary level, or even policy-makers/decision-makers

• Justifying a program or a project for continued funding

• Making visible the labor and other resources required for its production

Holistic evaluation of impact may involve all of the above (and more), and each of these concerns may implicate the others. Noting these different aims of assessment, however, calls attention to the fact that conversations about assessing expansive digital publications may unintentionally assume that the only use of assessment is to advance a scholar’s career. Yet the nature of these projects — interdisciplinary, collaborative, resource-intensive, public-facing and public-engaging — means that there are multiple stakeholders keen to measure impact for different purposes. While this section will primarily consider assessment as part of tenure and promotion, it will bring these other considerations to bear on that process.

Whether to assess. As part of their process of development, expansive digital publications can go through multiple stages of review (e.g., various members of the project critiquing and contributing to a grant proposal; proposal review and acceptance as a condition of funding; community feedback on the digital work through blog posts, comment forums, and social media).1 Before asking how to evaluate an expansive digital publication, we must first recognize and give weight to evaluations already conducted.

What to assess. Promotion and tenure committees have asked for, and scholars have voluntarily submitted, précis describing the scholarly value and impact of their digital work. Such documents assist committees with gauging the intellectual value of scholarly works while also allowing authors to direct attention to those salient aspects. Yet this approach risks undervaluing or ignoring entirely arguments inherent in the digital form. As has been argued elsewhere, a summary cannot substitute for evaluating a work in its original form.2

Lacking such a document, however, it may be difficult to discern a scholar’s contribution to an expansive digital publication, particularly if it reflects extensive collaboration. Expansive publications can involve multiple people (e.g., graduate students, programmers or other technical experts, community collaborators); if later evaluators remain confused about the contributions of individuals on an expansive publication, they may shy away from assigning credit to any single scholar for the work.3 Arguably, evaluating an expansive digital publication in terms of a single individual’s rewards works against the principles underlying such collaborations. Focusing exclusively on one individual’s creative role may serve the current academic reward structure but does not reflect the scholarly values and aims that drove the work in the first place.

Who will assess. When review committees lack members who can effectively evaluate aspects of a digital work (e.g., data structure, user interface, codebase), they will likely under-appreciate or even overlook those features. This problem raises particular concerns when such components help move the academic field into new areas of inquiry. Including reviewers with appropriate expertise helps ensure that promotion processes accurately assess a given digital work.4

How to assess. Fortunately, a number of scholarly societies have developed criteria for evaluating digital scholarship (e.g., MLA, AHA, CAA), thus helping creators and evaluators better communicate expectations about how these works will be assessed. Yet the nature of expansive projects requires an evaluative approach that also takes their distinctive aims and potential impacts into account, beyond disciplinary criteria. For example, while humanists increasingly recognize the potential of open-access digital publications to attract broad audiences, we have not codified methods for measuring those audiences and gauging impact, much less seen wide acceptance among those charged with evaluating scholarship in the promotion and tenure process. This is unfortunate since, through Web analytics and other alternative metrics of reception, digital publications can provide far more insights into readership, use, and impact than are possible with print.

Library support for evaluating and understanding impact

Here we offer a few low-barrier approaches to improving assessment of expansive digital works, ranging from early involvement in project planning to late-stage contributions.

Support promotion and tenure committees. In order to ensure that the promotion and tenure review process fairly evaluates digital scholarly work, libraries should look to establish relationships with the promotion and tenure committees and offer support with staff who are fluent in the technologies and publication types under consideration. This inclusion could also mean asking P&T committees to look outside their department and could involve librarians (particularly in cases where librarians hold faculty appointments), especially if they were part of the work’s development and are able to provide more insight into how to evaluate the work and its impact.

Advising on metrics for evaluating impact. Libraries have a broad view of the publication landscape and can help scholars identify the generic qualities of their work and appropriate assessment methods. Through integration of discoverability tracking tools, and other basic tools such as Altmetric Explorer and web analytics, libraries can advise scholars on ways to assess the reach and impact of their publications and effectively communicate this evaluation to review committees. This advisement can be especially useful early in publication development, to help scholars ensure that their digital publication enables this assessment (e.g., through creation and consistent use of DOIs).

Helping to plan and document assessment as part of project development. Scholars working on expansive digital publications have identified the value of building in phases, with article writing and other forms of peer review as part of that phased work. As partners in planning expansive digital publications, libraries can help define the assessment strategies for critical phases of the project to help ensure that scholars collect the data necessary to document impact. As participants, they can help document key information relevant to assessment (e.g., who worked on the publication and what they contributed; formal reviews of the publication at different stages of development).

Opportunities for partnerships

Digital publication peer review network. In addition to guidelines for promotion and tenure committees evaluating digital works, we also require external mechanisms for vetting and acknowledging the excellence and impact of expansive digital publications, such as the acquisition of such works by a university press. More libraries are playing a role in evaluating this scholarship, whether as part of library- or campus-wide programs for digital project development or as formal reviewers on tenure and promotion panels. As such, libraries can serve as key nodes in broader peer-review networks, alongside the presses that have provided and continue to provide this external credentialing function. Incorporating more incremental review for publications as they move from experiment to full-fledged publication, and building a broader review network of libraries, presses, and other organizations that can help better assess and document a work’s sustained contribution to a field, could help pave the way for higher levels of credentialing (e.g., the acquisition of such works by a university press; the work’s sustained support and development by libraries). Such a network would also help ensure that these publications are appropriately valued outside a scholar’s home institution.

Liaise with presses to take incubated publications further. Assuming that libraries are willing to help incubate publications (as they have been doing for decades now), a better working relationship with presses would help to move expansive digital publications into a broader sphere of access. Establishing such relationships early in any publication’s life cycle will help the press to sustain the project in partnership with the library and enable the richest forms of linking and discovery.