Asking individuals to disclose their expertise to increase engagement and improve problem-solving
Professionals working in a variety of institutional contexts – from regulatory agencies to social innovation labs – have a hard time identifying who among their colleagues and counterparts has the experience and expertise to help improve decisionmaking and problem-solving. The person sitting at the next desk with the ambiguous title of “manager” or “director” could have the know-how to make the difference between success and failure in an important public project. An individual on the other side of the world might know the secret to solving the exact problem her counterpart is experiencing at her institution. To make the City of Buenos Aires Government (full BA ExpertNet Project Plan and Research Agenda), participants in the Open Government Partnership (OGP), the Global Labs Network of Public Sector Innovators and the U.S. FDA’s regulatory review panels more effective, we want to be able to identify the talents, skills and capacity of relevant communities.
Working with partner institutions, the ExpertNet project at the Governance Lab is building open source tools to enable municipal employees working in the Buenos Aires city government to identify each other’s areas of expertise; open governance practitioners in OGP countries to gain insight from counterparts in other countries; Global Labs innovators to share best practices and the FDA to find the best experts to staff its medical device regulatory review panels. Additionally, this work will be instrumental in helping the GovLab Academy scale a mentor network that connects learners to the individuals most capable of helping with their projects.
ExpertNet builds on earlier work in connection with the White House Open Government Initiative on designing a platform for asking people to share more about their expertise – their experience, expertise, skills and passions. Using a collaborative wiki, the United States General Services Administration (GSA) and Open Government Initiative solicited feedback from the public on the development of a platform that 1) enables government officials to circulate notice of opportunities to participate in public consultations to members of the public with expertise on a topic; and 2) provides those volunteer experts with a mechanism to provide useful, relevant and manageable feedback back to government officials.
ExpertNet seeks to expand what we know by going beyond titles to capture a broad picture of people’s expertise. Useful expertise goes beyond what shows up in formal credentials. It can include a wide range of skills and past experience, encompassing what we know how to do and our ability to explain it to others.
What We Want to Understand
- what kinds of expertise are most helpful to identify;
- what are the best ways to collect that information;
- how expertise impacts people’s willingness to collaborate;
- whether identifying employee expertise helps institutions to be more effective at solving problems and sharing best practices; and
- the resulting real-world impact.
What We Seek To Build
In collaboration with partners, the GovLab will build an open source, linked data system for capturing relevant expertise in different contexts. In performing and designing platforms for varied experiments, we will leverage and learn from a variety of innovative tools and technologies already in existence. For instance:
- Expert networking platforms, like VIVO and Stack Exchange, will provide useful data to draw from as well as best practices to emulate for creating searchable platforms of individuals’ skills, interests and experiences.
- Ratings and endorsement systems, as found on LinkedIn and eBay, will allow users to vet skills, gain demonstrable recognition for good work, and make it possible to create badges and leaderboards to incentivize participation.
- Linked Data technologies, as used in CKAN and Google Now, will allow us to consolidate data on individuals’ skills, preferences, and experience from a diversity of sources.
- Predictive search functionality, as deployed by Google and Bing, will ensure consistency of self-reported data and make expert discovery easier.
- High-Performance databases, like Oracle DB2 and MSSQL, will satisfy our data management and storage needs.
What is Smarter Governance?
ExpertNet is part of the GovLab’s ongoing Smarter Governance efforts, which are based on the belief that people who are not traditionally included in certain governance decisions have expertise, experience, and ideas that can offer new insights and solutions for public problems.
Advances in science and technology have provided new tools for identifying who knows what and targeting requests to participate to those most likely to contribute. With the ability to identify and target engagement to a small number of relevant audiences, such as city government employees with a specific set of skills and real-world experience, and to offer them ways to participate meaningfully, city governments have a better chance of crafting a solution that is better suited to the material reality.
Opening an organization to the use of what Harvard Business School professor Henry W. Chesbrough calls “purposive inflows and outflows of knowledge” is well known in the business literature as a driver of innovation.[I] This “open innovation” paradigm is the “antithesis of the traditional vertical integration model where internal [research and development] activities lead to internally developed products that are then distributed by the firm.”[II] In many (but by no means all) businesses, the traditional assumption that working in secrecy leads to innovation is becoming obsolete.[III] In the contemporary environment businesses are finding expertise in every corner of their organizations and outside of them.[IV] Open innovation not only expands opportunities for new solutions based on previously unavailable or disconnected silos of knowledge, but also allows external actors the benefits of leveraging an institution’s resources, such as its investment in research and development.[V] The narrative of a lone entrepreneur bringing his or her innovation to the market has been replaced with the concept of networks and communities coming together in an interactive process of discovery.[VI] New businesses are thriving by supplying expertise to organizations such as banks looking to inform their investment decisions[VII] or law firms seeking expert witnesses.[VIII] Relying exclusively on internal research and development is seen as limiting the potential scope of innovation in many sectors.
Open innovation techniques can help the public sector as well as private enterprise. Consider this early example. In 2005, before Facebook and Twitter, New York Law School students and Beth Noveck designed and later launched Peer to Patent, the first expert network in government, which connected volunteer scientists and technologists to national patent offices to better inform how they decide who gets the twenty year grant of the monopoly right called a patent.[IX] They invented the process, technology and the policy by which a closed bureaucracy could open its doors and collaborate with knowledgeable people to do its work better. Time after time, the public was able to dig up quickly the expertise the official couldn’t find, enabling her to make the final determination informed by citizen participation but subject to the independent law and rules of the patent process. Despite the fact that patent practice hadn’t changed much since Thomas Jefferson began as the first examiner in 1791, under the new Patent Act, passed with near unanimity – Peer to Patent citizen engagement, once an experimental pilot undertaken only with the applying inventor’s consent – is now enshrined in law.
What Are the Methods for Identifying and Measuring Expertise?
Traditionally, credentials bestowed on individuals by widely recognized institutions have been the central means for demonstrating and identifying expertise. From doctorates earned at Ivy League universities to notary public licenses to plumbing certifications, people generally associate expertise with pre-defined systems of demonstrated aptitude administered and validated, through official documentation, by an authoritative institutional structure. While credentials remain the dominant means for demonstrating expertise, they don’t give a complete picture: people now often acquire skills through channels that exist outside traditional institutions.
For employees in most environments, experience-based expertise is the most important means for career advancement – following an initial reliance on credentials. The demonstrated ability to do good work, which may or may not be formally quantified, is a key barometer of who knows what in a workplace setting. Online, platforms like GitHub and Stack Exchange are using experience-based systems for quantifying and vetting expertise in a more useful and agile way than relying simply on credentials.
Recommendation and endorsement systems for expertise are becoming popular in response to the growing disconnect between individuals’ skills and traditionally recognized systems of credentials. The paradigmatic examples of LinkedIn, Yelp and eBay demonstrate how community-sourced ratings, recommendations and endorsements can provide useful metrics in addition to or in place of traditional certifications. For example, an individual who has completed online training with CodeAcademy can list that credential on her LinkedIn profile page and have it endorsed by colleagues and clients, whether or not she has any academic degrees or credentials in web development.
What ExpertNet Will Capture
ExpertNet will collect, categorize, and manage structured information on expertise and experience. With an eye toward platform iteration and linking existing databases, ExpertNet will initially capture information in the following broad categories: Biographical, Education and Credentials, Skills, Interests, Projects, Partnerships, Work Experience, and Future Plans.
Whether expertise rooted in credentials, experience or reputation, information can be gleaned automatically from mining online biosketches and data sources such as public and grants records or self reported. At least in its early phases, self-reporting will be the central method we test for identifying expertise. As the platform seeks to develop and populate databases with information that can be used to identify and measure expertise, gaps in available data will need to be filled by individuals self-reporting a variety of information on their backgrounds, skills, interests and experience.
Inputs traditionally associated with HR systems, like name, position, specialty, department and geographic area, will play an important role in establishing a person’s identity. In addition to providing basic searchable information, the bio section can be seen as a type of e-business card with both personal and professional aspects. Such a system has been implemented by, for instance, LinkedIn as a way to identify professionals in a searchable way.
Education and Credentials
Questions related to educational attainment and other traditional credentials and certifications are one way to estimate a person’s capability. While the ExpertNet system will seek to look beyond such traditional metrics of expertise, the information found here will still provide potentially useful data. However, ExpertNet will seek to emulate platforms like Stack Exchange in the sense that demonstrated proficiency is valued much more highly than traditional forms of certification.
Beginning with a self-reporting system, and eventually linking other data sources in later phases, the centerpiece of the ExpertNet system focuses on employees’ skills. Two important features will ensure that skills are accurately listed in an employee’s profile: an automated, predictive system for skills input to ensure data consistency, and an endorsement system allowing other users to vet skills and recognize satisfactory work.
ExpertNet will ask users to self-report their professional and personal interests to help identify the types of expertise they want to leverage and the types of projects they want to contribute to. In later phases, ExpertNet will link that data with information drawn from existing social networks, like Facebook, to develop a more fully formed image of an individual’s interests.
Information on work experience will provide context for self-reported skills and help identify any dormant professional expertise not being leveraged in an individual’s current position. Even more than the skills and interests sections, the ability to link self-reported data with existing professional data stores such as LinkedIn will enable the continued development of a rich and useful database on work experience, while simultaneously making it easier for the user to create her profile.
Potential Risks and Barriers
While the GovLab is confident that ExpertNet will benefit city governments and their employees in many ways, the project is not without risks and barriers, including:
- Data Availability. The biggest initial concern for the project is likely to be the availability of useful institutional data. If such data is not available, it will take time and manual entry to input data in a usable form.
- Low-Quality Self-Reported Data. Low-quality self-reported data could also pose challenges to the usefulness of the system. While it will not be easy to obtain accurate, uniform data through self-reporting, the project will take steps to mitigate concerns – including the use of a predictive entry system.
- Cultural Resistance. Traditional institutional culture will have to be transformed to one that embraces openness and greater collaboration. While the potential benefits and uses of ExpertNet will likely inspire many individuals to experiment with it, we expect some resistance at different organizational levels.
- Political Concerns. ExpertNet and its underlying principles are wholly apolitical. However, given the oft-partisan climate of many institutions, political issues are still likely to arise.
- Constraints and Challenges Unique to Certain Contexts. Many constraints and challenges will be impossible to predict until work on the project begins with different partners.
The GovLab’s Approach: Living Labs and Action Research
The late MIT Professor Kurt Lewin, a pioneer of modern social psychology who coined the term action research, said, “research that produces nothing but books will not suffice.” This notion is central to the GovLab’s Living Labs projects, which are undertaken using an action research methodology.
Action research involves experimenting in real-world settings with a partner organization. In this process, the research team and partners work together to collect and analyze data, evaluate performance based on various metrics, and change practices based on the feedback or evidence gathered.
In contrast to traditional research, which poses a question, collects and analyzes data, interprets the results and publishes findings – leaving to others the task of determining what can be done with the results – action research combines research and its application seamlessly. Action research is both a research process and an approach to affecting social or organizational change: It actually implements potential solutions and studies them as part of the research process. The outcomes of action research are both solutions to social and organizational problems and contributions to scientific knowledge and theory.
The Living Labs are intended to have a broad impact beyond their benefits to the GovLab’s partner institutions. The lessons learned in our Living Labs will inform overriding efforts to transform dysfunctional institutions and help create blueprints for effective, legitimate governance.
The GovLab’s Living Labs action research methodology is defined by the following objectives:
- Developing practical designs for collaborative democracy;
- Putting these designs into practice with institutional partners;
- Doing research on the outcomes to advance our understanding of different approaches; and
- Using that understanding to design subsequent implementations.
With these organizing objectives and this general methodology in mind, the GovLab’s Living Labs will follow a well-defined process, tailored to the needs, abilities and resources of our partner institutions.
The GovLab’s Living Labs Methodology:
ExpertNet Central Research Questions and Variables
The GovLab’s work on ExpertNet will be organized around key research questions, both to ensure a focused approach to developing the project itself and to help develop a blueprint for broader innovation of institutional governance. Our initial research questions, divided by experiment phase and variable, are the following:
- Expertise: Identification
- How do we know who knows what and target requests to participate?
- Can we supplement the use of the three paradigmatic techniques for identifying expertise – reputation, credential and experience – with self-reported information to obtain a full understanding of individuals’ expertise and experience?
- Through strategic data collection and analysis, is it possible to determine not only who is capable of helping to address a problem, but also who is willing to do so?
- How can we identify the types of problems that are best-suited to getting expertise in?
- How can different data sources – including internal government databases and existing social networks like LinkedIn – be linked together to provide a multi-faceted view of individuals’ expertise and experience?
- Expertise: Matching
- What can we learn from experiences in online search and advertising, academic discovery and other contexts regarding how to match people’s skills to work in different contexts, and apply those lessons to city governments?
- How do we match skills and expertise to problems?
- Can predictive information input and search functionalities improve expert matching?
- Expertise: Incentives and Motivation
- What incentives cause people to share their knowledge and expertise?
- Are gamification techniques – such as point systems, badges and leaderboards – effective in incentivizing individuals to engage in expert networking?
- What are the extrinsic (like prizes) and intrinsic (such as ego, altruism, autonomy, curiosity, empathy, etc.) motivations for participation, and how do they compare within different contexts?
- Barriers: Cultural and Legal Challenges and Readiness
- What are the legal and cultural impediments for different institutions to be open to expertise?
- How can intentionally closed institutions be pushed to open themselves to leveraging outside expertise and sharing their own?
- What absorptive capabilities must institutions develop to assimilate the diverse knowledge of identified experts?
- Framing: Problem Definition
- How can city employees define requests to expert colleagues so as to:
- Articulate the problem in a language that captures interest and expertise otherwise untapped;
- Clarify the urgency and need to find a solution;
- Develop sufficiently granular problem statements for productive ideation;
- Promote and embrace diverse and novel responses that expand the existing solution space; and
- Entice the relevant experts to respond?
- How does ambiguity in the problem statement or targeted request hinder success?
- How can ambiguity be avoided when defining the problem?
- How can problems be framed in a way that helps engage experts in unrelated fields who could contribute based on solutions encountered in their own work?
- How can city employees define requests to expert colleagues so as to:
Metrics of Success
The development of meaningful systems of metrics and evaluation is a key facet of the Planning stage of the Living Labs methodology. While stakeholder engagement and targeted due diligence will be required to craft a targeted evaluation system, the GovLab has identified the following initial metrics of success:
- Number of Active Users. The number of individuals in different institutions taking part in the system will provide a useful measure of interest and engagement. Changes in use over time will help determine if the system is experiencing a network effect, where use inspires more use.
- Total Endorsements, Recommendations and Ratings. Similar to the Active Users metric, the total number of endorsements, recommendations and ratings will provide a useful measure of individuals’ continued use of and engagement with the system.
- Number of Unique Users Earning Badges and Endorsements. Studying the number of unique users earning badges and endorsements, especially focusing on changes over time, will provide a means for measuring the success of the ExpertNet incentive and motivation system.
- Number Successful and Unsuccessful Expertise Search Queries. While also providing a view of which types of skills and experience do and don’t exist within different communities, the number of successful and unsuccessful search queries will also help identify any gaps in collected data.
Social Science Measures
- User Satisfaction. ExpertNet is meant to improve the decisionmaking and problem-solving capacity of institutional employees. As such, likely the best metric of success will come from surveys of participating employees asking about their satisfaction with the platform.
- Employee Engagement and Tendency Toward Collaboration. Obtaining insight into ExpertNet’s role in increasing individual engagement and interest in collaboration will prove more difficult, but will be essential for determining the platform’s success in relation to its objectives.
- Perceived Improvement in Problem-Solving. In theory, increasing collaboration and the capacity to identify and engage expertise will improve decisionmaking across contexts. However, testing this assumption by measuring the problem-solving success of people utilizing the platform will be essential.
- Assessment by Directors and Supervisors. In addition to surveying participating employees in general, studying the assessments of directors and supervisors will help determine whether the system has been useful for their employees. Perhaps more importantly, engaging directors and supervisors will help determine if the platform is successfully navigating cultural barriers within institutions.
[I] Henry Chesbrough, “Open Innovation: A New Paradigm for Understanding Industrial Innovation,” in Open Innovation: Researching a New Paradigm, eds. H. Chesbrough and W. Vanhaverbeke and J. West, (New York: Oxford University Press, 2006).
[III] Henry Chesbrough, Open innovation: The new imperative for creating and profiting from technology, (Boston: Harvard Business School Press, 2003).
[IV] David Weinberger, Too Big to Know: Rethinking Knowledge Now That the Facts Aren’t the Fact, Experts Are Everywhere, and the Smartest Person in the Room Is the Room, (New York: Basic Books, 2012).
[V] Linus Dahlander and David M. Gann, “How open is innovation?” Research Policy 29, no. 6 (July 2010): 699-709.
[VI] Frank Piller, Christoph Ihl and Alexander Vossen, “A typology of customer co-creation in the innovation process,” available at SSRN, December 29, 2010, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1732127
[VII] See Dawn Cowie, “Noble launches expert network,” Financial News, February 10, 2009, http://www.efinancialnews.com/story/2009-02-10/noble-launches-expert- network?ea9c8a2de0ee111045601ab04d673622
[VIII] See Andrew Lu, “When Should Attorneys Hire Expert Witnesses?” FindLaw, December 13, 2012, http://blogs.findlaw.com/strategist/2012/12/when-should-attorneys-hire-expert-witnesses.html
[IX] Beth Simone Noveck, “Peer to Patent: Collective intelligence, open review, and patent reform,” Harvard Journal of Law & Technology 20 (2006): 123.