As I mentioned in the last blog post about research, we’re sharing our thoughts about each social cause as we begin new research. The purpose of sharing this additional information is to explain why we think these are interesting or relevant areas to research, what we learned about the nuances of the cause, and the difficulties we faced in narrowing the scope of the research, so you, the reader, can understand what we considered as we refined our thinking about the research.

If you are an expert on any of these topics, you should be receiving an email from us soon and we hope you will be compelled to participate! If for some reason we have missed you and you think you have a valuable perspective to offer, please contact me at erinn.andrews@myphilanthropedia.org.

Additionally, I’m sure I haven’t been able to capture every nuance in these sectors, so I invite your feedback and thoughts about how you might think about this work. For those readers less familiar with this topic, I hope you will learn something new and tune in again when we have the results of this research. Thank you all for your participation!

Background

In the 1960s, there was an explosion of nonprofit arts organizations in the San Francisco Bay Area. In 1961, before the birth of the national Endowment for the Arts, San Francisco passed a law to require part of the hotel tax to go to funding the arts. Over time, each Bay Area region evolved to have different kinds of arts and culture groups across all disciplines, styles, and ethnicities: jazz, classical, folk, dance, choreography, visual arts, literary arts, poetry, film, video, and so on. In fact, at one point, the San Francisco Bay Area was second only to New York in terms of having the largest number of dance companies. Therefore, with decades of support and development, the vibrancy and variety among the arts in the Bay Area has led to a unique texture in the cultural fabric of this region. Today there are hundreds and hundreds of arts and culture nonprofits, of all sizes and varieties, all across the Bay Area.

The Arts and Culture Landscape

What does the world of arts and culture look like in the Bay Area? Through my conversations with arts experts, I learned that there are a number of ways to look at the landscape. First, one could examine nonprofits according to the two metrics: creation of work or engagement. Creation of work relates to supporting cultural heritage or preservation of art, as well as the support of new work, both in the development and production phase. Engagement in the arts could include introduction to the arts through performances for young audiences, it could include standards-based arts education in the schools, or it could include out-of-school arts learning for youth and/or adults.

Another way to look at arts and culture is by buckets or type of organization: nonprofits might focus on arts education, be considered major arts institutions, be community-based arts organizations (this might include service organizations, art therapy, low-income community programs, etc.), or be considered independent artists and organizations such as dance and theater companies.

A third way to examine this field is to look at organizations by budget size. Small arts and culture organizations might fall under $100,000 in annual budget. Mid-sized organizations might fall in the $100,000-$1,000,000 range. And large organizations might be above one million dollars. The largest arts nonprofits in the Bay Area might have a $4-6 million annual budget (still below that of some of the largest arts organizations in New York).

And yet another way to think about arts and culture nonprofits is to look at them by genre, discipline, or even location. I’m sure there are even more ways one could divide up “arts and culture” as a social cause.

As an organization, we value diversity of expert type in our network of professionals and diversity among the types of nonprofits represented in the expert mutual fund. Therefore, it’s important for us to keep the scope of our research broad enough to allow for this kind of diversity to emerge among the responses of our experts.

One difficult decision we faced was whether to limit the organization size in the scope of our research. An argument can certainly be made for asking experts to only highlight small-medium sized arts nonprofits because they most likely need the additional funding in this down-turned economy. However, we believe that the primary purpose of the research we run is to help individual donors understand which nonprofits are having the most impact in a given sector, rather than which nonprofits most need the additional funding. Therefore, we will invite participants to comment on any size nonprofit focusing on impact and encourage experts to think beyond organizations that have simply been around for the most number of years or are well established. We’re looking for quality of the organization above all else.

Who cares? Problem Statements

In most of our research, we begin by understanding what problems nonprofits in a particular sector are trying to solve. Again, this was a bit tricky to do with arts and culture nonprofits. Here are a few responses from experts to the question, “what problem are most arts nonprofits trying to solve?”

• Today, people have very little time to reflect on and distill their life experiences. The arts provide individuals with this opportunity.
• In an increasing capitalist, consumer-focused society, cultural heritage is being lost. Arts organizations must try to preserve these traditions.
• Commercial publishers seek monetary gains, therefore arts organizations must do what they can to encourage the literary arts to flourish.
• We are in danger of losing “art for arts sake.” We must encourage the creative expression of human beings because it’s good for the soul and for the mind.
• Funding for the arts in schools has been cut back over the last three decades. Arts organizations must fill in the gap and develop arts education programs to support students.
• Arts aren’t about solving problems, they’re intrinsically valuable to human culture, they’re about being alive and expressing yourself.

I found this part of my conversation with experts to be particularly interesting because while these are “problems,” most arts experts with whom I spoke don’t believe the purpose of arts organizations is to actually solve problems. This makes arts and culture a very unique social cause for Philanthropedia. I don’t think this will fundamentally change the way our research is run, but it’s important context to understand as we frame the survey and as we analyze the results in the weeks to come.

High-Impact Arts and Culture Nonprofits

Impact in the nonprofit sector is always hard to define, yet I have found that it’s even more difficult to define in arts and culture. How might we measure impact in the arts world? First, from a more objective standpoint, donors or funders might look at the “staying power” of an organization which could be evaluated by measuring the growth of support over time. One could look for evidence that someone cares about this work: attendance numbers, subscription numbers, percentage of the house that’s full, and/or donor support. Second, one might look for more subjective measures: what do organizations achieve with what they have or how much programming can they produce on a given budget; what’s the scope of this activity and what’s the quality of this activity? And then there’s the dimension of whether the performance, exhibit, reading, etc. was enjoyable, interesting, or thought-provoking to an audience member.

With the help of a few experts, I’ve come to this definition for what a high-impact arts and culture nonprofit might look like: A high-impact arts and culture nonprofit is one which is successful at creating or producing something of value to those who care about the arts and culture. A high-impact arts and culture nonprofit is able to contribute to the field by creating meaningful work and/or helping others develop an appreciation for the arts and culture.

Scope of the Research

Given all of this information and context, we decided that in the survey, we’d ask experts to recommend nonprofits that could be developing or producing new work, be focused on performance, preservation and promotion of traditional culture, have an educational component, serve any age or type of audience, have any budget size, and/or represent any genre or discipline of art. We are interested in arts nonprofits that have had real impact in their community and do outstanding work, NOT nonprofits that have simply been around for many years or most need additional funding. This research will be focused on organizations, not individual artists.

We’re excited to see the results of this research and share it with donors to help direct more funding to some of the great arts and culture nonprofits in the Bay Area who are really making an impact. As we begin this process, I’d love to hear what you think about these ideas. What other nuances did I miss? Do you agree with these definitions? If not, how might you define impact of arts and culture nonprofits? I invite you to share your thoughts and look forward to hearing from you.

We are excited to share this whitepaper, “Consolidating Expert Opinion about High-Impact Nonprofits: Review of Philanthropedia’s Methodology.” It summarizes more than two and a half years of research and highlights the results of our climate change research in much more depth. We are copying the executive summary below.  You can download the full whitepaper, as well as add your feedback at http://www.myphilanthropedia.org/whitepaper.

Executive Summary

The Philanthropedia whitepaper, “Collecting Expert Opinion about High-Impact Nonprofits: Review of Philanthropedia’s Methodology,” serves two main purposes: (1) to make the case for using experts to identify high-impact nonprofits, and (2) to share our progress towards a specific methodology to date.

Our overall conclusions are that:

• Our methodology captures expert opinion about high-impact nonprofits in different social causes.
• Using experts to identify high-impact nonprofits offers unique advantages in terms of high quality information about nonprofits at low cost to gather that information.

Because of the potential of this approach to evaluating nonprofits, we are investing in improvements to further strengthen our research.

The Case for Using Experts to Evaluate Nonprofits

Philanthropy is primarily concerned with distributing limited monetary resources to charities doing the best work in solving social problems.  Therefore, we believe there are two main problems the sector faces:

1. How to identify these high-impact nonprofits
2. How to how to distribute resources effectively to these nonprofits

In this whitepaper, we assess existing nonprofit evaluators and ourselves on two dimensions: quality and cost.

• Quality relates to how closely the measures used to evaluate a nonprofit are correlated with impact per dollar invested and the nonprofit’s capacity to absorb more resources.  We define impact as a measure of the lasting improvements produced by that nonprofit to address the core problems in a particular social cause.
• Cost is the combination of resources needed to perform the evaluation of a nonprofit including money, people, and time.  Therefore a good solution is one where there is a quick, scalable, low cost way to evaluate many nonprofits across multiple social causes.

To date, nonprofit evaluators have been able to make progress on either the cost or quality front, however, none have been able to strike a good balance between high quality and low cost.  The Philanthropedia approach fills this gap by using a low-cost method of surveying experts to identify high quality information about nonprofits:

Philanthropedia offers a high quality solution because we capture the opinions of experts who are uniquely qualified to assess nonprofits.  Experts are qualified because they have access to nonpublic data about charities and have advanced mental models for evaluating impact.

Philanthropedia offers a low cost solution because it takes experts only about 40 minutes to complete both online surveys, experts are not paid for their participation, and one trained employee can conduct four social cause research projects concurrently over the course of 2-3 months.

Philanthropedia’s Methodology

Philanthropedia’s methodology of surveying diverse and representative groups of social cause experts to identify high-impact nonprofits runs in six steps.

1. Research and define the social cause or scope of the research
2. Identify and recruit experts
3. Run a survey asking experts to identify high-impact nonprofit
4. Analyze survey data
5. Run a second survey to determine agreement among experts about high-impact nonprofits, collect strengths and areas for improvement for each nonprofit, and ask experts to allocate funding across nonprofits
6. Compile and analyze final results and publish a list of high-impact nonprofits for the social cause

When identifying experts, we require they have a minimum of two years of experience, relevant work experience as evidenced by current or past job titles or employers, and a minimum expert self-rating on a Philanthropedia-developed scale.  Our goal is to create a representative expert network along two dimensions: profession type (foundation professionals, nonprofit senior staff, researchers, and others) and geography.  And, we screen for high-quality responses.

For the top 15% of nonprofits recommended by experts, we collect the number of expert votes, the percentage of agreement across the expert network that the nonprofit is among the most effective in that sector, the percentage allocation of each nonprofit as part of an Expert Fund, and the strengths and areas for improvement.

Results and Data Analysis

To date, we have researched four social causes to test and develop our methodology: education, climate change, microfinance, and Bay Area homelessness.  To demonstrate our approach, we analyze the results of our climate change research.  We built a diverse and representative expert network of 139 experts with an average of 12.94 years of experience.  The expert breakdown was 19% foundation professional, 12% researchers, 47% nonprofit senior staff, and 22% other (policy makers, government officials, etc).  Forty-one percent of our experts came from the east coast, 9% from the midwest, 4% from the south, 6% from the northwest, 30% from the west coast, and 10% international and from other locations.   Our experts recommended 15 nonprofits which represent the top 15% of the mentioned nonprofits.

In order to determine how much our final results were influenced by each expert type, we ran correlations between the combined top nonprofit list and the lists recommended by each expert type.  While nonprofit senior staff had the most influence on the final list (due to their larger representation in the sample size), all three expert types (foundation professionals, nonprofit senior staff, and researchers) had a high degree of agreement about which nonprofits were most effective.

In order to determine which factors might be influencing the final ranking of nonprofits, we ran correlations with the external factors: nonprofit revenue, brand awareness (as measured by the number of Google mentions), age of organization, and size of organization (as measured by number of employees).  We found that none of these external factors influenced the final nonprofit rakings in any significant way.  Therefore we conclude that compiling expert opinion adds unique value when identifying high-impact nonprofits.

Areas for Improvement

The results from our climate change research and other studies are very encouraging.  However, we believe there are still many areas for improvement.  In particular, we intend to make changes in the way we sample experts, in the way we state the questions in our surveys, and in the way we share the results.

We intend to improve:

• the research quality in terms of expert responses and their ability to identify high-impact nonprofits.
• the clarity of the language used in our surveys in order to better communicate our goals to experts.
• our transparency by sharing more information publicly so that we can continue to build trust in the philanthropic community.

Conclusion

We are excited about the unique advantages that our methodology offers in terms of high quality and low cost, which is why we are investing in these further improvements.  We hope this whitepaper clarifies Philanthropedia’s approach of collecting expert opinion about high-impact nonprofits.

We do this research hoping to influence donors as they search for nonprofits to support.  We believe that donors can have a bigger impact in the nonprofit sector by directing more financial resources to some of the highest-impact nonprofits.

We invite feedback and discussion about this whitepaper and our work at: http://www.myphilanthropedia.org/whitepaper.

We are excited to launch research on a number of new social causes! We’re starting with a bundle of 4 local Bay Area causes and we’ll spread out the start of each survey by a few days or a week. At the start of each survey launch, we’ll share a blog post about the cause we’ll be studying. The purpose of sharing this additional information is to explain why we think these are interesting or relevant areas to research, what we learned about the nuances of the cause, and the difficulties we faced in narrowing the scope of the research, so you, the reader, can understand what we considered as we refined our thinking about the research.

If you are an expert on any of these topics, you should be receiving an email from us soon and we hope you will be compelled to participate! If for some reason we have missed you and you think you have a valuable perspective to offer, please contact me at erinn.andrews@myphilanthropedia.org.

Additionally, I’m sure I haven’t been able to capture every nuance in these sectors, so I invite your feedback and thoughts about how you might think about this work. For those readers less familiar with this topic, I hope you will learn something new and tune in again when we have the results of this research. Thank you all for your participation!

Background

Last summer, we conducted research to identify high-impact nonprofits working on climate change at the national level. However, as we thought about what additional causes to consider in the Bay Area, we realized that climate change is also a very important and meaningful cause at the local level. Here’s some of the reasoning why (starting at the national level and zooming into the Bay Area):

Climate change is a timely and important topic in the US. In the last few decades, scientific research has shown the effects that carbon emissions and other human-made pollutants in our atmosphere have had on global temperatures. These startling and sobering results have caused a number of environmental organizations to shift their focus to climate change mitigation.

Climate change is a relevant local cause as well. The Bay Area has a long history of involvement in the environment, especially around topics such as land conservation and open-space protection. And Silicon Valley specifically, (home to many tech start-ups) has a history of innovation around green tech, renewable energy, and sustainability. While there is work to be done at the national and corporate level, progress toward climate mitigation can also be made at the local level. Here, the Bay Area is uniquely suited to lead the way in reducing ones carbon footprint.

Individuals in the Bay Area are also in a position to help lessen the effects of climate change. Many of today’s solutions for climate mitigation at the individual and home-level are expensive, but because of the disproportionate wealth in these communities, individuals are at the forefront of adopting new policies and behaviors to work toward a solution. Therefore, research focused on climate change in the Bay Area is a timely, relevant, and interesting topic because of the unique combination of California history, entrepreneurial capacity, and individual financial strength and willingness to change.

Climate change has also been in the news lately at the state level. In 2006, California passed AB32, the Global Warming Solutions Act mandating that by 2020, California reduce greenhouse gas emissions to 1990 levels. However, in light of the state’s financial crisis, some politicians have proposed suspending the act for some period of time until the state is more financially stable. This would have major negative implications for the nonprofits working to mitigate the effects of climate change and for the community in general.

Who cares? Problem Statements

The experts with whom I spoke identified a number of problems that nonprofits might be trying to address around climate change. Here are a few examples that range from general to specific:

• Climate change is accelerating at a dangerously rapid pace due to human actions which is detrimental to the earth and the species that inhabit it.
• 40-50% of the greenhouse gas emissions in California come from cars and trucks.
• Emissions from transit vehicles are detrimental to the environment because they increase the amount of greenhouse gases released into the atmosphere which contributes to the premature acceleration of global warming. As temperatures on Earth rise, polar ice caps melt which cause the sea level to rise. With over 800 miles of shoreline, California is particularly vulnerable to the effects of sea level rise.
• As the state falls deeper into economic distress, climate change is seen as less urgent to address.

Scope of the Research

If Philanthropedia were interested in policy solutions alone, we would have more seriously considered looking at climate change at the state level. However, because of the Bay Area’s unique positioning, we think this more local level of analysis is equally valuable. Additionally, we are interested in identifying nonprofits that have a variety of approaches to solving the many problems in each sector, not just those with a policy focus.

Therefore, the scope of this research will be broad and could include organizations focused on transportation, energy, biodiversity, land use, agricultural activities, sustainable conservation, etc. We will ask experts to identify climate change nonprofits that could be policy organizations, research groups, trade associations, advocacy groups, utility commissions, traditional nonprofits or community based organizations, and/or energy commissions. Specifically excluded from consideration will be for-profit organizations such as solar or green tech companies, green architects, or climate/energy consultants.

To the climate change experts out there, I’m sure I’m only scratching the surface with these observations. I invite you to contribute your thoughts as well. In your opinion, what are the main problems that climate change nonprofits are trying to address in the Bay Area? Why is this topic unique to this region? What additional context is necessary to understand the complexity of this issue?

I look forward to hearing your thoughts.

We’re sorry it has been a few weeks since our last post, but we have been hard at work and are excited to share more about what we’ve been up to! Primarily, we have been spending our time on research in two main ways. First, over the last few months we’ve been writing a whitepaper. This whitepaper outlines in detail what problem we believe Philanthropedia is solving, detailed information about our results to date and the in-depth analysis we conducted, and the ways in which we plan to improve for our next rounds of research. Soon, we will be making this paper public, will summarize the main points and conclusions in a few accompanying blog posts, and will invite your feedback.

Second, over the last few weeks, we’ve been deciding which social causes to cover next. Our plan is to round out our national research by adding at least 4 more causes to those we already have (microfinance, education, and climate change). However, we also intend to continue our research at the local level, to build on our one local cause so far: homelessness in the Bay Area. So, we will add an additional 4 causes to the local Bay Area level.

What to research?

There are a number of things we take into consideration as we decide which social causes to look into. One major consideration is: are there donors who are interested in the topic we’re considering (i.e. is there already individual donor donation flow to nonprofits in this sector) or is there potential to have donor interest in the future. Another major consideration is: are there experts in the sector who have the depth and breadth of expertise necessary to evaluate and compare the nonprofits doing work in that space. These are the two most important criteria we consider as we decide which causes to study next.

Once we decide on a cause, we do research to understand the appropriate level of analysis. We try to strike a balance between being too broad and too narrow. We want to be broad so we can capture information about a variety of nonprofits, each potentially focused on different problems or implementing different approaches in their work. But we want to be narrow enough so experts understand what types of nonprofits to recommend and which ones to exclude from consideration.

Therefore, in order to learn more about each new research topic, I spoke with a number of experts to learn the nuances of their sector. Before going into the details of what I learned, I’d like to first say thank you to these experts. I had the privilege of speaking with over 50 foundation professionals, professors, nonprofit leaders, and many others (from six different social causes) who candidly and openly shared their perspective about their field. I am humbled by the amazing response and enthusiasm from these professionals, and grateful for the valuable time they spent answering my questions. It’s encouraging to see so many experts voluntarily participating in this work. In return, we hope that our research will help more donation dollars flow to some of the most high-impact nonprofits in their sector, helping these experts and their colleagues accomplish some of their own sector goals, as well.

In the coming blog posts, I will share what I learned from these discussions which helped us define the scope of our research. I’ll begin by exploring the local Bay Area research we plan to conduct first. In addition, we will also be sharing a number of other major announcements in the coming weeks, so stay tuned!

With this post, I want to close the loop on the recent conversation between GiveWell, Saundra, Ingvild, and us (among others) and provide a summary of the core issues and where we stand.

Issue 1: How are experts defined and who are they?

The answer to these questions is rather straightforward, as I explained here and here. In short, while there is certainly plenty of room to grow especially in terms of getting more foundation professionals to participate, we have a good grasp of how to find and recruit experts for our surveys. These experts come from the places you would expect and where, we argue, a lot of nonpublic knowledge about nonprofit impact is aggregated: among nonprofit senior staff, foundation professionals, and academics.

The related issue of transparency of our expert network is important, and we certainly plan to reveal more information in order to build credibility and trust, as well as promote open discussions about nonprofits. In their last blog post, GiveWell made an additional stronger claim that we need to associate individuals with their quotes, which I strongly disagree with. As a matter of fact, evidence and research points in the opposite direction: providing a forum for anonymous feedback creates opportunities to surface candid opinions about strengths and weaknesses (for a good example, consult literature on 360 evaluations). What is more, such a change would effectively prevent certain types of people from participating at all – most notably, foundation professionals, who need to make daily decisions about which nonprofits to support with grants. That is why we have no plans to mandate that individual experts be linked to their recommendations and we maintain committed to providing a safe forum for candid feedback about strengths and weaknesses of different nonprofits. That being said, we are planning changes to our survey and policies to increase the number of experts, who will be acknowledged as participants in our research. To us, it is a question of when rather than a question of if, so stay tuned for changes as Philanthropedia continues to mature as an organization.

Issue 2: How are organizations recommended?

In a comment in an earlier blog post, Ingvild built on GiveWell’s questions and asked how our experts recommend nonprofits (focusing on the difference between impact and effectiveness in particular). As I have mentioned before, we believe that the experts themselves have criteria for evaluating nonprofits. We collect that information in one of the first parts of our survey. Our goal is then to get recommendations from the experts about nonprofits based on these very same criteria, which we then compile and standardize to make available to the public.

Today, we ask experts about their criteria for effectiveness, which to me includes impact in addition to other elements such as marketing ability, fundraising efficiency, leadership, hiring, etc. We did that to experiment and see how far we could push our methodology and how much information we could collect. However, our number one priority remains assessing nonprofits based on impact, which is why we are planning some changes to our survey to make sure our we can collect enough specific information about the impact of nonprofits.

In summary, we have learned a lot from this debate – both in terms of how we could improve, as well as what we need to explain better. In future blog posts, I plan to address several topics including (1) our vision for the space, (2) specific changes motivated by this public discussion, (3) fundamental pros and cons of our model of using experts as a proxy (i.e. what we are and what we are not), (4) more information on our whitepaper and planned improvements to our methodology, (5) assessment of progress to date, and (6) planned social cause expansion for 2010.

Thank you all for your feedback!

Together with the whole world, we have been reading about the terrible disaster that has struck Haiti. Our thoughts are with the families that have been hurt and we hope for a quick international intervention to help the ailing country.

Please consider making a donation if you can. We have not performed any research on disaster relief to date, but our partner GiveWell has a few interesting notes about potentially good organizations here (even though they have not done specific research either). Of the few organizations that they mention, I would probably highlight Partners in Health, which seems to have moved rather quickly to provide aid in Haiti: http://www.pih.org/home.html.

This is another post in my series of articles explaining more about Philanthropedia. As always, I am extending an open invitation for others to comment and join the discussion. No point is too small or insignificant, so feel free to leave a comment! For those that prefer a private exchange, please email us at feedback@myphilanthropedia.org – as some of you already know, I answer almost all emails personally and promptly.

As a follow up to the recent exchange between GiveWell and us, Saundra from Good Intentions blogged about her questions, saying that “[u]nfortunately, instead of helping me understand your processes better, it left me with even more questions than before.” While I hope my blog post brought some clarity for readers, as a philosopher once said “the more I know, the more I realize how much more there is to know.” That is to be expected given the complexity of the subject matter and that we, ourselves are constantly coming up with new questions and searching for ways to improve our methodology. In any case, as I have said before, we are committed to transparency and plan to rework our webpage to better explain our approach.

Saundra had a number of excellent questions that I will attempt to answer below:

Choosing Experts

How do we choose experts? Our approach is rather straightforward and is based on three main categories: foundation professionals, academics, and nonprofit senior staff. When choosing experts, we currently look at a number of factors including years of experience working in the sector, job title and occupation, professional affiliations and/or academic background, with 2 years of experience and relevant career being minimum criteria. In addition, we ask experts to self-rate their expertise on a scale from 1-5 (where 5 means “most” expert), with a self-rating of “3” as a minimum requirement. Here is the scale that we used for microfinance:

On average, how would you characterize your expertise in microfinance?

Limited: “I have limited knowledge of this issue area and do not feel qualified to identify outstanding organizations.”

Basic: “I have basic knowledge of this issue area, and might be able to make a directional assessment of the organizations with which I am familiar. My professional experience and training might qualify me to identify and evaluate a few of the most outstanding organizations in the sector.”

Moderate: “I have moderate knowledge of this issue area and am confident in making a directional assessment of the organizations with which I am familiar. My professional experience and training probably qualifies me to identify and evaluate a few of the most outstanding organizations in the sector.”

Strong: “I have strong knowledge of this issue area that is both broad and deep. My professional experience and training qualifies me to identify and evaluate some, but perhaps not most or all, of the most outstanding organizations in the sector.”

Expert: “I have expert knowledge of this issue area that is both broad and deep, including experience with multiple sub-issues. My professional experience and training qualifies me to confidently identify and evaluate most or all of the most outstanding organizations in the sector.”

The vast majority of our experts are US based, reflecting the fact that this is the primary focus of our research and operations (microfinance being an exception). In addition, we ask experts to comment on the organizations that they know the best and we do not suggest or influence their responses in any way (i.e. these are open-response questions). We also always try to err on the side of inclusiveness: we do not set specific limits about a foundation’s size and we try to invite the all relevant experts within an organization. However, we do reserve the right to disqualify experts on the basis of very poor quality of responses or other related factors.

As for how many experts actually respond, here is a table that summarizes our response rates:

These numbers reflect a bit of the evolution of the venture. First, the main challenge in education was “can expert crowdsourcing be done at all.” As Bob Ottenhoff (GuideStar CEO and member of our Advisory Board) said, this is a challenge that the nonprofit sector has been trying to solve in the last 10 years.

Once we demonstrated some initial success with education, the next important issue was “can expert crowdsourcing be done at scale” (i.e. beyond just referrals from a few personal connections). Building on our initial momentum, we managed to overcome this challenge as well, building big and representative expert networks in these two national/international social causes. Microfinance was particularly difficult, given that it is very much an international issue. Even though we intend to focus on the US for the foreseeable future, we saw this social cause expansion as an important experiment in how far we could push our methodology and we have drawn a lot of lessons from it. And finally, we wanted to see if our approach could work on the local level, which is why we decided to expand into the Bay Area, starting with homelessness. We are quite happy with these initial results as well and plan to continue our work on the local level.

To summarize, we have put a lot of effort in the past 2 years to create a solid methodology that allows us to compile expert opinion about impactful organizations in different social causes. We now also have a really good grasp of how to go about finding and recruiting experts to participate in our research. Nevertheless, we see a lot of room for growth and continued iteration of our surveys and experimentation around the best ways to excite and educate even more experts.

Processing expert opinions

Saundra also asked how we handle disagreement, which is a great question that we constantly ask ourselves as well. The short answer is that our goal is to be able to identify all good approaches toward solving a particular problem rather than make judgment calls about what is “good” or “bad” (something that our experts are in a better position to answer). We believe that by publicly discussing the pros and cons of each solution, the sector will be able to collectively arrive at better ways to have more impact. For an example of this, you can see the organizations that we highlighted in climate change, which include policy advocacy, grassroots, and scientific-based organizations. I should repeat that in the next version of our website, we plan to better highlight the diversity of these approaches for donors and provide a forum for discussions of the pros and cons of each.

On the question of whether some experts have disproportionate value, the answer is yes, as can be seen in the graph below:

Note: “avg votes” refers to number of organizations recommended by the particular survey taker group (out of 5). We have not yet computed this for microfinance.

As you can see, foundations professionals and academics do bring disproportionate value – at least in the research that we have already done. That means that they have more influence over the final results as well. We do not adjust the scores in any other way, although we are continuously reevaluating this issue and looking for ways to have better results.

In summary, we recognize and celebrate the diversity of opinions that the different types of experts bring, and are committed to highlighting our research results in an unfiltered and actionable way on our website.

Criteria for evaluation of charities

Saundra also inquired about our criteria for evaluating charities. As our current FAQs state, we do not set our own criteria, but instead ask for the criteria that experts use to evaluate nonprofits. That is a fundamental difference between our model and GiveWell’s (both of which have their pros and cons in my opinion). Currently, we do not make the compiled criteria public, but this will change in the next iteration of our website.

In terms of charities, as a result of our methodology, we generate a long list of nonprofits (for example, in climate change we had a total of 169 organizations that our experts mentioned). But out of this long list, we only highlight the few nonprofits that are consistently mentioned by our experts. And in the near future, we certainly plan to have a response box for nonprofits that want to reply to the identified areas for improvement.

Expert reviews & difference between impact and effectiveness

Saundra also pointed to the Alliance for Climate Protection expert review, asking whether having a strong marketing ability should be a reason to recommend a nonprofit. I personally believe that this is not the case – we should be recommending nonprofits based on impact first and foremost. However, our previous research has focused on effectiveness in order to study just how much good information we could extract. (We have now decided to zoom in and only focus on impact).

Effectiveness includes a number of properties: impact of their solution, efficiency of deploying resources, attracting top talent, good governance, etc. In that line of thinking, good marketing is another important characteristic of effectiveness that our experts have legitimately pointed out given the question that we asked. Of course, by zooming into any particular organization, it is easy to miss the big point: namely, there are a number of organizations in any given social cause, each with its own strengths and weaknesses. By making these pros and cons visible, and collaborating and learning together, organizations have a much better shot of coming up with better solutions that have more impact.

In summary, our intention is to compile expert opinions to highlight strengths and weaknesses of existing models to help make them better. Many of the social problems that nonprofits are trying to solve are incredibly complex and making progress takes time, effort, resources, and patience. That is why our goal is to acknowledge which organizations are doing well and then help put them on the right track towards further improvement through a combination of public scrutiny and constructive debate. I will further elaborate on this point, and how it fits into my previously discussed idea for continuous improvement, in a separate blog post.

In the first part of this blog post, I gave some perspective around our progress so far and plans for the future as a first step towards responding to GiveWell’s excellent analysis on our microfinance report. In this second part, I will attempt to address the specific concerns that Holden raised. If you are short on time, you can scroll to the bottom of each paragraph, where I have summarized our plans for improvement in the future (where it made sense).

Issue 1: the definition of “expert” is unclear

I believe that Holden is right on the spot criticizing us that we have not done a good job of defining an expert publicly. The very fact that he had to search in multiple places to find something so fundamental already speaks volumes about the urgent need to bring more transparency and reorganize the way we present information. We intend to do that as soon as possible.

The more substantial issue that GiveWell is bringing up is basically how we came up with these particular experts. Let me provide some details on our process here (a better explanation will most certainly become a central part of our webpage).

When we decide to expand into a new social cause, our first step is to prepare by researching and mapping out the space. In doing so, we identify – to the best of our abilities – all relevant experts in a given field. We list all foundations, nonprofits, and academic institutions and then find the experts’ contact information in these organizations. In addition, in some social causes we include journalists, policy makers, researchers, and other types of experts. The end result is a list of hundreds of experts who we then invite to participate in our surveys.

Of course, not every invited expert participates today. And of those who participate, we don’t necessarily include all responses from all experts because we do additional sorting of experts during the data collection process. What is more, not all experts are “equal” in terms of their expertise or experience. And some – most notably foundation professionals and academics – bring disproportionate value to our research. However, we strongly believe in our current inclusive model, which also invites nonprofit executives and senior staff to participate for several reasons. First, this group is obviously much bigger than researchers or funders. Not including them would be a missed opportunity to capture knowledge from those “working in the trenches.” Second, getting nonprofit executives involved opens the door to fix the broken feedback loops by using expert reviews as a starting point. For example, many nonprofits have expressed interest in being able to respond to the reviews (a functionality that we will be developing soon) – which would get us one step closer to increased transparency. The reason why nonprofit executives form such a large part of the final expert network is that there are simply more of them, compared to academics or foundation professionals. And to address the issue of the “others” category, we plan to develop more specific categories.

So, overall, I completely agree with GiveWell that we need to disclose substantially more information to build more trust among donors. We have internally identified the same issue and are working to redesign our webpage to address the concern.

Issue 2: who are the experts?

I have already mentioned the challenge of answering the question of who our experts are because of our still incomplete research in part 1 of this blog post. Our goal is to continuously get more experts “on the record” and include their bios. Based on our experience so far, experts are excited to participate in the research and I am confident that we will end up featuring more than the 31 people we have today in microfinance.

On the more specific issue of featuring microfinance skeptics, I want to make two points. First, both microfinance skeptics and proponents have recently been announcing results from different research projects that point in different directions with respect to the effects of microfinance. Regardless of who is right (which is often somewhat subjective), one obvious limitation of any crowdsourced approach (such as ours) is that it takes time to change the perceptions and opinions of a group of people. Second, our research actually features a number of microfinance skeptics at the urging of Dean Karlan, who has kindly provided us with critical feedback, served as a sounding board, and helped us connect with more researchers in the space.

Nevertheless, GiveWell is making a terrific point here: our ability to build trust and engage donors depends on being able to attract and feature credible experts. It is our intent to make substantial progress by being selective about our experts, asking them for more participation, and continuing to build large and representative networks of experts. In addition, we intend to provide better social cause summaries, which highlight the core issues as well as most important debates.

Issue 3: the recommended charities

GiveWell makes an interesting observation that the “list reads like a who’s who in large U.S. microfinance charities.” This statement is quite surprising to me, because our own experience when testing the expert mutual fund with donors is quite different. Perhaps to insiders, the list reads like the who’s who, but to the average donor that is certainly not the case. In fact, most people’s knowledge is limited to Kiva.

In some industries, it is perfectly fine for the “average” individual to not know the “who’s who.” But in most social causes in philanthropy that is not the case, because individuals provide the vast majority of funding – and they do it on the basis of very little good information. So for our national social causes, the lists might read like the who’s who to experts, but that is certainly not the case for our target user: the individual donor. And I would personally argue that perhaps an even bigger opportunity is our work on a local level – highlighting top nonprofits in different communities, which inspires people to join forces, support good organizations, and work towards solving the important societal problems of the day.

GiveWell also criticizes the expert-recommended charities and I personally certainly respect their opinion as microfinance skeptics. However, the Philanthropedia vision of accelerating the pace of social change has a different approach. We know that no organization out there is perfect – that is why we identify both strengths and areas for improvement. Our goal is to engage experts, donors, and nonprofits to collectively find and fund the best organizations that are solving some of the world’s toughest social problems. I believe that this can only happen through an inclusive process of continuous improvement, focused on bringing more transparency and demanding more impact from these nonprofits.

Issue 4: expert quotes / reviews quality

GiveWell also uses the Kiva expert quotes to analyze the usefulness of our expert reviews. The main concern that Holden identifies is whether experts are indeed doing a good job of helping donors increase their impact. This is a concern that I personally share and that is at the core of our push to increase the quality of our research in 2010 that I wrote about in Part 1. Some of the upcoming changes that we will be making in our methodology and survey design should help us get higher quality reviews. Further, as we gain more traction and educate more experts about our mission, this problem will gradually be reduced.

However, I should note that we continuously evaluate all aspects of Philanthropedia and look for ways to improve. We have performed research on our expert reviews in Bay Area homelessness and climate change (both with foundation professionals and donors) and the feedback we got was that the comments capture the essence of what the different nonprofit’s strengths and weaknesses are. For example, one of Kiva’s big accomplishments is that it has been able to capture the attention and imagination of the individual. This is a fascinating and rather unique achievement that our experts have legitimately pointed out. So we feel that the starting point is not as low as GiveWell might portray.

In summary, I believe that our expert reviews can be further improved by better surveying and educating experts on the importance of sharing their opinion. This issue is front and center for us as we look to expand into more social causes in 2010.

Issue 5: reinforcing bad incentives

Holden also hypothesizes that our methodology encourages experts not to disclose their names, and encourages charities to do whatever to get them recommended by Philanthropedia’s set of experts. This is the main issue identified by GiveWell that I disagree with. In my opinion, Philanthropedia has the opposite effect from the one outlined in the GiveWell post. First, we are increasing the transparency of experts. Even if only 31 experts agreed to publish their bios and stand behind the results, that is exactly 31 more than before. And our plans are to further push the bar, both in terms of what experts disclose and in terms of how many experts participate.

Second, Philanthropedia is getting areas for improvement of different charities “on the record” and thus inspiring them to participate in the discussion, increase their transparency, and further improve. The idea that charities will be able to manipulate the results is certainly plausible, but there are a number of reasons why I think it is highly unlikely:

• Experts have high morals and ethics – foundation professionals, nonprofit executives, and academics operate in an environment characterized by conflicts of interest. For example, researchers might need access and data to assess nonprofits’ impact, or foundation professionals might receive multiple grant proposals from competing nonprofits, while collaborating with them on other projects. In general, in our dealings with experts, we have found them to have very high personal and professional integrity.
• The Philanthropedia methodology relies on a high number of experts, which makes gaming the system challenging.
• We currently have a number of internal checks that makes manipulating the results difficult. Moreover, we will be adding other policies such as random checks, more due diligence, better conflict of interest management, etc. to further reduce such risks.
• Of course, I fully expect that if someone is dedicated enough, they will be able to game the system. That’s why we plan on very aggressively punishing any offenders, who violate the our trust, as well as the trust of the community.

In summary, I personally disagree that we are promoting the wrong incentives. I believe that the public discussions, engagement of new donors, and reactions that nonprofits have speak to that as well.

Conclusion

In conclusion, let me again thank GiveWell for their time and analysis. The rest of the Philanthropedia team and I very much agree with the vast majority of the issues that were identified, as well as the suggestions for improvement. We absolutely plan on raising the bar in terms of transparency and quality in order to be able to gain the trust of both donors and experts. Stay tuned for our reorganized webpage that addresses some of the concerns that GiveWell brought up.

GiveWell, in addition to many others, have kindly said that they see a lot of potential in our model. We at Philanthropedia certainly aspire to live up to that expectation, and with more analyses and constructive feedback, we have a real shot of getting there. So thanks again for your leadership Holden and Elie and for helping us get to the next level.

This is a response to GiveWell’s excellent analysis of our microfinance report that you can find here. You are reading Part 1, Part 2 will be posted tomorrow.

About a month ago, I was having lunch at the Hewlett Foundation with President Paul Brest, when he asked me what I thought the main challenge would be for Philanthropedia in 2010. Paul, as well as our other close advisors have done a terrific job reviewing our work, helping us improve, and being supportive – and we are very thankful for that.

That’s why I was very excited when I heard that GiveWell had decided to take a deeper look into our microfinance results. Philanthropedia the organization is built on several important principles, including feedback, continuous improvement, advice-seeking, and humility. And who better suited to provide feedback on our new and developing model than an organization known for incredible rigor and great analytical thinking? We are honored that GiveWell has taken the time to review our work and we urge you to do the same. Take a good look and let us know what you think!

In the next two blog posts, I will elaborate on our future plans and reply to Holden’s recent article (read it here). I have split the response into three paragraphs (spread out over two posts due to length):

• Our vision – I take a step back and discuss how we see ourselves and our commitment to making the philanthropic sector better.
• Intervening variables – I look at factors that I believe need to be taken into account when discussing results from microfinance (and all of our social cause research), as well as the GiveWell analysis.
• Detailed response – to address the majority of specific concerns that GiveWell brought up.

I will cover the first two points below, while in Part 2 I will focus on answering the specific concerns that were raised.

Our Vision

Philanthropedia’s vision is to make everyone give with impact. We envision a world:

• where all experts openly share their knowledge about nonprofits,
• where all donors give to impactful nonprofits in social causes they care about
• and where all nonprofits improve their practice because of increased transparency, accountability, and effectiveness within the nonprofit sector.

As I look forward to realizing that vision and in response to Paul Brest’s question over lunch a few weeks ago, I see several main challenges for Philanthropedia in 2010:

First, on our programmatic side, we need to raise the quality of our research even further. That can happen in three main ways:

• Improve our methodology – so far, we have put 2.5 years into developing a solid methodology for compiling expert opinion. And yet, there is more work that we can do to further improve our main tool for collecting information – our surveys (see below for more information). For the past 4 months we have been analyzing our progress to date and writing a whitepaper that discusses the pros and cons of our methodology and outlines ways to improve. We are still reviewing it internally and sharing it with close advisors to critique. We look forward to gradually making a lot of the whitepaper public in the next 2 months. In addition, we have learned a lot from expanding into our first four social causes and plan to use this knowledge to get better results as we keep adding more social causes in 2010.
• Improve our selection & recruitment of experts – we already have a good grasp of how to select and recruit experts (in Part 2, I will address Holden’s concerns on this issue). Nevertheless, there is a lot of room to improve, especially in terms of exciting experts at academic institutions and foundations, which often have significant expertise and a great holistic perspective on social causes. In 2010, we plan to actively recruit “high quality” experts by focusing on sharing the Philanthropedia vision with major foundations, universities, and nonprofits, as well as partnering with other organizations that already have existing relationships.
• Improve our expert reviews – we need to educate our experts on the importance of sharing their evaluations of nonprofits to help guide donors’ decisions. As straightforward as this sounds, it is also radically different from the status quo, which is characterized by lack of cooperation and information sharing.

Overall, I would summarize our aspiration to higher quality as being able to carry out research and credibly claim that we have recruited the right experts, and compiled their knowledge of top performing nonprofits in a specific social cause. I am confident that for our upcoming research (planned for the first 6-9 months of 2010) we will be able to push the bar much higher through a combination of our own efforts outlined above as well as the support and constructive feedback of GiveWell and others in the field.

Second, organizationally we need to continue building trust and become more transparent. We can do more to disclose our approach, provide more information about our experts (while respecting their privacy), and publish more of the information that we collect (such as what criteria experts use to evaluate impact in a specific social cause). We are committed to improving on this dimension as soon as possible, which will be reflected on our webpage soon. The blog post from GiveWell is another major inspiration to get there faster.

We have a number of other challenges, related to our customers (individual donors) such as earning their trust, gaining their attention, and developing engaging products (such as our webpage, mutual funds, and gift cards). I will not elaborate on these topics here in order to keep this blog post focused. However, I will be blogging more about this in January, for those who are interested.

To sum up, we are committed to further improving our model and have already put a lot of work into identifying many areas for improvement, with a number of changes planned for the next couple of months. GiveWell’s recent blog post also highlighted many of the same concerns, which I acknowledge and elaborate on in Part 2 of this blog post. But don’t take my word for it – please do continue to monitor our progress and provide constructive feedback and suggestions for improvement.

However, GiveWell’s recent blog post is important not just because it gives me an opportunity to discuss our vision, strengths, areas for improvement, and commitments for the future, but because it is a perfect example of the future that we want to create in the philanthropic sector. We envision a future characterized by open discussions, active feedback cycles, and continuous improvement within the sector. I know that it is not enough for Philanthropedia to talk the talk – we have to walk the walk as well, and I hope this blog post serves as additional proof that we plan to.

Intervening Variables

Before I discuss the specifics of Holden’s blog post, I want to mention a few factors (“intervening variables” if you will) that I think need to be taken into consideration when discussing our reports .

First, it is important to take into account the organizational lifecycle stage that Philanthropedia is in right now. I would call this stage “formative,” which means that a lot of things are still very much under development and not yet at the level, where I ultimately see them being. For example, our efforts to compile expert opinion are by and large groundbreaking (especially on a mass scale). Despite some early success, our journey to improve our methodology and research is by no means over (even though our upcoming whitepaper should help a lot). To further complicate matters, as we have been working on our methodology, we have also discovered that qualitative analysis on a large scale is very poorly understood. In other words, even though the idea of crowdsourcing with experts sounds very logical and straightforward in theory, in practice it is extremely challenging to “get right.” It is due to this complexity that I am advocating for looking at Philanthropedia as a work in progress. We plan to continue rigorously researching the subject matter, collecting data, and then learning and improving continuously from it.

Let me give a specific example to illustrate this last point on the importance of continuously learning based on collecting data. One of the ways that we grow our list of experts is by asking for referrals from experts, who we have identified in advance. Some of our advisors voiced a legitimate concern that this creates an opportunity to introduce bias and potentially “game” the system. Let me say right away that a lot of these concerns paint scenarios which are extremely unlikely – due to the large number of experts involved, the experts’ own honor system, and our monitoring and curating processes. Nevertheless, the concern is legitimate, which is why we collected data and demonstrated that there is no statistically significant difference between the predictive powers of cold called experts vs. referred experts.

It is because of this idea of continuous improvement on the basis of real data that I personally disagree with GiveWell’s conclusion that our model is “reinforcing bad incentives” (pretty much my only disagreement with their analysis). While there are plenty of ways that Philanthropedia could improve that I will write about in Part 2, I personally would argue that we have an approach that promotes the “right” incentives – in particular, educating experts and inspiring them to share actionable information with the world.

To sum up, the first intervening variable that I want to highlight is the notion of a “formative” stage in our organizational development marked by continuous improvement. Just like Wikipedia was initially heavily criticized for its strength/weakness – user-generated content that sometimes had errors – it has since improved dramatically. Therefore, I urge the reader to see Philanthropedia not as a snapshot in time, but as a process of gradual improvement. Today, we have excited and educated some 400 experts on the virtues of sharing their knowledge. A year from now, that number will triple. And in five years it will be in the high thousands. Imagine a world, where everyone has access to the compiled knowledge of experts to make better giving decisions. That is the vision that we are driving towards.

The second intervening variable that I think needs to be mentioned with respect to GiveWell’s blog post is microfinance itself. I almost wish GiveWell’s research was on our report in Bay Area homelessness or perhaps climate change, because microfinance is such a unique field. Our initial thinking in choosing microfinance was (in addition to helping donors) to eventually be able to correlate measures of impact with insight from our experts, in order to better study how and whether experts are a good proxy for measuring impact. The reason why I thought microfinance might be a good field to try to answer that question was that the financial nature of the sector makes it a bit easier to assess impact. Of course, in practice that is hardly the case: both GiveWell (here) and our friend at Yale, Dean Karlan (here, here, and here), as well as others have written extensively about the challenges in measuring and achieving real impact. What makes this even harder for an organization such as Philanthropedia that identifies top nonprofits is that microfinance is in many respects a special case, because there is a mixture of nonprofits and for profits and a complex chain of organizations that work together to deliver the complete product.

But an even more important consideration about microfinance is that our research is not yet complete. In particular, we collect biographies of experts after concluding the main part of our research. As a result (and due to the holidays), we currently still have only 31/131 or 23% of experts that are “on the record” and publicly identified – a figure, which is an order of a magnitude lower than the same indicator in both Bay Area homelessness and climate change, where we have a bit more than 40% of experts publicly identified. Just to be clear: I am by no means content with a 40% rate – we are constantly pushing the number up and reconsidering our policies. However, the limited data set does present challenges from an analytical point of view and can lead to misleading or limited conclusions.

So, with these two notes in mind, I explore and answer to GiveWell’s analysis in Part 2 of this already very long post.

Happy Holidays from Philanthropedia!

As we reflect on 2009, we are certainly excited about the progress we’ve made, but we are particularly grateful for the colleagues who have helped us along the way.

After about two years of thinking and planning, we were able to take the Philanthropedia idea and turn it into a full-time organization. We were legally incorporated in June and received our official nonprofit status in September. Thanks to our colleagues at the Hewlett Foundation and the support from the Philanthropy program, we were able to continue our research and complete three new social causes this year: climate change, microfinance, and homelessness in the Bay Area.

We offer a warm thank you to the 392 experts who participated in this research and shared their candid opinions about some of the strongest nonprofits in their sector. We’d like to particularly applaud this group for embracing the Philanthropedia mission and for the first time, sharing their professional opinions with the public in this manner. We really value the perspective our experts offer and look forward to continuing engaging professionals in many more sectors in 2010.

With the help of an extraordinary web development and graphic design team, we were able to launch our website on November 1 and make publicly available high quality information about top nonprofits, for free. We now highlight 48 top-nonprofits in 4 social causes on our website. Thank you to our donors who, in just a few weeks, not only donated thousands of dollars to these outstanding organizations, but also gave us feedback on how to improve Philanthropedia to better serve their needs. And thank you to the outstanding people who work at these great nonprofits, doing such important work for our communities.

One of the most exciting moments this year was certainly the joint press release on December 1 with GuideStar, Charity Navigator, GiveWell, GreatNonprofits, Philanthropy Action, and the Hewlett Foundation. That moment marked a turning point in the charitable giving sector for two main reasons. (1) Major charitable information and rating sites publicly denounced the overhead ratio as a useful stand-alone metric for measuring nonprofit success, and (2) these online philanthropy websites showed they are really willing to collaborate to improve online charitable giving and to push the sector forward. While there is still a lot of work to be done, we were honored to be part of that movement and look forward to continue developing these relationships and ideas in 2010.

As we look forward to 2010, in addition to continuing our research and continuing to develop our website for donors, we especially look forward to building on the relationships we’ve begun over these last few months. Thank you to Sean Stannard-Stockton, the Chronicle of Philanthropy, Richard Marker, Phil Cubeta, Jeff Raderstrong, SSIR, FLiP, Jennifer Q. Chen, and the many others who gave us feedback and engaged us in discussion about these important topics.

In order to improve the nonprofit sector, we believe in the importance of sharing ideas, giving feedback, and self-improvement. This can only truly be accomplished if we can create effective feedback cycles. Whether we are attempting to accomplish this via our expert reviews or trying to engage experts in philanthropy and online giving in general to improve our own organizational practice, we believe change is possible and necessary to move the sector forward.

And of course, we remain focused on our mission of presenting donors with good information about top nonprofits in an actionable way so it’s easy for individuals to make a bigger impact with their giving.

We wish you a very happy holiday season and look forward to collaborating more in the New Year!

–The Philanthropedia Team

Erinn, Deyan, and Dawn