Democratising charity evaluations

Project summary

Create or adapt an existing platform to

pool individual-users' charity evaluations,
provide aggregated information on these evaluations to would-be donors
encourage would-be donors to give financial support to evaluations they've found helpful

What are this project's goals? How will you achieve them?

The EA Community Choice competition has been a great experience, engaging many of us with numerous microprojects we'd never have otherwise have heard of. It's inspired me to think of a new approach to to charity evaluation, which to my knowledge the EA movement hasn't explored: pooled and publicly funded public individual evaluations.

The idea will be some combination of the following:

Create a website, or repurpose an existing service, such as GiveWiki, Patreon, Manifund, Google Worksheets, and/or an instance of Forum Magnum (the EA forum/Less Wrong codebase), to collect and display individual evaluations of any charities they choose to evaluate.
Structure as much of the evaluation process in standardised formats as possible (for example, rating out of 10, ranking, estimated room for more funding, text fields for track record, $ per output, estimated value of outputs, hours spent on evaluation, conditional yes/no recommendation given interest in some particular field, disclosures of pre-existing relationships with evaluatees (that is, the people they're evaluating), other metainformation about the evaluator, whether they're evaluations of an existing charity or of a potential area for a charity to be created into, and many more)
Develop an API that will allow users to run queries on multiple individual evaluations in the database (to get e.g. mean rating, mean rating weighted by number of evaluations, number of ratings above x etc). This API could be a part of GiveWiki, of Manifund, or a standalone service. If standalone have a very basic frontend that tabulates the data (maybe using a service like Streamlit to present it).
Give a simple pipeline for would-be funders to give some amount of their money to express appreciation for helpful evaluations.
Tentatively: give a simple-pipeline for charities to pay evaluators to assess them (if so, this payment would also need to be a field of public note)

This would address various problems with the current evaluation landscape:

Bandwidth limitation preventing large funders from effectively supporting small projects
Specialisation in the evaluatee's focus area often being high value, and too few professional grantmakers to have sufficient specialisation
Physical proximity to the evaluatee often being high value (being able to go into their office, for example)
Lack of good feedback mechanisms or accountability for existing evaluators
'Democratic' concerns around most current evaluation being ultimately funded by a couple of very wealthy individuals
Allow pooling 'microevaluations', where different individuals may feel like they have an important insight on a charity or could generate some particular insight with a small amount of work, but don't want to commit to a full evaluation of other aspects of the charity

Ideally the project would be self-sustaining, through a further option to donate to its maintenance/development if it needed bespoke software, or through the community's efforts if it only needed (for example) a shared spreadsheet. Conceivably it could become a social benefit corporation, if many charities are willing to pay to be evaluated, and if that seems like a good direction to go in.

How will this funding be used?

The project has a number of stages, each of which could be used to rule out whether it's worth proceeding. The funding would be used to support me some part of the way through the following steps:

Gauge level of support for this type of project. How much does the community feel like this would address a need that isn't being sufficiently met? I'm hoping the Manifund response itself will give some indication of this.
Choose an existing platform, or resolve to create one. If need be, build an ultrasimple prototype - for example, a seeded database with a limited API.
Reach out to various people who've already posted evaluations or work that could be adapted into an evaluation on the forum for permission to include their work in the evaluations database.
Present the idea to a further subset of the EA community for input, feature requests, etc.
Ideally find some non-neglible seed funding available to contributors to incentivise use of the platform.
Present an MVP (minimum viable product) to the whole EA community.
If it gains tractions, iterate and improve.

I imagine steps 1-5 would take approx 4-6 months, which is what I'm seeking funding to cover - though if I got substantial encouraging signals from early funders and possible early adopters, I'd be willing to stretch that budget out longer to bridge the gap long enough to get funding to cover more substantial development.

So the level of funding will take me some If I start a step, I'll commit to at least finishing that step when funding runs out.

Who is on your team? What's your track record on similar projects?

My career has taken me to many places, all of which have some relevance to this project:

I worked as a full stack web developer for various companies including Founders Pledge. My personal site is here.

More recently, I've been working on an LTFF project to allow evaluations of extinction risk, global catastrophic risk and 'shortermist' work within the same framework. Explanatory sequence. Simple interactive calculator. Code for full calculator.

I was also a Trustee of CEEALAR, aka the EA hotel for 5 years (and remain an advisor), where I assessed hundreds of potential beneficiaries of the project.

I've also been engaged with independent community building in the EA community for many years - I cofounded the forum version of Felicifia, a proto-EA utilitarianism forum, and more recently started the EA Gather Town.

What are the most likely causes and outcomes if this project fails?

The most likely causes are:

Not enough engagement from evaluators, especially those who've written evaluation-like work already. If fewer than half of them show any interest, that would be an orange flag.
Not enough theoretical buy-in from would-be donors - people are content with the current evaluation landscape, or don't think this would improve it.
Not enough practical buy-in from would be donors: either too few people both have money and the motivation to use the platform, or the awareness is not high enough.
In particular, I'm aware that it's hard to get sustained awareness for projects within the EA community. I run a semi-regular series of intra-EA advertising posts on the forum because of this difficulty. I think this is substantially the highest risk of failure, and, uncomfortably, also the hardest to get reliable early feedback on.

If I sunk serious effort into the project before finding it to be unworkable, I would write a post-mortem for the forum, explaining what assumptions were wrong, what mistakes I'd made in the process, and any other generalisable lessons I'd learned.

What other funding are you or your project getting?

None. I have only just consolidated this idea, though I've been thinking for a long time about problems with incentives and feedback mechanisms within the EA world.

I might apply for funding from the Infrastructure Fund, but this would be competing with them - and is implicitly critical of their work.