Our Independent Project Evaluation Process
Contributors: Chelsea Robinson, Rachel WilliamsTwice a year, we take a pause on coaching to evaluate the current direction, level of support, and future outlook of our projects. This piece will explain how we conduct this process and the important role of outside evaluators.
Taking a balanced perspective on whether our projects are working well and adding value is important to us. We have a deep relationship with all our projects and project leads, including generating the approach and strategies, hand-selecting project leads, and coaching them even when they are mature. In this regard, we are more like a venture studio than an accelerator program. This is a deep relationship enhanced by evaluations that encourage us all to do our best.
Twice a year, we take a pause on coaching to evaluate the current direction, level of support, and future outlook of our projects. This piece will explain how we conduct this process and the important role of outside evaluators.
About the Evaluation Process
Open Lunar leverages the expertise of hand-picked members of our community to seek input on project evaluations. The goal of the evaluation is to determine whether a project is on track or off track and whether it should be granted more funding
In July 2023 we evaluated two initiatives: Breaking Ground Trust and The Lunar Policy Platform. This is a mid-year evaluation, which acts as a checkpoint. When we conduct a mid-year evaluation, it is a chance to see what advice, input and redirection may be valuable for an initiative for the second half of the year. For this evaluation, we invited six individuals to participate as evaluators to see how our Initiatives are performing.
What Criteria we are Evaluating
When we evaluate Initiatives, which are mature projects independently being led and operated by a self-managing team, we use a comprehensive set of assessments. There are four major things we’re looking for:
Scoring highly on our impact criteria
Scoring highly on our viability criteria
Scoring highly on our sustainability criteria
Progress against stated milestones
Each is explained below. We assess all of these traits and indicators when deciding how best to support these more mature projects.
Impact Criteria
If you’re familiar with our ideas-to-hunches process, you’ll know that before we invest in anything, we consider a set of nine impact criteria. These focus on assessing whether an idea is going to impact the future of the Moon in the way we want.
Examples include “Shared, Open Access Infrastructure/ Resources: This evaluates the project’s ability to develop infrastructure and resources that are openly accessible to various stakeholders. This can include data, research findings, or physical assets that can be shared for collaborative benefit.”
Another example is “Developing Effective Non-State Commons Management: This measures the extent to which the project supports a management model that involves multiple stakeholders, such as non-governmental organizations and the international community, in the governance of common resources. Non-State Commons Management refers to managing shared resources like the lunar surface or space, in a way that is not solely controlled by any government but involves collaborative stewardship.”
Viability Criteria
Assessing a project’s viability involves three points.
Is it fundable?
Is it staffable?
Is there appetite from other partners?
In essence, the most important part of assessing viability is whether there is a problem-solution fit and whether that is validated by other parties.
Sustainability Criteria
We consider whether the project has a positive trajectory towards sustainability. This includes elements of whether there’s good entrepreneurial leadership in the team that can be strategic and adaptive, whether there’s a fundraising or revenue model which can support the cost of operating the venture, and whether the team is passionate and committed to making it happen.
Milestones/Progress
Finally, we also track one other factor: Milestones being met. We look at whether the project is progressing as planned, or with good momentum and traction. We work with the leadership of each Initiative to determine on a rolling 6-month basis what their key goals and milestones are, and we use that to manage our expectations of what we believe we should anticipate to see by the next evaluation period. These are created collaboratively.
Role and Responsibility of External Evaluators
Six people agreed to be external evaluators with us for this mid-year review of two Initiatives. They were asked to read reports provided by our project teams, and reports from our staff, and then fill out a survey about how they rate the projects against the criteria explained above. Once they had submitted their views, we all met for discussions on Zoom to compare notes.
Evaluators are not decision-makers, they are submitting their input to Open Lunar’s leadership team from an external point of view. Open Lunar staff take this input very seriously and use it as strong guidance for decisions on funding, and future support levels for projects.
Evaluators were handpicked to provide this voluntary input based on our team’s judgment of what kinds of expertise would be most appropriate and valuable for reviewing the work of Breaking Ground and Lunar Policy Platform. Not all six evaluators were involved in each evaluation process, and we are deliberately anonymising who evaluated which project to enable evaluators to be open and honest in their responses.
Meet our July 2023 Evaluators:
Below we are proud to share the evaluators for July 2023. In cases of contractual obligation to their employer, some evaluators must remain anonymous. None of these individuals work for Open Lunar or have direct day-to-day involvement in any of the projects. We are enormously grateful for their time, energy and insight to help improve our work and the work of our community. The list is in alphabetical order.
1. Anonymous
2. Christopher Johnson is the Space Law Advisor at the Secure World Foundation and an Adjunct Professor of Law at Georgetown University where he co-teaches the Space Law Seminar. Mr. Johnson has written widely on space law and policy issues and represents the Secure World Foundation at the United Nations Committee on the Peaceful Uses of Outer Space (COPUOS). He has recently begun research for a PhD in space law for a thesis entitled "The International Legal Order for the Moon".
3. Erik Franks is CEO of Cislune. Previously, Erik was CEO of Tesseract Space, Inc., responsible for business operations, program management, sales, and fundraising. Tesseract raised $2.3M, has three customers in space utilizing our propulsion technology, and exited to Benchmark Space. Erik led Masten Space Systems Mojave Operations from 2013-2017. Erik designed, built, and test flew a four-passenger airplane using an automotive engine. Prior to that effort, Erik founded and was CEO of an online and retail bookstore with 50 employees and $3M per year in revenue.
4. Olga Volynskaya is a qualified international lawyer with 15 years of academic and practical experience in space law, policy and diplomacy. She was Chief International Law Advisor of the Russian space agency (Roscosmos), then practiced space law at the Russian Ministry of Foreign Affairs.
Dr. Volynskaya has extensive expertise in academic legal research and teaching. She is author and co-author of 88 scientific publications, lectured on international law, space law and space policy in universities of France, Germany, Japan, the Netherlands, Russia, South Africa and Saudi Arabia.
Olga is a member of the International Institute of Space Law (IISL).
5. Ranjana Kaul, Partner, Dua Associates Law Offices, New Delhi, India advises commercial space companies and provides advisory on international space treaty mandates at the intersection of policy and statutory requirements for national activities in outer space for peaceful purpose. She is Member, Board of Directors, International Institute of Space Law, Paris; Member, National Advisory Committee of Indian Space Association; and Founding Member of Spaceport SARABHAI (S2), India’s first space focused independent think tank. She holds LLM from Institute of Air & Space Law, Faculty of Law, McGill University, Canada; LLB from Delhi University and PhD from University of Pune, India. ranjanakaul@duaassociates.com
6. Yoav Landsman is a visionary and accomplished multidisciplinary professional in the field of space. With a long track record as a space engineer, innovator, and mentor, Yoav has played pivotal roles in numerous missions, including missions to geostationary orbit and the surface of the Moon.
As a highly regarded consultant and mentor, Yoav has provided technical solutions, mission planning guidance, and business development support to international space startups and organizations. His exceptional achievements in space have earned him the prestigious honor of being a Karman Fellow, symbolizing his leadership and significant contributions to shaping the future of space.
Currently residing in the Grand Duchy of Luxembourg, Yoav has recently taken on the role of Business Development Manager at ClearSpace, a European startup committed to offering commercial solutions for the pressing challenge of space debris.
