Guest Articles

Friday
December 21
2018

Nicky Khaki

Ease Off on the Accelerators: Why GALI’s Latest Study on Accelerator Programs May Be Overstating Their Impact

 

Note: This post is one of NextBillion’s 12 most influential posts of 2018.

 

There’s been a great deal of focus on early-stage venture accelerators as of late, particularly those in emerging and developing markets. And for good reason: Their numbers are growing so quickly –  seemingly spurred by the emergence of unicorns, gazelles, and other creative zoomorphisms – that it’s been hard to keep up. GUST’s 2016 Global Accelerator Report cited 579 accelerator programs worldwide, a roughly 50 percent increase over 2015; 2017 and 2018 are likely to show similar trends. Throw in tech hubs, incubators, hacker spaces, innovation centers and other enterprises dedicated to helping early-stage businesses and that figure would no doubt balloon. On the African continent alone, GSMA referenced 442 active startup-supporting entities earlier this year. And as someone who has invested in and worked closely with startups in Sub-Saharan Africa, I’ve taken keen interest.  

These programs certainly seem like a good thing: efforts to help nascent businesses overcome obstacles that would otherwise keep promising commercial and social enterprises from reaching their potential. But they have meaningful costs: direct costs to implement, direct costs to participate, and opportunity costs for participants and the supporters of the programs. Some of these programs, especially those supporting social enterprises, are backed by scarce donor or aid-based funds. So, while it’s important to give these programs an adequate runway to hit their stride, it’s also important to objectively review and benchmark them to ensure that the precious resources that are being invested in them are being used wisely.

That’s why I’m encouraged to see organizations putting in the effort to publish reports on the efficacy of these programs, such as the recently released Accelerating the Flow of Funds into Early-Stage Ventures, by the Global Accelerator Learning Initiative (GALI). These organizations should, however, be careful to structure their studies to yield instructive results, explicitly convey the limitations of their data and, subsequently, what can be concluded – an area where GALI, in this case, misses the mark.

 

A Fundamental Problem with the Analysis

The GALI report analyzes data collected from surveys with candidates of 52 accelerator programs across the globe who accepted applicants over the period of 2013 to 2016. Out of the 4,463 total applicants in these programs over this period, the survey collected data on changes in revenue, raised equity, debt and philanthropic funds from both the 526 participants in the programs and 1,733 ventures rejected by the programs. This data was also collected for the subsequent reported calendar year for each cohort. GALI aggregated these four figures into what it calls the Net Flow of Funds (NFF) and concluded that, “After one year, these accelerators were responsible for an additional $30,846 of incremental funding for each of the ventures they worked with.”

This sounds great, but critically, the study doesn’t control for the selection criteria that the programs use to choose their cohorts. By comparing ventures that were accepted into the programs to those that did not participate, accelerators are, in effect, tipping the scales on both sides. Selecting likely high-performers (in the parlance of the study, those more likely to achieve a high NFF in the year after the program) and comparing them to those that, overwhelmingly, did not make the cut means the selection criteria could very well be the sole factor in the difference in post accelerator performance. Based only on these selection criteria, it could be possible to get the same results that GALI presumptively attributes directly to participation in the programs. It’s even possible that those participants that were selected could have done as well or even better by skipping the program and investing the saved time into their businesses.

In other words: One can’t make meaningful inferences about the effect of accelerator programs by comparing the average performance of two separate sets of ventures (participants vs. rejected) when these groups are significantly different from the outset.

 

Distorted Groupings Can Lead to Flawed Assessments

You may think that successfully selecting for ventures with superior relative performance is an impressive skill in and of itself – it isn’t. Let’s take a simple example: If I took a random sampling of startups in a specific geography and pulled out those with revenues above the group average and revenue growth in the top quartile, it wouldn’t be hard to guess which group would be more likely to attract more investment on average, irrespective of what a particular accelerator program did. This is just for illustration; some accelerators do consider pre-revenue companies, but that doesn’t mean there still aren’t dramatic and easily identifiable differences between these companies, which are highly correlated to their ability to grow and attract funds.

Nor is this a purely academic stance: I recently managed an impact investment vehicle, and from 2015 to 2017 my team evaluated over 600 startups in Sub-Saharan Africa, many of which applied to accelerators. In my experience, there was a clear and sizeable gap between those that could be regarded as ready for acceleration and those that were not. Consider also that many accelerators’ selection criteria are informed by what investors in their ecosystem look for – some even have investors on their selection committees. So it’s not surprising how much this selection process, if not accounted for, can misrepresent the value added by participation in accelerator programs.

GALI could better control for selection bias by comparing the average results among accelerator participants to those that were selected but ultimately chose not to participate. It would be even better if participants were randomly chosen from a broader set that passed the selection process and compared to the rest of the selected group that did not participate – because there may be a systematic reason why ventures choose not to participate that would bias the comparison. There are also other methods to mitigate selection bias even if the programs are not on board with randomization, such as only comparing those near the cutoff point separating accepted and rejected applicants, but you get the point. These approaches would result in sparser comparison groups, but without them one can’t responsibly draw conclusions about the real impact of these programs.

 

The Cost of Miscalculation

This is a big deal, because this report will be used (as its predecessors were) to promote accelerator programs. Quotes will be pulled from the report, asserting that “Many programs are most effective at stimulating net revenue growth, while others are best at increasing the supply of outside equity investment.” Articles like this NextBillion post from GALI will connect dots that aren’t there by stating that “Accelerators in emerging markets do seem to be working – accelerated ventures raise more revenue and investment than unaccelerated ventures.” And favorable headlines will proliferate, declaring that “Startup Accelerators Have Become More Popular in Emerging Markets — and They’re Working,” along with the attendant social media chatter. Meanwhile, none of these assertions have been supported by an apples-to-apples comparison, and they can mislead entrepreneurs – particularly when taken out of context.

Add these to the stats that many accelerators post about their programs – aggregate revenue, equity, jobs created, carbon-captured and other accomplishments that their graduates accumulate, implying full credit to the program – and the decision to participate seems obvious. Yet if the benefits are not actually that clear cut, a miscalculation here can be extremely costly for an enterprise: Accelerators may often take equity (typically 2 – 10 percent accompanying a modest capital investment), or charge participation fees. They invariably require a significant investment of time (typically three to six months) from participating entrepreneurs, who are already stretched thin. And opportunity costs can be particularly pronounced for very early-stage startups with lean teams that need to temporarily relocate or travel to participate.

 

A More Responsible Approach

To be clear, my point isn’t that accelerators have no positive impact. Nor is this critique meant to pick on GALI or to bring other parts of the report into question – the comparative analysis between programs is particularly informative. GALI has done great work in painstakingly building out and generously sharing a veritable treasure trove of data, but as stewards of this data they have a unique opportunity to use it to give entrepreneurs a fuller picture.

Early-stage entrepreneurs and accelerators across the globe – particularly those in developing markets – would derive immense benefit from a GALI study that goes even further. Ideally, this study would not only control for initial selection, but would also control for other distinct benefits of accelerators, such as promoting/networking, training, mentoring, talent identification, signaling, etc. Further data and analysis to tease out the variability in bias-controlled post-performance across different programs and geographies would also be extremely helpful

This approach would allow entrepreneurs to better assess the value of the services provided and make informed decisions relevant to their geographies. It would also encourage accelerators and other entities to focus on developing those services that generate the most value for entrepreneurs. Where data is sparse or inconclusive, such studies would do the ecosystem a service by explicitly pointing this out and allowing entrepreneurs to use their own judgement until there is sufficient data to support an unambiguous claim.

This may be asking a lot – this additional data may be more challenging to gather, and it may even result in less excitement and glowing headlines. But it would better advance GALI’s goal of exploring the effectiveness of accelerators while also looking out for early-stage entrepreneurs around the world.

 

Editor’s Note: This article is a response to this recent NextBillion article from GALI, and it expands on the author’s initial views – and GALI’s response – in the comment section of that article.

 

 

Nicky Khaki is an emerging markets investor based in Johannesburg, South Africa. He has worked in private equity, venture capital, and impact investing across multiple geographies and sectors.

 

Image credit: Pixabay.com

 


 

 

Categories
Impact Assessment, Social Enterprise
Tags
accelerators, data, impact measurement, social enterprise, startups, venture capital