Since its inception, the State of JS survey has struggled to accurately reflect the diversity of the web development community, especially when it comes to gender demographics.
While women are a minority in the web development community to begin with, an even lower proportion participate in the State of JS survey, with the consequence that their experiences and voices can easily get lost in the resulting dataset.
This is problematic for obvious reasons, and this year, we implemented additional efforts to increase participation from gender minorities.
How We Got Here
We realized that part of the issue was our recruitment strategy. The survey is spread through channels such as Discord, X, Bluesky, YouTube, Reddit, and many other social platforms, which can themselves exhibit biased demographics.
While I was aware of this issue from the start, a lack of resources, time, and organization on my part meant that this bias was not properly addressed, bringing us to the current state of things.
Getting Professional Help
This year, to help reevaluate our recruitment strategy I sought professional help from Sidney Brandhorst, a data scientist and researcher who has experience with designing sampling strategies to mitigate bias.
Sidney started out by taking the survey herself and writing a detailed report outlining a couple ideas to improve it. As she puts it, “re-working the survey itself to promote inclusivity is a crucial step in improving audience diversity.”
Sidney then also implemented a targeted outreach strategy focused on increasing the proportion of women participating in the survey.
Note: it should go without saying that women are not the only minoritized group deserving of being better represented in the survey. But for practical reasons it made sense to focus on one demographic at a time, and this seemed like a good place to start.
Targeted Outreach Report
Sidney contacted 26 organizations focused on supporting diversity and inclusivity in the tech community. Of those 26 organizations:
- 3 (Coding Black Females, Black Tech Pipeline and DiversifyTech) shared the survey for free with their audience.
- 1 (Codebar) shared the survey with their audience in exchange for a sponsorship fee and a one-hour workshop.
- 2 responded to the inquiry but declined to share the survey at this time.
Leaving aside Codebar (which is covered in the next section), those organizations sent us a combined 21 respondents (that we were able to track), 2 of which were women.
This illustrates some key drawbacks with this approach. First of all, the fact that so many organizations did not reply at all makes it hard to get your foot in the door.
But even when an organization does share the survey with their audience, the proportion of that audience who actually participates in the survey might still be extremely low.
After all, we can barely get 10% of our own mailing list subscribers to take the survey. So it should be no surprise that conversion rates for an unrelated community would be low.
That being said, a huge thanks to Coding Black Females, Black Tech Pipeline and DiversifyTech who are all amazing organizations worth learning more about.
Paid Sponsorships Report
Another thing we tried this year was sponsoring content creators directly, with the parallel goals of expanding our audience, and supporting creators who bring something new to the community.
We ended up sponsoring three individual creators, and one organization (the aforementioned Codebar):
We spent a total of USD $1,281 on these four sponsorships, which resulted in 57 extra respondents according to our tracking, 8 of which were women.
While this is still not much compared to the size of the dataset, it provided us with some valuable datapoints:
- Afor Digital performed the best, which points to live streamers being able to drive higher engagement compared to newsletters or even YouTube videos.
- While Codebar did not send as many respondents, the audience it did send consisted of 54% women, which is extremely high for a developer community.
Talking with these creators, it also became apparent that being a woman yourself doesn’t necessarily mean reaching a diverse audience will be easy. As Afor stated: “I’m trying to reach a wider audience of women, but it’s certainly a challenge even as a woman, […] even trying to use inclusive language, to prioritize women, to always have 50% of [women] guests […] so that other women can have references.”
Snowball Sampling Report
Finally, a note about “snowball sampling”. I implemented this sampling strategy by showing a small note to any women respondents asking them to share the surveys with more women around them.
This proved to be the most effective outreach method out of everything we tried, resulting in 69 extra respondents, 64% of them women (28 respondents) or non-binary (5 respondents).
The fact that the strategy that was built into the survey itself was also the most successful is quite encouraging, because the survey is the one thing we can actually control!
Overall Results
We went from 4% women in 2023 to 6% this year. This may seem small, but at least it’s good to know that our efforts did have a measurable impact.
Here’s everything we tried –including both successes and failures– as well as what we’re doing next.
Doing Better in 2025
In addition to broadening these outreach strategies in 2025, we will focus on improving survey timing and organization overall.
We only started our efforts once the survey was already well underway, and as Black Tech Pipeline’s Pariss Athena highlighted, the lack of proper advance notice hampered their outreach efforts on the survey’s behalf: “I heard about the State of JS survey a week before the deadline, so I had to rush to get the word out. […] If someone had reached out much sooner, the response rate from us would have been higher.”
This is another “easy” fix (or at least something that’s under our control) and definitely an area we will improve in for the next round of surveys.
Additionally, we are working on increasing our sponsorship budget to try to find new organizations and creators to partner with – and if you think that should be you, drop us a line!.
Conclusion
My goal throughout this process was to identify an approach that results in meaningful, measurable change, and while I don’t think we’re there just yet, I do believe we’re on the right path.
After all, organizations such as Codebar show that it’s possible to build diverse, inclusive developer communities. And if they can do it, there’s no reason why we can’t learn from them as well!
So I’m looking forward to more outreach, more sponsored creators, and more experiments in 2025. And as usual, I’ll make sure to document all of it here!
To help counteract this, we provide a built-in query builder that lets anybody tailor our data to any audience subset. ↑
While Sidney did not specialize in studying the developer community, I also came across the profiles of other researchers who do, such as Orsolya Vásárhelyi and Siân Brooke. I would love to work with them in the future, and I encourage you to check out their work. ↑
Huge thanks to Pariss Athena, Sidney Brandhorst, and Stéphanie Walter for reading drafts of this post.
Source link
lol
