Behind the scenes of the Umbraco Package Awards 2022

Ever wondered what the process of nominating and selecting the Package Award winners is like? 2022 was my second year on the Umbraco Package Award jury panel, and in the name of transparency and openness, I thought I would share with you what this process looks like and how we selected this year's winning packages and contributors.

Nominations, jury selection, and introduction

A few months prior to Codegarden, a blog post is shared on Umbraco.com letting the Community members know that the nominations for this year's Umbraco Package Awards are open for submissions. This blog post also includes the deadline, eligibility criteria, the jury selection process, and so on.  

Once the Community has submitted all of their nominations, a jury panel is selected. This jury - between 6 and 8 judges as the aim - is hand-picked by Umbraco HQ with support from the Community Package Team, but anyone interested in joining can reach out to The Package Team. Note, however, that you cannot compete in the Umbraco Package Awards and also be part of the jury panel that same year.

The 2022 jury panel consisted of:

The jury members are invited to a private Slack channel and an introduction Zoom call, where they go over each nominee and filter out any package or contributor that doesn’t meet the eligibility criteria. 

During this introduction meeting, the jury also aims to set this year's award categories and which package should go into each category (although this is an ongoing process since some packages are hard to categorize without first trying them out).

The 2022’s categories included:

• Best Editor Experience
• Best Expansion
• Best Newcomer
• Best Innovation/Hack
• Best Package Care/Holistic Experience
• Best Developer Tool
• Jury’s Choice (previously Overall winner)
• Package Contributor of the Year

The jury also goes over each nominated package contributor and share their experience with each contributor as a foundation for the group evaluation meeting, where the year’s winner and runner ups will be selected.

Individual testing

Once the categories and packages are set, each jury member will test out each package individually over the upcoming weeks. During this period, jury members will test if and how the package works, what problem it solves, and how it holds up against other packages in that same category.

Each jury member also gives each package a score from A to D that later will be used during the group evaluation process. (Fun fact: one package even got A+++ this year.)

During these weeks, each package and its contributions are heavily discussed in the private Slack channel to help each jury member form an opinion about each contestant. Jury members also share notes in a shared document which is then used in the “the jury said” presentation summaries at Codegarden.

Mid-process meeting

About half-way into the evaluation process, the jury meet again for another Zoom call to go over the packages that have been tested so far, and if there have been any issues installing or testing any of the packages. At this stage each jury member will try not to share their personal opinion yet to avoid any influence on other jury members.

In some cases the packages up for evaluation might be too big to give a fair test in a local environment and in these cases the jury reaches out to partners and contacts known for working with these packages and have them share their opinion. This is just to give the jury an idea of how much a specific package might be appreciated, but it’s still up to each jury member to give their own score based on their own opinion.

After this meeting, each jury member continues to evaluate each package individually before the last upcoming group evaluation.

Group evaluation meeting

At the last meeting before Codegarden the jury meet once again for another Zoom call and now it's time to select the winners in each category. Each jury member shares their experiences, and deciding on a winner in each category is usually a smoother process than you’d think!

Sure, there have been cases where the jury was not 100% united, but this is where the individual scoring system comes in handy, and on some rare occasions a vote has been needed to settle a winning package. Besides the winner, the package runner ups are also selected and last but not least the jury decides on this year's winning contributor.

Summarizing and presenting the award

Once all the winners have been selected, the Community Package Team steward, Ronald Barendse, puts together a presentation and slidedeck for when this will be presented at the upcoming Codegarden. During the awards ceremony,  each jury member is also invited to come up on stage and present a winner in a specific category, like we see in the previous image when Paul Seal presented the winner in the “Best Editor Experience” category (congratulations again Marc!) 

As soon as the winners have been announced, a blogpost on Umbraco.com is published listing all the winners in each category. Also, the Codegarden Twitter account does live announcements during the awards show!

See you again next year!

So now you know a little more about the process behind selecting the Umbraco Package Award winners. As I mentioned at the beginning of this article, I want to highlight the fact that this jury panel is not a closed group, so if you are interested in joining us next year, don’t hesitate to reach out to The Package Team.

Lastly, I just want to say thank you to everyone who competed in this year's Package Awards, everyone who watched the award ceremony, my fellow jury members, and to all the other amazing package creators and contributors out there who add more value to the Umbraco ecosystem and help make Umbraco a better product. #h5yr!