OpenWave
OpenWave is invite-only. Sign in with the username and password from your invitation email.
OpenWave is a week of shared AI challenges. Everything you need is linked from this page.
What each challenge asks, what its data is like, and exactly how entries are scored.
How to read the data on your machine or straight from S3, and a viewer to listen in the browser.
Each day's focus, and when its challenges' data and submissions open.
Upload predictions for scoring. One submission per challenge per hour, up to 10 a day.
Standings for every challenge, with each entry's approach and source, and points for every participant across the event.
Once the event starts, team up with up to five others, split the challenges between you, and submit under one name. Strength in numbers, within reason.
Loading…
How it all works, with step-by-step examples: connecting, the data, storing and sharing files, shared services and submitting.
Private repositories for your code. Every submission names the repository and commit it came from.
You can read the challenges, browse the data and follow the leaderboards. Observers have no VM and don't submit.
To take part, put an SSH key on GitHub and post your GitHub username in the Discord (with your OpenWave username): see Getting ready in Start here. We'll tell you on Discord when your machine is ready; then sign out and back in.
For submit.py and your own scripts. Which to copy
Your first challenge, and the one open before the event starts. It's a trail through the environment: each stop gives you the clue to the next, and along the way you'll use what you need all week. It's encouraged before the event, not required.
Stuck? Ask in the Discord for a hint, but don't give away answers. Rules, points and what to do if VG says no: Guide → Check in.
No description has been published for this challenge yet.
| Split | Clips | Speakers | Audio | Clip length | Ground truth |
|---|
Every file and field, and why each property matters: on your machine, beside the challenge's README.md.
Applied in this order to both the ground truth and your prediction before they are compared.
| Step | Rule | Example |
|---|
Any scored submission earns the taking-part points. Early-bird points (for the first five people to reach the qualifying score) and standing points (for the top three) need a submission at or past it. Standing places are held twice over: once among submissions that run offline, and once among those that use an external API (alone or alongside local models). The full rules are in the guide: Leaderboards and points.
A submission is one JSON file (UTF-8) with a single key, predictions, holding one entry per test clip. Each key is a clip id exactly as it appears in test/ids.txt, without .wav.
For example, with this challenge's real test ids:
A file that breaks a rule is rejected with a message saying what is wrong, and it doesn't count towards your submission limits.
Your trail starts with the first clip below. The same files are on your machine, at ~/share/s3/datasets/welcome/.
| Clip | Audio | Length | Transcript |
|---|
The recordings at the end of the trail aren't here: they come from VG's whiteboard.
On your VM: a read-only folder, ~/share/s3/datasets, read like any other.
| Utterance | Audio | Ground truth |
|---|
| Challenge | Variant | Best, runs offline | Best, uses an external API | Qualifying score | Entries |
|---|
A solution that runs offline needs no external API: every model runs on the participant's own machine. One that uses an external API calls a hosted model API (Amazon Bedrock or another provider) for at least one step, even if the rest runs locally. Each kind has its own leaders and standing points. ✓ the leader has reached the challenge's qualifying score; ✗ not yet.
Points for taking part in every challenge, for reaching a challenge's qualifying score early, and for holding a top place. Choose a name to see where their points come from. How points are earned: Leaderboards and points in the guide.
| # | Who | Points ranks | Taking part | Early bird | Standing | Full set | Awards | Challenges | Leading |
|---|
The people whose submissions the most others built on, from what each submission lists under builds on, across every challenge. Recognition only: it earns no points. Build on others' work
| # | Who | People who built on them | Submissions crediting them | Most built-on entries (challenge, people) |
|---|
A submission passes someone else's only by beating it by at least one percentage point; closer than that is a tie, and the earlier one keeps the place. So a slightly better score can rank below an earlier one. Leaderboards and points
The last step of the check-in trail. Once VG has given you the six recordings, transcribe each in its own language and script (in the sixth, only the speaker from the first), and submit them below as one file: {"predictions": {"message_1": "…", …, "message_6": "…"}}. Like every submission, it needs a repository; a small one is fine: a README saying how you transcribed it, and your script or notes.
Submissions open when the event starts. Until then, your first task is the check-in: choose Check in above.
You're signed in as an observer, so you can't submit. This page shows what participants do.
Every submission must point to the code that produced it: a commit that is pushed to a repository the organizers can read, holding exactly the code that wrote this predictions file (nothing uncommitted), and the command that runs it. Explain in the repository's README what else it needs: data, model downloads, versions and hardware.
So others can see and build on it, the repository is checked before scoring. It must be in the event's GitHub organization, private, and shared with the participants team (repository Settings → Collaborators and teams, Read). At the submitted commit it needs a README.md, a dependency file (pyproject.toml, requirements.txt, a lock file…) and the scripts your reproduce command runs. Details: Make a submission in the guide.
The predictions file has a different shape for each challenge. See Submission format on the challenge's page for the exact structure, the rules it is checked against and an example.
Check the file against this challenge's submission format.
The same submission with curl, for example straight from your VM once your predictions are written. Run it from your repository so git rev-parse HEAD picks up the commit; commit and push first, because the scorer can't see uncommitted code. Your VM has curl and jq.
1. Copy your access token and paste it in place of <token> below. It lasts until your sign-in expires (24 hours).
2. Set where to send it:
3. Wrap your predictions file with the details the form asks for, then send it. Leave builds_on empty or list submission ids.
The reply is the same JSON the form shows: your scores, or an error such as the rate limit or a malformed file. There is no group token: you always send your own. To submit as your group, the submission names it with group: "<group-id>" (or --group <group-id> with submit.py); the snippet above includes it once you can submit as one, and the scorer checks that you're a member.
Nobody wins a speech week alone. Well, somebody might, but they'd be very tired.
A group is up to 6 participants who submit together under one name. Start one and you're its captain, the person who lets people in (with great power comes a small admin burden). Or ask to join one, and wait for the captain to notice. Any member can submit to any challenge as the group, except the check-in, which is a solo mission. On the leaderboards and in points the group counts as one entrant (👥), with one hourly limit between you. Members can still submit as themselves too.
Why a cap? Because a group of thirty is just "everyone", and we still need someone to beat. Why form one, and how to split the work: Guide → Work in a group.
Groups open when the event starts. That's on purpose: the best teams form once you've met people, seen who's good at what, and found who else is stuck on the same diarisation bug. Until then, start a conversation on Discord, not a group. (And do the check-in: it's a solo mission.)
No groups yet. Somebody has to go first; it might as well be you.
| Group | Captain | Members | Size |
|---|
Each day has its focus. Until 3 pm the day before you can peek at the first files of its challenges; the full data opens then (Monday's on the Friday before, at 9 am), and submissions at 9 am on the day. Once open, a challenge stays open.
Before the event: we encourage you to take on the check-in challenge, open now. It's a short trail through the environment, a good way to settle in before Monday: Check in.
Loading the schedule…
Loading the participant guide…
The guide, example code and submit.py are also on your machine in ~/share/s3/resources/.
Organizers only. What each challenge is worth on the participant leaderboard (100 × its tier) and the qualifying score that early-bird and standing points need. Both are set in the challenge's challenge.yaml (tier, qualifying_score, as a fraction in the metric's units) and go live with publish_challenge.py <dir> --metadata-only. A challenge without a qualifying score counts every scored entry as qualifying, so set one before the event: about the published baseline, so that the early bird is the first to beat it.
| Challenge | Tier | Worth (W) | Qualifying score | Entered | Qualified | Leading |
|---|
Organizers only. Every participant VM, the jump host (SSH) and the NAT (internet for the VMs). The jump host and NAT start with any VM and stop once no VM has run for a while.
| Machine | Role | State | Since | Type | Address |
|---|
Organizers only. The addresses allowed to reach the jump host's SSH port. Seen: where someone used the portal or SSH from, kept for a set time after their last use. Approved: added by an organizer, until its expiry. Requests wait here for a decision.
| User | Address | Kind | Allowed until | Detail |
|---|
Organizers only. Invited: the account exists but its owner has not signed in yet.
| User | Role | Status | Enrolled |
|---|
Organizers only: never shown to participants. Kept in challenges/datasets.json.
Organizers only: names the source datasets. Every submission's test score, worked out again on each source dataset's clips alone (a mixed clip counts for every dataset it contains). A dataset that most approaches score far better on than on the rest of the test set may be in their models' training data; one they score far worse on may be hard, or flawed. The harness stores the breakdown with each new submission; rescore_submissions.py --write adds it to older ones.
| Dataset | Languages | Likely in models' training data? |
|---|
Best and median: over every submission, on that dataset's clips. Gap: the best on that dataset against the best on the whole test set, positive where the dataset is easier.
| Challenge | Dataset | Clips | Best | Median | Whole test, best | Gap |
|---|
Organizers only. What this AWS account has used, in US dollars, including everything credits paid for. From the account's own billing export, which AWS refreshes about once a day, so the latest day is usually incomplete. A participant's share is their VM, its disk and the Bedrock calls their VM made; infrastructure is everything shared (NAT, jump host, storage, the platform). The page is rebuilt every hour.
| Used (before credits) | Credits | Paid | Participants | Infrastructure | Not attributed |
|---|
| User | Used | Of which Bedrock | Bedrock, last 7 days (estimate) | By service |
|---|
| Service | Infrastructure | Participants | Not attributed | Total |
|---|
| Day (UTC) | Used | Credits | Participants | Infrastructure | Not attributed |
|---|
From the invocation log (which role called which model, and the tokens; never prompts or responses), within minutes of each call. The cost is an estimate from a price list in modules/costs; the billed figure arrives in the export a day or so later.
| Day (UTC) | Caller | Model | Calls | Input | Output | Cache read | Cache write | Estimate |
|---|
Organizers only. Every submission to every challenge, newest first, not just each person's best.
| When | Challenge | User | Score | Approach and source | Submission |
|---|