- Thread starter
- #1
So I recently did a survey which asked players to rate some of the hardest black maps on the server. The maps included on the survey were Thrice, Skyrunner 2, Ragnarok, Magisterium: Ash, Oculus, Oculus v3, Lost Soul, Crimson Planet, Final-E, Happy Thoughts, Heartless, and Cataclysm. If I was re-doing this survey, I would also have included Patience (although I doubt it would break the top 7), and although some people asked about Chaoskampf, I intentionally did not include it as I felt too large of a percentage of its difficulty was based on non-parkour elements, which I wanted to avoid for the sake of this survey.
Analysis:
There were 22 23 responses, 11 12 13 of which ended up coming from players with all maps done, or only missing a map that was not in the top 5 in difficulty according to the results from players with all completed. I was surprised by how much disagreement there was in general, but there was still enough agreement to draw some conclusions.
Results:
Thrice, Cataclysm, and Lost Soul are the clear top 3.
Cataclysm received 1 vote of 6th
Thrice received 1 vote of 6th, and 1 vote of 9th
Lost Soul received 1 vote of 11th, 1 vote of 12th
The rest of their results were top 5 hardest, shown in the table below
Fun Nonsense:
If we take the averages of these scores as the accepted community average, we can then see how well each person's individual scores fit this average, and find the MSE (mean squared error, standard measurement of error) for each player who fully (or nearly fully with a little fiat) filled out the survey. Doing so, we find that the top 5 players who most closely matched the community ratings overall were
1. Bahnarli - MSE 2
2. Mercata - MSE 2.82
3. Anrein - MSE 4
3. TheGloryThief - MSE 4
5. AverageIPA - MSE 4.17
I will not say who the least matching was, because that seems mean, but the worst MSE was tied, at 12.8. (DM me if you want to find out your score lol).If you want to better understand these metrics from an intuitive standpoint, you could instead think of Absolute Error, and take the square root of these numbers, i.e. the least community agreeing player was different by a score of 3.57 per ranking, on average, and the most community agreeing player was different by a score of 1.4 per ranking, on average (for most players, this constituted general agreement on most, and high disagreement on a few maps).
I forgot that MSE cannot be directly translated to Absolute error, as it penalizes more off guesses more, i.e. if I rated a map 1st that the community rated 4th, MSE would consider that worse than me rating 2 maps 1st and 2nd, when the community rated them 2nd and 4th, but Absolute error would see these as the same. If you want to know Absolute error of your data, dm me and I can tell you.
How many maps are "high black"?
This was the other question asked on the survey, of which there were 14 usable responses here. The responses are as follows (those who gave a range I took the average of)
8, 5, 4, 5, 3, 7, 10, 5, 8, 6, 6, 9, 9, 7
Which is an average of 6.6 and a median of 6.5
Several players also explained their criteria for their answers (paraphrased):
-any map I consider harder than happy thoughts
-any map I consider harder than oculus v3
-any map I consider harder than heartless
-any map I consider harder than heartless
-Mag: Ash + any map I consider harder than Mag: Ash
and my personal favorites
-idk gut feeling
-at least some of them (thanks marjho)
-Maps that I remember having a particularly hard time on idk
-The best way would likely be to base it off of community consensus in a survey like this one
If we were just to decide to split high black on the largest gap in average community rating, we would split between magisterium: ash and heartless, which 1: lines up pretty well with the community average of 6.5 as the amount of maps they'd consider high black, as that would constitute 6 maps included, and 2: lines up quite well with the splitting criteria used by several of these players. That's pretty cool, and in the future, I think I will probably refer to Cataclysm, Thrice, Lost Soul, Ragnarok, Happy Thoughts, and Mag: Ash as "high black", and Heartless and Crimson planet as "borderline high black". So "harder than heartless" seems to be a very principled decision criteria for what "high black" is, and I could probably give or take hea and crmp in this consideration, but would strongly not consider any other maps in this category.
Thanks for your time, I hope you found this as interesting as I did!
Edit: Added in one more result that I hadn't originally included because it had ties, but it was ultimately usable, did not change any positional results.
Edit2: Added in one more result that came in later, did not change any positional results
Analysis:
There were
Results:
Thrice, Cataclysm, and Lost Soul are the clear top 3.
Cataclysm received 1 vote of 6th
Thrice received 1 vote of 6th, and 1 vote of 9th
Lost Soul received 1 vote of 11th, 1 vote of 12th
The rest of their results were top 5 hardest, shown in the table below
Hardest | 1st | 2nd | 3rd | 4th | 5th |
Cataclysm | 4 | 5 | 0 | 1 | 2 |
Thrice | 4 | 2 | 1 | 3 | 1 |
Lost Soul | 3 | 1 | 6 | 1 | 0 |
Median | Average | Average (Drop Low and High 2) | Highest Rating | Lowest Rating | Standard Deviation | |
Cataclysm | 2 | 2.61 | 2.33 | 1 | 6 | 1.76 |
Thrice | 3 | 3.31 | 2.89 | 1 | 9 | 2.39 |
Lost Soul | 3 | 3.84 | 2.77 | 1 | 12 | 3.66 |
Ragnarok | 5 | 4.76 | 4.77 | 1 | 8 | 1.92 |
Happy Thoughts | 4 | 4.85 | 4.67 | 1 | 11 | 2.82 |
Magisterium: Ash | 6.5 | 6.58 | 5.88 | 1 | 12 | 3.82 |
Heartless | 7 | 7.00 | 7.11 | 3 | 10 | 2.08 |
Crimson Planet | 7 | 7.54 | 7.56 | 4 | 11 | 1.98 |
Oculus | 9 | 8.69 | 9 | 3 | 11 | 2.21 |
Oculus v3 | 9 | 8.77 | 8.89 | 4 | 12 | 2.55 |
Final-E | 9 | 9.23 | 9.56 | 4 | 12 | 2.24 |
Skyrunner 2 | 11 | 9.66 | 9.22 | 3 | 12 | 2.84 |
Fun Nonsense:
If we take the averages of these scores as the accepted community average, we can then see how well each person's individual scores fit this average, and find the MSE (mean squared error, standard measurement of error) for each player who fully (or nearly fully with a little fiat) filled out the survey. Doing so, we find that the top 5 players who most closely matched the community ratings overall were
1. Bahnarli - MSE 2
2. Mercata - MSE 2.82
3. Anrein - MSE 4
3. TheGloryThief - MSE 4
5. AverageIPA - MSE 4.17
I will not say who the least matching was, because that seems mean, but the worst MSE was tied, at 12.8. (DM me if you want to find out your score lol).
I forgot that MSE cannot be directly translated to Absolute error, as it penalizes more off guesses more, i.e. if I rated a map 1st that the community rated 4th, MSE would consider that worse than me rating 2 maps 1st and 2nd, when the community rated them 2nd and 4th, but Absolute error would see these as the same. If you want to know Absolute error of your data, dm me and I can tell you.
How many maps are "high black"?
This was the other question asked on the survey, of which there were 14 usable responses here. The responses are as follows (those who gave a range I took the average of)
8, 5, 4, 5, 3, 7, 10, 5, 8, 6, 6, 9, 9, 7
Which is an average of 6.6 and a median of 6.5
Several players also explained their criteria for their answers (paraphrased):
-any map I consider harder than happy thoughts
-any map I consider harder than oculus v3
-any map I consider harder than heartless
-any map I consider harder than heartless
-Mag: Ash + any map I consider harder than Mag: Ash
and my personal favorites
-idk gut feeling
-at least some of them (thanks marjho)
-Maps that I remember having a particularly hard time on idk
-The best way would likely be to base it off of community consensus in a survey like this one
If we were just to decide to split high black on the largest gap in average community rating, we would split between magisterium: ash and heartless, which 1: lines up pretty well with the community average of 6.5 as the amount of maps they'd consider high black, as that would constitute 6 maps included, and 2: lines up quite well with the splitting criteria used by several of these players. That's pretty cool, and in the future, I think I will probably refer to Cataclysm, Thrice, Lost Soul, Ragnarok, Happy Thoughts, and Mag: Ash as "high black", and Heartless and Crimson planet as "borderline high black". So "harder than heartless" seems to be a very principled decision criteria for what "high black" is, and I could probably give or take hea and crmp in this consideration, but would strongly not consider any other maps in this category.
Thanks for your time, I hope you found this as interesting as I did!
Edit: Added in one more result that I hadn't originally included because it had ties, but it was ultimately usable, did not change any positional results.
Edit2: Added in one more result that came in later, did not change any positional results
Last edited: