Hardest Blacks Survey Results

[Bahnarli]

So I recently did a survey which asked players to rate some of the hardest black maps on the server. The maps included on the survey were Thrice, Skyrunner 2, Ragnarok, Magisterium: Ash, Oculus, Oculus v3, Lost Soul, Crimson Planet, Final-E, Happy Thoughts, Heartless, and Cataclysm. If I was re-doing this survey, I would also have included Patience (although I doubt it would break the top 7), and although some people asked about Chaoskampf, I intentionally did not include it as I felt too large of a percentage of its difficulty was based on non-parkour elements, which I wanted to avoid for the sake of this survey.

Analysis:
There were 22 23 responses, 11 12 13 of which ended up coming from players with all maps done, or only missing a map that was not in the top 5 in difficulty according to the results from players with all completed. I was surprised by how much disagreement there was in general, but there was still enough agreement to draw some conclusions.

Results:
Thrice, Cataclysm, and Lost Soul are the clear top 3.
Cataclysm received 1 vote of 6th
Thrice received 1 vote of 6th, and 1 vote of 9th
Lost Soul received 1 vote of 11th, 1 vote of 12th
The rest of their results were top 5 hardest, shown in the table below

Hardest	1st	2nd	3rd	4th	5th
Cataclysm	4	5	0	1	2
Thrice	4	2	1	3	1
Lost Soul	3	1	6	1	0

	Median	Average	Average (Drop Low and High 2)	Highest Rating	Lowest Rating	Standard Deviation
Cataclysm	2	2.61	2.33	1	6	1.76
Thrice	3	3.31	2.89	1	9	2.39
Lost Soul	3	3.84	2.77	1	12	3.66
Ragnarok	5	4.76	4.77	1	8	1.92
Happy Thoughts	4	4.85	4.67	1	11	2.82
Magisterium: Ash	6.5	6.58	5.88	1	12	3.82
Heartless	7	7.00	7.11	3	10	2.08
Crimson Planet	7	7.54	7.56	4	11	1.98
Oculus	9	8.69	9	3	11	2.21
Oculus v3	9	8.77	8.89	4	12	2.55
Final-E	9	9.23	9.56	4	12	2.24
Skyrunner 2	11	9.66	9.22	3	12	2.84

Fun Nonsense:
If we take the averages of these scores as the accepted community average, we can then see how well each person's individual scores fit this average, and find the MSE (mean squared error, standard measurement of error) for each player who fully (or nearly fully with a little fiat) filled out the survey. Doing so, we find that the top 5 players who most closely matched the community ratings overall were

1. Bahnarli - MSE 2
2. Mercata - MSE 2.82
3. Anrein - MSE 4
3. TheGloryThief - MSE 4
5. AverageIPA - MSE 4.17

I will not say who the least matching was, because that seems mean, but the worst MSE was tied, at 12.8. (DM me if you want to find out your score lol). If you want to better understand these metrics from an intuitive standpoint, you could instead think of Absolute Error, and take the square root of these numbers, i.e. the least community agreeing player was different by a score of 3.57 per ranking, on average, and the most community agreeing player was different by a score of 1.4 per ranking, on average (for most players, this constituted general agreement on most, and high disagreement on a few maps).
I forgot that MSE cannot be directly translated to Absolute error, as it penalizes more off guesses more, i.e. if I rated a map 1st that the community rated 4th, MSE would consider that worse than me rating 2 maps 1st and 2nd, when the community rated them 2nd and 4th, but Absolute error would see these as the same. If you want to know Absolute error of your data, dm me and I can tell you.


How many maps are "high black"?
This was the other question asked on the survey, of which there were 14 usable responses here. The responses are as follows (those who gave a range I took the average of)
8, 5, 4, 5, 3, 7, 10, 5, 8, 6, 6, 9, 9, 7
Which is an average of 6.6 and a median of 6.5

Several players also explained their criteria for their answers (paraphrased):
-any map I consider harder than happy thoughts
-any map I consider harder than oculus v3
-any map I consider harder than heartless
-any map I consider harder than heartless
-Mag: Ash + any map I consider harder than Mag: Ash
and my personal favorites
-idk gut feeling
-at least some of them (thanks marjho)
-Maps that I remember having a particularly hard time on idk
-The best way would likely be to base it off of community consensus in a survey like this one

If we were just to decide to split high black on the largest gap in average community rating, we would split between magisterium: ash and heartless, which 1: lines up pretty well with the community average of 6.5 as the amount of maps they'd consider high black, as that would constitute 6 maps included, and 2: lines up quite well with the splitting criteria used by several of these players. That's pretty cool, and in the future, I think I will probably refer to Cataclysm, Thrice, Lost Soul, Ragnarok, Happy Thoughts, and Mag: Ash as "high black", and Heartless and Crimson planet as "borderline high black". So "harder than heartless" seems to be a very principled decision criteria for what "high black" is, and I could probably give or take hea and crmp in this consideration, but would strongly not consider any other maps in this category.


Thanks for your time, I hope you found this as interesting as I did!

Edit: Added in one more result that I hadn't originally included because it had ties, but it was ultimately usable, did not change any positional results.
Edit2: Added in one more result that came in later, did not change any positional results

 

[Bahnarli]

Here's the raw data scrambled up in case anyone else wants to do analysis on it or look at it for fun:
(note that 1 player had not done mag, and 1 player had not done skr2, so those only have 12, not 13 results)
Cataclysm: 1,1,1,1,2,2,2,2,2,4,5,5,6
Thrice: 1,1,1,1,2,2,3,4,4,4,5,6,9
Lost Soul: 1,1,1,2,3,3,3,3,3,3,4,11,12
Ragnarok: 1,2,3,4,5,5,5,5,5,6,6,7,8
Happy Thoughts: 1,2,2,3,4,4,4,5,5,6,8,8,11
Mag: Ash: 1,2,3,4,4,5,8,9,10,10,11,12
Heartless: 3,4,6,6,6,7,7,7,8,8,9,10,10
Crimson Planet: 4,5,6,7,7,7,7,8,8,8,10,10,11
Oculus: 3,7,7,7,9,9,9,10,10,10,10,11,11
Oculus v3: 4,6,6,7,8,8,9,9,11,11,11,12,12
Final-E: 4,7,7,9,9,9,9,10,10,11,11,12,12
Skyrunner 2: 3,6,8,9,9,11,11,11,12,12,12,12

Funny to look at the actual numbers and see, seeing it in this format, you can really see how divided players are on especially Magisterium ash, also you can see where people really diverge from the pack (I'm looking at you, players who are the only player to rate a map under 5)

The players whose data was included in this survey were Bahnarli, Solawr, Soker, KittyCatElite, fruitlukes, Tetration, AverageIPA, LendKaru, Pray, MarvelFamily, Viceinc, TheGloryThief, Mercata

If you think that I made an error not including your data, or if you didn't fill out the survey but have done all these maps (or are only missing 1 or 2 near the bottom), let me know so I can include the data (although I would prefer if you didn't look at the data first, then go "hmm, I want that changed, let me fill out the form to push it in that direction" (although disagreement with some things and having looked at the data before filling out your survey is totally fine)).

Hardest Black Maps Survey

A survey to help determine public consensus on the very hardest black maps. Feel free to answer this form as long as you've completed at least 2 of the maps on this list.

forms.gle