(12/18 Note: Sports Interactive have examined the match engine code and not found any egregious errors, while also pointing out a flaw in the testing methodology. There is further testing to be done with updated methodology. Give them serious credit for the speed with which they’ve looked into this!)
The response to the last article, where I determined that it’s likely that the Decisions attribute is bugged and behaves the opposite of the expected behavior, has been immediate and passionate on both the FM Subreddit and in the Discord. The most common response was that the information is interesting, but not definitive because the testing of the one stat in a vacuum takes out too many variables. That’s certainly possible, so a new set of tests has been devised and executed, and the results should put this one to bed.
To come up with a real-world scenario, we want an average team for its league. To that end, I simulated the 2017-18 Premier League 20 times, and decided on Newcastle:
|League Table Place
|Average results after 20 seasons, 760 match-days (n=760)
So that is our baseline Newcastle side. I created two more Workshop files; both edit only Newcastle, and only the Decisions attribute of all players. In one file, all players have Decisions set to 1. In the other, all players have Decisions set to 20. Those two files were used to each simulate the 2017-18 season a further 20+ times. (Thank you to T6 who also submitted data for the Decisions 1 file.) Then each of the three files (Unmodified, Decisions 1, Decisions 20) had their best and worst seasons dropped.
|Average League Position
I have a suspicion that the Decisions 20 file would actually perform worse than that if there hadn’t been constant coach sackings and owners splashing cash to bring new players in with lower Decisions attributes.
You are reading that correctly. The team with Decisions set to 1 pulls in nearly double the amount of points over the course of the season than the team with Decisions 20.
I’ve made a couple charts to show how the seasons went from a league table perspective, and I need to reiterate that the only thing changed between the 3 sets of data is the Decisions attribute:
Or, to visualize it with the results together…
A couple of other interesting things…
I want to point out something else that has come up frequently: All I’m after, here, is for this attribute to behave in a logical manner. I’m not making any assertions on that you should look for players with Decisions set to 1, or avoid players with Decisions set to 20. What I’m finding is that it’s possible that players with high Decisions attributes may not be playing to their fullest ability due to what appears to be an inversion somewhere in the match engine. What I’m also finding, is that Decisions has a massive impact on the team, far more than a single attribute probably should, and that’s definitely the case when the behavior is opposite of what one would consider logical.
One more comparison I want to look at, how the Newcastle Decisions 20 squad compares to the Major League Statistics Decisions 20 squad (where all other stats are set to 10 for all players):
|Newcastle – Decisions 1
|Newcastle – Decisions 20
|MLS18 – Decisions 20
So Newcastle, full of players that are capable of playing in England’s top flight, would likely do fine matched up against a squad of players with all 10s and the identical 20 in Decisions.
I’m not sure what there is left to say. Decisions does not behave as it should, when tested both in a vacuum as the only variable, and in real-world testing where many factors are in play. We still see huge swings in team performance that is solely due to the Decisions attribute, and they work in the exact opposite manner as they logically should. SI needs to fix this.
I believe I’m done working on Decisions unless someone comes up with a huge revelation or oversight on my part. I’ll be working more with the results of Major League Statistics in further articles, as well as new tests still to come. The Decisions Linearity Testing is over at this point and I’m marking the experiments as completed on Bearpuncher Labs. So if you want to help, please keep submitting those Major League Statistics results!