I mean, it's not necessarily stealing cores from your games.
If you have a high core-count CPU, many games don't use all of them and will benefit more from having exclusive high-priority access to 3/4 of your CPU, rather than shared normal priority access to the entire CPU.
Here's an example from
SOMA, since I was playing that earlier today.
The game places the majority of its load on two threads - one main thread and one supplementary.
Having 22% headroom with game mode on vs only 6% with game mode disabled seems like it could matter.
And I don't have much open in the background there, just Steam, Afterburner, and a couple of minimized browser windows. (Edge)
The resources being used by them are less than 2% of the CPU's total capability - but because they were sharing a core with the game, they were close to potentially affecting gameplay.
I can see 2c/4t CPUs greatly benefitting from Game Mode if they give 3 threads to a game and reserve 1 thread for non-game processes.
Quad-cores probably have the most to lose here, since you're giving up an entire core whether it has simultaneous multithreading or not.
It's really going to be game dependent whether or not it is beneficial to use Game Mode with them.
In most new games I would think not, assuming that you don't have anything running the background which is using a lot of resources.
I do wish that Game Mode was a bit more configurable though.
On an 8c/16t CPU, I'm not sure that it's necessary to reserve a full 2c/4t for non-game processes.
I'd prefer that enabling Game Mode would give you an option to set the split for each application.
In well multithreaded games that can use 8c/16t, there's a good chance that sacrificing 2c/4t would hurt performance more than it would help, but it may be beneficial to set aside 1c/2t instead.
For older games that don't use as many resources, I might prefer to set the split to 4c/8t for each to keep the game on one CCX.