I am not exactly 100% sure why Turkey is so hell bent on S400
on the other hand I know its a top end system
but still
My theory is that the selection was originally based on merit, i.e. capability/cost/tech transfer. When the Americans started threatening to reneg on their existing contracts, that both made it an issue of national pride
and reinforced the argument for not relying solely on American hardware. Obviously the broader situation in Syria (where the American position involves close alignment with the Kurds who are anathema to Turkey) and the personal qualities of Erdogan contributed to the current situation also.
Given the historical differences between Turkey and Russia, and even the recent past where they were on opposite sides re: Assad in Syria, punctuated by the shooting down of a Su-24, the current relationship between Turkey and Russia, and deteriorating relationship between Turkey and the US, and (to a lesser extent) Turkey and Europe, represents an enormous coup for Russia. In Moscow they probably can't believe their luck.
(It also helps Russia that Turkey's biggest advocate in Europe, the UK, is currently doing its best to alienate itself from other European nations.)