There is a very simple reason for this happening - Google needs to keep making more money, so over time they need to raise advertising prices or increase click volume. There isn't going to be a dramatic increase in click volume anytime soon, so for Google to grow at 20%+ a year, they need to raise prices over time, which they do via quality score and social engineering.
I've watched Google do this relentlessly since 2006. They are quite good and the fact that people in here are blaming the advertiser for not managing their account well is proof of how good Google is at social engineering their way to profits with AdWords.
Can you explain more what you mean by QS and social engineering being the way Google is raising prices?
My understanding is that AdWords is a an auction system, where your position in the auction is a combination of your Max bid, inventory, and a QS multiple. If Google is manipulating QS, it would just change the ordering of auction winners, rather than overall CPC, right?
My first thought was that prices are rising because 1) more competition on the ads [more companies, higher prices], and 2) as a kw cost/value saturates, more companies were "overpaying" [bidding more than expected ROI], due to any number of causes.
AdWords started as a pure auction system where to rank higher it was basically CPC * CTR, which maximized profit in a simple way for Google and it worked well. The problem is that some people are too good at getting high CTR ads, so their CPC's got too low.
What Quality Score does is it adds a layer of indirection to the system where they can tweak some invisible dial so to speak to increase the "quality threshold" for ads. Google will tell you what each keywords QS is, but it's not as explicit as it could/should be as to why.
The point of QS is to enable Google to maximize profit while maintaining a decent user experience, but the problem is, at any point Google can turn the knob of QS and increase their ad rates. If Google needs to make more money, they can drop QS and people will bid higher to rank higher. Or, new advertisers who aren't very good will bid the "recommended" bids of $2 or $3 per click, where in the pre-QS world, those numbers were less than $0.50 in most cases.
They also can use QS to make you redesign your pages to be more what google wants, even if it doesn't increase your conversions or align with your goals. Google totally can and does dictate what kind of pages you can advertise and how you can advertise them.
So it's not just a auction system. It's much more clever than that. It's more like if eBay auctions had a bidder quality rating where you would have to pay more if your bidder quality score was too low.
One of the things that Google seems to generally understand is that a good business is built on delivering ever-higher value.
But depending for growth on getting an ever-higher percentage of the total value generated, rather than generating ever-more value, is the road to ruin. Happy customers rarely switch, even if somebody else is offering an as-good or somewhat-better deal. But having a bunch of mildly irritated customers is a massive opportunity for a competitor who isn't trying to hit an arbitrary set of revenue targets.
It's experience and observation. I started doing PPC and SEO back in 2006 and what I've seen over and over again is that Google is trying to black box as many of the success factors as possible so that it is harder for people to game the system.
For example, in SEO, Google continues to devalue on page optimization, and various methods of getting links because it break's Google's algorithms. So, they invent/demand the use of rel="nofollow" because otherwise they couldn't detect paid links. Google works very hard to tell people they should nofollow advertising links and whatnot under the guise that it's better for the internet or it's more ethical, but the reality is they need the nofollow "link condom" to keep their search engine from looking stupid and being easily gamed.
In the PPC space, AdWords also does a lot of things to black box their system to make it harder to succeed. For example, they introduced "Quality Score" which is a metric they can use to rank ads, which seems nice. But originally they had a simple CPC * CTR bidding system which was simpler and easier to game. More importantly, it made it hard for Google to maximize profits because if someone had a really high CTR ad, it would effectively lower Google's ad rates.
In reality, QS adds a layer of indirection where Google can tweak various quality metrics to raise their advertising rates in subtle ways that people won't complain about because instead of saying that they are raising rates, they can just lower a quality score and you have to bid more if you want traffic for that keyword. After they rolled out the quality score system initially, I believe then CEO Eric Schmidt was quoted as saying something along the lines that they are able to show fewer ads and charge more while creating a better user experience. That's a nice way of saying they're raising their advertising rates.
In short, Google is always struggling to balance between making more money and maintain a certain user experience. To achieve that goal, they create these black box systems that are harder to influence individually. That allows them to periodically raise their rates and "improve quality".
I'm not sure I would do anything dramatically different if I were Google, but their black boxes and social engineering efforts should not be underestimated.
I've watched Google do this relentlessly since 2006. They are quite good and the fact that people in here are blaming the advertiser for not managing their account well is proof of how good Google is at social engineering their way to profits with AdWords.