Have You Been Throttled Recently?

Have You Been Throttled Recently?

As long-time users will know, Amazon over the past few years have taken steps to control their excessive server usage. Some might say the steps taken have bordered on excessive, but certainly no one can doubt that as one of the most visited sites on the planet, they do rack up an outrageous number of hits, day in and day out. We’re going to examine one of these controversial steps, how it impacts you as a user of Associate-O-Matic, and what countermeasures you can take in order to limit the number of times you or your site visitors encounter this particular problem.

To many new AOM users, it can appear out of nowhere as an unwelcome surprise; while surfing your site, you click on a category link and instead of a page of product, you see a line of text: REQUEST FROM AKDG2GGQAJY6IKKAZQJY IS THROTTLED. What? What does this mean? You click on another category, and you may see it again. Eventually you find a category that seems to work normally. Going back to the first category, it now also works like always. But when you try page two, the message returns. It’s an intermittent error, but why?

What the message means is that the site(s) linked to your Amazon Associate ID are making too many requests to their servers (they prefer one request/second). Like a vehicle rolling down a steep grade, you’re moving too fast to maintain a safe speed, and in order to keep from crashing (the server), Amazon is literally ‘applying the brakes’ to your account, in order to maintain control. You may think that your one little store is hardly capable of disrupting Amazon’s powerful bank of servers, and normally that would be the case. However, you have to multiply your single site by the hundreds of thousands (if not literally millions) of hits that Amazon’s network gets every second from around the world. Just like any highway, the method for ensuring the road is safe for everyone is to prevent the speeders from overloading the system. So it’s not just your site, it’s the thousands of others who are also creating multiple hits too quickly that can disrupt the entire network.

At this point, you’re probably saying to yourself, “Hold on here; my one (or two or three) site(s) are not exactly overloaded with traffic. According to the Statcounter/Google Analytics/Amazon Sales Reports, I’m not drowning in hits. So where is all this activity coming from that I’m not seeing?”

The answer to this is: Bots. These are search engine robots that travel the web, seeking out new sites, and then endlessly indexing them, over and over. They rarely appear in popular site statistic trackers such as Statcounter or Analytics because those services detect visitors by their use of Javascript. AOM uses Javascript for various features such as the lightboxes (the ‘enlarge’ button for product images, for example). Visitors (via their web browsers such as Internet Explorer, FireFox, Opera, Safari, etc.) interact with the Javascript, and this is picked up by the tracking program. Bots on the other hand, tend to ignore Javascript, so they do not register as site traffic.

If you go through your web host control panel (such as cPanel, Plesk, etc.) you can find functions that track all traffic to your sites. Things like Site Error Logs, or AW Stats or even how much Bandwidth you use can give you important clues to how busy your sites really are. And to many people, it’s a shock to find their rarely-visited little store is actually a hotbed of frantic bot indexing. Sites with almost no human activity at all can be in danger of being shut off by webhosts who warn you about excessive server load or overuse of CPU cycle time. And yet the traffic counter never moves off of “000000”. In these cases, the Throttle message from Amazon acts like a circuit breaker, Allowing your site to reset, and hopefully keep the bots from overwhelming everything.

Statcounter Traffic
AOM Site, April 1-19, 2014 showing traffic with Statcounter
AW Stats Traffic
Same AOM site, same dates, showing Bot traffic from AW Stats not seen by Statcounter

Now that you know what the message means, what steps can you take to minimize or eliminate the chances of your customers ever seeing it? There are several things you can do:

1) Caching – AOM comes with a caching system in place by default. It is strongly recommended you do not turn it off. It works by storing the data from accessed pages on your website server. This data can be accessed by other visitors to your site for a maximum of 24 hours. Then the data is refreshed, and the new page is stored (the old data is eventually erased, depending on your cache settings). This means that instead of having to put a call into Amazon every time someone wants to see a particular page featuring product, the saved data is used instead. This can also speed up your site performance, since the information is closer to hand than having to contact Amazon and wait for a reply. The tradeoff is it does require more disk space. But it does help in dealing with repetitive bot requests.

2) Blocking Bots – While bots may be invisible to most tracking programs, the traffic functions mentioned previously (Error Logs, AW Stats, etc.) can see them. And some can record the Bot’s IP Address. You can use this to block or deny access to this IP. Many website control panels will have methods of denying IP access. You enter the IP, and it puts a line in your site .htaccess file. You can also do this manually by typing the command into your .htaccess file as well. In addition, there are Bad-Bot scripts you can set up in your website that will automatically add any bots that try to poke their nose into areas they should not go (which most human visitors would not do). Look in the Popular Articles section at the top of this page for Build Your Own Bot Trap.

Highlighted cPanel Features
It should be noted that some bots will fake, or spoof an IP address that isn’t their own. Many times they will impersonate Google’s own Googlebot.

3) Shut Poor-performing Sites – If you have a large number of sites set up, it’s possible that only a small handful are really generating any revenue for you. If so, the other sites are probably serving no other purpose than providing fodder for bots. So when Amazon throttles your account, the sites that generate little or no sales are creating damage to your profitable sites. You might want to consider shuttering the poor performers, and decreasing the chance for bots to cause temporary blackouts to your good sites.

You can also search online for alternate methods of blocking bots. Some bots are good and useful if you want your site on search engines. Many are harmful and serve no purpose other than to waste your site resources. If you’re not interested in being featured on Chinese search engines, you may want to block the bad version of the Baidu bot, for instance. There are hundreds of bots, if not thousands, and as old ones become blocked, new ones spring up to take their place. It can be almost a full-time job just trying to control the flood. But with at least some diligence, you can markedly decrease the waste on your sites and make sure the Throttle message is kept to a minimum.

Finally, you can customize the ‘Throttled’ message via the Error tab of your AOM control panel (either Error for API Limit or Error API Limit File). This can allow you to display a more reassuring message to your site visitors.

Latest posts

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.