Owners of sites who have a problem with high load often contact us at "Revision". The situation when the owner receives a "letter of happiness" from the hoster, which is a harbinger of blocking the site, is by no means rare, absolutely any site owner or publisher can face it, so we decided to consider in detail the causes and solutions to this problem.
Usually publishers learn about excess load from their hosters, who strictly regulate and control the process of CPU time consumption and, at the tariff plan level, set the permissible load that an account can create (usually it is measured in% of a certain allowed value or in CP / CPU minutes).
The hoster tries to distribute the processor resources evenly among all the clients of the server. If someone's hosting account "eats" 90% of the processor resources, then the rest will get only 10%. Therefore, in such cases, the owner of an account that exceeds the limits will receive a warning. And in case of systematic violations, the account is blocked so as not to interfere with the work of other sites located on the same server. And this, by no means, is not an attempt to "dilute" the client to a more expensive tariff, as some webmasters think, since it is not the hoster's fault that the site has required more resources for some time.
Let's try to figure out what the increase in the load on hosting can be connected with and how this problem can be solved.
It is important to note that the high load can be due to both external and internal in relation to the site and hosting factors.
External factors
External factors that create a high load are everything that does not depend on the hosting setup, the work of scripts and the site administration process. This is the result of external requests to the site by various services, bots or other sites. There are many factors.
- Site scan for vulnerabilities, search for "sensitive files", search for the admin panel.
Any site whose pages are indexed in a search engine can become a “target” for hackers and bots, every day someone will scan it, look for “holes”, try to hack it. It is impossible to stop this process, but you can counteract it.
Site requests, especially if they are performed intensively and using the POST method, consume a lot of processor resources. Therefore, the process of scanning a site with an external scanner is expressed in an increase in the load. If, as a result of scanning, an attacker discovers a vulnerability or a variant of hacking the site, then most likely he will upload malicious code to the site or commit some destructive actions. If no security problems are detected as a result of scanning, the site will continue to operate normally, and the load will return to normal. Until next scan ... - Password guessing from the site admin panel (brute force attack).
One of the popular attacks, the purpose of which is to gain administrative access by brute-force attacks on popular combinations of administrator usernames / passwords, is a brute-force attack. The hacker bot uses a special dictionary with TOP1000 popular combinations (admin / admin, admin / 123456, ...) and tries to enter the admin panel of the site with them. The brute-force process itself increases the load, since constant requests are made to the page of the administrative panel, and the requests are executed by the resource-intensive POST method. - Bulk user registration or bulk sending of spam through insecure feedback forms.
Often sites use feedback forms or user registration forms with weak anti-bot protection mechanisms. It is good if the form has at least some kind of “captcha” from the “prove that you are not a bot” series. If the site is included in the spammers' database, but there is no "captcha" or other protection mechanism against "http flood", then mass registration of users with spam profiles begins, sending mail through a form, etc. All this creates a load on the hosting, and in addition, it can provoke spam mailing, for which the hosting company turns off the mail service or blocks the site completely.
It should be noted that at the moment all simple defense mechanisms are easily bypassed by modern bots, so you need to immediately install something serious, for example, Google Recaptcha2.
- Site indexing by search bots.
Sometimes, with a sufficiently large search index (when a large number of pages fall into the search base of Yandex and Google), the reindexing process can take a long time and create a large load on the server. If your site has only a dozen pages, you may also face a similar problem, for example, if the site was hacked and hosted a doorway with 50,000 pages that hit the search results. Or the search index could be spammed by a competitor who took advantage of errors in the scripts on your site. There are a lot of options here.
- Content grabbing and scrubbing.
Owners of unique content should be concerned about the problem of downloading content from the site (scrubbing and grabbing). This can be done by special bots that bypass the pages of the site and copy the texts and pictures posted on the site in order to create clones. If the process of crawling your site is regular, and the site has a lot of pages, this can create an impressive load on the hosting.
- Data import (feeds, unloading of commodity items).
Often e-commerce resources use a data exchange mechanism with external services. For example, a list of commodity items can be downloaded from online stores, data from 1C can be loaded into them, news sites can regularly export news feeds, etc. If the content is not static, then each such request will create a high load on the server.
- Using images or links to your site.
One of the non-obvious moments that create a load can be the placement of a link to the site or the use of an image from the site on a more visited resource. One of the sources of the problem is the so-called “habraeffect”, when the site cannot cope with the flow of visitors from a more popular resource. The second option is when someone (or you yourself) posted a picture from your site on the blog you visit (for example, in the comments), and it is loaded by each visitor and creates a load on your hosting. Especially this can create serious problems if the picture is generated by scripts (for example, scaled using timthumb / phpthumb scripts). - Attacks on other sites (for example, a vulnerability in xmlrpc.php).
Often sites containing vulnerabilities are used by hackers to carry out attacks on other resources. Sometimes an attacker does not even need to hack the site for this. For example, owners of not the most recent versions of Wordpress may encounter this problem (attack through the xmlrpc.php file). In this case, your site will act as an intermediate link, and the work of site scripts will create a large load on the server.
- DDOS attack
If there is a DDOS attack on the site, then it will not be possible to cope with it without connecting special technical means proxying traffic (hosting service or DDOS protection service). It's hard not to notice DDOS. Due to the fact that a huge load will be created on the server, the hoster can do different things: offer a DDOS protection service, move the site to another server, or completely block (disable) the site. Therefore, to protect against DDOS, it is advisable to have a prepared solution in advance, so that when a problem arises, it can be quickly resolved. - Traffic growth
If the traffic is organic, then this is the most positive reason for the increase in load. So it's time to scale the site and think about optimizing scripts for higher traffic.
To find the reason for the load generated by external factors, you need to analyze the web server logs. To do this, you can use special applications or a combination of commands in the SSH console.
In the analysis results, you should look at TOP 20 requests by the POST method, TOP 20 requests by the GET / HEAD method, TOP 20 IP addresses by the number of hits, TOP 20 referring pages by the number of hits. All this will allow identifying the source and type of traffic, as well as the entry points to the site or scripts that are called most often. Most likely, they will be the cause of the high load.
To reduce the load in case of external attacks or intensive requests, in most cases, it is enough to enable protection against http flooding (for example, the classic “cookies on the client + redirect with verification”) or connect the site to traffic proxying services that will block dangerous or especially active requests, but good and legitimate - skip. In addition, static content (images, scripts and styles) will be served not from your site, but from CDN servers, which will also significantly reduce the load.
You can try connecting a caching plugin in a CMS or a caching service on a hosting, but in the case of external factors affecting the load, this may not help.
Internal factors
Internal factors include everything that affects the performance of the site at the level of scripts and settings. That is, something that can be controlled by the webmaster (site owner).
- Unoptimized scripts and sprawling database.
Due to a poorly designed architecture of a web application or inefficient implementation of scripts by inexperienced developers, it is possible that simply opening the start page or displaying search results on the site can seriously load the server. And the growth in the volume of the database (for example, an increase in the number of commodity items) with each update of the site will slow it down more and more, increasing the load on the hosting. Individual pages of a site with a large number of information blocks can send dozens of queries to the database, repeatedly perform the same operations with files, and sometimes even block the operation of other elements of the site. We often face a similar problem with online stores running on the old version of Joomla with the Virtuemart plugin. In some cases, more than 100 database queries are made when opening a catalog page. - Infecting the site with viruses
Hacking and infecting a website with malicious scripts is a fairly common cause of increased load. It increases due to viral activity arising from the introduction of malicious fragments into legitimate site scripts, the launch and operation of resident processes, as well as the connection of scripts to external resources at the moment of opening any page of the site. - External legitimate connections to third-party resources
Few people take into account the load created by connections to external sources of information (widgets, weather and exchange rate informers, news feeds, etc.). Often, data that is downloaded from other sites is not cached locally, and when the page is opened, each time the page is opened, content is connected and loaded from another server. If for some reason the external source stops responding quickly, it will affect the load and load speed of the main site. - Errors in scripting
When running scripts, errors may occur that are not displayed to visitors, but are written to the web server log or php log. If the site is visited or there are a lot of errors, this can also increase the load on the hosting. Most often, errors start to be generated at the moment of switching the site to a more recent version of PHP, with which the scripts are not compatible. Or when not all site components are updated, and conflicts arise between the new CMS core and old versions of plugins.
To analyze the problem of high load caused by internal factors, you need to check the site for malicious code (for example, check the site for free), and, if no malicious code is found, then profiling the scripts using the xhprof or xdebug modules.
To solve the problem of high load caused by the activity of malicious code, it is necessary to disinfect the site and install protection against repeated hacking. It will be better if the site curing and protection is done rather than by web developers.
If the cause of problems is in the architecture of the site or errors, then optimization of the site by an experienced web developer will help. One of the fallback solutions for the second case is the installation of a caching plugin, which in some cases can reduce the consumption of processor resources (hosting load) and speed up the site.
Duration
In conclusion, I would like to consider one more property of the processor load - its duration. It can be either a short-term burst on the chart during the day, or a constant growth over a long time.
If you see a one-time spike on the CPU consumption graph, don't worry. It is practically invisible, does not affect the availability of the site and does not interfere with the hosting “neighbors”. It is worse if the graph creeps up for a long time or shows the maximum (or exceeding the limit) processor load for several days. What to do in this case? It is necessary to audit the sites on the account as described above, checking both external and internal factors causing problems.
Our cybersecurity services
Promotion "2 for the price of 1"
The promotion is valid until the end of the month.
When you connect the service "Site under supervision" for one site, the second on the same account is connected free of charge. Subsequent sites on the account - 1500 rubles per month for each site.
A situation that can happen to every owner of a developing site with an increasing load -receiving a warning about exceeding the resource limit.
If your hosting provider tells you about the load on the processor, but you are sure that you should have enough resources, try to understand what could be the reason for this load. Knowing the reason, you can quickly fix the problem.
CPU load when running PHP or Perl scripts
In such cases, the hoster cannot always clearly say which script is causing the problem, and you will need to determine this yourself. If you are using a modular CMS, for example Joomla, Wordpress or Drupal, then the reason may be incorrect work of a separate module.
Almost always, the problematic link in the system of your site can be determined by comparing the time of the problem with the logs of requests to your site at that moment in time. You can usually get access to the web server logs in the hosting control panel, even without contacting the provider.
Load on the CPU and / or disk system from the Apache web server
Excessive load can be created by the Apache web server when the site is rendering a large amount of graphics and other statics. Apache's architecture is designed in such a way that only one request is processed by one thread or process. Accordingly, if your site has a lot of graphic information, then an unreasonably large amount of processor and RAM resources will be spent on its return. The load on the server disk will also be significant.
In this case, you can use the Nginx web server as the front-end to Apache. Nginx, due to its asynchronous architecture, allows you to handle thousands of connections within a single process and provides static content much easier and faster. The only problem is that on many hosting services only Apache is used as a web server, and the Nginx + Apache bundle is used quite rarely.... Nevertheless, when switching to a VPS, you can configure this bundle without problems. Usually, transferring sites to VPS from hosting and setting up the necessary software can be performed by your hosting provider even at no additional charge.
Excessive site requests
A large number of similar requests to a site from one IP or multiple IP addresses is the so-called HTTP flood, one of the types of DDoS attacks. Blocking problematic IPs in the .htaccess file using the "deny from" directive can help.
If the hosting uses only Apache, and the hoster is unable to reflect problem requests by means of the same Nginx, and the attack is intense, the hoster can block your account on the server and ask you to transfer the sites to a VPS or a dedicated server.
If an attack occurs for the first time, then it is logical to contact the hosting provider for help in repelling it, or at least to demand that the account be unlocked and the possibility of continuing to work with the hosting immediately after the attack is terminated. If such a problem arises systematically, then switching to a VPS or server and its competent configuration for fault-tolerant operation of the site (for example, installing scripts for autoblocking IP addresses of bot machines based on the analysis of logs, installing Nginx as a front-end to Apache and filtering problem requests at his level) really makes sense.
Significant increase in project attendance
With the development of the site, the growth of its traffic is a natural phenomenon, and sooner or later the site can really become cramped on the current tariff plan. This is normal for developing projects. In most cases, on ordinary shared hosting for dynamic sites, the traffic threshold is 2000-4000 unique visitors per day.
If the traffic to your site is close to these figures, then the transition to orwill be the right decision, which will positively affect its further development.
The load on the CPU and disk system from the MySQL side
The normal time to execute a query against a MySQL database is considered to be several tens of milliseconds. Queries that take longer (especially more than 0.5 seconds) often place excessive load on both the server's disk system and its processor. If the hoster warns you about a similar problem, ask him for the logs of slow queries and optimize the database structure, as well as clear the database of irrelevant information.
Intensive use of the hosting service for instant mass mailings to hundreds or thousands of recipients can cause a significant load on the server. For this reason, most providers set certain restrictions on sending mail - usually 25-50 letters per hour, or about 500-1000 letters per day. This limitation is aimed both at combating spam mailings and at reducing the load on the server's mail subsystem. For normal work with mail on the site, such restrictions are usually sufficient, and for mass mailings it is better to use mail hosting, instant mailing services, or adapt a VPS for this purpose.
You need to understand that a hoster can face dozens and hundreds of problems similar to yours every day, which means that it can help you solve them. The hoster is your ally, not your enemy, in the fight against such problems.
It is more profitable for a hosting provider to help solve a problem and retain a client than to refuse to provide services, while not only losing profit, but also damaging its reputation.
If the provider offers a transition to a higher tariff or a higher class service, specify what exactly this will give you. It is worth listening to the arguments if they are really weighty.
Choosing a hosting, you probably understand what the load will be. Today, there are only two options that are resistant to heavy loads, as well as meeting the performance requirements: a cloud or a cluster solution. Both one and the other are quite worthy, so the price is more likely to play a role here.
Choosing the best hosting
What is the difference between a cluster and a cloud?
Clustered solutions are created on the basis of two or more dedicated servers, usually physical. Moreover, server equipment is used constantly, and when the load rises, they rent an additional number of servers or buy in addition.
Cluster solutions are usually paid on a monthly basis.
Cloud solutions are also built on the basis of a number of virtual servers. How many servers are involved at a given time depends on the load. At high - a large number, and at low - servers that are not involved are simply shut down.
Cloud hosting is often paid on an hourly basis. An hour of operation of each individual server in the cloud is necessarily charged. The amount of payment depends entirely on the load. If it is high, then the amount to be paid is higher, and if there is no load, then the payment is minimal.
Comparing the cash costs per hour of running similar servers in the cloud and in a cluster, you will see that clusters are somewhat cheaper. At the same time, after a good analysis of the overall percentage of effective use of paid resources, it becomes quite obvious - in the "clouds" it is higher.
With such a hosting, you pay exclusively for actual consumption, while in a cluster, billing also applies to idle hardware.
Cloudy best hosting in terms of scaling speed outstrips cluster. Moreover, it lends itself to automation. The construction of a cluster is more time-consuming, there are much fewer automatic operations.
Let's dwell on a few more features of each of the described hosting solutions.
Clusters
Cluster hosting should be chosen if the load is high all the time or is steadily increasing.
In such circumstances, all cluster resources are constantly involved, so the equipment is not idle. Thus, there is time for scaling (increasing the number of servers). This happens gradually - as the load increases.
More often than others, the cluster is chosen by web services, online stores, business applications, social networks.
"Clouds"
Cloud hosting in Europe is justified for sites of events, exhibitions, special promotions and others, which have low traffic most of the time, but at some time can dramatically increase significantly.
In other words, we are talking about sites where jumps in load are closely related to marketing activity or it rises like an avalanche, and there is no time to deploy a cluster. In this case, the higher price per hour of work compensates for the fact that the equipment is not idle.
Thus, one of the advantages of cloud hosting is that the "cloud" reduces the time to deploy, as well as to market projects and applications.
Output
Of course, there will be opponents and supporters of both solutions. Nevertheless, since they exist, they serve a large number of projects, then they are a reasonable way to the Web. Which one to go is up to you.
As mentioned above, cloud hosting in Ukraine is suitable for periodically high loads and an unpredictable increase in the number of visitors at the same time.
Cluster solutions are used with constant high loads or predicted systematic growth in traffic.
Focusing on such very simple factors, the choice will be much faster and easier.
