- Smoke test on Jargon File
Wikimedia Foundation. 2010 .
See what "Smoke test" is in other dictionaries:
smoke test- noun A method of testing for leaks in drain pipes or chimneys by introducing dense smoke, often by using a smoke bomb Main Entry: smoke … Useful english dictionary
smoke test- Test made to determine the completeness of combustion … Dictionary of automotive terms
smoke test- 1. noun a) A test for leaks involving blowing smoke into a tube or pipe. b) A preliminary test on a newly constructed piece of electronic equipment, consisting simply of the application of electrical power, to make sure that no egregious wiring… … Wiktionary
smoke testing- is a term used in plumbing, woodwind repair, electronics, computer software development, and the entertainment industry. It refers to the first test made after repairs or first assembly to provide some assurance that the system under test will… … Wikipedia
smoke testing- bzw. Rauchtest ist ein Begriff aus dem Englischen, gebräuchlich im handwerklichen Bereich (z. B. in der Klempnerei, Elektronik oder beim Bau von Holzblasinstrumenten) wie auch in der Softwareentwicklung. Es bezeichnet den ersten… … Deutsch Wikipedia
Smoke- is the collection of airborne solid and liquid particulates and gases [SFPE Handbook of Fire Protection Engineering] emitted when a material undergoes… ... Wikipedia
test suite- In software development, a test suite, less commonly known as a validation suite , is a collection of test cases that are intended to be used to test a software program to show that it has some specified set of behaviors. A test suite often… … Wikipedia
smoke bomb- A smoke bomb is a firework designed to produce smoke upon ignition. While there are smoke generating devices that are dropped from airplanes, the term smoke bomb is used to describe the three types of devices:# A smoke ball is a hollow, cherry… … Wikipedia
Smoke and sanitary testing begins immediately after the release of the next version of the project. For many young testers, this process seems like absolute chaos. Did you recognize yourself? Then this article is for you. Now we will look at the definitions of smoke and health testing, and also show the difference between them with easy-to-understand examples.
Smoke testing:
Smoke testing is carried out in order to make sure that the resulting build is suitable for testing. It is also called a zero-day check.
It is this type of testing that will not let you waste time. It is logical that testing the entire application does not make sense if there are problems with key features and critical bugs have not been fixed.
Sanitary testing:
Sanitary testing is carried out at the release stage to check the main functionality of the application. They usually don't go further. Such testing is sometimes referred to as a shortened version of regression testing.
When a release is under pressure, doing thorough regression testing is next to impossible. In this case, sanity testing does a great job, which checks the operation of the main functions of the application.
An example to better understand the difference between smoke and sanitation testing:
There is a project for which an initial release is planned. The development team releases a build for testing, the test team starts work. The very first test is the suitability test. You need to find out if you can work with this version or not. This is smoke testing. If the team gives the go-ahead for further work with the build, it goes to deeper stages of testing. Imagine that the build has three modules: “Login”, “Admin” and “Employee”. The test team checks the performance of only the main functions of each of the modules, without delving into the particulars. This will be health testing.
A few more differences between smoke and sanitation testing:
- Smoke testing is done by both developers and testers;
- Sanitary testing is carried out only by testers.
- Smoke testing covers all the main functionality of the application from start to finish;
- Sanitary testing only tests a specific component of an application.
- Smoke testing goes through both stable and non-stable builds;
- A relatively stable version of the build is undergoing sanitation testing.
Kirill Flyagin, game designer, QA Lead
Let's draw a summer analogy with these types of testing. Let's say you want to buy a watermelon. Smoke testing is when you check it visually, look at the strips, squeeze, knock, evaluate. There are masters who manage to buy a really tasty berry in this way. In health testing, you cut out a pyramid at the top and check its color (as one of the components), while not knowing at all whether the whole watermelon is like that. But for the cut part you are completely sure.
After making the necessary changes, such as fixing a bug/defect, software must be retested to confirm that the problem has indeed been resolved. The following are the types of testing that should be performed after installing the software to confirm that the application is working or that the defect has been corrected:
- Smoke testing(Smoke testing)
- Regression Testing(Regression Testing)
- Build testing(Build Verification Test)
- Sanitary testing or consistency/health check(Sanity Testing)
concept smoke testing came from engineering. When commissioning new equipment ("iron"), it was considered that the test was successful if no smoke came out of the installation. In the field of software testing, it is aimed at a superficial check of all application modules for operability and the presence of quickly found critical and blocking defects. According to the results of smoke testing, a conclusion is made whether it is accepted or not. installed version software for testing, operation or delivery to the customer. To facilitate work, save time and human resources, it is recommended to implement automation of test scenarios for smoke testing.
Regression Testing is a type of testing aimed at verifying changes made to an application or environment(fixing a defect, merging code, migrating to another operating system, database, web server, or application server) to confirm that pre-existing functionality works as before (see also Sanitary Testing or Consistency/Health Checking). Regression can be functional, so non-functional tests.
As a rule, for regression testing, test cases are used, written in early stages development and testing. This ensures that changes in new version the apps didn't hurt existing functionality. It is recommended to automate regression tests to speed up the subsequent testing process and detect defects in the early stages of software development.
The term “Regression testing” itself, depending on the context of use, can have a different meaning. Sam Kaner, for example, described 3 main types regression testing:
- Bug regression is an attempt to prove that a fixed bug is not actually fixed.
- Old bugs regression- an attempt to prove that a recent change in code or data broke the correction of old errors, i.e. old bugs started to play again.
- Regression side effect(Side effect regression)- an attempt to prove that a recent code or data change broke other parts of the application being developed.
Sanity Testing or Consistency Testing (Sanity Testing) – this is a narrow test, sufficient to prove that a particular function works according to the requirements stated in the specification. It is a subset of regression testing. Used to determine the health of a particular part of the application after changes have been made to it or the environment. Usually done manually.
The difference between sanitary testing and smoke testing. Some sources mistakenly believe that sanitary and smoke testing are the same thing. We believe that these types of testing have "vectors of movement", directions in different sides. Unlike smoke testing, sanity testing is directed deep into the tested function, while smoke testing is directed in breadth, in order to cover as much functionality as possible with tests in the shortest possible time.
Build testing(Build Verification Test) is a test aimed at determining whether the released version complies with the quality criteria for starting testing. According to its goals, it is an analogue of Smoke Testing, aimed at accepting a new version for further testing or operation. It can penetrate further into the depths, depending on the quality requirements of the released version.
Installation Testing - is aimed at verifying the successful installation and configuration, as well as updating or uninstalling the software. At the moment, the most common installation of software using installers(special programs that themselves also require proper testing). In real conditions, there may not be installers. In this case, you will have to install the software yourself, using the documentation in the form of instructions or readme files, step by step describing all the necessary actions and checks. In distributed systems where the application is deployed on an already running environment, a simple set of instructions may not be enough. To do this, often, a deployment plan is written (Deployment Plan), which includes not only the steps for installing the application, but also the rollback steps (roll-back) to the previous version, in case of failure. The installation plan itself must also go through a testing procedure to avoid problems when released into actual operation. This is especially true if installation is performed on systems where every minute of downtime is a loss of reputation and a large amount of funds, for example: banks, financial companies or even banner networks. Therefore, installation testing can be called one of the most important tasks for ensuring software quality.
Exactly like this A complex approach with writing plans, step-by-step verification of the installation and rollback of the installation, it can rightly be called installation testing or Installation Testing.
If you want to create a simple computer program, which consists of a single file, you just need to collect and link all the code you wrote into this file. Actually regular project, which is handled by the development team, there are hundreds, even thousands of files. This "contributes" to the fact that the process of creating an executable program becomes more complex and time consuming: you must "assemble" the program from various components.
The practice used, for example, in Microsoft and some other software development companies, is to build the program daily, which is supplemented by smoke testing. On a daily basis, after each file is compiled (built, built), linked (linked), and combined into an executable program, the program itself is subjected to a fairly simple set of tests, the purpose of which is to see if the program "smokes" during work. These tests are called smoke tests (from the English smoke - smoke). Most often, this process is quite well automated (or should be).
BENEFITS. This simple process provides several significant benefits.
Risk minimization during integration
One of the most significant risks faced by a development team is that the developers themselves work on the code separately, independently of each other, as a result of which a complex program does not work as expected when building the developed code. Depending on when the incompatibility was discovered in the project, the debugging of the program may take longer than with earlier integration, especially if the program interface changes or after the implementation of major changes to the main parts of the program.
Daily assembly and running of smoke tests makes it possible to reduce the risk of integration errors, respond to them in time and prevent their accumulation.
Reducing the risk of poor software product quality
The low quality of the product directly depends on the failures and problems during integration. Running a minimal set of smoke tests daily keeps errors and problems from taking over the project. If you bring the project to a stable state once, it will remain stable forever. This way you will never allow the quality to decrease to the level at which errors occur.
Help in diagnosing errors
If one day the product didn't build (built with errors), it's much easier to find the cause of the problem by building daily and running a set of smoke tests. A working product yesterday and not working today is a clear hint that something went wrong between the two builds.
Morale Improvement
If the product works and acquires more and more new qualities and functions every day, the morale of the developers, in theory, should grow and it does not matter at all what exactly this product should do. It is always a pleasure for a developer to watch his working “brainchild”, even if the product displays a rectangle on the screen :)
Using daily builds and smoke tests
Here are some details of this principle.
Daily app build
A fundamental part of the daily build is the build of the part that was made last. Jim McCarthy in Dynamics of Software Development (Microsoft Press, 1995) called the daily build of the project his heartbeat. If there is no heartbeat, there is no project, it is dead. Less figuratively, the daily build has been described by Michael Cusumano and Richard W. Selby as the project's clock pulse (Microsoft Secrets, The Free Press, 1995). Each developer writes code in his own way and he, the code, can go beyond the framework generally accepted on the project - this is normal, but with each exposure to a synchronizing pulse, the code returns to the standard. By insisting on developing with the sync pulse constantly, you prevent the project from getting completely out of sync.
In some companies, it is customary to collect the project not every day, but once a week. This system is erroneous, because in the event of a “breakdown” in the project this week, it may take another couple of weeks before the next successful build. In this case, the company loses all the benefits of the daily project build system.
Check for failed build
In the case of a daily build of the project, it is assumed that the project should work. However, if the project is not working, then fixing it becomes a task with priority 1.
Each project has its own standard and a sign of what is called "build failure". This standard should specify a level of quality that is sufficient to keep track of minor defects and not overlook defects that "block" the project.
A good build is one where at least:
- all files, libraries and other components are successfully compiled;
- links to all files, libraries and other components are valid;
- does not contain any stable systemic, excluding the possibility correct operation application program blocking errors;
- all smoke tests pass.
Daily smoke tests
Smoke tests must be performed on the entire project from start to finish. They do not have to be exhaustive and comprehensive, but should contain a test of all major functions. Smoke testing should be deep enough that, if it passes successfully, you can call the project stable and call it such that it can be subjected to deeper testing.
The point of daily assembly is lost without smoke testing. This process guards the quality of the product and does not allow any integration problems. Without this, the daily build process is a waste of time, the purpose of which is to check the compilation.
Smoke testing should evolve with the project. In the beginning, the smoke tests will check for something as simple as whether the project can produce a "Hello, World!" message. As the system evolves, smoke tests become more in-depth. The time spent on the first smoke tests is calculated in a few seconds, however, as the system grows, the amount of time required for smoke testing also increases. At the end of a project, smoke testing can last for hours.
Build group definition
On most projects, there is a designated person responsible for checking the daily build of the system and performing smoke tests. This work is part of the duties of this employee, but on big projects there may be more such employees and such work is their main responsibility. For example, there were four people in the Windows NT 3.0 project's build team (Pascal Zachary, show stopper!, The Free Press, 1994).
Only add a revision to a build if it makes sense.
Typically, developers individually write code slowly enough that meaningful changes can be added to the system on a daily basis. They have to work on a large part of the code and integrate it into the system every few days.
|
Enter a system of penalties for disrupting the release of the next assembly (release of a non-working assembly). |
Most projects have a system of penalties for failing to release the next build. At the very beginning of the project, it is worth making it clear that the preservation of the working draft is the task of the highest priority. Breaking the release of the next build may be the exception, but by no means the rule. Insist that developers leave everything until the system works again. In the event of frequent build failure (release of a non-working build), it is quite difficult to get the project back on track.
Minor fines emphasize a high degree the need to monitor the build quality of the system. On some projects, developers who cause the assembly to crash are given lollipops for releasing a non-working assembly. A corresponding sign hangs on the door of the office of such a developer until he fixes the assembly (provided that the developers have separate offices :)). On other projects, guilty developers are required to wear artificial goat horns or contribute a certain amount to a "morale fund" (examples taken from the history of real companies).
But on some projects, more serious penalties are introduced. For example, Microsoft developers on high-priority projects (Windows NT, Windows 95, Excel) wore pagers and were required to report to work if an inspection was detected. Even if a breakdown or error was discovered at 3 am.
Build the system and "smoke" it even under pressure
When pressure on a project's release schedule intensifies, the job of checking a system build every day can seem like a waste of time. However, it is not. V stressful situations Developers often make mistakes. They feel such a pressure to release implementations that they simply don’t have under normal conditions. They check their code with unit tests much less carefully than usual. In such situations, the code tends to the state of entropy much faster than in less stressful situations.
Who benefits from this process? Some developers protest against daily builds, justifying their protests with the impracticality of this activity and its large time costs. But everything complex systems have recently been subjected to daily assembly and smoke tests. At the time of its release, Microsoft Windows NT 3.0 contained 5.6 million lines in 40,000 files. A full build took 19 hours and ran on multiple computers. Despite this, the developers managed to assemble the system daily. As a professional team, the NT development team owes much of its success to daily builds. Those developers who work on less complex projects and therefore don't take advantage of the daily build process should consider coming up with some reasonable explanations for themselves.
Simple mistakes can be fatal for your site - especially if you are a SaaS (eng. Software as a Service) company, like us. If a user visits your site and is unable to complete a simple task such as registering or resetting their forgotten password, You risk losing this user forever.
We have experienced it in our own skin. Of course, having your own people in the team who test the application and look for bugs is important, but not always appropriate or not thorough enough. In this article, we want to introduce you, the humanists, to the world of smoke testing.
If you still have questions, you can fill in the gaps by visiting
Smoke testing was originally conceived to explain how electrical engineers tested whether their appliance worked - turned it on and if smoke came out ...
Wait, how can this apply to applications?
The importance (and cost-effectiveness) of smoke testing is generally unknown to humanities managers and co-founders. Systematic smoke testing can be seen as an essential part of preventing an increase in the likelihood of a hack. They minimize the chance that your web or phone app will crash - and as we all know, just one failure and you can lose a customer forever.
This is an introductory guide on what it is, how it can be implemented, what resources are used to carry it out, and examples to guide readers.
Smoke tests are designed to test basic functionality and should be an integral part of your testing process. They might include something as simple as "Can I check in?".
Smoke testing helps ensure that none of the major and obvious failures are left to chance. You shouldn't do a deeper test until you've done 100% smoke tests, because they clean up fundamental bugs in the software.
Step 1: Decide what to test
Determine what your application is trying to achieve. What are the most obvious "baby" steps it takes to get into it? What are the minimum vital requirements and in what logical order would you list them?
Create a test suite. A test suite is a grouped collection of test cases (test cases) related in a certain way (for example, by functionality).
Smoke testing will not include variables or “what if?” questions. It assumes only yes/no answers, but before moving on to more detailed testing, all test cases must be passed with a positive result.
Let's take as an example the creation of an interactive forum. For it to work I have to:
- register.
- create Username.
- upload an avatar photo.
- write messages.
- reply to messages.
Step 2: Write the results to a table
The image above is an example from our team. You can find the pattern. This is necessary in order to keep records of what works for us and what does not - the basic organization will save a lot of time later. We divided our scores into Pass, Partial and Fail.
- Passed: everything works perfectly.
- Partial: Initially, you may not understand that some actions can be further subdivided, and therefore one part works and the other does not.
- Failed: Doesn't work.
We have described the exact steps we wanted to reproduce, and then, in the next column, we have added a brief description of what we expect the output to be. Example:
Step 3: Automating Smoke Tests
It is very important not to take it as an axiom that if any action was completed once, it will always have a positive outcome. Smoke tests allow you to constantly check that the main functions have not suffered over time or have not been broken over a long period.
Don't stop smoke testing. Never.
When your set of smoke test cases completes successfully on 100%, consider automating them. The recommended frequency of smoke tests is every day if your company develops every day.
The bare minimum is to run smoke tests before every release and after every patch.
Rule of thumb for smoke testing:
- Minimum time: 30 minutes.
- Maximum time: 60 minutes.
V further perspective automating smoke tests saves time, but when running the same tests over and over again, the human eye can stop noticing details, but the machine does not.