Installation Testing is aimed at checking the successful installation and configuration, as well as updating or uninstalling the software.

Currently, the most common is installation software using installers (special programs that also require proper testing on their own).

In distributed systems, where an application is deployed on an already running environment, a simple set of instructions can be small. To do this, you often write a Deployment Plan, which includes not only steps to install the application, but also roll-back steps to the previous version, in case of failure. The installation plan itself must also undergo a testing procedure to avoid problems when issued in real operation. This is especially true if the installation is performed on systems where every minute of idle time is a loss of reputation and a large amount of funds, for example: banks, financial companies or even banner networks. Therefore, testing the installation can be called one of the most important tasks to ensure the quality of software.

Such an integrated approach with writing plans, step-by-step testing of the installation and rollback of the installation, can rightfully be called Installation Testing.

Features of testing installers

The installer is an “ordinary” program, the main functions of which are Installing, Updating and Uninstalling the Software.

As the saying goes: “Beauty is only skin-deep”. The installation application is the beauty that creates the first impression of your product. That is why installation testing is one of the most important tasks.

As a regular program, the installer has a number of features, some of which are worth noting:

• Deep interaction with the operating system and dependence on it (file system, registry, services and libraries)
• Compatibility of both native and third-party libraries, components or drivers, with different platforms
• Ease of use: intuitive interface, navigation, messages and tips
• Design and style of the installation application
• Compatibility of user settings and documents in different versions of the application
• And much more

If these features did not charge you for a serious attitude to testing installation programs, then we will provide a small list of risks that will show all the importance of the correct operation of installers:

• risk of losing user data
• risk of operating system failure
• application failure risk
• risk of incorrect application operation

At the same time, some functional requirements are imposed on the installer. Combining them with a list of features, we will get a more complete picture showing the volume of upcoming testing work. And further, based on the list of requirements, you will need to answer the questions: “What to test in the installation programs?”, And only then – “How to test the Installation?”.

In most cases, the installer is an application in the form of a Wizard, which may have specific requirements.

With the modern abundance of personal computers, servers and operating systems, there is a need to install the same software on different platforms. To do this, installers should understand what and where they install, depending on the environment.

What to test in Installation programs?

Let’s consider more in detail “What?” you need to check to assess the correctness of the installer:

• Installation

          1. The availability of sufficient resources for installing the application resources such as: RAM, disk space, etc.

          2. The correctness of the list of files in the installation package:

• when selecting different installation types, or setting parameters, the file list and paths to them may also differ
• the absence of unnecessary files (project files not included in the installation package should not get on the user’s disk)

  3. Application registration in the OS

  4. Registration of extensions for working with files:

• for new extensions
• for existing extensions5. The access rights of the user who puts the application:
• the right to work with the system registry
• permissions to access files and folders, for example, %Windir%\system32

6. The correctness of the installation wizard (Installation Wizard)

   7. Installing multiple applications in a single call

   8. Installing the same application in different working directories of the same workstation

• Update

           1. The correctness of the file list, and the absence of unnecessary files:

• checking the file list for different installation parameters

• no extra files

          2. Backwards compatibility of generated data:

• safety and correct operation of the data created before the update

• the possibility of correct operation of old versions of the application with data created in new versions

          3. Update when the application is running

          4. Aborting update

• Uninstalling

     1.  Correct application removal:

• removing from the system registry installed during the installation of libraries and service records
• remove physical application files
• deleting / restoring previous file associations
• safety of files created during the time of work with the application

     2.   Deleting with the application running

     3.   Deleted with restricted access to the application folder

     4.   Deleting by user without proper rights

How to Test Installations

Having understood “What to test in installation programs?”, we will consider “How to test Installations?”.

• Installation

1. The resources necessary to install the software should be analyzed exactly at the beginning of the installation process. In case of their shortage, the user should receive an appropriate warning.
2. Getting the list of files should be done before, and checking the files themselves – after installation!

• the most correct thing is to try to get a list of files from the installer of the installation package. The phrase: “Take and make a list after installing the software, everything is right” – this is a provocation and it is better not to succumb to it.
• if the program contains files signed by a certificate from the MS, it is possible that temporary files that are no longer needed for running the application (*.tmp, etc.) may come across. Note that the same installation package can install different sets of files for different operating systems. Therefore, you should test for each OS separately.

3. Registration of working libraries and service records is important for almost any OS. For example, in Windows, you will need to check the system registry for correct recording of new data and registration of  new applications.

4.After installing the application, it is necessary to check the correctness of registration of extensions for working files of the installed program in the OS. Double-click on the document, as a result it should be opened by your application. If you get a dialog for selecting a program to open files of this type, or it will be opened by another application, then there will be an error registering the extension in the OS.

 5.  In a good way, the installation program should check the user account at startup and immediately report any problems with the rights correctly. But it is not excluded that additional restrictions will be imposed on any service folders. You can try to simulate situations when one of the folders is closed for writing, for example “\Program Files”, “\Windows”, “%Windir%\system32”, and also check how the application will behave if it is impossible to write any of the files on the desired path. The option is not excluded that without some files the health of the entire application will not be broken. In this case, it is sufficient to indicate a problem with copying the file (s) to the log, and NOT output an error message.

6. It is necessary to conduct a full Installation Wizard Testing.

7. Quite often there are situations when the application itself installs another product. This can be either a separate third-party application or a demonstration of your own product.  During the installation process, this will usually not be another set of steps of the installation wizard, but just a small message that such a product is being put. When testing, it will be necessary to pay special attention to the fact that we have a consistent installation of several products. If the first one requires a reboot after its installation, then the second one may not be installed correctly, especially if both drivers are placed in the depths of the system. Interest is represented by errors that occur at the junction of the installation of two applications.

8. There are applications that can be installed in different working directories of the same workstation, and at the same time they will work independently of each other, without creating any conflict situations. But it is not always the case. There may be conflicts with access to shared resources on disk, in memory and/or in the system. All this should be tested in the most thorough manner.

• Update

1. By analogy with paragraph 2 of the chapter “Installation”.
2. Backward compatibility checks include the following steps:

• after installing the update, all previously created objects, such as documents, forms, saving (if this is a game) should open and work without errors. This behavior is called backward compatibility. Custom settings should remain the same, of course, if updates did not affect their change.
• created in the new version of the same type of documents should open correctly in older versions, of course if the purpose of the update was not to change the format and structure of the files. If the new format was introduced, then the new version should be able to save the document in the old format.

3. If the updated application is launched, the user should be warned that you can not upgrade, when running application.

4. Note that the update process can be stopped at any time, while the original application must remain unchanged and in working order. Suppose you have installed version 1 of the application, you are trying to update it to version 2, but during the installation, changed your mind and interrupted the installation. In this case, the installer should return all the changes already made, clean the temporary files used for the update and complete its work. In this case, the version 1 application remains operational.

• Uninstalling

1. A good practice is to delete created during installation (registry entry in the registry, the library in the system catalogs% Windir%\system32, files, etc.). Conditionally, the uninstallation testing can be divided into several parts:

• check that the service records and links to the libraries installed during the installation have been deleted from the system registry.
• check physical removal of application files.
• verifying that after the uninstallation of the application, the file extensions registered during installation have been removed, and previously existing ones (registered before the installation) have been restored.
• verification of the safety of data created during the work with the application. It is likely that they lie somewhere in the depths of the catalog of the program itself. This can be service scripts, save from games or other user-created data, the removal of which will cause damage to the user. Just assume that when you uninstall MS Office, all your documents will be deleted with it. Therefore, such data can not be deleted without user confirmation.

1. If the application to be deleted is running, the user should receive a warning that the removal is not possible while the application is running.
2. It’s worth checking the installer’s behavior if the program’s directory is closed for removal by access rights. In this case, the deletion process should not be performed, and the user should receive a corresponding message.
3. If the user is not authorized to delete the application, then he must receive a corresponding message and the process of removal must be interrupted.

Installation Wizard Testing

Smart people wrote: “Wizards are evil.” You can agree or not, but you still have to test them. We offer the following test plan for the installation wizard:

1. Identify all the paths from beginning to end, and then prioritize for each one. This will help us avoid unnecessary costs and efforts in the passage of low-priority paths.
2. Forget the GUI. Try to describe the test cases without reference to the interface elements. For example, a GUI checkbox / radiobutton or a two-item menu is just a choice between true and false, what’s important is what it ultimately affects.
3. If the results of passing the wizard, you get either a file (a file describing the properties in the form of a list: property = value), which is then passed on to the export procedure. In this case, you can split the checks into two stages – the first one, create (generate) such files and check that the export is working correctly. The second is to check that the correct files are obtained through the GUI.
4. Do not forget to do such routine testing of wizards, like going back and forth on pages:

• without changing anything, all the answers must be preserved;
• changing something on the previous page, the next one should have an adequate change or reset the answers.

1. Make sure that the wizard adequately responds to the wrong answers and does not allow you to move on.
2. The Cancel (Close) button should always work on all pages of the wizard.
3. Create a template result for each of the possible ways of the installation wizard (ideally, make several for different input data). Then, if possible, automatically or manually compare the result with the template result.
4. Select those options that do not affect any others, and which others do not influence. The work of these options can be tested in isolation from others.

Cross-platform testing of installers

A separate point we want to highlight cross-platform testing of installers, which must necessarily be carried out for all three functions – installation, update and removal:

1. The correctness of the installer with different versions of OS, ServicePack and installed updates
2. Checking files, drivers and libraries when installing under different OS
3. Checking access rights to files, folders, and system records for different OSes
4. Checking Permissions

To simplify the testing procedure, it is recommended to create a table where the required configurations will be used by the columns, and the lines – test cases or test functions. In the testing process at the intersection of columns and rows, fill the result, which can visually show the progress of testing and compliance with “cross-platform” requirements. Now let’s pass directly to the description of testing:

1. If the application should work in several OSes, then the installer should check the required set of functions in the course of its work. It is possible that somewhere you need to install some update package or just need to work with another library.
2. Case from practice: “The application should be closed for installation on Windows Vista. We run it, the program thinks and gives a message that this version will not work under this OS. Everything is correct, we press OK. Then the computer thinks for a couple of seconds and reports the following : “The application was not properly completed. Let’s try to run it in compatibility mode?”. We confirm and start, our application starts again automatically , but this time it’s installed to cheers. “We do not know what and how Vista does there, but the protection code had to be altered.”
3. Checking the list of installed files is done by analogy with point 2 of the chapter “Installation”, as for the installation of drivers and libraries, then it should be noted the special importance of this check. Not all drivers and libraries work equally well on different platforms.
4. Case from practice: “For some drivers, there is a dependence on the file system on which the OS runs, more specifically for some types of errors.There was a case when everything worked fine on NTFS, and on FAT32 the OS could not get up after installing the software.  The reason was in the wrong entry when installing the driver in the registry. “
5. It is necessary to check that the installer has the rights to access files, folders and system records. This is especially important when installing under the UNIX family OS, with their strict limitations in accessing resources for different categories of users. Also see item 5 of the chapter “Installation”, and also item 4 of the chapter “Uninstallation”
6. When / after installation on unix systems, the files must have the appropriate Permissions. Those. if the file is intended for startup, then it must be run, if it is a configuration file, for example, there should be permissions for modification, etc.

We hope that the information above will help you during testing the installation applications.