Virtualization seems to be the latest IT buzzword, with promises of cost savings, ROI, and ease of administration. With all of these benefits, it is difficult to imagine that there could be any problems. But, like any new technology, there are security risks inherent in virtualization that need to be addressed.

As noted in InfoQ's earlier article An Introduction to Virtualization, virtualization consists of two main themes: consolidation of many resources to appear as one and making one resource appear to be many. With either of these virtualization strategies, there are security considerations that should be addressed. These include data leakage, authorization and access, and corruption of information assets.

Before getting too far into the article, it is important to point out that its intention is to neither praise nor malign virtualization. There are several benefits to virtualization, but these benefits come at the cost of increased vigilance with respect to security. This is also definitely not to say that virtualizing systems or networks makes them inherently more insecure.

What is security?

When talking about security, three areas are typically addressed:

• Confidentiality
• Integrity
• Accessibility

Confidentiality has been defined by the International Standards Organization as, "ensuring that information is accessible only to those authorized to have access." Confidentiality limits data to specific users based on needs and restricts access to everyone else. Typical confidentiality tools include file access permissions, network access control lists, and firewall rulesets.

According to Wikipedia, Integrity "means ensuring data is "whole" or complete, the condition in which data are identically maintained during any operation (such as transfer, storage or retrieval), the preservation of data for their intended use, or, relative to specified operations, the a priori expectation of data quality." Integrity ensures that users are accessing the information that is expected. Typical tools for system integrity checks are checksums and hashes such as MD5 and version control tools such as CVS or Subversion.

Accessibility to data is simply the ability to access the systems and data desired as needed. This means that outages and deletions can both impact security if they cause data to not be available when it is requested. In order to ensure accessibility, companies typically use high availability configurations, failovers, and hot swappable drives.

It is also important to identify what our systems are being securing from. The following definitions will be helpful in discussing security:

• Vulnerability - a weakness in a system which allows an attacker to violate the integrity of that system. This vulnerability may result from a software bug, a misconfiguration, a weak password.
• Threat - A vulnerability gives rise to a threat, that is the possibility that the vulnerability can and will be exploited. Threats take into account the probability of an attack. For this reason, weak passwords are often exploited first, while a software vulnerability may exist for years before exploitation.
• Exploit - is a piece of software, a chunk of data, or sequence of commands that take advantage of a bug, glitch or vulnerability in order to cause unintended or unanticipated behavior. In short, an exploit is the code or sequence that is used to take advantage of a vulnerability.

When there is a one to one correlation between a piece of hardware and a server, security is relatively easy. System administrators can configure an operating system and install scripts to secure the device on boot up, and DBAs and developers can configure the applications with fixed paths. Network engineers and security teams can configure fixed rulesets on firewalls and routers, restricting access to specific servers, disks, and services. Because systems are tied to one piece of hardware, located in a specific data center on a specific subnet, these static security practices work well. The physical server has to be moved to another subnet, or network cables have to place the server into another network. By using virtualization, however, servers can be brought down, copied, moved, and brought onto another network. The ease with which systems can move around the network, or even off of the network, means that new practices will need to be put into place.

Typical Security Uses of Virtualization

Security was one of the first adopters of virtualization, focusing on availability and confidentiality as drivers. In fact, some old buzzwords were originally the first attempts at virtualization:

VPN - Virtual Private Networks were invented to allow private or confidential data to traverse the dangerous Internet safely. By encrypting the payload prior to transmission, the data was expected to be confidential. While it did not strictly multiplexing the network, it did allow for multiple private channels across one public network.

MPLS/VLAN - Multiprotocol label switching and virtual local area networks both used tagging to allow different networks to traverse over the same hardware. This tagging separates different networks from one another, and hopefully prevents jumping from one virtual network to another.

Firewall partitioning - By separating the firewalls into distinct networks, multiple firewall rulesets can be created, and can allow traffic to be split to a logging device or an intrusion detection system. Additionally, firewall partitioning can allow for multiple rulesets to be applied based on which ports are being used. Firewalls can be expanded beyond a simple gateway device to act more like an advanced traffic cop.

Virtual hosts - Software such as VMware have allowed system administrators to roll out consistent systems simply by copying the image. Enhanced security can be obtained by configuring a system, including the latest patches, and then moving the system out. Additionally, the image can be restored from a known good source on a regular basis, or in the event of a system compromise.

Network Port Obfuscation - By installing port obfuscation on a host, it is possible to prevent hackers from effectively scanning your network. When an attacker performs a port scan against a network, they scan all network ports (both TCP and UDP) from 1-65535. When the attacker finds an open port, they will look for vulnerabilities based on the banner supplied. With port obfuscation software, a host can answer to all 65536 ports, even supplying inaccurate banners. A windows host, for instance may respond like a Unix host, or not even respond at all.

Security Aspects

Accessibility - By having a server farm that answers as one system, users are unaware of outages, as half of the farm can be taken down for maintenance, while the other half stays up. Single or even multiple hardware failures go unnoticed, as the rest of the systems will keep on responding to user requests. Additionally, upgrades and patches can be rolled to systems in a staged manner, to keep information available. While the use of virtualization can improve accessibility, the introduction of another layer can also impact it:

1) The loss of the hypervisor will cause numerous virtual systems to go down at once. While redundant systems should be put on different physical hosts, several different applications may lose a system at once. Bringing these systems back up in order may be difficult, especially given that formal documentation is typically at the bottom of the administrators' to-do list.

 

Confidentiality - This can be by far the most difficult aspect of security to ensure with virtualization. Because virtual systems can be brought up and down, and virtual networks can change based on a typo by an administrator, the entire security landscape can change quickly, before security administrators have time to react. Two examples can show the danger with virtualization:

1) A server has been configured with relatively lax host security, with the expectation that it will be in a highly secured virtual LAN, vlan 4. However, when the system is brought up, the network administrator accidentally configured that system to be on vlan 5, which is accessible to the Internet, instead of vlan 4. Since the host is accessible to its users, this problem isn't noticed. A week later there is a security breach which compromises not only the accessible hosts, but several others on the network. With hard wired networks and dedicated switches, it would be difficult for this scenario to occur, but because virtualization was in place, and a typing error occurred, several hosts were compromised.

2) A virtual host is configured as an application server for a system accessing a human resources database. Security is enabled on the host, and it accesses the database correctly through it's application. Because this server configuration is working so well, a member of another application team asks for and receives a copy of the virtual image. When the copy is made, the scripts and backend configuration that allow the server to connect to the database, including usernames and passwords, are copied with it. Once the developer loads the virtual host, he realizes that he can now access and read the HR application information, including payroll and personally identifiable information.

 

Virtualization is not the culprit in either of these cases, but because it was used without identifying the security ramifications, data was exposed to unintended users.

Integrity - Data integrity issues can occur with the use of virtual systems. One of the biggest threats comes from the ease with which virtual machines can be copied and moved. Two examples will identify issues:

1) Virtual machine A connects to database number 1, then all reads and writes will perform as expected. Once VM A is copied to VM B, both virtual machines are still access database number 1. Depending on the mechanisms being used, this can result in data being overwritten or not committed correctly. As a result, when applications are being written, and the expectation is that they will be placed on virtual machines, care needs to be taken to ensure that data access is correctly managed.

2) A server farm has been built from a copy of a virtual machine, which has been copied to VM's 1-5. An outage has occurred on VM 1, and the night shift system administrator, not realizing that the down server is a virtual machine, makes a correction to bring it back up. Now the 5 systems which are supposed to be exact replicas are out of sync. This can lead to performance issues, and may also lead to compliance issues once the dreaded audit commences. For this reason, when an outage occurs, and a change is made to any of the virtual machines, that change should be made to a master copy. After the emergency is eliminated, the fix should be evaluated, copied to a master, and the master should be copied to the other hosts.

 

Caveats

Whenever a new technology is introduced, someone (usually the developer of the godsend) comes out quickly to announce that it can cure all woes, operate with fewer resources and gets more miles to the gallon. Virtualization has been no difference, and one of the greatest selling points has been the above mentioned security features that virtualization can assist with. These benefits do not come without concerns though:

Attacking the hypervisor - Just as there are multiple layers of attack vectors with physical systems, adding virtual machines adds another layer. In addition to the physical, network, system, application, and perhaps database layers, using virtualization may allow an attack on the virtual machine hypervisor. These attacks may include any of the following:

• Denial of Service - Always a favorite for script kiddies, the ability to shut down a hypervisor, and by extension several machines, saves a lot of time, and allows the attacker to move on. Compared to a denial of service attack against one physical system, the same multiplexing ability that is the boon of virtualization may be the bane.
• Virtual machine jumping - By exploiting a security hole in the hypervisor, it is conceivable that a user logged into one virtual host could jump to another host, for instance from one windows host to another. Although no such holes have been identified as yet, it is conceivable that they will. In fact, when virtual LANs (vlans) were first implemented, it was possible for a system to retag itself with a different 802.1q tag, enabling it to jump vlans.
• Host traffic interception - By exploiting the hypervisor, it may be possible listen to system calls being made, as at somepoint there has to be a system call from the hosted operating system to the hosting system, and eventually to the hardware. Additionally, it may be possible to track paging files, memory, and disk writes if an attacker is able to break out of their assigned system.

While few vulnerabilities have been identified in the hypervisors as yet, this may be due to virtualization being in its infancy. With any new technology, with more eyes on it, more vulnerabilities will be found. The creator of OpenBSD, Theo de Raadt, has been quoted as saying, "You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes." This highlights the fact that virtualization can introduce as many issues as it solves.

New Modes of Thinking

When implementing virtualization, it is important to assess the impact before moving virtual hosts and hypervisors onto production servers. Among the issues that should be addressed are the policies around which applications can reside within the same hypervisors, and how virtual systems should be moved.

Policies need to be updated to prevent key combinations of applications from residing on the same hypervisors. For instance, a low security application, that has had little or no hardening, should not reside on the same system as a virtual host being exposed to the Internet. While there may not be any threat today, a vulnerability identified tomorrow may put both applications at risk. Many security policies already dictate how systems should be separated from a logical standpoint, and whether the application server and database should be segregated, however these policies will be out of date when virtualization is applied. By updating the policies and guidelines prior to implementation, it may be possible to limit the number of headaches, as well as the push back incurred by security staff when performing the migration.

The second area where policies may need updating are the rules around movement of the virtual hosts. When an application, database, or system resides on a physical host, it is difficult to move the data off, and often times systems are rebuilt. Because of this, data can remain relatively in the control of those it was intended to be used by. With virtualization, the ability to copy a virtual host creates new issues. A copy of the data can be taken and moved offsite, while the original stays intact. Data theft can occur, and if two virtual machines are configured to access the same database, data corruption can occur. For these reasons, policies should be rewritten to include virtualized instances of systems.

Another practice that you may want to consider is the use of "master" virtual machines. Using a master, production copies can be pushed to the hypervisors. When changes need to be made, the change can be made in one location, and migrated out in waves. Moving new changes in waves should allow the system to continue to respond, and can allow for rolling back updates in the event that there is an issue. Additionally, by using a master copy, refreshes can be made to development and test environments to get up-to-date copies of the host.

Conclusion

Virtualization may very well prove to be the boon to system administration and development that is being promised. Before it can realize its full potential, however, the security aspects need to be carefully considered. As with any technology, this consideration needs to occur prior to full rollout to production. By gauging virtualization against both the needs of the business and the confidentiality, integrity, and accessibility triangle of security, virtual machines can be used in places that make sense, rather than in places where the vendors want to push them.