IIS6 doesn’t like NewSID

Our PowerShell-based VM building framework is still in use. This is because some of the components for the new system were sent First Class instead of Next Day, and got lost in the mail. In future I’ll be sure to demand Next Day delivery on everything.

So our test server still operates (or not) at the whim of a mess of scripts. This seems to be working out OK so far, although human intervention is required when the VM registration process fails to restart properly, ie. roughly once a week. At least the terminal server I set up lets me fix it from home before I leave for work in the morning, which means the server might actually be built and working by the time I arrive.

The original test server runs Windows XP, which is pretty much the minimum requirement for our product. Recently we ran into a bug that only appeared on Windows 2003, requiring a second test server running that OS. As I mentioned back in May, we use NewSID plus yet more Powershell to deploy images since it suits our system better than Sysprep.

NewSID is a great tool. It generates a new Security ID for the machine, updates the Registry and other system files accordingly, fixes the filesystem’s Access Control Lists to use the new SID, and reboots the machine. Unfortunately, it doesn’t update the IIS Metabase.

On Windows XP this is not an issue for us. IIS versions prior to 6 use only a single application pool and our product’s installer sets up all permissions appropriately anyway. On II6, however, our app pool identity is a member of IIS_WPG, the IIS Worker Process Group, and relies on that group’s permissions. NewSID leaves the old SID in the Metabase and doesn’t add the new one, effectively nuking IIS_WPG’s permissions on all IIS directories.

IIS7 doesn’t have this problem. It uses built-in NT accounts not dependent upon the machine or domain SID, making a clean IIS7 Metabase effectively immune to this sort of corruption. Once you start adding permission sets it’s a different story, but in most cases only a clean system would be the subject of NewSID.

The conventional workaround for the issue is to remove IIS prior to running NewSID and reinstall it afterwards. This is fine for manual setup but falls short in our situation. It is by all means possible to automate the addition and removal of Windows components with the ’sysocmgr’ command, but setting up the necessary answer files is tedious. I wanted a better way.

So I poked around in Powershell, made heavy use of Google, and wrote some exploratory C#. Through Directory Services the Metabase can be edited just like Active Directory. I wrote some code to look through all the Access Control Lists for entries with trustees in a given domain and replace the trustee with its equivalent in another domain. This was all done at the SID level; ‘equivalent’ trustees are considered to be those with the same RID, since NewSID does not change that. Trustees that mapped to known accounts are ignored, since they aren’t broken.

It took me a while to figure out why the ‘patched’ entries were being removed by the original version of the tool. It seems that the AccessControlEntry interface doesn’t like raw SIDs being put into its Trustee property. Resolving the SID to a human-readable account beforehand solved this problem.

The result is a tool called MetabaseACL, which takes a directory path as its first argument:

MetabaseACL "IIS://Localhost"

This will display all ACLs in the Metabase within the given path. To fix SIDs, it can be given the ‘fixup’ command:

MetabaseACL "IIS://Localhost" fixup <old SID> <new SID>

It doesn’t matter if the SIDs are specific NT accounts or domain/machine SIDs; only the domain info will be used.

This integrates nicely with the ‘newsid’ package our VM builder uses. The script which does the work now looks something like this:

if($env:COMPUTERNAME -eq $compName)
{
if(Test-Path “C:\Machine.sid”)
{
$id = New-Object System.Security.Principal.NTAccount(”Administrator”);
$newSid = $id.Translate([System.Security.Principal.SecurityIdentifier]).AccountDomainSid.Value;
$oldSid = Get-Content “C:\Machine.sid”;
MetabaseACL “IIS://Localhost” fixup $oldSid $newSid;
Remove-Item “C:\Machine.sid”;
}
[We're done here. Call next script in the chain.]
}
else
{
net stop iisadmin /y ;
if(-not (Test-Path HKCU:/SOFTWARE/Sysinternals))
{
New-Item HKCU:/SOFTWARE/Sysinternals
}
# Record old Machine SID.
$id = New-Object System.Security.Principal.NTAccount(”Administrator”);
$sid = $id.Translate([System.Security.Principal.SecurityIdentifier]).AccountDomainSid.Value;
Set-Content “C:\Machine.sid” $sid;
# Prepare for NewSID.
$key = New-Item HKCU:/SOFTWARE/Sysinternals/NewSID;
$key.SetValue(”EulaAccepted”, 1);
$p = [diagnostics.process]::Start(”newsid.exe”, “/a ${compName}”);
$p.WaitForExit();
}

This script is run when the VM image boots. First time through it will record its SID and run NewSID. On the second boot it will retrieve the old SID, get the new SID, run MetabaseACL to do the fixup, and call the next script in the image building procedure.

And so ends another adventure with VM image deployment. Judging by the number of forum topics I came across while googling for a solution to the IIS/NewSID problem, this new tool will prove useful to people besides me.

Download MetabaseACL

Zip file includes source code and binaries. Runs on WinXP and Win2003. I haven’t tried it on Vista but it should be fine.

VirtualCloud bears fruit

Our testing system is finally coming together. Once the last few parts for the server cluster arrive, integration testing can begin. Meanwhile, one of its components is finding uses outside of its parent project.

Managed.VIM, our C# proxy for the VIM webservice, is available for download from the VirtualCloud Sourceforge site. It is currently in beta (revision 105) and probably not ready for use in a production environment. It doesn’t yet implement all the functionality of VIM either; only the bits we need for VirtualCloud have been written so far.

It’s a nice tool for PowerShell monitoring of VirtualCenter, though. Once the Managed.VIM assembly has been loaded into PowerShell, the proxy object can be created as follows:

$vim = New-Object Managed.VIM.Vim(”https://localhost:8443″, “vc_username”, “vc_password”);

Replace the URL, username and password with values appropriate to your VirtualCenter configuration.

The Vim object exposes collections of VMs, Hosts, Farms, Datastores and Templates. This is the root of the VirtualCenter object hierarchy. Nodes in the hierarchy expose children by type, in the same way as the root. For instance, a Farm can contain Hosts, VM groups and VMs; these are accessible via the Hosts, Groups and VirtualMachines indexed properties respectively, and can be retrieved by name.

The root of the hierarchy also exposes collections of all VMs and Hosts managed by the VirtualCenter service. Hosts are accessible by name, as they are within Farms, but the collection of all VMs is indexed by UUID. This is because you can have multiple VMs with the same name as long as they are in different groups.

The following will get the object corresponding to the host ‘testserver1′ in the server farm ‘TestFarm’:

$testServer = $vim.Farms["TestFarm"].Hosts["testserver1"];

Operations supported by VirtualMachine objects currently include:

• Start, stop and suspend,
• Change persistence mode of hard disks (VirtualCenter does not permit this, but VIM does),
• Get or set device or ISO image used for CD drives,
• Get CPU count and memory allocated to the VM,
• Get Host.

Operations supported by Host objects include:

• Get CPU and memory info,
• Get accessible datastores,
• Get hosted VMs,
• Get Farm.

Operations supported by Template objects include:

• Get size, memory allocation, datastore and guest OS,
• Deploy template to Host, with optional customisation of Windows guests.

Customisation of guest operating systems has not actually been tested yet and, like most of this assembly, is not documented yet either. It involves the Managed.VIM.Customisation.WindowsCustomisation class, which exposes some fairly self-explanatory properties and methods. Use at your own risk.

The leaf nodes of the hierarchy (Hosts, VirtualMachines, etc) are heavyweight objects. They are kept synchronised with the VirtualCenter service and therefore generate a certain amount of network traffic. For this reason, each collection also exposes a list of the names of all objects in it, like the Keys property of a dictionary. For example:

$hosts = $vim.Farms["TestFarm"].Hosts.Names;

This is to be preferred when you don’t actually need any of the properties or functions on a Host or VirtualMachine, but merely need to determine which ones are available or where they are.

Datastores are a bit of a sore point at present. Multiple datastores can technically have the same name, although I think this only applies to the default datastore on a host, called ‘local’. Working on this assumption I’ve allowed for multiple ‘local’ datastores, accessed via $vim.Datastores.Local, which is indexed by Host. All other stores are accessed by name (through $vim.Datastores) and must therefore have unique names. Otherwise Managed.VIM’s internals don’t work correctly and will attempt to log lots of errors through log4net. They’ll do this on pretty much every update request too, which will eat lots of CPU and effectively kill the update mechanism. I’m not going to try to fix this until I know it’s a problem, because the fix breaks some other things.

Custom properties on Hosts and VirtualMachines are supported by Managed.VIM. Getting and setting values is possible, but adding new properties requires the VirtualCenter client. This is apparently a limitation of the VIM SDK. This was a bit worrying initially since VirtualCloud will require several custom properties. Fortunately, VirtualCenter doesn’t appear to mind having new properties poked directly into its database, so the VirtualCloud installer (when written) will probably want to know where this database is in order to update it. I’d very much like to know if this is considered acceptable, or whether most VirtualCenter admins want other apps to consider that database inviolate; the alternative is asking the admin to add about twelve properties to VirtualCenter by hand…

By the way, Console is made of pure win. Tabbed PowerShell is extremely useful.

VMware, PowerShell and much automation

We run many of our servers as virtual machines on VMware GSX Server. This makes backups easy; simply power down the VM and copy the relevant disk image. It’s also rather nice for setting up a test server from a known-good state every day.

We already have a VM that builds in the early hours of the morning, right after the nightly production build. It has a non-persistent disk image, which means that all changes to the hard disk are lost when it is switched off. A script shuts it down prior to the production build, then starts it again afterwards. Another script starts on the VM at boot time to run the necessary installers.

This setup is simple and effective, but the installers can take a while to run. The sample data is a particular problem, sometimes taking as long as forty-five minutes to insert into the database. This means that, should the VM fail to build during the night, there’s a very long period of time between fixing the problem and having a working test server. Plus, the server image is connected to the Windows domain and Active Directory sometimes gets confused by the disk image reverting. This always requires manual intervention and the process for fixing it is prone to error.

With the new requirement for multiple similar servers running automatic tests in parallel, I decided to solve all our problems at once by writing a set of scripts to build server images independently of the network and the domain. The first criterion, network independence, arises from the fact that all the servers will come from a single base image with a single hostname, and running more than one at a time is going to really confuse the network. Changing the hostname is high on the list of priorities for the image builder.

The system I decided upon involves attaching a ‘parameter disk’ to the VM, containing a Bootstrap.ps1 script and all the programs necessary to configure the VM. The base image will search for the bootstrap script every time it starts and run it if possible. This lets us shut down a built image without losing everything on it (as we did with the old system) and without having the installers run on every boot. Once the image is set up, we just disconnect the parameter disk and it’ll behave as an ordinary VM.

Implementing all this has taken most of a fortnight and expanded our PowerShell script library significantly. Most of the complexity comes from the scripts’ interaction with VMware. This interaction occurs primarily through VMware’s COM API, but some things aren’t supported, particularly modifications to the VMs’ hardware configuration.

On the host machine, where the VMs are manipulated:

The image building script does the following:

1. Copies the base image to the target directory,
2. Calls another script which sets up the parameter disk,
3. Boots the VM and waits for it to exit,
4. Disconnects the parameter disk.

Our test server VM requires a few more things to be done afterwards, like checking the logs on the parameter disk, setting the new image’s MAC address, attaching it to the network, shutting down the old test server and booting the new one.

I needed to be able to set the VM’s MAC address because all our other machines are on the domain, the test server cannot be, and I need to give the VM a known IP so it can coincide with the IP of the old test server, which was on the domain and is the host most of our other machines have bookmarked.

The API provides a means for modifying configuration info like MAC addresses and hard disk devices in memory (VmCtl.Config) but any changes made this way are lost when the VM process terminates. Making permanent changes that will persist when you move the VM image elsewhere requires direct manipulation of the .vmx file, which does not appear to be very well documented. I take this to mean that it’s not really meant to be tinkered with, but that’s just an invitation…

Fortunately, Google turned up this page among the documentation, describing the procedure for setting a static MAC address.

On the VM, where the parameter disk is consumed:

Building two dozen servers for automated testing isn’t much use if they can’t connect to the network because they all have the same hostname. We need some means of changing the VM’s hostname. All Windows machines have a supposedly-unique System ID (SID) as well, and I’d like to change that during image construction just in case we need build machines for the domain.

Windows does not support changing the SID after the GUI phase of system installation has begun. The base image contains a thoroughly cooked Windows system, which has not only fully completed the OS installation but has several apps installed as well. Fortunately for Windows sysadmins everywhere, there’s a nice little app called NewSID, a part of the Sysinternals collection, which can change the SID after installation is complete. Running it in non-interactive mode can be done via a command-line switch, ‘/a’.

At least, in theory.

Microsoft has added a EULA to each of the Sysinternals tools, which means that NewSID will sit and wait for someone to click a button before doing anything, even in ‘non-interactive’ mode. This can be circumvented by creating a registry key before you run NewSID:

[HKEY_CURRENT_USER\SOFTWARE\Sysinternals\NewSID]
“EulaAccepted”=dword:1


Of course, NewSID will reboot the machine after it runs. For our purposes it should only be run once, no matter how many times the system is rebooted subsequently.

So the PowerShell script for running NewSID on a machine and changing its hostname to $compName looks something like this:

if($env:COMPUTERNAME -eq $compName)
{
[do subsequent setup tasks here];
}
else
{
if(-not (Test-Path HKCU:/SOFTWARE/Sysinternals))
{
New-Item HKCU:/SOFTWARE/Sysinternals
}
$key = New-Item HKCU:/SOFTWARE/Sysinternals/NewSID;
$key.SetValue(”EulaAccepted”, 1);
$p = [diagnostics.process]::Start(”newsid.exe”, “/a ${compName}”);
$p.WaitForExit();
}


Construction of the parameter disk:

Parameter disks are defined by INCLUDES configuration files which contain a list of files to put on the disk. This turned out to be a bit limiting; NewSID will need to be run for all our server configurations and adding that functionality isn’t as simple as just putting files on the disk.

So packages can also be included. A package is basically a directory with a BUILD.ps1 script in it, which takes a path to the parameter disk and can assume it’s running from the package’s own directory. For most packages this script just copies files, but the NewSID one has to do something a little more complicated.

A package is not allowed to make any assumptions about the parameter disk apart from the presence of a Bootstrap.ps1 file. All parameter disk configurations will have this file and it will be the first thing run. NewSID’s BUILD.ps1 has to set up the disk so that its own Bootstrap.ps1 gets run first, which is done by renaming the existing Bootstrap.ps1 and chaining to it from its own.

This does come with the caveat that NewSID must be the last package included, otherwise it might get displaced by another.

An INCLUDES file looks something like this:

Bootstrap.ps1
@Installers
@NewSID [hostname]


Filenames are relative to the directory containing the INCLUDES file. Any line starting with an ‘@’ is assumed to be a package name. Packages can be given a list of comma-delimited arguments, which will be passed as an array as the second argument to BUILD.ps1.

At present, our scripts are rather tied to our filesystem layout and server configurations. Once they’re a bit more portable I’ll make them available for download.

UPDATE: These scripts are rapidly approaching obsolescence, what with our decision to manage our VMs with VirtualCenter. Managed.VIM and VirtualCloud will supercede the scripts and be a lot less nasty to maintain.