Protecting Administrators from themselves

We all insist, as far as we can, that our end-users are not administrators. Or at least I really hope we do. There are a number of pieces of research that show how the cost of administration rises dramatically when users are given administrative rights as they are then able to wreak havoc over the once nicely locked down, and functioning, system.

However, what about those of us who are administrators because we do indeed trust ourselves? We all naturally run anti-virus products and possibly a few other products too that will offer some protection such as personal firewalls and VPN clients. But what do we actually need to protect ourselves from? There will always be the silly/accidental/idiotic file or folder deletion, which is why we have confirmation dialogs and backups, but what is there to stop us from running potentially malicious, trojaned, code, particularly if it manifests itself as something legitimate such as an executable called notepad.exe with the right icon and other trimmings? Not all malware unfortunately is classed as such by anti-virus vendors and there is always a window of opportunity between malware being released into the wild and anti-virus signatures being available that blocks them.

Enter the AppSense Application Manager product and its “Self-Authorization” feature. As you may know, Application Manager has a powerful feature called “Trusted Ownership Checking” which allows an out of the box configuration to instantly protect against malware threats without the tedium of white lists, black lists or  (frequent) signature updates. It achieves this by looking at the owner of any executable file, and any file that is not owned by a Trusted User or Group, as in an account that we expect to own the executables we wish to execute, such as “Administrators” or “System”, is prevented from running.

Whilst administrators generally do not subject themselves to the full protection of Application Manager by making themselves “restricted” users, as they feel it may make their job harder, there is a “half way house” between being a “restricted” and “unrestricted” user – This is the “Self-Authorizing” user.  Any attempts to execute files, which includes scripts and dll’s as well as .exe’s, will produce a pop-up warning that the file is not owned by a “Trusted Owner” and asking if the file should be allowed to execute or not.

You must still however, when presented with the pop-up message, make the decision as to whether you trust the file you are about to execute or install.  For example, a non-administrative, restricted user downloads a (seemingly legitimate) file/installer and asks you as a (Self Authorizing) Administrator to install the application for them as they are prevented from doing so by Trusted Ownership Checking.  You will of course expect to see the pop-up message as you attempt to execute the file, but you must now ask yourself  if you really do trust this user, and that they have not accidentally downloaded malware.

Secondly, if you are running a standard tool on your Windows build and an Application Manager Self-Authorization dialogue is unexpectedly seen, then this should set alarm bells ringing and the reason for the dialogue should be investigated before the file is run, just in case it is malware and has a nasty payload.

In the example above, I was in a command prompt window as a local administrator (actually started via a “Run As” since I was logged on interactively as a non-administrator, naturally) and I needed to run regedit, as you do. Little did I know, in this ever so slightly contrived scenario, that there was already a regedit.exe executable in that folder. Obviously the dialogue tells me where the executable resides so I know immediately that I am not running what I thought I was. On a Terminal Server/XenApp server or a shared workstation, this could easily happen, either accidentally or deliberately. You could of course set the global SafeDllSearchMode registry value (http://msdn.microsoft.com/en-us/library/ms682586(VS.85).aspx) to stop this particular threat but this is just one example of a number of ways that you could unwittingly run programs that you didn’t actually want to.

Similarly, also watch out for PATH environment variables that have potentially untrustworthy folders in them, particularly when they precede key folders such as %systemroot% and %systemroot%\system32. Or just use Application Manager with the self-authorization feature for your administrative users.

The case of the failing signed driver install

I was asked recently to look at a couple of support cases that had been logged where installations of our Application Manager and Performance Manager products were failing. The logs from the failed installations, obtained from invoking msiexec with the /l*vx syntax, gave the following error:

(Error code 0x800B0109: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.)

A web search for the error gave many matches which didn’t really help so I then tried to reproduce the error in a Windows Server 2003 x86 virtual machine but the installation worked fine, as it usually does. Analysis of the msiexec log from the failing system indicated that the error was occurring when installing our signed device drivers. So next I ran the great Process Monitor tool from SysInternals, now Microsoft, to try and understand what was happening, file system and registry wise, during the installation, particularly around the area where the msiexec process installs the device drivers.

What this showed me was immediately before our driver catalog (.cat) file was read, the “State” registry value in the following key was being read:

HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing

Given the error text from the failed installation, this looked relevant. A quick web search threw up a number of interesting articles, namely:

http://msdn.microsoft.com/en-us/library/aa388201(VS.85).aspx

and

http://blogs.msdn.com/spatdsg/archive/2006/06/05/618082.aspx

which led me to try changing the “state” value in the registry in my test VM from 0x23c00 to 0×40000  (WTPF_ALLOWONLYPERTRUST as per the MSDN link above and the wintrust.h header file so effectively much more restrictive than what was in this value by default).

Retrying the previously successful installation in my test VM then gave exactly the same error that our customers had been experiencing. On passing this information on, both customers confirmed that their “state” registry values were either not as per the default or were missing, due to the parent key being absent, and that setting the “state” value to the default allowed the drivers to be successfully installed.

Case(s) solved! But this leaves me with the desire to know what caused this to happen, particularly as we have had two cases from different customers logged so closely together, given that I stopped believing in coincidences many years ago. This is the main reason for me blogging about this issue – I hope that by the power of search engine indexing that if others suffer this issue then they will be brought here and their problem solved.

Guy Leech

10th Feb 2010

Deleting Orphaned Disks in Citrix XenServer 5.5

Recently I ran out of storage space on a Citrix XenServer because a number of local virtual disks had become orphaned and deleting them proved to be a little difficult, so, I thought it might be useful to share what I found lest others suffer the same issue.

I think the problem occurred because I removed disks which were attached to some VMs which had been created from templates, because I was using them with Provisioning Server and had run the XenConvert stage to copy the hard disk to the vDisk. This was so I could test that they worked without local disks, but didn’t want to delete them in case there was a problem.

I found two methods – the “hard” way and the “easy” way.  Guess which I found first?! ;)

Method 1.

From a console shell prompt run the command:

xe vdi-list

Note that vdi is short for “virtual disk image” in this context. This should result in a display similar to the following with one entry per storage item so will include ISO images as well as virtual disks. Note that this will list all items, not just orphaned ones. Pipe the output to “more” or redirect to a file and use “vi” or similar on the file produced to find the oprhaned items.

uuid ( RO)                : cb5781e0-c6f9-4909-acd6-5fd4b509d117

          name-label ( RW): Vista master for UIA

    name-description ( RW): Created by template provisioner

             sr-uuid ( RO): 72cc0d44-bea7-6c15-cf8d-c2965ca495b2

        virtual-size ( RO): 25769803776

            sharable ( RO): false

           read-only ( RO): false

Fortunately, I knew the name of the disks that were orphaned so passed their uuid values as an argument to the “xe vdi-destroy” command thus:

xe vdi-destroy uuid=cb5781e0-c6f9-4909-acd6-5fd4b509d117

The storage space freed up by deleting the disks will eventually be realized but to force this, you can manually instigate a re-scan of the containing storage repository. For this, we need to know the uuid of the storage repository which we have in in the “sr-uuid” field in the original “xe vdi-list” command above:

xe sr-scan uuid=72cc0d44-bea7-6c15-cf8d-c2965ca495b2

Method 2.

Add the disk to an existing VM and then delete the disk from the “Storage” tab in XenCenter for the VM when it is powered down :)

I hope this is of use to someone?…

 

 

Inaugural Meeting of the Northern VMware User Group

On Wednesday 18^th November I had the pleasure of attending the inaugural meeting of the Northern VMware User Group (UK) at the Wellington pub in Leeds. There were about thirty five people in attendance, mostly administrators of VMware infrastructures, including some big ones, which was a pretty impressive turnout given the far from ideal weather conditions. A couple of VMware vExperts also attended the event.

A very informative presentation from Ross Bisby of b2net covering the details of investigating performance issues in ESX/ESXi environments kicked things off. Hats off to Ross for a top job given he was drafted in at the last minute. This was followed by informal breakout sessions covering topics such as VCP certification, iSCSI storage and VDI.  There was certainly a good deal of interest in VDI from many people there with a variety of experience from planning through testing to having, successfully, deployed it. It was encouraging too to find a number of the attendees already familiar with AppSense products.

Many thanks to the committee for organising such a successful event and to VMware and Veeam for sponsoring the bar – when in Rome …

Looking forward already to the next event!

Sharing Personalization Settings between Operating Systems – Debunking the Myth

I have recently heard, from several different sources, that it is “best practice” not to share user profiles, or personalization settings, between different operating system platforms. On the surface, this seems a sensible limitation since different operating systems have different user profile structures. 

Vista and Windows Server 2008 (WS08) put most profile data somewhere in “\users\%username%\appdata”, whereas XP and Windows Server 2003 (W2K3) may place it in “\documents and settings\%username%\application data” or “\documents and settings\%username%\local settings” or somewhere else entirely.

We can’t predict where the data will go for a given application which doesn’t help us understand the “splatter” that it makes in the file system. This folder lottery is further compounded by the fact that Vista and WS08 implicitly add the “.v2” extension to any profile path you define for a user. What this results in is that with a roaming profile solution, you are forced to have different profiles, and therefore different settings, between XP/W2K3, which implicitly use a “v1” profile, and Vista/WS08 which explicitly use a “v2” profile (even though the path defined for this profile does not actually include the “.v2” extension).

Applications should get the paths to use within the profile folder hierarchy by using operating system API calls that are the same between the different operating systems but will yield the correct folder for the operating system it is being run on. Unfortunately, not all applications are written this way and some will make assumptions about paths and maybe even hard code them which is likely to cause problems even before operating system migration, particularly in Terminal Server/Citrix XenApp environments.

There is also the class of setting that is actually different between the different operating systems. Take for instance the good old desktop wallpaper which most people, if pushed, will confess is the one item that makes their PC experience “personal” (while this is not an essential productivity related personalization setting, it does however provide a good example as to how even the most basic of settings fail to migrate between OS platforms)  Although users don’t know, and indeed do not need to know, they are actually stored in different file formats between XP/W2K3 and Vista/WS08. Therefore if the setting for this, which is stored in the user’s registry hive, was just unintelligently transplanted between the two operating systems then one of the desktops wouldn’t show the correct wallpaper.

Some implementers may say that it is a good idea to start with clean profiles when moving from one operating system to another system since it is a good opportunity, in their view, for a clean start and to leave all the myriad of settings behind that aren’t apparently used for anything and just clutter the profile. However, against this has to be weighed the cost of the user having to re-personalize their applications and desktop. This costs both in terms of time (both users being interrupted during their workflow as they find a toolbar or application setting they need is missing, and then having to remember where and how to re-make the customization, which could be different to how they would have changed the option on their old OS)  and also can cause a certain amount of resistance when these users tell their yet-to-be-upgraded colleagues is that this great new operating system, which has been months in planning, has lost all of their settings and they are struggling to find the new ways to set things the way “they should be”.

Enter AppSense Environment Manager. All of the technical issues outlined above are addressed by Environment Manager making the migration from one operating system to another, and back again if required, a much less painless experience and instead now becomes an automated, seamless process for both the user and administrator alike. The files used by an application within the locally cached profile folders are stored in a relative, rather than absolute, form in the Environment Manager database which then allows them to be subsequently put back in the correct, operating system specific, folder hierarchies. Because Environment Manager functions on a per-application basis, it can much more accurately target which settings need to be brought over onto the new operating system and it also silently transmogrifies items and their settings, such as desktop wallpapers, to help ensure that seamless migration that administrators dream of. All this, of course, is done with next to no configuration by administrators so they do not need to understand the intricacies of any of the applications and subsequent registry settings and profile structures the user uses. This helps make for quick and easy migrations, although I don’t personally like the term “migration” since it implies a one way movement whereas Environment Manager provides bi-directionality with no extra effort.

So in summary…While it is right to say that it is NOT best practice to share ‘roaming profiles’ across OS platforms, AppSense Environment Manager dispels the myth that sharing ‘personalization settings’ between operating systems is not a recommended best practice –in fact AppSense recommend you embrace it…

Some Citrix XenDesktop Troubleshooting Tips

I have recently had to build a new Citrix XenDesktop environment for some testing which included Citrix Provisioning Server and Citrix XenServer. Along the way, I had various issues and struggled to find a single, comprehensive, troubleshooting article so I am going to have a stab at it here since I had to go through various tests in order to sort my issues. Having said this, there are some very good technotes on the Citrix web site here – http://support.citrix.com/product/xd/v3.0/technote/

1. Enable logging for the Workstation Agent and ensure that access to the C$ share of the master XenDesktop image is enabled, including a firewall exception for file sharing. This is so that you can get at the log file without having to log on interactively to the image. See this article for how to enable the logging by a simple edit to the WorkstationAgent.exe.config file:  http://support.citrix.com/article/CTX117452 

   Obviously ensure that the Workstation Agent (Citrix Desktop Service) is successfully starting, as are other Citrix services, and the log shows it registering with the DDC.

2. The event logs are also obviously another place to look when things fail although this can be tricky if your VM has been connected enough to want to reboot when the connection attempt has failed.
3. You can also enable logging for the Desktop Delivery Controller service which is detailed in the link above. Ensure that the DDC service and other Citrix ones start successfully.
4. PortICA logging can be enabled – http://support.citrix.com/article/CTX118837 – which could show potential ICA problems. It didn’t for me but will stay enabled in my base image whilst I am still testing.
5. Citrix tracing tool (CDF – Citrix Diagnostic Facility) – this didn’t help me as it only currently supports a small number of client side features such as USB. It can also be run on the machine running the Workstation Agent but I didn’t do this.  http://support.citrix.com/article/CTX120269
6. I did have some errors when using the XenDesktop Setup Wizard so I followed the steps to get a log file for this. I couldn’t get the log produced via the command line so ended up modifying the .config file as described here: http://support.citrix.com/article/CTX118278 

   My issues actually turned out, I think, to do with the fact that the template I was specifying in the wizard had an 8GB disk attached (it was my Gold Build VM that was booting off the PvS disk but still had the original hard drive in case I needed to rebuild the PvS disk) so each new VM created by the wizard was creating a new 8GB disk and I simply didn’t have the storage for it (not that I got an error suggesting this). I therefore created a new VM in XenServer that had the memory, NIC and CPUs I wanted but had no hard disk so actually didn’t have an OS installed (it never even got booted). This doesn’t matter since the OS comes from the vdisk/vhd you specify, separately, in the wizard.

7. Check that you can logon with the required accounts to the VMs in your XenCenter/XenServer console. This should show any domain joining or account issues, e.g. expiry or permissions. Also check network connectivity to/from them.
8. Fire up your gold image VM, since it should be on a standard image disk so the changes will be lost when it shuts down, add it to a new desktop group without a hosting infrastructure so that you just use the name of the VM in the group. This should tell you if the problem was something funny about the desktop group or the VMs that comprised it.
9. My issue was that I was launching the connection from Web Interface but I wasn’t getting a session, just a failure popup – “Unable to connect to the desktop. This may be a temporary problem. Click OK and then try starting the desktop again. If the problem persists, contact your system administrator”. Before acknowledging the failure popup, look in your %temp% folder for the ICA file that it dynamically created. It won’t be a .ica file but instead will most likely be a .tmp file although will probably start “ica*”– easily spotted by modification time, particularly if you sort on modification time. It is actually the argument to cdsbar.exe if you look in Task Manger on Vista or with SysInternals/Microsoft Process Explorer. Open the ica file in notepad and check that it makes sense – e.g. is connecting to the right thing (“Address=”) and that the entity can be resolved/contacted. Note that the ica file, in best Mission Impossible style, will self destruct, i.e. be deleted, when you ok the failure popup thanks to the “RemoveICAFile=On” line. Note also that there is little point in saving the ica file for later use since it has a logon ticket in there which most likely will have expired.
10. This leads on to checking that port 1494 is accessible in the virtual desktop by telnetting to it. However, port 1494 is only alive for a brief while after the connection is initiated so wait a few seconds after you have clicked on the icon to launch the session in Web Interface, or Program Neighborhood, before trying the telnet. When accessing a pool, look at the temporary ICA file to figure out which machine to check or reduce the pool to a single machine. We are not really looking for anything here other than the connection succeeds although you will probably see the characters “ICA” displayed.
11. As by this stage all logs were looking fine and port 1494 was working, I put on a network monitor, in this case SysInternals/Microsoft Process Monitor, on my client machine (the one accessing Web Interface) and filtered on wfica32.exe. This is when I found that some traffic was going through my proxy that I hadn’t allowed for – bingo, problem solved when the proxy was disabled. In my defence, I had tried accessing from a different client (this should probably be a separate line item in this troubleshooting “guide”) but that had also failed, albeit probably for different reasons as it wasn’t using a proxy.
12. Watch for proxies! Obviously configure them as necessary or disable them.
13. I did have some “funnies” with my XP VMs created by the XenDesktop Setup Wizard and running off PvS. I think they were because after creation I had switched the master disk away from Standard Image mode. My excuse is that you have to manually hit F5 to do a refresh after changing vDisk properties and I didn’t! I was actually getting the error described here: http://forums.citrix.com/message.jspa?messageID=1393521 

    Sometimes the streaming console (StreamConsole.exe) on the PvS box can help diagnose these kinds of issue. Unfortunately it didn’t in this case.

14. I also got caught by my base image having miniscule event log sizes (64KB) so even though they weren’t up for long, it was enough for them to fill up and not to overwrite so it was back to the base image to set larger sizes and set them to overwrite as needed.

How to find what has changed in HKCU

There are times when you need to find out what has changed in a user’s registry hive, either during their session or, more often, when they have logged out. This may be to try and understand why an application isn’t behaving the way it should, or because you are trying to find specific settings to extract and put into a mandatory profile or an environment provisioning mechanism such as that provided by AppSense Environment Manager. Here we reveal how to do you this even if you weren’t actively monitoring the registry in the session.

The SysInternals, now Microsoft, Process Monitor tool is very, very good at this sort of analysis but not if the changes to the registry have already occurred. Check it out here anyway:

http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

With traditional local and roaming profiles, you can load the ntuser.dat hive file containing the changes into regedit, but unfortunately regedit does not show the timestamps that are present on every registry key. This is where the free regrecent tool comes in as it allows you to search a registry key for changes made in a given time (and date) range. Note though that only registry keys have timestamps, not values, so a registry analysis performed this way won’t tell you what values have been modified, added or deleted unfortunately but the information can still be incredibly useful. Of course, if you have a copy of the hive file before the user logged in, either from a backup, the base mandatory profile or from the roaming profile location if the user is still logged on (HINT: take a copy of the original ntuser.dat at this point and work with this file) , then you can compare the changed registry key’s values in the two hives. It also does not require administrator privileges so can be used by the (test) user before they log off.

Download it here:

http://www.appsense.com/mstools

As an aside, when doing on-site troubleshooting in my consultancy days, I used to use regrecent to tell me what had changed 5 minutes after I left site to 5 minutes before arriving to see what had actually changed even though the customer would usually swear blind that they had not changed anything at all!

Microsoft Windows Server 2008 TS/RDS CAL now includes App-V

 As of 1st September 2009, Microsoft is including an App-V (formerly SoftGrid)  licence within the Client Access Licence (CAL) for Windows Server 2008 and 2008 R2, meaning that a separate App-V licence is no longer required and shortly will not even be available.

http://www.microsoft.com/systemcenter/appv/howtobuy/default.mspx

 http://www.microsoft.com/windowsserver2008/en/us/rds-product-licensing.aspx

The media for the client, sequencer and server is available here:

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=0890d6cd-0d3b-4c9d-b208-231c65d3e55a

Some Mandatory Profile Best Practices *** Updated April 16th 2010.

There are a number of different ways that you can capture a profile that you want to subsequently use as a mandatory profile. My preferred approach is to logon as a non-administrative test user, run whatever applications are needed and configure as appropriate, logoff and then take the resulting ntuser.dat, obviously renamed to ntuser.man, as the mandatory profile’s registry hive. I generally do not have any folders in the folder specified for the mandatory profile – it just contains the ntuser.man file and nothing else. *** Update:  However, on Vista, Win7 and WS08, the empty folder AppData\Roaming does need to be created. In addition, if none of the folders that by default are used for items such as “My Pictures” and “My Music” exist in the base profile, these special folders will not be available to the user who is assigned this mandatory profile. However, it is strongly recommended that folder redirection is used to provide these special folders, if required, rather than using the defaults provided in the locally cached profile folder hierarchy. ***

Once the ntuser.man file has been copied away, I load it as a hive in regedit and then check various elements of it; namely:

1. Security – the Access Control Entries (ACEs) for the user used to generate the profile should be removed and an Everyone – Full Control ACE added in its place. It is not actually ideal to open up security to this extent but since we don’t know what user is going to use the profile, we cannot lock it down much further although it could be done with a tool such as subinacl.exe [http://www.microsoft.com/downloads/details.aspx?FamilyID=e8ba3e56-d8fe-4a91-93cf-ed6985e3927b] at logon. For VDI environments, which are necessarily single user, it probably doesn’t matter but for Terminal Services, it means that a user with access to HKEY_USERS through regedit or other tools/scripts/macros can read and write/delete any other logged on user’s registry settings.
2. Search the hive for the username of the user used to generate the hive and delete/replace the values as appropriate.  Note that there is no guarantee that changing a REG_SZ value to a REG_EXPAND_SZ and using “%Username%” or “%UserProfile%” in place of the actual username or locally cached profile folder respectively will work since it is up to the application that reads the value to implement environment variable expansion. Don’t be tempted to delete a whole key unless you are prepared to test that no ill effects occur. For instance, deleting the key “HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders”, because it contains values with the path to the generating user’s locally cached profile folder, will cause problems at logon whereas deleting all of the values in the key, but not the key itself, does not cause issues.
3. Delete all policy registry keys such as “HKCU\Software\Microsoft\Windows\CurrentVersion\Policies” and “HKCU\Software\Policies” (unless of course you want to apply GPO like lockdown this way but it can cause confusion).
4. Strip out anything that you do not want – the best mandatory profiles are generally the simplest. There is, unfortunately, no easy way of deciding what should be stripped out. I tend to focus on Most Recently Used (MRU) lists such as those for opened documents, searches, runs and so on. The benefit of starting with the default user profile rather than a “contaminated” user profile is that this step, generally, is not required.
5. Check all autorun locations, such as “HKCU\Software\Microsoft\Windows\CurrentVersion\Run” and “RunOnce”. It is usually best to have nothing in these keys and have things run at logon via other means.
6. Set application defaults, such as disabling splash screens, either by running the application and configuring it or by directly editing the registry if you know what keys/values need setting.

Once you have unloaded the hive and quit regedit, delete all .log and similar files that may have been created when the hive was loaded. Also check that the folder containing the ntuser.man file and the file itself are owned by the local administrators group and have no write/delete access for non-administrators. This is particularly important if the mandatory profile will be local to the system it is used on rather than through a share since share level permissions can also help protect the hive from accidental or deliberate damage.

Finally, thoroughly test the mandatory profile works as desired when assigned to a representative, non-administrative, user and the available applications are run.

I hope this has been of use, and if you have any questions or comments, please do let us know.