One of the sets doesn’t have this error, but wouldn’t because they are physical servers.  They have been in production a long time, and we have plans to virtualize them. 

The second set doesn’t get the errors, but it is fewer servers and fewer users.

The third set:

• is virtual
• runs on 2008 R2 Hyper-V
• has more servers (6 as opposed to 4 or 5 for the other two)
• supports more users and more users per server (averages around 20 users per server during business hours)

Around November, we started upgrading our hosts from 2008 to 2008 R2.  The problems have been getting progressively worse peaking in the last 2 months.  Our last 2008 host was converted in March. 

After some event log review, we were able to correlate some of the issues to the following error in the event log:

Event Type:    Warning
Event Source:    netvsc
Event Category:    None
Event ID:    5
Date:        4/19/2010
Time:        3:49:53 PM
User:        N/A
Computer:    <ServerNameChangedToProtectTheGuilty>
Description:
The miniport ‘Microsoft Virtual Machine Bus Network Adapter #4′ hung.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 02 00 52 00   ……R.
0008: 00 00 00 00 05 00 00 80   …….€
0010: 00 00 00 00 00 00 00 00   ……..
0018: 00 00 00 00 00 00 00 00   ……..
0020: 00 00 00 00 00 00 00 00   ……..

and right behind that would be this message:

Event Type:    Information
Event Source:    netvsc
Event Category:    None
Event ID:    4
Date:        4/19/2010
Time:        3:49:53 PM
User:        N/A
Computer:    <ServerNameChangedToProtectTheGuilty>
Description:
The miniport ‘Microsoft Virtual Machine Bus Network Adapter #4′ reset.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 02 00 52 00   ……R.
0008: 00 00 00 00 04 00 00 40   …….@
0010: 00 00 00 00 00 00 00 00   ……..
0018: 00 00 00 00 00 00 00 00   ……..
0020: 00 00 00 00 00 00 00 00   ……..

After doing a bit of searching and getting a lot of nothing, and doing some on site troubleshooting without much luck, I finally broke down and called Microsoft.  I spent a day e-mailing back and forth with someone who was suggesting that I try all the things that I had already tried, so I contacted our TAM and had the case escalated. 

The technician then informed me that there was an internal hotfix that had not been fully tested yet, that related to my issue.  It seems that in 2008 R2 Hyper-V guests running Server 2003, the network adapter will hang and then reset under heavy load.  The hotfix has to be applied to the host and then the integration services on the guest have to be updated.  In my environment, when I updated, I had to remove the integration services from the guest before the updated NIC driver would install.  I reported this behavior to the technician I was working with, but he said that he couldn’t reproduce that particular problem and that he had no issues updating his test environment.

It is my understanding that the hotfix will be released under KB981836.  When you install this, it changes the integration services version from 6.1.7600.16385 to 6.1.7600.20683.  You can see this if you look at the driver version on the guest NIC.

This is all simple.  Just drop the LUNs from the file server, give them to the cluster and assign them as pass through disk to the guest OS.  Simple…

Except, I kept getting the following errors when I try to do the import:

An online pack already exists.

The operation failed, because an online pack object already exists.

The provider encountered an error while merging two packs.

I mean really…  what does all that mean.  I couldn’t find anything pertinent in my searches, I tried working in Diskpart, and got some of the same errors.

After lots of searches, I couldn’t find anything to help.  So I opened a ticket with Microsoft.  I got a very helpful person who told me that it basically sounds like a driver issue, but he couldn’t say for certain.  He couldn’t find much on it either.

So on a whim, I moved my test LUN from my production file server to another 2003 server.  That worked just fine, so I then moved it from there to the 2008 R2 machine that I had been working with.  That worked just fine.  Weird…

So I checked the driver for the production file server:

and compared that to the other 2003 server:

As you can tell, the culprit seems to be an outdated driver on the current file server.  Seems I don’t keep things as up to date as I should.

One of the REG_EXPAND_SZ values on the right is the UninstallString.  If you copy that value and paste it into the command line, it will politely ask:  Are you sure you want to uninstall this product?  If you are sure you want to uninstall it, go ahead and hit Ok.

If you want to get rid of both pieces of Forefront ( the Security State Assessment Service, and the Client Security Antimalware Service) you will need to find both registry keys and run both uninstalls.  There is a catch though…  the Client Security Antimalware Service has a value that looks like this: MsiExec.exe /I{436028CD-6476-4224-9274-8F0320F30FD1}.  To get it to uninstall, you need to change the /I to /X like this MsiExec.exe /X{436028CD-6476-4224-9274-8F0320F30FD1}.

There was however enough space, but for some reason Microsoft Forefront decided that I shouldn’t be allowed to extend a volume remotely.   So I disabled the services and did it anyway.

Server Core: Setting a Static IP address

Server Core: Setting a Static IP address – Techinical Blog of Jay-R Barrios

http://msdn.microsoft.com/en-us/library/aa387102(VS.85).aspx

And every time I forget the command to do something in Server Core, when I do a search to find it, I always come back with a link to this post:

http://blogs.dirteam.com/blogs/sanderberkouwer/archive/2007/08/13/changing-server-core-s-look-and-feel-part-1.aspx

And just because I am lazy, and don’t particularly like the command line (even though I like Server Core), I have created a folder with a few batch files that I use when I am setting up a Server Core machine.  For instance, I have a batch file that has the product key in it already and I run it to install the product key and activate. 

Add or remove programs

Again there are two ways to get information on the software that is installed onto your Server Core installation in absence of the “Add or Remove programs” Control Panel applet. (also known as appwiz.cpl)

Registry

The first way is by checking the registry keys underneath the following key:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall

Luckily Server Core offers regedit.exe and regedt32.exe to digg deep into the registry of our Server Core box. Actually the “Add or Remove programs” Control Panel applet checks these registry keys at well. Since we all know how long it takes to get a complete list of all the software on your windows box it’s obvious this method takes a lot of time.

As this TechNet page points out the software packages that are listed underneath the Uninstall registry key all have a registry value associated with the UninstallString. You can use this value as a command to uninstall the software.

WMI

The things that are better left unspoken : Add or Remove programs in Server Core

http://technet2.microsoft.com/windowsserver2008/en/library/47a23a74-e13c-46de-8d30-ad0afb1eaffc1033.mspx?mfr=true