Michael Phillips Blog

Error installing Integration Services on Hyper-V VM

I was trying to change from a “Legacy Network Adapter” to a “Network Adapter” on one of my Hyper-V VMs.   I added the “Network Adapter” and removed the “Legacy Network Adapter”, and started the machine up.  When the machine came up, it wouldn’t connect to the network.  Having seen this before I knew exactly how to fix it.  Run the Integration Services install again.  I did that, and got:

An error has occurred: One of the update processes returned error code  61658

So, being the smart guy that I am, I rebooted and tried again.  Same result.  The hits I got on google suggested that I was still running an CTP or Beta, but I am running the RTM version (2008 Hyper-V, but not R2 on this one). 

I was logging into this machine with a Domain account, but it wasn’t on the network.  We use some restrictions on the server desktops and redirect application setting and such, so I thought that might be related. 

Here is what worked:

I added back the Legacy Network Adapter (leaving the non-Legacy adapter as well)

I installed integration services again (and it worked just fine)

I removed the Legacy Network Adapter

Now the machine is working just as expected.

Error installing DPM 2010 Beta

I was installing the DPM 2010 Beta (finally) and had an issue trying to get the SQL 2008 to install.  Finally figured out that I had the install files stored too deeply in a network share.  I figure this out by running the SQL install directly and when it when to check prereq’s it had an error on one section and when you click for more info this is what you get:

Rule "Long path names to files on SQL Server installation media" failed.

SQL Server installation media on a network share or in a custom folder can cause installation failure if the total length of the path exceeds 260 characters. To correct this issue, utilize Net Use functionality or shorten the path name to the SQL Server setup.exe file.

So, I moved it to a shorter path and it installed just fine.

Data Protection Manager 2010

So I am a bit late realizing this, but the Beta for DPM 2010 is available now on the Connect site.  I haven’t read anything on it yet, so mainly I am just posting this to make myself look into it.

https://connect.microsoft.com/Downloads/DownloadDetails.aspx?SiteID=840&DownloadID=22070

Remote Server Administration Tools for Windows 7

As a follow up to my earlier post about the Remote server Administration Tools for the Windows 7 Release Candidate (RC), I thought I would post a link to the RTM version.  It took a bit to find it.  A search on Microsoft’s site wasn’t helpful to me.  I finally found a reference on Ben’s Blog here: http://blogs.msdn.com/virtual_pc_guy/archive/2009/08/12/rsat-for-windows-7-rtm-now-available.aspx

Which sent me here:

Remote Server Administration Tools for Windows 7

Brief Description

Remote Server Administration Tools for Windows® 7 enables IT administrators to manage roles and features that are installed on computers that are running Windows Server® 2008 R2, Windows Server® 2008, or Windows Server® 2003, from a remote computer that is running Windows 7.

Download details: Remote Server Administration Tools for Windows 7

Temporary Registry Profiles – Finally an answer

We have been having issues at work for a while where a user gets “cannot load the locally stored profile” and a temporary profile is created.  It gets to be quite frustrating for the users because each time they log in they lose any settings that they have made.  Apparently, we arent’ the only company that has been having this issue.  The complete article has a very interesting discussion on the Windows Logical Prefetcher and what the cause of the problem was, but the gist of what you need to fix the problem is update your Citrix client or use the work arounds mentioned below:

Now that the problem was understood, Microsoft and Citrix brainstormed on workarounds customers could apply while Citrix worked on an update to the ICA Client that would prevent the sharing violation. One workaround was to disable application prefetching and another was to write a logoff script that deletes the Ssonsvr.exe prefetch files. Citrix published the workarounds in this Citrix Knowledge Base article and Microsoft in this Microsoft Knowledge Base article. The update to the ICA Client, which was made available a few days later, changed the network provider DLL to 10 seconds after Ssonsvr.exe launches before returning control to Mpnotify.exe. Because Winlogon waits for Mpnotify to exit before logging on a user, the Logical Prefetcher won’t associate Winlogon’s accesses of the user’s hive with Ssonsvr.exe’s startup.

Mark’s Blog : The Case of the Temporary Registry Profiles

Follow up to the DPM recovery point expiration issues

Previously, I blogged about issues I was having where old recovery points were not being expired/removed from my DPM servers.  I had to open a ticket with Microsoft, and worked with them to determine the cause, and since then, they have released a fix.

The fix that Microsoft developed is here: http://www.microsoft.com/downloads/details.aspx?FamilyID=aee949aa-d3e7-4b0f-b718-00b7c20f1257&displayLang=en

A few people have asked for the PowerShell script “show-pruneshadowcopies.ps1” that Microsoft provided and I mentioned in my previous post (here).  The script looks like this:

#displays all RP for data sources and shows which RP’s would be deleted by the regular pruneshadowcopies.ps1
# Outputs to a logfile:  C:\Program Files\Microsoft DPM\DPM\bin\SHOW-PRUNESHADOWCOPIES.LOG

#Author    : Mike J
#Date    : 02/24/2009
$version="V1.0"

$date=get-date
$logfile="SHOW-PRUNESHADOWCOPIES.LOG.txt"

function GetDistinctDays([Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.ProtectionGroup] $group,
[Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.Datasource] $ds)
{   
    if($group.ProtectionType -eq [Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.OMCommon.ProtectionType]::DiskToTape)
    {
        return 0
    }
    $scheduleList = get-policyschedule -ProtectionGroup $group -ShortTerm
    if($ds -is [Microsoft.Internal.EnterpriseStorage.Dls.UI.ObjectModel.FileSystem.FsDataSource])
    {
        $jobType = [Microsoft.Internal.EnterpriseStorage.Dls.Intent.JobTypeType]::ShadowCopy
    }
    else
    {
        $jobType = [Microsoft.Internal.EnterpriseStorage.Dls.Intent.JobTypeType]::FullReplicationForApplication
        if($ds.ProtectionType -eq [Microsoft.Internal.EnterpriseStorage.Dls.Intent.ReplicaProtectionType]::ProtectFromDPM)
        {           
            return 2
        }
    }
    write-host   "Look for jobType $jobType"

    foreach($schedule in $scheduleList)
    {
        write-host("schedule jobType {0}" -f $schedule.JobType)
        if($schedule.JobType -eq $jobType)
        {
            return [Math]::Ceiling(($schedule.WeekDays.Length * $ds.RecoveryRangeinDays) / 7)
        }
    }

    return 0
}

function IsShadowCopyExternal($id)
{
    $result = $false;

    $ctx = New-Object -Typename Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlContext
    $ctx.Open()

    $cmd = $ctx.CreateCommand()
    $cmd.CommandText = "select COUNT(*) from tbl_RM_ShadowCopy where shadowcopyid = ‘$id’"  
    write-host $cmd.CommandText
    $countObj = $cmd.ExecuteScalar()
    write-host $countObj
    if ($countObj -eq 0)
    {
        $result = $true
    }
    $cmd.Dispose()
    $ctx.Close()

    return $result
}

function IsShadowCopyInUse($id)
{
    $result = $true;

    $ctx = New-Object -Typename Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlContext
    $ctx.Open()

    $cmd = $ctx.CreateCommand()
    $cmd.CommandText = "select ArchiveTaskId, RecoveryJobId from tbl_RM_ShadowCopy where ShadowCopyId = ‘$id’"  
    write-host $cmd.CommandText
    $reader = $cmd.ExecuteReader()
    while($reader.Read())
    {
        if ($reader.IsDBNull(0) -and $reader.IsDBNull(1))
        {
            $result = $false
        }
    }
    $cmd.Dispose()
    $ctx.Close()

    return $result
}

"**********************************" > $logfile
"Version $version" >> $logfile
get-date >> $logfile

$dpmservername = &"hostname"

$dpmsrv = connect-dpmserver $dpmservername

if (!$dpmsrv)
{
    write-host "Unable to connect to $dpmservername"
    exit 1
}

write-host $dpmservername
"Selected DPM server = $DPMservername" >> $logfile
$pgList = get-protectiongroup $dpmservername
if (!$pgList)
{
    write-host   "No PGs found"
    disconnect-dpmserver $dpmservername
    exit 2
}

write-host("Number of ProtectionGroups = {0}" -f $pgList.Length)
$replicaList = @{}
$latestScDateList = @{}

foreach($pg in $pgList)
{
    $dslist = get-datasource $pg
    if ($dslist.length -gt 0)
    {
    write-host("Number of datasources in this PG = {0}" -f $dslist.length)
    ("Number of datasources in this PG = {0}" -f $dslist.length) >> $logfile
     }
    Foreach ($ds in $dslist)
    {
       write-host("DS NAME=  $ds")
       ("DS NAME=  $ds") >>$logfile
    }
    foreach ($ds in $dslist)
    {       
        $rplist = get-recoverypoint $ds | where { $_.DataLocation -eq ‘Disk’ }
        write-host("Number of recovery points for $ds {0}" -f $rplist.length)
        ("Number of recovery points for $ds {0}" -f $rplist.length) >>$logfile 
        $countDistinctDays = GetDistinctDays $pg $ds
        write-host("Number of days with fulls = $countDistinctDays")
        ("Number of days with fulls = $countDistinctDays") >>$logfile
        if($countDistinctDays -eq 0)
        {
            write-host   "D2T PG. No recovery points to delete"
            "D2T PG. No recovery points to delete" >>$logfile
            continue;
        }
        $replicaList[$ds.ReplicaPath] = $ds.RecoveryRangeinDays
        $latestScDateList[$ds.ReplicaPath] = new-object DateTime 0,0
        $lastDayOfRetentionRange = ([DateTime]::UtcNow).AddDays($ds.RecoveryRangeinDays * -1);       
        write-host("Distinct days to count = {0}. LastDayOfRetentionRange = {1} " -f $countDistinctDays, $lastDayOfRetentionRange)
        ("Distinct days to count = {0}. LastDayOfRetentionRange = {1} " -f $countDistinctDays, $lastDayOfRetentionRange) >>$logfile
        $distinctDays = 0;
        $lastDistinctDay = (get-Date).Date
        $numberOfRecoveryPointsDeleted = 0

        if ($rplist)
        {
            foreach ($rp in ($rplist | sort-object -property UtcRepresentedPointInTime -descending))
            {                       
                if ($rp)
                {                   
                    if ($rp.UtcRepresentedPointInTime.Date -lt $lastDistinctDay)
                    {
                        $distinctDays += 1
                        $lastDistinctDay = $rp.UtcRepresentedPointInTime.Date
                    }
                    write-host(" $ds")
                    (" $ds") >>$logfile
                    write-host("  Recovery Point #$distinctdays RPtime={0}" -f $rp.UtcRepresentedPointInTime)
                    ("  Recovery Point #$distinctdays RPtime={0}" -f $rp.UtcRepresentedPointInTime) >>$logfile
                    if (($distinctDays -gt $countDistinctDays) -and ($rp.UtcRepresentedPointInTime -lt $lastDayOfRetentionRange))
                    {
                        write-host ("Recovery Point would be deleted ! – RPtime={0}" -f $rp.UtcRepresentedPointInTime)  -foregroundcolor red
                        ("Recovery Point would be deleted ! – RPtime={0} <<<<<<<" -f $rp.UtcRepresentedPointInTime) >>$logfile
#remove-recoverypoint $rp -ForceDeletion -confirm:$true | out-null
                        $numberOfRecoveryPointsDeleted += 1
                    }
                    else
                    {
                        write-host "    Recovery point not expired yet"
                        "    Recovery point not yet expired" >>$logfile
                    }
                }
                else
                {
                    write-host "Got a NULL rp"
                    "Got a NULL rp" >>$logfile
                }   
            }

            write-host "Number of RPs that would be deleted = $numberOfRecoveryPointsDeleted"  
            "Number of RPs that would be deleted = $numberOfRecoveryPointsDeleted" >>$logfile            
        }
    }
}

disconnect-dpmserver $dpmservername
write-host "Exiting from script"

exit

Hyper-V and DPM – Some issues that you may see

We have several (15 or so) Hyper-V hosts running a number (126 or so) guests.  We use DPM to backup our servers, but only a few of our VMs are backed up at the host level.  Most are backed up as regular clients.  I have been having trouble with a couple of the ones that we do backup at the host level and just got to looking for the answer to what is going on.  Lucky for me I waited long enough for the Core Team to come up with some suggestions:

Ask the Core Team : DPM 2007 – Troubleshooting protection for Hyper-V

This post is about Windows Server 2008 with the Hyper-V role installed, that are being protected by System Center Data Protection Manager 2007.  There may be one or many Virtual Machines on each Host/Parent Partition, and they may be running Windows 2003 and/or Windows 2008. 

Ask the Core Team : DPM 2007 – Troubleshooting protection for Hyper-V

Remote Server Administration Tools for Windows 7 Release Candidate (RC)

Admin tools for Windows 7 (RC).  Windows 7 rocks so far.  I just got to the point where I decided to install the admin tools though. 

Installing Remote Server Administration Tools for Windows 7 RC
You must be either a member of the Administrators group on the computer on which you want to install the Administration Tools pack, or you must be logged on to the computer by using the built-in Administrator account.
Important: Remove all versions of Administration Tools Pack or Remote Server Administration Tools for Windows Vista with SP1 from the computer before installing Remote Server Administration Tools for Windows 7 RC.
Important: Only one copy of Remote Server Administration Tools for Windows 7 RC can be installed on a computer at one time. Before you install a new package, remove any existing copies of Remote Server Administration Tools for Windows 7 RC. This includes any copies that are in different languages. To remove existing copies of Remote Server Administration Tools for Windows 7 RC, see Removing the complete Remote Server Administration Tools for Windows 7 RC pack on this page.
1. On a computer that is running the RC release of Windows 7 RC, download the Remote Server Administration Tools for Windows 7 RC package from the Microsoft Download Center.
2. Open the folder into which the package downloaded, and double-click the package to unpack it, and then start the Remote Server Administration Tools for Windows 7 RC Setup Wizard.
Important: You must accept the License Terms and Limited Warranty to start to install the Administration Tools pack.
3. Complete all the steps that you must follow by the wizard, and then click Finish to exit the wizard when installation is completed.
4. Click Start, click Control Panel, and then click Programs.
5. In the Programs and Features area, click Turn Windows features on or off.
6. If you are prompted by User Account Control to enable the Windows Features dialog box to open, click Continue.
7. In the Windows Features dialog box, expand Remote Server Administration Tools.
8. Select the remote management tools that you want to install.
9. Click OK.

Download details: Remote Server Administration Tools for Windows 7 Release Candidate (RC)

Time Travel?

Evidently, Windows Server 2003, running on Hyper-V, is confused and thinks it must be time traveling.  Read below for the details…

(Event ID 1054)
Windows cannot obtain the domain controller name for your computer network. (An unexpected network error occurred.). Group Policy processing aborted.

The customer explained to us that if he removes one of the Hyper-V virtual processors from his Windows Server 2003 Guest, the issue goes away. Based on this statement we asked the customer to gather a userenv log while forcing a group policy refresh with the additional virtual processor enabled, and this what we saw during the initial ping test before we process group policy:

USERENV(15c.858) 15:55:09:080 PingComputer: First time: 2069
USERENV(15c.858) 15:55:09:080 PingComputer: Second time: 2069
USERENV(15c.858) 15:55:09:080 PingComputer: First and second times match.
USERENV(15c.858) 15:55:09:080 PingComputer: First time: 2069
USERENV(15c.858) 15:55:09:080 PingComputer: Second time: -2069
USERENV(15c.858) 15:55:09:080 PingComputer: First time: -2069
USERENV(15c.858) 15:55:09:080 PingComputer: Second time: 0

We have a knowledgebase article that pertains to this issue on servers that uses dual-core or multiprocessor AMD Opteron processors:

938448 A Windows Server 2003-based server may experience time-stamp counter drift if the server uses dual-core AMD Opteron processors or multiprocessor AMD Opteron processors

http://support.microsoft.com/default.aspx?scid=kb;EN-US;938448

Now in the case of our customer they were not running an AMD Processor server so they felt this resolution did not apply to them. Even though the article did not apply to the type of processor in their servers, the behavior was identical so we applied the resolution outlined in the knowledgebase article and this resolved the customer’s issue. I am in the process of having a knowledgebase article created to specifically address this issue with Windows Server 2003 virtual machines running in Hyper-V.

So we did a little digging and found the following blog post from Tony Voellm, who is a Principal Software Test Engineer in the Windows Kernel development team:

Negative ping times in Windows VM’s – whats up?

http://blogs.msdn.com/tvoellm/archive/2008/06/05/negative-ping-times-in-windows-vm-s-whats-up.aspx

The following is from the above blog post:

If you see negative ping times in multiprocessor W2k3 guest OSes you might consider setting the /usepmtimer in the boot.ini file.

The root issue comes about from the Win32 QueryPerformanceCounter function.  By default it uses a time source called the TSC.  This is a CPU time source that essentially counts CPU cycles.  The TSC for each (virtual) processor can be different so there is no guarantee that reading TSC on one processor has anything to do with reading TSC on another processor.  This means back to back reads of TSC on different VP’s can actually go backwards. Hyper-V guarantees that TSC will not go backwards on a single VP.

So here the problem with negative ping times is the time source is using QueryPerformanceCounter which is using TSC.  By using the /usepmtimer boot.ini flag you change the time source for QueryPerformanceCounter from TSC to the PM timer which is a global time source.

Ask the Directory Services Team : Userenv 1054 events as a result of time-stamp counter drift on Windows Server 2003 guests running in Hyper-V

DPM v 3

I just watched a webcast on DPM v3 and thought I would share some of what I got from that.

In the last 18 months, DPM 2007 (v2) delivered application protection for Exchange ,SQL Server, SharePoint and virtualization environments running Virtual Server and Hyper-V.  Disaster recovery with Iron Mountain, Local Datasource Protection and Client backups have also come out through DPM 2007, its first feature update and Service Pack 1.  Now it is time to show what is coming next for DPM. 

A few top line items are support for the following:

• support for Exchange 14, and more granular restore
• protect the entire SQL instance, and auto discover new DB’s
• protect 1000’s of DBs per DPM server
• End User Recovery by the SQL Admin (role based access from the DPM console)
• Office 14
• AD appears as a data source in DPM UI
• Image restore from centrally managed DPM server – executed locally
• Support for Windows guest on VMware hosts
• SAP running on MS SQL

and some other improvements:

• up to 100 servers, 1000 laptops, 2000 databases per DPM server
• management pack updates
• automatic re-running of jobs and improved self-healing  —  This is a huge one in my book
• auto protect new sources for SQL and MOSS
• improved scheduling capabilities
• one click DPM DR failover and failback
• continued support for SAN (scripts/whitepapers)

platform requirements:

• DPM Server must be 64-bit Windows Server 2008 R2
• Integration capability with Windows EBS 2008 R2