Problems with Large Files (multi-gigabyte)

First, thanks to anyone who reads this. My issue is not trivial and is explained in depth.

I'm using Windows 7 Ultimate 64-bit, and my problem applies to that system.

This problem is stumping me but I've been able to gather evidence of the onset of the problem, an experiment which is repeatable. It seems either Windows Update and/or MSE delivers a payload that interferes with proper access to large files the first time they access the Internet and transfer any data.

(1) First: Signs of a problem

I had problems with a program that stores data in its program directory using the zip format. Integrity checks of the files would fail (the program internally computes some sort of hash or checksum). Since the files were large I decided to test this issue.

(2) Second: Test preparation on a different computer (Windows XP 32-bit)

I made a large zip file on another computer (about 3.8GB) and computed its MD5 hash. I tested the integrity of the zip file and everything extracted okay. I burned the file to a DVD, and computed the hash of the file on the DVD (it matched the previous hash of the file on the disk drive).

(3) Third: Testing MD5 file hashes on Windows 7

I took the DVD to the Windows 7 computer and computed the MD5 hash of the file on the DVD. It was different. Interestingly, on each reboot of the computer, the computed MD5 would change. When trying to extract the zip file, Windows own zip handler would encounter problems, WinZip would indicate CRC errors, and WinRar would error as well.

(4) Fourth: Safe mode and diagnostic Startup

When booting into safe mode, MD5 hashes on large files compare exactly to the computed hashes on the Windows XP 32-bit machine. In Diagnostic startup, the problem returns. (I have boot logs of the two modes and can post them if that helps).I can't see any difference that stands out and I'm thinking maybe, just as a matter of fact, more is being loaded than what's reported in the boot logs.

(5) Fifth: The onset of the problem on a clean installation

I reinstalled the operating system several times, each time zeroing out my disks before installation. I have successfully repeated the onset of the problem to occur when Windows Update and/or MSE access the Internet for the first time. The problem occurred after Windows Update alerted me to pending updates for the first time (notification but no download) and when MSE downloaded its first set of new definitions. These things happened concurrently. I'm in the process of trying to isolate this issue further.

Note: Before I connected to the Internet for the first time, I installed MSE and download the "standalone" definitions manually on another computer and transferred them to the Windows 7 machine and installed the definitions. This after this, MD5 hashes computed fine, it was only after Windows Update and MSE access the Internet and downloaded data that the problem occurred.

 (6) Sixth: Other Info

Using the Command Prompt to copy a large file from a DVD to the hard disk of the affected computer the line "copy largfile.ext c: /v" returns with "ERROR Verify - c:largefile.ext", and "1 file(s) copied." Through Windows Explorer, a copy operation throws no errors. THIS PROBLEM HAPPENS WITH ANY LARGE FILE I TEST NOT JUST ZIP FILES. I'm not sure where the boundary is in file size that this problem occurs. It's not clear to me that the files are actually corrupted as when I copied large zip files files from my Windows 7 machine to an external drive for backup, and checked those files on another computer, the zip files were fine. I think something is intercepting "correct" access to the large files.

(7) Seventh: When did the problem first happen

I started noticing issues about 3 to 4 weeks ago.

(8) Eight: Another test

This testing has consumed a lot of time. Now that I have a repeatable issue, I'm going to disable the Windows Update Service before I connect to the Internet for the first time, after reinstalling, and see if I can narrow the behavior specifically to either Windows Update or MSE. It may be that KB2533552 could help, I’m not sure. Testing takes a lot of time. Something like “Windows7FirewallControl” may help isolate MSE and Windows Update until this issue is resolved.

-----

I think this is a serious issue as any program that uses integrity checks on large files would be affected. And, anyone wishing to use large archives that perform integrity checks, like zip files, would be affected too -- assuming my experience is not an isolated incident. I believe it is not, as a search over the Internet shows problems with large zip files in an ongoing issue.

Any help would be appreciated.

Best Regards,

Davis

 

 

Question Info


Last updated March 24, 2018 Views 2,689 Applies to:

* Please try a lower page number.

* Please enter only numbers.

* Please try a lower page number.

* Please enter only numbers.

Hi,

Go back to the point before the problem started after you reinstalled the operating system using the system restore and install the updates one at a time to identify the conflict and let us know which update caused the issue.

 

Refer the steps in the site mentioned below for steps to perform system restore:

http://windows.microsoft.com/en-US/windows7/products/features/system-restore.

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

With what file system is the drive / partition formatted?
Gerry
Stourport-on-Severn, Worcestershire, England
Enquire Plan Execute

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

Hi Deepak,

Thanks for responding! My "test bed" starts with a clean install from the Windows 7 retail DVD with no connection to the Internet. After the installation, I made an image of my disc using Windows 7 backup. I use this back up to restore my installation each time the system is no longer able to compute consistent MD5 hashes for large files. (I use the System Repair disc + the backup).

During the course of testing, once I found that the problem occurs after connecting to the Internet for the first time, then I began testing with (1) the clean retail DVD install as well as (2) the clean retail DVD install + Windows 7 SP1 (applied using a DVD created from the downloadable SP1 iso, which was created on another computer, so as not to connect to the Internet with the computer in question).

The thing is the problem happens before selecting any patches to install from Window Update. By the time I'm connected to the Internet and simply notified of updates, the problem with incorrect MD5 hashes surfaces at some point.

So I'm testing now to see if the problem comes right after my computer makes its first contact with Windows Update or if a reboot is required after first contact with Windows Update for the problem to surface. And I'm also testing first contact with MSE definition updates or first contact plus a reboot.

At this time I don't think the problem is necessarily any update in particular.

Still testing, will get back.

Best Regards, Davis

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

Hi, my system drive is formatted with NTFS (2x500GB RAID 0) and I have a third drive, also formatted with NTFS, on which I store a backup of a clean DVD install of Windows 7 (made with no connection to the Internet). The clean install doesn't have any problem producing consisent MD5 hashes for large files.

Best Regard, Davis

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

What was the process for creating the clean DVD install of Windows 7.

In Windows 7 use Ctrl+Shift+Esc rather than Ctrl+Alt+Del. It gets you to Task Manager quicker. When the computer locks select Task Manager, the Performance tab, Resource Monitor, and Memory tab. What are the figures for Hardware Reserved, In Use, Modified, Standby and Free?

How much RAM is installed?

Gerry
Stourport-on-Severn, Worcestershire, England
Enquire Plan Execute

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

Hi Gerry,

 

I appreciate the follow up on this issue!

 

>> What was the process for creating the clean DVD install of Windows 7.

-----

 

After trying to figure out what process could be causing the issue (and me coming up with nothing), I realized I needed to start from scratch. I backed up my data, and also realized I would need some consistent test scheme so I could narrow down the issue on a step-by-step basis.

 

I disconnected from Internet access. Since my data was already backed up on a removable drive, I zeroed out my Raid array and the third internal disk I use for backups. Before I zeroed out the RAID array, I dismantled it in the BIOS and RAID setup so that the disks were seen as two single drives. To zero out each disk I booted into DOS mode and used utilities to ensure I wiped any Host Protected Area and DCO as well, because I wanted to make sure that I caught any areas malicious code may hide.

 

I powered down/unplugged the computer and disconnected the third backup drive.

 

I plugged-in/powered up the computer, reset the RAID array, restarted, and boot to the Windows retail DVD to start installation.

 

After the DVD install, I tested to make sure MD5 hashes were consistent with some large files and that the files copied without verify errors in the Command Prompt window. (This tested out okay.) I had Windows create a "System Repair Disc". Then I powered down/unplugged the computer, connected the backup drive, powered up the computer, and boot into Windows. I formatted the (third) backup with NTFS created an image of my system drive using Windows backup.

 

Now, when the issue occurs again, I'll use the System Recovery Disc to restore the disc image on the backup drive, which then gets me to an error-free state. This helps me to further narrow down the occurrence of the issue each time, as I get time to test.

I am getting closer to the exact onset of the issue, but once I find "it" I want to give exact steps to reproduce the issue so that the information is out there. It’s still unclear, but it’s either Windows Update, MSE definition updates, or (now I’m leaning toward this) ActiveX controls installed by Creative AutoUpdate. The thing that’s making the onset of problem hard to narrow down is that it’s unclear to me how soon after “something” is downloaded does the problem arise -- i.e. is a restart needed, etc. A search of the Internet shows that there were issues with Creative AutoUpdate, but also indications such problems have been fixed. The issue with AutoUpdate was an ActiveX vulnerability. With my issue it seems some program stays resident (and hidden) in the system even after disconnecting from the Internet and the system is restarted. Such a “hook” into the operating system as I’m seeing is annoying and somewhat flabbergasting. (How many people, like me, have an OEM driver disc from Creative that installs AutoUpdate? It wouldn’t be what’s on the driver disc that would be the problem, but what happens after connecting to the Internet and using AutoUpdate – still the issue is foggy, for me, at the moment.)

 

>>>In Windows 7 use Ctrl+Shift+Esc rather than Ctrl+Alt+Del. It gets you to Task Manager quicker. When the computer locks select Task Manager, the Performance tab, Resource Monitor, and Memory tab. What are the figures for Hardware Reserved, In Use, Modified, Standby and Free?

-----

 

Hardware Reserved: 3MB

In Use: 588MB

Modified: 15MB

Standby: 7585MB

Free: 1MB

 

These numbers are after a clean install on a system not experiencing the issue. I'll see if these numbers change when the system is affected again. I’m still testing as I get time.

 

>>>How much RAM is installed? 8GB DDR2

 

This system is a Dell XPS-630i, Core 2 Quad Q9550 on an nVidia nForce 650i motherboard

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

Creative AutoUpdate -Do you have the latest version issued in November2011 . I found researching this software suggested very buggy software.

Standby: 7585MB -What is causing this monster?

Select Start, Control Panel. Action Center, Click on the Arrow down to the right of Maintenance and then on View reliability history. In the graph you will see red orbs, Click on one and it gives details and to the right is another link saying View technical details. Can you copy and paste the description in a further message. Repeating reports are more relevant than one off problems.

Gerry
Stourport-on-Severn, Worcestershire, England
Enquire Plan Execute

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

Hey Gerry,

 

Thanks for the reply.

 

At the moment I'm testing the operation of Creative AutoUpdate. The version of AutoUpdate that comes with my Dell OEM Creative driver disc doesn't cause the problem (in and of itself). Once I connect my computer to the Internet (for the first time) and run AutoUpdate, I'm prompted to download three ActiveX controls (each contained in separate *.cab files) in the following order: CTSUEng.cab, CTPID.cab, and CTPIDPDE.cab. Just running AutoUpdate and installing these ActiveX controls alone (but then not downloading or installing any Creative drivers or software updates) doesn't cause the problem (even after restarting my computer twice, testing after each restart). Now, I'm looking at individual downloads of Creative drivers/software. It seems AutoUpdate does what it's suppose to do (detect hardware and offer relevant updates) -- but is it doing something extra it should not? And/or are the downloadable drivers and software doing something they should not? It's unclear to me for now, but I feel like I’m getting close.

 

In terms of the Standby (Memory) being at 7585MB, I didn't think that was surprising -- I thought this was an indication Windows was attempting to make as much memory as possible readily available. I did some searching and found this article:

 

http://www.zdnet.com/blog/bott/windows-7-memory-usage-whats-the-best-way-to-measure/1786

 

I did look at the Reliability History as you suggested (by the way, thanks for pointing this out – it looks like a good tool), but I did not see any red orbs. I did see blue orbs with an “i” in them – they simply indicated “Successful driver installation” in the window below the graph.

 

Even if I find a definitive smoking gun that's Creative Labs related, I'm still left with the question why it was only 3 to 4 weeks ago I started having issues, especially if their last update for anything pertaining to my system was November 2011.

 

Thanks very much for the continued help as well as the information about AutoUpdate. I think I'm getting close to something – as I have the time to test this stuff.

 

Best Regards, Davis

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

Standby: 7585MB -What is causing this monster?

I stiil think it's a monster. What is the software making this figure so large? Whatever it is has a potential to gobble RAM!

Gerry
Stourport-on-Severn, Worcestershire, England
Enquire Plan Execute

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

Problem Solved

 

I found the problem: the Creative Labs driver for the SB X-Fi (Titanium, PCIe) version 2.17.0008.1 dated July 2010. Not only is this driver available through Creative AutoUpdate, but (much to my chagrin) is also available through Windows Update.

 

Here is a problem description and work-around that I hope will help others.

 

Possibly Affected Systems: Windows 7 64-bit equipped with the SB X-Fi PCIe add-on card and Creative Labs driver version 2.17.0008.1 dated July 2010.

 

Problem: Large zip-archives fail CRC checks, incorrect MD5 hashes are generated for large files, and when large files are copied (using the verify option) within the Command Prompt window (cmd.exe), an error message is generated “ERROR Verify –“.  This behavior does not occur in “Safe Mode” but will reappear in “Diagnostic Mode” as well as when Windows is loaded normally. A large file may be considered to be 3GB or more in size. It is unclear what the file size boundary is for this behavior to manifest.

 

Solution: Uninstall the current Creative Labs driver and install an older version.

 

(Step 1)

 

The 2.17.0008.1 July 2010 driver downloaded by Creative Labs’ AutoUpdate (XFTI_PCDRV_LB_2_17_0008.exe) and Windows Update install the following items (in addition to the driver) which can be uninstalled:

 

Creative Audio Control Panel

Creative Software AutoUpdate

Creative Sound Blaster Properties x64 Edition

OpenAL

 

(Step 2)

 

Remove the driver:

 

In the search box type "Device Manager", select and open it.

 

In the Device Manager window, expand the section labeled "Sound, video and game controllers"

 

Highlight the entry "Creative SB X-Fi"

 

Right-click on the entry and select "Properties" from the context menu.

 

Click on the "Driver" tab of the “Creative SB X-Fi Properties” window.

 

You will see:

Driver Provider: Creative

Driver Data: 7/7/2010

Driver Version: 6.0.1.1348

Digital Signer: Microsoft Windows Hardware Compatibility Publisher

 

Click the "Uninstall" Button

 

In the "Confirm Device Uninstall" message box that appears, check the box that is labeled "Delete the driver software for this device".

 

Click the "Okay" button.

 

A message box will indicate the driver is being uninstalled. After the driver is uninstalled, reboot the computer.

 

Microsoft Windows 7 by default will load the "High Definition Audio Device" driver which will enable sound output with SB X-Fi sound card.

 

(Step 3)

 

As an alternative, use the SB X-Fi driver version 2.16.0003.6 dated 18 May 2009 which does not cause the behavior noted in the “Problem” section. This driver may be available on the Creative-supplied driver disc bundled with your SB X-Fi add-on card.

 

-----

Although this was my specific issue, I hope that this information may help anyone with the same (or a similar) issue.

 

As I post this I want to note that my specific system is:

 

Windows 7 Ultimate 64-bit

Dell XPS-630i

Intel Core 2 Quad Q9550

nVidia nForce 650i SLI motherboard (Dell OEM version)

8GB DDR2 RAM

RAID 0 (2x500GB) - System Partition

160GB - internal backup drive

Did this solve your problem?

Sorry this didn't help.

Great! Thanks for marking this as the answer.

How satisfied are you with this reply?

Thanks for your feedback, it helps us improve the site.

How satisfied are you with this response?

Thanks for your feedback.

* Please try a lower page number.

* Please enter only numbers.

* Please try a lower page number.

* Please enter only numbers.