I see see ECC

I wanted to create a well written post about using ECC RAM, but I just don't have the time.  Therefore you're getting a condensed version that's low on wit and high on content (but not necessarily facts or truths).  Nothing I write here is original (except for any inaccuracies), it's all been plagiarized from the Internet.  Like I said before I don't have the time to properly credit all of my sources and I'm sure I don't remember them all either.

Why create this post?  Because I couldn't find a good website that showed how ECC should be configured for an above average home ECC user like me.  I've found the answers to some of my questions and have taken educated guesses at what other settings should be.  I hope this post can either help someone else out, or at least get someone to point out where I can find an easy to understand authoritative source on these settings.

Basically ECC RAM is able to correct a single bit memory error on the fly (and report on double bit memory errors).  In theory this gives you better system uptime (less BSODs).  The downside to ECC RAM is that you take a 0.5-2% performance hit (depending on the type of app you're running) and it costs more.

According to this article the more ECC features you enable the more of a performance hit you take.  So if you're a gamer you might want to enable just the basic ECC checking.  Whether it costs a little or a lot more depends on timing and how savvy of a shopper you are.

More details on what ECC RAM is and the theory behind it is easily found via Google.  One interesting site is this summary of Google's 2.5-year study of DRAM error rates.

What do you need to run ECC RAM?  Well for starters you must have ECC RAM.  Most RAM is not ECC RAM.  ECC RAM is special.  In my "Gigabyte GA-790FXTA-UD5" motherboard I use this RAM:
CT2KIT25672BA1339 - 4GB kit (2GBx2), 240-pin DIMM , DDR3 PC3-10600 from Crucial.com
http://www.crucial.com/store/partspecs.aspx?IMODULE=CT2KIT25672BA1339
    * Module Size: 4GB kit (2GBx2)
    * Package: 240-pin DIMM
    * Feature: DDR3 1333 (PC3 10600)
    * Specs: DDR3 PC3-10600 • CL=9 • Unbuffered • ECC • DDR3-1333 • 1.5V • 256Meg x 72 

You also need a motherboard that supports ECC RAM.  AFAIK all (or maybe all non-laptop) AMD CPUs support ECC RAM (the ECC RAM controller is actually in the CPU), so every AMD motherboard should support ECC RAM.  Some motherboard manuals will show you all the ECC settings.  However my GA-790FXTA-UD5 manual didn't even mention ECC RAM.  Its website did say that ECC was supported, so I took a gamble and bought this motherboard (which paid off).  My CPU is a "AMD Phenom II X4 905e Deneb 2.5GHz 4 x 512KB L2 Cache 6MB L3 Cache Socket AM3 65W Quad-Core Processor".

So now that you've got an AMD CPU, AMD motherboard and ECC RAM what do you need to do next?  Just slap it all together and turn your computer on and everything should work.  However it's not quite that simple.  Every motherboard I've seen has the ECC function disabled by default.

All (AMD) motherboard ECC settings should be similar.  My ECC settings are found under "MB Intelligent Tweaker (M.I.T.)->DRAM Configuration".  My (mainly plagiarized) definitions of these settings are:

• DRAM ECC enable =        Turn on ECC
• DRAM MCE enable    =        Generate Machine Check Exception logs
• Chip-Killl mode enable =     Can correct some multibit errors.  Aka "4-Bit ECC"
• DRAM ECC Redirection =        When a single bit ECC error is found write the corrected data back to RAM (i.e. scrub just the location with the ECC error)
• DRAM background scrubber =     Interval between main memory scrubs (64 bytes)
• L2 cache background scrubber =    Interval between main L2 cache scrubs (one single L2 cache line tag address)
• DCache background scrubber =    Interval between data cache scrubs (64 bits)

Below is a list of how I've configured my ECC settings.  Below each setting is a comment why I made each choice.

• DRAM ECC enable            Enabled
• Why have ECC RAM if you're not going to use it?
• DRAM MCE enable            Enabled
• MCE logs let you keep track of how many ECC errors you have (i.e. how many times ECC silently corrected a single bit error and saved you from a crash)
• Chip-Killl mode enable        Enabled
• Correcting multi-bit errors sounds good to me.
• DRAM ECC Redirection        Enabled
• See my section below about scrubbing
• DRAM background scrubber    10.49ms (scrub 4GB in ~187 hours)
• See my section below about scrubbing
• L2 cache background scrubber    Disabled
• DCache background scrubber    Disabled
• I don't scrub either of my caches since this PDF says it's not worth it.

Back to ECC 101.  When your CPU reads a section of RAM with an single bit error ECC automatically corrects it (and generates a MCE log to tell ya that it corrected it).  By default this corrected RAM value is only given to the CPU, it's not written back to RAM.  I.e. the bad RAM value is not changed!  By turning on "DRAM ECC Redirection" the good (corrected) value is written back to RAM (over the bad value) to correct the bad entry in RAM.  Why they don't do this by default?  I supposed this takes a microsecond longer to do.

The other reason you wouldn't have to correct ECC errors found in RAM is that you can turn on scrubbing.  Scrubbing is basically a background process (normal ECC checks only RAM values that your CPU has requested) that goes through all of your RAM, reading small 64 byte sections of RAM and checking it for ECC errors.  If it finds an error it corrects the error and writes the updated (and corrected) value back to RAM.

The main reason to scrub your RAM is to avoid getting two bit errors.  I.e. a single cosmic ray flips one of your bits.  If you leave your computer on 24x7 and you don't have "DRAM ECC Redirection" enabled you will eventually have another cosmic ray flip another bit in the same "ECC area" as your first bit flip.  With a 2 bit error ECC can't fix it, only report it (and freeze your computer).  In summary , scrubbing greatly decreases the odds that you'll get a two bit ECC error.

Note that with Windows Vista and Windows 7 by default your computer sleeps when you shut it down.  In sleep mode your RAM is still powered and therefore still capable of taking errors.  I.e. as far as your RAM is concerned,if you use sleep mode your computer is on 24x7 and you should be scrubbing your RAM.  If you truly power off your computer or reboot it regularly (like Windows Update is known to do) you probably don't need to scrub your RAM.

I configure my system to scrub 64bytes (the amount of RAM scrubbed is not configurable) of RAM every 10.49ms.  This will scrub my entire 4GB of RAM in ~187 hours.  That may seem like a long time, but you need to remember that ECC errors are pretty rare anyway.  Since you really only need to be scared of two bit errors, the odds of getting any two bit errors are pretty slim.  Additionally since I have "DRAM ECC Redirection" enabled I'm in effect running another scrubber.

Another random source I found useful is the "AMD Hammer Family Processor BIOS and Kernel Developer's Guide."  It's not the easiest read, but there's some good info in there.


Now you know how/why I configure my ECC RAM to work for me.  I'm not an authoritative source on ECC RAM.  My settings may not be the best for you.  They may not even be the best for me!   With that dire warning aside, there is one more thing I should mention.

How do you monitor your ECC RAM?  How do you know that it's actually working and correcting errors?  Via the magic of MCE (Machine Check Exception) logs of course!

In Linux the MCE logs are in "/var/log/mcelog".  Fedora 12 (64 bit) writes the MCE logs to disk once an hour in this cron job "/etc/cron.hourly/mcelog.cron". If you're having system stability issues (i.e. your computer freezes before the MCE logs get written to disk) you could delete the hourly cron job and instead add this job to "/etc/cronttab" so that the MCE logs would get written out every minute:
* * * * * root /usr/sbin/mcelog --ignorenodev --filter >> /var/log/mcelog
This site has more good Linux info:

Here's what an corrected ECC error  log entry might look like from "/var/log/mcelog":
MCE 0
Fri Oct 12 22:11:47 1492
HARDWARE ERROR. This is *NOT* a software problem!
Please contact your hardware vendor
CPU 0 4 northbridge MISC c008000101000000 ADDR afe00
  Northbridge RAM Chipkill ECC error
  Chipkill ECC syndrome = b542
       bit46 = corrected ecc error
       bit59 = misc error valid
  bus error 'local node origin, request didn't time out
             generic read mem transaction
             memory access, level generic'
STATUS 9c214000b5080813 MCGSTATUS 0
CPUID Vendor AMD Family 16 Model 4
WARNING: SMBIOS data is often unreliable. Take with a grain of salt!
DDR DIMM 1333 Mhz Synchronous Width 64 Data Width 64 Size 2 GB
Device Locator: DIMM3
Bank Locator: BANK3
Manufacturer: Manufacturer03
Serial Number: SerNum03
Asset Tag: AssetTagNum3
Part Number: ModulePartNumber03

Note that this Linux mcelog reports the error in "DIMM3/BANK3".  It's important to know that Linux starts counting your memory banks at 0.  So if you have four RAM slots Linux numbers them from 0-3.  I.e. "DIMM3/BANK3" is actually your 4th memory bank.  As you'll see below Windows (unlike Linux) starts counting at 1, so this same error in Windows would be reported as coming from "Bank Number: 4".

In Windows you'll get a log like this in your system event log:
Log Name:      System
Source:        Microsoft-Windows-WHEA-Logger
Date:          10/12/1492 11:20:48 AM
Event ID:      19
Task Category: None
Level:         Warning
Keywords:      
User:          LOCAL SERVICE
Computer:      hal
Description:
A corrected hardware error occurred.  
Error Source: Corrected Machine Check
Error Type: Bus/Interconnect Error
Processor ID Valid: Yes
Processor ID: 0x0
Bank Number: 4
Transaction Type: N/A
Processor Participation: Local node responded to the request
Request Type: Generic Read
Memory/Io: Memory
Memory Hierarchy Level: Generic
Timeout: No

Note how it's even nice enough to tell you which memory bank had the error?  If you start getting a lot of MCE logs from one bank, maybe you simply have a stick of RAM going bad, get it replaced.

In Windows you can use the "AMD MCAT Machine Check Analysis Tool" to get more details on the MCE logs.
Machine Check Analysis Tool (MCAT) is a command line utility that takes a Windows System Event Log (.evt) file as an argument and decodes the MCA Error logs into human readable format. MCAT can alternatively take as an argument the raw register hexadecimal values from an MCE Error.

When I used this on Vista it couldn't read the event log directly. However I could manually give it the numbers from the event log XML details for it to decode. I used commands like this:
mcat /cmd 4 0xf4782000e0080a13 0x90b2c700 0x2231eaa00000000
mcat 4 0x94744000eb080a13 0x30a4bc0 0x226dbd200000000
mcat 4 0x94214000b5080813 0x15ba7c00 0x225385200000000

Some example mcat output:
mcat 4 0x94214000b5080a13 0x3a1d8500 0x1be542027ef95c
Processor Number  : 0
Bank Number       : 4
Time Stamp    (0x): 00000000 00000000
Error Status  (0x): 94214000 B5080A13
Error Address (0x): 00000000 3A1D8500
Error Misc    (0x): 001BE542 027EF95C
Status Bit Decode :
   Correctable ECC error
   Error address valid
   Error enable
   Error valid
Error Code    (0x): 0A13
   Error Type - Bus
   Participation Processor (PP) - Local node responded to the request (RES)
   Timeout (T) - Request did not time out
   Memory Transaction Type (RRRR) - Generic read (RD)
   Memory or IO (II) - Memory Access (MEM)
   Cache Level (LL) - Generic, includes L3 cache (LG)
Bank 4 North Bridge Errors:
   ECC Error - DRAM ECC error detected in the NB.
   Error address at 929 MB rage
   Syndrome  (0x): B542
      Data   (0x): 1F
      Bitmap (0x): 02
      Error on bit(s) (dec): 125 
   Address decode: 000000003A1D8500
      Node ID: 0
      Channel Select: 1
      Chip Select: 3



Lastly, in Windows it's helpful to create a custom log view that shows serious errors. Then by using this custom log view you can quickly see if you had any serious errors logged. My custom view is set up like this:

• Logged: Any time
• Event logs: System
• Event Sources:  BugCheck, eventlog, Eventlog, MemoryDiagnostics-Results, MemoryDiagnostics-Schedule, StartupRepair, WHEA-Logger, amdsata
• Includes Event IDs: 19,1001,6008,1002,1137,1208,1213,1101,1201,1103,11

March 2015 update, a few useful ECC pages I've come across lately:

• How to Check ECC RAM Functionality - Puget Custom Computers
• Advantages of ECC Memory - Puget Custom Computers
• ECC and REG ECC Memory Performance - Puget Custom Computers

30JAN2019 update.  I'm now running CentOS 7.6 on my same ancient hardware.  With CentOS 7 mcelog no longer supports my AMD CPU.  Per this article the mcelog package isn't used by AMD processors.  If you have mcelog installed with an AMD processor you'll get messages like these:
    grep mce /var/log/messages
    mcelog: ERROR: AMD Processor family 16: mcelog does not support this processor.  Please use the edac_mce_amd module instead.#012: Success
    : CPU is unsupported
The easiest way to get rid of these messages is to simply remove the mcelog package:
yum erase mcelog

And install this instead:
yum install edac-utils

To diagnose ECC errors with EDAC run these commands:
edac-util
edac-util --report=full
grep "\[Hardware Error\]\|EDAC" /var/log/messages


Some sample output below from my PC shows my DIMM in row 1 is giving me the CE (correctable errors).  Since they were corrected by my ECC RAM they didn't cause a problem.  I was able to replace my bad stick of RAM before things got bad (i.e. before I had any UE/Uncorrectable Errors).

Note that EDAC starts counting at 0, so csrow#1 actually corresponds to what my motherboard has labeled as DIMM slot 2.

Another thing to note is that the [Hardware Error] entries in /var/log/messages as also printed to any logged in terminal session (which is how I first became aware of this issue).
 
[jim@c64 ~]# edac-util
mc0: csrow1: mc#0csrow#1channel#1: 1 Corrected Errors

[jim@c64 ~]$ edac-util --report=full
mc0:csrow0:mc#0csrow#0channel#0:CE:0
mc0:csrow0:mc#0csrow#0channel#1:CE:0
mc0:csrow1:mc#0csrow#1channel#0:CE:0
mc0: csrow1: mc#0csrow#1channel#1: 1 Corrected Errors
mc0:csrow1:mc#0csrow#1channel#1:CE:1
mc0:csrow2:mc#0csrow#2channel#0:CE:0
mc0:csrow2:mc#0csrow#2channel#1:CE:0
mc0:csrow3:mc#0csrow#3channel#0:CE:0
mc0:csrow3:mc#0csrow#3channel#1:CE:0

[root@c64 ~]# grep "\[Hardware Error\]\|EDAC" /var/log/messages
Jan 30 05:16:59 c64 kernel: EDAC MC: Ver: 3.0.0
Jan 30 11:17:05 c64 kernel: AMD64 EDAC driver v3.4.0
Jan 30 11:17:05 c64 kernel: EDAC amd64: DRAM ECC enabled.
Jan 30 11:17:05 c64 kernel: EDAC amd64: F10h detected (node 0).
Jan 30 11:17:05 c64 kernel: EDAC amd64: MC: 0:  2048MB 1:  2048MB
Jan 30 11:17:05 c64 kernel: EDAC amd64: MC: 2:  2048MB 3:  2048MB
Jan 30 11:17:05 c64 kernel: EDAC amd64: MC: 4:     0MB 5:     0MB
Jan 30 11:17:05 c64 kernel: EDAC amd64: MC: 6:     0MB 7:     0MB
Jan 30 11:17:05 c64 kernel: EDAC amd64: MC: 0:  2048MB 1:  2048MB
Jan 30 11:17:05 c64 kernel: EDAC amd64: MC: 2:  2048MB 3:  2048MB
Jan 30 11:17:05 c64 kernel: EDAC amd64: MC: 4:     0MB 5:     0MB
Jan 30 11:17:05 c64 kernel: EDAC amd64: MC: 6:     0MB 7:     0MB
Jan 30 11:17:05 c64 kernel: EDAC amd64: using x4 syndromes.
Jan 30 11:17:05 c64 kernel: EDAC amd64: MCT channel count: 2
Jan 30 11:17:05 c64 kernel: EDAC MC0: Giving out device to 'amd64_edac' 'F10h': DEV 0000:00:18.3
Jan 30 11:17:05 c64 kernel: EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.2' (POLLED)
Jan 30 11:21:58 c64 kernel: mce: [Hardware Error]: Machine check events logged
Jan 30 11:21:58 c64 kernel: [Hardware Error]: Corrected error, no action required.
Jan 30 11:21:58 c64 kernel: [Hardware Error]: CPU:0 (10:4:3) MC4_STATUS[-|CE|MiscV|-|AddrV|CECC]: 0x9c00c00041080813
Jan 30 11:21:58 c64 kernel: [Hardware Error]: Error Addr: 0x000000037ceb8980
Jan 30 11:21:58 c64 kernel: [Hardware Error]: MC4 Error (node 0): DRAM ECC error detected on the NB.
Jan 30 11:21:58 c64 kernel: EDAC MC0: 1 CE on mc#0csrow#1channel#1 (csrow:1 channel:1 page:0x37ceb8 offset:0x980 grain:0 syndrome:0x4101)
Jan 30 11:21:58 c64 kernel: [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: SRC (no timeout)

Hunting the elusive great white system freeze

***********************************************************
***********************************************************
                                        31JAN2010 Update

My computer ended up not even being able to get past the BIOS POST screen. I ended up RMAing my Asus M4A79T Deluxe. While I was waiting for my Asus board to come back I bought a "Gigabyte GA-790FXTA-UD5" to use instead. The GA-790FXTA-UD5 has worked flawlessly since I installed it about three weeks ago. When my Asus board gets back I plan on eBaying it right away.

Morale of the story? When the going gets tough quit. The sooner the better. I wish I would've just RMAed the Asus board right away. On the plus side I do love my new Gigabyte board.

***********************************************************
***********************************************************
                                     Original Post

For a shorter version of this blog entry see my Asus forum post here.

This concise tale of woe and misery describes some of the suffering I endured to isolate an intermittent system freeze issue. The primary purpose of this post is to generate sympathy for me (and to get my bi-weekly blog entry online in time). Perhaps some will be so moved by my tragic story that they'll send monetary gifts to help ease my pain. Any gifts in excess of $100 will be great appreciated, don't get cheap on me now.

The troubleshooting steps described in this post may also help give some more troubleshooting ideas to others who are suffering from from similar computer instability issues. If my suffering can help even one person fix their computer more quickly, than all of my suffering will not have been worth it. Even if I help a million people with their computers, it still won't have been worth it. Slaying this instability beast was a horrible painful time drain. This is the kind of torture I would wish on my worst enemies. Only many large gifts of money could even begin to help ease my pain.

It all started July 2009. My wife's complaining about her system instability has seemed to increase with the frequency of her lock ups. I try to comfort her by repeatedly telling her that I'd be happy to install Fedora Linux on her computer to make all of her problems go away. Although she acknowledges that running an OS that supports about 1% of her apps should greatly improve her computer stability, she illogically (and repeatedly) turns down my generous offer of a fresh install of Linux. Perhaps if I had offered her Ubuntu instead? Alas it's too late for that now.

Our household computer standard is to refresh our desktop machines every 10 years. We're only about a year away from our desktops' tenth birthday, so I decided to bend the rules a bit and go ahead and build her a new PC now. I figured if I bought quality components for a change that building a new PC should be faster than troubleshooting her ancient desktop. I'm not sure that a simple re-install would've fixed her old PC, some hardware may have been going bad too.

In all my years of building computers with budget parts I've never had a major issue. Ironically the one time I shell out some extra green for "quality" parts I suffer the worse hardware problems I've ever had! My computer build part list is listed below. My two primary goals for this PC were:
1. Rock solid stability (ha ha ha ha ha)
2. Quiet (and low power)


[Parts list]
-Antec Sonata Designer 500 case
-Noctua NF-S12B ULN 120mm case fan
-Corsair VX450W power supply
-Asus M4A79T Deluxe with BIOS 2304 (I also tried BIOSes 2002-2205, all had the same symptoms)
-AMD Phenom II X4 905e Deneb 2.5GHz 4 x 512KB L2 Cache 6MB L3 Cache Socket AM3 65W Quad-Core Processor
-CT2KIT25672BA1339 - 4GB kit (2GBx2), 240-pin DIMM , DDR3 PC3-10600 from Crucial.com
http://www.crucial.com/store/partspecs.aspx?IMODULE=CT2KIT25672BA1339
* Module Size: 4GB kit (2GBx2)
* Package: 240-pin DIMM
* Feature: DDR3 1333 (PC3 10600)
* Specs: DDR3 PC3-10600 CL=9 Unbuffered ECC DDR3-1333 1.5V 256Meg x 72
-Scythe Mugen 2 CPU cooler
-SAPPHIRE 100252HDMI Radeon HD 4550 512MB 64-bit GDDR3 PCI Express 2.0 x16
-Western Digital Caviar Green WD10EADS 1TB (I also tried a "SAMSUNG Spinpoint F1 HD103UJ 1TB 7200 RPM 32MB Cache SATA 3.0Gb", but it didn't change anything)
-SAMSUNG S223L DVD±RW 22x SATA WHITE
-FA-8V08-WH 18-in-1 Floppy + Internal Flash Card Reader


I'm very pleased with the sound level of this PC. Very quiet. If I was to do it again I probably wouldn't worry about getting a special "quiet case". Just eliminating as many fans as possible and reducing the noise of the fans you do have to use seems to get you about 99% of the way to quiet PC goodness.

As you've probably guessed I'm not so pleased with my system stability. It all started out so well. As part of my computer build process I run memtest86+ (available on "System Rescue CD"), prime95 and OCCT. These all came back fine. Vista installed fine, everything installs fine. No problems whatsoever.

The first sign of something amiss was when I was taking an image of my C drive with the most excellent fsarchiver (ran from my favorite "System Rescue CD"). Note that fsarchiver is really a file system archiver, not a partition imager, but that just means it does a better job of creating "partition images" than a true partition imager does.

Anyway, System Rescue CD froze while running fsarchiver. This was would be quite embarrassing if Heather ever found out. Fortunately the only place where I've revealed this faux pas is in this blog entry which is far too long and boring for her to read. I chalked this freeze up to System Rescue CD using too new of a kernel. Perhaps in the latest kernel Linus replaced the SCO code he stole with some M$ code that supports system freezing?

So after rebooting and finishing taking the image of her C drive I give Heather her shiny new computer (complete with a "champagne pink" bezel). She seemed quite happy with it at first, but then the unthinkable happened, Vista froze! After the first freeze I was disappointed, but not worried. I just assumed that Vista had some sort of backwards compatibility with WinXP freezes and enduring a system lockup every month or two was a small price to pay to be a Bill Gates groupie.

Unfortunately these freezes weren't that infrequent. They'd happen once or twice a week. Infrequent enough to make troubleshooting and isolating the problem a nightmare, yet frequent enough to be fairly annoying. It also really rubbed me the wrong way to have this system that I paid a premium for (to get the best stability) locking up twice a week. She also reported that these freezes always seemed to happen when she clicked on something (start menu, starting a program etc.). This bit of info may prove useful....

Note that when I say freeze I mean that my computer just stops, the screen freezes with whatever's on the screen. Keyboard & mouse are unresponsive. Can't ping it over the network. I.e. it behaves just like a BSOD, except that instead of a blue screen it's just your regular Windows screen that's "frozen".

Something had to be done. Someone brave enough to face the dreaded "random & infrequent system freeze of death" had to be found! But what kind of man could have the infinite amount of courage and patience required to not only face this fearsome beast, but actually have a slim chance of victory? It hit me one morning after staring in the mirror at my ruggedly handsome face for half an hour that only I had the jeanious, good lucks, valor, courage and wit to face down this dastardly foe.

After saying goodbye to my family I began my journey into the heart of darkness (seems like there should be a witty Ubuntu reference here, but there isn't, get over it). My first goal was to be able to reproduce this problem at will. If the system only freezes once a week it could take years to isolate the cause. Fortunately I had actually used fsarchiver several times to take images of the C drive at different points (base install, with minimal apps, with full apps, etc.). I had several freezes while using fsarchiver so I started out using fsarchiver as my test program since the normal stress test programs (like prime95, occt, memetest86+) could not reproduce the issue.

I found that by running fsarchiver in a continuous loop (see below) I could get the system to freeze in a matter of hours. The commands I used to image my ntfs C drive to a file on my ntfs D drive (D:\1 directory) in an infinite loop looked like this:
ntfs-3g /dev/sda2 /mnt/backup
cd /mnt/backup/1
while [ 1 ]; do date; rm -f cdrive.fsa; fsarchiver -j 4 -z 8 savefs cdrive.fsa /dev/sda1; sleep 60; done

Unrelated to fsarchiver I also noticed some corrrected hardware error WHEA MCE (Machine Check Exception) logs within Vista (a few a week). These are found in the system log, source = "Microsoft-Windows-WHEA-Logger", Event ID = 19. Using AMD's mcat utility I was able to verify that these are reporting corrected ECC errors. They always reported a problem with memory bank 4. Maybe the RAM in slot 4 was bad? I swapped the two RAM modules, but the freezes continued and the MCE logs continued to point to bank 4. At this point I removed the RAM from slot 3 (I only have one 2GB stick on bank 4) and the freezes and logs continue.

I've now proved that RAM bank 4 was part of the problem, but was it the whole problem? Just for the fun of it I reset my BIOS settings to defaults and tested again with fsarchiver. I then went 24 hours without error! I added all of my BIOS settings back in and the freezes returned. So obviously the problem is a combination of RAM slot 4 and a BIOS setting(s).

At this point I could've started a binary search of my BIOS settings to see which one(s) were part of the problem. However, during my fsarchiver testing I found that (although I usually got a freeze within a few hours) I could go up to a dozen or so hours before getting a freeze. I.e. it could take quite a while to finish isolating this problem with my current fsarchiver freeze re-creation method. I decided to try to find a better way to (more quickly) re-produce the system freezes.

I was running fsarchiver with pretty intensive compression (the "-z 8" option). This caused fsarchiver to max out my CPU most (but not not all) of the time. When fsarchiver is waiting for more data from the disk it can't be compressing, so the CPU usage drops way down. I believe it actually will stop and go into halt state (very briefly) while waiting for more data from the hard drive to compress.

If maxing out the CPU with prime95 and OCCT doesn't cause a problem, but yet running fsarchiver with its varying levels of CPU usage does freeze the computer, maybe it's the act of bringing the CPU out of the halt state that contributes to this problem? This would also explain why Heather notices the freezes when she clicks on (i.e. starts) something.

I decided to try cycling prime95 by continuously starting prime95, stopping prime95 (and sleeping for a second) and then starting prime95 again. Success! With this prime95 cycle I cut my average time to freeze down from 3 hours to 30 minutes! The simple batch file I use to do this is listed in the [prime95cycle.cmd] section below.

prime95cycle.cmd is the batch file I used to cycle prime95 continuously. This works for me with Vista Business 32 bit. I believe it will work under all versions of Windows 7 as well. I know that WinXP doesn't have the "taskkill" or "timeout" commands, but it's easy enough to Google for WinXP replacements to those.


[prime95cycle.cmd]

REM Give Windows a minute to finish starting up
timeout 60 > nul

:loop
REM start prime95 in torture test mode (-t).
start "" D:\1\prime95.exe -t

REM Sleep while prime95 runs
timeout 1 > nul

REM Gracefully kill the prime95 torture test we started above
taskkill /im prime95.exe

REM Sleep for 2 seconds to allow prime95 time to close. Without this pause the next prime95 won't start since you can't have two instances of prime95 running at once.
timeout 2 > nul

REM rinse, repeat
goto loop

If you're running Linux prime95 is called "mprime" instead, so you'll need to run some commands like this:
while [ 1 ]; do mprime -t >/dev/null & sleep 1; pkill mprime; sleep 1; done

Maybe it's just superstition, but I believe performing frequent reboots also makes the problem more likely to occur. It seems that the computer is more likely to freeze after a reboot or cold start. Therefore I created a another simple script (see [reboot15min.cmd] below) that reboots my computer after 15 minutes. Even if rebooting isn't really necessary, this has the added benefit of clearing out any hung prime95 processes that taskkill couldn't get rid of (which happens occasionally).


[reboot15min.cmd]
REM 900=15 minutes
timeout 900 /nobreak > nul

REM Reboot
shutdown -r -c "Stress test reboot" -d p:0:0



Now all I need to do is put prime95cycle.cmd and reboot15min.cmd in my startup folder, reboot and then wait for the computer to freeze. Like I said before I averaged about 30 minutes until freezing using this method. However, for reasons I'm sure I'll never understand, my computer could still occasionally go hours without a freeze. Therefore while I was isolating which BIOS setting(s) were part of this problem I'd wait for 12 hours of stress testing to complete before assuming whatever set of BIOS settings I was using were safe.

The BIOS settings I changed from default are:
-Main->Storage Configuration = Set all SATA ports to "AHCI"
-Ai Tweaker->CPU Spread Spectrum = Disabled //Default is "Auto"
-Ai Tweaker->PCIE Spread Spectrum = Disabled //Default is "Auto"
-Advanced->CPU Configuration->Cool'n'Quiet = Enabled
-Advanced->CPU Configuration->C1E = Enabled
-Advanced->Chipset->Northbridge Configuration->ECC Configuration->ECC Mode = Good
-Advanced->USB Configuration->Legacy USB Support = Auto //Default is "Enabled"
-Power->ACPI 2.0 support = Enabled
-Power->APM Configuration->Power On By PME = Enabled //WOL
-Power->Hardware Monitor->CPU Q-Fan Function = Enabled
-Power->Hardware Monitor->Select Fan Type = PWM Fan
-Power->Hardware Monitor->-CPU Q-Fan Mode = Silent
-Boot->Boot Settings Configuration->Full Screen Logo = Disabled
-Tools->Express Gate = Disabled



After testing various combinations of BIOS settings I noticed that when I have ECC enabled in the BIOS I get freezes (not BSODs) and the occasional WHEA MCE log telling me about a corrected memory error in RAM bank 4. If ECC is disabled I just got a BSOD instead. Therefore if you're like 99.99999% of home computer users who don't use ECC RAM you'll need to ensure that your computer doesn't automatically reboot after a BSOD by following the steps here: http://www.internetfixes.com/vista_tips/IF01186.htm

After all this work I can finally state what my problem is. With C1E enabled and RAM in slot 4 (the black slot farthest away from the CPU) I get intermittent freezes and/or BSODs. The problem seems to be with the CPU coming out of the C1E state and delivering power to RAM slot 4. Enabling Q-Fan and frequently rebooting the computer seem to help the problem occur more quickly/often, but they are not required to re-produce this issue.

I think the problem is with the CPU coming out of C1E state for two reasons:
1. Most obvious reason is that the problem only occurs if C1E is enabled. The only default BIOS setting you need to change to get this problem is to enable C1E (however enabling QFan does help the problem to occur more often, more on this later).
2. I can run stress tests (like OCCT and prime95) for days without error. This proves my system is stable under load and the problem is not heat related. However, if I continuously start prime95, stop (and sleep for a second) and then start prime95 again I can usually get the problem to occur within an hour (with Q-Fan enabled, more on that below).

I believe this issue is power related because enabling Qfan seems to help reproduce the problem quicker. In my testing with a standard 60mm size CPU fan I can get the problem to happen most quickly with Q-Fan enabled and set to "PWM Performance", "DC Silent" or "DC Optimal". Using a larger 120mm fan (which runs at a much lower RPM) instead it seems that "PWM Silent" or "PWM Optimal" are the settings most likely to reproduce this problem. "DC Silent" also works, but since my 120mm fan never actually spins with this setting I don't like to use it.

Note, however, that enabling Q-Fan is NOT required to have this problem, it just makes the problem much more likely to happen. I've even gotten my computer to freeze with no CPU fan connected at all (yes, my temps were fine, I have huge heatsink and don't even really need a fan). My best attempt at an explanation for this is that whenever the system is using "some magic amount of power" and the computer comes out of C1E it can't deliver the right amount of power to RAM slot 4.

I know the problem is with RAM slot 4 because:
1. All of my WHEA MCE (Machine Check Exception) logs always indicate a problem with slot 4
2. When I try different slots and sticks of RAM the problem only occurs if I have a stick of RAM in slot 4.


[Other things I tried that didn't help]
Changing RAM voltage didn't help. My Crucial RAM specs says that it should use 1.50 volts. With the BIOS default setting of "AUTO" for the RAM voltage, the voltage actually gets set to 1.60 volts. I've tried setting a range of RAM voltages from 1.50-1.66 volts (in 2/100th increments), but it didn't help. I don't want to go higher than 1.66 volts for fear of frying my RAM.

Changing the OS didn't help. I got the same freezes when using Fedora 12 Linux. This obviously exonerates all of my Windows drivers, programs, etc. as being part of the problem. I should also note that the vast majority of my testing was done with a base install of Vista business 32 bit. Pretty much just did a default install and ran Windows Update.


[To keep this post from being too short here are some free bonus bug hunting tips]
1. In Linux the MCE logs are in /var/log/mcelog
2. Fedora 12 (64 bit) writes the MCE logs to disk once an hour in this cron job "/etc/cron.hourly/mcelog.cron". Since my system would usually freeze before the MCE logs got written to disk I deleted the hourly cron job and instead added this job to "/etc/cronttab" so that the MCE logs would get written out every minute:
* * * * * root /usr/sbin/mcelog --ignorenodev --filter >> /var/log/mcelog
3. In Windows you can use the "AMD MCAT Machine Check Analysis Tool" to get more details on the MCE logs.
http://support.amd.com/us/Pages/dynamicDetails.aspx?ListID=c5cd2c08-1432-4756-aafa-4d9dc646342f&ItemID=178
-Machine Check Analysis Tool (MCAT) is a command line utility that takes a Windows System Event Log (.evt) file as an argument and decodes the MCA Error logs into human readable format. MCAT can alternatively take as an argument the raw register hexadecimal values from an MCE Error.
-When I used this on Vista it couldn't read the event log directly. However I could manually give it the numbers from the event log for it to decode. I used commands like this:
mcat /cmd 4 0xf4782000e0080a13 0x90b2c700 0x2231eaa00000000
mcat 4 0x94744000eb080a13 0x30a4bc0 0x226dbd200000000
mcat 4 0x94214000b5080813 0x15ba7c00 0x225385200000000
4. In Vista it's helpful to create a custom log view that shows serious errors. Then by using this custom log view you can quickly see if you had any serious errors logged. My custom view is setup like this:
-Logged: Any time
-Event logs: System
-Event Sources: BugCheck, eventlog, Eventlog, WHEA-Logger
-Includes Event IDs: 19,1001,6008



In the end, the problem is probably with my "Asus M4A79T Deluxe" motherboard. Since disabling C1E fixes the issue, I'm hopeful that a simple BIOS update can fix this (I've got a ticket open with ASUS on this issue). I know I could avoid this issue simply by using the orange RAM slots and/or disabling C1E. However, I paid good money for this board, so everything should work dadgummit! C1E saves me ~ 7 watts of of power and who knows when I'll want to run 4 sticks of RAM.

Other than this issue, I really like this motherboard. It's got good support for ECC RAM. Plenty of old school connections like PS2 keyboard and mouse, serial port (need to buy the cable separately), PCI, IDE and floppy connections. All this in addition to modern conveniences like a very customizable BIOS, easy BIOS updates, SATA, eSATA, 4 PCIe slots, lots of USB2.0 ports, firewire, etc.

Real Ultimate Kid-proof Entertainment Center Cooling

My trusty ReplayTV 5040 quit working the other day, turned out the hard drive had gone bad. I suspect this had something to do with the balmy 94 degree temperature I keep inside my entertainment center. The reason it's so hot is that I have to keep the glass door to my entertainment center shut to keep my kids from terrorizing my electronic equipment.

I got a new hard drive installed in my Replay, but I don't want this one to get cooked too. So how can I keep my Replay cool while at the same time keeping little hands off of my electronics? Read on for the exciting duct tape filled pictorial!

This first pic shows what the Replay's entertainment center shelf looks like before I started hacking.


Here you see the two perfectly circular holes I cut, but what could they be for?


The top left hole is for an exhaust fan (a Thermaltake Mobile Fan II External USB Cooling Fan Mfr. Part# A1888) and the bottom right hole is for my custom cut "Solo cup exhaust dock". Basically I cut the bottom off of a Solo cup to that my Replay's Solo cup would dock into the entertainment center more easily.


I think grey duct tape looks really good on a Replay. The Replay's cup channels the air into the entertainment center cup. This way none of the Replay's hot exhaust air gets stuck inside the entertainment center.




The next three pics show the cup and fan mounted from the back of the entertainment center. Notice again the professionalism of the holes I cut.




My creation with my Squeezebox and Replay installed.


Another shot with the cooking door closed.


How well does my setup work? Before I started it was 94 degrees in the ReplayTV section of my entertainment center. Adding the Solo cup duct work lowered the temp to 88.5 degrees. Then turning my exhaust fan on (to its lowest speed so I can't hear it) further lowered the temp to 83.5 degrees. So I got just over 10 degrees of cooling from my modifications. I.e. I'm happy and I hope this lets my new Replay TV hard drive live a little longer.

When my kids are older and I can leave my entertainment center door off the temps inside my entertainment center are: 85.0 with no modifications, 82.5 with the cup duct work and 80.5 with the cup duct and fan turned on. So even with no door my hacks help. Note that all temps were recorded when the house was 77 degrees.

Below is the last pic of my entire entertainment center. This is the ultimate low-cost high-value kid proof entertainment center setup. Notice the duct tape on the TV? I cut a bottle of baby wash in half and taped it over the TV's buttons. This prevents my youngest from playing his favorite game of "shut the TV off". I.e. our TV is now operated by remote only. The best part of the grey duct tape is that it's practically invisible on our black TV. You really have to look to spot it.


If you think you have a better kid proof entertainment center you are wrong and I will send my kids over to your house to show you just how wrong you are.

One last note to any critics out there who say that simply removing the back panel of my entertainment center would've been better. I choose to do it my way instead because:

1. I think it looks better to leave the back panel on to hide the plethora of cables in back of the entertainment center.
2. This is geekier.
3. This involved duct tape