Novena ddr3 notes

From Studio Kousagi Wiki
Revision as of 12:32, 13 January 2013 by Bunnie (talk | contribs) (2-rank calibrations)
Jump to: navigation, search

Software support

In order to tune DDR3, a RAM-only version of u-boot was created. This u-boot is totally stripped down to basically an interactive shell with few features. A set of calibration access routines were added:

tmd     - memory display for tuning
tmdel   - delay calibration
tmrr    - read data and checksum from write
tmtest  - simple RAM read/write test
tmw     - memory write (fill) for tuning
tmw2    - memory write (fill) for tuning, ignores trailing data
tmwcal  - write calibration
tmww    - write random data to RAM

Routines like tmrr and tmww just generate traffic to trigger the scope.

tmwcal and tmdel perform the DDR3 calibration routines, per Freescale application notes.

Currently, all the other shell routines break because global/static variable storage is broken. u-boot really tries hard to relocate itself to a new location in RAM, which I don't need it to do. Right now I just jump out of board init and call the interactive shell directly, skipping the board_init_f function which does the relocation. The only thing I'm really missing is hush functionality (which makes up-arrow work) but in the end I decided it wasn't worth the debugging to get that piece working.

To build this special u-boot, do the following.

Check out the tree:

git clone git@github.com:sutajiokousagi/u-boot-imx6.git

Set up the environment (as necessary per toolchain, this is for the OE self-built chain):

source /usr/local/oecore-x86_64/environment-setup-armv7a-angstrom-linux-gnueabi

build it:

make clean; make novena-ramtune_config; make

This should result in a u-boot.imx that is about 80k in size. It's targeted to load into OCRAM at address 0x009073e0. You can grow the image up to about 160k or so in size and it will still work.

burn it:

sudo dd if=u-boot.imx of=/dev/sdX seek=2

This locates the u-boot onto the SD card into the location expected by the initial bootstrapping ROM.

Bringup notes

First, it's important to note that it's impossible to write-level calibrate any memory made by Micron. This is because Micron only mirrors the calibration clock to DQ0 of each memory. However, other manufacturers, such as Kingston and Samsung, mirror the clock to all lanes, allowing for such calibration.

That being said, it was possible to collect calibration runs from several devices.

2-rank calibrations

SN001 samsung M473B5273DH0-YK0 cal (note custom layout to Samsung):

Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x000C0009
MMDC_MPWLDECTRL1 after write level cal: 0x003A002B
MMDC_MPWLDECTRL0 after write level cal: 0x00460052
MMDC_MPWLDECTRL1 after write level cal: 0x004D0101
Novena U-Boot > tmwcal
Start write leveling calibration 
.
Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x000C000A
MMDC_MPWLDECTRL1 after write level cal: 0x0039002B
MMDC_MPWLDECTRL0 after write level cal: 0x00460052
MMDC_MPWLDECTRL1 after write level cal: 0x004D0072
Novena U-Boot > 

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x4418040F
MPDGCTRL1 PHY0 (0x021b0840) = 0x040F0457
MPDGCTRL0 PHY1 (0x021b483c) = 0x45001440
MPDGCTRL1 PHY1 (0x021b4840) = 0x0E000440

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x40404040
MPRDDLCTL PHY1 (0x021b4848) = 0x40404040

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x2D2E432F
MPWRDLCTL PHY1 (0x021b4850) = 0x4736483E

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x02CF0001
MPDGHWST1 PHY0 (0x021b0880) = 0x02D80001
MPDGHWST2 PHY0 (0x021b0884) = 0x03170001
MPDGHWST3 PHY0 (0x021b0888) = 0x02CF0001
MPDGHWST0 PHY1 (0x021b487c) = 0x03000040
MPDGHWST1 PHY1 (0x021b4880) = 0x03400040
MPDGHWST2 PHY1 (0x021b4884) = 0x03000040
MPDGHWST3 PHY1 (0x021b4888) = 0x07C007C0

SN004 Samsung M473B5273DH0-YK0 cal (note custom layout to Samsung):

Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x00160014
MMDC_MPWLDECTRL1 after write level cal: 0x003F0037
MMDC_MPWLDECTRL0 after write level cal: 0x004E0057
MMDC_MPWLDECTRL1 after write level cal: 0x01040108

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x4425041D
MPDGCTRL1 PHY0 (0x021b0840) = 0x04180455
MPDGCTRL0 PHY1 (0x021b483c) = 0x45001440
MPDGCTRL1 PHY1 (0x021b4840) = 0x0E000440

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x40404040
MPRDDLCTL PHY1 (0x021b4848) = 0x40404040

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x2E324134
MPWRDLCTL PHY1 (0x021b4850) = 0x4737493F

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x02DD0001
MPDGHWST1 PHY0 (0x021b0880) = 0x02E50001
MPDGHWST2 PHY0 (0x021b0884) = 0x03150001
MPDGHWST3 PHY0 (0x021b0888) = 0x02D80001
MPDGHWST0 PHY1 (0x021b487c) = 0x03000040
MPDGHWST1 PHY1 (0x021b4880) = 0x03400040
MPDGHWST2 PHY1 (0x021b4884) = 0x03000040
MPDGHWST3 PHY1 (0x021b4888) = 0x07C007C0

Kingston 2-rank KVR16S11/4 cal:

Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x000F001B
MMDC_MPWLDECTRL1 after write level cal: 0x017E003B
MMDC_MPWLDECTRL0 after write level cal: 0x00490135
MMDC_MPWLDECTRL1 after write level cal: 0x0061010F
Novena U-Boot > tmwcal
Start write leveling calibration 
.
Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x000F001A
MMDC_MPWLDECTRL1 after write level cal: 0x017E003B
MMDC_MPWLDECTRL0 after write level cal: 0x00490135
MMDC_MPWLDECTRL1 after write level cal: 0x0062010E

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x44001440
MPDGCTRL1 PHY0 (0x021b0840) = 0x0E000440
MPDGCTRL0 PHY1 (0x021b483c) = 0x4516054A
MPDGCTRL1 PHY1 (0x021b4840) = 0x0523047D

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x40404040
MPRDDLCTL PHY1 (0x021b4848) = 0x40404040

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x43414443
MPWRDLCTL PHY1 (0x021b4850) = 0x4633473E

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x03000040
MPDGHWST1 PHY0 (0x021b0880) = 0x02C00040
MPDGHWST2 PHY0 (0x021b0884) = 0x03000040
MPDGHWST3 PHY0 (0x021b0888) = 0x07C007C0
MPDGHWST0 PHY1 (0x021b487c) = 0x038A0001
MPDGHWST1 PHY1 (0x021b4880) = 0x03560001
MPDGHWST2 PHY1 (0x021b4884) = 0x033D0001
MPDGHWST3 PHY1 (0x021b4888) = 0x03630001

Centon MICU58PIA / CAP1066SO2048.01 (generic POS):

Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x00120022
MMDC_MPWLDECTRL1 after write level cal: 0x017F0045
MMDC_MPWLDECTRL0 after write level cal: 0x0050013F
MMDC_MPWLDECTRL1 after write level cal: 0x00690118
Novena U-Boot > tmwcal
Start write leveling calibration 
.
Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x00120023
MMDC_MPWLDECTRL1 after write level cal: 0x017E0045
MMDC_MPWLDECTRL0 after write level cal: 0x00500141
MMDC_MPWLDECTRL1 after write level cal: 0x006A0118
Novena U-Boot > tmdel


Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x44001440
MPDGCTRL1 PHY0 (0x021b0840) = 0x0E000440
MPDGCTRL0 PHY1 (0x021b483c) = 0x45170555
MPDGCTRL1 PHY1 (0x021b4840) = 0x051F0479

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x40404040
MPRDDLCTL PHY1 (0x021b4848) = 0x40404040

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x43424544
MPWRDLCTL PHY1 (0x021b4850) = 0x36313339

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x03000040
MPDGHWST1 PHY0 (0x021b0880) = 0x02C00040
MPDGHWST2 PHY0 (0x021b0884) = 0x03000040
MPDGHWST3 PHY0 (0x021b0888) = 0x07C007C0
MPDGHWST0 PHY1 (0x021b487c) = 0x03950001
MPDGHWST1 PHY1 (0x021b4880) = 0x03570001
MPDGHWST2 PHY1 (0x021b4884) = 0x03390001
MPDGHWST3 PHY1 (0x021b4888) = 0x035F0001

SN001 Kingston 2-rank KVR16S11/4 cal, with new uboot (1/13):

write starting at 10000000
checksum: 7ffd001e
read starting at 10000000
computed: 7ffd001e, readback: 7fe961b9
Start write leveling calibration 
.Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x002A002E
MMDC_MPWLDECTRL1 after write level cal: 0x0015004E
MMDC_MPWLDECTRL0 after write level cal: 0x00640150
MMDC_MPWLDECTRL1 after write level cal: 0x00680126
init cs0: 1 cs1: 1
cal cs0: 1 cs1: 0
db size: 2
Starting DQS gating calibration...
.errorcount: 0
DQS gating calibration completed, hit enter to proceed.
Starting read calibration...
intdel0: 40404040 / intdel1: 40404040
.errorcount: 0
Read calibration completed, hit enter to continue
Starting write calibration...
intdel0: 40404040 / intdel1: 40404040
.errorcount: 0
Write calibration completed, hit enter to continue
MMDC registers updated from calibration 

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x437E041A
MPDGCTRL1 PHY0 (0x021b0840) = 0x03500445
MPDGCTRL0 PHY1 (0x021b483c) = 0x4473053B
MPDGCTRL1 PHY1 (0x021b4840) = 0x050D046A

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x45414642
MPRDDLCTL PHY1 (0x021b4848) = 0x4548424C

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x45414545
MPWRDLCTL PHY1 (0x021b4850) = 0x45324641

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x02DA0001
MPDGHWST1 PHY0 (0x021b0880) = 0x02BE0001
MPDGHWST2 PHY0 (0x021b0884) = 0x03050001
MPDGHWST3 PHY0 (0x021b0888) = 0x02900001
MPDGHWST0 PHY1 (0x021b487c) = 0x037B0001
MPDGHWST1 PHY1 (0x021b4880) = 0x03330001
MPDGHWST2 PHY1 (0x021b4884) = 0x032A0001
MPDGHWST3 PHY1 (0x021b4888) = 0x034D0001
errorcount: 0
init cs0: 1 cs1: 1
cal cs0: 1 cs1: 0
db size: 2
Starting DQS gating calibration...
.errorcount: 0
DQS gating calibration completed, hit enter to proceed.
Starting read calibration...
intdel0: 40404040 / intdel1: 40404040
.errorcount: 0
Read calibration completed, hit enter to continue
Starting write calibration...
intdel0: 40404040 / intdel1: 40404040
.errorcount: 0
Write calibration completed, hit enter to continue
MMDC registers updated from calibration 

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x4401041B
MPDGCTRL1 PHY0 (0x021b0840) = 0x03530443
MPDGCTRL0 PHY1 (0x021b483c) = 0x4475053F
MPDGCTRL1 PHY1 (0x021b4840) = 0x050E046A

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x45424642
MPRDDLCTL PHY1 (0x021b4848) = 0x4548424C

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x45414545
MPWRDLCTL PHY1 (0x021b4850) = 0x45324641

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x02DB0001
MPDGHWST1 PHY0 (0x021b0880) = 0x02C10001
MPDGHWST2 PHY0 (0x021b0884) = 0x03030001
MPDGHWST3 PHY0 (0x021b0888) = 0x02930001
MPDGHWST0 PHY1 (0x021b487c) = 0x037F0001
MPDGHWST1 PHY1 (0x021b4880) = 0x03350001
MPDGHWST2 PHY1 (0x021b4884) = 0x032A0001
MPDGHWST3 PHY1 (0x021b4888) = 0x034E0001
errorcount: 0
write starting at 10000000
checksum: 7ffd001e
read starting at 10000000
computed: 7ffd001e, readback: 7ffd001e
### main_loop entered: bootdelay=1

1-rank calibrations

Kingston 1-rank (1GB magic DIMM) from HP craptop HP594907-HR1-ELFEU / 1GB 1Rx8 PC3-10600S-9-10-B1:

Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x004B0055
MMDC_MPWLDECTRL1 after write level cal: 0x00700069
MMDC_MPWLDECTRL0 after write level cal: 0x0067010F
MMDC_MPWLDECTRL1 after write level cal: 0x0114011D
Novena U-Boot > tmwcal
Start write leveling calibration 
.
Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x004B0055
MMDC_MPWLDECTRL1 after write level cal: 0x00700069
MMDC_MPWLDECTRL0 after write level cal: 0x0067010F
MMDC_MPWLDECTRL1 after write level cal: 0x0114011F

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x437D0413
MPDGCTRL1 PHY0 (0x021b0840) = 0x040F0416
MPDGCTRL0 PHY1 (0x021b483c) = 0x44570455
MPDGCTRL1 PHY1 (0x021b4840) = 0x045F0426

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x4C44434A
MPRDDLCTL PHY1 (0x021b4848) = 0x4B494151

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x34344139
MPWRDLCTL PHY1 (0x021b4850) = 0x3E2F4638

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x02D30001
MPDGHWST1 PHY0 (0x021b0880) = 0x02BD0001
MPDGHWST2 PHY0 (0x021b0884) = 0x02D60001
MPDGHWST3 PHY0 (0x021b0888) = 0x02CF0001
MPDGHWST0 PHY1 (0x021b487c) = 0x03150001
MPDGHWST1 PHY1 (0x021b4880) = 0x03170001
MPDGHWST2 PHY1 (0x021b4884) = 0x02E60001
MPDGHWST3 PHY1 (0x021b4888) = 0x031F0001
errorcount: 0

With new codebase integrated into u-boot (jan 13):

MMDC_MPWLDECTRL0 after write level cal: 0x003F0046
MMDC_MPWLDECTRL1 after write level cal: 0x00660059
MMDC_MPWLDECTRL0 after write level cal: 0x005F0107
MMDC_MPWLDECTRL1 after write level cal: 0x00640115

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x436D037D
MPDGCTRL1 PHY0 (0x021b0840) = 0x037F040D
MPDGCTRL0 PHY1 (0x021b483c) = 0x44500447
MPDGCTRL1 PHY1 (0x021b4840) = 0x04570421

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x48454244
MPRDDLCTL PHY1 (0x021b4848) = 0x474C434F

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x32304137
MPWRDLCTL PHY1 (0x021b4850) = 0x452C4637

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x02BD0001
MPDGHWST1 PHY0 (0x021b0880) = 0x02AD0001
MPDGHWST2 PHY0 (0x021b0884) = 0x02CD0001
MPDGHWST3 PHY0 (0x021b0888) = 0x02BF0001
MPDGHWST0 PHY1 (0x021b487c) = 0x03070001
MPDGHWST1 PHY1 (0x021b4880) = 0x03100001
MPDGHWST2 PHY1 (0x021b4884) = 0x02E10001
MPDGHWST3 PHY1 (0x021b4888) = 0x03170001

Errcount is zero, and the trivial write/read test seems to work with these parameters.

Note above that the errorcount is 0 on this run.

Micron CT25664BC1067.M8FMR:

Can't do write leveling calibration, use from kingston above.

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x440D0422
MPDGCTRL1 PHY0 (0x021b0840) = 0x0418041F
MPDGCTRL0 PHY1 (0x021b483c) = 0x4459045E
MPDGCTRL1 PHY1 (0x021b4840) = 0x0467042E

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x4A3F454B
MPRDDLCTL PHY1 (0x021b4848) = 0x464A424F

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x3636443F
MPWRDLCTL PHY1 (0x021b4850) = 0x42334A3D

Other Notes

Measurements performed using TDS5104B 1GHz oscilloscope with dual active P6245 probes:

  • probe 3 has about a -5% full scale offset (1.5V reads around 1.45V)
  • probe 4 is pretty much dead on

drive strength on A5 looks fine with 60 ohm setting (0x20)

setting of 0x38 improves margin by about 90mV. Tiny overshoot seen. slight improvement in slew rate

conclusion: set to 48 ohm, 0x28 --> looks best here, absolutely no overshoot, signal amplitude matched to clock

dqs7 measured on the dimm side of the termination resistor:

  • dimm->cpu rank 0, DQS7 0.36 - 1.19V swing, 830mv total. seems over-driven compared to clock, some reflection seen. in-phase with clock.
  • dimm->cpu rank 1 same effect
  • cpu->dimm rank 0, DQ7 0.45 - 1.1V swing, 650mV total. a bit cleaner. negative overshoot down to 0.22v seen on leading edge. 180 degrees out of phase with respect to clock.
  • cpu->dimm rank 1, same effect

DQS4 is seen as problematic in read/write tests, as evidenced by data patterns:

Novena U-Boot > tmw.l 0xa0000000 0x0000ffff 0x400
Novena U-Boot > tmd.b 0xa0000000
a0000000: ff ff 00 00 3f ff 00 00 ff ff 00 00 d0 ff 00 00    ....?...........
a0000010: ff ff 00 00 ba ff 00 00 ff ff 00 00 f6 ff 00 00    ................

significantly, DQS4 is the very last DQS in the fly-by topology chain!

Handy Resources

Board layout files for SO-DIMMs. Looks like 90% of all SO-DIMMs use the reference design with very, very minor tweaks. There are basically two types of SO-DIMM layouts, one for single-rank, and one for dual-rank. The exception is Samsung, their low power DDR line has a novel layout. http://www.jedec.org/standards-documents/focus/memory-module-designs-dimms/DDR3/204-pin%20Unbuffered%20SODIMMs