what is best achievable bit error rate with RAD?

[rsquared68]

I assembled a RAD that is perfectly usable for nuvies, Sonic, etc. but it has a non-negligible mean time between errors on the order of minutes to hours. These are errors that maybe most people won't care about, and half are errors in verify of otherwise good data, but I'm curious about RAD as a general platform for DMA and other experiments so I'm exploring a bit. Some errors I can greatly suppress by disabling VIC DMA and by using ROM addresses on the C64 side, so I think these types can ultimately be conquered by tweaking the timing config. However, there are some errors that I call "first bytes" errors which at first glance do not seem sensitive to what's happening on the bus. These errors occur only in the first byte after a switch to a different REU bank; e.g. if I am verifying bank $ab and then start verifying bank $ac the probability of getting an error on $ac::0000 is much higher than finding an error anywhere else within the whole 64k of $ac. Half of these errors are real (actual wrong bytes in RAD) and half are not (RAD contents correct but verify fails) so I guess there is the expected symmetry between the stash and verify error rates when changing to a different bank. There is a line of RAD code that tries to detect if the simulated REU is being tested by an external program and it vetoes the verify error...however, I do know from inspecting the REU dumps that around half the time it was the stash that failed and a verify error was appropriate. Sorry for the long setup, but my question is "Is this an unavoidable consequence of the Pi's cache performance, or should I be searching elsewhere for a root cause?" Timing still can't be ruled out (e.g. CPU write to $df04-06 not sampled correctly) and before I spend 100 hours looking under the wrong rocks ;) I wonder if you and the community might share your experiences.

[MC6809E089]

I know this is not conlusive but I had errors happening constantly on both of my C64s, these errors were causing several demos and Sonic to crash pretty much all of the time, until Frntc suggested changing timings in this thread...
#25 (comment)
I ended up changing my settings to...
WAIT_ENABLE_RW_ADDRLATCH 600
WAIT_ENABLE_DATA_WRITING 800
WAIT_RW_BEFORE_ADDR 60
...and everything has worked fine without a single problem ever since. I've left the things running for hours without there ever noticing a glitch or a crash, so as far as I can tell my problems were entirely due to the timings being incorrect for my mainboards.

[MC6809E089]

Hi again,

I've so far tried running your test program configured a few different ways, to see what it did when it crashed out... If the REU size is set to any amount lower than 16MB, I get an error reported at address "00:0001" within a minute or two... is that the expected crashing behaviour you mentioned? I also tried the other two test programs you mentioned, and the behaviour of both was the same as you've observed, and the errors being reported were identical to what you are witnessing on your machine.

I am now running your vic halt program again, with the REU set to 16MB. It's been running continuously for a few hours, and so far hasn't reported any failures at all. I'll leave it running for the rest of the weekend (or at least until the custom chips get too hot!) and report back on what happens.

All of the above tests so far were done so far on a SixtyClone shortboard 250469 Rev A, the version which still has external colour ram. I also have a 250466 long board I could try, and another built RAD unit, which I could swap out for testing to see if either makes any difference to the timings and stability if you think that might be worthwhile?

[rsquared68]

I can update as I just tested for around 20 hours after using the different #defines in that code to try to isolate the different challenges. With the VIC allowed to do badline DMA (I also added sprite access), the mean time between errors was 86 minutes. In that sample I had 8 total verify errors, and only one of them was deep in a bank. The other 7 were in the first byte or two, if I can believe the address counter (see below). With the VIC DMA disabled during transfer/verify, the mean time between errors flagged by read of the verify bit was 122 minutes. There were 4 total errors and 2 were first byte errors. So possibly I still have collisions with VIC DMA and the BA timing can be improved further...IF this is accurate.

I say IF because this testing uncovered some evidence that while the RAD DMA timing appears much improved, at least sometimes the CPU read of the verify bit and address counter registers could be wrong. I know it's not wrong all of the time, because I can inspect the RAD contents and see bad bytes at one minus the last location on the address counter. However, that independent confirmation only matches the dump to the SD card about half the time. Originally I hypothesized this was just a DMA error in the opposite direction. Yesterday I saw some evidence that if I do sta $df01 to start a RAD verify DMA and then the very next instruction is lda $df04 to fetch the stop address, the address comes back with some value. However if I wait a dozen additional C64 cycles and read $df04 again it may have a different value than the first. I've not yet tuned the timing related to the RAD behavior on CPU read/write access to the simulated REU registers, so that could be most of my problem. I'll need to write a different routine to test that.

[rsquared68]

Yes, if it is set to <16MB you will get an error because it will try to write all 256 banks (256*64k = 16MB) and if the REU is smaller, the memory fill will just wrap around. That is, if you set to 8MB, when the code gets to the first byte after 8MB the REU will wrap around overwriting the lower banks starting from 00:0000. When my code reads it back, it will see that the first address 00:0000 contains the value expected at 80:0000 so that's a mismatch/error. The REU reports the verify error address plus 1 and my program just prints that = 00:0001.

As you can see from my long comment on my own testing overnight, I now have some additional doubts about what I should be checking. However it's great to have confirmation from another user that I'm not crazy--thanks :) I would only do additional testing if you are as curious about it as I am, or if you need to optimize the timings for each machine independently. I have three C64s both stock and heavily modded and the "best" timings are different. Or you can wait for me...If I find a systematic way to do it, I'll maybe put together a full testsuite to share that automatically evaluates the different aspects of timing versus bus, VIC, CPU etc. and gives a hint as to in which direction you might adjust the config parameters to improve. So far for me the biggest improvement came from comparing the error rates when the C64 source address was in RAM versus ROM; ROM was like 3x better for the default config, so I knew immediately that the RAD was too slow in sampling the data lines for the RAM.

[MC6809E089]

I'm don't think I am smart enough to work out how to further optimise my settings on my own, I had to ask Frntc to help me last time but I'm happy to try and help where I can be useful, so if you think of anything you'd like me to test or help with, please just let me know. :)

[rsquared68]

I'm not smart enough either but I'm trying to learn by doing :) I don't know if you've used the "sprite timing" checker in https://csdb.dk/release/?id=198460 but that also fails for me. The test seems to be checking the amount of time the VIC steals for sprite access while the REU is trying to write on those same lines, important for a certain demo trick (part 9 of Hable's article in http://www.ffd2.com/fridge/chacking/c=hacking8.txt ) I've heard it's hard to emulate so a discrepancy would not be surprising. Since it's a pretty extreme corner case it's way down on my list of things to play around with (and no one else may care).

[rsquared68]

I'll write a final comment to close this out. I've spent a fair number of hours looking at this and my conclusion is that my Raspberry Pis--a Zero 2W and a 3B--are generally not able to keep up with the C64 under all circumstances when running v0.12 of the RAD firmware.
The easiest way for me to observe this is with a logic analyzer looking at the multiplexer select line and the bus transceiver enable line. On some CPU cycles the RAD seems "somewhere else" in the code and misses the CPU access to the IO2 address space. With the default rad.cfg and config.txt the attached .prg which only writes to $df04,5 and reads back will run for just a few seconds on my setup. If I drop the number of preloads/cycle from 10 to 6 and pull all of the other timings in to the shortest possible values the CPU read/write test .prg will run forever without errors. Similarly, if I overclock the Pi further than the default core and sdram speeds I can also make the program (which does no DMA) run forever with more frequent preloads. However, both of my Pis become unstable at around core_freq=620 (heatsinks but no fans). If I add DMA activity to the test and use it as an REU, I find around 1 verify error every 2 hours at core_freq=620 on the Pi 3B and 1 verify error every 1 hour on both the 3B and Zero 2W with core_freq=600. The verify errors are always on the first byte of a (new) bank. I hypothesize it is a related issue of a timing excursion due to not having the required data in the cache for the first byte because the verify error rate improves dramatically with core speed. I don't rule out that something else is slowing me down because one other user on lemon64 who tested for me reported >18hrs between verify errors...however, I'm at a dead-end for now. I may try to test in the future another RAD assembled with ICs that have known quality--some of mine are probably Chinese.

cpu-rw-test.zip