[BUG] WARN validate(), ll drift detected #3854
Comments
|
@oleggtitoff hi, can you please check the releases/tgl/v11.0-hotfix2 tag and test it? Thanks. |
|
Hi @slawblauciak, thanks for your advice. Unfortunately, we are not able to build FW from this tag. We build it the same way as in the reproduction steps above, the only difference is to |
|
looks the rimage version with the release branch is different, can you try 'git submodule update' in the sof/ folder before building. |
|
Hi @keyonjie, @slawblauciak, thanks for your comments. Added submodule update command after checkout: But got the same error. Could you please suggest, what else can we try? |
|
@oleggtitoff are you able to clone rimage repo and build ? Something may be out of sync, sub repo may need to be pulled as well. |
|
@oleggtitoff yes, do something like 'rm rimage -rf' before the 'git submodule update'. |
|
Hi guys, thank you very much. |
|
hi @oleggtitoff I think the master is already aligned with tgl/v11 after the #3768 was merged, can you try the latest master? |
|
Hi @keyonjie, |
|
Hi @oleggtitoff thanks for information, looks we need to define different warn_timeout for TGL platforms, the reason why you didn't see it with tgl-011 is that SA is disabled there. |
|
@keyonjie could this issue explain the very recent Its The more visible error is: |
|
Same failure in https://sof-ci.01.org/sof-pr-viewer/#/build/PR3891/build6130418 but on TGL this time: The |
|
PR #2686 downgraded this error to a warning, because of "non-critical" drifts (~ "soft" real-time?). However I don't see anything about "critical" drifts in #2686 Should it be upgraded to an error again when the drift is too large or when there are too many in a row or both? https://github.com/thesofproject/sof/issues?q=%22ll+detected+drift%2C+delta%22+ |
|
Hi all, looks the drift comes from re-requesting HPRO during exiting from WFI, @oleggtitoff can you try if #3762 can fix the issue at your end? @marc-hb I am not sure if the drift can lead to bad SNR or other errors, but at least it raise flag of abnormal issues happened, so at least a warning here is legit to me. BTW, PR #3762 should fix the drift issue so it worth to try with it for other quality issues IMO. |
To avoid any regression the #3762 should be taken alongside with the #3865 (merged to master). |
|
@keyonjie both merged now, can we close ? |
|
Hi everybody, the issue is fixed. Thank you very much! |
Probably the main change is fixing the huge etrace test gaps thesofproject#321 and thesofproject/sof#3281 Also fixes DMA trace gaps thesofproject#297 and thesofproject#298 I initial tried to preserve some of the existing code but it was just too bad. PR thesofproject#161 / commit 9136776 seemed especially bad: - It tried to ignore a specific `ll drift` error but instead it filtered out almost every log statement out of... stderr, that does not have show log statements!! (Just for the record this `ll drift` error has been downgraded to warning now, see thesofproject/sof#2686 and thesofproject/sof#3854) - That same commit also added code that merely starts the DMA trace with "there is an error below" (without failing the test) but that's eclipsed by the entire log that follows. Later, the firmware started printing ERROR every single time when the ERROR FW ABI prefix was introduced yet no one ever noticed which proves how useless this prefix is was. So remove this DMA trace prefix as the purpose of this test is - as clearly stated in thesofproject#167 - not to find firmware errors but errors with the sof-logger itself (even though we never had anything looking at firmware errors so far) Don't grep for "error" on stderr: anything on stderr is a logger failure (not a firmware failure). Don't require whitespace before the TIMESTAMP header. Add set -e. Use shell functions. Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Probably the main change is fixing the huge etrace test gaps thesofproject#321 and thesofproject/sof#3281 Also fixes DMA trace gaps thesofproject#297 and thesofproject#298 I initial tried to preserve some of the existing code but it was just too bad. PR thesofproject#161 / commit 7274f49 seemed especially bad: - It tried to ignore a specific `ll drift` error but instead it filtered out almost every log statement out of... stderr, that does not have show log statements!! (Just for the record this `ll drift` error has been downgraded to warning now, see thesofproject/sof#2686 and thesofproject/sof#3854) - That same commit also added code that merely starts the DMA trace with "there is an error below" (without failing the test) but that's eclipsed by the entire log that follows. Later, the firmware started printing ERROR every single time when the ERROR FW ABI prefix was introduced yet no one ever noticed which proves how useless this prefix is was. So remove this DMA trace prefix as the purpose of this test is - as clearly stated in thesofproject#167 - not to find firmware errors but errors with the sof-logger itself (even though we never had anything looking at firmware errors so far) Don't grep for "error" on stderr: anything on stderr is a logger failure (not a firmware failure). Don't require whitespace before the TIMESTAMP header. Add set -e. Use shell functions. Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Probably the main change is fixing the huge etrace test gaps thesofproject#321 and thesofproject/sof#3281 Also fixes DMA trace gaps thesofproject#297 and thesofproject#298 I initial tried to preserve some of the existing code but it was just too bad. PR thesofproject#161 / commit 7274f49 seemed especially bad: - It tried to ignore a specific `ll drift` error but instead it filtered out almost every log statement out of... stderr, that does not have show log statements!! (Just for the record this `ll drift` error has been downgraded to warning now, see thesofproject/sof#2686 and thesofproject/sof#3854) - That same commit also added code that merely starts the DMA trace with "there is an error below" (without failing the test) but that's eclipsed by the entire log that follows. Later, the firmware started printing ERROR every single time when the ERROR FW ABI prefix was introduced yet no one ever noticed which proves how useless this prefix is was. So remove this DMA trace prefix as the purpose of this test is - as clearly stated in thesofproject#167 - not to find firmware errors but errors with the sof-logger itself (even though we never had anything looking at firmware errors so far) Don't grep for "error" on stderr: anything on stderr is a logger failure (not a firmware failure). Don't require whitespace before the TIMESTAMP header. Add set -e. Use shell functions. Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Probably the main change is fixing the huge etrace test gaps thesofproject#321 and thesofproject/sof#3281 Also fixes DMA trace gaps thesofproject#297 and thesofproject#298 I initial tried to preserve some of the existing code but it was just too bad. PR thesofproject#161 / commit 7274f49 seemed especially bad: - It tried to ignore a specific `ll drift` error but instead it filtered out almost every log statement out of... stderr, that does not have show log statements!! (Just for the record this `ll drift` error has been downgraded to warning now, see thesofproject/sof#2686 and thesofproject/sof#3854) - That same commit also added code that merely starts the DMA trace with "there is an error below" (without failing the test) but that's eclipsed by the entire log that follows. Later, the firmware started printing ERROR every single time when the ERROR FW ABI prefix was introduced yet no one ever noticed which proves how useless this prefix is was. So remove this DMA trace prefix as the purpose of this test is - as clearly stated in thesofproject#167 - not to find firmware errors but errors with the sof-logger itself (even though we never had anything looking at firmware errors so far) Don't grep for "error" on stderr: anything on stderr is a logger failure (not a firmware failure). Don't require whitespace before the TIMESTAMP header. Add set -e. Use shell functions. Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Probably the main change is fixing the huge etrace test gaps thesofproject#321 and thesofproject/sof#3281 Also fixes DMA trace gaps thesofproject#297 and thesofproject#298 I initial tried to preserve some of the existing code but it was just too bad. PR thesofproject#161 / commit 7274f49 seemed especially bad: - It tried to ignore a specific `ll drift` error but instead it filtered out almost every log statement out of... stderr, that does not have show log statements!! (Just for the record this `ll drift` error has been downgraded to warning now, see thesofproject/sof#2686 and thesofproject/sof#3854) - That same commit also added code that merely starts the DMA trace with "there is an error below" (without failing the test) but that's eclipsed by the entire log that follows. Later, the firmware started printing ERROR every single time when the ERROR FW ABI prefix was introduced yet no one ever noticed which proves how useless this prefix is was. So remove this DMA trace prefix as the purpose of this test is - as clearly stated in thesofproject#167 - not to find firmware errors but errors with the sof-logger itself (even though we never had anything looking at firmware errors so far) Don't grep for "error" on stderr: anything on stderr is a logger failure (not a firmware failure). Don't require whitespace before the TIMESTAMP header. Add set -e. Use shell functions. Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Probably the main change is fixing the huge etrace test gaps #321 and thesofproject/sof#3281 Also fixes DMA trace gaps #297 and #298 I initial tried to preserve some of the existing code but it was just too bad. PR #161 / commit 7274f49 seemed especially bad: - It tried to ignore a specific `ll drift` error but instead it filtered out almost every log statement out of... stderr, that does not have show log statements!! (Just for the record this `ll drift` error has been downgraded to warning now, see thesofproject/sof#2686 and thesofproject/sof#3854) - That same commit also added code that merely starts the DMA trace with "there is an error below" (without failing the test) but that's eclipsed by the entire log that follows. Later, the firmware started printing ERROR every single time when the ERROR FW ABI prefix was introduced yet no one ever noticed which proves how useless this prefix is was. So remove this DMA trace prefix as the purpose of this test is - as clearly stated in #167 - not to find firmware errors but errors with the sof-logger itself (even though we never had anything looking at firmware errors so far) Don't grep for "error" on stderr: anything on stderr is a logger failure (not a firmware failure). Don't require whitespace before the TIMESTAMP header. Add set -e. Use shell functions. Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Describe the bug
See the bunch of warnings after building SOF from master upstream:
[180448660.052083] ( 87622544.000000) c0 sa src/lib/agent.c:79 WARN validate(), ll drift detected, delta = 41010 [182488660.000000] ( 2040000.000000) c0 sa src/lib/agent.c:79 WARN validate(), ll drift detected, delta = 41007 [190708671.250000] ( 8220011.000000) c0 sa src/lib/agent.c:79 WARN validate(), ll drift detected, delta = 41013 [196196674.010417] ( 5488003.000000) c0 sa src/lib/agent.c:79 WARN validate(), ll drift detected, delta = 41013To Reproduce
rm -rf tmp
mkdir tmp
cd tmp
cmake -DTOOLCHAIN=xt -DROOT_DIR=$CORE_PATH -DMEU_OPENSSL= '' -DRIMAGE_PRIVATE_KEY=/home/otitov/xcc/meu/otc_private_key_3k.pem ..
make tgplp_defconfig
cp ../src/arch/xtensa/configs/override/tigerlake_chrome.config override.config
make overrideconfig
make bin -j
cd ..
./scripts/build-tools.sh -T
Reproduction Rate
Every time.
Expected behavior
Expected not to see these warnings.
Impact
Can impact on third party integration.
Environment
Screenshots or console output
sof_logger_log.txt
dmesg_log.txt
The text was updated successfully, but these errors were encountered: