Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Same instruction sometimes executed TWICE when stepping through code (Herc s+ command) #321

Closed
YossiReichman opened this issue Aug 27, 2020 · 125 comments
Assignees
Labels
BUG The issue describes likely incorrect product functionality that likely needs corrected. Related This issue is closely related to another issue. Consider this issue a "sub-issue" of the other.

Comments

@YossiReichman
Copy link

YossiReichman commented Aug 27, 2020

I don't know if any one uses the B+ and S+ commands to trace code, and I think this is old bug, but I have observed that at certain points instruction are repeated (i.e. executed twice). Maybe the CPU that is stopped is not re-awoken?

The relevant code is in process_trace (cpu.c) where, after displaying the instruction, the CPU is put in Wait state. It is then awoken by pressing the <enter key> by the code in hscpufun.c function start_cmd_cpu.

The attached log is of a step/trace I did. I saw no less than 4 instructions that were repeated: The first is a CLM instruction at address 1F5941EA. The next is a STM at 1F594276, The third is a MVC at 1F5943D8. And finally there is a STG instruction at 1F59447E.

Many times, as in the case of a LR instruction for example, there is no harm done, but this has happened to me in the expansion of macros to a PC instruction where the sequence of code is L 14,772(14,0) followed by L 14,212(14,0) where, if the first load is repeated, I get a translation error.


Note:  Issue #322 ("Likely incorrect Hercules breaking/stepping handling/support?") is closely related to this one.

@YossiReichman
Copy link
Author

YossiReichman commented Aug 27, 2020

Since, whenever a z/OS processor stops executing instructions (such as when stepping through code instruction by instruction such as what occurs during a Hercules debugging session), z/OS might mistakenly believe there is a hardware problem (malfunction) because its other CPUs (that are still running!) may end up "spinning" trying to acquire a lock that the code being stepped through may be holding (thereby causing z/OS to mistakenly believe the "spin" is taking much too long) , the following PARMLIB member (and associated command to activate it) can be used to reduce the likelihood of z/OS complaining in such situations:

(contents):

SPINRCVY SPIN
SPINTIME=999

The attached PARMLIB member effectively lets a CPU spin for 999 seconds. Use it before starting a Hercules debugging session to step through z/OS code. To use it, first, place the member in the appropriate PARMLIB PDS as member name EXSPAT00, and then, before beginning your Hercules debug session (i.e. before stepping through code via Hercules s+ command), issue the z/OS command SET EXS=00 on the master console.

@Fish-Git Fish-Git added BUG The issue describes likely incorrect product functionality that likely needs corrected. Researching... The issue is being looked into or additional information is being gathered/located. labels Aug 27, 2020
@Fish-Git
Copy link
Member

Thank you for this report, Joe!

One question (just so we're on the same page): what version of Hercules is this? The BUGTRCE.TXT log you attached is not a complete log so it doesn't contain the Hercules version information.

Also, while I don't doubt you (I believe you!), if possible, I would very much like to see evidence of the double load issue you described (i.e. L 14,772(14,0) followed by L 14,212(14,0)). If the instruction is truly being executed twice, then we should be able to see evidence of that in the register display. That is to say, when Load is executed the first time, R14 will be loaded with one value, but then if it is executed a second time, R14 should then contain a completely different value, and we should see that in the register display.

What I am trying to determine here is whether what you are seeing is cosmetic in nature (where, as you speculated in your opening remarks, Hercules is simply not processing the start request correctly and is simply re-displaying the same instruction again), or whether this is indeed a true bug where the same instruction is indeed being executed again twice in a row.

Cosmetic bugs are confusing annoyances and should certainly be fixed whether they cause any problems or not, but an improper instruction execution bug such as the one you describe is a very serious, nay, catastrophic bug that needs to be found and fixed ASAP!

So if you can manage to catch evidence of the "double L Load" bug, I would very much appreciate seeing it!

In the mean time I am going to try and throw together a standalone test to see if I can reproduce this bug. I rather suspect it will prove to be very difficult to reproduce! But I will try.

Thank you for reporting it.

@YossiReichman
Copy link
Author

YossiReichman commented Aug 27, 2020 via email

@Fish-Git Fish-Git changed the title While Stepping thru code at certain points the same instruction is repeated Same instruction sometimes executed TWICE when s+ stepping through code Aug 27, 2020
@Fish-Git Fish-Git changed the title Same instruction sometimes executed TWICE when s+ stepping through code Same instruction sometimes executed TWICE when stepping through code (Herc s+ command) Aug 27, 2020
@YossiReichman
Copy link
Author

YossiReichman commented Aug 27, 2020

This time I have a trace where by executing/displaying the same instruction a register is changed. It happened at the expansion of a IBM macro that expands to PC instruction. If you look to the bottom of the log, I also output a Visual Studio trace and an image of the source code where I inserted the trace. The Hercules version is "4.3.9999.0-SDL" (4.3.9999.0).


TracePoint in CPU C


TracePoint in hscpfunc c

@YossiReichman
Copy link
Author

YossiReichman commented Aug 27, 2020

If you look at the bottom of the log you will see L 14,232(14,0) repeated twice with register 14 changing.

@Fish-Git
Copy link
Member

Fish-Git commented Aug 28, 2020

If you look at the bottom of the log you will see L 14,232(14,0) repeated twice with register 14 changing.

Wow. I see it.

This is bad. Very bad.

Very, VERY bad.   :(

I'll try to get right right on this for you, Joe!

One thing though: the Hercules version you reported is a very "generic" version number which is not helpful at all at identifying what specific version of Hercules you are using.(*) Perhaps posting the output of the Hercules version command might be better since it contains the build date and that might allow me to better guess what version you are likely using.

This is important since I have made some important changes recently (just the other day as a matter of fact) in the Hercules logic that deals with instruction tracing and I'm sure the version you're using doesn't have those changes. I very much need to be using the SAME version that YOU are using so I can be sure I am chasing the right bug. Thanks.

OR..., as an alternative, you might want to try downloading the most recent version of Hercules we have (updated just yesterday!) and build and test using that version instead of the one you're current using. I might have already fixed your bug! But I don't know that! Which Is why I would prefer to use the same version as what you are using so that once I identify the bug, I can know whether or not it's already fixed or not.

So please post the output of the Hercules version command. Hopefully the "build date" that it reports will get me a little bit closer to the version you are actually using.

Thanks.


(*) As you may recall in a much earlier private email to you back in early May, I explained that building Hercules from the source provided by GitHub's big green "Code" button ("Download ZIP") is NOT the preferred way to build Hercules from source.

Rather, it is much preferred that you instead do a git clone of the actual source code repository instead, and build that source code instead. Building from a clone of our source code repository will provide an exact version number, whereas building from the "Download ZIP" source code like you did only constructs a very generic (and very unhelpful) version string.

In the future, if you need/want to build Hercules from source, please try to remember to build it from a CLONE of our git source code repository and not from the generic "Download ZIP" source code.   Thanks!

@YossiReichman
Copy link
Author

YossiReichman commented Aug 28, 2020 via email

@Fish-Git
Copy link
Member

Fish Here is it thanks I’ll make it worth your effort. Ill follow your instruction and down load the latest copy and it test it out again most likely Saturday night as I am going away for the weekend. thanks

PLEASE do not respond to GitHub via email. (Pretty please?)

Instead, please respond (reply) via GitHub's web interface.

Thanks.

@YossiReichman
Copy link
Author

YossiReichman commented Aug 30, 2020

Fish, I think I downloaded or cloned the latest version. I have the github desktop installed on my desktop and used it to "clone" what I think is the latest version. Afterwards I tried to build it and I got two errors (missing files): win32.mak and pre64para.h.

@YossiReichman
Copy link
Author

YossiReichman commented Aug 30, 2020

Trying to upload a picture I took doesn't seem to work, so I'll just type it: 4.3.9999.0-SDL (4.3.9999.0).

I also got a wait state for the two new config statements FACILITY ENABLE 073_TRANSACT_EXEC and FACILITY ENABLE 050_CONSTR_TRANSACT. Don't know if you made changes to these recently. Will look at it tomorrow.

@wrljet
Copy link
Member

wrljet commented Aug 30, 2020

Missing win32.mak sounds like you haven't followed the build instructions completely.

http://www.softdevlabs.com/hercules-vs2015-build.html

@YossiReichman
Copy link
Author

YossiReichman commented Aug 30, 2020

I got it built. What I am not sure of is, I thought I had the latest version. Since I am getting the wait state stated above, I am beginning to wonder if this is true. The Version I have is 4.3.9999.0-SDL (4.3.9999.0).

@wrljet
Copy link
Member

wrljet commented Aug 30, 2020

Joseph,

Are you using git clone to pull down the source code, or downloading a zip file? Fish's _dynamic_version.cmd script cannot work if you downloaded a zip from github.

With the information from git, you should see a version string like this:

(from an SVN checkout):

HHC01413I Hercules version 4.3.9999.10260-SDL (4.3.9999.10260)

(from git clone):

HHC01413I Hercules version 4.3.9999.10241-SDL-g019e7511 (4.3.9999.10241)

Without the information from git (or svn), the build process won't know how to fill in that last part of the version number, so it will be a zero:

HHC01413I Hercules version 4.3.9999.0-SDL (4.3.9999.0)

Bill

@YossiReichman
Copy link
Author

YossiReichman commented Aug 30, 2020

I tried to do a clone it asked me for a SSH key.

SSH_KEYS

Since I didn't have one I clicked "use download using HTTPS". Nothing seemed to happen. It gave me the option of downloading using GITHUB desktop. That worked. However, based on getting a transaction facility wait state code, GitHub desktop must of downloaded a really old version, as the transaction facility wait code seemed to have been resolved for a few months.

@wrljet
Copy link
Member

wrljet commented Aug 30, 2020

This "dialog" from GitHub:

image

The little icon to the right of the https:// URL simply copies that URL to your clipboard (just to make it easier, so you can paste it into your terminal to do a git clone).

That said, I just used GitHub Desktop that you mentioned (I had really never noticed that there or heard of it before), and then built that, and it worked. Version number line shows the current commit ID hash.

Bill

@wrljet
Copy link
Member

wrljet commented Aug 30, 2020

One other point, _dynamic_version.cmd cannot work if there isn't a git.exe installed and in your search path.

Personally, I've been using Git For Windows: https://gitforwindows.org/.

@YossiReichman
Copy link
Author

YossiReichman commented Aug 30, 2020

Got it. Thanks. Going to build shortly. One more thing: when update this current issue I click write tab and doesn't update the comment. I end up having to hit the green comment button which just start a space for another comment. So my question is, when I finish writing in order to post, do I have to click anything?

@Fish-Git
Copy link
Member

(from git clone):

HHC01413I Hercules version 4.3.9999.10241-SDL-g019e7511 (4.3.9999.10241)

Without the information from git (or svn), the build process won't know how to fill in that last part of the version number...

Correct. But it is the abbreviated git hash value (i.e. the "019e7511" digits following the "g" in the version string) that tells us exactly what version you are using, being the first 8 digits of the got commit hash. For example, in your above example version string, the "g019e7511" part tells us you are using the version of SDL Hyperion that was current as of commit 019e751 made on 8/29/2020 (which is indeed the most current version).

Without knowing that, with just a "generic" version 4.3.9999.0 string, we cannot know precisely what version of Hercules you are using and thus cannot know whether the version you are using has certain important fixes in it or not.

That is why it is highly recommended that you always clone our git repository and then build Hercules from that git clone.

@Fish-Git
Copy link
Member

One other point, _dynamic_version.cmd cannot work if there isn't a git.exe installed and in your search path.

Correct! That is probably what Joe is missing. I suspect Joe doesn't have git installed.

Personally, I've using Git For Windows. (https://gitforwindows.org/)

I myself personally prefer TortoiseGit since it integrates itself directly into the Windows User Interface itself (i.e. the "Shell"), making using git almost trivially easy.

But TortoiseGit is just the user-interface side of things of course. You still need to have git installed on your system too of course (and yes, I too use Git For Windows as well, since that's what TortoiseGit recommends).

Being a long time Window user I personally prefer GUI programs over command line programs. (I also have TortoiseSVN on my system as well for example!) I'm guessing you're more of a command line person?

Joe? You need to install git. I thought later versions of Visual Studio came with git, but maybe not (or maybe it's not a standard git client?). I recommend installing Git For Windows and, optionally, TortoiseGit as well.

If you need help configuring it, I (or Bill I'm sure!) can help you with that.

@wrljet
Copy link
Member

wrljet commented Aug 30, 2020

Now that you mentioned it, I do see that there is a git.exe from Visual Studio 2017, in:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw32\bin

So I imagine I just have Git For Windows closer to the front end of my search path.

It appeared from posts above that Joe was using a Hercules compiled by VS2015. I don't see a git anywhere in its directories.

Bill

@Fish-Git
Copy link
Member

Fish-Git commented Aug 30, 2020

Just click inside the "Leave a comment" text box and start typing your response. As soon as you type anything into the "Leave a comment" text box, the green "Comment" button should turn from a very dull green to a bright green. When you are finished typing your reply, just click the green Comment button! Sheesh.

@YossiReichman
Copy link
Author

YossiReichman commented Aug 30, 2020

This is the version going to try to single step:

Version

@YossiReichman
Copy link
Author

YossiReichman commented Aug 30, 2020

Bug is still there:

Right in the beginning of the log there is a XC 2256(4,3),0(3) (I think it repeats three times) and then a few instructions later L 4,548(0,0) is repeated.

@Fish-Git Fish-Git added IN PROGRESS... I'm working on it! (Or someone else is!) and removed Researching... The issue is being looked into or additional information is being gathered/located. labels Sep 21, 2020
@Fish-Git
Copy link
Member

The fix is ready:

Please note the following:

When you run your test you MAY still see what at first looks like duplicate instructions being executed, but if you look closely at the registers, you should see that the instruction is actually not being executed twice. Rather, it is simply being fetched twice and subsequently traced twice too as a result.

For those interested, it turns out the first EXECUTE_INSTRUCTION following the INSTRUCTION_FETCH (and preceding the UNROLLED_EXECUTE loop) could not be removed. It must remain there. But it needs to be preceded with a VALID_AIE test instead rather than the usual regs->ip vs. regs->aie test that UNROLLED_EXECUTE does.

When a CPU is being stepped there can be a considerable delay between the time an instruction is fetched and the time it is executed (due to the system waiting for you to press enter). And when the other CPUs that are not being stepped are allowed to continue running normally (which as I said I personally feel is wrong), this greatly increases the likelihood of the TLB being invalidated before the instruction being stepped can be eventually executed. This is why Joe had such a relatively easy time causing it to occur whereas it went unnoticed by everyone else for over 14 years.

This bug could easily also be the cause of other TLB related issues (i.e. unusual/unexpected Hercules behavior) such as Ivan's SIE OPTION_SIE_PURGE_DAT_ALWAYS issue and possibly others that are know to likely exist but just haven't been found yet.

Joe? I'll give you a couple of days (two? three? more? less?) to try the patch and confirm that the problem is indeed now fixed before eventually committing my changes.

I would like to thank you profusely for your cooperation and tremendous patience with me while working together on this bug. I really appreciate it buddy. This was a tough one!

But we got through it.   :)

Thanks.

@Fish-Git Fish-Git added the QUESTION... A question was asked but has not been answered yet, -OR- additional feedback is requested. label Sep 22, 2020
@YossiReichman
Copy link
Author

Ill test it out after work 4pm est thanks

@wrljet
Copy link
Member

wrljet commented Sep 22, 2020

Fish, this only affects execution when tracing/stepping, correct?

@Fish-Git
Copy link
Member

Fish-Git commented Sep 22, 2020

Fish, this only affects execution when tracing/stepping, correct?

As best as I can tell, Bill, yes, the problem can only(?) occur during instruction stepping.

But I have admittedly not performed a complete audit of all Hercules code to be 100% sure.

The way Hercules is currently coded (i.e. due to the nature of this particular bug), I believe it may be possible for other potentially bad? things to occur in certain specific situations as well, but I cannot say that with any high degree of certainty. As I said, I have not had time to examine all of Hercules to see where, and whether or not, the problem (or similar a TLB related problem) might also possibly occur.

The problem occurs whenever the IPTE (Invalidate Page Table Entry) instruction is executed by some other CPU thereby impacting all CPUs in the complex, including the one being stepped.

Now normally, the way our OBTAIN_INTLOCK and SYNCHRONIZE_CPUS handling is designed, the TLB update will usually (most likely) occur in between instructions, whenever an interrupt is being processed, since it is only then that the interrupt lock is obtained. The interrupt lock is not normally obtained during normal instruction fetching and dispatching.

Whenever instruction stepping is enabled for a given CPU however, the CPU "pauses" (stops) after an instruction is fetched and before it is executed. It is during this "pause" that the interrupt lock is obtained and the CPU is put to sleep while waiting for the user to press the enter key to indicate to the system it should go ahead and execute the instruction it just fetched.

It is during this time period (which could be seconds, minutes, hours, days, years, etc; we cannot know when the user will decide to press the enter key!) that the window of opportunity opens for the bug to occur. It is during this time period where it is possible for some other CPU to execute the IPTE instruction thereby invalidating the TLB for the CPU being stepped, and thus, as a result, invalidate the instruction just fetched as well.

Due to the way Hercules was coded, this invalidation wasn't noticed and the just fetched instruction was executed anyway. After it was executed, we returned to our instruction stepping logic and upon attempting to fetch what should have been the next instruction, noticed the invalidation, which caused us to fetch the next instruction not from where our internal instruction pointer said to fetch it from, but rather from where the instruction address in the PSW said to fetch it from (because our internal instruction pointer was rendered "invalid" due to the INVALIDATE_AIA that was necessarily done as part of the TLB clearing).

But in Hercules, the instruction address field in the PSW is not updated after every single instruction. Doing so would slow Hercules down to a crawl. It is only updated whenever "needed". As a result, the "next" instruction that was fetched came from where the PSW instruction address field said to get it from, which was (you guessed it!) the very same instruction that it had just executed! End result: the same instruction ends up being executed twice in a row! Oops!

But as I said, this is unlikely to occur during normal instruction execution since during normal instruction execution the interrupt lock is not obtained between when the instruction is fetched and when is is executed. It is only during instruction stepping that this window of opportunity ever opens.

Now it is interesting to note that this problem, as explained, occurs because:

  1. Some other CPU executes the IPTE instruction while the CPU being stepped is stopped waiting for the user to press the enter key, and:

  2. The IPTE instruction, as currently coded in Hercules, always clears the TLB of all other CPUs in the complex too, due to our not currently supporting the "Local-TLB-Clearing Facility" yet.

But it's important to note that even if the "Local-TLB-Clearing Facility" was supported, the problem could still occur if the local clearing option wasn't specified in the IPTE instruction. The "Local-TLB-Clearing Facility" simply gives the program (i.e. z/OS in this case) the option of whether or not to only clear its own TLB and not the TLBs of all other CPUs, but just because the option would be available to it, that doesn't mean it must therefore always specify that option! It is free to continue executing the IPTE instruction without specifying the local TLB clearing option, in which case the problem would still occur just like it does today without that option.

@YossiReichman
Copy link
Author

first of if this problem would occur in normal settings there would be abends all over place, Just think of how much code is executed by ipling z/os. Second the user cant wait an indefinite amount of time to press enter because by stopping a CPU to step z/os thinks there is a malfunction. it is only be executing the exspatxx parmlib that gives a user sometime to debug your code. But think about it there is absolutely no way to debug a SRB or PC rtn. The only product that does something like this is XDC and trust me it has it issues with this type code. The code basically establishes estae FRR this is all task related with this feature you can debug multitasking xmem code. One thing I will say from time to time I have abend messages from hyperion truth is if there is a problem with z/os component z/os has recovery in place and issues an SDUMP. So for very display of an exceptional message there most likely should be an SDUMP. I am going to have delay my testing of the patch till 6 pm tonite as something came up thank you again

@YossiReichman
Copy link
Author

Fish just put in diff7 seems like there was a problem. The reason this bug became so apparent now because in this code I have large number of PC instructions typically used by IBM macros. To get the PC number the expansion of the IBM macro involves using Register 14 in the first operand (as a register) and in the second referencing storage. This first ins was L R14,772(,R14) and the second was L 14,208(14,0) this instruction is repeated however RE also changes here is the log
hercules.log

@Fish-Git
Copy link
Member

Fish just put in diff7 seems like there was a problem.

Then try again until there ISN'T a problem! Sheesh!

You need to use the new "joe7.diff" patch I gave you!

You hercules log is WORTHLESS because you weren't using my new code!

Try again!

@YossiReichman
Copy link
Author

I did the following git apply -R joe6.diff and then git apply joe7.diff this time I'll delete the directory and clone it then apply the patch few minutes

@YossiReichman
Copy link
Author

Did as I said downloaded a new clone applied joe7 hyperion frooze i'll try it again tommorow thanks
HYPERIONFREEZE

@YossiReichman
Copy link
Author

wait a minute hit enter again continuing but PSW seems to freeze as if none of the other CPU's are running

@YossiReichman
Copy link
Author

YossiReichman commented Sep 23, 2020

There were a few dumps. Didnt notice any dupes, but I have to be honest in that my code never got this far before. Here's the log:

I'll continue tomorrow. Thanks.

@Fish-Git
Copy link
Member

Did as I said downloaded a new clone applied joe7 hyperion frooze

I doubt that very much. What you are likely seeing is a simple re-fetching of the same instruction due to TLB invalidation. That is to be expected and is the fix.

@Fish-Git
Copy link
Member

wait a minute hit enter again continuing but PSW seems to freeze as if none of the other CPU's are running

I doubt that very much. What you are likely seeing is a simple re-fetching of the same instruction due to TLB invalidation. That is to be expected and is the fix.

@Fish-Git
Copy link
Member

There were a few dumps
didnt notice any dupes
but I have to be honest code never got this far
here is the log
I'll continue tommorow
thanks

The "dumps" are to just provide confirmatory evidence that my fix is working as expected (and it appears it is).

I'll clean up my code and commit the official fix as soon as I can, and will let you know when it's ready.

Thanks for your help.

@Fish-Git Fish-Git removed the QUESTION... A question was asked but has not been answered yet, -OR- additional feedback is requested. label Sep 23, 2020
@Fish-Git Fish-Git self-assigned this Sep 23, 2020
@Fish-Git
Copy link
Member

first of if this problem would occur in normal settings there would be abends all over place,

Agreed! Which is why I suspect it very likely does NOT occur during normal instruction execution. The problem only seems to occur during instruction stepping when the "window of opportunity" opens due to the "pause" between the fetch of the instruction and the actual execution of that instruction.

The fix is to simply check the aie for NULL immediately before the actual execution of the instruction just fetched, and if NULL (indicating a TLB/AIA invalidation), to simply do the fetch over again. Only if the aie is valid is the fetched instruction then executed.

Fish-Git added a commit that referenced this issue Sep 24, 2020
@Fish-Git
Copy link
Member

Fixed by commit 2b136d0.

Please discard your current build of Hercules (and all patches as well), (re-)clone the current version, rebuild and re-test. Everything should work just fine now.

I will await your confirmation that your problem is now fixed before closing this issue.

Thanks for everything, Joe.

p.s. to Bill: I have confirmed that this problem is impossible to occur under normal circumstances. It only occurs when instruction stepping is active (and now with the fix in place, should not cause any problem even if it does occur).

@Fish-Git Fish-Git added Waiting to close... Waiting for user to report back whether problem still exists or not before closing as resolved. and removed IN PROGRESS... I'm working on it! (Or someone else is!) labels Sep 24, 2020
@YossiReichman
Copy link
Author

YossiReichman commented Sep 24, 2020

Thanks. Will test at 4pm EST. I will delete and re-clone.

@YossiReichman
Copy link
Author

YossiReichman commented Sep 24, 2020

Looks good. The ultimate test is that I had this instruction where operand 1 and 2 referenced the same register and the repeat would make it blow up. Here, as in other instructions, you refetch the registers which fixes the problem. Attaching the log:

Processor CP06: running state selected
17:12:23 HHC00835W Processor CP06: aie invalidated; instruction being refetched
17:12:23 HHC00007I Previous message from function 'z900_run_cpu' at cpu.c(2074)
17:12:23 HHC02324I CP06: PSW=0704100080000000 000000001F593896 INST=58E0E08C     L     14,140(0,14) 

@Fish-Git
Copy link
Member

Excellent!   Closing issue!   :)

@Fish-Git Fish-Git removed the Waiting to close... Waiting for user to report back whether problem still exists or not before closing as resolved. label Sep 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BUG The issue describes likely incorrect product functionality that likely needs corrected. Related This issue is closely related to another issue. Consider this issue a "sub-issue" of the other.
Projects
None yet
Development

No branches or pull requests

3 participants