R assignment returns all tests failed all the time

[DrKAG]

Hello!

I have been testing out using the autograder for R programming assignments as I'd like to use it for my programming class. I used the gradeR template from the community resources and was able to create my own working test assignment last week. It was a simpler 2-test assignment without any data uploading. (Completed on Windows machine.)

Yesterday, I went to create an actual assignment for my students to complete and it's not working (4 tests, included data upload). It's returning every test as incorrect. My test submission is definitely correct and I've used the gradeR functions interactively to check that the tests should pass (or fail on purpose for testing). (Initially done on Windows machine, also tried this on Mac.)

Just now, I downloaded the zip from the working assignment and created a new assignment from it and the "new" assignment isn't not working even though the original working assignment still works great. (Windows and Mac)

I redownloaded the community resource from here (https://github.com/tbrown122387/Using-gradeR-for-the-Gradescope-Autograder) and zipped and uploaded that without any changes and that even returns all tests failed. (Mac)

Has something changed since last week? Can anyone help me to troubleshoot this issue? I'm not sure what additional information to provide, so let me know if anything would be useful.

Thanks!
KAG

[adam-brucker]

It sounds like something may have been updated in a package you are using? I would check if anything was updated in library(gradeR)

[DrKAG]

Thanks for the suggestion! I have been working with the Gradescope help team all week and tried earlier versions of gradeR with no success. However, their latest suggestion that I haven't yet tried is a versioning issue with the "withr" package. They suggested to include this in the setup.sh file.

Rscript -e "install.packages('remotes')"
Rscript -e "remotes::install_version('withr', '2.5.2')"

As a bandaid, I went with writing the tests and results.json file out more manually like the other community resource and that seemed to work.

I'll test out this potential solution soon and post an update!

[srcaggiano01]

Hi! Were you able to figure out. We used Gradescope to grade out R files last semester and it worked fine. I am using the exact same autograder.zip and the exact same correct answer key, and it is now deducting points from the correct work. This consistently happens for every assignment. Ugh!

[DrKAG]

Yes! This did work to appropriately grade submissions.

However, I'm not back to using gradeR just yet since it doesn't seem to handle existence errors. just correct and incorrect work. So gradeR should work fine if there are no errors and you're only checking for correct/incorrect. You should instruct your students to ensure their scripts do not contain any errors prior to submitting, which is a totally reasonable request.

EXAMPLE: I asked the students to assign 5 to "aa" and 7 to "bb" then make "cc" be the product of "aa" and "bb".

Graded just fine since "cc" exists, but is incorrect:
aa <- 5
bb <- 7
cc <- aa+bb

Would not grade anything since "cc" doesn't exist. But this is a common mistake for beginner programmers, forgetting a character:
cc < aa*bb

Also wouldn't grade this since "b" doesn't exist which typos is another common error.
cc <- aa*b

Because I wanted them to possibly see and fix incorrect things AND errors, I programmed a workaround from the other Community Resource not using gradeR and programmed it to catch any errors from sourcing the submission AND from the tests.

[srcaggiano01]

Thank you! It is just so annoying that it worked perfectly fine last semester. I will play with it some more. :-)

[tbrown122387]

Hey, I'm the author of gradeR and I'm just seeing this thread now (Gradescope sent it over to me). Can you post an issue on my Github repo? In the meantime I'll investigate.

This sounds a lot like something I experienced last Monday. In my case I had to fiddle with environment variables. In run_autograder file I had to change

Rscript grade_one_submission.r 

to

LC_ALL= Rscript grade_one_submission.r 

and it worked. gradeR hasn't changed at all, so I figure it's something new with testthat (gradeR's only dependency). I submitted an issue with them to find out more.

Hello!

I have been testing out using the autograder for R programming assignments as I'd like to use it for my programming class. I used the gradeR template from the community resources and was able to create my own working test assignment last week. It was a simpler 2-test assignment without any data uploading. (Completed on Windows machine.)

Yesterday, I went to create an actual assignment for my students to complete and it's not working (4 tests, included data upload). It's returning every test as incorrect. My test submission is definitely correct and I've used the gradeR functions interactively to check that the tests should pass (or fail on purpose for testing). (Initially done on Windows machine, also tried this on Mac.)

Just now, I downloaded the zip from the working assignment and created a new assignment from it and the "new" assignment isn't not working even though the original working assignment still works great. (Windows and Mac)

I redownloaded the community resource from here (https://github.com/tbrown122387/Using-gradeR-for-the-Gradescope-Autograder) and zipped and uploaded that without any changes and that even returns all tests failed. (Mac)

Has something changed since last week? Can anyone help me to troubleshoot this issue? I'm not sure what additional information to provide, so let me know if anything would be useful.

Thanks! KAG

[srcaggiano01]

This is new to me! Where is your github repository? I found this one from the gradescope documentation. Something is definitely amiss. The autograder.zip worked fine last semester and the same autograder.zip does not work this semester. :-)

[tbrown122387]

@srcaggiano01 this is the one for the Gradescope gradeR example, and this one is the general entire gradeR package one. Yeah I'm pretty sure you're suffering fro the same thing that I was, though so it should be okay shortly

[srcaggiano01]

Thank you! I am prepping for class tonight but I will play with it more tomorrow.

[DrKAG]

Thanks! I'll try this out on my next assignment!

[tbrown122387]

What's an existence error? It is relatively straightforward to check that a variable was defined by a student. For instance, if you want to see if a student made a variable called, say, a, then you can stick exists("a") inside the expect_true() call and you're good to go.

Yes! This did work to appropriately grade submissions.

However, I'm not back to using gradeR just yet since it doesn't seem to handle existence errors. just correct and incorrect work. So gradeR should work fine if there are no errors and you're only checking for correct/incorrect. You should instruct your students to ensure their scripts do not contain any errors prior to submitting, which is a totally reasonable request.

EXAMPLE: I asked the students to assign 5 to "aa" and 7 to "bb" then make "cc" be the product of "aa" and "bb".

Graded just fine since "cc" exists, but is incorrect: aa <- 5 bb <- 7 cc <- aa+bb

Would not grade anything since "cc" doesn't exist. But this is a common mistake for beginner programmers, forgetting a character: cc < aa*bb

Also wouldn't grade this since "b" doesn't exist which typos is another common error. cc <- aa*b

Because I wanted them to possibly see and fix incorrect things AND errors, I programmed a workaround from the other Community Resource not using gradeR and programmed it to catch any errors from sourcing the submission AND from the tests.

[DrKAG]

At the time of testing, I tried that, however I didn't just want to know if it existed, I also wanted to check if it was correct. So when I would do one testthat function for existence there would still be an issue with the testthat function to check for correctness. I suppose I could do a wrapper function that does the existence check and then if it exists, check for correctness. Otherwise, an object that didn't exist in the student's script would cause the autograder to just stop and not report anything.

[tbrown122387]

Just to clarify: a variable not existing is not a problem for gradeR, but it could be a problem in a student submission.

I think you're referring to the situation where, if a student tries to do something like print(a), but they never previously defined a variable a, then they'll get Error: object 'a' not found. If gradeR cannot run the code, it can't give points, but gradeR doesn't error out.

On the other hand, if a variable doesn't exist, then that's not necessarily a catastrophic thing. If gradeR checks a variable that doesn't exist, the student will just lose points on that check only. This is in contrast to other Python autograders I use where, if a variable doesn't exist, it kills the whole process and the student gets a zero.

In short, it's up to the student to produce a runnable script. They can easily check this themselves by clearing out their memory and running their script form start to finish. If it doesn't run from start to finish, they'll see the same errors that Gradescope runs into. This also encourages debugging on their local machines instead of Gradescope because it's faster.

[DrKAG]

Hm, in the testing on Gradescope the other week it wouldn't give points to any of the questions if it ran into the 'object not found' error anywhere in the script. But that may be due to the fact it just wasn't working with Gradescope at the time overall.

Then I moved to the other community resource and I had to code my way around the existence issue (which is what gradeR already did, but I couldn't get it to work at the time).

I totally agree that the student script needs to run without errors! It's my biggest hope they listen when I say to clear the Environment and run it, but I know that won't be the case for some. It's definitely a skill I'm trying to teach them! I also wanted the students to be able to submit and check their work as they go for these first few assignments where we do the building blocks of R objects, so they would likely not have everything done prior to submission.

I plan to try out the updated setup.sh on HW 2 though, so I'll definitely pay attention to how the testthat checks are coded.

[tbrown122387]

Now fixed. The example from Using-gradeR-for-the-Gradescope-Autograder now runs fine after making this commit. No changes to gradeR required.

Hello!

I have been testing out using the autograder for R programming assignments as I'd like to use it for my programming class. I used the gradeR template from the community resources and was able to create my own working test assignment last week. It was a simpler 2-test assignment without any data uploading. (Completed on Windows machine.)

Yesterday, I went to create an actual assignment for my students to complete and it's not working (4 tests, included data upload). It's returning every test as incorrect. My test submission is definitely correct and I've used the gradeR functions interactively to check that the tests should pass (or fail on purpose for testing). (Initially done on Windows machine, also tried this on Mac.)

Just now, I downloaded the zip from the working assignment and created a new assignment from it and the "new" assignment isn't not working even though the original working assignment still works great. (Windows and Mac)

I redownloaded the community resource from here (https://github.com/tbrown122387/Using-gradeR-for-the-Gradescope-Autograder) and zipped and uploaded that without any changes and that even returns all tests failed. (Mac)

Has something changed since last week? Can anyone help me to troubleshoot this issue? I'm not sure what additional information to provide, so let me know if anything would be useful.

Thanks! KAG