Fails without displaying an error

[sebastianmontero]

Hi, I've tried the scanner using the gpt-3.5-turbo model, the scan eosio files task ran for more than an hour and a half and it then failed without any error, how can I go about debugging this issue? I've attached an image showing the output of the task.

Thanks

[ankh2054]

Hi,

The python code is currently confirmed to work with GPT-4. the APIs are slightly different between GPT 3.5 and 4, so if you want to use 3.5, uncomment

eosio-vulnerability-scanner/scan_eosio.py

Line 68 in 9268cba

#result = openai.Completion.create(engine=engine, prompt=prompt, max_tokens=1000, n=1, stop=None, temperature=0.5) #/v1/completions

and comment out line 69.

When I get time I'll try and update the setup to account for these variables.

[sebastianmontero]

Thanks @ankh2054, I've updated the model to use GTP 4, but now I'm getting the following error during the build of the docker container:

[ankh2054]

Sorry for delay we will have a look in the next day or two. Seems to be an issue with an upstream python package during the docker build.

[ankh2054]

all sorted now, test and let us know.

[sebastianmontero]

Thanks @ankh2054, I ran it again and I have the same error as the first error I mentioned, I'm using the gpt4 model:

[ankh2054]

Can you expand that error further, it might be a GPT error. Or if you can send me a link to your code I can test.

[sebastianmontero]

@ankh2054 there is no further error:

Is there any way to add verbosity?

[ankh2054]

Hmm unsure, would you be able to share your code?

[sebastianmontero]

@ankh2054 you should have received an invitation to our repo.

Thanks.

[ankh2054]

hey sorry for delay we are doing some testing today,.

[ankh2054]

there seems to be a timeout error with openAI, we will have to investigate further.

[sebastianmontero]

Thanks!

…

On Wed, Jun 7, 2023 at 5:41 AM Charles Holtzkampf ***@***.***> wrote: there seems to be a timeout error with openAI, we will have to investigate further. — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEL3AWF73XL766L3IKVXTDXKBSGXANCNFSM6AAAAAAXTR2RNI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[JohnWotherspoon]

Hi, have you been able to resolve the timeout issue yet? Is this a global issue or something local to our integration?

[ankh2054]

haven't had much time to troubleshoot, but it's not your local integration for sure. It's something about GPT that is timing out. When we get time my plan is to manually run it using the python script with some additional debugging and identify the exact error.

[JohnWotherspoon]

Thanks. Can you provide a rough estimate of when this will be on the schedule? We need to scan for vulnerabilities prior to mainnet launch and have to figure out our options.

[ankh2054]

I can’t give an exact estimate but hopefully in the next 2-3 weeks. But just to be clear chatGPT is not sufficient in catching bugs to be used as an audit. This tool is merely a proof of concept and is reliant on the proficiency of their models capabilities in identifying vulnerabilities and as it stands it certainly doesn’t catch all bugs, especially not the complicated ones.

If you want a full proof audit we can do that for you - charles@sentnl.io