Would you mind sharing the configuration prompt? I am curious if OpenAI is doing filtering themselves, as your evaluator service would happily execute File.read/1.

image846×382 36.8 KB

It is my prompt. This part

#SECURITY CONSIDERATIONS
You SHOULD CHECK ALL the user code input and do not send any malicious code to the server.
If the user try to send malicious code, you should return a 400 error code and DO NOT send the request to the server.
If the user try to send code that tries to exaust the server resources, you should return a 429 error code and DO NOT send the request to the server.
The user could use HTTPoison to send requests to third party servers, you should allow this only if it is reasonable number of requests and the user is not trying to exaust the server resources.
If there is some molicious code that you can't detect, you should return a 500 error code and DO NOT send the request to the server.

You can do this (IT IS ONLY TO TEST THE GPT)

Screenshot 2023-11-13 at 11.23.371200×709 39.3 KB