-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Extension Proposal] Antivirus #36611
[Extension Proposal] Antivirus #36611
Comments
/cc @aloubyansky (extension-proposal), @gastaldi (extension-proposal), @gsmet (extension-proposal), @maxandersen (extension-proposal) |
+1, That sounds interesting |
It looks very interesting, how would it work, will the request go the anti-virus server after the uploaded content has been saved on the disk or as part of the multipart upload processing ? Can it take a lot of request time ? |
Maybe it can happen after the file is uploaded, in a scheduled task but yeah, that's an interesting question |
In my clients we do it in real time so as the multi part upload comes in so a virus is never saved to disk first. We have done it on 100mb files and you would be shocked how fast it is to stream to clamAV how fast it is. I will post example code here of what it would look like. But I can tell you in 10 years in production we have never seen performance be an issue and even in theory if it's a little slower most security departments can live with slower if it's stopping a virus from being stored on disk. |
Do you also upload the 100mb file to the service or just the SHA? |
For clamAv you stream the whole 100mb to clamav. For virusTotal it's just the SHA and it checks its database to see if it has results for that file from over 70 antivirus engines. |
Forwarding SHA can be a good option when someone is concerned about the performance. How would, for example, RestEasy Reactive users would use Good you plan to make it pluggable, it would be interesting to plugin, for example, an NVD feed check too, though may be it is out of scope and a similar concept you propose but to work with CVEs can be realized in another extension |
Awesome! Yeah I wanted to give multiple options and allow for users to write their own plug-in for say a proprietary virus engine etc! I am not as familiar with Reactive but I will test it out also. I will provide performance numbers for 1mb, 10mb, and 100mb files. |
Looks wonderful. I was wondering though why you made it one extension with all options in it? Don't you think people will use either one or another? |
Yes we plan on 1) adding more if users donate them 2) made it pluggable so people can implement their own and 3) allow you to call multiple AV if you want to scan it against one or more engines to be sure! |
interesting. I'm wondering if antivirus is the right name/abstraction as it sounds like this is usable for any kind of content scanning/classification? but that can be evolved in future it it broadens. |
@maxandersen you might be right as we could also check Mime Type matching etc but for now I thought I would just focus on antivirus? We can always make a new extension after this one gets burned in? @sberyozkin OK tested with 139MB file ClamAV: 1.945 seconds Told you it was fast! |
I'm surprised that it takes less than 2s to upload a 139Mb file, scan, and get the results. |
Here is my test case:
The whole thing takes 1.9 seconds on my machine. @PUT
@Consumes(MediaType.MULTIPART_FORM_DATA)
@Produces(MediaType.TEXT_PLAIN)
@Path("/upload")
public Response upload(@MultipartForm @Valid final UploadRequest fileUploadRequest) {
StopWatch stopWatch = new StopWatch();
stopWatch.start();
final String fileName = fileUploadRequest.getFileName();
final InputStream data = fileUploadRequest.getData();
log.infof("Uploading document %s", fileUploadRequest);
try {
final ByteArrayInputStream inputStream = new ByteArrayInputStream(
IOUtils.toBufferedInputStream(data).readAllBytes());
engine.scan(fileName, inputStream);
inputStream.reset();
// write the file out to disk
final File tempFile = File.createTempFile("fileName", "tmp");
tempFile.deleteOnExit();
FileOutputStream outputStream = new FileOutputStream(tempFile);
IOUtils.copy(inputStream, outputStream);
log.infof("File '%s' is successfully uploaded.", fileName);
} catch (AntivirusException | IOException e) {
throw new BadRequestException(e);
} finally {
stopWatch.stop();
log.infof("File '%s' processed in %s.", fileName, stopWatch.toString());
}
return Response.ok(stopWatch.toString()).status(Response.Status.CREATED).build();
} |
The fact that you read the whole file contents in memory is a bit concerning (at least from a resource usage perspective), but I understand that this is a test case and that you can do it from a temporary file |
Yep and because the InputStream from JAXRS when you call |
I put a Beta 0.0.1 out if anyone wants to try it and submit feedback. @sberyozkin I documented in the README.MD how to plug your own AV engine in. |
Description
Interested in this extension, please +1 via the emoji/reaction feature of GitHub (top right).
These days most applications allow some kind of file upload to either bulk load data, or proof of documentation, or any other good reason. These files should always be scanned for viruses upon entry and rejected if a virus is found.
Implementations
Currently we have implementations for:
The implementation should allow checking either ClamAV or VirusTotal or BOTH if the user want to check multiple engines.
Future Proof
Allow a pluggable architecture to allow users to plug their own AV Engine in easily.
Dev Service
Allow users to connect to existing ClamAV or start a DevContainer if desired running the latest ClamAV.
Dev UI
Configuration
Health Check
Repository name
quarkus-antivirus
Short description
Virus scan files using ClamAV or VirusTotal
Repository Homepage URL
https://www.clamav.net/
Repository Topics
...
Team Members
Additional context
I have been using this code both in PrimeFaces and at my clients for years.
The text was updated successfully, but these errors were encountered: