Handeling ES, mongo DB and raw files.

[RubelMozumder]

@ka-sarthak, I think it would fix deleting a nomad entry from Mongo, ES, and staging file systems. Please, take a look into it.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request refactors the file writing process to handle Elasticsearch, MongoDB, and raw files more efficiently. It introduces a new function delete_entry_file to manage file deletion and updates the write_file function to use this new function. The changes also improve error handling and logging, and simplify the logic for writing different file types.

Sequence diagram for updated file writing process

sequenceDiagram
    participant Client
    participant WriteFile
    participant DeleteEntry
    participant Storage
    participant ES
    participant MongoDB

    Client->>WriteFile: write_file()
    alt nexus file
        WriteFile->>DeleteEntry: delete_entry_file()
        DeleteEntry->>Storage: Check if file exists
        DeleteEntry->>Storage: Delete raw files
        DeleteEntry->>MongoDB: Delete entry
        DeleteEntry->>ES: Delete by query
        WriteFile->>Storage: Write NeXus file
        WriteFile->>Storage: Process updated raw file
    else hdf5 file
        WriteFile->>Storage: Write HDF5 file
    end

Loading

Class diagram for file handling components

classDiagram
    class WriteFile {
        +write_file()
        -_write_nx_file()
        -_write_hdf5_file()
    }

    class DeleteEntryFile {
        +delete_entry_file(archive, mainfile, delete_entry)
    }

    class Upload {
        +upload_id
        +published
    }

    class Entry {
        +entry_id
        +upload_id
        +mainfile
    }

    class StagingUploadFiles {
        +delete_rawfiles(path)
    }

    WriteFile ..> DeleteEntryFile : uses
    DeleteEntryFile ..> Upload : checks
    DeleteEntryFile ..> Entry : manages
    DeleteEntryFile ..> StagingUploadFiles : uses

Loading

File-Level Changes

Change	Details	Files
Introduction of a new delete_entry_file function.	A new function delete_entry_file is introduced to handle the deletion of entry files and associated data in Elasticsearch and MongoDB. The function takes the archive, mainfile, and an optional delete_entry flag as arguments. It checks if the file exists and if the upload is published before proceeding with deletion. It uses StagingUploadFiles to delete raw files and delete_by_query to delete associated entries in Elasticsearch. If the delete_entry flag is set, it also deletes the entry from MongoDB.	src/nomad_measurements/utils.py
Refactoring of the write_file function.	The write_file function is updated to use the new delete_entry_file function. The logic for handling NeXus and HDF5 file writing is simplified. Error handling is improved by calling delete_entry_file with appropriate arguments in case of exceptions. The process_updated_raw_file function is called after writing the NeXus file. Redundant code for deleting files is removed.	src/nomad_measurements/utils.py
Removal of unused imports and code.	The unused import of Entry in the _write_nx_file function is removed. Redundant code for checking and deleting existing files before writing new ones is removed from _write_nx_file.	src/nomad_measurements/utils.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @RubelMozumder - I've reviewed your changes - here's some feedback:

Overall Comments:

• The new delete_entry_file() function needs proper documentation - please add a docstring describing its purpose, parameters, and possible exceptions (especially the PermissionError for published entries).

Here's what I looked at during the review

• 🟡 General issues: 1 issue found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

issue: Consider handling the case where Entry.first() returns None to avoid potential AttributeError

[TLCFEM]

Again, this whole thing relies on deleting things to make things work.
This often indicates other parts are wrongly implemented.
This kind of functionality should be discouraged and shall not be implemented in the first place, because it encourages potential risky and faulty code.

You have been warned.

[TLCFEM]

No need to pay the cost if not used.
So changes like this is better.

        if delete_entry and (entry := Entry.objects(
                upload_id=archive.m_context.upload_id, mainfile=mainfile
            ).first()):
                delete_by_query({'entry_id': entry.entry_id})
                entry.delete()

[TLCFEM]

The first input argument can directly be upload_id.

[TLCFEM]

Prefer explicit key-value argument for optional arguments.