Issues with HPC Storage and GitHub (GitHub v3.8): Any Advice?

[Oliviabarn]

Hello everyone,

I am facing an issue while integrating HPC storage systems with GitHub (v3.8). I’ve been using GitHub to manage code and datasets, but there are a few challenges I’m struggling to resolve when working with HPC storage.
Here are the main issues I’m encountering:

Synchronizing large files and datasets: Managing the transfer of large files between HPC storage and GitHub (v3.8) repositories has been difficult. The size of these files often exceeds standard repository limits, making it challenging to maintain version control effectively.

Collaboration and access management: With multiple users working on the same projects, ensuring smooth collaboration between team members and avoiding version conflicts has become a priority. HPC storage systems add another layer of complexity to this workflow.

Data integrity during transfer: I’ve noticed inconsistencies and errors when transferring files back and forth between HPC storage and GitHub (v3.8). These errors are affecting both performance and reliability.

Workflow optimization: I’m also exploring ways to automate repetitive tasks, like syncing datasets or updating repositories, but the setup has been tricky with the current tools I’m using.

I would greatly appreciate it if anyone could share advice or insights on resolving these challenges. What are the best practices for using GitHub (v3.8) with HPC storage? Are there any specific tools or configurations that can simplify this process?

Looking forward to your suggestions.

Thank you!

[mecodeatlas]

Thanks for posting in the GitHub Community, @Oliviabarn!

We're happy you're here. You are more likely to get a useful response if you are posting your question in the applicable category, the Discussions category is solely related to conversations around the GitHub product Discussions. This question should be in the Repositories category. I've gone ahead and moved it for you. Good luck!