Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graph database integration #692

Closed
5 tasks done
noboruma opened this issue Nov 28, 2022 · 0 comments
Closed
5 tasks done

Graph database integration #692

noboruma opened this issue Nov 28, 2022 · 0 comments
Assignees
Labels
area/agent All the agent related issues area/api All the backend and API issues Backend Issues those will be fixed through Backend enhancement New feature or request

Comments

@noboruma
Copy link
Collaborator

noboruma commented Nov 28, 2022

Is your feature request related to a problem? Please describe.
Today we compute the Graph data in-memory via scope.
This comes with a couple of drawbacks:

  • Pain point 1: Backend service cannot be scaled properly
  • Pain point 2: The data representation of the graph is not easily accessible outside our code base

Describe the proposed solution
We are proposing to replace scope's in-memory graph representation with neo4j.
With neo4j, we can scale up to at least 10k nodes, where previously we were limited to 1k nodes.
See the https://github.com/deepfence/ThreatMapper/tree/neo4j branch for more details.

Alongside the graph representation change, we will use that opportunity to revamp ThreatMapper code. Our plan is to:

  • Rework our backend architecture to bring in additional performance and scalability

See the architecture below:

new_backend_design drawio(4) drawio(4)

Components/Services

  • UI/Frontend
  • API/Backend
  • Agent
  • Deployment/YAMLs
  • CI/CD Integration
@noboruma noboruma added enhancement New feature or request area/api All the backend and API issues area/agent All the agent related issues Backend Issues those will be fixed through Backend labels Nov 28, 2022
@noboruma noboruma linked a pull request Nov 28, 2022 that will close this issue
Closed
noboruma added a commit that referenced this issue Dec 9, 2022
noboruma added a commit that referenced this issue Dec 10, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 12, 2022
noboruma added a commit that referenced this issue Dec 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/agent All the agent related issues area/api All the backend and API issues Backend Issues those will be fixed through Backend enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants