How to update Credential Digger

Update library

First of all, update the credential-digger project with git pull (or git up in case you configured it). This step is not mandatory (unless you manually run the UI via python ui/server.py), but strongly suggested.

If you use the python library, you can upgrade directly the credentialdigger package:

pip install --upgrade credentialdigger

In case you have installed Credential Digger in a virtual environment, remember to activate it before launching the upgrade command.

If you want to build the project from scratch, you can follow the normal steps described in the README.

Please note that in case of major upgrade of the project, some database definitions may have changed. So, you may need to refer to the following Major upgrades of the library section.

Update Docker container

Since there are 2 different setups for Docker, the update strategy depends on the setup you're running.

Single container (UI with sqlite database)

In this case, an upgrade will delete the database. In case you want to save your data, you can export it:

docker cp credential_digger_sqlite:/credential-digger-ui/data.db .

To update the container, we suggest to delete it, delete its image, and restart it from scratch:

# Stop container
docker stop credential_digger_sqlite  
# Remove container
docker rm credential_digger_sqlite  
# Remove image
docker image rm credential-digger_sqlite
# Recreate and restart
docker-compose up --build 

In case you want to restore the old database, you can copy it in the new container (please note that you must have exported it before):

docker cp ./data.db credential_digger_sqlite:/credential-digger-ui/.

Please note that in case of major upgrade of the project, some database definitions may have changed. So, you may need to refer to the following Major upgrades of the library section.

Two containers: UI and postgres

In case the Docker setup is based on postgres, then you will have 2 different containers running: the former for the UI (i.e., the backend), the latter for the database (postgres). Thus, with this configuration, we need to recreate only the container running the UI. All the data are stored in postgres (i.e, in the other container), and will be persisted with the update

# Stop container
docker stop credential_digger_backend  
# Remove container
docker rm credential_digger_backend
# Remove image
docker image rm credential-digger_pg
# Recreate and restart
docker-compose -f docker-compose.postgres.yml up --build credential_digger

Again, please note that in case of major upgrade of the project, some database definitions may have changed. So, you may need to refer to the following Major upgrades of the library section.

Major upgrades of the library

Most skilled developers may want to run a smooth upgrade of a major version without breaking their code and without losing data. In this case, refer to the version upgrade you are to run in the following subsections.

Upgrading Credential Digger from v3.x to v4.x

With this major upgrade we need to add one table (i.e., embeddings) in the database.

If using Sqlite and Docker you will need to run a shell inside the container by entering docker exec -it <container-id> /bin/bash in a terminal
Then type sqlite3 <db_name.db> to connect to the database, and create a new table by running the following query:

CREATE TABLE IF NOT EXISTS embeddings (
    id INTEGER REFERENCES discoveries,
    snippet TEXT,
    embedding TEXT,
    repo_url TEXT REFERENCES repos,
    PRIMARY KEY (id)
);

If using Postgres, connect to your database (e.g., by typing psql -h <hostname> -p <port> -U <username> -d <database> in a terminal - insert credentials stored in .env file) and create a new table by running the following query:

CREATE TABLE embeddings (
    id INTEGER REFERENCES discoveries,
    embedding TEXT,
    snippet TEXT,
    repo_url TEXT REFERENCES repos,
    PRIMARY KEY (id)
);

Embeddings for discoveries of previously completed scans will automatically be added when updates of similar snippets are triggered in the corresponding repositories.