[#11878] Merge master into account-request-form (#12972)

* Update chrome driver download link in e2e-testing.md (#12924) * [#12048] Add SQL configuration into build.properties and build-dev.properties (#12917) * Add production config * Remove forgotten host and password * Fix lint --------- Co-authored-by: Zhang Ziqing <69516975+ziqing26@users.noreply.github.com> * [#12048] Add SQL description for postgres config (#12931) * Add production config * Remove forgotten host and password * Fix lint * Address changes, include production_user * Linting * [#12588] Improve test code coverage of core components - ToastComponent (#12916) * add test cases * add test case for isTemplate() --------- Co-authored-by: Cedric Ong <67156011+cedricongjh@users.noreply.github.com> Co-authored-by: Dominic Lim <46486515+domlimm@users.noreply.github.com> * [#12588] Add unit tests to question edit answer form (#12935) * add unit tests to constsum-options-question-edit-answer-form * add unit tests to constsum-options-question-edit-answer-form --------- Co-authored-by: Zhang Ziqing <69516975+ziqing26@users.noreply.github.com> * add delay to task queuer for indexing account request (#12936) Co-authored-by: Nicolas <25302138+NicolasCwy@users.noreply.github.com> * Make account req data migration script rerunnable (#12932) * [#12048] Relax read notif verification for migration verification script (#12937) * Fix account requests with wrong field during seed * Relax account attributes verification * Fix lint errors * Fix order of account request variables * [#12920] Create script to migrate noSQL test data to SQL schema format (#12922) * Add classes to migrate test json data * Add toposort script * Add function to remove foreign key data * Cleanup * WIP * Simplify keys for students and instructors * Fix lint issues * Output SQL JSON in same folder as JSON * Change output file name * Fix bug: wrong jsonkey used * Fix lint error * Make section and team name unique * Set read notification key to be unique * Delete python file * [#12588] Improve test code coverage of core components - ViewResultsPanelComponent (#12918) * add test cases to ViewResultsPanelComponent * fix lint errors --------- Co-authored-by: Dominic Lim <46486515+domlimm@users.noreply.github.com> Co-authored-by: Zhang Ziqing <69516975+ziqing26@users.noreply.github.com> * fix resetAccountAction (#12934) Co-authored-by: Zhang Ziqing <69516975+ziqing26@users.noreply.github.com> * [#12048] Migrate Feedback Rank Option E2E test (#12902) * Initial commit * Fix lint * Follow convention and add test * Change file path * Fix requested changes * Fixed testcases * Fix lint * Add deepcopy * Fixed e2e test --------- Co-authored-by: Wei Qing <48304907+weiquu@users.noreply.github.com> Co-authored-by: Cedric Ong <67156011+cedricongjh@users.noreply.github.com> * [#12048] Migrate FeedbackMcqQuestionE2ETest (#12820) * Migrate MCQ E2E * Fix lint * Fix lint * Update xml --------- Co-authored-by: Cedric Ong <67156011+cedricongjh@users.noreply.github.com> * [#12048] Remove unnecessary loading of datastore entities in InstructorNotificationsPageE2ETest (#12911) * migrate instructor notif e2e --------- Co-authored-by: Cedric Ong <67156011+cedricongjh@users.noreply.github.com> * [#12048] Migrate InstructorCourseDetailsPageE2ETest (#12908) * Add teammates.e2e.cases.sql.InstructorCourseDetailsPageE2ETest * Remove data properly to prevent clashes * Add SQL data bundle * Verify loaded details * Use email address when getting a student row * Check student links * Verify the sending of invites * Verify the reminding of all students to join * Remove SQL data properly to prevent clashes * Verify the downloading of the student list * Implement helper methods for Student * Add BaseTestCaseWithSqlDatabaseAccess::verifyAbsentInDatabase * Add to testng-e2e-sql.xml * Verify the deleting of students * Verify the deleting of all the students * Fix lint * Remove duplicate equality check for students * [#12588] add unit tests for question submission form (#12897) Co-authored-by: Zhang Ziqing <69516975+ziqing26@users.noreply.github.com> * Update developers.json (#12958) * Merge pull request #12960 from TEAMMATES/master (#12961) * [#12048] Fix account request indexing (#12967) * Add isTransactionNeeded method to Action * Remove delay from taskqueuer * Change CreateAccountRequest to handle own transactions * configure agroal connection pool (#12971) * Fix comment style for merge * Remove unnecessary check for account request --------- Co-authored-by: Nada Ayesh <nayesh10@students.iugaza.edu.ps> Co-authored-by: FergusMok <FergusMok1@gmail.com> Co-authored-by: Maureen Chang <76696006+techMedMau@users.noreply.github.com> Co-authored-by: Cedric Ong <67156011+cedricongjh@users.noreply.github.com> Co-authored-by: Dominic Lim <46486515+domlimm@users.noreply.github.com> Co-authored-by: Nicolas <25302138+NicolasCwy@users.noreply.github.com> Co-authored-by: Ching Ming Yuan <cmingyuan123@gmail.com> Co-authored-by: Wei Qing <48304907+weiquu@users.noreply.github.com> Co-authored-by: DS <yeodisheng@gmail.com> Co-authored-by: Jay Aljelo Ting <65202977+jayasting98@users.noreply.github.com>
TEAMMATES · Mar 31, 2024 · 99eeac7 · 99eeac7
1 parent 00b85ce
commit 99eeac7
Show file tree

Hide file tree

Showing 44 changed files with 3,486 additions and 173 deletions.
diff --git a/build.gradle b/build.gradle
@@ -77,6 +77,8 @@ dependencies {
     implementation("org.jsoup:jsoup:1.15.2")
     implementation("org.hibernate.orm:hibernate-core:6.1.6.Final")
     implementation("org.postgresql:postgresql:42.7.2")
+    implementation("org.hibernate.orm:hibernate-agroal:6.1.6.Final")
+    implementation("io.agroal:agroal-pool:2.1")
 
     testAnnotationProcessor(testng)
 

diff --git a/docs/e2e-testing.md b/docs/e2e-testing.md
@@ -39,7 +39,7 @@ Before running tests, modify `src/e2e/resources/test.properties` if necessary, e
 <panel header="#### Using Chrome" no-close>
 
 * You need to use chromedriver for testing with Chrome.
-  * Download the latest stable chromedriver from [here](https://sites.google.com/a/chromium.org/chromedriver/downloads).
+  * Download the latest stable chromedriver from [here](https://chromedriver.chromium.org/downloads).
     The site will also inform the versions of Chrome that can be used with the driver.
   * Specify the path to the chromedriver executable in `test.chromedriver.path` value in `test.properties`.
 

diff --git a/src/client/java/teammates/client/scripts/sql/DataMigrationForAccountRequestSql.java b/src/client/java/teammates/client/scripts/sql/DataMigrationForAccountRequestSql.java
@@ -1,10 +1,16 @@
 package teammates.client.scripts.sql;
 
+// CHECKSTYLE.OFF:ImportOrder
 import com.googlecode.objectify.cmd.Query;
 
 import teammates.common.datatransfer.AccountRequestStatus;
+import jakarta.persistence.criteria.CriteriaDelete;
+
+import teammates.common.util.HibernateUtil;
 import teammates.storage.sqlentity.AccountRequest;
 
+// CHECKSTYLE.ON:ImportOrder
+
 /**
  * Data migration class for account request entity.
  */
@@ -34,7 +40,8 @@ protected boolean isPreview() {
      */
     @Override
     protected void setMigrationCriteria() {
-        // No migration criteria currently needed.
+        // Prepare clean db before migration
+        cleanAccountRequestInSql();
     }
 
     /**
@@ -76,4 +83,15 @@ protected void migrateEntity(teammates.storage.entity.AccountRequest oldEntity)
 
         saveEntityDeferred(newEntity);
     }
+
+    private void cleanAccountRequestInSql() {
+        HibernateUtil.beginTransaction();
+
+        CriteriaDelete<AccountRequest> cdAccountReq = HibernateUtil.getCriteriaBuilder()
+                .createCriteriaDelete(AccountRequest.class);
+        cdAccountReq.from(AccountRequest.class);
+        HibernateUtil.executeDelete(cdAccountReq);
+
+        HibernateUtil.commitTransaction();
+    }
 }
diff --git a/src/client/java/teammates/client/scripts/sql/SeedDb.java b/src/client/java/teammates/client/scripts/sql/SeedDb.java
@@ -157,7 +157,7 @@ protected void persistAdditionalData() {
                 String accountRequestEmail = String.format("Account Email %s", i);
                 String accountRequestInstitute = String.format("Account Institute %s", i);
                 AccountRequest accountRequest = AccountRequestAttributes
-                        .builder(accountRequestName, accountRequestEmail, accountRequestInstitute)
+                        .builder(accountRequestEmail, accountRequestInstitute, accountRequestName)
                         .withRegisteredAt(Instant.now()).build().toEntity();
 
                 String accountGoogleId = String.format("Account Google ID %s", i);

diff --git a/src/client/java/teammates/client/scripts/sql/VerifyAccountAttributes.java b/src/client/java/teammates/client/scripts/sql/VerifyAccountAttributes.java
@@ -1,12 +1,8 @@
 package teammates.client.scripts.sql;
 
 // CHECKSTYLE.OFF:ImportOrder
-import java.time.Instant;
-import java.util.ArrayList;
-import java.util.Collections;
 import java.util.LinkedList;
 import java.util.List;
-import java.util.Map;
 
 import jakarta.persistence.TypedQuery;
 import jakarta.persistence.criteria.CriteriaBuilder;
@@ -17,7 +13,6 @@
 
 import teammates.common.util.HibernateUtil;
 import teammates.storage.entity.Account;
-import teammates.storage.sqlentity.ReadNotification;
 
 /**
  * Class for verifying account attributes.
@@ -87,18 +82,25 @@ public boolean equals(teammates.storage.sqlentity.Account sqlEntity, Account dat
             return false;
         }
 
-        Map<String, Instant> datastoreReadNotifications = datastoreEntity.getReadNotifications();
-        List<ReadNotification> sqlReadNotifications = sqlEntity.getReadNotifications();
+        return true;
 
-        List<Instant> datastoreEndTimes = new ArrayList<Instant>(datastoreReadNotifications.values());
-        Collections.sort(datastoreEndTimes);
+        // Not verifying read notification as current datastore implementation does not remove notifications
+        // that have been deleted from account entities. During migration, the notification will not be
+        // migrated since it is deleted and read notification will fail during migration (foreign key error)
+        // causing the verification to fail
 
-        List<Instant> sqlEndTimes = new ArrayList<>();
-        for (ReadNotification sqlReadNotification : sqlReadNotifications) {
-            sqlEndTimes.add(sqlReadNotification.getNotification().getEndTime());
-        }
-        Collections.sort(sqlEndTimes);
+        // Map<String, Instant> datastoreReadNotifications = datastoreEntity.getReadNotifications();
+        // List<ReadNotification> sqlReadNotifications = sqlEntity.getReadNotifications();
+
+        // List<Instant> datastoreEndTimes = new ArrayList<Instant>(datastoreReadNotifications.values());
+        // Collections.sort(datastoreEndTimes);
+
+        // List<Instant> sqlEndTimes = new ArrayList<>();
+        // for (ReadNotification sqlReadNotification : sqlReadNotifications) {
+        //     sqlEndTimes.add(sqlReadNotification.getNotification().getEndTime());
+        // }
+        // Collections.sort(sqlEndTimes);
 
-        return datastoreEndTimes.equals(sqlEndTimes);
+        // return datastoreEndTimes.equals(sqlEndTimes);
     }
 }
diff --git a/...lient/java/teammates/client/scripts/testdataconversion/ConvertDatastoreJsonToSqlJson.java b/...lient/java/teammates/client/scripts/testdataconversion/ConvertDatastoreJsonToSqlJson.java
@@ -0,0 +1,228 @@
+package teammates.client.scripts.testdataconversion;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.commons.io.FilenameUtils;
+
+import com.google.gson.JsonObject;
+
+import teammates.common.datatransfer.DataBundle;
+import teammates.common.datatransfer.SqlDataBundle;
+import teammates.common.exception.InvalidParametersException;
+import teammates.common.util.JsonUtils;
+import teammates.storage.sqlentity.Account;
+import teammates.storage.sqlentity.AccountRequest;
+import teammates.storage.sqlentity.Course;
+import teammates.storage.sqlentity.DeadlineExtension;
+import teammates.storage.sqlentity.FeedbackQuestion;
+import teammates.storage.sqlentity.FeedbackResponse;
+import teammates.storage.sqlentity.FeedbackResponseComment;
+import teammates.storage.sqlentity.FeedbackSession;
+import teammates.storage.sqlentity.Instructor;
+import teammates.storage.sqlentity.Notification;
+import teammates.storage.sqlentity.ReadNotification;
+import teammates.storage.sqlentity.Section;
+import teammates.storage.sqlentity.Student;
+import teammates.storage.sqlentity.Team;
+import teammates.test.FileHelper;
+
+/**
+ * Class to create JSON test data in SQL format from a noSQL JSON file.
+ * File can be run using the gradle execScript task and accepts a single argument which is the JSON path
+ * ./gradlew execScript -PuserScript="testdataconversion/ConvertDatastoreJsonToSqlJson" --args="JSON_FILE_PATH_HERE"
+ */
+public class ConvertDatastoreJsonToSqlJson {
+    private DataStoreToSqlConverter entityConverter;
+    private DataBundle dataStoreBundle;
+    private SqlDataBundle sqlDataBundle;
+
+    private String[] entitiesReferencedForeignKeys = new String[] {
+            "course",
+            "feedbackSession",
+            "section",
+            "account",
+            "giverSection",
+            "recipientSection",
+            "notification"};
+
+    protected ConvertDatastoreJsonToSqlJson(File inputFile) throws IOException {
+        this.entityConverter = new DataStoreToSqlConverter();
+
+        this.dataStoreBundle = loadDataBundle(inputFile.getCanonicalPath());
+    }
+
+    private String removeWhitespace(String string) {
+        return string.replaceAll("\\s", "");
+    }
+
+    private DataBundle loadDataBundle(String pathToJsonFile) throws IOException {
+        String jsonString = FileHelper.readFile(pathToJsonFile);
+        return JsonUtils.fromJson(jsonString, DataBundle.class);
+    }
+
+    private void saveFile(String filePath, String content) throws IOException {
+        FileHelper.saveFile(filePath, content);
+        System.out.println(filePath + " created!");
+    }
+
+    /**
+     * Amends foreign key references to only have ID field.
+     */
+    private void removeForeignKeyData(JsonObject obj) {
+        for (String entityName : entitiesReferencedForeignKeys) {
+            if (obj.get(entityName) != null) {
+                JsonObject entity = obj.get(entityName).getAsJsonObject();
+                for (String field : entity.deepCopy().keySet()) {
+                    if (!"id".equals(field)) {
+                        entity.remove(field);
+                    }
+                }
+            }
+        }
+    }
+
+    /**
+     * Read datstore json file and creates a SQL equivalent.
+     */
+    private void createSqlJson(File outputFile) throws IOException, InvalidParametersException {
+        sqlDataBundle = new SqlDataBundle();
+
+        migrateIndepedentEntities();
+        migrateDependentEntities();
+
+        // Iterates through all entities in JSON file and removes foreign entitity data except its ID
+        JsonObject sqlJsonString = JsonUtils.toJsonObject(sqlDataBundle);
+        for (String entityCollectionName : sqlJsonString.keySet()) {
+            JsonObject entityCollection = sqlJsonString.get(entityCollectionName).getAsJsonObject();
+            for (String entityName : entityCollection.getAsJsonObject().keySet()) {
+                JsonObject entity = entityCollection.get(entityName).getAsJsonObject();
+                removeForeignKeyData(entity);
+            }
+        }
+
+        String jsonString = JsonUtils.toJson(sqlJsonString);
+        saveFile(outputFile.getCanonicalPath(), jsonString + System.lineSeparator());
+    }
+
+    /**
+     * Migrate entities with no foreign key reference.
+     * Entities are account requests, usage statistics, courses, accouns, notifications
+     */
+    private void migrateIndepedentEntities() {
+        assert sqlDataBundle != null;
+
+        dataStoreBundle.accounts.forEach((k, datastoreAccount) -> {
+            Account sqlAccount = entityConverter.convert(datastoreAccount);
+            sqlDataBundle.accounts.put(k, sqlAccount);
+        });
+
+        dataStoreBundle.courses.forEach((k, datastoreCourse) -> {
+            Course sqlCourse = entityConverter.convert(datastoreCourse);
+            sqlDataBundle.courses.put(k, sqlCourse);
+        });
+
+        dataStoreBundle.accountRequests.forEach((k, accountRequest) -> {
+            AccountRequest sqlAccountRequest = entityConverter.convert(accountRequest);
+            sqlDataBundle.accountRequests.put(k, sqlAccountRequest);
+        });
+
+        dataStoreBundle.notifications.forEach((k, notification) -> {
+            Notification sqlNotification = entityConverter.convert(notification);
+            sqlDataBundle.notifications.put(k, sqlNotification);
+        });
+    }
+
+    /**
+     * Migrate entities which have dependence on each other or on the independent entities.
+     * The order which the entities were migrated was generated using a topological sort
+     * of its foreign key dependencies.
+     * Dependent entities: feedback sessions, sections, teams, users, students, instructors,
+     * deadline extensions, feedback questions, read notifications,
+     * feedback responses and feedback response comments.
+     */
+    private void migrateDependentEntities() {
+
+        dataStoreBundle.feedbackSessions.forEach((k, feedbackSession) -> {
+            FeedbackSession sqlFeedbackSession = entityConverter.convert(feedbackSession);
+            sqlDataBundle.feedbackSessions.put(k, sqlFeedbackSession);
+        });
+
+        dataStoreBundle.students.forEach((k, student) -> {
+            String jsonKey = removeWhitespace(String.format("%s-%s",
+                    student.getCourse(), student.getSection()));
+
+            if (!sqlDataBundle.sections.containsKey(jsonKey)) {
+                Section sqlSection = entityConverter.createSection(student);
+                sqlDataBundle.sections.put(jsonKey, sqlSection);
+            }
+        });
+
+        dataStoreBundle.students.forEach((k, student) -> {
+            String jsonKey = removeWhitespace(String.format("%s-%s-%s",
+                    student.getCourse(), student.getSection(), student.getTeam()));
+
+            if (!sqlDataBundle.teams.containsKey(jsonKey)) {
+                Team sqlTeam = entityConverter.createTeam(student);
+                sqlDataBundle.teams.put(jsonKey, sqlTeam);
+            }
+        });
+
+        dataStoreBundle.instructors.forEach((k, instructor) -> {
+            Instructor sqlInstructor = entityConverter.convert(instructor);
+            sqlDataBundle.instructors.put(k, sqlInstructor);
+        });
+
+        dataStoreBundle.students.forEach((k, student) -> {
+            Student sqlStudent = entityConverter.convert(student);
+            sqlDataBundle.students.put(k, sqlStudent);
+        });
+
+        dataStoreBundle.deadlineExtensions.forEach((k, deadlineExtension) -> {
+            DeadlineExtension sqlDeadline = entityConverter.convert(deadlineExtension);
+            sqlDataBundle.deadlineExtensions.put(k, sqlDeadline);
+        });
+
+        dataStoreBundle.feedbackQuestions.forEach((k, feedbackQuestion) -> {
+            FeedbackQuestion sqlFeedbackQuestion = entityConverter.convert(feedbackQuestion);
+            sqlDataBundle.feedbackQuestions.put(k, sqlFeedbackQuestion);
+        });
+
+        dataStoreBundle.accounts.forEach((k, account) -> {
+            List<ReadNotification> sqlReadNotifications = entityConverter.createReadNotifications(account);
+            sqlReadNotifications.forEach(notif -> {
+                String jsonKey = removeWhitespace(String.format("%s-%s",
+                            notif.getNotification().getTitle(), account.getEmail()));
+                sqlDataBundle.readNotifications.put(jsonKey, notif);
+            });
+        });
+
+        dataStoreBundle.feedbackResponses.forEach((k, feedbackResponse) -> {
+            FeedbackResponse sqlFeedbackResponse = entityConverter.convert(feedbackResponse);
+            sqlDataBundle.feedbackResponses.put(k, sqlFeedbackResponse);
+        });
+
+        dataStoreBundle.feedbackResponseComments.forEach((k, feedbackReponseComment) -> {
+            FeedbackResponseComment sqlFeedbackResponseComment = entityConverter.convert(feedbackReponseComment);
+            sqlDataBundle.feedbackResponseComments.put(k, sqlFeedbackResponseComment);
+        });
+    }
+
+    public static void main(String[] args) throws IOException, InvalidParametersException {
+        if (args.length > 0) {
+            File inputFile = new File(args[0]);
+            String fileExtension = FilenameUtils.getExtension(inputFile.getName());
+            if (!"json".equals(fileExtension)) {
+                throw new InvalidParametersException("The file provided is not a JSON file");
+            }
+
+            ConvertDatastoreJsonToSqlJson script = new ConvertDatastoreJsonToSqlJson(inputFile);
+            String outputFileName = FilenameUtils.getBaseName(inputFile.getName()) + "Sql.json";
+            File outputFile = new File(inputFile.getParent(), outputFileName);
+            script.createSqlJson(outputFile);
+        } else {
+            throw new InvalidParametersException("Required the path of the script to convert");
+        }
+    }
+}