Skip to content

Commit

Permalink
feat(search-indexer): Nested document resolution (#15949)
Browse files Browse the repository at this point in the history
* Add explicit email and phone fields to team member content type

* Tag article with what children are below it

* Revert "Add explicit email and phone fields to team member content type"

This reverts commit 48c8ccc.

* Tag documents with their children ids

* Change nested resolution logic

* Also trigger a re-index on asset deletion

* Remove unnecessary variable

* Reverse conditional when removing duplicates

* Handle paginated responses from ES

* Move nested entry resolution logic into a method

* Change comment

* Add logs

* Update comment

* Add type hint to extractChildEntryIds function

* Chunk root entry fetches

* Add comment

* Paginate re-sync fetches

* Paginated fetching during initial sync

* Check for chunk size

* Add a way to override sync token

* Only get next sync token on non delta updates

* Change return

* Copy parameter

* Remove if check from indexing.service

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
  • Loading branch information
RunarVestmann and kodiakhq[bot] authored Sep 19, 2024
1 parent 92485f0 commit 5d3444a
Show file tree
Hide file tree
Showing 26 changed files with 552 additions and 155 deletions.
20 changes: 20 additions & 0 deletions libs/cms/src/lib/search/cmsSync.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,26 @@ export class CmsSyncService implements ContentSearchImporter<PostSyncOptions> {
return hashResult.hash.toString()
}

async getNextSyncToken(syncType: SyncOptions['syncType']) {
if (syncType === 'fromLast') {
return ''
}

let nextPageToken: string | undefined = ''
let nextSyncToken = ''

// We don't get the next sync token until we've reached the last page
while (!nextSyncToken) {
const response = await this.contentfulService.getSyncData({
initial: true,
nextPageToken,
})
nextPageToken = response.nextPageToken
nextSyncToken = response.nextSyncToken
}
return nextSyncToken
}

// this is triggered from ES indexer service
async doSync(
options: SyncOptions,
Expand Down
Loading

0 comments on commit 5d3444a

Please sign in to comment.