Old RECAPDocuments incorrectly re-indexed into the recap_sweep index

[albertisfu]

@mlissner reported receiving a RECAP alert email containing old alerts mostly related to documents last updated in November 2024, which is unusual.

After investigating further, I confirmed that these alerts were triggered by the sweep index. I found the documents related to the alerts in the recap_sweep index. Additionally, I confirmed that the recap_sweep index was recreated today, Jan 17, at 8:01 UTC, so the issue does not appear to be caused by an outdated version of the index.

Using the following query:

GET recap_sweep/_search?pretty
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "docket_child": "recap_document"
          }
        }
      ]
    }
  },
  "_source":{
      "includes":[
         "timestamp"
      ]
   },
   "sort":[
      {
         "timestamp":{
            "order":"desc"
         }
      }
   ],
  "size":100,
  "track_total_hits":true
}

And then changing the sort order to asc:

I found that the newest RD in the index has a timestamp of: 2024-11-16T07:59:56,
and the oldest one has a timestamp of: 2024-11-15T08:00:00.

This is strange, as it seems it's re-indexing documents from November 16 instead of today.

The query that performs this re-indexing is:

{
                        "has_parent": {
                            "parent_type": "docket",
                            "query": {
                                "range": {
                                    "timestamp": {
                                        "gte": today_datetime_iso,
                                        "lt": next_day_utc_iso,
                                    }
                                }
                            },
                        }
                    }

I suspect that today_datetime_iso and next_day_utc_iso might be incorrect.

We need to investigate further to confirm whether this is a code bug or if the server time in the deployment environment is incorrect.