Reintegrate dev/v0.10 (#87)

* Bump version to 0.10a0

* Follow v0.8 -> v0.9 API churn

AbsPlainT3Unit -> AbsT4Unit
AbsReviewT3Unit -> AbsT3Unit

* Follow AlertConsumer API churn

* RapidBase: slack -> slack_sdk

* ruff: ignore more files

* ci: run on py311

PyTorch stable does not support 3.12 yet

See: pytorch/pytorch#110436

* T2RiseDeclineBase: inherit from LogicalUnit

AmpelBaseModel subclasses can't set extra attributes by default.

* test: init parameterized model via dict

* conf: follow T3ReviewUnit churn

* chore: activate warnings

* ligo not our problem

* ignore numpy header size warning

* fixup! chore: activate warnings

* T2RiseDeclineStat: squash numpy scalar conversion warning

* SlackSummaryPublisher: fillna in bulk for pandas 3.0 compat

* chore(deps): refresh lockfile

* Bump version to 0.10.0a0

* HealpixCorrPlotter: conform to matplotlib 3.9 API

* Remove vestigial ResourceDependentConsumer

* Bump version to 0.10.0a1

* T2RiseDeclineBase: demote to plain class

Since this will always be mixed into the inheritance chain of
AmpelUnit or BaseModel, both of which harvest annotations from their
bases, there's not actually a reason for this to be a BaseModel
itself.

* PlotTransientLightCurves: require no slack token by default

* Convert stellar_outburst example to v0.10 syntax

* chore(deps): advance ampel-core to 0.10.1

* chore: explciitly annotate typeddict arg

* chore: cast t2 payloads to dict

Ideally there would be a declared result type, such that downstream consumers can be type checked

* ruff: remove empty comments

* ruff: simplify

* ruff: % to f-strings

* ruff: simplify min calls

* T2RunTDE: unnecessary staticmethod

* ci: update ruff

* T2ElasticcReport: bail if either z is None

* ruff: format

* ruff: fixup line endings

* chore: hide type checking behind facade

* Remove remaining casts

* Ensure payload is Mapping

* fix(deps): advance ampel-interface to v0.10.3a0

* fix(deps): advance ampel-interface to v0.10.3a1

* ruff: cleanup

* thou shalt not print()

* renovate: baseBranches

* fix: use typing_extensions.TypedDict in models

for python<3.12 compatibility.  typing.TypedDict lacks necessary
metadata before python 3.12; see
https://docs.pydantic.dev/2.3/usage/types/dicts_mapping/#typeddict

* CostCounter: return TypedDict

* chore(deps): flip snpy back to mainline

now compatible with both PEP517 _and_ scipy 1.14

* DCachePublisher: reimplement in terms of requests

* TNSClient: replace aiohttp with requests

good riddance, i say

* chore(deps): remove ruff

* chore: move t4 units to t4 directory

* chore: sort unit config

* chore: remove more commented-out print()

* chore: Update README for 73f2591

.github/workflows/main.yml

@@ -12,6 +12,7 @@ on:
     branches:
       - main
       - dev/*
+      - stable/*
 
 jobs:
   ci:
@@ -20,6 +21,9 @@ jobs:
       PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
     with:
       mongo: true
-      python-version: "3.10"
+      # renovate: datasource=python-version depName=python versioning=python
+      python-version: "3.11"
+      # renovate: datasource=pypi depName=ruff versioning=pep440
+      ruff-version: "0.7.4"
       lint: true
       format: true

.pre-commit-config.yaml

@@ -3,7 +3,8 @@
 
 repos:
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.1.14
+  # renovate: datasource=pypi depName=ruff versioning=pep440
+  rev: v0.7.4
   hooks:
     - id: ruff
       args: [--fix]

README.md

@@ -81,19 +81,21 @@ requires an access token if data is to be retrieved.
 - [CostCounter](ampel/contrib/hu/t3/CostCounter.py): Derive metrics for the total cost, as parsed by the provided documents.
 - [ElasticcClassPublisher](ampel/contrib/hu/t3/ElasticcClassPublisher.py): This unit is intended to submit classifications to the DESC TOM db during the ELAsTICC LSST alert simulation.
 - [HealpixCorrPlotter](ampel/contrib/hu/t3/HealpixCorrPlotter.py): Compare healpix coordinate P-value with output from T2RunSncosmo..
-- [HealpixTokenGenerator](ampel/contrib/hu/t3/HealpixTokenGenerator.py): Based on a URL to a Healpix map.
 - [PlotLightcurveSample](ampel/contrib/hu/t3/PlotLightcurveSample.py): Unit plots results from lightcurve fitters (RunSncosmo, RunParsnip).
 - [PlotTransientLightcurves](ampel/contrib/hu/t3/PlotTransientLightcurves.py): Create a (pdf) plot summarizing lightcurves of candidates provided to the unit.
-- [RandomMapGenerator](ampel/contrib/hu/t3/RandomMapGenerator.py): Generate smoothed circular healpix probability values around a random coordinate..
 - [RapidBase](ampel/contrib/hu/t3/RapidBase.py): Trigger rapid reactions.
 - [RapidLco](ampel/contrib/hu/t3/RapidLco.py): Submit LCO triggers for candidates passing criteria..
 - [RapidSedm](ampel/contrib/hu/t3/RapidSedm.py): Select transients for rapid reactions.
 - [ScoreSingleObject](ampel/contrib/hu/t3/ScoreSingleObject.py): Calculate score based on how early a specific SN is detected.
 - [ScoreTNSObjects](ampel/contrib/hu/t3/ScoreTNSObjects.py): Calculate score based on detection time reported to TNS, if any..
 - [SlackSummaryPublisher](ampel/contrib/hu/t3/SlackSummaryPublisher.py)
-- [StreamTokenGenerator](ampel/contrib/hu/t3/StreamTokenGenerator.py): Stream based token generator for.
 - [SubmitTNS](ampel/contrib/hu/t3/SubmitTNS.py): Submit candidates to TNS (unless already submitted).
 - [TransientInfoPrinter](ampel/contrib/hu/t3/TransientInfoPrinter.py)
 - [TransientTablePublisher](ampel/contrib/hu/t3/TransientTablePublisher.py): Construct a table based on selected T2 output values.
 - [TransientViewDumper](ampel/contrib/hu/t3/TransientViewDumper.py)
 - [VOEventPublisher](ampel/contrib/hu/t3/VOEventPublisher.py): Unit for creating a VOEvent pased on T2output and transient LightCurve.
+
+### T4 units (control):
+- [HealpixTokenGenerator](ampel/contrib/hu/t4/HealpixTokenGenerator.py): Based on a URL to a Healpix map.
+- [RandomMapGenerator](ampel/contrib/hu/t4/RandomMapGenerator.py): Generate smoothed circular healpix probability values around a random coordinate..
+- [StreamTokenGenerator](ampel/contrib/hu/t4/StreamTokenGenerator.py): Stream based token generator for.

ampel/contrib/hu/alert/DynamicShaperAlertConsumer.py

@@ -9,6 +9,7 @@
 
 
 from ampel.alert.AlertConsumer import AlertConsumer
+from ampel.core.EventHandler import EventHandler
 from ampel.ingest.ChainedIngestionHandler import ChainedIngestionHandler
 from ampel.log import AmpelLogger
 from ampel.model.ingest.CompilerOptions import CompilerOptions
@@ -46,7 +47,10 @@ class DynamicShaperAlertConsumer(AlertConsumer):
 
     # Overload
     def get_ingestion_handler(
-        self, run_id: int, updates_buffer: DBUpdatesBuffer, logger: AmpelLogger
+        self,
+        event_hdlr: EventHandler,
+        updates_buffer: DBUpdatesBuffer,
+        logger: AmpelLogger,
     ) -> ChainedIngestionHandler:
         # Update shaper
         # print('config first', self.shaper)
@@ -71,7 +75,7 @@ def get_ingestion_handler(
             shaper,
             self.directives,
             updates_buffer,
-            run_id,
+            event_hdlr.get_run_id(),
             tier=0,
             logger=logger,
             database=self.database,

ampel/contrib/hu/t0/RcfFilter.py

@@ -23,7 +23,9 @@ class RcfFilter(AbsAlertFilter):
 
     min_ndet: int = 1  #: number of previous detections
     min_dist_to_sso: float  #: distance to nearest solar system object [arcsec]
-    min_gal_lat: float  #: minium distance from galactic plane. set to negative to disable cut.
+    min_gal_lat: (
+        float  #: minium distance from galactic plane. set to negative to disable cut.
+    )
     min_age: float  #: Age as calculated based on previous PP in alert
     max_ipac_age: float  #: Age as calculated based on alert keywords
     max_magpsf: float
@@ -340,7 +342,7 @@ def is_variable_star(
         ):
             return True
 
-        if (
+        if (  # noqa: SIM103
             age > 90
             and alert["distnr"] > 0  # shouldn't this be distnr?  was distnbr
             and alert["distnr"] < 0.5
@@ -400,10 +402,8 @@ def process(self, alert: AmpelAlertProtocol) -> None | bool | int:
                 or al["isdiffpos"] == "0"
             ):
                 continue
-            if al["jd"] < jd_first_pps:
-                jd_first_pps = al["jd"]
-            if al["magpsf"] < m_peak:
-                m_peak = al["magpsf"]
+            jd_first_pps = min(al["jd"], jd_first_pps)
+            m_peak = min(al["magpsf"], m_peak)
             if al["magpsf"] < self.max_magpsf:
                 bright_detections += 1
 
@@ -426,8 +426,7 @@ def process(self, alert: AmpelAlertProtocol) -> None | bool | int:
                     return codes.max_ipac_age
             except KeyError:
                 self.logger.debug(
-                    "%s No jd end or start alert keywords. Letting through."
-                    % (alert.id)
+                    f"{alert.id} No jd end or start alert keywords. Letting through."
                 )
 
         # SEARCH POINT SOURCE UNDERNEATH