JobProcessor Improvements v2

[vonnieda]

This is a second attempt at the JobProcessor Refactor which was originally done in #265. The work accomplished most of the goals we needed, but it left a couple important ones out and it resulted in a somewhat inflexible system. Based on learning since then, and comments from users, I think we can do better.

This issue can be considered an epic for the following tasks:

Improvements

• Job Placements need a status field that shows the current status of that placement during job run. #205 Job Placement Status: Store placement status in job so that interrupted jobs can be recovered.
  • Save to disk after each status change.
• Job Placements Panel #397 Job Placements Panel: A new job panel which shows all placements in a job. Note: See new thoughts in the Ideas section below.
  • Allows reordering (Users should be able set the order of placements #219)
  • Shows status of each placement
  • Columns: Board Name (RO), Placement ID (RO), Status (RO), NozzleTip (RO), Ordinal (RW).
  • Actions: Plan (Reorders According to Planner)
  • Run is modified to check for selected placements in new panel. If there are selections, prompt user "Run full job or selected placements only?". Selections are marked Pending and all others are marked Disabled. (Run Job For Selected Placements Only #258)
• Users should be able set the order of placements #219 Decouple planning and processing so that planning can be done ahead of time and the user can make adjustments by changing placement order.
• Add an error handling option in job placement that will determine how the job continues in the case of an error. This field will be settable on the new job placements panel. (Improve handling of alignment errors #275)
  • Raise: The error will be raised to the user and they will be able to determine what to do next.
  • Skip: The error will be stored, the placement marked as skipped and the job will continue uninterrupted.
• Replace JobProcessor FSM with job placement state, possibly using command pattern:
  • Primary state moves to JobPlacement, or PlannedJobPlacement.
  • JobProcessor loop just does whatever the next action is for the next JobPlacement. The only way to get past that JobPlacement is for it to be marked Complete, Errored or Skipped.
  • The JobProcessor still has simple state but it is only related to what the JobProcessor itself is doing: Idle, Running, Stepping,
  • Rework the JobProcessor and PnpJobProcessor interfaces according to the API changes below. The changes allow for placement status, placement reordering, error handling, and disabling placements from running during a job which provides the ability to run only selected placements.
  • Placement.type Place becomes Placement, Ignore is removed, functionality is replaced with enabled in JobPlacement. Make sure to load and default old jobs.
• Stopping a job should stop the job as soon as possible #278 Stopping a job should stop the machine as soon as possible.
  • Fiducial checks are a big source of this issue. They run unimpeded during a stop since they are considered a single "step".

JobProcessor API Changes

interface JobProcessor
    void initialize(Job) throws Exception
    boolean next() throws Exception
    void stop() throws Exception
    JobProcessor.Status getStatus()

enum JobProcessor.Status
    Idle        Job can be started.
    Running Job is active, next can be called.
    Done	   Job finished normally.

interface PnpJobProcessor < JobProcessor
    List<JobPlacement> getJobPlacements()
    List<JobPlacement> plan() // plans the job, reordering the placements by the planner's criteria

interface JobPlacement
    JobPlacement.Status getStatus()
    boolean isEnabled()
    void setEnabled(boolean)
    int getOrdinal()
    void setOrdinal(int ordinal)
    JobPlacement.ErrorOption getErrorOption()
    void setErrorOption(JobPlacement.ErrorOption)

enum JobPlacement.Status
    Disabled     Disabled for job run, will not be processed.
    Pending      Enabled, not yet processed.
    Processing	Currently being worked on.
    Error		Processing failed, error is retained.
    Done		Finished without errors.

enum JobPlacement.ErrorOption
    Raise
    Skip

Notes and Ideas

• With the recent Placed status addition ( new approach to follow up: https://github.com/openpnp/openpnp/pull/626 #632 ) I'm starting to see a way this can come together without adding a whole new UI.

  • Make boards in the job panel multi select and add appropriate logic for button state.
    • This is a good opportunity to also add right click for Enable and Check Fids on the boards table.
  • When more than one board is selected or when no board is selected, show the placements for all the selected boards (or all boards) in the placements panel and add two new columns: Board name, ordinal.

  The idea here is that users can select all or none to see all the placements in the job. In this state they can set ordinals to reorder placements and this will be the order that placing happens. This state can also be used to easily monitor the entire job as it runs - you will be able to see each placement as it happens.

• Move lesser used placements columns into a popup dialog.
  The placements table is getting too wide. Should move "Check Fids?", type, and maybe the coordinates into a popup dialog. The type can be implied by a row color.

References

• Pick fail detect and discard Pick Fail Detect and Discard #102
• Job Placement Status Field Job Placements need a status field that shows the current status of that placement during job run. #205
• Usability Improvements Usability Improvements #207
• User defined placement order Users should be able set the order of placements #219
• Run Job For Selected Placements Only Run Job For Selected Placements Only #258
• Bottom Vision v2 Bottom Vision v2 #264
• Original JobProcessor Refactor JobProcessor Refactor #265
• Solder Paste Dispense Broken After JobProcessor Refactor Solder Paste Dispense Broken After JobProcessor Refactor #271
• Improve handling of alignment errors Improve handling of alignment errors #275
• Stopping a job should stop the job as soon as possible Stopping a job should stop the job as soon as possible #278
• Run Placement from this component on Run placement from this component on #325
• Feeder System Improvements Feeder System Improvements #357
• Job Placements Panel Job Placements Panel #397

[vonnieda]

Note, per discussion in https://groups.google.com/d/msgid/openpnp/4b284469-fb53-4e17-a5f4-45b696c23f7a%40googlegroups.com there is a bug where discard might discard for the wrong nozzle. I think what is happening is that skip is not taking the right planned placement into account.

[KarlZeilhofer]

Hi, I just crashed the nozzle into a 2.5mm high chip, while placing a C0805 next to that chip. The job processor seam to decided to place the big chip first, because the big nozzle was inserted. After that the small parts got placed, an the crash occurred.

Is there any chance for the user to see, what is going to happen?

[vonnieda]

@KarlZeilhofer Sorry to hear that. No, there is currently no way for the user to see or modify the order of placements. That's an important part of this issue's work.

[mgineer85]

hi @vonnieda,
I just made a patch to have in a multinozzle-setup the right nozzle cleared if skipping a part due to errors: #693

[markmaker]

Referring to the point...

Actions: Plan (Reorders According to Planner)

...is this rework in any way related to #686 ?

If so, some thoughts about this:

1. Ideally the Planner would be a separate object/class that one can choose in a Drop-Down much like the machine driver.

The Planner should then have a simple API, mostly just being notified when it is time to do the planning.

The Planner should in turn have access to the full job and machine data, because it may vary wildly from machine to machine by which criteria the Planner will order/group the placements. A multi-nozzle machine might want to order/group parts by feeder proximity to reduce moving time between pickups. Conversely a single nozzle machine user doesn't care about feeder position and might favour assembly by PCB X or Y. Another [retrofitted] machine with small Z headroom might require strict part height sorting. Yet another one might want to implement this crazy idea :-).

See also:
https://youtu.be/y_Fp3wLdmtk?t=1m19s

Perhaps the rework could also include the following considerations:

2. The placement could include states "not yet planned", "auto-planned", "manually planned" to allow for gradual determination of the placement ordinal.
3. Planning could then be incremental, the Planner may choose to implement a lazy evaluation and/or background and/or iterative optimization scheme.
4. There may be dominant order/grouping properties (such as similar part heights) pre-determining a sub-selection of candidate placements for lazy evaluation. This gradual (hierarchic) planning may offload the CPU on large jobs and/or complex optimization criteria.
5. Iterative planning may allow for algorithms that gradually improve the placement solution (i.e. Simulated Annealing) without unduly slowing down job start/failure recovery (i.e. after a feeder fails). The Planner may revisit placement ordinals which are "auto-planned", but not those that are "manually planned".
6. The Planner should be notified when various events happen (job started, paused, resumed, next placements needed, feeder failed, user changed data, idle, etc.) but it would be up to the Planner implementation, when the actual planning (i.e. the determination of the placement ordinal) takes place.

_Mark

[vonnieda]

Hi @markmaker,

Yes, the two are certainly related.

OpenPnP used to have a separate planner interface. I removed it because it was complex, hard to maintain, and did not fit well with the first major refactoring of the JobProcessor. And I also didn't think anyone was really using it.

One thing I'm struggling with is the competing requests of:

1. Users want to be able to manage the order of placements.
2. Developers want a planning interface.

I have been thinking that the order of placements might just be defined by the order in the table, and that the planner would be something you can run as an action to set the order of placements.

I would say that there have been far more requests for the ability to modify the placement order than there have been for a custom planner. I'd be interested in ideas on how to manage both.

Finally, I should note that the addition of a new planning system will probably be secondary to the JobProcessor refactor. My primary goal with the refactor is to solve some major bugs, and to fix some major usability issues. Adding the planner later on should not really be much more additional work since the area where planning happens is already quite self contained.

Jason

[vonnieda]

@markmaker See #219 for some more information and discussion on this.

[markmaker]

Hi Jason

thanks for your response.

One thing I'm struggling with is the competing requests of:

1. Users want to be able to manage the order of placements.
2. Developers want a planning interface.

Yes that's why I asked for the implementation to include states "not yet planned", "auto-planned", "manually planned". This way both demands could be met and even mixed.

This can be implemented very simply, say ordinal unplanned=0, auto-planned < 0, manually planned >0. The JobProcessor would then take the abs(ordinal) to sort the placements.

The only thing that would be nice at this point is for the ordinal handling/storing to support such a thing. The actual Planner on top of it can wait.

Why the states?

The idea is that the [future] planner is free to change the ordinal <= 0 on all placements but not those > 0 determined by the user. The best analogue that comes to my mind is auto-routing with Fritzing. Not that I use Fritzing nowadays but this was a very nice experience.

_Mark

[vonnieda]

Most of this is now fixed in 97e5bfc. I'm closing this issue and will open new issues for the parts that remain. Much of the above is outdated with the new work, so I think it's best to start fresh. If 97e5bfc does not address something you were concerned with in this issue, please file a new one.