Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CIP-0097? | Representing the Script Context as a SOP term #507

Closed

Conversation

michaelpj
Copy link
Contributor

@michaelpj michaelpj commented Apr 20, 2023

This is the other part split out from #455 that talks about changing the representation of the script context to be a SOP term rather than Data. It depends on CIP-85.


(rendered proposal in branch)

@rphair rphair changed the title Representing the Script Context as a SOP term CIP-???? | Representing the Script Context as a SOP term Apr 20, 2023
@rphair rphair added the Category: Plutus Proposals belonging to the 'Plutus' category. label Apr 20, 2023
@L-as
Copy link
Contributor

L-as commented Apr 20, 2023

What is blocking us from simply making handling Data faster? Why is it so much slower than SOPs?

@MicroProofs
Copy link
Contributor

Now that I understand how SOPs will work in the vm. I'll make some examples comparing and contrasting using data in Aiken vs using sop in Aiken. That way I can show off use case costs. Gonna work on the these examples this week.

2. Translating ("decoding") the script context into a native form ([Scott encoding][] or SOPs) is far too slow because of 1 and because script contexts become large

These combine to make decoding real world script contexts prohibitive in many cases:
budget profiling of examples in `plutus-use-cases` that naively decode the script context revealed that many were spending 30-40% of their budget just on the decoding step, which is unacceptable.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these tests have been performed only on scripts generated using PlutusTX?

Copy link
Member

@KtorZ KtorZ May 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that only PlutusTx (as far as I know?) is trying to translate the script context into some intermediate representation, I'd assume that yes.

So it could be interesting to get some before/after benchmarks for solutions like Plu-ts or Aiken regarding the impact of using SOP vs Data. Though, if the script context still comes in the form of Data from the ledger, I believe that the overhead of turning a Data into a SOP will be still high. (forget that last bit, my mind was still on CIP-0085 for a second; this proposal here is precisely about representing the script context as a SOP)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is really only for people who actually try to convert the script context into another form. Although I do still think this is desirable - it's just obviously too expensive to be worth it at the moment.

So the numbers here are more meant to indicate how unacceptably high the cost is, not to argue that these are reductions that real users might see (the test cases in question are not real). I can see that's misleading, I'll rephrase.

The full specification of the new translation is to be determined in collaboration with the ledger team, but broadly:
- Integers and bytestrings are translated to Plutus Core integers and bytestrings instead of using the `I` and `B` Data constructors.
- Constructors are translated to Plutus Core `constr` terms instead of using the Data `Constr` constructor.
- Uses of the `Map` and `List` Data constructors will need more specific handling but will be translated in line with their underlying dataype representation in Plutus Core.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious on the current approach to handle maps and lists. Has this been fleshed out anymore?

CIP-????/README.md Outdated Show resolved Hide resolved
CIP-????/README.md Outdated Show resolved Hide resolved
CIP-????/README.md Outdated Show resolved Hide resolved
Co-authored-by: Ryan Williams <44342099+Ryun1@users.noreply.github.com>
@MicroProofs
Copy link
Contributor

MicroProofs commented May 2, 2023

I like SOP for its fast destructuring, but the problem is the blocker on missing out equals data and serialise_data. As a different compromise how about a new builtin that is similar to case. Say case_constr.

Takes in a constant of type data that is a constr and any term. Then it applies all field arguments from constr plus the constr index to that term. Then that single term can handle constr index and whatever fields it has applies. The current approach is to unconstr_data then fst_pair to check index, then access each field by doing some combination of head_list and tail_list. This is vastly improved by only requiring one builtin to get a constr index and its fields from a constant data that is type constr.

Now I know this step is computationally more expensive than SOP. But since we have a lot of builtins that operate on data, I wonder if operating on data in a more efficient way might be the solution. Even converting script context to SOP does not change that we still need better ways to operate on data because of datum, redeemer, and script context still has data fields in datums, inputs, outputs, and redeemers fields.

Here's an example @michele-nuzzi

[[(builtin caseConstr) (con data #D87A820540)] other_term]

this evaluate to

[[[other_term (con integer 1)] (con data #05)]  (con data #40)]

you could imagine other_term is:

(lam constr_index (force [
    [
      [
        (force (builtin ifThenElse)) 
        [[(builtin equalsInteger) (con integer 0)] constr_index]
      ]
      (delay (con bool True))
    ]

    (delay
        (force [
          [
            [
              (force (builtin ifThenElse)) 
              [[(builtin equalsInteger) (con integer 1)] constr_index]
            ]
            (delay (lam field1
                (lam field2 
                    (con bool True)
                )
            ))
          ]
        ])
    )
  ])
)

EDIT: and perhaps this builtin should be moved to a new CIP, but the more I think about it, even with SOP, the more essential a builtin like this is.

@michele-nuzzi
Copy link

not sure case_constr would turn more efficient than manual destructuring, especially for structs with many fields, but if it serves to keep only one representation I'm down to it

@MicroProofs
Copy link
Contributor

MicroProofs commented May 2, 2023

Typically I am seeing dApps that have use cases that check values, addresses, and datums of outputs against input. I'll come back and clarify some more in a bit but here's what I got.

Common Cases where I see SOP as more efficient:

  • Working with value, i.e. List<Pair<ByteArray, List<Pair<ByteArray, Int>>>>
  • Pulling out fields out of a constr like pulling out outputs, inputs, etc.

Common Cases where I see SOP as less inefficient:

  • Comparing addresses, like comparing an address that comes from a redeemer, datum, or input address, and comparing it to an output address. In this case especially if the address comes from a datum or redeemer. This is where equals data is really powerful
  • Comparing utxo references, i.e. transaction hash plus output index. Again comparing utxo references inputs to the Spend purpose field is much faster thanks to equals data. Also comparing a utxo ref to an output datum or output datum field that holds a utxo ref.

Cases where SOP has no effect:
Comparing redeemer/datum to output datums. This is a pain point where have a builtin like case_constr is nice since datum is still going to be data. I mean that datums and redeemers are given to us as data regardless of SOP on the script context.

@MicroProofs
Copy link
Contributor

not sure case_constr would turn more efficient than manual destructuring, especially for structs with many fields, but if it serves to keep only one representation I'm down to it

It would be more efficient than having to do head and tail list to get each constr field and get the constr index.

@KtorZ KtorZ changed the title CIP-???? | Representing the Script Context as a SOP term CIP-0097? | Representing the Script Context as a SOP term May 30, 2023
@@ -0,0 +1,205 @@
---
CIP: ?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
CIP: ?
CIP: 97

@rphair
Copy link
Collaborator

rphair commented Jun 9, 2023

(just noted this on Discord but probably better to address here)

@michaelpj would you come to our next CIP meeting - currently scheduled 20 June, 9:30 am UTC - to review this & invite any other Plutus representatives you think would have a stake in that issue? (cc @L-as)

@MicroProofs
Copy link
Contributor

(just noted this on Discord but probably better to address here)

@michaelpj would you come to our next CIP meeting - currently scheduled 20 June, 9:30 am UTC - to review this & invite any other Plutus representatives you think would have a stake in that issue? (cc @L-as)

Could I get an invite @rphair? The discord links don't work for me.

@rphair
Copy link
Collaborator

rphair commented Jun 9, 2023

yes @MicroProofs it would be good to have you there 🤓 https://discord.gg/kyaTyzkBqd

@rphair
Copy link
Collaborator

rphair commented Jun 9, 2023

p.s. @MicroProofs seems @michaelpj is unavailable for meeting this month so we have bumped the discussion until meeting # 69 (which if regularly scheduled would fall on 04 July).

@L-as
Copy link
Contributor

L-as commented Jun 22, 2023

I don't think we should do this. Being able to hash the script context is highly convenient for evaluating snarks on it, since you can use the hash of the script context as a circuit witness trivially this way. If you can't hash it cheaply, it gets an order of magnitude harder, which is not nice.

The real issue is that the Plutus interpreter is unperformant.

@rphair
Copy link
Collaborator

rphair commented Jul 2, 2023

@michaelpj @L-as @MicroProofs @michele-nuzzi this, with other Plutus items, is on the agenda for review early in the CIP editors' meeting Tuesday 04 July 16:00 UTC: https://hackmd.io/@cip-editors/69

@effectfully
Copy link

@michaelpj @L-as @MicroProofs @michele-nuzzi this, with other Plutus items, is on the agenda for review early in the CIP editors' meeting Tuesday 04 July 16:00 UTC: https://hackmd.io/@cip-editors/69

Michael is on leave, so I attempted to join instead of him, but it seems I wasn't let in. If you have any questions, I'd be happy to try to answer them, although I'm not sure what medium we should use.

@rphair
Copy link
Collaborator

rphair commented Jul 4, 2023

@effectfully: but it seems I wasn't let in

Are you talking about "let in" to Discord? If not, here's the invite link: https://discord.gg/kyaTyzkBqd ... if so, I don't see any problems reported in the #verification channel there... let's confirm you're using Discord first, then try to follow the event link (in less than 2 hours): https://discord.com/events/971785110770831360/1120631775404761139

@effectfully
Copy link

effectfully commented Jul 4, 2023

Are you talking about "let in" to Discord?

No, I took a meeting from Michael's calendar. Sorry, I should've just asked here. Thank you for the clarification.

I've now joined the Discord server.

let's confirm you're using Discord first, then try to follow the event link (in less than 2 hours): https://discord.com/events/971785110770831360/1120631775404761139

Unfortunately, in 2 hours from that moment (i.e. now in an hour) I'm going to be traveling likely without access to the Internet. I'll try to make it, but chances of that are not high.

@rphair
Copy link
Collaborator

rphair commented Jul 4, 2023

Tagging Waiting for Author due to @effectfully / @michaelpj report at last hour's CIP Editors' meeting that Plutus team is not currently sure this solution is preferable to alternatives they are also considering.

@rphair rphair added the State: Waiting for Author Proposal showing lack of documented progress by authors. label Jul 4, 2023
@rphair
Copy link
Collaborator

rphair commented Aug 23, 2024

@effectfully at your advice we can try to find some help to confirm, update as necessary & merge this... but without MPJ's original inspiration it seems this idea isn't going to move forward. Maybe you or @L-as @MicroProofs @michele-nuzzi could give us a tip if anyone in the community has the vision & persistence to document this in a way that will be acceptable to the current Plutus team. If so we'll remove the Likely Abandoned tag & work on moving this forward again.

On the off-chance that the Plutus team is doing this anyway, I'm tagging @zliu41 to indicate if this is so.

@rphair rphair added State: Likely Abandoned Close if confirmed abandoned (long waiting). and removed State: Waiting for Author Proposal showing lack of documented progress by authors. labels Aug 23, 2024
@MicroProofs
Copy link
Contributor

This was decided against due to the disadvantages that are brought by not having a Plutus Data encoded script context. A couple examples were using equalsData builtin for cheaper equality and using serializeData builtin for cheap access to hashing different parts of the script context extremely efficiently.

@zliu41
Copy link
Contributor

zliu41 commented Aug 23, 2024

This has been superseded by configurable script interfaces. The latter is an early stage idea, no CIP has been written, and we have no plan to pursue it in the near future. Instead we are working on making it easy to work with BuiltinData.

As such this CIP should be considered obsolete.

For anyone interested in learning about configurable script interfaces, look here.

@rphair rphair added State: Likely Deprecated Close if confirmed deprecated (or long waiting). and removed State: Likely Abandoned Close if confirmed abandoned (long waiting). labels Aug 23, 2024
@rphair
Copy link
Collaborator

rphair commented Aug 23, 2024

Thanks @MicroProofs @zliu41 for the conclusion & references - therefore closing in the Deprecated state.

@rphair rphair closed this Aug 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Plutus Proposals belonging to the 'Plutus' category. State: Likely Deprecated Close if confirmed deprecated (or long waiting).
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants