Salting Partitions problem

[brianfhead]

I'm experiencing a problem after adding salting_partitions. I'm wondering if anything has changed with the syntax or you can help identify a problem with my code. I'm using this page to inform my approach. For a previous problem in the same step it was a change in syntax, which is why I'm asking.

Here's my code:
settings = { "link_type": "link_only" ,"blocking_rules_to_generate_predictions" : [ {"blocking_rule": "substr(l.mlg_addr,1,4) = substr(r.mlg_addr,1,4) and l.mlg_zip = r.mlg_zip", "salting_partitions": 200} ] "comparisons" : [ cl.DamerauLevenshteinAtThresholds("mlg_addr") ,cl.ExactMatch("mlg_zip").configure(term_frequency_adjustments=True) }

(this is a snippet of code I pulled out after commenting a bunch of other code out to try to isolate the problem).

Here's the error:
Py4JJavaError: An error occurred while calling o1318.checkpoint. : java.util.NoSuchElementException: key not found: 765

The error is basically the same one I was getting on the page I shared above where the problem was a change in syntax.

I'm dealing with a large data skew problem, thus the need for salting. Also, I've tried multiple values for the partitions. Given my skew problem I think what I have now is reasonable, but I'm pointing out I don't think that's the problem here as I've tried a small number (e.g., 4) too.

[brianfhead]

The problem does seem to be with "salting_partitions": 200} as when I remove that one piece the error does not appear and it runs--just a really long run and I'm trying to make it more efficient and faster.

[RobinL]

I don't think you're doing anything wrong. I haven't seen this error before - it's a bit difficult for us to look into it without a reproducible example on our side

[brianfhead]

Thanks, Robin. I understand it's not ideal to ask questions without being able to show code. Unfortunately, in my position I must be very careful about showing code.

The good news is I was able to work it out. I started by going back to the example here. I was able to replicate that without problems. So I knew it was something else on my end. I pared down my data base sampling 10% and worked through until I was able to get it to work. The only thing I can find that seems to be causing it is use of .configure(term_frequency_adjustments=True) on more than one field. If I run it with the adjustment on one field it runs without problems. If I run it on two or more I get the error. It turns out that my data are highly skewed toward the first block, so it makes me wonder if I should use the adjustment on that one block or not at all. All of the blocks are address related. Appreciate any insight you might have with this description.

[RobinL]

Still not sure, but you could try pre-computing the term frequency tables which will reduce the complexity of the overall workflow that Sprark is trying to compute.

You can do that using this:
https://moj-analytical-services.github.io/splink/api_docs/table_management.html#splink.internals.linker_components.table_management.LinkerTableManagement.compute_tf_table

Once it's run Splink 'knows' about the table, and so will not re-compute it e.g. when you run predict()

(There's no guarantees this fixes your problem, but seems worth a shot)

[brianfhead]

Thank you, Robin. I will look into this.