Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance DataCollector to Validate model_reporters Functions #2606

Open
peter-kinger opened this issue Jan 10, 2025 · 3 comments
Open

Enhance DataCollector to Validate model_reporters Functions #2606

peter-kinger opened this issue Jan 10, 2025 · 3 comments

Comments

@peter-kinger
Copy link

Describe the bug
Currently, the DataCollector's model_reporters validation has limitations when handling string attributes. The issue arises because:

  1. Model reporters can accept four types of inputs:

    model_reporters = {
        "Count": lambda m: len(m.agents),              # Lambda function
        "Status": "market_status",                     # Attribute name (string)
        "Metric": model.calculate_metric,              # Method reference
        "Custom": [calculate_wealth, [param1, param2]] # Function with parameters
    }
  2. When using string attributes (e.g., "market_status"), there's no validation to check if these attributes actually exist in the model. This leads to:

    • Silent failures when collecting undefined attributes
    • Confusion when attributes are misspelled or not initialized
    • Delayed error detection until runtime

Adding a validation mechanism to ensure model_reporters keys mapped to callable functions or valid strings would greatly improve error detection and provide better debugging experiences for users.

Expected behavior
When using DataCollector, if an invalid value is provided in model_reporters (e.g., a non-callable object or undefined function):

Raise a ValueError with a clear error message, such as:
"Invalid reporter for 'Gini': must be a callable function/method or a valid attribute name as a string."
This prompt will help users:

  1. Understand their mistake.
  2. Quickly identify whether the provided function (e.g., compute_gini) is missing, incorrectly implemented, or improperly referenced.

To Reproduce

Valid Example:

`from mesa.datacollection import DataCollector  

def compute_gini(model):     
	return 0.5  

collector = DataCollector(     
	model_reporters={"Gini": compute_gini},     
	agent_reporters={"Wealth": "wealth"} 
)`

Invalid Example (Missing Function):

`from mesa.datacollection import DataCollector  

collector = DataCollector(     
	model_reporters={"Gini": "compute_gini"},  # Error: compute_gini is not callable     
	agent_reporters={"Wealth": "wealth"} 
)`

Expected Output:

`ValueError: Invalid reporter for 'Gini': must be a callable function/method or a valid attribut`

Proposed Solution
Modify the DataCollector class to include validation for model_reporters, similar to the following:

def _validate_model_reporter(self, name, reporter): 
	# Type 1: Lambda function
	if callable(reporter): 
		return 
	# Type 2: Class attributes (string)
	elif isinstance(reporter, str): 
		return 
	# Type xxx: other types mechanism to warn
	else: 
		raise ValueError(
		 f"Invalid reporter for '{name}': must be a callable function/method or a valid attribute name as a string." 
		)

Additional context

@peter-kinger
Copy link
Author

I have created a PR to address this issue. Please refer to: #2605

@quaquel
Copy link
Member

quaquel commented Jan 10, 2025

I like the idea of this issue and the PR. I hope to find time to look at it more closely soon. Its a part of the code base I am not particularly familiar with, however.

@peter-kinger
Copy link
Author

I like the idea of this issue and the PR. I hope to find time to look at it more closely soon. Its a part of the code base I am not particularly familiar with, however.

Thanks for the attention to issues. During the progress of the PR, there were some problems. Let me try to fix them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants