Initial AI validator script (Draft) #357

rajlearner17 · 2025-06-25T08:10:11Z

No description provided.

…Summary and restructure AI section > Draft (#352) Co-authored-by: raj <raj@turbot.com> Co-authored-by: Sumit Popat <sumit@turbot.com> Co-authored-by: David Boeke <david@boeke.com>

Co-authored-by: raj <raj@turbot.com> Co-authored-by: David Boeke <david@boeke.com>

…for troubleshooting permission issues across multiple guides to ensure consistency and accuracy. (#364) Co-authored-by: raj <raj@turbot.com>

…oses #365 (#366)

…database upgrade guide and ensure caution message formatting is consistent.(#368) Co-authored-by: raj <raj@turbot.com>

…to include `Enforce: No Boundary` in case permission boundary is used. Closes #354 (#372)

#378)

Co-authored-by: Raj <raj@turbot.com>

…Checklist page. Closes #362 (#380)

#369 (#379)

Co-authored-by: Sumit Popat <sumit@turbot.com> Co-authored-by: raj <raj@turbot.com>

Co-authored-by: raj <raj@turbot.com>

… (#387)

github-actions · 2025-09-08T10:30:34Z

Preview Available 🚀

Commit Author: abhiturbot
Commit Message: Add changelog for te v5.51.8

Preview Link: turbot-com-git-docs-guardrailsadd-ai-validator-turbot.vercel.app

…ort. Closes #388 (#389) Co-authored-by: Sumit Popat <sumit@turbot.com>

…, and validation scripts. Remove deprecated validation scripts and prompts. Enhance documentation with examples and manual validation instructions.

github-actions · 2025-09-15T11:20:02Z

Preview Available 🚀

Commit Author: abhiturbot
Commit Message: Add changelog for te v5.51.8

Preview Link: turbot-com-git-docs-guardrailsadd-ai-validator-turbot.vercel.app

…o emphasize automatic and manual options, enhance installation instructions, and clarify usage examples. Improve overall structure and readability.

github-actions · 2025-09-15T11:44:31Z

Preview Available 🚀

Commit Author: abhiturbot
Commit Message: Add changelog for te v5.51.8

Preview Link: turbot-com-git-docs-guardrailsadd-ai-validator-turbot.vercel.app

…n script, and validation scripts. Implement rule-based and LLM validation methods, enhance documentation with usage examples, and provide manual validation instructions. Include example guides for both good and bad formatting.

Copilot

Pull Request Overview

Draft adds a rule-based and optional LLM-backed validator for Markdown guides, plus CLIs, prompts, examples, and documentation.

Adds core Python validator with rules loaded from prompt files and optional Anthropic LLM checks
Provides simple CLI wrappers (validate, validate-prompt), install script, examples, and docs
Introduces prompts for each section and a test harness to run sample validations

Reviewed Changes

Copilot reviewed 20 out of 22 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
guide-validator/validate-prompt	Bash wrapper to display prompts/rules via prompt_helper.py
guide-validator/validate	Bash wrapper to run validation against a guide using scripts/validate_guides.py
guide-validator/scripts/validate_guides.py	Main validator combining rule-based checks and optional LLM; config, I/O, and output formatting
guide-validator/scripts/test_validator.py	Simple test runner that validates example guides and checks exit codes
guide-validator/scripts/rule_based_validator_v2.py	Rule-based validator loading YAML rules from prompts and applying them to sections
guide-validator/scripts/prompt_helper.py	CLI for listing sections and showing rules/prompts from prompt files
guide-validator/requirements.txt	Python dependencies (PyYAML, optional anthropic)
guide-validator/prompts/*.md	Validator prompts and embedded YAML rules for each section
guide-validator/install.sh	Installer: checks Python/pip, installs deps, chmods scripts, runs tests
guide-validator/examples/*.md	Good/bad example guides for validation
guide-validator/docs/*.md	Docs showing manual validation usage and base style guide
guide-validator/config.yaml	Default configuration (validators, LLM, output)
guide-validator/README.md	Full usage guide, configuration, examples, CI tips

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-21T10:07:52Z

guide-validator/scripts/validate_guides.py

+        prompt_file = f"prompts/{validator_name.lower().replace(' ', '_')}-section-validator-prompt.md"
+        try:
+            with open(prompt_file, 'r', encoding='utf-8') as f:


The LLM prompt path ignores the configured prompts_dir and assumes a working-directory-relative "prompts/". If --prompts-dir points elsewhere (or CWD differs), LLM prompts won't be found. Use the same resolved prompts directory as the rule-based validator: prompt_file = Path(self.rule_validator.prompts_dir) / f"{validator_name.lower().replace(' ', '_')}-section-validator-prompt.md" and open(str(prompt_file), ...).

Suggested change

prompt_file = f"prompts/{validator_name.lower().replace(' ', '_')}-section-validator-prompt.md"

try:

with open(prompt_file, 'r', encoding='utf-8') as f:

prompt_file = Path(self.rule_validator.prompts_dir) / f"{validator_name.lower().replace(' ', '_')}-section-validator-prompt.md"

try:

with open(str(prompt_file), 'r', encoding='utf-8') as f:

Copilot · 2025-09-21T10:07:53Z

guide-validator/scripts/rule_based_validator_v2.py

+                if rules:
+                    # Extract the validator name from filename
+                    validator_name = file_name.replace('-section-validator-prompt.md', '').replace('-', '_')
+                    self.rules[validator_name] = rules


self.rules[validator_name] is set to the entire YAML mapping (e.g., {'overview': {...}}) instead of the section’s inner dict, causing validators to read the wrong level and silently skip checks (e.g., rules.get('required') reads None). Extract the section node: section_rules = rules.get(validator_name) or rules.get(f"{validator_name}s"); if section_rules is None and len(rules) == 1: section_rules = next(iter(rules.values())); then assign self.rules[validator_name] = section_rules.

Suggested change

self.rules[validator_name] = rules

section_rules = rules.get(validator_name) or rules.get(f"{validator_name}s")

if section_rules is None and len(rules) == 1:

section_rules = next(iter(rules.values()))

self.rules[validator_name] = section_rules

Copilot · 2025-09-21T10:07:53Z

guide-validator/scripts/rule_based_validator_v2.py

+        if 'callout' not in self.rules:
+            result['status'] = 'FAIL'
+            result['issues'].append('Callouts validation rules not found')
+            return result
+
+        rules = self.rules['callout']
+


There is a singular/plural mismatch across code, config, and prompt YAML (code expects 'callout', config.yaml uses 'callouts', prompt YAML root is 'callouts'), which makes enabling/disabling and rule loading inconsistent. Standardize on a single key (e.g., 'callout') across code, config, and prompt YAML, or add a plural fallback when loading and when reading config.

Suggested change

if 'callout' not in self.rules:

result['status'] = 'FAIL'

result['issues'].append('Callouts validation rules not found')

return result

rules = self.rules['callout']

# Support both singular and plural keys for callout rules

rules = None

if 'callout' in self.rules:

rules = self.rules['callout']

elif 'callouts' in self.rules:

rules = self.rules['callouts']

else:

result['status'] = 'FAIL'

result['issues'].append('Callouts validation rules not found')

return result

Copilot · 2025-09-21T10:07:53Z

guide-validator/scripts/validate_guides.py

+            for validator_name in ['Overview', 'Prerequisites', 'Steps', 'Troubleshooting', 'Callout']:
+                if self.config['validators'].get(validator_name.lower(), True):


The config lookup for Callout uses key 'callout' but config.yaml provides 'callouts', so toggling callout LLM validation via config has no effect. Derive the key with a mapping, e.g., key = 'callouts' if validator_name == 'Callout' else validator_name.lower(), or standardize the config key as 'callout' and update config/docs accordingly.

Suggested change

for validator_name in ['Overview', 'Prerequisites', 'Steps', 'Troubleshooting', 'Callout']:

if self.config['validators'].get(validator_name.lower(), True):

# Map validator names to config keys

validator_key_map = {

'Callout': 'callouts',

'Overview': 'overview',

'Prerequisites': 'prerequisites',

'Steps': 'steps',

'Troubleshooting': 'troubleshooting'

}

for validator_name in ['Overview', 'Prerequisites', 'Steps', 'Troubleshooting', 'Callout']:

config_key = validator_key_map.get(validator_name, validator_name.lower())

if self.config['validators'].get(config_key, True):

Copilot · 2025-09-21T10:07:53Z

guide-validator/scripts/prompt_helper.py

+                section_key = section_name.replace('-', '_')
+                if section_key in rules:
+                    rule_data = rules[section_key]
+                    for key, value in rule_data.items():
+                        if isinstance(value, list):
+                            print(f"  {key}: {', '.join(map(str, value))}")
+                        else:
+                            print(f"  {key}: {value}")
+                else:
+                    print(f"  No rules found for {section_name}")


This only prints rules if the YAML contains a key equal to the section name; it fails for singular/plural mismatches (e.g., 'callout' vs 'callouts') or when the YAML contains a single top-level key. Improve by: section_key_alt = f"{section_key}s"; node = rules.get(section_key) or rules.get(section_key_alt) or (len(rules) == 1 and next(iter(rules.values()))); if node: print node, else show the existing message.

Copilot · 2025-09-21T10:07:55Z

guide-validator/scripts/validate_guides.py

+            return f"[Validator prompt not found: {prompt_file}]"
+
+        # Inject content into prompt
+        prompt = prompt_template.replace('<PASTE FILE CONTENT HERE>', content)


[nitpick] Using a raw replace risks accidental replacement if the file content contains the placeholder text. Prefer a distinct template marker unlikely to appear in guides (e.g., {{GUIDE_CONTENT}}) and update prompt files accordingly, or split on a known delimiter.

Suggested change

prompt = prompt_template.replace('<PASTE FILE CONTENT HERE>', content)

prompt = prompt_template.replace('{{GUIDE_CONTENT}}', content)

Copilot · 2025-09-21T10:07:55Z

guide-validator/docs/base-format/base-style-guide.md

+- Stay in markdown where possible; use HTML only where necessary:
+  - prefer `![](/link/to/img)` over `<img src="/link/to/img" />`.
+    - You may need to use the `<img>` tag to specify an image size for an image that is not full screen (like a modal), but for full-screen images, use markdown and don't specify the size.
+  - use HTML `<table>` only when you absolutely have to, because all the nested elements then have to be HTML - you cant use markdown inside an HTML table like you can in a markdown table.


Typo: "cant" -> "can't".

Suggested change

- use HTML `<table>` only when you absolutely have to, because all the nested elements then have to be HTML - you cant use markdown inside an HTML table like you can in a markdown table.

- use HTML `<table>` only when you absolutely have to, because all the nested elements then have to be HTML - you can't use markdown inside an HTML table like you can in a markdown table.

Copilot · 2025-09-21T10:07:55Z

guide-validator/docs/base-format/base-style-guide.md

+- Use light mode for screenshots.
+- Screenshots should be PNG images.
+- Full-page screenshots should not have a border.
+- Modal dial screenshots should show the border because they are not full-width.


Typo: "dial" -> "dialog".

Suggested change

- Modal dial screenshots should show the border because they are not full-width.

- Modal dialog screenshots should show the border because they are not full-width.

Copilot · 2025-09-21T10:07:55Z

guide-validator/docs/base-format/base-style-guide.md

+Each guide will have the following sections:
+1. An [overview](#guide-overview)
+2. [Prerequisites](#prerequisites)
+3. One of more [Steps](#step-sections), conccluding with **Monitor** and **Review** steps


Typo: "conccluding" -> "concluding".

Suggested change

3. One of more [Steps](#step-sections), conccluding with **Monitor** and **Review** steps

3. One of more [Steps](#step-sections), concluding with **Monitor** and **Review** steps

Copilot · 2025-09-21T10:07:56Z

guide-validator/docs/base-format/base-style-guide.md

+
+### Guide Overview
+
+The overview should apprear immediately after the page title; do not include a `## Overview` header.


Typo: "apprear" -> "appear".

Suggested change

The overview should apprear immediately after the page title; do not include a `## Overview` header.

The overview should appear immediately after the page title; do not include a `## Overview` header.

Initial validator script

99dd6ee

rajlearner17 self-assigned this Jun 25, 2025

rajlearner17 marked this pull request as draft June 25, 2025 08:10

SumitPopat and others added 20 commits August 18, 2025 17:29

Add concepts doc for Intelligent Assessment guardrail (#360)

009b49a

Add > Guide > Intelligent Assessment, Intelligent Fixes, Policy Pack …

3af27f9

…Summary and restructure AI section > Draft (#352) Co-authored-by: raj <raj@turbot.com> Co-authored-by: Sumit Popat <sumit@turbot.com> Co-authored-by: David Boeke <david@boeke.com>

Add > Guide > Upgrade AWS Redis OSS Cache to Valkey (#359)

713f631

Co-authored-by: raj <raj@turbot.com> Co-authored-by: David Boeke <david@boeke.com>

Update guide for Turbot > Notifications > Webhook #349 (#363)

09b638a

Add > Guide > Enable PgBouncer in Turbot Guardrails and Update links …

075c70d

…for troubleshooting permission issues across multiple guides to ensure consistency and accuracy. (#364) Co-authored-by: raj <raj@turbot.com>

Add guide > How to retrieve Resource ID in the Guardrails console. Cl…

60becc0

…oses #365 (#366)

Update guide to add example for cleaning up leftover publications in …

186e8d9

…database upgrade guide and ensure caution message formatting is consistent.(#368) Co-authored-by: raj <raj@turbot.com>

Update > Disconnect an AWS account from a Guardrails workspace guide …

8ca420d

…to include `Enforce: No Boundary` in case permission boundary is used. Closes #354 (#372)

Update azure permissions page to remove duplicate value

bd662cb

Fix minor typo in header in notifications/filter-rules.md (#376)

ba1c00b

Fix context and broken urls (#377)

0f0a367

Update guide - Typo on Managing AWS Permissions Docs page. Closes #374 (

4f2c164

#378)

Find Deleted Resources History. Closes #320 (#373)

b93a21b

Co-authored-by: Raj <raj@turbot.com>

Fix Broken CFT link on Turbot Guardrails Enterprise Pre-Installation …

663ef53

…Checklist page. Closes #362 (#380)

Update guide – Add instructions for creating the Sandbox folder. Closes

087e502

#369 (#379)

Update guide > with newly added Native Stacks. Closes #383 (#384)

740480e

Co-authored-by: Sumit Popat <sumit@turbot.com> Co-authored-by: raj <raj@turbot.com>

Add guide - Blue <> Green deployment. Closes #328 (#381)

dc678c8

Co-authored-by: raj <raj@turbot.com>

Add guide - Installing mod using CLI. Closes #344 (#370)

1cc699f

Co-authored-by: raj <raj@turbot.com>

Add > Checking maintenance container in enterprise hosting. Closes #367…

e967a40

… (#387)

Merge branch 'main' into add-ai-validator

2465192

Subhajit97 and others added 3 commits September 15, 2025 16:49

Update AI configuration guide with latest GPT-5 and Claude model supp…

7449bc4

…ort. Closes #388 (#389) Co-authored-by: Sumit Popat <sumit@turbot.com>

Update guide format validator: add configuration, installation script…

5e12ffd

…, and validation scripts. Remove deprecated validation scripts and prompts. Enhance documentation with examples and manual validation instructions.

Merge branch 'main' into add-ai-validator

59c3525

Revise README for Guide Format Validator: update validation methods t…

acdffbc

…o emphasize automatic and manual options, enhance installation instructions, and clarify usage examples. Improve overall structure and readability.

rajlearner17 marked this pull request as ready for review September 21, 2025 10:02

rajlearner17 requested a review from Copilot September 21, 2025 10:02

Copilot AI reviewed Sep 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Initial AI validator script (Draft) #357

Initial AI validator script (Draft) #357

Uh oh!

rajlearner17 commented Jun 25, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

github-actions bot commented Sep 15, 2025

Uh oh!

github-actions bot commented Sep 15, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Copilot AI Sep 21, 2025

Uh oh!

Uh oh!

-                    self.rules[validator_name] = rules
+                    section_rules = rules.get(validator_name) or rules.get(f"{validator_name}s")
+                    if section_rules is None and len(rules) == 1:
+                        section_rules = next(iter(rules.values()))
+                    self.rules[validator_name] = section_rules

		for validator_name in ['Overview', 'Prerequisites', 'Steps', 'Troubleshooting', 'Callout']:
		if self.config['validators'].get(validator_name.lower(), True):

-            for validator_name in ['Overview', 'Prerequisites', 'Steps', 'Troubleshooting', 'Callout']:
-                if self.config['validators'].get(validator_name.lower(), True):
+            # Map validator names to config keys
+            validator_key_map = {
+                'Callout': 'callouts',
+                'Overview': 'overview',
+                'Prerequisites': 'prerequisites',
+                'Steps': 'steps',
+                'Troubleshooting': 'troubleshooting'
+            }
+            for validator_name in ['Overview', 'Prerequisites', 'Steps', 'Troubleshooting', 'Callout']:
+                config_key = validator_key_map.get(validator_name, validator_name.lower())
+                if self.config['validators'].get(config_key, True):

	prompt = prompt_template.replace('<PASTE FILE CONTENT HERE>', content)
	prompt = prompt_template.replace('{{GUIDE_CONTENT}}', content)

	- use HTML `<table>` only when you absolutely have to, because all the nested elements then have to be HTML - you cant use markdown inside an HTML table like you can in a markdown table.
	- use HTML `<table>` only when you absolutely have to, because all the nested elements then have to be HTML - you can't use markdown inside an HTML table like you can in a markdown table.

	- Modal dial screenshots should show the border because they are not full-width.
	- Modal dialog screenshots should show the border because they are not full-width.

	3. One of more [Steps](#step-sections), conccluding with Monitor and Review steps
	3. One of more [Steps](#step-sections), concluding with Monitor and Review steps


		### Guide Overview

		The overview should apprear immediately after the page title; do not include a `## Overview` header.

	The overview should apprear immediately after the page title; do not include a `## Overview` header.
	The overview should appear immediately after the page title; do not include a `## Overview` header.

Initial AI validator script (Draft) #357

Are you sure you want to change the base?

Initial AI validator script (Draft) #357

Uh oh!

Conversation

rajlearner17 commented Jun 25, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Preview Available 🚀

Uh oh!

github-actions bot commented Sep 15, 2025

Preview Available 🚀

Uh oh!

github-actions bot commented Sep 15, 2025

Preview Available 🚀

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!