Answers Integration Beta v1

Add support for Airtable text field content answerai brand added contentful removed publisher from this branch removed textField from the node to fix errors Update docker-compose to build the repo Update image added youtube document loader added updated contentful loader Add chatflow API key and sync with Sidekick on create/update Update .gitignore and constant.js Update API key functionality and add chatflow domain and API key to request payload added hidden nav and manu when in iframe Remove command from compose optimize dockkerfile Make dark theme the default Add DOMAIN env Add API_KEY env Add DOMAIN env Add auth0 integration Update dockerfile Add env passthrough for auth Add env passthrough for auth Add environment variables for authentication in production Add production environment variables Update compose env Update authentication logging and chatflow domain Update Dockerfile and env.sh, fix sed command in env.sh, and modify index.js to handle undefined organization ID Refactor Dockerfile and env.sh scripts, and update App.js and index.js files Add embed & embed-react, enable credentials for API comms Improvements Add SST Fix authorization issue in API endpoints Update .gitignore file to ignore .pem files and .env.* files added option for string or document fo routput and adjusted config Add lock files to git Update Docker files Add CDN and remove quotes from chat prompt so JSON is not malformed updated to account for queryoveride Allow public access to icons Remove parallel from turbo dev Fix Dockerfile and cookies missing on some requests Add CreateDalleImage tool and DallePost node class Add flowise-configs package and tools Added full Serp results access from LangChain Task and Planning Chatflow Remove unused SerpAPI initialization and console.log statement added new links for web scrapers Small improvements to match the AnswerAI styles ADd public-prediction endpoint Improve 401 error handling Add copilot deploys Add deployment documentation Update database secret Ignore *.env Fix missing errorg Merge pull request #3 from Answers-AI/feature/AAI-3-copilot-deployment Feature/aai 3 copilot deployment Merge pull request #4 from Answers-AI/feature/AAI-6-chatbot-config Feature/aai 6 chatbot config Add DEPLOYMENT_COPILOT.md and update README.md Improve session handling Merge branch 'feature/beta-v1' into bt/tools-sandbox Merge pull request #2 from Answers-AI/bt/tools-sandbox Tools Sandbox Update gitignore Update deployment docs Update how public/private enforcing works Add db secret, add environment file, fix launch json node version Merge pull request #5 from Answers-AI/feature/AAI-3-copilot-deployment Copilot deployment Add db secret, add environment file, fix launch json node version Add pipelines Add pipelines Update launch json Update copilot files Added embed as part of the workspace, updateded deps Add embed dist Update showing source files Fix source documents not showing Improve source documents Update source documents list in chatbot Update Badge powered by url Add visibility settings for AnswersAI Merge pull request #7 from Answers-AI/feature/AAI-2 Visibility settings for Chatflow on AnswerAI Answers config improvements Fix messages not displaying Fix the typo Update colors Increase the resources for flowise app Lower the service count Add and filter by userId on chatflow entities Add User model and handle for Chatflow and Credentials Add override for langfuse tracing Add roo tracer for metadata customization Improve Langfuse Tracing Update gitignore Update embed package names Merge branch 'feature/aai-embed' into feature/beta-v1 Fix sqlite user migration
the-answerai · Jun 4, 2024 · 5fca7f0 · 5fca7f0
1 parent 60f1df4
commit 5fca7f0
Show file tree

Hide file tree

Showing 195 changed files with 43,629 additions and 33,859 deletions.
diff --git a/.daisyrc b/.daisyrc
@@ -0,0 +1,138 @@
+const path = require('path')
+
+/** @typedef {DaisyConfig} Config  */
+const config = (CODE_BASE_PATH) => {
+    const PINECONE_INDEX_NAME = process.env.DAISY_PINECONE_INDEX_NAME || 'daisy'
+    const PINECONE_NAMESPACE = process.env.DAISY_PINECONE_NAMESPACE || 'default'
+    const DAISY_DIRECTORY_NAME = process.env.DAISY_DIRECTORY_NAME || 'daisy'
+    const DAISY_DIRECTORY = path.join(CODE_BASE_PATH, DAISY_DIRECTORY_NAME)
+    const MARKDOWN_DIRECTORY = process.env.MARKDOWN_DIRECTORY || path.join(DAISY_DIRECTORY, 'markdown')
+    const PROMPTS_FILE_PATH = process.env.PROMPTS_FILE_PATH || path.join(DAISY_DIRECTORY, 'prompts')
+
+    const TEMPLATE_FILE_PATH = process.env.TEMPLATE_FILE_PATH || path.join(DAISY_DIRECTORY, 'templates')
+    const ANSWERAI_API_KEY = process.env.ANSWERAI_API_KEY
+    const ANSWERAI_EMBEDDINGS_URL = process.env.DAISY_ANSWERAI_EMBEDDINGS_URL || 'https://app.theanswer.ai/api/codebase/embeddings'
+    const ANSWERAI_CHAT_COMPLETION_URL = process.env.DAISY_ANSWERAI_CHAT_COMPLETION_URL || 'https://app.theanswer.ai/api/ai/chat-completion'
+    const OPENAI_API_KEY = process.env.OPENAI_API_KEY
+    const PINECONE_API_KEY = process.env.PINECONE_API_KEY
+    const PINECONE_ENVIRONMENT = process.env.PINECONE_ENVIRONMENT
+    return {
+        codeBasePath: CODE_BASE_PATH,
+        pineconeIndexName: PINECONE_INDEX_NAME,
+        pineconeNamespace: PINECONE_NAMESPACE,
+        daisyDirectoryName: DAISY_DIRECTORY_NAME,
+        markdownDirectory: MARKDOWN_DIRECTORY,
+        promptsFilePath: PROMPTS_FILE_PATH,
+        templateFilePath: TEMPLATE_FILE_PATH,
+        openAiApiKey: OPENAI_API_KEY,
+        pineconeApiKey: PINECONE_API_KEY,
+        pineconeEnvironment: PINECONE_ENVIRONMENT,
+        answerAI: {
+            apiKey: ANSWERAI_API_KEY,
+            embeddingsUrl: ANSWERAI_EMBEDDINGS_URL,
+            chatCompletionUrl: ANSWERAI_CHAT_COMPLETION_URL
+        },
+        invalidPaths: [
+            'node_modules',
+            'dist',
+            'build',
+            'coverage',
+            'public',
+            'static',
+            'assets',
+            'images',
+            'img',
+            '.nextjs',
+            '.next',
+            '.git',
+            'generated',
+            '.yarn',
+            'patches',
+            DAISY_DIRECTORY
+        ],
+        invalidFileTypes: [
+            '.prompt',
+            '.csv',
+            '.tsv',
+            '.log',
+            '.docx',
+            '.xls',
+            '.xlsx',
+            '.ppt',
+            '.pptx',
+            '.svg',
+            '.png',
+            '.jpg',
+            '.jpeg',
+            '.gif',
+            '.bmp',
+            '.tiff',
+            '.tif',
+            '.ico',
+            '.mp3',
+            '.mp4',
+            '.wav',
+            '.wma',
+            '.avi',
+            '.mov',
+            '.mpg',
+            '.mpeg',
+            '.zip',
+            '.rar',
+            '.7z',
+            '.tar',
+            '.gz',
+            '.dmg',
+            '.iso',
+            '.exe',
+            '.bin',
+            '.pkg',
+            '.deb',
+            '.rpm',
+            '.lock'
+        ],
+        invalidFileNames: ['pnpm-lock.yaml', 'package-lock.json', 'app_pairs.yml', '.env', '.DS_Store', '.gitignore'],
+        fileTypes: {
+            docs: {
+                fileTypes: ['.md', '.mdx', '.txt', 'LICENSE.md', 'LICENSE'],
+                skipCompletion: true
+            },
+            react: {
+                fileTypes: ['.jsx', '.tsx'],
+                prompt: 'react.prompt',
+                template: 'react.md'
+            },
+            api: {
+                pathIncludes: ['/api/', '/routes/', '/controllers/', '/services/'],
+                prompt: 'api.prompt',
+                template: 'api.md'
+            },
+            config: {
+                fileTypes: ['.json', '.yaml', '.yml', '.toml', '.ini', '.env', '.env.example', '.daisyrc'],
+                prompt: 'config.prompt',
+                template: 'config.md'
+            },
+            script: {
+                fileTypes: ['.js', '.ts'],
+                prompt: 'script.prompt',
+                template: 'script.md'
+            },
+            contentModel: {
+                pathIncludes: ['content_types'],
+                prompt: 'contentModel.prompt',
+                template: 'contentModel.md'
+            },
+            server: {
+                pathIncludes: ['packages/server'],
+                prompt: 'server.prompt',
+                template: 'server.md'
+            },
+            default: {
+                prompt: 'default.prompt',
+                template: 'default.md'
+            }
+        }
+    }
+}
+
+module.exports = config
diff --git a/.dockerignore b/.dockerignore
@@ -1,10 +1,47 @@
-node_modules
-dist
-build
+# Include any files or directories that you don't want to be copied to your
+# container here (e.g., local build artifacts, temporary files, etc.).
+#
+# For more help, visit the .dockerignore file reference guide at
+# https://docs.docker.com/go/build-context-dockerignore/
 
+**/.classpath
+**/.dockerignore
+**/.env
+**/*.env
+env.*
+**/.git
+**/.gitignore
+**/.project
+**/.settings
+**/.toolstarget
+**/.vs
+**/.vscode
+**/.next
+**/.turbo
+**/.cache
+**/*.*proj.user
+**/*.dbmdl
+**/*.jfm
+**/charts
+**/docker-compose*
+**/compose*
+**/Dockerfile*
 **/node_modules
+**/npm-debug.log
+**/obj
+**/secrets.dev.yaml
+**/values.dev.yaml
 **/build
 **/dist
 
 packages/server/.env
 packages/ui/.env
+**/.sst
+**/copilot
+copilot
+
+docker
+node_modules
+
+LICENSE
+README.md
diff --git a/.gitignore b/.gitignore
@@ -1,11 +1,9 @@
 # editor
 .idea
-.vscode
-
 # dependencies
 **/node_modules
 **/package-lock.json
-**/yarn.lock
+# **/yarn.lock
 
 ## logs
 **/logs
@@ -26,6 +24,7 @@
 .DS_Store
 
 ## env
+*.env
 .env.local
 .env.development.local
 .env.test.local
@@ -38,6 +37,8 @@
 ## secrets
 **/*.key
 **/api.json
+**/*.pem
+.env.*
 
 ## uploads
 **/uploads
@@ -78,7 +79,7 @@
 *.csr
 
 ## Certificate
-*.der 
+*.der
 
 ## Key database file
 *.kdb
@@ -101,3 +102,11 @@
 ## S/MIME message
 *.smime
 *.vsix
+daisy
+
+# sst
+.sst
+
+#copilot/environments
+!copilot/environments/addons/**
+copilot/environments
diff --git a/.nvmrc b/.nvmrc
@@ -0,0 +1 @@
+20
diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -0,0 +1,21 @@
+{
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Flowise: dev",
+            "type": "node",
+            "request": "launch",
+            "runtimeExecutable": "pnpm",
+            "runtimeVersion": "21",
+            "runtimeArgs": ["dev"],
+            "timeout": 30000,
+            "stopOnEntry": false,
+            "sourceMaps": true,
+            "restart": true,
+            "console": "integratedTerminal",
+            "internalConsoleOptions": "neverOpen",
+            "skipFiles": ["<node_internals>/**"],
+            "smartStep": true
+        }
+    ]
+}
diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md
@@ -0,0 +1,60 @@
+## Deploying an Environment for an Existing Application with AWS Copilot CLI
+
+Before you start, make sure you have the following prerequisites:
+
+-   AWS account
+-   AWS CLI installed and configured with the new customer's AWS account
+-   Docker installed
+-   Copilot CLI installed
+-   Environment variables for the application
+
+Here are the environment variables required:
+
+| Variable                         | Description                                           |
+| -------------------------------- | ----------------------------------------------------- |
+| PORT                             | The port the application runs on                      |
+| APIKEY_PATH                      | The path to the API key                               |
+| SECRETKEY_PATH                   | The path to the secret key                            |
+| LOG_PATH                         | The path to the log files                             |
+| DISABLE_FLOWISE_TELEMETRY        | Flag to disable telemetry                             |
+| IFRAME_ORIGINS                   | Origins allowed to embed the application in an iframe |
+| MY_APP_VITE_AUTH_DOMAIN          | Auth0 domain                                          |
+| MY_APP_VITE_AUTH_CLIENT_ID       | Auth0 client ID                                       |
+| MY_APP_VITE_AUTH_AUDIENCE        | Auth0 audience                                        |
+| MY_APP_VITE_AUTH_ORGANIZATION_ID | Auth0 organization ID                                 |
+| DOMAIN                           | The domain of the application                         |
+| ANSWERAI_DOMAIN                  | The domain of AnswerAI                                |
+| AUTH0_JWKS_URI                   | URI of Auth0 JWKS                                     |
+| AUTH0_ISSUER_BASE_URL            | Base URL of Auth0 issuer                              |
+| AUTH0_BASE_URL                   | Base URL of Auth0                                     |
+| AUTH0_CLIENT_ID                  | Client ID of Auth0                                    |
+| AUTH0_CLIENT_SECRET              | Client secret of Auth0                                |
+| AUTH0_AUDIENCE                   | Audience of Auth0                                     |
+| AUTH0_SCOPE                      | Scope of Auth0                                        |
+| AUTH0_TOKEN_SIGN_ALG             | Token signing algorithm of Auth0                      |
+| AUTH0_ORGANIZATION_ID            | Organization ID of Auth0                              |
+
+Here are the steps to deploy an environment for an existing application:
+
+1. **Clone the application repository**. This will get you the application code and configuration.
+
+```bash
+git clone https://github.com/answers-AI/Flowise
+cd Flowise
+```
+
+2. **Create a new environment**. This will create a new environment, which is a standalone instance of your application.
+
+```bash
+copilot env init --name <env-name> --profile default
+
+3. **Deploy the service**. This will build your Docker image, push it to Amazon ECR, and deploy it to Amazon ECS.
+```
+
+````bash
+copilot svc deploy  --env <env-name>
+
+4. **Access the service**. You can see the URL of your service by running:
+```bash
+copilot svc show
+````
diff --git a/DEPLOYMENT_COPILOT.md b/DEPLOYMENT_COPILOT.md
@@ -0,0 +1,67 @@
+## Deploying an Environment for an Existing Application with AWS Copilot CLI
+
+Before you start, make sure you have the following prerequisites:
+
+-   AWS account
+-   AWS CLI installed and configured with the new customer's AWS account
+-   Docker installed
+-   AWS Copilot CLI installed or ran via `npx`
+-   Environment variables for the application saved into a `${env}.env` file
+
+Here are the environment variables required:
+
+| Variable                         | Description                                           |
+| -------------------------------- | ----------------------------------------------------- |
+| PORT                             | The port the application runs on                      |
+| APIKEY_PATH                      | The path to the API key                               |
+| SECRETKEY_PATH                   | The path to the secret key                            |
+| LOG_PATH                         | The path to the log files                             |
+| DISABLE_FLOWISE_TELEMETRY        | Flag to disable telemetry                             |
+| IFRAME_ORIGINS                   | Origins allowed to embed the application in an iframe |
+| MY_APP_VITE_AUTH_DOMAIN          | Auth0 domain                                          |
+| MY_APP_VITE_AUTH_CLIENT_ID       | Auth0 client ID                                       |
+| MY_APP_VITE_AUTH_AUDIENCE        | Auth0 audience                                        |
+| MY_APP_VITE_AUTH_ORGANIZATION_ID | Auth0 organization ID                                 |
+| DOMAIN                           | The domain of the application                         |
+| ANSWERAI_DOMAIN                  | The domain of AnswerAI                                |
+| AUTH0_JWKS_URI                   | URI of Auth0 JWKS                                     |
+| AUTH0_ISSUER_BASE_URL            | Base URL of Auth0 issuer                              |
+| AUTH0_BASE_URL                   | Base URL of Auth0                                     |
+| AUTH0_CLIENT_ID                  | Client ID of Auth0                                    |
+| AUTH0_CLIENT_SECRET              | Client secret of Auth0                                |
+| AUTH0_AUDIENCE                   | Audience of Auth0                                     |
+| AUTH0_SCOPE                      | Scope of Auth0                                        |
+| AUTH0_TOKEN_SIGN_ALG             | Token signing algorithm of Auth0                      |
+| AUTH0_ORGANIZATION_ID            | Organization ID of Auth0                              |
+
+Here are the steps to deploy an environment for an existing application:
+
+1. **Clone the application repository**. This will get you the application code and configuration.
+
+```bash
+git clone https://github.com/answers-AI/Flowise
+cd Flowise
+```
+
+2. **Create a new environment**. This will create a new environment, which is a standalone instance of your application.
+
+```bash
+copilot env init --name <env-name> --profile default
+```
+
+4. **Deploy the service**. This will build your Docker image, push it to Amazon ECR, and deploy it to Amazon ECS.
+
+````bash
+copilot svc deploy  --env <env-name>
+```
+
+5. **Access the service**. You can see the URL of your service by running:
+```bash
+copilot svc show
+````
+
+6. **Show the logs of your service**. This command will show you the logs of your service. You can specify the number of lines to show with the `--limit` flag.
+
+```bash
+copilot svc logs --limit 100
+```