diff --git a/language/README.md b/language/README.md index c07335b54a..85cac447a4 100644 --- a/language/README.md +++ b/language/README.md @@ -14,6 +14,7 @@ Learning API. * [Setup](#setup) * [Samples](#samples) * [Analyze](#analyze) + * [Slackbot](#slackbot) ## Setup @@ -61,3 +62,11 @@ For more information, see https://cloud.google.com/natural-language/docs [analyze_docs]: https://cloud.google.com/natural-language/docs [analyze_code]: analyze.js + +### Slackbot + +The example in the [slackbot](./slackbot) subdirectory shows a Slack bot built using the +[Botkit](https://github.com/howdyai/botkit) library. +It runs on a Google Container Engine (Kubernetes) cluster, and uses one of the Google Cloud Platform's ML +APIs, the Natural Language (NL) API, to interact in a Slack channel. +See its [README](./slackbot/README.md) for more information. diff --git a/language/slackbot/.gitignore b/language/slackbot/.gitignore new file mode 100644 index 0000000000..3997beadf8 --- /dev/null +++ b/language/slackbot/.gitignore @@ -0,0 +1 @@ +*.db \ No newline at end of file diff --git a/language/slackbot/Dockerfile b/language/slackbot/Dockerfile new file mode 100644 index 0000000000..9b602efd14 --- /dev/null +++ b/language/slackbot/Dockerfile @@ -0,0 +1,28 @@ +# Copyright 2016 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM node:5.4 + +RUN apt-get update +RUN apt-get install -y sqlite3 + +# Install app dependencies. +COPY package.json /src/package.json +WORKDIR /src +RUN npm install + +# Bundle app source. +COPY demo_bot.js /src + +CMD ["node", "/src/demo_bot.js"] diff --git a/language/slackbot/README.md b/language/slackbot/README.md new file mode 100644 index 0000000000..cd7eee3fa7 --- /dev/null +++ b/language/slackbot/README.md @@ -0,0 +1,204 @@ + +# Building a Botkit-based Slack Bot that uses the GCP NL API and runs on Google Container Engine + + +This example shows a Slack bot built using the [Botkit](https://github.com/howdyai/botkit) library. +It runs on a Google Container Engine (Kubernetes) cluster, and uses one of the Google Cloud Platform's ML +APIs, the Natural Language (NL) API, to interact in a Slack channel. + +It uses the NL API in two different ways. +First, it uses the [Google Cloud NL API](https://cloud.google.com/natural-language/) to assess +the [sentiment](https://cloud.google.com/natural-language/docs/basics) of any message posted to +the channel, and if the positive or negative magnitude of the statement is +sufficiently large, it sends a 'thumbs up' or 'thumbs down' in reaction. + +Additionally, it uses the NL API to identify [entities](https://cloud.google.com/natural-language/docs/basics) in each +posted message, and tracks them in a database (using sqlite3). Then, at any time you can query the NL slackbot to ask +it for the top N entities used in the channel. + +The example uses [Google Container +Engine](https://cloud.google.com/container-engine/), a hosted version of +[Kubernetes](http://kubernetes.io), to run the bot, and uses [Google Container +Registry](https://cloud.google.com/container-registry/) to store a Docker image +for the bot. + + +## Setting up your environment + +### Container Engine prerequisites + +First, set up the Google Container Engine +[prerequisites](https://cloud.google.com/container-engine/docs/before-you-begin), including [installation of the Google Cloud SDK](https://cloud.google.com/sdk/downloads). + +### NL API prerequisites + +Next, enable the NL API for your project and authenticate to your service account as described [here](https://cloud.google.com/natural-language/docs/getting-started). (The service account step is not necessary when running the bot on Container Engine, but it is useful if you're testing locally). + +### Create a cluster + +Next, +[create a Kubernetes cluster](https://cloud.google.com/container-engine/docs/clusters/operations#creating_a_container_cluster) using Container Engine as follows: + +```bash +gcloud container clusters create "slackbot-cluster" --scopes "https://www.googleapis.com/auth/cloud-platform" +``` + +You can name the cluster something other than "slackbot-cluster" if you like. + +### Install Docker + +If you do not already have [Docker](https://www.docker.com/) installed locally, follow the [installation instructions](https://docs.docker.com/engine/installation/) on the Docker site. + +## Get a Slack token and invite the bot to a Slack channel + +Then, create a [Slack bot user](https://api.slack.com/bot-users) and get an +authentication token. + +Then, 'invite' your new bot to a channel on a Slack team. + +## Running the slackbot on Kubernetes + +### Upload the slackbot token to Kubernetes + +We will be loading this token in our bot using +[secrets](http://kubernetes.io/v1.1/docs/user-guide/secrets.html). + +Run the following script to create a secret .yaml file (replacing `MY-SLACK-TOKEN` with your token), then use that yaml file to create a secret on your Kubernetes cluster. + +```bash +./generate-secret.sh MY-SLACK-TOKEN +kubectl create -f slack-token-secret.yaml +``` + +### Build the bot's container + +We'll run the slackbot app in our Kubernetes cluster as a [Replication Controller](http://kubernetes.io/docs/user-guide/replication-controller/) with one replica. + +So, first, we need to build its Docker container. Replace `my-cloud-project-id` below with your +Google Cloud Project ID. This tags the container so that gcloud can upload it to +your private Google Container Registry. + +```bash +export PROJECT_ID=my-cloud-project-id +docker build -t gcr.io/${PROJECT_ID}/slack-bot . +``` + +Once the build completes, upload it to the Google Container registry: + +```bash +gcloud docker -- push gcr.io/${PROJECT_ID}/slack-bot +``` + + +### Running the container + +First, create a Replication Controller configuration, populated with your Google +Cloud Project ID, so that Kubernetes knows where to find the Docker image. + +```bash +./generate-rc.sh $PROJECT_ID +``` + +Now, tell Kubernetes to create the bot's replication controller. This will launch 1 pod replica running the bot. + +```bash +kubectl create -f slack-bot-rc.yaml +``` + +You can check the status of your bot with: + +```bash +kubectl get pods +``` + +Now your bot should be online. As a sanity check, check that it responds to a "Hello" message directed to it. + +Note: if you have forgotten to create the secret first, the pod won't come up successfully. + +## Running the bot locally + +If you want, you can run your slackbot locally instead. This is handy if +you've made some changes and want to test them out before redeploying. To do +this, first run: + +```bash +npm install +``` + +Then, set GCLOUD_PROJECT to your project id: + +```bash +export GCLOUD_PROJECT=my-cloud-project-id +``` + +Then, create a file containing your Slack token, and point 'SLACK_TOKEN_PATH' to that file when you run the script +(substitute 'my-slack-token with your actual token): + + echo my-slack-token > slack-token + SLACK_TOKEN_PATH=./slack-token node demo_bot.js + +## Using the Bot + +Once you've confirmed the bot is running, you can start putting it through its paces. + +### Sentiment Analysis + +The slackbot will give a 'thumbs up' or 'thumbs down' if it thinks a message is above a certain magnitude in positive or negative sentiment. + +E.g., try posting this message to the channel (you don't need to explicitly mention the bot in this message): + +``` +I hate bananas. +``` + +You should see that bot give a thumbs down in reply, indicatig that the NL API +reported negative sentiment for this sentence. Next, try: + +``` +I love coffee. +``` + +This should generate a thumbs up. Posted text won't get a reply from the bot +unless the magnitude of the sentiment is above a given threshold, 30 by +default. E.g., with a neutral statement like `The temperature is seventy +degrees.` the bot is unlikely to give a response. + +### Entity Analysis + +For every message posted to the channel, the bot-- behind the scenes-- is +analyzing and storing information about the entities it detects. At any time +you can query the bot to get the current N most frequent entities, where N is +20 by default. It will be more interesting if you wait until a few messages +have been posted to the channel, so that the bot has the chance to identify +and log some entities. + +E.g., suppose your bot is called `nlpbot`. +To see the top entities, send it this message: + +``` +@nlpbot top entities +``` + + +## Shutting down + +To shutdown your bot, we tell Kubernetes to delete the replication controller. + +```bash +kubectl delete -f slack-bot-rc.yaml +``` + + +## Cleanup + +If you have created a container cluster, you may still get charged for the +Google Compute Engine resources it is using, even if they are idle. To delete +the cluster, run: + +```bash +gcloud container clusters delete slackbot-cluster +``` + +(If you used a different name for your cluster, substitute that name for `slackbot-cluster`.) +This deletes the Google Compute Engine instances that are running the cluster. + diff --git a/language/slackbot/demo_bot.js b/language/slackbot/demo_bot.js new file mode 100755 index 0000000000..aac4c09c62 --- /dev/null +++ b/language/slackbot/demo_bot.js @@ -0,0 +1,235 @@ +/* ***************************************************************************** +Copyright 2016 Google Inc. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +******************************************************************************** + +This is a Slack bot built using the Botkit library (http://howdy.ai/botkit). It +runs on a Kubernetes cluster, and uses one of the Google Cloud Platform's ML +APIs, the Natural Language API, to interact in a Slack channel. It does this in +two respects. + +First, it uses the NL API to assess the "sentiment" of any message posted to +the channel, and if the positive or negative magnitude of the statement is +sufficiently large, it sends a 'thumbs up' or 'thumbs down' in reaction. + +Second, it uses the NL API to identify the 'entities' in each posted message, +and tracks them in a database (using sqlite3). Then, at any time you can +query the NL slackbot to ask it for the top N entities used in the channel. + +The README walks through how to run the NL slackbot as an app on a +Google Container Engine/Kubernetes cluster, but you can also just run the bot +locally if you like. +To do this, create a file containing your Slack token (as described in +the README), then point 'SLACK_TOKEN_PATH' to that file when you run the script: + + echo my-slack-token > slack-token + SLACK_TOKEN_PATH=./slack-token node demo_bot.js + +See the README.md in this directory for more information about setup and usage. + +*/ + +'use strict'; + +const Botkit = require('botkit'); +const fs = require('fs'); +const Language = require('@google-cloud/language'); +const path = require('path'); +const sqlite3 = require('sqlite3').verbose(); + +const controller = Botkit.slackbot({ debug: false }); + +// create our database if it does not already exist. +const db = new sqlite3.cached.Database(path.join(__dirname, './slackDB.db')); + +// the number of most frequent entities to retrieve from the db on request. +const NUM_ENTITIES = 20; +// The magnitude of sentiment of a posted text above which the bot will respond. +const SENTIMENT_THRESHOLD = 30; + +const ENTITIES_SQL = `SELECT name, type, count(name) as wc +FROM entities +GROUP BY name +ORDER BY wc DESC +LIMIT ${NUM_ENTITIES};`; + +const TABLE_SQL = `CREATE TABLE if not exists entities ( + name text, + type text, + salience real, + wiki_url text, + ts integer +);`; + +function startController () { + if (!process.env.SLACK_TOKEN_PATH) { + throw new Error('Please set the SLACK_TOKEN_PATH environment variable!'); + } + + let token = fs.readFileSync(process.env.SLACK_TOKEN_PATH, { encoding: 'utf8' }); + token = token.replace(/\s/g, ''); + + // Create the table that will store entity information if it does not already + // exist. + db.run(TABLE_SQL); + + controller + .spawn({ token: token }) + .startRTM((err) => { + if (err) { + console.error('Failed to start controller!'); + console.error(err); + process.exit(1); + } + }); + + return controller + // If the bot gets a DM or mention with 'hello' or 'hi', it will reply. You + // can use this to sanity-check your app without needing to use the NL API. + .hears( + ['hello', 'hi'], + ['direct_message', 'direct_mention', 'mention'], + handleSimpleReply + ) + // If the bot gets a DM or mention including "top entities", it will reply with + // a list of the top N most frequent entities used in this channel, as derived + // by the NL API. + .hears( + ['top entities'], + ['direct_message', 'direct_mention', 'mention'], + handleEntitiesReply + ) + // For any posted message, the bot will send the text to the NL API for + // analysis. + .on('ambient', handleAmbientMessage); +} + +function handleSimpleReply (bot, message) { + bot.reply(message, 'Hello.'); +} + +function handleEntitiesReply (bot, message) { + bot.reply(message, 'Top entities: '); + + // Query the database for the top N entities + db.all(ENTITIES_SQL, (err, topEntities) => { + if (err) { + throw err; + } + + let entityInfo = ''; + + // Uncomment this to see the query results logged to console: + // console.log(topEntities); + + topEntities.forEach((entity) => { + entityInfo += `entity: *${entity.name}*, type: ${entity.type}, count: ${entity.wc}\n`; + }); + + bot.reply(message, entityInfo); + }); +} + +function analyzeEntities (text, ts) { + // Instantiates a client + const language = Language(); + + // Instantiates a Document, representing the provided text + const document = language.document({ + // The document text, e.g. "Hello, world!" + content: text + }); + + // Detects entities in the document + return document.detectEntities() + .then((results) => { + const entities = results[1].entities; + + entities.forEach((entity) => { + const name = entity.name; + const type = entity.type; + const salience = entity.salience; + let wikiUrl = ''; + if (entity.metadata.wikipedia_url) { + wikiUrl = entity.metadata.wikipedia_url; + } + + // Uncomment this to see the entity info logged to console: + // console.log(`${name}, type: ${type}, w url: ${wikiUrl}, salience: ${salience}, ts: ${ts}`); + + db.run( + 'INSERT INTO entities VALUES (?, ?, ?, ?, ?);', + [name, type, salience, wikiUrl, Math.round(ts)] + ); + }); + + return entities; + }); +} + +function analyzeSentiment (text) { + // Instantiates a client + const language = Language(); + + // Instantiates a Document, representing the provided text + const document = language.document({ + // The document text, e.g. "Hello, world!" + content: text + }); + + // Detects the 'sentiment' of some text using the NL API + return document.detectSentiment() + .then((results) => { + const sentiment = results[0]; + + // Uncomment the following four lines to log the sentiment to the console: + // if (results >= SENTIMENT_THRESHOLD) { + // console.log('Sentiment: positive.'); + // } else if (results <= -SENTIMENT_THRESHOLD) { + // console.log('Sentiment: negative.'); + // } + + return sentiment; + }); +} + +function handleAmbientMessage (bot, message) { + // Note: for purposes of this example, we're making two separate calls to the + // API, one to extract the entities from the message, and one to analyze the + // 'sentiment' of the message. These could be combined into one call. + return analyzeEntities(message.text, message.ts) + .then(() => analyzeSentiment(message.text)) + .then((sentiment) => { + if (sentiment >= SENTIMENT_THRESHOLD) { + // We have a positive sentiment of magnitude larger than the threshold. + bot.reply(message, ':thumbsup:'); + } else if (sentiment <= -SENTIMENT_THRESHOLD) { + // We have a negative sentiment of magnitude larger than the threshold. + bot.reply(message, ':thumbsdown:'); + } + }); +} + +exports.ENTITIES_SQL = ENTITIES_SQL; +exports.TABLE_SQL = TABLE_SQL; +exports.startController = startController; +exports.handleSimpleReply = handleSimpleReply; +exports.handleEntitiesReply = handleEntitiesReply; +exports.analyzeEntities = analyzeEntities; +exports.analyzeSentiment = analyzeSentiment; +exports.handleAmbientMessage = handleAmbientMessage; + +if (require.main === module) { + startController(); +} diff --git a/language/slackbot/demo_bot.test.js b/language/slackbot/demo_bot.test.js new file mode 100644 index 0000000000..1c9fde810c --- /dev/null +++ b/language/slackbot/demo_bot.test.js @@ -0,0 +1,156 @@ +/** + * Copyright 2016, Google, Inc. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +'use strict'; + +const fs = require(`fs`); +const path = require(`path`); +const proxyquire = require(`proxyquire`).noCallThru(); +const sqlite3 = require(`sqlite3`).verbose(); + +const DB_PATH = path.join(__dirname, `./slackDB.db`); +const SLACK_TOKEN_PATH = path.join(__dirname, `./.token`); +const text = `President Obama is speaking at the White House.`; + +describe(`demo_bot`, () => { + let db, controllerMock, botkitMock, botMock, program; + + before((done) => { + fs.unlink(DB_PATH, (err) => { + if (err && err.code !== `ENOENT`) { + done(err); + return; + } + + db = new sqlite3.cached.Database(DB_PATH); + controllerMock = { + spawn: sinon.stub().returnsThis(), + startRTM: sinon.stub().returnsThis(), + hears: sinon.stub().returnsThis(), + on: sinon.stub().returnsThis() + }; + + botkitMock = { + slackbot: sinon.stub().returns(controllerMock) + }; + + botMock = { + reply: sinon.stub() + }; + + program = proxyquire(`./demo_bot`, { + botkit: botkitMock + }); + + db.run(program.TABLE_SQL, done); + }); + }); + + after((done) => { + fs.unlink(DB_PATH, (err) => { + if (err) { + done(err); + return; + } + fs.unlink(SLACK_TOKEN_PATH, done); + }); + }); + + it(`should analyze sentiment in text`, () => { + return program.analyzeSentiment(text) + .then((sentiment) => { + assert.equal(sentiment > 0, true); + }); + }); + + it(`should analyze entities in text`, () => { + return program.analyzeEntities(text, 1234) + .then((entities) => { + assert.equal(entities.some((entity) => entity.name === `Obama`), true); + assert.equal(entities.some((entity) => entity.name === `White House`), true); + + return new Promise((resolve, reject) => { + setTimeout(() => { + db.all(program.ENTITIES_SQL, (err, entities) => { + if (err) { + reject(err); + return; + } + assert.equal(entities.some((entity) => entity.name === `Obama`), true); + assert.equal(entities.some((entity) => entity.name === `White House`), true); + resolve(); + }); + }, 1000); + }); + }); + }); + + it(`should reply to simple hello message`, () => { + const message = {}; + + program.handleSimpleReply(botMock, message); + + assert.equal(botMock.reply.callCount, 1); + assert.deepEqual(botMock.reply.getCall(0).args, [message, `Hello.`]); + }); + + it(`should reply to entities message`, (done) => { + const message = {}; + + program.handleEntitiesReply(botMock, message); + + setTimeout(() => { + assert.equal(botMock.reply.callCount, 3); + assert.deepEqual(botMock.reply.getCall(1).args, [message, `Top entities: `]); + assert.deepEqual(botMock.reply.getCall(2).args, [message, `entity: *Obama*, type: PERSON, count: 1\nentity: *White House*, type: LOCATION, count: 1\n`]); + done(); + }, 1000); + }); + + describe(`startController`, () => { + let originalToken; + + before(() => { + originalToken = process.env.SLACK_TOKEN_PATH; + }); + + after(() => { + process.env.SLACK_TOKEN_PATH = originalToken; + }); + + it(`should check SLACK_TOKEN_PATH`, () => { + process.env.SLACK_TOKEN_PATH = ``; + + assert.throws(() => { + program.startController(); + }, Error, `Please set the SLACK_TOKEN_PATH environment variable!`); + }); + + it(`should start the controller`, () => { + let controller; + + fs.writeFileSync(SLACK_TOKEN_PATH, `test`, { encoding: `utf8` }); + process.env.SLACK_TOKEN_PATH = SLACK_TOKEN_PATH; + + controller = program.startController(); + + assert.strictEqual(controller, controllerMock); + assert.equal(controllerMock.spawn.callCount, 1); + assert.equal(controllerMock.startRTM.callCount, 1); + assert.equal(controllerMock.hears.callCount, 2); + assert.equal(controllerMock.on.callCount, 1); + }); + }); +}); diff --git a/language/slackbot/generate-rc.sh b/language/slackbot/generate-rc.sh new file mode 100755 index 0000000000..e367d2eb44 --- /dev/null +++ b/language/slackbot/generate-rc.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# Copyright 2016 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +if [[ $# -ne 1 ]] ; then + echo "Your project ID must be specified." + echo "Usage:" >&2 + echo " ${0} my-cloud-project" >&2 + exit 1 +fi +cloud_project=$1 + +cat < slack-bot-rc.yaml +apiVersion: v1 +kind: ReplicationController +metadata: + name: slack-bot +spec: + replicas: 1 + template: + metadata: + labels: + name: slack-bot + spec: + containers: + - name: master + image: gcr.io/${cloud_project}/slack-bot + volumeMounts: + - name: slack-token + mountPath: /etc/slack-token + env: + - name: SLACK_TOKEN_PATH + value: /etc/slack-token/slack-token + - name: GCLOUD_PROJECT + value: ${cloud_project} + volumes: + - name: slack-token + secret: + secretName: slack-token +END diff --git a/language/slackbot/generate-secret.sh b/language/slackbot/generate-secret.sh new file mode 100755 index 0000000000..157ec02513 --- /dev/null +++ b/language/slackbot/generate-secret.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# Copyright 2016 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +if [[ $# -ne 1 ]] ; then + echo "Your slack token must be specified." + echo "Usage:" >&2 + echo " ${0} MY-SLACK-TOKEN" >&2 + exit 1 +fi + +token=$1 +token_base64=$(python -c "import base64; print base64.b64encode(\"${token}\")") + +cat < slack-token-secret.yaml +apiVersion: v1 +kind: Secret +metadata: + name: slack-token +type: Opaque +data: + slack-token: ${token_base64} +END diff --git a/language/slackbot/package.json b/language/slackbot/package.json new file mode 100644 index 0000000000..2f9ab75520 --- /dev/null +++ b/language/slackbot/package.json @@ -0,0 +1,20 @@ +{ + "name": "kubernetes-slack-botkit-example", + "version": "0.0.1", + "private": true, + "license": "Apache Version 2.0", + "author": "Google Inc.", + "description": "A Slack Botkit bot, running on Kubernetes.", + "main": "demo_bot.js", + "dependencies": { + "botkit": "^0.0.5", + "@google-cloud/language": "^0.6.3", + "sqlite3": "^3.1.8" + }, + "scripts": { + "test": "cd ../..; npm run st -- language/slackbot/*.test.js" + }, + "engines": { + "node": ">=4.3.2" + } +}