-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathgopherbot-doc.txt
6144 lines (5338 loc) · 756 KB
/
gopherbot-doc.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# concatenation of gopherbot-doc-main/ - all the files from gopherbot-doc-main/
# have been included in this single file, each preceded by a
# descriptive preamble that includes the filename in the opening
# tag.
# All the ".md" files are official Gopherbot documentation. Most
# of the other files are related to "mdbook", the tool used in
# generating the html version of the documentation.
<preamble file: gopherbot-doc-main/.github/workflows/build.yml>
</preamble>
<file_content file: gopherbot-doc-main/.github/workflows/build.yml>
name: Gopherbot Docs
on:
push:
branches:
- main
pull_request:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Checkout gopherbot/gh-pages
uses: actions/checkout@v2
with:
repository: lnxjedi/gopherbot
persist-credentials: false
fetch-depth: 0
path: gopherbot-doc
ref: gh-pages
- name: Setup mdBook
# peaceris/actions-mdbook v1.1.14
uses: peaceiris/actions-mdbook@4b5ef36b314c2599664ca107bb8c02412548d79d
with:
mdbook-version: 'latest'
- name: Build
run: |
mv gopherbot-doc/.git ghpages.git
mdbook build -d ../gopherbot-doc ./doc
mv ghpages.git gopherbot-doc/.git
- name: Commit and push
shell: bash
env:
GITHUB_TOKEN: ${{ secrets.REPO_PUSH_TOKEN }}
run: |
cd gopherbot-doc
git config --local user.email "parsley@linuxjedi.org"
git config --local user.name "David Parsley"
git add -A
git commit -m "Automated update" || exit 0
git push https://parsley42:${GITHUB_TOKEN}@github.com/lnxjedi/gopherbot gh-pages
</file_content file: gopherbot-doc-main/.github/workflows/build.yml>
<preamble file: gopherbot-doc-main/.gopherci/mkdocs.sh>
</preamble>
<file_content file: gopherbot-doc-main/.gopherci/mkdocs.sh>
#!/bin/bash
# mkdocs.sh - generate updated mdbook docs
mv gopherbot-doc/.git gopherbot-doc.git
cd doc/
mdbook build -d ../gopherbot-doc/
cd -
mv gopherbot-doc.git gopherbot-doc/.git
</file_content file: gopherbot-doc-main/.gopherci/mkdocs.sh>
<preamble file: gopherbot-doc-main/.gopherci/pipeline.sh>
</preamble>
<file_content file: gopherbot-doc-main/.gopherci/pipeline.sh>
#!/bin/bash
# pipeline.sh - trusted pipeline script for gopherci for Gopherbot
source $GOPHER_INSTALLDIR/lib/gopherbot_v1.sh
if [ -n "$NOTIFY_USER" ]
then
FailTask notify $NOTIFY_USER "Gopherbot build failed"
fi
FailTask email-log parsley@linuxjedi.org
# Initialize ssh for updating docs repo
AddTask ssh-init
# Make sure github is in known_hosts
AddTask ssh-scan github.com
# Clone existing gh-pages
AddTask git-clone https://github.com/lnxjedi/gopherbot.git gh-pages gopherbot-doc
# Build new
AddTask exec ./.gopherci/mkdocs.sh
# Publish doc updates (if any)
AddTask exec ./.gopherci/publishdoc.sh
AddTask notify $NOTIFY_USER "Completed successful documentation build"
</file_content file: gopherbot-doc-main/.gopherci/pipeline.sh>
<preamble file: gopherbot-doc-main/.gopherci/publishdoc.sh>
</preamble>
<file_content file: gopherbot-doc-main/.gopherci/publishdoc.sh>
#!/bin/bash
# publishdoc.sh - check for updated docs and push if needed
COMMIT=$(git rev-parse --short HEAD)
cd gopherbot-doc
if [ -z "$(git status --porcelain)" ]
then
echo "No updates to documentation"
exit 0
fi
git add .
git commit -m "Updates from master branch, commit $COMMIT"
git remote add update git@github.com:lnxjedi/gopherbot.git
git push -u update gh-pages
</file_content file: gopherbot-doc-main/.gopherci/publishdoc.sh>
<preamble file: gopherbot-doc-main/README.md>
</preamble>
<file_content file: gopherbot-doc-main/README.md>
This repository holds the markdown sources for the documentation at [lnxjedi.github.io/gopherbot](https://lnxjedi.github.io/gopherbot). The rendered pages are kept in the `gh-pages` branch of the main [gopherbot repository](https://github.com/lnxjedi/gopherbot), just to make the documentation URL look nice (and stick with what it has been previously).
</file_content file: gopherbot-doc-main/README.md>
<preamble file: gopherbot-doc-main/doc/.gitignore>
</preamble>
<file_content file: gopherbot-doc-main/doc/.gitignore>
book
</file_content file: gopherbot-doc-main/doc/.gitignore>
<preamble file: gopherbot-doc-main/doc/README-dev.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/README-dev.md>
## Notes on Writing Documentation for Developers
Gopherbot documentation is written in Markdown and processed by [`mdbook`](https://github.com/rust-lang/mdBook) to be published with [Github Pages](https://lnxjedi.github.io/gopherbot).
These instructions are primarily for me (David Parsley) to remind me of the few steps in setting up a dev environment for writing Gopherbot documentation, which I primarily do on a Chromebook with Linux (Crostini) installed.
1. Check the VM IP with `ip addr show dev eth0` and make sure `/etc/hosts` lists the IP ad `penguin.linux.test`
1. Download the `mdbook` binary to `$HOME/bin/mdbook`; make sure `$HOME/bin` is added to `$PATH` in `~/.bashrc`
1. Run `mdbook serve -n penguin.linux.test` from the `doc/` directory
1. View the work in progress in the browser at `http://penguin.linux.test:3000`
## CI/CD Notes
Whenever the `master` branch is updated, the pipeline automatically updates the `gh-pages` branch (if documentation changed).
</file_content file: gopherbot-doc-main/doc/README-dev.md>
<preamble file: gopherbot-doc-main/doc/Unsorted.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/Unsorted.md>
## Job Arguments
A **Gopherbot** job can be started with an arbitrary number of space-separated arguments, either using e.g. `AddJob myjob foo bar baz` (with `bash`) in a a pipeline, or interactively in chat with `run job myjob foo bar baz`. If there are configured `Arguments:` in the `<jobname>.yaml` file, the robot will prompt for these if not given, and check supplied arguments against configured regular expressions.
</file_content file: gopherbot-doc-main/doc/Unsorted.md>
<preamble file: gopherbot-doc-main/doc/book.toml>
</preamble>
<file_content file: gopherbot-doc-main/doc/book.toml>
[book]
authors = ["David Parsley"]
language = "en"
multilingual = false
src = "src"
title = "Gopherbot DevOps Chatbot"
[build]
build-dir = "/tmp/mdbook"
</file_content file: gopherbot-doc-main/doc/book.toml>
<preamble file: gopherbot-doc-main/doc/cserve.sh>
</preamble>
<file_content file: gopherbot-doc-main/doc/cserve.sh>
#!/bin/bash
mdbook serve -p 8888 -n 0.0.0.0
</file_content file: gopherbot-doc-main/doc/cserve.sh>
<preamble file: gopherbot-doc-main/doc/serve.sh>
</preamble>
<file_content file: gopherbot-doc-main/doc/serve.sh>
#!/bin/bash
mdbook serve -p 8888
</file_content file: gopherbot-doc-main/doc/serve.sh>
<preamble file: gopherbot-doc-main/doc/src/Admin.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/Admin.md>
# Managing Your Robot and Adding Extensions
**Gopherbot** robots are designed to be remotely administered and updated, for common cases where a robot runs behind network firewalls, in virtual cloud networks, or in a container environment. Many of the frequently desired updates - such as changing the schedule of an automated job - can be safely and easily updated by pushing a commit to your robot's repository and instructing it to update. More significant updates can be tested by modelling with the **terminal** connector before committing and saving, then updating your production robot.
This chapter covers:
* How to update your robot with **git**
* The two primary ways to set up a dev environment
* **Gopherbot** CLI commands
You should have a robot deployed "in production" (connected to your team chat) to work the examples in the following sections.
</file_content file: gopherbot-doc-main/doc/src/Admin.md>
<preamble file: gopherbot-doc-main/doc/src/Basics.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/Basics.md>
# Robot Basics
Central to the design of Gopherbot is the idea that once your robot connects to your team chat and joins one or more channels, the robot "hears" every message in those channels, just as a user would. Every time the robot hears a message[^1]:
* It checks to see if the message was directed to it by name
* It checks for "ambient" message matches
* It checks for matching job triggers
This chapter focuses on the first case - sending commands directly to your robot.
[^1]: Depending on the configured values for `IgnoreUnlistedUsers` and `IgnoreUsers`, messages may be dropped entirely without any processing. This would show up in the logs at log level **Debug**.
</file_content file: gopherbot-doc-main/doc/src/Basics.md>
<preamble file: gopherbot-doc-main/doc/src/Configuration.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/Configuration.md>
# Configuring Gopherbot
> NOTE: This chapter has not been updated for **Gopherbot** version 2, but has reference material that may be of use when examining default and standard configuration.
**Gopherbot** has very powerful and flexible configuration capabilities based on *yaml* templates. The core concept is simple; **Gopherbot** ships with default configuration in the `conf/` directory of the installation archive, and individual robots can modify and override the default configuration with environment variables and custom configuration files. This chapter examines the configuration system in detail.
</file_content file: gopherbot-doc-main/doc/src/Configuration.md>
<preamble file: gopherbot-doc-main/doc/src/DevNotes.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/DevNotes.md>
# Gopherbot Development Notes - Current
`DevNotes.md` - TODO items and design notes for future developmen
## 2.0 Goals
* Job scheduling (DONE)
* Pipelines (DONE)
* Remote plugin execution over ssh (ON HOLD)
* GopherCI for CI/CD (DONE)
## Version 2 Final Release TODOs
- TODO: Advanced gitpod workflows for demo and devel
- Demo should create $BOTNAME-gopherbot repo and help user set up initial repo / install
- Devel should clone existing dev-gopherbot* repos
- Figure out how demo and devel should differ
- TODO: fix "stop debugging" crash; `debug task citools`, `stop debugging`
- TODO: Add admin "monitor \<channel\>" / "stop monitoring" to DM admin with all messages to a channel similar to debug, for use in plugin devel & troubleshooting
- TODO: `makerobot.sh <name>` script the copies robot.skel and creates unit file and .env to be edited
- TODO: Security:
- DONE?: Squelch arguments in debugging messages when it's a DM - audit
- DONE/TEST: Security: Return to drop priv full when thread OS state propagation fixed; issue #29613
- TODO: Bugs & buglets:
- DONE: Don't allow plugins to be scheduled, only jobs; too easy to create nil ptr deref by not supplying correct args to plugin
- DONE: Squelch Replies to 'BotUser' users - use Say "UserName: foo"; slack webhooks can't be @mentioned. Easiest solution - add BotUser flag to Robot that's set in makeRobot and cloned.
- TODO: Add tests for:
- Parameter precedence (configured params, stored params, SetParameter)
- Pipelines
- Triggers only
- Pipeline failures
- Users listed in the UserRoster w/ alt names
- Test repository configured parameters
- Decrypting values encrypted in yaml files
- TODO: Write documentation for:
- Configuration updates: config merging, default config, template funcs like 'decrypt'
- Need for BotInfo to provide the robot with a Name and Email
- Pipelines
- Namespaces / extended namespaces and histories; histories include branch, namespaces don't
- store repository parameter can include a branch - TEST; e.g. store repository parameter xxx/master would override value for xxx in branch master
- CI/CD
- repositories.xml
- UserRoster
- BotUser & cross-robot triggering
Fix this:
```
fatal error: sync: Unlock of unlocked RWMutex
goroutine 1 [running]:
runtime.throw(0xb91930, 0x20)
/usr/local/go/src/runtime/panic.go:617 +0x72 fp=0xc000159108 sp=0xc0001590d8 pc=0x42d702
sync.throw(0xb91930, 0x20)
/usr/local/go/src/runtime/panic.go:603 +0x35 fp=0xc000159128 sp=0xc000159108 pc=0x42d685
sync.(*RWMutex).Unlock(0x1178710)
/usr/local/go/src/sync/rwmutex.go:124 +0xa2 fp=0xc000159158 sp=0xc000159128 pc=0x466a22
github.com/lnxjedi/gopherbot/bot.(*botContext).loadConfig(0xc000159bb8, 0x1, 0xc000232060, 0x7fb0c992b008)
/workspace/gopherbot/bot/conf.go:250 +0x5840 fp=0xc0001599f0 sp=0xc000159158 pc=0x7c6fc0
github.com/lnxjedi/gopherbot/bot.initBot(0xb7a884, 0x1, 0xc000024180, 0x14, 0xc0001fe320)
/workspace/gopherbot/bot/bot_process.go:109 +0x2ee fp=0xc000159e58 sp=0xc0001599f0 pc=0x7b4cae
github.com/lnxjedi/gopherbot/bot.Start(0xb84682, 0xf, 0xcb81f8, 0x7)
/workspace/gopherbot/bot/start_unix.go:114 +0x889 fp=0xc000159f68 sp=0xc000159e58 pc=0x7f74d9
main.main()
/workspace/gopherbot/main.go:69 +0x51 fp=0xc000159f98 sp=0xc000159f68 pc=0xa14821
runtime.main()
/usr/local/go/src/runtime/proc.go:200 +0x20c fp=0xc000159fe0 sp=0xc000159f98 pc=0x42f06c
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000159fe8 sp=0xc000159fe0 pc=0x45ca51
goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x41
```
### Wishlist
These items aren't required for release, but desired soonish
- TODO: Clean up platform-specific code; esp. runtasks_(unix|linux|windows).go to reduce duplicate code
- TODO: Clean up job running output; remove messages in ExtendNamespace, emit message in localbuild
- TODO: Add 'build reponame (branch)' shorthand command with configurable reponame->repo URL mapping
- TODO: (maybe later) clean up IncomingMessage / botContext struct to eliminate dupes from the ConnectorMessage
- TODO: (f) skip to final (failed) task for history; may need to modify Section history breaks for non-primary pipeline tasks
- TODO: Add tests that check behavior of UserRoster / attributes, user w/ no username, etc.
- TODO: Update 'Starting job xxxx' message to include arguments; e.g. 'Starting job localtrusted github.com/lnxjedi/gopherbot master'
- TODO: Ansible playbook / Dockerfiles: /var/lib/gopherbot/(.env, custom) gopherbot:550:bin:root; custom:700:robot:robot; /opt/gopherbot/gopherbot suid robot
- TODO: CommandOnlyUsers - to allow bots to talk to each other without matching ambient messages;
- TODO: Decrypt brain utility for removing 2nd level of encryption (secrets still encrypted)
## Protecting Secrets
This section outlines potential means of providing secrets to the robot in a secure manner, such that even malicious external scripts / plugins / jobs would not be able to obtain the Slack token or encryption key.
### SUID robot executable
This is the most promising.
Create e.g.:
* $GOPHER_INSTALLDIR/custom, root:root:500
* custom/repo, robot:robot:700 - location of robot configuration repository
* custom/brain, robot:robot:700 - location for robot file brain
* custom/.env, robot:robot:400 - starting environment
* Docker WORKDIR=$GOPHER_INSTALLDIR/custom
* systemd WorkingDirectory=$GOPHER_INSTALLDIR/custom
* /opt/gopherbot/gopherbot runs SUID robot
TODO:
* Update calling convention in runtasks.go to always use "/dev/stdin"
* DONE - GopherCI needs GetRepoData() to run in WorkSpace
* DONE - Return interpreter args - and use them - in getInterpreter; e.g. `#!/bin/bash -e` should return `/bin/bash`, [ "-e" ]
* DONE - Add "Protected" flag for jobs that run in configpath instead of workspace
* DONE - When relpath == true, run tasks by connecting script to stdin and running `<interpreter> /dev/stdin args`, otherwise use usual method
* DONE - Mark `update` job as protected and test
* DONE/obsoleted - Load gopherbot.env from configpath, in both start_* and conf.go
* DONE - Integrate godotenv for loading environment from $cwd/.env (obsoleted: & $cwd/gopherbot.env)
* DONE - Update startup to allow for relative path to repo & brain
* DONE - Put workings in to allow config repo update to happen in the jail
* DONE - Prevent scripts/plugins that are NOT update from setting the working dir relative to cwd
Add to documentation:
#### Running a robot with the 'term' connector
In many cases you can develop plugins and test your robot with the simple `terminal` connector. For developing tests, use `make test` to build a binary with Event gathering and display enabled. To run the robot, `cd` to the configuration directory and e.g.:
```shell
$ GOPHER_PROTOCOL=term path/to/gopherbot -l /tmp/term.log
```
The test suite configuration directories set the protocol and log file in `gopherbot.env`, so with a test configuration directory you can simply use `/path/to/gopherbot`.
### Environment Scrubbing
Gopherbot should make every attempt to prevent credential stealing by any potential rogue code. One ugly 'leak' of sensitive data comes from `/proc/\<pid\>/environ` - once a process starts, the contents of `environ` remain unchanged, even after e.g. `os.Unsetenv(...)`. One way to scrub the environment is to use `syscall.Exec`:
```go
package main
import (
"bufio"
"fmt"
"os"
"syscall"
)
func main() {
reader := bufio.NewReader(os.Stdin)
if len(os.Environ()) > 0 {
fmt.Println("Scrubbing the environment, check my PID/environ and press <enter> to start")
fmt.Print("-> ")
_, _ = reader.ReadString('\n')
self, _ := os.Executable()
syscall.Exec(self, []string{}, []string{})
}
os.Setenv("SCRUBBED", "true")
fmt.Println("Done. Check PID/environ and press <enter> to finish")
fmt.Print("-> ")
_, _ = reader.ReadString('\n')
}
```
When the program is run and prints `Done. ...`, `/proc/\<pid\>/environ` is empty. This algorithm is incomplete, though - now NONE of the environment is available to the new process. The 'fix' for this (needs to be written/tested) is for the initial process to spawn a goroutine that re-runs `gopherbot` with the single argument `export`, and keeps an open FD reading from stdout. The main process can then call `syscall.Exec(...)` and read the env from the FD in to potentially a memguard LockedBuffer, where it can be safely dealt with.
## Robot Configuration
The stuff here needs to be checked / verified. Some may already be done.
To simplify locking:
* Some of the content of gopherbot.yaml should only be processed when the robot first starts, e.g. the brain, connector, listening port, logger
* Start-up items should be stored globally and readable without a lock
* These start-up items should be processed during initbot
* Items that can change on reload should be stored in a config struct similar to plugins and jobs; when the botContext is registered, it should get a copy of this struct that doesn't change for the life of the context, just like the task list
* Items that are processed to binary representations (e.g. string to loglevel) should be stored in non-public struct members; e.g. LogLevel(string) and logLevel(int)
## TODOs:
- DONE/TEST/DOCUMENT: fix configuration merging to include plugin default config
- DONE/TEST/DOCUMENT: Make sure plugins/tasks/jobs can be disabled in gopherbot.yaml
- TODO: Move *.sample files to new resources/custom - sample remote config repository
- TODO: Evaluate / update Ansible role to simplify use
- TODO: Low Priority - Support for Go jobs / tasks (can wait 'til first need)
- DONE/DOCUMENT: try converting lists of tasks/jobs/plugins to map[name]struct
- To obtain the `BXXXXXXXX` ID for a webhook, visit the settings page, e.g.: https://myteam.slack.com/services/BC7DLEPA8 <- that's it!
- TODO: Low Priority - Audit reload cases where something not configured needs to be un-configured, similar to scheduled jobs n->0
- TODO: Add WorkSpace to output from `info`
- TODO: paging for show log similar to history; same 2k page size, but showing in reverse (e.g. 'p' for previous page, 'q' to quit)
- TODO: Low Priority - Brain optimization; keep a global seenCache map[string]bool:
- if the entry exists, it's been seen and the bool indicates whether the memory exists
- if the entry doesn't exist, check the memory and store whether it exists
- when storing a memory, always store that the memory exists
- best of all, the brain loop should mean no mutex!
- TODO/VERIFY: (This may be complete) Audit use of 'debug' - most calls should be debugT? (before bot.currentTask is set)
- TODO: (LP) Move slack send loop into anon func in Run
- TODO: Documentation for events, emit, development and testing - Developing.md
- DONE/DOCUMENT: Stop hard-coding 'it' for contexts; 'server:it:the server' instead of
'server', (context followed by generics that match); e.g. "reboot server",
"reboot the server", "reboot it" would all check the "server" short-term
memory to get e.g. "webtest1.example.com"
- TODO: This needs more thought - the 'catchall' plugin should be in gopherbot.yaml (There Can Be Only One)
- TODO/VERIFY: Plugin debugging verbose - emit messages for user message matching(?); when the user requesting debugging sets verbose, message match checks should trigger debug messages as well if the plugin being debugged has message matchers
- TODO: Re-sign psdemo and powershell lib (needs to be done on Windows)
- DONE: Update ansible role to stop creating installed gopherbot.yaml, favoring version
from install archive
## Consider
Stuff to think about:
- Consider: Use globalLock for protocolConfig, brainConfig, elevateConfig
- Consider: should configuration loading record all explicit values, with intentional defaults when nothing configured explicitly? (probably yes, instead of always using the zero-values of the type)
- Are all commands checked before all message matchers? Multiple matching commands should do nothing, but what about multiple message matchers? Probably, all matching message matchers should run.
# Historical/Completed
(This stuff can be removed later, but is left for now as a reminder of what needs to be documented)
- Fix history paging; goes to next page if no reply
- Update jobbuiltins / history to list namespaces
- Create `run` task for running a command/script in a repository
- When a task in a pipeline is a job:
- If it's a different job, create a new botContext and call startPipeline with a parent argument != nil == pointer to current bot
- Start with inheriting environment, user, channel, msg, and namespace; can re-visit what's inherited in child jobs later
- Let registerActive take a parent arg; if non-nil, registerActive will set parent and child members of bots while holding the lock
- Add WorkSpace top-level item to the robot; the r/w wdirectory where it clones repositories and does stuff; export GOPHER_WORKSPACE in environment
- Remove WorkDirectory from jobs; jobs should consult environment vars and call SetWorkingDirectory
- Ansible role should create WorkSpace and put it in the gopherbot.yaml
- Change Exclusive behavior: instead of just returning true when bot.exclusive, make sure tag matches, and return false if not and log an error; also fail if jobName not set
- Put .gopherci/pipeline.sh in repositories
- Create gitea-ci job triggered by messages in dev
- Clone repository in e.g WorkSpace/ansible/deploy-gopherbot
- SetParameter PIPE_STARTED=true
- If .gopherci/pipeline.sh exists, run it with no args
- pipeline.sh should call and check e.g. Exclusive deploy-gopherbot
- pipeline.sh can add self with stages: AddTask $GOPHER_JOBNAME tests
- Since PIPE_STARTED is true, it'll just run pipeline.sh with the arg
- Export GOPHER_JOBNAME
- Repositories like role-foo should just have pipelines that call several SpawnPipeline "$GOPHER_JOBNAME" ansible deploy-gopherbot
- Stop 'massaging' regexes; instead, collapse multiple spaces in to a single space before checking the message in dispatch
## Pipelines
Jobs and Plugins can queue up additional jobs/tasks with AddTask(...) and FinalTask(...), and if the job/plugin exits 0, the next task in the list will be run, subject to security checks. FinalTask() is for cleanup; all FinalTasks run regardless of the last exit status, and without security checks.
### Pipeline algorithm model
When a task in the middle of the pipeline adds tasks, it creates a new pipeline; when a task at the end of the pipeline adds tasks, they're just added to the end
of the currently running pipeline.
```go
package main
import (
"fmt"
)
func runPipe(p []int) {
fmt.Println("Running pipeline")
l := len(p)
n := make([]int, 0)
for i := 0; i < l; i++ {
switch i {
case 2:
n = append(n, 7, 8)
case 4:
n = append(n, 10, 11, 12)
}
t := len(n)
fmt.Printf("Task: %d, index %d, added: %d\n", p[i], i, t)
if t > 0 {
if i == l - 1 {
fmt.Println("Appending...")
p = append(p, n...)
l += t
} else {
fmt.Println("Running new pipeline...")
runPipe(n)
}
n = []int{}
}
}
fmt.Println("End pipeline")
}
func main() {
t := []int{ 0, 1, 2, 3, 4 }
runPipe(t)
fmt.Println("Hello, playground")
}
```
### How Things Work
* Everything the robot does will be a pipeline; most times it will just be a single job or plugin run
* Every item in the pipeline will be a job, plugin, or task defined in `gopherbot.yaml`
* Builtins will be augmented with commands for listing all current pipelines running, which can possibly be canceled (killed). Certain builtins will be allowed to run even when the robot is shutting down:
* Builtins that run almost instantly
* Builtins that read internal data but don't start new pipelines
* Builtins that report on running pipelines or allow aborting or killing/cancelling pipelines
* The `botContext` object, created at the start of a pipeline, will take on a more important role of carrying state through the pipeline; in addition to other struct data items, it will get a `runID` incrementing integer for each job/plugin
* When a pipeline starts a pointer to the Robot will be stored in a global table of running pipelines
* Pipelines can be started by:
* A job `Trigger`
* A message or command matching a plugin
* A scheduled job running
* A job being triggered manually with the `run job...` builtin (subject to security checks)
* New jobs started from Triggers, Scheduled tasks, or `run job ...` will take arguments
* The Robot object will carry a pointer to the current plugins and jobs
* In addition to the pluginID hash, there will be runID, a monotonically incrementing int that starts with a random int and is OK with rollovers
* runIDs will be initilized and stored in RAM only
* The Robot will carry a copy of the runID
* The runID will be passed to plugins and returned in method calls to identify the Robot struct
* A global hash indexed by pluginID and runID will be set before the first job/plugin in a pipeline is run
* The global hash will be used to get the Robot object for the pipeline in http method calls
* The Robot needs a mutex to protect it, since admin commands can read from the Robot while a pipeline is running
* The Robot will also get a historyIndex array for each job with histories:
* The historyIndex is only needed when the number of histories kept is > 0
* It has a monotically increasing int of every run of the pipeline
* It has a datestamp
* It has a record of the args to the pipeline
* AddTask(...) will replace CallPlugin
### TODO
* Move KeepLogs and WorkingDirectory to BotJob
* Make "update" a job triggered by the update plugin
* Initialize history on the first job in the initial pipeline that has KeepLogs > 0
* Create new botContext for sub-pipelines; if parent context has no history, the child context may start one
* Implement "SpawnPipeline" - create a new goroutine / botContext and run startPipeline; allow trigger / plugin to start multiple jobs in parallel whose success or failure doesn't affect the running pipeline
* Add SetWorkingDirectory(...) bool method, allow relative paths relative to pipeline WorkingDirectory
* Set loglevel to Debug and clean up redundant log entries in runtasks
* Audit / update logging & history sections for starting pipelines & sub-pipelines
* Add builtins for listing and terminating running tasks
Algorithm mock-up for what to do after callTask in runPipeline:
```go
package main
import (
"fmt"
)
func main() {
requeued := false
for i := 0; i < 5; i++ {
fmt.Printf("Index is %d\n", i)
if i == 0 && !requeued {
i--
requeued = true
}
}
}
```
## Histories
Whenever a new pipeline starts, if the initiating job/plugin has KeepLogs > 0, a history file will be recorded, tagged with the name of the job/plugin.
### TODO
* Add a *botConf member to the Robot
* Use confLock.RLock() in registerActive() to obtain a copy of config
* Modify bot methods to query config items from the config (e.g. in logging) without locks
## Encrypted Brain
TODO: Add this info to documentation
When the robot starts with an encrypted brain, it'll look for a provided key to
decrypt the "bot:brainKey" memory to retrieve the actual key used to en-/de-crypt memories.
* If brainKey doesn't exist, it will be randomly generated and encrytped with the provided key
* If it exists but can't be decrypted, encryptBrain will be true but cryptKey.initialized will be false
* Low-level brain functions can check the encryptBrain bool w/o locking
* cryptKey is protected by an RWLock so an admin can later provide a key for unlocking the brainKey
* When encryptBrain is true and cryptKey.initialized is true, a failure to decrypt a memory should be interpreted as an unencrypted memory that gets encrypted and stored, then returned
* Add admin command 'convert \<key\>' to forces the robot to read the memory and re-store
* Robot should take a new EncryptBrain bool parameter
* BrainKey is optional, can be specified at start or runtime
* The BrainKey should unlock the 'real' brainKey, for later re-keying if desired, e.g. if the admin switches from a configured key to a runtime-supplied key
### Datum Processing
Datum are protected by serializing all access to memories through a select loop. It should be reviewed / rewritten:
* Replace "seen" with a timestamp
* Guarantee 2 seconds exclusive access
* Lower the brain cycle to 0.1s, guaranteeing 2.0-2.1s access to a memory
### TODO
* Write reKey function
* Write 'rekey brain \<foo\>' admin commands
## Plugins and Jobs
### Augmentations
Plugins and jobs share a common botCaller struct. Both can have a defined NameSpace,
which determines sharing of long-term memories and environment variables (parameters).
Plugins can now also reference parameters which are set as environment variables. An example might be credentials that an administrator sets with the 'set environment ...' built-in.
#### NameSpaces
Changes to NameSpaces:
- Get rid of PrivateNameSpace - no more inheritance
- Add NameSpace to tasks so they can share with jobs, plugins
- Exclusive is for jobs only (1 per pipeline), so c.jobName must be set to use it
Long-term memories have always been scoped by plugin name, so when "rubydemo" stores a long-term memory called e.g. `preferences`, internally the key has been set to `rubydemo:preferences`. Now, both plugins and jobs will be able to explicitly set a `NameSpace`, which determines what other jobs/plugins share the long-term memories. Also, environment variables will be scoped by namespace, and organized under a `bot` top-level namespace, e.g. `bot:environment:rubydemo:secretkey`.
The `bot` top-level namespace will be an illegal name for a job or plugin.
Jobs or plugins that don't explicitly set `NameSpace` will default to NameSpace==job or plugin name.
### Security
#### Change in the model
Previously, Gopherbot design tried to provide some separation between plugins, to protect local plugins from third-party. This added extra complexity to configuration, which leads to brittleness. For Gopherbot to become more of a DevOps bot capable of running scheduled jobs, and with built-in knowledge of pipelines, security between plugins has been mostly removed. NameSpace support provides some level of separation, but mainly so that sharing between jobs/plugins is intentional and not accidental.
Security now focuses mainly on determining which users can run which plugins and start which jobs, and in what channels (for visibility of actions).
Documentation on elevation and authorization should be clarified:
#### Authorization
`Authorization` answers the question "is this user allowed to perform this action?" This is normally done by checking group membership of some sort (using the value of `AuthRequire`), but could also be based on values for command and its arguments.
#### Elevation
`Elevation` answers the question "how certain is it that this request REALLY came from the user, and not somebody else who found an unlocked chat session?" This is best accomplished by an easy two-factor method such as providing an OTP code or using Duo.
#### How Things Work
##### Scheduled Jobs / Plugins
Scheduled jobs configured in `gopherbot.yaml` should never trigger elevation or authorization checks. An `automaticTask` bool in the created Robot object should indicate this.
##### Interactive Jobs
Jobs run interatively with the `run job ...` built-in will Elevate if an Elevator is configured, and Authorize if an Authorizer and AuthRequire are set.
For jobs triggered interactively, authorization and elevation will be evaluated for each job in the pipeline.
* Elevation will only ever be checked once in a pipeline, after which the `elevated` flag will be set in the Robot object for the pipeline; if elevation fails at any point the pipeline will stop with an error
* Authorization will be checked for the triggering user for each job/plugin in the pipeline; failure at any point will fail the pipeline with an error
#### Elevation
If a job/plugin requires elevation, the configured elevator for that job/plugin will run, and if Elevation succeeds
### Differences
#### Parameters vs. Arguments
Plugins, as always, will take arguments starting with a command. Jobs will instead take
parameters, which get set as environment variables.
* Administrators can use a built-in 'set environment \<scope\> var=value' command, direct
message only, to set environment vars that will be attached to the Robot when the
pipeline starts. The `global` scope will be set for all jobs & plugins, otherwise scope is a NameSpace.
* Plugins and Jobs can use a SetSessionParameter method that sets an environment variable in the current plugin, and in the Robot object for future jobs/plugins in the pipeline
#### Requirements
Jobs can be normal scripts in ANY scripting language, and need not import or
use the bot library. Jobs don't get called with "configure" and "init" the way
a plugin does.
#### Triggers
Plugins are triggered when a user issues a command or sends a message matching
a plugins CommandMatchers or MessageMatchers. Jobs can be triggered in one of
three ways:
* Configuring a `ScheduledJob` in `gopherbot.yaml`
* A user using the `run job ...` builtin command
* Another bot or integration triggers the job by matching one of the job's
`Triggers`
</file_content file: gopherbot-doc-main/doc/src/DevNotes.md>
<preamble file: gopherbot-doc-main/doc/src/Environment-Variables.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/Environment-Variables.md>
# Gopherbot Environment Variables
**Gopherbot** makes extensive use of environment variables, both for configuring the robot and plugins, and for providing parameters to external scripts. This article describes the various environment variables and their use; for the environment applicable to a given running task, see [per-task environment](pipelines/TaskEnvironment.md).
## Robot Execution Environment
Certain environment variables can be supplied to the running **Gopherbot** process to configure and/or bootstrap your robot. These environment variables can be set by:
* `systemd` - Not recommended; while systemd can provide environment variables to your robot, it's very insecure and will allow local users on the system to view the values
* `docker`, `docker-compose` or [Kubernetes](https://kubernetes.io) - these and other container environments provide more secure means of providing environment variables to containers
* `$GOPHER_HOME/.env` - the most secure means of providing environment variables to your robot is creating a `.env` file in `$GOPHER_HOME`, outside of any git repository, mode `0600`
The last two options are recommended for production deployments of a **Gopherbot** robot.
### Start-up Environment
The following values can be provided to your robot on start-up:
* `GOPHER_ENCRYPTION_KEY` - 32+ character encryption key used for decrypting the `binary-encrypted-key`
* `GOPHER_CUSTOM_REPOSITORY` - clone URL for the robot's custom configuration, used in bootstrapping
* `GOPHER_CUSTOM_BRANCH` - branch to use if other than `master`
* `GOPHER_LOGFILE` - where to write out a log file
* `GOPHER_CONFIGDIR` - absolute or relative path to configuration directory
* `GOPHER_DEPLOY_KEY` - ssh deploy key for cloning the custom repository
For the optional `state` and `private` repositories, the included jobs will use the `GOPHER_CUSTOM_REPOSITORY` value with `s/gopherbot/state/` and `s/gopherbot/private/` (same branch). If desired, the values can also be supplied:
* `GOPHER_STATE_REPOSITORY` - repository holding state, normally just a file-backed brain, defaults to `$GOPHER_CUSTOM_REPOSITORY` and `robot-state` branch
* `GOPHER_STATE_BRANCH` - if `GOPHER_STATE_REPOSITORY` is set, this defaults to `master`, otherwise `robot-state`
* `GOPHER_PRIVATE_REPOSITORY` - non-public repository with `environment`, for dev only
* `GOPHER_PRIVATE_BRANCH` - branch to use if other than `master`
### Configuration Environment Variables
**Gopherbot** normally takes almost all of it's configuration from the collection of `*.yaml` files in the custom configuration directory, but for easy flexibility, a collection of environment variables are referenced in the default configuration. These are some of the values that are expanded; the actual configuration files are the definitive reference.
* `GOPHER_PROTOCOL` - used to select a non-default protocol (e.g. "terminal")
* `GOPHER_LOGLEVEL` - error, warn, info, debug, trace
* `GOPHER_BOTNAME` - the name the robot will answer to, e.g. "floyd"
* `GOPHER_ALIAS` - the one-character alias for the robot, e.g. ";"
* `GOPHER_BOTMAIL` - the robot's email address
* `GOPHER_BOTFULLNAME` - the robot's full name
* `GOPHER_HISTORYDIR` - directory for storing file-based historical job logs
* `GOPHER_WORKSPACE` - workspace directory where e.g. build jobs clone and run
* `GOPHER_BRAIN` - non-default brain provider to use
* `GOPHER_STATEDIR` - default dir for storing state, normally just the brain
* `GOPHER_BRAIN_DIRECTORY` - directory where file-based memories are stored, overrides above
* `GOPHER_JOBCHANNEL` - where jobs run by default if not otherwise specified
* `GOPHER_TIMEZONE` - UNIX tz, e.g. "America/New_York" (default)
## External Script Environment
**Gopherbot** always scrubs the environment when executing tasks, so environment variables set on execution are not automatically passed to child processes. The only environment variables that are passed through from original execution are:
* `HOME` - this should rarely be used; for portable robots, use `GOPHER_HOME`, instead
* `HOSTNAME`
* `LANG`
* `PATH` - this should be used with care since it can make your robot less portable
* `USER`
In addition to the above passed-through environment vars, **Gopherbot** supplies the following environment variables to external scripts:
* `GOPHER_INSTALLDIR` - absolute path to the gopherbot install, normally `/opt/gopherbot`
* `RUBYLIB` - path for Ruby `require 'gopherbot_v1'`, normally `/opt/gopherbot/lib`
* `PYTHONPATH` - path for Python `import`, normally `/opt/gopherbot/lib`
## Automatic Environment Variables
During startup, **Gopherbot** will examine it's environment and potentially set values for a few environment variables to support the bootstrap and setup plugins, and simplify common operations.
First, **Gopherbot** will check for custom configuration or the presence of a `GOPHER_CUSTOM_REPOSITORY` environment variable. In the absence of either, the following will be automatically set:
* `GOPHER_UNCONFIGURED` - set true
* `GOPHER_LOGFILE` - set to "robot.log" if not already set
* `GOPHER_PROTOCOL` - set to "terminal" so the default robot will start
If no custom configuration is present but `GOPHER_CUSTOM_REPOSITORY` is set:
* `GOPHER_PROTOCOL` - set to "nullconn", the null connector, to allow the bootstrap plugin to bootstrap your robot
If the robot is configured but `GOPHER_PROTOCOL` isn't set:
* `GOPHER_PROTOCOL` - set to "terminal" for local operations
* `GOPHER_LOGFILE` - set to "robot.log" if not already set
Finally, if encryption is initialized on start-up, `GOPHER_ENCRYPTION_INITIALIZED` will be set to `true`, regardless of whether the robot is configured.
## Pipeline Environment Variables
The following environment variable are set for all pipelines, whether started by a plugin or a job:
* `GOPHER_CHANNEL` - the channel where the plugin/job is providing output
* `GOPHER_CHANNEL_ID` - the protocol channel ID
* `GOPHER_MESSAGE_ID` - the opaque message id of the matching message
* `GOPHER_THREAD_ID` - an opaque string value identifying the conversation thread
* `GOPHER_THREADED_MESSAGE` - set "true" if the message was received in a thread
* `GOPHER_USER` - the user whose message created the pipeline (if any)
* `GOPHER_PROTOCOL` - the name of the protocol in use, e.g. "slack"
* `GOPHER_BRAIN` - the type of brain in use
* `GOPHER_ENVIRONMENT` - "production", unless overridden
* `GOPHER_PIPE_NAME` - the name of the plugin or job that started the pipeline
* `GOPHER_TASK_NAME` - the name of the running task
* `GOPHER_PIPELINE_TYPE` - the event type that started the current pipeline, one of:
* `plugCommand` - direct robot command, not `run job ...`
* `plugMessage` - ambient message matched
* `catchAll` - catchall plugin ran
* `jobTrigger` - triggered by a JobTrigger
* `scheduled` - started by a ScheduledTask
* `jobCommand` - started from `run job ...` command
The following are also supplied whenever a job is run:
* `GOPHER_JOB_NAME` - the name of the running job
* `GOPHER_START_CHANNEL` - the channel where the job was started
* `GOPHER_START_CHANNEL_ID` - the protocol ID for the channel where the job was started
* `GOPHER_START_MESSAGE_ID` - the opaque message id for the message that started the job
* `GOPHER_START_THREAD_ID` - the opaque thread id where the job was started
* `GOPHER_START_THREADED_MESSAGE` - whether the job was started from a message in a thread
* `GOPHER_REPOSITORY` - the extended namespace from `repositories.yaml`, if any
* `GOPHER_LOG_LINK` - link to job log, if non-ephemeral
* `GOPHER_LOG_REF` - log reference used for email log and tail log commands
The following are set at the end of the main pipeline, and can be referenced in final and fail tasks:
* `GOPHER_FINAL_TASK` - name of final task that ran in the pipeline
* `GOPHER_FINAL_TYPE` - type of last task to run, one of "task", "plugin", "job"
* `GOPHER_FINAL_COMMAND` - if type == "plugin", set to the plugin command
* `GOPHER_FINAL_ARGS` - space-separated list of arguments to final task
* `GOPHER_FINAL_DESC` - `Description:` of final task
* `GOPHER_FAIL_CODE` - numeric return value if final task failed
* `GOPHER_FAIL_STRING` - string value of robot.TaskRetVal returned
Pipelines and tasks that have `Homed: true` and/or `Privileged: true` may also get:
* `GOPHER_HOME` - absolute path to the startup directory for the robot, relative paths are relative to this directory; unset if `cwd` can't be determined
* `GOPHER_WORKSPACE` - the workspace directory (normally relative to `GOPHER_HOME`)
* `GOPHER_CONFIGDIR` - absolute path to custom configuration directory, normally `$GOPHER_HOME/custom`
### GopherCI Environment Variables
In addition to the environment variables set by the **Gopherbot** engine, the `localbuild` GopherCI builder sets the following environment variables that can be used to modify pipelines:
* `GOPHERCI_BRANCH` - the branch being built (`GOPHER_REPOSITORY` is set by `ExtendNamespace`)
* `GOPHERCI_DEPBUILD` - set to "true" if the build was triggered by a dependency
* `GOPHERCI_DEPREPO` - the updated repository that triggered this build
* `GOPHERCI_DEPBRANCH` - the updated branch
* `GOPHERCI_CUSTOM_PIPELINE` - pipeline being run if other than "pipeline"
</file_content file: gopherbot-doc-main/doc/src/Environment-Variables.md>
<preamble file: gopherbot-doc-main/doc/src/Foreword.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/Foreword.md>
# Foreword
My work with ChatOps began with Hubot around 2012 when I was working as an systems engineer for a small hosting provider. The owner had sent me a link to Hubot, asking me to have a look. I took to the concept immediately, and started automating all kinds of tasks that our support team could easily access from our team chat. Soon they were using our robot to troubleshoot email routing and DNS issues, migrate mailboxes, and build new cPanel server instances.
Not being a very talented (or motivated) Javascript/NodeJS programmer, my Hubot commands invariably followed the same pattern: write it in Javascript if it was _trivially_ easy to do so, otherwise shell out to bash and return the results. This was productive and gave results, but it was ugly and limited in functionality.
When I began teaching myself Go, I needed a good project to learn with. After my experience with Hubot, I decided to write a robot that was more approachable for Systems and DevOps engineers like myself - tasked with providing functionality most easily accessible from e.g. bash or python scripts. Towards that end, Gopherbot's design:
* Is CGI-like in operation: the compiled server process spawns scripts which can then use a simple API for interacting with the user / chat service
* Supports any number of scripting languages by using a simple json-over-http localhost interface
* Uses a multi-process design with method calls that block
Ultimately, Gopherbot gives me a strong alternative to writing Yet Another Web Application to deliver some kind of reporting, security, or management functionality to managers and technical users. It's a good meet-in-the-middle solution that's nearly as easy to use as a web application, with some added benefits:
* The chat application gives you a single pane of glass for access to a wide range of functionality
* The shared-view nature of channels gives an added measure of security thanks to visibility, and also a simple means of training users to interact with a given application
* Like a CGI, applications can focus on functionality, with security and access control being configured in the server process
It is my hope that this design will appeal to other engineers like myself, and that somewhere, somebody will exclaim "Wait, what? I can write chat bot plugins _**in BASH**_?!?"
David Parsley, March 2017 / September 2019
```bash
#!/bin/bash
# echo.sh - trivial shell plugin example for Gopherbot
# START Boilerplate
[ -z "$GOPHER_INSTALLDIR" ] && { echo "GOPHER_INSTALLDIR not set" >&2; exit 1; }
source $GOPHER_INSTALLDIR/lib/gopherbot_v1.sh
command=$1
shift
# END Boilerplate
configure(){
cat <<"EOF"
---
Help:
- Keywords: [ "repeat" ]
Helptext: [ "(bot), repeat (me) - prompt for and trivially repeat a phrase" ]
CommandMatchers:
- Command: "repeat"
Regex: '(?i:repeat( me)?)'
EOF
}
case "$command" in
# NOTE: only "configure" should print anything to stdout
"configure")
configure
;;
"repeat")
REPEAT=$(PromptForReply SimpleString "What do you want me to repeat?")
RETVAL=$?
if [ $RETVAL -ne $GBRET_Ok ]
then
Reply "Sorry, I had a problem getting your reply: $RETVAL"
else
Reply "$REPEAT"
fi
;;
esac
```
</file_content file: gopherbot-doc-main/doc/src/Foreword.md>
<preamble file: gopherbot-doc-main/doc/src/GopherDev.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/GopherDev.md>
# Working on Gopherbot
This chapter outlines the tools and methods for working on **Gopherbot** itself.
</file_content file: gopherbot-doc-main/doc/src/GopherDev.md>
<preamble file: gopherbot-doc-main/doc/src/InstallOverview.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/InstallOverview.md>
# Installation Overview
Version 2 of **Gopherbot** introduced a new, unified and simplified install method for both host- and container- based installs. Whether creating an instance of a new robot, or running an existing robot in a new location, basic installation to a host consists of a few simple steps:
1. Un-zip or un-tar the download archive in `/opt/gopherbot`
2. Create a new empty directory for your robot; by convention `\<name\>-gopherbot`, e.g. `clu-gopherbot`
1. Optionally, create a symlink to the binary, e.g.: `cd clu-gopherbot; ln -s /opt/gopherbot/gopherbot .`
3. Run the gopherbot binary from the new directory: `/opt/gopherbot/gopherbot` or `./gopherbot`
</file_content file: gopherbot-doc-main/doc/src/InstallOverview.md>
<preamble file: gopherbot-doc-main/doc/src/Installation.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/Installation.md>
# Installing and Configuring a Gopherbot Robot
> Up-to-date with v2.6
There are three distinct tasks involved in installing and running a **Gopherbot** robot:
1. The first section discusses installing the **Gopherbot** from the pre-built distribution archive or source code, normally in `/opt/gopherbot`; this provides the `gopherbot` binary, default configuration, and an assortment of included batteries (libraries, plugins, jobs, tasks, helper scripts and more); if you're using a [Gopherbot container](https://github.com/orgs/lnxjedi/packages), installation is essentially a no-op.
1. Configuring a runnable instance of a robot for your team; the included **autosetup** plugin should make this an "easy button" - discussed in the chapter on [Initial Configuration](RobotInstall.md).
1. Deploying and running your robot on a server, VM, or in a container - covered in the chapter on [Running your Robot](RunRobot.md).
## **Gopherbot** and *Robots*
It's helpful to understand the relationship between **Gopherbot** and the individual robots you'll run. It's apt to compare Gopherbot with *Ansible*:
* *Gopherbot* is similar to *Ansible* - a common code base with an assortment of included batteries, but with limited functionality on it's own; several **Go** tasks and plugins are part of the compiled binary, but the bulk of the base robot configuration is stored in **yaml** and script files under `/opt/gopherbot`
* A *Robot* is comparable to a collection of playbooks and/or roles - this is your code for accomplishing work in your environment, which uses *Gopherbot* and it's included extensions to do it's work; in the case of **yaml** configuration files your robot configuration is merged with the base, while jobs and plugins can be overridden by providing replacements in your robot's repository
Similar to Ansible playbooks and roles, individual robots may periodically require updates as you upgrade the **Gopherbot** core.
## Running in a Container
If you plan on running your **Gopherbot** robot in a container, you can skip ahead to [Team Chat Credentials](/botsetup/credentials.html). For reference, it might be useful to peruse the contents of the [Gopherbot install archive](/appendices/InstallArchive.html) describing the contents of the Gopherbot install directory and default robot.
</file_content file: gopherbot-doc-main/doc/src/Installation.md>
<preamble file: gopherbot-doc-main/doc/src/Introduction.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/Introduction.md>
# Introduction
**Gopherbot** DevOps Chatbot is a tool for teams of developers, operators, infrastructure engineers and support personnel - primarily for those that are already using Slack or another team chat platform for day-to-day communication. It belongs to and integrates well with a larger family of tools including *Ansible*, *git* and *ssh*, and is able to perform many tasks similar to *Jenkins* or *TravisCI*; all of this functionality is made available to your team via the chat platform you're already using.
To help give you an idea of the kinds of tasks you can accomplish, here are a few of the things my teams have done with **Gopherbot** over the years:
* Generating and destroying [AWS](https://aws.amazon.com) instances on demand
* Running software build, test and deploy pipelines, triggered by git service integration with team chat
* Updating service status on the department website
* Allowing support personnel to search and query user attributes
* Running scheduled backups to gather artifacts over `ssh` and publish them to an artifact service
* Occasionally - generating silly memes
The primary strengths of **Gopherbot** stem from its simplicity and flexibility. It installs and bootstraps readily on a VM or in a container with just a few environment variables, and can be run behind a firewall where it can perform tasks like rebooting server hardware over IPMI. Simple command plugins can be written in `bash`, `python` or `ruby`, with easy to use encrypted secrets for accomplishing privileged tasks. Like any user, the robot can also have its own (encrypted, naturally) ssh key for performing remote work and interfacing with *git* services.
The philosophy underlying **Gopherbot** is the idea of solving the most problems with the smallest set of general purpose tools, accomplishing a wide variety of tasks reasonably well. The interface is much closer to a CLI then a Web GUI, but it's remarkable what can be accomplished with a shared CLI for your team's infrastructure.
The major design goals for **Gopherbot** are reliability and portability, leaning heavily on "configuration as code". Ideally, custom add-on plugins and jobs that work for a robot instance in [Slack](https://slack.com) should work just as well if your team moves, say, to [Rocket.Chat](https://rocket.chat). This goal ends up being a trade-off with supporting specialized features of a given platform, though the **Gopherbot** API enables platform-specific customizations if desired.
Secondary but important design goals are configurability and security. Individual commands can be constrained to a subset of channels and/or users, require external authorization or elevation plugins, and administrators can customize help and command matching patterns for stock plugins. **Gopherbot** has been built with security considerations in mind from the start; employing strong encryption, privilege separation, and a host of other measures to make your robot a difficult target for potential attackers.
Version 2 for the most part assumes that your robot will employ encryption and get its configuration from a *git* repository. Other deployments are possible, but not well documented. This manual will focus on working with **Gopherbot** instances whose configuration is stored on [GitHub](https://github.com), but other *git* services are easy to use, as well.
That's it for the "marketing" portion of this manual - by now you should have an idea whether **Gopherbot** would be a good addition to your DevOps tool set.
</file_content file: gopherbot-doc-main/doc/src/Introduction.md>
<preamble file: gopherbot-doc-main/doc/src/Modules.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/Modules.md>
## Gopherbot Loadable Module Support
**Gopherbot's** loadable module support now allows new or updated **Go** plugins to be loaded in to a running robot (though it requires a full restart).
> NOTE: Go loadable module support has been REMOVED from Gopherbot.
</file_content file: gopherbot-doc-main/doc/src/Modules.md>
<preamble file: gopherbot-doc-main/doc/src/PrivilegeSeparation.md>
</preamble>
<file_content file: gopherbot-doc-main/doc/src/PrivilegeSeparation.md>
## Privilege Separation
One of **Gopherbot**'s primary tasks is executing external scripts, which may in some cases be third-party plugins. To harden the environment in which these scripts are run, the `gopherbot` binary can be installed setuid to a non-privileged user such as `nobody`. The following rules apply to external scripts as privileged/unprivileged:
* Unless a plugin is configured with `Privileged: true` in `robot.yaml`, the plugin will run as the unprivileged user
* Jobs will always run privileged
* Only a privileged plugin will be able to add a job to the current pipeline
* If an external plugin command is added to a pipeline with `AddCommand`, that plugin will still run unprivileged