Skip to content

Commit

Permalink
Merge branch 'master' into stable
Browse files Browse the repository at this point in the history
  • Loading branch information
jrtcppv committed Aug 3, 2021
2 parents 98e13aa + af6599e commit e91e89c
Show file tree
Hide file tree
Showing 17 changed files with 239 additions and 329 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ jobs:
command: make test
- run:
name: Front end tests
command: pytest test --host=http://$TATOR_UNIT_TEST_HOST_IP --username=admin --password=admin --screenshots $HOME -s
command: pytest test --base-url=http://$TATOR_UNIT_TEST_HOST_IP --browser=chromium --username=admin --password=admin --videos=$HOME -s
- run:
name: Copy test directories
command: cp -r scripts/packages/tator-py/test ./tatorpy_test && cp -r scripts/packages/tator-py/examples .
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,7 @@ FILES = \
projects/delete-project.js \
projects/projects-dashboard.js \
account-profile/account-profile.js \
token/token-page.js \
new-project/new-project-close.js \
new-project/custom/custom-form.js \
new-project/custom/custom.js \
Expand Down
30 changes: 29 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,15 @@ What is Tator?
* **Collaboration** - Invite your team to help annotate, analyze, and review data
* **REST API** - Develop custom scripts and tools. Clients available in Python and R

**IMPORTANT**: Only Chromium-based browsers are supported (Chrome and Edge).
Quick start
===========

From a node with a fresh [Ubuntu 20.04 LTS](https://releases.ubuntu.com/20.04/) install and a [reserved IP address](https://lifehacker.com/how-to-set-up-dhcp-reservations-and-never-check-an-ip-5822605):
```bash
git clone --recurse-submodules -b stable https://github.com/cvisionai/tator
cd tator
./install.sh
```

Tutorials
=========
Expand Down Expand Up @@ -49,6 +57,26 @@ Annotation
* [Speed up annotation with favorites](https://www.tator.io/tutorials/2021-06-29-speed-up-annotation-with-favorites/)
* [Pick up where you left off with bookmarks](https://www.tator.io/tutorials/2021-06-29-pick-up-where-you-left-off-with-bookmarks/)

Analysis
* [Explore and edit localizations](https://www.tator.io/tutorials/2021-07-09-localization-analytics-view/)
* [Apply advanced localization filters](https://www.tator.io/tutorials/2021-07-09-apply-advanced-localization-filters/)

Collaboration
* [Make a project public](https://www.tator.io/tutorials/2021-07-27-make-a-project-public/)

REST API
* [Use Swagger UI for documentation and requests](https://www.tator.io/tutorials/2021-06-21-use-swagger-ui-for-documentation-and-requests/)
* [Create a project with python](https://www.tator.io/tutorials/2021-03-11-create-a-project-with-python/)

Administration
* [Make announcements](https://www.tator.io/tutorials/2021-05-12-make-announcements/)
* [Update your account profile](https://www.tator.io/tutorials/2021-06-12-update-your-account-profile/)
* [Reset your password](https://www.tator.io/tutorials/2021-06-11-reset-your-password/)

Advanced installation
* [Set up a domain](https://www.tator.io/tutorials/2021-03-30-set-up-a-domain/)
* [Enable HTTPS](https://www.tator.io/tutorials/2021-03-30-enable-https/)

API Reference
=============
* [REST API documentation](https://www.tatorapp.com/anonymous-gateway?redirect=/rest)
Expand Down
4 changes: 2 additions & 2 deletions helm/tator/values-microk8s.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ domain: &domain <Insert static IP or domain>
# # Cert secret name that will be used in secret for this domain.
# tlsCertSecretName: mysite-tls-cert
# Enables debug mode for gunicorn. Do NOT enable this in production.
tatorDebug: true
tatorDebug: false
# Enables javascript minification.
useMinJs: false
useMinJs: true
# Enable this to turn on "down for maintenance" page.
maintenance: false
# Secret key for django. Feel free to change this.
Expand Down
13 changes: 9 additions & 4 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,12 @@ sudo snap install microk8s --classic --channel=1.19/stable
# Install apt packages.
sudo apt-get update \
&& sudo -E apt-get -yq --no-install-suggests --no-install-recommends install \
iproute2 net-tools gzip wget unzip jq ffmpeg python3 python3-pip build-essential \
chromium-chromedriver
iproute2 net-tools gzip wget unzip jq ffmpeg python3 python3-pip build-essential

# Install Chrome for front end testing.
echo "Installing Chrome."
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ./google-chrome-stable_current_amd64.deb

# Get IP address if it is not set explicitly.
# Credit to https://serverfault.com/a/1019371
Expand Down Expand Up @@ -119,7 +123,8 @@ kubectl delete pod sleepy
echo "Installing pip packages."
pip3 install --upgrade pip
pip3 install setuptools
pip3 install /tmp/*.whl pandas opencv-python pytest pyyaml selenium
pip3 install /tmp/*.whl pandas opencv-python pytest pyyaml playwright pytest-playwright
playwright install

# Install tator.
echo "Installing tator."
Expand All @@ -137,7 +142,7 @@ kubectl exec -it $GUNICORN_POD -- \

# Print success.
echo "Installation completed successfully!"
echo "Open a browser to http://$HOST_IP and enter credentials:"
echo "Open a browser (must be Chrome or Edge) to http://$HOST_IP and enter credentials:"
echo "username: admin"
echo "password: admin"
echo "If this installation is accessible by others please change your password!"
2 changes: 1 addition & 1 deletion main/rest/_annotation_query.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ def _get_annotation_psql_queryset(project, filter_ops, params, annotation_type):
return qs

def _use_es(project, params):
ES_ONLY_PARAMS = ['search', 'media_search']
ES_ONLY_PARAMS = ['search', 'media_search', 'section']
use_es = False
for es_param in ES_ONLY_PARAMS:
if es_param in params:
Expand Down
25 changes: 25 additions & 0 deletions main/rest/_attribute_query.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

from ..models import LocalizationType
from ..models import StateType
from ..models import Section
from ..search import TatorSearch

from ._attributes import KV_SEPARATOR
Expand Down Expand Up @@ -85,6 +86,30 @@ def get_attribute_es_query(query_params, query, bools, project,
attr_query['media']['filter'] += bools
attr_query['annotation']['filter'] += annotation_bools

section = query_params.get('section')
if section is not None:
section_object = Section.objects.get(pk=section)
if section_object.lucene_search:
attr_query['media']['filter'].append({'bool': {
'should': [
{'query_string': {'query': section_object.lucene_search}},
{'has_child': {
'type': 'annotation',
'query': {'query_string': {'query': section_object.lucene_search}},
},
},
],
'minimum_should_match': 1,
}})
if section_object.media_bools:
attr_query['media']['filter'].append(section_object.media_bools)
if section_object.annotation_bools:
attr_query['annotation']['filter'].append(section_object.annotation_bools)
if section_object.tator_user_sections:
attr_query['media']['filter'].append({'match': {'tator_user_sections': {
'query': section_object.tator_user_sections,
}}})

if is_media:
# Construct query for media
has_child = False
Expand Down
23 changes: 0 additions & 23 deletions main/rest/_media_query.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,29 +94,6 @@ def get_media_es_query(project, params):
if name is not None:
bools.append({'match': {'_exact_name': {'query': name}}})

if section is not None:
section_object = Section.objects.get(pk=section)
if section_object.lucene_search:
bools.append({'bool': {
'should': [
{'query_string': {'query': section_object.lucene_search}},
{'has_child': {
'type': 'annotation',
'query': {'query_string': {'query': section_object.lucene_search}},
},
},
],
'minimum_should_match': 1,
}})
if section_object.media_bools:
bools.append(section_object.media_bools)
if section_object.annotation_bools:
annotation_bools.append(section_object.annotation_bools)
if section_object.tator_user_sections:
bools.append({'match': {'tator_user_sections': {
'query': section_object.tator_user_sections,
}}})

if dtype is not None:
bools.append({'match': {'_dtype': {'query': dtype}}})

Expand Down
7 changes: 7 additions & 0 deletions main/schema/_annotation_query.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@
'items': {'type': 'integer'},
},
},
{
'name': 'section',
'in': 'query',
'required': False,
'description': 'Unique integer identifying a media section.',
'schema': {'type': 'integer'},
},
{
'name': 'type',
'in': 'query',
Expand Down
1 change: 1 addition & 0 deletions main/static/js/annotation/entity-selector.js
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ class EntitySelector extends TatorElement {

const details = document.createElement("details");
details.setAttribute("class", "position-relative");
details.setAttribute("id", "current-index");
controls.appendChild(details);

const summary = document.createElement("summary");
Expand Down
4 changes: 2 additions & 2 deletions main/static/js/project-settings/project-main-edit.js
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ class ProjectMainEdit extends TypeForm {
this._thumbUpload.setAttribute("for", "thumb");
this._thumbUpload.projectId = this.projectId;
this._thumbUpload.setValue(this.data.thumb);
this._thumbUpload.default = this.data.thumb;
this._thumbUpload.default = this.data.thumb === null ? "" : this.data.thumb;
this._thumbUpload.addEventListener("change", this._formChanged.bind(this));
this._form.appendChild(this._thumbUpload);

Expand Down Expand Up @@ -307,4 +307,4 @@ class ProjectMainEdit extends TypeForm {

}

customElements.define("project-main-edit", ProjectMainEdit);
customElements.define("project-main-edit", ProjectMainEdit);
2 changes: 1 addition & 1 deletion main/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ def updateProjectTotals(force=False):
projects=Project.objects.all()
for project in projects:
temp_files = TemporaryFile.objects.filter(project=project)
files = Media.objects.filter(project=project)
files = Media.objects.filter(project=project, deleted=False)
num_files = temp_files.count() + files.count()
if force or num_files != project.num_files:
project.num_files = num_files
Expand Down
2 changes: 1 addition & 1 deletion scripts/packages/tator-py
Submodule tator-py updated 1 files
+15 −0 test/test_search.py
81 changes: 17 additions & 64 deletions test/_common.py
Original file line number Diff line number Diff line change
@@ -1,67 +1,20 @@
import os

from selenium.common.exceptions import NoSuchElementException
from urllib.parse import urlparse, urljoin
def get_video_path(page):
""" Gets a page with video name set to the current test.
"""
dir_name = os.path.dirname(page.video.path())
test_name = os.getenv('PYTEST_CURRENT_TEST')
test_name = test_name.replace('/', '__').replace('.py::', '__').split('[')[0]
file_name = f"{test_name}.webm"
path = os.path.join(dir_name, file_name)
return path

def print_page_error(err):
print("--------------------------------")
print("Got page error:")
print(f"Message: {err.message}")
print(f"Name: {err.name}")
print(f"Stack: {err.stack}")
print("--------------------------------")

class ShadowManager:
def __init__(self, browser):
self._browser = browser

def expand_shadow_element(self, element):
return self._browser.execute_script('return arguments[0].shadowRoot', element)

def expand_shadow_hierarchy(self, tags):
""" Given a list of shadow element tag names, return the shadow dom of the
last element.
"""
shadow_root = self._browser
for tag in tags:
element = shadow_root.find_element_by_tag_name(tag)
shadow_root = self.expand_shadow_element(element)
return shadow_root

def find_shadow_tree_elements(self, dom, by, value, single=False):
""" Given description of the element you want to find, searches full shadow tree
on the page for any elements that match and returns a list.
"""
elements = []
# Try to find the specified element in the dom.
elements += dom.find_elements(by, value)
# Keep searching in nested shadow doms.
all_elements = dom.find_elements_by_tag_name("*")
for element in all_elements:
shadow = self.expand_shadow_element(element)
if shadow:
elements += self.find_shadow_tree_elements(shadow, by, value)
if single and len(elements) > 0:
break
return elements

def find_shadow_tree_element(self, dom, by, value):
""" Given description of the element you want to find, searches full shadow tree
on the page for that element and returns the first one.
"""
elements = self.find_shadow_tree_elements(dom, by, value, True)
if len(elements) > 0:
element = elements[0]
else:
raise NoSuchElementException
return element

def go_to_uri(browser, uri):
parsed = urlparse(browser.current_url)
goto = urljoin(f"{parsed.scheme}://{parsed.netloc}", uri)
browser.get(goto)

class ScreenshotSaver:
def __init__(self, browser, directory):
self._dir = directory
self._browser = browser
self._count = 0

def save_screenshot(self, description):
fname = f"{self._count:02d}_{description.lower().replace(' ', '_')}.png"
out = os.path.join(self._dir, fname)
self._browser.save_screenshot(out)
self._count += 1

Loading

0 comments on commit e91e89c

Please sign in to comment.