-
Notifications
You must be signed in to change notification settings - Fork 1
/
usage.html
231 lines (193 loc) · 12.9 KB
/
usage.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
<!DOCTYPE html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<title>HiCAL - A System for Efficient High-Recall Retrieval</title>
<!-- Bootstrap & Font Awesome CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.6/css/bootstrap.min.css" integrity="sha384-rwoIResjU2yc3z8GV/NPeZWAv56rSmLldC3R/AZzGRnGxQQKnKkoFVhFQhNUwEyJ" crossorigin="anonymous">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
<link rel="stylesheet" href="static/style.css">
<link rel="shortcut icon" href="images/favicon.ico" type="image/x-icon">
<link rel="icon" href="images/favicon.ico" type="image/x-icon">
<script src="https://code.jquery.com/jquery-3.1.1.slim.min.js" integrity="sha384-A7FZj7v+d/sdmMqp/nOQwliLvUsJfDHW+k9Omg/a/EheAdgtzNs3hpfag6Ed950n" crossorigin="anonymous"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/tether/1.4.0/js/tether.min.js" integrity="sha384-DztdAPBWPRXSA/3eYEEUWrWCy7G5KFbe8fFjk5JAIxUYHKkDx6Qin1DkWx51bBrb" crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.6/js/bootstrap.min.js" integrity="sha384-vBWWzlZJ8ea9aCX4pEW3rVHjgjt7zpkNpZk+02D9phzyeVkE+jo0ieGizqPLForn" crossorigin="anonymous"></script>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-122235632-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-122235632-1');
</script>
</head>
<body>
<div class="navbar-container">
<nav class="navbar navbar-toggleable-md navbar-light ml-4 mr-4">
<a class="navbar-brand" href="index.html"><img src="images/hical.png" width="100px"></a>
<span class="navbar-text mb-3 mr-5">
<span class=" mr-1"></span> A System for Efficient High-Recall Retrieval.
</span>
<div class=" navbar-collapse-right" id="navbarTogglerDemo03">
<ul class="navbar-nav mr-auto mb-3">
<li class="nav-item">
<a class="nav-link" href="index.html">Install</a>
</li>
<li class="nav-item active">
<a class="nav-link" href="#">Usage<span class="sr-only">(current)</span></a>
</li>
<li class="nav-item">
<a class="nav-link" href="documentation.html">Documentation</a>
</li>
<li class="nav-item">
<a class="nav-link" href="paper.html">Paper</a>
</li>
<li class="nav-item">
<a class="nav-link" href="https://github.com/hical">Github</a>
</li>
</ul>
</div>
</nav>
</div>
<div class="content">
<h4>Create a user and login</h4>
<p>Once your docker images are up and running, open your browser to <a href="http://localhost:9000/" class="code-inline">http://localhost:9000/</a>.
You should be able to access the web interface for the system. Create a user (or click on practice) and login.
</p>
<div class="row">
<div class="col-md-4">
<figure class="figure">
<img src="images/login.png" class="figure-img img-fluid rounded p-1 " alt="Login page of HiCAL.">
<figcaption class="figure-caption ">Figure 1: Login page of HiCAL. You can also use the practice account by clicking on the "click to practice" button.</figcaption>
</figure>
</div>
<div class="col-md-4">
<figure class="figure">
<img src="images/home_notopic.png" class="figure-img img-fluid rounded p-1 " alt="Homepage HiCAL.">
<figcaption class="figure-caption ">Figure 2: After logging in to HiCAL, create or select a topic of search. This will start a session and train the Machine learning model.</figcaption>
</figure>
</div>
<div class="col-md-4">
<figure class="figure">
<img src="images/home_topicinit.png" class="figure-img img-fluid rounded p-1 " alt="Homepage HiCAL.">
<figcaption class="figure-caption ">Figure 3: After initiating a topic, you can select any of the retrieval component (CAL or Search) from the navigation bar.</figcaption>
</figure>
</div>
</div>
<p>Once you login, create a topic of search. Once created, navigate to the CAL or Search components using the side bar.</p>
<h2>How to use</h2>
<h4>HiCAL Web</h4>
<p>There are 2 retrieval components, CAL (Figure 4a) and Search (Figure 4b). Documents judged from the Search component are also used to update prime the machine learning model used in the CAL interface.</p>
<figure class="figure">
<img src="images/posterinterfacesv2.png" class="figure-img img-fluid mb-4" alt="Homepage HiCAL.">
<figcaption class="figure-caption text-center">Figure 4: a) Left, Document Queue Interface (CAL/AutoTAR). b) Right, Interactive Search and Judging Interface.</figcaption>
</figure>
<h5 class="mt-5">CAL Interface</h5>
<p>In this interface, the machine learning model will select the next most-likely relevant document to show you.
The model initially uses the seed query you have entered when creating the topic of search to train the model.
After each iteration of reviewing and judging, the model improves by using your previous judgments to further update the model.
<div class="alert alert-warning" role="alert">
<p class="m-0">Note! It is a good idea to first "prime" the model by judging documents from the search engine.
After a few judgments from Search, go to the CAL interface to view the documents the machine learning model has selected.</p>
</div>
<h6>Keyword Highlighting</h6>
<p>
You can highlight keywords by entering it in the highlighting search bar.
Words or part of words that match your entered keywords will be highlighted.
</p>
<h6>Judging</h6>
<p> You can judge a document by clicking at any of the judging buttons (not relevant, relevant, highly relevant) or using the keyboard shortcuts (<span class="text-danger">s</span>, <span class="text-warning">r</span>, <span class="text-success">h</span>).
To rejudge a document you have previously judged, you can click on the "<span class="text-primary">Latest judgment</span>" buttons on the top right corner, or use the keyboard shortcut "<span class="text-primary">u</span>").
</p>
<p>
Once you judge a document, the model will retrain and the next most-likely relevant document will be returned.
A cache-queue is implemented in the front-end to speed the reviewing process while waiting for the model to complete the training.
Once the training is done, the queue is automatically flushed and updated with the new set of documents.
This increases the responsiveness of the system and removes any perceptible interface lag.
</p>
<p>The model will not return anything once all documents in the corpus are judged.</p>
<div class="alert alert-primary mt-5 mb-5" role="alert">
<h4 class="alert-heading">What features people like?</h4>
<hr>
<p>An experiment with several people using our system showed that participants liked different features.
<p>
Take a look at our <a href="static/de27-abualsaudA.pdf">paper</a> to learn more.
</p>
</div>
<h6>Judging using a paragraph excerpt</h6>
<p>Often times, documents contains few paragraphs that can used to determine the relevancy of the document.
Instead of reading or searching the whole document for relevant material,
assessing the relevancy of a document by reading the most relevant paragraph saves you the time of reading unnecessary content.
</p>
<p>
By default, the interface will show you what the model has selected as the most-likely relevant paragraph.
Reviewing paragraphs instead of full documents reduces effort and review time.
</p>
<p>
If you believe you need to review the full document to determine its relevancy, you can click on the
"view full document" button to view the whole document content. The full document content will appear below the
paragraph.
</p>
<p> You can also customize the interface (see <a href="documentation.html">documentation</a>) to your needs (e.g. only show paragraph excerpts, only show full document content, etc).
</p>
<h5 class="mt-5">Search Interface</h5>
<p>The search interface allows you to search for documents using your own queries.
By default, the interface will return 10 results from the search engine.
You can select the number of documents you want to return using the dropdown button next to the search bar.
</p>
<p>Each Search Engine Result Page (SERP) item contains a snippet generated by the search engine and a set of judging buttons.
Judged documents have a vertical indicator bar showing the relevancy judgment of the document.
</p>
<p>You can click at any SERP item to view or search the full document content.</p>
<p>Any document you judge in the Search interface will be used to update the machine learning model (accessible in the CAL interface).</p>
<div class="alert alert-primary mt-5 mb-5" role="alert">
<h4 class="alert-heading">What interface should I be using?</h4>
<hr>
<p>We've ran an experiment with several people using our system to complete several search tasks. Do people like to use the Search interface? Do they like the CAL interface? Do they like the ability to view the full document along with a paragraph excerpt?
</p>
<p>Our participants have indicated that they like to have the full-fledged system (Search enabled, CAL enabled with paragraph excerpt and ability to view the full document).
Users want full control of the system!
<p>However, this comes at a cost! Full control causes users to be slow and waste time.
Our analysis shows that performance is highest when their interactions are limited to producing relevance judgments on paragraph length excerpts from the CAL interface.</p>
</p>
<p>
To learn more, click <a href="static/de27-abualsaudA.pdf">here</a> to read our paper.
</p>
</div>
<h5>Exporting judgments</h5>
<p>You can click on the archive button on the side bar to access the list of documents you have judged for your current topic. Click on "export to csv" to get a csv file of your judgments.</p>
<div class="row">
<div class="col-md-6">
<figure class="figure">
<img src="images/archive.png" class="figure-img img-fluid mb-4 rounded p-1 border gray-background" alt="Export judgments from the archive page.">
</figure>
</div>
<div class="col-md-6">
<p>Figure 5. A list of the documents you have judged for your current topic is shown in the archive page. You can export your document judgments by clicking on the "export to csv" button.</p>
</div>
</div>
<h4 class="mt-3">HiCAL Engine</h4>
<p>The CAL Interface is powered by a modified C++ implementation of the AutoTAR algorithm. This tool can be used through the command line interface or HTTP API. Detailed documentation on how to use this tool is available <a href="https://github.com/hical/HiCAL/blob/master/CALEngine/README.md">here</a>.</p>
<h2 class="mt-5">Bugs, issues or feature requests?</h2>
<p>Please report <a href="https://www.github.com/hical/hical/issues">here</a>.</p>
<h2 class="mt-5">LICENSE</h2>
<a href="http://www.gnu.org/licenses/gpl.html"><img src="images/gpl.png"></a>
</div>
<div class="content">
<ul class="sm" style="list-style: none;">
<li>
<a href="https://github.com/HiCAL" target="_blank"><i class="fa fa-github"></i></a> <span class="ml-3">/</span>
</li>
<li class="small mt-1">
GitHub code.
</li>
<li class="small mt-1">
GNU General Public License v3.0
</li>
</ul>
<p class="copy"></p>
</div>
</body>
</html>