-
Notifications
You must be signed in to change notification settings - Fork 0
/
HACKING
1726 lines (1279 loc) · 71.3 KB
/
HACKING
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Instructions for hacking on Xapian
==================================
.. contents:: Table of contents
This file is aimed to help developers get started with working on
Xapian. The documentation contains a section covering various internal
aspects of the library - this can also be found on the Xapian website
<https://xapian.org/>.
Extra options to give to configure
==================================
Note: Non-developer configure options are described in INSTALL
You will probably want to use some of these if you're going to be developing
Xapian.
--enable-assertions
This enables compiling of assertion code which will throw
Xapian::AssertionError if the code detects violating of
preconditions, postconditions, or fails other consistency checks.
--enable-assertions=partial
This option enables a subset of the assertions enabled by
"--enable-assertions", but not the most expensive. The intention is
that it should be suitable for use in a real-world system for tracking
down problems without imposing too much of an overhead (but note that
we haven't yet performed timings to measure the overhead...)
--enable-log
This enables compiling code into the library which generates verbose
debugging messages. See "Debugging Messages", below.
--enable-log=profile
In 1.2.0 and earlier, this used to use the debug logging macros to
report to stderr how long each method takes to execute. This feature
was removed in 1.2.1 - you are likely to get better results using
dedicated profiling tools - for more information see:
https://trac.xapian.org/wiki/ProfilingXapian
--enable-maintainer-mode
This tells configure to enable make dependencies for regenerating build
system files (such as configure, Makefile.in, and Makefile) and other
generated files (such as the stemmers and query parser) when required.
These are disabled by default as some make programs try to rebuild them
when it's not appropriate (e.g. BSD make doesn't handle VPATH except
for implicit rules). For this reason, we recommend GNU make if you
enable maintainer mode. You'll also need a non-cross-compiling C
compiler for compiling the Lemon parser generator and the Snowball
stemming algorithm compiler. The configure script will attempt to
locate one, but you can override this autodetection by passing
CC_FOR_BUILD on the command line like so::
./configure CC_FOR_BUILD=/opt/bin/gcc
--enable-documentation
This tells configure to enable make dependencies for regenerating
documentation files. By default it uses the same setting as
--enable-maintainer-mode.
Debugging Messages
==================
If you configure with --enable-log, lots of places in the code generate
debugging messages to tell us what they're up to - this information can be
very useful for debugging both the Xapian library and code which uses it. But
the quantity of information generated is potentially vast so there's a
mechanism to allow you to select where to store the log and which types of
message you're interested by setting environment variables. You can:
* set XAPIAN_DEBUG_LOG to be the path to a file that you would like debugging
output to be appended to, or to the special value ``-`` to indicate that you
would like debugging output to be sent to stderr. Unless XAPIAN_DEBUG_LOG
is set, no debug logging will be performed. Occurrences of %p in
XAPIAN_DEBUG_LOG will be replaced with the current process-id.
* set XAPIAN_DEBUG_FLAGS to a string of capital letters indicating the types
of debugging message you would like to display (the default is to log calls
to API functions and methods). These letters are shown in the first column
of the log output, and are also listed in ``common/debuglog.h``. If the
first character is ``-``, then the letters indicate those categories of
message *not* be shown instead. As a consequence of this, setting
``XAPIAN_DEBUG_FLAGS=-`` will give you all debugging messages.
These environment variables only have any effect if you ran configure with the
--enable-log option.
The format is::
<message type> <pid> [<this>] <message>
For example::
A 16747 [0x57ad1e0] void Xapian::Query::Internal::validate_query()
Each nested call adds another space before the ``[`` so you can easily see
which function call and return messages correspond.
Debugging memory allocations
============================
The testsuite can make use of valgrind 3.3.0 or newer to check for memory
leaks, reads from uninitialised memory, and some other bugs during tests.
Valgrind doesn't support every platform, but Xapian contains very little
platform specific code (and most of what there is is Microsoft Windows
specific) so even just testing with valgrind on one platform gives good
coverage.
If you have a new enough version of valgrind installed, it's automatically
detected by configure and used when running the testsuite. The testsuite runs
more slowly under valgrind, so if you wish to disable this auto-detection you
can run configure with:
./configure VALGRIND=
Or you can disable use of valgrind during a particular run of "make check"
like so:
make check VALGRIND=
Or disable it while running a test directly (under sh or bash):
VALGRIND= ./runtest ./apitest
Running test programs
=====================
To run all tests, use ``make check``. You can also run just the subset of
tests which exercise the inmemory, remote progserver, remote TCP,
multi-database, glass, or chert backends using ``make check-inmemory``,
``make check-remoteprog``, ``make check-remotetcp``, ``make check-multi``,
``make check-glass``, or ``make check-chert``
respectively.
Also, ``make check-remote`` will run the tests on both variants of the remote
backend, and ``make check-none`` will run those tests which don't use any
backend. These are handy shortcuts when doing development work on a particular
backend.
The runtest script (in the tests subdirectory) takes care of the details of
running the test programs (including setting up the environment so they work
when srcdir != builddir and handling libtool dynamically linked binaries). To
run a test program by hand (rather than via make) just use:
./runtest ./apitest
You can specify options and arguments. Individual test programs optionally
take one or more test names as arguments, and you can also pass ``-v`` to get
more verbose output from failing tests, e.g.:
./runtest ./apitest -v deldoc1
If the number of the test is omitted, all tests with that basename are run,
so to run deldoc1, deldoc2, etc:
./runtest ./apitest deldoc
You can also use runtest to run a test program under gdb (or most other tools):
./runtest gdb ./apitest -v deldoc1
./runtest valgrind ./apitest -v deldoc1
Some test programs take special arguments - for example, you can restrict
apitest to the glass backend using ``-bglass``.
There are a few environment variables which the testsuite harness checks for
which you might find useful:
XAPIAN_TESTSUITE_SIG_DFL:
By default, the testsuite harness catches signals and handles them
gracefully - the current test is failed, and the testsuite moves onto the
next test. If you want to suppress this (some debugging tools may work
better if the signal is not caught) set the environment variable
XAPIAN_TESTSUITE_SIG_DFL to any value to prevent the testsuite harness
from installing its own signal handling.
XAPIAN_TESTSUITE_OUTPUT:
By default, the testsuite harness uses ANSI escape sequences to give
colour output if stdout is a tty. You can disable this feature by setting
XAPIAN_TESTSUITE_OUTPUT=plain (alternatively, piping the output (e.g.
through ``cat`` or ``more``) will have the same effect). Auto-detection
can be explicitly specified with XAPIAN_TESTSUITE_OUTPUT=auto (or empty).
Any other value forces the use of colour. Colour output is always disabled
on Microsoft Windows, so XAPIAN_TESTSUITE_OUTPUT has no effect there.
XAPIAN_TESTSUITE_LD_PRELOAD:
The runtest script will add this to LD_PRELOAD if it is set, allowing you
to easily load LD_PRELOAD libraries when running the testsuite. The
original intended use was to allow use of libeatmydata
(https://www.flamingspork.com/projects/libeatmydata/) which makes fsync
and related calls no-ops, but configure now checks for the eatmydata
wrapper script and this is used automatically. However, there may be
other LD_PRELOAD libraries which are useful, so we've left the machinery
in place.
Speeding up the testsuite with eatmydata
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The testsuite does a lot of small database operations, and the calls to fsync,
fdatasync, etc which Xapian makes by default can slow down testsuite runs
substantially. There's a handy LD_PRELOAD library called eatmydata
(https://www.flamingspork.com/projects/libeatmydata/), which can help here, by
turning fsync and related calls into no-ops.
You need a version of eatmydata with the eatmydata wrapper script (version 37
or newer), and then configure should auto-detect it and it'll get used when
running the testsuite (via runtest). If you wish to disable this
auto-detection for some reason, you can run configure with:
./configure EATMYDATA=
Or you can disable use of eatmydata during a particular run of "make check"
like so:
make check EATMYDATA=
Or disable it while running a test directly (under sh or bash):
EATMYDATA= ./runtest ./apitest
Using various debugging, profiling, and leak-finding tools
==========================================================
GCC's libstdc++ supports a debug mode, which checks for various misuses of
the STL - to enable this, define _GLIBCXX_DEBUG when building Xapian:
./configure CPPFLAGS=-D_GLIBCXX_DEBUG
For documentation of this option, see:
https://gcc.gnu.org/onlinedocs/libstdc++/manual/debug_mode.html
Note: all C++ code must be compiled with this defined or you'll get problems.
Xapian's API headers include a check that the same setting is used when
building code using Xapian as was used to build Xapian.
To use valgrind (http://www.valgrind.org/), no special build options are
required, but make sure you compile with debugging information (on by default
for GCC) and the valgrind documentation recommends disabling optimisation (with
optimisation, line numbers in error messages can be confusing due to code
inlining, etc):
./configure CXXFLAGS='-O0 -g'
To use gdb (https://www.gnu.org/software/gdb/), no special build options are
required, but make sure you compile with debugging information (on by default
for GCC). You'll probably find debugging easier if you compile without
optimisation (with optimisation, line numbers in error messages can be
confusing due to code inlining, etc, and the values of some variables can't be
printed because they've been eliminated from the code completely):
./configure CXXFLAGS='-O0 -g'
To enable profiling for gprof:
./configure CXXFLAGS=-pg LDFLAGS=-pg
To use Purify (a proprietary tool):
./configure CXXLD='purify c++' --disable-shared
To use Insure (another proprietary tool):
./configure CXX=insure
To use lcov (at least version 1.10) to generate a test coverage report (see
`lcov.xapian.org <http://lcov.xapian.org/>`_ for reports) there are three make
targets (all in the `xapian-core` directory):
* `make coverage-reconfigure`: reruns configure in the source tree. See
Makefile.am for details of the configure options used and why they
are needed. If you're using ccache, make sure it's at least version
3.0, and ideally at least 3.2.2.
* `make coverage-reconfigure-maintainer-mode`: does the same thing, except
the tree is configured in "maintainer mode", which is what you want if
generating coverage reports while working on the code.
* `make coverage-check`: runs `make check` and generates an HTML report in a
directory called `lcov`.
+ You can specify extra arguments to pass to the ``genhtml`` tool using
`GENHTML_ARGS`, so for example if you plan to serve the generated HTML
coverage report from a webserver, you might use:
`make coverage-check GENHTML_ARGS=--html-gzip`
You ideally want lcov 1.11 or later, since 1.11 includes patches to reduce
memory usage significantly - lcov 1.10 would run out of memory in a 1GB VM.
If you have runes for using other tools, please add them above, or send them
to us so we can.
Snapshots
=========
If you want to try unreleased Xapian code, you can fetch it from our git
repository. For convenience, we also provide bootstrapped tarballs (much like
the sourcecode download for any release version) which get built every 20
minutes if there have been any changes checked in. These tarballs need to
pass "make distcheck" to be automatically uploaded, so using them will help
to assure that you don't pick a "bad" version. The snapshots are available
from the "Bleeding Edge" page of the Xapian website.
Building from git
=================
When building from a git checkout, we *strongly* recommend that you use
the ``bootstrap`` script in the top level directory to set up the tree ready
for building. This script will check which directories you have checked out,
so you can bootstrap a partial tree. You can also ``touch .nobootstrap`` in
a subdirectory to tell bootstrap to ignore it.
You will need the following tools installed to build from git:
* GNU m4 >= 4.6 (for autoconf)
* perl >= 5.6 (for automake; also for various maintainer scripts)
* python >= 2.3 (for generating the Python bindings)
* GNU make (or another make which support VPATH for explicit rules)
* GNU bison (for building SWIG, used for generating the bindings)
* Tcl (to generate unicode/unicode-data.cc)
For a recent version of Debian or Ubuntu, this command should ensure you have
all the necessary tools and libraries::
apt-get install build-essential m4 perl python zlib1g-dev uuid-dev wget bison tcl
If you want to build Omega, you'll also need::
apt-get install libpcre3-dev libmagic-dev
On Fedora, the uuid library can be installed by doing::
yum install libuuid-devel
On Mac OS X, if you're using macports you'll want the following:
* file (magic.h in configure)
If you're using homebrew you'll want the following::
brew install libmagic pcre
If you're doing much development work, you'll probably also want the following
tools installed:
* valgrind for better testsuite error finding
* ccache for faster rebuilds
* eatmydata for faster testsuite runs
The repository does not contain any automatically generated files
(such as configure, Makefile.in, Snowball-generated stemmers, Lemon-generated
parsers, SWIG-generated code, etc) because experience shows it's best to keep
these out of version control. To avoid requiring you to install the correct
versions of the tools required, we either include the source to these tools in
the repo directly (in the case of Snowball and Lemon), or the bootstrap script
will download them as tarballs (autoconf, automake, libtool) or
from git (SWIG), build them, and install them within the source tree.
To download source tarballs, bootstrap will use wget, curl or lwp-request if
installed. If not, it will give an error telling you the URL to download from
by hand and where to copy the file to.
Bootstrap will then run autoreconf on each of the checked-out subdirectories,
and generate a top-level configure script. This configure script allows you to
configure xapian-core and any other modules you've checked out with single
simple command, such that the other modules link against the uninstalled
xapian-core (which is very handy for development work and a bit fiddly to set
up by hand). It automatically passes --enable-maintainer-mode to the
subprojects so that the autotools will be rerun if configure.ac, Makefile.am,
etc are modified.
The bootstrap script doesn't care what the current directory is. The top-level
configure script generated by it supports building in a separate directory to
the sources: simply create the directory you want to build in, and then run the
configure script from inside that directory. For example, to build in a
directory called "build" (starting in the top level source directory)::
./bootstrap
mkdir build
cd build
../configure
When running bootstrap, if you need to add any extra macro directories to the
path searched by aclocal (which is part of automake), you can do this by
specifying these in the ACLOCAL_FLAGS environment variable, e.g.::
ACLOCAL_FLAGS=-I/extra/macro/directory ./bootstrap
If you wish to prevent bootstrap from downloading and building the autotools
pass the --without-autotools option. You can force it to delete the downloaded
and installed versions by passing --clean.
If you are tracking development in git, there will sometimes be changes
to the build system sources which require regeneration of the generated
makefiles and associated machinery. We aim to make the build system
automatically regenerate the necessary files, but in the event that a build
fails after an update, it may be worth re-running the bootstrap script to
regenerate the build system from scratch, before looking for the cause of the
error elsewhere.
Tools required to build documentation
-------------------------------------
If you want to be able to build distribution tarballs (with "make dist") then
you'll also need some further tools. If you don't want to have to install all
these tools, then pass --disable-documentation to configure to disable these
rules (the default state of this follows the setting of
--enable-maintainer-mode, so in a non-maintainer-mode tree, you can pass
--enable-documentation to enable these rules). Without the documentation,
"make dist" will fail (to prevent accidentally distributing tarballs without
documentation), but you can configure and build.
The documentation tools are:
* doxygen (v1.8.8 is used for 1.3.x snapshots and releases; 1.7.6.1 fails to
process trunk after PL2Weight was added).
* dot (part of Graphviz. Doxygen's DOT_MULTI_TARGETS option apparently needs
">1.8.10")
* help2man
* rst2html or rst2html.py (in python-docutils on Debian/Ubuntu)
* pngcrush (optional - used to reduce the size of PNG files in the HTML
apidocs)
* sphinx-doc (in python-sphinx and python3-sphinx on Debian/Ubuntu, or as
sphinx via pip install)
For a recent version of Debian or Ubuntu, this command should install all the
required documentation tools::
apt-get install doxygen graphviz help2man python-docutils pngcrush python-sphinx python3-sphinx
Documentation builds on OS X
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Mac OS X, if you're using homebrew, you'll want the following::
brew install doxygen help2man graphviz pngcrush
(Ensure you're up to date with brew, as earlier packaging of graphviz
didn't properly install dot.)
You also need sphinx and docutils, which are python packages; you can
install them via pip::
pip install sphinx docutils
You may find it easier to use homebrew to install python first, so
these packages are separate from the system python::
brew install python
If you install both python (v2) and python3 (v3) via homebrew, you
will be able to build bindings for both; you'll then need to install
sphinx for python3::
pip3 install sphinx
PDF versions of docs
~~~~~~~~~~~~~~~~~~~~
As of 1.3.2, we no longer build PDF versions of the API docs by default, but
you can build them yourself with::
make -C docs apidoc.pdf
Additional tools are needed for these:
* gs (part of Ghostscript)
* pdflatex (in texlive-latex-base on Debian/Ubuntu)
* epstopdf (in texlive-extra-utils on Debian/Ubuntu)
* makeindex (in texlive-binaries on Debian/Ubuntu, or texlive-base-bin for older releases)
Note that pdflatex, epstopdf, gs, and makeindex must all currently be on your
path (as specified by the environment variable PATH), since doxygen will look
for them there.
For a recent version of Debian or Ubuntu, this command should install these
extra tools::
apt-get install ghostscript texlive-latex-base texlive-extra-utils texlive-binaries texlive-fonts-extra texlive-fonts-recommended texlive-latex-extra texlive-latex-recommended
On Mac OS X, if you're using macports you'll want the following:
* texlive (pdflatex during build)
* texlive-basic (for makeindex in configure)
* texlive-latex-extra (latex style)
Alternatively, you can install MacTeX from https://www.tug.org/mactex/ instead
of texlive, texlive-basic and texlive-latex-extra.
The homebrew texlive package only supports 32 bit systems, so even if you're
using homebrew, you'll probably want to install MacTeX from
https://www.tug.org/mactex/ instead.
Autotools versions
------------------
* autoconf 2.69 is used to generate snapshots and releases.
autoconf 2.64 is a hard minimum requirement.
autoconf 2.60 is required for docdir support and AC_TYPE_SSIZE_T.
autoconf 2.62 generates faster configure scripts and warns about unrecognised
options passed to configure.
autoconf 2.63 fixes a regression in AC_C_BIGENDIAN introduced in 2.62
(Omega uses this macro).
autoconf 2.64 generates smaller configure scripts by using shell functions.
* automake 1.15.1 is used to generate snapshots and releases.
automake 1.12.2 is a hard minimum requirement. This version fixes a
security issue (CVE-2012-3386) in the generated `make distcheck` rules.
automake 1.12 is needed to support using LOG_COMPILER to specify a testsuite
driver (used by xapian-bindings).
* libtool 2.4.6 is used to generate snapshots and releases.
libtool 2.2.8 is the current hard minimum requirement.
libtool 2.2 is required for us to be able to override link_all_deplibs_CXX
and sys_lib_dlsearch_path_spec in configure. It also fixes some
long-standing issues and is significantly faster.
Please tell us if you find that newer versions of any of these tools work or
fail to work.
There is a good GNU autotools tutorial at
<http://www.lrde.epita.fr/~adl/autotools.html>.
Building from git on Windows with MSVC
--------------------------------------
Building using MSVC is now supported by the autotools build system. You need
to install a set of Unix-like tools first - we recommended MSYS2:
https://www.msys2.org/
For details of how to specify MSVC to ``configure`` see the "INSTALL" document.
When building from git, by default you'll need some extra tools to generate
Unicode tables (Tcl) and build documentation (doxygen, help2man, sphinx-doc).
We don't currently have detailed advice on how to do this (if you can provide
some then please send a patch).
You can avoid needing Tcl by copying ``xapian-core/unicode/unicode-data.cc``
from another platform or a release which uses the same Unicode version. You
can avoid needing most of the documentation tools by running configure with
the ``--disable-documentation`` option.
Using a Vagrant-driven Ubuntu virtual machine
---------------------------------------------
Note: Vagrant support is experimental. Please report bugs in the
normal fashion, to https://trac.xapian.org/newticket, or ask for help
on the #xapian IRC channel on Freenode.
If you have Vagrant (http://www.vagrantup.com/, tested on version
1.5.2) and VirtualBox (https://www.virtualbox.org/, tested on version
4.3.10) installed, `vagrant up` will make a virtual machine suitable
for developing Xapian:
* Ubuntu 13.04 with all packages needed to build Xapian and its
documentation
* eatmydata (to speed up test runs) and valgrind (for debugging
memory allocations) both also installed
* source code from this checkout in /vagrant; edit it on your host
operating system and changes are reflected in the VM. The source
tree is bootstrapped automatically (ensuring that the right
versions of the build tools are available on the VM)
* build tree in /home/vagrant/build, configured to install into
/home/vagrant/install, with maintainer mode and documentation
both enabled
Setting up can take a long time, as it downloads a minimal base box
and then installs all the required packages; once this is done you
don't have to wait so long if you need to reprovision the VM. (Once
Ubuntu 14.04 is released the plan is to build our own base box with
these packages already installed, which should make the process much
faster.)
`vagrant ssh` will log you into the VM, and you can type `cd build &&
make` to build Xapian. `make check` will run the tests.
(As noted above, in maintainer mode most changes that require
reconfiguration will happen automatically. If you need to do it by
hand you can either run the configure command yourself, or you can run
`vagrant provision`, which also checks for any system package
updates.)
The VM has a single 64 bit virtual processor, with 384M of memory; it
takes about 8G of disk space once up and running.
Use of C++ Features
===================
* As of Xapian 1.3.3, a compiler with decent support for C++11 is required to
build Xapian. We currently aim to allow users to use a non-C++11 compiler
to build code which uses Xapian.
There are now several compilers with good C++11 support, but there are a
few shortfalls in commonly deployed versions of most of them. Often we can
work around this, and we should do where the effort is low compared to the
gain (so a compiler version which is widely used is more worth supporting
than one which is hardly used by anyone).
However, we shouldn't have to jump through hoops to cater for compilers where
their authors aren't putting in the effort to keep up with the language
standards.
Please avoid the following C++11 features for the time being:
* ``std::to_string()`` - this is completely missing on current versions of
mingw and cygwin - in the library, you can ``#include "str.h"`` and then
use the ``str()`` function instead for most cases. This is also usually
faster than ``std::to_string()``.
* C++ features we currently assume:
* We assume <sstream> is available. GCC < 2.95.3 didn't have it but GCC
2.95.3 includes a backported version. We aren't aware of any other
compilers still in use which lack it.
* Non-".h" versions of standard ISO C++ headers (e.g. ``#include <list>``
rather than ``#include <list.h>``). We aren't aware of any compiler still
in use which lacks these, and GCC 4.3 no longer has the old versions. If
there are any, we could add a directory full of forwarding headers to work
around this issue.
* Standard header ``<limits>`` (for ``numeric_limits<>``) - for GCC, this was
added in GCC 3.0.
* Standard header ``<streambuf>`` (GCC < 3.0 only has ``<streambuf.h>``).
* Working auto_ptr in header ``<memory>`` (some old version of some compiler
had a buggy implementation - the details are lost to history, but it may
have been GCC 2.95, or perhaps EGCS).
* RTTI (dynamic_cast<>, typeid, etc): Needing to use RTTI features in the
library most likely indicates a design flaw, and you should avoid use
of these features. Where necessary, you can use a technique similar to
Database::as_networkdatabase() to replace dynamic_cast<>.
* Exceptions: In hindsight, throwing exceptions in the library seems to have
been a poor design decision. GCC on Solaris can't cope with exceptions in
shared libraries (though it appears this may have been fixed in more recent
versions), and we've also had test failures on other platforms which only
occur with shared libraries - possibly with a similar cause. Exceptions can
also be a pain to handle elegantly in the bindings. We intend to investigate
modifying the library to return error codes internally, and then offering the
user the choice of exception throwing or error code returning API methods
(with the exception being thrown by an inlined wrapper in the externally
visible header files). With this in mind, please don't complicate the
internal handling of exceptions...
* "using namespace std;" and "using std::XXX;" - it's OK to use these in
applications, library code, and internal library headers. But in externally
visible headers (such as anything included by "#include <xapian.h>") you MUST
use explicit "std::" qualifiers - it's not acceptable to pull anything from
namespace std into the namespace of an application which uses Xapian.
* Use C++ style casts (static_cast<>, reinterpret_cast<>, and const_cast<>)
or constructor-syntax (e.g. ``double(value)``) in preference to C style
casts. The syntax of the C++ casts is ugly, but they do make the
intent much clearer which is definitely a good thing, and they avoid issues
such as casting away const when you only meant to cast the type of a pointer.
* std::pair<> with an STL class as one (or both) of the members can produce
very long symbols (over 4KB!) after name mangling - long enough to overflow
the size limits of some vendor compilers or toolchains (so this can affect
GCC if it is using the system ld or as). Even where the compiler works, the
symbol bloat in an unstripped build is probably best avoided, so it's
preferable to use a simple two member struct instead. The code is probably
more readable anyway, and easier to extend if more members are needed later.
* We try to avoid putting the full definition of virtual methods in header
files. This is because current compilers can't (as far as we know) inline
virtual methods, so putting the definition in the header file simply slows
down compilation (and, because method definitions often require further
header files to be included, this can result in many more files needing
recompilation after a change to a header file than is really necessary).
Just put the declaration in the header file, and put the definition in a .cc
file with the same basename.
Include ordering for source files
---------------------------------
To help us move towards a consistent ordering of #include lines in source
files, please follow the following policy when ordering them:
* #include <config.h> should be first, and use <> not "" (as recommended by the
autoconf manual). Always include config.h from C/C++ source files, but don't
include it from header files - the autoconf manual recommends that it should
be included first, so including it from headers is either redundant, or may
hide a missing config.h include in the source file the header was included
from (better to get an error in this case).
* The header corresponding to the source file should be next. This means that
compilation of the library ensures that each header with a corresponding
source file is "self supporting" (i.e. it implicitly or explicitly includes
all of the headers it requires).
* External xapian-core headers, alphabetically. When included from other
external headers, use <> to reduce problems with finding headers in the
user's source tree by mistake. In sources and internal headers, use "" (?) -
practically this makes no difference as we have -I for srcdir and builddir,
but <> suggests installed header files so "" seems more natural).
* Internal headers, alphabetically (using "").
* "Safe" versions of library headers (include these first to avoid issues if
other library headers include the ones we want to wrap). Use "" and order
alphabetically.
* Library headers, alphabetically.
* Standard C++ headers, alphabetically. Use the modern (no .h suffix) names.
C++ Portability Issues
======================
Web Resources
-------------
The "C++ Super-FAQ" covers many frequently asked C++ questions:
https://isocpp.org/faq
Header Portability Issues
-------------------------
<fcntl.h>:
----------
Don't directly '#include <fcntl.h>' - instead '#include "safefcntl.h"'.
The main reason for this is that when using certain compilers on certain
versions of Solaris, fcntl.h does '#define open open64'. Sadly this breaks C++
code which has methods called open (as we do). There's a cunning workaround
for this problem in common/safefcntl.h.
Also, safefcntl.h ensures the O_BINARY is defined (to 0 if not required) so
calls to open() and creat() can specify O_BINARY unconditionally for the
benefit of platforms which discriminate between text and binary files.
<windows.h>:
------------
Don't directly '#include <windows.h>' - instead '#include "safewindows.h"'
which reduces the bloat of header files included and prevents some of the
more egregious namespace pollution. It also defines any constants we need
which might be missing in older versions of the mingw headers.
<winsock2.h>:
-------------
Don't directly '#include <winsock2.h>' - instead '#include "safewinsock2.h"'.
This ensure that safewindows.h is included before <winsock2.h> to avoid
winsock2.h including windows.h without our namespace pollution reducing
workarounds.
<sys/select.h>:
---------------
Don't directly '#include <sys/select.h>' - instead '#include "safesysselect.h"'
which supports older UNIX platforms which predate POSIX 1003.1-2001 and works
around a problem on Solaris.
<sys/socket.h>:
---------------
Don't directly '#include <sys/socket.h>' - instead '#include "safesyssocket.h"'
which supports older UNIX platforms which predate POSIX 1003.1-2001 and works
on Windows too.
<sys/stat.h>:
-------------
Don't directly '#include <sys/stat.h>' - instead '#include "safesysstat.h"'
which under MSVC enables stat to work on files > 2GB, defines the missing
POSIX macros S_ISDIR and S_ISREG, pulls in <direct.h> for mkdir() (which is
provided by sys/stat.h under UNIX) and provides a compatibility wrapper for
mkdir() which takes 2 arguments (so code using mkdir can always just pass
two arguments).
<sys/wait.h>:
-------------
To get `WEXITSTATUS` or `WIFEXITED` defined, '#include "safesyswait.h"'.
Note that this won't provide `waitpid()`, etc on Microsoft Windows, since
these functions are only really useful to use when `fork()` is available.
<unistd.h>:
-----------
Don't directly '#include <unistd.h>' - instead '#include "safeunistd.h"'
- MSVC doesn't even HAVE unistd.h!
The various "safe" headers are maintained in xapian-core/common, but also used
by Omega. Currently bootstrap sorts out setting up a copy of this subdirectory
via a secondary git checkout.
Warning-Free Compilation
------------------------
Compiling without warnings on every platform is our goal, though it's not
always possible to achieve. For example, some GCC 3.x compilers produce the
occasional bogus warning (e.g. warning that a variable may be used
uninitialised, despite it being initialised at the point of declaration!)
You should consider configure-ing with:
./configure CXXFLAGS=-Werror
when doing development work on Xapian. This promotes warnings to errors,
which should ensure you at least don't introduce new warnings for the compiler
you're using.
If you configure with --enable-maintainer-mode, and are using GCC 4.1 or newer,
this is done for you automatically. This is intended to be an aid rather than
a form of automated punishment - it's all too easy to miss a new warning as
once a file is compiled, you don't see it unless you modify that file or one of
its dependencies.
With Intel's C++ compiler, --enable-maintainer-mode also enables -Werror.
If you know the equivalent of -Werror for other compilers, please add a note
here, or tell us so that we can add a note.
Miscellaneous Portability Issues
--------------------------------
Make sure that the last line of any source file ends with a linefeed character
since it's undefined behaviour if it doesn't (most compilers accept it, though
at least GCC gives a warning).
Branch Prediction Hints
=======================
For compilers which support ``__builtin_expect()`` (GCC >= 3.0 and some others)
you can provide manual hints to assist branch prediction. We've wrapped these
in macros which evaluate to just their argument for compilers which don't
support ``__builtin_expect()__``.
Within the xapian-core library code, you can mark the expressions in ``if`` and
``while`` statements as ``rare`` (if the condition is rarely true) or ``usual``
(if the condition is usually true).
For example::
if (rare(something_unusual())) deal_with_it();
while (usual(!end_condition()) keep_going();
It's easy to make incorrect assumptions about where hotspots are and which
branches are usually taken or not, so except for really obvious cases (such
as ``if (!consistency_check()) throw_exception();``) you should benchmark
that new ``rare`` and ``usual`` hints help rather than hinder before committing
them to the repository. It's also likely to be a waste of effort to add them
outside of areas of code which are executed very frequently.
Don't expect miracles - the first 15 uses added saved approximately 1%.
If you know how to implement the ``rare`` and ``usual`` macros for other
compilers, please let us know.
Configure Options
=================
Especially for a library, compile-time options aren't a good solution for
how to integrate a new feature. An increasingly large number of users install
pre-built binary packages rather than building from source, and unless the
package is capable of being split into modules, the packager has to choose a
set of compile-time options to use. And they'll tend to choose either the
standard ones, or perhaps a broader set to try to keep everyone happy. For a
library, similar issues occur when installing from source as well - the
sysadmin must choose the options which will keep all users happy.
Another problem with compile-time options is that it's hard to ensure that
a change doesn't break compilation under some combination of options without
actually building and running the test-suite on all combinations. The fewer
compile-time options, the more likely the code will compile with every
combination of them.
So please think carefully before adding more compile-time options. They're
probably OK for experimental features (but should go away once a feature is no
longer experimental). Options to instrument a build for special purposes
(debug, profiling, etc) are also acceptable. Disabling whole features probably
isn't (e.g. the --disable-backend-XXX options we already have are dubious,
though being able to disable the remote backend can be useful when trying to
get Xapian going on a platform).
Makefile Portability
====================
We don't want to force those building Xapian from the source distribution to
have to use GNU make. Requiring GNU make for "make dist" isn't such a problem
but it's probably better to use portable constructs everywhere to avoid
problems when people move or copy code between targets. If you do make use
of non-portable constructs where it's OK, add a comment noting the special
circumstances which justify doing so.
Here's an incomplete list of things to avoid:
* Don't use "$(RM)" - it's defined by GNU make, but using it actually harms
portability as other makes don't define it. Use plain "rm" instead.
* Don't use "%" pattern rules - these are GNU make specific. Use an
implicit rule (e.g. ".c.o:") if you can. Otherwise, write out each version
explicitly.
* Don't use "$<" except in implicit rules. This is an annoying restriction,
as using "$<" makes it much easier to make VPATH builds work. But it's only
portable in implicit rules. Tips for rewriting - if it's a source file,
write it as::
$(srcdir)/foo.ext
If it's a generated object file or similar, just write the name as is. The
tricky case is a generated file which isn't in git but is shipped in the
distribution tarball, as such a file could be in either the source or build
tree. Use this trick to make sure it's found whichever directory it's in::
`test -f foo.ext || echo '$(srcdir)/'`foo.ext
* Don't use "exit 0" to make a rule fail. Use "false" instead. BSD make
doesn't like "exit 0" in a rule.
* Don't use make conditionals. Automake offers conditionals which may be
of use, and these are implemented to work with any make. See the automake
manual for details, and a few caveats.
* The list of portable utilities is:
cat cmp cp diff echo egrep expr false grep install-info
ln ls mkdir mv pwd rm rmdir sed sleep sort tar test touch true
Note that versions of these (GNU versions in particular) support switches
which aren't portable - notably, "test -r" isn't portable; neither is
"cp -a". And note that "mkdir -p" isn't portable - the semantics vary.
The autoconf manual has some useful information about writing portable
shell code (most of it not specific to autoconf)::
https://www.gnu.org/software/autoconf/manual/autoconf.html#Portable-Shell
* Don't use "include" - it's not present in BSD make (at least some versions
have ".include" instead, but that doesn't really seem to help...) Automake
provides a configure-time include, which may provide a replacement for some
uses of "include".
* It appears that BSD make only supports VPATH for implicit rules (e.g.
".c.o:") - there's certainly a restriction there which is not present in GNU
make. We used to try to work around this, but now we use AM_MAINTAINER_MODE
to disable rules which are only needed by those developing Xapian (these were
the rules which caused problems). And we recommend those developing Xapian
use GNU make to avoid problems.
* Rules with multiple targets can cause problems for parallel builds. These
rules are really just a shorthand for multiple rules with the same
prerequisites and commands, and it is fine to use them in this way. However,
a common temptation is to use them when a single invocation of a command
generates multiple output files, by adding each of the output files as a
target. Eg, if a swig language module generates xapian_wrap.cc and
xapian_wrap.h, it is tempting to add a single rule something like::
# This rule has a problem
xapian_wrap.cc xapian_wrap.h: xapian.i
SWIG_commands
This can result in SWIG_commands being run twice, in parallel. If
SWIG_commands generates any temporary files, the two invocations can
interfere causing one of them to fail.
Instead of this rule, one solution is to pick one of the output files as a
primary target, and add a dependency for the second output file on the first
output file::
# This rule also has a problem
xapian_wrap.h: xapian_wrap.cc
xapian_wrap.cc: xapian.i
SWIG_commands
This ensures that make knows that only one invocation of SWIG_commands is
necessary, but could result in problems if the invocation of SWIG_commands
failed after creating xapian_wrap.cc, but before creating xapian_wrap.h.
Instead, we recommend creating an intermediate target::
# This rule works in most cases
xapian_wrap.cc xapian_wrap.h: xapian_wrap.stamp
xapian_wrap.stamp: xapian.i
SWIG_commands
touch $@
Because the intermediate target is only touched after the commands have
executed successfully, subsequent builds will always retry the commands if an
error occurs. Note that the intermediate target cannot be a "phony" target
because this would result in the commands being re-run for every build.