Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce the correct order of responses. Handle non-idempotent requests. #660

Merged
merged 67 commits into from
Mar 3, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
193d362
Send responses in correct order - first working version. (#419)
keshonok Sep 20, 2016
363b52b
Small refactoring of tfw_sg_for_each_srv().
keshonok Oct 5, 2016
c4b0d07
Introduce 'connect_retries' directive. (#419)
keshonok Oct 5, 2016
c13332f
Multiple changes. Remove the HOLD flag. (#419)
keshonok Oct 7, 2016
31f0f7d
Simplify the code by removing unnecessary logic. (#419)
keshonok Oct 12, 2016
e032033
Provide the count of non-idempotent reqs in a server conn. (#419)
keshonok Oct 13, 2016
b6f4f2a
RR scheduler: consider the presence of non-idempotent reqs. (#419)
keshonok Oct 14, 2016
5cda301
A small fix after rebase on top of master branch. (#419)
keshonok Oct 14, 2016
af18f8e
Few cosmetic changes for style. (#419)
keshonok Oct 17, 2016
c4c79ea
Re-sending of requests within a restored server connection. (#419)
keshonok Oct 20, 2016
ed89a57
Bug fixes related to the previous commit. (#419)
keshonok Oct 21, 2016
2dc1f91
Re-schedule requests from a dead server connection. (#419)
keshonok Oct 21, 2016
f245c8f
Minor bug fixes. (#419)
keshonok Oct 24, 2016
52c3593
Multiple changes, fixes, tune-ups, and options. (#419)
keshonok Oct 27, 2016
b633cd3
Bug fixes, refactorings, etc. (#419)
keshonok Nov 2, 2016
160a954
Implement a way to define non-idempotent requests in config. (#419)
keshonok Dec 4, 2016
6bd9011
Rework the handling of non-idempotent req definitions in config. (#419)
keshonok Dec 5, 2016
d113a32
Fix a number of rebase glitches. (#419)
keshonok Dec 10, 2016
7479c7d
Fix sticky cookie unit tests. (#419)
keshonok Dec 10, 2016
8b4b526
Fix schedulers unit tests. (#419)
keshonok Dec 10, 2016
8c9afa4
Port to 4.8.15
krizhanovsky Dec 27, 2016
dbe3adf
More careful handling of various data types in vhost.[ch]
keshonok Jan 2, 2017
c8dccc0
Better handling of the number of reconnect attempts.
keshonok Jan 2, 2017
2736aca
Consider that srv_conn->qsize is always modified under a lock.
keshonok Jan 2, 2017
10d818e
__init_{req/resp}_ss_flags() to tfw_http_{req/resp}_init_ss_flags().
keshonok Jan 2, 2017
7cce5a0
Remove 'resp' from a union in TfwHttpReq{}.
keshonok Jan 4, 2017
6ba9b0d
Separate message's SKB data set up from message allocation.
keshonok Jan 7, 2017
6a873a8
Move fwd_list from TfwMsg{} to TfwHttpReq{}.
keshonok Jan 7, 2017
cc0829c
Fix an bug where sets of ordered responses may be mixed up.
keshonok Jan 9, 2017
c19388d
Eliminate TfwConnection{}->msg_resent.
keshonok Jan 11, 2017
5470e37
Hold the client connection while sending responses.
keshonok Jan 11, 2017
cb8760f
Fix handling of non-idempotent requests in case of connection repair.
keshonok Jan 12, 2017
2828644
Replace spaces with tabs in vhost.c
keshonok Jan 12, 2017
84542c1
Return 502 or 504 error message instead of 404.
keshonok Jan 13, 2017
ed9b2f4
Don't call connection repair if the connection is not restricted.
keshonok Jan 16, 2017
fac5cea
Separate APM timestamps and eviction (age) timestamp.
keshonok Jan 17, 2017
9fd20bb
Better code unification.
keshonok Jan 18, 2017
676f205
Rename @srv->qjtimeout to srv->qjtmo_max for uniformity.
keshonok Jan 18, 2017
5fe74f5
Print current queue size for each server's connection in procfs stats.
keshonok Jan 18, 2017
4787c63
Count responses that were not forwarded to a client in perf stats.
keshonok Jan 19, 2017
d6fdddf
Call the repair correctly when a server connection is dead.
keshonok Jan 21, 2017
1f96814
Correctly handle requests in a closing client connection.
keshonok Jan 22, 2017
a8fa850
Rename *_max vars and structure members for unification.
keshonok Jan 22, 2017
84e7cc0
Move flags and max_* members from TfwServer{} to TfwSrvGroup{}.
keshonok Jan 22, 2017
1dddfb5
Better config processing to align with ALB branch.
keshonok Jan 23, 2017
98634d3
Minor fixes prompted by the code review.
keshonok Jan 23, 2017
ab8716a
Release server connection's resources when Tempesta is shut down.
keshonok Jan 23, 2017
b2f0343
Fix several oversights in cases of sending an error response.
keshonok Jan 24, 2017
3f25a47
Split TfwConnection{} into TfwCliConnection{} and TfwSrvConnection{}.
keshonok Jan 27, 2017
3984c9b
Rename TfwConnection{} to TfwConn{}.
keshonok Jan 27, 2017
7dc8fe1
Minor APM-related changes, better/clearer comments.
keshonok Jan 30, 2017
7f867d2
Describe queues in client and server connections.
keshonok Jan 30, 2017
4131412
Fix var names in debug statements. Also, fix TfwSrvConn{} init.
keshonok Feb 1, 2017
1510439
Release all server connection's resources on shutdown.
keshonok Feb 13, 2017
7d6ea58
Merge branch 'master' into ab-msg-seq
keshonok Feb 14, 2017
fb3b579
Add the description of all new options to README.md.
keshonok Feb 14, 2017
328ff60
Evict timed out requests from an inactive server connection.
keshonok Feb 15, 2017
77d1d13
Correctly release server connection's resources at STOP time.
keshonok Feb 17, 2017
9db23ed
Address latest code review comments - the first batch of changes.
keshonok Feb 20, 2017
6478c70
Add TODO items for further improvements of the architecture.
keshonok Feb 20, 2017
bcfcf0f
Retrofit each HTTP error response with a reason message.
keshonok Feb 22, 2017
4365346
Better/stricter implementation of the request eviction procedure.
keshonok Feb 27, 2017
69a5a2f
Add a TODO comment; retract the description of zero forward_retries.
keshonok Feb 27, 2017
6969c76
Remove erroneous response free() in popreq().
keshonok Feb 27, 2017
50ade60
Add the number of schedulable connections to perfstat.
keshonok Mar 1, 2017
8273de0
Minor changes according to the latest code review.
keshonok Mar 2, 2017
5d5983f
Merge branch 'master' into ab-msg-seq
keshonok Mar 2, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 135 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ ssl_certificate_key /path/to/tfw-root.key;

Also, `proto=https` option is needed for the `listen` directive.

#### Self-signed certificate genration
#### Self-signed certificate generation

In case of using a self-signed certificate with Tempesta, it's
convenient to use OpenSSL to generate a key and a certificate. The
Expand Down Expand Up @@ -301,6 +301,31 @@ just one example:
curl -X PURGE http://192.168.10.10/
```

#### Non-Idempotent Requests

The consideration of whether a request is considered non-idempotent may
depend on specific application, server, and/or service. A special directive
allows the definition of a request that will be considered non-idempotent:
```
nonidempotent <METHOD> <OP> <ARG>;
```
`METHOD` is one of supported HTTP methods, such as GET, HEAD, POST, etc.
`OP` is a string matching operator, such as `eq`, `prefix`, etc.
`ARG` is an argument for `OP`, such as `/foo/bar.html`, `example.com`, etc.

One or more of this directive may be specified. The directives apply to one
or more locations as defined below in the [Locations](#Locations) section.

If this directive is not specified, then a non-idempotent request in defined
as a request that has an unsafe method.

Below are examples of this directive:
```
nonidempotent GET prefix "/users/";
nonidempotent POST prefix "/users/";
nonidempotent GET suffix "/data";
```

### Locations

Location is a way of grouping certain directives that are applied only
Expand All @@ -321,9 +346,10 @@ location <OP> "<string>" {
Multiple locations may be defined. Location directives are processed
strictly in the order they are defined in the configuration file.

Only caching policy directives may currently be grouped by the location
directive. Caching policy directives defined outside of any specific
location are considered the default policy for all locations.
Only caching policy directives and the `nonidempotent` directive may
currently be grouped by the location directive. The directives defined
outside of any specific location are considered the default policy for
all locations.

When locations are defined in the configuration, the URL of each request
is matched against strings specified in the location directives and using
Expand Down Expand Up @@ -351,6 +377,7 @@ location prefix "/society/" {
cache_bypass prefix "/society/breaking_news/";
cache_fulfill suffix ".jpg" ".png";
cache_fulfill suffix ".css";
nonidempotent GET prefix "/society/users/";
}
```

Expand All @@ -367,14 +394,72 @@ server <IPADDR>[:<PORT>] [conns_n=<N>];
IPv6 address must be enclosed in square brackets (e.g. "[::0]" but not "::0").
`PORT` defaults to 80 if not specified.
`conns_n=<N>` is the number of parallel connections to the server.
`N` defaults to 4 if not specified.
`N` defaults to 32 if not specified.

Multiple back end servers may be defined. For example:
```
server 10.1.0.1;
server [fc00::1]:80;
```

if a connection with a server is terminated for any reason, an effort is made
to restore the connection. Sometimes the effort is futile. The directive
`server_connect_retries` sets the maximum number of re-connect attempts after
which the server connection is considered dead. It is defined as follows:
```
server_connect_retries <N>;
```
If this directive is not defined, then the number of re-connect attempts
defaults to 10. The value of zero specified for `N` means unlimited number
of attempts.

This is an important directive which controls how Tempesta deals with
outstanding requests in a failed connection. If the connection is restored
within the specified number of attempts, then all outstanding requests are
re-forwarded to the server. However if it's not restored, then the server
connection is considered dead, and all outstanding requests are re-scheduled
to other servers and/or connections.

If a server connection fails intermittenly, then requests may sit in the
connection's forwarding queue for some time. The following directives set
certain allowed limits before these requests are considered failed:
```
server_forward_retries <N>;
server_forward_timeout <N>;
```

`server_forward_retries` sets the maximum number of attempts to re-forward
a request to a server. If not defined, the default number of attempts is 5.
The value of zero specified for `N` means unlimited number of attempts.

`server_forward_timeout` set the maximum time frame in seconds within which
a request may still be forwarded. If not defined, the default time frame
is 60 seconds. The value of zero specified for `N` means unlimited timeout.

When one or both of these limits is exceeded for a request, the request is
evicted and an error is returned to a client.

Note that while requests in a connection are re-forwarded or re-scheduled,
that connection is not schedulable, which means it's not available to
schedulers for new incoming requests.

When re-forwarding or re-scheduling requests in a failed server connection,
a special consideration is given to non-idempotent requests. Usually
a non-idempotent request is not re-forwarded or re-scheduled. That may be
changed with the following directive that doesn't have arguments:
```
server_retry_nonidempotent;
```

Each server connection has a queue of forwarded requests. The size of the
queue is limited with `server_queue_size` directive as follows:
```
server_queue_size <N>;
```
Each connection to the server has the same limit on the queue size set
with this directive. If not specified, the queue size is set to 1000.


#### Server Groups

Back end servers can be grouped together into a single unit for the purpose of
Expand All @@ -383,33 +468,39 @@ The load is distributed evenly among servers within a group.
If a server goes offline, other servers in a group take the load.
The full syntax is as follows:
```
srv_group <NAME> [sched=<SCHED_NAME>] {
srv_group <NAME> {
server <IPADDR>[:<PORT>] [conns_n=<N>];
...
}
```
`NAME` is a unique identifier of the group that may be used to refer to it
later.
`SCHED_NAME` is the name of scheduler module that distributes load among
servers within the group. Default scheduler is used if `sched` parameter is
not specified.

Servers that are defined outside of any group implicitly form a special group
called `default`.

All server-related directives listed in [Servers](#Servers) section above
are applicable for definition for a server group. Also, a scheduler may be
speficied for a group.

Below is an example of server group definition:
```
srv_group static_storage sched=hash {
srv_group static_storage {
sched hash;
server 10.10.0.1:8080;
server 10.10.0.2:8080;
server [fc00::3]:8081 conns_n=1;
server_queue_size 500;
server_forward_timeout 30;
server_connect_retries 15;
}
```

#### Schedulers

Scheduler is used to distribute load among known servers. The syntax is as
follows:
Scheduler is used to distribute load among servers within a group. The group
can be either explicit, defined with `srv_group` directive, or implicit.
The syntax is as follows:
```
sched <SCHED_NAME>;
```
Expand All @@ -423,14 +514,22 @@ scheduler.
Requests are distributed uniformly, and requests with the same URI/Host are
always sent to the same server.

If no scheduler is defined, then scheduler defaults to `round-robin`.
The round-robin scheduler is the fastest scheduler. However, the presence
of a non-idempotent request in a connection means that subsequent requests
may not be sent out until a response is received to the non-idempotent
request. With that in mind, an attempt is made to put new requests to
connections that don't currently have non-idempotent requests. If all
connections have a non-idempotent request in them, then such a connection
is used as there's no other choice.

Only one `sched` directive is allowed per explicit or implicit group.
A scheduler defined for the implicit group becomes the scheduler for an
explicit group defined with `srv_group` directive if the explicit group
is missing the `sched` directive.

The defined scheduler affects all server definitions that are missing a
scheduler definition. If `srv_group` is missing a scheduler definition,
and there is a scheduler defined, then that scheduler is set for the group.
If no scheduler is defined for a group, then scheduler defaults
to `round-robin`.

Multiple `sched` directives may be defined in the configuration file.
Each directive affects server groups that follow it.

#### HTTP Scheduler

Expand Down Expand Up @@ -713,22 +812,31 @@ and running. Below is an example of the command to show the statistics,
and the output:
```
$ cat /proc/tempesta/perfstat
Client messages received : 450
Client messages forwarded : 450
SS pfl hits : 5836412
SS pfl misses : 5836412
Cache hits : 0
Cache misses : 0
Client messages received : 2918206
Client messages forwarded : 2918206
Client messages served from cache : 0
Client messages parsing errors : 0
Client messages filtered out : 0
Client messages other errors : 0
Client connections total : 30
Clients online : 0
Client connection attempts : 2048
Client established connections : 2048
Client connections active : 0
Client RX bytes : 47700
Server messages received : 447
Server messages forwarded : 447
Client RX bytes : 309329836
Server messages received : 2918206
Server messages forwarded : 2918206
Server messages parsing errors : 0
Server messages filtered out : 0
Server messages other errors : 0
Server connections total : 2220
Server connections active : 4
Server RX bytes : 153145
Server connection attempts : 8896
Server established connections : 8896
Server connections active : 32
Server connections schedulable : 32
Server RX bytes : 11494813434
```

Also, there's Application Performance Monitoring statistics. These stats show
Expand Down
Loading