Skip to content

Commit

Permalink
Introduce model level API (kv.*, stream.*, container.*, object.*) (#48)
Browse files Browse the repository at this point in the history
  • Loading branch information
@pavius
pavius authored Sep 14, 2020
1 parent 3ad4e8f commit 0edd8da
Show file tree
Hide file tree
Showing 15 changed files with 2,054 additions and 330 deletions.
86 changes: 44 additions & 42 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,48 +35,40 @@ The client supports a handful of low level APIs, each receiving different argume
* `output`: An object containing the parsed response (each API returns a different object)
* `headers` and `body`: The raw headers and body of the response

You would normally only access the `output` field unless an API was called that returns raw data like `get_object` (in which case `body` holds the response). Consult the reference for each API call to see how to handle its `Response` object. In the example below, we perform a simple request to get the containers available in our tenant, print the returned status code and containers:
You would normally only access the `output` field unless an API was called that returns raw data like `object.get` (in which case `body` holds the response). Consult the reference for each API call to see how to handle its `Response` object. In the example below, we perform a simple request to get the containers available in our tenant, print the returned status code and containers:

```python
# list the containers we can access
response = v3io_client.get_containers()
# list the contents of a container
response = v3io_client.container.list('users', '/')

# print the status code. outputs:
#
# Status code: 200
#
print(f'Status code: {response.status_code}')

# iterate over the containers and print them
#
# #0: bigdata
# #1: users
#
for container_idx, container in enumerate(response.output.containers):
print(f'#{container_idx}: {container.name}')
```

We can also get help information about the parameters this API call receives:
```python
help(v3io_client.get_containers)
help(v3io_client.container.list)
```

### Handling errors
By default, making a request will raise an exception if any non-200 status code is returned. We can override this default behavior in two ways.

The first is to simply never raise an exception and handle the status manually:
```python
# list the containers we can access and never raise an exception
response = v3io_client.get_containers(raise_for_status=v3io.dataplane.RaiseForStatus.never)
# list the contents of a container and never raise an exception
response = v3io_client.container.list('users', '/', raise_for_status=v3io.dataplane.RaiseForStatus.never)

# do anything we want with the status code
# some_logic(response.status_code)
```

The second is to indicate which status codes are acceptable:
```python
# list the containers and raise if the status code is not 200 or 204
response = v3io_client.get_containers(raise_for_status=[200, 204])
# list the contents of a container and raise if the status code is not 200 or 204
response = v3io_client.container.list('users', '/', raise_for_status=[200, 204])
```

### Creating batches
Expand All @@ -87,15 +79,15 @@ To get the highest possible throughput, we can send many requests towards the da
for idx in range(16):

# returns immediately
v3io_client.batch.put_object(container='bigdata',
v3io_client.batch.object.put(container='bigdata',
path=f'/object{idx}',
body=f'object-{idx}')

# wait for all writes to complete
v3io_client.batch.wait()
```

The looped `put_object` interface above will send 16 `put object` requests to the data layer in parallel. When `wait` is called, it will block until either all responses arrive (in which case it will return a `Responses` object, containing the `responses` of each call) or an error occurs - in which case an exception is thrown. You can pass `raise_for_status` to `wait`, and it behaves as explained above.
The looped `object.put` interface above will send 16 `put object` requests to the data layer in parallel. When `wait` is called, it will block until either all responses arrive (in which case it will return a `Responses` object, containing the `responses` of each call) or an error occurs - in which case an exception is thrown. You can pass `raise_for_status` to `wait`, and it behaves as explained above.

> Note: The `batch` object is stateful, so you can only create one batch at a time. However, you can create multiple parallel batches yourself through the client's `create_batch()` interface
Expand All @@ -107,12 +99,12 @@ Put data in an object, get it back and then delete the object:

```python
# put contents to some object
v3io_client.put_object(container='users',
v3io_client.object.put(container='users',
path='/my-object',
body='hello, there')

# get the object
response = v3io_client.get_object(container='users', path='/my-object')
response = v3io_client.object.get(container='users', path='/my-object')

# print the contents. outputs:
#
Expand All @@ -121,7 +113,7 @@ response = v3io_client.get_object(container='users', path='/my-object')
print(response.body.decode('utf-8'))

# delete the object
v3io_client.delete_object(container='users', path='/my-object')
v3io_client.object.delete(container='users', path='/my-object')
```

### Accessing key-values (NoSQL)
Expand All @@ -136,20 +128,22 @@ items = {

# add the records to the table
for item_key, item_attributes in items.items():
v3io_client.put_item(container='users', path='/bobs-burgers/' + item_key, attributes=item_attributes)
v3io_client.kv.put(container='users', table_path='/bobs-burgers', key=item_key, attributes=item_attributes)

# adds two fields (height, quip) to the louise record
v3io_client.update_item(container='users',
path='/bobs-burgers/louise',
attributes={
'height': 130,
'quip': 'i can smell fear on you'
})
v3io_client.kv.update(container='users',
table_path='/bobs-burgers',
key='louise',
attributes={
'height': 130,
'quip': 'i can smell fear on you'
})

# get a record by key, specifying specific arguments
response = v3io_client.get_item(container='users',
path='/bobs-burgers/louise',
attribute_names=['__size', 'age', 'quip', 'height'])
response = v3io_client.kv.get(container='users',
table_path='/bobs-burgers',
key='louise',
attribute_names=['__size', 'age', 'quip', 'height'])


# print the item from the response. outputs:
Expand All @@ -159,10 +153,10 @@ response = v3io_client.get_item(container='users',
print(response.output.item)

# create a query, and use an items cursor to iterate the results
items_cursor = v3io_client.new_items_cursor(container='users',
path='/bobs-burgers/',
attribute_names=['age', 'feature'],
filter_expression='age > 15')
items_cursor = v3io_client.kv.new_cursor(container='users',
table_path='/bobs-burgers',
attribute_names=['age', 'feature'],
filter_expression='age > 15')

# print the output
for item in items_cursor.all():
Expand All @@ -174,9 +168,9 @@ Creates a stream with several partitions, writes records to it, reads the record

```python
# create a stream w/8 shards
v3io_client.create_stream(container='users',
path='/my-test-stream',
shard_count=8)
v3io_client.stream.create(container='users',
stream_path='/my-test-stream',
shard_count=8)

# write 4 records - 3 with explicitly specifying the shard and 1 using hashing
records = [
Expand All @@ -186,13 +180,21 @@ records = [
{'data': 'some shard record #1'}
]

v3io_client.put_records(container='users', path='/my-test-stream', records=records)
v3io_client.stream.put_records(container='users',
stream_path='/my-test-stream',
records=records)

# seek to the beginning of the shard of #1 so we know where to read from
response = v3io_client.seek_shard(container='users', path='/my-test-stream/1', seek_type='EARLIEST')
response = v3io_client.stream.seek(container='users',
stream_path='/my-test-stream',
shard_id=1,
seek_type='EARLIEST')

# get records from the shard (should receive 2)
response = v3io_client.get_records(container='users', path='/my-test-stream/1', location=response.output.location)
response = v3io_client.stream.get_records(container='users',
stream_path='/my-test-stream',
shard_id=1,
location=response.output.location)

# print the records. outputs:
#
Expand All @@ -203,7 +205,7 @@ for record in response.output.records:
print(record.data.decode('utf-8'))

# delete the stream
v3io_client.delete_stream(container='users', path='/my-test-stream')
v3io_client.stream.delete(container='users', stream_path='/my-test-stream')
```

# Controlplane client
Expand Down
Loading

0 comments on commit 0edd8da

Please sign in to comment.