Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding support for input encoding and output decoding #50

Merged
merged 36 commits into from
Nov 13, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
c33654a
adding Base64IO context manager stream transformer
mattsb42-aws Oct 14, 2017
e892919
adding input decoding and output decoding arguments and handling
mattsb42-aws Oct 14, 2017
45471e7
disabling pylint duplicate-code message due to volume of false positives
mattsb42-aws Oct 14, 2017
64e795c
adding integration tests for source decoding and destination encoding
mattsb42-aws Oct 14, 2017
77771b1
updating docs to talk about --decode/--encode
mattsb42-aws Oct 15, 2017
f4a5baa
disabling seek on Base64IO - it creates too many hairy problems
mattsb42-aws Oct 15, 2017
a598937
adding filesize read when encrypting files to pass expected source le…
mattsb42-aws Oct 15, 2017
06c0399
renaming _b variable in Base64IO read/write to be more descriptive an…
mattsb42-aws Oct 19, 2017
6efc171
removing errant print statement
mattsb42-aws Oct 19, 2017
ea9b841
adding unit test to verify that Base64IO read behaves properly when o…
mattsb42-aws Oct 19, 2017
f13ef80
removing overly chatty debug log records
mattsb42-aws Oct 19, 2017
e68820c
properly handling closing wrapped stream and adding creation option t…
mattsb42-aws Oct 20, 2017
1c78874
adding Base64IO.writelines
mattsb42-aws Oct 20, 2017
8df803e
converting Base64IO from subclassing ObjectProxy to subclassing IOBas…
mattsb42-aws Oct 20, 2017
cedab23
updating io_handling._encoder to return stream directly if not using …
mattsb42-aws Oct 20, 2017
91be46d
condensing base64 integration tests into single parameterized test
mattsb42-aws Oct 20, 2017
92a234f
updating Base64IO readline and readlines interfaces to match IOBase
mattsb42-aws Oct 20, 2017
60738dc
mypy, linting, and docs tweaks
mattsb42-aws Oct 20, 2017
7057720
adding additional test cases for Base64IO read and write
mattsb42-aws Oct 20, 2017
4fe9937
adding Base64IO option to ignore whitespace and tests to verify corre…
mattsb42-aws Oct 20, 2017
41c079b
mypy and docs tweaks
mattsb42-aws Oct 20, 2017
0eda7de
relaxing pylint max-attributes to 10
mattsb42-aws Oct 20, 2017
c677ea2
adding tighter bounds for providing file source length when decoding …
mattsb42-aws Oct 20, 2017
1fb5453
removing potentially noisy debug log record in Base64IO
mattsb42-aws Oct 20, 2017
9866901
enabling Base64IO.write outside of a context manager
mattsb42-aws Oct 20, 2017
0a6f947
simplifying Base64IO to flush write buffer to __wrapped directly on c…
mattsb42-aws Nov 1, 2017
36d6616
fixing outdated docstring on io_handling._encoder
mattsb42-aws Nov 1, 2017
a7b6acd
moving call to wrapped interactive check outside of try block to avoi…
mattsb42-aws Nov 1, 2017
d7e1ceb
removing ability to turn off whitespace ignoring
mattsb42-aws Nov 2, 2017
7b985cd
refining whitespace unit tests
mattsb42-aws Nov 2, 2017
f2eab56
refining Base64IO unit test case generation
mattsb42-aws Nov 2, 2017
a64f7f0
updating linting rules: large numbers of locals in tests are ok
mattsb42-aws Nov 2, 2017
ee87928
refining documentation for Base64IO
mattsb42-aws Nov 11, 2017
2119369
adding proper b < 0 support for Base64IO.read()
mattsb42-aws Nov 11, 2017
97796c1
adding tests to verify that Base64IO is closed and unusable after a c…
mattsb42-aws Nov 11, 2017
b379b1a
flake8 formatting fix for separating __future__ imports
mattsb42-aws Nov 13, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,40 @@ references to other configuration files.

aws-crypto @my-encrypt -i $INPUT -o $OUTPUT


Encoding
--------
By default, ``aws-crypto`` will always output raw binary data and expect raw binary data
as input. However, there are some cases where you might not want this to be the case.

Sometimes this might be for convenience:

* Accepting ciphertext through stdin from a human.
* Presenting ciphertext through stdout to a human.

Sometimes it might be out of necessity:

* Saving ciphertext output to a shell variable.

* Most shells apply a system encoding to any data stored in a variable. As a result, this
often results in corrupted data if binary data is stored without additional encoding.

* Piping ciphertext in PowerShell.

* Similar to the above, all data passed through a PowerShell pipe is encoded using the
system encoding.

In order to address these scenarios, we provide two optional arguments:

* ``--decode`` : Base64-decode input before processing.
* ``--encode`` : Base64-encode output after processing.

These can be used independently or together, on any valid input or output.

Be aware, however, that if you target multiple files either through a path expansion or by
targetting a directory, the requested decoding/encoding will be applied to all files.


Execution
=========

Expand Down Expand Up @@ -381,6 +415,8 @@ Execution
-o OUTPUT, --output OUTPUT
Output file or directory for encrypt/decrypt
operation, or - for stdout.
--encode Base64-encode output after processing
--decode Base64-decode input before processing
-c ENCRYPTION_CONTEXT [ENCRYPTION_CONTEXT ...], --encryption-context ENCRYPTION_CONTEXT [ENCRYPTION_CONTEXT ...]
key-value pair encryption context values (encryption
only). Must a set of "key=value" pairs. ex: -c
Expand Down
1 change: 1 addition & 0 deletions doc/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Modules
aws_encryption_sdk_cli
aws_encryption_sdk_cli.internal
aws_encryption_sdk_cli.internal.arg_parsing
aws_encryption_sdk_cli.internal.encoding
aws_encryption_sdk_cli.internal.identifiers
aws_encryption_sdk_cli.internal.io_handling
aws_encryption_sdk_cli.internal.master_key_parsing
Expand Down
22 changes: 17 additions & 5 deletions src/aws_encryption_sdk_cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,9 @@ def process_cli_request(
recursive, # type: bool
interactive, # type: bool
no_overwrite, # type: bool
suffix=None # type: Optional[str]
suffix=None, # type: Optional[str]
decode_input=False, # type: Optional[bool]
encode_output=False # type: Optional[bool]
):
# type: (...) -> None
"""Maps the operation request to the appropriate function based on the type of input and output provided.
Expand All @@ -123,6 +125,8 @@ def process_cli_request(
:param bool interactive: Should prompt before overwriting existing files
:param bool no_overwrite: Should never overwrite existing files
:param str suffix: Suffix to append to output filename (optional)
:param bool decode_input: Should input be base64 decoded before operation (optional)
:param bool encode_output: Should output be base64 encoded after operation (optional)
"""
_catch_bad_destination_requests(destination)
_catch_bad_stdin_stdout_requests(source, destination)
Expand All @@ -134,7 +138,9 @@ def process_cli_request(
source=source,
destination=destination,
interactive=interactive,
no_overwrite=no_overwrite
no_overwrite=no_overwrite,
decode_input=decode_input,
encode_output=encode_output
)
return

Expand All @@ -154,7 +160,9 @@ def process_cli_request(
destination=_destination,
interactive=interactive,
no_overwrite=no_overwrite,
suffix=suffix
suffix=suffix,
decode_input=decode_input,
encode_output=encode_output
)

elif os.path.isfile(_source):
Expand All @@ -172,7 +180,9 @@ def process_cli_request(
source=_source,
destination=_destination,
interactive=interactive,
no_overwrite=no_overwrite
no_overwrite=no_overwrite,
decode_input=decode_input,
encode_output=encode_output
)


Expand Down Expand Up @@ -236,7 +246,9 @@ def cli(raw_args=None):
recursive=args.recursive,
interactive=args.interactive,
no_overwrite=args.no_overwrite,
suffix=args.suffix
suffix=args.suffix,
decode_input=args.decode,
encode_output=args.encode
)
return None
except AWSEncryptionSDKCLIError as error:
Expand Down
11 changes: 11 additions & 0 deletions src/aws_encryption_sdk_cli/internal/arg_parsing.py
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,17 @@ def _build_parser():
help='Output file or directory for encrypt/decrypt operation, or - for stdout.'
)

parser.add_argument(
'--encode',
action='store_true',
help='Base64-encode output after processing'
)
parser.add_argument(
'--decode',
action='store_true',
help='Base64-decode input before processing'
)

parser.add_argument(
'-c',
'--encryption-context',
Expand Down
Loading