Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(toolkit): Produce an Asset manifest document #951

Closed
wants to merge 1 commit into from
Closed

feat(toolkit): Produce an Asset manifest document #951

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 66 additions & 1 deletion packages/aws-cdk/bin/cdk.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import 'source-map-support/register';
import cxapi = require('@aws-cdk/cx-api');
import childProcess = require('child_process');
import colors = require('colors/safe');
import crypto = require('crypto');
import fs = require('fs-extra');
import YAML = require('js-yaml');
import minimatch = require('minimatch');
Expand Down Expand Up @@ -347,14 +348,78 @@ async function initCommandLine() {

fs.mkdirpSync(outputDir);

const extension = json ? 'json' : 'yaml';
const hashCache: { [path: string]: string } = {};
for (const stack of stacks) {
const finalName = renames.finalName(stack.name);
const fileName = `${outputDir}/${finalName}.template.${json ? 'json' : 'yaml'}`;
const fileName = path.join(outputDir, `${finalName}.template.${extension}`);
highlight(fileName);
await fs.writeFile(fileName, toJsonOrYaml(stack.template));
const manifestName = path.join(outputDir, `${finalName}.manifest.${extension}`);
await fs.writeFile(manifestName, toJsonOrYaml(await prepareManifest(stack, outputDir, hashCache)));
}

return undefined; // Nothing to print

/**
* Prepares an asset manifest document, staging assets in a specified directory.
* @param stack the stack for which assets are to be manifested.
* @param stackName the name of the stack (accounting for renames)
* @param dir the directory under which to stage the assets (an assets sub-directory will be used).
* @param cache a cache for asset files hashes, improving performance when the same assets are used across multiple stacks
*/
async function prepareManifest(stack: cxapi.SynthesizedStack, dir: string, cache: { [path: string]: string }) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting... My intuition was that all the assets and asset manifest should be emitted to the working directory by the App and not by the toolkit. Now that the cx protocol enables this, I wonder if it doesn't make more sense to move all this logic to App so that we'll have less magic in the toolkit.

What do you think?

I am okay if you feel this can be done in two steps, but it would be nice to have some agreement on where we want this to go, so we'll not take too many redundant steps.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That actually sounds like it'd be better. I had started this before you made the changes to the App API, and I missed the fact I could refactor this away to the other side of the wall.

const manifest = { version: '1', assets: {} as any };
for (const key of Object.keys(stack.metadata)) {
const entries = stack.metadata[key].filter(entry => entry.type === cxapi.ASSET_METADATA);
for (const entry of entries) {
const asset = entry.data! as cxapi.AssetMetadataEntry;
const basePath = path.join('assets', await hash(asset.path), path.basename(asset.path));
const fullPath = path.join(dir, basePath);
if (!await fs.pathExists(fullPath)) {
await fs.copy(asset.path, fullPath);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use symlinks to make things faster, or would this break CI/CD?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CodeBuild documentation doesn't really state anything with respects to symlinks behavior in output artifacts, so I'd rather avoid assuming they are replaced by the referent file... But on the other hand this should be relatively easy to test out and verify. That'd also save disk space on a user machine, which is not a bad thing.

}
manifest.assets[asset.id] = {
constructPath: key,
packaging: asset.packaging,
parameters: {
s3bucket: asset.s3BucketParameter,
s3key: asset.s3KeyParameter,
},
path: basePath,
};
}
}
return manifest;

function hash(filePath: string): Promise<string> {
return new Promise<string>(async (ok, ko) => {
if (filePath in cache) { return ok(cache[filePath]); }
debug(`Hashing asset file ${filePath}`);
const stat = await fs.stat(filePath);
const digest = crypto.createHash('sha256');
digest.once('readable', () => {
const binaryHash = digest.read() as Buffer;
if (binaryHash) {
const strHash = binaryHash.toString('hex');
debug(`Hash of asset file ${filePath}: ${strHash}`);
return ok(cache[filePath] = strHash);
}
ko(new Error('No hash was generated!'));
});
if (stat.isDirectory()) {
for (const element of (await fs.readdir(filePath)).sort()) {
digest.write(await hash(path.join(filePath, element)) + '\0');
}
digest.end();
} else {
const reader = fs.createReadStream(filePath);
digest.write(path.basename(filePath) + '\0');
reader.pipe(digest);
}
});
}
}
}

/**
Expand Down