Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mir-opt] Optimize calls to CopyNonOverlapping #83785

Closed
wants to merge 1 commit into from
Closed

[mir-opt] Optimize calls to CopyNonOverlapping #83785

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 34 additions & 10 deletions compiler/rustc_mir/src/transform/lower_intrinsics.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ pub struct LowerIntrinsics;

impl<'tcx> MirPass<'tcx> for LowerIntrinsics {
fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
let param_env = tcx.param_env(body.source.def_id()).with_reveal_all_normalized(tcx);
let (basic_blocks, local_decls) = body.basic_blocks_and_local_decls_mut();
for block in basic_blocks {
let terminator = block.terminator.as_mut().unwrap();
Expand Down Expand Up @@ -43,22 +44,36 @@ impl<'tcx> MirPass<'tcx> for LowerIntrinsics {
sym::copy_nonoverlapping => {
let target = destination.unwrap().1;
let mut args = args.drain(..);
block.statements.push(Statement {
source_info: terminator.source_info,
kind: StatementKind::CopyNonOverlapping(
box rustc_middle::mir::CopyNonOverlapping {
src: args.next().unwrap(),
dst: args.next().unwrap(),
count: args.next().unwrap(),
},
),
});
let src = args.next().unwrap();
let dst = args.next().unwrap();
let count = args.next().unwrap();

assert_eq!(
args.next(),
None,
"Extra argument for copy_non_overlapping intrinsic"
);
drop(args);

let stmt = if let (Some(1), Some(src), Some(dst)) = (
eval_operand_to_usize(tcx, param_env, &count),
src.place(),
dst.place(),
) {
StatementKind::Assign(box (
tcx.mk_place_deref(dst),
Rvalue::Use(Operand::Copy(tcx.mk_place_deref(src))),
))
Comment on lines +63 to +66
Copy link
Member

@bjorn3 bjorn3 Oct 20, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was an interesting conversation on zulip around the semantics of copy_nonoverlapping. If it does an "untyped copy", lowering it to a regular assignment is not a sound optimization, so this probably needs to wait on a decision if copy_nonoverlapping does a typed or untyped copy. https://rust-lang.zulipchat.com/#narrow/stream/136281-t-lang.2Fwg-unsafe-code-guidelines/topic/copy_nonoverlapping.20and.20provenance/near/258410065

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, thanks for digging this up, I knew this proposal was living somewhere... (and I think I was advocating for it, not realizing what equating copy_nonoverlapping and assignment would entail).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If copy_nonoverlapping does an untyped copy does that mean that the implementation of ptr::read needs to change so that it does a typed read?

Copy link
Member

@RalfJung RalfJung Oct 20, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is correct to replace a typed by an untyped copy, just not the other way around.

So, read purporting to be typed but actually doing an untyped operation under the hood is fine. (Typedness is force on it anyway because the result is returned by-value at type T.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but that may pessimize the read. See #73258

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy_nonoverlapping does an untyped copy as per #97712, so this lowering is off the table now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we close it then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be made an assignment at type MaybeUninit<T>.

} else {
StatementKind::CopyNonOverlapping(
box rustc_middle::mir::CopyNonOverlapping { src, dst, count },
)
};

block
.statements
.push(Statement { source_info: terminator.source_info, kind: stmt });

terminator.kind = TerminatorKind::Goto { target };
}
sym::wrapping_add | sym::wrapping_sub | sym::wrapping_mul => {
Expand Down Expand Up @@ -126,6 +141,15 @@ impl<'tcx> MirPass<'tcx> for LowerIntrinsics {
}
}

fn eval_operand_to_usize(
tcx: TyCtxt<'tcx>,
param_env: rustc_middle::ty::ParamEnv<'tcx>,
operand: &Operand<'tcx>,
) -> Option<u64> {
let constant = operand.constant()?;
Some(constant.literal.try_eval_usize(tcx, param_env)?)
}

fn resolve_rust_intrinsic(
tcx: TyCtxt<'tcx>,
func_ty: Ty<'tcx>,
Expand Down
111 changes: 111 additions & 0 deletions src/test/mir-opt/lower_intrinsics.copy_nonoverlapping.LowerIntrinsics.diff
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
- // MIR for `copy_nonoverlapping` before LowerIntrinsics
+ // MIR for `copy_nonoverlapping` after LowerIntrinsics

fn copy_nonoverlapping() -> () {
let mut _0: (); // return place in scope 0 at $DIR/lower_intrinsics.rs:76:30: 76:30
let _1: i32; // in scope 0 at $DIR/lower_intrinsics.rs:77:9: 77:12
let _4: (); // in scope 0 at $DIR/lower_intrinsics.rs:82:9: 82:88
let mut _5: *const i32; // in scope 0 at $DIR/lower_intrinsics.rs:82:46: 82:62
let mut _6: *const i32; // in scope 0 at $DIR/lower_intrinsics.rs:82:46: 82:62
let _7: &i32; // in scope 0 at $DIR/lower_intrinsics.rs:82:46: 82:50
let mut _8: *mut i32; // in scope 0 at $DIR/lower_intrinsics.rs:82:64: 82:84
let mut _9: *mut i32; // in scope 0 at $DIR/lower_intrinsics.rs:82:64: 82:84
let mut _10: &mut i32; // in scope 0 at $DIR/lower_intrinsics.rs:82:64: 82:74
let _11: (); // in scope 0 at $DIR/lower_intrinsics.rs:83:9: 83:88
let mut _12: *const i32; // in scope 0 at $DIR/lower_intrinsics.rs:83:46: 83:62
let mut _13: *const i32; // in scope 0 at $DIR/lower_intrinsics.rs:83:46: 83:62
let _14: &i32; // in scope 0 at $DIR/lower_intrinsics.rs:83:46: 83:50
let mut _15: *mut i32; // in scope 0 at $DIR/lower_intrinsics.rs:83:64: 83:84
let mut _16: *mut i32; // in scope 0 at $DIR/lower_intrinsics.rs:83:64: 83:84
let mut _17: &mut i32; // in scope 0 at $DIR/lower_intrinsics.rs:83:64: 83:74
scope 1 {
debug src => _1; // in scope 1 at $DIR/lower_intrinsics.rs:77:9: 77:12
let mut _2: i32; // in scope 1 at $DIR/lower_intrinsics.rs:78:9: 78:18
scope 2 {
debug dst_1 => _2; // in scope 2 at $DIR/lower_intrinsics.rs:78:9: 78:18
let mut _3: i32; // in scope 2 at $DIR/lower_intrinsics.rs:79:9: 79:18
scope 3 {
debug dst_2 => _3; // in scope 3 at $DIR/lower_intrinsics.rs:79:9: 79:18
scope 4 {
}
}
}
}

bb0: {
StorageLive(_1); // scope 0 at $DIR/lower_intrinsics.rs:77:9: 77:12
_1 = const 42_i32; // scope 0 at $DIR/lower_intrinsics.rs:77:15: 77:17
StorageLive(_2); // scope 1 at $DIR/lower_intrinsics.rs:78:9: 78:18
_2 = const 1_i32; // scope 1 at $DIR/lower_intrinsics.rs:78:21: 78:22
StorageLive(_3); // scope 2 at $DIR/lower_intrinsics.rs:79:9: 79:18
_3 = const 2_i32; // scope 2 at $DIR/lower_intrinsics.rs:79:21: 79:22
StorageLive(_4); // scope 4 at $DIR/lower_intrinsics.rs:82:9: 82:88
StorageLive(_5); // scope 4 at $DIR/lower_intrinsics.rs:82:46: 82:62
StorageLive(_6); // scope 4 at $DIR/lower_intrinsics.rs:82:46: 82:62
StorageLive(_7); // scope 4 at $DIR/lower_intrinsics.rs:82:46: 82:50
_7 = &_1; // scope 4 at $DIR/lower_intrinsics.rs:82:46: 82:50
_6 = &raw const (*_7); // scope 4 at $DIR/lower_intrinsics.rs:82:46: 82:50
_5 = _6; // scope 4 at $DIR/lower_intrinsics.rs:82:46: 82:62
StorageLive(_8); // scope 4 at $DIR/lower_intrinsics.rs:82:64: 82:84
StorageLive(_9); // scope 4 at $DIR/lower_intrinsics.rs:82:64: 82:84
StorageLive(_10); // scope 4 at $DIR/lower_intrinsics.rs:82:64: 82:74
_10 = &mut _2; // scope 4 at $DIR/lower_intrinsics.rs:82:64: 82:74
_9 = &raw mut (*_10); // scope 4 at $DIR/lower_intrinsics.rs:82:64: 82:74
_8 = _9; // scope 4 at $DIR/lower_intrinsics.rs:82:64: 82:84
- _4 = std::intrinsics::copy_nonoverlapping::<i32>(move _5, move _8, const 1_usize) -> bb1; // scope 4 at $DIR/lower_intrinsics.rs:82:9: 82:88
- // mir::Constant
- // + span: $DIR/lower_intrinsics.rs:82:9: 82:45
- // + literal: Const { ty: unsafe extern "rust-intrinsic" fn(*const i32, *mut i32, usize) {std::intrinsics::copy_nonoverlapping::<i32>}, val: Value(Scalar(<ZST>)) }
+ (*_8) = (*_5); // scope 4 at $DIR/lower_intrinsics.rs:82:9: 82:88
+ goto -> bb1; // scope 4 at $DIR/lower_intrinsics.rs:82:9: 82:88
}

bb1: {
StorageDead(_8); // scope 4 at $DIR/lower_intrinsics.rs:82:87: 82:88
StorageDead(_5); // scope 4 at $DIR/lower_intrinsics.rs:82:87: 82:88
StorageDead(_10); // scope 4 at $DIR/lower_intrinsics.rs:82:88: 82:89
StorageDead(_9); // scope 4 at $DIR/lower_intrinsics.rs:82:88: 82:89
StorageDead(_7); // scope 4 at $DIR/lower_intrinsics.rs:82:88: 82:89
StorageDead(_6); // scope 4 at $DIR/lower_intrinsics.rs:82:88: 82:89
StorageDead(_4); // scope 4 at $DIR/lower_intrinsics.rs:82:88: 82:89
StorageLive(_11); // scope 4 at $DIR/lower_intrinsics.rs:83:9: 83:88
StorageLive(_12); // scope 4 at $DIR/lower_intrinsics.rs:83:46: 83:62
StorageLive(_13); // scope 4 at $DIR/lower_intrinsics.rs:83:46: 83:62
StorageLive(_14); // scope 4 at $DIR/lower_intrinsics.rs:83:46: 83:50
_14 = &_1; // scope 4 at $DIR/lower_intrinsics.rs:83:46: 83:50
_13 = &raw const (*_14); // scope 4 at $DIR/lower_intrinsics.rs:83:46: 83:50
_12 = _13; // scope 4 at $DIR/lower_intrinsics.rs:83:46: 83:62
StorageLive(_15); // scope 4 at $DIR/lower_intrinsics.rs:83:64: 83:84
StorageLive(_16); // scope 4 at $DIR/lower_intrinsics.rs:83:64: 83:84
StorageLive(_17); // scope 4 at $DIR/lower_intrinsics.rs:83:64: 83:74
_17 = &mut _3; // scope 4 at $DIR/lower_intrinsics.rs:83:64: 83:74
_16 = &raw mut (*_17); // scope 4 at $DIR/lower_intrinsics.rs:83:64: 83:74
_15 = _16; // scope 4 at $DIR/lower_intrinsics.rs:83:64: 83:84
- _11 = std::intrinsics::copy_nonoverlapping::<i32>(move _12, move _15, const 1_usize) -> bb2; // scope 4 at $DIR/lower_intrinsics.rs:83:9: 83:88
- // mir::Constant
- // + span: $DIR/lower_intrinsics.rs:83:9: 83:45
- // + literal: Const { ty: unsafe extern "rust-intrinsic" fn(*const i32, *mut i32, usize) {std::intrinsics::copy_nonoverlapping::<i32>}, val: Value(Scalar(<ZST>)) }
+ (*_15) = (*_12); // scope 4 at $DIR/lower_intrinsics.rs:83:9: 83:88
+ goto -> bb2; // scope 4 at $DIR/lower_intrinsics.rs:83:9: 83:88
}

bb2: {
StorageDead(_15); // scope 4 at $DIR/lower_intrinsics.rs:83:87: 83:88
StorageDead(_12); // scope 4 at $DIR/lower_intrinsics.rs:83:87: 83:88
StorageDead(_17); // scope 4 at $DIR/lower_intrinsics.rs:83:88: 83:89
StorageDead(_16); // scope 4 at $DIR/lower_intrinsics.rs:83:88: 83:89
StorageDead(_14); // scope 4 at $DIR/lower_intrinsics.rs:83:88: 83:89
StorageDead(_13); // scope 4 at $DIR/lower_intrinsics.rs:83:88: 83:89
StorageDead(_11); // scope 4 at $DIR/lower_intrinsics.rs:83:88: 83:89
_0 = const (); // scope 4 at $DIR/lower_intrinsics.rs:81:5: 84:6
StorageDead(_3); // scope 2 at $DIR/lower_intrinsics.rs:85:1: 85:2
StorageDead(_2); // scope 1 at $DIR/lower_intrinsics.rs:85:1: 85:2
StorageDead(_1); // scope 0 at $DIR/lower_intrinsics.rs:85:1: 85:2
return; // scope 0 at $DIR/lower_intrinsics.rs:85:2: 85:2
}

bb3 (cleanup): {
resume; // scope 0 at $DIR/lower_intrinsics.rs:76:1: 85:2
}
}

12 changes: 12 additions & 0 deletions src/test/mir-opt/lower_intrinsics.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,15 @@ pub fn discriminant<T>(t: T) {
core::intrinsics::discriminant_value(&());
core::intrinsics::discriminant_value(&E::B);
}

// EMIT_MIR lower_intrinsics.copy_nonoverlapping.LowerIntrinsics.diff
pub fn copy_nonoverlapping() {
let src = 42;
let mut dst_1 = 1;
let mut dst_2 = 2;

unsafe {
std::intrinsics::copy_nonoverlapping(&src as *const _, &mut dst_1 as *mut _, 1);
std::intrinsics::copy_nonoverlapping(&src as *const _, &mut dst_2 as *mut _, 1);
}
}