Allow calculating the layout behind a pointer

[CAD97]

There was some discussion around allowing this previously.

This does make the requirement for raw pointers to have valid metadata exposed as part of the std API (as a safety invariant, not validity invariant), though I think this is not strictly necessarily required as of current. cc @rust-lang/wg-unsafe-code-guidelines

Naming is hard; I picked the best "obvious" name I could come up with.

If it's agreed that this is actually a desired API surface, I'll file a tracking issue and update the attributes.

[rust-highfive]

r? @LukasKalbertodt

(rust_highfive has picked a reviewer for you, use r? to override)

[rust-highfive]

The job x86_64-gnu-llvm-7 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2020-02-11T23:44:34.5455606Z ========================== Starting Command Output ===========================
2020-02-11T23:44:34.5456989Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/96d04040-5bbb-49af-b75b-8b15ec91a50c.sh
2020-02-11T23:44:34.5457019Z 
2020-02-11T23:44:34.5460088Z ##[section]Finishing: Disable git automatic line ending conversion
2020-02-11T23:44:34.5466474Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-02-11T23:44:34.5467962Z Task         : Get sources
2020-02-11T23:44:34.5468023Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-11T23:44:34.5468050Z Version      : 1.0.0
2020-02-11T23:44:34.5468076Z Author       : Microsoft
---
2020-02-11T23:44:35.4004341Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-02-11T23:44:35.4083640Z ##[command]git config gc.auto 0
2020-02-11T23:44:35.4132978Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-02-11T23:44:35.4193723Z ##[command]git config --get-all http.proxy
2020-02-11T23:44:35.4329319Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/69079/merge:refs/remotes/pull/69079/merge
---
2020-02-12T00:40:26.1820796Z .................................................................................................... 1700/9627
2020-02-12T00:40:30.8031908Z .................................................................................................... 1800/9627
2020-02-12T00:40:41.7520658Z ..............................i..................................................................... 1900/9627
2020-02-12T00:40:48.7187185Z .................................................................................................... 2000/9627
2020-02-12T00:41:02.1922826Z ....................iiiii........................................................................... 2100/9627
2020-02-12T00:41:11.1459883Z .................................................................................................... 2300/9627
2020-02-12T00:41:13.4184781Z .................................................................................................... 2400/9627
2020-02-12T00:41:17.8242452Z .................................................................................................... 2500/9627
2020-02-12T00:41:38.3528825Z .................................................................................................... 2600/9627
---
2020-02-12T00:44:07.8550302Z .......................................................................i...............i............ 4900/9627
2020-02-12T00:44:15.3964022Z .................................................................................................... 5000/9627
2020-02-12T00:44:23.3741524Z .................................................................................................... 5100/9627
2020-02-12T00:44:28.0231706Z .............i...................................................................................... 5200/9627
2020-02-12T00:44:38.9183433Z .......................................................................................ii.ii........ 5300/9627
2020-02-12T00:44:42.6557674Z i...i............................................................................................... 5400/9627
2020-02-12T00:44:54.1436485Z .................................................................................................... 5600/9627
2020-02-12T00:45:02.0626504Z ...........................................................................i........................ 5700/9627
2020-02-12T00:45:09.2394236Z .................................................................................................... 5800/9627
2020-02-12T00:45:15.2200904Z .................................................................................................... 5900/9627
2020-02-12T00:45:15.2200904Z .................................................................................................... 5900/9627
2020-02-12T00:45:24.9223938Z ...................................................................ii...i..ii...........i........... 6000/9627
2020-02-12T00:45:45.9169532Z .................................................................................................... 6200/9627
2020-02-12T00:45:53.1928270Z .................................................................................................... 6300/9627
2020-02-12T00:46:00.6350720Z ...............................................................................................i..ii 6400/9627
2020-02-12T00:46:17.5757234Z .................................................................................................... 6500/9627
---
2020-02-12T00:48:16.5922871Z .................................................................................................... 7600/9627
2020-02-12T00:48:20.5183509Z .................................................................................................... 7700/9627
2020-02-12T00:48:25.6551734Z .................................................................................................... 7800/9627
2020-02-12T00:48:33.7427868Z .................................................................................................... 7900/9627
2020-02-12T00:48:42.1217416Z .....................................................................iiiiiii.i...................... 8000/9627
2020-02-12T00:48:57.3986695Z .........i......i................................................................................... 8200/9627
2020-02-12T00:49:02.6374003Z .................................................................................................... 8300/9627
2020-02-12T00:49:16.1201892Z .................................................................................................... 8400/9627
2020-02-12T00:49:25.2522678Z .................................................................................................... 8500/9627
---
2020-02-12T00:51:43.6065635Z  finished in 6.992
2020-02-12T00:51:43.6240644Z Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-02-12T00:51:43.8467001Z 
2020-02-12T00:51:43.8468271Z running 178 tests
2020-02-12T00:51:46.6261062Z iiii......i...........ii..iiii...i....i...........i............i..i..................i....i......... 100/178
2020-02-12T00:51:48.8293731Z ...i.i.i...iii..iiiiiiiiiiiiiiii.......................ii.i...........ii......
2020-02-12T00:51:48.8298955Z 
2020-02-12T00:51:48.8303942Z  finished in 5.206
2020-02-12T00:51:48.8468138Z Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-02-12T00:51:49.0062071Z 
---
2020-02-12T00:51:50.8266115Z  finished in 1.980
2020-02-12T00:51:50.8442505Z Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-02-12T00:51:50.9959553Z 
2020-02-12T00:51:50.9959964Z running 9 tests
2020-02-12T00:51:50.9960880Z iiiiiiiii
2020-02-12T00:51:50.9961679Z 
2020-02-12T00:51:50.9963300Z  finished in 0.152
2020-02-12T00:51:51.0140458Z Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-02-12T00:51:51.2142392Z 
---
2020-02-12T00:52:10.5027156Z  finished in 19.489
2020-02-12T00:52:10.5245891Z Check compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-02-12T00:52:10.7266829Z 
2020-02-12T00:52:10.7272189Z running 116 tests
2020-02-12T00:52:23.4126990Z iiiii..i.....i..i...i..i.i.i..i..i..ii....i.i....ii..........iiii..........i.....i..i.......ii.i.ii. 100/116
2020-02-12T00:52:25.0981531Z ....iiii.....ii.
2020-02-12T00:52:25.0985977Z 
2020-02-12T00:52:25.0987310Z  finished in 14.574
2020-02-12T00:52:25.0990233Z Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-02-12T00:52:25.0992576Z Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
2020-02-12T01:05:30.8524418Z 
2020-02-12T01:05:30.8525298Z    Doc-tests core
2020-02-12T01:05:35.5637543Z 
2020-02-12T01:05:35.5637930Z running 2473 tests
2020-02-12T01:05:44.3956642Z ......iiiii......................................................................................... 100/2473
2020-02-12T01:05:52.8368833Z ..................................................................................ii................ 200/2473
2020-02-12T01:06:12.5365361Z .................i.................................................................................. 400/2473
2020-02-12T01:06:12.5365361Z .................i.................................................................................. 400/2473
2020-02-12T01:06:21.3215587Z ......................................................................i..i..................iiii.... 500/2473
2020-02-12T01:06:36.0707211Z .................................................................................................... 700/2473
2020-02-12T01:06:43.8487705Z .................................................................................................... 800/2473
2020-02-12T01:06:51.6567839Z .................................................................................................... 900/2473
2020-02-12T01:06:59.4287644Z .................................................................................................... 1000/2473
---
2020-02-12T01:08:52.8428156Z ...............................................................................i.................... 2400/2473
2020-02-12T01:08:58.6746985Z ............i............................................................
2020-02-12T01:08:58.6747591Z failures:
2020-02-12T01:08:58.6747629Z 
2020-02-12T01:08:58.6748584Z ---- mem/mod.rs - mem::align_of_ptr (line 427) stdout ----
2020-02-12T01:08:58.6748881Z error[E0658]: use of unstable library feature 'layout_for_ptr'
2020-02-12T01:08:58.6749081Z  --> mem/mod.rs:430:15
2020-02-12T01:08:58.6749524Z   |
2020-02-12T01:08:58.6749590Z 6 | assert_eq!(4, mem::align_of_ptr(&5i32));
2020-02-12T01:08:58.6749697Z   |
2020-02-12T01:08:58.6749736Z   = help: add `#![feature(layout_for_ptr)]` to the crate attributes to enable
2020-02-12T01:08:58.6749763Z 
2020-02-12T01:08:58.6749817Z error: aborting due to previous error
2020-02-12T01:08:58.6749817Z error: aborting due to previous error
2020-02-12T01:08:58.6749842Z 
2020-02-12T01:08:58.6750304Z For more information about this error, try `rustc --explain E0658`.
2020-02-12T01:08:58.6750495Z Couldn't compile the test.
2020-02-12T01:08:58.6750706Z ---- mem/mod.rs - mem::size_of_ptr (line 316) stdout ----
2020-02-12T01:08:58.6750905Z error[E0658]: use of unstable library feature 'layout_for_ptr'
2020-02-12T01:08:58.6751067Z   --> mem/mod.rs:323:16
2020-02-12T01:08:58.6751138Z    |
2020-02-12T01:08:58.6751174Z 10 | assert_eq!(13, mem::size_of_ptr(y));
2020-02-12T01:08:58.6751251Z    |
2020-02-12T01:08:58.6751308Z    = help: add `#![feature(layout_for_ptr)]` to the crate attributes to enable
2020-02-12T01:08:58.6751335Z 
2020-02-12T01:08:58.6751369Z error: aborting due to previous error
---
2020-02-12T01:08:58.7007083Z   local time: Wed Feb 12 01:08:58 UTC 2020
2020-02-12T01:08:59.2540165Z   network time: Wed, 12 Feb 2020 01:08:59 GMT
2020-02-12T01:08:59.2540263Z == end clock drift check ==
2020-02-12T01:08:59.6489185Z 
2020-02-12T01:08:59.6581544Z ##[error]Bash exited with code '1'.
2020-02-12T01:08:59.6592137Z ##[section]Finishing: Run build
2020-02-12T01:08:59.6619318Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-02-12T01:08:59.6620815Z Task         : Get sources
2020-02-12T01:08:59.6620851Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-12T01:08:59.6620907Z Version      : 1.0.0
2020-02-12T01:08:59.6620938Z Author       : Microsoft
2020-02-12T01:08:59.6620938Z Author       : Microsoft
2020-02-12T01:08:59.6620972Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-02-12T01:08:59.6621026Z ==============================================================================
2020-02-12T01:09:00.0521246Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-02-12T01:09:00.0561362Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-02-12T01:09:00.0666009Z Cleaning up task key
2020-02-12T01:09:00.0666770Z Start cleaning up orphan processes.
2020-02-12T01:09:00.0765553Z Terminate orphan process: pid (5094) (python)
2020-02-12T01:09:00.1138971Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[LukasKalbertodt]

Could you explain how Layout::for_ptr would be used? Are people supposed to create a dummy/aliasing pointer and pass it to the function? Is that really something useful? (Or just link to that previous discussion, so I can read up on it there.)

[RalfJung]

This does make the requirement for raw pointers to have valid metadata exposed as part of the std API (as a safety invariant, not validity invariant), though I think this is not strictly necessarily required as of current.

I don't think that is a good decision -- and certainly it is a decision that requires FCP at least.

Given the general description of raw pointers as being geared towards unsafe code, I think having the safety invariant require more than the validity invariant is going against the expectation of most people.

So I'd prefer if this function were unsafe, with a safety precondition saying that the metadata on the raw pointers is "valid" or "proper" or whatever adjective we use. If we end up with stronger invariants for raw pointers later down the road, we can always remove the unsafe qualifier, that should be backwards-compatible.

[rust-highfive]

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2020-02-12T15:51:43.0188717Z ========================== Starting Command Output ===========================
2020-02-12T15:51:43.0190100Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/11c79fed-5083-47ba-b044-3cf0c7b10966.sh
2020-02-12T15:51:43.0190135Z 
2020-02-12T15:51:43.0192578Z ##[section]Finishing: Disable git automatic line ending conversion
2020-02-12T15:51:43.0198411Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-02-12T15:51:43.0200961Z Task         : Get sources
2020-02-12T15:51:43.0200992Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-12T15:51:43.0201071Z Version      : 1.0.0
2020-02-12T15:51:43.0201103Z Author       : Microsoft
---
2020-02-12T15:51:43.9662207Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-02-12T15:51:43.9750228Z ##[command]git config gc.auto 0
2020-02-12T15:51:43.9824787Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-02-12T15:51:43.9882297Z ##[command]git config --get-all http.proxy
2020-02-12T15:51:44.0033191Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/69079/merge:refs/remotes/pull/69079/merge
---
2020-02-12T15:56:29.8165551Z     | ------- previous doc comment
2020-02-12T15:56:29.8165827Z 317 | #![feature(layout_for_ptr)]
2020-02-12T15:56:29.8166177Z     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ not permitted following an outer attibute
2020-02-12T15:56:29.8166411Z     |
2020-02-12T15:56:29.8166845Z     = note: inner attributes, like `#![no_std]`, annotate the item enclosing them, and are usually found at the beginning of source files. Outer attributes, like `#[test]`, annotate the item following them.
2020-02-12T15:56:29.8208249Z error: an inner attribute is not permitted following an outer doc comment
2020-02-12T15:56:29.8208568Z    --> src/libcore/mem/mod.rs:429:1
2020-02-12T15:56:29.8208795Z     |
2020-02-12T15:56:29.8209045Z 428 | /// ```
2020-02-12T15:56:29.8209045Z 428 | /// ```
2020-02-12T15:56:29.8209359Z     | ------- previous doc comment
2020-02-12T15:56:29.8209814Z 429 | #![feature(layout_for_ptr)]
2020-02-12T15:56:29.8210208Z     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ not permitted following an outer attibute
2020-02-12T15:56:29.8210457Z     |
2020-02-12T15:56:29.8211205Z     = note: inner attributes, like `#![no_std]`, annotate the item enclosing them, and are usually found at the beginning of source files. Outer attributes, like `#[test]`, annotate the item following them.
2020-02-12T15:56:39.2686597Z    Compiling libc v0.2.66
2020-02-12T15:56:40.0104652Z error: aborting due to 2 previous errors
2020-02-12T15:56:40.0104746Z 
2020-02-12T15:56:40.0272395Z error: could not compile `core`.
---
2020-02-12T15:56:40.2168856Z   local time: Wed Feb 12 15:56:40 UTC 2020
2020-02-12T15:56:40.3779843Z   network time: Wed, 12 Feb 2020 15:56:40 GMT
2020-02-12T15:56:40.3780081Z == end clock drift check ==
2020-02-12T15:56:48.2293291Z 
2020-02-12T15:56:48.2390157Z ##[error]Bash exited with code '1'.
2020-02-12T15:56:48.2403192Z ##[section]Finishing: Run build
2020-02-12T15:56:48.2419872Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-02-12T15:56:48.2421618Z Task         : Get sources
2020-02-12T15:56:48.2421682Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-12T15:56:48.2421728Z Version      : 1.0.0
2020-02-12T15:56:48.2421768Z Author       : Microsoft
2020-02-12T15:56:48.2421768Z Author       : Microsoft
2020-02-12T15:56:48.2421830Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-02-12T15:56:48.2421879Z ==============================================================================
2020-02-12T15:56:48.6547206Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-02-12T15:56:48.6588242Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-02-12T15:56:48.6702241Z Cleaning up task key
2020-02-12T15:56:48.6703243Z Start cleaning up orphan processes.
2020-02-12T15:56:48.7049404Z Terminate orphan process: pid (4474) (python)
2020-02-12T15:56:48.7070086Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[CAD97]

IIRC, the current proposed validity requirement for fat pointers would be enough for these functions. I only note this as a safety rather than validity requirement because the safety of these APIs only impacts the safety requirement. I'm perfectly happy to mark these APIs as unsafe until that question is resolved, though. (I agree that this should probably go through an FCP, though.)

I know the availability of size_of_val/align_of_val for pointers rather than references has been discussed previously, though I can't find any of said discussion currently.

The specific case that prompted me to actually do the legwork to see if this were possible is liballoc::rc::data_offset.

unsafe fn data_offset<T: ?Sized>(ptr: *const T) -> isize {
    // Align the unsized value to the end of the `RcBox`.
    // Because it is ?Sized, it will always be the last field in memory.
    // Note: This is a detail of the current implementation of the compiler,
    // and is not a guaranteed language detail. Do not rely on it outside of std.
    data_offset_align(align_of_val(&*ptr))
}

In this case the pointer is known to always be valid, but being able to directly query the layout of a type behind a pointer would eliminate the need to manifest a reference in this function.

Getting the layout behind a pointer also makes typed dealloc easier. Just as a quick example,

unsafe fn dealloc_t<T: ?Sized>(a: &mut impl AllocRef, ptr: NonNull<T>) {
    a.dealloc(ptr.cast(), Layout::for_ptr(ptr.as_ptr()))
}

The other primary potential usage I see for getting layout from the raw pointer rather than a reference is if the pointer is being passed around and the user wants to get the layout back without manifesting a reference, thus requiring grabbing exclusive access (&mut) or ruling out shared mutability (temporarily) (&).

To be completely honest, I'd be fine if the decision is that this isn't needed/desired. If some code has created a pointer that has valid pointer metadata but isn't valid as a reference, they probably already know the layout (from allocation or when it was a valid reference); in that case this would only be allowing to retrieve said layout information back from the type system rather than a formal parameter.

[CAD97]

I've not touched the safety of the intrinsics (yet) but I marked the exposed API as conservatively unsafe. I'll let FCP decide any further details.

Also, potential footgun I just realized: mem::align_of_ptr as implemented in this PR is "min_align_of_ptr", returning the minimum required alignment, where it could be instead interpreted to return the actual current alignment of the pointer. (Layout::for_ptr probably doesn't have the same issue.)

[rust-highfive]

The job x86_64-gnu-llvm-7 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2020-02-12T16:25:16.9216445Z ========================== Starting Command Output ===========================
2020-02-12T16:25:16.9218698Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/03518e6f-d810-47ec-b3f8-9e9e16a0fbaf.sh
2020-02-12T16:25:16.9218871Z 
2020-02-12T16:25:16.9221968Z ##[section]Finishing: Disable git automatic line ending conversion
2020-02-12T16:25:16.9227699Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-02-12T16:25:16.9229670Z Task         : Get sources
2020-02-12T16:25:16.9229700Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-12T16:25:16.9229741Z Version      : 1.0.0
2020-02-12T16:25:16.9229770Z Author       : Microsoft
---
2020-02-12T16:25:17.9164376Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-02-12T16:25:17.9174308Z ##[command]git config gc.auto 0
2020-02-12T16:25:17.9177090Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-02-12T16:25:17.9179704Z ##[command]git config --get-all http.proxy
2020-02-12T16:25:17.9186657Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/69079/merge:refs/remotes/pull/69079/merge
---
2020-02-12T16:52:55.0887595Z   local time: Wed Feb 12 16:52:55 UTC 2020
2020-02-12T16:52:55.6422287Z   network time: Wed, 12 Feb 2020 16:52:55 GMT
2020-02-12T16:52:55.6425607Z == end clock drift check ==
2020-02-12T16:52:57.1865731Z 
2020-02-12T16:52:57.1962446Z ##[error]Bash exited with code '1'.
2020-02-12T16:52:57.1973106Z ##[section]Finishing: Run build
2020-02-12T16:52:57.1992105Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-02-12T16:52:57.1994242Z Task         : Get sources
2020-02-12T16:52:57.1994298Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-12T16:52:57.1994350Z Version      : 1.0.0
2020-02-12T16:52:57.1994384Z Author       : Microsoft
2020-02-12T16:52:57.1994384Z Author       : Microsoft
2020-02-12T16:52:57.1994440Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-02-12T16:52:57.1994480Z ==============================================================================
2020-02-12T16:52:57.5885695Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-02-12T16:52:57.5926750Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-02-12T16:52:57.6034298Z Cleaning up task key
2020-02-12T16:52:57.6035117Z Start cleaning up orphan processes.
2020-02-12T16:52:57.6134839Z Terminate orphan process: pid (3475) (python)
2020-02-12T16:52:57.6316521Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[RalfJung]

IIRC, the current proposed validity requirement for fat pointers would be enough for these functions.

One proposal for validity requirements for vtables is that this is the same as NonNull<()>. That would certainly not be enough for these functions.

I am perfectly on board with adding these methods for raw pointers, that has a clear use. I am just saying we should be cautious about making these methods safe.

[LukasKalbertodt]

I know way too little about unsafe stuff to be a useful assignee for this PR.

r? @RalfJung

[nikomatsakis]

I agree with @RalfJung that this precise point -- if I'm understanding correctly -- has been a point of contention. I recall discussing it with @Manishearth, for example, who felt strongly that one should be able to construct *mut dyn Foo pointers with a dummy/NULL vtable, so long as you knew that the pointer wouldn't be used.

[RalfJung]

one should be able to construct *mut dyn Foo pointers with a dummy/NULL vtable, so long as you knew that the pointer wouldn't be used.

(Just to be clear, right now NULL specifically is insta-UB because raw wide pointers use the vtable as niche for layout optimizations. But we could conceivably change that.)

[RalfJung]

From a UCG perspective, this LGTM.

However, this is a public (but unstable) libstd addition, so maybe @rust-lang/libs should be involved?
Also, the new libs feature needs a tracking issue.

[Centril]

From @RalfJung's comments it seems like the language level component of the PR was removed cause of the addition of unsafe, so removing T-Lang accordingly.

[SimonSapin]

unsafe fns should have a Safety section in their doc-comment that explains what requirements are expected of callers.

[CAD97]

What exactly should the safety section say for these functions? The technical requirement is that the pointer is thin, or that the fat pointer metadata is "enough" to determine size/align. And I'd argue that a safety requirement of "it is safe to do this" is worse than not having one. Of course, on top of this, with current defined Rust, it's not possible to violate this requirement.

A "safe to deref" requirement is over-constraining, though it is still looser than making a real reference. "Has valid metadata" is too loose, as validity refers to the validity invariant, and the point of it being unsafe is that there are some valid pointers which are not safe to use with these functions.

I definitely think these functions would already be useful, but I don't know how to specify their safety requirements while it's not possible to break them.

[SimonSapin]

Regardless of whether "valid" is the right term for these docs, as far as I can tell the requirements are something like:

• If T is Sized or an extern type, this function is always safe and returns a well defined result
• If T is a [_] slice type, this function is always safe but may return an arbitrary result if the computation overflows
• If T is a dyn Trait trait object, the pointer must have been originally created by unsizing coercion from a pointer to a concrete type

(In 1.41.0 on x64, mem::size_of_val(slice::from_raw_parts(&0_u32, (1 << 62) + 1)) return 4.)

I agree with @RalfJung that this precise point -- if I'm understanding correctly -- has been a point of contention. I recall discussing it with @Manishearth, for example, who felt strongly that one should be able to construct *mut dyn Foo pointers with a dummy/NULL vtable, so long as you knew that the pointer wouldn't be used.

But until we add something like <*const T>::from_raw_parts(data_ptr, metadata) there is no way to do this without something transmute-like that makes assumptions about undocumented details of the memory layout of fat pointers, is there?

[rust-highfive]

The job x86_64-gnu-llvm-7 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2020-03-09T23:46:41.4761581Z ========================== Starting Command Output ===========================
2020-03-09T23:46:41.4767028Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/f820627d-60eb-45d8-9b79-715ae0257a05.sh
2020-03-09T23:46:41.4767439Z 
2020-03-09T23:46:41.4772680Z ##[section]Finishing: Disable git automatic line ending conversion
2020-03-09T23:46:41.4790745Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-03-09T23:46:41.4794014Z Task         : Get sources
2020-03-09T23:46:41.4794268Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-03-09T23:46:41.4794559Z Version      : 1.0.0
2020-03-09T23:46:41.4794728Z Author       : Microsoft
---
2020-03-09T23:46:42.7074710Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-03-09T23:46:42.7087424Z ##[command]git config gc.auto 0
2020-03-09T23:46:42.7094959Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-03-09T23:46:42.7098266Z ##[command]git config --get-all http.proxy
2020-03-09T23:46:42.7106908Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/69079/merge:refs/remotes/pull/69079/merge
---
2020-03-10T00:45:06.7189551Z .................................................................................................... 1700/9754
2020-03-10T00:45:10.9532037Z .................................................................................................... 1800/9754
2020-03-10T00:45:21.8616713Z ...........................................................i........................................ 1900/9754
2020-03-10T00:45:28.9700186Z .................................................................................................... 2000/9754
2020-03-10T00:45:42.4668770Z .................................................iiiii.............................................. 2100/9754
2020-03-10T00:45:52.0422385Z .................................................................................................... 2300/9754
2020-03-10T00:45:54.1151283Z .................................................................................................... 2400/9754
2020-03-10T00:45:57.2768173Z .................................................................................................... 2500/9754
2020-03-10T00:46:18.1591041Z .................................................................................................... 2600/9754
---
2020-03-10T00:48:47.4898896Z ....................i...............i............................................................... 5000/9754
2020-03-10T00:48:56.6129698Z .................................................................................................... 5100/9754
2020-03-10T00:49:01.8374935Z ...............................................................i.................................... 5200/9754
2020-03-10T00:49:07.9487939Z .................................................................................................... 5300/9754
2020-03-10T00:49:16.4160651Z ............................................ii.ii........i...i...................................... 5400/9754
2020-03-10T00:49:23.9349651Z .................................................................................................... 5600/9754
2020-03-10T00:49:33.0385146Z .................................................................................................... 5700/9754
2020-03-10T00:49:39.4702751Z ...................................i................................................................ 5800/9754
2020-03-10T00:49:45.0036529Z .................................................................................................... 5900/9754
2020-03-10T00:49:45.0036529Z .................................................................................................... 5900/9754
2020-03-10T00:49:54.9356790Z .................................................................................................... 6000/9754
2020-03-10T00:50:03.7991945Z ............................ii...i..ii...........i.................................................. 6100/9754
2020-03-10T00:50:20.2603812Z .................................................................................................... 6300/9754
2020-03-10T00:50:26.4643089Z .................................................................................................... 6400/9754
2020-03-10T00:50:26.4643089Z .................................................................................................... 6400/9754
2020-03-10T00:50:37.6676766Z ...........................................................i..ii.................................... 6500/9754
2020-03-10T00:51:02.7624087Z .................................................................................................... 6700/9754
2020-03-10T00:51:05.9464417Z .....................................................i.............................................. 6800/9754
2020-03-10T00:51:07.8145115Z .................................................................................................... 6900/9754
2020-03-10T00:51:09.7111649Z ....................................................................................i............... 7000/9754
---
2020-03-10T00:52:44.3069479Z .................................................................................................... 7700/9754
2020-03-10T00:52:48.1635377Z .................................................................................................... 7800/9754
2020-03-10T00:52:53.7766549Z .................................................................................................... 7900/9754
2020-03-10T00:53:00.5033962Z ..................................i................................................................. 8000/9754
2020-03-10T00:53:09.1937415Z ...................................................................................iiiiiiiiii.i..... 8100/9754
2020-03-10T00:53:24.3125460Z ...........................i.......i................................................................ 8300/9754
2020-03-10T00:53:28.8817270Z .................................................................................................... 8400/9754
2020-03-10T00:53:39.6755019Z .................................................................................................... 8500/9754
2020-03-10T00:53:51.2118575Z .................................................................................................... 8600/9754
---
2020-03-10T00:56:09.8103002Z  finished in 6.897
2020-03-10T00:56:09.8274882Z Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-03-10T00:56:10.0184953Z 
2020-03-10T00:56:10.0185254Z running 179 tests
2020-03-10T00:56:12.7717230Z iiii......i...........ii..iiii....i....i...........i............i..i..................i....i........ 100/179
2020-03-10T00:56:15.0115308Z ....i.i.i...iii..iiiiiiiiiiiiiiii.......................iii............ii......
2020-03-10T00:56:15.0117945Z 
2020-03-10T00:56:15.0124466Z  finished in 5.185
2020-03-10T00:56:15.0299998Z Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-03-10T00:56:15.1896018Z 
---
2020-03-10T00:56:16.9892306Z  finished in 1.959
2020-03-10T00:56:17.0068548Z Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-03-10T00:56:17.1564739Z 
2020-03-10T00:56:17.1564975Z running 9 tests
2020-03-10T00:56:17.1567417Z iiiiiiiii
2020-03-10T00:56:17.1569008Z 
2020-03-10T00:56:17.1569204Z  finished in 0.150
2020-03-10T00:56:17.1732343Z Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-03-10T00:56:17.3570257Z 
---
2020-03-10T00:56:36.1743094Z  finished in 19.001
2020-03-10T00:56:36.8532854Z Check compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-03-10T00:56:37.0352562Z 
2020-03-10T00:56:37.0352811Z running 115 tests
2020-03-10T00:56:50.1766104Z iiiii..i.....i..i...i..i.i.i..i..i..ii....i.i....ii..........iiii.........i.....i..i.......ii.i.ii.. 100/115
2020-03-10T00:56:51.8231857Z ...iiii.....ii.
2020-03-10T00:56:51.8242763Z 
2020-03-10T00:56:51.8242901Z  finished in 14.971
2020-03-10T00:56:51.8248256Z Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-03-10T00:56:51.8249012Z Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
2020-03-10T01:09:20.7197651Z 
2020-03-10T01:09:20.7212159Z    Doc-tests core
2020-03-10T01:09:25.2651059Z 
2020-03-10T01:09:25.2652295Z running 2482 tests
2020-03-10T01:09:33.8638022Z ......iiiii......................................................................................... 100/2482
2020-03-10T01:09:42.4238803Z ....................................................................................ii.............. 200/2482
2020-03-10T01:10:02.1196136Z ...................i................................................................................ 400/2482
2020-03-10T01:10:02.1196136Z ...................i................................................................................ 400/2482
2020-03-10T01:10:11.3855136Z ........................................................................i..i..................iiii.. 500/2482
2020-03-10T01:10:26.9499558Z .................................................................................................... 700/2482
2020-03-10T01:10:34.9893075Z .................................................................................................... 800/2482
2020-03-10T01:10:43.0932678Z .................................................................................................... 900/2482
2020-03-10T01:10:51.0923736Z .................................................................................................... 1000/2482
---
2020-03-10T01:14:08.6246234Z .................................................................................................... 500/760
2020-03-10T01:14:08.6440876Z .................thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: RecvError', src/libstd/sync/mpsc/mod.rs:2741:22
2020-03-10T01:14:08.6461769Z ....thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', src/libstd/sync/mpsc/mod.rsthread '<unnamed>:2766:' panicked at 'called `Result::unwrap()` on an `Err` value: RecvError', src/libstd/sync/mpsc/mod.rs:277817
2020-03-10T01:14:08.6463583Z :21
2020-03-10T01:14:08.6496943Z .......thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: RecvError', src/libstd/sync/mpsc/mod.rs:2645:13
2020-03-10T01:14:08.9089996Z ..........................................thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: RecvError.', src/libstd/sync/mpsc/mod.rs:1997:22
2020-03-10T01:14:08.9119492Z ...thread '.<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', src/libstd/sync/mpsc/mod.rsthread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: RecvError', src/libstd/sync/mpsc/mod.rs:2034:21
2020-03-10T01:14:08.9120520Z :2022:17
2020-03-10T01:14:08.9139494Z ......thread '.<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: RecvError', src/libstd/sync/mpsc/mod.rs:1916:13
2020-03-10T01:14:10.9568200Z .....................thread '<unnamed>' panicked at 'explicit panic', src/libstd/sync/mutex.rs:633:13
2020-03-10T01:14:10.9570378Z ..thread '<unnamed>' panicked at 'test panic in inner thread to poison mutex', src/libstd/sync/mutex.rs:587:13
2020-03-10T01:14:10.9576569Z ..thread '<unnamed>' panicked at 'test panic in inner thread to poison mutex', src/libstd/sync/mutex.rs:563:13
2020-03-10T01:14:10.9583177Z thread '<unnamed>' panicked at 'explicit panic', src/libstd/sync/mutex.rs:694:13
---
2020-03-10T01:14:19.8732091Z 
2020-03-10T01:14:19.8732867Z running 1010 tests
2020-03-10T01:14:36.8694603Z i................................................................................................... 100/1010
2020-03-10T01:14:46.6569238Z .................................................................................................... 200/1010
2020-03-10T01:14:53.3666492Z ..................iii......i......i...i......i...................................................... 300/1010
2020-03-10T01:14:58.1688163Z .................................................................................................... 400/1010
2020-03-10T01:15:04.5594455Z ............................................i..i......................................ii............ 500/1010
2020-03-10T01:15:16.3309469Z .................................................................................................... 700/1010
2020-03-10T01:15:16.3309469Z .................................................................................................... 700/1010
2020-03-10T01:15:22.7295706Z ....................................iiii............................................................ 800/1010
2020-03-10T01:15:36.0869029Z .................................................................................................... 900/1010
2020-03-10T01:15:42.4756066Z ..........................................................iiii...................................... 1000/1010
2020-03-10T01:15:42.8663309Z test result: ok. 990 passed; 0 failed; 20 ignored; 0 measured; 0 filtered out
2020-03-10T01:15:42.8663516Z 
2020-03-10T01:15:42.8767362Z  finished in 158.911
2020-03-10T01:15:42.8782266Z Testing term stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
2020-03-10T01:31:24.2950297Z     Checking rustc-std-workspace-core v1.99.0 (/checkout/src/tools/rustc-std-workspace-core)
2020-03-10T01:31:25.4665713Z  Documenting alloc v0.0.0 (/checkout/src/liballoc)
2020-03-10T01:31:28.3793832Z     Finished release [optimized] target(s) in 23.76s
2020-03-10T01:31:28.7304706Z  Documenting core v0.0.0 (/checkout/src/libcore)
2020-03-10T01:31:40.6412534Z error: `[slice]` cannot be resolved, ignoring it.
2020-03-10T01:31:40.6414431Z     |
2020-03-10T01:31:40.6414431Z     |
2020-03-10T01:31:40.6416357Z 448 | ///     - a [slice], then the length of the slice tail must be an intialized
2020-03-10T01:31:40.6418136Z     |
2020-03-10T01:31:40.6418586Z note: the lint level is defined here
2020-03-10T01:31:40.6419259Z    --> src/libcore/lib.rs:64:9
2020-03-10T01:31:40.6419868Z     |
2020-03-10T01:31:40.6419868Z     |
2020-03-10T01:31:40.6420435Z 64  | #![deny(intra_doc_link_resolution_failure)] // rustdoc is run without -D warnings
2020-03-10T01:31:40.6421757Z     = help: to escape `[` and `]` characters, just add '\' before them like `\[` or `\]`
2020-03-10T01:31:40.6422053Z 
2020-03-10T01:31:41.0532013Z error: aborting due to previous error
2020-03-10T01:31:41.0532850Z 
2020-03-10T01:31:41.0532850Z 
2020-03-10T01:31:41.0834501Z error: Could not document `core`.
2020-03-10T01:31:41.0834948Z 
2020-03-10T01:31:41.0835242Z Caused by:
2020-03-10T01:31:41.0837088Z   process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustdoc --edition=2018 --crate-type lib --crate-name core src/libcore/lib.rs --target x86_64-unknown-linux-gnu -o /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/x86_64-unknown-linux-gnu/doc --error-format=json --json=diagnostic-rendered-ansi --markdown-css rust.css --markdown-no-toc --generate-redirect-pages --resource-suffix 1.43.0 --index-page /checkout/src/doc/index.md -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-std/release/deps` (exit code: 1)
2020-03-10T01:31:41.0853789Z 
2020-03-10T01:31:41.0853789Z 
2020-03-10T01:31:41.0856162Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "rustdoc" "-Zconfig-profile" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "2" "--release" "--locked" "--color" "always" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/checkout/src/libtest/Cargo.toml" "-Z" "unstable-options" "-p" "core" "--" "--markdown-css" "rust.css" "--markdown-no-toc" "--generate-redirect-pages" "--resource-suffix" "1.43.0" "--index-page" "/checkout/src/doc/index.md"
2020-03-10T01:31:41.0857795Z 
2020-03-10T01:31:41.0857990Z 
2020-03-10T01:31:41.0865248Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2020-03-10T01:31:41.0865719Z Build completed unsuccessfully in 1:38:51
2020-03-10T01:31:41.0865719Z Build completed unsuccessfully in 1:38:51
2020-03-10T01:31:41.0920852Z == clock drift check ==
2020-03-10T01:31:41.0939859Z   local time: Tue Mar 10 01:31:41 UTC 2020
2020-03-10T01:31:41.3866439Z   network time: Tue, 10 Mar 2020 01:31:41 GMT
2020-03-10T01:31:41.3868859Z == end clock drift check ==
2020-03-10T01:31:42.2899942Z 
2020-03-10T01:31:42.2978513Z ##[error]Bash exited with code '1'.
2020-03-10T01:31:42.2991982Z ##[section]Finishing: Run build
2020-03-10T01:31:42.3042395Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-03-10T01:31:42.3048322Z Task         : Get sources
2020-03-10T01:31:42.3048594Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-03-10T01:31:42.3048843Z Version      : 1.0.0
2020-03-10T01:31:42.3049048Z Author       : Microsoft
2020-03-10T01:31:42.3049048Z Author       : Microsoft
2020-03-10T01:31:42.3049326Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-03-10T01:31:42.3049804Z ==============================================================================
2020-03-10T01:31:42.6382626Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-03-10T01:31:42.6425066Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/69079/merge to s
2020-03-10T01:31:42.6512814Z Cleaning up task key
2020-03-10T01:31:42.6513878Z Start cleaning up orphan processes.
2020-03-10T01:31:42.6675681Z Terminate orphan process: pid (3797) (python)
2020-03-10T01:31:42.6849641Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

[RalfJung]

LGTM (modulo the last nits), thanks for enduring!

@SimonSapin what do you think?

[RalfJung]

@CAD97 okay @Amanieu says this is okay for t-libs. Could you squash the commits?

[CAD97]

Squashed.

[RalfJung]

Thanks! @bors r+

[bors]

📌 Commit dd973d1 has been approved by RalfJung

[bors]

⌛ Testing commit dd973d1 with merge d1e81ef...

[bors]

☀️ Test successful - checks-azure
Approved by: RalfJung
Pushing d1e81ef to master...

[arslan-raza-143]

There was some discussion around allowing this previously.

This does make the requirement for raw pointers to have valid metadata exposed as part of the std API (as a safety invariant, not validity invariant), though I think this is not strictly necessarily required as of current. cc @rust-lang/wg-unsafe-code-guidelines

Naming is hard; I picked the best "obvious" name I could come up with.

If it's agreed that this is actually a desired API surface, I'll file a tracking issue and update the attributes.

[arslan-raza-143]

@CAD97