Error writing block to datastore: Access is denied. #3971

jaynagpaul · 2017-06-11T20:52:52Z

Fixed by ipfs/go-ds-flatfs#34, needs to be bubbled up.
~ Kubuxu

Version information:

go-ipfs version: 0.4.9-
Repo version: 5
System version: amd64/windows
Golang version: go1.8.1

Type: Bug

Severity: Medium

Description:

Started ipfs daemon:
ipfs daemon
Runs for a bit and connects to peers.
Starts throwing errors in the console.
�[0;37m16:48:35.891 �[31mERROR �[0;34m bitswap: �[0mError writing block to datastore: Access is denied. �[0;37mbitswap.go:322�[0m
and
�[0;37m16:48:39.904 �[31mERROR �[0;34mcommands/h: �[0merr: write tcp4 127.0.0.1:5001->127.0.0.1:56762: wsasend: An established connection was aborted by the software in your host machine. �[0;37mhandler.go:288�[0m

The text was updated successfully, but these errors were encountered:

adi-dev · 2017-07-06T08:48:40Z

is there any progress on it, still happening on 0.4.10, similar setup

wolfsgrove · 2017-07-15T21:33:16Z

Same problem here.

Agent Version go-ipfs/0.4.10/
Protocol Version ipfs/0.1.0

Win 10 amd64 binary install

leerspace · 2017-07-21T13:09:41Z

I also have been getting these errors every once in a while, and was finally able to reproduce it while capturing --debug output from the daemon. At the time of the "error writing block to datastore..." there is also an ACCESS DENIED error at that time in procmon. I'm also seeing quite a bit of activity from Windows' SearchIndexer.exe, SearchProtocolHandler.exe, and MsMpEng messing with files in .ipfs\blocks\. More importantly, I see them messing with the same file when ipfs is also trying to use it.

In the example of the error on my windows machine it was related to the following file:

C:\Users\John\.ipfs\blocks\LU\put-675545035

I'm going to disable Windows' file indexing on the .ipfs directory and try later to see if I still get this. If I do I'll add an exclusion to Windows Defender for the .ipfs directory too.

leerspace · 2017-07-22T00:48:50Z

FWIW since turning off Windows indexing on the .ipfs directory I haven't encountered this error again. I haven't done anything with antivirus.

@jaynagpaul @khokkanen @adi-dev I'd be curious if turning off indexing for the .ipfs directory also works for you.

Kubuxu · 2017-07-22T15:35:07Z

It would make sense, when we are writing to some key we first create temporary file and then move it under correct key. If windows runs search indexer and it accesses the file between those two happen and is still accessing the file when we are trying to move it it will fail.

leerspace · 2017-08-02T02:38:27Z

I just moved my .ipfs directory from my small OS SSD onto a larger HDD and started getting this error again. I thought this might be related to the hash I was pinning containing some virus content (the textfiles.com archive from archives.ipfs.io), but I still have some examples even after excluding the directory from Windows Defender. FWIW for that archiving I was also getting some "Error writing block to datastore: Operation did not complete successfully because the file contains a virus or potentially unwanted software." messages, but those went away when I excluded the .ipfs directory (not necessarily recommended).

I got an example in procmon and it looks like two different threads under the ipfs.exe process are attempting to rename put-* files to the same *.data files before the first thread has closed the file. Maybe there's some kind of race condition exacerbated by having the datastore on a relatively slow disk (?).

Disclaimer in case it's relevant: when moving my .ipfs directory to the HDD I created a symbolic link in the original location, so IPFS is accessing it through a symbolic link.

djdv · 2017-08-02T09:20:28Z

FWIW I get this error seemingly randomly, I have both indexing and defender completely off on my main machine, anything that can exclude the .ipfs directory is set up to ignore it as well.

PCSmith · 2017-08-30T07:03:19Z

This is making the system un-usable for me. Thousands of these access denied errors... Most attempts to access files fail with something like this shown in the browser:

When browsing to: http://localhost:8080/ipfs/QmZpc3HvfjEXvLWGQPWbHk3AjD5j8NEN4gmFN8Jmrd5g83/cs/bob.jpg

"ipfs cat /ipfs/QmZpc3HvfjEXvLWGQPWbHk3AjD5j8NEN4gmFN8Jmrd5g83/cs/bob.jpg: Failed to get block for QmdrCNKCexqTz8cW1PVc3jSQNCNU4NGwEL9iC85KAM8NYc: open C:\Users****.ipfs\blocks\3F\CIQOM4HRVP7HYYMWM5WPMR6LYQGV3VNTVEMSKAFO6OO27356ZWYS3FI.data: The process cannot access the file because it is being used by another process."

Refresh a few times and it works.

This should be rated critical as it's a show stopper for usage.

leerspace · 2017-08-30T14:08:52Z

@PCSmith mostly out of curiosity, have you disabled Windows indexing on the .ipfs directory? Is the .ipfs directory on an SSD or HDD?

whyrusleeping · 2017-08-30T17:58:28Z

@PCSmith do you have an easy way to reproduce the error? Is it just "run ipfs on windows?"

leerspace · 2017-08-31T02:15:36Z

@whyrusleeping I know I'm not @PCSmith, but I can readily reproduce this error on a fresh windows 10 VM using the following steps to reproduce this error. This VM is running on a cheap consumer HDD if it matters.

install ipfs, ipfs init, then ipfs daemon in a cmd window
disable indexing for the .ipfs directory: right click > Properties > Advanced... > Uncheck "Allow files in this folder to have contents indexed..."
ipfs get Qmb8wsGZNXt5VXZh1pEmYynjB6Euqpq3HYyeAdw2vScTkQ in another cmd window (this is the XKCD archive)
try loading various albums in the browser from the Project Apollo Archive http://127.0.0.1:8080/ipfs/QmSnuWmxptJZdLJpKRarxBMS2Ju2oANVrgbr2xWbie9b2D/frontend/frontend.html

PCSmith · 2017-08-31T03:26:59Z

leerspace basically covers it.

I tried it a few different ways with no changes to the issues. Complete delete re-init each time.

HDD
SSD
symlinked to a network drive iSCSI
indexed and non-indexed
anti-virus and none

I'm windows 10 (latest release). 64 GB RAM w/ high end proc.

whyrusleeping · 2017-08-31T03:37:45Z

Great, I just set up a Windows 10 VM. I will be taking a look soon.

…

On Wed, Aug 30, 2017, 8:27 PM Patrick ***@***.***> wrote: leerspace basically covers it. I tried it a few different ways with no changes to the issues. Complete delete re-init each time. 1. HDD 2. SSD 3. symlinked to a network drive iSCSI 4. indexed and non-indexed 5. anti-virus and none I'm windows 10 (latest release). 64 GB RAM w/ high end proc. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3971 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABL4HPuSomT0IgjTNVfXDSLl3EgIzr8Sks5sdigGgaJpZM4N2gba> .

hoffmabc · 2017-09-08T16:52:19Z

Seeing this on OpenBazaar as well with fresh Windows VMs.

PCSmith · 2017-09-11T19:45:52Z

Any luck duplicating it on your end?

whyrusleeping · 2017-09-11T19:51:47Z

Hrm... I was unable to reproduce the error. Could be my ssd is too fast?

hoffmabc · 2017-09-11T20:18:11Z

Not sure. It might not be an issue on SSD drives but can't confirm. We added a looping hack to our code to make sure it goes through, but it's well..hacky.

whyrusleeping · 2017-09-11T20:55:25Z

This should not be a problem if you use badger, so can someone try out latest master to confirm? (`ipfs init —profile=badgerds`)

…

On Sep 11, 2017, 4:18 PM -0400, Brian Hoffman ***@***.***>, wrote: Not sure. It might not be an issue on SSD drives but can't confirm. We added a looping hack to our code to make sure it goes through, but it's well..hacky. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Stebalien · 2017-09-11T21:43:58Z

Related issue from rust land: rust-lang/rust#29497

We could set FILE_ATTRIBUTE_NOT_CONTENT_INDEXED to disable indexing but go doesn't make it easy to make syscalls like that...

leerspace · 2017-09-12T23:14:58Z

I tried building current master from source using the manual steps and then the cygwin way using this documentation, but I ran into errors I didn't know how to fix in both cases. The manual way ends up throwing a bunch of messages that look like this:

..\..\vendor\gx\ipfs\QmQUmDr1DMDDy6KMSsJuyV9nVD7dJZ9iWxXESQWPvte2NP\go-libp2p-swarm\swarm.go:36:2: cannot find package "github.com/whyrusleeping/multiaddr-filter" in any of:
        C:\Users\John\go\src\github.com\ipfs\go-ipfs\vendor\github.com\whyrusleeping\multiaddr-filter (vendor tree)
        C:\Go\src\github.com\whyrusleeping\multiaddr-filter (from $GOROOT)
        C:\Users\John\go\src\github.com\whyrusleeping\multiaddr-filter (from $GOPATH)

And like this using the cygwin way (after having to run dos2unix on multiple files that make install didn't like):

error: couldn't download with ftp (2)
Unable to download https://ipfs.io/ipfs/QmR27Do9gqx9VmuQTEX1UGXETSWYJTQzPzxS5FNUnySCv1/gx/v0.12.0/gx_v0.12.0_windows-amd64.zip. exiting.
make: *** [bin/Rules.mk:19: bin/gx-v0.12.0] Error 1

If someone has a current windows build of master I'd be happy to test. Or if there are updated build instructions for Windows that I can get to work I can also test.

whyrusleeping · 2017-09-13T00:12:52Z

@leerspace you should be able to do:

go get github.com/whyrusleeping/gx
go get github.com/whyrusleeping/gx-go
gx install
go install ./cmd/ipfs

kpcyrd · 2017-09-13T00:21:45Z

@leerspace the error of the second block looks like it's from dist_get:
https://github.com/ipfs/go-ipfs/blob/c14a995fab81f894371ed06ec989c32bdd628e3d/bin/dist_get#L46-L52
Make sure you have either wget or curl installed and in your path.

leerspace · 2017-09-13T02:10:15Z

@whyrusleeping @kpcyrd thank you both for the help. While neither suggestion got me all of the way there, it at least got me past a couple of hurdles where the subsequent errors were relatively easy to solve (though there are a few things I'm not sure how I got past since I didn't keep track of everything I tried). Both of your posts make it sound like this should not be nearly as difficult as it was for me, so maybe I'm just doing something wrong or our environments are different.

The good news is that I have not been able to reproduce the "access denied" error while using the badger datastore (ipfs init --profile=badgerds)! :)

If anyone else who was getting the error wants to test this and is having trouble building this on windows, I'll happily share the executable I used to test. I'm just not sure whether it's good form to post a link to the binary in this issue.

ipfs version 0.4.11-dev-c14a995

piravp · 2017-09-28T00:48:49Z

Anyone has a solution for this yet?
I tried running ipfs daemon, but got these errors trying to access the webui:
02:35:37.193 ERROR bitswap: Error writing block to datastore: Access is denied. bitswap.go:321
and then a blue screen.
I'm running: Version v0.4.10 for windows 64bit on Windows 10.

leerspace · 2017-09-28T00:53:22Z

I don't know if anyone else has tested it, but using badgerds instead of leveldb for the datastore seems to fix it. There's an issue with badger on Windows in 0.4.11-rc2 that prevents badgerds from working on Windows, but it's been fixed in badger and the changes just need to make it up to IPFS.

The blue screen sounds like a separate issue. I haven't seen one of those in a long time (and never from IPFS).

sandersaares · 2017-10-07T06:15:33Z

I confirm issue remains with version go-ipfs_v0.4.11_windows-amd64.zip.

The badgerdb option, whatever that does, also fails:

I:\>ipfs init --profile=badgerds
initializing IPFS node at I:\Data
generating 2048-bit RSA keypair...done
peer identity: QmbMd4xxSWHQYPWNiMTxp37U3GDTXvW7C4vMkTGGwwrYWL
Error: Unable to open value log: Unable to mmap value log file: Not enough storage is available to process this command.

This defect is a complete blocker for my use of IPFS.

djdv · 2017-10-07T08:47:02Z

@whyrusleeping
Is there any plans to implement migration methods that allow you to move from 1 datastore type to another? In this case leveldb -> badger. If not I assume we can just run 2 instances at once with different datastore paths, copying pins from the leveldb instance to the badger one.

@sandersaares
This issue has been resolved upstream dgraph-io/badger@273f40c
I think the ipfs team is waiting for the badger api to become stable before updating though.
dgraph-io/badger#161

magik6k · 2017-10-07T09:28:41Z

@djdv For moving data there is https://github.com/ipfs/ipfs-ds-convert, though it currently has problems with moving data to badger, see ipfs/ipfs-ds-convert#4.

The badger fix for windows happened before badger 0.8, so we can pull it in.

leerspace · 2017-10-07T17:40:11Z

#4242 is related to the badger mmap value log error on Windows and can probably also be closed once badger is updated

leerspace · 2017-10-10T11:46:01Z

For anyone else who was anxious to use badger on Windows, the badgerds option is now working again on Windows (at least as of ipfs version 0.4.12-dev-1e73c3f) and the ipfs-ds-convert utility's previously referenced issue is now fixed.

whyrusleeping · 2017-10-11T09:49:24Z

Though note: badger support in ipfs is experimental. We can't make any guarantees about it yet. That said, please do try things out and give us as much feedback as you can, try to break it :)

dang0ld · 2017-12-06T14:35:34Z

I had this problem and was able to solve it.. mostly .. by opening up the daemon inside an Administrator Command Prompt instead of a user one.

This eliminated all the webui/permissions-related errors I was seeing in the console output - but now I am seeing other ones when actually just running the daemon. Will report back if I figure out anything else..

kiahmed · 2018-03-04T01:43:09Z

running on admin mode in win 10 pro, still get this error
�[0;37m20:10:29.518 �[31mERROR �[0;34m bitswap: �[0mError writing block to datastore: Access is denied. �[0;37mbitswap.go:317�[0m

avatar-lavventura · 2018-03-22T07:17:48Z

I am having same bug on linux as well.

Kubuxu · 2018-03-22T12:57:36Z

@avatar-lavventura the cause for that will be competently different.

Stebalien · 2018-04-02T21:02:15Z

Fixed by ipfs/go-ds-flatfs#34, needs to be bubbled up.

dragon9783 · 2018-06-05T00:56:47Z

I got same issue on win 10, and ipfs version 0.4.15

�[0;37m08:43:17.459 �[31mERROR �[0;34m   bitswap: �[0mError writing block to datastore: rename C:\Users\win\.ipfs\blocks\2L\put-355713921 C:\Users\win\.ipfs\blocks\2L\CIQCYDJ23XAVLSZMK4JDHQK2TCHN6ASKTWAKP6VDP5D7LW5Z54KV2LA.data: Access is denied. �[0;37mbitswap.go:385�[0m

i change directory to full access control, it does not work

Stebalien · 2018-06-05T18:49:05Z

Crap... well, let's move this to a new issue as this one has definitely been fixed.

whyrusleeping added kind/bug A bug in existing code (including security flaws) topic/repo Topic repo topic/windows Windows specific labels Jun 12, 2017

magik6k mentioned this issue Oct 7, 2017

datastore: update badgerds to 0.4.1 #4291

Merged

djdv mentioned this issue Dec 28, 2017

IPFS repo suddenly read-only #4528

Closed

djdv mentioned this issue Jan 7, 2018

gx install sometimes fails to fetch IPFS objects on windows: "Access denied." whyrusleeping/gx#88

Closed

Falsen mentioned this issue Feb 15, 2018

IPFS cannot write to datastore #4710

Closed

Stebalien mentioned this issue Mar 3, 2018

cant find resources on other peers ipfs/ipfs#309

Closed

avatar-lavventura mentioned this issue Mar 22, 2018

Linux: ERROR bitswap: Error writing block to datastore: open /.ipfs/blocks/MX/put-253548744: permission denied bitswap.go:317 #4860

Closed

leerspace mentioned this issue Mar 22, 2018

Win10: WebUI does not show any peers #4862

Closed

Stebalien mentioned this issue Mar 22, 2018

Windows initiative 2018 #4808

Closed

9 tasks

djdv mentioned this issue Mar 23, 2018

gx: update go-ds-flatfs #4872

Merged

ghost assigned djdv Mar 23, 2018

ghost added the status/in-progress In progress label Mar 23, 2018

whyrusleeping closed this as completed in #4872 Apr 26, 2018

ghost removed the status/in-progress In progress label Apr 26, 2018

ri3tn mentioned this issue Jun 3, 2022

[gui error report] Error: Initializing daemon... ipfs/ipfs-desktop#2164

Closed

Error writing block to datastore: Access is denied. #3971

Error writing block to datastore: Access is denied. #3971

Comments

jaynagpaul commented Jun 11, 2017 • edited by Kubuxu Loading

Version information:

Type: Bug

Severity: Medium

Description:

adi-dev commented Jul 6, 2017

wolfsgrove commented Jul 15, 2017

leerspace commented Jul 21, 2017 • edited Loading

leerspace commented Jul 22, 2017 • edited Loading

Kubuxu commented Jul 22, 2017

leerspace commented Aug 2, 2017

djdv commented Aug 2, 2017

PCSmith commented Aug 30, 2017

leerspace commented Aug 30, 2017

whyrusleeping commented Aug 30, 2017

leerspace commented Aug 31, 2017 • edited Loading

PCSmith commented Aug 31, 2017

whyrusleeping commented Aug 31, 2017 via email

hoffmabc commented Sep 8, 2017

PCSmith commented Sep 11, 2017

whyrusleeping commented Sep 11, 2017

hoffmabc commented Sep 11, 2017

whyrusleeping commented Sep 11, 2017 via email

Stebalien commented Sep 11, 2017

leerspace commented Sep 12, 2017 • edited Loading

whyrusleeping commented Sep 13, 2017

kpcyrd commented Sep 13, 2017

leerspace commented Sep 13, 2017

piravp commented Sep 28, 2017 • edited Loading

leerspace commented Sep 28, 2017

sandersaares commented Oct 7, 2017

djdv commented Oct 7, 2017

magik6k commented Oct 7, 2017

leerspace commented Oct 7, 2017

leerspace commented Oct 10, 2017 • edited Loading

whyrusleeping commented Oct 11, 2017

dang0ld commented Dec 6, 2017 • edited Loading

kiahmed commented Mar 4, 2018

avatar-lavventura commented Mar 22, 2018

Kubuxu commented Mar 22, 2018

Stebalien commented Apr 2, 2018

dragon9783 commented Jun 5, 2018 • edited Loading

Stebalien commented Jun 5, 2018

jaynagpaul commented Jun 11, 2017 •

edited by Kubuxu

Loading

leerspace commented Jul 21, 2017 •

edited

Loading

leerspace commented Jul 22, 2017 •

edited

Loading

leerspace commented Aug 31, 2017 •

edited

Loading

leerspace commented Sep 12, 2017 •

edited

Loading

piravp commented Sep 28, 2017 •

edited

Loading

leerspace commented Oct 10, 2017 •

edited

Loading

dang0ld commented Dec 6, 2017 •

edited

Loading

dragon9783 commented Jun 5, 2018 •

edited

Loading