Large database created under Windows is not the same as created under Linux or Mac

[m3ng9i]

I created bolt databases under Windows/Linux/Mac and insert 100000 and 1000 records into the database. Then I got the result below:

file	records	created under	file size	md5sum
win_1000.db	1000	Windows	262144	05d62c9653360e15d35625759947e8ea
linux_1000.db	1000	Linux	262144	05d62c9653360e15d35625759947e8ea
mac_1000.db	1000	Mac	262144	05d62c9653360e15d35625759947e8ea
win_100000.db	100000	Windows	33554432	aa11783743dc42f44778ffbcbc4b0ba9
linux_100000.db	100000	Linux	34795520	b7cc6445179f70911d27f682903632d9
mac_100000.db	100000	Mac	34795520	b7cc6445179f70911d27f682903632d9

Things I found:

1. If the number of the data is small, no matter which OS you use to create the database, the database files are all the same (same size and same content).
2. If insert a large number of date into the database, the one created under Windows is different with the one created under Linux or Mac. See win_100000.db above.
3. All database files can be recognized under Linux or Mac no matter which OS the database is created from.
4. A database with a large number of data created under Linux or Mac can not be recognized under Windows. There will be an error when opening the database: CreateFileMapping: Not enough storage is available to process this command.

Could the 2nd and 4th point above be fixed?

Below is the code for test:

// file: test-bolt.go
package main

import (
    "crypto/sha512"
    "flag"
    "fmt"
    "io"
    "os"
    "github.com/coreos/bbolt"
)


func fileExist(filename string) bool {
    _, err := os.Stat(filename)
    if err == nil {
        return true
    }
    if os.IsNotExist(err) {
        return false
    }
    return true
}


func genData(n int) []byte {
    h := sha512.New()
    io.WriteString(h, fmt.Sprintf("%d", n))
    return h.Sum(nil)
}


func create(filename string, n int) (err error) {
    if fileExist(filename) {
        err = fmt.Errorf("%s already exist, please use another file name", filename)
        return
    }

    db, err := bolt.Open(filename, 0600, nil)
    if err != nil {
        return
    }
    defer db.Close()

    fmt.Fprintf(os.Stderr, "add %d records to %s:\n", n, filename)

    err = db.Update(func(tx *bolt.Tx) error {
        b, err := tx.CreateBucket([]byte("data"))
        if err != nil {
            return err
        }

        var i int
        for i = 1; i <= n; i++ {
            err = b.Put([]byte(fmt.Sprintf("%d", i)), genData(i))
            if err != nil {
                return err
            }

            if i % 2000 == 0 {
                fmt.Printf("%d records added\n", i)
            }
        }
        i--

        if i % 2000 != 0 {
            fmt.Printf("%d records added\n", i)
        }

        return nil
    })

    if err == nil {
        fmt.Println("done")
    }

    return
}


func check(filename string) (err error) {
    if !fileExist(filename) {
        err = fmt.Errorf("%s not exist", filename)
        return
    }

    db, err := bolt.Open(filename, 0600, &bolt.Options{ReadOnly:true})
    if err != nil {
        return
    }
    defer db.Close()

    err = db.View(func(tx *bolt.Tx) error {
        b := tx.Bucket([]byte("data"))
        if b == nil {
            return fmt.Errorf(`bucket "data" not found`)
        }
        fmt.Printf("%s has %d records.\n", filename, b.Stats().KeyN)
        return nil
    })

    return
}


const usage = `test-bolt

Usage:
    test-bolt -m create -n <integer> <filename>
    test-bolt -m check <filename>

Options:
    -m <create|check>   create or check bolt database
    -n <integer>        number of records to insert into the database
    -h                  help message`


func main() {
    var method, filename string
    var n int
    var help bool
    flag.StringVar(&method, "m", "", "create or check bolt database")
    flag.IntVar(&n, "n", 0, "number of records to insert into the database")
    flag.BoolVar(&help, "h", false, "help message")
    flag.Parse()

    if help {
        fmt.Println(usage)
        os.Exit(0)
    }

    if method != "create" && method != "check" {
        fmt.Fprintln(os.Stderr, `value of -m must be "create" or "check"`)
        os.Exit(1)
    }

    if method == "create" && n <= 0 {
        fmt.Fprintln(os.Stderr, "value of -n must > 0")
        os.Exit(1)
    }

    args := flag.Args()
    if len(args) == 0 {
        fmt.Fprintln(os.Stderr, "please input database filename")
        os.Exit(1)
    }

    filename = args[0]

    if method == "create" {
        err := create(filename, n)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
    } else {
        err := check(filename)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
    }
}

Test command:

Create database with 1000 records under Windows:

go run test-bolt.go -m create -n 1000 win_1000.db

Create database with 100000 records under Windows:

go run test-bolt.go -m create -n 100000 win_100000.db

Create database with 1000 records under Linux:

go run test-bolt.go -m create -n 1000 linux_1000.db

Create database with 100000 records under Linux:

go run test-bolt.go -m create -n 100000 linux_100000.db

Create database with 1000 records under Mac:

go run test-bolt.go -m create -n 1000 mac_1000.db

Create database with 100000 records under Mac:

go run test-bolt.go -m create -n 100000 mac_100000.db

Check database under Windows, Linux and Mac:

go run test-bolt.go -m check win_1000.db
go run test-bolt.go -m check win_100000.db
go run test-bolt.go -m check linux_1000.db
go run test-bolt.go -m check linux_100000.db
go run test-bolt.go -m check mac_1000.db
go run test-bolt.go -m check mac_100000.db

[m3ng9i]

I found that in function mmap in bolt_windows.go, if the parameter sz's value is greater than the database's file size, an error of CreateFileMapping: Not enough storage is available to process this command. will return.

If run go run test-bolt.go -m check linux_100000.db under Windows, the value of sz in mmap is 67108864, and the size of linux_100000.db is 34795520, so the error return.

The parameter sz of function mmap is returned by function mmapSize in db.go.

[cenkalti]

Is this still an issue?

If yes;

• Is Windows 32 or 64 bit?
• Does the machine have enough memory?

Related #252