Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add encoding.iconv #22332

Merged
merged 30 commits into from
Oct 2, 2024
Merged

add encoding.iconv #22332

merged 30 commits into from
Oct 2, 2024

Conversation

kbkpbot
Copy link
Contributor

@kbkpbot kbkpbot commented Sep 27, 2024

Module iconv provides functions convert between vstring(UTF8) to/from different encodings.

assert iconv.vstring_to_encoding('V大法好', 'GB18030' ) == [u8(86), 180, 243, 183, 168, 186, 195]
assert iconv.encoding_to_vstring([u8(86), 180, 243, 183, 168, 186, 195], 'GB18030') == 'V大法好'

@kbkpbot
Copy link
Contributor Author

kbkpbot commented Sep 28, 2024

CI macOS show error

OK    [ 150/2006] C:   396.9 ms, R:     4.383 ms vlib/encoding/csv/writer_test.v
OK    [ 151/2006] C:   385.4 ms, R:     4.364 ms vlib/encoding/hex/hex_test.v
OK    [ 152/2006] C:   426.6 ms, R:     4.921 ms vlib/encoding/html/escape_test.v
 FAIL  [ 153/2006] C:  1644.2 ms, R:     0.000 ms vlib/encoding/iconv/iconv_test.v
>> compilation failed:
================== C compilation error (from clang): ==============
cc: ld: symbol(s) not found for architecture arm64
cc: clang: error: linker command failed with exit code 1 (use -v to see invocation)
... (the original output was 47 lines long, and was truncated to 2 lines)
===================================================================
(You can pass `-cg`, or `-show-c-output` as well, to print all the C error messages).
builder error: 
==================
C error found. It should never happen, when compiling pure V code.
This is a V compiler bug, please report it using `v bug file.v`,
or goto https://github.com/vlang/v/issues/new/choose .
You can also use #help on Discord: https://discord.gg/vlang .

OK    [ 154/2006] C:   461.2 ms, R:     7.025 ms vlib/encoding/leb128/leb128_test.v

It seems missing libiconv on macOS?

@spytheman
Copy link
Member

It seems missing libiconv on macOS?

Yes, I will install it through brew on the CI.

@spytheman
Copy link
Member

The windows gcc CI failure is not related to the PR.

@kbkpbot
Copy link
Contributor Author

kbkpbot commented Oct 1, 2024

// write_file_encoding write_file convert `text` into `encoding` and writes to a file with the given `path`. If `path` already exists, it will be overwritten.
// For `encoding` in UTF8/UTF16/UTF32, if `bom` is true, then a BOM header will write to the file.
pub fn write_file_encoding(path string, text string, encoding string, bom bool) ! 

// read_file_encoding reads the file in `path` with `encoding` and returns the contents
pub fn read_file_encoding(path string, encoding string) !string 

@JalonSolov
Copy link
Contributor

JalonSolov commented Oct 1, 2024

Making the BOM an option is probably the best way to go... when saving the data to a file. If you're writing a file specifically for use by something that doesn't need/want a BOM, then it makes sense to leave it out. Very handy to have, though if the file might be read by things that do know how to handle a BOM properly.

While a string is in in memory, it really doesn't do anything other than take up extra space. Your program should know what is in the strings it creates, so it doesn't need a BOM to tell it what format they are.

Copy link
Member

@spytheman spytheman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not find anything more substantial to nitpick anymore.

Good work.

@spytheman spytheman merged commit 7477949 into vlang:master Oct 2, 2024
68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants