A possibility to rebuild tools

[bigfatbrowncat]

Hi!

I appreciate your work very much and like it, but I have a problem with using it.

I want to use your library with cygwin environment that differs from the recommended msys. But I can't. The tools and compilers are using the msys paths and so on. They are build upon msys infrastructure...

The problem is - we don't have the sources. Are your tools closed source? I tried rebuilding tcc-65816 from scratch by taking the code from a snes-sdk mirror, but, unfortunately, it needs some patching.

Could you please provide us with the links or the sources for (at least) tcc-65816 sources that you are using?

[alekmaul]

I am not the owner of tcc, i'm using the one shipped with snes-sdk. I will try to compile it or chat with mic_ who did that some time ago.

[Chief-Detektor]

Hi!
While I was trying to compile all tools for linux I found out that the only way to get a working 816-tcc binary was compiling it with a recent version of tcc instead of gcc.
Simply set CC and HOST_CC vars to tcc instead of gcc in config.mak and run make 816-tcc
I used the tcc sources from here sned-sdk

I wonder if it is worth the effort porting the 816-tcc which is based on version 0.9.23 of tcc to the most recent version of tcc 0.9.27.

Edit: Add link to source

[hoit]

Hi,

I think it will be useful to port tcc816 on the latest version of tcc :)
The 0.9.23 is more than 10 years old and there are certainly improvements, new OS support, bug resolution that will serve to tcc 816.
To get support or help on implementation, it is better to work on a latest version.

I already compared tcc816 version with the official tcc and tcc 0.9.23 with 0.9.27 to see modifications (with winmerge software) but i don’t know th difficulty to adapt it.

[Chief-Detektor]

I also compared the versions and it seems to be quite a bit of work to port 816-tcc to 0.9.27.
Also the 816-tcc modifications remove a lot of code from the official 0.9.23. code. Mainly code that is not needed for the 816 version.
The Versions 0.9.23. and 0.9.27 differ a lot. There are some major structural changes.

[hoit]

I saw that you used the tcc link from github. The first one was on google code (https://code.google.com/archive/p/snes-sdk/source/default/commits) and the latest commit was done on August, 2011. I don’t know if there are a gap.

So it is a real challenge to port it, but it could be a big improvement for the snes dev community (pvsneslib and more).

Are you interested?

An other point is the version of wla embedded with with it (9.4). I thought it was the first thing to do (upgrade to 9.8) but i have some difficulties for asm part.

[Chief-Detektor]

I am interested in a nice portable snes-toolchain. So yes. But before I try to update 816-tcc I'd like to build all tools and the toolchain on linux and build a nice Dockerfile/Image so snes programming becomes platform independent.
(Which is my main motivation right now)
Since the 816-tcc is working as long as one complies it with a recent version of tcc I think it's okay for now.
TASM is the real bummer..
To compile for the SPC-700 I thought I could add support for it to uz80as assembler.
Then the toolchain would be completely opensource. :D

[hoit]

ok so please keep us informed when you will have something to try :)

[hoit]

To continue this thread and now that tcc is available on the repository, i think if the project need to be independent, we should upload this sources :

• tasm.exe
• stripcom.exe
• optimore-816.exe (replaced by python script ?)
• bin2h.exe (replaced by bin2txt ?)
• bass.exe
• snesbrr.exe

snesbrr is less important because available here and only to help the developer.

[Chief-Detektor]

I can rebuild all tools except of TASM from which I could not find the source code.
And even if I did have the TASM sources we simply can't distribute it because the code is proprietary.
So this tool needs to be replaced with something like uz80as
The optimore.exe is not a replacement for the optimore script. As far as I understood the exe is for further optimization after the python script has been executed.
In this context it might be worth finding out whether a more recent TCC version would make these optimizations obsolete, or if these optimizations can be implemented into TCC itself.

Instead of simply uploading the sources of the tools mentioned I would rather include them as git submodules since most of the tools are hosted on github already and often by the original mantainers.

[hoit]

I am not sure to understand where is used TASM but i think it is for sound module ?
Because wla dx compile code for SPC-700 too.

I would like to test few modifications of tcc816 sources to understand how it works but the one provided with pvsneslib give me difficulties.

When downloaded, you need to do :
./configure
then i changed few path in config.mak and did "make".
I get "Nothing to be done for 'all'.

I tried with .PHONY: all in makefile but still the same.

Did you progress on Docker ?

[Chief-Detektor]

Yes, TASM is needed to compile code for the SPC-700. If wla dx is able to compile for the SPC-700 that's cool but is it compatible with the code provided with this toolchain? I think the TASM assembler code is somewhat different than wla dx asssembler code. So someone would need to port the SPC-700 code to wla dx.

To build tcc816 one needs to run make 816-tcc.
Make sure you replaced gcc with tcc in config.mak and have a recent version of tcc in your path.

I did make some progress on docker but I am busy with my work right now.
I can make a fork next week and push my progress so far so that you guys can take a look.

[hoit]

From this thread it was not advised to use wla for SPC 700 but there a no informations and it was 5 years ago. And Vhelin (owner of wla) is really open (and available) to improve the tool.

Ok thanks, make 816-tcc.exe (i am on windows) works for me, using gcc too :)

Take the time you need to do something cool and useful for everybody, but i will be happy to see it !

[Chief-Detektor]

You might run in problems using the resulting 816-tcc.exe when you try to compile with it. The gcc does a lot of optimizations and on linux the resulting 816-tcc gives a segfault when I tried to compile code. Ironically I could fix this by simply using tcc. I actually tried a lot of compilers like clang and different versions of gcc (down to 4.x.x). The problem is the 816-tcc was compiled using gcc2 or gcc3. Both versions are incompatible with modern gnu/linux distributions.
Does your 816-tcc work?

[hoit]

Yes it seems to work correctly for me but i didn’t tried a lot of code for now.
I saw that the initial tcc 65816 project was on google code and has been uploaded to github.
This fork is the most recent, i don’t know if it can solve your case:
https://github.com/Ace17/snes-sdk

[hoit]

i need some help to build pvsneslib for wla 9.8. I don't know if someone here can help ?
i would like to add this condition in snes_rules :

%.obj: %.asm
	@echo Doing obj files ... $(fname)
#USER := $<
#ifeq ($(USER), libc.asm)
ifeq ($($<), libc.asm)
	$(AS) -x -v -o $@ $<
else
	$(AS) -v -o $@ $<
endif

but i get snes_rules:68: *** commands commence before first target. Stop.
i tried differents way but ifeq is never triggered...

Then, i would like to generate dynamically the linkfile for wlalink.

%.sfc:
	@echo Linking ... $(notdir $@)
	rm -f linkfile	
	$(foreach dir,$(SOURCES), $(echo $(notdir $(wildcard $(dir)/*.obj)) >> linkfile;))
	$(LD) linkfile $@

but linkfile is not created.

And finally, in hello_world sample, i would like to find where comes from this lines generated in asm :

.section ".rodata" superfree
__tccs_L.6: .db $48,$65,$6c,$6c,$6f,$20,$57,$6f,$72,$6c,$64,$20,$21,$0
__tccs_L.7: .db $57,$45,$4c,$43,$4f,$4d,$45,$20,$54,$4f,$20,$4c,$49,$42,$53,$4e,$45,$53,$0
__tccs_L.8: .db $48,$54,$54,$50,$3a,$2f,$2f,$57,$57,$57,$2e,$50,$4f,$52,$54,$41,$42,$4c,$45,$44,$45,$56,$2e,$43,$4f,$4d,$0
.ends

I thought it cames from 816-gen.c but i replaced char* static_prefix = "__tccs_"; by char* static_prefix = "tccs_"; and it still the same. I didn't find anywhere in sources a label like __tccs_L*

[Chief-Detektor]

Hi! Recipes are evaluated in second pass during makefile processing so you need to inject some shell scripting.
For conditional compilation use this:

%.obj: %.asm
	@echo Doing obj files ... $(fname)
	@FILE=$<; \
		if [ $$FILE = libc.asm ]; then \
		echo "Building using -x switch: $<"; \
		$(AS) -x -v -o $@ $<; \
	else \
		echo "Building: $<"; \
		$(AS) -v -o $@ $<; \
	fi

[hoit]

Hi,
Thanks, this parts now works :)
I guess it will be the same for foreach, i will try it tomorrow.

For hello_world, it was a stupid error, i included an other folder in 816-tcc command line..

[hoit]

really strange. i think tcc is completely bugged. I compiled at least 100 times PVSneslib this last weeks without difficulties. I did change in tcc 816 to replace char* static_prefix = "__tccs_"; by char* static_prefix = "tccs_"; in 816-gen.c file provided with it and i built again PVSneslib. Then i saw that in interrupt.c and .h of PVSneslib that nmi_handler was declared with "__" like that : __nmi_handler; and it produce local item in new wla 9.8.

I just changed it to nmi_handler and built the tool a new time but i got an error on background.h (definition of bgState).

I changed it to :

//background state
typedef struct BgState BgState;
struct BgState {
	unsigned char type;
	unsigned char active;
	unsigned short gfxaddr;
	unsigned short mapaddr;
};

and it seems to work because building continue.

Then i get the same on sprite.h :

In file included from libc_c.c:5:
In file included from background.c:25:
In file included from e:/pvsneslib/source/../include/snes/background.h:38:
In file included from e:/pvsneslib/source/../include/snes/dma.h:37:
e:/pvsneslib/source/../include/snes/sprite.h:60: ';' expected
make[1]: *** [libc_c.ps] Error 1

/*!	\brief Sprite definition : OAM entry for DMA copy if needed */
typedef struct  {
  u8 x;	/*!< xxxxxxxx X coordinate*/
  u8 y;	/*!< yyyyyyyy Y coordinate*/
  u8 tilenumber;	/*!< cccccccc Starting tile num */
  u8 attribute;	/*!< vhoopppc v : vertical flip h: horizontal flip o: priority bits p: palette num c : last byte of tile num */
} oamEntry;

Line 60 is "typedef struct".
i tried all possibilities to change the code but nothing works and i don't understand why it worked before.

I don't know if i will find a solution to it with this version of tcc. If you have any idea, please tell me !

[alekmaul]

Here is the last version mic_ sent me last week, perhaps it could help with this issue.
http://www.portabledev.com/pvsneslib/816-gen.zip
I need to prepare a convention for next day, so i will check the problem this sunday.

[hoit]

I tried it and the result is the same :( i compared files with winmerge and it seems to be only optimisations.

To be sure, i will retry with sources from github. I should also try to build tcc with an other compiler than gcc like advised by Chief-Detektor.

Without solution, i think the 1st step will be to upgrade tcc816 on 0.9.27 then wla.

[hoit]

Hi !

My problem is solved by using sources from github and re-apply modifications to all files. It is not the first time, i think that it is related to Programmer's Notepad which corrupt files some times. I will check to use an other (and better?) tool.

Now, the standard hello world sample build correctly with wla 9.8 but the text doesn't appear to the screen :( i will investigate on it.

edit : -s (small s) flag in wlaling generate .sym file for No$sns

The last compilation rule (after the command wlalink.exe) in snes_rules contains :

	@sed 's/://' <$(TARGET).sym >$(TARGET).tmp
	@mv $(TARGET).tmp $(TARGET).sym

Is it still useful for PVSneslib ?

[hoit]

I spent some hours to understand why the text doesn't appear but without success.
I saw in no$sns that it is loaded correctly into VRAM, it is only a display issue i think.

I uploaded hello_world sources :
hello_world20190610.zip

Does somebody can help me to find where the bug comes from ?
No$sns debugger is not user friendly, i don't know how to use it correctly.

[hoit]

i did test with others sample, this issue is more global than i thought : no display in all samples (background, sprites or text...).
Everything seems to be loaded correctly in VRAM but something wrong during display.

[hoit]

Did you progress on your work Chief-Detektor? :)

[alekmaul]

Not yet, sorry, so much things in real life (i'm preparing a big trip to another country).
As soon I work on it, i will post here

[hoit]

Ok cool! I am curious to know which country you chose ;) enjoy it

[alekmaul]

You see see on FB & Twitter next sunday with my friends I will meet in these mysterious country

[vayan]

@alekmaul @Chief-Detektor did you manage to have a first draft of the docker file? I'm trying to setup pvsneslib on Linux 😄

[alekmaul]

HI, nothing on my side, sorry. I will try to think how to handle dockerfile for the project.

[Chief-Detektor]

Hi!
Well I issued the possibility to create a Dockerfile that can be used to compile C code along with this library.
While I archived something that could possibly be used, I've put it aside.
This is because of 2 reasons:

1. The used C compiler tcc-65816 in this project is a "patched" version of the original tcc to output asm code for the 65816 snes cpu.
   I was able to compile it on Linux but I found it extremely unsatisfying how I archived it.
   All newer C compilers (GCC and CLANG) produced a binary that segfaulted when used.
   The only way to get a working binary was by using the original TCC to compile the tcc-65816.
   When I dig into the tcc-65816 code I was overwhelmed by the dense code originating from the original author Fabrice Bellard and on top of that by the code changes that were made to "patched" it make it compile for the snes.
2. Not all components are available in source form.
   The assembler used to compile the SPC700 code called tasm.exe is closed source. It works without any problems using wine. But including wine just for one little part of the toolchain was another unsatisfying thing that annoyed me.

I really like this project and the idea to be able to write C code for the SNES. But in order to have a really clean and sound toolchain around it I think we ideally need a better C compiler.
One that produces specifically asm code for the SNES from C code and is more maintainable than the tcc-65816.
Also the tasm.exe should be replaced ideally. That shouldn't be that hard as it is just a table driven assembler that reads a table definition and a source file and compiles that to the object file.

This all challenged me but the SNES itself was still too mysterious to me.
In order to get a better understanding of the SNES I started a project on my own in which I was able to get the perspective form the SNES view.
It's mainly a thin execution layer that could eventually become something like no$nes.
I have an initial disassembler as well as a debugger working.
You may take a look if you like: https://gitlab.chief-detektor.dev/chiefdetektor/superrustendo

[vayan]

Thanks for the write up!! I'll try to figure something out with all the infos that you gave me.

I also found this http://emureview.ztnet.com/developerscorner/SoundCPU/spc.htm which has a SPC700 TASM table

[hoit]

Hi,

Spc700 is available in wla too: https://wla-dx.readthedocs.io/en/latest/man/wla-cpu.html
I don’t know the work to adapt current tool but wla is already used for 65816 asm, it could be interesting to use it also for that.

[vayan]

I made this Dockerfile quickly, everything seems to work: https://github.com/vayan/docker-snes-toolchain/blob/master/Dockerfile, I need to try with some big project to make sure everything is ok.

We can use it like this docker run -it --rm -v$(pwd):/game vayan/snes-toolchain make inside your project folder

[Chief-Detektor]

That's cool!
Looks promising.
I will test it as soon as I can.
I like the fact that you seemed to have solved the segfault and compilation errors mainly by selecting a 32bit older ubuntu.
I tried to compile everything on an bleeding edge Arch.

[vayan]

I tried to compile everything on an bleeding edge Arch.

I also wasted lot of time trying to do that 😭

Do you have a link to a big open source C SNES game so I can test things?

[Chief-Detektor]

Unfortunately not..
Due to the fact that traditionally SNES games are written in 65816 assembly.
This is what the project makes so special and cool but also pretty exotic in the approach to create SNES games.

[vayan]

I'm trying things with this one for now 😄 https://drludos.itch.io/keeping-snes-alive

[Chief-Detektor]

Yep this should do it! Fingers crossed! :)

[hoit]

Maybe this one can be used to test:

https://github.com/1r3n33/bomberworld

Or this one but it is older:

http://forums.nesdev.com/viewtopic.php?f=12&t=11382

[vayan]

Pushed a fix to add snesbrr which I forgot. but everything seems to work!

We don't need to work on Windows anymore 😅 and we can do CI and all sort of magic stuff

There's some issue with #36 with the project SNESHEADER not being the same as the devkit, not sure on how to fix that yet for project that really need SRAM

[hoit]

Wow, really cool :) i am not familiar with docker but i will try to install it and use your image.

For the SNESHEADER issue (where parameters used to build the lib and the game need to be the same), it is one of the reason i found to upgrade the lib with the latest version of wla (#29) because i think we can overwrite some properties in game header (like sram) by using directly wla directives (and snestool would become useless)

[Chief-Detektor]

Awesome! Great Job! We should definitely use ci from now on to ensure quality and functionality when working on the tools and the library. Thanks for the effort! RetroAntho <notifications@github.com> schrieb am Mi., 3. Juni 2020, 14:50:

…

Wow, really cool :) i am not familiar with docker but i will try to install it and use your image. For the SNESHEADER issue (where parameters used to build the lib and the game need to be the same), it is one of the reason i found to upgrade the lib with the latest version of wla (#29 <#29>) because i think we can overwrite some properties in game header (like sram) by using directly wla directives (and snestool would become useless) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRYLW64NPRBFODNV3TLOGDRUZBKRANCNFSM4E22ZLFA> .

[alekmaul]

Argggg, i'm too much busy to read all comments, you're all crazy, thanks a lot guys :D !
Also, thanks a lot about your interest on this lib for SNES.
For your information, i will work on PVSneslib upgrade during this summer as I can't do the trip I prepared to USA to met my friend during a retro ocnvention in Phoenix...
So just continue to post comments and everything you want, I will merge all of them for a new version during end of July, the next dates for my vacations.

[Chief-Detektor]

Hi all!
I was researching the net to find a replacement/alternative for the tcc-65816.
There are some nice projects like cc65 but those target only popular CPUs like 6502 or the 65815 from the AppleII.
Since the 65816 CPU used in the SNES has some special features (multiplication/division via register usage of 0x4202-0x4206) it would be desirable to have a compiler framework that is maintained, well documented and more accessible like the most popular once GCC or LLVM.
I found a compiler framework written in python that looks really promising and is well documented.
It's called ppci and has even an 6502 backend implemented (which is the base/subset of the 65816).

It might be worth it to evaluate it for further use and adaption since it is written in python it is definitely easier to extend and maintain than the tcc-65816.
What do you guys think?

[vayan]

T'inquiete pas de soucis @alekmaul 😄

@Chief-Detektor outside my knowledge, what the issue with tcc-65816?

[hoit]

I saw some discussions about compiler comparison. This one is really interesting:
http://forums.nesdev.com/viewtopic.php?t=18361&p=233080

I think the better is still tcc816 for now (because of specific SNES architecture) and we can also remove tools that are not free (like wdc). Tcc816 version is very old but i think it could be updated and improved to solve some existing bugs, merge pieces of code everywhere (constify, optimiser, python script...). From my point of view, it should be better to invest time in the current tools and document it instead of use new one.

Maybe the python compiler ppci you said could be an alternative but i think it will ask as much work as update current tools and it will probably have other constraints/limitations

[Chief-Detektor]

@vayan I think the version of tcc we use is very hard to maintain and extend due to it's nature being tiny and being written by a genius. Also if you compare the original tcc with the patched version we use with this library you can see that these changes seem more a proof of concept than a solid and sound solution.

@hoit I absolutely agree that we should stick with this tcc as long as we don't have a working alternative. The version tcc-65816 is based on is V0.9.23 official repo is here. It might be worth it porting the 65816 specific changes to the newest version but I am really not sure..
I was also wondering how do we handle CPU mode changes? How can the user switch between emulation and native mode and how between 16 and 8 bit Registers etc.
It would be really nice if the compiler provided directives or something similar for this.

[hoit]

Hello,
I am looking what is needed to "finish" this issue and close it officially.
From the 1st post, tcc is now available on git repo.
tasm.exe is not open source so i hope we will replace it by wla-spc700.
doxygen is provided directly but already exists for linux.
I think only snesbrr.exe is missing and the sources are available here : https://www.smwcentral.net/?p=section&a=details&id=8976 but it do not build on my computer : cc1plus.exe: error: CPU you selected does not support x86-64 instruction set

@alekmaul do you remember where you took sources or if you changed anything ?

[vayan]

In this Dockerfile you can see how I managed to rebuild everything: https://github.com/vayan/docker-snes-toolchain/blob/master/Dockerfile

But that's inside a 32-bit docker image :)

[alekmaul]

nice @vayan, I will check the difference with the docker image provided by redbug :)

[alekmaul]

@hoit , regarding snesbrr, I have the source code and here are the author & version
snesbrr - An SNES BRR encoder / decoder.
Version: 2006-12-13
Author: DMV27
Don't know if you got the same that is not 32 bits compliants

[hoit]

I confirm, same author & version !
To solve the error cc1plus.exe: error: CPU you selected does not support x86-64 instruction set, i need to add "-m32" in CFLAGS (file : makefileinclude)
but i get these kind of errors :

MemoryStream.cpp: In member function 'virtual void base::MemoryStream::write(con
st void*, base::Stream::size_type)':
MemoryStream.cpp:60:7: error: 'memcpy' is not a member of 'std'
  std::memcpy(&mem[pos], buf, nmax);

../base/basemath.h:83:17: error: left operand of shift expression '(-1 << 15)' i
s negative [-fpermissive]
../base/basemath.h:83:6: error: enumerator value for 'low' is not an integer con
stant
 enum { low = -1 << (N - 1) };

[hoit]

Hi, i think we can close the issue: snesbrr now build correctly. All tools needed for pvsneslib (and open sources) are available in the repo.

is it ok for you too?