-
Notifications
You must be signed in to change notification settings - Fork 215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversion of WSPR decoder to FT8 #689
Comments
Hi Andy, It's great that you've managed to build and to run the FT8 decoder. With the 1 second decoding time, it could be still possible to simultaneously decode several bands. If I'm not mistaken, the FT8 decoder requires 12000 samples per second. I've just checked the code in ft8_decode.f90 and it looks like Since 12 kSPS can be obtained from 125 MSPS as I've put together a make file that builds So, looks like the FPGA configuration can be built based on the multiband WSPR transceiver project. It would be nice to find a working FT8 decoder that doesn't require building the entire wsjtx package. Best regards, Pavel |
I've just added a FPGA configuration with all the required modifications: |
Pavel, Thanks for the quick response and all the information and modifications. I believe the "jt9" binary is the executable we need to run for decoding FT8, I don't think the "ft8d" is up to date or supported. Here is a copy of the help output for "jt9" as run on my Redpitaya command line https://docs.google.com/document/d/1FVfpCKlMhSnXaePuC9wQM8OY__xB7V4y-MRuvWZ9om4/edit?usp=sharing As you can see the "-8" or "--ft8" options force decoding of FT8 mode WAV files using this 422K executable (see below). There are several other modes this can do as well "-6" JT65, "-4" JT4, "-9" JT9, etc. Since I already built WSJT-X on my Redpitaya and installed in /usr/local, today I tar/gzip it up, which only includes just the WSJT-X files from the bin, include, lib and shared folders. I've made available for download if you wish to try running "jt9" on your system. https://drive.google.com/open?id=1kKnrWl7XFr2fJ_uC3HLeuyrdAsUJn4zK Then just do: cp wsjt-x-usr-local.tgz /usr/local And hopefully you will have all the needed binary files to try for yourself. Thanks again! andyz - K1RA |
Thank you for sharing the pre-built WSJT-X binaries. However, my point is to have a FT8 decoder that is easy to build and to modify. The examples of such decoders are wsprd by Steven Franke, K9AN and weakmon by Robert Morris, AB1HL. BTW, the FT8 decoder by Robert Morris, AB1HL supports two sample rates 12 kSPS and 6 kSPS. It would be interesting to do some tests and to compare the performance of the decoder with different sample rates. |
You're right Now, I can build and run |
I've tested the FT8 decoder with the .wav file from this links. There are 7 .wav files and 71 decodes. The decoding time is 32-33 seconds when the decoder runs on one CPU. So, if the receiver works with 8 bands and the FT8 decoder runs on two CPUs, then the decoding time is around 20 seconds. Looks like the CPU isn't fast enough to decode 8 bands every 15 seconds. A solution could be to run it every 30 seconds. |
I think that this problem could be solved by slightly modifying the |
I haven't compiled your ft8d decoder yet, but did download the zip file of WAVs you used and copied to the RP. I re-ran my earlier timing test, this round using the earlier "jt9" binary I sent and the command "time jt9 -8 170711*.wav" which represents all seven, 15 second decode sequences and got 65 decodes with the following time: real 0m9.010s Full output here https://docs.google.com/document/d/1VUiG6j6-yWv6NqbwLnl8rx0VrmNUnKbfOogdAR50fdU/edit?usp=sharing I also ran just a single 15 second decode sequence WAV file and got root@rp-f01e82:~# time jt9 -8 170711_215815.wav real 0m1.353s I'm not sure why yours is taking so much longer, but will compile and re-run tests to validate here. Thanks for all your help and work to get this system closer to operational. |
Thanks for checking the timings. I've just found that I can reproduce your result if I set the
|
Since we now have a standalone FT8 decoder and the timings are OK, I think that the next steps are
|
Pavel, andyz |
Hi Andy, I tried to modify the FT8 decoder but it turned out to be less trivial than I thought while I was writing my previous comment. The structure of the code isn't very uniform and some parts of the code are parameterized differently than some other parts. Some parameters are centrally defined in ft8_params.f90, some parameters are hardcoded in ft8_downsample.f90 and in subtractft8.f90. After several unsuccessful attempts, I put it on hold for a while. Best regards, Pavel |
Pavel, OK, understand and appreciate your efforts on attempting to convert the FT8 decoder. Would the FPGA code still be able to generate 6000 sps and write to the memory buffer space similar to WSPR? If so, I would like to at least be able to read that into Linux space and then play with CSDR library to convert to USB and then WAV to attempt using with the existing jt9 executable. If not, then I guess it'll have to wait until I become more fluent in Vivado, FPGA and able to take it on myself. Take care andyz |
Just noticed that I wrote too many zeros in some of my previous comments. It's fixed now.
I don't think that the existing jt9 executable requires the 6000 SPS sample rate. The current version of the FPGA configuration outputs the complex samples at 12 kSPS. The unmodified FT8 decoder works with the real samples at 12 kSPS. So, the sample rates are OK.
I'd say that using CSDR for converting complex samples to real samples is an overkill. Just take every other sample and that's it. It can be done either in the program that reads the samples from the FPGA or in the FT8 decoder.
Maybe it wasn't clear from one of my previous comments but the FPGA configuration is ready: What is needed is some C or Fortran code to efficiently decode the FT8 signals. Would you be interested to do some coding for this project or do you expect me to write all the code? |
I've just tested the FPGA configuration with the FT8 decoder and they seem to work well together. I can see many decodes. Here is a link to the SD card image that I'm using: The application starts automatically at boot. The decodes can be seen in In this version, the FPGA configuration outputs the real samples at 12 kSPS and the FT8 decoder reads them at 12 kSPS. However, I still think that it would be better to switch to the complex samples at 6 kSPS. |
Here is a new version: In this version, the FPGA configuration and the FT8 decoder both work with complex samples at 12 kSPS. The number of decodes doubled. |
Finally, I have a version that works with the complex samples at 4 kSPS. With this sample rate, the FPGA configuration contains only one FIR filter: I've started to put together some notes about this project: |
Pavel, Best regards, andyz - K1RA |
I've modified write-c2-files.c to record 236000 complex samples at 4000 samples per second. This way, it records the four 15s transmission periods every minute and can be started by cron once a minute. With this new version, I see more than 200 decodes per minute and more than 10k decodes per hour. If you want to give this new version a try, here is a link to the new version of the SD card image: Looks like the only missing part is an uploader to a remote database. |
Great job again, Pavel! Thanks. This is from the skimmertalk reflector: May be that does already what you need to upload spots. |
Hello Holger, I've looked at the WSJT-X's UDP protocol description at https://sourceforge.net/p/wsjt/wsjt/HEAD/tree/branches/wsjtx/NetworkMessage.hpp and at the Qt serialization algorithm at https://github.com/qt/qtbase/blob/5.11/src/corelib/serialization/qdatastream.cpp Looks like this protocol is quite easy to implement. I'll try to add it to the FT8 receiver application for Red Pitaya. Best regards, Pavel |
For comparison, or as an alternative, PSKreporter at https://pskreporter.info/pskmap.html uses a format that can be found as implemented in https://sourceforge.net/p/wsjt/wsjt/HEAD/tree/branches/wsjtx/psk_reporter.cpp and https://sourceforge.net/p/wsjt/wsjt/HEAD/tree/branches/wsjtx/psk_reporter.h which is discussed in detail here https://www.pskreporter.info/pskdev.html andyz - K1RA |
Pavel, andyz |
For those who want a single, local network, CW Skimmer-like, DXcluster accessible stream for the FT8 decoder, for use with programs like RXClus by HB9BZA http://www.hb9bza.net/rxclus-overview - I've got a simple awk script that can be used with the Redpitaya's onboard 'tail' and 'nc' commands to provide a TCP listener which you can Telnet too, or point your favorite DXcluster client at. It can be downloaded here https://drive.google.com/open?id=15wYQwUah1JFLNotUkglsY_2mUNiuFbcR Once you SCP to the RP set execute permissions
Then start by using following command line
Then from your favorite PC client either get the redpitaya IP address and
or put redpitaya:7373 in your DXcluster client app And you should start seeing CQ only spots like this
This method currently only supports a single TCP inbound connection. 73 andyz - K1RA |
Very nice job |
I'm no programmer, but I've recently written a few other scripts (in PERL) that parse the /dev/shm/decode-ft8.log file. One script creates a WSJT-X UDP compatible stream that will feed GridTracker - https://tagloomis.com/ and the other will post to PSKReporter - you can see what my Redpitaya is copying over the last 15 minutes here: https://pskreporter.info/pskmap.html?preset&callsign=k1ra&txrx=rx&timerange=900 You can change the time frame to see more. If anyone is interested in playing with these scripts let me know. I hope to convert over to C at some point, but if you can get PERL loaded on your RP, I'd be happy to share and let you give it a try. I've been running the decoder and scripts on my RP running Ubuntu 16.04.3 LTS Red Pitaya GNU/Linux Ecosystem V0.98 Build 617 andyz - K1RA |
@azwirko |
Hi Andy, Have you managed to implement the following requirements listed at https://pskreporter.info/pskdev.html?
Best regards, Pavel |
Pavel, I can easily buffer 5 minutes worth of data before I send it and will work on that next. Randomizing the post timing with some sort of delay shouldn't be too hard to add as well. More to follow. Regards andyz |
Tks again Pavel and Andy for your investment in these project :-))
Andy i am not a specialist in programmation, but i am ready to testing the
pskreporter upload script, i running the standalone Alpine version of Pavel
software.
73
Sylvain
F1EYG
|
Suggest to start a separate Issue for this 77-bit conversation, as I just did ;-) PE3ES / Erwin |
Thanks for the .c2 files. They'll help me to test the new version of the FT8 decoder. |
I've updated the FT8 decoder. It now works with the new 77-bit message protocol. Here is a link to a pre-release version of the SD card image: |
Running the new image and it appears to be working. Here are my decodes of 2.0 packets for the last hour: I may attempt to run the old decoder as well in an effort to catch WSJT-X users who haven't converted over to the new version. Not sure there is enough horse power on the ARM for this though. |
You could always modify the ft8.cron to run the new decoder even minutes and the old one the uneven or any other timing combination.
I have a cron that switches between the antennas every 15 minutes and changes Day frequencies to Night frequencies and vice versa. I have 2 upload.sh to differentiate the results on PSKReporter as well. See F/PE3ES-1 and -2
Erwin de PE3ES
|
To see propagation of the new 2.0 version of WSJT-x
https://pskreporter.info/cgi-bin/pskstats.pl
End of the page, General.
General
The following software was used over the last seven days.
Many older versions still being used as well.
de PE3ESErwin
|
I know very little about writing Linux commands/files. I am a Windows man! Some help would be appreciated with:
|
waterwin - I've actually got both decoders running off the same set of .c2 files every minute. I had to modify decode-ft8.sh to handle running the older v1.x decoder. You can see my modified script here: https://www.dropbox.com/s/wxupfhdpp9ucqms/decode-ft8-v2-1.sh?dl=0 It doubles the decode time from ~15 secs. to ~30 secs. Note I also made some other modifications and comments in that script to support several PERL scripts I'd written for the older v1.x decoder and converted for use with the new v2.0 decoder. I've update my dxc.pl (DXCluster server) and jtudp.pl (JTUDP packet broadcast) PERL scripts to handle the new .txt file structure Pavel uses for saving date/time and calls/grids. These new PERL scripts are here JTUDP - https://github.com/azwirko/perl-jtudp-v2 Along with having to install PERL on the RP, you also have to use my modified decode-ft8.sh script above. This prints a few extra lines in /dev/shm/decode-ft8.log so I know when the .txt decode files have been completely written. Note if you use GridTracker https://tagloomis.com/grid-tracker/ with my JTUDP script, unfortunately the map no longer shows different color grids since Pavel new log format no longer saves the actual decoded FT8 message copied off the air. It now only saves a call and/or grid of the transmitting station. It still makes for a nice real-time, local mapping system without need for waiting for PSKreporter. |
Hello all |
There is now a release that includes the updated FT8 decoder: |
Hello Pavel
What is difference with 20181215 ?
Tks
|
The FT8 application is the same in both versions. I've just updated the Linux kernel and some links in the vna and mcpha applications. |
Tks :-)
|
Hello
i testing your version with double decoding 1&2, all working but the
decode_xxxxx .txt
is strange at the end i have i lot of records with an incorrect time like
these :
...
0_1812 1_1400 45.6 0 0.27 3574371 OZ2LP JO55
0_1812 1_1415 3.9 -15 -0.16 3573849 RA4FEQ LO13
0_1812 1_1415 13.0 -7 -1.45 3573940 RA2FCL KO04
0_1812 1_1430 27.9 -4 0.29 3574371 OZ2LP JO55
0_1812 1_1445 5.0 -14 -0.16 3573849 RA4FEQ LO13
...
all other records arent correct :
...
181221 143945 17.2 21 -0.19 14075095 SP7TF KO01
181221 143945 26.4 0 0.00 14075365 K3WW
181221 143945 25.2 2 -0.11 14075660 RX4CD LO22
181221 143945 68.8 1 -0.94 14075797 SP1JQJ
181221 143945 44.8 13 -0.05 14075875 RK4FD LO22
181221 143945 2.7 -12 0.06 14076072 W9DUB
...
the stations corresponding arent not reported on psk
and when i test the call, these station using 1.xx version of FT8
have you the same probleme ?
Best regards ans 73
F1EYG
|
Looks like the old decoder is too old and it's incompatible with the current filename format. I'd suggest to use the old FT8 decoder from the 20181022 release. |
Sorry...........................;-(
Forget my last post i using a wrong version of old FT8 decoder (20181020) i
change for 20181022 and all records arent correct ;-))
System running with day/night band selection and decoding V1&2
I stay like this for 24 hours :-)) and look the result on psk
PS i see the numbers of pitayauser growing from 8 to 26 !
Best regards
F1EYG
|
Yes
Is allready done ! system working
Tks
73 F1EYG
Le ven. 21 déc. 2018 à 16:19, Pavel Demin <notifications@github.com> a
écrit :
… Looks like the old decoder is too old and it's incompatible with the
current filename format.
I'd suggest to use the old FT8 decoder from the 20181022
<https://github.com/pavel-demin/red-pitaya-notes/releases/tag/20181022>
release.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#689 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/APItdymTJqylBFI1MNpTRjaKfTUuNkZaks5u7PwFgaJpZM4S5LTA>
.
|
Hi Pavel & all |
Once you remote access the Redpitaya using SSH and default username and password of root and changeme , you should change directory to /dev/shm by doing a "cd /dev/shm" and then after a few minutes look for files being created there once a minute. You can use "ls" to list new files. You will being to see date/time stamped .txt files which should hold decoded call signs. Use the "more filename.txt" command to view the contents. 73 & HNY |
I looked at the pskreporter logs, and it turns out that there are bunch of "<...>" callsigns being reported. There isn't much point in sending these.... |
Can you post one of the files that is doing this please. The scripts do not start sending spots if there is no callsign in the script. The callsign doing the reporting is important. Might help in debugging. Also for instance the "antenna descriptor" or the system "using" it comes from. My own callsigns by the way F/PE3ES-1 and -2 reported almost 50.000 spots to your wonderful website and I have as antenna descriptor "test for 77 bit". The report itself could contain composite callsigns or other specials. But these are normally reported well: 190121 093830 12.7 -5 0.20 10137499 SM5DAJ JO89 But I also found this: 73 de Erwin |
I've checked the .c2 files (ft8-c2-3573.tgz) provided by Andy and found some decodes with the <...> callsign. Here are some examples:
|
Here is how the FT8 message with the <...> callsign look like:
|
I think that it should be enough to filter out all the callsigns starting with <. Here is a link to the corresponding commit: |
The empty callsigns are filtered out now: |
There is now a release with the updated FT8 decoder that filters out empty and <...> callsigns: |
Hi again Pavel, the following is a continuation of a post I made awhile back
#629
Thanks for your advice, which I followed and I was able to successfully compile the latest WSJT-X 1.8 branch - https://sourceforge.net/projects/wsjt/files/wsjtx-1.8.0/ on the Redpitaya. I was then able to run the "jt9" command line tool with the "-8" option to decode a pre-recorded FT8 transmission (.wav file) - https://sourceforge.net/projects/wsjt/files/samples/FT8/ . The decode took ~1 second on Redpitaya.
My long term goal is to learn Vivado and FPGA, as well as use your WSPR code as a basis to create a multi-band FT8 decoder only, no transmit. Since FT8 seems more popular and there exists a propagation, mapping server called PSKreporter - https://pskreporter.info/pskmap.html , similar to WSPRnet.org , I am hoping to create a receiver to feed into this propagation reporting system.
So, if there is any chance you could answer some questions or point me in the right direction, or provide ideas, I'd much appreciate it, thanks! If this is too much, I completely understand as I'm just a beginner when it comes to DSP on FPGA.
Questions
Is using your WSPR multi-band decoder as a basis for my idea a good starting point, or do you recommend one of your others? As a starting point I would reset the multiple HF frequencies from WSPR to FT8 frequencies - http://www.qsl.net/sv1grb/psk31.htm
Would it be feasible and what would I need to change such that I could capture ~2500-3000 hz of bandwidth above a given FT8 base frequency, instead of the narrow band 375 hz that you are currently capturing for WSPR?
The above stated question implies a ~8x rate increase over your current operation. Do you think buffer sizes and write speeds will be exceeded with this requirement? See next question for buffer changes.
Can the buffer capture length / time be changed easily? The needed capture time at the above rate is only 13 seconds of data, instead of ~110 seconds for WSPR. I see you use system clock and cron to kick off process every 2 minutes. I would need to start capture at 0, 15, 30 and 45 seconds after the minute. Or to state differently, write a file at 13, 28, 43 and 58 seconds after the minute. So cron would not be usable, unless I start every minute and run 4 sequential decodes (15 seconds each). See next question for another issue regarding the file data format.
The wsprd decoder expects a complex .c2 data file. Do you think it would be possible in the FPGA to create a WAV file of the USB pass band? "jt9 -8" expects an FT8 audio file not a c2 complex file. It looks like you have an embedded transceiver, so does any of that project produce straight audio? I see there is a CSDR DSP library - https://github.com/simonyiszk/csdr that converts IQ to SSB (RTL example) so I guess I could do that conversion on the CPU, but was wondering what could be done on the FPGA. Are there any existing blocks/code?
Even if the Redpitaya could only decode 4 or 5 bands, I think that would still be useful tool for the community.
As far as the call sign, spot upload feature, WSJT-X already has the routines to send to PSKreporter, so that code I could reuse as is, once I extracted the data from the jt9 decoder.
I've enjoyed looking over your various project notes online and hope I can create something as useful for the ham radio community.
73
andyz - K1RA
The text was updated successfully, but these errors were encountered: