hunting failed due to the backend issue - even on sample images #10
Comments
|
I am getting this as well |
|
I have installed and processed some sample images OK. I have not studied in detail what was found but the processing got through to making queries on the MPC and showing the images and a list of potential objects. Running with images from Slooh.com I very quickly get "the backend error" and the CMD box has Running with some images from iTelescopes the processing gets to "70%" then I get "the backend error" and the CMD box says Caused by: These are, I presume, different errors caused by the different usages of FITS. If this program is going to be useful then it needs to support all/most of the common FITS formats and I don't feel that modifying the camera control software or changing the source code of this program is a viable solution (as suggested in another thread). |
|
All, we're going to update the user guide with instructions on the required format of the .FITS files, very soon. As soon as we do, we'll post here. |
|
Rashid, Have you got agreement from the major online remote telescope operators to produce FITS to a common standard? Tony Evans From: Rashid Sial [mailto:notifications@github.com] All, we're going to update the user guide with instructions on the required format of the .FITS files, very soon. As soon as we do, we'll post here. — |
|
Each camera provider tends to produce their own versions. So even if we asked, that is unlikely to be the case. However, this is editable in post processing by adding header information that is required. I've left a fair bit of information on #5 about the correct format. |
|
The java issue I'm less sure about at the moment. I think if the format is correct it'll go through the java - it has expectations of certain pieces of header information. |
|
It is the camera control software that produces the FITS file. Surely you should support at least the formats produced by the main scope/camera/observatory control software packages. |
|
iTelescope.net FITS Images have passed tests of format run against the NASA Often these software packages are designed by well intentioned people who The key issue here - it doesn't work as per the install instructions and if We are not all "script kitties". With the best of intentions .... Peter On Tue, Mar 24, 2015 at 10:04 AM, Tony notifications@github.com wrote:
|
|
Hi @astroswanny I am an iTelescope user and a software developer by profession - so more than a "script kiddie" :-). I may have some time over the next few weekends to make a pull request to the project code, to get the values it needs from iTelescope FITS files. Thanks @Beasley197 for the WCS header info provided on #5. Most helpful. |
|
@pjcabrera - that would be much appreciated. |
|
Thanks PJ, I could forsee us spinning this up on a VM and really doing something with I have already raised it with Brad. We'll keep across it. Regards On Thu, Mar 26, 2015 at 6:03 AM, Tony notifications@github.com wrote:
|
|
Gentlepeoples, I have found a solution to our woes, without much undue strain on my poor addled brain, and without spending a lot of time on it. I came upon the solution entirely by accident, while noodling around on something else. Can anyone give me a "Eureka!"? The astrometry.net project has created a web service at nova.astrometry.net, that can provide astrometric calibration data for any input image, in many formats, including FITS. Among the output it produces, is a new-image.fits file, with the WCS header info we need! astrometry.net provides the code for their astrometry solver at http://github.com/dstndstn/astrometry.net , as well as a JSON API web service, that can be called from other software. In summary, as Asteroid Data Hunter users, we can do one of 2 things: 1 ) process your source images with astrometry.net, by uploading your FITS files to their web service, then, for each of your uploads, download the generated new-image.fits, and process these WCS calibrated files with Asteroid Data Hunter or
Installation of these command-line utilities is cumbersome and requires also downloading large star catalogs. So for a quick solution, use the web service. As a developer, I am more interested in integrating astrometry.net into the Asteroid Data Hunter code, allowing the upload of any astrometry.net supported file format into Asteroid Data Hunter, not just FITS files, and being almost guaranteed to have a FITS file with the required WCS headers. I'll be working on this integration on my own fork of Asteroid Data Hunter, and will keep you updated on my progress. Feel free to ask any questions here, or on my fork at https://github.com/pjcabrera/NTL-Asteroid-Data-Hunter |
|
PJ, Thanks, good info. What I was referring to was this ....... Anything that passes NASA own test page should work. In the iTelescope.net Portal there is a check box for your image session When I forget to check "Platesolve your images" I usually just have to open Images which have been through this process when uploaded to "FITS Verify" For example I just ramdomly picked an image taken over a year ago and 73 header keywords 16-bit integer pixels, 2 axes (4008 x 2672), ++++++++++++++++++++++ Error Summary ++++++++++++++++++++++ HDU# Name (version) Type Warnings Errors **** Verification found 0 warning(s) and 0 error(s). **** I am just saying I would be VERY SURPRISED if a FITS file which passed Can you have a go at running some of your images on the NASA FITS verify. Thanks and Cheers On Sun, Mar 29, 2015 at 8:15 AM, PJ Cabrera notifications@github.com
|
|
Hi Peter, The FITS verify website says "Capabilities for validation of remote URLs are temporarily disabled" and it definitely wasn't working when I tried it. So I downloaded the source for FITS verify and built my own binary. My test images download right off of the iTelescope.net FTP server all passed the FITS verify with my build. And I'm certain I didn't use the "Platesolve all images" setting when I took them. But that's irrelevant. Until the NTL developers modify Asteroid Data Hunter to be less strict about the format, we're stuck. I agree with you and think it's utterly ridiculous that they release this to the public and 1) it only accepts FITS (huh? the average member of the public doesn't even know what FITS is) 2) has very strict header requirements. But what the hell, let's make lemonade. I'm saying, using the astrometry.net service is a way for people to input valid files to Asteroid Data Hunter, until they modify their source, or until I add astrometry.net to my fork of the processing pipeline. |
|
Also, I've been using Xephem to platesolve my images from iTelescope.net, without the "Platesolve all images" setting and without any extra processing - other than star alignment and integration in Xephem itself - and it platesolves just fine. So good job to you and Brad and whomever else was involved in the FITS work at iTelescope.net |
|
Thanks to everyone who is working on this problem. I normally use Astrometrica to plate solve images and measure asteroid positions for reporting to the MPC. Astrometrica has a moving object detector but it would be great to try this one. The first thing I notice is that FITS Verify will OK an image regardless whether it has been plate solved. It seems they check that what is there is OK but does not check what is missing. This image of a NEOCP earlier this month was obtained using Slooh.com (images are not plate solved on arrival). The image was solved by Astrometrica. The image passed FITS Verfy before and after solving. Four such images spanning a few hours were submitted to Asteroid Hunter and immediately gave the backend error and: Caused by: The date in question can only come from the header card: DATE = '2015-03-14' / File creation date It cannot come from the observation time as that contains day 13 not 14: DATE-OBS= '2015-03-13T23:23:55' / Start of exposure The complete header for the Astrometrica-solved image that caused the error is shown below - if anyone can see anything wrong with it please let me know.: SIMPLE = T / Conforms to FITS standard (NOST 100-2.0) |
|
Rashid, From a Kepner Tregeo problem solving point of view the key question is: The Keywords are standard key words they are either there or they aren't, the information they contain presumably should conform to a the standard. NASA FITS Verify tool clearly identifies each of the key words and tells you if the data complies with the standard. For yet another perspective, I just ran one of the ADH test images provided with the downloads. Interesting, it fails, with 1 warning and 4 errors: 1 Header-Data Units in this file. =================== HDU 1: Primary Array =================== *** Error: Byte #3 in Card#98 is a null(\0).
16-bit integer pixels, 2 axes (4110 x 4096), *** Error: Error trying to read last byte of the file at byte 33678720. HDU# Name (version) Type Warnings Errors So back to the Drawing board I think as the sample images they have provided don't even meet NASA's FITS Verify standards. There is not much more I can offer on any of this as I can't even install it. The best I can do is offer to chair a webex discussion with stakeholders if you are interested. |
|
@pjcabrera, this is awesome, and exactly what we intended when we made this open source. Good luck on your fork! @astroswanny, the correct for format for input can be found in #5. |
|
Rashid, Hi, understand. As discussed the concern I had was that the test images provided don't conform/pass NASA's FITS Verify tool. Secondly I am surprised the Algorithm is not able to cope with standard methods that people currently use. The Keywords are standard things. It shouldn't be too dificult for us to add the extra keywords you require - however thats not the whole story. If someone asks the question how do I stack images of a faint asteroid such that the stacking is done for movement of the object (ie the Track and Stack) How is your algorithm going to cope with that given the only acceptable time is MJD and I see no reference in your headers to Position Angle. Are you going to support stacked images? If we can't stack the images and get them to you with accurate timestamps you will loose a good portion of your intended user base. You should probably spell out how you can/would support stacking and the methods supported. I can see this would be good for survey work where the parameters are always the same, but that is not how amateur astronomers always operate. If you have an amateur astronomer, they require the flexibility to slice and dice the data a number of ways to get a detection. Depending on the brightness and speed of the asteroid there are a significant number of skills/techniques involved that may not always sit with your algorithm and the simple keywords it uses. Just trying to help the discussion ;-) Peter |
|
pjcabrera can you please give me an hint what astrometry.net command you are referring to in your post above? Is this wcs-resample? And could you give an comandline example? |
|
@pauledd I'm just using solve-field, and the only parameter is the relative path to the original_name.fit file. During processing, this will produce a file original_name.new, with the WCS fields in the FITS header. solve-field takes many parameters, but the defaults seem to work for me. But there may be more efficient ways to run it if you know the arcsec width/height and arcsecs per pixel of your image ahead of time. |
|
@pjcabrera thanx. Using solved-field worked. I just had to rename original_name.new to original_name.fits to load it into ADH. It processes until 57% has reached and quits with "Hunting Failed due to the backend issue." Here is some more detail but my guess is that is another issue and not fits-format related: |
|
I'm entering late in the game here, but I tried pjcabrera's suggestion above: "1 ) process your source images with astrometry.net, by uploading your FITS files to their web service, then, for each of your uploads, download the generated new-image.fits, and process these WCS calibrated files with Asteroid Data Hunter" And it didn't work. Specifically, I received the same error from Asteroid Data Hunter: "Hunting Failed due to the backend issue." Not surprisingly (to me at least) my files are too large (> 40 mb) for the on-line version of NASA's fitsverify tool, and at my advanced age I am incapable of becoming sufficiently expert at software development to get the "downloadable" version of fitsverify to work on my local machine. At this date should I expect some future version of ADH to become generally usable for any .fits file format? Is there any specific and explicit example available of what an acceptable .fits header should and should not contain in order to be acceptable to ADH? Or can anyone suggest a more generally useful tool for detecting moving objects? All suggestions are most welcome. Thanks. |
|
@GomezToth as nothing seems to have happened for weeks to progress this problem I don't have great expectations. I'm surprised that a piece of software announced with such a fanfare by NASA does not appear to work. Have you tried using the moving object detector in Astrometrica - it seems to work sometimes. |
|
Tony - Thanks for the rapid reply. I agree that it's surprising that NASA I haven't yet tried the astrometrica option, but it seems that's where I'm GT On Tue, May 26, 2015 at 5:47 PM, Tony notifications@github.com wrote:
|
|
GT, Hi, just a suggestion ...... 40M is a big file size. You must have quite a I share your frustration. Regards On Wed, May 27, 2015 at 11:12 PM, GomezToth notifications@github.com
|
|
Just in case anyone is looking at this code I tried processing some images where the WCS data had been added by Canopus. I got a similar (to when I used Astrometrica for WCS) error in the "backend": Caused by: The only place this date appears in the header is in: |
|
@Tony-E I think I can fix that. @pauledd & @GomezToth uploading to astrometry.net (or running the astrometry.net code locally) has worked with my test images. I don't know why your images don't work. But it seems to me, from the error messages that that have been posted here, the parsing of the FITS content is way too strict. For example, '2015-05-25' is a perfectly valid date, IF the code had used the correct date parser format. Rather than fail, it should have tried a less strict format, until it exhausted all possible ways to encode a date as text. @pauledd & @GomezToth email me separately at pj dot cabrera at gmail dot com and I will add you to a shared folder on Dropbox where you can upload the files that failed for you, and I'll try to figure out the cause and see what changes I can make in the code. As for the fanfare and hoopla that went into announcing this, and the extent to which it doesn't "just work", it seems to me this was really never meant to be used as we expected. It almost looks like they expected citizen scientists (I hate that term, but it's all I got) to use images from a survey, rather than our own images. Or the developers were so, so naive, they assumed everybody's FITS files had the expected format. |
|
PJ, Thanks. Another way of looking at it is that images taken by amateurs which comply But thats the nature of open-source crowd developments - They provide a I am still working on it our end, just waiting for a few things to fall Regards On Thu, May 28, 2015 at 3:37 AM, PJ Cabrera notifications@github.com
|
|
@pjcabrera I just shared four fits in your dropbox. Sorry for the filesize. I just want to be sure they are "raw" as possible. Those files came directly from my Canon700D, processed by astrometry and renamed to fits. |
|
Hello, I try to use the FIT files and load in the application but all the time I have message of error. I'm not sure if my FIT format files was fine, maybe is better define the type of FIT file the application need, normally I use the RAW file of Nikon and use one tools to convert the RAW files to FIT files, maybe the tools is not the good solution. The problem after read the documentation in Hunter Asteroid, not define the parameter FIT files need to load at the application hunter Asteroid. Thank you, Leandro |
|
Hello, The error when load the FIT files. voke(TransactionInterceptor.java:94) on.java:82) on.java:82) on.java:82) Thank you Leandro |
|
@rapalarapala converting a camera RAW file to FIT isn't going to pre-fill many of the required header fields. |
|
Hi folks, I'm going to try to spend some time this week and weekend deploying my latest changes somewhere and having you test on that. Sorry I haven't had as much time as I thought to develop & test to create the pull request. |
when I run the sample images it appears to attempt to work but I get the "hunting failed due to the backend issue" error message. The CMD window shows "failed to load TrainedRF" a bit prior to the error message being displayed.
When I run any of my own images, it fails sooner. If I read correctly on that other backend issue thread, this is setup for only one .fit format????? If that is true then it doesn't seem setup for most of the public to help out using their own images.
The text was updated successfully, but these errors were encountered: