Script uploaded to JSS not populating

[golbiga]

When adding a script to a jss.recipe the contents of the script are not showing up in the JSS. The script itself is there, but the script contents are empty. In the RECIPE_DIR is the script and app specific scripttemplate.xml. The actual jss.recipe has the appropriate fields added for scripts.

When I try to download the script from within the JSS GUI it's downloading the file with filename.sh.txt and the contents of that file are empty. Not sure if I'm doing something wrong here.

Thanks
Allen

[sheagcraig]

Can you post your jss recipe (scrubbed for personal info of course), as well as your ScriptTemplate.xml that you specify in the recipe?

That would give me a good start.

I assume you're not getting any errors in AutoPkg, right?

[golbiga]

No errors in AutoPkg, it says it uploaded the script and/or updated it. I just launched Casper Admin and it says the script is missing and not Stored in Database.

JSS Recipe - https://github.com/golbiga/recipes/blob/master/JSS/TextMate2.jss.recipe
ScriptTemplate - https://github.com/golbiga/recipes/blob/master/JSS/TextMate2ScriptTemplate.xml

[sheagcraig]

Hmmm. Recipe and template look good.

What kind of distribution points are you using?

[golbiga]

smb distribution point

[sheagcraig]

I suspect the .txt thing is not coming from python-jss/JSSImporter. (But from where? No clue!)

The fact that you're getting the red missing script text in Casper Admin means that there should be a script object created for it, but either the file is missing, or it is not replicated across all distribution points.

Do you have more than one SMB distribution point, and if so, are they all configured in your com.github.autopkg.plist?

I'll keep looking through the code to see if I can come up with any other ideas.

[sheagcraig]

Another thing you could check is to hit the API for the list of scripts and see what it reports for TextMate2Preinstall.sh.

If you're not already digging around with python-jss, easiest way would be to go the /api page on your JSS and just hit the Scripts get button and confirm the script object exists.

Then use the ID from that to GET the specific script and double-check that it looks as expected. You could compare to one that you already have too.

[golbiga]

Only a single DP at the moment.

[golbiga]

Can see the script from the /api page, but the contents are empty there as well. I'm guessing when you import it's supposed to take the contents of the script and combine with the template?

Other scripts when brought have the actual script content when i look them up by id.

[sheagcraig]

So that's weird.

On our JSS (9.61...) the "script_contents" are empty for ALL scripts.

You're saying that some of your scripts, when you look at them through the API, have the actual code in the object?

With a SMB (and AFP) DP, I'm pretty sure it just defers to the file specified in the "filename" field.

Once you have "migrated" your server to using a JDS (as mentioned in the "JDS Instances" page of Computer Management, you then have all of your script text included in the script_contents field.

So my guess it that something is jacked up because you have migrated your system to using the DB style storage for the packages and scripts. With a SMB DP configured, JSSImporter is trying to copy the script file, but the JSS ignores it because it really wants it to be in the database.

This is not a situation that I had foreseen!

A few things:

• Can you confirm that you've "migrated"?
• Do packages still work correctly? For sure (like you've downloaded and inspected/installed them)
• I know this sounds nutty, but I have a hunch that you may be able to change your JSS_REPOS key to reference only a single JDS (i.e. have one dict, with key of "type" and value of "JDS"), and it might just do the right thing.

Otherwise, I'll probably have to add in some kind of override key or something that allows you to do this.

[golbiga]

JSS is 9.6, recently migrated/upgraded from 8.73. Post migration I hit the migration button so all flat pkgs were zipped up and scripts copied to database. Not using a JDS.

I just checked a bunch of scripts using GET /scripts/id/{id} and they all had the actual code. This includes scripts that were added post migration to v9.

Packages work just fine, other JSS recipes work perfectly.

[sheagcraig]

Okay, that's almost certainly it then.

python-jss/JSSImporter don't handle that correctly then.

Did you try the "JDS" trick above?

If that doesn't work no matter what, it sounds like I'll have to find some way to handle this condition.

[golbiga]

Is this what you mean?

"JSS_REPOS" = (
{
URL = "smb://servername";
name = CasperShare;
password =;
port = 139;
"share_name" = CasperShare;
type = SMB;
username =;
},
{
type = JDS;
}
);

[sheagcraig]

Close, but remove the SMB DP configuration. The only element of the array should be the "type = JDS".

This will use the same mechanism to upload scripts and packages that JDS' use. The package likely won't work that way, but I just want to see if the script will work. If you already have a package object with the current name JSSImporter will skip it.

[golbiga]

Ok, removed the SMB DP configuration and only had "type = JDS", same results.

[sheagcraig]

Dang.

I'll have to add in something to handle this for sure then.

Until then, you'll have to manually do the script stuff for this policy. Each JSSImporter run will probably replace the script_contents, which could get irritating!

[sheagcraig]

Here's one more question, to help me write the code to handle this.

On your SMB share, do you have a Scripts subdirectory, and if so, does it have the script files in it?

[golbiga]

It does and guess what, the script is there. Maybe if I remove the Scripts folder? It will write to the database instead of trying to write to DP?

[sheagcraig]

Oh right, but you said that the file is empty? Or it's just empty in the DB script_contents field?

Are all of your other scripts in there? Or just the JSSImported one?

[golbiga]

Only scripts that were there pre jss migration/upgrade (moved from os x server to rhel plust 8.73 to 9.6). Newer ones are not there. Which makes me think I could remove that folder?

[sheagcraig]

If you have a backup... Give it a go.

Then give the JDS thing a try again. Make sure you manually erase the script object in the JSS->Computer Management->Scripts page first. Just to be safe.

[sheagcraig]

Actually, hold on. I'm working on my SketchUp recipe, and it requires a script (don't get me started...)

And I'm getting the same thing, only we haven't migrated. There's no script file on the SMB share. So maybe there is something going on in the code that I missed previously.

[golbiga]

Ok, I'm going to find out as well If I can remove the Scripts folder since their technically in the database now.

[golbiga]

I got the OK from JAMF to remove the Scripts folder. When I did that and ran the TextMate2 recipe I get this error when it gets to the point of copying the script.

https://gist.github.com/golbiga/f28dacd967d6acd2e429

[golbiga]

Tried the JDS trick, empty script uploaded but no errors.

[sheagcraig]

Okay, that's as to be expected. You can't shutil.copy to a nonexistent folder.

It's super weird about how the scripts are just empty.

I think that this means that we have to add the script to the script object rather than upload the script file.

[golbiga]

Does that require a change to the code or just a section added to the script template?

[sheagcraig]

This will require some changes to the code. I'm thinking that a global "JSS_MIGRATED" flag will be added. As far as I can tell so far, the only time this affects anything is when a script is uploaded (as packages still work, right?)

I'll need to add mostly documentation. The JSSImporter will need a block to handle JSS_MIGRATED in the handle_scripts() method. Basically, if the JSS has been migrated, it will have to add the contents of the script file to the XML to be POSTed as part of the script object, rather than as a discrete file to be stored in the SMB shares' Scripts folder.

I'm hoping I can test this on our test server, as it has been migrated so we can use a JDS; but it also has a SMB share.

I'll let you know when the testing version drops so you can test it out for me.

[sheagcraig]

@golbiga I have this "working". It has thrown my object hierarchy into a tizzy, so I'm contemplating how to refactor and restructure to make it "make sense". But it does indeed work.

[golbiga]

@sheagcraig I'm still having issues post upgrade. It's still trying to copy the script to a distribution point and not to the database. I added JSS_MIGRATED true to com.github.autopkg.plist

If /data/caspershare/Scripts exists the script is copied there but is blank in the JSS and shows up as missing. If /data/caspershare/Scripts is removed i get the following:

JSSImporter: Script: TextMate2Preinstall.sh created.
JSSImporter: Copying /Users/admin/Library/AutoPkg/RecipeRepos/com.github.golbiga.recipes/JSS/TextMate2Preinstall.sh to all distribution points.
Traceback (most recent call last):
File "/usr/local/bin/autopkg", line 1469, in
sys.exit(main(sys.argv))
File "/usr/local/bin/autopkg", line 1463, in main
exit(subcommands[verb]'function')
File "/usr/local/bin/autopkg", line 1287, in run_recipes
autopackager.process(recipe)
File "/Library/AutoPkg/autopkglib/init.py", line 466, in process
self.env = processor.process()
File "/Library/AutoPkg/autopkglib/init.py", line 295, in process
self.main()
File "/Library/AutoPkg/autopkglib/JSSImporter.py", line 672, in main
self.scripts = self.handle_scripts()
File "/Library/AutoPkg/autopkglib/JSSImporter.py", line 511, in handle_scripts
self.copy(script['name'], id=script_object.id)
File "/Library/AutoPkg/autopkglib/JSSImporter.py", line 381, in copy
self.j.distribution_points.copy(source_item, id=id_)
File "/Library/Python/2.7/site-packages/python_jss-0.5.4-py2.7.egg/jss/distribution_points.py", line 204, in copy
repo.copy_script(filename, id_)
File "/Library/Python/2.7/site-packages/python_jss-0.5.4-py2.7.egg/jss/distribution_points.py", line 385, in copy_script
'Scripts', basename))
File "/Library/Python/2.7/site-packages/python_jss-0.5.4-py2.7.egg/jss/distribution_points.py", line 414, in _copy
shutil.copyfile(full_filename, destination)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: u'/Volumes/CasperShare/Scripts/TextMate2Preinstall.sh'

I tried the JDS trick but it simply adds a blank TextMate2Preinstall.sh file as well.

Thanks
Allen

[sheagcraig]

Hey @golbiga!

Based on the traceback, I think I see what is happening. The very last bit mentions the copy is using shutil.copyfile, which should not be the case here. It should be using distribution_points.SMBDistributionPoint._copy_script_migrated()

AFP and SMB repos can be configured with just the share's "name" and password. The rest of the information to connect is pulled from the Distribution Points API call. As it stands, the AFPDistributionPoint and SMBDistributionPoint have no reference to a jss.JSS object, so they don't know whether JSS_MIGRATED has been set or not.

And the conditions I check for are whether a jss.JSS object has been passed to the AFP/SMB repo constructor AND that it has JSS_MIGRATED=True.

So I'm guessing you didn't divine the magical incantation! This is primarily a documentation issue, but also a python-jss vs. JSSImporter issue. I'm going to add in some code to pass the correct data around for the auto-detection scheme mentioned above so you don't have to add anything extra into your config. Only people who are explicitly configuring AFP and SMB shares outside of the DistributionPoints class will ever need to worry about this, and it is documented in the docstring.

Just to see it "work" and confirm that this methodology works, you can give this a try:

# curl -u <username> 'https://<URL>:<PORT>/dbfileupload' -H 'DESTINATION: 1' -H 'OBJECT_ID: -1' -H 'FILE_TYPE: 3' -H 'FILE_NAME: <FILENAME>' -d '@<FULL_PATH_TO_FILE>' -X POST

e.g.:

# curl -u admin 'https://turtlepower.org:8443/dbfileupload' -H 'DESTINATION: 1' -H 'OBJECT_ID: -1' -H 'FILE_TYPE: 3' -H 'FILE_NAME: find_printer.py' -d '@/Users/scraig/Desktop/find_printer.py' -X POST

If you get it wrong, curl and the JSS will often give you a response of "Success", even though nothing happened. You'll know it worked because

• The script will show up on your JSS
• The JSS will respond with an md5 checksum and a received size
  • e.g. <?xml version="1.0" encoding="UTF-8" standalone="yes"?><uploadResponse><id>38</id><md5>571073a50c394367b05737ff4056e89a</md5><message></message><receivedSize>683</receivedSize><successful>true</successful></uploadResponse>

Can I slip you a testing version once I get it updated? That way I can release again with the comfort of knowing that it works correctly!

[sheagcraig]

Yeah-try grabbing jss/distribution_points.py from the testing branch and copying it in over your installed one. Works for me (and I was able to recreate your issue).

[golbiga]

Just swapped out distribution_points.py and the scripts are now being uploaded properly. Thanks!

[sheagcraig]

Phew! I'm glad that works.

I've got a PR to review; after that I'll push a release. Thanks for bringing to my attention.

[grahampugh]

I've also got this problem. I've set JSS_MIGRATED to True but it is still "copying" the script to the distribution point:

JSSImporter: Script: Carbon Copy Cloner-postinstall.sh created.
JSSImporter: Copying /Users/Me/Library/AutoPkg/RecipeOverrides/Carbon Copy Cloner-postinstall.sh to all distribution points.
JSSImporter: Copying to jamfproserver
JSSImporter: Copied /Users/Me/Library/AutoPkg/RecipeOverrides/Carbon Copy Cloner-postinstall.sh

It cannot really be copying it, because I do not have a Scripts folder - this is a new setup built originally as 9.100, so there's never been a scripts folder. But still, Casper Admin shows the script as "missing" and there is an empty script object in the database. I'm using the default way of setting up the JSSImporter settings. Is there some other value that has to be set?

$ defaults read com.github.autopkg
{
    "API_PASSWORD" = blahblah;
    "API_USERNAME" = AutoPkg;
    "JSS_MIGRATED" = True;
    "JSS_REPOS" =     (
                {
            name = JPShare;
            password = "blahblahblah";
        }
    );
    "JSS_URL" = "https://jamfproserver/template";
}

[sheagcraig]

@grahampugh are you using or testing the JSSImporter testing release? It should solve this issue (it does away with the JSSI_MIGRATED value entirely.

[grahampugh]

Hi Shea Yes we started using that and it works well. Is that release now the one that your AutoPkg recipe points to? PS your JSSImporter came up a LOT in JNUC this year. I’m very interested in helping extend its functionality to the new Patch Policies, which will be separately populated. I guess it needs a new Processor. Cheers Graham

…

On 26 Oct 2017, at 14:12, Shea Craig ***@***.***> wrote: @grahampugh <https://github.com/grahampugh> are you using or testing the JSSImporter testing release? It should solve this issue (it does away with the JSSI_MIGRATED value entirely. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFiK5c97Ln29fNsXUmitEn-SLC5HtUx7ks5swNmdgaJpZM4DJoeM>.