Bash / Python question

[cjboat]

Hi there. I am currently developing a series of python scripts to reach into S3 and do some CSV insertion into our DB. My thing is, I want to use them in Cronicle for ease of use and scheduling. I have #!/usr/bin/env python3 at the top of the script, is that all that is needed?

I also have another python file with login credentials that I have stored on the local box. I have it being called from creds_prod import HOST, PORT, USER, PASSWORD but it is reliant on having this file in the same directory as the script calling it. How would I do that in Cronicle?

[jhuckaby]

I have #!/usr/bin/env python3 at the top of the script, is that all that is needed?

That will probably work, but it may be better to use the fully-qualified path to your python3 binary, e.g. #!/usr/bin/python3 or #!/usr/local/bin/python3 or wherever it is installed on your system.

Doing it this way ensures that it will always be able to find the binary, even if the PATH environment variable is missing (i.e. after a server reboot, if Cronicle is set to launch at boot).

I also have another python file with login credentials that I have stored on the local box. I have it being called from creds_prod import HOST, PORT, USER, PASSWORD but it is reliant on having this file in the same directory as the script calling it. How would I do that in Cronicle?

Hmmm, I don't know what any of this stuff is. But I can tell you this: I'm assuming you're using the Shell Plugin based on your previous question. So, Cronicle just creates a temp script file, and launches it. It's up to you to issue the correct commands inside that script.

Since you are inling the python code directly, it would be the same as creating a temp file called myfile.py and then running it on the terminal. Cronicle is basically just doing this.

To make things simpler, maybe just create a shell script instead? Then launch your python from inside the shell script?

[cjboat]

Thanks for the replay! Let me play around with the shell side of things and calling it.

[cjboat]

I have #!/usr/bin/env python3 at the top of the script, is that all that is needed?

That will probably work, but it may be better to use the fully-qualified path to your python3 binary, e.g. #!/usr/bin/python3 or #!/usr/local/bin/python3 or wherever it is installed on your system.

Doing it this way ensures that it will always be able to find the binary, even if the PATH environment variable is missing (i.e. after a server reboot, if Cronicle is set to launch at boot).

I also have another python file with login credentials that I have stored on the local box. I have it being called from creds_prod import HOST, PORT, USER, PASSWORD but it is reliant on having this file in the same directory as the script calling it. How would I do that in Cronicle?

Hmmm, I don't know what any of this stuff is. But I can tell you this: I'm assuming you're using the Shell Plugin based on your previous question. So, Cronicle just creates a temp script file, and launches it. It's up to you to issue the correct commands inside that script.

Since you are inling the python code directly, it would be the same as creating a temp file called myfile.py and then running it on the terminal. Cronicle is basically just doing this.

To make things simpler, maybe just create a shell script instead? Then launch your python from inside the shell script?

Hey there. Do you know where the "default" place for the temp location is for this? I'm trying to test something on the python side of things, and doing it through bash isn't giving me any of my prints like it should. Though, NGL, I'm a complete newbie when it comes to things like that.

The script is simple:

#!/bin/bash

# Set the path to your Python script
PYTHON_SCRIPT="/root/scripts/cheq_dl_insert_yesterday.py"

# Ensure the script has execute permissions
chmod +x "$PYTHON_SCRIPT"

# Run the Python script
python3 "$PYTHON_SCRIPT"

and it just kind of ... runs? Maxing out my big prod box's memory, even though when I run the python locally it barely peaks anything.

[jhuckaby]

Well, this is all stuff that is very specific to your unique case. I can't explain the behavioral differences in your script, but I can offer some advice that helps a lot of other people:

1. Python will by default "buffer" all output and not print it until the buffer is full or the script exits. This is probably why you aren't seeing any output. Call python3 with the -u flag to fix this. python3 -u $PYTHON_SCRIPT.
2. You are probably missing key environment variables that are present when you run the script by hand in a terminal, but aren't present in the Cronicle environment. Examine your current environment in your terminal (type env in most shells). You may need to explicitly set some of these vars in your shell script so the behavior is the same.
3. By default the CWD (current working directory) for Cronicle shell scripts is set to the system temp dir (Whatever is returned by os.tmpdir()). You can change this by calling cd from inside your shell script at the very top, e.g. cd /root/scripts/.

Good luck!

[cjboat]

Yeah I know it's super customized, but not sure why it won't run on Cronicle, but it does on the main server. Maybe when we move over to Cronicle 2.