Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance document #1197

Merged
merged 9 commits into from
Nov 25, 2022
Merged

Enhance document #1197

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
cc113fb
Update readme.md
NewUserHa Nov 19, 2022
b0d7bf2
Update readme.md
NewUserHa Nov 20, 2022
cc08b6a
Update readme.md
NewUserHa Nov 20, 2022
e5d503e
Update PixivConfig.py
NewUserHa Nov 20, 2022
d696c90
Update readme.md
NewUserHa Nov 23, 2022
2f12865
Update readme.md
NewUserHa Nov 23, 2022
05720c5
Update PixivConfig.py
NewUserHa Nov 23, 2022
8f14cad
Update readme.md
NewUserHa Nov 23, 2022
b8fc3f7
Merge branch 'master' into patch-2
Nandaka Nov 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions PixivConfig.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@ class PixivConfig():
ConfigItem("Settings", "downloadAvatar", False),
ConfigItem("Settings", "useSuppressTags", False),
ConfigItem("Settings", "tagsLimit", -1),
ConfigItem("Settings", "writeImageInfo", False),
ConfigItem("Settings", "writeImageJSON", False),
ConfigItem("Settings", "writeImageInfo", False),
Nandaka marked this conversation as resolved.
Show resolved Hide resolved
ConfigItem("Settings", "writeRawJSON", False),
ConfigItem("Settings", "RawJSONFilter",
"id,title,description,alt,userIllusts,storableTags,zoneConfig,extraData,comicPromotion,fanboxPromotion"),
Expand Down Expand Up @@ -187,19 +187,20 @@ class PixivConfig():

ConfigItem("Ugoira", "writeUgoiraInfo", False),
ConfigItem("Ugoira", "createUgoira", False),
ConfigItem("Ugoira", "deleteZipFile", False),
ConfigItem("Ugoira", "createWebm", False),
ConfigItem("Ugoira", "createWebp", False),
ConfigItem("Ugoira", "createGif", False),
ConfigItem("Ugoira", "createApng", False),
ConfigItem("Ugoira", "deleteUgoira", False),
ConfigItem("Ugoira", "createWebm", False),
ConfigItem("Ugoira", "createWebp", False),
ConfigItem("Ugoira", "deleteZipFile", False),

ConfigItem("DownloadControl", "minFileSize", 0),
ConfigItem("DownloadControl", "maxFileSize", 0),
ConfigItem("DownloadControl", "checkLastModified", True),
ConfigItem("DownloadControl", "alwaysCheckFileSize", False),
ConfigItem("DownloadControl", "overwrite", False),
ConfigItem("DownloadControl", "backupOldFile", False),
ConfigItem("DownloadControl", "dayLastUpdated", 7),
ConfigItem("DownloadControl", "alwaysCheckFileSize", False),
ConfigItem("DownloadControl", "checkUpdatedLimit", 0),
ConfigItem("DownloadControl", "useBlacklistTags", False),
ConfigItem("DownloadControl", "useBlacklistTitles", False),
Expand All @@ -208,7 +209,6 @@ class PixivConfig():
ConfigItem("DownloadControl", "enableInfiniteLoop", False),
ConfigItem("DownloadControl", "useBlacklistMembers", False),
ConfigItem("DownloadControl", "downloadResized", False),
ConfigItem("DownloadControl", "checkLastModified", True),
ConfigItem("DownloadControl", "skipUnknownSize", False),
ConfigItem("DownloadControl", "enablePostProcessing", False),
ConfigItem("DownloadControl", "postProcessingCmd", ""),
Expand All @@ -223,7 +223,7 @@ def proxy(self):
value = getattr(self, "proxyAddress", None)
if not value:
return None
match = re.match(r"^(?:(https?|socks[45])://)?([\w.-]+)(:\d+)?$", value)
match = re.match(r"^(?:(https?|socks[45]h?)://)?([\w.-]+)(:\d+)?$", value)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we add check for socks4a too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can. But I don't have any socks4a server for test.

if not match:
return None
scheme, netloc, port = match.groups()
Expand Down
73 changes: 40 additions & 33 deletions readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ Q3: Error at process_image(): (<type 'exceptions.AttributeError'>,
id, dump html, and log file (check on the application folder).

Q4: URLError: <urlopen error [Errno 11004] getaddrinfo failed>
- Update version to > pixivutil20221029.
- This is because the Pixiv downloader cannot resolve the address to
download the images, please try to restart the network connection or do
ipconfig /flushdns to refresh the dns cache (windows).
Expand Down Expand Up @@ -341,6 +342,9 @@ Please refer run with `--help` for latest information.
- r18mode

Only list images tagged R18, for member, member's bookmark, and search by tag. Set to `True` to enable.
- r18Type

0: All; 1: R18; 2: R18G
- dateformat

Pixiv DateTime format, leave blank to use default format (YYYY-MM-DD).
Expand Down Expand Up @@ -500,24 +504,24 @@ Please refer run with `--help` for latest information.

Number of tags to be used for %tags% meta in filename.
Use -1 to use all tags.
- writeimageinfo

Set to `True` to export the image information to text file.
The filename is following `filename(Manga)Infoformat` + .txt.
- writeImageJSON

Set to `True` to export the image information to JSON.
Set to `True` to export the entire original information for the image from the source to JSON.
The filename is following `filename(Manga)Infoformat` + .json.
- writeimageinfo

Set to `True` to export the image information to text file. This is a subset info extracted from the `writeImageJSON`.
The filename is following `filename(Manga)Infoformat` + .txt.
- writeRawJSON

Set to `True` to export the image JSON untouched.
The filename is following `filename(Manga)Infoformat` + .json.
- RawJSONFilter

Enter the JSON keys which you want to filter out. Keys are seperated by a comma.
- writeSeriesJSON
- includeSeriesJSON

Set to `True` to export the series information to JSON.
Set to `True` to export the series information to JSON. Non-series artwork doesn't have this info.
The filename is following `filenameSeriesJSON` + .json.
- writeImageXMP

Expand All @@ -527,10 +531,10 @@ Please refer run with `--help` for latest information.
Set to `True` to export the image information to a .XMP sidecar file, one per image in the album. The data contained within the file is the same but some software requires matching file names to detect the metadata. If set to `True`, then `writeImageXMP` is ignored.
- verifyimage

Do image and zip checking after download. Set the value to `True` to enable.
Check if downloaded files are valid image or zip. Set the value to `True` to enable.
- writeUrlInDescription

Write all url found in the image description to a text file. Set to `True` to enable. The list will be saved to to the application folder as url_list_<timestamp>.txt
Write all url found in the image description to a text file at the root directory. Set to `True` to enable. The list will be saved to to the application folder as url_list_<timestamp>.txt
- stripHTMLTagsFromCaption

Remove all HTML tags and their contents from the image caption/description when writing metadata to files. The contents of any links will be lost, so consider enabling writeUrlInDescription to retain them.
Expand All @@ -542,7 +546,7 @@ Please refer run with `--help` for latest information.
Use different database.
- setLastModified

Set last modified timestamp based on pixiv upload timestamp.
Set last modified timestamp based on pixiv upload timestamp to the file.
- useLocalTimezone

Use local timezone when setting last modified timestamp/works date.
Expand All @@ -557,20 +561,24 @@ Please refer run with `--help` for latest information.
- maxFileSize

Skip if file size is more than minFileSize, set `0` to disable.
- checkLastModified

If the last-modified timestamp of the local files is the same with the uploaded date of the artwork, it'll log "match" and skip to process the current image_id.
Require `setlastmodified = True` in config.ini to work properly
- alwaysCheckFileSize

Actually, it'll always check the file size. But if `this` is false, if the `overwrite` is also false and this file is recorded in db, it'll skip to process the current image_id.
This will override the image_id checking from db (always fetch the image page to check the remote size).
- overwrite

Overwrite old files, set `False` to disable.
If is true, when found file size different, it'll just delete the file (unless the backupOldFile is true), then start to re-download the image.
- backupOldFile

Set to True to backup old file if the file size is different.
Old filename will be renamed to filename.unix-time.extension.
- daylastupdated

Only process member_id which were processed at least x days since the last check.
- alwaysCheckFileSize

Check the file size, if different then it will be downloaded again, set `False` to disable.
This will override the image_id checking from db (always fetch the image page to check the remote size).
- checkUpdatedLimit

Jump to the next member id if already see n-number of previously downloaded images.
Expand Down Expand Up @@ -600,10 +608,6 @@ Please refer run with `--help` for latest information.
- downloadResized

Download the medium size, rather than the original size.
- checkLastModified

Compare local file's last-modified timestamp with works date.
Require `setlastmodified = True` in config.ini to work properly
- skipUnknownSize

Skip downloading if the remote size is not known when `alwaysCheckFileSize` is set to True.
Expand All @@ -626,39 +630,39 @@ Please refer run with `--help` for latest information.
Codec to be used for encoding webm, default is using `libwebp`.
- webpparam

Parameter to be used to encode webm, default: `-lossless 0 -compression_level 5 -quality 100 -loop 0 -vsync 0`
Parameter to be used to encode webm, default: `-lossless 0 -compression_level 5 -quality 100 -loop 0 -vsync 0`.

## [Ugoira]
- writeugoirainfo

If set to `True`, it will dump the .js to external file.
If set to `True`, it will write the info of ugoira frames to a `filename(Manga)Infoformat`+.zip.js file. `writeImageJSON` contains this info as well.
- createugoira

If set to `True`, it will create .ugoira file.
This is Pixiv own format for animated images. You can use Honeyview to see the animation.
- deleteZipFile
- createwebm

If set to `True`, it will delete the zip files from ugoira.
Only active if `createUgoira = True`.
Set to True to create webm file (video format). The default encoding settings is lossy encoding but high quality with smallest file size.
Required `createUgoira = True` and ffmpeg executeable.
- createwebp

Set to True to create webp file (image format). The default encoding settings is lossy encoding but high quality with smaller file size.
Required `createUgoira = True` and ffmpeg executeable.
- creategif

Set to True to convert ugoira file to gif.
Set to True to convert ugoira file to gif. The default encoding settings is lossy encoding but moderate quality with smaller file size.
Required `createUgoira = True` and ffmpeg executeable.
- createapng

Set to True to convert ugoira file to animated png.
Set to True to convert ugoira file to animated png. The default encoding settings is lossless encoding but very large file size.
Required `createUgoira = True` and ffmpeg executeable.
- deleteugoira

Set to True to delete original ugoira after conversion.
- createwebm

Set to True to create webm file (video format).
Required `createUgoira = True` and ffmpeg executeable.
- createwebp
- deleteZipFile

Set to True to create webp file (image format).
Required `createUgoira = True` and ffmpeg executeable.
If set to `True`, it will delete the zip files from ugoira.
Only active if `createUgoira = True`.

## [Filename]
- filenameformat
Expand Down Expand Up @@ -703,6 +707,9 @@ Please refer run with `--help` for latest information.

For sanitizing filenames with custom rules. Supports regular expressions.
For detailed syntax, please refer to 'Bad chars' section.
- customCleanUpRe

TODO.

# Filename Format Syntax
Available for filenameFormat, filenameMangaFormat, avatarNameFormat, filenameInfoFormat,
Expand Down