Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release/0.17.1 #72

Merged
merged 4 commits into from
Dec 27, 2024
Merged

Release/0.17.1 #72

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
169a9ca
Fix incomplete ARD image link checking
emschu Dec 27, 2024
3570927
update changelog
emschu Dec 27, 2024
3ecf6b3
remove code smells
emschu Dec 27, 2024
d453e7e
update readme
emschu Dec 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Changelog oerc

## 0.17.1 - 2024/12/27
- Fix problem validating ard image urls

## 0.17.0 - 2024/12/26
- Golang 1.23, toolchain and dependency updates
- Changed fetching ARD program information using new ARD program and tv show API
Expand Down
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,12 +223,12 @@ to be available too, so ensure to enable the postgres database service as well.

### Channel list

**Channels:** ARD, ZDF, 3Sat, ARTE, ZDFInfo, ZDFNeo, Phoenix, KiKa, ARD One, Tagesschau24, ARD Alpha, SWR RP Fernsehen,WDR Fernsehen, SWR BW Fernsehen,SR Fernsehen, Radio Bremen TV, RBB Fernsehen, NDR Fernsehen, MDR Fernsehen, HR Fernsehen, BR Fernsehen, ORF eins, ORF 2, ORF III, ORF Sport +, SRF-1, SRF-zwei, SRF-info.
**Channels:** ARD, ZDF, 3Sat, ARTE, ZDFInfo, ZDFNeo, Phoenix, KiKa, ARD One, Tagesschau24, ARD Alpha, SWR, WDR, SR, Radio Bremen TV, RBB, NDR, MDR, HR, BR, ORF eins, ORF 2, ORF III, ORF Sport +, SRF-1, SRF-zwei, SRF-info.
### Time range import limits

| Channel family | Earliest date | Latest date |
|----------------|-----------------| --------------- |
| ARD | ~ 2010/01 | Today + 6 weeks |
|----------------|-----------------|-----------------|
| ARD | Today - 8 days | Today + 8 days |
| ZDF | 2015/03 | Today + 6 weeks |
| ORF | Today - 14 days | Today + 22 days |
| SRF | Today - 14 days | Today + 29 days |
Expand Down
69 changes: 35 additions & 34 deletions ard.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,45 @@
const (
ardHost = "programm-api.ard.de"
ardHostWithPrefix = "https://" + ardHost
ardMediaThekApiHost = "api.ardmediathek.de"

Check failure on line 34 in ard.go

View workflow job for this annotation

GitHub Actions / Code linting 

const ardMediaThekApiHost should be ardMediaThekAPIHost
ardMediaThekApiTvShowPath = "https://" + ardMediaThekApiHost + "/page-gateway/widgets/ard/editorials/"

Check failure on line 35 in ard.go

View workflow job for this annotation

GitHub Actions / Code linting 

const ardMediaThekApiTvShowPath should be ardMediaThekAPITvShowPath
ardDefaultImageWidth = "600"
)

var (
ardProgramURLMatcher = regexp.MustCompile(`^https:\/\/www\.ardmediathek\.de\/video\/.+`)
ardProgramPageMatcher = regexp.MustCompile(`^https:\/\/programm\-api\.ard\.de\/program\/api\/teaser\?teaserId=.+`)
ardImageLinkUrlMatcher = regexp.MustCompile(`^https:\/\/api\.ardmediathek\.de\/image-service\/image.+`)
ardProgramURLMatcher = regexp.MustCompile(`^https:\/\/www\.ardmediathek\.de\/video\/.+`)
ardProgramPageMatcher = regexp.MustCompile(`^https:\/\/programm\-api\.ard\.de\/program\/api\/teaser\?teaserId=.+`)
ardImageLinkMatcher = regexp.MustCompile(`^https:\/\/api\.ardmediathek\.de\/image-service\/image.+`)
ardImageLinkMatcher2 = regexp.MustCompile(`^https:\/\/programm-api.ard.de\/images\/.+`)
ardTvShowCategories = []string{
"a",
"b",
"c",
"d",
"e",
"f",
"g",
"h",
"i",
"j",
"k",
"l",
"m",
"n",
"o",
"p",
"q",
"r",
"s",
"t",
"u",
"v",
"w",
"x",
"y",
"z",
"#",
}
)

// ARDParser struct of ard parser code
Expand Down Expand Up @@ -144,7 +174,7 @@

if len(programEntry.ImageLinks) > 0 {
for _, img := range programEntry.ImageLinks {
if !ardImageLinkUrlMatcher.MatchString(img.URL) {
if !ardImageLinkMatcher.MatchString(img.URL) && !ardImageLinkMatcher2.MatchString(img.URL) {
appLog(fmt.Sprintf("Found invalid image link '%s' for program entry with hash '%s'. Skipping.", img.URL, programEntry.Hash))
atomic.AddUint64(&status.TotalSkippedPE, 1)
continue
Expand Down Expand Up @@ -186,7 +216,7 @@
return flattenedProgramItems, false
}

func getArdApiResponseForDailyProgramByChannel[T any](url string) (*T, error) {

Check failure on line 219 in ard.go

View workflow job for this annotation

GitHub Actions / Code linting 

func getArdApiResponseForDailyProgramByChannel should be getArdAPIResponseForDailyProgramByChannel
headers := map[string]string{}
resp, err := doGetRequest(url, headers, 3)
if resp == nil || err != nil {
Expand Down Expand Up @@ -236,37 +266,8 @@
}

// build set of urls to fetch tv shows from
var tvShowCategories = []string{
"a",
"b",
"c",
"d",
"e",
"f",
"g",
"h",
"i",
"j",
"k",
"l",
"m",
"n",
"o",
"p",
"q",
"r",
"s",
"t",
"u",
"v",
"w",
"x",
"y",
"z",
"#",
}
var tvShowApiURLs = make([]string, 0)
for _, category := range tvShowCategories {
for _, category := range ardTvShowCategories {
categoryString := strings.TrimSuffix(base64.StdEncoding.EncodeToString([]byte("ARD."+category)), "=")
tvShowApiURLs = append(tvShowApiURLs, fmt.Sprintf("%s%s", ardMediaThekApiTvShowPath, categoryString))
}
Expand Down
6 changes: 5 additions & 1 deletion server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,11 @@ func initRouter() *gin.Engine {
}
} else {
// trust no proxies - except the user specified one
r.SetTrustedProxies(nil)
err := r.SetTrustedProxies(nil)
if err != nil {
log.Fatal(fmt.Sprint("Problem to trust nil proxy url"))
return nil
}
}
r.Use(gin.Recovery())

Expand Down
Loading