Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry only for transient error codes #950

Merged
merged 1 commit into from
Jan 16, 2025
Merged

Retry only for transient error codes #950

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/autolabel/transforms/webpage_transform.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

logger = logging.getLogger(__name__)

MAX_RETRIES = 5
MAX_RETRIES = 3
BACKOFF = 2
DEFAULT_TIMEOUT = 60000 # in milliseconds
PREMIUM_PROXY_PARAM = "premium_proxy"
Expand All @@ -36,20 +36,20 @@


class WebpageTransform(BaseTransform):
COLUMN_NAMES = [
"content_column",
]

Check failure on line 41 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (RUF012) 

src/autolabel/transforms/webpage_transform.py:39:20: RUF012 Mutable class attributes should be annotated with `typing.ClassVar`

def __init__(

Check failure on line 43 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (PLR0913) 

src/autolabel/transforms/webpage_transform.py:43:9: PLR0913 Too many arguments in function definition (8 > 5)
self,
cache: BaseCache,
output_columns: Dict[str, Any],

Check failure on line 46 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (FA100) 

src/autolabel/transforms/webpage_transform.py:46:25: FA100 Add `from __future__ import annotations` to simplify `typing.Dict`
url_column: str,
timeout: int = DEFAULT_TIMEOUT,

Check failure on line 48 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (ARG002) 

src/autolabel/transforms/webpage_transform.py:48:9: ARG002 Unused method argument: `timeout`
scrapingbee_api_key: str = None,

Check failure on line 49 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (RUF013) 

src/autolabel/transforms/webpage_transform.py:49:30: RUF013 PEP 484 prohibits implicit `Optional`
max_retries: int = MAX_RETRIES,
v2_enabled: bool = False,

Check failure on line 51 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (FBT001) 

src/autolabel/transforms/webpage_transform.py:51:9: FBT001 Boolean-typed positional argument in function definition

Check failure on line 51 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (FBT002) 

src/autolabel/transforms/webpage_transform.py:51:9: FBT002 Boolean default positional argument in function definition
v2_api_key: str = None,

Check failure on line 52 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (RUF013) 

src/autolabel/transforms/webpage_transform.py:52:21: RUF013 PEP 484 prohibits implicit `Optional`
) -> None:
super().__init__(cache, output_columns)
self.url_column = url_column
Expand All @@ -74,8 +74,8 @@
headers = {"Authorization": f"Bearer {self.v2_api_key}"}

url = f"https://r.jina.ai/{url}"
response = requests.get(url, headers=headers)

Check failure on line 77 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (S113) 

src/autolabel/transforms/webpage_transform.py:77:24: S113 Probable use of `requests` call without timeout
return response.text

Check failure on line 78 in src/autolabel/transforms/webpage_transform.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (TRY300) 

src/autolabel/transforms/webpage_transform.py:78:13: TRY300 Consider moving this statement to an `else` block
except Exception as e:
logger.warning(f"Error fetching content from URL: {url}. Exception: {e}")
raise TransformError(
Expand Down Expand Up @@ -109,7 +109,7 @@
return text
except Exception as e:
logger.warning(f"Error fetching content from URL: {url}. Exception: {e}")
if response.status_code in [408, 425, 429, 500, 502, 503, 504]:
if response.status_code in [408, 429, 504]:
await asyncio.sleep(BACKOFF**retry_count)
return await self._load_url(url, retry_count=retry_count + 1)

Expand Down
Loading