fix: use curl_cffi to bypass Cloudflare TLS fingerprinting#1
Open
gimmelovej wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix Cloudflare 403 blocks by replacing urllib with curl_cffi
Problem
The current implementation uses Python's built-in
urllibto make requests tox.com. While functional in theory,urllibproduces a TLS handshake fingerprint that is easily identifiable as non-browser traffic. Cloudflare's bot detection system flags this fingerprint and responds with a403 Forbiddenerror before the request even reaches X's API — regardless of whether valid cookies (auth_token,ct0) are provided.This makes the tool effectively unusable for many users, even when they supply a fully authenticated browser session.
Error observed:
Root Cause
Cloudflare performs TLS fingerprinting (also known as JA3/JA4 fingerprinting) on incoming connections. Python's
urlliband evenrequestspresent a TLS ClientHello that differs significantly from what real browsers (Chrome, Firefox) send — in cipher suites, extensions order, and other handshake parameters. Cloudflare uses this signal to classify the connection as automated traffic and blocks it at the edge, before any cookie or authentication header is evaluated.Solution
This PR replaces
urllibwithcurl_cffi, a Python library that wrapslibcurlwith support for browser TLS impersonation. By settingimpersonate="chrome124", the library produces a TLS handshake indistinguishable from Chrome 124, allowing requests to pass through Cloudflare's bot detection layer.All existing behavior is preserved:
--cookie,--cookie-file)ct0) extractionThe only structural change is that
urllib.requestcalls are replaced by a persistentcurl_cffi.requests.Session, which also improves performance via connection reuse.Changes
xcompare.py: replacedurllib-based HTTP calls withcurl_cffi.requests.Session(impersonate="chrome124")curl_cffi(available on PyPI)New Requirement
Users will need to install
curl_cffibefore running the tool:It is recommended to add this to the README under Prerequisites.
Testing
Tested against two public X accounts that previously returned
403 Forbiddenwith the originalurllibimplementation. After this change, both accounts resolved correctly and follower data was fetched successfully.Notes
curl_cffiis actively maintained and supports Windows, macOS, and Linuxchrome131) if future Cloudflare rules require a newer fingerprintSo much depends upon a red wheelbarrow