Skip to main content

​
Download Python SDK

Download the Python SDK. 
pip install webtranspose
Not a Python user? Use Web Transpose via API.

​
Get your API Key

Get your API key from the Web Transpose Dashboard. 
os.environ['WEBTRANSPOSE_API_KEY'] = "YOUR API KEY"

​
Crawl a Website

Your crawls can be viewed and monitored in the Web Transpose Crawls Dashboard.

Crawl a Website (3 lines of code)

import webtranspose as webt

crawl = webt.Crawl(
"https://www.example.com",
max_pages=100,
render_js=True,
api_key="YOUR_WEBTRANSPOSE_API_KEY", # optional, if you want to run on cloud
)
await crawl.crawl() # crawl.queue_crawl() for async

# download to local disk
crawl.download()

# get page data
page_urls = crawl.get_visited()
for x in page_urls:
print(crawl.get_page(x))

​
Scrape a Website

Your scrapers can be viewed and monitored in the Web Transpose Scraper Dashboard.

Build a Scraper (1 line of code)

import webtranspose as webt

schema = {
"Merchant Name": "string",
"Title of Product": "string",
"Product Photo URL": "string",
}

scraper = webt.Scraper(
schema, 
render_js=True, 
api_key="YOUR_WEBTRANSPOSE_API_KEY", # optional, if you want to run on cloud
)

out_json = scraper.scrape("https://www.example.com")

​
SERP API

Run the SERP API doing the following:

Regular web search.

import webtranspose as webt

results = webt.search_filter("Paul Graham's Blog")
# result.keys()
# \['results', 'filtered_results']
⌘I