adds license
[west_allis_property_tax] / README.txt
1 Docker containers you will need:
2 - selenium server for headless scraping (https://github.com/SeleniumHQ/docker-selenium)
3
4
5 -------------------------------------------
6 Run West Allis
7 -------------------------------------------
8 ./main scrape --driver-type='REMOTE' --driver-url='http://localhost:4444/wd/hub' --endpoint='WEST_ALLIS' --tax-ids-filepath='/home/jake/Code/oldkent/west_allis_property_tax/single_family_ids.txt' --current-index-filepath='/home/jake/Code/oldkent/west_allis_property_tax/index' --seconds-between-requests='23'
9
10 The --tax-ids-filepath is a text file containing the tax ids that you're looking to scrape
11 The index text file contains the current index to grab from the --tax-ids-filepath
12
13 # export DRIVER_TYPE='LOCAL'
14 export DRIVER_TYPE='REMOTE'
15 export DRIVER_URL='http://localhost:4444/wd/hub'
16 export ENDPOINT='WEST_ALLIS'
17 export TAX_IDS_FILEPATH='/home/jake/Code/oldkent/west_allis_property_tax/single_family_ids.txt'
18 export CURRENT_INDEX_FILEPATH='/home/jake/Code/oldkent/west_allis_property_tax/index'
19 export SECONDS_BETWEEN_REQUESTS='23'
20
21
22 -------------------------------------------
23 Run West Allis Sales History
24 -------------------------------------------
25 ./main scrape --driver-type=REMOTE --driver-url='http://localhost:4444/wd/hub' --endpoint='WEST_ALLIS_SALES_HISTORY' --tax-ids-filepath='/home/jake/Code/oldkent/data/all_ids.txt' --current-index-filepath='/home/jake/Code/oldkent/data/index.txt' --seconds-between-requests=7 --output-directory='/home/jake/Code/oldkent/data/sales_history_out'
26 ./main json --endpoint='WEST_ALLIS_SALES_HISTORY' --html-directory='/home/jake/Code/oldkent/data/sales_history_out'
27 ./main csv --endpoint='WEST_ALLIS_SALES_HISTORY' --json-directory='/home/jake/Code/oldkent/data/sales_history_out'
28
29
30 -------------------------------------------
31 Test
32 -------------------------------------------
33
34 Run tests with:
35 ENDPOINT='test' python -m unittest discover
36
37 Run coverage report with:
38 coverage report -m --omit='tests/**/*,venv/**/*,tests/*.py'
39
40
41
42
43 -------------------------------------------
44 Scraper Todo List
45 -------------------------------------------
46 https://www.auction.com/residential/WI/Milwaukee-county/active_lt/resi_sort_v2_st/y_nbs/
47 https://www.hubzu.com/searchResult/state/1386027/wi?searchBy=Wisconsin&srchBtnClk=1&view=list
48 https://www.homepath.com/listings/milwaukee-county-wi
49 https://www.hudhomestore.com/Listing/PropertySearchResult.aspx?zipCode=&city=&county=&sState=WI&fromPrice=0&toPrice=0&fcaseNumber=&bed=0&bath=0&street=&buyerType=0&specialProgram=&Status=0&indoorAmenities=&outdoorAmenities=&housingType=&stories=&parking=&propertyAge=&sLanguage=ENGLISH
50
51 https://www.homesteps.com
52 https://www.williamsauction.com/government-bank-owned
53