Just some minor web crawlers.
Pull requests are always welcome.
- Required: License and copyright notice + State Changes + Disclose Source
- Permitted: Commercial Use + Modification + Distribution
- Forbidden: Hold Liable + Sublicensing
Get bill data from the National Assembly and structurize to json formats. (See attributes)
pip install -U celery-with-redis # Install dependencies
cd bills
cp settings.py.sample settings.py # Input data directory
python main.py
Get Korean politicians' data from Korea Election Commission (중앙선거관리위원회).
This data contains the list of all people that have run for office in the National Asssmbly.
cd election_commission
python main.py
Get and merge data for POPONG Glossary from:
committee: Standing committee and Special Committee (국회상임위원회 및 특별위원회),
likms: Integrated Legislation Knowledge Management System (입법통합지식관리시스템),
nas: National Assembly Secretaritat (국회사무처).
python get.py # To get source data files
python merge.py # To create glossary.csv
Get Google search counts.
cd google
python ndocs.py
Get member information from the Korean National Assembly.
cd national_assembly
python crawl.py
Get People Power 21 (열려라국회) webpages. (Currently broken)
cd peoplepower
scrapy crawl peoplepower21
Get pledges from NEC (선거관리위원회) for 19th National Assembly officials.
cd pledges
python crawler.py
Get Korean politicians' data from ROKPS(헌정회).
cd rokps
python crawler.py
python parser.py
Get Korean lastnames from Wikipedia.
cd wikipedia
python wiki_lastnames.py
Get Wikipedia links for assembly members.
cd wikipedia
python assembly_members.py