Rental Listings ETL

A streamlined and efficient ETL pipeline for online rental listing data.


This project gathers rental listing data from the Rentfaster.ca API. A custom data engineering process cleans the data and stores it in the appropriate format for analytical queries.

The data collected by this service forms the foundation for The Calgary Project.

Prerequisites

Dependencies can be installed via:

pip install requirements.txt

Tests

Testing for this project is handled by pytest.

Testing the database connection & API status

In the project directory, type:

$ pytest tests/test.py

All tests should pass before running app.py.

Built With

Code

• Requests
  • HTTP library
• Pandas
  • Python data analysis library
• mysql-connector-python
  • MySQL driver written in python
• Schedule
  • Python job scheduling for humans.
• pytest
  • Testing library

Services

• Microsoft Azure
  • Cloud service
• Docker
  • Virtualization

Contributing

Feedback and constructive criticism is more than welcome!

Author

• Andrew Moss
  • Creator
    • agmoss

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Acknowledgments

• Rentfaster.ca