calafou / uzta

uzta

fork from github.com/maxigas/uzta

1
 

uzta

Lorea scraper. Lorea means flower in Basque, Uzta means harvest.

Uzta is the missing export module that enables users to save their data from the online platform to their hard disk. It accepts a website address, a username and a password. It yields a collection of plain text files or as a zip file.

Demo

Sample run of uzta:

https://vimeo.com/141580679

Installation

Option 1: The pip way

Install system dependencies

Tested on Debian stable and testing:

sudo apt install git python3 python3-pip python3-dev

Clone the repo and enter the dir

git clone https://gitlab.com/calafou/uzta.git
cd uzta

Install script dependencies

sudo pip3 install -r requirements.txt

Option 2: The Debian way

Install the dependencies

Tested on debian stable and testing:

sudo apt install git python3 python3-joblib python3-bs4 python3-html5lib python3-requests python3-yaml

Clone the repo and enter the dir

git clone https://gitlab.com/calafou/uzta.git

cd uzta

Usage

Option 1: Harvest the seed to a directory

./uzta.py yourusername yourpassword https://the.lorea.seed/

Output arrives to the output directory.

Option 2: Harvest the seed to a zip file

./uzta.py -z yourusername yourpassword https://the.lorea.seed/

Output arrives to uzta.zip.

Detailed help

Running the uzta script without any options prints a detailed help message.

The same message is printed as a result of the -h or –help switch:

./uzta --help

usage: uzta.py [-h] [-o OUTPUT] [-z] [-d ZIPOUT] [-j JOBS] [-t] username password website

positional arguments:
  username              Your username
  password              Your password
  website               The full URL of the lorea seed you want to harvest.

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        The directory name to store the data output. If
                        doesn't exist, it will be created. Default is
                        'output'.
  -z, --zip             The result of the harvesting will be an uzta.zip file.
                        The output directory will be treated as a temp dir and
                        will be permanently deleted!
  -d ZIPOUT, --zipout ZIPOUT
                        The directory name to store the zip file. If doesn't
                        exist, it will be created. Default is 'output'.
  -j JOBS, --jobs JOBS  Number of parallel CPU cores to use. Default is all
                        CPUs.
  -t, --debug           Debug mode. Print traceback on errors. Useful for
                        reporting bugs to developers. Default is False

Support

If you have comments or questions you can get help:

Option 1: Issue tracker

There is a public issue tracker that is part of the online repository:

https://gitlab.com/calafou/uzta/issues

Option 2: IRC

There is an IRC channel where you can talk about uzta with other users and developers:

irc://irc.freenode.net/uzta

You can use an IRC client software to connect.

Alternatively, there is a web interface:

https://webchat.freenode.net/

Option 3: Email

You can write an email to the authors:

efkin (a) riseup net or maxigas (a) anargeek net

Please include “uzta” in the subject line of the email.

Frequently Asked Questions

1. Which particular Lorea seeds are supported by uzta?

In theory uzta should work with any Lorea seed (e.g. any website powered by the Lorea software). In practice it is implemented as a screen scraper which makes it vulnerable to small variations in data of users, on the particular versions of the Elgg library and the Lorea software used on the website, etc. Please feel free to report bugs when you run into difficulties (see next question).

uzta is being tested with some user accounts on the following websites:

Drop us a line if you know about more seeds where we can test uzta.

2. How to report bugs?

Run uzta with the --debug option and send us the output, using one of the three options given in the Support section above. Preferably, open an issue on the public bug tracker. Please be prepared to test again once we think that we addressed your issue. Thanks for helping to develop uzta.

Credits

The development of this software was supported by the Cooperativa Integral Catalana.