Meet Parltrack 2.0

For the last year and thanks to the grant by the Dutch SIDN Foundation, Parltrack has been the object of a complete re-writing. We are proud to introduce Parltrack 2.0 to the world!

Browse our data by meps or by dossiers.

The general objectives of Parltrack remain unchanged: To dispose of a comprehensive, easily reusable data-set from the European Parliament, to allow for many different views over its members and their activites, but also to enable the tracking of changes over time.

Our main hope is that through its scraped and aggregated data, Parltrack 2.0 will be instrumental to enable many other projects aiming at shedding transparency and scrutiny over the Parliament.

From a technological perspective, the re-writing of Parltrack aims at achieving the main following objectives:

• More robust code that would accomodate from the many quirks and oddities found over the years in the data from the European Parliament.

• Better structured data, so more operations can be easily performed (including the tracking of changes over time).

• Future-proofing Parltrack so it can easily be adapted whenever the EP website changes (again) its design and/or format.

A pile-load of new features!

Here is a non exhaustive list of Parltrack 2.0 new features, including the ones currently under development:

Totally rewritten scraper orchestrator.

Parltrack can now scrape in parallel, and handle certain events during scraping (like something is failing hard: EP website is down, or the design changed again, or some other fault). Parltrack can now trigger various actions at other events (like sending out notifications, or rebuilding some static pages)

Brand new MEPs and Dossiers scrapers.

They should be much more robust and also much better at detecting that something is wrong (inconsistent data, etc.).

A much faster (non-)database backend.

We used to have mongoDB, we tried (hard and decided to not use it) to migrate to PostgreSQL, and now have something completely uncommon: Parltrack just loads all the data in RAM and uses Python dictionaries and Python code to "query" the data. according to our tests this is significantly faster and more economic than using mongoDB.

An improved search function.

Parltrack now allows to search for MEPs and parliamentary dossiers, and soon also in the tabled amendments.

A new, flexible notification system.

Parltrack 2.0 will allow visitors or groups of interest to subscribe to EP subjects, keywords in dossier titles, specific dossiers and MEPs.

A todo-list for the near future...

Before the end of the year, Parltrack should include the following planned features:

• A better viewer for the evolution of various versions of the data over time.

• Scraping of roll-call votes in opinion committees.

• Records of attendance of committee meetings

Stay tuned as the data-set will soon be updated, when the new Parliament enters in function, shortly after the May 2019 elections!

"We hope Parltrack 2.0 will enable a zillion uses and re-uses of the European Parliament data. We want, especially by allowing to follow the history over time of the MEPs and their activities, that they will feel the pressure of the scrutiny from their constituency(, and therefore deliver on their electoral promises)." concludes Sandro Niggurath, one of the many lead developers of Parltrack.

If you want to contribute, or for any inquiry, feel free to [contact the Parltrack development team].