News Direct Wales

Scrapey and Archie Bots: The future of political monitoring

bot development | web app design

Scrapey displayed on an iPhone.
Newsdirect logo
Wasting time visiting websites

To give accurate updates from the political landscape, the Newsdirect Wales team crawled Welsh Gov, Senedd, and party websites constantly, looking for new stories and blogs. The entire process was tedious and manual, and it was hard to actively report on client mentions.

Have a similar challenge?
Implementing Automation

The first step was to get a real strong idea of how the team worked, and analyse what parts of the process could be automated. Then we got to building the first version of Scrapey, a bot that monitors and scrapes websites 5 times a day for new content, compiling the new stuff into an accessible format for the team. Now, the team had the latest content delivered right to them, exactly when they needed it. Zing!

Archie the Assembly Search Engine

Without the required methods in place within the Assembly, Newsdirect Wales needed a search engine to look for mentions of a particular keyword in the Assembly contribution transcripts. The Archie engine worked hard to pull in all transcripts from the last three years and break them into contribution content and who made the contribution. With a database with over 240,000 contributions, the team can use it to report directly to their customers.

Newsdirect Wales are doing some amazing work and we’re really excited to be working with them to build the next big thing in political monitoring.